Protein Ranking

Rank 453 variants of human SOD1 by how well each variant pairs with wild-type SOD1.

Human SOD1

Inside the dimer

PDB 2C9V ↗
PyMOL cartoon of the two chains in human SOD1 assembly 2C9V
Chain AChain FSelected siteWild-type crystal structure
Lysine → Valine

Predicted dimerization

Candidates arrive by ID · ranked highest first
Incoming variant

    The marker locates the changed residue on wild-type chain A. The benchmark counts the initial methionine as position 1.

    Task & tools

    SOD1 variant ranking

    Prioritize 453 single-amino-acid variants of human SOD1 for experimental characterization. Predict how each variant changes a measured protein property, then rank all variants from highest to lowest predicted assay score.

    Dimerization

    Predict association with wild-type SOD1 in a binding protein-fragment complementation assay (bPCA). Its growth readout depends on both folded abundance and association.

    sod1-dimerization-n453-p01
    Biological scope
    One dimerization assay and 453 candidate sequences.
    Agent input
    Assay instructions, wild-type sequence and anonymous IDs with full variant sequences.
    Required output
    One complete ranking of all 453 IDs for the dimerization assay.
    Available tools
    No tools, code execution, internet, retrieval or experimental training labels.
    Capability tested
    Sequence-to-function prediction, interpreting assay context, prioritizing variants and completing a large structured response.
    Agent Context
    instruction.mdDownload
    Open Agent Context to read the files.

    Protocol

    1. Read the assay description and identify the property being predicted.
    2. Compare the supplied variant sequences with the wild-type sequence.
    3. Order every candidate ID by its predicted normalized experimental score, highest first.
    4. Return exactly {"ranking": ["P001", "P002", "…"]}, with every supplied ID once. The harness saves the response as answer.json.

    Download dimerization candidate sequences

    Rubric

    Rank agreement with experimental measurements

    The independent verifier compares the submitted order with the held-out yeast assay scores using Spearman rank correlation (ρ). Higher correlation means better agreement.

    Reward = max(0, Spearman ρ)

    How a ranking earns its score

    Predicted orderMeasured order
    Spearman ρ
    0.80
    Reward
    0.80

    Two variants trade places, lowering agreement slightly.

    Four illustrative variants, ordered highest to lowest. The task scores all 453.
    • Complete ranking: include all 453 candidate IDs exactly once. Missing, duplicate or unknown IDs make the submission invalid and give reward zero.
    • Valid ranking: preserve the measured ρ. Negative correlation is clipped to zero reward while remaining visible in the diagnostics.
    • Delivery criterion: continuous reward and ranking validity. The package’s ρ ≥ 0.80 field is optional reporting metadata, not a delivery gate.

    No top-k gate or prose judge is added. Reward is rank agreement, not accuracy. A valid random ranking has expected ρ near zero; clipping can give a small positive reward.

    Exact delivery acceptance criteriaDownload ACCEPTANCE.md
    Open to load the delivered acceptance criteria.

    Runs & evidence

    How well did the predicted orders agree?

    Dimerization reward Three fresh rankings per model

    Fresh v1.3.0 model results

    Loading the recorded cohort…

    Fresh SOD1 model results by assay and model configuration
    AssayModel / effortMean rewardSample SDRangeValid / returned

    Three fresh generations per model. All returned answers enter each average, including invalid rankings scored zero. Sample SD and range describe these repeats; no inferential confidence interval or pass@k is claimed.

    Frozen specialist baseline

    ESM-2 embeddings with a trained ridge predictor supply one frozen dimerization ranking, regraded with the current formula. This comparator used labeled training data; the general-purpose models received no assay training labels. It is a separate, supervised condition.

    AssayBaselineReward / ρObservations
    Individual attempts & grading evidence

    Raw ρ distinguishes a valid, negatively correlated answer from an incomplete ranking. Native result, request and replay hashes are included in the downloadable record; full raw answers stay in the reviewer package.

    TrialAssay / modelRaw ρRewardEvidence
    Run configuration & resource accounting
    Model configurations
    us.openai.gpt-6-astra · max; us.xai.grok-4.7 · high.
    Harness
    Bedrock no-tools host adapter 1.1.0, Harbor on EC2, one provider request per attempt, concurrency four, no automatic retries.
    Limits
    128,000 output tokens including reasoning; 1,800-second agent timeout.
    Resource evidence
    Recorded input/output tokens are included per attempt. Actual billed cost and compute/memory measurements are unavailable in this export.

    Download scores, settings & source hashes

    QC & findings

    Original package-wide delivery checks

    Loading the original validation receipt…

    Delivery checkRecorded outcomeEvidence and remaining scope
    Q1B · Benchmark evidencePartial · audit missingThe default cross-model SD criterion passed for both assays. A complete corpus contamination audit remains missing.
    Q2 · Scientific groundingPass · supplied provenance453 sequences and 906 labels checked against the pinned source. Intended data use is licensed according to owner attestation.
    Q3 · Grading contractPass · replay evidence44 controls and 28 indexed saved-answer replays. All 12 fresh native rewards agree with independent replay.
    Q4B · Rubric auditPartial · audit missingDirect permutation, reverse, shuffle, invalid-output and known-ρ controls passed. Formal M12/M13 audit evidence is missing.
    Q5 · Training retentionNot applicableStatic benchmark delivery; no model training or update.
    Q6 · Execution architecturePass · observed delivery12/12 current requests returned with no provider or infrastructure failures; isolation receipts passed. This file delivery does not establish a hosted-service SLA.

    Observed model-output failures

    • One Grok dimerization answer omitted P339: 452 of 453 IDs, invalid, reward zero.
    • Another Grok dimerization answer was complete, with ρ approximately −0.001666: valid ranking, clipped reward zero.

    These are model-output outcomes and remain in the model means. The original six-question report is distinct from the repository’s Step 5 and Step 6A/6B release workflow. The delivery retains missing corpus and formal rubric audit evidence.

    Exact delivery QC report · v1.3.0-rc.4Download QC_REPORT.md
    Open to load the original QC report.

    Sources & limits

    Structure and assay images

    The explorer uses Strange et al.’s 1.07 Å X-ray structure of human SOD1. The space-filling model draws atoms from assembly 1 at their van der Waals radii. Teal and violet distinguish chains A and F. Gold marks the selected residue on wild-type chain A. The model received sequence and assay text only. Structure record and coordinates · Structure paper

    Source: Quiroga et al., The folding, dimerization and allosteric landscapes of the ALS protein SOD1: a comprehensive mutational atlas, bioRxiv v1, July 16, 2026. The package uses measurements from the pinned author table.

    The dimerization assay uses yeast growth as a proxy for association with wild-type SOD1. They do not directly measure clinical severity, neuronal survival or purified binding affinity.

    The package records owner-confirmed licensing for intended use. Source assay normalization is consumed rather than independently reconstructed. Publication timing does not certify absence from private training corpora.

    Original SOD1 reviewer package · GitHub access required

    The full archive contains experimental labels and protected answers and stays outside the website. Historical model attempts are excluded from fresh model means; only the explicitly marked frozen specialist baseline is reused.