Protein Ranking
Rank 453 variants of human SOD1 by how well each variant pairs with wild-type SOD1.
Inside the dimer
The marker locates the changed residue on wild-type chain A. The benchmark counts the initial methionine as position 1.
Task & tools
SOD1 variant ranking
Prioritize 453 single-amino-acid variants of human SOD1 for experimental characterization. Predict how each variant changes a measured protein property, then rank all variants from highest to lowest predicted assay score.
Dimerization
Predict association with wild-type SOD1 in a binding protein-fragment complementation assay (bPCA). Its growth readout depends on both folded abundance and association.
sod1-dimerization-n453-p01- Biological scope
- One dimerization assay and 453 candidate sequences.
- Agent input
- Assay instructions, wild-type sequence and anonymous IDs with full variant sequences.
- Required output
- One complete ranking of all 453 IDs for the dimerization assay.
- Available tools
- No tools, code execution, internet, retrieval or experimental training labels.
- Capability tested
- Sequence-to-function prediction, interpreting assay context, prioritizing variants and completing a large structured response.
Agent Context
Open Agent Context to read the files.
Protocol
- Read the assay description and identify the property being predicted.
- Compare the supplied variant sequences with the wild-type sequence.
- Order every candidate ID by its predicted normalized experimental score, highest first.
- Return exactly
{"ranking": ["P001", "P002", "…"]}, with every supplied ID once. The harness saves the response asanswer.json.
Rubric
Rank agreement with experimental measurements
The independent verifier compares the submitted order with the held-out yeast assay scores using Spearman rank correlation (ρ). Higher correlation means better agreement.
Reward = max(0, Spearman ρ)
How a ranking earns its score
- Spearman ρ
- 0.80
- Reward
- 0.80
Two variants trade places, lowering agreement slightly.
- Complete ranking: include all 453 candidate IDs exactly once. Missing, duplicate or unknown IDs make the submission invalid and give reward zero.
- Valid ranking: preserve the measured ρ. Negative correlation is clipped to zero reward while remaining visible in the diagnostics.
- Delivery criterion: continuous reward and ranking validity. The package’s ρ ≥ 0.80 field is optional reporting metadata, not a delivery gate.
No top-k gate or prose judge is added. Reward is rank agreement, not accuracy. A valid random ranking has expected ρ near zero; clipping can give a small positive reward.
Exact delivery acceptance criteria
Download ACCEPTANCE.mdOpen to load the delivered acceptance criteria.
Runs & evidence
How well did the predicted orders agree?
Fresh v1.3.0 model results
Loading the recorded cohort…
| Assay | Model / effort | Mean reward | Sample SD | Range | Valid / returned |
|---|
Three fresh generations per model. All returned answers enter each average, including invalid rankings scored zero. Sample SD and range describe these repeats; no inferential confidence interval or pass@k is claimed.
Frozen specialist baseline
ESM-2 embeddings with a trained ridge predictor supply one frozen dimerization ranking, regraded with the current formula. This comparator used labeled training data; the general-purpose models received no assay training labels. It is a separate, supervised condition.
| Assay | Baseline | Reward / ρ | Observations |
|---|
Individual attempts & grading evidence
Raw ρ distinguishes a valid, negatively correlated answer from an incomplete ranking. Native result, request and replay hashes are included in the downloadable record; full raw answers stay in the reviewer package.
| Trial | Assay / model | Raw ρ | Reward | Evidence |
|---|
Run configuration & resource accounting
- Model configurations
us.openai.gpt-6-astra· max;us.xai.grok-4.7· high.- Harness
- Bedrock no-tools host adapter 1.1.0, Harbor on EC2, one provider request per attempt, concurrency four, no automatic retries.
- Limits
- 128,000 output tokens including reasoning; 1,800-second agent timeout.
- Resource evidence
- Recorded input/output tokens are included per attempt. Actual billed cost and compute/memory measurements are unavailable in this export.
QC & findings
Original package-wide delivery checks
Loading the original validation receipt…
| Delivery check | Recorded outcome | Evidence and remaining scope |
|---|---|---|
| Q1B · Benchmark evidence | Partial · audit missing | The default cross-model SD criterion passed for both assays. A complete corpus contamination audit remains missing. |
| Q2 · Scientific grounding | Pass · supplied provenance | 453 sequences and 906 labels checked against the pinned source. Intended data use is licensed according to owner attestation. |
| Q3 · Grading contract | Pass · replay evidence | 44 controls and 28 indexed saved-answer replays. All 12 fresh native rewards agree with independent replay. |
| Q4B · Rubric audit | Partial · audit missing | Direct permutation, reverse, shuffle, invalid-output and known-ρ controls passed. Formal M12/M13 audit evidence is missing. |
| Q5 · Training retention | Not applicable | Static benchmark delivery; no model training or update. |
| Q6 · Execution architecture | Pass · observed delivery | 12/12 current requests returned with no provider or infrastructure failures; isolation receipts passed. This file delivery does not establish a hosted-service SLA. |
Observed model-output failures
- One Grok dimerization answer omitted
P339: 452 of 453 IDs, invalid, reward zero. - Another Grok dimerization answer was complete, with ρ approximately −0.001666: valid ranking, clipped reward zero.
These are model-output outcomes and remain in the model means. The original six-question report is distinct from the repository’s Step 5 and Step 6A/6B release workflow. The delivery retains missing corpus and formal rubric audit evidence.
Sources & limits
Structure and assay images
The explorer uses Strange et al.’s 1.07 Å X-ray structure of human SOD1. The space-filling model draws atoms from assembly 1 at their van der Waals radii. Teal and violet distinguish chains A and F. Gold marks the selected residue on wild-type chain A. The model received sequence and assay text only. Structure record and coordinates · Structure paper
Source: Quiroga et al., The folding, dimerization and allosteric landscapes of the ALS protein SOD1: a comprehensive mutational atlas, bioRxiv v1, July 16, 2026. The package uses measurements from the pinned author table.
The dimerization assay uses yeast growth as a proxy for association with wild-type SOD1. They do not directly measure clinical severity, neuronal survival or purified binding affinity.
The package records owner-confirmed licensing for intended use. Source assay normalization is consumed rather than independently reconstructed. Publication timing does not certify absence from private training corpora.
Original SOD1 reviewer package · GitHub access required
The full archive contains experimental labels and protected answers and stays outside the website. Historical model attempts are excluded from fresh model means; only the explicitly marked frozen specialist baseline is reused.