Introducing VEP-bench: predicting variant effects with language models

VEP-bench v0.1

Early draft: This post was generated by an AI agent and has not yet been reviewed by the authors. We’re sharing it to invite early feedback; it may contain errors or overstate conclusions. We plan to replace it with an author-written version.

Can general-purpose language models predict the effects of genetic variants from DNA sequence and experimental context? VEP-bench tests this directly, without tools or internet access.

The first release covers fitness, gene expression, and splicing across 52 experimental panels and 2,600 variant appearances. Performance varies across these tasks, providing a view of model capabilities that an overall score alone cannot capture.

Dataset

Prediction tasks and questions

The three tasks measure different outcomes: functional effects in saturation genome editing (SGE), reporter activity in satMutMPRA, and exon inclusion in OpenSplice.

Each question supplies sequence and assay context for a panel of 50 variants. Models predict an effect for every variant. We score the predicted ordering against the measurements using Spearman correlation within each panel, then average panels equally within each task. The overall score is the equally weighted mean of the three task scores.

Example prompt: MSH6 exon 7

This is the complete prompt for MSH6 exon 7, the panel selected for our later response case study. The gene name identifies the example for readers; it is not included in the model's prompt.

Explore this question and model responses.

Dataset composition

Counts describe the selected benchmark panels. Each variant is counted once per panel; an allele appearing in two panels contributes twice.

Results

We evaluate nine models at their highest available reasoning effort, using complete runs from the September 18, 2026 snapshot. Effort selection does not depend on score. The publication note documents recovered responses and replacements; the published DeepSeek configuration includes selective token-limit retries.

This snapshot adds GPT-6 Astra at maximum effort, with 52 valid answers under a 128,000-token output cap. Its completed responses cost $37.54 and used 1,610,754 tokens, including input and generated output with reasoning. Three OpenSplice serving errors were retried with unchanged settings; their original attempts remain in provenance and are excluded from benchmark cost and token usage. None of Astra's retained responses reached the cap. The interrupted DeepSeek maximum-effort evaluation is documented on the leaderboard and is excluded from these comparisons.

The bar plot shows overall Spearman scores across all three tasks, displayed as percentages to match the leaderboard. GPT-6 Astra has the highest overall score (0.610), followed by Gemini 3.8 Flash (0.559) and GPT-5.6 Sol (0.558).

Performance across tasks

The radar plot compares fitness, expression, and splicing on the same 0–1 Spearman scale.

Astra max has the highest observed score on all three tasks. Its splicing score is close to Gemini's (0.773 versus 0.770). All nine models score highest on splicing and lowest on expression in this dataset.

Explore all models and reasoning efforts on the leaderboard.

Comparison with specialist models

We compare AlphaGenome's signed molecular predictions for splicing and expression, and AlphaGenome Variant Impact (AVI) scores for fitness, with saved LLM answers on the same variants within each original panel. Each panel has equal weight. The full benchmark leaderboard retains its original variant set.

Coverage is 1,000/1,000 splicing variants, 800/800 expression variants, and 487/800 fitness variants. The molecular predictions include indels; the AVI fitness comparison covers SNVs only. Each LLM is rescored on the same covered variants as the specialist.

Astra's fitness score is 0.611, compared with 0.586 for AVI, on the same 487 SNVs across 16 genes. Excluding AVI model-selection studies gives 0.589 versus 0.557 on 382 SNVs across 13 genes. These are higher observed scores on the matched SNV subsets; they do not establish an advantage on indels or statistically conclusive superiority.

Comparison with other benchmarks

The plot below compares overall VEP-bench scores with the Artificial Analysis Intelligence Index v4.3.

Comparison methodology and sources.

Assay dates and model knowledge cutoffs

Correction — September 16, 2026. Our original 9 before / 7 after split used recent MaveDB releases without following their predecessors, other public repositories, or preprints. That overstated the number of later assays. The corrected split is 13 before / 2 after / 1 mixed, for all five models with known cutoffs in this snapshot. The figure, statistics and downloads below have been regenerated from the same frozen model outcomes using the corrected provenance. Model predictions and benchmark scores have not changed.

SGE's gene panels have different recorded public dates, so we can split them relative to each model's reported knowledge cutoff. The expression and splicing tasks each use a single assay date in this snapshot; they do not offer the same within-task split.

Each point below is the mean within-gene Spearman correlation, with equal weight per gene panel. Blue circles show panels before the cutoff; orange diamonds show panels after it. Counts are gene panels, not individual variants. Horizontal bars show 95% confidence intervals for each group mean. For each model we select its highest available reasoning effort among complete published SGE runs, breaking ties by the most recent run. Selection does not depend on performance. Models without a known cutoff are excluded.

The five cutoffs (February 16, March, and April 30, 2026) select the same genes. BARD1, CTCF, RAD51D and XRCC2 move to the earlier group. SFPQ is excluded from both means because its panel combines earlier and later evidence. The two remaining later panels are SBDS and TINF2; we verified no earlier scored release for either in this audit. This is a search result, not proof that they were absent from training data.

The audit also corrected older study dates using preprints and a BAP1 archived dataset. Individual IGVF file release dates determine availability: a file can be uploaded privately months before release, or added after its parent dataset became public.

The bars are symmetric 95% Student's t confidence intervals: mean ± t(0.975, n − 1) × s / √n, where s is the sample standard deviation of the gene-level scores in that date group. We compute them with SciPy's t distribution. The interval has exact 95% coverage at finite sample sizes when the gene scores are independent and identically normally distributed; it does not require a large-sample approximation under those assumptions. Normality is an approximation for these bounded correlation scores, so coverage here is approximate, especially with only two later panels. The intervals estimate uncertainty in the mean across genes, rather than variability across repeated model runs. We do not clip their bounds to the correlation range. A group with fewer than two genes or no variation has no estimated interval. Overlap between the two intervals is not the significance test; the p-value below tests the before-cutoff advantage directly.

The assay date records the earliest verified public scored-assay evidence covering the selected panel, including preprints, archived datasets and superseded releases. Earlier experimental scores qualify even if later revised; this comparison concerns prior assay evidence, not exact answer identity. For historical papers, study-level coverage is used and score identity across versions has not been established. A partial earlier release spanning the cutoff is Mixed availability and excluded from both means. These dates are evidence bounds, not demonstrated first indexing dates or proof of when a model saw data. For a cutoff specified only to the month, panels dated in that month are excluded because their ordering is unknown. An exact cutoff date includes that day in the “before” group. Missing or unmatched assay dates are also excluded.

A gap is descriptive, not a causal estimate of training-data exposure: the before and after groups contain different genes and experiments, with potentially different difficulty. Group membership can also change across models with different cutoffs. Completed format failures retain their published zero scores; negative correlations are retained. An empty group has no mean.

Does this model have a before-cutoff advantage?

The hypothesis is per model: does this model perform better on genes whose assays were available before its knowledge cutoff? We use SciPy's independent-sample permutation test with the statistic mean Spearman before minus mean Spearman after, a one-sided alternative (before greater than after), and exact enumeration. The gene panel is the independent unit. SciPy reassigns whole genes between the groups while keeping their sizes fixed; it counts allocations with a difference at least as large as observed. With 13 and 2 genes, all 105 allocations are included. This does not require normally distributed scores.

The test assumes independent genes whose scores are exchangeable between date groups under the null (the same score distribution). These genes and assays were not randomly assigned to publication dates, so the test describes an exploratory association. It cannot separate date from gene or assay differences. The reported p-values are unadjusted and interpreted separately for each model, with a per-model threshold of 0.05. A detected before-cutoff advantage would be consistent with source exposure, but would not prove overfitting. With only two later panels, a nonsignificant result cannot rule out overfitting or establish equivalence. Groups with fewer than two genes are not tested.

No model has a detected before-cutoff advantage at the per-model 0.05 threshold after this correction. The one-sided p-values are 0.781 for Luna, 0.695 for Terra, 0.429 for Sol, 0.619 for Astra, and 0.886 for Gemini. This small, uneven comparison provides little evidence about memorization; it should not be read as validation of a contamination-free evaluation.

The cutoff analysis is a saved snapshot generated with SciPy from the published run and outcome indexes. Its CSV downloads retain run IDs, question-set digests, cutoffs, and assay-date sources for the displayed comparison.

Do explanations match the measured biology?

For supplementary case studies, we selected MSH6 exon 7 and the LDLR promoter before inspecting model explanations because the source papers describe their regulatory structures. We asked Astra Max for a substantive biological justification grounded in the supplied sequence, followed by predictions for the same 50 variants per panel. The prompt added no measured effects or paper annotations. This is a deliberately chosen pair of examples, and each explanation is the model's stated justification rather than a verified account of its internal reasoning.

For MSH6, Astra correctly located the splice signals and reconstructed sequence changes, but a plausible mechanism did not always predict the measured effect:

The V19 discrepancy also exposes a limitation in our question. Astra explicitly warned that its rescue prediction depends on whether shifted splice products count toward inclusion. The paper's Methods classify expected inclusion and skipping sequences by exact matching. Our prompt does not specify that counting rule. We therefore cannot attribute the mismatch solely to biological reasoning; defining the assay's measured target precisely matters alongside supplying the sequence.

This pattern extends across the panel: the 12 alleles overlapping the acceptor average −86.4 measured ΔPSI, versus −18.2 predicted with explanation. All 11 interior alleles are within 1.2 points of zero, despite predictions as large as −16 or +8. Earlier branchpoint experiments support the upstream TTCAT motif, with partial skipping after substitutions and evidence compatible with alternative branchpoints. They do not validate the model's specific enhancer and silencer assignments inside the exon. The expanded MSH6 analysis maps all 50 alleles, reconstructs the proposed rescue sites and mutant donors, and distinguishes those earlier experiments from this reporter's measurements.

For LDLR, the established promoter architecture gives a more demanding test than recognizing isolated motifs. SatMutMPRA maps large activity losses to known regulatory elements, and GPN-Star Fig. 4D examines dependencies among regions labelled FP1, SREBP1, SREBP2, and SP1. The numbered labels refer to promoter repeats in the SatMutMPRA supplement; they do not establish separate occupancy by the SREBP-1 and SREBP-2 protein isoforms. Classic experiments support cooperating Sp1 sites flanking the SRE.

Astra identifies the canonical SRE and predicts losses for all five selected substitutions within it. It also recognizes the flanking C/T-rich sequences and proposes cooperation. But it misses FP1 and substantially underestimates losses in the cooperating Sp1-associated elements:

For FP1, all six sampled variants reduce activity; the mean measured effect is −2.08, versus Astra's −0.05. For V34, the known c.-142C>T allele, earlier experiments directly observed impaired Sp1 binding and reduced reporter activity. Creating a short ETS-like core does not establish compensation for losing a cooperating activator. GPN-Star's dependencies are model-derived and its MPRA track reuses the Kircher measurements; we use it as a reference for examining architecture, not as an independent experiment or a quantitative comparison with Astra.

The provider-exposed summary also names the LDL receptor promoter, so the sequence-focused instruction does not establish that recognition or recall was absent. This expanded element audit was performed after reading the responses; it leaves the original selection and all 50 scored alleles unchanged.

Each score uses all 50 variants. One completion per condition does not establish a benefit from requesting explanations: ordering improved slightly in both cases, while Pearson correlation decreased. The detailed responses are useful for locating unsupported assumptions and gaps in the task definition. These supplementary predictions remain separate from the baseline leaderboard.

The case-study analysis maps the literature to the exact sequence, compares all four LDLR elements, and plots every allele against both sets of predictions. The complete prompts and responses are readable and downloadable, with committed data for offline replay.

Conclusion

VEP-bench shows that general-purpose language models can recover part of the ordering of measured variant effects from sequence and experimental context. Astra max leads the selected configurations on all three tasks; the rankings below it vary by task, and performance also varies across variant classes. These differences make the task and subset results useful alongside the overall ranking.

The specialist comparisons and broader analyses help put these results in context, but the limited set of models and assay panels leaves substantial uncertainty. This release provides a public starting point for evaluating progress across fitness, expression, and splicing.

• Explore the leaderboard
• Task methodology
• Code on GitHub