VEP-bench

VEP-bench is an experimental benchmark of language models' native ability to predict genetic variant effects. Models answer without internet access or tools, and every response and deterministic score can be inspected. The benchmark has not yet been fully reviewed.

See our early blog draft for additional results, including a comparison with AlphaGenome.

Leaderboard

Each model appears in the bar chart at its highest available reasoning effort with complete results for this view. “All tasks” requires the same configuration across every task. Equal-effort configurations use the latest results; selection does not depend on score. Stopped and otherwise incomplete evaluations are listed under Unscored model attempts.

Scroll horizontally to see all models. Hover over a bar for cost, token usage, and knowledge cutoff.

Score by cost and token usage

Compare the selected task's score with benchmark cost and total token usage across all available reasoning efforts and configurations. Both plots share model colors and the score scale; each line connects configurations from the same family. Tokens include input and generated output, including reasoning. Both cost and tokens count completed responses, including completed truncations before selective retries, and exclude serving errors.

Unscored model attempts

An attempt is reported here benchmark-wide when refusals, content filtering, or an early stop leave the evaluation without a complete, rankable model result. These attempts remain visible regardless of the task selected above and are not included in the leaderboard. Recorded responses may have individual scores, but an incomplete evaluation has no overall score or rank.

DeepSeek V4.1 Flash (max), September 17, 2026. We stopped this evaluation to avoid further spend after 9 of 16 Fitness (SGE) responses exhausted the 384,000-token output allowance without a final answer; the other 7 were valid. Requests used DeepSeek's own provider route through OpenRouter, with maximum reasoning effort and provider fallback disabled. The allowance includes reasoning and final output and was the selected route's advertised maximum, rather than an OpenRouter-wide limit.

The 20 recorded responses include all 16 fitness questions, 3 of 16 expression questions, and 1 of 20 splicing questions, including the initial pilot. Four additional requests were interrupted in flight and have no recorded outcome; they are not counted as model failures. Completed response cost was $2.48. At least one truncated response degenerated into repeated single letters. The cause is unconfirmed, and these observations do not establish performance on the unanswered questions or on other providers or reasoning settings. No retries were made.

The saved responses are retained in the September 17 audit publication.

Output limits and usage

Across all reasoning efforts, only configurations with recorded formatting failures, refusals, output-limit stops, or retried failures in the selected tasks appear below. Configurations with no recorded failures are omitted; incomplete attempts remain in the section above.

• Valid answers follow the required answer format, regardless of prediction accuracy. For ranking tasks, the last well-formed FINAL: {JSON object} line must contain every candidate ID exactly once, with finite numeric predictions.
• Formatting failures have no usable final answer: for example, a missing FINAL: line, JSON split across lines, missing or duplicate candidate IDs, or nonnumeric predictions. These responses receive zero for both correlations and remain in the score average. A refusal or output-limit stop can also leave no usable answer, so this rate does not identify the underlying cause.
• Truncated counts responses the provider reports as stopped at an output limit, even when they contain a valid final answer. It is based on the reported stop reason, not a comparison of token usage with the configured ceiling. A model can therefore have 100% valid answers and a nonzero truncation rate; formatting failures and truncation can also overlap.

Output tokens include reasoning. Answer rates and output figures describe the scored responses. Benchmark cost and total tokens include input and output for completed model responses, including completed token-limit attempts before selective retries; serving errors are excluded. Retries keep recovered failures visible here. Selective retries replace every initially invalid truncated answer once at the larger listed limit; all retry outcomes, including failures, are retained. Other settings stay the same. Unavailable measurements appear as “—”.