K-Dense BYOK on CompBioBench: Three Models, One Harness

We ran CompBioBench v1 three times through K-Dense BYOK, once each with DeepSeek V4 Pro (r1), GPT-5.6 SOL (r2), and Gemini 3.8 Flash (r3), all called through OpenRouter. Everything else stayed the same: the prompt, the sandbox, the skills, the tools, the data files, and one isolated project per question.

CompBioBench is a Genentech benchmark of 100 computational-biology tasks, introduced by Nair, Gunsalus, and colleagues in Agentic systems are adept at solving well-scoped, verifiable problems in computational biology (bioRxiv, 2026). Every question asks the agent to fetch data, run a tool or pipeline, and report one number, which is what agents are being asked to do in computational biology today. CompBioBench keeps its answers held-out on the leaderboard and grades by exact match, so a score on it is via external verification. This post describes where the three runs agreed, where they split, how each one failed, and what each one cost.

Ranks will move as submissions arrive. With 100 questions, a score near 93% carries a standard error of about 2.6 points, so the top two are a statistical tie: a Flash-tier model matched a frontier reasoning model on 100 hard computational-biology tasks. DeepSeek's gap is real, though up to 3 of its 11 points belong to the harness rather than the model (Result 2).

On the same harness, all three models gave the identical final answer on 81 of 100 questions, and GPT and Gemini agreed on 92 of the 99 questions they both answered. The other 19 questions are where the model still matters, in three ways: finishing, verifying, and knowing when to stop.

Setup

The harness itself was operated by an agent. Claude Opus 5, running in Cursor, wrote the driver script, launched and monitored all three runs, read the failure transcripts, made the patches listed under "Extra turns" above, added a per-turn wall-clock cap after r3's overnight hangs, and proposed which r3 answers to re-run. The three models in the harness saw only the prompt and the sandbox. DeepSeek does not expose xhigh, so it ran at high.

Result 1: The harness does most of the work on well-scoped tasks

CompBioBench tags each question with a style, and unanimity tracks the style. On Routine Analysis all three models agreed on 20 of 22 questions; on Retrieval, 14 of 17. These are tasks like "compute GC content over these intervals" or "convert these identifiers." Once the sandbox has bedtools, samtools, uv, and a skill file that names the right tool, the model's job is to drive it, and three very different models drove it to the same place.

Unanimity drops on Metadata Recovery (21 of 27), Synthetic/Augmented Data (20 of 26), and Tooling (6 of 8), where no standard tool produces the answer. The agent has to decide what "the source study" or "the contaminant" or "least consistent" means, and then commit.

Offline questions split more than online ones: 73% unanimous (16 of 22) with no internet, against 83% (65 of 78) where the agent could fetch a canonical record. When there is a database to check, all three check it and converge. When there is only the data file, they diverge.

Result 2: Each model fails in its own way

GPT answered every question on the first attempt. DeepSeek needed 128 attempts and Gemini 129 (119 overnight, 3 backfilling questions the overnight run left blank, and 7 in the operator's re-run pass), and they reached those near-identical totals by very different routes.

Five questions needed retries from both DeepSeek and Gemini. Four of Gemini's 16 multi-attempt questions were operator re-runs to check an answer rather than failed first attempts.

DeepSeek: thinks to the token cap and never writes the answer

In 30 of the 31 failed r1 attempts, the final assistant message is a reasoning block of 8,500 to 17,000 characters with no answer text, and the turn ends with stopReason: length. The model does the analysis, then cannot convert it into the one-line commitment the grader needs. The 31st failure was a dropped stream.

The stall has two variants. The fast stall, 13 of the 31, happened inside five minutes with at most 17 tool calls. On a question about enhancer-promoter interactions, DeepSeek read two small CSV files, computed the right summary table in one script, and then deliberated. Its reasoning trace weighs two readings of the question, notes that "both interpretations [are] defensible," decides it "need[s] to figure out the benchmark's intent," and runs out of tokens. It did this five times, at five reasoning levels down to minimal, in 10 minutes and $0.28, and the lowest level still produced a 16,900-character thinking block. The slow stall came after 15 minutes to 11 hours of real work. On a question that asks which public study a dataset came from, DeepSeek's first attempt ran 167 tool calls over 11.1 hours, enumerating repository records by date range and downloading candidate matrices one by one. The trace ends mid-plan: "Conclusion: source not among 14 ... Let's broaden to all dates." Two more attempts, 14.7 hours in total, produced nothing, and the cost counter returned nothing on two of the three, so the $1.67 recorded is a floor.

Retries recovered 14 of the 17 stalled questions, but only 8 of those 14 answers match r2's, and the 6 that do not account for most of DeepSeek's content disagreements in this post.

Gemini: issues a command that never returns

Gemini's failures happen earlier in the turn. In 14 attempts across 9 questions, the agent issued a tool call that never came back. In the overnight run the harness waited out its budget, 126 to 148 minutes, and recorded the attempt at $0 because the cost event never arrived. Those 14 hung turns account for 30 of r3's 60 agent-hours.

Six of the fourteen hung commands were remote queries to a public sequence-search service, sent from inside the sandbox and never answered. Others were a whole-genome alignment of a full read set, a WebSocket test against an ortholog database, and a multi-hour processing pipeline. Every one is a command a careful analyst would put a timeout on. Gemini put a timeout on none of them, and neither did the harness, which trusted the socket-level timeout that never fires while a connection dribbles bytes. We added a wall-clock cap per turn after the overnight run; it fired correctly on the first stall it saw the next morning.

Gemini also hit provider errors none of the other models saw: HTTP 429 "resource exhausted" on four attempts, and at least four 400s reading "Invalid thought signature" or "Corrupted thought signature," a failure specific to Gemini's thinking API when a multi-turn conversation is resumed. Both were handled by retry.

GPT-5.6 SOL: finishes, but waits for a user who is not there

GPT never stalled and never hung. Its cost was waiting. It called the harness's interview tool on 21 questions to ask a user who does not exist in a benchmark to confirm its analysis plan. Each call waited for its timeout (60 seconds on 13 questions, 600 on 7) and then proceeded with the option it had already marked "recommended," which adds up to 85 minutes of r2's 22 agent-hours. The format-check turn we added after every answer cost $1.87 and changed nothing, because GPT already formats.

The four blanks

Four question-run pairs ended with no answer after every retry. Three are DeepSeek's: the two stalls described above, plus a DNA language model question where it correctly diagnosed that the sandbox had no GPU, named the right fix, and then thought to the token cap three times instead of applying it (GPT answered the same question in 4 minutes). The fourth is Gemini's on an ATAC-seq pipeline question, and mostly our fault: the pipeline needs 1.6 to 3.9 hours, and we re-ran it under a new 45-minute per-turn cap, so it was unanswerable by construction. That cap also exposed a second harness bug: closing the stream did not stop the agent, and one orphaned session ran for another hour and read files from a sibling project's directory before the provider killed it. A cap needs to cancel the session too, and ours does not yet.

The three r1 blanks also matter for the score. GPT and Gemini both answered all three and gave the same answer on each. r1 ran before the recovery turn existed, and that turn later rescued three text-less Gemini turns of exactly the kind DeepSeek produced here. Had it rescued DeepSeek's three, we estimate DeepSeek would have scored between 81% and 84%, so up to 3 of the 11 points between DeepSeek and GPT belong to the harness. The other 8 or more are DeepSeek's.

Result 3: Three models, three relationships with the harness

The tool mix tells the same story from the other side. DeepSeek made 3,898 tool calls and 85% were bash. It worked like someone who only has a terminal: it used shell commands to write a script to disk, ran the script, read what it printed, and repeated. It touched the lab notebook 14 times and never opened a specialist.

GPT-5.6 SOL made 3,241 calls and 44% were bash. The rest spread across the harness: 370 notebook entries, 277 file reads, 272 page fetches, 68 structured result records, and 51 calls to spawn or poll specialist sub-agents on 10 questions.

Gemini 3.8 Flash made 6,999 calls, more than the other two combined, and 76% were bash. It ran 904 web searches, five times GPT's 181, used the notebook 68 times, and never used a specialist or called interview. It is the most active model of the three and the least structured.

A question about protein structure shows the three styles side by side. GPT rendered projections to PNG, looked at the images, and handed the same file to a data-visualizer specialist for an independent view: five minutes, $0.35. DeepSeek rendered the same kind of image, tried to read it, and got back "Current model does not support images." It then spent 3.2 hours and $2.76 across two attempts rendering ASCII silhouettes and matching them against letter templates. Gemini recognized the structure, searched for the paper that first described it, downloaded that paper's figure, cropped the relevant panels, and compared them to its own renders: six minutes, $0.25. The three methods produced three different letters.

Result 4: Verification is where the money goes

GPT finished the benchmark in an evening; DeepSeek took a day and Gemini most of a night, and in both cases a handful of long or hung turns held a worker slot for hours while the other three slots emptied. Wall-clock matters if you are waiting on results; agent-hours matter if you are paying for a sandbox by the hour.

r1 and r3 costs are understated, because the cost counter returned nothing on DeepSeek's 11-hour attempt or on any of Gemini's 14 hung turns. The costs are also not comparable across models. They are the harness's own ledger, computed from token counts and OpenRouter list prices on the day of the run rather than read from a provider invoice. Gemini used 2.5 times GPT's tokens and cost less because 92% of its prompt-side tokens were served from cache, and the ledger bills cached tokens at zero, so a model with aggressive implicit caching gets a systematic accounting advantage. OpenRouter's own usage endpoint disagreed with the ledger by about $9 for r3. Treat the accuracy numbers as measured and the cost numbers as estimates that favor Gemini.

One question moves the r2 total more than any other. It asks the agent to identify the GWAS behind a stripped summary-statistics file. GPT spent $13.16 and 118 minutes on it, 18% of the entire r2 bill and 53 million tokens. DeepSeek spent $1.92 over three attempts, the last one 10 minutes. Gemini spent $2.13 and 84 minutes. All three named a different paper.

The transcripts show three definitions of "done." DeepSeek mapped the top loci to genes, recognized the trait, found the best-known GWAS of that trait, confirmed the PubMed ID, and stopped. GPT did the same mapping in the first 15 minutes and reached the same paper. Then it tried to match the file's exact numbers. It downloaded that paper's supplementary tables, searched them for the file's effect sizes and P values, did not find them, and rejected the candidate. It went on through the GWAS Catalog, PheWeb, a literature specialist, and the supplementary tables of two more papers until one table matched the file's positions row for row. Gemini also went numbers-first: it pulled full summary-statistics files from the GWAS Catalog FTP for a series of candidate studies and compared variant counts and coordinates, and after 349 tool calls it landed on a third study, a proteomics GWAS whose tables overlapped the file.

We do not know which answer is right. We do know that "find a plausible source" and "find the source that reproduces the file" are different tasks, and that two agents pursuing the second one still landed in two different places. Metadata recovery from a stripped file is the one part of this benchmark where more effort did not produce convergence.

Two smaller cases make the same point. On a question that asks for a count rounded to the nearest hundred, the three runs produced three different hundreds. On a contamination question, DeepSeek and GPT both found the same non-human signal and named the same genus. Gemini found that signal too, then kept going: it aligned the reads to a species closely related to human, saw a large number of hits, and named that species instead. Reads from a human sample will always map to a close relative of human, so that is not evidence of contamination, and a model that has already found the real contaminant should not be talked out of it by a second alignment.

Result 5: All three models went looking for the answer key

The prompt says "get any files or tools you need from the internet." All three models read that to include the question itself.

DeepSeek ran 54 web searches on 23 questions that quoted the input filename or the run label. GPT-5.6 SOL did it least, 13 searches on 8 questions, but most precisely: it searched site:huggingface.co/datasets/genentech/compbiobench-data-v1 and site:github.com/genentech/compbiobench for the input filenames. Gemini did it most: 135 searches on 51 questions, more than half the benchmark, quoting filenames, the run label cbb-r3, and whole sentences from the question text in quotation marks.

None of them found anything, because the answers are not public. An agent that searches for the benchmark's own source is looking for a shortcut around the problem, and on a public benchmark with published answers this would be contamination in real time. Filenames that look like benchmark fixtures (something.q1.tsv.gz) invite it, and a model that runs 900 searches will try it more often than a model that runs 180.

Result 6: The clock is advisory

The prompt tells the agent it has 120 minutes, and in r1 and r2 nothing enforced it. DeepSeek's longest single turn was 11.1 hours. GPT's was 3.9 hours, running a full ATAC-seq pipeline end to end on a Modal instance; DeepSeek ran the same pipeline in 1.6 hours and got a peak count that differed from GPT's by 41, about 0.14%. That is a pipeline determinism question rather than a model question, and exact-match grading cannot tell the two apart. A stated time budget that the harness does not enforce is a suggestion, and two of three models ignored it. When we did enforce it, the cap broke the one question that needs hours.

What we take from this

A Flash-tier model tied a frontier reasoning model. That is the most useful thing these runs measured. Weigh it against the rest of this list, not against the two-point gap.

The model matters least where the tooling is mature. Routine analysis and retrieval converged three ways. If you run a bioinformatics core, the question is whether the skill and tool layer is in place, more than which model sits on top of it.

Each model has one dominant failure, and they are different. DeepSeek reasons past the output cap and emits no text. Gemini issues network commands with no timeout and hangs. GPT asks a question and waits for an answer that will never come. A harness can catch all three with a recovery turn, a wall-clock cap, and a headless flag that disables the interview tool, and ours now has all three. The cheapest failure is still the one that never happens, and only GPT delivered that.

A cap shorter than the task is a failure you built. Caps need to be set per question rather than per run, and they need to kill what they cap.

Recovery turns need a sanity check. r3's rescued three answers and corrupted one. Anything that re-asks the model for its answer should check that the reply looks like one.

Verification is a spend decision. GPT-5.6 SOL's instinct to reproduce the file before naming the source is the behavior you want from a scientist and the behavior that turned one question into 18% of the bill. Gemini had the same instinct at a fifth of the price and reached a different conclusion. Per-project spend caps exist for this reason. So does a human who can say "the first candidate was fine."

More activity is not more agreement. Gemini made the most tool calls, ran the most searches, and used the most tokens. It agreed with GPT almost everywhere GPT and DeepSeek already agreed, and where it went its own way, on the contaminant and the rounded count, the extra work led it off rather than on.

Watch for answer-key hunting. All three models tried it. Held-out answers are the only reason it did not work.

Caveats

• n=1 per model per question. No temperature or seed was set. McNemar on the GPT vs. Gemini pairs: p = 0.5 to 0.8.
• Reasoning level was high for DeepSeek, xhigh for the others, and stepped down on retry. Runs happened on different days.
• Harness changed between runs (see Setup). r3's recovery turn changed three answers and corrupted one.
• r3's three backfills and five re-runs were chosen by the operator agent and approved by a human.
• The operator was a frontier model. Its failure diagnoses were checked against transcripts, but it chose where to look.
• Costs are estimates (see Result 4).
• Wall-clock is computed from session timestamps, first start to last end, including idle gaps. The r3 re-run pass is estimated.

How we ran it

All three runs used K-Dense BYOK, the open-source version of Kady. Every project had its own sandbox, environment, and spend cap, and every transcript is on disk. We can share the driver script that runs the benchmark through BYOK. We will not share the answer files, for the same reason we did not print answers here. If you want to run the same class of computational-biology work on your own data, that is what K-Dense Web is for.

Start your own analysis →

Reference

Nair S, Gunsalus L, Orcutt-Jahns B, Rossen J, Lal A, De Donno C, Çelik MH, Fletez-Brant K, Xie X, Corrada Bravo H, Eraslan G. Agentic systems are adept at solving well-scoped, verifiable problems in computational biology. bioRxiv 2026.04.06.716850. doi:10.64898/2026.04.06.716850. Dataset: genentech/compbiobench-v1. Runner: Genentech/compbiobench-runner.