--- title: "The Silent 97%: Introducing the waypoint-bio Agent Skill" description: "A microbiome foundation model silently discards 97% of a malformed table and returns a valid embedding. We measured what an Agent Skill does about that." updatedAt: "2026-08-18" author: "Timothy Kassis, PhD" authorTwitter: "timothykassis" authorLinkedIn: "https://www.linkedin.com/in/timothykassis/" tags: ["AI", "Skills", "Scientific Agent Skills", "Benchmarks", "Microbiome", "Open Source"] canonical: "https://www.k-dense.ai/blog/introducing-waypoint-agent-skill" ---

What is the waypoint-bio skill? The waypoint-bio skill packages Outpost Bio's open microbiome foundation models for an AI agent: a SKILL.md, four reference documents, and two scripts covering the waypoint CLI, the Atlas pretraining corpus, the Compass benchmark, and the conversion of MetaPhlAn, Kraken2, QIIME 2 and MGnify abundance tables into the format these tools expect. It is part of K-Dense's open-source Scientific Agent Skills library, and it is the subject of the benchmark below.

A microbiome foundation model will accept a malformed taxonomy table, discard almost everything in it, and return a perfectly well-formed embedding. This post measures what an Agent Skill does about that, on the model's own benchmark and on the agents that use it. | | | |---|---| | **Subject** | the `waypoint-bio` skill · Outpost Bio Waypoint / Atlas / Compass | | **Measured on** | Compass, 62,809 samples · Waypoint-6m and Waypoint-45m · waypoint-bio 1.0.2 | | **Hardware** | Apple Silicon, PyTorch MPS · torch 2.13, Python 3.12 | A microbiome sample can be read as a sentence. Each taxon is a token, the tokens are ordered by how unusually abundant they are, and a transformer is trained to predict the next one. That is the idea behind [Waypoint](https://github.com/Outpost-Bio/waypoint), Outpost Bio's open family of microbiome foundation models, released with a 539,308-sample pretraining corpus (Atlas) and an eight-task benchmark (Compass). The idea works, but it inherits a property of every language model: a fixed vocabulary. Waypoint's vocabulary was frozen at pretraining time from Atlas, so anything outside it becomes ``, and `waypoint embed` drops those tokens before it does anything else. It emits no warning and no log line, and it exits zero. This is where an agent gets into trouble. Ask any capable AI agent to "embed my MetaPhlAn profiles with Waypoint" and every step will appear to succeed: a parquet file will be written, it will have the right shape, and every downstream number computed from it will be meaningless. The `waypoint-bio` skill exists to make that failure hard to reach, and this post measures whether it does, with the package actually installed, the gated checkpoints actually downloaded, and the benchmark actually run. | | | |---:|---| | **3%** | of abundance mass survives when MetaPhlAn lineages are passed in unconverted | | **514** | Compass rows that ship with no taxa at all, embedded and scored anyway | | **133** | tokens the skill costs when it is not being used | | **16 to 0** | paired wins for skill-equipped agents on a blind-graded rubric | ## 1. What a skill actually is An Agent Skill is a folder. At its root is `SKILL.md`, the instructions an agent reads when it decides the skill is relevant, plus reference files and scripts it can open on demand. Only the skill's *name and description* live permanently in the agent's context; everything else loads when it is needed and not before. That structure is what makes a skill cheap to keep installed, and it is measurable. ![Context cost of the skill by layer](./figures/f1_context.png) **Fig 1.** Context cost of the `waypoint-bio` skill, by layer, against the upstream repository. Counted with `tiktoken o200k_base` as a stand-in for a frontier model's tokenizer. The skill's standing cost is the top bar: 133 tokens, about two sentences' worth. The full skill including both bundled scripts is 18,178 tokens; the upstream README, source and configs come to 36,571. The skill is roughly half the size of the material it summarises, and in ordinary operation it is 0.4% of that, because 133 tokens is all an agent carries until a microbiome task actually appears. You do not pay for a skill on every turn. You pay for the name and description, and for the rest only on the turns that use it. ## 2. Where the knowledge lives Most of what the skill says is not written down anywhere upstream. We took 22 load-bearing claims from the skill and classified each by where an agent could have learned it, verifying every classification with a grep against the cloned repository. | Where the fact lives | Facts | What it costs an agent to find | |---|---:|---| | Upstream README | 4 | read 5,543 tokens of README | | Source code or shipped config only | 9 | read the relevant module | | Determined only by running it | 4 | **not documented anywhere** | | The paper, not the code | 4 | **not in the repository** | | Model card | 1 | read the Hub page | ![Where the skill's facts are written down](./figures/f1b_provenance.png) **Fig 2.** Where the skill's 22 load-bearing claims are written down. Thirteen are in the repository: four in the README, nine only inside source or a shipped config. The remaining nine are not written anywhere in it, with four establishable only by running the code and five living in the paper or the model card. Nine of twenty-two are recoverable only by reading source, four are stated nowhere and can only be established by running the code, and five more are documented outside the repository entirely. Some of the relevant vocabulary never appears upstream at all: we grepped the entire README and source tree for `MetaPhlAn`, `Kraken`, `QIIME`, `d__Bacteria`, `random forest`, `out-of-vocabulary` and `batch effect`, and got zero hits on all seven. Those are the terms describing how you would get data into this model and how you would know whether to trust what comes out. The mechanisms are in the code (`filter_unk_taxa` sits in `dataset.py`), but nothing tells a reader that it matters, or what to do about it. ## 3. One character, 97% of the data The tokenizer splits a lineage on `;`, reads each segment's three-character rank prefix, and returns the genus segment, falling back to the most specific higher rank when genus is missing. MetaPhlAn joins its lineages with `|` instead. The whole string therefore arrives as one segment, its first three characters are `k__` so it matches at kingdom rank, and the entire pipe-joined string is returned as a single token that is not in the vocabulary. ``` waypoint format k__Bacteria; p__Firmicutes; …; g__Lactobacillus → g__Lactobacillus ✓ in vocab MetaPhlAn k__Bacteria|p__Firmicutes|…|g__Lactobacillus → (whole string) ✗ QIIME 2 / SILVA d__Bacteria; p__Firmicutes; …; g__Lactobacillus → g__Lactobacillus ✓ in vocab bare tip name Lactobacillus → None ✗ ``` To size the damage we took 3,000 real Compass samples that are 100% in-vocabulary when correctly formatted, re-expressed their lineages the way each profiler actually emits them, and measured how much of each community the model could still see. ![Abundance mass retained by input format](./figures/f3_formats.png) **Fig 3.** Median fraction of abundance mass that survives tokenisation, by input format. The same 3,000 samples are used throughout, so every loss is caused by formatting alone. MetaPhlAn's pipe separators cost 97% of abundance mass and leave 243 of 3,000 samples with nothing at all. Bare names, species-only tips and domain-truncated lineages lose everything. The QIIME 2 `d__` prefix costs 3.8% in practice, because deeper ranks in the same lineage still resolve. > **Why these numbers moved.** An earlier version of this figure reported 99.6% loss, not 97%. It was computed on the abundance column as Compass ships it, and for `mgnify-biomes` that column does not hold relative abundances. Its values are sorted descending, run negative, and sum to about −3.2 per sample: they are per-taxon abundance *z-scores*, while the other three configurations hold genuine relative abundances. The z-scores invert with the checkpoint's own `token_std_means.parquet` (`ra = z × std + mean`), which reconstructs row sums of 1.00, and the figure above is computed on the reconstruction. We did not catch this ourselves; an agent did, while running the bake-off in section 9. > **A result that argues against the skill's own emphasis.** The skill warns prominently about the `d__` prefix. Measured, it is the mildest of the four failure modes, retaining 0.962 of abundance mass, and it only destroys a lineage that *stops* at domain. Worth knowing and worth fixing, but not the thing to worry about first. ## 4. The benchmark has the problem too We ran the skill's own `vocab_coverage.py` across all four Compass source studies, 62,809 samples and the complete benchmark, using the checkpoint's tokenizer and vocabulary. ![Vocabulary coverage per Compass study](./figures/f2_coverage.png) **Fig 4.** Median abundance-weighted vocabulary coverage per Compass study, Waypoint-6m. The vocabulary holds 14,394 entries: 8,717 genus, 3,417 family, 1,443 order, 531 class, 274 phylum, 7 kingdom, and 5 special tokens. | Compass study | samples | taxon cov. | abundance cov. | below 0.8 | empty embedding | |---|---:|---:|---:|---:|---:| | mgnify-biomes | 41,399 | 1.000 | 1.000 | 0.0% | 0 | | roswall (infant gut) | 2,031 | 0.636 | 0.809 | 47.5% | 1 | | mastrorilli (drug degradation) | 15,419 | 0.933 | **0.736** | **88.3%** | **514** | | handuo (drug exposure) | 3,960 | 0.778 | **0.536** | **80.2%** | 17 | Five hundred and thirty-two benchmark samples, 0.85% of Compass, encode to `[BOS][EOS]`, an embedding of the empty sentence, and are scored anyway. They get there two different ways, and the distinction matters. Eighteen are true vocabulary casualties, where every taxon they carry is out-of-vocabulary. The other 514 are not a vocabulary problem at all: Compass ships them with an empty `Taxa` list, 3.3% of the `mastrorilli` regression task, nothing in them to measure in the first place. Dropping those empty rows moves `mastrorilli`'s median coverage only from 0.736 to 0.738, so the picture above is unaffected, but "the model cannot see this sample" and "there is nothing to see" are different findings, and our first pass had them conflated. In `handuo`, where the loss is genuinely a vocabulary problem, the median sample has 46% of its community invisible. The reasons are legible once you look at what is missing. `mastrorilli` uses the revised NCBI kingdom names, `k__Bacillati` and `k__Pseudomonadati`, which post-date Atlas. `handuo` and `roswall` use SILVA genus conventions such as `g__Escherichia-Shigella`, `g__Lachnospiraceae NK4A136 group` and `g__[Ruminococcus] torques group`. These are real taxa carrying real abundance, and the model cannot see any of them. Waypoint's own paper names vocabulary as its main limitation, so the point here is narrower: it bears on what a number from this benchmark means, and it is what the skill instructs an agent to measure before reporting anything. ## 5. What silence looks like downstream We embedded 3,310 held-out Compass samples twice with the same checkpoint and the same command, once in waypoint format and once with the lineages pipe-joined, and projected both into two dimensions. ![PCA of embeddings, correct vs corrupted input](./figures/f6_structure.png) **Fig 5.** PCA of Waypoint-6m embeddings, 1,500 held-out MGnify samples, three largest biome classes. On the left, with correct input, there are 1,500 distinct embeddings and the biomes separate cleanly along PC1. On the right, the identical samples with MetaPhlAn pipe separators collapse onto 80 distinct embedding vectors. Both runs completed successfully and wrote valid output. ![Macro-F1 by feature set](./figures/f4_downstream.png) **Fig 6.** Macro-F1 on MGnify biome, 12,000 training samples and 3,310 held-out test samples. Blue is Waypoint representations from correctly formatted input, green is classical baselines, and orange is two ways to silently destroy the signal. The pipe-separated run scores 0.207, and the pipeline reported no error. Two secondary results bear on advice the skill gives. The first is that `first_token` pooling scores 0.061: the skill says the first position "carries little signal in a causal LM", and measured, it carries none a linear probe can reach, since always predicting the majority class scores 0.061 as well. The second is that mean pooling beats the default. `last_token` is the documented default and the skill tells you to keep it, but on this task mean pooling scores 0.899 against 0.866, a real 3.3-point advantage. The skill does allow that "mean is a reasonable alternative for unsupervised use", which is what this probe is, but an agent reading only the headline advice would leave that on the table. > **What these probes measure.** These are linear probes on frozen embeddings, not fine-tuning runs. They measure how much signal the pretrained representation exposes linearly, not the ceiling of a fine-tuned model, and not a Compass score. ## 6. Coverage, not sample size, decides who wins The skill carries a blunt warning: below roughly 1,000 labelled samples, Waypoint underperforms a random forest on raw abundances, and the crossover sits near 10,000. We tried to reproduce that shape, and on the in-distribution task we could not, so we ran the same comparison on the low-coverage studies as well. One thing had to be fixed first. Compass's own `mgnify-biomes` split is not grouped by study: 99% of its test samples come from a study that also appears in training, with 751 of 789 test studies shared. That is the leak the skill warns about, present in the benchmark's own splits. Scores computed on it are inflated for both methods, so we rebuilt the MGnify comparison with every `Study Accession` confined to one side, across three random grouped splits. ![What the study-level leak was worth](./figures/f5b_leak.png) **Fig 7.** What the study-level leak was worth. Same task, same features, same models, with only the split changing. On Compass's own split both methods score around 0.3 macro-F1 higher than they can justify. The ranking survives the correction and the gap between them doubles, from +0.064 to +0.128. Both methods lose about 0.3 macro-F1 once studies cannot be memorised, which is a measurement of exactly the batch-effect caveat the skill lists. The ranking survives and the gap doubles, and the MGnify panel below uses the grouped split. ![Learning curves by vocabulary coverage](./figures/f5_curves.png) **Fig 8.** Macro-F1 against labelled training-set size, mean of three seeds. MGnify uses a study-grouped split because Compass's own split leaks; `handuo` and `roswall` use their Compass test splits, and their coverage figures are for the exact subset analysed. Panels are ordered by vocabulary coverage. On `mgnify-biomes`, with the leak removed, the probe leads by 0.102 macro-F1 at n = 50 and never loses, ending 0.631 to 0.503. On `handuo`, at 52% coverage over the subset analysed, the forest is ahead from n = 250 on, finishing 0.276 to 0.221. On `roswall`, at 81% coverage, the two cross: the forest leads between n = 250 and n = 500, the probe from n = 1,000, ending 0.759 to 0.725. What moves the crossover is how much of each community the model is allowed to see, not how many labelled samples there are. A random forest reads every taxon in the table, while the transformer reads only what survived tokenisation. When those are the same set, pretraining pays off immediately; when they are not, the baseline is competing with a handicapped opponent, and wins. This refines the skill's caveat rather than contradicting it: *measure coverage, then decide* is the sharper rule, and the sample-size threshold is a proxy for it. ## 7. Does any of this change how an agent behaves? Five realistic tasks, each written so the traps above are live: a MetaPhlAn table with 240 samples, a QIIME 2 / SILVA table to embed, a longitudinal infant cohort with repeated measures, benchmarking a custom model on Compass, and debugging a fine-tune that terminates immediately. Each task was answered four times with the skill installed and four times without. The no-skill agents kept full web access and the public repository, and were barred only from the skill directory. That is forty answers, each scored against a five-item task-specific rubric fixed *before* the answers existed, by graders who saw the answers anonymised and shuffled. ![Rubric scores with and without the skill](./figures/f7_ab.png) **Fig 9.** Rubric items satisfied (of five) per task, mean of four runs, blind-graded. Every task improves and none regresses, with an overall 4.80 ± 0.41 against 3.65 ± 0.81. Across the twenty matched pairs there are 16 wins, 0 losses and 4 ties (exact sign test, two-sided p = 0.00003). The gap is not spread evenly across the rubric. It sits almost entirely on the two items that require having been burned before. ![Rubric items regrouped by what they test](./figures/f7b_items.png) **Fig 10.** The same 25 rubric items regrouped by what they test. The two bars that move are checking vocabulary coverage (100% against 33%) and fitting the classical baseline first (100% against 50%). Grouped splits and training-config details sit at 100% in both conditions. Unskilled agents handled the textbook statistics perfectly: every one demanded a subject-level split for the longitudinal cohort, and they got the training-configuration diagnosis right every time. What they missed was package-specific and undocumented. Fabricated flags, subcommands and config keys counted by the graders across the forty answers came to 4 with the skill and 11 without. Two graders read every answer in the first two replicates independently, blind and reshuffled, and over those 100 item-level judgements they agreed on 96% (Cohen's κ = 0.85), so the rubric measures something reproducible rather than one grader's taste. ### The skill is cheaper to use than not having it ![What each run cost](./figures/f8_cost.png) **Fig 11.** What each run cost, replicates 1 and 2. Agents with the skill used 7.0 tool calls against 15.4, and 3,368 output tokens against 5,494. Wall-clock followed at 51s against 119s, but runs shared a laptop, so treat tool calls and tokens as the clean measures and wall-clock as indicative. The composition is the explanation. Without the skill, agents made 63 web searches and fetches hunting for documentation; with it, they made zero, replaced by 10 local reads of the reference files. Unskilled agents are not helpless here. They recovered the MetaPhlAn `|` conversion in seven of eight attempts by reasoning about it. What they consistently missed was the part no amount of reasoning produces, which is that out-of-vocabulary taxa vanish without a warning, so you must measure coverage before you believe anything. ## 8. Which layer carries it, and where it does nothing Two follow-up experiments ask what the effect actually is. The first strips the skill down, answering the same three tasks with *only* `SKILL.md`, the agent barred from opening the reference files or the scripts, against no skill and against the full skill, with all six answers per task graded together by one grader on one scale. The second runs four microbiome tasks the skill says nothing about: alpha diversity, PERMANOVA, differential abundance, and study design. ![Ablation and negative control](./figures/f9_ablation.png) **Fig 12.** On the left, the 3,406-token `SKILL.md` carries essentially the whole effect (4.83 of 5, against 3.17 with no skill), and adding the four reference files and two scripts does not raise the rubric score further (4.67, a one-item difference on one answer). On the right, across four tasks the skill does not cover, the two conditions are the same within noise (4.75 against 4.50). The gain is package-specific knowledge, not a generic "more context helps" effect. For anyone writing a skill, the left panel is the result to read. The expensive part, 8,798 tokens of reference files and 5,974 tokens of scripts, did not move the rubric; what moved it was the cheap part, the 3.4k-token entry document that says *check coverage, fit the baseline first, the separator is a semicolon*. That does not make the references useless, since they carry the specifics an agent needs once it is actually running commands: which converter flag, which config key, what `benchmark_results.json` contains. But the judgement calls, the ones that decide whether a result is trustworthy, fit on a page. The right panel is the control that makes the rest of the post harder to dismiss. If the skill were simply making answers longer or more careful in general, it would have helped on PERMANOVA too. It did not, and it did not hurt either: no skill-equipped agent went off on a Waypoint tangent when asked about rarefaction. ## 9. Advice is not the same as outcome Everything so far scores what an agent *says*. The harder question is whether it changes what an agent *produces*, so we ran a bake-off. Both conditions got the same task: 4,000 labelled and 1,000 unlabelled MGnify samples, lineages pipe-joined the way MetaPhlAn emits them, a working environment with the checkpoints cached, and one instruction, to predict the biome with Waypoint, write `predictions.csv`, and state the macro-F1 they expected. The split is grouped by study, and we kept the test labels and scored the submissions ourselves. ![Bake-off results](./figures/f13_bakeoff.png) **Fig 13.** Eight independent runs, four per condition, scored on 1,000 held-out samples. Dashed lines are the condition means. With the skill: 0.706 ± 0.018. Without: 0.693 ± 0.018. The difference of +0.013 is smaller than the spread between runs of the same condition. A majority-class baseline scores 0.039. There is no detectable difference. Every run landed between 0.676 and 0.726, and the condition means are within one run-to-run standard deviation of each other. We are reporting it as a null. The transcripts show why: *every* agent found the pipe-separator trap, because when you actually run the pipeline, the trap announces itself. Coverage collapses to zero, embeddings come back identical, a cross-validation score sits at chance. Several agents also independently noticed that the abundance column held z-scores and inverted them, and two rediscovered that mean pooling beats the `last_token` default, which is the same thing Fig 6 found. Given a real dataset, an execution loop and twenty minutes, an unskilled agent can rediscover most of what the skill would have told it up front. So the skill's value is concentrated where there is no feedback loop: the one-shot answer, the plan written before any code runs, the reviewer's question about whether a number can be trusted. That is exactly what sections 7 and 8 measure, and it is a narrower claim than "the skill makes the model better at microbiome ML", which it does not. > **The result nobody was calibrated for.** All eight runs stated an expected macro-F1, and seven overestimated, by a mean of 0.12, predicting 0.80 to 0.85 and delivering 0.68 to 0.73. The single run that came in under its own estimate predicted 0.68 and scored 0.711; it got there by running adversarial validation against the test set and discounting its cross-validation number. Both conditions were overconfident (skill +0.10, no skill +0.14, four runs each). The skill did not fix that, and nothing in it addresses calibration. If we were writing it again, that is the gap we would fill. ## 10. What this means if you build for scientists Scientific tooling fails without saying so far more often than it fails loudly. A malformed lineage does not raise, it returns an embedding. A leaked split does not raise, it returns an excellent test score. A frozen vocabulary does not raise, it returns a shorter sentence. Every one of those produces output that passes every check an agent knows how to run. A skill carries what someone learned the first time this went wrong, and on these measurements it does so cheaply. It costs 133 tokens while idle and 3,406 once a microbiome task appears, and runs that used it took 7 tool calls instead of 15, with 63 web searches replaced by zero. Nine of the twenty-two load-bearing facts it encodes are written nowhere in the repository, and seven of the relevant technical terms never appear upstream at all. Where that changes behaviour is the one-shot answer: 16 to 0 on matched pairs, and a third as many fabricated API details. Where it does not is the final score, once the agent can run the pipeline and iterate. The strongest part of the `waypoint-bio` skill is its *Scientific caveats* section and the two scripts that let an agent check its own assumptions before reporting a number, `vocab_coverage.py` above all. A skill that only taught an agent to run commands would have produced Fig 5's right-hand panel and called it a result. > **What we would change in the skill.** Lead the coverage guidance with the measured consequence, since a 0.866 to 0.207 collapse is more persuasive than "taxa are dropped". Demote the `d__` warning relative to the `|` one, since we measured 3.8% loss against 97%. And restate the small-sample rule in terms of coverage, since coverage is what actually predicted the crossover in all three studies we tested. ## 11. Methods Everything here was run once, on one laptop, against the real artefacts. No numbers are illustrative. **Environment.** `waypoint-bio` 1.0.2 from PyPI, Python 3.12, torch 2.13 on Apple Silicon MPS, transformers 5.15, scikit-learn 1.9. Checkpoints `outpost-bio/Waypoint-6m` and `-45m`, dataset `outpost-bio/Compass` (all four configurations, 62,809 rows). Embedding throughput was 189 samples/s on Waypoint-6m and 53 samples/s on Waypoint-45m. **What was run.** - *Coverage.* The skill's own `scripts/vocab_coverage.py` plus a reimplementation over all four studies, using the checkpoint's `TaxonomicTokenizer._extract` and vocabulary, which is the same code path `embed`, `finetune` and `benchmark` use. - *Format degradation.* 3,000 `mgnify-biomes` test samples, re-expressed in six formats, scored on taxon and abundance retention. - *Embeddings.* The real `waypoint embed` CLI, ten runs, 12,000 train and 3,310 test samples, three pooling modes, two checkpoints, correct and corrupted input. - *Probes.* Standardised features, multinomial logistic regression (max_iter 3,000), and a baseline random forest of 300 trees on the top 3,000 taxa by training-set prevalence. Neither model was tuned. Macro-F1 on a study-grouped split for MGnify and on each study's own Compass test split otherwise, with learning curves at eight sizes by three seeds. - *Split audit.* `Study Accession` overlap between the Compass MGnify train and test splits, `Sample ID` overlap for the three study configs (none), and a rebuilt grouped split for the MGnify comparison. - *Agent A/B.* 5 tasks by 2 conditions by 4 replicates, giving 40 answers, with rubrics fixed in advance, grading blind and shuffled, and an exact sign test on the 20 matched pairs. Inter-grader agreement comes from a second independent reading of the 20 first-wave answers. - *Cost.* Tool calls, output tokens and wall-clock parsed from the run transcripts, with condition labels recovered from each run's own prompt rather than a hand-kept list. - *Ablation.* 3 tasks by 3 conditions (no skill, `SKILL.md` only, full skill) by 2 replicates, all conditions for a task graded together by one grader on one scale. - *Negative control.* 4 microbiome tasks the skill does not cover, 2 conditions, same procedure. - *Bake-off.* 4,000 train and 1,000 test MGnify samples, study-grouped, lineages pipe-joined, with 4 runs per condition in isolated directories and predictions scored against held-out labels kept outside the agents' data directory. Every transcript was checked for access to the truth file, and none touched it. - *Abundance units.* Row sums and sign checks per Compass configuration, plus inversion of the `mgnify-biomes` z-scores through the checkpoint's `token_std_means.parquet`. **Limits on what these numbers support.** - Probes are not fine-tuning, and none of these are Compass scores. A Compass score requires `waypoint benchmark` with the shipped config, no `--max_samples`, and multiple seeds. - The learning-curve comparison uses three studies. The coverage-decides-the-crossover claim is a hypothesis those three are consistent with, not an established law. - n = 20 for the agent comparison. The sign test is exact and the effect is one-sided across every task, but this is a small study on one model family, and rubric grading has irreducible judgement. - Token counts use `tiktoken o200k_base`, a proxy for the tokenizer the agent actually uses, accurate to a few percent. - The random forest's 3,000 features are selected by prevalence over the whole training pool, including for the small-n points on the learning curves. That favours the baseline slightly, and it does not change any ranking reported here. - Compass exposes no subject identifier for the longitudinal `roswall` cohort, so a subject-grouped split cannot be built from the released data. Its numbers are optimistic for both methods in the same way the un-grouped MGnify numbers were. - The ablation and negative control are small, at 6 and 4 answers per condition. They are consistent with "SKILL.md carries the effect" and "the effect is package-specific", but neither would detect a difference of less than roughly half a rubric item. - The bake-off's study-grouped split shifts the class priors hard, and several agents independently estimated the test set at about 60% Marine against 16% in training. That makes the task realistic but noisy, and macro-F1 on rare classes swings by several points on a handful of samples. - Wall-clock in the cost figure was measured while several runs shared one laptop, so tool calls and token counts are the reliable comparisons there. ### Reproducing this ```bash pip install waypoint-bio export HF_TOKEN=hf_... # request access on each gated repo first python analysis/00_fetch.py # Compass + checkpoints python analysis/01_context_economics.py # Fig 1, provenance table python analysis/02_coverage.py # coverage table python analysis/03_formats.py # format degradation python analysis/04_prep_embed_inputs.py bash analysis/05_run_embeddings.sh # the real `waypoint embed` CLI python analysis/06_probes.py # downstream probes python analysis/07_blind_sets.py # anonymise the A/B answers python analysis/08_unblind.py # A/B scores python analysis/09_lowcov.py # low-coverage learning curves python analysis/10_structure.py # PCA of embeddings python analysis/11_runtime.py # throughput python analysis/13_item_breakdown.py # rubric items by what they test python analysis/14_empty_vs_oov.py # empty rows vs vocabulary casualties python analysis/15_grouped_split.py # MGnify panel, leak-free python analysis/17_agent_cost.py # run cost python analysis/18_prep_bakeoff.py # bake-off dataset python analysis/19_score_bakeoff.py # bake-off, scored on held-out labels python analysis/20_blind_sets_v2.py # blind the ablation + negative control python analysis/21_score_neg_abl.py # ablation, inter-grader agreement python analysis/22_abundance_units.py # corrected abundance units python analysis/23_unblind_full.py # all four A/B replicates python analysis/12_figures.py python analysis/16_verify_claims.py # asserts every number in this post ``` Access to `Waypoint-6m`, `Waypoint-45m`, `Atlas` and `Compass` is gated per repository and auto-approved. A token alone is not enough, because the request has to be made on each repository page, which is the first thing that went wrong when we started. `analysis/16_verify_claims.py` re-checks every number printed above against the raw results and exits non-zero if the prose and the data have drifted apart. The skill is open source, along with the rest of the library: [github.com/K-Dense-AI/scientific-agent-skills](https://github.com/K-Dense-AI/scientific-agent-skills). Drop `waypoint-bio` into your agent's skills directory (`.claude/skills/` in Claude Code, or wherever your agent of choice loads Agent Skills from), and the next time you ask it to embed a taxonomy table, it will check what the model can actually see before it hands you a number. --- Waypoint, Atlas and Compass are released by Outpost Bio under Apache 2.0 and described in Treloar, Ur-Rehman, Yang and Outpost Bio (2026), *Learning the Language of the Microbiome with Transformers*, [bioRxiv 2026.05.02.722381](https://www.biorxiv.org/content/10.64898/2026.05.02.722381v2). The `waypoint-bio` skill is authored by K-Dense and is not affiliated with Outpost Bio. Waypoint is not a clinical or diagnostic tool, and nothing here should be read as validating it for that use.