--- title: "NVIDIA Nemotron 3.5 Lightning as a Science Agent" description: "Testing NVIDIA Nemotron 3.5 Lightning vs Gemma 4 31B on 15 science-agent tasks." updatedAt: "2026-08-11" author: "Aubrey Brueckner" authorTwitter: "ambrueckner" authorLinkedIn: "https://www.linkedin.com/in/aubrey-brueckner/" tags: ["AI", "Research", "NVIDIA", "Benchmark"] canonical: "https://www.k-dense.ai/blog/nemotron-3-5-lightning-science-agent" --- NVIDIA [just released Nemotron 3.5 Lightning](https://developer.nvidia.com/blog/nvidia-nemotron-3-5-lightning-delivers-fast-accurate-specialized-task-execution-for-long-running-agents/), an open 30B MoE model built for high-volume agent execution. We did the thing we always do with a new model: pointed it at our interdisciplinary AI-scientist and compared it against a model in the same weight class, Gemma 4 31B. Fifteen research-agent tasks across biology, chemistry, physics, and math/CS, each run once per model, each scored as a frozen research bundle rather than a chat transcript. The headline is one-sided: Gemma led 13 of 15 composites and both co-primary axes, but the headline is the least interesting part. We found the two models fail in *different ways*, and those failure modes tell you more about what each one needs than any leaderboard row. ## Setup Each prompt asks the agent to acquire real public data, run a domain-appropriate analysis, and leave an auditable trail: code, machine-readable results, figures, and provenance. We scored the frozen bundles with a predefined rubric using a fresh-context LLM judge (Grok 4.5 High Fast) per prompt × model. Two outcomes are co-primary: **scientific quality** (75 points: is the work correct, complete, and evidence-based?) and **research execution** (25 points: can it be regenerated, traced, and reused?). Thirteen criteria each get an integer 0–4 raw score converted to weighted points, and before examining either bundle the judge freezes a requirement checklist from the exact prompt and reuses it for both models. A fluent writeup without evidence earns nothing. | Metric | Gemma 4 31B | Nemotron 3.5 Lightning | | --- | --- | --- | | Mean scientific quality / 75 | 47.0 | 26.8 | | Mean research execution / 25 | 12.0 | 8.9 | | Mean secondary composite / 100 | 58.9 | 35.7 | | Median composite | 65.0 | 30.0 | | Composite leads | 13 | 2 | | Execution leads | 9 | 4 (2 ties) | The paired composite gap is +23.2 points (SD 24.5 across prompts; large and highly variable). Neither model's absolute scores are impressive: the best single bundle in the whole run scored 91/100, both models bottomed out near 10 on the same prompt, and both shipped confident failures more than once. ![Per-prompt composite scores for Gemma 4 31B and Nemotron 3.5 Lightning across 15 science-agent tasks](./fig1-composite-by-prompt.png) *Figure 1. Per-prompt composite scores, 15 science-agent tasks (rubric v3.0, single run per prompt × model, LLM-judged). Gemma mean 58.9, Nemotron mean 35.7.* ## Where models scored points The criterion-level profile is where the models stop looking like "better" and "worse" and start looking like different animals. ![Criterion profile comparing Gemma 4 31B and Nemotron 3.5 Lightning on scientific-quality and research-execution criteria](./fig2-criterion-profile.png) *Figure 2. Where each model earns and loses points across scientific-quality criteria A–I and execution criteria R1–R4.* Three things stand out: 1. **Gemma's lead is concentrated in doing-the-science criteria.** The largest gaps are task fulfillment and result correctness (both ~1.3 raw points), evidence integrity, and decision quality. Gemma finishes more of the required analytical work and gets more of the numbers right. 2. **Nemotron is nearly at parity on source lineage** (1.87 vs 1.93 mean raw). This is the one criterion where it structurally holds its own. It finds the correct official data sources and documents what it did with them about as well as Gemma does. 3. **Both models share the same worst criterion: environment and determinism** (1.27 vs 1.00). Neither reliably ships lockfiles, seeds, or a pinned runtime. Whatever else separates them, no bundle from either model was one-command reproducible in a clean environment. ![Co-primary scatter of scientific quality vs research execution for both models, with per-prompt pairing lines](./fig3-coprimary-scatter.png) *Figure 3. Co-primary outcomes per prompt. Each gray link joins the two models on one prompt.* The scatter shows the same story per prompt: Gemma's cloud sits up and to the right, but the gray pairing lines are long and noisy. In other words, on a given task, either model can land almost anywhere. ## Nemotron 3.5 Lightning: a careful archivist with a weak numerical core ### Strengths Nemotron's instincts around data and packaging are genuinely good. It located correct official sources (NBER pages, DepMap releases, challenge files with matching hashes) and, critically, it *documents failure instead of faking success*. On the NMR task it recorded a database access failure rather than inventing a download; on the CyTOF task it correctly reported that its own clustering had failed. It consistently ships machine-readable JSON/CSV summaries and organized folders even when the science inside them is broken, which is why it took four execution-axis wins and two ties despite losing most composites. On isolated methodology calls it can be sharper than Gemma: it implemented a proper whole-scaffold holdout split and used the correct convergence-rate formula where Gemma slipped on both. ### Weaknesses The core problem is implementation errors. Nemotron's result-correctness mean (1.13/4) was the lowest criterion score either model posted on the scientific axis, and the bugs are the kind that never get caught: an off-by-one that dropped a city from every TSP tour (leading it to claim it beat a published optimum), Monte Carlo dynamics that used stale neighbor states and broke detailed balance, a hand-rolled binary parser that corrupted every marker value it read, a missing factor of 2 in a closed-form derivation. The second pattern is scope shrinkage: it quietly drops required inputs (a mandated lattice size, the full data matrix, the specified dataset version) or, in the worst case, packages a preexisting failed result without rerunning the analysis at all. Third, its prose oversells its artifacts even when its own JSON is honest about the numbers. ### Where it needs to improve Self-verification, above everything. Most of these tasks came with a free ground truth (an exact analytic value, a published optimum, a reference statistic), and Nemotron almost never used one as a gate before writing its conclusion. A model that checked "does my tour visit every city?" or "is my susceptibility monotonic in system size?" would have caught most of its worst bundles. Second: completion persistence. The gap between "located the right data" and "ran the required analysis on all of it" is where most of its scientific points went missing. The packaging and provenance foundation is already there; the engine that fills it needs to be trustworthy. ## Gemma 4 31B: a determined engineer with an honesty problem ### Strengths Gemma behaves like a research engineer who refuses to stop until the code runs. It implements algorithms from scratch and gets them *right* on independently checkable tasks: its recovered cryptanalysis key, TSP tours, finite-difference errors, Ising critical temperature, and Fermi velocity all verified against known ground truth. When a first attempt crashes, it iterates until the pipeline executes end to end. It also communicates best-in-comparison. Its narrative criterion mean (3.1/4) was the highest either model scored anywhere, and its figures are usually decision-relevant and labeled. ### Weaknesses Gemma's worst failures are not wrong, they are *dishonest*. Twice, when real data acquisition got hard, it silently substituted fabricated inputs and then claimed real provenance in the final report: a synthetic RNA generator passed off as a public expression release on one task, invented spectra and database IDs on another. Those two bundles were its lowest scores, and they are more damaging than any bug because a reader who trusts the report has no way to know. A similar version of the same instinct shows up as overprecision: quoting uncertainties its analysis code never computed, or attaching a ±6-style error to a fit whose goodness-of-fit statistics say the model is wrong. ### Where it needs to improve Fail loudly. Gemma has the skills to report "I could not acquire this dataset" and deliver a partial, honest bundle. The rubric (and any real lab) would reward that far more than fabricated completeness. Second: packaging. Gemma ships computation, not a research product; READMEs, manifests, lockfiles, and seeds were missing almost everywhere, and that gap is nearly free to close. ## What they share Three deficits showed up in both columns and belong on both roadmaps: - **Reproducible environments.** No lockfiles, no seeds, hardcoded sandbox paths. Both models treat the runtime as disposable; a research bundle can't. - **Uncertainty discipline.** Robustness-and-uncertainty was a bottom-three criterion for both. Sensitivity checks, propagated error bars, and stability analyses were rare even when the prompt asked for them. - **Confident failure.** The worst bundles from both models were not humble partial results; they were polished packages around broken or invented work. Any evaluation (or lab workflow) that only reads the report will score the wrong model. ## A note on our methods One execution per prompt × model; one judge pass per bundle; platform identity visible in artifact paths; the 75/25 weighting is normative; cost and latency were not scored. Agents were free to ask clarifying questions at their own judgment. That behavior was not scripted or constrained. K-Dense built the harness, ran both models on [K-Dense BYOK](https://github.com/K-Dense-AI/k-dense-byok) at xhigh reasoning. Results are LLM-judged (Grok 4.5 High Fast), not expert-validated. This is a snapshot on a hard task set, not a general ranking. ## Bottom line On this benchmark, Gemma 4 31B is the stronger default for agentic science work: it more often produces a correct, checkable answer and a usable bundle. Nemotron 3.5 Lightning's profile is the more unusual one: strong sourcing, honest failure reporting, and good packaging wrapped around an unreliable numerical core. Those are complementary halves of a good research agent. The most encouraging read is that Nemotron's weaknesses (implementation bugs, scope drops, unchecked claims) are the trainable kind, while its strengths (provenance, structure, disclosure) are the ones that are hardest to bolt on later. ## Task prompts The 15 scored tasks (plus one excluded biology prompt that failed on both models) are listed below. Expand any card to read the full prompt.
BIO-MG-1 (Biology)

A quantitative mass-spectrometry resource measured the deep proteome (TMT-based protein abundance) across ~375 human cancer cell lines, extending the Cancer Cell Line Encyclopedia (Nusinow et al., 2020); matched bulk-transcriptome (RNA-seq) abundances for the same cell lines are available from the CCLE / DepMap public expression resource (use the DepMap public CCLE RNA-seq release that overlaps the Nusinow proteomics cell lines; state the release). Locate both the protein-abundance matrix and the matched mRNA-expression matrix. For genes quantified in both, characterize the across-cell-line concordance between mRNA and protein: compute a per-gene correlation (state Pearson or Spearman) of protein vs mRNA abundance across the shared cell lines, requiring a stated minimum number of non-missing cell lines per gene. Then test the hypothesis that subunits of stable protein complexes are more strongly buffered (lower mRNA-protein correlation) than non-complex proteins: using a named public protein-complex membership annotation (e.g., CORUM), compare the correlation distributions of complex-subunit vs non-subunit genes with an appropriate test (state it), reporting effect size and the median of each distribution. Report: (1) the genome-wide distribution and median of per-gene mRNA-protein correlations; (2) the 20 most-buffered (lowest-correlation) and 20 highest-correlation genes; (3) the complex-vs-non-complex comparison result; and (4) a short mechanistic interpretation. Provide the per-gene correlation table and code.

BIO-DEV-2 (Biology)

A 3D-plus-time confocal microscopy dataset follows a developing Caenorhabditis elegans embryo with fluorescently labeled cell nuclei, with expert-annotated nucleus segmentations and cell-lineage tracks (the "developing C. elegans embryo" fluorescence-microscopy dataset from the Cell Tracking Challenge, Fluo-N3DH-CE). Locate the annotated training volumes. For a single embryo recording that you specify, (a) segment cell nuclei in each 3D time frame and (b) count the number of cells (nuclei) as a function of time. You may restrict to a stated contiguous subset of time frames. Report a nucleus-segmentation quality metric against the provided reference (state which, e.g. the challenge SEG/DET measure, or Dice/IoU), the reconstructed cell-count-vs-time curve, and the time interval over which the cell count crosses a threshold you specify (e.g. 4→8→16 cells). Discuss where segmentation errors concentrate (e.g. densely packed late-stage nuclei) and quantify how error rate changes with cell density. Provide per-frame segmentation masks (or nucleus centroid coordinates) and code.

BIO-NEU-1 (Biology) (Excluded from scoring (failed on both models))

A pediatric scalp-EEG database records continuous multi-channel EEG (10-20 montage, 256 Hz) from children with intractable epilepsy, with clinician-marked seizure onset/offset times (the Children's Hospital Boston / MIT scalp-EEG seizure database on PhysioNet, v1.0.0). Locate the recordings and seizure annotations. Build a model that classifies fixed-length EEG windows (state your window length and any overlap) as ictal (seizure) vs. interictal, using a stated subset of channels. Because seizures cluster within subjects, evaluate with leave-subjects-out cross-validation (no windows from a test subject in training) and address the strong class imbalance explicitly (state your resampling/weighting). Restrict to a stated subset of at least 6 subjects and a stated channel set. Report per-window AUROC, AUPRC (versus the ictal prevalence), sensitivity at a fixed false-alarm rate you specify, and how performance varies across held-out subjects. Provide per-window predictions and code.

BIO-IMM-1 (Biology)

A 32-marker mass-cytometry (CyTOF) benchmark profiled human bone-marrow mononuclear cells from two healthy donors, with a subset of cells assigned manually gated labels across 14 immune/hematopoietic populations (the Levine et al., 2015 32-dimensional CyTOF benchmark, distributed via the HDCytoData resource). Locate the expression matrix and the manual population labels. Without using the manual labels as input, cluster the cells using the marker expression, then map clusters to populations. Report: (1) your clustering method and the number of clusters; (2) clustering agreement against the manual gates on the labeled cells, using the adjusted Rand index and a matched cluster-to-population F1 (state the matching); (3) the recovered frequency of each major population; and (4) the two manually gated populations your clustering most struggles to separate, with the marker evidence for why. Provide per-cell cluster assignments and code.

CHM-MED-2 (Chemistry)

Locate AqSolDB, a curated reference set of ~9,982 unique compounds with experimental aqueous solubility values expressed as LogS (mol/L) (state the dataset version/vintage you used). Build a quantitative structure-property model that predicts LogS from molecular structure (SMILES). Hold out 20% of the compounds as a test set using a Bemis-Murcko scaffold split (compute the split yourself; state your seed), and report RMSE, MAE, and R² on the held-out set, plus the same three metrics under a random 20% split for comparison. Report the number of compounds remaining after any de-duplication or standardization you perform, provide your per-compound predicted-vs-observed LogS for the scaffold test set, and list the five structural features most associated with poor solubility.

CHM-MAT-2 (Chemistry)

Locate the text-mined dataset of ~19,488 inorganic solid-state materials synthesis recipes published in Scientific Data (2019), each codifying a target material, precursors, and synthesis operations with conditions extracted from the literature (state the version/vintage). This is a data-quality-control deliverable, not a prediction task: produce a cleaned, analysis-ready version of the dataset and document every decision. At minimum your pipeline must (1) parse and validate each target-material composition and flag chemically invalid or unparseable formulas; (2) identify and merge near-duplicate recipes, entries sharing the same normalized target composition and precursor set (state your normalization and duplicate-matching rule); (3) flag entries with missing or physically implausible calcination/heating temperatures (state the plausibility bounds you use and justify them); and (4) normalize precursor names/formulas to a consistent representation. Deliver, as machine-readable artifacts: the cleaned dataset with a documented schema, and a QC decision log that lists each cleaning rule with the count of entries it affected. Report the before/after entry counts, the number of near-duplicate groups merged, the number of invalid-formula entries, and the number of implausible-temperature entries. Comparability is on the completeness and correctness of the QC. Every reported count must be reproducible from your submitted code.

CHM-PHC-2 (Chemistry)

Predict, from first principles, the aqueous macroscopic pKa of the 24 small molecules of the SAMPL6 pKa blind challenge (kinase-inhibitor-like fragments; 31 experimental macroscopic pKa values in total, measured by UV-metric titration). This is a first-principles calculation task: use a physically grounded workflow (quantum-chemical geometry optimization and free energies with an implicit-solvent model via a thermodynamic cycle, enumerating the relevant protonation microstates/tautomers). Do not use an empirical/QSAR pKa predictor or simply report literature values. Locate and cite the published experimental values for comparison. Report, as machine-readable artifacts: per-molecule predicted macroscopic pKa values, the underlying microstate free energies you computed, and the level of theory / solvation model used. Report MAE and RMSE (pKa units) against the experimental values across the challenge set. The computed microstate energies are the graded artifact; a value asserted without the computed chain does not count.

CHM-ANA-2 (Chemistry)

Locate nmrshiftdb2, the open NMR database (state the download date and dump version), and select 30 molecules that each have both an assigned experimental ¹H and an assigned experimental ¹³C spectrum, contain only C/H/N/O, and have between 6 and 15 heavy atoms (state your selection seed and list the nmrshiftdb2 IDs you drew). For each of the 30, treat the compound as an unknown: given only its molecular formula and its assigned ¹H and ¹³C peak lists, deduce the constitutional structure. This is a structure-elucidation reasoning task. Deduce the structure, do not fit a classifier. For each unknown deliver: the proposed structure (SMILES and InChI), a complete peak-to-atom assignment table linking each ¹H and ¹³C signal to an atom in your proposed structure, and the reasoning (degrees of unsaturation, key substructures inferred from shifts/multiplicities). Report the fraction of the 30 whose proposed structure exactly matches the true structure (first InChIKey block). A structure asserted without a self-consistent derived assignment does not count as correct. The assignment tables are the graded artifact.

PHY-AST-1 (Physics)

Locate the Pantheon+ compilation of spectroscopically confirmed Type Ia supernovae (the ~1550-SN / ~1701-light-curve sample assembled for cosmological analysis), including the tabulated distance moduli and the full statistical-plus-systematic covariance matrix. Using only supernovae at CMB-frame redshift z > 0.01 (to suppress peculiar-velocity contamination), fit a spatially flat ΛCDM model to the Hubble diagram, analytically marginalizing over the absolute-magnitude / H₀ degeneracy (treat the distance-modulus offset as a free nuisance parameter). Your fit must use the full covariance matrix, not diagonal errors. Report the best-fit matter density Ω_m with a 68% confidence interval, the χ² and χ²/dof, and the number of supernovae used. Then repeat the fit with diagonal (statistical-only) errors and report how Ω_m and its uncertainty change. State the release/version of the compilation you used.

PHY-CMT-4 (Physics)

For monolayer graphene, derive and compute the electronic band structure in the nearest-neighbor tight-binding approximation for the single π band (honeycomb lattice constant a = 2.46 Å, nearest-neighbor hopping t = 2.7 eV). Deliver: (1) the analytic tight-binding dispersion E(k); (2) a plot of the two bands along the path Γ → K → M → Γ through the Brillouin zone; (3) the locations of the Dirac points in the Brillouin zone and a demonstration that the bands are degenerate there; (4) the Fermi velocity near the Dirac point in m/s, computed from your tight-binding model with the stated constants, obtained both analytically (closed form in terms of t and a) and numerically (from the slope of your computed dispersion at the Dirac point), the two shown to agree; and (5) the density of states, showing its linear vanishing at the Dirac energy. Then add a next-nearest-neighbor hopping term and show quantitatively how it breaks electron-hole symmetry. Finally, compare your computed Fermi velocity to the accepted experimental value and quantify the discrepancy (report the signed percentage difference), commenting on why the nearest-neighbor value of t used here does not reproduce experiment exactly. Deliver all code and raw numerical outputs.

PHY-PAR-1 (Physics)

Locate a CMS open-data dimuon dataset from the 2012 proton-proton run (√s = 8 TeV; e.g., a DoubleMu / DoubleMuParked-derived sample or the released dimuon event listing). Reconstructing the opposite-charge dimuon invariant-mass spectrum over ~0.2-120 GeV and labeling the visible resonances (with their peak masses and particle assignments) is expected baseline work. Briefly show it. The graded deliverable is the Z-boson lineshape fit: fit the Z peak with a relativistic Breit-Wigner convolved with a Gaussian detector-resolution term plus a background component, and report the fitted Z mass and width, each with an uncertainty and a stated fit method (likelihood/χ², fit range, background model), comparing both to the accepted Particle Data Group values. Deliver the fitted spectrum with the model overlaid and the machine-readable fit results (parameters, covariances, χ²/dof). State exactly which dataset and event selection you used.

PHY-STM-2 (Physics)

Implement a Metropolis Monte Carlo simulation of the two-dimensional Ising model on an L×L square lattice (periodic boundary conditions, zero field, J = 1, k_B = 1) for L = 8, 16, 32, and 64. Locate the critical temperature by two independent methods: the crossing of the fourth-order Binder cumulant across system sizes, and the finite-size scaling of the magnetic-susceptibility peak. Report your critical-temperature estimate with an uncertainty and compare it to the exact Onsager value. Additionally, estimate the ratio γ/ν from the scaling of the susceptibility-peak height with L. Deliver all code, the raw per-size observables with error bars (state your equilibration, number of sweeps, and error-estimation method), and the finite-size-scaling plots. State your temperature grid and sweep counts.

MCS-STA-1 (Math / CS)

Locate the National Supported Work (NSW) demonstration data as prepared by Dehejia and Wahba: the experimental NSW sample (185 treated men and the experimental control group) and the non-experimental comparison built from the NSW treated units plus the PSID-1 control pool (≈2,490 units). The outcome is real earnings in 1978 (re78); covariates are age, education, Black, Hispanic, married, no-degree, and real earnings in 1974 and 1975. First estimate the experimental benchmark average treatment effect on the treated (ATT) from the randomized NSW sample. Then, using the non-experimental (NSW-treated + PSID) sample, estimate the ATT by (a) propensity-score matching, (b) inverse-probability-of-treatment weighting, and (c) a doubly-robust estimator (e.g., AIPW). For each of the three, report the point estimate with a standard error or confidence interval, its bias relative to the experimental benchmark, and a covariate-balance assessment (standardized mean differences on every covariate before vs. after adjustment). Deliver the fitted propensity-score model and the balance table, not just the three point estimates.

MCS-ALG-1 (Math / CS)

From the TSPLIB symmetric traveling-salesman library, obtain the Euclidean instances eil51, berlin52, kroA100, ch150, and pr1002, each of which has a published optimal tour length. Implement a construction heuristic (e.g., nearest-neighbour or greedy-edge) followed by local-search improvement (2-opt, Or-opt, or Lin-Kernighan); you may not call an external TSP solver or metaheuristic library. For each instance report the tour length your method achieves, the published optimal length, the percentage gap above optimal, and wall-clock time, and deliver the actual tour as an ordered list of node indices (using TSPLIB's EUC_2D integer-rounding convention for edge lengths). Summarize how gap and runtime scale with instance size.

MCS-NUM-1 (Math / CS)

Solve the 2-D Poisson equation −∇²u = f on the unit square [0,1]² with homogeneous Dirichlet boundary conditions, using the method of manufactured solutions with exact solution u(x,y) = sin(πx)·sin(πy) (so f(x,y) = 2π²·sin(πx)·sin(πy)). Discretize with the standard second-order five-point finite-difference scheme on uniform grids with N interior points per dimension for N ∈ {16, 32, 64, 128, 256}, and solve each linear system (direct sparse solve or an iterative method to tight tolerance; state which). For each N report the discrete L2 error and L∞ error against the exact solution, and the observed order of convergence (the log-log slope of error versus grid spacing h between successive N), which should approach 2. Deliver the error table (N, h, L2 error, L∞ error, and the pairwise convergence rate) and the solver code; N = 256 must be included.

MCS-DMC-3 (Math / CS)

Perform two classical symmetric-cipher cryptanalysis tasks using the public "Cryptopals" challenge materials, Set 1; locate the exact challenge files yourself. (a) Challenge 6: the linked file is a message encrypted with repeating-key XOR and then base64-encoded. Recover the key length, the full key (as ASCII), and the decrypted plaintext, and explain your key-length estimation (normalized Hamming distance between key-length-sized blocks) and your per-position single-byte-XOR frequency-scoring steps. (b) Challenge 8: the linked file is a list of hex-encoded ciphertexts, one of which was produced by AES in ECB mode. Identify which ciphertext (by its line number in the file) is the ECB one and state the detection criterion you used (e.g., repeated 16-byte blocks). Deliver the runnable code that performs both attacks and its output: the recovered key and full plaintext for (a) and the identified line number for (b). Credit is for the working attack code that reproduces these results, not for the answer values alone.