Golden Fish · First Notes

Golden Fish began with a narrow request. We wanted a model that could receive state and context, a typed question and the candidates valid for that request, then return scores or probabilities without writing a paragraph first. That interface covers routing and policy decisions. It also covers tool selection, risk scoring, code and agent decisions, preference ranking, binary contextual choices and ordinal scoring. The restriction is deliberate. It gives us something fast enough to measure as a decision system instead of treating text generation as the default answer to every machine facing problem.

System One is the operational name we use for that path. We did not invent the concept or the models. Jev at TypeSafe AI pioneered this direction, and Golden Fish uses the term in that existing sense. It means a direct forward pass with no autoregressive decode, no chain of thought requirement and no token by token conversation. The output is typed and probabilistic. The name is not a claim about human cognition or a claim that the model is intuitively wise. It describes an engineering choice. If the job is to choose an action from a small candidate set, generating an explanation before choosing it can be an expensive detour.

The question is how far this can go at roughly six hundred million parameters. Golden Fish is fast and reasonably capable in the current measurements, but the training recipe has not reached the accuracy we want. That gap is the subject of this note. The rest of the article follows the attempts to explain it, including the runs that made the explanation less comfortable.

That target pulls the architecture in a specific direction. The backbone is BAAI/bge-m3 with approximately 576 million parameters, used as a bidirectional encoder. The state and typed question share a representation. Candidates receive their own representations and those representations can be cached when the inventory is stable. We normalize the embeddings before scoring, apply a learned temperature and send a small top k set through a lightweight set or listwise refinement layer. The current implementation uses refine_top_k = 8. The result is a probability distribution over the request candidates after the typed decision primitive has been applied.

The first pass therefore asks a compatibility question. How well does the shared state representation match each candidate representation. A useful simplified view is

p_i = exp(s_i / τ) / Σ_j exp(s_j / τ)

Here τ is learned and the candidate index belongs to the request. The refinement step then sees the small surviving set together instead of treating every score as independent. That matters for ranking and for cases where one candidate changes the meaning of another. The model still has one direct decision path. There is no autoregressive decode hiding behind a friendly API name.

The parameter budget is intentional because latency is part of the model design. Previous measurements on an A100 showed roughly 29 to 32 milliseconds for uncached BF16 inference and roughly 15 to 16 milliseconds when candidate representations were cached. Those are measurements from one implementation and one hardware setup, not universal latency guarantees. They still answer the first engineering question. Caching candidates can remove a meaningful part of request time without changing what the model is asked to decide.

A single classifier head per dataset would make the first result easier to organize and the later system harder to use. Golden Fish instead shares a decision representation and uses typed primitives to state what relation is being measured. The current families are commonsense_mcqa for multiple choice reasoning, contextual_binary for state dependent binary choices, entailment for whether a claim follows from context, intent_routing for intent selection, ordinal_score for ordered scores, policy_plan for an allowed action or plan and ranking_preference for preference decisions. The common representation is the bet that these tasks share enough structure to train together.

That bet only works if candidates remain part of the request. The model does not need one closed label vocabulary for every deployment. A caller can provide the valid choices for an action menu, a tool set, a policy branch or a temporary routing inventory. The flexibility has a cost. Candidate encoding and candidate retrieval become part of accuracy, and a good scorer cannot choose a candidate that never reaches refinement. This is why the first useful baseline has to measure more than the final top one answer.

G1 was the first serious test of the premise. It used BAAI/bge-m3 with approximately 576.2 million parameters. The split contained 202,871 training rows, 10,547 calibration rows, 12,859 development rows, 12,905 test rows and 3,996 gauntlet rows. The final result reached 78.98 percent development accuracy and 78.88 percent test accuracy. Test macro family accuracy was 75.57 percent. Negative log likelihood was 0.5567. Expected calibration error was 2.41 percent. Gauntlet accuracy was 45.77 percent. The premise survived the first test. The score did not mean the training problem was understood.

G1 was competent enough to make the remaining failures worth studying. Its sampler drift changed the effective family and language mixture. Its out of distribution performance was weak. Checkpoint selection was fragile and calibration behaved differently across slices. An INT8 export did not preserve enough quality to make the latency trade obvious. Small fast development subsets made the run look cleaner than the full evaluation. The important observation was not that one component was broken. It was that the headline score could stay healthy while the experiment quietly changed underneath it.

G2 was a response to that ambiguity rather than an attempt to make the model larger. We changed family and language sampling, made typed losses explicit, added replay and hard example mining, separated checkpoint roles and selected against the full development set. Forgetting guards and rollback were added because a later stage should not be allowed to erase a capability without making that loss visible. Gradient diagnostics, per pool diagnostics and stronger leakage and dedup auditing served the same purpose. Before asking for a better score, we wanted a run whose score meant the same thing on Tuesday and on Friday.

The strongest Stage 2 result reached 76.73 percent full development accuracy, 74.11 percent macro family accuracy, 0.5903 negative log likelihood, 3.27 percent expected calibration error and 41.87 percent on the mini gauntlet. The headline number was lower than G1. That was not automatically a regression because G2 had changed the evaluation surface and exposed more of the tradeoff. The result was still a warning. Better bookkeeping had not yet produced a better model.

Then we found a mistake that changed how the next stages should be interpreted. An old selection rule chose the weaker Stage 2 step 8000 checkpoint even though step 16000 was substantially stronger overall. The selected checkpoint became the starting point for later stages, so a local bookkeeping error became the prior for a new experiment. The model did exactly what the training loop asked. The selection rule had asked for the wrong model.

The next question was whether specialization could recover the weak slices without spending the general model. Stage 3 did improve some out of distribution slices, but full development fell to about 68.48 percent and macro family accuracy to about 68.00 percent. Stage 4 reached about 70.80 percent on full development and improved some hard negative, NOTA and long context slices, while entailment and ranking deteriorated badly. The expected trade was small. The observed trade was a collapse in general capability. A model that gets better at hard examples while getting worse overall is not progressing. It is spending one ability to buy another.

Our first explanation was sampler imbalance. The logs made that explanation too small. The run also started from the stale checkpoint, specialized sequentially, used insufficient replay and produced very large gradients. Nearly every update was clipped. Several losses were pulling on the same parameters, and row count was a poor proxy for useful signal. The stage boundaries did not create the interference. They made it visible.

At that point the training loop was not just an implementation detail. It was preventing us from telling which explanation was true. We added explicit stage initialization and separate checkpoint identities for best_id, best_ood and best_composite. Stage final checkpoints are saved even when another checkpoint wins one metric. Forgetting guards can stop a stage and rollback can return to a known state. We also log gradient contribution by loss, by data pool and by dataset window. The goal is not a busy dashboard. It is a trace of what actually moved the weights.

The new logs made the optimization failure concrete. Before the changes, clipping affected nearly 100 percent of optimizer updates. Pre clip norms were sometimes in the 20 to 80 plus range, and individual spikes approached or exceeded 100. After adaptive clipping was added, Stage 4 clipping fell to roughly 8 to 11 percent with an adaptive threshold around 40 to 47. This does not establish that the later threshold is right for every run. It establishes that the optimizer stopped leaning against the clip boundary on every step.

The correction improved interpretability before it improved accuracy. Fixing optimization stability did not magically raise the score. A stable run can still optimize the wrong objective. The clean gradient trace was not a result by itself. It was permission to run the next experiment without guessing whether the optimizer was fighting the logging system.

The same instrumentation changed how we thought about data volume. Agentic examples often reached 93 to 99 percent training accuracy with very high gold probability and almost zero gradient contribution. The model appeared to have mostly solved that pool, so sampling it heavily spent rows without spending much useful optimization signal. Row percentage is therefore not a training utility measure. A large pool can be nearly silent while a small pool supplies the updates that matter.

We used that view to mine errors across approximately 180,000 training rows. The pass found 829 confidently wrong rows, 28,956 wrong rows, 12,334 near misses, 130,272 correct rows and 7,609 correct but uncertain rows. The mined set contained 49,728 rows. Those counts map the model's behavior. They do not certify every selected row as a useful training example.

The mining pass changed the question from how many errors we could find to which errors were learnable. Hard mining from a dataset where the model is already close to random may not produce useful hard examples. It may produce random failures with a convincing score attached. That is a hypothesis, not a conclusion. A controlled comparison between mined rows, uniform replay and rows selected with an external signal is still needed. The table is evidence about behavior. It is not yet a recipe.

The short remediation pilot was the cheapest way to test whether the mined rows were useful. Its mixture was approximately 40 percent replay, 42.5 percent remediation rows, 10 percent NOTA rows and 7.5 percent agentic rows. It reached about 76.52 percent full development accuracy, about 73.81 percent macro family accuracy, about 0.6068 negative log likelihood, about 2.97 percent expected calibration error and about 41.80 percent on the mini gauntlet. Per family it reached 44.62 percent on commonsense_mcqa, 75.06 percent on contextual_binary, 74.07 percent on entailment, 85.68 percent on intent_routing, 78.80 percent on ordinal_score, 97.08 percent on policy_plan and 61.38 percent on ranking_preference.

The result was effectively flat against the strongest Stage 2 checkpoint. A difference of a few tenths is not meaningful without confidence intervals and repeated seeds. The pilot did not show that remediation works. It showed that this mixture did not obviously repair the observed failures in a short run. That is a useful null result because it removes one easy story without replacing it with a better one.

NOTA then became the clearest example of a formulation problem. It means none of the above, but it did not behave like an ordinary semantic candidate. NOTA training accuracy was often around 10 to 20 percent. Gold probability stayed around 0.18 to 0.20. Its gradient contribution was disproportionately large while held out NOTA remained extremely weak. More optimization pressure did not solve the formulation. It mostly made the formulation louder.

The leading hypothesis is that NOTA describes the candidate set rather than an item inside it. In an A B or C choice, the gold item is one of the objects being compared. None of the above says that the supplied set does not contain the right object or that the set fails a condition. Treating NOTA as one more semantic embedding may ask the encoder to represent a set level relation as an item level relation. The logs are consistent with that explanation. They do not prove it.

The planned test changes the output object rather than adding more pressure to the same one. It uses an explicit null or abstention logit instead of a semantic candidate, plus synthetic drop gold examples where the correct candidate is removed. Selective prediction metrics would measure whether the model knows when to abstain. The point is to separate a bad representation from a bad optimizer. It is too early to say that a null logit will work.

NOTA also pushed us toward a broader hypothesis. Training accuracy estimated from the mining pass is not dramatically above development accuracy. That is not a clean train evaluation and it may be misleading. If the gap is real, high bias or underfitting is more plausible than classic overfitting. Several causes remain open. The candidate scoring formulation may leave capacity unused. Multi task gradients may interfere. Effective optimization may be insufficient even with a large nominal budget. Candidate counts may differ between training and inference. The top eight limit may hide a retrieval bottleneck. Losses may be regularizing before the base task is learned. Lower encoder layers appear to move very little in some windows. The data itself may simply be difficult for this architecture.

Banking77, MNLI, SNLI and BoolQ can separate those explanations. They ask different versions of a basic question. Can the model distinguish close intents, follow entailment under lexical variation and handle ordinary contextual binary decisions. If isolated models solve these tasks while the shared model does not, multi task interference becomes a strong suspect. If isolated models also fail to memorize a fixed train subset, the pressure moves toward formulation, optimization or capacity.

The refine_top_k = 8 setting makes candidate retrieval a first class question. In Banking77, CLINC150, high_k_intent and other high K intent tasks, the gold candidate must survive the first score pass. If it misses the top eight, listwise refinement cannot recover it. The refiner may be receiving blame for errors that happen before it runs. That is a hard ceiling, not a subtle failure.

Before changing the refiner, we should measure Recall at 1, Recall at 4, Recall at 8, Recall at 16, Recall at 32 and Recall at all for every dataset. A compact definition is

R at k = mean of the indicator that rank of the gold candidate is at most k

We also need accuracy conditioned on the gold candidate surviving the top k and accuracy conditioned on the gold candidate not surviving the top k. The first tells us what the scorer and refiner can do when the candidate is available. The second tells us how much final error was already decided by retrieval. This experiment can tell us whether the next improvement belongs in scoring or in refinement.

The loss list also grew faster than the evidence. It now includes cross entropy, contrastive loss, pairwise loss, ordinal loss, score loss, anchor loss, consistency loss, Brier loss and a confidence penalty. Gradient logs suggest that only a subset contributes materially in most windows. Cross entropy usually dominates. Pairwise and anchor matter. Brier contributes modestly. Contrastive, score and consistency often contribute very little. The uncomfortable question is whether we are regularizing a model that has not learned the base task well enough.

That is why the next ablations are deliberately plain. CE only tests the base task. CE plus pairwise tests whether ranking signal helps. CE plus ordinal applies the extra structure where it belongs. The current full objective tells us what the loss zoo buys in aggregate. CE with teacher distillation asks a different question about the supervision itself. The point is to improve the decision function, not to make the run more difficult to read.

Teacher distillation is a major next experiment and is not implemented in the result described here. The current label is usually one hot. The gold candidate receives one and every other candidate receives zero. An offline teacher could instead provide a distribution such as A with 0.71, B with 0.22 and C with 0.07. The student would combine gold label cross entropy with a KL term against the teacher distribution at a chosen temperature.

L = CE gold label + λτ² KL teacher distribution at τ against student distribution at τ

The reason to try it is that a distribution contains relationships a hard label discards. It can preserve ambiguity, expose possible label errors and give a low capacity student structure that is absent from the one hot target. We would start with entailment, contextual binary, intent routing and other typed decisions, then consider commonsense only if teacher quality is high enough. The risk is ordinary but serious. Teacher errors and teacher overconfidence can be distilled too. A soft target is not automatically a better target.

The data pipeline changed for the same reason the instrumentation changed. We needed to know whether a score reflected learning or repeated exposure. There were about 363,000 initially valid rows. After duplicate and template controls there were about 303,000. The current final training split has about 255,800 rows, with about 14,800 development rows, about 20,900 test rows, about 11,900 calibration rows and about 6,000 gauntlet rows. The controls remove duplicate candidates, exact duplicates that differ only in candidate order and template families that exceed their caps. Train, validation and test leakage audits now run explicitly. Gauntlet collision checks are part of the pipeline.

Cleaning the split did not answer the accuracy question by itself. Deduplication is hygiene, not an automatic improvement. Removing repeated templates can reduce memorization and can also remove useful coverage if the controls are too aggressive. The new counts make the data easier to reason about. They do not tell us whether the remaining data is sufficient or whether the formulation is the real limit.

The next phase is a sequence of small experiments rather than another long run with a new name. E0 measures variance so we know which accuracy delta is believable. E1 measures single task ceilings on Banking77, MNLI, SNLI, BoolQ and CommonsenseQA. If isolated models solve these tasks with the same backbone while the shared model does not, multi task interference becomes a concrete target. If they do not, the model may be underfitting or the candidate formulation may be wrong for the task.

The roadmap is a decision procedure. If train accuracy is close to development accuracy, optimization and formulation come first. If train accuracy is much higher, generalization and data become the leading problem. If a single task is much stronger than the shared model, test interference. If Recall at 8 is low, test the candidate pipeline before changing the refiner. If distillation helps, the supervision signal was probably too thin for the student. These branches are simple on purpose. They turn the next run into a test of a belief instead of another month spent interpreting one aggregate score.

Golden Fish has demonstrated that a sub billion bidirectional model can make useful typed decisions quickly. It has not demonstrated that the current training recipe can reach the desired accuracy. We also know that several explanations were incomplete. Stable optimization did not remove the plateau. Remediation was effectively null. NOTA receives gradient without becoming good, and candidate retrieval may set a ceiling before refinement begins. The remaining leverage is in optimization, multi task interference, candidate formulation, supervision quality or genuine capacity. The next experiments are designed to separate those possibilities.