The first model finished training on 30 September, the same day I had the idea. It had 30 million parameters, it had read 629 million tokens of FineWeb-Edu on a free Colab T4, and it took about four and a half hours. One of the sentences it wrote was "The capital of Pakistan is Bangladesh."
Grammatically perfect. Wrong about the country I live in.
That sentence ended up being the whole project.
The model does not need to know the world
What I wanted was a tiny language model that answers everyday questions by reading search results, small enough for phones, watches and a Raspberry Pi. The app runs the search and passes the question and the results in. The model replies with a few words, or exactly not found when the results do not hold the answer. It does not search and it is not a chatbot.
Language comes from the model, facts come from search. My 30M model had learned English and not much that was true, and for this job that is fine. It never has to know the capital of Pakistan. It has to read the page.
Then I tuned it on SQuAD 2.0, a reading test where some questions have no answer in the passage. It did read. It scored 43.14 F1 on the answerable questions, where a baseline that returns the first sentence scores 10.2. But on the unanswerable ones it made up an answer 37% of the time.
It had learned to find answers. It had not learned that sometimes there is none.
Two days of NaN
Over 1 and 2 October, training broke in four different ways.
I re-ran the 30M on Kaggle in fp16, and from step 11,000 the validation loss was NaN. The strange part was the speed. It went up, from 41k to 50k tokens a second, because every step was being skipped. The weights were frozen, and the speed counter had never looked better. The first fix was to stop on a non-finite loss instead of saving it.
Then I built the 60M, and its attention collapsed. In all 18 layers the largest attention logits were between 260 and 750, and attention entropy was 0.05 to 0.27 nats. That is one-hot: every head staring at a single token. A healthy 30M sat at logits of 20 to 45 and entropy of 2 to 3 nats. The fix was QK-norm, an RMSNorm on the queries and keys of each head, as in Qwen3 and OLMo 2.
The long 60M run on Kaggle, also fp16, hit NaN at step 12,750. Muon without weight decay lets every weight matrix keep growing. The old 30M, after 19k steps, had weights 18 to 217 times their starting size and a residual stream peaking at 15,888. The largest number fp16 can hold is 65,504. Sooner or later one MLP output passes it. The fix for both fp16 runs was decoupled weight decay on Muon, 0.01 on the 60M.
The fourth was not maths: my Colab runtime died about 30 minutes after my Mac went to sleep. I had upgraded to Pro+ shortly before, but that runtime was started under the old plan. A fresh one started under Pro+ finished with the Mac closed and the WiFi off.
The loss looked normal right up to the step it went NaN.
One clean pretraining run, then the job
With QK-norm in, the 60M beat the 30M on the same 629M tokens and tune, 56.82 F1 on SQuAD against 53.70, so one 60M model would cover every device. A 3B-token run reached 66.56 F1. SmolLM2-135M scores 75.53 on the same test.
The real base read 10 billion tokens of FineWeb-Edu on one RTX PRO 6000, with Muon plus AdamW, in bf16. It ran 10 hours 35 minutes with zero restarts, used about 95 compute units, and finished at validation loss 2.7127. For the last 30% of training, while the learning rate came down, I mixed in search-reading examples. The finished model has 62,018,304 parameters, so I call it 62M here, though the repo name says 60M. It loads as Qwen3 in transformers.
Search results are messier than SQuAD paragraphs, so I collected 213,509 real rows: MS MARCO (real Bing queries with web passages), TriviaQA with Bing results, SQuAD 2.0, and hard negatives. A hard negative is an answerable row with the result that holds the answer swapped out, so the right reply becomes not found.
None of that is laid out like a Google results page, so I also generated Google-style rows with Qwen3.6-35B-A3B on vLLM. Python picks the facts that decide each answer. The big model writes result text around them. A second pass of the same model then answers from the results alone, and I keep the row only when both agree.
After a first round on fewer real rows, the model got 79.2% of my 24-question dev set right. The SQuAD-only model got 50.0%. The released version trained on 258,542 examples for two epochs. But 24 questions is far too few to trust.
A Google test I could not argue with
I had already tried a small real-Google check: 25 questions on the SQuAD-only 60M. Every answer was copied from the results. It also answered a Fahrenheit question with "100 °C", gave a date when asked how tall K2 is, and answered a trick question about Lahore hosting the Olympics with "1948". It could read. It did not know what kind of answer a question wanted.
So I built a proper set from real Google results, with a slow script in my own Chrome that ran the searches one at a time. After about 38 searches in a row Google started answering with a bot check. I stopped searching from my browser there and paid a scraping service, Apify, for the rest: 172 searches for under half a dollar.
For the biggest set, two AI labellers wrote the gold answers independently and a third settled the disagreements. I dropped 35 questions whose top results did not hold an answer, and a few more that were unclear or matched my training rows. With the three smaller sets I already had, that made 258 questions in four sets of 24, 53, 19 and 162. About a quarter of them, 65, are about Pakistan. 50 of the 258 are trick questions where the right reply is not found, like "When did Pakistan win the FIFA World Cup?"
Ten billion tokens moved the loss more than the score
| Version | What changed | All 258 |
|---|---|---|
| v1.1a | More SQuAD, harder trick questions, unit questions, long numbers | 69.4% (179) |
| v2a | Same job data on the 10B-token base | 73.3% (189) |
| v3a | Synthetic "latest / current / most recent" rows | 77.1% (199) |
| v3.1a, released | Rebalanced: more answer-type, hard-trap and unit rows | 79.8% (206) |
v2a is the one I have to be honest about. Ten billion tokens of pretraining moved validation loss from 2.79 to 2.71. On the 77 questions I had at the time, v1.1a got 60 right and v2a got 59. A tie. On the full 258, which I only had later, v2a does come out ahead, 189 to 179, so I will not pretend the base did nothing.
Data aimed at one weakness did move the score. v3a added synthetic rows about the latest, current or most recent something, built around invented entities. On the 60 questions of that kind, v2a got 41.7% and v3a got 65.0%, McNemar p = 0.001.
They also broke something. v3a started copying dates where a value belonged: asked for the boiling point of mercury in Celsius, it answered "01-Jun-2018". v3.1a rebalanced the mix. It reached 79.8% overall, and the latest questions slipped back to 61.7%, 37 of 60.
What I released on 4 October
I published v3.1a as tiny-search-reader-60M on Hugging Face, under Apache-2.0. It gets 206 of 258 right, so about one answer in five is wrong, and the misses are not spread evenly. On the 162-question set:
| Kind of question | Right |
|---|---|
| Fact | 30 / 34 |
| Live value | 29 / 35 |
Trick, should say not found | 24 / 28 |
| Unit | 21 / 25 |
| Most recent winner | 14 / 26 |
| Latest version | 6 / 14 |
On made-up results that only cover 2025 and 2026, "Who won the 2031 Harbor Street Chess Open?" gets not found. Give it only two of those results, though, and it answers with the 2026 winner.
70 MB, and about 5x faster than Qwen3-0.6B
I converted it to GGUF at Q8_0, keeping the embedding at F16. The file is 70.2 MB and gets the same 206 of 258, with all 258 verdicts identical to the full model. Q4_K_M is 41 MB and gets 205. On my M1 Pro, CPU only, with llama.cpp on 4 threads, an average test question takes 168 ms and peaks at 174 MB of memory.
Against Qwen3-0.6B, both at Q8_0 on 4 threads:
| Mine | Qwen3-0.6B | |
|---|---|---|
| Reading the prompt | 1,900 tokens/s | 340 tokens/s |
| Writing (llama-bench) | 561 tokens/s | 132 tokens/s |
| 449-token prompt, 24-token answer | 0.28 s | 1.50 s |
| File size | 70 MB | 634 MB |
llama-bench writes from an empty context, so both writing numbers are higher than when the answer follows a long prompt.
Nothing has been measured on a real phone yet, so any phone speed I gave here would be a guess.
Ten models, the same 258 questions
Then I ran nine other models on the same 258 questions and grading.
| Model | How | Parameters | All 258 | Trick /50 | Latest /60 |
|---|---|---|---|---|---|
| Qwen3-4B | zero-shot | 4.02B | 236 (91.5%) | 44 | 52 |
| Qwen3-1.7B | zero-shot | 1.72B | 218 (84.5%) | 26 | 54 |
| SmolLM2-135M | fine-tuned on my rows | 135M | 214 (82.9%) | 32 | 47 |
| Qwen3-0.6B | zero-shot | 596M | 207 (80.2%) | 30 | 45 |
| Mine | trained for this | 62M | 206 (79.8%) | 41 | 37 |
| RoBERTa-base SQuAD2 | extractive | 124M | 184 (71.3%) | 29 | 39 |
| MiniLM SQuAD2 | extractive | 33M | 179 (69.4%) | 27 | 35 |
| SmolLM2-360M-Instruct | zero-shot | 362M | 170 (65.9%) | 0 | 44 |
| Qwen2.5-0.5B-Instruct | zero-shot | 494M | 168 (65.1%) | 0 | 40 |
| SmolLM2-135M-Instruct | zero-shot | 135M | 154 (59.7%) | 0 | 40 |
Mine ties Qwen3-0.6B, 206 against 207, at a tenth of its size. On the questions where they disagree, 36 went my way and 37 went Qwen's, p = 1.000. It catches 41 trick questions, more than any model here except Qwen3-4B. The three instruct models under 0.6B never once replied not found, even when the prompt told them to.
Qwen3-4B is better. It got 236, with 42 questions only it got right against 12 only mine got, p < 0.001. And my 37 of 60 on the latest questions is below every generative model I tested.
The caveats: every zero-shot model got the same fixed prompt, which may not suit it, and my dev set and question types lean slightly in my favour. Some training rows were also written after I had seen mistakes on the test, so it is not fully blind.
A model trained on my data scored higher than mine
I took SmolLM2-135M and fine-tuned it on my exact training rows. It scored 214 to my 206. Across all 258 that is not significant: 33 questions only it got right, 25 only mine, p = 0.358. But on the latest questions it was significantly better, 47 against 37, p = 0.021. It was worse on trick questions, 32 against my 41.
It is 2.2x my size, and its base was pretrained on about 2 trillion tokens, 200x my 10 billion. The same rows did more on a better-pretrained base.
What I would tell myself on 30 September
The data did most of the work. Ten billion tokens of pretraining moved the loss and bought a tie on the 77 questions I had then. Rows aimed at one weakness, the latest questions, took them from 41.7% to 65.0%.
not found has to be trained in. Telling a model to say it is not enough.
Check the attention logits and the size of the weights before trusting a long run in fp16.
And publish the result you did not want. The easiest way to make this post look better was to leave out the SmolLM2 row. It taught me more than the tie did.
Next is more pretraining or a stronger base, and the latest questions, where this model is weakest. The model is on Hugging Face. Four days after the Bangladesh sentence, it reads the page right four times out of five. The fifth is the next problem.