A tiny model that learned SQL from a score, not from answers.
A 0.5-billion-parameter model was trained with reinforcement learning (GRPO). It was never shown a correct query: it wrote several attempts, each was run on a real database, and it was told only whether the rows were right. Ask a question and compare it with its untrained self and a frontier model.
Show the tables
Small model, untrained
Qwen2.5-0.5B as downloaded
Small model, GRPO-trained
Same weights + the trained adapter
Frontier model
DeepSeek V4.1 Flash over its API
Measured results
Execution accuracy: the share of test questions where the model's query returns the right rows.
Databases seen in training (new questions)
Databases never seen in training
Training curve (GRPO)
Left to right: training steps. Line: share of sampled queries that returned the right rows, averaged over 25 steps. Dots: accuracy on a held-back development set, with greedy decoding.
Details
What the numbers say
The first run cheated
What the model learned to write
The first scoring rule accepted any subset of the reference columns. One easy column with the right number of rows got full reward.
The same run, two scoring rules
Fix: every question now lists the columns it really asks for, and an answer must contain all of them. The results above come from a new training run with the fixed reward.
How it works
GRPO (Group Relative Policy Optimization) is the reinforcement-learning method behind DeepSeek-R1. The training loop here is written from scratch in PyTorch, about 150 lines, and ran on a laptop.
- Sample a groupFor each question the model writes 8 different queries by sampling.
- Check, don't judgeEach query is run on the database. Right rows with the columns the question asks for: reward 1. Runs but wrong: 0.2. Fails: 0.
- Compare inside the groupEach attempt's advantage is its reward minus the group average, divided by the spread. No extra "critic" network.
- Nudge the modelTokens of better-than-average attempts become more likely, the others less. Only a small LoRA adapter is trained.