For a Columbia NLP course project, my group fine-tuned Qwen3-4B-Instruct on the BIRD benchmark with a fairly narrow goal: not just correct SQL, but fast SQL. A model that returns the right rows via a three-way self-join when an indexed lookup would do is "correct" by execution-accuracy metrics and useless in production. That distinction — correctness vs. efficiency — is where most of the interesting problems showed up.
Why one alignment stage wasn't enough
Our first instinct was to just run DPO on pairs of efficient vs. inefficient SQL for the same question and call it done. DPO is cheap, stable, and doesn't require an online rollout loop — you curate pairs once and train offline. It worked, to a point: the model learned general stylistic preferences (avoid unnecessary subqueries, prefer joins over correlated subqueries where equivalent) reasonably well.
What it didn't do well was generalize to query shapes we hadn't paired. DPO optimizes relative preference between two specific completions — it has no signal about what happens when you actually run the query. A model can learn "pattern A looks more like the efficient examples than pattern B" without any grounding in whether pattern A is actually faster on a given schema and data distribution. Preference pairs are a proxy for efficiency, not efficiency itself.
Adding execution-aware RL — and immediately regretting the naive version
The fix was a second stage: GRPO-style reinforcement learning with a reward built directly from execution — actually running the generated SQL and measuring latency, alongside correctness. This closes the gap DPO leaves open, because now the model gets gradient signal from ground truth about what's actually fast, not just "faster-looking."
WHERE clauses executes very fast and sometimes
still passes loose correctness checks on certain query types. Fast and wrong scored better than
slow and right until we tightened the correctness gate to be a hard filter (zero reward if incorrect,
rather than a soft penalty) before latency ever entered the reward calculation.
That's the core lesson from stacking DPO and GRPO: the reward for the online stage has to be adversarially robust before you scale it up, because RL will find the shortest path to a high reward, not the path you intended. A correctness-gated reward — where efficiency only matters at all once correctness is satisfied — was non-negotiable, not an optimization.
DPO's stability vs. GRPO's variance
Practically, the two stages felt very different to run. DPO training was stable — loss curves behaved, no reward collapse, easy to reproduce. GRPO was noisier: reward variance across rollout groups was high early in training, and it took more careful reward normalization (group-relative advantage, per the GRPO formulation) to keep updates from being dominated by a few outlier rollouts with unusually fast or unusually broken queries.
Running DPO first and GRPO second wasn't arbitrary — it mattered in that order. DPO gave the policy a reasonable starting distribution over "SQL that looks efficient," which meant GRPO's rollouts started from a saner place and needed fewer steps to converge, and reward variance from truly degenerate outputs (the empty-result-set problem above) was already partially suppressed before online RL even started putting pressure on the reward.
What it added up to
The combined DPO → GRPO pipeline got a 7% improvement in execution accuracy and Relative Valid Efficiency Score (R-VES) over baseline on the BIRD Mini-Dev set. The qualitative error analysis afterward showed the remaining failures clustering around genuinely hard query structures — deeply nested aggregations, multi-table joins with ambiguous column references — rather than the "technically correct but absurdly inefficient" pattern we started with. That shift is really the whole point: preference learning alone gets you stylistic efficiency, execution-aware RL gets you actual efficiency, and you need the correctness gate to keep the second stage honest.
Full project report →