At Stepfun, part of my job was turning raw crosstalk (相声) recordings — a very specific,
very Chinese style of comedic dialogue — into training-quality conversational data.
The pipeline had three stages: rule-based filtering, LLM-as-a-judge classification, and a manual
spot-check. The middle stage is where multi-model routing became a real engineering problem
instead of a single openai.ChatCompletion.create() call.
Why route across models at all
The naive approach — pick one model, prompt it, move on — breaks down fast once you're processing 10K+ raw dialogues and need to benchmark judge quality against ground truth. We were comparing GPT, Claude, and Qwen as candidate judges, and each had a different rate limit, different latency profile, and different failure modes on messy conversational Chinese. Locking into one model early would have meant re-running the entire pipeline every time we found a better judge.
So the pipeline was built model-agnostic from the start: a thin routing layer took a prompt template and a target model identifier, normalized the response format, and logged raw outputs regardless of which backend answered. That abstraction is unglamorous but it's the thing that let us swap judges without touching the filtering logic.
Two different parallelization problems
There were actually two distinct speed problems, and conflating them would have made both worse:
- Production throughput — running the chosen judge model over the full dataset. Here the bottleneck was API round-trip latency, not compute. Async execution (asyncio + a bounded semaphore per provider's rate limit) got us roughly a 5× speedup over sequential calls, just by not idling on I/O.
- Benchmarking throughput — running all three candidate models over the same sample set to compare judge quality. This is embarrassingly parallel across models and across samples, so local parallelization (multiprocessing across CPU cores, since we were mostly waiting on network I/O across many concurrent connections) got us closer to a 10× speedup for the comparison runs.
The lesson: "make it faster" isn't one problem. Production throughput and experimentation throughput have different bottlenecks and want different solutions. Async concurrency fixes I/O idling; local parallelism fixes CPU-bound orchestration overhead when you're fanning out to many models at once. Applying the wrong fix to the wrong problem gets you a smaller win for more complexity.
The prompt-placement finding nobody warns you about
The part that actually changed how we wrote prompts: we ran ablations on whether instructions belonged
in the system role or the user role, and the optimal placement was not consistent
across models.
This meant our prompt templates couldn't be a single string swapped across providers — they needed a per-model injection strategy. It's a small thing, but it's exactly the kind of detail that doesn't show up until you're running the same task across multiple models side by side and start diffing failure rates. If you're building anything that routes across providers, budget time for this ablation early; retrofitting per-model prompt strategies after your pipeline assumes one format is annoying.
What it added up to
The three-stage pipeline took the raw 10K+ dialogues down to roughly 6K gold-standard samples, with few-shot-guided rewriting filling in gaps where dialogue density was too low to be useful as-is. That curated set contributed a 6–7 point gain on internal reasoning and language benchmarks — enough to clear the bar for consumer deployment. None of that would have been feasible at the volume we needed without treating "which model judges this" and "how fast can we run it" as two separate, deliberately solved problems.