targeted RL beats scaling for tool-use
Snorkel 4B model beats 235B on financial SQL tasks after $500 RL fine-tune AI EngineerTL;DW
- 4B-parameter model outperformed 235B model on financial tool-use tasks using RL with high-quality, expert-validated datasets and $500 compute cost.
- Tool use discipline, not reasoning ability, was the core failure mode: larger model hallucinated answers after failed queries; smaller model learned to discover tables first.
- RL training on single-table questions only yielded best results, yet generalized to harder multi-table benchmarks with similar performance doubling (13.9% to 26.6%).
- High-quality data generation requires experts in the loop (PhD-level domain specialists and industry practitioners) for verification and data quality assurance.
- GRPO-based RL with FinQA environment (self-contained, no external dependencies) enabled efficient training without expensive infrastructure or data export concerns.
- FinQA environment published on HuggingFace Spaces and OpenEnv; includes 290 single-table and 79 multi-table reasoning benchmarks for reproducible evaluation.
- Model behavior analysis via evaluation rubrics—breaking feedback into specific questions about correctness—reveals which behaviors to target in dataset generation rather than guessing.
- Enterprise production deployments favor smaller fine-tuned models over large ones: lower cost, faster inference, on-premise deployment options, better data control for financial/healthcare sectors.
- Smaller models with specialized RL training outperformed reasoning-optimized larger models; reasoning ability alone insufficient without learned tool-use discipline.
- Self-correction via error observation was critical behavior: model learned to detect SQL errors and adjust queries, not just initial query formulation.
TL;DW
- 4B-parameter model outperformed 235B model on financial tool-use tasks using RL with high-quality, expert-validated datasets and $500 compute cost.
- Tool use discipline, not reasoning ability, was the core failure mode: larger model hallucinated answers after failed queries; smaller model learned to discover tables first.
- RL training on single-table questions only yielded best results, yet generalized to harder multi-table benchmarks with similar performance doubling (13.9% to 26.6%).
- High-quality data generation requires experts in the loop (PhD-level domain specialists and industry practitioners) for verification and data quality assurance.
- GRPO-based RL with FinQA environment (self-contained, no external dependencies) enabled efficient training without expensive infrastructure or data export concerns.
- FinQA environment published on HuggingFace Spaces and OpenEnv; includes 290 single-table and 79 multi-table reasoning benchmarks for reproducible evaluation.
- Model behavior analysis via evaluation rubrics—breaking feedback into specific questions about correctness—reveals which behaviors to target in dataset generation rather than guessing.
- Enterprise production deployments favor smaller fine-tuned models over large ones: lower cost, faster inference, on-premise deployment options, better data control for financial/healthcare sectors.
- Smaller models with specialized RL training outperformed reasoning-optimized larger models; reasoning ability alone insufficient without learned tool-use discipline.
- Self-correction via error observation was critical behavior: model learned to detect SQL errors and adjust queries, not just initial query formulation.
Partnering with UC Berkeley's RLLM team, Snorkel applied GRPO fine-tuning on 290 FinQA samples in under 21 hours. The 4B model doubled pass@1 by learning to discover table schemas before querying, eliminating the hallucinated-table failures that plagued the larger model. Single-table-only training generalized to multi-table tasks.