4B-parameter model outperformed 235B model on financial tool-use tasks using RL with high-quality, expert-validated datasets and $500 compute cost.
Tool use discipline, not reasoning ability, was the core failure mode: larger model hallucinated answers after failed queries; smaller model learned to discover tables first.
RL training on single-table questions only yielded best results, yet generalized to harder multi-table benchmarks with similar performance doubling (13.9% to 26.6%).
High-quality data generation requires experts in the loop (PhD-level domain specialists and industry practitioners) for verification and data quality assurance.
GRPO-based RL with FinQA environment (self-contained, no external dependencies) enabled efficient training without expensive infrastructure or data export concerns.
FinQA environment published on HuggingFace Spaces and OpenEnv; includes 290 single-table and 79 multi-table reasoning benchmarks for reproducible evaluation.
Model behavior analysis via evaluation rubrics—breaking feedback into specific questions about correctness—reveals which behaviors to target in dataset generation rather than guessing.
Enterprise production deployments favor smaller fine-tuned models over large ones: lower cost, faster inference, on-premise deployment options, better data control for financial/healthcare sectors.
Smaller models with specialized RL training outperformed reasoning-optimized larger models; reasoning ability alone insufficient without learned tool-use discipline.
Self-correction via error observation was critical behavior: model learned to detect SQL errors and adjust queries, not just initial query formulation.
Partnering with UC Berkeley's RLLM team, Snorkel applied GRPO fine-tuning on 290 FinQA samples in under 21 hours. The 4B model doubled pass@1 by learning to discover table schemas before querying, eliminating the hallucinated-table failures that plagued the larger model. Single-table-only training generalized to multi-table tasks.
AI adoption alone doesn't improve outcomes—organizational systems and capabilities determine success; teams without foundational systems experience more acute pain when AI is introduced.
Individual productivity increases most with AI, but software delivery instability (rollbacks) increases second most—indicating a verification tax problem from shipping code faster than testing scales.
Seven distinct team profiles emerged in DORA research; outcomes vary dramatically across organizations despite near-universal AI adoption, proving AI is an amplifier, not a solution.
Working in small batches is under threat; agentic workflows cause developers to context-switch constantly across multiple simultaneous tasks, leading to burnout and exhaustion after one hour of effective work.
Code review capacity is the new bottleneck—AI can generate 10x-100x more code, but reviewers default to 'looks good to me' on massive diffs, creating a massive verification tax.
DORA's seven AI capabilities model: clear communicated AI stance, healthy data ecosystems, version control practices, small batches, user-centric focus, quality internal platforms, and reduced cognitive load—these combine with adoption to drive outcomes.
Use AI to help review code, not replace code writing—code reviews were a bottleneck before AI and are worse now; invest in automated testing and better feedback mechanisms instead.
Quality internal platforms must now serve two user types: developers and AI agents; also serve non-engineers (business analysts, finance, marketing) to safely enable anyone to ship features.
Organizations experience a J-curve productivity dip when adopting new tools; many abandon changes mid-dip instead of staying disciplined—learning by doing, not token leaderboards, reduces curve width and depth.
Build systems and platforms to minimize burnout and verification tax, not just drive token consumption; bad systems beat good people every time—fix the system, not just adoption metrics.
Findings from DORA's 2025 survey of ~5,000 practitioners show 90% use AI yet software delivery instability rises with adoption—AI amplifies existing system quality, good or bad. Seven capabilities predict better outcomes: strong version control, small-batch workflows, healthy data ecosystems, and quality internal platforms.
Cloudflare migrated from NGINX to Tokio-based Rust proxies (Pingora, FL2) across 93 million requests/second infrastructure to fix architectural limitations, not just for Rust performance gains.
Shared-nothing architecture forces connection affinity to single worker threads, causing terrible connection pool reuse (multiple redundant pools per machine) and hundreds of gigabytes of redundant cached data per server.
Work-stealing allows all worker threads to share one I/O driver and steal tasks from each other's queues, eliminating connection affinity issues and simplifying worker load balancing without kernel patches.
Moving to work-stealing reduced origin connections by 66%, saving ~434 years of TLS handshake time per day and enabling shared connection pools instead of per-worker pools.
FL2 (work-stealing) uses ~20GB memory vs Engine X-FL using ~85GB on same machine handling substantially more traffic—4:1 memory savings from eliminating redundant per-worker caches.
NGINX's requirement to disable keep-alive between proxies (forcing new connection per request) caused significant latency that work-stealing solved by allowing any worker to handle any connection.
Architectural changes (shared-nothing → work-stealing) delivered larger tail latency improvements than switching to Rust, proving runtime design matters more than language choice for web servers.
Work-stealing is "incredibly fast out of the box" and easier to operate than shared-nothing, making it a better default for web servers despite shared-nothing advantages in specific cache-sensitive workloads.
Unix domain socket communication between Cloudflare's proxies enabled the shift to work-stealing without changing transport layer fundamentals.
Noah Kennedy details Cloudflare's edge proxy rewrite, showing shared-nothing event loops caused per-worker connection pool fragmentation and 85 GB config caches. Tokio's work-stealing runtime unified the IO driver, shrunk caches to 20 GB, cut origin connections by 66%, and improved tail latency—without kernel patches or disabled keep-alives.
Stoflow challenges distributed stream processing default by designing for vertical scaling on single instances instead of horizontal scaling across multiple nodes.
Horizontal scaling requires repartitioning data through network and Kafka, serialization overhead up to 40-50% of compute, state migration complexity, and rebalancing storms—collectively called the 'distribution tax'.
Stoflow architecture uses single consumer, dispatcher for event-time ordering and key hashing, parallel lanes with virtual threads, shared RocksDB state, and barrier-based exactly-once transactions—eliminating repartition topics.
P99 end-to-end latency 13x lower, CPU usage 3.5-4x less, memory 9x less than Kafka Streams in benchmarks across five representative topologies.
Single 8-core machine processes 124k events/sec (1KB payloads) or 2M events/sec (word count), enabling 10 billion records/day throughput—ceiling higher than most real-world mission-critical workloads.
Visa processes ~8,200 transactions/sec globally; SWIFT ~600 msgs/sec; most stream applications operate far below horizontal scaling thresholds but default to complex distributed setups.
Eliminates Kafka Streams operational pain: no rebalancing, no state migration, simple stop-upgrade-restart cycles with persistent volumes preserve local state.
Virtual threads enable blocking I/O (database enrichment, REST APIs, LLM calls) natively without blocking cores—previously considered anti-pattern in Kafka Streams.
All state globally accessible in memory without sharding; concurrent processing with atomic primitives and locking; enables foreign-key joins without data movement.
Stoflow decouples parallelism from source topic partition count using virtual threads; achieves hundreds/thousands of concurrent lanes with minimal resource cost versus Flink's practical limits.
Hartmut Armbruster's Stoflow uses a single consumer with virtual threads, in-memory lane-based processing, and transactional epochs for exactly-once semantics—eliminating repartition topics and rebalancing storms. Benchmarks on an 8-core VM show 3-4x less CPU, 9x less memory, and up to 2M events/sec, handling ~10B records/day on modest hardware.
Specula automatically generates TLA+ specifications from code using agents, discovering 130+ previously unknown bugs in production systems like MongoDB and Raft with 4.7% false positive rate.
TLA+ specs require separate modeling from code and multi-month engineering effort; Specula automates this workflow for ~$36-40 per project, making formal verification economically feasible at scale.
LLMs alone score poorly on spec generation (especially conformance and invariance metrics at ~30-40%), but agents equipped with TLC model checking, trace validation, code instrumentation, and step-through debugging achieve 100% on key quality metrics.
Specula's four-phase pipeline: repository mining (identifies suspicious modules via code churn heuristics), spec generation with syntax validation, model checking and trace validation (automated debugging that took 2 engineer-months manually), and bug reproduction with evidence.
Discovered bugs require average 24 sequential actions to trigger, explaining why manual testing missed them; formal methods' exhaustive state exploration catches complex interleaving bugs humans cannot find.
Trace validation maps real execution logs to abstract TLA+ actions, catching specs that don't match implementation; critical because specs useless if they don't reflect actual code behavior.
Agent tools should include GitHub issues, git commits, code comments, and execution traces—context humans use but LLMs lack by default—to ground generated specs in reality and reduce hallucinations.
Tool provides rigor for AI-generated code verification: instead of manually reviewing hundreds of thousands of lines, developers examine formally model-checked invariants, creating virtuous cycle of software correctness.
Specula found bugs across 47 systems including serious issues: deadlocks, data corruption, memory safety errors (even in Rust), with C/Erlang/Java projects showing higher bug rates than Go/Rust.
15% of codebase remains unverified by traces; false positives stem from undocumented developer assumptions; affordable formal verification adoption now feasible, shifting from human-effort constraint to cost-efficient automated verification.
Emily Ma's Specula pipeline chains LLM agents with repository mining, syntax validation, and model-checking tools across four phases to produce TLA+ specs without manual effort. Evaluated on MongoDB, HashiCorp Raft, ZooKeeper, and others: 95.3% maintainer-confirmed bug rate at ~€38 and 3 hours per project.
