Monday, June 1, 2026 — front page

LLM code quality reality

Sonar tests 53 LLMs on 4,444 Java tasks, finds GPT-5 4x more verbose than Gemini and Claude most vulnerable AI Engineer
TL;DW
Sonar evaluated 53+ LLMs on 4,444 Java assignments; Gemini 3.1 Pro High leads with 84.17% pass rate but generates verbose, complex code with 210 security issues per million lines.
Claude Sonnet 4.6 generates 627,000 lines of code for test suite with 300 security issues per million lines; GPT-4 variants produce 1.2 million lines, creating severe verbosity problems.
LLM code quality issues stem from mixed-quality training data, inbuilt security flaws in datasets, hidden logic errors, probabilistic generation, limited context, and lack of explainability.
Newer LLM models generate progressively more lines of code and higher cyclomatic/cognitive complexity than earlier versions, creating harder-to-understand and harder-to-maintain solutions.
55% of developers regularly use AI agents for coding per March 2024 Pragmatic Engineer Survey, but functional correctness metrics (80%+ pass rates) miss real-world reliability, security, and maintainability gaps.
Sonar's ACDC framework addresses LLM code quality through three stages: Guide (context augmentation, data treatment), Verify (runtime agentic analysis in 1-5 seconds before commit), Solve (remediation agent for tech debt).
SonarQube agentic analysis runs in 1-5 seconds before code commit, catching issues faster than CI/CD (1-5 minutes) and enabling agents to fix problems before code submission.
Sonar remediation agent auto-fixes issues and tech debt by creating per-issue PRs, re-analyzing and recompiling to prevent regressions before delivering code to developers.
Cyclomatic complexity measures code branches (ifs, loops, conditions); cognitive complexity measures human readability and maintainability difficulty—LLMs score high on both, requiring expert review.
LLM leaderboards focus on functional correctness (80%+ accuracy) but ignore security density, bug rates (614+ per million lines), complexity, verbosity, and maintainability—critical for enterprise readiness.

Evaluated against enterprise dimensions—security, maintainability, reliability—LLMs that pass functional benchmarks at 80–84% still generate excessive complexity and bugs. GPT-5 produces 1.2M lines vs. Gemini's 300K for identical tasks; Claude Sonnet logs 300 security issues per million LOC. Sonar's ACDC framework adds real-time agentic analysis and auto-remediation before merge.

Spec-driven agent evaluation

AI Engineer: test agents against behavioral specs, not benchmarks, to catch capability-boundary failures AI Engineer
TL;DW
Spec-driven testing for agents requires more than datasets: include rules, ontologies, domain knowledge, rights/roles, and robustness requirements under perturbations like typos and rephrasing.
Larger models aren't always better or safer—they may understand jailbreaks in poetry, perform unnecessary tasks slowly, or create broader vulnerability surface area.
Explicit specs enable security testing by defining agent domains (where it's most vulnerable) and the infrastructure tasks it can perform (where it has most power).
Agent cards from A2A spec describe what agents do, but you still need envelopes specifying valid ranges—e.g., which people meetings can be booked for—to evaluate properly.
Build integration tests independent of implementation platform (LangSmith, Vertex, etc.) so tests remain portable if you switch infrastructure later.
Robustness testing should vary inputs systematically: can the agent handle typos, rephrasing, and other stress conditions before users get frustrated?
Domain-specific substitutions matter in testing—LLMs may confuse gross profit vs. gross sales, but finance agents must distinguish them correctly.
Use specs to close the loop: run agents automatically, iterate to fill robustness gaps, and version control everything in a GitHub repo for tool portability.

Willmott lays out a spec structure covering explicit rules, domain ontologies, role-based permissions, and robustness requirements, then shows how security and robustness testing both target the spec rather than the training set—so evaluations survive model swaps and infrastructure changes.

AI-powered scam disruption

Apache deploys AI bot armies to waste scammers' time and harvest fraud intelligence BSides Sydney
TL;DW
Apache deploys armies of tens of thousands of AI bots mimicking victims to waste scammers' time—averaging 14-23 minutes per call, with records exceeding 2 hours—while extracting intelligence on tactics, accounts, and infrastructure before scammers reach real victims.
Blocking scams at telco level is a vanity metric: scammers using auto-dialers don't notice blocked calls and simply retry. Australians still receive 10-15 scam calls monthly despite telcos blocking 2.5-3.5 billion calls yearly.
Apache bots use reinforcement learning with hidden Markov models to maintain conversation context and adapt strategies dynamically. Bots compete against fleet averages, with probabilistic rewards—bots never know if they succeeded, preventing fingerprinting.
Bots operate across voice (30+ languages, 18+ dialects in some regions) and text platforms (WhatsApp, Telegram, Signal), extracting mule accounts, crypto wallet addresses, and merchant IDs to dismantle scammer infrastructure in real time.
Commonwealth Bank deployment uses 20,000 voice bots and 200 text bots daily. Bot assets include fake credit cards that trigger pre-auth transactions, revealing fraudulent merchant accounts banks can block.
Bots must respond within 350 milliseconds to avoid awkwardness. They use natural disfluencies (ums, ahhs, laughter) to mimic human thinking time while computing next moves.
SIP redirect integration with telcos redirects flagged scam calls to bot infrastructure without blocking. Deterministic blocking (unallocated numbers, do-not-originate lists) reduces false positives when routing to bots.
40% of US bot-call volume is already bot-versus-bot encounters. Scammer bots are less sophisticated than Apache bots; if scammers widely adopt bots, humans become shielded while bot-fight intelligence still extracts actionable data.
Command-and-control interface lets banks launch targeted bot missions—e.g., hunt all calls impersonating bank executives, extract mule accounts, shut down specific campaigns—through intuitive deployment commands.
Billions lost monthly to scams globally (trillion-dollar economy). Each successful scammer call erodes public trust in communication systems; proactive bot-disruption shifts from reactive blocking to offensive intelligence gathering.

Instead of blocking scam calls, Apache redirects them to tens of thousands of AI voice bots with unique personas that average 14-23 minutes per call while extracting mule account and infrastructure details. Text bots on WhatsApp and Telegram pull crypto wallet addresses and merchant IDs; telco and bank partnerships feed the intelligence back to dismantle campaigns.