CI/CD supply chain attack post-mortem
Grafana contains GitHub Actions supply-chain attack in 10 hours via canary tokens Grafana LabsTL;DW
- GitHub's `pull_request_target` trigger grants attackers access to secrets and repo; combining it with user-controlled inputs like branch names creates exploitable CI/CD vulnerabilities.
- Gato-X tool automatically scans GitHub orgs for vulnerable Actions workflows and actively exploits them; attackers gained access within 10 hours of the malicious PR merge.
- Canary tokens—high-permission-looking but zero-access decoys—detected the attack when TruffleHog validated them; without canaries, attackers operated undetected in a 'chocolate factory.'
- GitHub Secrets are overly permissive; Grafana migrated to HashiCorp Vault (OIDC-gated, multi-step access) to block attackers from exfiltrating all secrets automatically.
- Zizmor static analysis tool flags pull_request_target, unpinned actions, and injection vulnerabilities; now mandatory in Grafana's CI pipeline post-incident.
- TruffleHog finds secrets in repos and validates them against live services; the dual-use tool revealed compromised tokens but attackers bypassed Thinkst canary detection.
- Loki log aggregation retained GitHub logs that attackers could have deleted; powerful query language enabled forensic investigation faster than GitHub's limited tools.
- IRM (incident response management) coordinated communication across teams; integration with Slack, Google Docs, and Grafana Alerting moved response from alert to action seamlessly.
- Preparation beats reaction: canary tokens, static analysis, secret hygiene, and CI/CD observability allowed Grafana to detect and contain breach in under 10 hours with zero customer impact.
- GitHub App permissions were over-scoped; Grafana broke down super-user access and severely reduced application scope to limit blast radius of future supply-chain attacks.
TL;DW
- GitHub's `pull_request_target` trigger grants attackers access to secrets and repo; combining it with user-controlled inputs like branch names creates exploitable CI/CD vulnerabilities.
- Gato-X tool automatically scans GitHub orgs for vulnerable Actions workflows and actively exploits them; attackers gained access within 10 hours of the malicious PR merge.
- Canary tokens—high-permission-looking but zero-access decoys—detected the attack when TruffleHog validated them; without canaries, attackers operated undetected in a 'chocolate factory.'
- GitHub Secrets are overly permissive; Grafana migrated to HashiCorp Vault (OIDC-gated, multi-step access) to block attackers from exfiltrating all secrets automatically.
- Zizmor static analysis tool flags pull_request_target, unpinned actions, and injection vulnerabilities; now mandatory in Grafana's CI pipeline post-incident.
- TruffleHog finds secrets in repos and validates them against live services; the dual-use tool revealed compromised tokens but attackers bypassed Thinkst canary detection.
- Loki log aggregation retained GitHub logs that attackers could have deleted; powerful query language enabled forensic investigation faster than GitHub's limited tools.
- IRM (incident response management) coordinated communication across teams; integration with Slack, Google Docs, and Grafana Alerting moved response from alert to action seamlessly.
- Preparation beats reaction: canary tokens, static analysis, secret hygiene, and CI/CD observability allowed Grafana to detect and contain breach in under 10 hours with zero customer impact.
- GitHub App permissions were over-scoped; Grafana broke down super-user access and severely reduced application scope to limit blast radius of future supply-chain attacks.
A pull_request_target misconfiguration exposed repo secrets to external contributors; attackers used Gato-X to find and exploit the flaw before a triggered canary token blew the operation. Response included Loki log analysis, Zizmor workflow scanning, TruffleHog secret detection, and a full migration from GitHub-native secrets to HashiCorp Vault.