---
name: frontier-index
description: Evidence-graded rules of thumb for building with AI agents (verification, orchestration, context, skills, memory, security, churn). Load when designing or reviewing agent workflows, subagents, verification loops, skills, memory or agent security.
---

# The Frontier Index

What actually works with AI agents, graded by evidence. Last checked 2026-11-19. Source: https://bitby.ai/frontier

Evidence grades: **A** (strong) = several independent papers, or one large peer-reviewed study. **B** (good) = one large systematic study, not yet replicated. **C** (early) = a single preprint, vendor self-report or practitioner claim. **P** (practice) = practitioner consensus: shown in repos, posts or talks, not measured.
Most arXiv items are unreviewed preprints: weigh the grade, and re-check anything older than three months.

## Verification

The gate on all autonomy.

- [P] Autonomy is bounded by verification capacity. Build the verifier before the loop. Sources: [Boris Cherny, '#1 tip'](https://x.com/bcherny/status/2007179861115511237); [Andrej Karpathy, 'automate what you can verify'](https://singjupost.com/andrej-karpathy-from-vibe-coding-to-agentic-engineering-w-stephanie-zhan-transcript/); [poteto, prove-it-works](https://github.com/cursor/plugins/tree/main/pstack); [Matt Pocock, skills](https://github.com/mattpocock/skills).
- [A] Same-context self-correction is unreliable: a 64.5% failure rate on a model's own errors, yet the same errors get fixed when attributed to someone else. Sources: [arXiv 2310.01798](https://arxiv.org/pdf/2310.01798); [arXiv 2507.02778](https://arxiv.org/html/2507.02778v2).
- [C] Exception: frontier models with a very low error-introduction rate (about 0.5%) gain slightly from self-correction. Others get worse. Sources: [arXiv 2604.22273](https://arxiv.org/html/2604.22273v2).
- [B] Rank your checks: executable ground truth (tests, types, build, the real app) first, then an agent-written check validated against fixtures, then a fresh-context or other-model reviewer. Same-context self-review is weakest. Sources: [arXiv 2601.11868](https://arxiv.org/abs/2601.11868); [arXiv 2610.28679](https://arxiv.org/abs/2610.28679); [arXiv 2606.05976](https://arxiv.org/html/2606.05976v1).
- [C] An agent-written check counts only after it passes a known-good solution, passes another valid solution, and fails a broken one. Otherwise it inherits the agent's misunderstanding. Sources: [VerifierBench, arXiv 2610.28679](https://arxiv.org/abs/2610.28679).
- [C] For LLM review, present the work as a colleague's claim: correction rates rose 23 to 93 points. 'Distrust yourself' prompts do nothing. Sources: [arXiv 2606.05976](https://arxiv.org/html/2606.05976v1).
- [C] Task success is not compliance: 31.1% of successful runs broke at least one harness rule. Check the trajectory, not just the output. Sources: [TraceTrial-Bench, arXiv 2610.32285](https://arxiv.org/abs/2610.32285).
- [C] Gate accuracy is not workflow quality: a claimed 23.9% saving was really 4.3%. Evaluate end to end. Sources: [arXiv 2610.02267](https://arxiv.org/abs/2610.02267).
- [P] Write done criteria as binary pass/fail lines of about 8 to 12 words each. Sources: [Daniel Miessler](https://danielmiessler.com/blog/the-most-important-ideas-in-ai).
- [P] Gates default to fail. A test run is not a judgement call: the agent proposes, code disposes. Sources: [IndyDevDan, super-simple-software-factory](https://github.com/disler/super-simple-software-factory).
- [C] Cap retries at 2 to 3. A model's own past errors in context make more errors likely. Sources: [arXiv 2509.09677](https://arxiv.org/abs/2509.09677).
- [B] Plan unattended work around the 80% time horizon (about 3 hours at the last measurement), not the 50% headline. Sources: [METR time horizons, via agitracker](https://agitracker.io/blog/metr-time-horizons-2026/).
- [P] Routines at scale still need human tuning: one team's agent routines merged about 46% of the PRs they opened. Sources: [Boris Cherny, Aug 2026](https://x.com/bcherny/status/2088014489438621990).
- [P] Pick work where an executable verifier exists or can be built: headless playtests and replays for games, geometry and manufacturability checks for CAD. The moat is the verifiers, not the agent count. Sources: own projects.
- [P] Research starts from what you already know and ends in a measured change, or an explicit 'no action'. Sources: own projects.

## Orchestration

One agent or many.

- [B] Default to one agent with skills and tools. Multi-agent ranged from +80.8% on decomposable work to -70% on sequential planning, and gains shrink as the single agent gets stronger. Sources: [arXiv 2512.08296 (260 configurations)](https://arxiv.org/abs/2512.08296).
- [C] Never split a user-facing conversation across agents: multi-agent lost 12 of 13 cells (-28 points) and at best tied while spending 5.5x the tokens. Sources: [arXiv 2610.27655](https://arxiv.org/abs/2610.27655v1).
- [C] A single agent with skills matched multi-agent accuracy with about 54% fewer tokens. Sources: [arXiv 2601.04748](https://arxiv.org/html/2601.04748v2).
- [B] Use subagents for parallel, read-mostly fan-out (search, research, file-by-file migration), merged by one central verifier. Uncentralised setups propagate errors. Sources: [arXiv 2512.08296](https://arxiv.org/abs/2512.08296); [MAST, arXiv 2503.13657](https://arxiv.org/abs/2503.13657).
- [P] Budget 2 to 6x tokens for multi-agent. Cheap models for bulk work, a strong model to orchestrate and verify, and have the strong model write code filters before the cheap one reads the data. Sources: [Andrej Karpathy (secondhand notes)](https://brycewatson.com/research/karpathy-ai-habits/).
- [P] Split agents by trust domain and identity (reader vs actor, outside-facing vs internal), not by role in one workflow. Sources: [Daniel Miessler, ideas](https://danielmiessler.com/ideas); own projects.
- [P] Earn scale: start with 1 to 5 watched agents, build verification skills, then scale. Sources: [poteto, talk](https://www.youtube.com/watch?v=NjoZoUm85x0).
- [P] Vendors ship swarms anyway. A product trend is not evidence of benefit. Sources: [Google Antigravity /swarm](https://antigravity.google/blog/gemini-4-carbon-in-google-antigravity/); [Claude platform release notes](https://platform.claude.com/docs/en/release-notes/overview).
- [P] Code owns control flow: the agent proposes, code disposes, with typed JSON handoffs and rollback at the write boundary. Sources: [IndyDevDan, super-simple-software-factory](https://github.com/disler/super-simple-software-factory).

## Context

Less is more, and long context rots.

- [P] Subtract. Audit always-loaded context: unused skills and long instruction files cost tokens every session. Sources: [Boris Cherny, /checkup](https://x.com/bcherny/status/2074997571563479143); [LifeOS releases](https://github.com/danielmiessler/LifeOS/releases).
- [P] Newer models need less: shorter prompts, often no separate plan step. Keep a spec-first step where misalignment, not capability, is the risk. Sources: [Claude blog, context engineering](https://claude.com/blog/the-new-rules-of-context-engineering-for-claude-5-generation-models); [Matt Pocock, skills](https://github.com/mattpocock/skills).
- [A] Long context rots: accuracy falls with length even when the facts are present. Prefer a fresh session plus a handover file over compressing a long one. Sources: arXiv 2605.08580; [Matt Pocock (workshop notes)](https://cussid-huaz.github.io/agent-brain/queries/matt-pocock-ai-coding-workflow-2026/).
- [C] Compaction can silently drop constraints (violations 0% to 30%). Keep safety rules in always-reloaded instructions and treat summaries as untrusted. Benchmarks look compaction-tolerant, but they measure completion, not rule-following. Sources: [arXiv 2606.22528](https://arxiv.org/abs/2606.22528); [arXiv 2610.31778](https://arxiv.org/html/2610.31778v1).

## Skills

Curated beats generated.

- [B] Human-curated skills: +16.2 points on average. Self-generated skills: about -1.3. Never auto-promote agent-written learnings into always-on rules. Sources: [SkillsBench, arXiv 2602.12670](https://arxiv.org/html/2602.12670v1).
- [B] One skill = one workflow. Focused skills of 2 to 3 modules beat comprehensive docs. Sources: [SkillsBench, arXiv 2602.12670](https://arxiv.org/html/2602.12670v1).
- [P] Repository instructions are ambient guidance; a skill is an on-demand capability. Push standards to reviewers as always-on rules; let implementers pull skills when needed. Sources: [Matt Pocock, skills](https://github.com/mattpocock/skills).
- [B] Skills are supply chain: 76 malicious skills were found among 3,984 audited. Pin versions, review diffs, and never install a skill you haven't read. Sources: [Snyk ToxicSkills](https://snyk.io/blog/toxicskills-malicious-ai-agent-skills-clawhub/).
- [P] Keep skills portable (the SKILL.md format). Harnesses are commoditising and swappable. Sources: [Claude platform release notes](https://platform.claude.com/docs/en/release-notes/overview).

## Memory

The least settled area.

- [C] Vendor memory benchmarks are self-reported, and memory evaluation has hidden confounds. Sources: [arXiv 2606.29914](https://arxiv.org/pdf/2606.29914); [Mem0 blog (vendor claim)](https://mem0.ai/blog/state-of-ai-agent-memory-2026).
- [P] File-based memory is what practitioners actually run: an index file, typed notes, and a periodic distill pass. Sources: [LifeOS releases](https://github.com/danielmiessler/LifeOS/releases); own projects.
- [P] Track where each memory came from. Memory poisoning is a different threat from context poisoning. Sources: own projects.

## Security

Boundaries, not prompts.

- [A] Prompt and classifier guardrails are not a boundary: adaptive attacks beat 12 defenses at over 90%, and classifiers let about 70% of evasions through. Use them for telemetry only. Sources: [The Attacker Moves Second, arXiv 2510.09023](https://arxiv.org/abs/2510.09023); [arXiv 2504.11168](https://arxiv.org/html/2504.11168).
- [B] Only architecture holds: a quarantined reader (dual-LLM), code-enforced policy and capability limits. Sources: [CaMeL](https://css.csail.mit.edu/6.5660/2026/readings/camel.pdf); [arXiv 2606.26479](https://arxiv.org/abs/2606.26479).
- [C] Small planners write broken security plans (up to 68% fail to run), so give them a policy they didn't write. Sources: [CapLite, arXiv 2611.17115](https://arxiv.org/abs/2611.17115v1).
- [P] Remove soft guards (prompt rules, hooks) only after something harder replaces them: permission rules, git hooks, an OS sandbox. Sources: [LifeOS v8 release](https://github.com/danielmiessler/LifeOS/releases).
- [P] Keep credentials out of agent reach: a broker, injection after requests leave the sandbox, or policy enforced outside the agent's process. Sources: sandbox and secrets-broker vendor docs.
- [B] Treat PR and issue text, fetched pages, skills, tool output and compaction summaries as untrusted input. Sources: [Comment and Control (VentureBeat)](https://venturebeat.com/security/ai-agent-runtime-security-system-card-audit-comment-and-control-2026); [arXiv 2608.30686](https://arxiv.org/pdf/2608.30686); [arXiv 2609.03884](https://arxiv.org/pdf/2609.03884).
- [C] Agents are attackers now: one evaluation agent chained a zero-day into code execution, secrets and CI. Give unattended agents least-privilege network egress. Sources: [Hugging Face incident timeline](https://huggingface.co/blog/agent-intrusion-technical-timeline).
- [P] Honeytoken credentials and honey tools are cheap prompt-injection tripwires. Sources: own projects.

## Harnesses & churn

Everything moves; pin what matters.

- [P] Pin model IDs in long-lived agents. Model IDs and tool versions change often, and deprecations break things. Sources: [Claude platform release notes](https://platform.claude.com/docs/en/release-notes/overview).
- [P] The harness matters more than the model, but harnesses are becoming swappable. Don't build anything you can't move. Sources: own projects.
- [B] Agent-manager apps die fast. Depend only on ones with a business model or that you can fork, behind your own data layer. Sources: [Vibe Kanban shutdown](https://vibekanban.com/blog/shutdown).
- [P] Protocols are settling: MCP now has a stateless core, and MCP and A2A sit under one foundation. Sources: [AAIF, MCP 2026-07-28](https://aaif.io/blog/mcp-2026-07-28-whats-changing-and-how-to-migrate).

## Game design

From building a party game with agents.

- [A] When two players must agree, each commits privately first, then both picks show and they confirm. Otherwise the more confident partner wins, right or wrong. Sources: Smith et al., Science 2009; Bahrami et al., Science 2010; Koriat, Science 2012.
- [B] Pitch multiple-choice questions at about 74% correct (4 options) or 77% (3), not 50%. Three plausible options, under 15 words. Sources: Lord 1952; Haladyna et al. 2002.
- [B] Eliminated players keep the same action and get one way back. Never leave them idle. Sources: Among Us developer interview (Inverse); Trivia Murder Party ghosts.
- [B] Vary a one-action game by changing one thing per round (timing, sync, tempo, a decoy), announced in five words. Never add a second action. Sources: Iwata Asks: WarioWare; Keep Talking and Nobody Explodes, GDC 2016.
- [A] Never accept an AI's fun rating. Fun is judged by humans playing; use bots for balance and broken states only. Sources: [Collins et al., ICLR 2026, arXiv 2510.10930](https://arxiv.org/abs/2510.10930).
