Skip to content

■Free · checked 2026-11-19

The Frontier Index.

What actually works with AI agents, graded by evidence.

Rules of thumb for building with AI agents. Each one carries an evidence grade, the sources behind it and the date it was last checked. Most papers here are preprints; the grade says how much to trust each rule.

SKILL.md · drop it into your agent's skills folder and it loads the rules when they matter.

fig. 01 verification · the gate on all autonomy

Verification

  1. P

    Autonomy is bounded by verification capacity. Build the verifier before the loop.

    Boris Cherny, '#1 tip'Andrej Karpathy, 'automate what you can verify'poteto, prove-it-worksMatt Pocock, skills

  2. A

    Same-context self-correction is unreliable: a 64.5% failure rate on a model's own errors, yet the same errors get fixed when attributed to someone else.

    arXiv 2310.01798arXiv 2507.02778

  3. C

    Exception: frontier models with a very low error-introduction rate (about 0.5%) gain slightly from self-correction. Others get worse.

    arXiv 2604.22273

  4. B

    Rank your checks: executable ground truth (tests, types, build, the real app) first, then an agent-written check validated against fixtures, then a fresh-context or other-model reviewer. Same-context self-review is weakest.

    arXiv 2601.11868arXiv 2610.28679arXiv 2606.05976

  5. C

    An agent-written check counts only after it passes a known-good solution, passes another valid solution, and fails a broken one. Otherwise it inherits the agent's misunderstanding.

    VerifierBench, arXiv 2610.28679

  6. C

    For LLM review, present the work as a colleague's claim: correction rates rose 23 to 93 points. 'Distrust yourself' prompts do nothing.

    arXiv 2606.05976

  7. C

    Task success is not compliance: 31.1% of successful runs broke at least one harness rule. Check the trajectory, not just the output.

    TraceTrial-Bench, arXiv 2610.32285

  8. C

    Gate accuracy is not workflow quality: a claimed 23.9% saving was really 4.3%. Evaluate end to end.

    arXiv 2610.02267

  9. P

    Write done criteria as binary pass/fail lines of about 8 to 12 words each.

    Daniel Miessler

  10. P

    Gates default to fail. A test run is not a judgement call: the agent proposes, code disposes.

    IndyDevDan, super-simple-software-factory

  11. C

    Cap retries at 2 to 3. A model's own past errors in context make more errors likely.

    arXiv 2509.09677

  12. B

    Plan unattended work around the 80% time horizon (about 3 hours at the last measurement), not the 50% headline.

    METR time horizons, via agitracker

  13. P

    Routines at scale still need human tuning: one team's agent routines merged about 46% of the PRs they opened.

    Boris Cherny, Aug 2026

  14. P

    Pick work where an executable verifier exists or can be built: headless playtests and replays for games, geometry and manufacturability checks for CAD. The moat is the verifiers, not the agent count.

    own projects

  15. P

    Research starts from what you already know and ends in a measured change, or an explicit 'no action'.

    own projects

fig. 02 orchestration · one agent or many

Orchestration

  1. B

    Default to one agent with skills and tools. Multi-agent ranged from +80.8% on decomposable work to -70% on sequential planning, and gains shrink as the single agent gets stronger.

    arXiv 2512.08296 (260 configurations)

  2. C

    Never split a user-facing conversation across agents: multi-agent lost 12 of 13 cells (-28 points) and at best tied while spending 5.5x the tokens.

    arXiv 2610.27655

  3. C

    A single agent with skills matched multi-agent accuracy with about 54% fewer tokens.

    arXiv 2601.04748

  4. B

    Use subagents for parallel, read-mostly fan-out (search, research, file-by-file migration), merged by one central verifier. Uncentralised setups propagate errors.

    arXiv 2512.08296MAST, arXiv 2503.13657

  5. P

    Budget 2 to 6x tokens for multi-agent. Cheap models for bulk work, a strong model to orchestrate and verify, and have the strong model write code filters before the cheap one reads the data.

    Andrej Karpathy (secondhand notes)

  6. P

    Split agents by trust domain and identity (reader vs actor, outside-facing vs internal), not by role in one workflow.

    Daniel Miessler, ideasown projects

  7. P

    Earn scale: start with 1 to 5 watched agents, build verification skills, then scale.

    poteto, talk

  8. P

    Vendors ship swarms anyway. A product trend is not evidence of benefit.

    Google Antigravity /swarmClaude platform release notes

  9. P

    Code owns control flow: the agent proposes, code disposes, with typed JSON handoffs and rollback at the write boundary.

    IndyDevDan, super-simple-software-factory

fig. 03 context · less is more, and long context rots

Context

  1. P

    Subtract. Audit always-loaded context: unused skills and long instruction files cost tokens every session.

    Boris Cherny, /checkupLifeOS releases

  2. P

    Newer models need less: shorter prompts, often no separate plan step. Keep a spec-first step where misalignment, not capability, is the risk.

    Claude blog, context engineeringMatt Pocock, skills

  3. A

    Long context rots: accuracy falls with length even when the facts are present. Prefer a fresh session plus a handover file over compressing a long one.

    arXiv 2605.08580Matt Pocock (workshop notes)

  4. C

    Compaction can silently drop constraints (violations 0% to 30%). Keep safety rules in always-reloaded instructions and treat summaries as untrusted. Benchmarks look compaction-tolerant, but they measure completion, not rule-following.

    arXiv 2606.22528arXiv 2610.31778

fig. 04 skills · curated beats generated

Skills

  1. B

    Human-curated skills: +16.2 points on average. Self-generated skills: about -1.3. Never auto-promote agent-written learnings into always-on rules.

    SkillsBench, arXiv 2602.12670

  2. B

    One skill = one workflow. Focused skills of 2 to 3 modules beat comprehensive docs.

    SkillsBench, arXiv 2602.12670

  3. P

    Repository instructions are ambient guidance; a skill is an on-demand capability. Push standards to reviewers as always-on rules; let implementers pull skills when needed.

    Matt Pocock, skills

  4. B

    Skills are supply chain: 76 malicious skills were found among 3,984 audited. Pin versions, review diffs, and never install a skill you haven't read.

    Snyk ToxicSkills

  5. P

    Keep skills portable (the SKILL.md format). Harnesses are commoditising and swappable.

    Claude platform release notes

fig. 05 memory · the least settled area

Memory

  1. C

    Vendor memory benchmarks are self-reported, and memory evaluation has hidden confounds.

    arXiv 2606.29914Mem0 blog (vendor claim)

  2. P

    File-based memory is what practitioners actually run: an index file, typed notes, and a periodic distill pass.

    LifeOS releasesown projects

  3. P

    Track where each memory came from. Memory poisoning is a different threat from context poisoning.

    own projects

fig. 06 security · boundaries, not prompts

Security

  1. A

    Prompt and classifier guardrails are not a boundary: adaptive attacks beat 12 defenses at over 90%, and classifiers let about 70% of evasions through. Use them for telemetry only.

    The Attacker Moves Second, arXiv 2510.09023arXiv 2504.11168

  2. B

    Only architecture holds: a quarantined reader (dual-LLM), code-enforced policy and capability limits.

    CaMeLarXiv 2606.26479

  3. C

    Small planners write broken security plans (up to 68% fail to run), so give them a policy they didn't write.

    CapLite, arXiv 2611.17115

  4. P

    Remove soft guards (prompt rules, hooks) only after something harder replaces them: permission rules, git hooks, an OS sandbox.

    LifeOS v8 release

  5. P

    Keep credentials out of agent reach: a broker, injection after requests leave the sandbox, or policy enforced outside the agent's process.

    sandbox and secrets-broker vendor docs

  6. B

    Treat PR and issue text, fetched pages, skills, tool output and compaction summaries as untrusted input.

    Comment and Control (VentureBeat)arXiv 2608.30686arXiv 2609.03884

  7. C

    Agents are attackers now: one evaluation agent chained a zero-day into code execution, secrets and CI. Give unattended agents least-privilege network egress.

    Hugging Face incident timeline

  8. P

    Honeytoken credentials and honey tools are cheap prompt-injection tripwires.

    own projects

fig. 07 harnesses & churn · everything moves; pin what matters

Harnesses & churn

  1. P

    Pin model IDs in long-lived agents. Model IDs and tool versions change often, and deprecations break things.

    Claude platform release notes

  2. P

    The harness matters more than the model, but harnesses are becoming swappable. Don't build anything you can't move.

    own projects

  3. B

    Agent-manager apps die fast. Depend only on ones with a business model or that you can fork, behind your own data layer.

    Vibe Kanban shutdown

  4. P

    Protocols are settling: MCP now has a stateless core, and MCP and A2A sit under one foundation.

    AAIF, MCP 2026-07-28

fig. 08 game design · from building a party game with agents

Game design

  1. A

    When two players must agree, each commits privately first, then both picks show and they confirm. Otherwise the more confident partner wins, right or wrong.

    Smith et al., Science 2009Bahrami et al., Science 2010Koriat, Science 2012

  2. B

    Pitch multiple-choice questions at about 74% correct (4 options) or 77% (3), not 50%. Three plausible options, under 15 words.

    Lord 1952Haladyna et al. 2002

  3. B

    Eliminated players keep the same action and get one way back. Never leave them idle.

    Among Us developer interview (Inverse)Trivia Murder Party ghosts

  4. B

    Vary a one-action game by changing one thing per round (timing, sync, tempo, a decoy), announced in five words. Never add a second action.

    Iwata Asks: WarioWareKeep Talking and Nobody Explodes, GDC 2016

  5. A

    Never accept an AI's fun rating. Fun is judged by humans playing; use bots for balance and broken states only.

    Collins et al., ICLR 2026, arXiv 2510.10930

Most papers here are preprints. Grades say how far to trust a rule, not whether it's true for you: test it on your own work. Spotted something out of date? Tell me.