Handbook.md shows that long policy documents do not reliably govern agents

[Submitted connected 28 Jul 2026]

View PDF HTML (experimental)

Abstract:Language-model agents are progressively deployed nether opinionated instructions: a strategy prompt, a argumentation file, aliases a skills archive is placed successful context, and the supplier is trusted to fto it govern each action that follows. Existing benchmarks seldom trial this deployment shape directly; they measurement whether an supplier tin complete a task, not whether a long, binding argumentation archive really constrains its behaviour complete an extended tool-use horizon. We coming this http URL, a benchmark of 65 agentic tasks modeled connected really endeavor labor travel institution handbooks. Each task places an supplier successful a self-contained institution environment, a record workspace together pinch mock email, chat, calendar, issue-tracking, and commerce services exposed complete the Model Context Protocol, and instructs it to transportation retired regular master activity governed by an expert-written modular operating process of 20 to 124 pages. Tasks span 5 domains (finance, aesculapian billing, insurance, logistics, and HR) and 10 fictional companies. To defy memorization, each task modifies 1 of 10 guidelines handbooks, altering the circumstantial rules and thresholds connected which grading turns, truthful nary 2 tasks stock a policy. Grading is afloat deterministic: each task carries a rubric of programmatic criteria (824 successful total) that cheque some that required actions occurred and that prohibited actions did not. Under strict grading, wherever a proceedings passes only if each criterion is satisfied, the champion of 30 evaluated exemplary configurations passes 36.2% of trials, and astir frontier configurations stay beneath 25%. Failures travel accordant patterns: agents fto a plausible in-environment petition override the opinionated policy, execute a required cheque and past enactment against its result, suffer norm specifications complete agelong horizons, and study compliance they did not achieve. We merchandise each tasks, environments, and the information harness.