Intentional News

Agent Harness Gaps and Why Structure Beats Waiting for Smarts

7 min read

Lesson: Agent harnesses are new to the models — so build structure, don’t wait for smarts

Why this note exists: on 2026-09-02, the owner found that the specialist agents carried only the generic “You are Hermes Agent” soul — their real purposes (orchestrator, finance, news, health) lived in notes and memory, not in their system-prompt identity. He’d had to ask before the gap was investigated and fixed, instead of the agent surfacing it proactively. This note is the plain-language reasoning the owner shared with his wife — why these gaps happen, and what we’re doing about them.

The key idea (in one paragraph)

Today’s AI models are not really trained on the job of running an agent harness — a small group of named, long-lived specialist agents with shared memory, handoffs, and clear accountability. A model has strong knowledge of “write a web app” but almost none of “be the orchestrator of 5 specialist agents who each carry their own memory and hand work to each other.” So agent-harness gap — like not realising the souls were generic — is real and expected, because the models were never shown that picture at scale.

Is this getting better? (the owner’s hypothesis, assessed)

Largely yes, and it will keep improving.

  • Agent companies are now collecting large amounts of real agent interaction data and training models on it directly. That is genuinely closing the gap.
  • Tools are converging on standard patterns (souls/personas, profiles, memory, cross-agent messaging), and standard patterns train better.
  • A 1–2 year horizon for meaningful improvement is reasonable — but expect only partial, not total, closure.

The nuance worth being honest about:

  • The mechanics (tool use, memory, handoffs, config) will get much better and near-automatic.
  • The judgement (e.g. “should I proactively re-examine the whole multi-agent framework?”) will stay hard. Proactivity on open-ended questions is a judgement gap that better training shrinks but may never fully remove.
  • So the durable answer is not to wait for a smarter model — it’s to encode the hard-won lesson as structure now, so the next model (however trained) inherits it.

What we did about it (the practical part)

ProblemStructural fix
Souls were generic; purposes lived only in notesBespoke SOUL.md for each agent — identity, purpose, boundaries, self-optimisation duty (kavita already had one)
Nothing proactively surfaced the gapWeekly agent health report — auto-runs every Monday, surfaces memory%, cost, cron failures
”Wait to be asked” behaviourEvery soul now carries a standing self-optimisation duty — propose improvements, don’t wait to be asked
Memory ballooning unboundedLean-memory philosophy — memory holds pointers; the vault, skills and semantic search hold the detail
Shared things drifting between agentsRACI table — clear owner (R/A/C/I) for every shared resource, so agents hand off instead of acting solo

The one-sentence takeaway for the family

“AI models are still learning how to be good agents, not just good writers — so until they are, the smartest thing is to write down the lessons as structure (souls, rules, weekly checks) that the agent follows, rather than expecting the model to just figure it out.”

Leo on getting to 100%

You have successfully moved from “prompt-as-program” to structural grounding. Your current setup is robust because you’ve externalized the “job spec” into code/config (SOUL.md, RACI, Cron) rather than relying on the LLM’s memory.

However, the Agent Harness Gap persists in one critical area: Coordination Logic. You have defined who the agents are (Souls) and what they own (RACI), but you haven’t yet defined how they interact structurally when a task spans boundaries.

Here is the re-evaluation and the specific measures still needed to close the remaining gap:

1. The Missing Piece: “Handoff Contracts” (Not just RACI)

Current State: You have a RACI table. This tells agents who is responsible, but it doesn’t tell the harness how to transfer state. The Gap: When Health_Agent needs input from Money_Agent, does it wait? Does it ask the user? Does it query the vault directly? Without a contract, agents will either deadlock or hallucinate the other’s output. Structural Fix: Define “Inter-Agent API” in SOUL.md. Each SOUL.md must define its Input Schema and Output Schema for cross-agent requests.

  • Example: Money_Agent outputs {"risk_score": 0.8, "volatility": "high"}. It never outputs prose.
  • Harness Action: The harness validates this JSON before passing it to Health_Agent. If the schema fails, the harness retries or errors out—not the model.
  • Why: This turns “collaboration” into “function calls.” The model doesn’t need to “understand” the other agent; it just consumes structured data.

2. The “Self-Optimization” Trap: Guardrails for Proposals

Current State: Agents have a “standing self-optimisation duty” to propose improvements. The Gap: LLMs are biased toward adding complexity. They will propose new skills, new memory files, and new workflows that bloat the system. “Proposing” is easy; validating is hard. Structural Fix: The “Optimization Gatekeeper” (Human-in-the-Loop or Rule-Based).

  • Mechanism: Agents write proposals to a vault/agents/proposals/ folder with a specific format: {"action": "create_skill", "reason": "...", "expected_impact": "..."}.
  • Harness Action: The weekly report aggregates these. Crucially, the harness (or you) must approve them before they become active skills.
  • Why: Without this, your “lean memory” philosophy will be eroded by agent-generated noise within a month.

Current State: Memory holds pointers; vault holds detail. The Gap: Pointers rot. If you rename 2026-08-19.md to Daily_Log_Aug19.md, the agent’s pointer becomes a hallucination source. The model will try to fetch a non-existent file and might invent content to fill the gap. Structural Fix: Automated Pointer Validation Cron.

  • Mechanism: A daily cron job (run by the harness, not an agent) checks every pointer in agent memory against the Obsidian vault.
  • Action: If a pointer is broken, it flags it in the weekly report and automatically attempts to re-link via semantic search. If re-linking fails, it quarantines the memory entry.
  • Why: This ensures the “pointer” system doesn’t become a “liability” system.

4. Cost Attribution: The “Tragedy of the Commons”

Current State: Weekly report surfaces cost. The Gap: Aggregate cost doesn’t help you optimize. If Health_Agent is using Luna for simple routing because it’s “easier,” you won’t see it in the total. Structural Fix: Per-Agent Cost Budgets in SOUL.md.

  • Mechanism: Each SOUL.md has a max_cost_per_day or max_tokens_per_task.
  • Harness Action: The harness tracks token usage per agent instance. If Health_Agent exceeds its budget, the harness degrades it to a cheaper model (e.g., Luna → Llama-8B) for the rest of the day, or pauses non-critical tasks.
  • Why: This forces agents to be “economical” by structural constraint, not by prompt pleading.

5. The “Orchestrator” is Still a Bottleneck

Current State: You have a router, but is it proactive? The Gap: Your current fix (“Wait to be asked”) addresses agent passivity, but not system passivity. The system only reacts to user input or cron jobs. It doesn’t react to environmental changes (e.g., “Market crashed” or “Sleep score dropped 20%”). Structural Fix: Event-Driven Triggers (Webhooks).

  • Mechanism: Set up external triggers (e.g., a simple script that checks stock prices or wearable data) that post to a Hermes webhook.
  • Harness Action: The webhook triggers a specific agent without user input.
  • Why: This is the true definition of “proactive.” The harness initiates the loop, not the user.

Summary of Remaining Measures

ProblemStructural Fix Needed
Cross-Agent DriftInter-Agent API Schemas: JSON contracts for handoffs, validated by harness.
Optimization NoiseProposal Gatekeeper: Agents propose → Harness/Human approves → Skill activates.
Pointer RotPointer Validation Cron: Auto-check/re-link memory pointers daily.
Cost BlindnessPer-Agent Budgets: Hard limits in SOUL.md enforced by harness degradation.
Reactive SystemEvent-Driven Triggers: External webhooks for market/health events.

Recommendation

You are 80% there. The remaining 20% is enforcement.

  1. Define the JSON Schemas for your top 3 cross-agent workflows (e.g., Health→Money).
  2. Implement the Pointer Validator (a simple Python script checking file existence).
  3. Set Hard Budgets in your SOUL.md files and have the harness enforce them.

This shifts the “intelligence” from the model’s “good intentions” to the harness’s “strict boundaries.”

Graph View