Reliability, Safety, and Governance

The central problem of agent technology is not making an agent act. It is making the agent act correctly, predictably, and within acceptable limits.

Common failure modes

• Choosing the wrong goal or interpreting the goal too broadly
• Calling the wrong tool or using correct tools in the wrong order
• Continuing after the task is already complete
• Trusting inaccurate data
• Failing to notice that an action did not succeed
• Exposing private information or exceeding authority

Guardrails

Useful controls include read-only modes, approval gates, transaction limits, tool-specific permissions, simulation before execution, immutable audit logs, and automatic escalation.

Human-in-the-loop design

Human review should be concentrated at high-consequence decision points rather than added randomly. For example, an agent may prepare a payment but require approval before releasing funds.

Measurement

Agents should be evaluated on completed outcomes, not just the quality of individual messages. Important measures include task success, error rate, recovery rate, cost per completed task, time saved, and percentage of cases escalated.

Autonomy without observability is not efficiency. It is hidden risk.