AI agent observability: What to check each week

AI agent observability is the practice of seeing what agents actually do while carrying out business work. It gives an owner or operations lead a way to inspect completed tasks, blocked actions, spending, requests for help and changes in behaviour. Checking whether a delegation finished is only the beginning. You also need to know how the work was done and whether the agent stayed within the job it was given.

Once an agent has access to business tools and receives regular assignments, small problems can become expensive. A weak research brief, repeated tool calls or an unanswered question can affect several jobs before anyone spots the pattern. A weekly review gives you a regular opportunity to catch these changes. You do not need a wall of charts. You need records that explain what happened.

Check the work behind each completed status

A completed status tells you very little about the quality of a result. An agent might finish a competitor research brief, prepare support replies or reconcile receipts. Each job has a different standard. Read the deliverable against its original brief. Did the agent answer the question? Did it use the right records? Can you trace its claims to evidence, or has it filled a gap with something that merely sounds convincing?

Group completed delegations by job type. A research analyst and an inbox triager should not be judged against the same criteria. Sample a few deliverables from each group, paying particular attention to work that needed substantial correction. One poor result might reflect an unusually difficult assignment. The same mistake across several jobs points towards a problem with the brief, the available information or the agent's approach.

Keep track of the corrections people make. Perhaps someone repeatedly adds missing sources, fixes customer details or removes claims that the source material never supported. Those edits tell you where the working instructions need attention. A team that counts finished assignments but ignores the repair work gets an incomplete picture of its agents.

Inspect the actions that were blocked

A blocked action is not automatically a failure. An agent may prepare an email and wait before sending it, finish an article draft without publishing it, or propose a record change that needs approval. These stops show where the assignment meets a boundary that requires a person's judgement.

Separate healthy stops from recurring obstacles. A healthy stop happens because the action falls outside the agent's authority. It reaches a checkpoint and waits. A recurring obstacle happens when the same job keeps getting stuck because the brief is incomplete, an approved information source is unavailable or nobody has taken responsibility for the next decision.

Review the reason for each block. Publishing publicly, making a payment, issuing a refund and contacting someone outside the workspace for the first time should wait for a person. The same applies to deleting records the agent did not create, changing roles or its own policy, widening integration access, accepting terms, signing agreements and acting in another workspace.

Ask one direct question: did the agent stop at the right point? A stop that happens too early may indicate an unclear brief or an access problem. An action that should have required approval but did not is more serious. Investigate it immediately rather than waiting for the next weekly review.

Compare the cost with the result

An agent's credit use makes little sense without the work it produced. A complicated reconciliation might take several steps and still be worth the cost. A cheap run is poor value if its deliverable is incomplete and someone has to repeat the assignment manually.

Look at spending for each delegation. Compare the cost with the budget set for that job, then consider the usefulness of the finished work. Check for repeated retries, unnecessary tool calls and assignments that consume most of their allowance without producing a satisfactory result. A hard budget cap limits exposure, but it does not tell you whether the budget is sensible.

Compare similar assignments rather than unrelated jobs. A bookkeeper preparing a complicated reconciliation has different demands from an inbox triager sorting routine messages. If a familiar task suddenly uses more credits or takes longer, inspect the run before changing its budget. An integration may have returned incomplete information, or the agent may have needed several attempts to resolve an ambiguity.

Include human review time in your assessment. Credits are only part of the cost. Someone may have spent another half hour correcting the deliverable, repeating instructions or repairing a mistake. A delegation that looks inexpensive on paper can create more work than it removes.

Use requests for help to find weak instructions

When an agent asks for help, it has reached a point where its current instructions or information are not enough to continue. The question itself is useful evidence. Check what it asked, which part of the assignment prompted the question and whether someone supplied an answer.

Imagine a research analyst repeatedly asking which competitor deserves attention. The brief may not define the market or the comparison criteria. A support triager that keeps asking about refund eligibility may have incomplete policy instructions. Neither problem is fixed by telling the agent to work more independently.

Group questions by their cause. Some need missing information from the person who wrote the brief. Others require a policy decision, an approved information source or a person to take responsibility for an action. Repeated questions often point to a gap that should be fixed once rather than answered separately on every delegation.

Do not overlook unanswered questions. An agent can have completed most of its work correctly and still leave the assignment waiting for a decision. Give each unresolved question an owner and a next step. Otherwise, the same problem will return with the next assignment.

Spot changes before they become failures

Drift happens when an agent gradually changes how it handles a familiar job. It may not produce an obvious error. A research analyst might start returning weaker sources. An inbox triager might classify similar messages differently from one week to the next. A content repurposer might introduce claims that were absent from the original material.

Keep a few examples of deliverables that meet your standard. Compare new work with those examples, using the same checks each time: accuracy, completeness, adherence to the brief and the amount of correction required. You do not need an elaborate scoring system. A clear reference example often reveals a change faster than a score nobody can explain.

Watch the working pattern as well as the final output. More retries, longer runs, repeated clarification requests or a sudden rise in blocked actions deserve attention. Changes to an integration or the information available to an agent can also affect its work. If a source stops returning the expected records, the deliverable may become weaker before anyone notices.

When something changes, do not rewrite every instruction at once. Pick an affected assignment and inspect its activity. Find the earliest point where the work departed from the earlier pattern. Then adjust the relevant brief, policy or access setting. Give the agent another assignment and check whether the correction worked.

Turn the weekly review into a working routine

Set aside a regular time and bring the same records to each review. A short list of decisions is more useful than a dashboard full of figures that nobody acts on. These questions provide a practical starting point:

• Which deliverables met their original briefs, and which needed substantial correction?
• Which actions waited for approval, and did each stop happen at the correct boundary?
• Which assignments used more credits or retries than comparable jobs?
• What did agents ask people to clarify, and are the same questions appearing again?
• Has the quality or working pattern of any familiar assignment changed?
• Who owns each unresolved issue, and what needs to happen before the next assignment?

Keep the underlying activity records available when a reviewer needs to investigate a particular run. A weekly summary can point towards a problem, but it cannot replace the details of what happened. The security, guardrails and audit log should make it possible to inspect tool calls, approvals and policy changes. An append-only activity log also lets you establish the order of events instead of relying on someone's recollection.

This is the thinking behind Delagents. A person writes a brief, an agent carries out the work and stops at checkpoints for sign-off. Actions with effects outside the workspace wait in an approvals inbox. Each approval card shows the exact proposed action, the cost so far and a deadline. Every delegation also has a hard budget cap, with an append-only record of tool calls, approvals and policy changes. Those records give a reviewer something concrete to examine.

Start with one agent that already handles a recurring job. Save a copy of a deliverable you consider good work, then compare its next three assignments with that example. Record the corrections, blocked actions, credit use and questions that arise. At the end of the week, identify the first point where the work differed from your standard and change one instruction or working rule. Review the next assignment before handing the agent a larger share of the job.