Skip to content
All insights

AI & Automation

Always-On AI Agents That Recover From Retries

Build an always-on AI agent with durable state, safe retries, and a clear recovery plan for actions such as sending messages or updating records.

An always-on AI agent wakes up for a message, webhook, or timer, does a bounded piece of work, and saves enough state to resume later. The first design decision is what happens when an action succeeds but the reply goes missing. If a customer already received an acknowledgment, the next run should recover the receipt or ask for help before attempting another send.

Key takeaways

  • Keep workflow state and delivery receipts outside the model session.
  • Give each outbound action a stable ID and reconcile ambiguous delivery.
  • Rehearse duplicate events, restarts, and lost responses in a test environment before enabling live writes.
Applying this to your product?Plan your agent workflow

Choose the event that wakes the agent

Start with one trigger and one outcome. A new support ticket can wake a worker that reads the message, checks whether the case needs escalation, and drafts an acknowledgment. A timer can wake it again when a review is due. Between those events it can wait without running the model.

Anthropic’s November 13 guide describes an event inbox, timers, memory, permissions, and an outbox. A first rollout can keep those pieces in a single database and worker. Use the model where interpretation helps, such as classifying a vague request. Enforce record uniqueness, authorization, and action limits in application code.

Anthropic’s engineering guide to building always-on agents

Save progress where the next run can find it

Store each input with its provider, event ID, timestamp, and affected work item. A uniqueness constraint on the source and event ID lets the worker accept a repeat delivery without creating a new job. Claim work with a short lease and a way to recover expired claims. If two events can change the same ticket, serialize changes per ticket or reject writes against an old version.

Keep the ticket’s current state, approval, and provider receipts in a durable record. A conversation summary helps a model interpret what happened. The receipt and stored version tell a worker which action has already happened and where it can safely continue.

Handle duplicate incoming events in an n8n webhook workflow

Recover from a timeout after a successful send

Consider a hypothetical support ticket, T-184, that qualifies for a canned acknowledgment. Commit the decision and a single outbox row in one transaction. Give the row a stable effect ID such as T-184:ack:v1. It stores the evaluated recipient, approved message, attempt state, and provider message ID when available. If the provider supports idempotency keys, pass that same effect ID on every valid retry.

An outbox keeps the local decision and pending send together. Delivery still needs its own recovery logic. Check how long the provider honors a key and keep automatic retries inside that window. When the result is unknown and there is no valid key or receipt lookup, route the row to a named operator for reconciliation.

Illustrative recovery states for one ticket acknowledgment
Where work stoppedWhat to checkNext action
Before outbox commitLatest ticket state and approvalResume and create the effect once
Outbox committed, provider reply lostSame effect ID, receipt lookup or valid retry keyReconcile or retry the same effect if covered
Provider acceptance recordedProvider message ID on the effect recordClose the task with the receipt attached
Delivery unclear and lookup unavailableLast attempt and affected ticketPause sending for this effect and assign a review

AWS guidance on the transactional outbox pattern

FROM READING TO DOING

Where should your agent pick up after an interruption?

Bring one workflow, the events that start it and the actions it needs to take. We can help define durable state, safe retries and the steps your team should review.

Give operators a way to pause and intervene

Put a named owner and expiration on approvals. Save the approved text and recipient. Before delivery, confirm the ticket is still open and the action still matches the approval. Send changed drafts back for review. Limit credentials to the approved mailbox and action, and keep a pause control that stops new dispatches while leaving events and receipts available.

  • Watch the age of the oldest event, expired claims, unresolved deliveries, and approval wait time.
  • Record the input ID, work item, effect ID, status and timestamp. Keep sensitive ticket contents in the approved record store.
  • Begin in draft mode with representative tickets. Review when the workflow would send, wait, or escalate before opening live actions.

Set approval and escalation rules for human-in-the-loop AI workflows

Run a three-failure release drill

Use a synthetic ticket and a sandbox mailbox. Deliver the same event twice, stop the worker after it commits the outbox row, and simulate losing the response after the provider accepts a send. Check the database and the mailbox after each fault. A passing run ends with one confirmed acknowledgment, or with a review item that has enough evidence to resolve an unknown delivery.

Copy the agent recovery test brief

Use it in a test environment before enabling live customer contact.

Workflow and owner: Support acknowledgment, support operations lead.
Fixture: One test ticket and one authorized test destination.
Faults: Duplicate event, crash after commit, and reply lost after acceptance.
Evidence: One effect ID, attempt history, provider receipt when available.
Pass: One confirmed delivery or an assigned review with dispatch paused for the effect.

Frequently asked questions

Does an always-on AI agent run a model all day?

An event-driven agent starts a bounded run when work arrives. It can store pending tasks between runs and use separate limits for execution time and concurrency.

When do I need a transactional outbox?

Use one when an external action must follow a database decision and the application needs to recover if it stops between the two. Pair it with a retry and reconciliation strategy for your actual delivery provider.

YOUR NEXT STEP

Let’s work through your next step.

Tell us what you’re building, what you’ve tried, and where you need a hand. We’ll work with you to define a practical way forward.

Keep reading.

Choose which optional tools can run. You can reopen these settings from the footer. These choices do not subscribe you to marketing emails.

Cookie Policy