Retry, Idempotency, and Failure Recovery in SitecoreAI Agents

A retry policy should begin with the effect the workflow may already have produced. A request can fail before reaching a service, return a confirmed rejection, or lose its response after the remote action completed. Those outcomes require different decisions. Treating all of them as “run it again” can repeat content creation, notifications, or other consequential operations. Treating all of them as permanent failure can abandon work that is recoverable.

This article proposes a recovery architecture for a hypothetical SitecoreAI campaign agent that prepares a draft, writes approved content, and optionally requests an external notification. The records and state names are application examples, not native Sitecore schemas. No universal retry count, timeout, or exactly-once guarantee is claimed. The actual tool and destination contracts determine which operations can be repeated and what evidence can establish their outcome.

My position is that a visible uncertain state is preferable to an automatic repeat of a consequential action whose outcome cannot be established. That creates manual reconciliation work in some cases. I would accept that cost until the integration has a reliable correlation mechanism. The practical goal is to recover the intended business task while preserving completed effects, authorization scope, and evidence about what remains unresolved.

1. Classify outcomes before deciding to retry

Distinguish transport evidence from business evidence

Start by recording what the caller actually knows. A local validation failure before submission differs from a remote response rejecting the payload. A connection interruption after submission may leave the effect uncertain. A successful response may confirm only acceptance rather than completion. Use the documented interface to interpret each observation. Do not assign a business state solely from a generic exception name. The recovery decision should depend on both the error class and the operation’s possible side effect.

The Amazon Builders’ Library discussion of timeouts and retries explains that a timeout does not establish absence of side effects and that retries can increase load. These are general distributed-systems principles, not Sitecore-specific guarantees. Sitecore’s agent selection guidance recommends workflow agents for checkpointed work and consequential actions. Apply those references by inspecting the exact boundary being recovered rather than assuming a resumable workflow makes all downstream effects safe to repeat.

For the campaign example, classify reading a source, generating a draft, creating a content item, updating an item, and requesting a notification separately. A repeated read may observe newer data. Repeated generation may produce different content. A repeated creation may produce an additional item unless the interface prevents it. A repeated notification may reach the audience twice. These are risk scenarios to assess, not assertions about a particular tool’s undocumented behavior. The operation inventory should state the verified contract for each one.

Build an outcome classification the operator can use

Propose application outcomes such as not issued, confirmed rejected, accepted awaiting verification, confirmed completed, and uncertain. Define which evidence permits each transition. A local record saying issued does not prove the remote effect happened. A missing local confirmation does not prove it did not. An operator should see the distinction without reading every transport log. Keep the state linked to the intended action and the provider response reference under the appropriate access rules.

Separate retry eligibility from completion state. An uncertain operation may be eligible for a documented idempotent repeat, or it may require inspection first. A confirmed rejection may be retryable only after correcting an input or permission. A confirmed completion should generally lead to the next stage rather than another attempt at the same effect. The policy should state these decisions explicitly. Avoid one boolean called failed that hides whether the process needs new evidence, changed input, or a repeated request.

Treat invalid input as a correction problem. If a destination identifier is wrong or a required field is missing, repeating the identical request without a changed condition is not a useful recovery plan. Route the issue to the component that can correct it. A model may help draft a corrected proposal, but it should not silently change an approved payload and submit it under the old authorization. Preserve the relationship between the rejected attempt and the new proposed effect so the reviewer can inspect what changed.

Define the evidence required to continue

For each operation, list the postcondition that demonstrates the business requirement. A draft creation may require a readable destination item with the intended source relationship. An update may require the expected field revision. A notification may need a documented delivery-state observation appropriate to the business process. Do not invent stronger confirmation than the provider exposes. If the interface only reports acceptance, retain that state and specify what later evidence or manual check is needed.

Keep warnings visible when they matter to recovery. A response may indicate partial success or a condition that requires follow-up. Normalize the result into a narrow application record without discarding those distinctions. Sitecore’s Invoke Tool guidance describes success and failure envelopes and potentially nested details. Inspect the actual output shape before classifying it. A syntactically valid envelope should not automatically become a completed business operation.

Test the classifier with representative provider responses and controlled interruptions. Include a failure before submission, a rejection, an accepted operation awaiting verification, and a lost response after a remote effect. State the expected application outcome for each fixture. The test should check the next permitted action, not only the error label. A classifier that calls every exception transient may look simple while allowing repetition at the most consequential boundary.

End this stage with an operation-specific recovery table. Give each outcome an owner and a permitted next step. The drafting stage may return for correction; an uncertain write may enter reconciliation; a confirmed completed effect may resume from the next checkpoint. My preference is a small table tied to real operations over a broad retry decorator applied to the entire workflow. The table forces the team to identify the evidence that makes recovery appropriate.

2. Define operation identity around business intent

Identify intent before sending the request

Assign an operation identity to the business effect the user authorized. In the hypothetical workflow, that identity could represent a particular campaign, destination, language, and approved packet revision. Determine the exact composition according to the task. It should remain stable across transport attempts for the same effect. Do not generate a new identity merely because the caller timed out. Also do not reuse an identity for a later intentional change just because the destination and campaign happen to be the same.

The Amazon Builders’ Library article on idempotent APIs describes identifying repeated intent with a client request identifier and handling the same identifier with different parameters. Use the principle with the provider’s documented mechanism where one exists. The source discusses AWS designs; it does not prove that an arbitrary Sitecore tool accepts an idempotency key. Verify supported fields, scope, retention, and conflict behavior before relying on them.

Keep business-request identity separate from execution-attempt identity. A user request can have several attempts while still representing one intended effect. An attempt record helps diagnose transport behavior. The business identity helps prevent an operator from confusing a recovery with a new campaign action. Preserve both where the integration needs them. Avoid using a conversation title as the sole key, because a title may not uniquely identify the destination, revision, or requested operation.

Bind the identity to the payload and scope

Associate the operation identity with a canonical representation of the approved effect. Define what that representation covers: destination, action class, field assignments, and relevant authorization references. If the integration computes a digest, record the representation and normalization rules. A digest of only the article text would not detect a changed target. The purpose is to detect conflicting reuse of an identity, not to pretend the digest establishes approval or factual truth.

Reject or route a conflict when the same identity appears with a different intended effect. Do not silently overwrite the original payload and call the new request a retry. The organization may choose to cancel an old intent and create a new one, but that should be explicit and traceable. A reviewer approving the first packet should not appear to authorize a later generated revision merely because the integration kept the same operation key for convenience.

Define deduplication scope and lifetime. A provider’s idempotency guarantee may apply only to a particular identity scope and retention period. Your own ledger also needs retention and lookup rules. Do not assume that a key remembered indefinitely by one component remains meaningful to every downstream system. If a recovery happens after the supported correlation window, inspect the destination or escalate according to the documented policy. The correct action may differ from the one permitted immediately after the original attempt.

Use a durable intent record where the application needs one

The following example is an integration-owned operation record. It is not a Sitecore API payload or a claim that a specific tool accepts these fields.

{
  "operationRef": "campaign-guide-approved-change-17",
  "attemptRef": "transport-attempt-2",
  "approvedPacketRef": "review-packet-17",
  "destinationRef": "verified-item-and-language",
  "payloadRevision": "approved-field-set-4",
  "state": "uncertain",
  "providerResultRef": null
}

Record the intended effect before issuing it where the surrounding architecture supports durable storage. Then retain the provider result reference when received and verified. This gives an operator a starting point after a process interruption. The admitted limitation is that a local intent record does not make a remote side effect and a local confirmation one atomic transaction. A crash between them still creates a gap that needs idempotency or reconciliation.

For several effects in one workflow, use distinct operation identities. Creating a draft and sending a notification should not be hidden under one generic completed flag. They have different postconditions and recovery limits. Link them to the same campaign request while preserving their separate outcomes. A later correction to the draft should not automatically repeat a previously confirmed notification unless that new notification is itself an authorized effect with a new identity and evidence requirements.

Test duplicate and conflicting submissions deliberately. Submit the same intended operation through the supported mechanism twice and inspect whether the documented behavior holds. Then submit a changed payload under the same identity in a disposable environment. Verify the application’s conflict policy and the provider’s response. Do not report an exactly-once guarantee from one successful example. The useful result is evidence that your chosen identity and conflict rules behave as expected under the specific contract you intend to use.

3. Put durable checkpoints around consequential effects

Place checkpoints at business boundaries

A useful checkpoint records a confirmed result that later work can reuse without repeating its effect. In the hypothetical campaign, the approved content packet can be one checkpoint. A verified destination write can be another. A notification confirmation can be a separate checkpoint. Choose boundaries according to what would be expensive, unsafe, or misleading to repeat. Do not checkpoint every internal text transformation simply to create a long execution history. The record should answer where recovery can resume with the intended meaning intact.

Preserve the artifact revision needed by the next stage. A retry of content generation may produce a different draft. If the original draft was approved, recovery should not silently substitute the new one under the old decision. Determine whether to reuse the existing artifact or create a new proposal and obtain the required review. This is a governance choice tied to operation identity. Re-running a generation step is not necessarily a transparent technical retry when its output determines a consequential effect.

Separate the workflow’s execution checkpoint from the remote action’s outcome. A step may be marked completed while additional business verification remains. Conversely, a remote action may have completed while the workflow failed before recording the checkpoint. The application record should preserve those possibilities. Establish what the native run state means for the actual action and interface. Do not infer a stronger destination guarantee from a generic workflow completion label.

Record transitions with the evidence behind them

For a write operation, record planned intent, issuance, provider response, and verified postcondition as distinct facts when needed. Use supported durable mechanisms in the platform or the surrounding service. An event history can help explain changes, but the implementation also needs a consistent current decision about what may happen next. Keep the relationship between history and current state clear. A collection of logs is not automatically an operational state machine that prevents duplicate execution.

When the application owns a database transaction, use it for the local records it can actually control. Do not extend that local guarantee rhetorically to an unrelated remote service. If the architecture uses an outbox or similar coordination pattern, describe the specific delivery and deduplication contract, and test it. This article does not prescribe an unverified implementation library. The core requirement is to make the remaining gap explicit rather than hide it behind the word durable.

Consider simultaneous recovery attempts. Two workers or an operator and a background process might try to resume the same operation. The application needs a supported concurrency rule for claiming and updating the local work record. Determine what happens if a worker loses ownership while its remote request is still in flight. Do not assume a local lock can cancel a request already sent to another system. Preserve the uncertainty and rely on documented remote idempotency or reconciliation where needed.

Keep post-approval recovery within authorization

Before a resumed action executes, check whether the approved packet and destination assumptions remain valid under policy. A campaign can expire while the workflow is interrupted. A destination may change through ordinary editorial work. Use a documented conditional-write mechanism when the interface supports it, or choose a controlled process that states its limitations. A successful recovery should not overwrite legitimate newer work merely because the old request was once approved.

Preserve side-effect ordering where the business depends on it. A notification about a completed draft should follow the verification required by the process, not merely the issuance of a write request. If the write is uncertain, decide whether the notification must wait. Make that dependency explicit in the state model. An external message saying work is ready can mislead people even if the underlying content eventually appears. Recovery correctness includes what people are told about the result.

Define checkpoint retention according to the period in which the workflow may need to explain or recover an effect. Apply the organization’s access and retention policy to payloads and evidence. Keep stable references when detailed records expire under that policy. Do not retain sensitive source material indefinitely by default. An operator should understand when the available record is no longer sufficient for automatic recovery and a separate investigation or renewed authorization is required.

The acceptance exercise should interrupt the process at several points around one side effect. Stop before issuance, after issuance without confirmation, after provider confirmation, and after local checkpoint storage. Resume through the supported mechanism and inspect the destination. Record the last confirmed evidence and the next permitted action in each case. These tests reveal whether the checkpoint design expresses actual recovery boundaries or merely mirrors the visual sequence of boxes in the workflow editor.

4. Reconcile uncertain actions without inventing certainty

Treat uncertainty as work to resolve

Reconciliation asks whether the intended effect exists and whether it belongs to the operation under investigation. It is not a search for any vaguely similar resource. In the campaign example, inspect the supported destination reference, operation correlation data, approved payload revision, and relevant scope. A draft with the same title may have been created by another campaign or a person. The reconciler needs enough evidence to link the observed artifact to the specific intended action before marking the operation complete.

Choose the correlation mechanism during design. A documented provider operation identifier is preferable when it gives the required relationship. A supported external reference field may work for a custom integration if the destination model and permissions permit it. An application ledger can preserve known returned identifiers. Do not invent a hidden metadata field or assume that arbitrary request properties are stored remotely. If the interface cannot support reliable correlation, make manual inspection part of the recovery process before automating writes at scale.

For an uncertain update, compare the destination with the approved postcondition and current editorial state. A matching value may be relevant evidence, but determine whether it establishes the required operation history or merely the present state. The business may care only that the desired content exists, or it may need attribution to a particular approved action. State that distinction. Do not mark an operation verified under a stronger claim than the observation can support.

Define reconciliation outcomes explicitly

Propose outcomes such as confirmed effect, confirmed absent under a documented check, conflicting destination state, and unresolved. Each should have a permitted next action. A reliable confirmed effect can resume from the next checkpoint. A conflict may return to an editor. An unresolved case should preserve the uncertainty and stop automatic repetition if the operation is consequential and lacks a safe repeat contract. Avoid collapsing unresolved into absent simply because the reconciler did not find a match immediately.

Consider the visibility behavior of the destination. A newly completed effect might not appear through every read path at the same instant. Establish the actual documented consistency and lookup behavior for your chosen interface. Do not assume an immediate failed search proves absence. Where the contract permits bounded verification polling, use it with a deadline and an explicit exhaustion state. If the required evidence cannot be obtained automatically, route the case to an operator rather than extending the search indefinitely.

Handle partial effects separately. A page may exist while a related content assignment or asset reference did not complete. The reconciler should inspect the postconditions for each operation rather than treating existence of the parent as full success. Decide whether the incomplete result remains a draft, can be completed through a new permitted operation, or requires correction. Preserve existing human edits. A cleanup process that removes a partially completed item without inspecting subsequent use can cause a new problem while trying to fix the old one.

Give operators a decision packet

Show the operator the intended destination, operation identity, approved payload reference, and known observations. Provide links to the supported destination and run evidence. Keep secrets and unnecessary sensitive content out of the default view. The operator should know what they are being asked to establish and which action they may take afterward. A vague message saying “retry failed job” does not provide enough context to judge whether a previous effect could already exist.

Record the operator’s conclusion and supporting evidence. If they confirm that the intended draft exists, link the verified destination and resume from the appropriate checkpoint. If they authorize a new attempt after investigation, preserve the relationship to the original uncertain attempt. If the effect must change, create a new approved intent rather than rewriting history. The goal is an explainable resolution, not simply a green status. Future investigation should distinguish automatic confirmation from a human decision made under limited evidence.

Design correction separately from deduplication. If two drafts do exist, deciding which one to keep may require editorial context. If a notification was duplicated, deleting a local record does not retract what recipients already saw. Define supported corrective actions and their limitations before release. Do not describe compensation as a universal reversal. The business recovery goal may be to publish a correction or contact an owner, which is a different authorized task from undoing a technical write.

Test reconciliation with ambiguous evidence, not only a perfectly tagged artifact. Include a similar title from another operation, a changed destination revision, a partial write, and a lookup that temporarily fails. The expected outcome should preserve uncertainty where the evidence is insufficient. My preference is for the reconciler to ask a person a precise question rather than infer certainty from weak similarity. That may slow exceptional recovery, but it prevents a convenient lookup heuristic from becoming an undocumented authorization to repeat or delete content.

Consider an illustrative investigation of a draft creation. The local record says the request was issued but contains no returned item identifier. The operator finds two similar drafts. One has a supported correlation reference matching the operation and the expected destination; the other belongs to a separate campaign. The first is evidence relevant to this investigation. The second should remain untouched. If neither draft has a reliable relationship to the operation, the operator should preserve that uncertainty rather than choose the more recently created title as a convenient answer. This scenario is a proposed rehearsal fixture, not a report of an observed deployment incident.

After establishing the correlated draft, inspect the intended postcondition. If the required field assignment is incomplete, creation may be confirmed while the overall campaign task remains unfinished. Record those two facts separately. A later corrective update can then be evaluated as its own permitted effect without repeating creation. This distinction lets the operator continue useful work while keeping the history accurate about what the original attempt did and did not establish.

5. Bound retries and protect dependent systems

Use an operation-specific retry budget

Choose retry eligibility, attempt limits, and total deadlines according to the dependency and the operation. Verify any provider guidance and client-library behavior first. A budget should account for time spent waiting, not merely the number of requests. A campaign with a near deadline may need an earlier escalation. A background draft task may tolerate a longer recovery window. This article supplies no universal Sitecore timeout or retry count because those values require observed dependency behavior and business constraints.

The AWS retry-with-backoff guidance discusses delaying repeated attempts and considering idempotency. Apply that general pattern to the contract you actually call. Backoff and jitter can spread retry traffic, but they do not make a non-idempotent effect safe to repeat. Keep the permission to retry separate from the timing algorithm. First establish that repeating the request is appropriate; then decide when and how often the application may do so.

Account for retry behavior at every layer. A transport library, connector, service, and workflow may each have its own policy. Inspect the actual configuration so one user-visible attempt does not conceal many downstream submissions. Choose a responsible layer for the operation’s recovery decision and keep other layers consistent with that choice where you control them. Do not add an outer “retry everything” loop without understanding what the inner components already repeat and which effects they can produce.

Protect the dependency during failure

Limit new work and repeated work when the dependency is degraded according to your operating policy. A growing queue of retried campaigns can compete with new authorized work or other users of the same service. Define a way to pause intake or restrict the affected action while preserving existing records. Do not assume that cancelling a local task cancels an already issued remote request. The operator view should distinguish prevented future actions from requests still in flight.

Use provider retry hints where documented and supported. Verify their interpretation in the actual client. A general error classification based only on status-code ranges may be too coarse for a particular API. An authorization problem may require an identity correction; a transient service condition may permit a delayed repeat; an uncertain write may require reconciliation. Keep those decisions in the operation policy. Avoid asking the model to invent an arbitrary delay or decide that a repeated denial will eventually become permission.

Bound verification polling as well as request retries. A workflow waiting for a destination postcondition can otherwise become indefinite background work. Record the observation deadline and the state reached when it expires. The correct outcome may be unresolved rather than failed or complete. An operator should know whether the request was issued, what evidence was checked, and why the automatic path stopped. That makes the budget a practical recovery control rather than a hidden implementation setting.

Observe amplification and exhausted work

Track attempts per operation identity, time spent in recovery, uncertain outcomes, and the reasons automatic recovery stops. These are proposed measures, not observed results from this article or native telemetry field names. Define the units and populations so operators can distinguish many attempts for one intent from many separate user requests. Preserve enough correlation to follow the work across the components you control without copying sensitive payloads into every log event.

Review retry exhaustion by cause. A repeated invalid payload needs a different fix from a slow dependency or an unavailable correlation mechanism. Route the issue to the owner who can change the relevant condition. Do not make every exhausted run a prompt-tuning task. Likewise, do not increase attempt limits merely to reduce the number of visible failures. A higher limit may postpone escalation while leaving the business uncertainty unchanged. Inspect representative records before changing the policy.

Plan what happens when the recovery queue grows beyond the team’s ability to inspect it promptly. Prioritize according to business impact and retain clear ownership. Restrict new consequential work if the organization cannot safely reconcile the existing uncertain operations. A smaller bounded workload with known outcomes can be preferable to a large backlog whose eventual repeats are poorly understood. The appropriate threshold is a local operating decision; document the reasoning and the action the owner will take when it is reached.

The acceptance test for the budget should show that the workflow stops automatic recovery at the agreed boundary and preserves the information needed for the next decision. Include a dependency that remains unavailable and an invalid request that never becomes valid. Verify that neither produces an unbounded loop. Then test a permitted transient case and confirm that the recovered result remains correlated with the original intent. A good retry policy recovers eligible work without treating persistence itself as proof that another attempt is justified.

6. Rehearse recovery before production depends on it

Rehearse the uncertain-write window

Use a controlled destination and disposable content to reproduce the difficult gap. Let the remote write complete, then interrupt the caller before local confirmation is stored. Resume through the supported workflow or integration mechanism. The expected result is a correlated existing effect, a documented safe repeat under the provider contract, or an explicit uncertain case for reconciliation. It should not be a blind second creation presented as ordinary success. Preserve the evidence so the team can explain exactly which window the exercise covered.

Test interruptions before and after approval separately. Before approval, the process may return to drafting or review. After approval, it must preserve the authorized packet and avoid substituting newly generated content. Include a changed destination while the run is stopped. The recovery should follow the concurrency and invalidation policy. Do not describe a successful restart as adequate if it silently overwrites a newer human edit. The rehearsal should inspect the destination and the authorization record together.

Include a notification boundary if the workflow has one. Confirm what the provider response means and what the business considers completion. Simulate a lost response under controlled conditions and follow the actual reconciliation procedure. If the provider offers no reliable repeat or correlation mechanism, document the manual decision required. Do not claim that a workflow checkpoint can retract a message or guarantee that it was delivered exactly once. The recovery plan must state the limitations of that effect.

Hand the runbook to someone else

Ask an operator who did not implement the workflow to resolve a prepared uncertain case. Give them the ordinary evidence surfaces and the runbook, not private hints from the builder. They should identify the intended effect, last confirmed observation, and permitted next action. Record the questions they cannot answer. Those gaps may belong to missing identifiers, unclear state labels, or an incomplete destination inspection procedure. Fix the evidence path rather than expecting operators to memorize hidden implementation details.

Write each runbook entry around a decision. Explain how to inspect the remote state, what supports confirmation, and which conditions require escalation. Include the owner who can authorize a changed effect or a corrective action. Avoid “rerun the agent” as the default instruction for every problem. A runbook should help the operator preserve completed work and distinguish recovery from a new request. It should also explain how to stop further downstream actions while an uncertain operation is being investigated.

Use Sitecore’s workflow troubleshooting guidance for the supported run-inspection path. Supplement it with your integration’s operation records and destination evidence where needed. State what remains invisible to the operator and how to obtain additional support. Do not claim that one logging surface explains every external effect. A practical runbook connects the evidence the deployment actually provides and identifies the owner of any remaining uncertainty.

Release recovery behavior with the workflow

Treat changes to retries, identity rules, or reconciliation as production behavior changes. A new timeout can alter how often the workflow enters uncertainty. A changed deduplication key can alter which requests the application considers the same intent. A new tool can introduce an effect the old runbook does not cover. Review and test those changes with the same care as the forward path. Do not hide them under a generic maintenance label merely because the campaign’s visible content did not change.

Start with a bounded action scope and collect real outcomes. Suggested observations include how often reconciliation is needed, whether operators can resolve cases from the available evidence, and which failure classes recur. These are proposals for measurement, not performance results. Use them to improve the specific boundary causing work. A decrease in visible exceptions is not automatically an improvement if the new behavior silently repeats actions or marks uncertain outcomes complete. Review actual examples alongside aggregate counts.

Before broad release, require an operation inventory, stable intent identity, checkpoint evidence, and a tested reconciliation path for every consequential effect. Add bounded retry and verification policies, plus a named recovery owner. Scale the machinery to the consequence, but do not remove the reasoning. A draft-only workflow may need fewer controls than one that writes shared content and contacts an external audience. The release record should explain why the selected design fits the actual scope.

The next implementation step is to choose one side effect and trace its failure windows. Ask what happens if the caller stops immediately before submission, immediately after the remote effect, and immediately after local confirmation. For each window, identify the evidence and permitted recovery. Close the first gap where the current answer is simply to try again. That gives the team a concrete improvement in recovery behavior without claiming that any distributed workflow can eliminate uncertainty by adding another retry loop.

Keep the rehearsal results tied to the configuration tested. Record the client retry settings, relevant operation contract, and evidence used to resolve the uncertain effect. If those settings or interfaces change, identify which exercise must be repeated. A runbook validated against an older connector may no longer describe the actual attempts being issued. Preserve that limitation visibly and choose a bounded release while the new behavior is checked. The goal is a recovery procedure supported by current evidence rather than a document whose reassuring title outlives the implementation it originally described.