Observability for SitecoreAI Agents: Traces, Metrics, Failures, and Recovery

Consider this illustrative failure: a SitecoreAI agent finishes a request. The chat says the work is complete. The campaign owner opens the destination and finds an incomplete page. Somewhere between selecting a tool, invoking an API, and reviewing the resulting artifact, the system confused activity with completion. Observability should make that mismatch visible before an operator has to reconstruct it from screenshots.

For SitecoreAI agents, observability means connecting a business request to its execution steps, dependency calls, resulting artifacts, and verified outcome. Correlation identifies the work. Metrics reveal changes across many runs. Traces explain individual executions. Recovery records show whether a corrective action actually restored the intended state.

This article proposes an operational design for teams integrating applications, Agent API, workflows, and MCP tools. It includes a telemetry contract, measurement definitions, failure classification, and recovery procedures. The design prioritizes one question: can the operator determine what happened and choose a safe next action without opening sensitive prompts or repeating an uncertain write?

Scope and evidence: Product references were checked on September 7, 2026. Sitecore documents Marketer MCP as a bridge to Agent API, while Agentic studio provides its own agent and workflow experiences. These are related surfaces, not a mandatory four-hop call chain. The custom fields, metric names, run states, thresholds, and runbooks below are proposed integration conventions. They are not claims about built-in Sitecore telemetry exports, automatic trace propagation, or universal rollback support. No production measurements or customer incident anecdotes are asserted. Sitecore: Marketer MCP and Agent API overview.

1. Map the execution and preserve correlation

Begin with the execution paths that actually exist in your deployment. An external application can call Agent API directly. An AI client can invoke Marketer MCP, which uses Agent API. A workflow can execute tools and pass results to subsequent steps. Draw each route separately and mark who owns its instrumentation. This prevents the architecture diagram from promising visibility inside a managed component where you only control the calling boundary.

Sitecore distinguishes standard agents from workflow agents. Workflow agents follow a predefined sequence with checkpoints and approvals, and produce artifacts during the process. This matters operationally because a request may contain more than one observable result and may pause for a person. A workflow waiting for approval should not look identical to a stalled network call. Sitecore: Agentic Studio toolkit.

Use different identifiers for different questions

I recommend assigning a durable operation identifier when the application accepts a business request. Preserve it across retries, approval pauses, and recovery attempts. Use a separate execution identifier for each run, a step identifier for a logical workflow action, and an attempt identifier for each actual call. A new retry must never erase the evidence that a previous attempt existed.

A trace identifier serves a different purpose. It joins spans in distributed tracing, while the durable operation identifier can join several traces over a long business process. W3C Trace Context defines the traceparent and tracestate headers; OpenTelemetry provides context propagation mechanisms that let participating services preserve causal relationships. Use the supported instrumentation APIs to produce valid context instead of treating a business operation ID as a trace ID. W3C Trace Context.

These identifiers are not interchangeable with an MCP session identifier or a JSON-RPC request identifier. Keep protocol identifiers only where needed for protocol diagnosis, with appropriate access controls. The operation ledger should maintain explicit mappings, including an origin field stating whether each value came from your application, the workflow, or the remote service. A missing remote identifier should remain missing rather than being populated with a convenient local value.

Propagate where supported; correlate at boundaries elsewhere

For services you control, configure the HTTP client and server instrumentation to inject and extract trace context. Carry the durable operation reference in your own queue envelope or approved application metadata. If a workflow engine supports custom variables, retain the reference there. Do not add undocumented fields to a tool’s input or assume that a hosted endpoint accepts arbitrary correlation headers.

At an opaque boundary, create a client span around the actual outbound call. Record the locally measured duration, the normalized outcome, and any returned request or job reference. Mark the boundary as externally observed. If the service provides a separate audit view, use verified references to join the records. Two observations at either side of an opaque service are useful evidence; they are not proof that a continuous trace exists inside that service.

For asynchronous work, persist the correlation envelope before handing work to the queue. When a worker resumes, restore the operation and execution references and create new execution context. Use a parent relationship or span link according to the actual lifecycle and your tracing backend. I prefer linked execution segments for long approval waits because they make the distinction between active work and business waiting explicit.

Review incoming correlation data as untrusted input. A browser or external integration must not choose arbitrary tenant context merely by sending a header. Enforce limits on identifier length and format, and derive authorization from the authenticated request. OpenTelemetry also cautions that propagated baggage can expose sensitive information across service boundaries. Keep credentials, user content, and personal identifiers out of baggage. OpenTelemetry: context propagation.

Make correlation a tested acceptance criterion

Create a synthetic request in a test environment and follow it through every boundary you own. Search by operation ID and confirm that the accepted request, workflow step, outbound tool call, returned result, and final validation can all be found. Repeat the exercise with a retry and with a worker restart. The acceptance criterion is a reconstructable history, not merely a populated trace_id field.

Also test what happens when the remote service supplies no request ID. Your support packet should still contain the UTC time window, environment alias, operation name, and locally observed outcome. Keep that packet compact enough to produce automatically. If operators must guess which of several identical calls failed, the correlation design has not yet done its job.

2. Design structured logs that keep sensitive content out

A useful log should explain the event without reproducing the material the agent was working on. For a content workflow, record that an input satisfied the schema, which tool was selected, whether the response passed validation, and which protected artifact reference resulted. Avoid logging the complete brief, retrieved page body, generated copy, or uploaded file just because those objects are available in memory.

OWASP’s logging guidance identifies access tokens, passwords, encryption keys, sensitive personal information, and commercially sensitive data as information that generally should not appear directly in logs. It also recommends validation and sanitization of event data. Apply those principles at the producer before telemetry leaves the process. A downstream dashboard filter does not undo exposure in an exporter queue or storage index. OWASP Logging Cheat Sheet.

Build records from an allowlist

My proposed default is an allowlist serializer: construct a fresh event object from approved fields. Do not serialize an arbitrary request and then attempt to remove suspicious keys. An allowlist makes code review practical because reviewers can see exactly which data crosses the telemetry boundary. Treat each new field as a change to the telemetry contract with an owner, a purpose, and a retention decision.

Use a fixed event name and machine-readable classifications. Put service identity and release information in resource attributes or consistently named fields. Include the trace and span references when available. OpenTelemetry’s logging model supports correlation through execution context, so a log record can connect to a trace without copying its entire payload. OpenTelemetry: log correlation.

The following is a synthetic application event. Every value is illustrative. It represents an observed tool failure and a pending reconciliation decision; it does not reproduce a Sitecore response format.

{
  "event": "agent.tool.attempt.finished",
  "telemetry_schema": "1",
  "service": "campaign-orchestrator",
  "environment": "test",
  "operation_id": "op_demo_17",
  "execution_id": "run_demo_02",
  "step_id": "prepare_content",
  "attempt_id": "attempt_demo_03",
  "trace_id": "4bf92f3577b34da6a3ce929d0e0e4736",
  "span_id": "00f067aa0ba902b7",
  "tool_name": "update_content",
  "duration_ms": 8200,
  "outcome": "unknown",
  "error_layer": "transport",
  "error_class": "transient",
  "error_code": "response_timeout",
  "retry_decision": "reconcile_first",
  "side_effect_state": "unknown",
  "request_body_recorded": false
}

Keep error codes stable and bounded. An operator should be able to group response_timeout events without depending on the wording of a vendor exception. Preserve a sanitized diagnostic category and, where justified, a protected support reference. Never put the unfiltered exception message into a supposedly safe error_message field. Exceptions can contain URLs, request fragments, identifiers, and user-supplied text.

Design a separate path for diagnostic content

Sometimes an investigation needs the exact artifact or input version. Provide that through a restricted artifact store with explicit authorization and short-lived access, rather than through the general log index. Log an opaque reference and a reason code for the diagnostic capture. Record who accessed the object through a separate access audit. The telemetry viewer should not become an alternate route around content permissions.

For artifact URLs, store a reference that your application resolves after authorization. Avoid putting signed URLs into traces or alert messages. A signature or query parameter can grant access even when the rest of the URL looks harmless. Likewise, consider whether a page title or path exposes a planned campaign. Prefer a nonsemantic identifier over a recognizable business name in routine telemetry.

If correlation requires pseudonymous user or tenant information, document the transformation and the access boundary. A keyed pseudonym can support joins without placing raw identifiers in every log, but it should still be treated as controlled data. Do not assume that hashing a short, predictable value makes it anonymous. In this design, raw identity belongs in the authorized business system; routine operational records use scoped aliases.

Make sanitization recursive where diagnostic structures are explicitly permitted, and test size limits before serialization. Handle strings with line breaks or control characters so an attacker cannot forge an apparent second event. OWASP’s developer guidance explicitly calls for encoding and validating dangerous characters to prevent log injection. JSON serialization helps with representation, but it does not decide which content is appropriate to collect. OWASP: security logging and monitoring.

Test the actual telemetry pipeline

Create fixtures containing a fake token, a private campaign sentence, a signed-looking URL, and nested fields with misleading names. Exercise both successful and failing calls. Inspect what reaches the exporter and storage backend, including exception events and automatic HTTP instrumentation. A hand-written application logger can be safe while a middleware package captures the complete request beside it.

Set explicit retention and access policies for operational logs, audit records, and diagnostic artifacts. These stores serve different purposes and should not inherit one indefinite retention setting. Add an automated check that produces a harmless marker event and confirms its arrival. When logging is delayed or dropping events, show that condition on the operational dashboard instead of letting an empty error panel imply a healthy system.

Finally, define how the application behaves when telemetry is unavailable. For routine diagnostics, a bounded buffer and a visible dropped-event count may be appropriate. For an action that requires a durable approval or audit record, the application may need to pause before performing the action. Make that choice explicit per operation. An observability outage should not silently change the authorization or evidence requirements of a content mutation.

3. Measure outcomes, latency, retries, tool failures, and usage

Choose the unit of measurement before choosing a dashboard. A business operation can include several workflow runs, and each run can include multiple attempts at the same tool. If the system reports a successful retry as a new successful request while ignoring the original failure, its charts will describe call volume rather than user experience. Keep operation, run, and attempt measurements separate.

For the proposed operation ledger, use explicit states such as accepted, running, awaiting_review, succeeded, failed, canceled, and outcome_unknown. Document which transitions are allowed and which component owns them. Count a terminal business result once. A recovery execution may resolve an unknown operation, but it should not silently create a second completed business request in the denominator.

Define success at the right boundary

Track technical completion separately from accepted outcome. Technical completion means the expected steps completed without an unresolved execution error. Accepted outcome means the artifact or destination passed the checks promised by that workflow. For a draft-generation workflow, success can mean an approved draft is available. For a publication workflow, it may require verifying the published destination. The definition belongs to the workflow contract.

Use a time-bounded cohort for deadline success. For example, group operations accepted during a defined interval and evaluate whether each eligible operation met its delivery deadline. This is preferable to comparing completions in one minute with starts in the same minute when work takes much longer. Report operations still pending evaluation separately so a burst of new work does not appear to be a sudden reliability failure.

Keep user cancellations and expected review waits visible even if they are excluded from a particular SLO. State those exclusions beside the chart. A service can look reliable by excluding the hardest cases, so periodically inspect the excluded population. I would rather see a lower, well-defined completion rate than a high percentage that hides stalled work behind an unspecified pending status.

Separate active duration from waiting

Measure queue wait, active execution, dependency time, retry delay, and human review wait independently. Also keep the total elapsed time the user experienced. An eight-hour approval wait and an eight-hour API call should lead to different investigations. For overlapping steps, do not add all child durations and present the sum as wall-clock latency. Use the actual operation timestamps for that measurement.

Record duration distributions rather than relying only on averages. Google SRE’s monitoring guidance explains why percentiles reveal slow portions of a workload that a mean can hide. Choose histogram boundaries suitable for the expected call and workflow durations, then examine the slower tail by workflow type and tool. An interactive lookup and a long content-generation job should not share an unexplained universal latency target. Google SRE: monitoring.

For retries, report the fraction of logical tool invocations needing an additional attempt, the number of extra attempts, and the eventual recovery result. Track retries suppressed by policy as well as retries performed. A reduction in attempts can mean a fixed dependency, but it can also mean that a retry budget is being exhausted sooner. Pair the retry chart with failure class and unresolved outcome counts.

Control dimensions and attribute costs honestly

Use bounded dimensions such as environment, workflow type, controlled tool name, outcome, and error class. Keep operation IDs, artifact references, exception text, and page paths in logs or traces rather than metric labels. Review the possible combinations before deploying a new label. If tenant-level breakdowns are necessary, choose a controlled aggregation strategy instead of automatically creating a time series for every customer identifier.

For model consumption, record the usage information actually returned by a provider or available in authorized billing data. Separate input tokens, output tokens, image units, and any product-specific credits. Do not add unlike units into a single token count. If a managed Sitecore capability does not expose per-run model usage, label that portion unavailable and retain the operation reference for later reconciliation.

Cost estimates need their own provenance. Record the source of the price, the effective date, the model or service being priced, the currency, and whether discounts or cached-input rates are included. Mark estimated and reconciled amounts separately. This article intentionally provides no universal SitecoreAI per-run price because the observed usage and applicable commercial terms must come from the reader’s environment.

For an internal efficiency measure, divide the known total cost of an evaluated cohort by its accepted outcomes. Include the cost of failed attempts and recovery work within that cohort. Display the share of operations with complete usage data beside the result. If coverage is partial, call it a partial estimate; a precise decimal amount does not make incomplete attribution accurate.

Version the metric contract when meanings change. A renamed workflow, a new validator, or a revised definition of completion can move a chart without a real reliability change. Annotate those releases and retain enough release metadata to compare equivalent cohorts. This discipline makes an observability dashboard useful during an incident instead of turning the first ten minutes into an argument about what its numbers mean.

4. Connect execution traces to artifact lineage

A trace should explain the sequence of observable work. Start with spans for the orchestration you own: receiving a request, validating input, selecting a workflow, invoking a tool, validating a response, and recording a result. Add model-request spans only where your application actually observes those requests. Do not invent internal model or Sitecore server spans to make a trace tree look complete.

For each dependency call, record the operation name, attempt reference, normalized result, and duration. Where permitted, attach a remote request reference. A successful transport response does not imply the tool succeeded, so the wrapper should interpret the response before finishing its logical tool span. An HTTP span and a tool span can legitimately report different outcomes because they describe different boundaries.

Record the observable execution, not private reasoning

Capture the selected tool, validated arguments’ schema version, explicit workflow decisions, and externally visible results. Do not require hidden chain-of-thought or internal reasoning text as an observability signal. A concise decision category such as approval_required or input_schema_rejected is more suitable for operational grouping than a long explanation generated by the model.

When a model produces an explanation, treat it as another untrusted output with its own validation and privacy rules. It is not proof of which service calls occurred. The authoritative execution history should come from the orchestrator and dependency responses. Keep the model’s narrative separate from the record that establishes whether a mutation happened and who approved it.

Sitecore documents that running an agent generates an artifact and recommends reviewing generated artifacts and validating key information. Use that distinction in the operational design: artifact creation and artifact acceptance are separate events. A generated object can exist before it is approved for use. Sitecore: run an agent.

Create a minimal artifact manifest

For integrations you control, maintain an artifact manifest that connects a protected object to its origin. Include an opaque artifact reference, revision, producing execution, source references, validator result, and approval state. Record the identifiers supplied by the product when available; otherwise keep your own manifest explicitly outside the product’s namespace. Never manufacture a Sitecore artifact ID to fill a gap.

The manifest should answer whether the operator is reviewing the same revision that passed validation. If an editor changes the content after a successful check, create or record a new revision and invalidate any approval that no longer applies. In this proposed design, approval binds to an artifact revision and destination scope, rather than to a mutable filename or a chat message saying looks good.

{
  "artifact_ref": "artifact_demo_09",
  "revision": 3,
  "produced_by_execution": "run_demo_02",
  "source_refs": ["source_demo_04"],
  "validator_revision": "content-checks-7",
  "validation": "passed",
  "approval": "required",
  "destination_state": "not_applied",
  "content_in_telemetry": false
}

A content digest can help detect whether an object changed, but do not assume it proves that the content was correct or authorized. Where digest disclosure could reveal information about predictable content, keep it inside the protected manifest or use an approved keyed construction. For most routine dashboards, the artifact reference and revision are enough to navigate to the evidence.

Preserve state across pauses and partial work

Write a checkpoint before a potentially consequential step and update it after the result is reconciled. The checkpoint should say which prerequisites were satisfied, what was intended, and what is known about the destination. If the worker restarts, it can inspect that state instead of replaying every preceding action. The proposed ledger is a recovery aid, not a promise of atomicity across remote systems.

For a multi-artifact workflow, track the result of each required artifact and define whether partial delivery is acceptable. A localization workflow could finish several languages and fail another. Report that state explicitly with an incomplete deliverable set. Do not compress the entire outcome into succeeded because one language passed, or failed without preserving the usable work already produced.

When joining traces to the manifest, account for retention differences. A trace may expire before the business artifact does. Retain a compact execution summary with the artifact so later reviewers can identify the workflow revision and validation status even when detailed spans are gone. Conversely, removing a sensitive artifact should not leave its content embedded in archived telemetry.

Sample detail without losing the operation history

OpenTelemetry distinguishes head sampling, which decides early, from tail sampling, which can use later information about a trace. Tail sampling can support selection based on errors or latency, but it requires the relevant spans to reach the sampling system. It cannot recover spans discarded before they arrived. OpenTelemetry: sampling.

My suggested policy is to maintain the required durable operation record independently of trace sampling, then retain detailed traces according to an explicit budget. Favor unusual failures and slow runs where your pipeline supports that decision, and retain a representative sample of normal runs. Keep all collected attributes subject to the same privacy rules; selecting an error trace is not permission to capture its full prompt.

Test the investigator’s path from an alert to a trace and then to an authorized artifact view. Verify behavior when the trace was not sampled, the artifact has been deleted, or the execution crossed a managed boundary. Each case should show a truthful unavailable state and the remaining evidence. A tool that admits a visibility limit is more useful than a polished interface that silently substitutes a different revision.

5. Classify failures before choosing recovery

Failure class and recovery action should be separate fields. A transient network problem can leave a write’s result unknown. A contract failure can sometimes be corrected by changing an argument. An authentication failure can be resolved by an approved sign-in flow, but repeated identical calls will not provide the missing authority. Classify the evidence first, then decide what action is safe.

I recommend recording error_layer, error_class, retry_decision, and side_effect_state. The layer describes where the failure was observed: transport, protocol, tool, workflow, or outcome validation. The class describes the best current explanation. The remaining fields state the operational decision and what is known about changes. Allow an unknown class when evidence is insufficient; uncertainty should be visible rather than disguised as model_error.

Inspect every response layer

The MCP tools specification separates JSON-RPC protocol errors from tool execution errors. A tool can return a result with isError set to true even though the protocol exchange completed. Structured output can also have a declared output schema. Pin and test against the protocol revision negotiated by your client and server instead of assuming all versions have identical contracts. MCP tools specification, revision 2025-11-25.

Sitecore’s Invoke Tool documentation shows another relevant distinction: workflow tool responses can expose success and error fields, and downstream steps must reference the exact value they need from upstream output. These examples are not a universal envelope for every tool. Implement an adapter for each supported tool contract and normalize it into your internal outcome model. Sitecore: using Invoke Tool in a workflow.

Validate the request structure before dispatch, interpret the protocol response, inspect the tool’s error indicator, validate the expected output, and then check business postconditions. Keep the raw payload outside routine telemetry. This sequence helps prevent a successful HTTP exchange from masking a failed tool, and helps prevent a syntactically correct tool response from masking the wrong content change.

Transient failures: retry eligibility is not retry safety

For a known read operation, a temporary dependency failure may justify another attempt within the operation’s deadline. For a write, first establish whether the request could have taken effect. A connection error before dispatch and a timeout after the request was transmitted carry different uncertainty. If your transport cannot tell the difference, record the more cautious state and reconcile through an authoritative read or supported job lookup.

Bound retries by attempts, elapsed time, and available work budget. Choose one owner for the retry policy and inspect retries hidden in SDKs or lower layers. AWS guidance recommends limiting retries and using backoff with jitter, while emphasizing the need to verify idempotency. Treat those as design principles; they do not establish that a particular Sitecore tool supports an idempotency key. AWS: retry with backoff.

Keep a logical action reference constant across a genuine retry, but do not label a changed request as the same attempt. If an API documents idempotency support, use its documented mechanism and retention rules. If it does not, an application ledger can coordinate local workers and reconcile known outcomes, but it cannot create a server-side exactly-once guarantee. Escalate an ambiguous create instead of hoping that a second create will repair it.

Identity failures: repair the authority, not the prompt

Distinguish authentication from authorization in your internal taxonomy even when the user-facing error is simply access denied. Check the intended organization and tenant, the connector’s supported session state, and the permissions of the authenticated identity. Sitecore’s Marketer MCP troubleshooting guidance directs users to verify organization, tenant, session expiry, and role access. Do not broaden privileges merely to make a failing agent run green. Sitecore: Marketer MCP troubleshooting and tool reference.

Allow a controlled refresh only through the connector or identity mechanism that owns it. If interactive authentication is required, transition the operation to a visible waiting state and tell its owner how to resume. Do not ask a model to reconstruct tokens, place credentials in a prompt, or keep retrying with the same rejected identity. After access is repaired, verify a minimal authorized operation before resuming the affected work.

Contract failures: find the producing boundary

A missing page identifier can have several causes. The model may have omitted a required argument, the workflow may have mapped the full previous response instead of its ID field, or the remote tool may have changed its output shape. Preserve the producing and consuming schema revisions in protected diagnostic evidence. The error class is contract-related; the owner depends on where the invalid structure originated.

Use deterministic validation wherever possible. Test each adapter with valid results, tool errors, missing fields, and additional fields. Avoid immediately labeling an unfamiliar but valid response as a failure just because it contains new metadata. Conversely, do not silently coerce an object into a string when the next tool expects an identifier. A contract repair should make the intended mapping explicit and include a representative fixture.

Model failures: evaluate the result independently

Reserve model or quality classifications for evidence about generated behavior: an unsuitable tool choice, a fabricated reference, an output that fails the declared structure, or content rejected by the workflow’s acceptance criteria. A model provider timeout belongs to the dependency layer until there is evidence of a generation-quality problem. This distinction keeps a model-quality dashboard from becoming a collection of unrelated infrastructure faults.

For correctable output, permit a small, explicit repair budget using sanitized validation feedback. Stop when the same condition repeats or the remaining operation budget is insufficient. Route uncertain factual content or consequential changes to review under the workflow’s policy. A refusal or policy block can be an expected controlled outcome, so track it distinctly rather than encouraging the system to bypass it in pursuit of a higher completion rate.

6. Turn alerts into recovery runbooks

An alert should identify a condition that needs action and provide the evidence needed to start. For agent workflows, useful signals include sustained failure of accepted operations, an expanding queue of unknown write outcomes, repeated identity failures across a connector, and a sharp increase in a particular tool’s contract errors after a release. Each signal should have a named owner and a bounded first response.

Use service-level objectives for user-visible reliability and separate diagnostic alerts for causes. Google SRE describes burn-rate alerting with multiple windows so teams can detect both rapid and sustained consumption of an error budget. Low-volume services need additional care because a small number of requests can produce large ratios. Adapt that method to the workflow’s traffic and delivery promises instead of copying thresholds without a denominator. Google SRE: alerting on SLOs.

Choose severity by impact and uncertainty

As a proposed starting policy, treat confirmed unauthorized changes or uncontrolled consequential writes as urgent incidents. Treat widespread inability to complete a business-critical workflow as a high-priority availability incident. Route an isolated schema regression in a noncritical draft workflow to the owning team with the affected executions attached. The exact severity names should match your incident process; a model’s wording should never set severity on its own.

Distinguish an alert from a diagnosis. A rise in unknown outcomes says that reconciliation is needed; it does not prove a database outage. Group related symptoms under the same incident where possible and suppress repetitive child notifications while preserving the underlying events. Include the first-seen time, affected workflow, release, failure class distribution, and a link to representative safe telemetry. Avoid attaching generated content to pager messages.

Watch the age of the oldest unresolved operation as well as backlog size. A stable queue can still contain one campaign that has been forgotten. For approval waits, route a reminder to the business owner according to the agreed workflow instead of paging an infrastructure team. For active executions with no progress evidence, investigate the execution mechanism. These conditions can have similar elapsed time and entirely different owners.

Runbook: response timeout after a write

1. Pause additional attempts for the affected logical action and preserve its operation reference.
2. Inspect the last durable checkpoint, outbound attempt record, and any returned request or job reference.
3. Read the authoritative destination through a supported interface and determine whether the intended change exists.
4. If completion is confirmed, record the evidence and continue from the next safe checkpoint.
5. If noncompletion is established and the action remains authorized, retry within the configured budget.
6. If the result cannot be determined, retain outcome_unknown and route it to the operation owner for reconciliation.

Do not infer completion merely because an item with a similar title exists. Match the expected target, revision, language, and relevant state using the operation’s protected evidence. If another actor could have changed the destination, compare against the stored precondition and assess concurrency. A rollback that overwrites a later valid edit can turn a small incident into a content-loss incident.

Where a supported product operation offers a documented job-based recovery mechanism, capture the job identifier and verify its scope before using it. The Agent API reference discusses job tracking and reverting unintended actions. That is not a universal rollback promise for every workflow, external MCP tool, or downstream side effect. Check the specific operation and environment before building it into a runbook. Sitecore Agent API reference.

Runbook: authentication or permission failure

First identify whether the incident affects one user, one connector, or many operations. Stop repeated identical calls for that scope. Check the configured target and the supported authentication flow without copying tokens into incident notes. Have the identity owner repair an expired session or the access owner review a missing permission. Test the smallest appropriate operation with the repaired identity, then resume eligible work gradually.

Record how long work waited for identity repair separately from active retry time. If the business deadline has passed or approval no longer applies, return the operation to its owner instead of automatically executing stale work. A restored session answers whether the system can authenticate now; it does not answer whether every action queued under the old session should still happen.

Runbook: contract regression after deployment

Compare the first failures with the release timeline and identify the producing adapter or workflow mapping. Capture a sanitized fixture that reproduces the validation failure. If a known compatible release exists, consider rolling back the integration configuration or adapter after checking compatibility with in-flight work. Correct the schema mapping in a test environment and run the affected workflow with a harmless fixture.

Do not resume the entire queue immediately after one successful test. Select a bounded set of eligible executions, verify their outcomes, and inspect whether the same error signature recurs. Preserve the old failure records and link the recovery execution. This gives the team evidence that the regression is fixed without rewriting history to make the failed deployment look successful.

Runbook: unacceptable model output

Quarantine the affected artifact revision from downstream use while retaining its protected reference. Determine whether the problem originated in source material, retrieval, instructions, model behavior, or the validator itself. Reproduce it with a sanitized test case where possible. Apply a specific correction, such as a clearer output schema or a repaired source mapping, and evaluate the new artifact independently.

A model change is one possible intervention, but it should not be the default response to every poor result. If the workflow supplied the wrong page or omitted a required brand source, switching models may leave the root cause intact. Keep the output review criteria stable during the comparison, and document any deliberate change in those criteria. Approval should bind to the revised artifact rather than carrying over from an earlier rejected version.

Exercise recovery before depending on it

Run a focused failure drill in a test environment: interrupt a read, simulate an expired session, supply a malformed tool result, and lose the response after a controlled write. Confirm that each event reaches the expected failure class and runbook. Inspect whether the operator can recover without reading a secret or repeating a write whose state remains uncertain. These are test scenarios, not claims that Sitecore exposes a built-in fault-injection facility.

Include a worker restart during recovery. Verify that the ledger still identifies the original operation and that two workers cannot independently claim the same recovery action without coordination. Where you use a lease or lock, document its expiry and reconciliation behavior. A local lock prevents competing local actions while held; it does not prove that a remote request was never executed.

Define exit criteria before declaring the incident resolved. New operations should meet the intended outcome checks, the affected backlog should be reconciled or explicitly assigned, and telemetry should be arriving normally. Record any remaining unknown results with an owner. A green health endpoint or one successful tool call is insufficient evidence that all interrupted content work has been recovered.

Put the first operational slice into service

Start with one workflow that matters to its users. Write down its accepted outcome and deadline, its consequential actions, and who can resolve an uncertain result. Add the durable operation reference and boundary instrumentation. Implement the minimal artifact manifest, then connect a failure alert to a runbook that names the exact evidence and verification step. Expand only after the team can demonstrate that complete path.

The strongest readiness test is practical: give an operator an operation ID for a failed synthetic run. They should be able to locate the responsible step, classify the failure, identify any affected artifact, and establish whether replay is safe. If they can finish that exercise with protected evidence and a verified destination state, the system has an operational foundation worth extending.