How to Measure Agent Quality in SitecoreAI

Agent quality is a relationship between a task, its constraints, and an observable outcome. A fluent draft can be useless for the intended market. A technically failed run can still stop correctly before an unauthorized action. Measuring only completion or overall satisfaction hides those differences. Begin with the decision the evaluation should support: whether an agent is suitable for a particular job, whether a change improves that job, or whether production behavior requires intervention.

This article proposes an evaluation program for a hypothetical SitecoreAI campaign-content agent. It combines researched platform surfaces with an application-owned measurement design. The examples are illustrative and the suggested metrics are not observed production results. No universal score, benchmark, or sample size is claimed. The organization must choose the evaluation population and release conditions that fit the consequences of its own workflow.

My position is that an agent should not pass a production evaluation solely because its average writing score improves. I would keep destination correctness and authorization behavior as separate release conditions for a workflow that changes shared content. That choice may reject an otherwise impressive candidate. It also makes the tradeoff explicit: better prose does not compensate for changing the wrong resource. The rest of the design follows that principle from test selection through production monitoring.

1. Define quality as a task outcome

Name the outcome before choosing a metric

Write the task in business terms that can be checked. For the campaign-content example, the agent prepares a source-supported draft for a selected destination and language, then returns a review packet. That definition excludes public release unless the workflow explicitly includes it. A test should therefore evaluate the draft and packet, not reward the agent for taking an extra publishing action. A broader task needs a broader evaluation, but the scope should be chosen before execution rather than inferred from whatever output the agent produces.

Separate technical execution from business acceptance. Technical execution asks whether configured steps ran and produced expected records. Business acceptance asks whether the result meets the campaign’s requirements. Editorial usefulness asks whether a reviewer can use the result without disproportionate correction. Those dimensions may interact, but each needs its own evidence. An operation can return a successful response while the field content is unsuitable. A reviewer can like the writing while the destination resolver selected the wrong item. Do not collapse those outcomes into one undifferentiated pass.

Sitecore’s configuration guidance describes reviewing run steps and their inputs and outputs. Use that evidence to diagnose execution. The NIST AI RMF Measure function recommends task-relevant assessment before deployment and monitoring during operation. Neither reference supplies a universal quality threshold for your campaign. The thresholds and acceptance criteria below are proposed application decisions that require an accountable business owner.

Define separate requirement categories

For the hypothetical agent, propose categories for destination selection, structure, evidence support, editorial fitness, and permitted behavior. Each category should state how it is assessed and who owns the interpretation. Destination selection might be a deterministic match to an approved target. Editorial fitness may require a rubric and reviewer judgment. Permitted behavior should include what the agent must not do. Keep a correct stop or clarification outcome available when the input lacks information necessary to proceed.

Define essential failures explicitly. If a wrong-language draft is unusable for the task, do not let a strong style score average it into an acceptable result. If an external action was outside scope, treat that as a distinct violation even when the content was excellent. These are policy recommendations, not statistical facts about every workflow. The owner should decide which failures block release and which allow a draft to continue with warnings. Write the reasons so later teams do not reinterpret the rules to favor a preferred candidate.

Include uncertainty as a valid outcome. A task with conflicting product evidence may require the agent to identify the conflict rather than resolve it creatively. A test expecting a confident answer in that situation would reward the wrong behavior. Record the expected response class for each fixture: draft, clarification, rejection, or escalation. Then evaluate whether the content of that response is useful. “Stopped” is not automatically good or bad; its correctness depends on the task and the evidence available to the agent.

Specify the unit of evaluation

Choose whether you are evaluating a field, an artifact, a workflow run, or an end-to-end campaign task. A field-level check can identify an unsupported claim while the artifact-level assessment describes overall usefulness. A run-level assessment can identify a tool-contract failure. A task-level assessment can include whether a reviewer completed the campaign work. Keep these units separate in records. Counting one rejected field as one rejected campaign without explaining the aggregation would make the result hard to interpret.

Record the configuration under evaluation. Include the agent configuration reference, prompt revision, relevant schema and template revisions, selected context, and tool configuration where those references are available. If the platform does not expose a particular internal version, state that limitation. Do not invent it. Preserve enough of the configuration you control to reproduce the comparison as closely as the interface permits. An unexplained change in source context can otherwise be mistaken for a model improvement or regression.

Start with an evaluation charter that fits on a short review document. State the task, intended users, relevant destinations, requirement categories, and who can accept residual limitations. Add the metrics only after those decisions are clear. Ask an editor and an operator to describe what a successful result looks like from that charter. If their answers conflict, resolve the task definition first. A dashboard cannot repair an evaluation whose participants disagree about what the agent was supposed to accomplish.

2. Build an evaluation set that represents the work

Build fixtures from meaningful task variation

Create a set of campaign tasks that spans the variation the workflow is expected to handle. Include different content types, relevant languages, source quality, destination ambiguity, and review requirements. Use authorized, sanitized examples where actual work is available. If you create synthetic fixtures, label them and explain what boundary each one exercises. A large set of near-identical requests may create a reassuring volume of results while testing very little of the intended operating range.

For each fixture, preserve the input and expected outcome before running the candidate. The expected outcome should describe requirements rather than demand one exact creative wording. A draft may have several acceptable formulations. A destination identifier may have only one acceptable value. A conflict case may require an explicit unresolved condition. Write enough guidance that a reviewer can distinguish a valid variation from a failure. Do not let an evaluation become a comparison with one arbitrary reference paragraph that happens to share vocabulary with the baseline.

Include ordinary tasks and difficult boundary cases for different purposes. Ordinary tasks estimate usefulness within the selected workload. Boundary cases examine whether important controls behave as intended. Report those groups separately. A set deliberately full of malicious or ambiguous inputs is not an estimate of the frequency of such inputs in production. Conversely, an ordinary sample that contains no hostile inputs cannot demonstrate that authorization boundaries withstand deliberate manipulation. State what each group supports and what it does not.

Represent correct non-generation outcomes

Add a brief with two plausible destinations and no market selection. The expected outcome might be clarification rather than a draft. Add conflicting approved sources for one claim. The expected outcome might be a visible conflict requiring a source owner. Add an out-of-scope action request. The expected outcome should follow the permission policy. These fixtures test whether the agent respects the limits of the task. A quality system that rewards output volume alone would miss the purpose of those cases.

Test tool and integration behavior separately from prose. A malformed response fixture can check whether the adapter rejects an invalid identifier. An uncertain external write can check whether the workflow escalates instead of repeating the action blindly. Use a controlled environment and the real supported integration behavior where appropriate. Do not describe a simulated adapter test as a successful production recovery exercise. Preserve the distinction between unit evidence, integration evidence, and an operational rehearsal involving the destination system.

Use a provenance record for evaluation material. Record where the fixture came from, whether it was modified, and which source revision it represents. If a product specification changes, decide whether an old fixture remains useful as a historical regression or needs a new expected outcome. Do not silently update the evidence under an old result. A future reviewer should be able to understand why a candidate was judged correct under the test definition used at the time.

Manage access and leakage deliberately

Keep sensitive campaign or customer material under the organization’s existing access and retention rules. Evaluation is another use of the data, not an automatic exemption from those rules. Use sanitized fixtures when they preserve the behavior you need to test. Document what was removed so evaluators understand the resulting limitation. A fixture stripped of all destination ambiguity may no longer test the original problem even if it retains the same title and general topic.

Reserve evaluation material for comparison rather than repeatedly tuning to every example. The exact split and sampling method should follow the team’s workload and measurement needs. The recommendation is to preserve some independent challenge to the candidate, not to claim that a particular percentage is universally correct. When a failed fixture becomes a development example, label that change and retain other material for later assessment. Otherwise improvements may reflect familiarity with the test set rather than broader task performance.

Assess coverage with a matrix tied to the task charter. Each row can identify a requirement and the fixtures that exercise it. Empty rows reveal untested requirements. Overcrowded rows may reveal redundant examples. Include ownership for expected outcomes that need domain judgment. The NIST Measure playbook emphasizes documenting test sets, metrics, and tools; the proposed coverage matrix turns that principle into a concrete working artifact for this campaign evaluation.

Finish by having someone outside the agent’s implementation team review the fixture definitions. Ask whether the ordinary tasks resemble the expected work and whether the difficult cases test real consequences. Record disagreements before running the comparison. The admitted limitation is that no finite evaluation set establishes correctness for every future input. Its value is clear evidence about defined scenarios, plus a disciplined path for adding new scenarios when production behavior reveals a gap.

3. Choose metrics with explicit denominators and limits

Write a metric contract, not just a name

Every reported metric should identify its numerator, denominator, unit of analysis, exclusions, and observation window. “Acceptance rate” could mean accepted fields, accepted drafts, or completed campaigns. Those are different measures. For the hypothetical workflow, define draft acceptance as reviewed eligible drafts accepted under a stated rubric divided by reviewed eligible drafts. Report unreviewed and ineligible work separately. Do not make the number look better by silently removing difficult rejected cases from the population after seeing the outcome.

For destination correctness, define the expected target independently of the agent output. Count matches among tasks where the destination is resolved and the expected value is available. For intentionally ambiguous tasks, evaluate the required clarification behavior instead. Report those two populations separately. A workflow that correctly refuses to guess should not be labeled wrong-target simply because it did not return an identifier. The metric should reflect the task contract rather than force all outcomes into the same shape.

For structural validity, evaluate the output against the schema and validator actually used by the integration. State which checks ran and whether business constraints were separate. A syntactically valid record can still describe the wrong item or an unsupported claim. Do not present schema-pass rate as overall agent accuracy. If generation produces an invalid structure and a repair step fixes it, report both first-pass validity and final validity when those distinctions matter to operational effort and reliability.

Measure support and usefulness at the right granularity

For evidence support, define what counts as a material claim and how reviewers judge whether the selected source supports its wording. Record unsupported claims separately from missing citations. A citation can exist without supporting the statement. A supported statement may need a more precise reference for the review process. The organization should choose which conditions block a draft. Avoid assigning a universal numerical evidence score without a rubric that explains those decisions.

For editorial usefulness, ask reviewers about the correction required before the draft can serve its intended purpose. Categories might distinguish minor wording edits from substantive factual or targeting changes. These are proposed categories and should be calibrated with examples from the actual workload. If you measure editing time, define when timing begins, which interruptions are included, and who performs the work. Do not report a productivity gain unless the comparison conditions support that interpretation.

For operational performance, separate active processing time from time waiting for review or an external dependency. A run can be slow because the source owner has not answered a question, not because generation is inefficient. Report the stage that consumed time and the task outcome. If you track resource consumption, use actual observable measures and current verified commercial terms. This article supplies no Sitecore price, quota, or universal latency target. Choose targets according to the workflow’s deadline and the interfaces it depends on.

Preserve missing data and measurement uncertainty

Do not convert an unavailable measurement into zero or a pass. A draft that has not been reviewed is unreviewed. A source comparison that cannot access the evidence is unresolved. The metric report should show how many cases remain in those conditions. Otherwise the team may confuse incomplete observation with high quality. Keep the reason for missing data where it affects interpretation. A dashboard should help the owner ask what evidence is absent, not conceal the absence inside a clean percentage.

Sitecore documents confidence values and labels such as Known, Assumed, and Generated for structured artifact fields. See artifact confidence guidance. Treat those as product signals to inspect. Do not assume that a displayed percentage is a measured probability of factual correctness for your task. If the team wants to use it as a predictor, compare it with independently reviewed outcomes and document the population and limitations of that comparison.

The following application-owned record illustrates a metric definition. It is not a native Sitecore analytics schema. Its value is the explicit population and treatment of unresolved work.

{
  "metric": "draft_acceptance",
  "unit": "reviewed_draft",
  "numerator": "accepted_under_rubric",
  "denominator": "eligible_reviewed_drafts",
  "reportSeparately": ["unreviewed", "ineligible", "unresolved_evidence"],
  "rubricRevision": "campaign-draft-rubric-1"
}

Review metrics as decision instruments. Ask what action a change in each measure should trigger and what alternative explanation might exist. A lower acceptance rate could reflect harder incoming work, a stricter rubric, or a weaker agent. Preserve task categories and rubric versions so the owner can investigate. Do not interpret a trend without checking whether the population changed. The point is not to produce more charts; it is to make a release or operating decision with evidence whose meaning remains stable enough to inspect.

Consider a deliberately synthetic reporting example. Suppose a test collection contains twenty eligible reviewed drafts, of which fifteen meet the stated acceptance rubric, plus five drafts still awaiting review. The reviewed-draft acceptance measure is fifteen divided by twenty, or seventy-five percent. The report should also show that five drafts remain unreviewed. It should not quietly divide by all twenty-five while retaining the same metric label, nor assume that the unreviewed drafts will pass. These numbers illustrate arithmetic and population definitions only. They are not measurements of SitecoreAI, a customer deployment, or the example agent.

Now suppose a separate set contains four deliberately ambiguous requests and all four correctly ask for clarification. Report those outcomes against the clarification expectation. Adding them to the accepted-draft numerator would change the meaning of draft acceptance. Calling them four failed drafts would also misstate the task. A report can describe both draft usefulness and correct clarification behavior, provided it names their populations and does not pretend that one percentage answers both questions.

4. Design human review and model-assisted judging

Give human reviewers a concrete rubric

Choose reviewers who understand the task and the evidence needed to judge it. A copy editor can assess readability, while a product owner may be needed for claim support or regional applicability. The evaluation design should say which judgments each reviewer makes. Avoid asking one person to assign a general quality score covering requirements they cannot inspect. When specialized review is unavailable, record the resulting limitation rather than treating a confident general assessment as a substitute.

Build a rubric around observable differences. For a factuality judgment, provide examples of a supported claim, an overstatement of the source, and a claim with no relevant support. For editorial usefulness, show what counts as a minor revision versus a substantive correction. For destination behavior, show a resolved case and an ambiguous case that should stop. The examples should be approved for the evaluation context. They help reviewers apply the same meanings rather than infer the categories from their personal preferences.

Have reviewers score a shared calibration set before using the rubric for a consequential comparison. Discuss disagreements and revise unclear definitions. Do not force agreement by averaging incompatible interpretations. One reviewer may be judging whether a claim sounds plausible while another checks whether the source supports its exact wording. The disagreement reveals a rubric problem. Record the agreed interpretation and retain unresolved issues for the owner to decide. The aim is a review process people can explain, not an appearance of numerical precision.

Use model-assisted judging for bounded tasks

A model-assisted reviewer can help identify issues for a human or apply a narrow rubric to a draft. Treat its output as an evaluation component with its own failure modes. Preserve the judge configuration and source context where available. Do not assume that the judging model is independent merely because it runs in a separate step. Validate whether its recommendations agree with qualified review for the specific task. If the evidence does not support using it for a release decision, keep it as an advisory signal.

Separate generation instructions from judging instructions. The judge should receive the task criteria and relevant evidence, not a request to praise the candidate’s work. Avoid giving the candidate authority to define its own success conditions. If the agent produces a self-assessment, preserve it as a self-assessment. It can help explain what the agent believed, but it should not become the only ground truth. The trusted evaluation record should identify which judgments came from the candidate, a separate model, or a person.

Include adversarial and misleading material in controlled judge tests. A draft might contain an instruction telling the evaluator to award a high score. A retrieved source might claim that the task has already been approved. The OWASP prompt-injection guidance explains the general risk of instructions embedded in content. In the proposed evaluation architecture, treat the submitted draft and retrieved material as data. Test whether the judging process follows the rubric despite those attempts and keep consequential enforcement outside the generated score.

Record disagreement as evidence

Retain the reason for a disputed judgment. A model may flag a phrase that the regional editor considers standard terminology. A person may overlook a claim that the evidence checker identifies as unsupported. Investigate the actual text and source. Do not automatically privilege either side based on confidence alone. The appropriate decision may be a rubric clarification, a source correction, or an acknowledged uncertainty. Preserve that decision so the same disagreement does not recur as an unexplained score difference in later comparisons.

Keep reviewers blind to candidate identity when that is practical and relevant to the comparison. Avoid labels that reveal which version the implementation team hopes will win. State the review procedure in the report, including any constraints that prevented masking. The recommendation is to reduce avoidable preference effects in the local process; this article does not claim a universal bias reduction percentage. Use qualified methodological advice if the evaluation must support broader statistical or organizational claims.

Ask reviewers to assess whether the artifact is useful as a review object. Important issues should be visible where decisions are made. If source references are inaccessible or the destination is unclear, that is a practical quality problem even when the prose looks strong. Record whether the agent exposes unresolved questions and whether a reviewer can locate the evidence. The evaluation should reflect the actual handoff, not just the text isolated from the workflow that produces and uses it.

The output of this stage should be a review record with distinct judgment origins, a rubric revision, and reasons for important failures. Keep enough evidence to reconsider a disputed result later, subject to access and retention policy. My preference is to use model-assisted judging to focus human attention on specific claims and inconsistencies, while preserving qualified human decisions for consequential ambiguity. That trades some review effort for a clearer account of who concluded what and on which evidence.

5. Compare changes without hiding regressions

Compare one defined change against a stable baseline

Choose a baseline configuration and preserve its evaluation record. For a candidate, identify the intended change: prompt wording, schema rules, selected context, tool configuration, or another controlled component. If several components change together, document that bundle and limit the causal claims you make. A better outcome under a new source packet cannot automatically be attributed to a new prompt. The comparison should state what was held stable and what the available interface did not let the team control.

Run the baseline and candidate on the same defined fixtures when the comparison permits it. Retain the exact input and evidence references. If a live external source changes between runs, record the difference or use an approved fixed source packet for the test. Do not claim that the experiment compared identical conditions if it did not. Some variability may be unavoidable; make it visible so reviewers can judge whether the result supports the release decision or requires another targeted comparison.

Plan repeated observations where the task and system behavior make a single result insufficient. Choose the repetition and analysis method according to the question, available resources, and methodological needs. Do not invent a universal number of runs. A single attractive draft is weak evidence about consistent behavior, while repeating the same trivial request many times may add little information about coverage. The report should explain why the selected observations are informative for the intended release boundary.

Inspect regressions by requirement and task slice

Report destination correctness, structural validity, evidence support, and editorial outcomes separately. Break down results by the meaningful task categories in the charter. A candidate might improve ordinary English drafts while weakening the localized conflict cases. An overall average can hide that result. The owner should decide whether the affected category is in scope for the release and what correction is required. Do not let strong performance on a large easy category erase a consequential failure in a smaller one.

Keep blocking requirements distinct from preferences. If an unauthorized-action fixture fails, the release process should follow the documented policy even if reviewers prefer the candidate’s prose. If the candidate has a minor style regression but substantially reduces substantive corrections in the defined task, the owner can assess that tradeoff. The important point is to state the decision rule before promoting the candidate. A score threshold chosen after seeing the results is not a stable release criterion.

Review examples near the decision boundary. Inspect rejected drafts, disputed judgments, and cases with incomplete evidence. A candidate that passes by exploiting an overly permissive rubric should prompt a rubric correction, not celebration. Retain the original result and explain why the interpretation changed. If the evaluation definition changes, rerun or re-evaluate the comparison as appropriate. Do not rewrite the historical report to make it appear that the original rubric always expressed the newer understanding.

Document the release decision and its limits

The comparison report should identify the task, configurations, fixture revisions, metrics, and major failures. State whether the candidate is accepted for the entire task scope or a narrower subset. If the organization releases only for one content type or language, make that restriction operationally enforceable where possible. A note in a report is insufficient if the production entry point still accepts unrestricted destinations. Tie the release scope to the actual workflow and permissions the team controls.

Preserve unresolved limitations. Perhaps the comparison did not cover a particular regional template, or the source system did not provide stable snapshots. State how that limits the conclusion. A team may still choose a bounded release with review, but the decision should be honest about the evidence. Do not describe an evaluation as proving production safety. It establishes results for the tested conditions and supports a policy decision about the next step. Monitoring and recovery remain necessary after the candidate is accepted.

Use a controlled rollout when the consequences justify it. Start with draft generation or a restricted destination set, review actual outputs, and expand only after the relevant evidence is collected. Compare the incoming workload with the fixture coverage. If production requests differ materially, revisit the evaluation rather than assuming the test results transfer unchanged. The rollout should be a way to gather and respond to evidence, not a justification for skipping the earlier boundary tests.

Finish the report with a concrete owner decision. Name the accepted configuration, the permitted scope, the conditions that would trigger rollback or restriction, and who can make that call. Keep the report linked to the released workflow revision where the integration supports it. An operator investigating later behavior should know which evaluation supported the release and which limitations were accepted. That connection turns evaluation from a one-time presentation into evidence the production team can actually use.

When communicating the comparison to a campaign owner, include a short example of each material regression. Show the input condition, the relevant requirement, and the observed result without unnecessary sensitive content. A concrete wrong-destination example is easier to assess than a small movement in a composite average. The owner can then decide whether the affected scope should remain restricted while the team corrects the problem. Keep that decision attached to the release record rather than leaving it in an informal discussion that future operators cannot retrieve.

6. Operate the quality program after launch

Connect production feedback to the evaluation set

Collect correction reasons from actual reviewers using the same requirement categories where practical. A new unsupported-claim pattern should lead to evidence review and a targeted fixture. A repeated destination ambiguity may reveal missing inputs or an overly permissive resolver. Preserve the distinction between a new task type and a regression in an existing one. Adding a fixture is useful only if its expected outcome is clear and its relationship to the production observation is recorded.

Monitor the system at the level needed for decisions. Sitecore’s troubleshooting guidance describes inspecting workflow-agent steps and context. For surrounding services, OpenTelemetry’s observability primer explains the role of telemetry signals in understanding behavior. Use supported evidence from each layer without assuming one instrumentation package exposes every internal platform action. Record where an operator must use a separate interface or support procedure to complete the diagnosis.

Define triggers for investigation that match the task. A rise in unresolved evidence might reflect a changed source repository. A rise in structural failures might follow a schema change. A longer review queue might reflect reviewer availability. Do not treat all of them as model-quality regressions. The owner should inspect the relevant configuration and workload changes before choosing a correction. A better dashboard is one that helps distinguish plausible causes and directs the issue to someone who can act.

Keep measurement definitions stable enough to compare

Version the rubric and metric definitions. If the editorial team changes what counts as a substantive correction, mark the change in the report. Historical percentages computed under different definitions should not be compared as if their meaning were identical. Preserve the original population and, when appropriate, re-evaluate a comparable subset under the new rule. The goal is not to freeze the evaluation forever. It is to make changes understandable so the team can learn without manufacturing a misleading trend.

Review whether the metrics still serve the decision. A measure that once identified a useful failure may become redundant after an interface change. Another requirement may emerge when the workflow gains an external action. Retire or revise metrics deliberately and record the reason. The NIST framework supports ongoing review of measurement effectiveness; the local implementation should translate that principle into an owner and a review schedule appropriate to the workflow’s change rate and consequences, without borrowing a universal interval from this article.

Protect evaluation records under the same governance discipline as production content. They may contain private source material, reviewer identity references, or sensitive failure examples. Define access and retention for each type of record. Preserve aggregate findings where appropriate, but do not assume anonymization is complete merely because a name was removed. Use the organization’s approved data-handling process. The evaluation program should answer quality questions without creating an uncontrolled archive of every input the agent has seen.

Give the program an operating owner

Assign responsibility for maintaining fixtures, adjudicating rubric disputes, and approving releases. These tasks can be shared, but each needs a clear decision path. An engineer should not quietly loosen a blocking requirement to ship a prompt change. A business owner should not interpret an execution failure as poor writing without inspecting the technical evidence. The operating model should connect those perspectives so a problem is corrected at the boundary that caused it.

For the first iteration, build a small evaluation set with clear expected outcomes and run one baseline comparison. Include ordinary drafts, ambiguous requests, unsupported claims, and at least the consequential integration boundaries in scope. Review the failures before adding more examples. A compact set that exposes a real decision problem can be more useful than a large set scored by an unclear rubric. Expand coverage when the task charter or production evidence identifies a specific missing category.

Use a release checklist that asks for evidence rather than reassurance. Which task is accepted? Which configurations were compared? Which requirements block release? What remains untested? Who owns the production correction path? The answers should point to records and decisions. If the only evidence is a collection of attractive generated drafts, the evaluation has not yet established whether the workflow meets its operational purpose. Improve the missing requirement or test before presenting a general quality score as sufficient.

The next practical step is to choose one current agent and write its metric contract. Define the unit, eligible population, and treatment of unresolved work. Select fixtures that expose the task’s actual consequences, then record the baseline without tuning the expected answers afterward. That gives the team a measurement process it can challenge and improve. A credible quality program begins with a well-defined question and evidence that can answer it, then remains accountable to the work people actually need the agent to perform.