Measure candidate outputs with pinned references and retain one comparison
POST
/v1/quality/checks/benchmark/{evaluationId}/experiments/{experimentId}/compareAn atomic, idempotent analysis; no target execution. Corrected references conflict instead of silently changing the protocol. Changed per-case eligibility, unknown rates or too few paired examples produce an inconclusive result. Returns descriptive percentage-point differences, not statistical or release approval. Existing comparisons remain readable after subsequent reference corrections.
How to call this endpoint
Every ACP API request uses bearer authentication. The examples here show the actual request path, auth header, and body shape that the platform expects.
Path, query, and header parameters
These parameters control which ACP object the endpoint acts on and how the request is processed.
Path parameters
| Name | Location | Type | Required | Description |
|---|---|---|---|---|
| evaluationId | path | string | Yes | — |
| experimentId | path | string | Yes | — |
Query parameters
None.
Body schema
Content type: application/json · Optional
None.
What the API returns
Each response code below includes the documented payload shape for the ACP API.
200Immutable comparison or its original retry receiptapplication/json
| Field | Type | Required | Description |
|---|---|---|---|
| comparison | object | Yes | — |
| comparison.schemaVersion | computer_agents_quality_comparison_v1 | Yes | — |
| comparison.experimentId | string | Yes | — |
| comparison.evaluationId | string | Yes | — |
| comparison.protocolFingerprint | string | Yes | — |
| comparison.candidateMeasurementId | string | Yes | — |
| comparison.candidateMeasurementFingerprint | string | Yes | — |
| comparison.outcome | improved | not_improved | inconclusive | Yes | — |
| comparison.metrics | object[] | Yes | — |
| comparison.metrics[].metricId | string | Yes | — |
| comparison.metrics[].label | string | Yes | — |
| comparison.metrics[].baseline | object | Yes | — |
| comparison.metrics[].baseline.id | string | Yes | Unique identifier. |
| comparison.metrics[].baseline.label | string | Yes | — |
| comparison.metrics[].baseline.description | string | Yes | Human-readable description. |
| comparison.metrics[].baseline.numeratorLabel | string | Yes | — |
| comparison.metrics[].baseline.denominatorLabel | string | Yes | — |
| comparison.metrics[].baseline.unit | percent | Yes | — |
| comparison.metrics[].baseline.aggregation | micro | Yes | — |
| comparison.metrics[].baseline.direction | higher_is_better | lower_is_better | informational | Yes | — |
| comparison.metrics[].baseline.state | measured | partial | unmeasured | invalid | Yes | — |
| comparison.metrics[].baseline.value | number | Yes | A percentage |
| comparison.metrics[].baseline.numerator | integer | Yes | — |
| comparison.metrics[].baseline.denominator | integer | Yes | — |
| comparison.metrics[].baseline.populationCount | integer | Yes | — |
| comparison.metrics[].baseline.includedCount | integer | Yes | — |
| comparison.metrics[].baseline.excludedCounts | object | Yes | — |
| comparison.metrics[].baseline.excludedCounts.unmeasured | integer | No | — |
| comparison.metrics[].baseline.excludedCounts.missing_reference | integer | No | — |
| comparison.metrics[].baseline.excludedCounts.source_changed | integer | No | — |
| comparison.metrics[].baseline.excludedCounts.not_applicable | integer | No | — |
| comparison.metrics[].baseline.excludedCounts.error | integer | No | — |
| comparison.metrics[].baseline.excludedCounts.skipped | integer | No | — |
| comparison.metrics[].baseline.excludedCounts.missing_output | integer | No | — |
| comparison.metrics[].baseline.excludedCounts.invalid | integer | No | — |
| comparison.metrics[].baseline.uncertainty | object | Yes | — |
| comparison.metrics[].baseline.uncertainty.kind | not_estimated | Yes | — |
| comparison.metrics[].baseline.uncertainty.reason | string | Yes | — |
| comparison.metrics[].candidate | object | Yes | — |
| comparison.metrics[].candidate.id | string | Yes | Unique identifier. |
| comparison.metrics[].candidate.label | string | Yes | — |
| comparison.metrics[].candidate.description | string | Yes | Human-readable description. |
| comparison.metrics[].candidate.numeratorLabel | string | Yes | — |
| comparison.metrics[].candidate.denominatorLabel | string | Yes | — |
| comparison.metrics[].candidate.unit | percent | Yes | — |
| comparison.metrics[].candidate.aggregation | micro | Yes | — |
| comparison.metrics[].candidate.direction | higher_is_better | lower_is_better | informational | Yes | — |
| comparison.metrics[].candidate.state | measured | partial | unmeasured | invalid | Yes | — |
| comparison.metrics[].candidate.value | number | Yes | A percentage |
| comparison.metrics[].candidate.numerator | integer | Yes | — |
| comparison.metrics[].candidate.denominator | integer | Yes | — |
| comparison.metrics[].candidate.populationCount | integer | Yes | — |
| comparison.metrics[].candidate.includedCount | integer | Yes | — |
| comparison.metrics[].candidate.excludedCounts | object | Yes | — |
| comparison.metrics[].candidate.excludedCounts.unmeasured | integer | No | — |
| comparison.metrics[].candidate.excludedCounts.missing_reference | integer | No | — |
| comparison.metrics[].candidate.excludedCounts.source_changed | integer | No | — |
| comparison.metrics[].candidate.excludedCounts.not_applicable | integer | No | — |
| comparison.metrics[].candidate.excludedCounts.error | integer | No | — |
| comparison.metrics[].candidate.excludedCounts.skipped | integer | No | — |
| comparison.metrics[].candidate.excludedCounts.missing_output | integer | No | — |
| comparison.metrics[].candidate.excludedCounts.invalid | integer | No | — |
| comparison.metrics[].candidate.uncertainty | object | Yes | — |
| comparison.metrics[].candidate.uncertainty.kind | not_estimated | Yes | — |
| comparison.metrics[].candidate.uncertainty.reason | string | Yes | — |
| comparison.metrics[].deltaPercentagePoints | number | Yes | — |
| comparison.metrics[].eligibleExamples | integer | Yes | — |
| comparison.metrics[].comparable | boolean | Yes | — |
| comparison.metrics[].reason | string | Yes | — |
| comparison.reasons | string[] | Yes | — |
| comparison.adoption | object | Yes | — |
| comparison.adoption.allowed | false | Yes | — |
| comparison.adoption.reasons | string[] | Yes | — |
| comparison.createdByUserId | string | Yes | — |
| comparison.createdAt | string | Yes | ISO 8601 timestamp. |
| comparison.fingerprint | string | Yes | — |
400Unknown request fields
None.
401Authentication required
None.
403Organization or entitlement unavailable; execution credentials cannot run this analysis
None.
404Experiment or current case-management permission unavailable
None.
409References or retained evidence differ from the pinned protocol
None.
503Durable storage unavailable
None.
Measure candidate outputs with pinned references and retain one comparison
Loading...
Response 200
Loading...