Eval plan / result テンプレート

このページは、記入用テンプレートを誤ってサイトのメタデータとして解釈しないよう、原文をコードとして表示しています。

原本をダウンロード

---
schema: codex-delivery/eval@1
document_id: "EVAL-<<required: change ID or stable eval ID>>"
change_id: "CHG-<<required: YYYYMMDD-slug>>"
risk_tier: "<<required: R0 | R1 | R2 | R3>>"
document_status: draft
decision_state: UNKNOWN
owner: "<<required: eval owner>>"
evidence_as_of: UNKNOWN
commit_sha: "<<required: candidate exact SHA or UNKNOWN before implementation>>"
supersedes: "<<required: prior eval plan ID or N/A>>"
refs:
  - TASK-<<required>>
---

# Eval plan and result: <<required: decision>>

This form covers deterministic behavioral verification and AI eval. Mark AI-only sections `N/A — reason` for a non-AI change. Freeze the plan before viewing candidate results.

## Decision contract

- Decision to make: `<<required: ship/route/model/prompt/retrieval/tool/policy/rollback>>`
- Eligible unit and denominator: `<<required: request/job/case/account; define exclusions>>`
- Risk tier / affected segment: `<<required>>`
- Baseline fingerprint: `<<required>>`
- Candidate fingerprint: `<<required>>`
- Primary criterion: `<<required>>`
- Critical vetoes: `<<required>>`
- Guardrails: `<<required: security, policy, latency, cost, human work>>`
- Required practical effect/range: `<<required or N/A with reason>>`
- Decision date / owner / rollback owner: `<<required>>`

Do not import a universal threshold from a benchmark. Set criteria from the product promise, risk, baseline, sample limits, and maximum loss.

## Freeze record

- Plan version/hash: `<<required>>`
- Frozen at/by: `<<required: ISO 8601 and actor>>`
- Code/config cutoff: `<<required>>`
- Cases visible to implementation author: `<<required>>`
- Holdout access owner: `<<required or N/A with reason>>`
- Leakage/duplicate check: `<<required>>`
- Changes after freeze: `<<required: new version and reason, or none>>`

Changing a case, segment, threshold, rubric, or treatment after freeze creates a new plan version. Preserve the old result.

## Test/eval layers

| Layer | Claim | Cases/command | Oracle | Required? | Evidence/result ID |
|---|---|---|---|---:|---|
| Static/type/schema | <<required>> | <<required>> | deterministic | yes/no | EVID-<<required>> |
| Unit/property | <<required>> | <<required>> | deterministic | yes/no | EVID-<<required>> |
| Integration/E2E | <<required>> | <<required>> | state/post-condition | yes/no | EVID-<<required>> |
| Security/abuse | <<required>> | <<required>> | safe end state | yes/no | EVID-<<required>> |
| AI task quality | <<required or N/A>> | <<required>> | rubric/reference | yes/no | EVID-<<required>> |
| AI tool effect | <<required or N/A>> | <<required>> | verified end state | yes/no | EVID-<<required>> |
| Cost/latency | <<required>> | <<required>> | budget contract | yes/no | EVID-<<required>> |
| Preview/canary | <<required>> | <<required>> | runtime signals | yes/no | EVID-<<required>> |

## Case-set identity and governance

| Set | Purpose | Count scheduled | Content hash | Provenance/license/consent | Privacy/retention | Author visibility |
|---|---|---:|---|---|---|---|
| development | iteration | <<required>> | <<required>> | <<required>> | <<required>> | visible |
| holdout | release comparison | <<required>> | <<required>> | <<required>> | <<required>> | restricted |
| adversarial | critical/abuse | <<required>> | <<required>> | <<required>> | <<required>> | <<required>> |
| production audit | distribution drift | <<required>> | <<required>> | <<required>> | <<required>> | after collection |

Never paste raw customer data or secret prompts into this ledger. Reference a controlled input artifact with provenance, deletion, and access rules.

## Case contract

Copy a row/block per case or keep cases in a machine-readable file and reference its immutable hash.

| Case ID | Set | Segment/risk | Input ref/hash | Preconditions | Expected properties/end state | Forbidden result/severity | Repeats | Oracle |
|---|---|---|---|---|---|---|---:|---|
| CASE-01 | <<required>> | <<required>> | <<required>> | <<required>> | <<required>> | <<required>> | <<required>> | <<required>> |

### Oracle independence

- Requirement/source used to define expected behavior: `<<required>>`
- Who authored/reviewed the oracle: `<<required>>`
- How circular self-confirmation is reduced: `<<required: frozen AC, external spec, independent fixture, human review, state constraint>>`
- Allowed variation: `<<required>>`
- Unknown/unjudgeable rule: `<<required>>`
- Author / fixture author / runner / reviewer separation: `<<required, or solo-developer compensation controls and remaining limitation>>`

## AI configuration fingerprint

Use `N/A — non-AI change` when appropriate.

| Component | Baseline | Candidate | Immutable ref/hash |
|---|---|---|---|
| Model/provider/tier | <<required>> | <<required>> | <<required>> |
| System/developer prompt | <<required>> | <<required>> | <<required>> |
| Retrieval index/query | <<required>> | <<required>> | <<required>> |
| Tools/schema/authority | <<required>> | <<required>> | <<required>> |
| Policy/guardrail | <<required>> | <<required>> | <<required>> |
| Grader/rubric | <<required>> | <<required>> | <<required>> |
| Runner/dependencies | <<required>> | <<required>> | <<required>> |

## AI tool action and effect ledger

Required for every AI/agent action that can write, send, pay, delete, publish, change permission, or invoke another consequential tool. Use `N/A — no consequential tool path; evidence <scan ID/hash>` only when proven inapplicable. This table indexes an append-only action artifact; do not paste secrets or sensitive arguments.

| Action ID | Proposal payload hash | Tool/target/args/policy/effect class hash | Approval ID/hash | Authorizer actor ID / role-authority basis / scope / expiry | Commit-time precondition/version | Idempotency key | External effect/receipt ID | Verification due/status/evidence | Compensation/owner |
|---|---|---|---|---|---|---|---|---|---|
| ACT-EFF-01 | <<required>> | <<required>> | <<required>> | <<required>> | <<required>> | <<required>> | <<required or pending>> | <<required>> | <<required>> |

Approval is valid only for the exact proposal payload, target, arguments, policy/effect class, scope, and expiry. Recheck mutable preconditions immediately before commit. A successful API response is not a verified effect; record the authoritative final state or keep it `UNKNOWN`. R2/R3 action promotion requires this ledger, attack cases, bounded authority, and a human decision event.

## Grading and maturity

- Goal success definition: `<<required>>`
- Policy-compliant success definition: `<<required>>`
- Critical failure definition: `<<required>>`
- Tool post-condition verification: `<<required or N/A>>`
- Judge type / calibration set / agreement: `<<required>>`
- Timeout/abort/retry treatment: `<<required>>`
- Unknown treatment: `<<required>>`
- Quality/value/incident maturity times: `<<required or N/A with reason>>`
- Multiple-run metric: `<<required: pass^k/pass@k/none and why>>`

## Execution snapshot

Append one row per run. A regrade or rerun creates a new row; do not overwrite history.

- Evidence manifest root ID/hash: `<<required: immutable source of truth for all result rows and raw artifacts>>`

| Run ID | Plan/case hash | Code/config fingerprint | Environment | Command/runner | Started/completed | Exit | Raw artifact hash/path | State |
|---|---|---|---|---|---|---:|---|---|
| RUN-01 | <<required>> | <<required>> | <<required>> | <<required>> | <<required>> | <<required>> | <<required>> | executed / judged / matured / reopened |

## Case results

| Run/case | Judged? | Goal success | Policy success | Critical/severity | Verified effect | Attempts | Latency/cost | Human work | Evidence |
|---|---:|---:|---:|---|---|---:|---|---|---|
| RUN-01/CASE-01 | yes/no | pass/fail/unknown | pass/fail/unknown | none/low/medium/high/critical | pass/fail/unknown/N/A | <<required>> | <<required>> | <<required>> | <<required>> |

## Aggregate without hiding cases

| Metric/segment | Baseline x/n | Candidate x/n | Unknown | Judged coverage | Difference/range | Criterion | Result |
|---|---|---|---:|---|---|---|---|
| Primary | <<required>> | <<required>> | <<required>> | <<required>> | <<required>> | <<required>> | PASS/FAIL/UNKNOWN |
| Critical cases | <<required>> | <<required>> | <<required>> | <<required>> | <<required>> | zero required | PASS/FAIL/UNKNOWN |
| Cost/latency | <<required>> | <<required>> | <<required>> | <<required>> | <<required>> | <<required>> | PASS/FAIL/UNKNOWN |

Report case-level critical failures, worst cases, and small `x/n`; do not let an average erase a veto. Scheduled cases must reconcile to pass + fail + unknown.

## Deviations and limitations

- Missing/failed/skipped cases: `<<required>>`
- Environment deviations: `<<required>>`
- Leakage/contamination: `<<required>>`
- Judge disagreement/drift: `<<required>>`
- Population and generalization limit: `<<required>>`
- Evidence expiry or next monitoring date: `<<required>>`

## Gate decision

- Required gates: `<<required: IDs>>`
- Active critical failure: `yes/no — IDs`
- Required evidence missing/stale/immature: `yes/no — IDs`
- Derived state: `STOP | PAUSE | UNKNOWN | SHIP`
- Decision / actor / time: `<<required>>`
- Rollback or promotion action: `<<required>>`
- Next production observation: `<<required or N/A with reason>>`