Small-data experiment and decision pack

最終確認: 2026-08-02
用途: 低トラフィックの Web app で、価格、onboarding、販売、AI 機能、運用変更を 記述 / 関連 / 因果 の境界を守って判断する

この pack は判断・計測・証拠管理の書式であり、統計、法律、医療、金融その他の個別助言ではない。高影響領域や重要な推定は、領域専門家・統計家・法律専門家の review を受ける。

記入例の企業、人物、ID、件数、率、金額、期間、効果は完全な架空であり、実績、予測、benchmark、推奨 sample size ではない。

氏名、email、住所、会話全文、cookie 値、IP address、決済情報、健康・信用・雇用等の機微情報、credential、raw prompt/output を公開書式や生成 AI へ貼らない。実運用は opaque ID、集約、redacted summary、hash、権限制御した正本への reference を使う。

章との責任境界

• 19 章: decision-first、time zero、ITT、integrity、小標本、設計、privacy、decision gate の原則。
• 04 章: marketing、funnel、channel experiment。
• 06 章: 日本の security、privacy、契約、表示等の共通確認。
• 12 章: AI job、quality、provider cost、human rescue、retention。
• 13 章: code/metric/query の exact change、release、rollback evidence。
• SQLite companion: 本書式の主要 invariant、snapshot freshness、decision chain を完全架空データと DQ01–DQ102 で検査する参考実装。

14 書式を使う順序

1. 書式 1 で「何を決めるか」と情報の価値を決める。
2. 書式 2 で eligible population、unit、identity、time zero を凍結する。
3. 書式 3 で control / treatment の exact version と exposure cap を固定する。
4. 書式 4 で primary、driver、guardrail、data-quality metric を version 化する。
5. 書式 5 で assignment と allocation を固定する。
6. 書式 6〜7 で event と expected outcome spine を作る。
7. 書式 8 で A/A、SRM、coverage 等を通す。
8. 書式 9 で assignment から analysis までの flow と欠測を閉じる。
9. 書式 10 で design、stopping、multiplicity、sensitivity を事前固定する。
10. 書式 11 で ITT-first readout を作る。
11. 書式 12 の risk/privacy/legal/ethics veto を先に評価する。
12. 書式 13 で action、段階 rollout、rollback、monitoring を決める。
13. Codex を使う場合だけ書式 14 で redacted handoff を作る。

共通 contract

ID、version、時刻

decision_id / decision_version:
design_id / design_version:
intervention_id / intervention_version:
metric_id / metric_version:
assignment_id / assignment_revision:
analysis_snapshot_id / analysis_revision:
evidence_id / evidence_revision:
supersedes_evidence_id: N/A | exact ID

effective_from / effective_until:
occurred_at:
recorded_at:
assigned_at / follow_up_start_at:
outcome_cutoff_at:
snapshot_as_of:
decision_due_at:

owner / reviewer / reviewed_at:
source_system / source_object_ref:
query_or_artifact_hash:
retention_due_at:

• 過去行を静かに update せず、新 revision と exact supersession を作る。
• occurred_at <= outcome_cutoff_at と recorded_at <= snapshot_as_of を別に判定する。
• future exposure や outcome で eligibility / assignment を書き換えない。
• hash は内容固定の補助であり、真正性、適法性、因果性を単独で証明しない。

Evidence class

E0_INFERENCE_OR_UNVERIFIED_NOTE
E1_APPROVED_PLAN_OR_SELF_REPORTED
E2_VERSIONED_CONFIG_OR_ASSIGNMENT_RECEIPT
E3_VERIFIED_PRODUCT_EVENT_OR_TEST
E4_CUSTOMER_OR_INDEPENDENT_OUTCOME
E5_CASH_REGULATORY_OR_INCIDENT_AUTHORITY

上位ほど常に正しいのではない。assignment には E2、顧客受入には E4、銀行着金には E5 のように、stage に適した正本を事前指定する。

状態を混ぜない

状態	意味
N/A	contract 上その項目が存在しない。理由と recheck trigger がある
UNKNOWN	必要だが未確認、期限切れ、矛盾、join 不能
0	観測対象と窓が確定した上でのゼロ
FAILED	実行した検査または必要条件が明示的に不合格
OBSERVED	必要 evidence class の肯定観測
NOT_DUE	outcome maturity がまだ来ていない

Decision priority

STOP > PAUSE > UNKNOWN > INCONCLUSIVE > CHANGE > MAINTAIN > EXPAND

UNKNOWN は計測不能、INCONCLUSIVE は計測は有効だが実務上有益・有害の両方が残る状態である。0、FAILED、no effect と置き換えない。

書式 1 — Decision / value-of-information contract

数字を見る前に複製する。変えたい action がなければ、まず dashboard を作らない。

# Decision and information value

decision_id / version:
decision_owner / independent_reviewer:
created_at / decision_due_at:

customer/business question:
current decision options:
  - option A:
  - option B:
  - option C:
claim sought: DESCRIPTION | ASSOCIATION | CAUSAL
estimand in plain language:

eligible population summary:
primary outcome / maturity:
minimum practically important effect (MPE):
harm / trivial / valuable zones:

maximum acceptable:
  customer/data/financial harm:
  live subjects/accounts:
  cash:
  founder minutes:
  elapsed time:

plausible upside if decision is correct:
plausible avoidable loss if decision is wrong:
probability new evidence changes action: LOW | MEDIUM | HIGH | UNKNOWN
instrumentation/build/analysis cost:
traffic opportunity cost:
delay cost:
exposure risk:
cheapest evidence that could change action:

run experiment?: YES | NO | UNKNOWN
reason:
if no, rollout/rollback or qualitative probe:

precommitted action map:
  integrity invalid:
  material harm:
  valuable effect with safe guardrails:
  trivial effect:
  interval spans harm and value:
  primary immature at due date:

approved_by / approved_at:
decision_contract_hash:

Review

[ ] 一つの decision と複数の現実的 option がある
[ ] metric の変化でなく顧客・事業 action を書いた
[ ] MPE と最大損失を別にした
[ ] 全 plausible result が同じ action なら test を中止した
[ ] test 費用に build、traffic、delay、exposure、founder time を含めた
[ ] external benchmark や fixture 数値を target にしていない

書式 2 — Population / unit / identity / time-zero contract

# Population, units, identity, and time zero

population_contract_id / version:
decision_id / decision_version:

eligibility rule:
eligibility source / evidence class:
eligibility evaluated_at:
exclusion rule and reason codes:
sampling frame coverage / known omissions:

eligibility unit:
randomization unit:
exposure unit:
analysis unit:
cluster_id rule: N/A | rule

identity keys allowed:
identity graph version:
merge / split rules:
shared device / multi-account / proxy-operation handling:
test/internal/bot handling:

time_zero:
  eligibility_at:
  assigned_at:
  follow_up_start_at:
required ordering:
natural outcome cycle:
maturity rule:

interference map:
  who can affect whom:
  shared queue/capacity/operator:
  referral/social/market effects:
  spillover contained by cluster?: YES | NO | UNKNOWN

privacy/applicability review ref:
owner / reviewer / approved_at:
contract_hash:

Review

[ ] eligibility は outcome/exposure より前の情報だけで判定する
[ ] eligibility、randomization、exposure、analysis unit を混ぜていない
[ ] 同一 unit の重複、merge、split、転職等を扱える
[ ] assignment と follow-up start を揃えた
[ ] future adoption で treatment cohort を定義していない
[ ] interference がある場合に individual independence を仮定していない

書式 3 — Intervention / control version and scope

# Intervention and control version

intervention_contract_id / version:
decision_id:
environment: SYNTHETIC | SANDBOX | LIMITED_LIVE | LIVE

control:
  experience description:
  code/config/artifact version:
  UI/copy/price/entitlement version:
  model/prompt/tool/routing/policy version: N/A | exact

treatment:
  one intended change:
  experience description:
  code/config/artifact version:
  UI/copy/price/entitlement version:
  model/prompt/tool/routing/policy version: N/A | exact

retry / fallback / cache policy:
start / assignment end / exposure end:
allocation ratio:
eligible scope:
maximum live subjects/accounts:
maximum data/cash/external effects:

change freeze:
approved exception process:
hotfix / provider change handling:
contamination paths:

feature flag / disable owner:
rollback method / rehearsal evidence:
recovery post-condition:

owner / reviewer / approved_at:
intervention_hash:

Review

[ ] treatment と control は再現可能な exact version である
[ ] 複数変更なら package 効果だけを主張する
[ ] provider alias や remote config の未知変更を扱う
[ ] exposure cap と reversible stop がある
[ ] hotfix 後を同じ version として黙って結合しない

書式 4 — Metric/KPI contract

metric ごとに複製する。同じ decision では primary を一つにする。

# Metric contract

metric_id / version:
decision_id / decision_version:
role: PRIMARY | DRIVER | GUARDRAIL | DATA_QUALITY
human-readable question:

unit of analysis:
eligible denominator rule:
numerator rule:
aggregation: COUNT | RATE | SUM | MEAN | MEDIAN | QUANTILE | DISTRIBUTION
direction: HIGHER_BETTER | LOWER_BETTER | TWO_SIDED | VETO_ONLY

source-of-truth event/table:
required evidence class:
deduplication key:
event-time field / timezone:
window start / end:
maturity rule:
late / reopened rule:
missing rule:
exclusions / reason codes:

minimum practically important effect: N/A | value
guardrail limit / veto: N/A | rule
allowed prespecified segments:
forbidden post-treatment dimensions:

baseline source / period: UNKNOWN | exact
target: TBD_BASELINE | exact with rationale
query / code version:
owner / reviewer / effective_from:
metric_contract_hash:

Review

[ ] 分子、分母、unit、窓、maturity、dedup がある
[ ] primary、driver、guardrail、data-quality の役割が一つだけである
[ ] missing を silent 0 にしない
[ ] ratio と同時に eligible 総数・絶対件数を表示する
[ ] post-treatment adoption/engagement を causal segment にしない
[ ] target は baseline と事業制約から別途設定した

書式 5 — Assignment manifest

# Assignment manifest

assignment_manifest_id / revision:
decision_id / design_id / population_contract_id:
intervention_contract_id / metric_contract_set_id:

randomization unit / cluster rule:
allocation method: HASH | RANDOM_SEED | PREGENERATED_SEQUENCE | OTHER
allocation ratio / strata / blocks:
seed or algorithm version:
persistence key / storage:
reassignment rule: NEVER | exact exceptional rule

eligibility snapshot_at:
assignment starts_at / ends_at:
expected units by arm:
maximum units:
assignment service/version:

pre-assignment exposure check:
duplicate-assignment prevention:
concurrency/idempotency control:
assignment receipt source:

generated_by / reviewed_by / approved_at:
manifest_hash:

一 unit の receipt

assignment_id:
manifest_id / manifest_hash:
subject_id / cluster_id:
eligibility_evidence_id:
arm: CONTROL | TREATMENT
assigned_at / recorded_at:
assignment_algorithm_version:
receipt_hash:

Review

[ ] founder が好みで treatment を選べない
[ ] assignment は一意・永続で retry に対して idempotent
[ ] allocation、strata、seed、algorithm が outcome 前に固定
[ ] expected arm ratio を後の SRM check に渡せる
[ ] assignment receipt が exposure event から独立して残る

書式 6 — Event / instrumentation contract

event type ごとに複製する。

# Event and instrumentation contract

event_contract_id / version:
decision_id / metric_ids:
event_name / business meaning:

producer / service / environment:
schema version / payload hash method:
event_id / idempotency key:
subject / assignment / intervention keys:
occurred_at / recorded_at / timezone:
trusted append authority / monotone ingestion sequence:
source-of-truth vs derived:

required fields:
nullable fields and meaning:
forbidden raw fields:
dedup rule:
ordering rule:
late-arrival / correction / supersession rule:
reopened outcome rule:

delivery path / vendor / destination legal entity:
observability for dropped events:
retention / deletion / access control:

positive fixture:
duplicate fixture:
late fixture:
missing-key fixture:
wrong-arm fixture:

owner / reviewer / verified_at:
network payload inspection evidence:
inspection data/environment: SYNTHETIC | STAGING | AUTHORIZED_MINIMIZED_PRODUCTION
production inspection authorization / redaction / expiry: N/A | exact
contract_hash:

Review

[ ] event 名でなく business meaning と正本性を定義した
[ ] occurred_at と recorded_at を分けた
[ ] recorded_at は client claim でなく trusted append clock で、backdate を拒否する
[ ] late fact 後は analysis / decision を新 revision で supersede する
[ ] delivery failure 自体を観測できる
[ ] treatment ごとの performance/error/log loss を比較できる
[ ] raw personal/sensitive data を便利さで追加していない
[ ] actual network payload と destination を確認した

書式 7 — Expected outcome / maturity spine

assignment 時に canonical expected_outcome_spine へ一行を作り、成功 event が来た人だけを分母にしない。

# Expected outcome spine row

expected_outcome_id:
decision_id / metric_id / metric_version:
assignment_id / subject_id / arm:
assigned_at / follow_up_start_at:
outcome_due_at:
maturity_rule_version:

status: NOT_DUE | DUE_OPEN | OBSERVED | MISSING | INELIGIBLE_CORRECTION
outcome_value: N/A | 0 | value
outcome_event_id: N/A | exact ID
missing_reason: N/A | INSTRUMENTATION | IDENTITY_LINK | WITHDRAWAL |
                SERVICE_UNAVAILABLE | LATE_UNRESOLVED | OTHER

occurred_by_cutoff: YES | NO | UNKNOWN
recorded_by_snapshot: YES | NO | UNKNOWN
closed_at / evidence_id:
correction / supersession ref:

Spine reconciliation

metric_id / arm:
assigned:
not_due:
due_open:
observed_0:
observed_nonzero:
missing:
ineligible_correction:

assertion:
assigned = not_due + due_open + observed_0 + observed_nonzero + missing + ineligible_correction
status: PASS | FAILED | UNKNOWN

Review

[ ] assignment 時に expected row を作る
[ ] NOT_DUE、0、MISSING、UNKNOWN_LINK を分ける
[ ] outcome event がなくても分母から消えない
[ ] ineligible correction は理由・revision を残し事後除外を隠さない
[ ] cutoff と snapshot watermark の両方を持つ

書式 8 — A/A, SRM, and integrity preflight

# Integrity preflight

integrity_review_id / revision:
decision_id / assignment_manifest_id:
reviewed_at / evidence_cutoff / snapshot_as_of:

A/A rehearsal:
  same experience exact versions:
  synthetic/live scope:
  assignment/delivery/logging/metric path covered:
  rerandomization or repeated calibration performed:
  expected false-alarm behavior:
  observed anomalies / resolution:
  result: PASS | FAILED | UNKNOWN

SRM:
  expected arm counts/ratio:
  actual assigned counts/ratio:
  randomization unit / assignment inclusion rule:
  method / threshold / predeclared_at:
  result: PASS | FAILED | UNKNOWN
  root cause status:

other integrity checks:
  eligible-to-assigned reconciliation:
  one assignment per unit/cluster:
  pre-assignment exposure:
  downstream event coverage / attrition by arm:
  outcome missingness by arm:
  duplicate / ordering / clock / timezone:
  test/bot/internal filtering:
  metric/query version match:
  contamination / crossover:
  outcome spine reconciliation:

business_result_visibility_before_review: HIDDEN | EXPOSED_WITH_REASON
overall: PASS | PAUSE | UNKNOWN
owner / independent reviewer / approved_at:
review_hash:

Review

[ ] A/A は一回の「差なし」で完全証明としていない
[ ] SRM fail は原因不明のまま補正して進めない
[ ] SRM pass を無 bias の証明にしていない
[ ] assignment-receipt SRM と下流 event loss/attrition/missingness を分けた
[ ] arm 別 event loss、latency、error、missing を比較した
[ ] integrity pass 前の business effect を意思決定者へ見せない運用を検討した

書式 9 — Assignment / exposure / contamination / missingness ledger

subject/assignment ごとに一行を作る。個人情報を入れず opaque ID を使う。

# Experiment flow row

flow_row_id / revision:
decision_id / assignment_id / subject_id / arm:

eligible: YES | NO | UNKNOWN
assigned: YES | NO | UNKNOWN
offered: YES | NO | N/A | UNKNOWN
exposed: YES | NO | N/A | UNKNOWN
adopted: YES | NO | N/A | UNKNOWN
mature: YES | NO | UNKNOWN
outcome_observed: YES | NO | UNKNOWN
analyzed_in_itt: YES | NO | UNKNOWN

assignment_at / first_offer_at / first_exposure_at:
outcome_due_at / outcome_at:

contamination: NONE | CROSSOVER | OTHER_CHANNEL | SHARED_OPERATOR |
               SHARED_ACCOUNT | UNKNOWN
contamination_at / evidence:

missingness: NONE | INSTRUMENTATION | IDENTITY_LINK | WITHDRAWAL |
             SERVICE_UNAVAILABLE | LATE_UNRESOLVED | OTHER | UNKNOWN
missingness may depend on arm/outcome?: YES | NO | UNKNOWN
exclusion_reason: N/A | exact predeclared code

event/evidence refs:
reviewed_at / reviewer:

Arm-level flow table

arm | eligible | assigned | offered | exposed | adopted | mature |
    | observed | missing | contaminated | excluded | analyzed_itt
CONTROL   |  |  |  |  |  |  |  |  |  |  |
TREATMENT |  |  |  |  |  |  |  |  |  |  |

Missingness sensitivity

primary assumption:
complete-case result:
all treatment missing=failure, control missing=success:
all treatment missing=success, control missing=failure:
other plausible mechanism/model:
does action change?: YES | NO | UNKNOWN
limitation:

Review

[ ] ITT principle では全 randomized/assigned unit を original arm に残し、事後判明の ineligibility も flow と sensitivity に残す
[ ] available-case x/n を missing のない完全な ITT effect と呼ばない
[ ] exposure/adoption comparison を主 causal estimate にしない
[ ] missing と exclusion を arm 別 reason 付きで示す
[ ] contamination を削除せず診断と sensitivity に使う
[ ] simple imputation で precision を水増ししていない

書式 10 — Analysis / stopping / multiplicity / design plan

business result を見る前に承認する。

# Analysis plan

analysis_plan_id / version:
decision_id / design_id:
claim cap: DESCRIPTION | ASSOCIATION | CAUSAL
primary estimand:

design: INDIVIDUAL_RANDOMIZED | CLUSTER_RANDOMIZED |
        RANDOMIZED_STAGED_ROLLOUT | SWITCHBACK |
        RANDOMIZED_ENCOURAGEMENT | INTERRUPTED_TIME_SERIES |
        MATCHED_OBSERVATIONAL | BEFORE_AFTER | QUALITATIVE_PROBE

assignment / analysis unit:
cluster/strata/block handling:
interference assumptions:
carryover assumptions:

primary metric / version:
confirmatory metric family:
guardrails / data-quality metrics:
prespecified segments / interaction tests:
exploratory outputs allowed:

planning objective: DECISION_PRECISION | POWER_FOR_EFFECT | FEASIBILITY
baseline / variance source and period:
MPE / effect used for planning:
alpha / power or interval-width target:
planned units / calendar / maturity delay:
allocation / cluster / attrition assumptions:
sample/precision calculation code and reviewer:

analysis population: ITT | OTHER_WITH_JUSTIFICATION
effect measures: ABSOLUTE | RELATIVE | BOTH
uncertainty level / sidedness / method / assumptions:
standard-error / cluster method:
missingness primary analysis:
sensitivity analyses:

stopping mode: FIXED_HORIZON | PREDECLARED_SEQUENTIAL
fixed assignment end / calendar minimum / maturity delay:
sequential method / look schedule / boundaries: N/A | exact
efficacy early-stop rule: NONE | exact
safety/legal/privacy early-stop rule:

multiplicity family / correction method:
post-hoc handling: EXPLORATORY_ONLY

query/code/artifact version:
analysis environment:
statistical review required?: YES | NO | UNKNOWN
approved_by / approved_at:
plan_hash:

Switchback appendix, when applicable

whole-system treatment reason:
time block length / number of blocks:
randomized block sequence ref:
time strata / calendar coverage:
plausible carryover horizon:
washout or transition exclusion:
serial dependence method:
shared queue/inventory/operator state:
outcome attribution window:
underestimated-carryover sensitivity:

Review

[ ] design は traffic でなく interference/carryover/ethics/feasibility に合う
[ ] fixed horizon か本物の sequential method のどちらかを選んだ
[ ] MPE と MDE/precision target を混ぜず、baseline・variance・cluster・attrition の仮定を示した
[ ] definitive impact を区別できない設計なら事前に FEASIBILITY objective とした
[ ] dashboard peeking を anytime-valid inference と呼ばない
[ ] confirmatory family と exploratory slice を分けた
[ ] subgroup は interaction を見ずに「片方だけ有意」で比較しない
[ ] observational design は仮定なしに CAUSAL へ昇格しない

書式 11 — Assignment-first ITT flow / practical-value readout

# Analysis result snapshot

analysis_snapshot_id / revision:
decision_id / analysis_plan_id / exact plan_hash:
generated_at / outcome_cutoff_at / snapshot_as_of:
query/code/artifact hash:
claim cap: DESCRIPTION | ASSOCIATION | CAUSAL

integrity review id / status:
deviations from plan:
deviation timing: PRE_OUTCOME | POST_OUTCOME | UNKNOWN

flow by arm:
  CONTROL: eligible / assigned / exposed / mature / observed / missing / contaminated
  TREATMENT: eligible / assigned / exposed / mature / observed / missing / contaminated

primary assignment-first result:
  assigned / outcome due / observed / missing / analyzed by arm:
  prespecified missing-outcome method:
  observed complete-case control x/n or distribution:
  observed complete-case treatment x/n or distribution:
  assignment-based effect estimate under stated assumptions:
  absolute difference:
  relative difference:
  uncertainty interval / level / sidedness / method:
  MPE:
  interval vs harm/trivial/valuable zones:

driver results:
guardrail results / maximum harm:
data-quality results:
prespecified segment interactions:
exploratory findings, clearly labeled:

missingness/contamination sensitivity:
adjusted estimate: N/A | value/method
unadjusted estimate:
assumptions that could reverse conclusion:

business economics:
  total experiment cash:
  founder minutes:
  traffic/opportunity/delay cost:
  expected fully loaded impact range:

result classification:
  INVALID | MATERIAL_HARM | VALUABLE | TRIVIAL | INCONCLUSIVE | UNKNOWN
limitations:
generated_by / independently_reviewed_by / reviewed_at:
snapshot_hash:

Readout rules

[ ] x/n と利用可能 outcome 数を arm 別に表示した
[ ] original assignment の全 unit を arm に残し、due/observed/missing/analyzed n を示した
[ ] 絶対差を相対差より先に説明した
[ ] control rate が 0 なら relative effect は N/A とした
[ ] confidence interval を「真値が入る確率」と説明していない
[ ] p-value を treatment 成功確率、重要性、出荷命令にしていない
[ ] not significant を no effect にしていない
[ ] interval が material harm と valuable benefit を跨ぐなら INCONCLUSIVE
[ ] winning point estimate をそのまま revenue forecast にしていない

Pre-period variance reduction, optional

method: NONE | CUPED | OTHER
covariate measured strictly before assignment?: YES | NO | UNKNOWN
coverage / pre-post correlation:
validated on A/A?: YES | NO | UNKNOWN
treatment-affected/post-trigger risk:
adjusted and unadjusted results both reported?: YES | NO
local variance reduction observed:

pre-period covariate が treatment の影響を受ける、または post-trigger なら使わない。外部事例の variance reduction を自 app の効果として見込まない。

書式 12 — Safety / privacy / legal / ethics gate

# Risk, privacy, legal, and ethics gate

risk_review_id / revision:
decision_id / intervention version:
facts_as_of / jurisdictions / customer types:
reviewer / professional consultation ref:

safety/security:
  maximum plausible harm:
  privileged/data/cash/external action:
  human review / appeal / recovery:
  incident detection and owner:
  status: PASS | STOP | UNKNOWN | N/A

privacy/data:
  exact purpose / notice version:
  data fields / subject / relationship / location:
  identifier / personal-related / personal-data classification:
  linkage expected:
  destination legal entity / role / country:
  consent or other applicability review:
  processor scope / own-use prohibition:
  retention / deletion / access / vendor receipt:
  status: PASS | STOP | UNKNOWN | N/A

children/special-care/regulated-sector:
  minor or guardian-consent applicability:
  special-care/sensitive information:
  regulated/high-impact domain:
  sector_review_required: YES | NO | UNKNOWN
  guardian/sector review evidence ref:
  status: PASS | STOP | UNKNOWN | N/A

external transmission:
  actual network payload inspected:
  inspection data/environment: SYNTHETIC | STAGING | AUTHORIZED_MINIMIZED_PRODUCTION
  production authorization / minimization / redaction / expiry: N/A | exact
  destination / sender purpose / destination purpose:
  user confirmation opportunity:
  separate APPI review:
  status: PASS | STOP | UNKNOWN | N/A

claim/price/checkout:
  variant claim / substantiation / expiry:
  actual and comparison price basis:
  quantity / total or recurring price:
  payment/provision timing:
  cancellation/return/deadline clarity:
  dark-pattern / vulnerable-user review:
  status: PASS | STOP | UNKNOWN | N/A

email/messaging:
  consent wording/screen/time/source:
  sender/purpose scope:
  global suppression joined immediately before send:
  sender identity/address/contact/unsubscribe:
  records retention by applicable regime:
  status: PASS | STOP | UNKNOWN | N/A

fairness/high-impact:
  affected segments / proxy risk:
  material adverse impact:
  explanation / human review / appeal:
  excluded high-impact domains:
  status: PASS | STOP | UNKNOWN | N/A

overall: PASS | STOP | UNKNOWN
recheck trigger / due_at:
approved_by / approved_at / review_hash:

Hard veto

[ ] purpose/entity/role/location/data class/destination が UNKNOWN なら live 停止
[ ] actual payload 未検証なら live 停止
[ ] inspection は synthetic/staging 優先で、production は authorization・最小化・redaction・短期廃棄を持つ
[ ] claim 根拠期限切れ、price/renewal/cancel 不明瞭なら停止
[ ] 子ども・要配慮情報・規制/高影響領域の guardian/sector review が UNKNOWN なら停止
[ ] email consent/global suppression が join 不能なら送信停止
[ ] high-impact/vulnerable decision に human review/appeal/recovery がなければ停止
[ ] deletion/vendor return-deletion が未実証なら拡大停止

書式 13 — Decision / rollout / reversal snapshot

# Decision snapshot

decision_snapshot_id / revision:
decision_id / decision_contract_hash:
analysis_snapshot_id / snapshot_hash:
risk_review_id / review_hash:
decided_at / decision_owner / independent reviewer:

claim permitted: DESCRIPTION | ASSOCIATION | CAUSAL
decision: STOP | PAUSE | UNKNOWN | INCONCLUSIVE | CHANGE | MAINTAIN | EXPAND
reason:
primary effect / uncertainty / MPE:
guardrail and maximum harm:
integrity status:
key assumptions / unresolved unknowns:

next action:
next evidence that could change action:
owner / due_at:

rollout stage:
eligible scope / exposure cap:
start / review / maturity dates:
monitoring metrics / alerts:
feature flag / release version:

rollback triggers:
rollback authority / maximum response time:
rollback action / rehearsal evidence:
post-condition / customer recovery:

post-launch realized impact review_due_at:
null/negative/invalid learning ledger ref:
supersedes decision snapshot:
approved_by / approved_at:
decision_snapshot_hash:

Decision rules

SRM/telemetry/time-zero/version failure      -> PAUSE or UNKNOWN, no causal ship proof
material safety/legal/privacy/ethics harm    -> STOP
interval spans material harm and value       -> INCONCLUSIVE
effect is precise but below practical value  -> MAINTAIN/CHANGE/STOP by cost, not EXPAND
valuable range + safe guardrails + readiness -> EXPAND candidate
post-hoc winning segment only                -> next-test hypothesis, not current proof

EXPAND は一度の全開を意味しない。段階ごとに新 exposure cap、maturity、monitoring、rollback readiness を確認する。

書式 14 — Codex analysis / implementation handoff

# Codex handoff

handoff_id / version:
decision_id / task_change_id:
requested outcome:

allowed inputs:
  synthetic/redacted dataset refs:
  schemas / metric contracts:
  analysis plan / exact hash:
  code/query/release refs:
forbidden inputs:
  raw PII / secrets / payment / sensitive content / customer conversations:

requested operations:
  SQL / tests / fixture generation:
  integrity diagnostics:
  analysis reproduction:
  documentation / diff review:

non-goals:
  choose business metric:
  approve causal assumptions:
  determine legal basis:
  authorize live rollout or external action:

required outputs:
  changed files / exact hashes:
  commands and receipts:
  x/n and flow reconciliation:
  deviations / assumptions / unknowns:
  negative/adversarial probes:

approval boundary:
tool/data/network permissions:
human reviewer / production authorizer:
rollback owner:

approved_by / approved_at:
handoff_hash:

Codex review prompt

次の decision contract、population、metric、assignment manifest、analysis plan を
exact version のまま review してください。

1. eligibility/assignment/follow-up の time zero がずれていないか。
2. post-treatment field を exclusion、segment、covariate に使っていないか。
3. ITT denominator から missing/contaminated unit が黙って消えていないか。
4. arm 別 SRM、coverage、duplicate、clock、late、maturity が照合するか。
5. fixed horizon / sequential rule / multiplicity family に違反していないか。
6. effect、uncertainty、MPE、guardrail、claim cap の整合を確認してください。
7. fact、assumption、limitation、UNKNOWN を分け、出荷判断はしないでください。

raw PII を出力せず、発見ごとに source ref、counterexample、影響する decision state、
最小修正、再検証 command を返してください。

完全架空の記入例 — KairoNote onboarding

この例の 24 account、結果、日付は操作練習用である。

decision_id/version: DEC-SYN-019 / 1
question: new-account onboarding B を限定拡大するか
claim sought: CAUSAL
eligible: 2026-07-01..07-10 に setup-ready となった synthetic account
randomization/analysis unit: account
primary: assignment 後14日以内の customer-confirmed completed workflow
MPE: absolute +15 percentage points (fictional exercise value)
guardrails:
  severe data error = 0 tolerated
  founder rescue >60m <= control

assignment: control 12 / treatment 12
outcome due (mature): control 10 / treatment 10
observable at snapshot: control 10 / treatment 9
observed complete-case primary: control 4/10 / treatment 6/9
not due: control 2 / treatment 2
missing: control 0 / treatment 1
contamination: control 1 / treatment 0
severe data error: control 0 / treatment 1
founder rescue >60m: control 1 / treatment 3

integrity: PAUSE — treatment missingness under investigation
risk: STOP — severe data error recovery not yet evidenced
decision: STOP
claim permitted: no business-effect claim while integrity/risk gates fail
next: disable B, recover affected synthetic record, fix cause, create intervention v2,
      rerun A/A and bounded rollout; do not delete the incident row

誤った読み方は「B は 67%、A は 40% なので +67% uplift」である。絶対件数、未成熟、欠測、contamination、重大 error、founder rescue を落としている。正しい練習成果は勝者選定ではなく、v1 を停止し、同じ data の後付け除外をせず、v2 の安全な検証条件を作ることである。

Experiment portfolio ledger

winner だけを残すと学習と forecast が歪む。decision ごとに一行を残す。

decision_id / intervention version:
question / claim cap:
started / ended / maturity:
result: INVALID | HARM | NEGATIVE | TRIVIAL | INCONCLUSIVE | VALUABLE
decision: STOP | PAUSE | UNKNOWN | INCONCLUSIVE | CHANGE | MAINTAIN | EXPAND
primary point/interval/MPE:
total cash / founder minutes / elapsed days:
root cause or learning:
post-launch realized impact review:
forecast revision:
reusable instrumentation/test:

raw winning uplift を合算して revenue plan にしない。選ばれた勝者は上振れしやすい。実運用では post-launch realization、replication、長期 holdout、縮小推定等を設計に応じて検討し、null、negative、invalid も残す。

Pack 完了チェック

[ ] 14 書式のうち decision に必要なものを exact ID で結んだ
[ ] decision と target を metric 作成より先に捏造していない
[ ] expected outcome spine が assignment 全件を保持する
[ ] integrity pass 前に business result で配分や停止日を変えていない
[ ] ITT、absolute/relative effect、uncertainty、MPE、guardrail を同時表示した
[ ] observational/exploratory claim を CAUSAL に昇格していない
[ ] privacy、表示、email、high-impact veto を uplift で相殺していない
[ ] Codex に raw PII や rollout authority を渡していない
[ ] null/negative/invalid と post-launch realized impact を ledger に残す