Two pull requests run the same automated review. The first flags a dependency with no clear owner; the second does not. The model appears unchanged, and so does the workflow, but nobody can say which exact set of instructions reached the agent in either run.
Without that identity, “the agent changed its behavior” is not a diagnosis. It is merely an observation that is difficult to reproduce. The result we want is more modest and more useful: keep the prompt in Git, calculate the SHA-256 of the bytes sent, record that identity with the run, and compare controlled changes in CI.
OpenAI’s current guidance is to treat production prompts like application code: store them in versioned files, review them through pull requests, and include fixtures and checks in the delivery process. This also avoids building a new solution around v1/prompts: creation of those objects was deprecated on June 3, 2026, and the endpoint is scheduled to shut down on November 30, 2026 (prompting documentation).
The minimum that belongs in the repository
For a review agent running in GitHub Actions, a small structure is enough:
.github/
codex/
prompts/
review.md
review.schema.json
workflows/
agent-review.yml
.agent/
tools.json
scripts/
record_agent_usage.py
usage.json
AGENTS.mdreview.md contains the task and rubric for that review. AGENTS.md holds durable repository guidance. In Codex, applicable AGENTS.md files are concatenated from the repository root down to the working directory; guidance from a closer file appears later and takes precedence. AGENTS.override.md provides a temporary override.
There is an important operational distinction: ~/.codex/AGENTS.md belongs to the machine environment and does not appear in the pull request. If a rule is mandatory in CI, it must live in the repository. Relying on the author’s home-directory file is a remarkably efficient way to review a configuration nobody else has.
The .agent/tools.json and .agent/prompt.toml paths are conventions used by our harness, not vendor APIs or reserved filenames. The same applies to usage.json: it is the run’s audit artifact, not the usage object returned by an API.
Checkout identity is not prompt identity
Record two different pieces of information:
git describe --tags --always --dirtyidentifies the checkout;- SHA-256 identifies the content that was actually loaded.
The first command may produce a value such as v1.4.0-7-g2414721. It helps locate the commit and reveals a dirty worktree, but it does not replace the prompt hash. The checkout must not be shallow if git describe is expected to see history and tags on the runner.
Do not confuse git hash-object with a raw SHA-256 digest either. Git calculates the identity of a blob object and may apply filters; hashlib.file_digest(..., "sha256"), with the file opened in binary mode, calculates the digest of the bytes read. Different CRLF bytes mean a different hash—the bytes remain unmoved by the argument that the file “looks the same.”
This script adds identity fields to an existing usage.json:
import hashlib
import json
import subprocess
from pathlib import Path
def sha256_file(path: Path) -> str:
with path.open("rb") as stream:
return hashlib.file_digest(stream, "sha256").hexdigest()
prompt_path = Path(".github/codex/prompts/review.md")
tools_path = Path(".agent/tools.json")
usage_path = Path("usage.json")
usage = json.loads(usage_path.read_text()) if usage_path.exists() else {}
usage.update({
"prompt_path": str(prompt_path),
"prompt_sha": sha256_file(prompt_path),
"tools_sha": sha256_file(tools_path) if tools_path.exists() else None,
"git_describe": subprocess.check_output(
["git", "describe", "--tags", "--always", "--dirty"],
text=True,
).strip(),
"git_sha": subprocess.check_output(
["git", "rev-parse", "HEAD"], text=True
).strip(),
})
usage_path.write_text(json.dumps(usage, indent=2, sort_keys=True) + "\n")This example assumes review.md is sent without transformation. If the harness renders templates, expands includes, or assembles messages, write the resolved content to a temporary file and calculate the digest from that file. The hash must represent the bytes sent, not the good intentions stored at the original path.
For production observability, publish usage.json as an artifact of every job and have CI recalculate the SHA of the committed file. The check should fail when the recorded value differs. Each output then carries an auditable link between the checkout, prompt, and tools.
Prompts and tools change the agent through different channels
The prompt defines policy, task, and rubric. The tools list, on the other hand, exposes functions with names, descriptions, and parameters expressed as JSON Schema. It is separate from the instructions in both function-tool calls and agent harnesses.
For that reason, do not embed tools.json in the prompt just to get a single hash. Keep the artifacts separate and record both prompt_sha and tools_sha. Renaming a tool or making a parameter required can change behavior while the prompt remains identical byte for byte.
Apply the same discipline to AGENTS.md: when the instruction chain loaded by the runner is part of the contract, record or verify it separately. The prompt_sha for review.md should not pretend to cover files whose bytes were never part of its input.
Do not use the Chat Completions seed as a prompt version either. A seed concerns sampling and still coexists with changes reported through system_fingerprint; it does not identify instructions, tools, or a checkout. Fixing it may help a specific experiment, but it does not make two reviews identical bit for bit.
A one-line prompt change needs a receipt
Consider two revisions of the same prompt file. review.v4.md contains everything from review.v3.md and adds one line to the rubric:
--- a/.github/codex/prompts/review.v3.md
+++ b/.github/codex/prompts/review.v4.md
@@
- Do not suggest new dependencies.
+- If the diff adds a dependency, require a human reviewer.
- End with a JSON object: {"ok": bool, "findings": [string]}The files executed in this example have these identities:
review.v3.md — 216 bytes
sha256: 4d93b509ce99e6e0d210fc96147e9ae13d04a03d3a1a5acdc1f0f29132fa4b9a
review.v4.md — 275 bytes
sha256: d2d3f27d4f42eace3af1ca6f3abdc50593df5588511b3b24342f71083cf92b36The hash difference proves that the input changed; it does not prove that the review improved. To evaluate behavior, freeze the pull request diff used as the fixture and run two jobs that differ only in the committed prompt. Preserve usage.json and the agent output from both runs. The test should assert that prompt_sha changed and then compare the results.
This design provides traceability: every result published by the job points to an exact prompt revision. If the pipeline also uses a token budget and circuit breaker, the identity fields belong in the same run artifact; they do not need a second accounting system.
A CI workflow without a hidden second source
The Codex GitHub Action accepts either prompt or prompt-file, never both. For a versioned review, choose the committed file. Checkout must run before the action, fetch-depth: 0 makes git describe useful, and codex-version receives an exact version approved by the repository. The official Codex Action example still shows actions/checkout@v5; this article pins actions/checkout@v7 because it is the current major on 2026-09-05.
name: Agent review
on:
pull_request:
jobs:
review:
runs-on: ubuntu-latest
permissions:
contents: read
steps:
- name: Checkout
uses: actions/checkout@v7
with:
fetch-depth: 0
persist-credentials: false
- name: Run Codex review
id: codex
uses: openai/codex-action@v1
with:
openai-api-key: ${{ secrets.OPENAI_API_KEY }}
prompt-file: .github/codex/prompts/review.md
output-file: artifacts/review.json
codex-version: ${{ vars.CODEX_VERSION_EXACT }}
codex-args: >-
["--output-schema", ".github/codex/review.schema.json"]
- name: Record prompt identity
run: python scripts/record_agent_usage.py
- name: Verify recorded SHA
run: |
python - <<'PY'
import hashlib, json
from pathlib import Path
prompt = Path(".github/codex/prompts/review.md")
with prompt.open("rb") as stream:
actual = hashlib.file_digest(stream, "sha256").hexdigest()
recorded = json.loads(Path("usage.json").read_text())["prompt_sha"]
assert recorded == actual, (recorded, actual)
PYAfterward, publish artifacts/review.json and usage.json through the artifact step already pinned by the repository.
CODEX_VERSION_EXACT should contain a complete, immutable version approved by the team, not latest or a range. Keeping it in a variable avoids inventing a version in the example; protecting it and recording its value in usage.json prevents convenience from becoming another invisible input.
In practice, the primary gate should validate the JSON against review.schema.json: required fields, allowed severity values, finding locations, and the absence of invalid structure. Exact Markdown should not be the only required golden. Residual nondeterminism remains even with model and version pinned, so scores, ordering, and wording may vary without breaking the contract.
You can keep representative outputs and inspect them with git diff --no-index or diff -u. Just do not turn identical punctuation into a pipeline health requirement. Given the chance, the machine will happily fail over a comma.
Failure modes the check must catch
Three failures show up regularly in this design:
- The workflow defines both
prompt-fileand an inlineprompt:. The action rejects the duplicate, so the fix is to keep one versioned source. - The hash is calculated before the template is rendered. Persist and hash the resolved payload instead, then compare that digest with the value recorded in the artifact.
- The test changes the prompt, fixture, tools, and model at the same time. Freeze the input pull request and change only the prompt file between the two runs.
Also recalculate tools_sha, record the commit as git_sha, and confirm that the runner loaded the expected AGENTS.md files. Durable project guidance belongs near the code; the details of assembling the remaining context are a separate agent concern, covered in project memory for coding agents.
When to adopt this contract—and when to step back
DevDojo would adopt this contract for a review agent that runs in CI, comments on pull requests, or can block a merge. At that point, the prompt, tools, runner version, and structured output are part of what goes into production: they pass through review, receive hashes, and run against frozen fixtures.
For an exploratory one-session prompt, this is unnecessary weight. Record the hypothesis, text, and result in a lab journal or daily note without turning the attempt into a required check. AGENTS.override.md can also hold temporary guidance, provided it is not mistaken for the shared CI contract.
The next step is small: move one real review into .github/codex/prompts/review.md, add prompt_sha and tools_sha to usage.json, and run the same fixture with review.v3.md and review.v4.md. If CI can prove which input it executed and validate the output schema, the next regression stops being a debate about memory and becomes a verifiable diff.