# Autonomous research for Death Gym

This guide helps a human operator run one or more coding agents as disciplined RL researchers. The directly executable agent instructions live at:

https://death-gym.up.railway.app/program.md

The design is inspired by [karpathy/autoresearch](https://github.com/karpathy/autoresearch), but the upstream repository must not be run unchanged. Its five-minute, single-GPU language-model loop optimizes validation bits per byte. Death Gym needs fixed environment-step budgets, multi-seed RL evidence, a frozen external evaluator, and strict separation from the public leaderboard.

## When this mode helps

Use autonomous mode when you have:

- a dedicated NVIDIA GPU for each active runner;
- enough time for repeated fixed-budget PPO runs;
- an isolated clone or worktree per agent;
- a clean evaluator checkout at the competition commit;
- an operator who can own hidden promotion seeds and final submission.

Keep the normal quickstart when the goal is only to understand the environment, reproduce a baseline, or make one manual experiment. Autoresearch adds overhead and should be optional.

## Recommended topology

Use three roles:

1. **Research agent** — edits only `research_train.py`, trains candidates, and records hypotheses.
2. **Frozen judge** — exports, validates, and scores candidate ZIPs from a clean checkout at `ba1f337038745f0d310e96542c5364257ed790d3`.
3. **Human operator** — allocates GPUs and budgets, retains hidden promotion seeds, and authorizes any external or paid action.

Run at most one training process per GPU. Multiple agents sharing a card destroy wall-time and memory comparability. Pin evaluation to one GPU model and explicit index; never rely on whichever device happens to be first visible.

The research agent must not control the judge checkout. Ideally the judge is a separate service or operator-owned process that returns aggregate metrics only.

## Why the upstream loop needs adaptation

Autoresearch's useful primitives are:

- one scoped editable file;
- one explicit hypothesis per experiment;
- fixed comparable budgets;
- a durable experiment ledger;
- automatic keep/discard decisions;
- continued work inside an approved campaign budget.

The stock assumptions do not transfer:

- RL training varies across seeds.
- Death Gym's learning signal can appear late, so five minutes may rank only startup noise.
- The bounty has no fixed training-compute rule; wall time is a resource, not the score.
- The public scoring seeds are known and therefore unsafe as an inner-loop validation set.
- The local scorer imports the canonical policy implementation, so an editable evaluator can disagree with the organizer.
- The final artifact must load into an evaluator-owned architecture and have zero truncated worlds.

## Operator setup

Choose unique absolute paths outside every other agent's checkout:

```text
<research-worktree>     agent-owned, one per campaign/GPU
<frozen-evaluator>      clean, operator-owned, pinned and read-only to the agent
<artifact-store>        bounded local storage for logs, checkpoints, and ZIPs
```

Never point an autonomous loop at a shared working tree.

Set explicit operator-owned paths and GPU indexes before running the examples:

```bash
export DM_RESEARCH_WORKTREE=/absolute/path/to/research-worktree
export DM_FROZEN_EVALUATOR=/absolute/path/to/frozen-evaluator
export DM_ARTIFACT_STORE=/absolute/path/to/artifact-store
export DM_DEV_JUDGE=/absolute/path/to/operator-owned-dev-judge
export DM_TRAIN_GPU_INDEX=0
export DM_EVAL_GPU_INDEX=0
```

Also set `DM_DEV_SEEDS` to exactly three distinct non-public decimal seeds chosen before launch. Abort if any overlaps the public practice seeds `3930,7717,20477`. All paths must be absolute and all variables must be nonempty. The operator-owned judge must enforce these checks itself.

Prepare the research checkout normally, then create the single editable surface:

```bash
cd "$DM_RESEARCH_WORKTREE"
test "$(git rev-parse HEAD)" = \
  "ba1f337038745f0d310e96542c5364257ed790d3"
if [ -e .venv ] || [ -L .venv ]; then
  test -x .venv/bin/python || {
    echo ".venv exists but has no executable interpreter; stop for operator repair" >&2
    exit 1
  }
else
  uv venv
fi
uv pip install --python .venv/bin/python -e '.[dev]'
just build
just test
just bench
test ! -e research_train.py || {
  echo "research_train.py already exists; refusing to overwrite it" >&2
  exit 1
}
cp train.py research_train.py
mkdir -p \
  "$DM_RESEARCH_WORKTREE/local/autoresearch/checkpoints" \
  "$DM_RESEARCH_WORKTREE/local/autoresearch/logs" \
  "$DM_ARTIFACT_STORE"
```

Prepare the evaluator in its own checkout:

```bash
git clone https://github.com/rvorias/death-gym.git "$DM_FROZEN_EVALUATOR"
git -C "$DM_FROZEN_EVALUATOR" checkout --detach ba1f337038745f0d310e96542c5364257ed790d3
cd "$DM_FROZEN_EVALUATOR"
if [ -e .venv ] || [ -L .venv ]; then
  test -x .venv/bin/python || {
    echo ".venv exists but has no executable interpreter; stop for operator repair" >&2
    exit 1
  }
else
  uv venv
fi
uv pip install --python .venv/bin/python -e '.[dev]'
just build
just test
```

Before every campaign, confirm:

```bash
test "$(git -C "$DM_FROZEN_EVALUATOR" rev-parse HEAD)" = \
  "ba1f337038745f0d310e96542c5364257ed790d3"
test -z "$(git -C "$DM_FROZEN_EVALUATOR" status --porcelain=v1 --untracked-files=all)"
```

Both assertions must pass before every score. Make the evaluator checkout filesystem read-only to the research agent. Do not let it patch `train.py`, `engine/`, `dmfast/`, `rewards.py`, `checkpoint.py`, `tools/`, `tests/`, or the evaluator checkout.

## Seed ownership

Create two evaluation layers before experiments:

- **Development banks:** three non-public seeds fixed for the campaign. The agent may see these if necessary.
- **Promotion banks:** a separate set held outside the agent's context and queried only by the operator for promoted candidates.

Never use the public practice seeds `3930,7717,20477` inside the research loop. Use them once after the final choice is frozen.

Even hidden seeds can be adaptively overfit through repeated score feedback. `evaluate_submission.py --quiet` still prints seed values, so the operator must capture its raw output privately. Return only aggregate metrics and anonymized bank means—never seeds or per-world trajectories.

## Campaign authorization

Give the agent one bounded authorization containing:

- its exact worktree;
- training and evaluation GPU indexes plus the pinned evaluation GPU model;
- maximum hours or GPU-hours;
- maximum disk usage;
- screen, replication, and scale step budgets;
- development judge command or frozen-evaluator path;
- starting checkpoint, if any;
- stop time.

This lets the agent continue unattended without treating “run forever” as an unlimited spending instruction. It still must stop before external uploads, wallet actions, legal acceptance, funding, or paid Taskmarket proofs.

Enforce the deadline, per-run timeout, GPU assignment, and disk quota outside the agent where possible. Before each run, require a remaining-budget and artifact-store-size check. Do not start a run that cannot fit. Retention may remove only files owned by a rejected experiment; never shared or incumbent artifacts. Long jobs should use the host's managed session mechanism, report at least every 60 seconds, and terminate the exact managed process on timeout or interruption.

Suggested kickoff prompt:

```text
Read https://death-gym.up.railway.app/program.md and follow it exactly.
Use only the isolated worktree I assigned. GPU: <index>.
Campaign budget: <hours> hours and <disk> GB.
Screen at <steps> environment steps. Do not exceed these limits.
The frozen judge is <path-or-command>.
Start from <fresh-or-checkpoint>.
Do not access Taskmarket or the public leaderboard during the loop.
```

## Successive-halving schedule

A practical default:

| Rung | Runs | Score bank | Purpose |
|---|---:|---|---|
| Smoke | 20M × 1 seed | 3 × 1,024 dev worlds | catch crashes and learning failures |
| Screen | 100M × 1 seed | 3 × 4,096 dev worlds | eliminate weak ideas |
| Replicate | 100M × 3 seeds | same dev banks | estimate training variance |
| Scale | 500M × 3 seeds | operator-held 3 × 16,384 | compare top two |
| Confirm | 1B or approved final budget | hidden promotion, then public once | freeze winner |

Treat the numbers as operator-set defaults, not permission to consume compute. Calibrate the first untouched baseline because the repository's prose is inconsistent about the learning achieved at 20M steps.

For replication, pair candidate and incumbent on the same training seeds and evaluation banks. First average the three banks within each trained policy, then compute one paired delta per training seed. With three seeds, use the predeclared one-sided 95% Student-t lower bound `mean(delta) - 2.920 × SE(delta)`. Do not treat nine seed/bank cells as independent. Promote only when this bound is above zero and every truncation count is zero.

Before a hidden query, choose one exact ZIP per treatment using a development-only rule fixed in advance, such as always taking training seed 11. Hidden results must not select the luckiest seed replica. Default caps are two ZIP queries total at Scale and one exact finalist ZIP at Confirm; public scoring cannot change the frozen artifact.

## Scoring template

The agent trains in its worktree:

```bash
CUDA_VISIBLE_DEVICES="$DM_TRAIN_GPU_INDEX" \
DM_CHECKPOINTS="$DM_RESEARCH_WORKTREE/local/autoresearch/checkpoints" \
"$DM_RESEARCH_WORKTREE/.venv/bin/python" \
  "$DM_RESEARCH_WORKTREE/research_train.py" \
  --total-steps 100000000 \
  --checkpoint-every 25000000 \
  --seed 11 \
  --run-name e001-seed11 \
  > "$DM_RESEARCH_WORKTREE/local/autoresearch/logs/e001-seed11.log" 2>&1
```

The clean evaluator packages and validates the candidate:

```bash
"$DM_FROZEN_EVALUATOR/.venv/bin/python" \
  "$DM_FROZEN_EVALUATOR/tools/export_submission.py" \
  "$DM_RESEARCH_WORKTREE/local/autoresearch/checkpoints/e001-seed11/final.safetensors" \
  --out "$DM_ARTIFACT_STORE/e001-seed11.zip"

"$DM_FROZEN_EVALUATOR/.venv/bin/python" \
  "$DM_FROZEN_EVALUATOR/tools/validate_submission.py" \
  "$DM_ARTIFACT_STORE/e001-seed11.zip" \
  --load

sha256sum "$DM_ARTIFACT_STORE/e001-seed11.zip"
```

The stock scorer below is useful for a manual reference check, but it is not an independent judge and its output is not machine-readable:

```bash
CUDA_VISIBLE_DEVICES="$DM_EVAL_GPU_INDEX" \
DM_COMPETITION_SEEDS="$DM_DEV_SEEDS" \
"$DM_FROZEN_EVALUATOR/.venv/bin/python" \
  "$DM_FROZEN_EVALUATOR/tools/evaluate_submission.py" \
  "$DM_ARTIFACT_STORE/e001-seed11.zip" \
  --worlds 4096 \
  --quiet
```

The autonomous agent instead calls `"$DM_DEV_JUDGE" "$DM_ARTIFACT_STORE/e001-seed11.zip"`. That operator-owned wrapper must validate the frozen commit, exact ZIP hash, three non-public development seeds, GPU assignment, 4,096 worlds per bank, batch width 2,048, finite metrics, and zero truncations; it must exit nonzero on any mismatch and return the `death-gym-development-v1` JSON contract specified in `/program.md`.

Score the ZIP itself. Keep the same evaluator commit, GPU model, batch width, world count, and bank set for paired comparisons. If the wrapper is unavailable, the agent may smoke-test but must not promote.

For hidden promotion, the operator runs that scorer privately and returns a redacted result with protocol `death-gym-promotion-v1`, evaluator commit, exact ZIP SHA-256, worlds per bank, batch width, anonymized bank means, aggregate mean XP, and truncation count. Never forward raw scorer stdout. Reject mismatched hashes or protocol fields, non-finite metrics, and any nonzero truncation.

## What agents may explore

Start with the submit-safe `dm_lstm_v1 256/256/1` actor.

Good early axes:

- rollout length, gamma, and GAE lambda;
- learning rate and entropy schedule;
- reward transform and reward-shaping ablations;
- permutation-consistent bag/market augmentation;
- curricula that are disabled for clean evaluation;
- privileged critics, search teachers, and distillation into the canonical actor.

The first controlled treatment should preserve 65,536 transitions per PPO iteration while lengthening credit assignment:

```python
NUM_ENVS = 512
ROLLOUT_STEPS = 128
GAMMA = 0.999
GAE_LAMBDA = 0.97
```

Do not initially widen the model. Never select a candidate whose final tensor set is not accepted by the canonical validator.

## Integrity gates

Reject a run if any of the following occurs:

- the research diff touches a frozen file;
- the effective step budget differs from its comparator;
- training or evaluation produces NaN, crash, timeout, or OOM;
- the ZIP is invalid or cannot load into the canonical model;
- any evaluated world truncates;
- the evaluation used an edited engine, mask, batch width, or public seed;
- the improvement appears only in training XP, max XP, or one lucky bank;
- the candidate needs `--shop-gate` or `--engine-param` behavior absent from deployment.

A training-only teacher is allowed only when its knowledge is distilled into a valid submitted actor.

## Taskmarket boundary

Autonomous research ends with a local, validated ZIP and a human-readable report. It must never upload every candidate or use Taskmarket as an experiment tracker.

After the operator selects the final artifact, follow:

https://death-gym.up.railway.app/llms.txt

Wallet creation/import, legal acceptance, funding, the exact artifact upload, and the paid proof retain their explicit human-approval gates.