# Results Logging Protocol Track every iteration in a structured log. Enables pattern recognition and prevents repeating failed experiments. ## Setup & Initialization Autoresearch creates the log automatically at Phase 0 (baseline). The agent runs these commands during initialization: ```bash # 1. Create log file with metric direction and header echo "# metric_direction: higher_is_better" > autoresearch-results.tsv echo -e "iteration\tcommit\tmetric\tdelta\tguard\tguard-metric\tstatus\tdescription" >> autoresearch-results.tsv # 2. Add to .gitignore (log is local, not committed) echo "autoresearch-results.tsv" >> .gitignore # 3. Run verify command to establish baseline metric BASELINE=$(npx jest --coverage 2>&1 | grep 'All files' | awk '{print $4}') # 4. Record baseline as iteration 0 COMMIT=$(git rev-parse --short HEAD) echo -e "0\t${COMMIT}\t${BASELINE}\t0.0\tpass\tbaseline\tinitial state — coverage ${BASELINE}%" >> autoresearch-results.tsv ``` ## Logging Function Called at Phase 7 of every iteration after the keep/discard/crash decision: ```bash # Function: log_iteration log_iteration() { local iteration=$1 commit=$2 metric=$3 delta=$4 guard=$5 guard_metric=$6 status=$7 description=$8 echo -e "${iteration}\t${commit}\t${metric}\t${delta}\t${guard}\t${guard_metric}\t${status}\t${description}" \ >> autoresearch-results.tsv } # Usage examples: log_iteration 1 "b2c3d4e" "87.1" "+1.9" "pass" "-" "keep" "add tests for auth middleware" log_iteration 2 "-" "86.5" "-0.6" "-" "-" "discard" "refactor test helpers (broke 2 tests)" log_iteration 3 "-" "0.0" "0.0" "-" "-" "crash" "add integration tests (DB connection failed)" log_iteration 4 "-" "-" "-" "-" "-" "no-op" "attempted to modify read-only config" log_iteration 5 "-" "-" "-" "-" "-" "hook-blocked" "pre-commit lint rejected formatting" log_iteration 6 "-" "-" "-" "-" "-" "metric-error" "verify output was 'PASS' — not a number" ``` ## Reading & Using the Log ```bash # Phase 1 (Review): Read recent entries for pattern recognition tail -20 autoresearch-results.tsv # Count outcomes for progress tracking KEEPS=$(grep -c 'keep' autoresearch-results.tsv || echo 0) DISCARDS=$(grep -c 'discard' autoresearch-results.tsv || echo 0) CRASHES=$(grep -c 'crash' autoresearch-results.tsv || echo 0) # Detect stuck state: >5 consecutive discards triggers recovery LAST_5=$(tail -5 autoresearch-results.tsv | awk -F'\t' '{print $6}') # If all 5 are "discard" → trigger "When Stuck" protocol (re-read all files, try radical change) # Pattern recognition: which file changes succeed? # Cross-reference "keep" rows with git log to find winning patterns grep 'keep' autoresearch-results.tsv | awk -F'\t' '{print $7}' # → Shows descriptions of all successful changes ``` ## Integration with the Autoresearch Loop Where logging fits in the loop lifecycle: ``` Phase 0 (Setup): → CREATE log file, record baseline (iteration 0) Phase 1 (Review): → READ last 10-20 log entries for pattern recognition Phase 3-6 (Loop): → Modify, Commit, Verify, Decide Phase 7 (Log): → APPEND new row after keep/discard/crash decision Phase 8 (Repeat): → Back to Phase 1 (reads updated log) ``` Complete end-to-end example: ``` $autoresearch Goal: Increase test coverage from 72% to 90% Scope: src/**/*.ts Verify: npx jest --coverage 2>&1 | grep 'All files' | awk '{print $4}' Guard: npm run typecheck # Internal lifecycle: # 1. Agent creates autoresearch-results.tsv with baseline 72.0 # 2. Agent reads log (empty except baseline) → decides first experiment # 3. Agent modifies code, commits, runs verify → gets 74.5 # 4. Agent appends: "1 b2c3d4e 74.5 +2.5 pass keep add auth middleware tests" # 5. Next iteration: agent reads log, sees auth tests worked → tries similar pattern # 6. Continues until coverage reaches 90% or iterations exhausted ``` ## Log Format (TSV) Create `autoresearch-results.tsv` in the working directory (gitignored): ```tsv iteration commit metric delta guard guard-metric status description ``` ### Columns | Column | Type | Description | |--------|------|-------------| | iteration | int | Sequential counter starting at 0 (baseline) | | commit | string | Short git hash (7 chars), "-" if reverted | | metric | float | Measured value from verification | | delta | float | Change from previous best (negative = improved for "lower is better") | | guard | enum | `pass`, `fail`, or `-` (no guard configured) | | guard-metric | float or `-` | Measured guard-metric value (metric-valued guards only). `-` for pass/fail guards or no guard. | | status | enum | `baseline`, `keep`, `keep (reworked)`, `discard`, `crash`, `no-op`, `hook-blocked`, `metric-error` | | description | string | One-sentence description of what was tried | ### Example (pass/fail guard) ```tsv iteration commit metric delta guard guard-metric status description 0 a1b2c3d 85.2 0.0 pass - baseline initial state — test coverage 85.2% 1 b2c3d4e 87.1 +1.9 pass - keep add tests for auth middleware edge cases 2 - 86.5 -0.6 - - discard refactor test helpers (broke 2 tests) 3 - 0.0 0.0 - - crash add integration tests (DB connection failed) 4 - 88.9 +1.8 fail - discard inline hot-path functions (guard: 3 tests broke) 5 c3d4e5f 88.3 +1.2 pass - keep add tests for error handling in API routes 6 d4e5f6g 89.0 +0.7 pass - keep add boundary value tests for validators ``` ### Example (metric-valued guard — bundle size with 5% threshold) ```tsv iteration commit metric delta guard guard-metric status description 0 a1b2c3d 85.2 0.0 pass 48200 baseline coverage 85.2%, bundle 48200 bytes 1 b2c3d4e 87.1 +1.9 pass 48500 keep add auth tests (bundle +300 bytes, within 5%) 2 - 88.0 +0.9 fail 51500 discard add integration tests (bundle +3300, exceeds 5% of 48200) 3 c3d4e5f 87.8 +0.7 pass 47900 keep add unit tests (bundle decreased) ``` **Note:** When guard fails, the metric may have improved but the change is still discarded. The guard column makes this visible in the log. For metric-valued guards, the guard-metric column lets you track drift over time even when individual iterations stay within threshold. ## Log Management - Create at setup (iteration 0 = baseline) - Append after EVERY iteration (including crashes) - Do NOT commit this file to git (add to .gitignore) - Read last 10-20 entries at start of each iteration for context - Use to detect patterns: what kind of changes tend to succeed? ## Summary Reporting Every 10 iterations (or at loop completion in bounded mode), print a brief summary: ``` === Autoresearch Progress (iteration 20) === Baseline: 85.2% → Current best: 92.1% (+6.9%) Keeps: 8 | Discards: 10 | Crashes: 2 Last 5: keep, discard, discard, keep, keep ``` ## Metric Direction Clarify at setup whether lower or higher is better: - **Lower is better:** val_bpb, response time (ms), bundle size (KB), error count - **Higher is better:** test coverage (%), lighthouse score, throughput (req/s) Record direction in first line of results log as a comment: ``` # metric_direction: higher_is_better ```