Results Logging Protocol

Track every iteration in a structured log. Enables pattern recognition and prevents repeating failed experiments.

Setup & Initialization

Autoresearch creates the log automatically at Phase 0 (baseline). The agent runs these commands during initialization:

# 1. Create log file with metric direction and header
echo "# metric_direction: higher_is_better" > autoresearch-results.tsv
echo -e "iteration\tcommit\tmetric\tdelta\tguard\tguard-metric\tstatus\tdescription" >> autoresearch-results.tsv

# 2. Add to .gitignore (log is local, not committed)
echo "autoresearch-results.tsv" >> .gitignore

# 3. Run verify command to establish baseline metric
BASELINE=$(npx jest --coverage 2>&1 | grep 'All files' | awk '{print $4}')

# 4. Record baseline as iteration 0
COMMIT=$(git rev-parse --short HEAD)
echo -e "0\t${COMMIT}\t${BASELINE}\t0.0\tpass\tbaseline\tinitial state — coverage ${BASELINE}%" >> autoresearch-results.tsv

Logging Function

Called at Phase 7 of every iteration after the keep/discard/crash decision:

# Function: log_iteration
log_iteration() {
  local iteration=$1 commit=$2 metric=$3 delta=$4 guard=$5 guard_metric=$6 status=$7 description=$8
  echo -e "${iteration}\t${commit}\t${metric}\t${delta}\t${guard}\t${guard_metric}\t${status}\t${description}" \
    >> autoresearch-results.tsv
}

# Usage examples:
log_iteration 1 "b2c3d4e" "87.1" "+1.9" "pass" "-" "keep" "add tests for auth middleware"
log_iteration 2 "-" "86.5" "-0.6" "-" "-" "discard" "refactor test helpers (broke 2 tests)"
log_iteration 3 "-" "0.0" "0.0" "-" "-" "crash" "add integration tests (DB connection failed)"
log_iteration 4 "-" "-" "-" "-" "-" "no-op" "attempted to modify read-only config"
log_iteration 5 "-" "-" "-" "-" "-" "hook-blocked" "pre-commit lint rejected formatting"
log_iteration 6 "-" "-" "-" "-" "-" "metric-error" "verify output was 'PASS' — not a number"

Reading & Using the Log

# Phase 1 (Review): Read recent entries for pattern recognition
tail -20 autoresearch-results.tsv

# Count outcomes for progress tracking
KEEPS=$(grep -c 'keep' autoresearch-results.tsv || echo 0)
DISCARDS=$(grep -c 'discard' autoresearch-results.tsv || echo 0)
CRASHES=$(grep -c 'crash' autoresearch-results.tsv || echo 0)

# Detect stuck state: >5 consecutive discards triggers recovery
LAST_5=$(tail -5 autoresearch-results.tsv | awk -F'\t' '{print $6}')
# If all 5 are "discard" → trigger "When Stuck" protocol (re-read all files, try radical change)

# Pattern recognition: which file changes succeed?
# Cross-reference "keep" rows with git log to find winning patterns
grep 'keep' autoresearch-results.tsv | awk -F'\t' '{print $7}'
# → Shows descriptions of all successful changes

Integration with the Autoresearch Loop

Where logging fits in the loop lifecycle:

Phase 0 (Setup):    → CREATE log file, record baseline (iteration 0)
Phase 1 (Review):   → READ last 10-20 log entries for pattern recognition
Phase 3-6 (Loop):   → Modify, Commit, Verify, Decide
Phase 7 (Log):      → APPEND new row after keep/discard/crash decision
Phase 8 (Repeat):   → Back to Phase 1 (reads updated log)

Complete end-to-end example:

$autoresearch
Goal: Increase test coverage from 72% to 90%
Scope: src/**/*.ts
Verify: npx jest --coverage 2>&1 | grep 'All files' | awk '{print $4}'
Guard: npm run typecheck

# Internal lifecycle:
# 1. Agent creates autoresearch-results.tsv with baseline 72.0
# 2. Agent reads log (empty except baseline) → decides first experiment
# 3. Agent modifies code, commits, runs verify → gets 74.5
# 4. Agent appends: "1  b2c3d4e  74.5  +2.5  pass  keep  add auth middleware tests"
# 5. Next iteration: agent reads log, sees auth tests worked → tries similar pattern
# 6. Continues until coverage reaches 90% or iterations exhausted

Log Format (TSV)

Create autoresearch-results.tsv in the working directory (gitignored):

iteration	commit	metric	delta	guard	guard-metric	status	description

Columns

Column	Type	Description
iteration	int	Sequential counter starting at 0 (baseline)
commit	string	Short git hash (7 chars), "-" if reverted
metric	float	Measured value from verification
delta	float	Change from previous best (negative = improved for "lower is better")
guard	enum	`pass`, `fail`, or `-` (no guard configured)
guard-metric	float or `-`	Measured guard-metric value (metric-valued guards only). `-` for pass/fail guards or no guard.
status	enum	`baseline`, `keep`, `keep (reworked)`, `discard`, `crash`, `no-op`, `hook-blocked`, `metric-error`
description	string	One-sentence description of what was tried

Example (pass/fail guard)

iteration	commit	metric	delta	guard	guard-metric	status	description
0	a1b2c3d	85.2	0.0	pass	-	baseline	initial state — test coverage 85.2%
1	b2c3d4e	87.1	+1.9	pass	-	keep	add tests for auth middleware edge cases
2	-	86.5	-0.6	-	-	discard	refactor test helpers (broke 2 tests)
3	-	0.0	0.0	-	-	crash	add integration tests (DB connection failed)
4	-	88.9	+1.8	fail	-	discard	inline hot-path functions (guard: 3 tests broke)
5	c3d4e5f	88.3	+1.2	pass	-	keep	add tests for error handling in API routes
6	d4e5f6g	89.0	+0.7	pass	-	keep	add boundary value tests for validators

Example (metric-valued guard — bundle size with 5% threshold)

iteration	commit	metric	delta	guard	guard-metric	status	description
0	a1b2c3d	85.2	0.0	pass	48200	baseline	coverage 85.2%, bundle 48200 bytes
1	b2c3d4e	87.1	+1.9	pass	48500	keep	add auth tests (bundle +300 bytes, within 5%)
2	-	88.0	+0.9	fail	51500	discard	add integration tests (bundle +3300, exceeds 5% of 48200)
3	c3d4e5f	87.8	+0.7	pass	47900	keep	add unit tests (bundle decreased)

Note: When guard fails, the metric may have improved but the change is still discarded. The guard column makes this visible in the log. For metric-valued guards, the guard-metric column lets you track drift over time even when individual iterations stay within threshold.

Log Management

Create at setup (iteration 0 = baseline)
Append after EVERY iteration (including crashes)
Do NOT commit this file to git (add to .gitignore)
Read last 10-20 entries at start of each iteration for context
Use to detect patterns: what kind of changes tend to succeed?

Summary Reporting

Every 10 iterations (or at loop completion in bounded mode), print a brief summary:

=== Autoresearch Progress (iteration 20) ===
Baseline: 85.2% → Current best: 92.1% (+6.9%)
Keeps: 8 | Discards: 10 | Crashes: 2
Last 5: keep, discard, discard, keep, keep

Metric Direction

Clarify at setup whether lower or higher is better:

Lower is better: val_bpb, response time (ms), bundle size (KB), error count
Higher is better: test coverage (%), lighthouse score, throughput (req/s)

Record direction in first line of results log as a comment:

# metric_direction: higher_is_better

6.9 KiB Raw Blame History