An autonomous research loop inspired by Karpathy's autoresearch pattern. Instead of editing training code and running short LLM training jobs, this project runs a Python research agent that drafts reports, evaluates them, iterates until quality improves, and persists the results.
The workflow is orchestrated by Inngest and powered by OpenRouter for model access.
When you submit a research topic, the system:
- Accepts the request over HTTP and queues an Inngest event.
- Runs a durable multi-step workflow that survives retries and restarts.
- Uses a ResearchAgent to write a markdown research brief via OpenRouter.
- Uses an EvaluatorAgent to score the draft and suggest the next improvement direction.
- Repeats the draft/evaluate cycle for up to
max_rounds, keeping the best-scoring draft. - Saves a final markdown report under
reports/. - Appends structured experiment metadata to
memory/experiments.json. - Returns a structured result from the Inngest function run.
This is not model training. It applies the same core idea from Karpathy's repo (propose a change, measure the outcome, keep or iterate) to research generation instead of weight updates.
Karpathy's autoresearch repo is an autonomous overnight experiment loop around a small real LLM training setup: the agent edits code, trains briefly, checks whether things improved, keeps or discards the change, and repeats. That repo also steers the "research org" through program.md context files instead of hand-editing Python files directly.
This project borrows that shape:
| Karpathy autoresearch | Python AutoResearch |
|---|---|
| Edit training code | Draft research brief |
| Run a short training job | Call OpenRouter for generation |
| Measure loss or metric | Evaluator scores clarity, depth, structure |
| Keep or discard change | Keep best draft, feed back weaknesses |
| Repeat overnight | Repeat for max_rounds via Inngest steps |
The durable orchestration layer (Inngest) replaces a manual or shell-based loop, which makes the research cycle observable, retryable, and easy to extend.
POST /research
|
v
Inngest event: research/requested
|
v
research_workflow.py (research-loop function)
|
+--> ResearchAgent --> OpenRouter (draft markdown)
|
+--> EvaluatorAgent --> OpenRouter (score + feedback JSON)
|
+--> report_store --> reports/*.md
|
+--> report_store --> memory/experiments.json
|
v
Return final structured result
flowchart TB
Client[HTTP Client]
API[FastAPI main.py]
InngestDev[Inngest Dev Server]
Workflow[research_workflow.py]
Research[ResearchAgent]
Evaluator[EvaluatorAgent]
OR[OpenRouter API]
Reports[reports/]
Memory[memory/experiments.json]
Client -->|POST /research| API
API -->|send research/requested| InngestDev
InngestDev -->|invoke steps| Workflow
Workflow --> Research
Workflow --> Evaluator
Research --> OR
Evaluator --> OR
Workflow --> Reports
Workflow --> Memory
API -->|serve /api/inngest| InngestDev
Each round of the loop is wrapped in ctx.step.run(...), which gives you:
- Durability: if a step fails, Inngest retries from the last successful step instead of restarting the whole run.
- Observability: every step (
draft-1,evaluate-1,save-report, etc.) appears in the Inngest dev UI. - Sleep between rounds:
ctx.step.sleep(...)paces iterations without blocking the server process. - Event-driven entry: the HTTP handler only enqueues work; the heavy lifting runs in the background function.
The FastAPI app registers the workflow with inngest.fast_api.serve(app, inngest_client, [research_loop]), exposing the standard Inngest sync endpoint at /api/inngest.
OpenRouter provides a unified OpenAI-compatible API over many models. This project uses the official OpenAI Python SDK with:
base_urlset tohttps://openrouter.ai/api/v1- Your API key from the OpenRouter dashboard
OpenRouter recommends protecting keys and deleting compromised keys immediately. Optional HTTP-Referer and X-OpenRouter-Title headers can be configured for app rankings (see .env.example).
For each round from 1 to max_rounds:
- Draft:
ResearchAgent.draft(topic, objective, feedback)produces markdown with sections for Summary, Current landscape, Key opportunities, Risks / unknown, and Next actions. - Evaluate:
EvaluatorAgent.evaluate(topic, draft)returns JSON withscore,strengths,weaknesses, andnext_direction. - Track best: the highest-scoring draft is kept as
best_report. - Early exit: if
score >= 8, the loop stops early. - Feedback: otherwise,
next_directionis passed into the next draft as feedback. - Pause: a 1 second sleep runs between rounds via Inngest.
After the loop, the workflow builds a final markdown document (iterations log + best draft), saves it, and appends metadata to the experiment memory file.
Request (POST /research):
| Field | Type | Default | Description |
|---|---|---|---|
topic |
string | required | Research subject (min 3 characters) |
objective |
string | "Produce a detailed, practical research brief." |
What the brief should accomplish |
max_rounds |
integer | 3 |
Iteration cap (1 to 10) |
Evaluator output (per round):
{
"score": 8,
"strengths": ["clear structure"],
"weaknesses": ["limited depth"],
"next_direction": "Add concrete examples and citations"
}Experiment memory record (appended to memory/experiments.json):
{
"topic": "AI coding agents in 2026",
"objective": "Produce a practical research brief...",
"best_score": 9,
"report_path": "reports/20260605-043453--ai-coding-agents-in-2026.md",
"iterations": [ ... ]
}| Layer | Technology |
|---|---|
| HTTP API | FastAPI |
| ASGI server | Uvicorn |
| Workflow engine | Inngest Python SDK |
| LLM access | OpenAI SDK via OpenRouter |
| Config | python-dotenv |
| Validation | Pydantic |
Requirements: Python 3.10 or newer.
python-autoresearch/
├── app/
│ ├── __init__.py
│ ├── agents/
│ │ ├── __init__.py
│ │ ├── evaluator_agent.py # Scores drafts, returns JSON feedback
│ │ └── research_agent.py # Generates markdown research briefs
│ ├── config/
│ │ ├── __init__.py
│ │ └── settings.py # Environment-based configuration
│ ├── services/
│ │ ├── __init__.py
│ │ ├── openrouter.py # AsyncOpenAI client for OpenRouter
│ │ └── report_store.py # Report and experiment persistence
│ ├── workflows/
│ │ ├── __init__.py
│ │ └── research_workflow.py # Inngest research-loop function
│ ├── inngest_client.py # Shared Inngest client instance
│ └── schemas.py # Pydantic request models
├── memory/
│ └── experiments.json # Append-only experiment history
├── reports/ # Generated markdown reports
├── .env # Local secrets (not committed)
├── .env.example # Environment template
├── main.py # FastAPI app + Inngest serve
├── requirements.txt
└── README.md
| File | Role |
|---|---|
main.py |
Health endpoint, POST /research event sender, Inngest handler registration |
app/inngest_client.py |
Creates the Inngest client with app id python-autoresearch |
app/workflows/research_workflow.py |
Defines the research-loop function and iteration logic |
app/agents/research_agent.py |
Prompts OpenRouter to write structured research briefs |
app/agents/evaluator_agent.py |
Prompts OpenRouter to score and critique each draft |
app/services/openrouter.py |
Configures the AsyncOpenAI client pointed at OpenRouter |
app/services/report_store.py |
Writes markdown files and appends JSON experiment records |
app/config/settings.py |
Loads and validates environment variables at startup |
Before setup, install:
-
Python 3.10+
python3 --version
-
An OpenRouter API key
- Create an account at openrouter.ai
- Generate an API key from the dashboard
- Keep the key private; never commit it to git
-
Node.js and npm (for the Inngest dev server)
- Used via
npxso no global Inngest install is required - See Quick Start Runbook for daily commands
- Used via
Once prerequisites are ready, jump to Quick Start Runbook for one-time and every-time setup.
This is the practical section. Read this first if you just want to run the project.
All commands run from inside python-autoresearch/.
git clone <your-repo-url>
cd python-autoresearch
python -m venv .venv
source .venv/bin/activate
pip install -r requirements.txt
cp .env.example .envEdit .env and set your real OpenRouter key:
OPENROUTER_API_KEY=sk-or-v1-your-actual-key-here
OPENROUTER_MODEL=openai/gpt-4o
OPENROUTER_HTTP_REFERER=http://localhost:8000
OPENROUTER_APP_TITLE=python-autoresearch
INNGEST_DEV=1Optional: create output directories upfront (they are also created automatically on first run):
mkdir -p reports memory
echo "[]" > memory/experiments.jsonYou do not need to install the Inngest CLI globally. Use npx in the daily run steps below.
You need two terminals open at the same time. Order matters: start Terminal 1 first, then Terminal 2.
Terminal 1: FastAPI app
cd python-autoresearch
source .venv/bin/activate
INNGEST_DEV=1 .venv/bin/uvicorn main:app --reloadWait until you see:
INFO: Uvicorn running on http://127.0.0.1:8000
INFO: Application startup complete.
Terminal 2: Inngest dev server
cd python-autoresearch
npx --yes inngest-cli@latest dev -u http://localhost:8000/api/inngestFirst run downloads the CLI (about 15 to 30 seconds). Later runs start faster.
Wait until you see logs like:
starting server ... addr":"0.0.0.0:8288"
apps synced, disabling auto-discovery
Confirm both are connected
In Terminal 1 you should start seeing:
PUT /api/inngest HTTP/1.1" 200 OK
That means Inngest is synced with your FastAPI app.
Quick health check (any terminal):
curl http://127.0.0.1:8000/Expected: {"status":"ok"}
# Terminal 1 (API server)
cd python-autoresearch
source .venv/bin/activate
INNGEST_DEV=1 .venv/bin/uvicorn main:app --reload
# Terminal 2 (Inngest dev server)
cd python-autoresearch
npx --yes inngest-cli@latest dev -u http://localhost:8000/api/inngest
# Terminal 3 (trigger a research run)
curl -X POST http://localhost:8000/research \
-H "Content-Type: application/json" \
-d '{
"topic": "AI coding agents in 2026",
"objective": "Produce a practical research brief with opportunities, risks, and next steps.",
"max_rounds": 3
}'
# Browser: Inngest UI
# http://127.0.0.1:8288| What | URL |
|---|---|
| FastAPI health check | http://127.0.0.1:8000/ |
| Start research (POST) | http://127.0.0.1:8000/research |
| Inngest sync endpoint | http://127.0.0.1:8000/api/inngest |
| Inngest dev portal (UI) | http://127.0.0.1:8288 |
Press Ctrl+C in each terminal.
If a port is stuck:
fuser -k 8000/tcp 2>/dev/null
fuser -k 8288/tcp 2>/dev/nullThe Inngest dev portal is the visual control panel for your local workflow. Open it in a browser:
This only works while Terminal 2 (inngest dev) is running.
| Section | What it shows |
|---|---|
| Apps | Your synced app (python-autoresearch) pulled from http://localhost:8000/api/inngest |
| Functions | Registered functions, including research-loop |
| Events | Incoming events such as research/requested |
| Runs | One entry per research job triggered by POST /research |
- Start both terminals (API + Inngest dev server).
- Open http://127.0.0.1:8288 in your browser.
- Confirm the app appears under Apps and
research-loopappears under Functions. - Trigger a run from the terminal:
curl -X POST http://localhost:8000/research \
-H "Content-Type: application/json" \
-d '{"topic": "Your topic here", "max_rounds": 3}'- Go to Runs in the Inngest portal. A new run should appear within a few seconds.
- Click the run to open the detail view.
Each research job executes as a sequence of Inngest steps. In the run detail view you will see steps appear and turn green as they complete:
| Step name | What happened |
|---|---|
draft-1 |
ResearchAgent wrote the first markdown draft via OpenRouter |
evaluate-1 |
EvaluatorAgent scored the draft and returned JSON feedback |
pause-1 |
1 second sleep between rounds |
draft-2 |
Second draft, using feedback from round 1 |
evaluate-2 |
Second evaluation |
save-report |
Final markdown written to reports/ |
save-memory |
Experiment record appended to memory/experiments.json |
If the evaluator score reaches 8 or higher, the loop may stop early (fewer steps than max_rounds).
- Step input/output: click any step to see what was passed in and what was returned.
- Final output: at the bottom of a completed run, see JSON with
best_score,report_path,memory_path, anditerations. - Failed steps: red steps show the error. Common causes: invalid OpenRouter key, model timeout, or non-serializable step output.
- Retries: Inngest automatically retries failed steps. You can see retry attempts in the run timeline.
When a run completes successfully:
- Copy
report_pathfrom the run output in the portal. - Open that file locally, for example:
cat reports/20260605-043453--ai-coding-agents-in-2026.md- Check structured history:
cat memory/experiments.json- List all generated reports:
ls -lt reports/| Signal | Where to look |
|---|---|
| App is up | curl http://127.0.0.1:8000/ returns {"status":"ok"} |
| Inngest is connected | Terminal 1 shows PUT /api/inngest" 200 OK |
| Event was queued | curl response has "queued": true and an event_ids value |
| Workflow is running | Inngest portal Runs tab shows an active run with steps updating |
| Workflow finished | Run status is Completed; reports/ has a new .md file |
All commands assume you are inside the python-autoresearch/ directory. If you followed the Quick Start Runbook above, you can skip this section.
git clone <your-repo-url>
cd python-autoresearchpython -m venv .venv
source .venv/bin/activateOn Windows (PowerShell):
python -m venv .venv
.venv\Scripts\Activate.ps1pip install -r requirements.txtInstalled packages:
fastapi
uvicorn[standard]
inngest
python-dotenv
openai
pydantic
Copy the example file and edit it:
cp .env.example .envSet your OpenRouter key in .env:
OPENROUTER_API_KEY=sk-or-v1-your-actual-key-here
OPENROUTER_MODEL=openai/gpt-4o
OPENROUTER_HTTP_REFERER=http://localhost:8000
OPENROUTER_APP_TITLE=python-autoresearch
INNGEST_DEV=1| Variable | Required | Description |
|---|---|---|
OPENROUTER_API_KEY |
Yes | Your OpenRouter API key |
OPENROUTER_MODEL |
No | Model id (default: openai/gpt-4o) |
OPENROUTER_HTTP_REFERER |
No | Referer header for OpenRouter rankings |
OPENROUTER_APP_TITLE |
No | App title header for OpenRouter rankings |
INNGEST_DEV |
Yes (local) | Enables Inngest dev mode |
INNGEST_BASE_URL |
Docker only | URL of the Inngest dev server (e.g. http://inngest:8288 in Compose) |
MAX_ROUNDS |
No | Default max rounds if not sent in the request |
The app will fail at startup if OPENROUTER_API_KEY is missing or empty.
These are created automatically on first run, but you can create them upfront:
mkdir -p reports memory
echo "[]" > memory/experiments.jsonThe recommended approach is npx (no install needed). If you want a permanent CLI:
npm install -g inngest-cli
inngest dev -u http://localhost:8000/api/inngestIf global install fails due to permissions, use npx or install locally:
npm init -y
npm install inngest-cli
npx inngest dev -u http://localhost:8000/api/inngestYou need two terminals running at the same time. See the Quick Start Runbook above for the short version.
Always use the Uvicorn binary from your virtual environment, not a system-wide install:
cd python-autoresearch
source .venv/bin/activate
INNGEST_DEV=1 .venv/bin/uvicorn main:app --reloadThe API will be available at:
http://127.0.0.1:8000
Verify the health endpoint:
curl http://127.0.0.1:8000/Expected response:
{"status":"ok"}If you do not have the Inngest CLI installed globally, use npx:
npx --yes inngest-cli@latest dev -u http://localhost:8000/api/inngestOr, if installed globally:
inngest dev -u http://localhost:8000/api/inngestThe Inngest dev UI will be available at:
http://127.0.0.1:8288
Use the UI to inspect function runs, step outputs, retries, and errors for the research-loop function. See Using the Inngest Dev Portal for a full visual walkthrough.
Docker lets you run the full stack (API + Inngest dev server) with one command instead of two terminals.
Run all commands from the python-autoresearch/ directory.
Recommended — full stack (API + Inngest):
# Build images and start both services (foreground)
docker compose up --build
# Start in background
docker compose up --build -d
# Stop and remove containers
docker compose down
# View logs
docker compose logs -f
docker compose logs -f api
docker compose logs -f inngest
# Check status
docker compose psBuild and run API image only:
# Build the image
docker build -t python-autoresearch .
# Run the API container
docker run --rm -p 8000:8000 \
--env-file .env \
-e INNGEST_DEV=1 \
-e INNGEST_BASE_URL=http://host.docker.internal:8288 \
-v "$(pwd)/reports:/app/reports" \
-v "$(pwd)/memory:/app/memory" \
python-autoresearch
# Stop a running container (if not using --rm)
docker stop <container_id>Useful one-offs:
# List running containers
docker ps
# List all images
docker images
# Remove the built image
docker rmi python-autoresearch
# Rebuild without cache (after Dockerfile changes)
docker compose build --no-cache
docker build --no-cache -t python-autoresearch .Quick health checks:
curl http://localhost:8000/
curl -X POST http://localhost:8000/research \
-H "Content-Type: application/json" \
-d '{"topic":"AI agents","objective":"Brief overview","max_rounds":1}'| Command | What it does |
|---|---|
docker compose up --build |
Build + start API and Inngest together |
docker compose up -d |
Start in background |
docker compose down |
Stop everything |
docker compose logs -f |
Stream logs from all services |
docker build -t python-autoresearch . |
Build only the API image |
docker run ... python-autoresearch |
Run API container alone |
docker ps |
Show running containers |
docker images |
Show built images |
Think of it like a recipe vs a meal.
| Image | Container | |
|---|---|---|
| What it is | A frozen snapshot / blueprint | A running (or stopped) instance of that blueprint |
| Analogy | Recipe or app installer | The actual running app |
| In this project | python-autoresearch image built from Dockerfile |
The live API process when you docker run or docker compose up |
- Build once → you get an image (
docker build -t python-autoresearch .) - Run it → you get a container (
docker run python-autoresearch)
You can run many containers from the same image, just like many people can cook the same recipe.
| Dockerfile | docker-compose.yml | |
|---|---|---|
| Purpose | Defines how to build one image | Defines how to run one or more containers together |
| Analogy | Recipe for one dish | Full dinner plan (main + sides + drinks) |
| In this project | Installs Python, deps, copies code, starts Uvicorn | Runs api + inngest, wires ports, env vars, volumes |
Dockerfile answers: "What goes inside the API image?"
FROM python:3.12-slim
COPY requirements.txt .
RUN pip install -r requirements.txt
CMD ["uvicorn", "main:app", "--host", "0.0.0.0", "--port", "8000"]docker-compose.yml answers: "How do I run the whole stack?"
- Start the API container on port 8000
- Start the Inngest dev server on port 8288
- Connect them (
INNGEST_BASE_URL=http://inngest:8288) - Mount
reports/andmemory/so files persist on your machine
One-line summary:
- Dockerfile → builds an image
- Image → becomes a container when you run it
- docker-compose.yml → runs multiple containers as one setup with one command (
docker compose up)
Ensure .env exists with your OpenRouter key:
cd python-autoresearch
cp .env.example .env # if needed, then edit
docker compose up --build| Service | URL |
|---|---|
| API health | http://localhost:8000/ |
| Inngest dev UI | http://localhost:8288 |
Stop the stack:
docker compose downIf you only need the API container (you must run the Inngest dev server separately):
docker build -t python-autoresearch .
docker run --rm -p 8000:8000 \
-e OPENROUTER_API_KEY=your_key \
-e INNGEST_DEV=1 \
-e INNGEST_BASE_URL=http://host.docker.internal:8288 \
-v "$(pwd)/reports:/app/reports" \
-v "$(pwd)/memory:/app/memory" \
python-autoresearchWithout INNGEST_BASE_URL, the API tries to reach Inngest at 127.0.0.1:8288 inside the container and registration fails. Use host.docker.internal when the Inngest dev server runs on your host machine.
Send a POST request to start the workflow:
curl -X POST http://localhost:8000/research \
-H "Content-Type: application/json" \
-d '{
"topic": "AI coding agents in 2026",
"objective": "Produce a practical research brief with opportunities, risks, and next steps.",
"max_rounds": 3
}'Example response:
{
"queued": true,
"event_ids": ["01KTB0XVMSGWJP3G2FNGAR4R5Y"]
}The HTTP response only confirms the event was queued. The actual research loop runs asynchronously inside Inngest. Open http://127.0.0.1:8288 to watch it execute.
| Result | Location | How to view |
|---|---|---|
| Live workflow progress | Inngest portal | http://127.0.0.1:8288 → Runs → click a run |
| Final markdown report | reports/ directory |
ls -lt reports/ then cat reports/<filename>.md |
| Experiment history | memory/experiments.json |
cat memory/experiments.json |
| API server logs | Terminal 1 | Step requests like POST /api/inngest?fnId=... |
| Inngest server logs | Terminal 2 | JSON log lines from the dev server |
Example: read the most recent report:
ls -lt reports/ | head -5
cat reports/$(ls -t reports/ | head -1)Health check.
Response:
{"status": "ok"}Queue a new autonomous research run.
Request body:
{
"topic": "string (required, min 3 chars)",
"objective": "string (optional)",
"max_rounds": 3
}Response:
{
"queued": true,
"event_ids": ["..."]
}Inngest sync endpoint. Used by the Inngest Dev Server and production Inngest Cloud. Do not call this manually under normal use.
Registered in app/workflows/research_workflow.py:
| Property | Value |
|---|---|
| Function id | research-loop |
| Trigger event | research/requested |
| Steps per round | draft-{n}, evaluate-{n}, pause-{n} |
| Final steps | save-report, save-memory |
Inngest step outputs must be JSON-serializable. That is why save_report and append_experiment return string paths rather than pathlib.Path objects.
Set OPENROUTER_MODEL in .env to any model listed on OpenRouter, for example:
OPENROUTER_MODEL=anthropic/claude-3.5-sonnet
OPENROUTER_MODEL=openai/gpt-4o-mini
OPENROUTER_MODEL=google/gemini-2.0-flash-001Either pass max_rounds in the API request, or set a global default:
MAX_ROUNDS=5The per-request value takes precedence over the environment default inside the workflow.
| Problem | Cause | Fix |
|---|---|---|
inngest: command not found |
CLI not installed globally | Use npx --yes inngest-cli@latest dev -u http://localhost:8000/api/inngest |
No module named 'inngest' |
System uvicorn used instead of venv | Activate .venv and run .venv/bin/uvicorn main:app --reload |
requirements.txt not found |
Wrong working directory | cd python-autoresearch before running commands |
| Event queued but nothing runs | Inngest dev server not started | Start Terminal 2 and confirm portal at http://127.0.0.1:8288 loads |
| Portal shows no app | API server not running or wrong URL | Start Terminal 1; confirm PUT /api/inngest" 200 OK in logs |
OPENROUTER_API_KEY is missing |
Empty or missing .env |
Copy .env.example to .env, add key, restart Terminal 1 |
| Run is slow | Two LLM calls per round | Normal; check current step in Inngest portal Runs tab |
| Port already in use | Previous session still running | Ctrl+C old terminals or fuser -k 8000/tcp and fuser -k 8288/tcp |
You are not using the virtual environment Uvicorn. Activate .venv and run .venv/bin/uvicorn instead of a system uvicorn.
Ensure .env exists in the project root and contains a non-empty OPENROUTER_API_KEY. Restart the server after editing .env.
Confirm the Inngest Dev Server is running and pointed at http://localhost:8000/api/inngest. Check the Inngest UI at http://127.0.0.1:8288 for connection status.
Verify your API key is valid and has credits. If a key was exposed, revoke it in the OpenRouter dashboard and create a new one.
Step functions must return JSON-safe values (strings, numbers, lists, dicts). Path objects are not allowed. This project returns string paths from report_store.py to satisfy that requirement.
Each round makes two LLM calls (draft + evaluate). Higher max_rounds values and slower models increase total runtime. Use the Inngest UI to see which step is currently executing.
- Never commit
.envor API keys to git. The repository.gitignoreexcludes.env. - Treat OpenRouter keys like production credentials. Rotate immediately if leaked.
- This local setup has no authentication on
POST /research. Add auth before exposing the API publicly. - Generated reports may contain model hallucinations. Review output before using it for decisions.
This README focuses on local development. For production deployment:
- Remove or override
INNGEST_DEV=1and connect to Inngest Cloud. - Deploy the FastAPI app to a host reachable by Inngest (Railway, Fly.io, AWS, etc.).
- Register the production
/api/inngestURL in your Inngest app settings. - Store secrets in your platform's secret manager, not in plain
.envfiles. - Add rate limiting and authentication to
POST /research.
Add your license here before publishing to GitHub.