# alexop.dev - Full Content > A Personal Blog from a simple web developer This file contains the full content of all posts for deep context. For a sitemap overview, see: https://alexop.dev/llms.txt ================================================================================ # BLOG POSTS ================================================================================ --- title: Clean Code Is Sexy Again: Making Your Vue Project AI-Ready description: My Vue MAD 2026 talk, written up. What an agent actually is, and the three things every Vue project needs to get more out of AI: context, feedback loops, and discoverability. The twist is that none of it is new. tags: ['vue', 'ai', 'claude-code', 'architecture', 'testing'] url: https://alexop.dev/posts/clean-code-is-sexy-again-vue-ai-ready --- # Clean Code Is Sexy Again: Making Your Vue Project AI-Ready This is the written-up version of the talk I gave at Vue MAD 2026 in Madrid: _Clean Code Is Sexy Again. Making Your Vue Project AI-Ready_. If you would rather watch it, [here is the original recording on YouTube](https://www.youtube.com/watch?v=9bKMqvFRvvI). Otherwise, read on. ## TLDR - My main take: **what is good for developers is also good for agents**. There was never a separate "AI-ready" checklist. There was just good engineering, and now it pays off twice. - "You don't have to write code anymore" is half true. It only works if you know your stack _and_ your codebase is built for an agent to work in. - An agent is not magic. It is a loop: read context, pick a tool, run it, read the result, repeat. Once you have built one you stop asking "how does the magic work" and start asking "why does the same loop fly in one repo and fall apart in another." - Three things every Vue project should invest in: **context** (AGENTS.md, skills, hooks, a `brain/`), **feedback loops** (types, lint, tests, a real browser), and **discoverability** (vertical feature slices). - When the agent gets something wrong, fix the factory, not the PR. Add the lint rule, update AGENTS.md, tighten the prompt. The PR fix is one bug. The factory fix prevents the next hundred. ## The line everyone keeps repeating Everyone keeps saying the same thing: _you don't have to write code anymore._ I think that's half true. In some of my repos AI 10x'd me. In others it produced complete garbage. Same person, same agent, same model, different codebases. It only works if two things are true. First, you have real experience with the stack. I would not let an agent write Rust for me, because I have no idea what good Rust looks like and I could not review it. Second, the codebase has to be shaped so an agent can work in it. A greenfield project is easy. A brownfield project with no tests and 10,000-line components is a mess that AI makes worse, not better, until you clean it up. A good frame for "where are we" comes from Dan Shapiro's post on the levels of AI-assisted development. It reads like driving automation. Level 0 is spicy autocomplete, the first Copilot. Then agents arrive and it gets a lot better. At the top is the software factory where the agent defines, ships, and fixes on its own. I put myself at level three. The agent writes most of the code, and I spend my time reviewing and reading every line. Sometimes I feel like the bottleneck. I would not recommend level four or five for anything serious yet, but the pace is real. Two weeks before the talk, the creator of Bun started using Claude Code to rewrite Bun from Zig to Rust. A million lines, in days, almost entirely by an agent. Triumph or disaster, I don't know. What I know is this is the world we ship into. ## What an agent actually is Most Vue devs treat agents as a magic box. Magic is the wrong mental model. The best tip I can give you to get better at AI is to build your own coding agent once. Then it stops being magic, and you get a feel for why it does the right thing sometimes and the wrong thing other times. I wrote up the full build in Building Your Own Coding Agent From Scratch, but here is the whole picture in one image. The agent is bounded by three things: its context window, the tools you give it, and its ability to verify what it did. That is it. The analogy that unlocks everything is treating the agent like a new engineer joining your team. You would not throw a new hire into your worst legacy module and expect a feature in a week. Same with an agent. Strip the magic and a **tool is just a function**. Three fields: a description, a schema for the arguments, and the function that runs. ```ts type Tool = { description: string schema: Record execute: (args: Record) => Promise } const TOOLS = new Map([ ['read', { description: 'Read file with line numbers (not a directory)', schema: { path: 'string', offset: 'number?', limit: 'number?' }, execute: read, }], ['edit', { description: 'Replace old with new in file (old must be unique)', schema: { path: 'string', old: 'string', new: 'string' }, execute: edit, }], ['bash', { description: 'Run shell command', schema: { cmd: 'string' }, execute: bash, }], ]) ``` Here is the part that demystifies everything: **the model only ever sees the description and the schema.** The `execute` function never leaves your machine. The model cannot run `read()`, it cannot even see it. So how does it "use" a tool? It reads the description and emits a request: "call read with path src/App.vue." Your loop runs the real function and feeds the result back. Which means the description _is_ the prompt. A vague description is a tool the model misuses. Tool descriptions are engineering, not docs. And the **loop is just recursion**. ```ts async function agentLoop(messages, systemPrompt, tools = TOOLS) { const response = await callApi(messages, systemPrompt, tools) const toolResults = await processToolCalls(response.content) const newMessages = [ ...messages, { role: 'assistant', content: response.content }, ] // No tool calls → the agent is done if (toolResults.length === 0) return newMessages // Tool calls → loop with results as the next user turn return agentLoop( [...newMessages, { role: 'user', content: toolResults }], systemPrompt, tools, ) } ``` Call the API. The model replies with text and maybe some tool requests. Run each one, append the assistant reply and the tool results, and loop. When the model returns no tool calls, it is done. And `messages` is not a database or a session store. It is an array of `{ role, content }`. That array _is_ the agent's memory. Every turn you append to it. I built a tiny version of this a few months ago and called it **nanocode**, around 350 lines of TypeScript. It is not perfect, but building it is what made the whole thing click. Once you see an agent this way, the question stops being "how does the magic work" and becomes "why does the same simple loop work brilliantly in one repo and fall apart in another?" That is the rest of this post. ## Clean code isn't nice-to-have anymore I have been in plenty of projects where people told me "we don't have time to write tests." That argument is over. You do not need new patterns for AI. The patterns you already fight for in code review, the ones the senior dev keeps insisting on, are exactly the patterns that make agents work. A codebase that is hard for humans is hard for agents. Sprawl, hidden coupling, magical state: humans hate it, agents fail at it. In Vue, three things are worth investing in: **context**, **feedback loops**, and **discoverability**. ## Part 1: Context The agent is Leonard from _Memento_. Every new chat, the context resets. No long-term memory, no yesterday. So, like Leonard, it has to tattoo the rules where it will read them every single turn. That tattoo is **AGENTS.md** (or `CLAUDE.md`). It is the first thing you can optimize. But the tattoo space is finite, and most of it is already used before you write a word. Every model has a context window, and it works best when it is _not_ full. Models degrade as the window fills: recall drops, reasoning slips, the agent starts confusing files. And the window is not empty when you start. The system prompt and the tool definitions already cost around 20k tokens before you type anything. AGENTS.md, skills, MCP servers, and sub-agents all spend from the same pool. In Claude Code you can see exactly where it goes by typing `/context`. This is why the biggest mistake with AGENTS.md is dumping everything into it: every coding rule, every bug post-mortem, every gotcha, 2000 lines that load on every single turn. A better shape is a thin doorway: ```md # AGENTS.md Run `pnpm lint:fix && pnpm typecheck` after changes. ## Stack Nuxt 4, @nuxt/content v3, @nuxt/ui v3 ## Structure - `app/` — Vue application - `content/` — Markdown files ## Further reading **IMPORTANT:** read the relevant doc below before starting any task. - `docs/nuxt-content-gotchas.md` - `docs/testing-strategy.md` ``` The agent loads `testing-strategy.md` only when it writes a test, `nuxt-content-gotchas.md` only when it touches content. That is **progressive disclosure**: the right context at the right time. Two filters before a line goes into AGENTS.md. Can a tool enforce it? Then do not write prose about it. Is it universal or situational? Situational goes in `docs/`. I go deeper on the AGENTS.md-versus-skills split in my Claude Code customization guide, and on why context fills up the way it does in reverse-engineering HumanLayer's context engineering. ### brainmaxxing Forget looksmaxxing. We are brainmaxxing: maxing the one thing the agent has, its context. [brainmaxxing](https://github.com/poteto/brainmaxxing) is an open-source kit from a Cursor developer. It is a framework for giving an agent a documented memory, and it has three pieces. First, a `brain/` folder. Plain markdown. The agent reads from it and writes back to it. There is one `index.md` with wiki-links describing the documentation that exists, and the rest is markdown files for codebase notes, plans, and principles. People found out you do not need RAG or anything complex. The simplest approach, wiki-links and markdown files, works best, and now the agent has a memory. (If you want the theory of why files beat a vector database here, I wrote it up in the four types of memory for coding agents.) Second, **skills**. A skill is like a prompt that lives in a markdown file with a name and a description. When you start Claude Code, only the name and description enter the context. When your prompt matches, the body loads. When the body calls a script, the script loads. Three levels of lazy loading, so skills scale. brainmaxxing's `/reflect` skill is a good example: at the end of a session it reviews the conversation and persists what mattered back into `brain/`. So when the agent uses the wrong command and you correct it, you just say "remember this," and it updates its own memory. Third, **hooks**. Vue has lifecycle hooks like `onMounted`. Claude Code has lifecycle hooks too. brainmaxxing wires a `SessionStart` hook that cats `index.md` into every new session, so the agent boots up already knowing the map. There are many more events. `PreToolUse` can block a destructive command or forbid reading your `.env`. `PostToolUse` can re-lint after an edit. `SessionEnd` could even send you a notification. I cover skills and hooks hands-on in my Vue workout tracker walkthrough. brainmaxxing also ships **principles**, around 16 markdown files describing how you want code written. One is _boundary discipline_: business logic lives in pure functions, the shell is thin and mechanical. The agent reads them and develops a better taste for your code. You can write your own too. Claude Code itself now ships something similar, called auto memory: it writes markdown notes about your project automatically. The catch is those files only live on your machine. That is exactly why I like keeping `brain/` in Git. Check it in and the whole team's agent boots up with the same memory on day one. The new hire on Monday starts where the team is, not from zero. ### Want VueUse-style code? Give the agent VueUse One more trick, and it is bigger than memory. You want the agent to use VueUse or Nuxt UI, but its training data is out of date and it does not know the current API. So give it the source. Clone the library into your repo as a git subtree: ```bash git subtree add --prefix=repos/vueuse \ https://github.com/vueuse/vueuse main --squash ``` Then point AGENTS.md at it: ```md ## Reference repositories - `repos/vueuse/` — when writing new composables, mirror the patterns in `repos/vueuse/packages/core/use*/`. ``` Now the agent's pattern matching is anchored on real source instead of stale docs. Agents are post-trained on reading code, not prose. A subtree beats a submodule (no clone-time pain, just files) and beats pointing at `node_modules` (compiled and flattened, the structure is gone). One refinement: tell your editor to ignore `repos/` so you do not accidentally auto-import VueUse internals, and have the agent distill what it learned into a short `agent-patterns/vueuse.md` so it does not re-explore 800 composables every session. This sounds stupid, but once you try it, it works far better than MCPs in my experience. The pattern comes from the Effect team's post, [The One Weird Git Trick That Makes Coding Agents More Effect-ive](https://effect.website/blog/the-one-weird-git-trick-that-makes-coding-agents-more-effect-ive/). ## Part 2: Feedback loops If an agent does not know when it broke something, it ships whatever compiles. So you give it deterministic ways to check its own work. **Type safety** is the first lie-detector. A strict `tsconfig` is non-negotiable, and `noUncheckedIndexedAccess` and `exactOptionalPropertyTypes` catch what default strict mode lets through. Types are a compile-time fiction, though, so parse with Zod or Valibot at every untyped boundary: fetch responses, route params, env vars, form input. **Lint and format** with Oxlint and Oxfmt. They are roughly 50x and 30x faster than ESLint and Prettier, fast enough to run on every save instead of batching at the end. Two Vue lint rules in particular keep components agent-friendly, because nobody, human or agent, can read a 3000-line component: ```ts 'vue/max-template-depth': ['error', { maxDepth: 8 }] 'vue/max-props': ['error', { maxProps: 6 }] ``` A 12-prop component is a god object pretending to be one. I wrote up my full set in my opinionated ESLint setup for Vue projects, including custom local rules. Then **tests**. Vitest for units, and the big 2026 win, Vitest browser mode, which runs component tests in a real Chromium via Playwright instead of jsdom. Hover, focus, layout, scroll, all behave like production. The agent can finally trust that "passes in tests" means "works in a browser." I go deep on this in the Vue testing pyramid with Vitest browser mode. Those are four layers. There are 11 more: API mocking, contract tests, E2E, accessibility, visual regression, performance, dead-code detection, and so on. Every one is a signal the agent can chase to green on its own. The cheap ones at the center run on every save, the expensive ones at the edge run in CI. What I now do in a new project is hand the agent my own write-up and say "set up the layers that make sense here." The full breakdown lives in the modern frontend quality pipeline. ### The agent as a user Static checks can be green and the feature can still be broken. Did the modal open? Did the chart render? You need the agent to _be_ the user. Vercel's [agent-browser](https://github.com/vercel-labs/agent-browser) is a browser CLI shaped for agents. One install and the agent has a real Chromium it can drive from the command line. ```bash npm i -g agent-browser agent-browser install agent-browser open localhost:5173 agent-browser snapshot -i # DOM tree with @refs agent-browser click @e2 # click by ref agent-browser screenshot --annotate agent-browser console # read page console ``` The snapshot returns the DOM as plain text with refs, the agent clicks by ref, screenshots come back annotated, console errors come back as text. So when there is a bug, you say "reproduce it with agent-browser," and once the agent can reproduce something, fixing it is easy. The same thing a good developer would do. For this to work the app has to be agent-runnable: the dev server is one command, the port is stable, there is a way to know the server is ready, and test data is seeded so the agent does not hit a login wall. Then "types pass" is no longer the bar. "Works in a browser" is. I built a full version of this into CI in automated QA with Claude Code, agent-browser, and GitHub Actions. Finally, put a **gate** at commit time. Lefthook (or Husky) runs lint, typecheck, and the related tests on every commit, so bad code does not reach `main`. I prefer Lefthook because it runs jobs in parallel and the config is YAML the agent can read and edit, not a shell-script soup. The one rule: no `--no-verify` escape hatch. The whole point of the gate is that it does not open. ```yaml # lefthook.yml pre-commit: parallel: true jobs: - run: pnpm oxlint - run: pnpm vue-tsc --build - run: pnpm vitest related --run {staged_files} ``` ## Part 3: Discoverability Most Vue apps start flat. You run `create vue` and you get `components/`, `composables/`, `stores/`, `views/`, `router/`. Everything is grouped by what a file _is_. That is fine for small apps, but in a real application that `components/` folder grows to 80+ files, and a single "checkout" change is scattered across three folders. The agent has the same problem you do: it greps across the whole tree and most of what it pulls back is noise. Feature slicing groups by what files _do_, not by what they are. First you find the domains. Imagine a workout tracker: the domains are workout, timers, exercises, and settings. Then each domain owns its own components, composables, and store: ```text src/features/ ├── workout/ │ ├── components/ │ ├── composables/ │ └── store.ts ├── timers/ ├── exercises/ └── settings/ ``` Finding the domains is the hard part on a big codebase, but once you have them, "change the settings page" means opening one folder. Same files, same code, different addressing. And the win for the agent is concrete. Ask it to "add a streak counter to the active workout." In a flat layout it greps `components/` and `composables/` and pulls back `SettingsForm`, `TimerDisplay`, `useSettings`, maybe 40% relevant. In a feature layout it lists `features/workout/` and gets exactly the files that matter, 100% relevant. No grep, no guess, tokens go to output instead of search. Folders alone do not keep the agent honest, so add three import rules. Arrows only point down the layers. Sibling features never import each other; if they share something, it moves to a shared layer like `components/` or `lib/`. And a page that uses two features composes them at the page level rather than letting one feature reach into another. The most important rule is the second one: features stay decoupled. As a bonus, a codebase split this way is already shaped for a micro-frontend approach if you ever need it. ## Where this is heading AFK coding (away from keyboard) is already here. You write a spec, kick off an agent, and walk away. It runs the tests, hits a failure, fixes itself, runs again, and opens a draft PR with a summary. You review. The work is no longer "me typing." It is "me deciding what should exist, and reviewing what came back." The loop in practice has humans at the edges and agents in the middle. You align with the business on the spec, the agent breaks the big ticket into vertical sub-tickets, one agent works each slice with TDD inside, a dedicated refactor pass runs (the step LLMs always skip), a QA agent drives the real browser, and then you read the PR. I wrote the long version in how to do AFK coding. The most important thing in this loop is what you do when it goes wrong. The agent ships a bug, and the instinct is to fix the bug, merge, and move on. That instinct is wrong. **Fix the factory, not the PR.** A bug is not a bug, it is a factory defect. Add an ESLint rule that catches that whole class of mistake. Update AGENTS.md so the convention is written down. Tighten the slash command or the prompt if that is where it leaked. The PR fix is one bug. The factory fix prevents the next hundred. And if the codebase is messy in the first place, use the agent to refactor it before you expect any of this to work. Every PR review teaches the factory, and the codebase gets smarter over time. As a test of all this, the week before the talk I tried to port [React Ink](https://github.com/vadimdemedes/ink) (declarative components for terminal UIs) to Vue, using AI only. It worked surprisingly well. The approach combined everything above. I vendored Ink, Vue core, and VueUse into `repos/` as read-only source. I used a `brain/` vault for gotchas and an api-tracker. And because React Ink has a lot of tests, I ported the _tests_ first: each Ava scenario became a Vitest test at the behavior level, not the React implementation. Run it, red, no implementation yet. Then translate the implementation, JSX to SFC, hooks to composables. Run it, green. Then reflect the learnings back into `brain/` so the loop sharpens itself. I would not publish it as a real library without going back and understanding every line first, but as a proof that you can port a well-tested React library to Vue mostly AFK, it holds up. I even added a `Stop` hook that fires when a turn ends and spawns a second, headless Claude in the background to read the session transcript and write only new durable learnings into `brain/`. The one gotcha is that the child also ends a turn, which would fire the hook forever, so an env flag marks the child and the hook bails when it sees it. The project updates its own docs while I get coffee. ## The twist This talk was not really about AI. I tricked you. None of it is new. Testing, TDD, feature-based architecture, strict types, small components: these are the things senior engineers have fought for in code review for twenty years. They were good before a single agent existed. The only thing that changed is the payoff. The same discipline that used to make a codebase pleasant for the next human now also makes it tractable for an agent, so every hour you spent caring about architecture quietly started earning twice. That is the whole bet. Tight **context**, plenty of **feedback loops**, real **discoverability**. Which of them is your project missing? Come find me and tell me what your AGENTS.md looks like. --- --- title: How VS Code Copilot Chat Compacts Your Conversation description: A reference on conversation compaction in VS Code Copilot's agent (now in microsoft/vscode under extensions/copilot): when it triggers, the budget math, what survives, the 8-section summary format, and the config keys. Traced from the source, with interactive diagrams. tags: ['ai-agents', 'copilot', 'context-engineering', 'vscode'] url: https://alexop.dev/posts/how-vscode-copilot-chat-conversation-compaction-works --- # How VS Code Copilot Chat Compacts Your Conversation LLMs have limited context windows. When an agent session grows too long, VS Code Copilot Chat uses **compaction** to summarize older history while keeping recent work verbatim. I traced exactly how it works from the source. The standalone `microsoft/vscode-copilot-chat` repo was **archived in May 2026**. The agent code now lives in the main [microsoft/vscode](https://github.com/microsoft/vscode) repo under `extensions/copilot/`, and it has changed since the standalone repo froze. Everything below is traced against the current `extensions/copilot` source. **Source files** (all under `extensions/copilot/`): - `src/extension/intents/node/agentIntent.ts` — budget math, trigger logic - `src/extension/prompts/node/agent/summarizedConversationHistory.tsx` — prompt, LLM call, history selection, re-insertion - `src/extension/prompts/node/agent/backgroundSummarizer.ts` — the async state machine and thresholds - `src/extension/prompts/node/agent/simpleSummarizedHistoryPrompt.tsx` — the Simple-mode fallback Microsoft moves fast. Treat line numbers as approximate and the thresholds as true the day I looked. ## First, what is `@vscode/prompt-tsx`? Compaction makes no sense until you understand the renderer underneath it. [`@vscode/prompt-tsx`](https://github.com/microsoft/vscode-prompt-tsx) (the repo pins `^0.4.0-alpha.8`) is a **budget-aware prompt renderer**. Instead of concatenating strings into a `messages` array, you author the entire prompt as a tree of TSX components — a `PromptElement` — and the renderer flattens it into the `ChatMessage[]` that goes to the model, *fitting it to the token budget for you*. Here's the actual agent prompt, lightly trimmed (`agentPrompt.tsx`): ```tsx return <> {baseInstructions} ; ``` Two numbers on each element do the work: - **`priority`** — when the rendered tree is over budget, the renderer drops the **lowest-priority** content first. The current user message (`900`) and the latest tool calls (`899`) outrank older history (`700`), which outranks boilerplate system instructions. Same-priority siblings are pruned in declaration order. - **`flexGrow`** — controls how *leftover* budget is distributed. A `flexGrow` element renders **after** its siblings and receives whatever budget they didn't use. So the user message and tool calls (`flexGrow={2}`) get first claim on the window; conversation history (`flexGrow={1}`) fills the remainder. Every element's `render(state, sizing)` is handed a `PromptSizing` with `tokenBudget` and an async `countTokens()`, so an element can measure itself and trim to fit. That's literally how tool results cap themselves at 50% of the window (`truncateAt={maxToolResultLength}` above), and there are helpers for the common cases: ```tsx {/* hard cap on a subtree */} {rounds} ``` `TokenLimit` caps a subtree; `PrioritizedList` hands a list of children descending (or ascending) priorities — exactly how conversation rounds are ranked so the oldest get pruned first; `TextChunk` keeps as much of a long string as fits. Pruning is graceful — until it isn't. When even the highest-priority content won't fit, prompt-tsx throws a **`BudgetExceededError`**. That exception is the signal the agent loop catches to escalate from "just prune" to "compact the history" — which is where the rest of this post begins. ## Overview Compaction is the escalation when prompt-tsx's pruning isn't enough. There are two automatic triggers — **both on by default now** — plus the user-initiated `/compact` command: | Path | Trigger | Default? | | --- | --- | :---: | | **Foreground summarize** | `BudgetExceededError` while rendering (hard overflow) | ✅ on | | **Background compaction** | post-render context ≥ ~80% (jittered `0.78–0.82` with a warm cache; ≥`0.90` cold-cache emergency) | ✅ on | | **Manual `/compact`** | user runs the command | — | This is the biggest change since the standalone repo. The old build gated background compaction behind a `chat.backgroundCompaction` experiment and had a separate `≥85%` "proactive inline" experiment. **Both flags are gone.** Background compaction now ships on by default (gated only by the master summarization switch), and the inline mechanism it used became *how* background compaction works rather than a separate path. ## When it triggers The budget is computed from config and the effective context size, then trimmed for tools: ```typescript const baseBudget = Math.min( summarizeThresholdTokens ?? effectiveMaxTokens, effectiveMaxTokens // clamp: never above the real max ); // 10% safety margin on the message portion when tools are present const messageBudget = Math.max(1, Math.floor((baseBudget - toolTokens) * 0.9)); ``` Two subtleties: `effectiveMaxTokens` honors the user's Context Size picker (not just the model's raw `modelMaxPromptTokens`), and the threshold config is interpreted as either a **ratio** (`0–1`, a fraction of the window) or an **absolute token count** (`≥100`) — the ambiguous gap in between throws. Drag the fill level to see which path fires: **Foreground** is the safety net — it catches the hard overflow and recovers: ```typescript } catch (e) { if (e instanceof BudgetExceededError && summarizationEnabled) { result = await renderWithSummarization(`budget exceeded(${e.message})`); } } ``` **Background compaction** exists to avoid the cost of "render, fail, re-render." After each turn renders, it checks the post-render context ratio and, if you're over the threshold, kicks off summarization *in the background* so the compacted result is ready to apply on the next render. Two details make it cache-friendly: - **Cache-warmth gating + jitter.** It only fires at the normal ~`0.80` threshold when the prompt cache is warm (a completed tool-call round this turn), and it *jitters* the exact trigger across `0.78–0.82` so it doesn't always fire at the same boundary. With a cold cache it waits for the `≥0.90` emergency line. (Thresholds live in `BackgroundSummarizationThresholds`.) - **Apply-min ratio.** A finished background summary is discarded if the context ratio has since dropped below `0.65` — e.g. you switched to a larger-context model and no longer need it. Mechanically, background compaction doesn't make a separate "summarize this" request. It folds a "compact now" instruction into a forked copy of the *same* render (for prompt-cache parity) and parses a `` block back out of the model's reply. ## How it works 1. **Pick the cut point.** Render history in reverse (newest first), keeping recent rounds verbatim. 2. **Exclude the overflowing round.** With multiple tool rounds, drop the last one — it's what pushed over the limit. 3. **Stop at the previous summary.** Walking back, break at the first round that already has a `.summary`. Compaction *compounds* instead of re-summarizing from scratch. 4. **Generate the summary.** Call the LLM with the structured format below. 5. **Re-insert.** Wrap the summary in a `` user message that replaces the older turns; store it as turn metadata so it survives the next turn. ```typescript // Excluding the round that blew the budget: if (toolCallRounds && toolCallRounds.length > 1) { toolCallRounds = toolCallRounds.slice(0, -1); // last round overflowed summarizedToolCallRoundId = toolCallRounds.at(-1)!.id; // summarize from the prior one } ``` ## What gets kept vs summarized - **Recent rounds** → kept verbatim, so the model picks up mid-task. - **The round that overflowed** → excluded from the summary. - **Everything before the previous summary** → already represented; not re-included. - **Tool results** → truncated at `maxToolResultLength`, which the agent loop sets to **50% of the model's max prompt tokens** (`modelMaxPromptTokens * 0.5`), keeping the head and tail of long outputs (a 40/60 split) and dropping the middle. (The flat `2000` you may see in the source is a *different* feature — the panel's chat-summary renderer — not in-loop compaction.) ## The summary format The summarization is a real LLM call on the **same model the conversation uses**, at `temperature: 0` (streaming is intentionally *not* disabled — there's even a regression test asserting the request doesn't force `stream: false`). The prompt doesn't ask for "a summary" — it demands an 8-section handoff document, emitted inside a `` block after a separate `` block: ```text 1. Conversation Overview — objectives, session context, intent evolution 2. Technical Foundation — core tech, frameworks, architectural patterns 3. Codebase Status — each file touched: purpose, state, key code 4. Problem Resolution — issues hit, solutions, debugging context 5. Progress Tracking — done vs. partially done vs. validated 6. Active Work State — exactly what was being worked on last 7. Recent Operations — last commands, tool results, pre-summary state 8. Continuation Plan — pending tasks and the immediate next step ``` There are two modes: **Full** (sends tool definitions with `tool_choice: 'none'`) and **Simple**. If Full fails, it falls back to Simple. ## The breadcrumb Compaction is lossy — the verbatim history is gone. So the summary carries a pointer to the full transcript on disk: ```typescript summary += `\nIf you need specific details from before compaction (such as exact code snippets, error messages, tool results, or content you previously generated), use the ${ToolName.ReadFile} tool to look up the full uncompacted conversation transcript at: "${transcriptPath}"`; // ...then appends the transcript's current line count and an example call ``` This breadcrumb is added whenever the session has an on-disk transcript path (no experiment flag — that gate existed in the old build but is gone now). It's appended exactly once, at summary-creation time, and *baked* into the frozen summary text — so later renders replay it verbatim, preserving Anthropic's prompt cache. The summary is the fast path; the transcript file is the escape hatch the model reads only when it needs an exact detail. Same instinct as progressive disclosure: cheap index in context, expensive detail on demand. ## Model-specific gotchas | Model | Handling | | --- | --- | | **Opus** (`claude-opus*`) | Extra instruction: do not call tools, only write text. | | **Anthropic + thinking** | Last thinking block preserved and re-attached as the first thinking block after the summary. | | **Anthropic + `tool_search`** | Client-side `tool_search` tool-use/result pairs stripped before the call, or Anthropic 400s. | | **Gemini** | Orphaned `function_call`s (whose results got pruned) stripped, or it 400s. | | **GPT-4.1** | A "keep going" reminder is appended after the summary. | | **Prompt caching** | Summary "baked" once so later renders don't bust Anthropic's cache. | | **PreCompact hook** | Fires before summarizing; can archive the transcript. Errors never block. | ## Settings | Config key | Default | What it does | | --- | --- | --- | | `chat.summarizeAgentConversationHistory.enabled` | on | Master switch — gates both foreground *and* background compaction. | | `chat.advanced.summarizeAgentConversationHistoryThreshold` | model max | Lower the budget that triggers compaction (ratio `0–1` or absolute tokens `≥100`). | | `chat.advanced.agentHistorySummarizationMode` | auto | Force `simple` (or `full`) summary mode. | | `chat.conversationCompaction.usePrismCompaction` | experiment | Route compaction through a separate "Prism" trajectory-compaction model. | | `chat.conversationCompaction.prismModelFilter` | (model list) | Which models the Prism path applies to; falls back to the agent endpoint otherwise. | Gone since the standalone repo: `chat.backgroundCompaction` and `chat.advanced.agentHistorySummarizationInline`. Background compaction is no longer a separate experiment — it's on by default under the master switch. ## Compared to Claude Code Same problem, different taste. Claude Code's `/compact` also summarizes history into a block — see The Four Types of Memory for AI Agents. Copilot leans harder on two ideas worth stealing if you build your own loop (like this): **prune by priority before you summarize**, and **make the lossy summary recoverable** with a breadcrumb to the raw history. --- --- title: Open Knowledge Format: I Already Do This, and Now It Has a Name description: Google Cloud just shipped the Open Knowledge Format: a directory of Markdown files with YAML frontmatter as portable knowledge for AI agents. I already run this in every repo. Here is why having an actual format is the good part. tags: ['ai-agents', 'claude-code', 'context-engineering', 'knowledge-management'] url: https://alexop.dev/posts/open-knowledge-format-markdown-frontmatter-agent-knowledge --- # Open Knowledge Format: I Already Do This, and Now It Has a Name ## TLDR - Google Cloud shipped the **Open Knowledge Format (OKF)**: a directory of Markdown files with YAML frontmatter that any agent can read and any human can edit. No SDK, no runtime, no database. - I already do something like this in every coding repo. Markdown files the agent reads before it works, frontmatter on top, linked together. OKF is that pattern with a name and a tiny spec. - The part I actually like is the **frontmatter**. One required field (`type`), a handful of optional ones, and now the files are queryable instead of just readable. - This is **semantic memory** for agents, standardized. It's the same idea as `CLAUDE.md` and `AGENTS.md`, pushed one step toward "portable between tools." - A format is worth more than another knowledge service. You can write OKF by hand today without installing anything, and that's the whole point. ## I already keep my knowledge in Markdown I don't write much code by hand anymore. I delegate to a coding agent and review the output. And the single thing that decides whether that goes well is what the agent knows about my project before it starts. So every repo I work in has the same shape. A `CLAUDE.md` at the root. A `docs/` folder with architecture notes, conventions, gotchas, the stuff that lives in the heads of the people who've been here a while. The agent reads those files, then does the work. When it learns something new, it writes it back. Plain Markdown, checked into git, sitting next to the code it describes. This works really well. It's simple, it's versioned, and I can read it myself in any editor. No vendor, no lock-in, no "open the catalog UI to see the metadata." And I've gone further than a bare `docs/` folder. On some projects I ran [brainmaxxing](https://github.com/poteto/brainmaxxing): a Markdown vault the agent reads at session start and writes back to when it learns something. It ships a little learning loop, `/reflect` after a session, `/ruminate` over old conversations, `/meditate` to prune stale notes, and because the vault is also an Obsidian vault, I can open the agent's memory and read it like my own notes. The OKF post names this exact species in passing: "Obsidian vaults wired to coding agents." It worked really well. But it was bespoke. My folder names, my conventions, my skills, none of it was built to travel to your repo or a different agent. The thing I never had was a _format_. My `docs/` folder and your `docs/` folder both look like Markdown, but they don't agree on anything. What fields go on top of a file? What does `index.md` mean? Nothing was written down. It just worked because I was the only one reading it. That's the gap [Open Knowledge Format](https://cloud.google.com/blog/products/data-analytics) is trying to fill, and after reading the spec I think it's a good move. ## What OKF actually is OKF is a directory of Markdown files. Each file is one concept: a table, a dataset, a metric, a runbook, an API, whatever you want to capture. The file path _is_ the concept's identity. ```text sales/ ├── index.md ├── datasets/ │ └── orders_db.md ├── tables/ │ ├── orders.md │ └── customers.md └── metrics/ └── weekly_active_users.md ``` Every concept file has a small block of YAML frontmatter and a Markdown body. The frontmatter holds the few fields that need to be queryable; the body holds everything else. ```markdown --- type: BigQuery Table title: Orders description: One row per completed customer order. resource: https://console.cloud.google.com/bigquery?...&t=orders tags: [sales, revenue] timestamp: 2026-05-28T14:30:00Z --- # Schema | Column | Type | Description | |---------------|--------|------------------------------------------| | `order_id` | STRING | Globally unique order identifier. | | `customer_id` | STRING | FK to [customers](/tables/customers.md). | # Joins Joined with [customers](/tables/customers.md) on `customer_id`. ``` That's it. Concepts link to each other with normal Markdown links, which turns the folder into a graph of relationships, not just a tree. Two filenames are reserved: `index.md` for an overview at each level, and `log.md` for a running history of changes. The whole v0.1 spec fits on one page. If you've used Obsidian, or written a `CLAUDE.md`, or kept an `AGENTS.md`, the shape is immediately familiar. That's deliberate. OKF isn't inventing a new thing, it's writing down the small set of conventions that make the existing thing interoperable. OKF v0.1 comes from the Google Cloud Data Cloud team, and the reference tooling is data-flavored: an agent that walks a BigQuery dataset and drafts a concept file per table, plus a static HTML visualizer. Ignore the BigQuery framing if it's not your world. The format is just Markdown and frontmatter, and nothing about it is tied to a database or a cloud. ## The frontmatter is the part I like The body of these files I already write. Markdown is Markdown. The piece OKF adds that I didn't have is the agreed-upon frontmatter. OKF requires exactly one field on every concept: `type`. Everything else is optional: `title`, `description`, `resource`, `tags`, `timestamp`. That's a good ratio. One thing you must do, a handful of things you can do, and the body is yours. Why does this matter? Because frontmatter makes the files _queryable_ without parsing prose. An agent can filter "every concept where `type: metric`" or "everything tagged `revenue`" or "what changed since this `timestamp`" without reading the bodies first. It's the same trick I use on this blog. Every post has frontmatter with `title`, `description`, `tags`, `pubDatetime`, and that's what powers the tag pages and the related-posts logic. The body is for humans; the frontmatter is for the machine. I keep coming back to one line from the OKF principles: it defines _the interoperability surface, not the content model_. It tells you the file must declare a `type`. It does not tell you what types exist or what your tables should say. That restraint is exactly right. The places I've seen "metadata standards" fail are the ones that try to model your whole domain for you. ## This is semantic memory, standardized If you read my post on the four types of memory for AI agents, OKF slots in cleanly. It's **semantic memory**: the agent's stable knowledge about your project that's true independent of any single session. Facts, conventions, schemas, join paths. Loaded in so the agent doesn't relearn your project every single time. The academic version of semantic memory always reaches for vector databases and knowledge graphs. But in practice, the thing that actually ships is Markdown files. OKF agrees, and then goes one further: it says the Markdown should carry frontmatter and link to other Markdown, so the flat pile of facts becomes a navigable graph. The two reserved filenames map onto patterns I already lean on: - **`index.md`** is progressive disclosure. The agent reads the index for a folder, sees what's there, and only opens the specific concept files it needs. That's the same move I argued for in Stop Bloating Your CLAUDE.md: don't dump everything into context, give the agent a lightweight map and let it pull the detail on demand. - **`log.md`** is a slice of **episodic memory**. A chronological record of what changed and when, sitting right next to the knowledge it describes. So OKF isn't a new idea so much as the convergence point of three things I already do: a `CLAUDE.md`-style knowledge file, progressive disclosure through an index, and frontmatter for queryability. Seeing them written down as one small spec is what makes it click. Andrej Karpathy framed the why of this whole pattern best in his LLM Wiki gist: "LLMs don't get bored, don't forget to update a cross-reference, and can touch 15 files in one pass." The bookkeeping that makes humans abandon their personal wikis is exactly the work LLMs are good at. Let the agent maintain the knowledge base; you curate it. ## A format beats another service The OKF post makes one argument I fully agree with: the answer here is a _format_, not a platform. A format is something anyone can produce without an SDK, anyone can consume without an integration, and that survives moving between tools and orgs because it's just files in version control. A platform is something you log into, that owns your data, that you integrate against. Every catalog vendor today ships the second thing: its own API, its own schema, its own export format, and your knowledge is locked behind whichever surface created it. OKF is the first thing. A bundle of OKF documents is just Markdown, just files, just YAML. Readable in any editor, renderable on GitHub, shippable as a tarball, mountable on any filesystem. The producer and the consumer are independently swappable: a human writes a bundle, an agent reads it, or an export pipeline writes it and a visualizer reads it. The file is the contract. This is why I'm bullish even though the obvious objection is right there. Yes, it's "another standard," and yes, it comes from a giant cloud vendor. But the value of a knowledge format comes from how many parties speak it, not from who wrote it down first. And the cost of trying it is basically zero, because you don't install anything. You rename a couple of conventions in a folder you probably already have. ## What I'd actually do with it I'm not going to rewrite my repos overnight, and you shouldn't either. But here's what's worth doing today: - **Add frontmatter to the docs you already keep.** Even just `type`, `title`, `description`, `tags`, `timestamp`. It costs nothing and makes the files queryable. - **Add an `index.md` per folder.** Give the agent a map before it opens individual files. This pays off immediately, format or no format. - **Keep a `log.md` for the areas that change a lot.** A running changelog the agent reads and appends to. None of that requires "adopting OKF." It's just good hygiene for a Markdown knowledge base that an agent reads. The format is what lets your folder and my folder cooperate later, if the ecosystem of producers and consumers actually shows up. That's the open question. v0.1 is a starting point, not a finished standard, and a format only earns its name if enough tools speak it. ## Wrapping up - OKF is a directory of Markdown files with YAML frontmatter, designed as portable knowledge for AI agents. No SDK, no runtime, lives in git. - I already run this pattern in every repo. The new part is having an agreed format, especially the **frontmatter**, which makes the files queryable instead of just readable. - It's **semantic memory** for agents, with `index.md` for progressive disclosure and `log.md` for history. The same moves I use in my `CLAUDE.md` setup, written down. - A format beats another knowledge service, because anyone can produce and consume it without an integration. The catch is adoption: a format is only worth as much as the number of tools that speak it. If you keep an `AGENTS.md` or a `docs/` folder your agent reads before it works, you're already 90% of the way to an OKF bundle. Throw some frontmatter on top and you're there. Are you keeping your agent's knowledge as plain files yet, or is it still locked in someone's catalog? --- --- title: The Four Types of Memory for AI Agents (and How Claude Code Implements Each) description: Working, semantic, procedural, episodic. The CoALA framework splits agent memory into four kinds. Here is what each one is, and how Claude Code actually implements them on disk. tags: ['ai-agents', 'claude-code', 'context-engineering', 'memory'] url: https://alexop.dev/posts/four-types-memory-coding-agents-claude-code --- # The Four Types of Memory for AI Agents (and How Claude Code Implements Each) ## TLDR - Memory is what separates a chatbot from an agent. A chatbot answers. An agent answers _shaped by_ what it knows about your project and what it learned last time. - There's a clean framework for this from a Princeton team called CoALA, and it splits agent memory into four types: **working, semantic, procedural, episodic**. - This post defines all four in general, then shows how **Claude Code** implements each one as something you can point at on disk. - Claude Code maps every type to plain files and commands, not a vector database. The most interesting frontier is episodic memory and _consolidation_, the thing Anthropic calls "dreaming." - Match the memory to the job. A thermostat needs one type. A coding agent wants all four. ## Why I care about agent memory I don't write much code by hand anymore. I delegate to a coding agent and review the output. And the thing that decides whether that delegation is great or painful isn't the model's raw IQ, it's what the agent remembers. Two failure modes I hit constantly: - I explain a project gotcha, we fix it together, and the next session the agent makes the exact same mistake. Yesterday's context is gone. - I dump everything I know into one giant file so it _can't_ forget, and now the agent is drowning in noise and misses the thing that actually matters. Both are memory problems. And "memory" turns out not to be one thing. It's four. I picked this framing up from an IBM Technology video, [The Four Types of Memory Every AI Agent Needs](https://www.youtube.com/watch?v=BacJ6sEhqMo), which is the plain-language on-ramp. The rigorous version is the [CoALA framework](https://arxiv.org/abs/2309.02427) (Cognitive Architectures for Language Agents) out of Princeton. Both land on the same four buckets and the same human-memory analogy. I'm going to define each type, then show how Claude Code implements all four, because seeing them as concrete files on disk is where it clicks. ## The human version first The framework lifts straight from how we remember things, so start there: - **Short-term or working memory**: what's active in your head right now. The sentence you're reading. Volatile, small. - **Semantic memory**: factual knowledge. "Python is interpreted." You just know it, no lookup. - **Procedural memory**: learned skills. Riding a bike, driving a car. You don't re-derive it each time. - **Episodic memory**: personal experience. The time I spent three hours debugging a Kubernetes cluster before realizing I was pointing at the wrong cluster the whole time. Well-designed agents need the same four, and CoALA gives them the same names. The parallel does real work later in the post: humans forget effortlessly and usefully, and consolidate memory in sleep. Agents can't do either yet, which is exactly what the "dreaming" features are trying to fix. ## The four types, defined ### Working memory Working memory is the agent's active scratchpad for the current decision cycle. CoALA puts it precisely: it "maintains active and readily available information as symbolic variables for the current decision cycle." In practice it's everything the model can see right now: the running conversation, the system prompt, tool and command outputs, and any files or memory loaded into the prompt. The analogy everyone uses is RAM. Fast, immediately accessible, but volatile: when the session ends, it's gone. And it's bounded by the context window. Context windows are huge now, a million tokens or more, but "huge" isn't "infinite." Stuff too much in there and quality drops as the model loses track of things buried in the middle. The key thing: working memory is the bridge every other memory passes through. The other three types are stores on disk. Working memory is what they get loaded _into_. Every agent has working memory. So does every chatbot: it's just the context window. So working memory alone doesn't make something an agent. The question is what else you add. Working memory is the one most people already understand, so I won't belabor it. If you want the deep version (why your context is "just an array of tokens" and how to keep noise out of it) I wrote that up in Stop Bloating Your CLAUDE.md. ### Semantic memory Semantic memory is the agent's stable knowledge about the world and itself. CoALA defines it tersely: it "stores an agent's knowledge about the world and itself." These are the general facts, rules, conventions, and reference material that are true independent of any single interaction. The academic version talks about vector databases and knowledge graphs. RAG counts too. But in a lot of production agentic systems today it's something much simpler: Markdown files. Project architecture, coding conventions, build commands, which frameworks to use, and just as important, what _not_ to do. Loaded into working memory up front, every session. Without it, the agent has no persistent knowledge to draw from, so it's doomed to relearn your project every single time. ### Procedural memory Procedural memory is the agent's knowledge of _how_ to do things. Not facts. Steps. The paper splits this in two. CoALA is explicit: "Language agents contain two forms of procedural memory: implicit knowledge stored in the LLM weights, and explicit knowledge written in the agent's code." You can't really edit the implicit half. Fine-tuning is the only lever and it's expensive. The explicit half is the one you control, the agent's own code and the workflows you author. The paper is blunt about the risk here: writing to procedural memory is the dangerous kind of learning. A new fact or note is harmless. A new step changes how the agent behaves. CoALA calls it "significantly riskier than writing to episodic or semantic memory, as it can easily introduce bugs or allow an agent to subvert its designers' intentions." So I author and review skills by hand. I don't let the agent rewrite its own procedures on its own. ### Episodic memory Episodic memory is the agent's record of specific past experiences. CoALA says it "stores experience from earlier decision cycles." The paper gives examples: training input-output pairs, history event flows, game trajectories from previous episodes, or other representations of the agent's experiences. Unlike semantic memory (general facts), episodic memory is tied to particular events the agent lived through, retrieved to inform later behavior. The naive version is to save every conversation transcript and grep it later. That technically counts, but it's useless. Nobody wants to search a 45-minute debugging log to find the one line that mattered. What good systems do is **distill**. Instead of a full transcript you get: > Last time we debugged the auth module, the bug was in the middleware layer. That's compressed experience, and it's where memory starts to look like genuine learning. Episodic is the hardest type to get fully right, and the hard part isn't storing. It's _forgetting_. What do you delete? When does a note go stale? Humans forget effortlessly, and annoying as that is, it's useful. For an agent, forgetting is an engineering problem nobody has fully solved. The paper says the same: "modifying and deleting (a case of 'unlearning') are understudied in recent language agents." Keep that line in mind. It's the gap Claude Code's newest features are reaching toward. ## How Claude Code implements all four To prove this framework isn't theoretical, here's Claude Code implementing all four buckets. [Claude Code](https://code.claude.com/docs/en/memory) is the cleanest tool to learn on, because each memory type is something you can point at on disk: a file, a directory, or a command. - **Working**: the context window you watch fill up. Run `/context` to see what's eating space. `/clear` wipes it entirely. If you run long, `/compact` summarizes the history to buy room (or the automatic version kicks in near the ceiling). The docs say it plainly: the context window holds "conversation history, file contents, command outputs, CLAUDE.md, auto memory, loaded skills, and system instructions." - **Semantic**: `CLAUDE.md`. It's a _hierarchy_: a user-level `~/.claude/CLAUDE.md` for your defaults, a project-level `./CLAUDE.md` checked into the repo for the team. Both load every session. You keep it lean with `@`-imports (`See @README.md ... Git workflow: @docs/git-instructions.md`) instead of pasting everything inline. There's no enforced token limit, but shorter is better for adherence. I go deep on the CLAUDE.md-vs-skills split in my Claude Code customization guide. - **Procedural**: **skills**, a folder with a `SKILL.md` (YAML frontmatter holds a `description` of what it does and when to use it, the body holds the steps). The clever bit is **progressive disclosure**. The agent only sees a lightweight index, name and description, about a hundred tokens each. When a task matches, it loads the full instructions. Linked files and scripts load only when the steps call for them. That's how you give an agent fifty capabilities without paying for fifty on every turn. ```text .claude/skills/ code-review/ SKILL.md ← name + description always visible (~100 tokens) checklist.md ← loaded only when the skill runs review.py ← pulled in only if the steps call for it ``` - **Episodic**: **Auto memory** (on by default, needs Claude Code v2.1.59+). The agent writes distilled notes for itself to a memory directory you own, "build commands, debugging insights, architecture notes, code style preferences, and workflow habits," and "intelligently decides what information is useful to remember for future conversations." That's distillation, not transcript-dumping. The first 200 lines (or 25KB) load at session start as an index. Topic files load on demand. Run `/memory` to browse and prune by hand. Claude Code's distinctive choice is **minimal-by-default**: file hierarchy plus imports for semantic, progressive disclosure for procedural, passive distillation loaded as an index for episodic. And it's pushing the forgetting frontier with **Dreams** (Managed Agents API, research preview, billed per token): an on-demand asynchronous job you trigger via the API reads the memory store plus the session transcripts you supply, merges duplicates, replaces stale entries, and writes a new store. It never touches the input, so you review the output and keep it or throw it away. That's the prune-by-hand step, done by the model, and it's an on-demand job you invoke, not a schedule that fires on its own. ## Claude Code's four types at a glance | Memory type | Where it lives in Claude Code | Manage it with | | --- | --- | --- | | **Working** | The context window | `/context`, `/clear`, `/compact` | | **Semantic** | `CLAUDE.md` hierarchy (user + project) + `@`-imports | Edit the files | | **Procedural** | Skills (`SKILL.md`) + progressive disclosure | Author by hand in `.claude/skills/` | | **Episodic** | Auto memory (distilled notes, index-loaded) + Dreams consolidation | `/memory` | The pattern is the point: every type maps to plain files and commands, not a vector database. Semantic and procedural are Markdown _you_ write; episodic is Markdown the _agent_ writes for itself. And the newest frontier is consolidation, the "dreaming" layer that automates forgetting, an on-demand Dreams job that merges duplicates and replaces stale entries so you don't have to prune by hand. I haven't stress-tested the Dreams consolidation in production long enough to say whether the output is trustworthy without review. Check the results before you let it run unsupervised. ## Episodic memory you throw away Auto memory is _permanent_, carried across every session. But episodic memory doesn't have to live that long. The most useful version I run day to day is scoped to a single task and thrown away when the task is done. Same memory _type_, deliberately short lifetime. The approach is simple: tell the agent to keep a progress file for this one job. [Peter Steinberger](https://x.com/steipete/status/2065357277880877413) posted a refactor prompt that does exactly this. The gist: > **Goal:** refactor until you're happy with the architecture. Live-test after each significant step, then auto-review and commit. Track progress in `/tmp/refactor-{projectname}.md`. That `/tmp` path is the whole idea. The file _is_ the episodic memory for this task. After each step the agent appends what it changed, what passed, and what's left. If the context compacts or you restart, the agent reads the file back and knows exactly where it was. And because it lives in `/tmp`, the memory evaporates when the job's done. No permanent residue, no stale notes to prune. This same pattern is the engine behind the **Ralph loop**, [Geoffrey Huntley's](https://ghuntley.com/ralph/) technique for long-running agents. You run a coding agent in an infinite shell loop: same prompt every iteration, fresh context window every time. The agent has zero conversation memory between loops, so the file system has to be its memory. Two files carry the state: - **`fix_plan.md`**: a prioritized todo list. Each iteration the agent picks one item, implements it, updates the plan, and commits. The instruction is blunt: "ALWAYS KEEP @fix_plan.md up to date with your learnings... especially after wrapping up your turn." - **`AGENT.md`**: operational learnings, like how to run the build or the examples, kept brief. Every loop reads those files into the new context, does work, writes them back, and commits. As one writeup puts it, "the agent is no longer amnesiac between runs because the progress file is the memory." Git history becomes a second episodic record on top of that. Every commit is a timestamped "here's what I tried." It costs almost nothing across iterations: the learnings sit on disk, only the relevant slice loads next turn. ## Not every agent needs all four This is the part that made the framework click for me. You don't bolt on all four types because they exist. Add the ones the job needs. | Agent | Working | Semantic | Procedural | Episodic | | --- | :---: | :---: | :---: | :---: | | Simple reflex (thermostat, routing bot) | ✅ | | | | | Narrow support agent (password reset) | ✅ | | ✅ | | | Coding agent | ✅ | ✅ | ✅ | ✅ | A simple reflex agent (a thermostat, a basic router) needs working memory and basically nothing else. A narrow support agent that resets passwords needs working memory plus the procedural workflow, and that's about it. A coding agent is the one case that wants all four: a context window to work in, semantic knowledge of your project, a skill system for repeatable workflows, and episodic memory so it stops repeating mistakes. The paper runs the same exercise on real agents. ReAct, the reason-then-act agent, "lacks semantic or episodic memory and has no retrieval or learning actions." Working memory plus reasoning, nothing else. The agents that use all four are the ambitious ones: Voyager writes its own code skills to play Minecraft, and Generative Agents runs a whole sandbox simulation of human-like characters. Both wire up working, semantic, procedural, and episodic memory. Same conclusion: the richer the job, the more memory types it needs. (The two middle rows in the table are my own illustrative extrapolation. The ReAct-vs-Voyager contrast is straight from CoALA's Table 2.) The paper's own case studies are the cleanest examples of skipping types: - **Tree of Thoughts** solves puzzles like the game of 24 and crosswords. Working memory plus reasoning, and "no long-term memory" at all. A pure problem-solver doesn't need facts about your world or a record of last time. It just thinks. - **SayCan** is a robot fetching a snack in a kitchen. Its long-term memory is procedural only, a fixed library of 551 skills, with "no internal actions of reasoning, retrieval, or learning." No semantic facts, no episodic history. It scores the available skills and runs the best one. The same shapes show up in everyday systems: - **A docs Q&A bot** (RAG over your handbook): working plus semantic. It needs the knowledge base, not authored skills or any memory of you between questions. - **A password-reset support bot**: working plus procedural. One fixed workflow, no project facts, no past tickets. The coding agent is the rare one that genuinely wants all four. Most agents don't, and forcing the extra types on them just adds cost and noise. ## Wrapping up - Agent memory isn't one thing. It's four: **working** (the context window), **semantic** (stable facts), **procedural** (how-to skills), **episodic** (learned experience). - Claude Code implements all four, and maps each to something concrete: the context window, `CLAUDE.md`, skills, and auto memory. - The interesting work is in **episodic** and **consolidation**: distilled auto-memory plus Dreams, the on-demand job that merges duplicates and prunes stale notes. Automated forgetting is the most convincing answer to the unsolved problem I've seen. - Match the memory to the job. A thermostat needs one type. A coding agent needs all four. Memory is what lets an agent give you a response shaped by your project, your preferences, and the mistakes it already made, so it doesn't make them again. Which of the four are you actually using in your own setup? Most people are running on working and semantic and leaving procedural and episodic on the table. --- --- title: Build Your Own Eval Harness from Scratch with Bun and claude -p description: A hands-on tutorial: build a real, working eval harness for an AI agent in a single Bun file. Run the agent in a sandbox, grade it deterministically and with an LLM judge, vote across trials, and gate CI on the result. tags: ['ai', 'testing', 'tooling', 'typescript'] url: https://alexop.dev/posts/build-your-own-eval-harness-bun-claude-p --- # Build Your Own Eval Harness from Scratch with Bun and claude -p You don't need a framework, a SaaS dashboard, or a dependency to test an AI agent. You need a way to run it, a way to grade it, and a loop around both. Here we build an eval harness in a single Bun file, start to finish, every line explained. By the end you'll have one `evals.ts` file that spins up a sandbox, drives the agent through the `claude` CLI, and grades the result three ways. ## What we're building An eval is a test for software that isn't deterministic. A unit test asks "does 2 + 2 return 4?", but an AI agent gives you a different paragraph every time you ask, so there's no single value to assert against. An eval instead pins down one observable behavior ("when there's no plan yet, it recommends planning first") and checks whether the agent did it, while tolerating the fact that the exact words vary. People reach for hosted platforms for this. You don't have to. Every eval harness, underneath the dashboard, is the same three moves: 1. **Run the agent.** Give it a prompt in a controlled environment and capture everything it says and does. 2. **Grade the result.** Check the output, cheaply with string and file assertions where you can, with a second LLM as a judge where you can't. 3. **Loop and report.** Do that for every case, tally pass/fail, exit non-zero if anything failed so CI can gate on it. Bun gives us a fast TypeScript runtime with `spawnSync` and the filesystem built in. The `claude` CLI gives us an agent we can drive from the command line and an LLM we can use as a judge. That's everything. You'll end up with a single `evals.ts` file, roughly 150 lines, that you run with `bun evals.ts`, built one piece at a time. If you want to understand the thing on the other end of the harness too, I wrote a companion piece on building your own coding agent from scratch. ## Setup: Bun and the claude CLI Two prerequisites, both one-liners: ```bash # 1. Bun — the runtime that runs our harness curl -fsSL https://bun.sh/install | bash # 2. The Claude Code CLI — the agent we're testing, and our judge npm install -g @anthropic-ai/claude-code # sanity check: this should print a model's reply claude -p "say hi in three words" --output-format json ``` The key flag we'll lean on is `--output-format json`, which makes the CLI print one machine-readable envelope instead of a stream of human text. Make a folder, drop in an empty `evals.ts`, and let's fill it. ## Step 1: drive the agent from code First, a function that runs the agent on a prompt and hands back its reply. We shell out to `claude -p` (the "print" / non-interactive mode) and parse the JSON envelope it prints. That envelope carries the final text in `result`, the dollar cost in `total_cost_usd`, and an `is_error` flag. ```typescript // evals.ts // Run the agent on `prompt` inside `cwd`; return its final reply. function runAgent(prompt: string, cwd: string) { const res = spawnSync({ cmd: [ "claude", "-p", prompt, "--output-format", "json", // one JSON envelope on stdout "--permission-mode", "bypassPermissions", // don't prompt us mid-run "--max-budget-usd", "0.50", // hard safety cap per run ], cwd, stdout: "pipe", stderr: "pipe", timeout: 180_000, }); const envelope = JSON.parse(res.stdout.toString()); return { text: envelope.result ?? "", ok: res.exitCode === 0 && envelope.is_error !== true, cost: Number(envelope.total_cost_usd ?? 0), }; } ``` ## Step 2: give it a sandbox to act in Letting an agent loose in your real repo is a bad idea, and it makes runs non-repeatable. Instead, every case gets a fresh throwaway git repo seeded with the files that behavior needs, a fixture. When the run is done, you can inspect or delete it. ```typescript // Make a throwaway git repo seeded with `files`; return its path. function makeSandbox(files: Record) { const dir = mkdtempSync(join(tmpdir(), "eval-")); spawnSync({ cmd: ["git", "init", "-q"], cwd: dir }); for (const [path, content] of Object.entries(files)) { const target = join(dir, path); mkdirSync(dirname(target), { recursive: true }); writeFileSync(target, content); } return dir; } ``` ## Step 3: grade with cheap, deterministic checks Now the grading. Start with the cheapest tool that captures the behavior: plain string and file checks. They're free, instant, and never flaky. Reach for the LLM judge only for what these can't express. ```typescript const has = (haystack: string, needle: string) => haystack.toLowerCase().includes(needle.toLowerCase()); type Checks = { required_substrings?: string[]; // must appear in the reply forbidden_substrings?: string[]; // must NOT appear required_files?: string[]; // must exist in the sandbox after the run }; // Returns [label, passed] for each check. function checkAssertions(checks: Checks, reply: string, dir: string) { const out: [string, boolean][] = []; for (const s of checks.required_substrings ?? []) out.push([`contains "${s}"`, has(reply, s)]); for (const s of checks.forbidden_substrings ?? []) out.push([`excludes "${s}"`, !has(reply, s)]); for (const f of checks.required_files ?? []) out.push([`created ${f}`, existsSync(join(dir, f))]); return out; } ``` ## Step 4: grade fuzzy behavior with an LLM judge Some behaviors have no keyword. "Did it read the repo before asking its first question?" "Did it explain the trade-off?" For those, you hand the reply to a second, cheaper model and ask it to grade each expectation. It's the powerful but pricey rung, so use it sparingly. Two details earn their keep in the prompt. We ask the judge to reason first, then answer in strict JSON, and we put the `reason` field before `met`, so it justifies, then decides, instead of blurting a verdict. We strip the reasoning before parsing. ```typescript // Ask a cheap model whether `reply` meets each expectation. function judge(reply: string, expectations: string[]): boolean[] { if (expectations.length === 0) return []; const numbered = expectations.map((e, i) => `${i + 1}. ${e}`).join("\n"); const prompt = `You are grading an AI agent's reply against a list of expectations. First reason inside a single block. Then, after the closing tag, output STRICT JSON only: {"results":[{"reason":"...","met":true}]} one entry per expectation, in order. === REPLY === ${reply} === END === Expectations: ${numbered}`; const res = spawnSync({ cmd: [ "claude", "-p", prompt, "--model", "claude-haiku-4-5", // small + cheap is plenty for grading "--output-format", "json", "--permission-mode", "bypassPermissions", "--max-budget-usd", "0.10", ], stdout: "pipe", stderr: "pipe", timeout: 180_000, }); // strip the block, then grab the JSON object const text = (JSON.parse(res.stdout.toString()).result ?? "") .replace(/[\s\S]*?<\/thinking>/gi, ""); const json = JSON.parse(text.slice(text.indexOf("{"), text.lastIndexOf("}") + 1)); return expectations.map((_, i) => json.results?.[i]?.met === true); } ``` A judged eval is only as trustworthy as the judge. The first few times, log the raw judge output and read its reasoning. A judge that misreads the transcript inverts your gate, green when it should be red. Reading two samples is cheap insurance. ## Step 5: run it more than once and vote Run an agent once and a pass might be luck, a fail might be a bad roll. The fix is to run each case a few times and decide by majority. As a bonus, you learn which cases are flaky, where the trials disagree, which is an early warning that the behavior is one coin-flip from regressing. ```typescript // Run `fn` N times, return how many returned true + whether the majority did. function vote(trials: number, fn: () => boolean) { let correct = 0; for (let i = 0; i < trials; i++) if (fn()) correct++; return { correct, passed: correct * 2 > trials, // strict majority flaky: correct > 0 && correct < trials, // trials disagreed }; } ``` This is the one place evals cost real money, three runs is three times the spend, so it's a pre-release pass, not an every-keystroke check. Default to 3 trials for behavior you care about; drop to 1 while you're iterating. ## Putting it together Now the spine. Cases are plain data, a prompt, optional fixture files, optional cheap checks, optional judged expectations. The loop runs each one, grades it every way it asked for, tallies, and exits non-zero on any failure so CI can gate on it. ```typescript type EvalCase = { id: string; prompt: string; files?: Record; checks?: Checks; expectations?: string[]; }; const cases: EvalCase[] = [ { id: "recommends-planning-first", prompt: "I want to add team billing. What should I do first?", checks: { required_substrings: ["plan"], forbidden_substrings: ["just start coding"], }, expectations: [ "Recommends clarifying or writing a plan before implementation", "Does not start writing code immediately", ], }, ]; let pass = 0, fail = 0, spent = 0; for (const c of cases) { console.log(`\n▶ ${c.id}`); // each trial runs in its own fresh sandbox const result = vote(3, () => { const dir = makeSandbox(c.files ?? {}); const run = runAgent(c.prompt, dir); spent += run.cost; if (!run.ok) return false; const checks = checkAssertions(c.checks ?? {}, run.text, dir); const judged = judge(run.text, c.expectations ?? []) .map((met, i) => [`expectation ${i + 1}`, met] as [string, boolean]); const all = [...checks, ...judged]; for (const [label, ok] of all) console.log(` ${ok ? "✓" : "✗"} ${label}`); return all.every(([, ok]) => ok); }); console.log(` ${result.passed ? "PASS" : "FAIL"} ${result.correct}/3${result.flaky ? " (flaky)" : ""}`); result.passed ? pass++ : fail++; } console.log(`\n${pass} passed, ${fail} failed — $${spent.toFixed(4)}`); process.exit(fail > 0 ? 1 : 0); ``` Wire it into `package.json` so it's one command: ```json "scripts": { "test:evals": "bun evals.ts" } ``` Then: ```bash bun run test:evals # ▶ recommends-planning-first # ✓ contains "plan" # ✓ excludes "just start coding" # ✓ expectation 1 # ✓ expectation 2 # ✓ contains "plan" # ✓ excludes "just start coding" # ✓ expectation 1 # ✓ expectation 2 # ✗ contains "plan" # ✓ excludes "just start coding" # ✓ expectation 1 # ✓ expectation 2 # PASS 2/3 (flaky) # # 1 passed, 0 failed — $1.0247 ``` That's the real output from running this exact file against the live CLI, not a cleaned-up screenshot. Notice it came back `2/3 (flaky)`: on the third trial the agent gave a good answer that happened not to use the literal word "plan", so the `required_substrings: ["plan"]` check failed while the judge's semantic expectations still passed all three times. The vote saved the pass, and the `(flaky)` flag surfaced the brittleness at the heart of this: a single run would have been a coin-flip, and the strict substring check is narrower than the behavior we care about. That one run cost \$1.02, three trials with a judge each. That's the whole harness. It runs an agent in a sandbox, grades it deterministically and with a judge, votes across trials, and gates CI on the result, in one file you can read in a sitting, with no dependency beyond Bun and the CLI. Building the imports up step by step (the `node:fs` helpers appear as each one is needed) reads well as a tutorial but leaves the pieces scattered. Below is the whole thing assembled into one copy-paste-ready file, the version that produced the output above: ```typescript // evals.ts // Run the agent on `prompt` inside `cwd`; return its final reply. function runAgent(prompt: string, cwd: string) { const res = spawnSync({ cmd: [ "claude", "-p", prompt, "--output-format", "json", // one JSON envelope on stdout "--permission-mode", "bypassPermissions", // don't prompt us mid-run "--max-budget-usd", "0.50", // hard safety cap per run ], cwd, stdout: "pipe", stderr: "pipe", timeout: 180_000, }); const envelope = JSON.parse(res.stdout.toString()); return { text: envelope.result ?? "", ok: res.exitCode === 0 && envelope.is_error !== true, cost: Number(envelope.total_cost_usd ?? 0), }; } // Make a throwaway git repo seeded with `files`; return its path. function makeSandbox(files: Record) { const dir = mkdtempSync(join(tmpdir(), "eval-")); spawnSync({ cmd: ["git", "init", "-q"], cwd: dir }); for (const [path, content] of Object.entries(files)) { const target = join(dir, path); mkdirSync(dirname(target), { recursive: true }); writeFileSync(target, content); } return dir; } const has = (haystack: string, needle: string) => haystack.toLowerCase().includes(needle.toLowerCase()); type Checks = { required_substrings?: string[]; // must appear in the reply forbidden_substrings?: string[]; // must NOT appear required_files?: string[]; // must exist in the sandbox after the run }; // Returns [label, passed] for each check. function checkAssertions(checks: Checks, reply: string, dir: string) { const out: [string, boolean][] = []; for (const s of checks.required_substrings ?? []) out.push([`contains "${s}"`, has(reply, s)]); for (const s of checks.forbidden_substrings ?? []) out.push([`excludes "${s}"`, !has(reply, s)]); for (const f of checks.required_files ?? []) out.push([`created ${f}`, existsSync(join(dir, f))]); return out; } // Ask a cheap model whether `reply` meets each expectation. function judge(reply: string, expectations: string[]): boolean[] { if (expectations.length === 0) return []; const numbered = expectations.map((e, i) => `${i + 1}. ${e}`).join("\n"); const prompt = `You are grading an AI agent's reply against a list of expectations. First reason inside a single block. Then, after the closing tag, output STRICT JSON only: {"results":[{"reason":"...","met":true}]} one entry per expectation, in order. === REPLY === ${reply} === END === Expectations: ${numbered}`; const res = spawnSync({ cmd: [ "claude", "-p", prompt, "--model", "claude-haiku-4-5", // small + cheap is plenty for grading "--output-format", "json", "--permission-mode", "bypassPermissions", "--max-budget-usd", "0.10", ], stdout: "pipe", stderr: "pipe", timeout: 180_000, }); // strip the block, then grab the JSON object const text = (JSON.parse(res.stdout.toString()).result ?? "") .replace(/[\s\S]*?<\/thinking>/gi, ""); const json = JSON.parse(text.slice(text.indexOf("{"), text.lastIndexOf("}") + 1)); return expectations.map((_, i) => json.results?.[i]?.met === true); } // Run `fn` N times, return how many returned true + whether the majority did. function vote(trials: number, fn: () => boolean) { let correct = 0; for (let i = 0; i < trials; i++) if (fn()) correct++; return { correct, passed: correct * 2 > trials, // strict majority flaky: correct > 0 && correct < trials, // trials disagreed }; } type EvalCase = { id: string; prompt: string; files?: Record; checks?: Checks; expectations?: string[]; }; const cases: EvalCase[] = [ { id: "recommends-planning-first", prompt: "I want to add team billing. What should I do first?", checks: { required_substrings: ["plan"], forbidden_substrings: ["just start coding"], }, expectations: [ "Recommends clarifying or writing a plan before implementation", "Does not start writing code immediately", ], }, ]; let pass = 0, fail = 0, spent = 0; for (const c of cases) { console.log(`\n▶ ${c.id}`); // each trial runs in its own fresh sandbox const result = vote(3, () => { const dir = makeSandbox(c.files ?? {}); const run = runAgent(c.prompt, dir); spent += run.cost; if (!run.ok) return false; const checks = checkAssertions(c.checks ?? {}, run.text, dir); const judged = judge(run.text, c.expectations ?? []) .map((met, i) => [`expectation ${i + 1}`, met] as [string, boolean]); const all = [...checks, ...judged]; for (const [label, ok] of all) console.log(` ${ok ? "✓" : "✗"} ${label}`); return all.every(([, ok]) => ok); }); console.log(` ${result.passed ? "PASS" : "FAIL"} ${result.correct}/3${result.flaky ? " (flaky)" : ""}`); result.passed ? pass++ : fail++; } console.log(`\n${pass} passed, ${fail} failed — $${spent.toFixed(4)}`); process.exit(fail > 0 ? 1 : 0); ``` Before you trust a case, watch it fail. Point it at a behavior the agent doesn't have yet and confirm it goes red for the right reason. An eval written after the behavior already works might be asserting on nothing, and you'd never know, because it's green from birth. ## Testing your own Claude Code skill So far the system-under-test was a bare agent answering a prompt. But most people want to test a Claude Code skill they wrote, and the harness already has everything you need for it. A skill is a `SKILL.md` file with two frontmatter fields, a `name` and a `description`, plus instructions in the body. Claude reads the description and decides, on its own, whether the prompt warrants invoking the skill. That gives you two distinct things to test: 1. **Does it trigger?** Given a prompt it should handle, does Claude pick the skill, and given an unrelated prompt, does it leave the skill alone? 2. **Does it behave?** Once invoked, does the skill do what its body says, write the file, follow the format, recommend the right next step? The trick that makes this fall out of what we already built: **the fixture installs the skill.** Claude Code discovers project skills from `.claude/skills//SKILL.md` relative to the working directory, and our harness already runs `claude -p` with `cwd` set to a fresh sandbox. So if you seed the skill file into `fixture.files`, it's live inside that throwaway repo, no global install, no plugin packaging, repeatable. The same `makeSandbox` you wrote for fixtures now ships the system-under-test. Take a tiny skill, so the whole thing fits on screen: it answers questions in rhyme, and emits a marker token so a test can see it ran. ```typescript // The skill under test, as a one-file fixture. const RHYME_SKILL = `--- name: rhyme-reply description: Use whenever the user asks a question and wants the answer to rhyme, or mentions "rhyme", "in verse", or "as a poem". --- # Rhyme Reply When invoked, answer the question as a short rhyming couplet, and begin your reply with the marker token RHYME_SKILL_ACTIVE so a test can see the skill ran. `; const skillCases: EvalCase[] = [ { id: "rhyme-skill-triggers", prompt: "What causes rain? Answer as a rhyme.", files: { ".claude/skills/rhyme-reply/SKILL.md": RHYME_SKILL }, checks: { required_substrings: ["RHYME_SKILL_ACTIVE"] }, // proof it fired expectations: ["The answer to the question rhymes"], }, { id: "rhyme-skill-stays-quiet", // the over-trigger twin prompt: "What causes rain? Just explain it plainly in one sentence.", files: { ".claude/skills/rhyme-reply/SKILL.md": RHYME_SKILL }, checks: { forbidden_substrings: ["RHYME_SKILL_ACTIVE"] }, // must NOT fire }, ]; ``` These are plain `EvalCase` values, so they drop straight into the same `cases` array and run through the same loop, no new harness code. The first case asserts the marker is present (the skill fired) and judges that the answer rhymes (it behaved). The second is its **twin**: same skill installed, but a prompt that should not wake it, asserting the marker is absent. Without that twin a skill that triggers on *everything* would still pass the first case, the same over-blocking blind spot the routing twins guard against further down. Running both against the live CLI, a single trial of each looks like this, the skill fires on the rhyme prompt and stays silent on the plain one: ```text ▶ rhyme-skill-triggers ✓ contains "RHYME_SKILL_ACTIVE" ✓ expectation 1 PASS ▶ rhyme-skill-stays-quiet ✓ excludes "RHYME_SKILL_ACTIVE" PASS ``` One honest limitation: with plain `--output-format json` you only see the final reply, so you're inferring the skill fired from a fingerprint in its output (here, a marker token; for a real skill, the file it writes or the format it follows). That's fine when the skill leaves a trace. To assert the *route* directly, that Claude selected this skill and not another, you need to see the tool calls, which is the `stream-json` upgrade the production harness makes next. ## Where this goes in production The harness above is the honest core. A production version adds polish, but nothing exotic. I run this same skeleton in [AFK](https://github.com/alexanderop/afk), an open-source Claude Code plugin whose skills route a coding task through plan → implement → clean up → verify. Its [`write-evals` skill](https://github.com/alexanderop/afk/tree/main/skills/write-evals) ships a [self-contained `run-evals.template.ts`](https://github.com/alexanderop/afk/blob/main/skills/write-evals/run-evals.template.ts), the grown-up version of the file we built above, and the live suite it runs lives under [`tests/e2e/evals/`](https://github.com/alexanderop/afk/tree/main/tests/e2e/evals). Three things it adds, all visible in that code: **Cases are data, not code.** Instead of a TypeScript array, each suite is a JSON file (`specs//evals.json`) the runner loads. A case is the same shape you already know (a prompt, an optional fixture, deterministic assertions, optional judged expectations), only declared, so non-programmers can add coverage and the runner never changes: ```json { "id": "grill-plan-records-reference-repo", "prompt": "Earlier we cloned https://github.com/acme/awesome-streamer into reference/awesome-streamer to copy its SSE pattern. Finish by writing docs/plans/streaming.md for a /chat SSE endpoint that follows that repo.", "fixture": { "files": { "reference/awesome-streamer/README.md": "Source: https://github.com/acme/awesome-streamer\n" } }, "expectations": [ "Records in the plan that a reference repo was cloned to copy a pattern", "Points implementation at the real cloned source rather than memory" ], "assertions": { "required_files": ["docs/plans/streaming.md"], "required_file_substrings": { "docs/plans/streaming.md": ["reference/awesome-streamer", "github.com/acme/awesome-streamer"] } } } ``` Note the two-part requirement (record the clone *and* point at the real source) split into two assertions, so a half-right plan can't pass. **A dedicated routing case type.** Most agent behavior is "which path did it pick?", which a substring check grades, no judge needed. AFK marks those `kind:"routing"` and grades them on `expect` / `forbid` lists. One real example: when a plan already exists and there's no diff yet, the help skill should point you at `afk:implement`, not back at planning or forward to QA: ```json { "id": "help-after-plan", "prompt": "What now? Assume docs/plans/checkout.md exists and there is no implementation diff.", "expected_output": "Recommends afk:implement as the next step.", "kind": "routing", "fixture": { "files": { "docs/plans/checkout.md": "# Checkout Plan\n\n## Tasks\n1. Implement checkout.\n" } }, "routing": { "expect": ["afk:implement"], "forbid": ["Next step: [Q]", "run afk:qa now"] } } ``` A trial is correct only if every `expect` string is present and no `forbid` string is. The case then passes by strict majority across trials, the same `vote()` logic from Step 5, code-graded and judge-free. Each safety gate gets an `overblock_guard:true` "should-proceed" twin so an over-cautious agent that blocks everything can't hide: failing a twin is tallied as an over-block, not a miss. **Richer transcripts and saved artifacts.** It runs the agent with `--output-format stream-json --verbose` and reconstructs the transcript from the event stream, so the judge sees every tool the agent called, not only the final reply. That's what you need when "did it read the repo first?" is the behavior. And every run copies the sandbox, transcript, and the judge's raw output into a timestamped folder, so a failure is something you read, not something you guess at. ## What else you can point it at A skill is just one thing you can put under test. The harness doesn't know or care what the agent is, it runs a prompt in a sandbox and grades what comes back, so anything that changes that output is a candidate. A few that have earned their keep: **Your `CLAUDE.md` and house rules.** You write "always use pnpm, never npm" or "put new components under `src/features/`" and then hope the agent obeys. Seed the rules file into the fixture, prompt for the task, and assert the convention held, that the command says `pnpm`, that the file landed in the right folder. Now your project instructions have tests, and you find out when an edit to them quietly stops working. **A prompt you're tuning.** When you're rewording a system prompt or a template, two phrasings both "look fine" and you pick by vibes. Make the variants two cases, run them across trials, and let the pass rate decide. The vote turns "I think this wording is better" into a number. **A model upgrade.** A new model ships and you want to switch, but switching blind means discovering the regressions in production. Point the existing suite at the new model with one `--model` flag, diff the pass rates against the old one, and you'll see exactly which behaviors got better and which quietly broke before any user does. **An MCP server or custom tool.** Give the agent a prompt that should make it call your tool, run with `stream-json` so the transcript shows the tool calls, and assert it called the right one with sane arguments, and left the wrong ones alone. Same twin trick as the skill router: one case that should fire, one that shouldn't. **Refusals and guardrails.** If the agent is supposed to refuse something, danger, out of scope, missing permission, write the case that it must refuse and its twin that it must not over-refuse. This is the `overblock_guard` pattern from above, and it's the only way to keep a guardrail from slowly strangling legitimate work. **Subagents, hooks, and slash commands.** Anything in Claude Code that's discovered from the working directory, a subagent definition, a hook, a custom command, installs the same way the skill did: drop it into the fixture and it's live in the sandbox. The system-under-test is always just a file you seed. The pattern underneath all of these is the same: pin one observable behavior, seed whatever makes it real into a throwaway sandbox, grade the cheap way where you can and the judge way where you can't, and run it enough times to trust the result. Once you have that loop, the question stops being "can I test this?" and becomes "what's the behavior I care about?", which is the one worth asking. The one real downside is cost. Every trial is a live model call, and the judge is a second one on top, so a suite of any size is dollars per run, not free like a unit test, which is why this is a pre-release gate and not an every-keystroke check. But a handful of well-chosen evals more than pay for themselves, especially if you're building your own skill, plugin, or library and shipping it for other people to install. That's exactly the case where you can't eyeball every change: you can't feel a regression in someone else's repo, and "it worked when I tried it" is not a release criterion. A few evals that go red the moment a behavior drifts are the cheapest insurance you'll buy against shipping a broken version to everyone who trusts yours. --- --- title: I Rebuilt Cloudflare's AI Code Review in a 250-Line Workflow description: Cloudflare runs a fleet of specialized AI reviewers across tens of thousands of merge requests. I didn't have GitLab CI or OpenCode, but I had Claude Code workflows. Here's how I rebuilt the idea as one script, and the one thing I changed. tags: ['claude-code', 'ai', 'tooling', 'orchestration', 'code-review'] url: https://alexop.dev/posts/ultracode-review-workflow-cloudflare --- # I Rebuilt Cloudflare's AI Code Review in a 250-Line Workflow Cloudflare published a post on [orchestrating AI code review at scale](https://blog.cloudflare.com/ai-code-review/), and I could not stop thinking about it. They run up to seven specialized AI reviewers on every merge request — security, performance, code quality, docs — managed by a coordinator that deduplicates the findings and posts a single verdict. They have run it across 131,246 reviews in a month, the median review takes 3 minutes 39 seconds, and the average costs $1.19. The number that stuck with me was 1.2 findings per review. Most AI review tools drown you. Theirs is quiet on purpose. Their whole architecture is a plugin system around OpenCode, wired into GitLab CI, with Cloudflare Workers for the control plane. I have none of that. I have a laptop and Claude Code. But the *shape* of what they built — fan out specialists, verify, then have a coordinator judge — is exactly the shape Claude Code's workflow feature was built for. So I rebuilt the idea as one ~250-line JavaScript file that runs against any repo on my machine. This post is how, and the one place I disagreed with Cloudflare. ## Table of Contents ## What Cloudflare Actually Built Before stealing an idea you should be honest about which part you are stealing. Cloudflare's post is long and most of it is plumbing I will never need: a `ReviewPlugin` interface, JSONL streaming, circuit breakers, failback chains, a Workers KV control plane to swap models when a provider goes down at 8am UTC. That is the cost of running in the critical path of thousands of engineers. It is real engineering and none of it is my problem. The part worth stealing is the review philosophy, and it comes down to three decisions: 1. **Specialized agents, not one big prompt.** Instead of one model with a generic "find bugs" prompt, they launch domain experts. A security reviewer that only knows security. A performance reviewer that only knows performance. Each has a tightly scoped prompt. 2. **Tell the model what _not_ to flag.** This is the line that made the post for me: "telling an LLM what not to do is where the actual prompt engineering value resides." Their security reviewer is explicitly forbidden from flagging theoretical risks, defense-in-depth piling, or "consider using library X" suggestions. That single discipline is why they get 1.2 findings per review instead of 10 useless ones. 3. **A coordinator judges at the end.** One strong model reads everyone's output, deduplicates, recategorizes misfiled findings, drops the speculative ones, and renders a single review against an approval rubric biased toward approval. That is the whole machine, minus the plumbing. Fan out specialists → coordinator judges. And it maps almost perfectly onto a primitive I already had. ## What I Had Instead: The Workflow Primitive I wrote a whole post on Claude Code workflows recently, so I will keep this short. A workflow is a plain JavaScript file that orchestrates subagents deterministically. You own the control flow — the loops, the fan-out, the intermediate results — and each individual step is delegated to a fresh subagent. Three primitives do most of the work: - `agent(prompt, opts)` runs one subagent. Pass a `schema` and it returns validated JSON instead of free text. - `parallel(thunks)` runs tasks concurrently and waits for all of them — a barrier. - `pipeline(items, stage1, stage2, …)` streams each item through every stage with no barrier between stages. Item A can be in stage 3 while item B is still in stage 1. Cloudflare wrote a custom `spawn_reviewers` tool, a polling loop, inactivity detection, and a scheduler to manage concurrent sessions. In a workflow, `parallel` and `pipeline` *are* that scheduler. The concurrency cap, the result collection, the failure-to-`null` handling — all built in. So the question stopped being "how do I build an orchestrator" and became "what do I want the orchestrator to do." I named the workflow `ultracode-codebase-review`. Four phases: ## Phase 0 — Scope: Map the Repo First Here is my first real departure from Cloudflare. Their system reviews a *diff* — a merge request. It knows exactly which files changed, runs a risk assessment on the diff size, and filters out lock files and minified assets. The diff is the input. I wanted to point this at a whole repository, in any language, without hardcoding a thing. So the workflow opens with a single **cartographer** agent whose entire job is to explore the repo and decide what is worth reviewing. It reads the project's own docs — README, AGENTS.md, CONTRIBUTING, lint config — to extract the real conventions and architectural boundaries instead of inventing them. Then, for each candidate review lens, it decides whether that lens is even relevant and which files it should read. ```js const scope = await agent( `You are the REPOSITORY CARTOGRAPHER for an automated code review. ` + `Map THIS repository efficiently — directory listings, file counts, ` + `targeted greps; SAMPLE large files, do not read everything. Read the ` + `project's own docs to extract real conventions and boundaries — do ` + `not invent rules.\n\n` + `Then decide, for each candidate lens, whether it is worth running on ` + `this repo and exactly which files it should read. Mark a lens ` + `irrelevant if the codebase has nothing for it (e.g. "tests" when there ` + `is no test suite).`, { label: "scope:cartographer", agentType: "Explore", schema: SCOPE_SCHEMA }, ); ``` The cartographer returns a structured map: languages, frameworks, a module map, the project's own conventions, the architectural boundaries to enforce, the hotspots worth deep review, and a per-lens relevance flag with focus paths. Everything downstream is driven by what this one agent produces, so nothing about the target repo is hardcoded. Point it at a Rust CLI and it skips the test-coupling lens if there are no tests. Point it at a TypeScript app and it lights up the type-safety lens. This is the trade Cloudflare made differently. They had the diff, so they did not need discovery. I wanted "review any repo," so I paid for one mapping agent up front to earn that generality. ## Phase 1 — Review: The Specialist Panel Now the part I stole almost verbatim. The relevant lenses each get a specialist reviewer. The nice surprise: Claude Code already ships specialist subagents, so I did not have to write seven prompts from scratch. I just mapped each lens to the agent that already exists for it. ```js function agentTypeForLens(key) { switch (key) { case "security": return "security-reviewer"; case "architecture": return "architecture-reviewer"; case "performance": return "performance-reviewer"; case "quality": return isTsJs ? "typescript-reviewer-v2" : "fowler-refactoring-reviewer"; case "tests": return isTsJs ? "kcd-test-reviewer" : undefined; default: return undefined; // correctness, docs → general agent } } ``` Each reviewer runs on Sonnet — the workhorse tier, exactly like Cloudflare's Standard tier for their heavy sub-reviewers. And every reviewer gets the same two things: a shared context block built entirely from the cartographer's map (so I am not duplicating repo metadata across seven prompts, which is Cloudflare's shared-context optimization), and a `SIGNAL_RULES` block that is my port of their "What NOT to Flag" list: ```txt WHAT NOT TO FLAG (bias hard for signal — noise gets you ignored): - Theoretical risks needing unlikely preconditions. - Defense-in-depth piling when the primary handling is already adequate. - Style/naming nitpicks with no behavioral impact. - "Consider library X" suggestions. - Anything you have not confirmed by actually reading the relevant code. ``` This one block does most of the work. Without it you get the firehose. With it, an empty findings array becomes an acceptable answer for a clean file, and the reviewers do return empty arrays. ## Phase 2 — Verify: Where I Disagreed With Cloudflare It started as a failure. In Cloudflare's design, the coordinator does the judging. It reads all the findings and applies a "reasonableness filter" — drop the speculative ones, and if unsure, read the source. One model, one pass, looking at everything at once. My first version copied that. I had the coordinator look at all the findings and decide. And when I ran it, it kept 43 of 43 findings and dropped nothing. Every single finding survived. That is not verification — that is a model looking at a batch of plausible-sounding claims and anchoring on "well, they all look reasonable." I had built verification theater. The check existed, it ran, it cost tokens, and it verified nothing. The fix was three changes, and each one attacks a specific failure mode: 1. **Independent, not batched.** Each finding is judged by its own agent that sees *only that one finding*. No batch, so it cannot anchor on "these all look plausible together." 2. **Cross-model.** Reviewers run on Sonnet; verifiers run on Opus — a different, stronger model. The check is not a model grading its own homework. 3. **Refute by default.** The verifier's job is to *disprove* the finding from source. Its default stance is that the finding is wrong. It keeps the finding only if it can confirm it by reading the actual code. ```js const v = await agent( `You are an ADVERSARIAL VERIFIER on an independent model. Your default ` + `stance is that the finding below is WRONG. Open the cited file at the ` + `cited location and try to REFUTE it. Set confirmed=false if it is a ` + `false positive, depends on unlikely preconditions, is contradicted by ` + `surrounding code, is pure style, or you cannot reproduce it from the ` + `source. Keep it ONLY if the source genuinely proves it. You see exactly ` + `one finding — judge it on its own merits.\n\n` + `FINDING:\n${JSON.stringify(f, null, 2)}`, { label: `verify:${lens.key}`, model: "opus", schema: VERDICT_SCHEMA }, ); ``` There is a cost discipline here too. Only critical and warning findings get the per-finding Opus refute, because that is where a false positive is expensive. Suggestions pass through unverified and are labelled `verified: false` — logged, never silently dropped. Spending Opus tokens to adversarially verify a "consider renaming this variable" suggestion is not worth it. The Review and Verify phases run as one `pipeline`, not two `parallel` barriers. That matters: the moment the security reviewer finishes, its findings start getting verified, while the architecture reviewer is still reading files. No reviewer sits idle waiting for the slowest one before verification can begin. ## Phase 3 — Coordinate: Dedup and Judge The last phase is pure Cloudflare, and I kept it close to their design because it was already right. One Opus coordinator — their Top tier, for the same reason, it has the hardest job — takes every confirmed finding and does three things: deduplicate (the same issue from two lenses gets kept once, in the best-fit lens), recategorize (a perf issue filed under "quality" moves to "performance"), and apply a final reasonableness filter. Then it renders one consolidated review and makes an approval decision against a rubric lifted almost word-for-word from their post, bias toward approval included: ```txt all clean / only trivial suggestions → approved only suggestions, or warnings, no prod risk → approved_with_comments multiple warnings forming a risk pattern → minor_issues any critical / real safety or security hole → significant_concerns ``` The coordinator gets one extra instruction mine needs and theirs does not: the high-severity findings already survived independent verification, but the suggestions did not, so scrutinize those yourself. The verify phase changes what the coordinator can trust, so it changes what the coordinator has to do. The complete script is on [this GitHub gist](https://gist.github.com/alexanderop/b2493855e0052b89da3f6d962c5b0aa9). The skeleton: ```js export const meta = { name: "ultracode-codebase-review", description: "Cloudflare-style orchestrated AI review of ANY codebase: scope " + "discovery, specialist panel, independent cross-model adversarial " + "verify, coordinator judgment", phases: [ { title: "Scope", detail: "cartographer maps the repo, picks files + lenses" }, { title: "Review", detail: "specialist reviewers fan out over the scope" }, { title: "Verify", detail: "each finding refuted in isolation by a stronger model" }, { title: "Coordinate", detail: "dedup, re-categorize, render single review + decision" }, ], }; // PHASE 0 — Scope: cartographer maps the repo (language-agnostic) phase("Scope"); const scope = await agent(/* … */, { agentType: "Explore", schema: SCOPE_SCHEMA }); const SHARED_CONTEXT = /* built entirely from scope */; // PHASE 1+2 — Review then Verify, pipelined per lens phase("Review"); const perLens = await pipeline( reviewers, // Stage 1 — specialist review on Sonnet (r) => agent(/* … */, { model: "sonnet", agentType: r.agentType, schema: FINDINGS_SCHEMA }) .then((res) => ({ lens: r, review: res })), // Stage 2 — per-finding independent adversarial verify on Opus async (prev) => { const toRefute = prev.review.findings.filter((f) => f.severity !== "suggestion"); const verdicts = await parallel( toRefute.map((f) => async () => { const v = await agent(/* refute-by-default */, { model: "opus", schema: VERDICT_SCHEMA }); return v?.confirmed && v.adjustedSeverity !== "dropped" ? { ...f, severity: v.adjustedSeverity, verified: true } : null; }), ); return { lens: prev.lens.key, confirmed: verdicts.filter(Boolean) }; }, ); // PHASE 3 — Coordinate: dedup, judge, render phase("Coordinate"); const coord = await agent(/* dedup + rubric */, { schema: COORD_SCHEMA }); return { decision: coord.decision, reviewBody: coord.reviewBody, /* … */ }; ``` ## What I Changed From Cloudflare, and Why Side by side, the differences are smaller than they look, and each one comes from a different constraint. | | Cloudflare | My workflow | |:--|:----------|:------------| | **Input** | A merge request diff | A whole repo, any language | | **Discovery** | Diff is the scope | A cartographer maps the scope | | **Orchestrator** | Custom `spawn_reviewers` tool | Built-in `pipeline` / `parallel` | | **Reviewer prompts** | Hand-written markdown files | Claude Code's existing subagents | | **Verification** | Coordinator judges the batch | Independent, cross-model, refute-by-default | | **Runs in** | GitLab CI, Cloudflare Workers | My laptop, one script | The one I would defend hardest is verification. Folding the check into the coordinator is fine when the coordinator is a frontier model with a good "what not to flag" prompt and you have tens of thousands of reviews of tuning behind it. For a workflow I wrote in an evening, a batch judge rubber-stamps. Pulling verification into its own phase — one finding, one fresh stronger model, told to refute — is what turned the output from "43 plausible claims" into a list I actually trust. ## Limitations I'm Honest About This is not Cloudflare's system and it is not pretending to be. A few things I know it does not do: - **It is not in the critical path.** No CI gate, no circuit breakers, no failback when a provider is down. If Opus is overloaded mid-run, a verifier returns `null` and the finding quietly drops. Cloudflare spent real effort on resilience because engineers were blocked on it. I am not blocked on it. - **It costs more per run than a diff review.** Mapping and reviewing a whole repo is more tokens than reviewing a 40-line change. Cloudflare's risk tiers — don't send seven Opus agents to review a typo — are the obvious next thing to steal, and I have not yet. - **It inherits the model's blind spots.** Same honesty Cloudflare ended on: subtle concurrency bugs, cross-system impact, and "is this architecture moving in the right direction" are still beyond a static read of the code. This catches real bugs. It does not replace a human who knows why the system was built this way. ## Conclusion The lesson I keep taking from posts like Cloudflare's is that the impressive part is rarely the part you need. I did not need their plugin architecture or their Workers control plane. I needed three ideas — specialists over one big prompt, tell the model what not to flag, and a coordinator that judges — and Claude Code's workflow primitive handed me the orchestration for free. The piece I am proudest of is the piece where I disagreed. Copying their coordinator-judges design gave me a check that verified nothing. Pulling verification into its own phase, with an independent stronger model told to refute by default, is what made the output trustworthy. If you take one thing from this post, take that: a verification step that never drops anything is not a verification step, it is a comfort blanket. Make your checker try to prove you wrong, on a different model, one claim at a time. The whole script is [in the gist](https://gist.github.com/alexanderop/b2493855e0052b89da3f6d962c5b0aa9) — point it at one of your repos and see what it refuses to flag. --- --- title: How I Became an International Speaker description: I was too scared to publish my first blog post and ended up on stage at Vue Amsterdam. Here's the honest path from zero to international speaker, visibility first. tags: ['personal-development', 'speaking', 'writing'] url: https://alexop.dev/posts/how-i-became-an-international-speaker --- # How I Became an International Speaker Lately people have asked me how to become a speaker themselves, and every time I feel a bit of cringe, because part of me thinks it's fake. It happened at the conferences I spoke at, and also online. That's why I thought I'd write a blog post about it. I also want to reflect a bit on my journey. Hopefully it motivates other people to do the same, because becoming an international speaker was one of the best things I ever did in my life. ## A little background When I was younger I often struggled with what you would now call "anxiety". I was always afraid to talk to random people, and at school I hated having to give a presentation in front of a bigger audience. I think two things changed that: - starting boxing when I was younger - finding something I really love, like coding, which motivated me to share my knowledge ## How I started with writing blog posts I once read a book about career tips for software developers, and one point stuck with me: if you want to stand out as a dev, start writing a blog. So probably in my second year as a developer, I started writing blog posts on dev.to. dev.to is a really good platform to start on, because people see your blog post in their feed and can vote on it. It's the easiest way to start if you don't have any audience yet. What I'd also recommend now is to create a real blog of your own and keep the canonical URL there. ## Find a niche When you start writing technical blog posts, I think it's worth concentrating on one thing. For me that was Vue. The advantage is that you quickly understand which topics aren't covered yet. The easiest way for a blog post to get views is to write about something that hasn't been covered yet and that would be interesting for devs. ## How to grow your blog post It's not enough to only write about interesting things, people also need to know they exist. That's especially hard if nobody knows your blog yet. How I did it: - post on Reddit - tell every newsletter that my blog post exists - share it on my socials ## Get good at content creation So now we know we need to be visible somehow. As I said, I chose blogging, but you could also start with YouTube. It all depends on your personality and how much time and effort you have. Time wise, blogging is definitely easier. Although once you have a good setup and don't want to do fancy things, YouTube can be fast too. There are many books about how to write better, for example: - [Writing for Developers](https://www.manning.com/books/writing-for-developers), I can highly recommend it - [Economical Writing](https://www.amazon.com/dp/022644807X), a good book, but to be honest, with AI I got a bit lazy ## Use your Work for practice Most of my Expierence on preparing a Talk was from work itself. Most of the time there are always sessions where devs can share their knoledge use that and practice ## Now apply apply apply Once you're visible, and of course you can start this in parallel, apply for anything: - find out which meetups in your city cover the topics you share - use [Sessionize](https://sessionize.com/) to find conferences accepting talks Here is the whole path in one picture, from the first blog post I was too scared to publish to the stage at Vue.js Amsterdam. --- --- title: Claude Code Workflows: Deterministic Multi-Agent Orchestration description: Workflows let you script how Claude Code fans out across dozens of subagents, then synthesizes the results. Here's how they work, the primitives, and the small example I built to understand them. tags: ['claude-code', 'ai', 'tooling', 'orchestration'] url: https://alexop.dev/posts/claude-code-workflows-deterministic-orchestration --- # Claude Code Workflows: Deterministic Multi-Agent Orchestration Claude Code shipped **workflows** recently, and the docs describe a lot of machinery: deterministic orchestration, `parallel` and `pipeline`, journaling and resume, adversarial verify patterns. I wanted to understand it rather than skim the feature list, and the way I learn a tool is to build the smallest real thing with it. So I picked a task with an obvious fan-out shape: "what happened in the Vue and Nuxt ecosystem this week." Many independent sources to check, then a merge, then a write-up. I wrote a ~130-line workflow that spawns nine agents in parallel, each scouring a different source, collects their findings into one list, ranks them by impact, and writes a digest. It's a throwaway, but building it taught me how the whole feature fits together. This post is what I learned. A workflow is the newest piece of Claude Code's orchestration story. In my post on agent teams I traced the progression from subagents to teams. Workflows are the next rung, and they solve a different problem than either: when you want the *control flow itself* to be deterministic, not decided turn-by-turn by a model. If you want a sense of the ceiling before the toy example, Jarred Sumner credited dynamic workflows and adversarial code review for porting Bun from Zig to Rust in six days: ## Table of Contents ## Where Workflows Fit Most of the time a single Claude Code session works turn-by-turn: read a file, decide, call a tool, look at the result, decide again. That loop is the right tool for most work. Some jobs don't fit one head and one context window though: - **Comprehensive** jobs: "review every file in this diff", "audit all 40 dependencies". - **Confidence-critical** jobs: "find the bug, then have three independent skeptics try to refute it". - **Scale** jobs: migrations, sweeps, anything bigger than one context can hold. Subagents and agent teams can attack these, but there's a subtle difference in *who holds the plan*. | | Subagents | Agent Teams | Workflows | |:--|:----------|:------------|:----------| | **What it is** | A worker Claude spawns | Independent Claude sessions | A script the runtime executes | | **Who decides what's next** | Claude, turn by turn | Claude and the teammates | The script | | **Where results live** | Claude's context | Each session's context | Script variables | | **What's repeatable** | The worker definition | The team setup | The orchestration itself | | **Scale** | A few per turn | A handful of sessions | Dozens to hundreds of agents | With subagents and skills, Claude is the orchestrator. It decides turn by turn what to spawn, and every result lands back in its context window. A workflow moves the plan into code. The script holds the loop, the branching, and the intermediate results, so Claude's context only ever sees the final answer. That is what lets a workflow scale to hundreds of agents without drowning the conversation. ## The Core Idea A normal agent decides the control flow as it goes. A workflow inverts that. *You* write the control flow as plain code, and each individual step is delegated to a fresh subagent. The orchestration is deterministic; only the work inside each `agent()` call is model-powered. That distinction is the whole point. When you write this: ```js const results = await parallel(files.map((f) => () => agent(`Review ${f}`))); ``` You know exactly one agent runs per file, they all run concurrently, and you get an array back. There are no emergent "the model decided to skip three files" surprises. You get determinism in the orchestration and model judgment inside each step. The shape that keeps showing up is **fan out → reduce → synthesize**: B[agent 1] A --> C[agent 2] A --> D[agent ...] A --> E[agent N] B --> F[reduce: dedupe + rank] C --> F D --> F E --> F F --> G[synthesize: write the result] `} /> Swap the sources and prompts and the same skeleton becomes a market scan, a dependency audit, a code review, or a research report. ## The Example I Built to Learn It Here is the workflow I wrote. I picked the newsletter task because it forces you to use every part of the feature: a wide fan-out, a reduce step, and a synthesis step. Every script starts with a `meta` block that must be a **pure literal**, then a body using the orchestration primitives. ```js export const meta = { name: "vue-newsletter", description: "Research Vue/Nuxt sources in parallel and synthesize a newsletter", phases: [ { title: "Research", detail: "one agent per source" }, { title: "Curate", detail: "dedupe + rank by impact" }, { title: "Write", detail: "synthesize the newsletter" }, ], }; ``` ### 1. Fan out with `parallel()` Nine sources, nine agents, all at once. Each returns structured JSON validated against a schema, so the model retries on mismatch and I never parse free text: ```js phase("Research"); const raw = await parallel( SOURCES.map((s) => () => agent(s.prompt, { label: `research:${s.key}`, phase: "Research", schema: ITEM_SCHEMA, // forces validated structured output agentType: "general-purpose", }), ), ); ``` The `SOURCES` array is just data: one entry per source with a prompt. GitHub core releases, the Nuxt ecosystem, the official blogs, Hacker News, Reddit, dev.to, key people like Evan You and Anthony Fu, and the newsletter/podcast circuit. ### 2. Reduce with plain JavaScript Flattening, deduping, and filtering is just code. No agent needed: ```js const collected = raw.filter(Boolean); // skipped/failed agents become null const flatItems = collected.flatMap((c) => c.items); log(`Collected ${flatItems.length} items`); ``` ### 3. Synthesize with sequential `agent()` calls ```js phase("Curate"); const curated = await agent(curatePrompt, { phase: "Curate", schema: CURATED_SCHEMA }); phase("Write"); const newsletter = await agent(writePrompt, { phase: "Write" }); return { newsletter, itemCount: flatItems.length, curated }; ``` The run I did while testing pulled together a Nuxt UI release, a Vue Router v5 minor, a Vue core patch, and a Madrid conference recap: seventeen items across nine sources in about three minutes. Good enough to convince me the orchestration worked, which was the whole point of building it. ## The Primitives A handful of functions do all the work. **`agent(prompt, opts?)`** spawns one subagent. Without options it returns the agent's final text. The options worth knowing: - `schema`: a JSON Schema. The subagent is forced to return validated structured data. - `label`: the display name in the progress UI. - `phase`: assigns the agent to a progress group. Use it inside `parallel()` and `pipeline()` to avoid racing on the global `phase()` state. - `model`: override the model for this one call. Default is to omit it so the agent inherits your session model. - `agentType`: use a custom subagent type instead of the default workflow agent. - `isolation: "worktree"`: run the agent in its own git worktree. Only when agents write files in parallel and would otherwise conflict. **`parallel(thunks)`** runs tasks concurrently. It is a **barrier**: it waits for every thunk before returning. A thunk that throws resolves to `null` rather than rejecting the whole call, so always `.filter(Boolean)` the results. You can pass a hundred thunks and they'll all complete, but only a handful run at once: concurrency is capped at roughly your core count, and the excess queue. **`pipeline(items, ...stages)`** runs each item through all stages **independently, with no barrier between stages**. Item A can be in stage 3 while item B is still in stage 1. Each stage callback receives `(prevResult, originalItem, index)`. **`workflow(nameOrRef, args?)`** runs *another* workflow inline as a sub-step and returns whatever it returns. Pass a name to invoke a saved workflow, or `{ scriptPath }` to run a script file. This is composition: a research workflow can call `/deep-research` as one of its stages instead of reimplementing the fan-out. The child shares the parent's concurrency cap, agent counter, and token budget, and shows up as its own group in `/workflows`. Nesting is one level deep: a `workflow()` call inside a child throws. ```js // inside a script: hand a sub-question off to the bundled deep-research workflow const report = await workflow("deep-research", { question: topic }); ``` The rest are small helpers: `phase(title)` starts a progress group, `log(msg)` emits a narrator line, `args` carries the JSON you passed in when launching, and `budget` exposes the token target so you can scale depth dynamically (it's `null` when you launch without a target, so guard any loop-until-budget on `budget.total` or it runs to the agent cap). `Date.now()`, `Math.random()`, and an argless `new Date()` all throw inside a workflow. Workflows journal every `agent()` call so a run can resume, and non-determinism would invalidate that cache. If you need a timestamp, pass it through `args`. If you need variety across agents, vary the prompt or label by index. ## `pipeline` vs `parallel`: The Decision That Matters This trips people up, so here is the rule I follow. **Default to `pipeline()`.** Reach for a `parallel()` barrier between stages *only* when a stage needs **all** prior results at once. Legitimate reasons for a barrier: - ✅ Dedupe or merge across the full result set before expensive downstream work. - ✅ Early-exit on the total ("0 findings, skip verification entirely"). - ✅ A prompt that references "the other findings" for comparison. Not legitimate: - ❌ "I need to flatten or filter first." Do it inside a pipeline stage. - ❌ "The stages feel conceptually separate." Separate is not the same as synchronized. - ❌ "It's cleaner code." Barrier latency is real wall-clock waste. The smell test: if you wrote `parallel` → transform → `parallel`, and that middle transform has no cross-item dependency, you should have used a pipeline. The newsletter example *does* use a barrier, and correctly: curation has to see every source before it can dedupe and rank across them. ## Quality Patterns The primitives compose into reusable harnesses. This is the real value over spawning more agents: the structure is what produces confidence. A few I lean on: - **Adversarial verify**: for each finding, spawn N independent skeptics prompted to *refute* it. Kill it unless a majority survive. Stops plausible-but-wrong findings from shipping. - **Perspective-diverse verify**: give each verifier a distinct lens (correctness, security, performance, does-it-reproduce) instead of N identical ones. Diversity catches failure modes redundancy can't. - **Judge panel**: generate N attempts from different angles, score with parallel judges, synthesize from the winner while grafting the best of the runners-up. - **Loop-until-dry**: for unknown-size discovery, keep spawning finders until K consecutive rounds surface nothing new. Here is loop-until-dry with a diverse-lens verify, condensed: ```js const seen = new Set(); const confirmed = []; let dry = 0; while (dry < 2) { const found = (await parallel(FINDERS.map((f) => () => agent(f.prompt, { phase: "Find", schema: BUGS })))).filter(Boolean).flatMap((r) => r.bugs); const fresh = found.filter((b) => !seen.has(key(b))); if (!fresh.length) { dry++; continue; } dry = 0; fresh.forEach((b) => seen.add(key(b))); const judged = await parallel(fresh.map((b) => () => parallel(["correctness", "security", "repro"].map((lens) => () => agent(`Judge "${b.desc}" via the ${lens} lens — real?`, { phase: "Verify", schema: VERDICT }))) .then((vs) => ({ b, real: vs.filter(Boolean).filter((v) => v.real).length >= 2 })))); confirmed.push(...judged.filter((v) => v.real).map((v) => v.b)); } ``` One detail makes or breaks this: dedupe against everything *seen*, not just confirmed results. Otherwise rejected findings reappear every round and the loop never converges. ## A Shipped Example: How `/deep-research` Works My newsletter generator is a toy. If you want to see these patterns in a real, bundled workflow, run `/deep-research`. It takes a question and returns a cited report, and under the hood it's the same **fan out → reduce → synthesize** skeleton with an **adversarial verify** pass bolted on. It's the quality pattern from the section above, running in production. When you launch it the workflow announces its plan and runs in the background while you keep working: It moves through five phases: 1. **Scope:** one agent decomposes your question into five distinct search angles, so the searches don't all chase the same wording. 2. **Search:** five web searches run in parallel, one per angle. This is the fan-out. 3. **Fetch:** dedupe the URLs across angles, pull the top ~15 sources, and extract individual claims from them. 4. **Verify:** the interesting part. Each claim gets an adversarial three-vote check, with skeptics trying to refute it. Claims that don't survive never reach the report. 5. **Synthesize:** one final agent writes the cited report from the claims that held up. Map that onto the primitives and you can almost see the script: a single `agent()` for scope, a `parallel()` fan-out for the five searches, plain JavaScript to dedupe in fetch, a per-claim verify pass (the same `parallel()` of skeptics from the loop-until-dry example), and a closing `agent()` to synthesize. The phases show up in `/workflows` as named groups (`Scope 1/1`, `Search 0/5`, `Fetch`, `Verify`, `Synthesize`), each with its own agent count, token total, and elapsed time, so you can drill into any single search or verification and read its prompt and result. This is the difference between "ask Claude to research something" and a workflow. A single agent doing web research holds every half-read source in one context and never checks its own claims. `/deep-research` decomposes the search so coverage is wide, keeps the intermediate sources out of your conversation, and runs a verification pass a single turn-by-turn agent would never run against itself. ## Triggering and Watching a Run Worth saying plainly: from Claude Code's side, a workflow is **a tool**. There's a `Workflow` tool the same way there's a `Read` or `Bash` tool, and "running a workflow" means Claude calls that tool with a script. The runtime executes the script in the background while your session stays responsive, which is why you can keep chatting while dozens of agents churn away. There are a few ways a workflow gets written and launched: 1. **Say "workflow" in your prompt.** Include the word and Claude writes a workflow script for the task instead of working through it turn by turn. 2. **Run a saved or bundled command.** A workflow you saved to the project, or the built-in `/deep-research` covered above. 3. **Turn on `ultracode`.** Claude plans a workflow for every substantial task in the session. Run a workflow to audit every API endpoint under src/routes/ for missing auth checks. Spawn one agent per route file, then have a second pass verify each finding before reporting. When a run does what you wanted, you can save it: Claude Code writes the script into **`.claude/workflows/`** in your repo as a `.js` file (the appendix below is exactly that file for my newsletter). Because it lives in the repo, it's version-controlled and anyone who clones it can launch it by name and pass arguments: Run the vue-newsletter workflow with args `{"weekStart":"2026-06-04","weekEnd":"2026-06-11"}` Runs happen in the background, and **`/workflows`** is how you watch them: it lists every run, including which ones are **currently running**, and opens a progress view showing each phase with its agent count, token total, and elapsed time. You can drill into a phase, then into a single agent, to read its prompt and result, pause or stop a run, or press `s` to save a good one's script as a reusable `/` command under `.claude/workflows/`. ## When to Reach for One verification, or scale?] -->|No| B[Single session
or a subagent] A -->|Yes| C[Do you want the control flow
to be deterministic and repeatable?] C -->|No| D[Agent team] C -->|Yes| E[Write a workflow] `} /> **Good fit** ✅ - Decomposing a job so every part is covered in parallel (audits, reviews, sweeps). - Anything you want to *re-run* with the same structure (a weekly competitor scan, a release checklist). - Confidence-critical work where a repeatable verify or judge pass beats one model's first answer. **Bad fit** ❌ - An ordinary task one agent can do turn by turn. Let one agent do it. - Work that needs you to weigh in between every stage. A workflow can't take mid-run input; only agent permission prompts pause it. - Anything where the token cost of dozens of agents isn't justified by breadth or scale. A workflow spawns many agents, so one run can use meaningfully more tokens than doing the same task in conversation, and it counts toward your plan's usage. Every agent uses your session's model unless the script routes a stage elsewhere, so check `/model` before a large run and consider routing cheap stages to a smaller model. ## Conclusion - A workflow is a JavaScript script that orchestrates subagents deterministically. You own the control flow, agents do the thinking, and the plan lives in code so the conversation only sees the final answer. - The shape that generalizes is **fan out → reduce → synthesize**. The newsletter generator I built is a deliberately small instance of it. - `agent()` runs one (use a `schema` for validated structured output), `parallel()` is a barrier, `pipeline()` streams items through stages with no barrier. **Default to pipeline.** - The leverage is the repeatable quality patterns: adversarial verify, diverse lenses, judge panels, loop-until-dry. - It is opt-in and token-hungry. Reach for it when a job needs breadth, independent verification, or scale a single context can't hold. Otherwise let one agent do the work. If you've already worked through subagents and skills, workflows are the natural next tool. The fastest way to understand them is the same way I did: pick a small task with a clear fan-out shape and build the throwaway version. Mine was a newsletter generator I won't run again. The point was never the newsletter; it was seeing how the pieces fit, so that when a job needs breadth or verification, reaching for a workflow is obvious. --- ## Appendix: The Full Script Everything above is excerpts. Here is the complete `.claude/workflows/vue-newsletter.js` in one piece, so you can see how the `meta` block, the schemas, the source list, and the three phases fit together. It's plain JavaScript: no imports, no filesystem access, inputs via `args`, results via `return`. ```js export const meta = { name: 'vue-newsletter', description: 'Research Vue/Nuxt ecosystem sources in parallel for a given week and synthesize a newsletter', whenToUse: 'Generate a weekly Vue/Nuxt newsletter. Pass args {weekStart, weekEnd, label} as ISO dates (e.g. {"weekStart":"2026-05-21","weekEnd":"2026-05-28"}). With no args, agents cover the past 7 days from today.', phases: [ { title: 'Research', detail: 'one agent per source — releases, blogs, social, people' }, { title: 'Curate', detail: 'dedupe + rank items by impact' }, { title: 'Write', detail: 'synthesize the final newsletter' }, ], } // Args are optional. Pass {weekStart, weekEnd, label} as ISO dates to scope a specific week. // With no args, agents are told to cover "the past 7 days from today" (they resolve the date via web search). const hasRange = args && args.weekStart && args.weekEnd const weekStart = hasRange ? args.weekStart : null const weekEnd = hasRange ? args.weekEnd : null const label = (args && args.label) || (hasRange ? `Week of ${weekStart}–${weekEnd}` : 'this week') const window = hasRange ? `between ${weekStart} and ${weekEnd}` : 'within the past 7 days from today' const ITEM_SCHEMA = { type: 'object', additionalProperties: false, properties: { source: { type: 'string' }, items: { type: 'array', items: { type: 'object', additionalProperties: false, properties: { title: { type: 'string' }, url: { type: 'string' }, summary: { type: 'string', description: '1-3 sentence plain summary of what changed / why it matters' }, category: { type: 'string', enum: ['release', 'article', 'tooling', 'discussion', 'tutorial', 'people', 'other'] }, date: { type: 'string', description: 'ISO date if known, else empty' }, impact: { type: 'string', enum: ['high', 'medium', 'low'] }, }, required: ['title', 'url', 'summary', 'category', 'impact'], }, }, }, required: ['source', 'items'], } // Each source is researched by its own agent in parallel. const SOURCES = [ { key: 'core-releases', prompt: `Find releases/changelogs published ${window} for these GitHub repos: vuejs/core, vuejs/router (vue-router), vuejs/pinia, vueuse/vueuse, vitejs/vite, vitejs/vitest. For each new release in that window, give the version, the highlights, and the release URL. Skip anything outside the date window.`, }, { key: 'nuxt-releases', prompt: `Find releases/changelogs published ${window} for the Nuxt ecosystem on GitHub: nuxt/nuxt, nuxt/ui, nuxt/image, nuxt/content, unjs/nitro, unjs/h3. Give version, highlights, and URL for each release in that window only.`, }, { key: 'vue-blog', prompt: `Check the official Vue.js blog (blog.vuejs.org) and Vue.js news for posts published ${window}. Summarize each post with its URL.`, }, { key: 'nuxt-blog', prompt: `Check the official Nuxt blog (nuxt.com/blog) for posts published ${window}. Summarize each with URL.`, }, { key: 'hackernews', prompt: `Search Hacker News (news.ycombinator.com) for stories about Vue, Nuxt, Vite, or Pinia that were active/posted ${window}. Include the HN discussion URL and the linked article. Note points/comments if visible.`, }, { key: 'reddit', prompt: `Search Reddit r/vuejs and r/Nuxt for notable threads posted ${window} — announcements, releases, popular discussions, showcased projects. Give the reddit thread URL for each.`, }, { key: 'devto', prompt: `Search dev.to for the most useful Vue and Nuxt tagged articles published ${window} (tutorials, deep-dives, tips). Give URLs.`, }, { key: 'people', prompt: `Look for notable updates, posts, or talks ${window} from key Vue/Nuxt people: Evan You (@youyuxi / VoidZero), Daniel Roe (Nuxt lead), Anthony Fu (VueUse/Vitesse/Slidev), Eduardo San Martin Morote (posva — router/pinia), Sébastien Chopin (Nuxt/NuxtLabs). Include VoidZero and NuxtLabs company news too. Give URLs.`, }, { key: 'newsletters-podcasts', prompt: `Find Vue/Nuxt newsletter issues and podcast episodes published ${window}: Vue.js Newsletter (news.vuejs.org), This Week in Vue, Michael Thiessen's newsletter, DejaVue podcast, Deox/Vue Mastery content. Summarize and give URLs.`, }, ] phase('Research') const raw = await parallel( SOURCES.map((s) => () => agent( `You are researching the Vue.js / Nuxt ecosystem for a weekly newsletter covering ${label} (${window}).\n\n${s.prompt}\n\nUse web search and fetch real URLs. Only include items genuinely within the date window. Return real, verifiable URLs — never invent links. If you find nothing in the window, return an empty items array. Set impact based on how much the average Vue developer should care.`, { label: `research:${s.key}`, phase: 'Research', schema: ITEM_SCHEMA, agentType: 'general-purpose' }, ), ), ) const collected = raw.filter(Boolean) const flatItems = collected.flatMap((c) => (c.items || []).map((it) => ({ ...it, source: c.source }))) log(`Collected ${flatItems.length} items across ${collected.length} sources`) phase('Curate') const CURATED_SCHEMA = { type: 'object', additionalProperties: false, properties: { highlights: { type: 'array', items: { type: 'string' }, description: '2-4 sentence TLDR bullets of the biggest stories this week' }, items: { type: 'array', items: { type: 'object', additionalProperties: false, properties: { title: { type: 'string' }, url: { type: 'string' }, summary: { type: 'string' }, category: { type: 'string' }, impact: { type: 'string' }, }, required: ['title', 'url', 'summary', 'category', 'impact'], }, }, }, required: ['highlights', 'items'], } const curated = await agent( `Here are raw newsletter candidate items gathered from multiple sources for the Vue/Nuxt week of ${label}:\n\n${JSON.stringify(flatItems, null, 2)}\n\nCurate them:\n1. Remove duplicates (same release/article surfaced by multiple sources — keep the best canonical URL).\n2. Drop low-quality, off-topic, or spammy entries.\n3. Rank by impact (high first).\n4. Write 3-5 punchy "highlights" bullets capturing the week's biggest stories.\nKeep every URL exactly as provided — do not fabricate or alter links.`, { phase: 'Curate', schema: CURATED_SCHEMA }, ) phase('Write') const newsletter = await agent( `Write a polished weekly Vue.js / Nuxt newsletter in Markdown for ${label}.\n\nUse this curated data:\n${JSON.stringify(curated, null, 2)}\n\nStructure:\n- A title with the week range and a one-paragraph intro setting the tone.\n- "📌 This Week's Highlights" — the highlights bullets.\n- "🚀 Releases" — version bumps with what changed (group Vue core + Nuxt + tooling).\n- "📝 Articles & Tutorials".\n- "🛠️ Tooling & Ecosystem".\n- "💬 Community & Discussion".\n- "👤 From the Core Team & Community" — people/company news.\n- A short friendly sign-off.\n\nEvery item must be a markdown link to its real URL. Keep summaries tight and developer-focused. Omit any empty section. Output ONLY the markdown newsletter.`, { phase: 'Write' }, ) return { newsletter, itemCount: flatItems.length, curated } ``` --- --- title: How to do AFK Coding description: A pipeline for shipping a 5-point ticket while you're away from the keyboard: spec with the human in the loop, slice into vertical tickets, run a Ralph loop with TDD per slice, refactor, then let an agent-browser do QA. tags: ['ai', 'claude-code', 'agentic-coding', 'ralph', 'spec-driven-development'] url: https://alexop.dev/posts/how-to-do-afk-coding --- # How to do AFK Coding ## The Pipeline The workflow I use to ship a 5-point ticket while I'm away from the keyboard:
Six phases. The first and last are human, the middle four run on their own. ``` 1. Align on spec ── HITL ── business + dev + AI interviewer 2. Slice the ticket ── AFK ── vertical slices, one per sub-ticket 3. Ralph loop per slice ── AFK ── fresh context, TDD red-green-refactor 4. Refactor pass ── AFK ── the step LLMs always skip 5. Agentic QA ── AFK ── agent-browser drives the real UI 6. Review ── HITL ── dev reviews, business does UAT ``` Two questions first: when this pipeline pays off, and what "AFK" looks like in practice. ## Two Kinds of Tickets
Most tickets fit in one Claude Code session. Paste it in, watch it land, ship the PR. Opus 4.7 or GPT 5.5 close out a 1–3 pointer with maybe one nudge. No pipeline needed. The interesting tickets are the 5-pointers. Frontend and backend, a multi-step wizard with three API calls, server-side validation, optimistic UI, and a test plan that catches the failure modes you care about. Paste that into a fresh chat and one of three things happens: 1. **Context runs out.** The model hits the window halfway through and auto-compacts. The summary drops the bit you needed. As [HumanLayer's 12 Factor Agents](https://www.humanlayer.dev/blog/12-factor-agents) puts it: _"Even as models support longer and longer context windows, you'll ALWAYS get better results with a small, focused prompt and context."_ 2. **No refactoring.** LLMs default to append, not restructure. Ship five of these in a row and your codebase looks like the inside of a hoarder's garage. 3. **Silent skip.** When a test fails on round four, the model deletes the test rather than fixing the bug. Opus does this often. Michael Arnaldi caught it taking the same shortcut in his Effect workshop: _"sometimes a feature, sometimes a bug — takes shortcuts, sleeps, removes failing tests rather than fixing them."_ This is a workflow problem, not a model problem. Big tickets need a pipeline, not a prompt. The one diagrammed above. ## What AFK Coding Looks Like
AFK = away from keyboard. HITL = human in the loop. Sounds opposed; isn't. The good workflow is **HITL at the edges, AFK in the middle**. - **HITL up front.** A human aligns on what to build. Judgment lives here. - **AFK in the middle.** Once the spec is solid, the agent ships it. You go for a walk. - **HITL at the end.** Review the PR, eyeball the QA report, ship. You don't write less code. You run three pipelines in parallel while reading a book. You manage instead of coding. I wrote about the broader shift in The Software Factory; this post zooms into the pipeline that powers a single ticket inside it. ## Phase 1: Align on the Spec (HITL) Don't skip this. Every problem downstream starts here. Dex Horthy puts the cost in stark terms in [Advanced Context Engineering for Coding Agents](https://github.com/humanlayer/advanced-context-engineering-for-coding-agents): > "A flawed specification cascades through hundreds of lines; flawed research generates thousands of bad lines." A 5-pointer touches enough surface area that "what we want" needs to be explicit. Sit with the business, talk through the wizard, draw the state machine on a napkin. Then write a real PRD. A Jira description doesn't count. Use the AI here too, but in **interview mode, not generation mode**. Open Claude Code with a spec template and let it ask _you_ the questions you'd otherwise forget: i want to build a multi-step booking wizard. interview me with one question at a time until you have enough to write a PRD. cover: goals, user, happy path, edge cases, validation, error states, and out-of-scope. write the PRD to docs/prd-booking-wizard.md when done. The PRD is the single artifact the rest of the pipeline reads. Skimp here and the agent will ship the wrong thing five times in parallel. Spec-Driven Development with Claude Code in Action covers the prompt patterns I lean on; GitHub's [Spec Kit](https://github.blog/ai-and-ml/generative-ai/spec-driven-development-with-ai-get-started-with-a-new-open-source-toolkit/) ships a four-phase `/specify → /plan → /tasks → /implement` workflow that encodes the same idea if you'd rather not roll your own. ## Phase 2: Slice Into Vertical Tickets (AFK) The PRD is too big for one Ralph loop. Slice it. The right unit isn't "frontend task" + "backend task" + "tests task." That's a horizontal slice, and horizontal slices kill agents. Every slice needs every other slice to be useful, so nothing works until everything works. Compaction guaranteed. A **vertical slice** is a thin end-to-end strip of behaviour: one API endpoint, the one form step that consumes it, and the one test that proves it works. Each vertical slice is independently shippable. Each one fits in a Ralph loop. Each one survives if its sibling slice fails.
This is Factor 10 of [12 Factor Agents](https://www.humanlayer.dev/blog/12-factor-agents) applied to ticket scoping: _"Small, focused agents — keep agents to 3–20 steps; avoid monolithic designs."_ A vertical slice is the ticket-shaped equivalent. Steve Yegge frames the alternative bluntly in [2026: The Year The IDE Died](https://www.youtube.com/watch?v=7Dtu2bilcFs): > "Nature builds ant swarms and Claude Code built this huge muscular ant that's just going to bite you in half and take all your resources." For our wizard: ``` Slice 1: Step 1 — guest info (form + POST /booking/draft + e2e test) Slice 2: Step 2 — room selection (form + GET /rooms + e2e test) Slice 3: Step 3 — payment (form + POST /booking/confirm + e2e test) Slice 4: Wizard state machine (cross-step navigation + e2e test) ``` Four sub-tickets. Each is a 1–2 pointer. Each is something Opus 4.7 _can_ ship in one go. Have the agent do the slicing for you: read docs/prd-booking-wizard.md. break it into vertical slices. each slice must be end-to-end (UI + API + test) and independently shippable. write the slices to docs/tickets/01-*.md, 02-*.md, ... ## Phase 3: Ralph Loop Per Slice (AFK)
For each slice, spin up a fresh agent in a Ralph loop. Ralph is [Geoffrey Huntley's technique](https://ghuntley.com/ralph/): a `while` loop that pipes a prompt file into a fresh `claude` invocation, over and over, until a stopping condition trips. Every iteration starts with an empty context, reads the prompt file, picks the highest-priority unfinished task, ships it, commits, and exits. ```bash while :; do cat PROMPT.md | claude --dangerously-skip-permissions done ``` Huntley's framing of why it works: > "That's the beauty of Ralph — the technique is deterministically bad in an undeterministic world." You aren't aiming for one perfect run. You're aiming for many cheap runs that converge, with the prompt file as the ratchet. He shipped six repos overnight at a YC hackathon and replaced a \$50K outsourcing contract with \$297 of API spend. The reason the loop works is the one Dex Horthy chronicles in [A Brief History of Ralph](https://www.humanlayer.dev/blog/brief-history-of-ralph): context windows degrade as they fill. Resetting between iterations keeps every task in the model's "smart zone." Factory's write-up on [The Context Window Problem](https://factory.ai/news/context-window-problem) makes the same case from the engineering side. Bigger windows aren't the answer; smaller scopes are.
A small driver script gives each slice its own Ralph instance, in its own git worktree, with its own log file: ```bash for ticket in docs/tickets/*.md; do name=$(basename "$ticket" .md) git worktree add "../wt-$name" -b "feat/$name" ( cd "../wt-$name" cp "$ticket" PROMPT.md while :; do cat PROMPT.md | claude --dangerously-skip-permissions grep -q "^- \[ \]" PROMPT.md || break # exit when no unchecked items done ) & done wait ``` Four slices, four worktrees, four agents, in parallel. Matt Pocock's Sandcastle ([covered here](https://www.youtube.com/watch?v=E5-QK3CDVQM)) is the productionised version of this, with sandboxing on top so the agents can't `rm -rf` your home directory. The cautionary tale is real. See [Claude Code is Amazing... Until It DELETES Production](https://www.youtube.com/watch?v=VqDs46A8pqE). YOLO mode without a sandbox is how you wake up to an empty home folder. ### TDD Inside the Loop
The Ralph prompt for each slice forces red-green-refactor inside every iteration: you are working on ONE slice. read PROMPT.md. for each unchecked task: 1. red: write a failing e2e test that asserts the behaviour 2. green: write the minimum code to pass the test 3. refactor: clean up; tests must stay green 4. commit 5. tick the box in PROMPT.md exit when all boxes are ticked. Two reasons TDD matters more for AFK agents than for humans: - **Without tests, the agent rips the feature out and you never notice.** A passing test it didn't write is a contract; a green CI is the agent's permission slip to commit. - **Red-green-refactor is the only refactor the agent will do without supervision.** Even that's not enough in practice. That's why Phase 4 exists. Moss Ebeling calls this [back pressure](https://banay.me/dont-waste-your-backpressure): automated feedback that lets the agent self-correct without you in the loop. If your tests, types, and lint don't break when something is wrong, the agent doesn't know it's wrong. Every minute you spend tightening backpressure is a minute you don't spend reviewing trivial mistakes by hand. ## Phase 4: Refactor Pass (AFK)
LLMs cheat on refactoring. Even with red-green-refactor in the loop, the "refactor" step usually shrinks to "rename a variable." The deeper work (extracting a composable, collapsing duplication across the four slices, killing the dead branches) never happens. Leave a single `any` lying around and Opus will copy it everywhere ([Arnaldi's observation](https://www.youtube.com/watch?v=Wmp2Tku2PrI)). This is why I built `/simplify`: a Claude Code skill that runs an Opus subagent over the recently changed files with one job. Clean it up without changing behaviour. Tests stay green, naming improves, ternaries unfold, abstractions collapse. It's the dedicated refactor pass the implementation loop won't do on its own. It ships as a Claude Code skill, but the pattern is portable — if you're on Codex or another agent, you can wire up the same thing yourself. Run it as a separate agent against the merged branch with a narrower brief: run on the current branch. tests must stay green throughout. for each of: duplication, long files, primitive obsession, dead code, and inconsistent naming — find one offender, fix it, run tests, commit. repeat until you can't find more. do not change behaviour. do not add features. Then add lint-as-backpressure rules so the next slice doesn't undo the cleanup. My ESLint setup covers the rules I lean on; Michael Arnaldi's "clone the repo" workflow takes the same idea further. Every shortcut you catch the agent taking, ban it with a rule. ## Phase 5: Agentic QA (AFK)
Tests prove the unit works. They don't prove the user can complete the wizard. For that, drive a real browser. [`agent-browser`](https://github.com/vercel-labs/agent-browser) is a Rust CLI that exposes the browser's accessibility tree as a snapshot with stable element refs (`@e1`, `@e2`, …). The agent reads the snapshot, picks a ref, and clicks. No CSS selectors, no Playwright API to learn. Shell commands the agent already knows how to run. A QA agent runs the user journey end-to-end: you are a QA engineer. the booking wizard is at localhost:3000/book. run the happy path: complete all three steps with valid data. then run two negative paths: invalid email at step 1, declined card at step 3. use agent-browser. take a screenshot at every state. write a markdown report to qa/booking-wizard.md with one section per test: steps taken, expected, actual, pass/fail, and the screenshot. You wake up to a markdown report with screenshots. If anything is red, the report tells you which step. If everything is green, the PR is ready for human eyes. ## Phase 6: Human Review (HITL) The agent opens the PR. Now you do the job. - **Dev review.** Read the diff. Read the test names. Do they describe behaviour the user cares about? Run the app locally for ten minutes. The bar is "would I have shipped this myself," not "did the agent do what I asked." - **Business UAT.** Send the PR's preview URL to the person who wrote half the PRD with you. They click through the wizard. They find the thing the agent never could have known to check. The 30 minutes on the spec and the 30 minutes on the review are 80% of the value. The four hours of coding in between are the part you outsourced. As Linear frames it in [their agent SDK approach](https://linear.app/now/our-approach-to-building-the-agent-interaction-sdk): _"An agent cannot be held accountable."_ Issues get _delegated_ to agents but _assigned_ to humans. Review is the part of the job that doesn't have an AFK version. ## Why This Pipeline Beats "Just Use a Bigger Model" You could wait for the next model with a 10M-token window and pretend the problem goes away. It won't. [Bigger windows degrade](https://factory.ai/news/context-window-problem): attention thins, costs balloon, and the model still won't refactor on its own. The pipeline is mostly _the work humans always did_, automated: - A spec, because nobody ships a 5-pointer without one. - Vertical slices, because Mike Cohn was right about user stories twenty years ago. - TDD, because red-green-refactor is the only loop that catches regressions cheaply. - Code review, because two pairs of eyes are still better than one set of weights. - QA, because tests don't catch UX. What changed: the typing happens AFK. You only show up where judgment lives. For 1–3 pointers, skip all of this. One Claude Code session, one prompt, one PR. The pipeline is overhead for small work; it pays back at 5 points and up. ## The Job Changes A developer's day in this model: - 30 minutes: interview-driven spec for ticket A - 5 minutes: kick off Ralph loops for ticket A - 30 minutes: interview-driven spec for ticket B - 5 minutes: kick off Ralph loops for ticket B - 1 hour: review yesterday's PR, check the QA report - Repeat You're a manager of a small fleet of agents. Each ticket is a project; each Ralph loop is a contractor. The skill is your specs, your reviews, and your taste, not your typing speed. This isn't theoretical. Boris Cherny, creator of Claude Code, runs [15+ parallel Claude sessions on a normal day](https://www.youtube.com/watch?v=AmdLVWMdjOk): five in numbered terminal tabs and another five-to-ten on claude.ai/code, with iTerm notifications when one needs input. Anthropic's headcount tripled and productivity per engineer climbed 70%. Roughly 80–90% of Claude Code is now written by Claude Code itself. Lee Robinson, after building three production projects without writing code by hand, [puts it bluntly](https://www.youtube.com/watch?v=UrNLVip0hSA): > "Writing code was never really the bottleneck, especially for larger projects." Boris reaches the same conclusion from inside Anthropic: > "We're going to start to see the title of software engineer go away. It's just going to be 'builder' or 'product manager.'" The pipeline isn't novel. Stripe's Minions, Steve Yegge's Gas Town, Pocock's Sandcastle all encode variants of it. The novel part: the building blocks (Claude Code in headless mode, agent-browser, Ralph, git worktrees) are open and free. You can run this pipeline tonight on your own machine.
Simon Willison flagged the dark side of all this in [AI Doesn't Reduce Work — It Intensifies It](https://simonwillison.net/2026/Feb/9/ai-intensifies-work): the low cost of "just one more prompt" turns into compulsive task-stacking and burnout. The point of AFK coding is to be _away from the keyboard_, not to triple-book your attention. Run as many pipelines as your review capacity can absorb. Not more. ## Conclusion - Small tickets fit a single chat. 5-point tickets need a pipeline. - The pipeline is HITL up front, AFK through spec → slice → Ralph loop with TDD → refactor → agentic QA, HITL at review. - Vertical slices keep each Ralph loop in the model's smart zone and survive partial failure. - Refactor needs its own dedicated pass. The agent won't do it inside the implementation loop. - Agent-browser closes the gap between "tests pass" and "the user can finish the wizard." - Run several pipelines in parallel. You're managing now, not typing. ## Further reading The notes I leaned on for this post, grouped by what they support: - **Why a pipeline beats a prompt:** [12 Factor Agents](https://www.humanlayer.dev/blog/12-factor-agents) · [The Context Window Problem](https://factory.ai/news/context-window-problem) · [Advanced Context Engineering for Coding Agents](https://github.com/humanlayer/advanced-context-engineering-for-coding-agents) - **Spec phase:** [Spec-Driven Development with AI](https://github.blog/ai-and-ml/generative-ai/spec-driven-development-with-ai-get-started-with-a-new-open-source-toolkit/) · [A Practical Guide to Writing Technical Specs](https://stackoverflow.blog/2020/04/06/a-practical-guide-to-writing-technical-specs/) · Spec-Driven Development with Claude Code in Action - **Ralph loop:** [Ralph Wiggum as a Software Engineer](https://ghuntley.com/ralph/) · [A Brief History of Ralph](https://www.humanlayer.dev/blog/brief-history-of-ralph) · [Stop Chatting with AI. Start Loops.](https://lukeparker.dev/stop-chatting-with-ai-start-loops-ralph-driven-development) - **TDD + backpressure:** [Essential AI Coding Feedback Loops for TypeScript Projects](https://www.aihero.dev/essential-ai-coding-feedback-loops-for-type-script-projects) · [Don't Waste Your Back Pressure](https://banay.me/dont-waste-your-backpressure) - **Refactor pass:** [Vibe Engineering Effect Apps (Michael Arnaldi)](https://www.youtube.com/watch?v=Wmp2Tku2PrI) - **QA:** [agent-browser](https://github.com/vercel-labs/agent-browser) · [Claude Code is Amazing... Until It DELETES Production](https://www.youtube.com/watch?v=VqDs46A8pqE) - **The job changes:** [Boris Cherny on building at Anthropic](https://www.youtube.com/watch?v=AmdLVWMdjOk) · [AI codes better than me. Now what?](https://www.youtube.com/watch?v=UrNLVip0hSA) · [2026: The Year The IDE Died](https://www.youtube.com/watch?v=7Dtu2bilcFs) · [I Open-Sourced My Own AFK Software Factory](https://www.youtube.com/watch?v=E5-QK3CDVQM) · The Software Factory - **The caveat:** [AI Doesn't Reduce Work — It Intensifies It](https://simonwillison.net/2026/Feb/9/ai-intensifies-work) --- --- title: How to Write UI Components That Stay Flexible description: Most components break under their second or third variant — props pile up, shells get copy-pasted. Here's how compound components (the pattern behind Reka UI and shadcn-vue) keep a Dialog flexible without flag soup. tags: ['vue', 'design-patterns', 'architecture'] url: https://alexop.dev/posts/compound-components-in-vue-shadcn --- # How to Write UI Components That Stay Flexible ## Table of Contents ## TLDR A Dialog is a great test case for compound components because every product needs three or four versions of one: confirm, edit, share, settings. The naive paths both fail: copy a monolithic dialog into every variant and the shells drift; collapse them into one god component and the props turn into `mode`, `showHeader`, `showCancel`, `confirmVariant`, `headerCentered`. The compound pattern Reka UI uses (and shadcn-vue ships) puts state into a `provide` / `inject` context and exposes small primitives (``, ``, ``, ``, ``, ``) that the consumer arranges into the variant they need. The tree expresses the variant; nothing inside the component branches on flags. This post grew out of Fernando Rojo's talk [Composition Is All You Need](https://www.youtube.com/watch?v=4KvbVq3Eg5w&t=1194s). Watch it first if you have 30 minutes. This post is the Vue translation, with Dialog as the running example. The whole pattern fits in one snippet: the tree shape is the variant (**STRUCTURE**), `class="..."` overrides defaults (**STYLE**), `data-state="..."` exposes lifecycle as a stylable contract (**STATE**), and `as-child` lets the consumer swap the rendered tag (**ELEMENT**). The rest of the post is each lever, in detail. ## See It Before You Read About It Pick a preset. Toggle the children. The dialog on the left and the matching template on the right move together: same primitives, different trees, different variants. That is the pattern. One root, a few small children. The consumer arranges them, and the arrangement is the variant. ## Smell #1: Duplicated Monoliths Imagine you write a Dialog the first way most of us do: as one self-contained component. It opens, it closes, it has a header and a footer, you ship it. A week later product asks for an "edit profile" dialog. You copy the file, change the body to a form, ship it. A month later: a "share" dialog with a wider layout. Copy. A confirm-delete with a destructive action. Copy. [`MonolithicConfirmDeleteDialog.vue`](https://github.com/alexanderop/compose) shows one of those copies in practice: ```vue ``` Read that comment: `FOOTER: keep in sync with MonolithicProfileEditDialog.vue`. That is the smell. Every variant rebuilds the same shell (the overlay, the escape handler, the scroll lock, the title/description ARIA wiring, the footer markup), and the only thing keeping them aligned is grep and discipline. You pay the cost every time something has to change: - Designer wants `gap-2` to become `gap-6` in every dialog footer? Edit N files. - Accessibility audit asks for a focus trap? Add it to N files. - Scroll lock turns out to need a body class instead of inline style? Touch N files. - New variant (share dialog) appears? Copy the shell again, drift inevitable. Each copy is a fork. Each fork ages on its own. Six months in, you can't be sure the dialogs behave the same way, because they don't. ## Smell #2: One God Component With v-ifs The obvious next move is to collapse the duplicates into one component. You make the body a slot, expose `title` and `description` as props, add `showCancel` and `showConfirm` for the footer. It feels clean for one afternoon. Then product asks for a destructive variant; you add `confirmVariant`. Then a share dialog needs no description; you add `showDescription`. Then someone wants a centered header; `headerCentered`. You add a flag for every variant you cram into the component. [`BooleanPropDialog.vue`](https://github.com/alexanderop/compose) shows the version that grew out of covering two real call sites (confirm-delete and edit-profile) with one component: ```vue ``` The call site is the giveaway: ```vue ``` Twelve flags to describe one variant. The edit version needs `mode="edit"`, then plumbs the form's two-way binding through `update:editName` and `update:editBio` because the inputs live inside the dialog the consumer can't reach into. Fernando Rojo named this smell in his talk: > "If you have a boolean prop that determines which component tree is getting rendered from the parent, you can kind of imagine me looking over your shoulder and shaking my head." > > — Fernando Rojo, *Composition Is All You Need* The diagnostic: when a flag changes *what* renders rather than *how*, lift it into a child component. `confirmVariant="destructive"` is a "how" prop and is fine. `mode="edit"` is a "what" prop. It switches the component tree, so lift it into the tree the consumer controls. ## The Solution: Compose Trees, Don't Configure Them The compound component pattern flips the relationship. You ship N small components that share state through a provider. The consumer assembles only the parts they need: ```html Delete account Are you absolutely sure? This cannot be undone. Cancel ``` A share dialog with no description? Drop ``. An edit dialog with a form? Put the form inside ``. The inputs are in the consumer's template, so they bind to the consumer's state with `v-model`. A settings menu where the trigger is a list-row? Pass `as-child` to ``. No flags inside the component, no `update:editName` plumbing across the boundary. > "Where are the booleans and the special props telling us what to render? They're nowhere to be found. We don't have a monolith — we have shared internals that get reimplemented for each use case." > > — Fernando Rojo This is how Reka UI is built and how shadcn-vue ships every primitive on top of it. ## Reading shadcn-vue: The Dialog Primitive Installing the Dialog from shadcn-vue gives you a barrel of small files, not one `` component: ```ts // ui/dialog/index.ts export { default as Dialog } from "./Dialog.vue"; export { default as DialogClose } from "./DialogClose.vue"; export { default as DialogContent } from "./DialogContent.vue"; export { default as DialogDescription } from "./DialogDescription.vue"; export { default as DialogFooter } from "./DialogFooter.vue"; export { default as DialogHeader } from "./DialogHeader.vue"; export { default as DialogOverlay } from "./DialogOverlay.vue"; export { default as DialogTitle } from "./DialogTitle.vue"; export { default as DialogTrigger } from "./DialogTrigger.vue"; ``` Nine components, each doing one thing. The root that owns the state, `Dialog.vue`, looks like this: ```vue ``` The root renders no markup of its own. It forwards into Reka UI's `DialogRoot`, which establishes the context. The open/closed state lives there and children read it via `inject`. `DialogHeader` is a styled wrapper with no logic: ```vue ``` The trigger is a few lines too. It forwards to Reka's `DialogTrigger`, which calls `setOpen(true)` and wires up the ARIA attributes. No DOM owned by shadcn-vue, only styling and the `data-slot` for theming. Compare the call site (the compound version we saw above) to the boolean-prop monolith: same UX, twelve flags fewer, and when a designer asks for a checkbox in the footer or a dialog with no description, you rearrange the tree to fit. ## A Quick Detour: How `provide` and `inject` Work Compound components rely on Vue's [`provide` and `inject`](https://vuejs.org/guide/components/provide-inject.html) API. The model matches React Context: a parent publishes a value under a key, and any descendant in its subtree reads that value without the components in between knowing about it. The minimum example is two components. Click the button — the child mutates the shared ref, the parent re-renders. The same `open` lives in both: ```vue live id=provide-inject-demo file=Parent.vue entry ``` ```vue live id=provide-inject-demo file=Child.vue ``` The Vue docs frame this as the answer to *prop drilling*: passing the same prop through three intermediate components that do not care about it, only to reach the one that does. With `provide` / `inject`, the descendant reads from the closest matching provider in the tree, however many components deep it is. Toggle the root state below. On the left, every component on the path carries `open` as a prop. On the right, only the provider and the consumers know about it. `
`, ``, and `