oh my pi · the harness

The conversation about AI coding assistants almost always centres on the model: GPT-5.3 versus Opus, Gemini versus whatever dropped this week. That framing misses something the maintainers of oh my pi (omp) consider the actual bottleneck — the harness. This article is a sourced, non-promotional look at what omp is, what problem it claims to solve, where it is genuinely useful, and where the right answer is still a smaller or different tool. Every claim below is cited to a primary source or to an external benchmark. Nothing here is hype framing from marketing copy, and there are no embedded AI-generated videos.

What "the harness" actually means here

A coding-agent harness is everything that sits between the LLM API and the developer's intent: the prompt format, the edit protocol, the tool namespace, error recovery, context compaction, subagent fan-out, debugger and language-server wiring, and the terminal UI. The project's own companion essay — *We improved 15 LLMs at coding in one afternoon. Only the harness changed.* — opens with the thesis that "one of the bottlenecks is something much more mundane: the harness. Tool schemas, error messages, state management, everything between 'the model knows what to change' and 'the issue is resolved.' This is where most failures happen in practice" (stencil.so/blog/the-harness-problem, 2026-02-12, author Can Bölük).

The upstream lineage reinforces the framing. Mario Zechner's Pi repo explicitly titles itself "Pi Agent Harness" and breaks the work into @earendil-works/pi-agent-core (agent runtime, tool calling, state) and @earendil-works/pi-coding-agent (interactive CLI). omp is positioned in its own README as "the Pi you love, with batteries included" — same harness, more pre-installed surface (can1357/oh-my-pi README).

The term is therefore not a marketing word in this codebase; it is the literal subject of the project.

What omp ships

From the README at github.com/can1357/oh-my-pi and the omp.sh project page, the headline numbers are:

• 40+ providers and hundreds of models behind one /model slash command (Anthropic, OpenAI, OpenAI Codex, Gemini, xAI, Mistral, Groq, Cerebras, Fireworks, Together, Hugging Face, NVIDIA, OpenRouter, Synthetic, Vercel AI Gateway, Cloudflare AI Gateway, Perplexity, plus subscription-routed coding plans: Cursor, GitHub Copilot, GitLab Duo, Kimi Code, Moonshot, MiniMax Coding Plan, Qwen Portal, Z.AI / GLM, OpenCode Go, OpenCode Zen).
• 32 built-in tools, 14 LSP operations, 28 DAP operations, ~55,000 lines of Rust core across pi-natives, pi-shell, pi-ast, pi-iso, plus a vendored brush-shell.
• Native binaries for macOS, Linux, Windows; no WSL required.
• Four execution roles: default, smol (cheap subagent fan-out), slow (deep reasoning), plan (plan mode), with mid-session /model swap and Ctrl+P cycling.

The distribution is real and reproducible: npm carries it as @oh-my-pi/pi-coding-agent, install via curl -fsSL https://omp.sh/install | sh, brew install can1357/tap/omp, mise use -g github:can1357/oh-my-pi, or bun install -g @oh-my-pi/pi-coding-agent. License is MIT.

Why it should be used — the sourced rationale

1. The edit format is a measurable bottleneck

The companion benchmark — published as the project's "harness problem" essay — runs sixteen models, three edit tools, 180 tasks, three runs each, on React-corpus mutation fixes. The headline finding: patch is the worst format for nearly every model; hashline matches or beats str_replace for most; and the weakest models gain the most. Specific numbers from that essay and the README table:

Sources: stencil.so/blog/the-harness-problem (full methodology) and can1357/oh-my-pi README §"Every tool, _benchmaxxed_".

omp's edit tool is built on the hashline protocol from @oh-my-pi/hashline: lines are tagged with short content-hash anchors, edits reference anchors (replace line 2:f1, replace range 1:a3..3:0e), and stale anchors are rejected before the file is corrupted. The react-edit-benchmark code that produced the numbers is in the public monorepo at packages/react-edit-benchmark (stencil.so/blog/the-harness-problem, footer).

2. Independent academic evidence supports the broader claim

Two peer-style benchmarks confirm the framing that the harness — particularly the edit protocol — is a first-class variable, not a solved detail:

• Diff-XYZ (JetBrains, arxiv.org/abs/2510.12487). Cross-format comparison of diff representations on a CommitPackFT-derived corpus: "different formats should be used depending on the use case and model size. For example, representing diffs in search-replace format performs best for larger models across most tasks, while structured udiff variants offer similar but slightly weaker performance. In contrast, smaller open models benefit little from any formatting choice."
• EDIT-Bench (arxiv.org/abs/2511.04486). 540 real-world instructed-editing problems, 40 LLMs evaluated: "EDIT-Bench is a challenging set of problems where only 1 model scores over 60%." Translation: even at frontier, model-level code-editing capability is far from saturated, which means the format and harness around the model still move the needle.

Aider's own historical data — referenced inside the harness problem essay — also shows format-only swings of GPT-4 Turbo from 26% to 59% (aider.chat/docs/benchmarks.html, as cited in stencil.so/blog/the-harness-problem). The point is not that omp is the only harness with a story; the point is that the broader evidence base already treats harness design as a meaningful axis.

3. The IDE, the debugger, and the version-control system are wired in

omp's own differentiator, per the README, is that the "batteries" are real and visible:

• LSP on every write. Renames go through workspace/willRenameFiles, so re-exports, barrels, and aliased imports update before the file moves. README §02.
• DAP drives real debuggers. lldb-dap for native binaries, dlv for Go, debugpy for Python. README §03 documents an attached session against /tmp/omp-native/demo reading the instruction pointer at a segfault site, not sprinkling print().
• Persistent execution kernels with tool-calling. A Python cell and a Bun worker run as persistent sessions; either kernel can call back into the agent's own read, search, task tools over a loopback bridge. README §01.
• Time-traveling stream rules. A regex match aborts the stream mid-token, injects the rule as a system reminder, and retries from the same point. README §04.
• Typed subagents. task fans out into isolated worktrees, results come back schema-validated. README §05.
• GitHub as paths. PRs and issues are addressed as pr://… and issue://…, the same read interface as local files. README §12. Twelve internal :// schemes (skill, rule, agent, conflict, ast, diff, …) resolve through the same tool.
• ACP for editors. Runs inside Zed against the real buffer, terminal, and permission prompts via the Agent Client Protocol JSON-RPC. README §14.
• Impervious config migration. Reads Cursor MDC, Cline .clinerules, Codex AGENTS.md, Copilot applyTo, and four others in their native shape. README §15.
• Rust-native, no shell-out. ripgrep, glob, find, and brush-shell are linked into the process. README §09.

These are claims from the maintainer's own README; the Rust module table (shell 3,700 LoC, grep 1,900, keys 1,490, text 1,450, summary 1,040, ast 1,000, fs_cache 840, highlight 470, pty …) is consistent with the surface area described (can1357/oh-my-pi README).

4. The vendor-economics argument

The companion essay closes with an argument about why this matters at the industry level: closed vendors have no incentive to tune harnesses for competing models, while open harnesses accumulate fixes across the model landscape because contributors use different models and fix their personal failure modes. The Anthropic / OpenCode block cited in the essay (HN 46625918) and the maintainer's own Gemini ban — "Not rate-limited. Not warned. Disabled. For running a benchmark" — are presented as evidence that closed stacks treat the harness as competitive surface rather than shared infrastructure (stencil.so/blog/the-harness-problem). The point is debatable; what is not debatable is that it is a real, named argument for why an open harness is worth investing in.

Concrete use cases

These are the situations where the README's feature set is most directly relevant:

1. Multi-file refactors with real rename semantics. LSP-aware, hashline-anchored, mid-flight drift-resistant. README §02, §11. 2. Triaging a segfault or a hang. Attach lldb-dap, dlv, or debugpy; step, read frames, evaluate expressions instead of adding println. README §03. 3. Long-horizon work across sessions. Hindsight memory (retain / recall / reflect) compresses prior runs into a mental model loaded on the first turn of the next one. README §13. 4. Live pairing or supervising. /collab exposes the session over a relay (read-only or read-write) with a joinable link and QR code. README §07. 5. Heavy code review with priorities. /review spawns reviewer subagents in parallel, ranks issues P0–P3 with confidence scores. README §10. 6. PR work without bespoke tool stubs. Treat GitHub as paths under pr:// / issue://. README §12, §17. 7. Dropping into an existing repo with existing rules. Existing Cursor / Cline / Codex / Copilot rule files are imported in place. README §15.

When not to use it — the rational counter-cases

omp is not the right tool for every developer, and the upstream Pi's own README is explicit about the boundaries:

• You only need chat completion over a repo. A simpler client (Claude Code, Codex CLI, Aider) is less surface to maintain. The upstream Pi is a deliberately smaller harness and is the more conservative choice if you do not need omp's 32 tools.
• You need a hardened sandbox out of the box. Pi's README, §"Permissions & Containerization," states verbatim: "Pi does not include a built-in permission system for restricting filesystem, process, network, or credential access. By default, it runs with the permissions of the user and process that launched it." Containerization (Gondolin micro-VM, Docker, OpenShell) is an operator responsibility, not a harness guarantee (badlogic/pi-mono README).
• You want a hosted SaaS. Both Pi and omp are local-first CLI tools. The harness framing is the wrong category if your requirement is managed cloud.

These are not weaknesses framed as features; they are the documented design choices of the project and its upstream.

What the benchmark numbers mean and do not mean

Honest reading of the numbers in the companion essay:

• The harness-problem benchmark is the project's own benchmark. It compares edit formats on a controlled mutation-reversion task. The 6.7% → 68.3% Grok Code Fast 1 lift is real and reproducible from the published code, but it measures the edit-format variable, not omp as a whole product.
• The "+5 percentage points" claim for Gemini 3 Flash is *over str_replace*, framed against "Google's own best attempt at the format." It is not a 5 pp lift over Claude Code with the same model.
• The 61% Grok 4 Fast output-token reduction is a token efficiency claim, not a quality claim. Fewer tokens spent on retry loops is a cost-and-latency win, not a guarantee that the produced patch is better.
• The 2.1× MiniMax pass-rate lift is the most interesting row for local-model users: it suggests the harness compounds with weaker base capability. The companion essay concludes with "Often the model isn't flaky at understanding the task. It's flaky at expressing itself. You're blaming the pilot for the landing gear" (stencil.so/blog/the-harness-problem). Read it as the project's thesis, not as a verified external benchmark.

The independent academic results above (Diff-XYZ, EDIT-Bench) confirm the *direction* — harness and format are first-class variables — but they do not validate omp's specific implementation.

Configuration reality, not configuration myth

A practical note that the README's marketing surface can paper over:

• 40+ providers is accurate, but several are routed through OAuth to subscription plans (Cursor, GitHub Copilot, GitLab Duo, Perplexity). Subscription coverage is the upstream provider's decision, not omp's.
• Local-model wiring is supported via OpenAI-compatible /v1/models (Ollama, Ollama Cloud, LM Studio, llama.cpp, vLLM, LiteLLM). A custom provider is one YAML block in ~/.omp/agent/models.yml. The README documents this in full (can1357/oh-my-pi README §"Forty-plus providers…").
• Fallback chains are first-class: per-role retry.fallbackChains so a 429 on the primary model hands the rest of the turn to the next entry, restored on cooldown.
• Path-scoped models: enabledModels and disabledProviders can be scoped to a path: prefix, so a single repo can pin a different model set without touching global config.

This is the part of omp that converts "40 providers" into something a working developer can rely on day-to-day.