← All experiments

No. 045 · Week 46 · Dan Shipper · Exploring

Turn your daily newsletter edits into AI checks—and watch every frontier model jump 25+ points.

Turn your daily newsletter edits into AI checks—and watch every frontier model jump 25+ points.

Almost every day at Every, we publish Context Window. Getting an AI to summarize an article or demo in our voice—tight, problem-first, click-worthy, and free of insider shorthand—usually meant repeating the same redlines in chat over and over, only to start from scratch on the next issue.

Instead of leaving those corrections trapped in chat history, we started saving real Context Window drafts alongside my actual edits and gut reactions in Every Checks. Each saved example pairs an original prompt with a gold case—the exact version we actually wanted to publish—plus the specific feedback behind it. Every time I flagged something off, we turned that reaction into a binary yes/no check: "does the opening explain what is being corrected and why a reader would want a test?", "does it avoid unexplained agent jargon like 'an orchestrator managing faceless subagents'?", "does the bolded opener avoid copying raw article labels like 'Level 9:'?", and "are the sentences crisp instead of jamming four clauses into one run-on?" Across eight real cases, that gave us 74 yes/no checks defining what good looks like.

From there, we built a reusable short-idea-summary skill—a shared set of writing instructions any model can load—and ran an optimization loop from v1 to v9. Each version was tested in isolated, zero-context sandboxes across four frontier models—Gemini 4 Hydrogen, Claude Opus 5.5, GPT-6 Astra, and Claude Sonnet 5.5 (all at High effort)—and blind-graded by claude-sonnet-5-5 against those 74 checks.

On v1, every model struggled, scoring between 34% and 56% on the initial seven cases (Claude Opus 5.5 at 56%, GPT-6 Astra at 46%, Gemini 4 Hydrogen at 40%, Claude Sonnet 5.5 at 34%). After encoding our redlines into v9, every model jumped into a tight band on the full 74-check Lab, with Gemini 4 Hydrogen taking the lead:

  • Gemini 4 Hydrogen: 60/74 (81.1%)
  • Claude Opus 5.5: 58/74 (78.4%)
  • GPT-6 Astra: 58/74 (78.4%)
  • Claude Sonnet 5.5: 56/74 (75.7%)

The takeaway: If you want AI to match your taste reliably, start by collecting gold cases—concrete examples of the exact outputs you'd actually ship. Once you have gold cases, you can turn the differences between a mediocre draft and a gold one into checks that grade any model automatically. And once you have checks, you can systematically optimize a skill against them until models hit your bar on the first try—turning daily editing feedback into a reusable asset that compounds every time you publish.