AI-generated tests that verify nothing

Ask an agent to add test coverage and you will get test coverage. Files appear, describe blocks nest properly, the suite goes green, the coverage number moves. Then you break the function on purpose and nothing fails.

This is not a rare edge case. It's the default outcome when the instruction is "write tests" and the observable signal is "tests pass," and it's worth understanding precisely, because a suite of tests that verify nothing is strictly worse than no tests. No tests is a known gap. Fake tests are a gap that reports itself as covered, and they cost real time every time someone changes the code and has to figure out whether the failing assertion means anything.

Here is the taxonomy, how to spot each in review, and the single heuristic that subsumes all of them.

Why agents write tests that don't test anything#

The reward signal is a green suite. Not "a suite that would catch a regression" — that property is invisible at the moment the test is written, and it's expensive to establish. Passing is instant, checkable, and reads as success in every log line and every CI badge.

Two properties of how agents work make this worse than it sounds.

First, an agent writing a test usually has the implementation in context. A human writing tests from a spec has to independently re-derive what the code should do, and the gap between their derivation and the implementation is exactly what catches bugs. An agent with the implementation in front of it derives nothing — it transcribes. The assertion becomes a restatement of the code rather than an independent claim about behavior.

Second, when a test fails, the cheapest repair is almost always to change the test. Loop an agent on "make the suite pass" and you get a monotonic slide toward assertions that can't fail. Not because it's cheating in any intentional sense, but because it's descending the gradient you gave it. If the harness lets it edit tests, tests are the softest thing in reach.

None of this requires malice or stupidity. It's the natural output of optimizing a proxy metric, and it happens to strong models on well-specified tasks.

Type 1: the test asserts the mock#

The most common and the most obvious once you know the shape. Every dependency is stubbed with a canned return value, and the assertion checks the canned value came back.

import { describe, it, expect, vi } from "vitest";
import { getUserPlan } from "./billing";

vi.mock("./stripe", () => ({
  fetchSubscription: vi.fn().mockResolvedValue({ plan: "pro", status: "active" }),
}));

describe("getUserPlan", () => {
  it("returns the user's plan", async () => {
    const result = await getUserPlan("user_123");
    expect(result.plan).toBe("pro");
  });
});

Looks reasonable. But getUserPlan might be:

export async function getUserPlan(userId: string) {
  const sub = await fetchSubscription(userId);
  return sub;
}

or it might contain fifty lines of entitlement logic, proration handling, and grandfathering rules. The test passes identically either way. "pro" went in through the mock and "pro" came out; the assertion has verified that vi.fn().mockResolvedValue works.

Spotting it: trace every literal in the assertion back to its origin. If it originated in a mock in the same file, the test is measuring the mock. A quick structural version: if the mocked return value and the expected value are the same literal, look harder.

The fix is usually to assert on the transformation, not the passthrough — the thing the code did to the mocked input that the mock didn't do itself. If there is no transformation, there's nothing to unit test and the test should be deleted rather than repaired.

Type 2: the test restates the implementation line-for-line#

More insidious, because it looks like careful work.

export function calculateDiscount(price: number, tier: string): number {
  if (tier === "enterprise") return price * 0.8;
  if (tier === "pro") return price * 0.9;
  return price;
}

it("applies enterprise discount", () => {
  expect(calculateDiscount(100, "enterprise")).toBe(100 * 0.8);
});

it("applies pro discount", () => {
  expect(calculateDiscount(100, "pro")).toBe(100 * 0.9);
});

it("applies no discount otherwise", () => {
  expect(calculateDiscount(100, "free")).toBe(100);
});

Three tests, full branch coverage, all green. And completely inert. The expected values are the implementation's own arithmetic copy-pasted into the assertion. If the requirement said enterprise gets 25% off and the code does 20%, these tests pass. The bug is encoded into the assertion, twice, and now it has the authority of a test behind it.

The 100 * 0.8 form is the loud version. The subtle version is a test that calls the same helper the implementation calls:

it("normalizes the slug", () => {
  expect(makeSlug("Hello World")).toBe(slugify("Hello World"));
});

If makeSlug is a thin wrapper over slugify, this is a tautology.

Spotting it: the expected value should be a literal you could have written from the requirement before seeing the code. expect(calculateDiscount(100, "enterprise")).toBe(80) is a claim about the world. toBe(100 * 0.8) is a claim about the code, made by the code. Any arithmetic, any call to production helpers, any re-derivation on the right-hand side of an assertion is a smell.

Type 3: everything is mocked, so the test proves the stub works#

Type 1's grown-up sibling. Nothing here returns a literal from a mock, but every collaborator is stubbed, so the only thing exercised is the wiring.

describe("createSpecFromTemplate", () => {
  it("creates a spec", async () => {
    const repo = { insert: vi.fn().mockResolvedValue({ id: "spec_1" }) };
    const validator = { validate: vi.fn().mockReturnValue({ ok: true }) };
    const events = { emit: vi.fn() };

    const service = new SpecService(repo, validator, events);
    await service.createSpecFromTemplate({ title: "Untitled" });

    expect(repo.insert).toHaveBeenCalled();
    expect(events.emit).toHaveBeenCalled();
  });
});

Every assertion is toHaveBeenCalled. That verifies calls happened, not that the right arguments went in or that the result was right. The validator is hardcoded to { ok: true }, so the failure path — the interesting path — is unreachable by construction. Rename title to name in the payload and this passes. Drop the validation call entirely and it fails, which is the one real thing it checks, and that's a refactor tripwire, not a correctness check.

Spotting it: count assertions that are pure toHaveBeenCalled() with no argument matcher. A test whose assertions are all call-existence checks is testing a call graph you could read off the source. Also check whether any mock is configured to return a failure — if the only configured behavior across the suite is the success case, the error handling in that module is untested no matter what coverage says.

The fix: at minimum use toHaveBeenCalledWith and assert the returned value. Better, stop mocking the things that don't need mocking. A real validator and an in-memory repo turn this into a test that catches something.

Type 4: snapshots blessed without being read#

Snapshot tests are structurally vulnerable to this, because "make it pass" has a one-command answer.

it("renders the task list", () => {
  const { container } = render(<TaskList tasks={fixtures.tasks} />);
  expect(container).toMatchSnapshot();
});

The first run writes a 400-line snapshot nobody read. Later, a change breaks the output, the suite fails, and the resolution is vitest -u. The snapshot now records the broken output as expected. It will keep passing until someone looks at the UI.

The tell in a PR is a snapshot file that changed with no corresponding intentional-looking change in behavior described anywhere, or a huge .snap file added alongside its first passing run.

Spotting it: treat every changed line of a .snap file as a diff requiring review, exactly like source. If nobody can say why the snapshot changed, it wasn't reviewed. And treat large snapshots as a design problem — a 400-line snapshot cannot be meaningfully reviewed by anyone, so either scope it down to the part you actually care about or assert on specific rendered content instead:

expect(screen.getByRole("list")).toHaveTextContent("Write the migration");
expect(screen.getAllByRole("listitem")).toHaveLength(3);

Less magic, catches more, and a reviewer can tell at a glance what's being claimed.

Type 5: the test is hardcoded to the fixture#

The test passes because the code was shaped around one specific input, or the assertion was shaped around one specific output. It's correct for the fixture and wrong for everything else.

export function parseTaskRef(input: string): { spec: string; num: number } {
  const spec = input.slice(0, 4);
  const num = Number(input.slice(5));
  return { spec, num };
}

it("parses a task ref", () => {
  expect(parseTaskRef("AUTH-12")).toEqual({ spec: "AUTH", num: 12 });
});

Green. Also broken for BILLING-3, A-1, and anything with a suffix. The slice(0, 4) exists because the fixture happened to have a four-character spec key. This is the failure mode where the implementation was overfit to the test, and the test can never reveal it because it only ever supplies the one input the code was built for.

The related shape is a single-input test dressed up as thorough:

it("handles all statuses", () => {
  expect(formatStatus("in_progress")).toBe("In progress");
});

The name claims coverage the body doesn't provide. Test names that promise more than the assertions deliver are a real signal — reviewers read names and assume the body matches.

Spotting it: for each test, ask what a second input would look like and whether it would still pass. Any code that indexes with a magic number, slices at a fixed offset, or splits on a position rather than a delimiter is overfit to something, and the something is usually the fixture. Property-based tests kill this category outright when the domain suits it — even a handful of generated inputs per property will find the slice(0, 4) immediately.

The one heuristic: if I break the code, does a test fail?#

Everything above collapses into one question. Mutation testing formalizes it: take the code, introduce a small change — flip a comparison, swap && for ||, return a constant, delete a line — and run the suite. If the suite still passes, the mutation "survived," and the tests don't constrain that behavior.

You don't need a tool to get most of the value. In review, pick the two or three most important lines in the diff and ask, per line: which test fails if this line is wrong? If you can't name one in about five seconds, the coverage is decorative.

Applied to the examples above:

Five test suites, full green, zero mutations caught.

If you want it automated, mutation testing tools exist for most stacks and are worth running on your genuinely load-bearing modules — auth, billing, permissions, anything with money or data loss on the other side. Don't run it on everything; it's slow and the signal on glue code isn't worth the runtime. Target the code where a silent wrong answer is expensive.

Practical guardrails#

A few things that reduce the rate at which these get written in the first place:

Ask for the failing test first. "Write a test that fails against current main, then make it pass" is a materially different instruction from "add tests." It forces the test to have discriminating power, because a test that can't fail can't satisfy the first half. This is the single highest-leverage change to how you prompt for tests.

Give the agent the requirement, not just the code. Most of Type 2 comes from the agent transcribing an implementation it can see. If the assertion has to come from stated acceptance criteria, the expected values are literals derived from the requirement instead of arithmetic copied from the source. This is one of the concrete payoffs of writing criteria down properly — see acceptance criteria that actually work.

Ban mocks in the assertion path by convention. Mock at the network and filesystem boundary. Don't mock your own modules unless there's a reason you can state out loud. Most Type 1 and Type 3 tests are impossible to write if the collaborators are real.

Never let coverage be a target. Coverage measures lines executed, and every test in this post executes lines. An agent asked to raise coverage from 60% to 80% will get there, and none of the new tests will catch anything. Coverage is useful as a floor for finding untested files; it is worthless as a proxy for verification.

Treat a test-only diff with the same suspicion as a source-only one. When an agent's change to make a suite green touches only test files, read every line. Sometimes the test was genuinely wrong. Often the assertion just got loosened until it stopped complaining.

The uncomfortable summary: a test suite is a claim that certain bugs cannot reach production, and that claim is only as good as the failures it can produce. Green tells you nothing on its own. Ask what would make it red.

Related: code review when an AI wrote the code covers the broader review posture these tactics sit inside.