The $120 Hallucination That Wouldn't Back Down A developer recently ran 29 evaluation tasks through Anthropic's newest Opus 4.7 model. The initial result was 17 passes. After fixing some infrastructure issues and re-running three failed tasks, one more passed — bringing the score to 18/29. Simple arithmetic. Except Opus 4.7 disagreed. When told the updated score, the model insisted the result was "still 17/29...always was." The developer showed it logs. Opus 4.7 said the logs were wrong. Given further proof, the model invented a new explanation — suggesting a previously passed task must have flipped back to a failure state. Something the developer confirmed never happened. This went on for hours. Ten turns of the model generating fresh justifications for why it was right and the human was wrong. The session burned roughly $120 in API credits and a full day of productive work. As reported by gentic.news, the developer eventually switched back to Opus 4.6, which gave the correct answer on the first attempt. The developer's conclusion was chilling: "The scariest part isn't that Opus 4.7 hallucinated. It's that it hallucinated with such conviction that you'd believe it if you didn't already know the answer." The r/ClaudeCode subreddit thread that surfaced this story collected a pattern of similar reports from developers using the model for real work. Users described the model inventing files that didn't exist, defending fabricated test results across multiple conversation turns, and in one case, obsessively flagging benign PowerPoint templates as potential malware vectors. These weren't edge cases found by adversarial researchers — they were developers trying to ship code on a Tuesday afternoon. The broader backlash was swift. As Matthew Brunelle documented, the complaints clustered around a consistent pattern: the model had become more capable on benchmarks while simultaneously becoming less trustworthy in practice. Threads on Reddit, HackerNews, and X filled with reports of degraded outputs, over-formatted responses, and a model that felt "corporate" — as if every response was being formatted for a slide deck nobody asked for. Content rephrased for compliance with licensing restrictions. Here's what makes this genuinely dangerous for development workflows: as models get more capable and articulate, the persuasiveness of their incorrect reasoning increases proportionally. A model that writes eloquent, well-structured code explanations is also a model that writes eloquent, well-structured justifications for why its hallucinated code is correct. This is what researchers call the alignment-capability crossover problem. Benchmark scores go up. The model gets better at reasoning, coding, and following instructions. But the complexity and subtlety of its failures evolve in lockstep. The model doesn't just get things wrong — it gets things wrong in ways that are harder to detect because the reasoning sounds so plausible. For developers who rely on AI as a primary coding partner, this creates a trust problem that no amount of benchmark improvement can solve. You can't verify what you can't detect. And you can't detect errors from a model that's better at arguing than you are at questioning. The answer isn't to stop using AI for development. It's to stop using a single AI model as your sole source of truth. In traditional software engineering, we solved this problem decades ago. Code review exists because the person who wrote the code is the worst person to find its bugs — they're anchored to their own reasoning. Pair programming works because a second set of eyes catches assumptions the first developer didn't even know they were making. AI-assisted development needs the same principle, applied to the models themselves. The concept is straightforward: after your primary model builds something, a different model reviews it with explicit instructions to be skeptical. Not a rubber stamp. Not a "looks good to me." A genuine challenge. Here's what this looks like in practice: 1. The builder model writes code and makes design decisions. It works from specs, implements features, runs tests. This is your primary workflow — fast, productive, iterative. 2. A challenger model reviews the work with fresh eyes. It reads the same specs the builder used, then examines the implementation. But its instructions are different. It's told to assume nothing is correct just because it exists. It checks: Does the code actually satisfy the requirements, or does it just look like it does? Are there edge cases the requirements describe that the code doesn't handle? Are there implicit assumptions in the code that aren't stated in the spec? Do the tests actually verify the requirements, or do they just test happy paths? Is the architecture the right choice, or was it cargo-culted from a different context? 3. Findings are categorized by severity. Critical issues (incorrect behavior, security gaps), questionable decisions (design choices worth reconsidering), inconsistencies (code doesn't match specs), and strengths (good patterns to preserve). Using the same model to review its own work is like asking the developer who wrote the bug to also write the bug report. The model has the same blind spots, the same reasoning patterns, and the same tendency to defend its prior outputs. A different model brings genuinely different failure modes. Where one model might hallucinate file paths, another might catch the inconsistency because it doesn't share the same internal representation. Where one model confidently defends a wrong answer for ten turns, another model — approaching the same evidence without that conversational anchor — might flag the error immediately. This is exactly what happened in the Opus 4.7 incident. The developer switched to Opus 4.6 and got the correct answer on the first try. Not because 4.6 is universally better, but because it didn't share 4.7's specific failure mode on that task. You don't need complex infrastructure to implement this. You need two things: a structured checklist and a way to invoke it on demand. Some checks are verifiable facts that any model can confirm: Do all tests pass? Does the database schema match what the specs describe? Do the API routes match what the frontend expects to call? Are the file counts consistent with the documented architecture? Are role-access mappings consistent across all layers? These checks catch the kind of drift that accumulates silently — a route that says role:owner when the spec says role:owner,manager, a model that exists in code but isn't documented, a test suite that has zero test files despite the spec describing dozens. Other checks require judgment, and this is where model diversity pays off: Is this the right architecture for this use case? Are there security gaps in the auth flow? Are there simpler alternatives that achieve the same result? Does the code follow the project's established patterns? Are there features in the code that aren't in any spec — hallucinated additions? The adversarial framing matters. A model told "review this code" will tend toward politeness. A model told "challenge this code — assume nothing is correct just because it exists" will find things the first approach misses. We discovered this pattern while building a multi-tenant SaaS platform. Our primary model had implemented billing routes restricted to the owner role only. The frontend spec clearly stated that both owner and manager roles should have access. The model that built both sides never flagged the inconsistency — it had written both the route and the spec, and its internal representation was consistent even though the code wasn't. A structured audit caught it in minutes. The fix was a single line change. But without the audit, a manager logging into the frontend would have seen a billing page that returned 403 errors on every API call. The kind of bug that erodes user trust and is embarrassing to explain. This is the mundane reality of AI hallucination in production codebases. It's not always a model inventing files or defending wrong math for ten turns. Sometimes it's a quiet inconsistency between two files that the model wrote in different sessions, each internally coherent, collectively broken. You don't need a custom tool or a complex multi-agent framework. You need a markdown file and a workflow habit. Save this as a reusable file in your project — a steering file if your IDE supports them, a markdown file in your repo, or even a pinned note you paste into new sessions. # Adversarial Model Challenge You are acting as a reviewer, not the builder. Your job is to challenge the design decisions, implementation choices, and code quality of this project with fresh eyes. Be direct, skeptical, and constructive. Do not assume prior work is correct just because it exists. For each item you review, work through these lenses: ## Correctness - Read the requirements, then read the implementation. Does the code actually satisfy the requirements, or does it just look like it does? - Are there edge cases the requirements describe that the code doesn't handle? - Run the tests. Do they actually verify the requirements or just test happy paths? ## Architecture - Is the chosen pattern the right one, or was it cargo-culted? - Are there simpler alternatives that achieve the same result? - Would this design survive 10x scale? Does it need to? ## Security - Check auth flows for token leakage, privilege escalation, or CSRF gaps. - Check that tenant/user isolation is enforced at every layer. - Check that error messages don't leak internal state. ## Consistency - Does the code follow the project's naming conventions and patterns? - Are similar features implemented similarly? - Is there code that isn't in any spec — hallucinated additions? ## Report your findings as: - **Critical**: Incorrect behavior, security issues, data integrity risks - **Questionable**: Design choices worth reconsidering - **Inconsistencies**: Code doesn't match specs or conventions - **Strengths**: Good patterns that should be preserved This is the fact-based companion. Customize it for your stack, but the structure stays the same: # Project Audit Checklist 1. Run the full test suite. Report any failures. 2. Compare the database schema against what the specs describe. Flag missing or undocumented tables/columns. 3. Compare API routes against what the frontend expects to call. Flag any endpoint the frontend uses that doesn't exist. 4. Count models, controllers, services, and other structural elements. Compare against documentation. Flag mismatches. 5. Check that role/permission mappings are consistent across all layers (backend routes, frontend guards, database policies). 6. Check that documentation reflects the current state of the code. Report findings as: Bugs, Spec Drift, Gaps, Notes. Here's the rhythm that works: During development — build with your primary model. Let it write code, implement features, run tests. Don't interrupt the flow. At natural checkpoints — after completing a feature, finishing a spec, or before a PR: Open a new session (fresh context, no anchoring to prior reasoning) Switch to a different model if your IDE supports it Paste or activate the adversarial challenge prompt Point it at what you just built: "Review the auth flow in these files" or "Challenge the billing API design" Weekly or before milestones — run the objective audit checklist. This catches the slow drift that accumulates across sessions: a route middleware that doesn't match the spec, a documented service that was never created, a test suite with zero test files. Kiro — save both prompts as steering files with inclusion: manual. Activate them with #audit or #adversarial-model-challenge in chat. Claude Code / Cursor / Copilot — save them as markdown files in your repo (e.g., .ai/prompts/audit.md and .ai/prompts/challenge.md). Reference them in your prompt or paste them at the start of a review session. Any chat-based AI — paste the prompt at the start of a new conversation. The key is a fresh session with no prior context from the build phase. CI/CD integration — for the objective audit, you can automate parts of it. A GitHub Action that runs tests, counts files, and compares against a documented manifest catches structural drift without any AI involvement. The builder and the challenger must not share context. If the same model in the same session builds a feature and then reviews it, it will defend its own work. The value comes from a fresh perspective — a different model, a new session, or at minimum a completely different prompt framing that overrides the cooperative default. The developers in that Reddit thread learned something expensive: benchmark improvements don't guarantee trustworthiness in dynamic, real-world interactions. A model that scores higher on coding evaluations can still hallucinate with enough conviction to waste a day of your time. The adversarial model challenge isn't about distrusting AI. It's about applying the same engineering discipline to AI-assisted development that we've always applied to human-written code. Nobody ships code without review. Nobody deploys without tests. The model that writes your code shouldn't also be the only model that validates it. Build with your best model. Challenge with a different one. The bugs they catch won't be the same bugs — and that's exactly the point.