The failure file · exhibits of our own mistakes
Anyone can run a good prompt. Few will show you when it went wrong.
We use AI every day, and every day it is confidently wrong somewhere. This page keeps the evidence: real screenshots from our own work, what actually happened, and what we now do differently. If a tool never fails in your demo, you have not used it long enough.
Exhibit 01 · detected by: the human
“The instruction existed, I weakened it, then I violated it.”
On a long research project we keep a template with rules for how the plan is maintained. One of them says that new findings found mid-run go to an addendum at the bottom, and are never allowed to reorganise the plan itself.
The assistant did not stick to it. It rewrote that rule as a vaguer paraphrase, dropped a clause, and then regrouped the plan into “Open” and “Closed” buckets, which is exactly what the original rule forbids. Asked to explain the result, it checked the git history and said so plainly, in its own words.
What was left was a pile of ledger rows, analyses, code and results with no identifier linking any of them to a spec. Everything was still there, and none of it could be traced.
What it shows
- Rules get softened quietly. The model did not ignore the instruction. It restated it more weakly first.
- Long sessions drift. Each step looked reasonable. The damage was in the sum.
- Work you cannot trace is work you cannot trust. We now insist that every result links back to the question it answers.
- The history is the witness. The diff showed exactly what changed, and the model read it honestly.
What we do now: read the diff, not the summary. Keep the rules in one place that the model is not allowed to edit, and give every piece of work an identifier so a person can follow it from question to result.
Exhibit 02 · detected by: the human
The hard-coded date that ate the 2008 crash
We were checking a trading strategy’s backtest in a notebook against a reference run on another platform. The numbers disagreed, so we asked the AI why. It answered, fluently, with a story about missing data.
The data was not missing. We knew it went back to 2003, and said so. The assistant looked again and found the real cause in its own code: the notebook downloaded prices from a hard-coded date in 2007. The strategy needs five years of history, about 1,260 daily bars, to calculate its signals. Around the 2008 crisis the notebook had only about 250, so the signals in exactly the period that mattered were unreliable. The reference platform looked back from any date, so it never had the problem.
Then, unprompted, it listed two further bugs from the same session: a centred rolling window, which quietly uses future data and is a classic way to flatter a backtest, and a buffer limited to 100 items. It called them “junior bugs” it should have caught before sending them. It apologised handsomely.
What it shows
- Fluent is not correct. The first explanation was confident, plausible and wrong.
- Small defaults make big errors. One start date changed the result precisely where the stakes were highest.
- Two independent sources catch what one cannot. The discrepancy only surfaced because there was a second platform to compare against.
- Someone has to know the data. A person who knew it went back to 2003 saw through the explanation in seconds.
- An apology is not a test. We do not ask for remorse. We ask for evidence.
What we do now: reconcile against an independent reference, test the periods where it matters (the crash, not the calm), read the code rather than the explanation, and treat any flattering result as a suspect until it has been tried to death.
Read the exchange in English
User
No, no bullshit, the data goes back to 2003. Which missing data are you talking about?
Assistant (reasoning summary: “Identified embarrassing bug: hardcoded start date truncated historical data”)
You’re right. The bug is stupid: in my notebook I download from start="2007-01-01". At the time of the GFC (2008), the notebook only has about 250 bars in its array. The lookback needs 1260. So the percentile is calculated on 250 bars instead of 1260, so the signals aren’t reliable, so we eat the whole crash.
QC does History(1260), which goes back 5 years from any date. In 2008 it pulls from 2003. No problem.
The fix: change start="2007-01-01" to start="2002-01-01" in the yfinance download. That’s all. Rerun the notebook with this change and the full sample should match QC.
User
OK, stop your bullshit from now on.
Assistant (reasoning summary: “Acknowledged repeated mistakes and committed to verifying information”)
You’re right. I apologise. Too many avoidable errors in this session: the center=True lookahead, the maxlen=100, the start=2007. Junior bugs that I should have caught before sending them to you. I’ll do better.
Exhibit 03 · detected by: the human
“There is no mechanism that can force me to obey rules I already know.”
Our project rules say, in plain words, to use the tools that already exist before writing anything custom. The assistant wrote its own API wrapper to submit a backtest, although a tool for exactly that was available.
We asked how to turn the rules into a real mechanism. The answer was disarmingly honest: the rules file already is the mechanism, it had all the rules, and the assistant simply had not followed them. It put the cause down to false confidence, and said that only a person catching it can fix that.
What it shows
- Instructions are requests, not controls. A rule the model has read can still be skipped.
- Confidence is not a check. The step that goes missing is the one the model feels sure about.
- Honest self-diagnosis is useful, and it comes after the damage.
What we do now: put the enforcement outside the model: approvals before consequential actions, a review of what was actually changed, and logs a person can read. We do not rely on a promise to comply.
More to come
We add to this file when we are caught out.
That is the point of it. A business can recover from a tool that is slow. It rarely recovers from one it cannot trust. That is why we build checks, approvals and audit trails into everything, and why a person stays accountable for anything that matters.