Blog

There's an email we talk about a lot at Mary Technology. Three characters, sent between two executives in the middle of negotiating a contract. "Hmm..."

The junior associates flagged it as irrelevant. So would every AI tool on the market. The partner who eventually found it would tell you that nine out of ten litigators may have walked past it too. He insisted on spot checking the irrelevant documents himself, and three characters sent at exactly the wrong moment became evidence of hesitancy about whether there was ever genuine agreement on the deal. He used it in court.

That instinct came from twenty years of watching how evidence behaves at trial. It came from him, not from a machine.

Most of the litigators we work with don't need to be sold on AI. They need to be shown what it's hiding from them. We've been having a version of this conversation for the last two years, and the pattern is always the same. The senior lawyers who are best at their jobs don't want AI to do the work for them. They want it to stop them missing things. They want more of the relevant record in front of them, not less. Speed matters, but only between the moments that actually require their judgment. At those moments, they want to be thinking harder, not less.

Earlier this year, Steven Shaw and Gideon Nave at the Wharton School published a study that put empirical weight behind what we'd been hearing anecdotally. Most lawyers already know Kahneman's framework for thinking: System 1, fast and intuitive, System 2, slow and deliberative. Shaw and Nave argue we now need a third. System 3: artificial cognition that operates outside the brain entirely. Across three preregistered experiments with more than 1,300 participants, they found that when people used the AI, they followed wrong recommendations roughly four times out of five. Their confidence in their own answers rose by nearly 12 percentage points when they used the AI, even though half the time the AI was feeding them errors. Shaw and Nave call this pattern "cognitive surrender."

The finding that matters most for firms evaluating AI tools is what happened at the extremes. Accuracy scaled with reliance. The more a participant leaned on the AI, the more their performance tracked the AI's performance rather than their own. Heavy users scored highest when the AI was right. They scored lowest when it was wrong. In a law firm, that means the lawyers getting the most out of a speed-to-output tool on a good day are the same lawyers getting hurt worst when the tool has a bad day. Your best AI users are your most exposed users.

The data also surfaced something hopeful. Participants who scored higher on deliberative thinking, the people who enjoy wrestling with a problem, overrode wrong AI advice at significantly higher rates. In a firm, those people are your senior partners. Their instinct, built over a career, is the best defence against cognitive surrender. The right design question for legal AI is how to keep that instinct in the room, not how to work around it.

Most legal AI on the market right now is built around a compression architecture. The tool reads everything, decides what matters, writes the memo, and the partner reviews it at the end. Everyone has seen the demos. But that workflow is cognitive surrender by design. It asks the most experienced person in the room to sign off on conclusions drawn from documents they never touched, from a set of facts someone else's algorithm already chose to surface.

The fix has to live inside the product itself, because behavioural solutions don't work. The Wharton researchers tried them. They paid participants for accuracy and gave them feedback after every answer. Override rates improved, but more than half the time, participants still followed the faulty AI. If direct financial incentives and real-time error signals can only do that much, asking lawyers to read more carefully won't either. The real difference is the difference between checking your answers on a test and knowing whether you answered every question.

This is the test we come back to every time we think about how Mary should work. Litigation is adversarial. If there's a fact in the record that can hurt your client, opposing counsel is going to find it. The only question is whether your team finds it first. Any AI tool that cannot guarantee that was not built for litigation.

So before you invest in the next platform that promises to do the thinking for you, ask yourself one question. When the tool runs, will you still be thinking too?

Mary is a fact management system for litigators who need to verify everything. We believe the profession deserves AI that makes judgment faster, not optional. If that resonates, you can book a time with us to see how it works at marytechnology.com, or come find us at Legal Innovation & Tech Fest in Sydney.