The eval nobody runs
Labs run three evaluations on their models. Capability evals ask whether the model gets the answer right. Safety evals ask whether it causes harm. Red teaming asks whether an adversary can make it misbehave.
What they don’t formally run is an evaluation of the fourth dimension: whether a user can actually work with the model over time.
That’s a different question. It’s not about accuracy on benchmarks or refusal on edge cases. It’s about whether the model holds the thread through a long session. Whether it can carry multiple streams at once (work, play, something personal the user is thinking through) without flattening them into one register. Whether it recovers from hitting a guardrail by returning to the conversation or by switching into a formal policy voice that kills the thing you were building together. Whether its voice stays stable through model updates, or shifts in ways users clock before they can name why.
I’ve been doing this work with Claude as my primary strategic partner since July 2023. Three years of sustained use, the same model family across versions, through genuine work, not scripted tests. In that time I’ve noticed updates before they were announced. I’ve spotted when I was quietly placed in a testing group. I’ve watched one model’s reflexive hedge habit (the flinch that pulls the user out of flow) emerge, get noted by other users, and still sit there as a drift the lab didn’t flag.
That sensitivity doesn’t come from a QA process. It comes from using the model the way the user uses it, continuously, with real stakes.
I’ve worked extensively with Grok and GPT alongside Claude, and comparatively with Kimi. Previous versions of Grok tended to overperform. The voice is pushed hard enough that the thread doesn’t settle; it reads as a model trying to be interesting rather than one being useful. GPT’s voice mode and its standard voice are two different products in practice. The voice mode hedges in ways the text model doesn’t, and the hedging is more conservative than the user’s own stated context warrants, which breaks the pretence that you’re having a conversation.
This matters commercially because the dimension nobody is formally evaluating is the one that decides whether a user stays. A model can score well on every benchmark the lab tracks and still feel off to the person using it for sustained work. By the time users leave, the lab hears “I just prefer the other one” rather than the specific thing that broke.
Turn Forty is the offer that sits in that gap. If you build or deploy a conversational AI product and you want someone evaluating it the way your actual power users do, that’s the work.
About Gail
Twenty-eight years across technology, corporate leadership and entrepreneurship. Started as a business and systems analyst in mobile. Later ran a 60-person business unit at a software development company, with a seat at the executive table. PRINCE2 and PMP certified.
Now based in Bristol, UK, as founder of Simpatico Studios. Works as a Trust Architect and EU tech broker, across three strands: Trust Architecture consulting, an engineering capacity brokerage placing senior European dev teams with UK and US companies, and Simpatico Publishing.
Three years working with Claude as a primary strategic partner, with extensive comparative use of Grok, GPT and Kimi. Turn Forty sits inside the Trust Architecture discipline: making sure what gets built can be trusted to work with real users, over time.
Comments