AI Is Not Logical

I asked my agent to calculate the average age of 8 people. It gets it wrong a not insignificant percentage of the time.

I do this all the time. That’s not a fringe failure mode. That’s basic arithmetic, on a task I run constantly, on the best model I have access to.

This is the part of building with AI that nobody warns you about. AI is not logical. It looks logical. It sounds logical. But every response is governed by probability. LLMs are trained on pattern matching, not computation. They recognize what correct math looks like. They aren’t actually computing anything.

That single fact is the source of almost every hard problem in agent engineering right now. How you reason about it changes how you build, what you trust agents to do, and where you put the guardrails.

I’ll be unpacking it over the next few posts.