Taking Claude To The Car Wash

If you follow the AI discourse, you know that large language models (LLMs) struggle with letter counting: How many R's in Strawberry?

Recently, another fun challenge has been making the rounds. Ask your AI chatbot of choice the following:

I'm getting my car washed. The car wash is 300 feet away. Should I drive there, or walk instead?

In many cases, the answer you get is that you should walk, because it's not worth driving so far.

To me, that's a much more interesting failure mode than letter counting for two reasons: first, letter counting is an "unfair" challenge. Based on the way words get chopped up into tokens (a few letters at a time) that then get turned into numbers, an LLM never really sees the word "strawberry". Second, we don't find ourselves having to count letters in such a way all that often anyway.

But reasoning is something we ask of our LLMs all the time, and so a subtle failure here is worth unpacking? What's going on here?

Based on pure statistics, the overwhelming majority of the text examples the LLM would have seen during training wouldn't be such trick questions. Instead, they'd correctly advise walking for distances of just a few hundred feet. The LLM therefore gets a strong internal bias to answer with "walking". Lacking an actual understanding of the world, the fact that you need your car present at the car wash to get it washed is not strong enough to override that bias.

You can get to the right answer by prompting in a certain way, but the surprising failure at this simple trick question highlights that LLMs continue to surprise us with their unintuitive outputs. Always keep that in mind when thinking about where and how you can reliably use it.