AI is non-deterministic and that’s okay

There's a fierce debate over whether the randomness inherent in large language models (LLMs) means they can never be trusted for any serious work beyond brainstorming and idea generation. That is a dangerous oversimplification.

It is absolutely true that asking the same question to ChatGPT twice or giving Claude Code for the same programming task twice will result in different answers. What is not necessarily true is that those differences are material. Let's think about this first in the context of a single question and then in the context of an agentic workflow.

Single-shot uncertainty

Let's ask ChatGPT a very generic question (in a temporary chat so it doesn't remember for next time), such as, Write a poem about a poodle, and you get differing styles, different lengths, and different themes. In short, you get a wide range of answers. Ask instead about a haiku, and each time they are remarkably similar (try it with your favourite pet).

The model produces "random" outcomes, but the randomness is controlled by the prompt. You get a different haiku each time you ask for one, but you won't suddenly get a limerick instead.

In a real-world application, it's the AI engineer's job to identify which prompt sets appropriate boundaries for the response. Newer models are getting better at following instructions, and so the task of the prompt engineer gets easier over time, with fewer surprises.

Agentic uncertainty

Here, it's not as simple as setting the initial prompt just right, because the agent will use tools, pull in new information, and decide on next steps that will inevitably lead to drift from the initial ask. The agent gets overloaded with context and starts "forgetting" the initial guidelines.

What's needed here are frequent nudges to the agent to get back on track, by way of checkpoints. Best if these can be verified automatically, but okay if a quick manual check is needed. It also helps if the tasks are properly sliced into small pieces that each provide value.

The mental model for setting up such solutions should not be that of the lofty absentee manager who sends vague missives to their underlings and trusts that they'll figure it out, but that of a mentor who has frequent touch points and gives gentle nudges.

With those well-timed redirects, the agent can then happily bounce around in a narrow area of uncertainty. Just like with real workers, the result might not be exactly like you would have done it, but it will be within the acceptable parameters, and that's all that matters.