Next Step for Web LLM Interaction: Probing

How to use an LLM?

Among many other things, OpenAI did something incredible 3 years ago - build a product that everyone is using. It is exceedingly rare to witness such a powerful tool as an LLM move to quickly from researc prototype into the palms of billions. Whether it's a middle schooler in California, or a gardener in India, everyone has learned how to use an LLM.

Where to go next, from this blinking cursor?

From Prompting

The mental model, and the LLM contract is simple. You tell the chat interface what you want from it, and it will activate from it's neurons the sum total of human knowledge, and deliver (almost) flawlessly.

Spot a problem? Telling the chat interface what you want from it isn't so easy after all. If the output isn't quite right, you "tweak" your prompt. More constraints, more adjectives and maybe some irritation, and hopefully this time the LLM understands what you mean.

To Probing

I think probing is the next iteration of this interaction contract, and one that will just as easily permeate the world. Instead of the AI delivering an instant gratification of what it thinks it understood, we try to change the response objective.

The goal of the LLM interaction is now not to give you an ansewr right away, but to find out what you mean, and what you need. By having our model probe before it produces, the relationship becomes a collaborative one, rather than transactional.

Building a context document, for every conversation

This has been seeping into agentic harnesses already, where the LLM tries to find out as much information about a project as possible before beginning work. The idea is simple - instead of jumping to provide a wall of text, the LLM works with you. It can figure out the right questions to fill in the gaps of your ask, probe you for details you didn't think to add, and ultimately give you, this time, a flawless answer.

Delayed Gratification?

Maybe this idea sounds counterintuitive, and it really does intentionally break the idea of instant gratification. But with an intentional shift to probing, we change what feels good about the product. The speed is less impressive lately, but with our models getting better and better by the minute, the value has moved to a clarity of understanding, and a quality of the answer.