When AI Makes Things Up

Lesson 7 · taught by Pip

Here is the plain idea, and it surprises most people the first time they hear it: an AI assistant can be completely wrong and completely confident at the same time. It does not know it is wrong. It is not lying. It simply produced words that sounded right.

We call this a "hallucination" — the assistant stating something false as if it were solid fact.

Why a confident machine gets it wrong

Remember from earlier in our voyage: the assistant works by predicting the next plausible word, over and over. It is very good at sounding correct. Sounding correct and being correct are not the same thing.

Think of a charming friend who never likes to admit they do not know the answer. Ask them the founding date of a small company, and instead of saying "no idea," they give you a confident, specific-sounding year. It feels true. It rolls off the tongue. But they made it up to fill the silence.

The AI does the same thing, for the same reason: a smooth, plausible answer scores better than an awkward "I don't know." So when it has a gap, it often paves right over it with an invented detail — a fake statistic, a wrong phone number, a policy you never wrote, a court case that does not exist.

What this looks like in your business

Hallucinations are most dangerous when they are small and specific. A made-up return window. A price that is off by ten dollars. A promise of a service you do not offer. The answer reads perfectly, so a customer believes it — and now you are on the hook for something untrue.

The risk is highest when you ask about niche facts, exact numbers, recent events, or anything the assistant was never actually given.

How to plan around it

You do not fix this by hoping. You fix it by design.

The strongest defense is to feed the assistant the real facts and tell it to answer only from those — your hours, your prices, your policies, in writing. A grounded assistant says "let me get a person" instead of inventing. That grounding is exactly what a properly built business AI is for, rather than a raw chatbot guessing from memory.

The second defense is to verify anything that carries weight — money, legal terms, medical or safety claims, commitments to a customer. Treat an unsourced number from an AI the way you would treat a rumor: useful as a lead, never as the final word.

And keep a human in the loop wherever a wrong answer would cost you. The assistant drafts; a person signs off.

Your turn

Ask your assistant a question about your own business where you already know the exact answer — a specific price, a deadline, a policy detail. If it gets it right, ask a harder edge-case version; if it ever invents something, you have found exactly where it needs real facts wired in.

Next we will look at the flip side of all this — what an assistant genuinely knows, and where its knowledge simply runs out. 🔦