How AI meeting summaries actually work — and what to look for

An AI meeting summary is the output of a two-stage pipeline. Audio goes in, structured Markdown comes out, and somewhere in the middle a lot of decisions get made about your privacy. Here is what each stage actually does — and the questions that separate a useful summary from a liability.

Stage 1: Speech-to-text (STT)

The first stage takes audio (typically 16 kHz mono PCM or compressed Opus) and produces text. This is the easier of the two problems, and the one with the longer history. Modern STT engines fall into two families:

• End-to-end neural transcribers (Whisper, Deepgram Nova, Mistral Voxtral, NVIDIA Parakeet). A single model takes audio frames and emits text. Trained on hundreds of thousands of hours of multilingual audio. Word error rate (WER) on clean audio: 4–7%.
• Hybrid HMM/DNN systems (legacy Kaldi-based pipelines). Faster on CPU, less accurate on noisy or accented input. Mostly retired in production by 2024.

For Punkto, we use Mistral Voxtral Mini for transcription. Two reasons: it is strong on European languages (French, German, Spanish, Italian, Polish are first-class, not afterthoughts), and the provider is EU-jurisdictional, which closes a sovereignty box other transcribers leave open.

The three things that break STT

1. Cross-talk. Two people speaking at once. WER doubles or worse. Diarization (who-said-what) collapses entirely.
2. Domain jargon.“Schrems II,” “K8s pod,” product names — all foreign to the training set, all transcribed phonetically. Custom vocabulary helps; few consumer tools expose it.
3. Accents. A heavy regional accent on a non-native English speaker can push WER past 15%. The fix is using a model trained for that accent, not better mics.

Diarization is harder than transcription

Knowing “who said what” requires speaker embeddings — vector representations of each voice — and clustering them across the call. Mistakes are common: short utterances get attributed to the wrong speaker, people who joined late get a fresh ID, and microphones picking up multiple people from the same room confuse the system.

For 90% of business meetings, diarization is “good enough but not great.” If you actually need legally-attributable speech, use a tool with explicit speaker labels (everyone joins with their own mic, model is told who they are), or fall back to humans.

Stage 2: Language model summarization

The transcript goes into a large language model with a prompt that asks for a structured rewrite. This is where AI summaries actually become useful — and where the failure modes get more interesting.

What a good summary prompt looks like

Behind the scenes, a vendor's summary prompt typically looks something like this (paraphrased from typical patterns, not any one vendor's proprietary version):

You are a meeting summarizer. The transcript below is from a meeting. Produce a Markdown summary with the following sections: Decisions (3–5 bullet points), Action items (with owner and due date if mentioned), Open questions (unresolved). Do not invent decisions. If you are not certain something was decided, list it as an open question.

That last sentence is the difference between a useful summary and a hallucinated one. Without it, LLMs love to confidently assert decisions that the meeting never made.

The three classic LLM failure modes

• Hallucinated decisions.The model says “the team decided X” when in fact someone said “maybe X.” This is the most dangerous failure for follow-up.
• Lost nuance. A 10-minute discussion with disagreement gets compressed to one bullet. The minority view disappears. Where decisions actually emerge from disagreement, this erases the context.
• Action item drift.The model invents owners or dates that were never assigned. “@theo will review by Friday” when neither Theo nor Friday were mentioned. Easy to spot if you check, dangerous if you do not.

How to mitigate hallucinations

• Use a recent model. Smaller, older models hallucinate more. As of 2026, Mistral Small, Llama 3.x 70B+, and equivalent EU-jurisdictional models are sufficient for meeting summarization.
• Constrained output format.Require structured Markdown sections. Open-ended “summarize this meeting” prompts produce more drift than “list decisions / actions / questions.”
• Lower temperature. Temperature 0.0–0.3 for summary tasks. Higher temperature increases creativity, which is exactly the wrong incentive for factual recap.
• Always show the transcript alongside.If a user can click from a summary line to the transcript that produced it, they spot hallucinations immediately. If they can't, they trust blindly.

The hidden privacy decisions

Behind every AI summary is a chain of providers, each with its own data policy:

1. The meeting tool itself (where the audio first goes).
2. The STT provider (third-party API, or self-hosted).
3. The LLM provider (different from STT in 90% of cases).
4. The storage layer (where the resulting transcript and summary live).

Each layer has its own retention policy, training rights, and jurisdiction. A US-based meeting tool sending audio to a US-based STT then to a US-based LLM creates a three-link chain of US-jurisdictional exposure. Each link can be subpoena'd independently. Each link is a question on your DPIA.

The most defensible architecture in 2026:

• EU-jurisdictional meeting tool.
• EU-jurisdictional STT provider, zero-retention contract.
• EU-jurisdictional LLM provider, zero-retention contract.
• Storage in the EU, audio discarded after transcription.

That is what we built into Punkto. Audio is transcribed in memory by an EU-jurisdictional STT, summarized by an EU-jurisdictional LLM, and the audio buffer is destroyed at the end of the request. Only the text remains.

What to ask before adopting an AI summary feature

1. What STT provider do you use, and where are they incorporated?
2. What LLM do you use for summary generation, and where are they incorporated?
3. Is the audio retained after transcription? For how long? Why?
4. Is my data used to improve your or your providers' models?
5. What temperature do you use for summaries? (0.3 or below is the right answer.)
6. Can I see the transcript that produced a given summary line?
7. What is the disclosed word error rate on my primary language?
8. How are speaker labels assigned, and what is their accuracy?

Vendors that can answer these in under five minutes have thought about the problem. Vendors that need a sales engineer to come back to you have not.

The honest summary about summaries

AI meeting summaries are useful, but they are not magic. They compress hours of audio into a few bullet points, which is exactly the trade-off you wanted — at the price of losing some accuracy and adding hallucination risk. Used as memory aids, they save time. Used as legal records, they will eventually burn you.

The right mental model: an AI summary is what a smart but slightly drunk colleague would write up immediately after the meeting. Useful. Almost always directionally correct. Read with a grain of salt and edit before sharing.

Curious about ours? Punkto generates an AI summary at the end of every recorded session — decisions, action items with owners and dates, open questions, plus the full transcript. EU-hosted, zero audio retention, free for 3 transcripts per month.