Whisper

Whisper is an open-source automatic speech recognition (ASR) system built and released by OpenAI in September 2022. Trained on 680,000 hours of multilingual audio scraped from the internet, it covers 99 languages and handles a wide range of accents, recording conditions, and domain-specific vocabulary. The model weights and inference code are published on GitHub under the MIT license, meaning anyone can download, modify, and use Whisper without paying a license fee. OpenAI also exposes Whisper through its API for teams that want hosted inference without managing their own GPU infrastructure.

As of April 2026, the recommended production version is Whisper large-v3-turbo, released in October 2024, which runs roughly eight times faster than large-v3 at near-identical accuracy on high-resource languages. Community projects have expanded what Whisper can do: faster-whisper repackages the model using CTranslate2 for additional speed gains; WhisperX adds word-level timestamps and speaker diarization; distil-whisper is a Hugging Face-trained distilled variant that is six times faster with 49% fewer parameters. Developers use Whisper across podcast production, meeting transcription, video captitling, voice data pipeline construction, and accessibility tooling.

What Whisper produces in April 2026

Whisper outputs plain text transcriptions with optional timestamps, available as plain text, SRT, VTT, TSV, or JSON. The model family spans five original sizes: tiny (39M parameters), base, small, medium, and large (1.5B parameters). The current production-recommended weights are large-v3-turbo, a distilled variant of large-v3 optimized for throughput without requiring the full memory footprint of its parent.

Processing is chunk-based: Whisper analyzes audio in 30-second segments. This architecture delivers high accuracy on recorded audio but means the model cannot stream transcriptions in real time. For batch workflows like post-processing interviews, lectures, or recorded calls, this is not a limitation. For live captioning or voice agents that need sub-second responses, it is a hard constraint.

Language support spans 99 languages. Quality across those languages is not uniform. English, Spanish, French, German, Japanese, and Portuguese perform near human-level on clean audio. Coverage drops significantly for lower-resource languages. Users transcribing Yoruba, regional Punjabi dialects, or Haitian Creole report error rates that require substantial manual correction. The translation feature converts audio from any supported language directly into English text, which is useful for multilingual content indexing.

The large-v3-turbo release in October 2024 coincided with a separate news cycle: an Associated Press investigation published the same month documented that Whisper systematically hallucinated text in medical transcription contexts, generating plausible-sounding sentences during silent audio segments or ambient noise. OpenAI's own documentation already included a caveat that the model "may generate text that was not spoken," but the AP report brought this to wider attention and raised questions about production deployments in healthcare and legal settings. The hallucination behavior is architectural, not model-size-dependent, and persists across all Whisper variants including turbo.

"I ran Whisper on silent audio segments to test and it generated plausible-sounding filler text. Not just noise, coherent sentences that were never spoken. This is a known failure mode and it's alarming for medical use." - u/quietcompute, r/MachineLearning, January 2025

Where Whisper sits versus AssemblyAI Universal-3 and Deepgram Nova-3

The honest comparison here is open versus closed, and batch versus real-time. Whisper is the only option in this comparison that you can run offline, fine-tune, and inspect fully. AssemblyAI and Deepgram are closed commercial APIs.

AssemblyAI Universal-3 (released March 2024) is a proprietary encoder-decoder model trained on licensed audio datasets, which AssemblyAI claims are cleaner than Whisper's internet-scraped training corpus. The practical difference: Universal-3 supports real-time streaming transcription via WebSocket with sub-300ms end-to-end latency, which Whisper's architecture cannot match. Universal-3 also includes built-in speaker diarization, eliminating the need to assemble a separate pipeline. On conversational English, independent benchmarks broadly support AssemblyAI's claim of lower word error rate than Whisper large-v3. Pricing starts with a free tier (five hours per month) then moves to $0.37 per hour. Whisper wins on language breadth (99 vs. fewer), zero self-hosted cost, code transparency, and no vendor dependency.

Deepgram Nova-3 (released January 2025) uses a non-autoregressive streaming architecture, architecturally opposite to Whisper's autoregressive encoder-decoder design. This allows Nova-3 to deliver streaming transcription with under 200ms latency and word-level confidence scores in real time, making it the benchmark choice for voice agent and live captioning deployments. Nova-3 covers 36 languages natively, fewer than Whisper but with higher consistency across those languages. API pricing is $0.0043 per minute, cheaper than OpenAI's Whisper API at $0.006 per minute. Whisper wins again on openness, self-hosting, and the full 99-language range.

The practical choice usually comes down to one question: do you need real-time output? If yes, Whisper is not viable without complex chunking workarounds that still introduce latency. If you are processing recorded audio in batch, Whisper (self-hosted) delivers competitive or better accuracy at zero per-minute cost at scale.

"whisper-large-v3-turbo is genuinely impressive. I'm transcribing 4-hour interview recordings and the WER is comparable to what I was paying Rev.com $0.25/min for. The speed improvement over v3 is enormous." - u/mlpraktiker, r/MachineLearning, November 2024

The licensing and copyright reality

Whisper's MIT license is one of the most permissive in AI: you can use it commercially, redistribute it, modify it, and build products on top of it without paying royalties or obtaining additional permissions. The license covers the model weights and the inference code. There is no "commercial use requires enterprise agreement" clause that some open models include.

The self-hosted path is entirely free. You download weights from Hugging Face Hub, install the Python package, and run transcription locally. GPU infrastructure costs are yours, not OpenAI's. A developer running large-v3-turbo on a rented A10G GPU through a cloud provider pays the GPU rate, not a per-transcription fee. At scale, the economics are compelling: a podcast production house processing 1,000 hours per month would pay roughly $360 at OpenAI API rates, or a fraction of that in GPU compute running faster-whisper locally.

The OpenAI API path charges $0.006 per minute of audio processed. This is a standard pay-as-you-go rate with no subscription requirement. Teams that want managed infrastructure without owning GPU hardware use this route. There is no monthly subscription or seat pricing. The API endpoint (`api.openai.com/v1/audio/transcriptions`) accepts audio files up to 25MB and returns transcription in the requested format.

Community forks like faster-whisper and distil-whisper are also MIT licensed. WhisperX uses pyannote.audio for diarization, and pyannote has its own access model on Hugging Face that requires account-gating before downloading certain models. This has caused friction for teams trying to automate their pipeline setup, since the gating requires a manual token request step.

Where Whisper reliably falls short

Hallucinations on silence and low-signal audio. This is the most serious documented limitation. Whisper generates confident-sounding text during segments that contain no speech, ambient noise, or inaudible audio. The `no_speech_threshold` and `logprob_threshold` parameters reduce this behavior but do not eliminate it. For verbatim accuracy requirements (medical, legal, official minutes), every output segment needs validation. This is not a theoretical concern: the October 2024 AP investigation documented specific incidents where Whisper inserted words never spoken into medical transcription output.

No built-in streaming. Whisper's 30-second chunk architecture is a fixed property of how the model was trained. It cannot produce partial transcriptions as audio arrives. Community workarounds involve voice activity detection (VAD) to split audio into sentence-level chunks, but these add latency and complexity. For voice agents, live meeting captions, or any use case where output must appear within 500ms of speech, Whisper requires significant engineering around its core limitations.

Speaker diarization requires third-party assembly. Whisper's output is a flat transcript with no speaker labels. Adding diarization requires WhisperX plus pyannote.audio, which involves separate installation, a Hugging Face account with gated model access, and additional GPU memory. Teams coming from commercial APIs that include diarization as a single toggle find this integration work significant.

GPU requirements for large-v3. Running large-v3 at useful speed requires at minimum 10GB VRAM in fp16. Consumer GPUs like the RTX 3060 (12GB) hit out-of-memory errors with default settings. large-v3-turbo improved this substantially, but developers who want maximum accuracy on large-v3 still need serious hardware. Small and medium models run acceptably on CPU but produce noticeably worse output on challenging audio.

Language quality variance. Whisper supports 99 languages, but the quality distribution is steep. The model performs near human-level on English and major European languages; performance on lower-resource languages is inconsistent enough that users processing Yoruba, regional Arabic dialects, or less-common Asian languages frequently describe the output as requiring complete manual review.

Who Whisper is for

Whisper is purpose-built for developers. There is no official web UI, no drag-and-drop interface, and no managed dashboard from OpenAI. Getting started requires Python, pip, and either a CUDA-enabled GPU or the patience to run smaller models on CPU. If that describes your setup, Whisper is one of the most capable free tools available for transcription at any scale.

The strongest use case is cost-sensitive batch processing: podcast production, video subtitle generation, archiving recorded lectures, transcribing interview audio for research, processing call center recordings. At 1,000 hours of audio per month, a self-hosted faster-whisper setup costs GPU compute only. The same volume through a commercial API runs $360 or more. The economics are not close.

Whisper also fits offline and air-gapped deployments. Healthcare IT teams, government contractors, and legal firms that cannot send audio to external APIs can run Whisper entirely inside their own infrastructure without any data leaving their environment. The MIT license means this use is permitted without restriction.

Skip Whisper when: you need live transcription under 500ms latency for voice agents or interactive applications; when your users are non-technical and need a web interface; when you process medical or legal audio where hallucination risk is unacceptable without a dedicated validation layer; or when you need production-grade speaker diarization out of the box. In those cases, AssemblyAI Universal-3 or Deepgram Nova-3 solve the real-time and diarization gaps, at the cost of vendor dependency and per-minute pricing.