Whisper on the Neural Engine: what on-device inference actually costs

Every AI feature eventually reaches the same fork: call a hosted API, or run the model on the user’s machine. The decision usually gets made on vibes — hosted because it is easier, or local because it sounds more private — and then nobody revisits it.

I had a reason to actually measure. I dictate a lot, I wanted the transcript on my clipboard, and I did not want my voice leaving the machine. So I built a menu bar dictation app for macOS: press a hotkey, speak, press it again, and the transcript is on the clipboard. Roughly 2,500 lines of Swift, one day, transcription via WhisperKit on the Apple Neural Engine. Nothing touches the network after the model downloads.

What follows are the four decisions that actually mattered. Only one of them was about the model.

The model name lied, so I benchmarked

WhisperKit’s model repository offers variants with a _turbo suffix, and the obvious reading is that this means OpenAI’s large-v3-turbo. It does not. The suffix denotes Argmax’s own Neural Engine optimisation. The component that actually identifies turbo is the date:

Variant	decoder_layers	Actual model
openai_whisper-large-v3-v20240930_turbo	4	large-v3-turbo, ANE-optimised
openai_whisper-large-v3_turbo	32	full large-v3, ANE-optimised

Four decoder layers against thirty-two. Had I gone by the name I would have shipped the wrong one and concluded that on-device Whisper was too slow.

Measured on an M3 Max, warm, second run, against real speech rather than text-to-speech samples:

Model	24.6 s English	16.8 s Russian	Language detection
large-v3-turbo	0.69 s (35.8x)	0.85 s (19.9x)	correct on every run
full large-v3	2.63 s (9.4x)	3.38 s (5.0x)	returned fr, ro, es

The smaller model was roughly four times faster, produced identical English text, and was the only one that got the language right. Full large-v3 misdetected Russian as Romanian badly enough that it transcribed the audio as Romanian — confidently, and in the wrong alphabet.

This is the part worth generalising. The bigger model was not a safer default; it was worse on both axes I cared about. If I had picked by parameter count, or by the name, I would have been wrong twice. The benchmark took an afternoon and a handful of real recordings.

Warm model load is about 5.6 s and resident memory about 0.8 GB, which for a tool that lives in the menu bar all day is the real cost — not the inference.

Clipboard, not synthesised keystrokes

The obvious design is to type the transcript into whatever app has focus. I deliberately did not.

Injecting keystrokes requires Accessibility permission, and macOS Secure Input silently discards synthetic events in password fields and terminals — so the feature would fail exactly where a failure is most confusing. Going through the clipboard means the microphone stays the only permission the app ever requests, and behaviour is identical everywhere.

The cost is one ⌘V and a clobbered clipboard. I took it. A permission you never request is a support burden you never carry, and “works everywhere except your terminal” is not a feature.

The silence threshold could not be a constant

Auto-stop needs to know when you have stopped talking, and the tempting implementation is a fixed threshold in dBFS. I tried that first. Measured speech on this hardware peaks at about −37 dBFS, so a fixed −42 dBFS threshold classified 92% of active speech as silence and cut recordings off mid-sentence.

Absolute levels are meaningless across microphones — they depend entirely on gain. The threshold is now relative: silence is anything 20 dB below the loudest speech in the current recording, bounded to −65…−35 dBFS.

I did not arrive at 20 dB by intuition. I built a small tool that replays a real recording through the actual detector and prints the level timeline, the derived threshold, the longest continuous silence, and where auto-stop would have fired. Across two real recordings, a 15 dB drop left gaps of 3.4–3.8 s inside continuous speech; 20 dB reduced that to 1.9 s; beyond 25 dB the threshold falls under the room’s noise floor and auto-stop never fires at all.

Tuning a heuristic against a recording you can replay takes an hour. Tuning it by using the app and getting annoyed takes weeks and never converges.

The shipping tax nobody budgets for

The genuinely expensive bug had nothing to do with audio or models.

SwiftPM generates a Bundle.module accessor for packages that carry resources. It looks for its bundle in Bundle.main.bundleURL — which for an application is the .app directory itself — and otherwise falls back to a path hardcoded into the build machine’s .build directory. Inside a signed app the only permitted placement is Contents/Resources, where the accessor never looks, so it calls fatalError. Put the bundle where the accessor does look, at the top level of the .app, and the signature becomes invalid with “unsealed contents present in the bundle root”. Symlinks fail identically. There is no placement that satisfies both.

On the machine that built it, everything works, because the hardcoded fallback path still exists. The crash only appears on someone else’s computer.

That class of defect — invisible on the build host, guaranteed on every other machine — is why “it works on my laptop” and “it ships” are separated by more than a build script. The fix was to replace the one view that used the accessor. Finding it cost considerably more than fixing it.

What I would take to a client conversation

The app is not the point; there are several good dictation tools already. The point is that four decisions on a one-day project each had a measurable right answer, and in three of the four the intuitive choice was wrong:

• The larger model was slower and less accurate, and the naming pointed the wrong way.
• The more capable integration (keystroke injection) was the more fragile one.
• The simpler heuristic (fixed threshold) misclassified 92% of speech.
• The build succeeded on the only machine where success proved nothing.

On-device inference is now genuinely viable for a real workload on consumer hardware: sub-second transcription, no network, roughly 1.5 GB on disk and 0.8 GB resident. That is a serious answer to “should this feature call an API”, and it is available to anyone willing to spend an afternoon measuring instead of an afternoon guessing.