Markdown Broke Our Text-to-Speech

For a while now, the most common complaint I've had about DemoPolish was some version of the same sentence: the voiceover doesn't line up with the video.

It's about the worst bug this product can have. The entire promise is that you upload a screen recording and get back the same video with a better voice on it. If the new voice is describing a button you clicked four seconds ago, there's nothing left to sell. It doesn't matter how good the writing is.

I found it this week. It wasn't what I thought it was, and the wrong answer was convincing enough that I nearly shipped a fix for it.

The obvious suspect

Here's how the pipeline works. Your video gets transcribed, which gives us the words and the timestamps — when each sentence started and ended in your original narration. An LLM rewrites the script. A speech engine reads the rewrite. We put that audio back on your video.

The timestamps are the whole game. They're the ground truth for when narration should be happening, because you recorded yourself describing what was on screen at that moment.

So when audio drifts, the obvious culprit is speaking rate. The rewriter is told to write a script that fits, and it budgets words using an assumed speaking rate — a constant, in a config file, that says roughly how many words per minute the voice will produce. If that constant is wrong, every script comes out the wrong length, and the error piles up as the video goes on.

That constant was 150 words per minute. I was fairly confident it was wrong, because it had been chosen when we were using a different speech engine and nobody had re-measured it since. Case closed, surely.

Measuring it, and getting it wrong

Before changing the constant I wanted a number, so I rendered the same 65-second demo through the pipeline and compared where speech happened in the original against where it happened in the output.

The result was startling. The original recording, by my measurement, was only 41% speech — the narrator talked, paused while clicking, talked again. The render was 90% speech. We weren't just drifting; we were filling every silence with words.

It was a great theory. It explained everything. It was also completely wrong.

My speech detector used a fixed loudness threshold to decide whether a chunk of audio contained talking. That's fine when you compare like with like. I was comparing a quiet screen-recording microphone against a synthesised voice mastered at a healthy level. Half the original narration sat below my cutoff, so I'd scored it as silence.

What gave it away was a number that couldn't be true. My measurement implied the original narrator was speaking at 342 words per minute. Nobody talks like that. Auctioneers don't talk like that.

I recalculated the threshold per file, relative to each recording's own loudness, and the finding evaporated. The original was 85% speech. The narrator talked almost continuously, exactly as you'd expect. My beautiful theory had been an artifact of my own measuring tape.

I'm including this because it's the part I'd want to read. It is genuinely easy to produce a confident, coherent, well-supported conclusion about a bug that doesn't exist, and then go and "fix" it. The only thing that saved me was one number being implausible enough to notice.

What was actually happening

With a working measuring tape, I went back and dumped the exact script the pipeline was handing to the speech engine. There it was:

To get started, click **Campaigns** and navigate to **Email Campaigns**. Next, enter your **From Name** and **From Email Address**.

Markdown. Asterisks, wrapped around every interface label, going straight into a text-to-speech engine.

Obvious in hindsight, and obvious in origin. The rewriter is an LLM. When you ask an LLM to write instructions about software, it bolds the buttons — that's what good technical writing looks like, and it has read an enormous amount of it. Nobody told it the text was going to be spoken aloud rather than displayed, so it formatted it the way it formats everything.

Nothing between the rewriter and the speech engine was checking. There was a second contaminant too: the rewriter returns its output as pipe-delimited rows, and a parsing bug was leaving a stray | at the end of the text. That went to the engine as well.

So I measured the cost. Same 141 words, once as we were shipping it and once with the markup stripped:

• With the Markdown: 68.87 seconds
• Cleaned up: 51.70 seconds

17.17 seconds of pure garbage on a 63-second video. A third of the runtime, spent on characters that nobody can hear.

Speech engines don't skip things they can't pronounce. They stall on them. Every ** was a little hitch, and there were sixteen of them.

Why it broke everything at once

Here's the part I found interesting. That inflation didn't cause one failure. It caused two different ones, depending on which way a video happened to be rendered.

The default path synthesised the whole script as one continuous piece of audio and laid it over the video. There's no timeline placement in that approach at all — it works only if the script's spoken length happens to match the original narration's length. Inflate the script by a third and it can't. The audio just slides, and it slides further the longer the video runs.

The other path places each line at its own original timestamp and gently adjusts pacing to make it fit. That one can't drift. But it also can't work miracles: there's a hard limit on how much you're allowed to speed a line up before it stops sounding human. Feed it a script that's 20% too long and it hits that ceiling and stays there. No drift — just narration that sounds rushed and clipped.

Two entirely different symptoms. One cause. Users experienced both and described both as "out of sync," which is a completely fair description of each.

The part I'm least proud of

We built that second path — line-by-line placement with pacing adjustment — back in July. It works. It's careful code, and it solved this problem properly.

It was only switched on for videos where you'd edited the script yourself.

The reasoning at the time was defensible: an edited script carries no guarantee about length, so it obviously needs fitting, whereas a freshly generated script was written to a word budget and should already fit. That assumption was doing an enormous amount of unexamined work. It was true only if the word budget was accurate — and the budget was being computed against a script full of characters that would be silently inflating its runtime by a third.

So the fix existed, in the codebase, for weeks. It was wired to the path almost nobody takes.

The pipeline even noticed the problem. Buried in the logs, on every affected render:

WARNING Duration mismatch: 4.9s difference. Script may need adjustment

It detected the drift, wrote it down, and merged the video anyway. A warning nobody reads is not a safety net. It's a diary.

What we changed

Four things, in the order they matter.

Nothing reaches the speech engine unsanitised. Markdown, backticks, headings, link syntax, stray delimiters — all stripped at the boundary, on every path. The prompt also now tells the model it's writing for a speech engine and to skip the formatting. Both, deliberately: the instruction reduces the problem, and the sanitiser means we don't depend on the instruction being followed. You should never let model output reach something that isn't a text box without cleaning it first.

Every render now gets timeline placement. Not just edited ones. Each line goes back at the timestamp it came from.

The render checks its own work. After the audio is fitted, the pipeline verifies that each line actually fits where it's supposed to go and produces a verdict that travels with the job.

And if that verdict is bad, you're told. Your video still renders and it's still yours to download — we're not going to withhold your work because a threshold tripped. But you'll see a note suggesting you watch it before you share it. The failure mode we just lived through was people receiving a broken video that looked, by every signal available to them, completely successful. That's the thing worth never repeating.

On the video I'd been testing with, the same render now reports every line fitting its slot, with no compression needed anywhere.

Three things I took from it

The obvious suspect deserves evidence too. The speaking-rate constant really was slightly wrong. It was a real bug, sitting in plain sight, and it was not the one hurting anyone. Had I "fixed" it without measuring, I'd have shipped a change, seen a small improvement, and stopped looking.

Measure the thing you actually ship. The rate constant came from a different speech engine. The word budget was calculated on text that included characters the engine would choke on. Every number in that chain was reasonable in isolation and wrong in combination, because none of them had been checked against the system as it actually ran.

A detector that only logs is not a detector. The single most useful line of code in this whole fix isn't the sanitiser. It's the check that turns a mismatch into something a human sees. We had the numbers to catch this the entire time. We just weren't looking at them.

If your voiceovers have been landing a beat behind the action, re-render them — the fix is live. And if you hit anything that still doesn't line up, tell me. That complaint turned out to be worth a great deal.