Aligning 800 audiobooks (8,000 hours) to text in 6 days, no LLM in the loop

On July 14th, I started a git repo called tale. Several months before I had built a simple prototype for an old manuscript from college that I unearthed called the man with the patchwork heart. The thesis was simple, both when I wrote it in 2009, and today in 2026: could literature work on the web? In addition to my own writing, which I only have so much of, I knew the corpus of Standard Ebooks could fill in thousands of public domain works, and if I could get them all working well on a website, with a beautiful reading experience, I could prove that the browser could be a great place to read literature.

Over the next 8 days, I applied the same agentic coding discipline that I had developed at Lemontree, and refined on another codebase in June, to develop an elegant single page reader, which stored your progress across devices with anonymous-first notations anyone could copy with simple, readable urls, no account required.

Then on July 22nd around midnight I had an idea which I think became the first truly magical feature of tale.fyi: what if we could listen to an audiobook along with our reading, so that we could seamlessly switch from listening to an audiobook in the car, to reading during our lunch break. After a quick Google, I discovered the incredible performances on LibriVox, all released as public domain, which could form our first proof-of-concept: a whisper-sync for the open web. My agents and I made two rules: the audiobooks must be aligned word-by-word to the text, and there must be high confidence that the alignment is solid before I ship the read-along.

With that decided, I set a /goal on my agents and went to bed. The agents originally thought the aeneas library would be the answer, because our measured test runs were fast (380x realtime on my MacBook's CPU, 60x faster than forced CTC alignment), but after running through a bunch of books, they decided at 3:12am that they just weren't matching our ASR ground truth: paragraph 27 of Dracula's first chapter placed at 2072s against the actual ~1490s (+582 seconds), the chapter's back third crushed into 146 seconds. More critically: aeneas does not export a confidence score, so I couldn't even flag where it was losing confidence. At 2:24pm, with me back at the keyboard, I landed on a more modern, higher confidence solution: a CTC neural alignment using Meta's Massively Multilingual Speech model. What I first rejected for being too slow was sped up dramatically by moving the processing to GPU: 106x realtime. It was 3x slower (not 60x), but more accurate, and most critically, emitted confidence scores (probabilities).

Why not transcribe, then align? Transcribe-then-align always emits something: on a mismatched edition, something subtly wrong; forced alignment aligns text to acoustics directly, one model and one error source. Word timing resolves to 20ms, and when it can't lock, it abstains. "We prefer abstaining" is the exact wording in the README, which has guided this whole project.

Evals ended up being critical for refining the sync script. Books with optional front matter caused trouble. In Beowulf, the narrator skips the translator's preface and endnotes, so Beowulf passed the median-only gate while beginning three minutes in. Aesop's Fables placed only 39 of 380 paragraphs, smeared confidently across the whole audio (span coverage 100%, median 0.93) and it shipped broken; I added a new whole-book gate to prevent that from happening again.

Out of memory errors crashed the aligner on long books. I developed a streaming from disk technique, which only applies on books longer than 15 hours, since it costs clock time.

I still had crashes, and "big books run out of memory" turned out to be the wrong tree: the 1.6-hour Tao Te Ching died while the 17.9-hour Tess of the d'Urbervilles sailed through. The real culprit was sparse alignment: memory blew up with the gap between anchored passages, not the length of the book. I capped the gap at 300 seconds.

Finally, on July 28th, six days after the midnight idea, I had a completed run, without crashes: all 816 mirrored audiobooks, about 8,000 hours of narration, each with a verdict. 553 shipped a read-along; the gates rejected the rest. Over the next few days, we kept iterating to rescue over a hundred of the initially rejected books.

Now you can enjoy 668 audiobooks, all aligned, all resumable from either text or audio on tale.fyi/audiobooks, all from a midnight idea on July 22nd. The cost? $0 compute (it all ran on my MacBook) and $6 per month in storage.