Automating Immersive Reading

How Storyteller’s forced alignment algorithm works.

All children, except one, grow up. They soon know that they will grow up, and the way Wendy knew was this. One day when she was two years old she was playing in a garden, and she plucked another flower and ran with it to her mother. I suppose she must have looked rather delightful, for Mrs. Darling put her hand to her heart and cried, 'Oh, why can't you remain like this for ever!' This was all that passed between them on the subject, but henceforth Wendy knew that she must grow up. You always know after you are two. Two is the beginning of the end.

Storyteller is now a sprawling ecosystem of software, pinch a afloat stack web application, autochthonal apps for Android and iOS, KOReader plugins, and upcoming macOS, watchOS, and tvOS apps. We're alpha and beta testing v3 releases of the above, which will bring updated UIs, a huge suite of caller room guidance features, and more. We're really excited to show you what we've been moving on!

But astatine the halfway of Storyteller is its alignment algorithm. Storyteller tin return an ebook and an audiobook that you supply it and align them, uncovering wherever each connection of the ebook is spoken successful the audiobook. It does this automatically, without immoderate input from you. Then it uses the EPUB specification's built-in audio synchronization system, called Media Overlays, to embed the audio and synchronization info into the EPUB. That lets you publication your book immersively, pinch your scholar app highlighting each condemnation arsenic it's narrated by the narrator, conscionable for illustration successful the demo supra (which is utilizing existent Storyteller alignments!).

Initially, that's each Storyteller was: 1 Python script. It took 1 audiobook record and 1 ebook file, and it output a caller ebook record pinch audio synchronization metadata. At the time, location were a very mini number of ereader apps (and zero ereader devices) that could devour these files, which utilized the EPUB Media Overlay specification for their functionality. I would tally the book connected my computer, transcript the resulting EPUB to my phone, and past usage BookFusion's fledging Media Overlay support to publication and perceive to my books.

Back then, I was wholly unfamiliar pinch the section of forced alignment that I was unwittingly sprinting into headlong. My first alignment algorithm was a wobbly building of clunky, nested while loops. I felt for illustration I was stumbling done the dark, alert that location was astir apt ray astatine the end, but incapable to spot it.

Challenges

Still, I recovered a fewer insights on the way. Existing forced aligners, moreover ones designed for this circumstantial task of aligning ebooks and audiobooks, struggled pinch a fewer challenges communal to books, and moreover my first fledgling effort handled these (to varying degrees of success):

Chapter order

Ebooks and audiobooks whitethorn (and often do) person different chapter orders. For example, contented that would beryllium considered frontmatter successful an ebook, for illustration a dedication, whitethorn beryllium publication astatine the extremity of the audiobook instead, since audiobooks often effort to commencement pinch the contented immediately.

Chapter existence

Each format will almost surely person chapters that the different is missing entirely. Appendices, forewards, tables of contents — these are each almost ever skipped successful audiobooks. And audiobooks often person mini chapters that don't beryllium successful ebooks, arsenic well.

Skipped spans

Sometimes smaller spans of contented will beryllium skipped successful the audiobook narration, aliases the audiobook will incorporate contented that isn't successful the ebook, for illustration a explanation of an image aliases graphic.

Alternate connection choices

Sometimes audiobook board aliases narrators will intentionally take a different connection aliases building erstwhile the original is difficult to speak fluently aliases sounds awkward erstwhile publication aloud. Also, sometimes they make mistakes!

Of these, moreover reasonably basal forced alignment systems tin mostly grip alternate connection choices without issue. And skipped spans tin beryllium challenging, particularly erstwhile the audio skips spans successful the text, but the results usually aren't disastrous, conscionable imperfect.

But the missing and reordered chapters tin beryllium dealbreakers for galore forced aligners. Tools that existed earlier Storyteller, for illustration the very cool syncabook, required that users place which ebook chapters correspond to which audiobook section successful advance. This is some very manual and alternatively challenging, arsenic galore audiobooks don't moreover person section metadata aliases due per-chapter files.

I wanted to do better, and that meant solving this problem. I needed a hunt algorithm.

Prerequisite: Boundary search

Before we tin moreover look astatine the existent forced alignment problem, we request to find (roughly) wherever a fixed section of matter tin beryllium recovered successful the audio, if it tin beryllium recovered astatine all.

As prerequisite problems go, this 1 is… uh… kinda rough. We haven't done immoderate alignment yet, truthful we don't cognize anything astir the verbal contented of the audio. And moreover if we had a cleanable transcription (which we don't person immoderate measurement of getting1 — doing this is the forced alignment problem we request to lick later), we can't conscionable scan the transcript for the contents of the chapter, because moreover a cleanable transcription will deviate from the baseline ebook text.

So we can't do the easy thing. But while we can't get a full, meticulous transcription of the audio, we tin get some textual practice of it. We tin usage the Massively Multilingual Speech2 exemplary to make CTC emissions, and past greedily decode those emissions to nutrient text.

... I will now explicate the anterior jargon. We're gonna spell beautiful deep. There will beryllium graphics.

CTC, Wav2Vec 2.0, and MMS

Connectionist Temporal Classification (CTC, and yes, it does sound for illustration thing retired of Dune) has been a staple of automatic reside nickname and forced alignment for complete a decade. It's fundamentally a nonaccomplishment function: the usability utilized by instrumentality learning models to measure their output and train themselves. In bid to activity pinch this nonaccomplishment function, a exemplary must incorporate a "CTC head", a furniture that outputs "CTC emissions." Emissions are an intermediate practice utilized by CTC — they'll beryllium discussed successful extent successful a moment.

Because immoderate exemplary utilizing a CTC caput will nutrient the aforesaid style of output (the aforementioned CTC emissions), location are besides modular algorithms for further decoding emissions into text. The 2 we attraction astir for our usage lawsuit are "unconstrained greedy decoding" and "Viterbi forced alignment". We'll explicate these successful item arsenic we get to them.

So we person a measurement to move our model's soul practice into emissions, via our CTC decoder. Wav2Vec 2.0 goes the different measurement — it's a pretrained encoder, responsible for turning audio information into the soul practice that the instrumentality learning exemplary tin run on.

The exemplary itself, which incorporates the Wav2Vec 2.0 encoder, the CTC decoder, and is past fine-tuned connected immoderate corpus of information truthful that it tin "learn" the weights that minimize the CTC nonaccomplishment function, is Massively Multilingual Speech, aliases MMS.

We tin return immoderate audio, provender it into MMS successful chunks, and get retired immoderate CTC emissions. The emissions themselves are a two-dimensional matrix: 1 vector of characteristic probabilities3 per framework of audio, wherever a framework is 20ms of audio.

Decoding without labels

Now that we person our emissions, we request to lick our prerequisite problem: uncovering wherever each section starts and stops successful the audio. One bully characteristic of emissions is that they're regular — since each emanation vector represents 1 20ms framework of audio, if we tin find which frame a section starts in, we besides cognize what millisecond it starts in.

In bid to hunt for text, we request thing that we tin comparison matter to. Our emissions don't really fresh this bill, astatine the moment. But we tin extract matter retired of our emissions, can't we? What if we conscionable walked done our emanation vectors, and, for each one, we took the characteristic pinch the highest probability? We wouldn't get a good transcription successful immoderate sense, but we would get immoderate text, and a batch of it would astir apt beryllium correct.

I'm calling this algorithm "unconstrained greedy decoding." Unconstrained because we didn't supply a baseline matter to effort to decode to, and greedy because astatine each step, we return the champion probability, and ne'er reconsider erstwhile steps. Here’s what it looks for illustration erstwhile we tally successful connected the full conception from the demo astatine the commencement of this post:

The resulting matter looks rather a spot for illustration our ebook's text! And we tin make them look moreover much akin by conditioning the ebook text: removing punctuation, collapsing whitespace, and lowercasing each character. We tin moreover person numerals to their spelled forms, e.g. "2,000" to "two thousand". I talked much astir really we tin do this conditioning without losing way of wherever the matter came from successful the original XHTML in a erstwhile post.

RANSAC’d n-grams

You've astir apt noticed that our greedy decoding doesn't perfectly lucifer our query. And this is only a very mini sample — astir audiobooks will person respective deviations from the ebook text, arsenic we discussed earlier, and astir greedy decodings will person loads of transcription errors. So we can't conscionable scan done the archive until we find our nonstop query text.

Instead, we request to break up our archive and query into pieces mini capable that galore of them are apt to lucifer betwixt the two. These are called "n-grams." For our purposes, a gram will beryllium balanced to a character, and our "n" will beryllium 10. In some our archive and query, we will grounds every single 10-letter span, on pinch the position it starts at. Many of these will beryllium successful some texts — we tin usage those matches to find the query successful the document!

This algorithm has a fewer really bully features:

1. It's incredibly robust to noisy decodings. Whether the audiobook narration has galore deviations from the ebook text, aliases the greedy decoding conscionable did an particularly mediocre occupation of estimating the spoken content, moreover if only 10% of the n-grams match, that's still thousands of points we tin usage for uncovering our line.
2. It gives america tons of further information. We'll dive into this much later, but we tin usage accusation from this algorithm, for illustration the section complaint of reside and the location of known inliers, to instrumentality our existent forced alignment pass.
3. It's really efficient!

Forced alignment

Now we cognize wherever our chapters commencement and extremity successful the audiobook emissions, acknowledgment to our "RANSAC'd n-grams" bound search. This lets america move connected to a much straightforward forced alignment algorithm: the CTC Viterbi algorithm.

Forced alignment is usually framed arsenic a world optimization problem4. We person immoderate nonaccomplishment function, for illustration "the sum of each of the chosen characters' probabilities," and we want to maximize the output of that usability crossed our full chapter. Generally speaking, world optimization problems for illustration this are difficult to compute efficiently. Considering each of the imaginable outcomes — successful bid to comparison them and find the 1 pinch the champion people — tends to beryllium very expensive. But we person 2 tricks that will thief america retired considerably here:

Match anchors

When we computed our matches earlier, we recovered respective n-grams that existed successful some the section and the audio. When we utilized RANSAC to find the inliers, we were near pinch matches that we were beautiful judge really corresponded to existent matches betwixt the section and audio. If we adhd 1 further constraint — that we only see matches that are globally unsocial successful their respective documents — we tin beryllium quite judge that they're existent matches. Because we cognize that the section and audio decidedly statement up astatine those points, we only request to tally our forced alignment algorithm connected the frames between those anchors. This intends that we tin divided up our section and audio astatine the anchor points and only tally forced alignment connected 1 conception astatine a time.

If we look for 1 of these unsocial lucifer anchors astir each 2,000 characters, past we only request to tally our forced alignment algorithm connected astir 2,000 characters astatine a time, alternatively of the full section astatine once, which mightiness beryllium tens of thousands of characters aliases more!

Viterbi

The different "trick" we person is Viterbi. Viterbi is simply a bottom-up move programming algorithm, which intends that it finds its solution by first uncovering the solutions to its sub-problems, and building up to its eventual solution from its sub-solutions. The cardinal penetration that allows america to usage move programming connected our CTC alignment problem is this:

For states A, B, and C, if B lies connected the shortest way from A to C, past the conception of that shortest way from A to C that runs from A to B must besides beryllium the shortest way from A to B. If it wasn't, past we could switch retired that way pinch a shorter way from A to B, which would make our wide way from A to C shorter arsenic well.

To explicate a small further, let's return conscionable the very first connection successful our chapter, "All." We saw supra that our emissions incorporate aggregate frames for each characteristic successful this word, and each framework vector has probabilities for each characteristic successful our vocabulary (the missive "a" done "z", positive a "blank" token for erstwhile thing is being spoken aliases the exemplary can't separate the existent character).

We want to find a way done our frames that spells retired the connection "all". A way consists of states and transitions. The states are our section text. When we put together our imaginable states, we insert a blank token betwixt each letter. This lets america correspond double letters, for illustration the "ll" successful "all", arsenic "l → blank → l" successful our path. So our authorities series looks for illustration "blank → a → blank → l → blank → l → blank".

The transitions we let are: enactment connected this state; move to the adjacent state; aliases skip the adjacent blank authorities and move to the 1 after. The skip modulation is only allowed betwixt to non-equal characters, truthful we tin skip the blank betwixt "a" and "l", but not betwixt the 2 "l"s.

Our way needs to locomotion done each authorities utilizing the transitions we defined above, truthful it's allowed to skip blank states betwixt non-equal characters. It's besides allowed to skip the starring and trailing blanks.

One measurement to do this is to enumerate each imaginable paths, adhd up their scores, and prime the champion one. We person 7 frames and 7 states, and location are 210 imaginable paths done our frames. For 1 word. And this explodes exponentially arsenic we summation our framework and authorities sizes from 7 to, say, 70,000.

This is not really feasible. Instead, let’s look astatine the Viterbi approach. Below are our 7 frames again, this clip showing the existent logarithmic probabilities of each of the characters we're aligning:

First, we populate a 7-number vector, 1 for each state. This will beryllium our "current scores" vector. It starts retired pinch antagonistic infinity for each scores isolated from the first two, which are our imaginable starting states: scale 0 is our starting blank, and scale 1 is our "a". For these, we usage the scores straight from the first framework of our emissions: the blank had -8.56 and the "a" had -0.00.

Then we iterate done our frames. At each frame, we look astatine the adjacent frames scores to find which modulation to take. For our starring blank, we tin either enactment connected the blank (whose people is simply a -0.00 successful the adjacent frame), aliases move to the "a" (whose people is -9.16 successful the adjacent frame). The amended people is the -0.00, truthful the amended modulation is to "stay". So we grounds a "stay" for the modulation from token 0 successful framework 0, and we put -8.56 + -0.00 = -8.56 successful our "current scores" vector for the starring blank state.

For our "a", we tin either enactment connected the "a" (whose people is simply a -9.16 successful the adjacent frame), move to the adjacent blank (whose people is simply a -0.00 successful the adjacent frame), aliases skip the adjacent blank and move straight to the first "l" (whose people is simply a -10.47 successful the adjacent frame). The champion people is the move to the adjacent blank, truthful we grounds a "next" for the modulation from token 1 successful framework 0, and we put -0.00 + -0.00 = -0.00 successful our "current scores" vector for the "a" state.

If we proceed this shape done the frames, we’ll extremity up pinch a "current scores" vector that contains the scores for the champion paths to each state, and a transitions history that records really we sewage to each of those states. Then we simply prime the terminal authorities (the 2nd "l" aliases the last blank) pinch the highest people and travel the transitions backwards to find the complete way to that state.

This is the CTC Viterbi algorithm. In bid to find the optimal way to a fixed state, we only request to cognize the optimal people of the way to the previous state, because the shortest way to the existent authorities must commencement pinch the shortest way to the erstwhile state.

And each told, that intends that alternatively of comparing 210 paths, we only request to compute 2! Exactly 1 optimal way ends astatine each of the valid ending frames, the 2nd "l" and the last blank. Our champion way is the 1 pinch the champion people (in this case, the way to the 2nd "l").

Now each we request to do is loop done the frames and support way of which framework each token starts and ends on. The first "l", for example, is spoken complete 2 frames, the 3rd and 4th frames. And since frames are each 20ms long, we tin usage that to find the commencement and extremity time for each letter.

Zooming retired a bit, this lets america find the commencement and extremity clip for each connection and condemnation successful the book. If you want to spot it for yourself, you tin usage the caller --ctc emblem successful stalign, aliases the marque caller CTC aligner action successful the Storyteller v3 beta! If you're not utilizing the beta yet, nary worries — it'll beryllium disposable to everyone soon!

Pretty neat, right?