Transcriber R&D project

I’ll present a custom-built tool designed to evaluate the quality of timestamps and text generated by various speech-to-text engines. At Capsule, precise transcriptions and timestamps are essential for video editing, making accuracy in these areas critical.

In just 4 minutes, I’ll demo a Next.js app that integrates with the media library and visualizes outputs from different versions of our transcription pipeline. This pipeline includes ML models such as VAD, language identification, speech-to-text engines (e.g., Whisper, AssemblyAI, Reverb), alignment model, and LLMs.

You’ll get a glimpse into how I address complex challenges using techniques like chunking, LLM, Diff, Merge algorithms, and logging. I’ll cover solutions for issues such as missed repetitions, punctuation improvements, timestamp alignment losses, hallucinations, Japanese tokenization errors, and how merging outputs from two transcription models enhances quality of text and timestamps.