Verbalyx

01Overview

Verbalyx takes a YouTube link or an uploaded audio or video file, transcribes the speech, generates a title, a summary, action items, key decisions and open questions, and then answers questions about what was said using retrieval-augmented generation. Every answer is displayed with the transcript passages it was built from.

It analyses speech only — not the images in a video — and keeps no timestamps or speaker labels.

The pipeline began from a public tutorial. It was extended with a web studio and job API, display of the retrieved evidence, a fix for answers drawing on previously analysed videos, a configurable LLM, and this write-up.

02Problem statement

Long recordings are slow to search. Given a recording, produce a usable written record and answer specific questions about it — while letting the reader verify each answer against the words that were actually spoken.

03Motivation

Sending a whole transcript to an LLM is expensive and can exceed its context window, and the answer cannot easily be traced back to a passage. Retrieving a few relevant chunks first keeps prompts short and makes the grounding visible, so wrong answers can be told apart from wrong retrieval.

04System architecture

1. Audio — YouTube via yt-dlp, or an uploaded file; converted to WAV with ffmpeg/pydub and split into 10-minute chunks.
2. Transcribe — Whisper “small” on the local CPU for English; for Hinglish, the Sarvam speech-to-text-translate API (25-second pieces, translated to English).
3. Analyse — Mistral generates a title from the first 2,000 characters, a map-reduce summary over 3,000-character chunks, and action items, decisions and open questions from the full transcript.
4. Index — the transcript is split into 500-character chunks (50 overlap), embedded with all-MiniLM-L6-v2 and stored in Chroma; the collection is cleared before each new recording.
5. Ask — the four most similar chunks are passed to Mistral with an instruction to answer only from that context, or to say “I could not find this information in the meeting transcript.”

05Technical implementation

• A FastAPI server runs one job at a time in a background thread and reports each stage’s status and duration to the studio as it progresses.
• Only YouTube URLs are accepted (no arbitrary hosts or local paths); uploads are limited by file type and size and deleted, with their derived audio, when the job finishes. API keys stay on the server and are redacted from logs.
• For each answer, the server repeats the retriever’s similarity search and returns the passages with their chunk index and distance; the interface flags any passage that does not occur in the current transcript.
• Bug fixed: the original code added each new transcript to the same persisted collection, so a question about one video could retrieve passages from another (4 of 4 in a local test). The collection is now cleared before indexing; the same test then retrieved none from the previous video.

06Model design

Three models with separate jobs: Whisper for speech recognition, a small sentence-transformer (MiniLM) for retrieval, and a Mistral chat model for generation. The Mistral model is configurable; the recorded run used ministral-8b-latest. Keeping retrieval separate from generation means the evidence for each answer can be inspected independently of the answer itself.

07Evaluation methodology

There is no systematic evaluation yet. The recorded run was checked in two ways:

• Transcription: word error rate against the known script of a synthetic recording made with macOS text-to-speech (three voices, 88 seconds).
• Answers: three questions with known answers, including one about something never discussed to test the “not found” behaviour.

Synthetic speech is clearer than real meetings, so these figures are a best case, not an estimate of real-world accuracy.

08Verified results

Nearly all transcription differences are number formatting (“seventeenth” → “17th”, “two hundred and forty” → “240”); the one real word error was “will” → “we’ll”.

The attribution error has a clear cause: the transcript has no speaker labels, so the model cannot know who said a sentence and guessed.

09Demonstration video

2:28Play Verbalyx demo, 2 minutes 28 seconds

Recorded locally on a Mac, 11 October 2026; no audio track. The input is a synthetic meeting generated with text-to-speech, labelled as such in the video. Shown in real time: upload, the six processing stages, the generated analysis, the transcript, and three questions with their retrieved evidence. The end card states the run’s measured results and its error.

10Deployment and source code

• Live demo: anshaiml369.github.io/verbalyx — Hugging Face Space (Docker, 2 vCPU, 16 GB), Whisper “small”. Locally the process peaked at about 1.45 GB of memory, too much for 512 MB free hosting. On the Space, the 88-second synthetic meeting was processed in 55 s and the launch-date question answered correctly (14 October 2026). Public limits: 10 analyses a day, 2 per visitor per hour, 10 questions per analysis, uploads up to 10 minutes and 25 MB; YouTube links are disabled.
• External services: Mistral (and Sarvam for Hinglish) via server-side API keys.
• Source code: available on request.

11Limitations and known failure modes

• No speaker labels or timestamps, so answers cannot say who said something or point to a moment in the video.
• Speech only: anything shown but not said is invisible to the system.
• Long recordings: the extraction prompts send the full transcript in one request and may exceed the model’s context.
• One index: analysing a new recording replaces the previous one.
• Retrieval can miss: if the relevant passage is not among the top four chunks, the model answers from the wrong context or says it cannot find the answer.
• YouTube downloads may be blocked from cloud servers; only one synthetic recording has been evaluated.

12Future improvements

• Keep Whisper’s segment timestamps and add speaker diarisation, so answers can cite who spoke and when.
• An evaluation set of real recordings with reference questions, measuring retrieval hit rate and answer accuracy separately.
• A hosted version on a larger instance, with the same usage limits as Scholaryx.