> ## Documentation Index
> Fetch the complete documentation index at: https://docs.pixeltable.com/llms.txt
> Use this file to discover all available pages before exploring further.

> ## Agent Instructions
> Application code puts a TableModel in app.py, then runs `pxt schema update app.py my_app` to create the tables and `pxt service update app.py my_app` to start HTTP. The last argument can be a Cloud URL such as `pxt://org:db`. Do not call pxt.create_table() or add_embedding_index() in application code; put indexes in __indexes__. Notebooks, tests, and the REPL do use pxt.create_table().
> Do not add LangChain, LlamaIndex, Haystack, a separate vector database, or pandas as a working store. Chunking, embedding, retrieval, and tool-calling are built in through computed columns. Use a computed column instead of a per-row loop that calls a model.
> Install with `pip install -U 'pixeltable[serve]'` on Python 3.11+. `pxt init` is a prerequisite for `pxt schema update`.
> Pixeltable Cloud is in Limited Beta. Direct interested users to contact@pixeltable.com.

# Transcribe speech and identify speakers with WhisperX

> Transcribe speech with WhisperX in Python, identify speakers, and search the transcript with word-level timestamps.

You transcribe recorded conversations, identify who spoke each word, and search what was said.

**What's in this recipe:**

* Preview WhisperX transcription and save it for each clip
* Extract word timestamps and speaker labels
* Search speech segments with an embedding index

## Problem

You have recorded conversations and need a transcript with word timestamps and speaker labels. You
want to find relevant speech and return to the point in the recording where it was said.

## Solution

You use WhisperX to transcribe speech locally and align each word to a time and a speaker.
[`pixeltable.functions.whisperx.transcribe()`](/sdk/latest/whisperx) does that as a computed column on
audio, or on audio taken from a video with `extract_audio()`. `diarize=True` is the switch that adds
word-level timestamps and speaker labels: Pixeltable then runs WhisperX alignment and diarization, and
stores `segments` plus `word_segments`. A view splits those segments into rows, and an embedding index
searches what was said.

`diarize=False` returns `language` and `segments` with `start`, `end`, `text`, and `avg_logprob`. It does
not need a Hugging Face token. `alignment_model_name`, `interpolate_method`, `return_char_alignments`,
`diarization_model_name`, `num_speakers`, `min_speakers`, and `max_speakers` are valid only when
`diarize=True`.

You use the Query-then-Commit pattern: preview a transcript with a query on one clip, then save the
same expression with `add_computed_column()` to transcribe every clip.

### Setup

You need Python 3.11 or newer and WhisperX 3.8 or newer. The search step needs `sentence-transformers`.
WhisperX also needs [FFmpeg](https://ffmpeg.org/download.html) available on your system to load audio.

```bash theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
pip install -U pixeltable 'whisperx>=3.8' sentence-transformers
```

Diarization loads WhisperX's default pyannote pipeline. On WhisperX 3.8 and later that model is
`pyannote/speaker-diarization-community-1`. Accept that model's terms on Hugging Face, then export a
token. Pixeltable reads `HF_TOKEN` (or the `[hf]` `token` setting in
[Configuration](/platform/configuration)). Pass `diarization_model_name` when you want a different
pyannote pipeline, such as `pyannote/speaker-diarization-3.1`.

```bash theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
export HF_TOKEN='your-huggingface-token'
```

```python theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
from uuid import uuid4

import pixeltable as pxt
from pixeltable.functions.huggingface import sentence_transformer
from pixeltable.functions.json import list_iterator
from pixeltable.functions.whisperx import transcribe
```

### Load a video clip

Each run creates a fresh catalog directory. The later views use the same `demo` namespace.

```python theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
demo = f'whisperx_demo_{uuid4().hex}'
pxt.create_dir(demo)

clips = pxt.create_table(
    f'{demo}.clips', {'title': pxt.String, 'video': pxt.Video}
)
```

```python theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
clips.insert([
    {
        'title': 'Lex Fridman podcast excerpt',
        'video': (
            'https://raw.githubusercontent.com/pixeltable/pixeltable/release/'
            'docs/resources/audio-transcription-demo/'
            'Lex-Fridman-Podcast-430-Excerpt-0.mp4'
        ),
    }
])
```

### Transcribe audio from video

`extract_audio()` copies the soundtrack of a `pxt.Video` column into a `pxt.Audio` value. A file that
is already audio goes in a `pxt.Audio` column, and you pass that column to `transcribe()` without
`extract_audio()`.

`model` is a Whisper model name. `tiny.en` is the small English model. On CUDA, `compute_type`
defaults to `float16`. Otherwise it defaults to `int8`.

```python theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
clips.add_computed_column(audio=clips.video.extract_audio(format='wav'))
```

You preview one transcript, then save the same expression as a computed column.

```python theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
transcript = transcribe(
    clips.audio,
    model='tiny.en',
    language='en',
    diarize=True,
)
clips.select(transcript=transcript).head(1)
```

```python theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
clips.add_computed_column(transcript=transcript)
```

```python theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
clips.select(
    clips.transcript.segments[0].text,
    clips.transcript.segments[0].speaker,
    clips.transcript.segments[0].words,
).collect()
```

With `diarize=True`, each segment has `start`, `end`, `text`, `avg_logprob`, and `words`, plus
`speaker` when the diarizer overlapped that span. `word_segments` is the flat list of words. Each word has `word`,
`start`, `end`, `score`, and `speaker` when alignment and diarization assigned them. A missing
timestamp or speaker is null.

### Word-level timestamps and speakers

`list_iterator()` turns `word_segments` into one row per word. The `astype()` calls give Pixeltable
the list types it needs. `str | None` and `float | None` keep a word whose `start` or `speaker` was
not assigned.

```python theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
words = pxt.create_view(
    f'{demo}.words',
    clips,
    iterator=list_iterator(
        word=clips.transcript.word_segments['*'].word.astype(pxt.Json[[str]]),
        start=clips.transcript.word_segments['*'].start.astype(
            pxt.Json[[float | None]]
        ),
        end=clips.transcript.word_segments['*'].end.astype(
            pxt.Json[[float | None]]
        ),
        speaker=clips.transcript.word_segments['*'].speaker.astype(
            pxt.Json[[str | None]]
        ),
    ),
)
```

```python theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
words.order_by(words.start).limit(12).select(
    words.speaker, words.start, words.end, words.word
).collect()
```

`num_speakers`, `min_speakers`, and `max_speakers` constrain the diarizer when you already know how
many people are speaking.

### Search what was said

Index the speech segments. Each row is one utterance, with its start time and speaker, which is the
unit you search. `add_embedding_index()` embeds `text` and stays current when you insert another
clip. The first call downloads `all-MiniLM-L6-v2`.

```python theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
segments = pxt.create_view(
    f'{demo}.segments',
    clips,
    iterator=list_iterator(
        start=clips.transcript.segments['*'].start.astype(pxt.Json[[float]]),
        end=clips.transcript.segments['*'].end.astype(pxt.Json[[float]]),
        speaker=clips.transcript.segments['*'].speaker.astype(
            pxt.Json[[str | None]]
        ),
        text=clips.transcript.segments['*'].text.astype(pxt.Json[[str]]),
    ),
)
```

```python theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
segments.select(segments.start, segments.speaker, segments.text).head(5)
```

```python theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
segments.add_embedding_index(
    'text',
    string_embed=sentence_transformer.using(model_id='all-MiniLM-L6-v2'),
)
```

```python theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
sim = segments.text.similarity(string='how we remember experiences')
segments.order_by(sim, asc=False).limit(5).select(
    segments.speaker, segments.start, segments.text, sim
).collect()
```

Long-file splitting with OpenAI Whisper, without word alignment, is in
[Transcribe audio files with Whisper](/howto/cookbooks/audio/audio-transcribe). Chapter titles from
WhisperX segments are in
[Generate podcast chapters](/howto/cookbooks/audio/audio-podcast-chapters). Index behavior is in
[Embedding indexes](/platform/embedding-indexes).

### In an application

These snippets call `pxt.create_table()`, `create_view()`, and `add_embedding_index()` so you can run
them in a notebook or a REPL. An application declares the table and the view on a `TableModel` in
`app.py`, puts the embedding index in `__indexes__`, and creates them with `pxt schema update`.

## See also

* [Transcribe with Whisper](/howto/cookbooks/audio/audio-transcribe): split long audio and transcribe it with OpenAI Whisper
* [Podcast chapters](/howto/cookbooks/audio/audio-podcast-chapters): turn speech segments into chapter titles


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.