Transcribing and Indexing Audio and Video in Pixeltable

This documentation page is also available as an interactive notebook. You can launch the notebook in Kaggle or Colab, or download it for use with an IDE or local Jupyter installation, by clicking one of the above links.

In this tutorial, we’ll build an end-to-end workflow for creating and indexing audio transcriptions of video data. We’ll demonstrate how Pixeltable can be used to:

Extract audio data from video files;
Transcribe the audio using OpenAI Whisper;
Build a semantic index of the transcriptions, using the Huggingface sentence_transformers models;
Search this index.

The tutorial assumes you’re already somewhat familiar with Pixeltable. If this is your first time using Pixeltable, the 10-Minute Tour tutorial is a great place to start.

Create a Table for Video Data

Let’s first install the Python packages we’ll need for the demo. We’re going to use the popular Whisper library, running locally. Later in the demo, we’ll see how to use the OpenAI API endpoints as an alternative.

%pip install -q pixeltable openai openai-whisper sentence-transformers spacy
!python -m spacy download en_core_web_sm -q

Now we create a Pixeltable table to hold our videos.

import pixeltable as pxt

pxt.drop_dir(
    'transcription_demo', force=True
)  # Ensure a clean slate for the demo
pxt.create_dir('transcription_demo')

# Create a table to store our videos and workflow
video_table = pxt.create_table(
    'transcription_demo/video_table', {'video': pxt.Video}
)

video_table

Connected to Pixeltable database at: postgresql+psycopg://postgres:@/pixeltable?host=/Users/asiegel/.pixeltable/pgdata
Created directory ‘transcription_demo’.
Created table ‘video_table’.

Next let’s insert some video files into the table. In this demo, we’ll be using one-minute excerpts from a Lex Fridman podcast. We’ll begin by inserting two of them into our new table. In this demo, our videos are given as https links, but Pixeltable also accepts local files and S3 URLs as input.

videos = [
    'https://github.com/pixeltable/pixeltable/raw/release/docs/resources/audio-transcription-demo/'
    f'Lex-Fridman-Podcast-430-Excerpt-{n}.mp4'
    for n in range(3)
]

video_table.insert({'video': video} for video in videos[:2])
video_table.show()

Inserted 2 rows with 0 errors in 2.04 s (0.98 rows/s)

Now we’ll add another column to hold extracted audio from our videos. The new column is an example of a computed column: it’s updated automatically based on the contents of another column (or columns). In this case, the value of the audio column is defined to be the audio track extracted from whatever’s in the video column.

from pixeltable.functions.video import extract_audio

video_table.add_computed_column(
    audio=extract_audio(video_table.video, format='mp3')
)
video_table.show()

Added 2 column values with 0 errors in 0.91 s (2.19 rows/s)

If we look at the structure of the video table, we see that the new column is a computed column.

video_table

We can also add another computed column to extract metadata from the audio streams.

from pixeltable.functions.audio import get_metadata

video_table.add_computed_column(metadata=get_metadata(video_table.audio))
video_table.show()

Added 2 column values with 0 errors in 0.02 s (95.47 rows/s)

Create Transcriptions

Now we’ll add a step to create transcriptions of our videos. As mentioned above, we’re going to use the Whisper library for this, running locally. Pixeltable has a built-in function, whisper.transcribe, that serves as an adapter for the Whisper library’s transcription capability. All we have to do is add a computed column that calls this function:

from pixeltable.functions import whisper

video_table.add_computed_column(
    transcription=whisper.transcribe(
        audio=video_table.audio, model='base.en'
    )
)

video_table.select(
    video_table.video, video_table.transcription.text
).show()

Added 2 column values with 0 errors in 4.63 s (0.43 rows/s)

In order to index the transcriptions, we’ll first need to split them into sentences. We can do this using Pixeltable’s built-in string_splitter iterator.

from pixeltable.functions.string import string_splitter

sentences_view = pxt.create_view(
    'transcription_demo/sentences_view',
    video_table,
    iterator=string_splitter(
        video_table.transcription.text, separators='sentence'
    ),
)

The string_splitter creates a new view, with the audio transcriptions broken into individual, one-sentence chunks.

sentences_view.select(sentences_view.pos, sentences_view.text).show(8)

Add an Embedding Index

Next, let’s use the Huggingface sentence_transformers library to create an embedding index of our sentences, attaching it to the text column of our sentences_view.

from pixeltable.functions.huggingface import sentence_transformer

sentences_view.add_embedding_index(
    'text',
    embedding=sentence_transformer.using(model_id='intfloat/e5-large-v2'),
)

modules.json:   0%|          | 0.00/387 [00:00<?, ?B/s]
README.md: 0.00B [00:00, ?B/s]
sentence_bert_config.json:   0%|          | 0.00/57.0 [00:00<?, ?B/s]
config.json:   0%|          | 0.00/616 [00:00<?, ?B/s]
model.safetensors:   0%|          | 0.00/1.34G [00:00<?, ?B/s]
tokenizer_config.json:   0%|          | 0.00/314 [00:00<?, ?B/s]
vocab.txt: 0.00B [00:00, ?B/s]
tokenizer.json: 0.00B [00:00, ?B/s]
special_tokens_map.json:   0%|          | 0.00/125 [00:00<?, ?B/s]
config.json:   0%|          | 0.00/201 [00:00<?, ?B/s]

We can do a simple lookup to test our new index. The following snippet returns the results of a nearest-neighbor search on the input “What is happiness?”

sim = sentences_view.text.similarity(string='What is happiness?')

(
    sentences_view.order_by(sim, asc=False)
    .limit(10)
    .select(sentences_view.text, similarity=sim)
    .collect()
)

Incremental Updates

Incremental updates are a key feature of Pixeltable. Whenever a new video is added to the original table, all of its downstream computed columns are updated automatically. Let’s demonstrate this by adding a third video to the table and seeing how the updates propagate through to the index.

video_table.insert([{'video': videos[2]}])

Inserted 10 rows with 0 errors in 4.20 s (2.38 rows/s)
10 rows inserted.

video_table.select(
    video_table.video,
    video_table.metadata,
    video_table.transcription.text,
).show()

sim = sentences_view.text.similarity(string='What is happiness?')

(
    sentences_view.order_by(sim, asc=False)
    .limit(20)
    .select(sentences_view.text, similarity=sim)
    .collect()
)

We can see the new results showing up in sentences_view.

Using the OpenAI API

This concludes our tutorial using the locally installed Whisper library. Sometimes, it may be preferable to use the OpenAI API rather than a locally installed library. In this section we’ll show how this can be done in Pixeltable, simply by using a different function to construct our computed columns. Since this section relies on calling out to the OpenAI API, you’ll need to have an API key, which you can enter below.

import getpass
import os

if 'OPENAI_API_KEY' not in os.environ:
    os.environ['OPENAI_API_KEY'] = getpass.getpass('OpenAI API Key:')

from pixeltable.functions import openai

video_table.add_computed_column(
    transcription_from_api=openai.transcriptions(
        video_table.audio, model='whisper-1'
    )
)

Added 3 column values with 0 errors in 6.49 s (0.46 rows/s)
3 rows updated.

Now let’s compare the results from the local model and the API side-by-side.

video_table.select(
    video_table.video,
    video_table.transcription.text,
    video_table.transcription_from_api.text,
).show()

They look pretty similar, which isn’t surprising, since the OpenAI transcriptions endpoint runs on Whisper. One difference is that the local library spits out a lot more information about the internal behavior of the model. Note that we’ve been selecting video_table.transcription.text in the preceding queries, which pulls out just the text field of the transcription results. The actual results are a sizable JSON structure that includes a lot of metadata. To see the full output, we can select video_table.transcription instead, to get the full JSON struct. Here’s what it looks like (we’ll select just one row, since it’s a lot of output):

video_table.select(
    video_table.transcription, video_table.transcription_from_api
).show(1)

Welcome to Pixeltable

Core Concepts

How-To

Transcribing and Indexing Audio and Video in Pixeltable

Create a Table for Video Data

Create Transcriptions

Add an Embedding Index

Incremental Updates

Using the OpenAI API

Welcome to Pixeltable

Core Concepts

How-To

Documentation Index

​Create a Table for Video Data

​Create Transcriptions

​Add an Embedding Index

​Incremental Updates

​Using the OpenAI API

Create a Table for Video Data

Create Transcriptions

Add an Embedding Index

Incremental Updates

Using the OpenAI API