module pixeltable.functions.whisperx

WhisperX audio transcription and diarization functions.

udf transcribe()

transcribe(
    audio: Audio,
    *,
    model: String,
    diarize: Bool = False,
    compute_type: String | None = None,
    language: String | None = None,
    task: String | None = None,
    chunk_size: Int | None = None,
    alignment_model_name: String | None = None,
    interpolate_method: String | None = None,
    return_char_alignments: Bool | None = None,
    diarization_model_name: String | None = None,
    num_speakers: Int | None = None,
    min_speakers: Int | None = None,
    max_speakers: Int | None = None
) -> Json

Transcribe an audio file using WhisperX. This UDF runs a transcription model locally using the WhisperX library, equivalent to the WhisperX transcribe function, as described in the WhisperX library documentation. If diarize=True, then speaker diarization will also be performed. Several of the UDF parameters are only valid if diarize=True, as documented in the parameters list below. Requirements:

pip install whisperx

Parameters:

audio (Audio): The audio file to transcribe.
model (String): The name of the model to use for transcription.
diarize (Bool): Whether to perform speaker diarization.
compute_type (String | None): The compute type to use for the model (e.g., 'int8', 'float16'). If None, defaults to 'float16' on CUDA devices and 'int8' otherwise.
language (String | None): The language code for the transcription (e.g., 'en' for English).
task (String | None): The task to perform (e.g., 'transcribe' or 'translate'). Defaults to 'transcribe'.
chunk_size (Int | None): The size of the audio chunks to process, in seconds. Defaults to 30.
alignment_model_name (String | None): The name of the alignment model to use. If None, uses the default model for the given language. Only valid if diarize=True.
interpolate_method (String | None): The method to use for interpolation of the alignment results. If not specified, uses the WhisperX default ('nearest'). Only valid if diarize=True.
return_char_alignments (Bool | None): Whether to return character-level alignments. Defaults to False. Only valid if diarize=True.
diarization_model_name (String | None): The name of the diarization model to use. Defaults to pyannote/speaker-diarization-3.1. Only valid if diarize=True.
num_speakers (Int | None): The number of speakers to expect in the audio. By default, the model with try to detect the number of speakers. Only valid if diarize=True.
min_speakers (Int | None): If specified, the minimum number of speakers to expect in the audio. Only valid if diarize=True.
max_speakers (Int | None): If specified, the maximum number of speakers to expect in the audio. Only valid if diarize=True.

Returns:

Json: A dictionary containing the audio transcription, diarization (if enabled), and various other metadata.

Examples: Add a computed column that applies the model tiny.en to an existing Pixeltable column tbl.audio of the table tbl:

tbl.add_computed_column(result=transcribe(tbl.audio, model='tiny.en'))

Add a computed column that applies the model tiny.en to an existing Pixeltable column tbl.audio of the table tbl, with speaker diarization enabled, expecting at least 2 speakers:

tbl.add_computed_column(
    result=transcribe(
        tbl.audio, model='tiny.en', diarize=True, min_speakers=2
    )
)

​module pixeltable.functions.whisperx

​udf transcribe()

module pixeltable.functions.whisperx

udf transcribe()