module pixeltable.functions.twelvelabs
Pixeltable UDFs that wrap various endpoints from the TwelveLabs API. In order to use them, you must firstpip install twelvelabs and configure your TwelveLabs credentials, as described in
the Working with TwelveLabs tutorial.
udf async_embed()
Signatures
embed.v_2.tasks.create and retrieve endpoints:
https://docs.twelvelabs.io/sdk-reference/python/create-embeddings-v-2/create-async-embeddings
Embeds audio and video of any duration supported by the model, and returns one EmbeddingResult per provider
embedding record. By default, the result contains embeddings for individual segments and for the whole input,
with separate visual and audio embeddings for Marengo 3.5 video. Use media_kwargs to choose modalities, scopes,
and segmentation. To index the embeddings, expand the result into a view with list_iterator(); use embed()
to create query embeddings.
Request throttling:
Applies the rate limit set in the config (section twelvelabs, key rate_limit). If no rate
limit is configured, uses a default of 600 RPM.
The UDF waits for each input up to the timeout set in the twelvelabs config section (default: 600 seconds),
including upload processing; TWELVELABS_TIMEOUT overrides the config file.
Requirements:
pip install 'twelvelabs>=1.3.6'
video(Video): The video to embed. An overload acceptsaudioinstead.model_name(String): The embedding model (marengo3.0ormarengo3.5).media_kwargs(Json | None, default:Literal(None)): Additional fields for the provider’sAsyncAudioInputRequestorAsyncVideoInputRequest, such asstart_sec,end_sec,embedding_option,embedding_scope,embedding_type,segmentation, andtime_based_metadata.media_sourceis managed by Pixeltable and cannot be set here.embedding_dimension(Int | None, default:Literal(None)): Output dimension. Omit to use the model’s default; specify it as a constant when indexing embeddings from an unrecognized model.embedding_uncertainty(Bool | None, default:Literal(None)): Whether to request per-dimension uncertainty. For audio and video, setmedia_kwargs['embedding_scope']withoutasset; the default scope includes it and the request fails.
pxt.Json[(Json[{'embedding': Array[(None,), float32], 'embedding_option': String | None, 'embedding_scope': String | None, 'start_sec': Float | None, 'end_sec': Float | None, 'embedding_uncertainty': Array[(None,), float32] | None, 'metadata': Json | None}], ...)]: A list of embedding records, one per provider embedding record.
udf async_embed_asset()
Signatures
embed.v_2.tasks.create and retrieve endpoints:
https://docs.twelvelabs.io/sdk-reference/python/create-embeddings-v-2/create-async-embeddings
Embeds audio and video of any duration supported by the model, and returns the provider’s asset-scope
embedding: for video, one fused embedding that combines the visual and audio content; for audio, one audio
embedding. Unlike embed(), it accepts Marengo 3.5 media longer than 30 seconds or larger than 32 MB. Use it
to index media, together with embed() to create query embeddings. Use async_embed() to get embeddings for
individual segments.
Request throttling:
Applies the rate limit set in the config (section twelvelabs, key rate_limit). If no rate
limit is configured, uses a default of 600 RPM.
The UDF waits for each input up to the timeout set in the twelvelabs config section (default: 600 seconds),
including upload processing; TWELVELABS_TIMEOUT overrides the config file.
Requirements:
pip install 'twelvelabs>=1.3.6'
video(Video): The video to embed. An overload acceptsaudioinstead.model_name(String): The embedding model (marengo3.0ormarengo3.5).
pxt.Array[float32]: The embedding vector for the whole input.
embed() for text queries:
udf embed()
Signatures
marengo3.0 and marengo3.5 return 512-dimensional vectors that can be used
with an embedding index. Embeddings from different model versions are incompatible; re-embed existing
content when changing models.
Marengo 3.5 uses the synchronous multi_input API. Images, audio, and video must be at most 32 MB.
Audio and video must also be at most 30 seconds. Split longer files with audio_splitter or
video_splitter before embedding them, or use async_embed_asset() to embed longer media into a single vector.
The start_sec, end_sec, and embedding_option parameters are supported only with Marengo 3.0.
Equivalent to the TwelveLabs Embed API:
https://docs.twelvelabs.io/sdk-reference/python/create-embeddings-v-2/create-sync-embeddings
Request throttling:
Applies the rate limit set in the config (section twelvelabs, key rate_limit). If no rate
limit is configured, uses a default of 600 RPM.
With Marengo 3.0, audio and video files over 2 MiB are uploaded first. The UDF waits for each uploaded
asset to become ready, subject to timeout in the twelvelabs config section.
TWELVELABS_TIMEOUT overrides the config file.
Requirements:
pip install twelvelabs- Marengo 3.5 requires
twelvelabs>=1.3.6.
model_name(String): The name of the model to use (marengo3.0ormarengo3.5).text(String): The text to embed.image(Image | None, default:Literal(None)): If specified, the embedding will be created from both the text and the image.
pxt.Array[float32] | None: The embedding.
embed for an embedding of a string column input: