Skip to main content

module  pixeltable.functions.twelvelabs

Pixeltable UDFs that wrap various endpoints from the TwelveLabs API. In order to use them, you must first pip install twelvelabs and configure your TwelveLabs credentials, as described in the Working with TwelveLabs tutorial.

udf  async_embed()

Signatures
Creates embeddings for segments of a video or audio input using the TwelveLabs async embeddings API. Wraps the TwelveLabs embed.v_2.tasks.create and retrieve endpoints: https://docs.twelvelabs.io/sdk-reference/python/create-embeddings-v-2/create-async-embeddings Embeds audio and video of any duration supported by the model, and returns one EmbeddingResult per provider embedding record. By default, the result contains embeddings for individual segments and for the whole input, with separate visual and audio embeddings for Marengo 3.5 video. Use media_kwargs to choose modalities, scopes, and segmentation. To index the embeddings, expand the result into a view with list_iterator(); use embed() to create query embeddings. Request throttling: Applies the rate limit set in the config (section twelvelabs, key rate_limit). If no rate limit is configured, uses a default of 600 RPM. The UDF waits for each input up to the timeout set in the twelvelabs config section (default: 600 seconds), including upload processing; TWELVELABS_TIMEOUT overrides the config file. Requirements:
  • pip install 'twelvelabs>=1.3.6'
Parameters:
  • video (Video): The video to embed. An overload accepts audio instead.
  • model_name (String): The embedding model (marengo3.0 or marengo3.5).
  • media_kwargs (Json | None, default: Literal(None)): Additional fields for the provider’s AsyncAudioInputRequest or AsyncVideoInputRequest, such as start_sec, end_sec, embedding_option, embedding_scope, embedding_type, segmentation, and time_based_metadata. media_source is managed by Pixeltable and cannot be set here.
  • embedding_dimension (Int | None, default: Literal(None)): Output dimension. Omit to use the model’s default; specify it as a constant when indexing embeddings from an unrecognized model.
  • embedding_uncertainty (Bool | None, default: Literal(None)): Whether to request per-dimension uncertainty. For audio and video, set media_kwargs['embedding_scope'] without asset; the default scope includes it and the request fails.
Returns:
  • pxt.Json[(Json[{'embedding': Array[(None,), float32], 'embedding_option': String | None, 'embedding_scope': String | None, 'start_sec': Float | None, 'end_sec': Float | None, 'embedding_uncertainty': Array[(None,), float32] | None, 'metadata': Json | None}], ...)]: A list of embedding records, one per provider embedding record.
Examples: Embed each video, expand the embeddings into a view, and index them:
Limit processing to the first minute and to segment embeddings by passing provider options:
Search the indexed embeddings:

udf  async_embed_asset()

Signatures
Creates a single embedding vector for a whole video or audio input using the TwelveLabs async embeddings API. Wraps the TwelveLabs embed.v_2.tasks.create and retrieve endpoints: https://docs.twelvelabs.io/sdk-reference/python/create-embeddings-v-2/create-async-embeddings Embeds audio and video of any duration supported by the model, and returns the provider’s asset-scope embedding: for video, one fused embedding that combines the visual and audio content; for audio, one audio embedding. Unlike embed(), it accepts Marengo 3.5 media longer than 30 seconds or larger than 32 MB. Use it to index media, together with embed() to create query embeddings. Use async_embed() to get embeddings for individual segments. Request throttling: Applies the rate limit set in the config (section twelvelabs, key rate_limit). If no rate limit is configured, uses a default of 600 RPM. The UDF waits for each input up to the timeout set in the twelvelabs config section (default: 600 seconds), including upload processing; TWELVELABS_TIMEOUT overrides the config file. Requirements:
  • pip install 'twelvelabs>=1.3.6'
Parameters:
  • video (Video): The video to embed. An overload accepts audio instead.
  • model_name (String): The embedding model (marengo3.0 or marengo3.5).
Returns:
  • pxt.Array[float32]: The embedding vector for the whole input.
Examples: Index a video column, using embed() for text queries:
Search the index:

udf  embed()

Signatures
Creates an embedding vector for the given text, audio, image, or video input. Each UDF signature corresponds to one of the four supported input types. If text is specified, it is possible to specify an image as well. Both marengo3.0 and marengo3.5 return 512-dimensional vectors that can be used with an embedding index. Embeddings from different model versions are incompatible; re-embed existing content when changing models. Marengo 3.5 uses the synchronous multi_input API. Images, audio, and video must be at most 32 MB. Audio and video must also be at most 30 seconds. Split longer files with audio_splitter or video_splitter before embedding them, or use async_embed_asset() to embed longer media into a single vector. The start_sec, end_sec, and embedding_option parameters are supported only with Marengo 3.0. Equivalent to the TwelveLabs Embed API: https://docs.twelvelabs.io/sdk-reference/python/create-embeddings-v-2/create-sync-embeddings Request throttling: Applies the rate limit set in the config (section twelvelabs, key rate_limit). If no rate limit is configured, uses a default of 600 RPM. With Marengo 3.0, audio and video files over 2 MiB are uploaded first. The UDF waits for each uploaded asset to become ready, subject to timeout in the twelvelabs config section. TWELVELABS_TIMEOUT overrides the config file. Requirements:
  • pip install twelvelabs
  • Marengo 3.5 requires twelvelabs>=1.3.6.
Parameters:
  • model_name (String): The name of the model to use (marengo3.0 or marengo3.5).
  • text (String): The text to embed.
  • image (Image | None, default: Literal(None)): If specified, the embedding will be created from both the text and the image.
Returns:
  • pxt.Array[float32] | None: The embedding.
Examples: Add a computed column embed for an embedding of a string column input:
Add an embedding index for cross-modal search:
Last modified on November 7, 2026