module pixeltable.functions.twelvelabs

Pixeltable UDFs that wrap various endpoints from the TwelveLabs API. In order to use them, you must first pip install twelvelabs and configure your TwelveLabs credentials, as described in the Working with TwelveLabs tutorial.

udf embed()

Signatures

# Signature 1:
@pxt.udf
embed(
    text: pxt.String,
    image: pxt.Image | None,
    *,
    model_name: pxt.String
) -> pxt.Array[float32] | None

# Signature 2:
@pxt.udf
embed(
    image: pxt.Image,
    *,
    model_name: pxt.String
) -> pxt.Array[float32] | None

# Signature 3:
@pxt.udf
embed(
    audio: pxt.Audio,
    *,
    model_name: pxt.String,
    start_sec: pxt.Float | None,
    end_sec: pxt.Float | None,
    embedding_option: pxt.Json[(String, ...)] | None
) -> pxt.Array[float32] | None

# Signature 4:
@pxt.udf
embed(
    video: pxt.Video,
    *,
    model_name: pxt.String,
    start_sec: pxt.Float | None,
    end_sec: pxt.Float | None,
    embedding_option: pxt.Json[(String, ...)] | None
) -> pxt.Array[float32] | None

Creates an embedding vector for the given text, audio, image, or video input. Each UDF signature corresponds to one of the four supported input types. If text is specified, it is possible to specify an image as well. Both marengo3.0 and marengo3.5 return 512-dimensional vectors that can be used with an embedding index. Embeddings from different model versions are incompatible; re-embed existing content when changing models. Marengo 3.5 uses the synchronous multi_input API. Images, audio, and video must be at most 32 MB. Audio and video must also be at most 30 seconds. Split longer files with audio_splitter or video_splitter before embedding them. The start_sec, end_sec, and embedding_option parameters are supported only with Marengo 3.0. Equivalent to the TwelveLabs Embed API: https://docs.twelvelabs.io/sdk-reference/python/create-embeddings-v-2/create-sync-embeddings Request throttling: Applies the rate limit set in the config (section twelvelabs, key rate_limit). If no rate limit is configured, uses a default of 600 RPM. Requirements:

pip install twelvelabs
Marengo 3.5 requires twelvelabs>=1.3.6.

Parameters:

model_name (String): The name of the model to use (marengo3.0 or marengo3.5).
text (String): The text to embed.
image (Image | None, default: Literal(None)): If specified, the embedding will be created from both the text and the image.

Returns:

pxt.Array[float32] | None: The embedding.

Examples: Add a computed column embed for an embedding of a string column input:

tbl.add_computed_column(
    embed=embed(model_name='marengo3.5', text=tbl.input)
)

Add an embedding index for cross-modal search:

tbl.add_embedding_index(
    tbl.input, embedding=embed.using(model_name='marengo3.5')
)

​module pixeltable.functions.twelvelabs

​udf embed()

module pixeltable.functions.twelvelabs

udf embed()