​

module  pixeltable.functions.vlmrun_gateway

​

udf  analyze()

# Signature 1:
@pxt.udf
analyze(
    image: pxt.Image,
    *,
    model: pxt.String,
    method: pxt.String | None,
    method_params: pxt.Json | None,
    model_kwargs: pxt.Json | None
) -> pxt.Json

# Signature 2:
@pxt.udf
analyze(
    image: pxt.Video,
    *,
    model: pxt.String,
    method: pxt.String | None,
    method_params: pxt.Json | None,
    model_kwargs: pxt.Json | None
) -> pxt.Json

• pip install openai

• image (Image): The image to analyze. A video (pxt.Video) is also accepted by video-capable models, such as 'facebook/sam3.1' with method='track' or 'usyd-community/vitpose-plus-large'.
• model (String): The model to use, such as 'facebook/sam3.1' (segmentation), 'microsoft/florence-2-base-ft' (detection and captions), 'usyd-community/vitpose-plus-large' (body pose) or 'geopavlakos/hamer' (hand pose). For the available models, see https://vlm.run/gateway/models.
• method (String | None, default: Literal(None)): The model entrypoint to run, such as 'segment' or 'od'. If omitted, the model’s default method is used. For each model’s methods, see https://docs.vlm.run/gateway/methods.
• method_params (Json | None, default: Literal(None)): Arguments for the method, such as {'prompt': 'car'} for 'facebook/sam3.1' segmentation.
• model_kwargs (Json | None, default: Literal(None)): Additional Gateway fields, such as precision, video_fps or video_max_frames. 'facebook/sam3.1' with method='track' reads video_fps and video_max_frames from method_params instead. For details, see https://docs.vlm.run/gateway/extra-kwargs.

• pxt.Json: The Gateway’s JSON result, unchanged: the input’s metadata (such as image_width and image_height) and content, which is {'object': ..., 'items': [...]} (for a video, also frames), or a string for caption methods. Image coordinates are fractions of the image width and height, and pose keypoints can fall outside 0 to 1; 'geopavlakos/hamer' also returns metric 3D joints in kpts_xyz. For each object type, see https://docs.vlm.run/gateway/response-types.

tbl.add_computed_column(
    cars=analyze(
        tbl.img,
        model='facebook/sam3.1',
        method='segment',
        method_params={'prompt': 'car'},
    )
)

​

udf  chat_completions()

@pxt.udf
chat_completions(
    messages: pxt.Json[(Json, ...)],
    *,
    model: pxt.String,
    model_kwargs: pxt.Json | None = None,
    tools: pxt.Json[(Json, ...)] | None = None,
    tool_choice: pxt.Json | None = None
) -> pxt.Json

• pip install openai

• messages (pxt.Json[(Json): A list of messages for the chat conversation. Media content parts may reference Pixeltable columns directly (see above).
• model (Any): The model to use, such as 'qwen/qwen3.5-0.8b'. For the available models, see https://vlm.run/gateway/models.
• model_kwargs (Any): Additional keyword args for the Gateway chat/completions API, such as response_format, max_tokens, or Gateway fields like method, method_params and document_dpi. For details, see https://docs.vlm.run/gateway/extra-kwargs.
• tools (Any): An optional list of Pixeltable tools to use for the request. Supported by some models only, such as 'qwen/qwen3.8-27b' and the partner Gemini, Kimi, MiniMax and Muse models.
• tool_choice (Any): An optional Pixeltable tool choice to use for the request.

• pxt.Json: A dictionary containing the API response, including the cost of the call in USD in usage.cost.

messages = [
    {
        'role': 'user',
        'content': [
            {'type': 'text', 'text': 'What is in this image?'},
            {'type': 'image_url', 'image_url': tbl.img},
        ],
    }
]
tbl.add_computed_column(
    response=chat_completions(messages, model='qwen/qwen3.5-0.8b')
)

receipt_messages = [
    {
        'role': 'user',
        'content': [
            {
                'type': 'text',
                'text': 'What is the total on this receipt?',
            },
            {'type': 'image_url', 'image_url': tbl.img},
        ],
    }
]
tbl.add_computed_column(
    receipt=chat_completions(
        receipt_messages,
        model='qwen/qwen3.8-27b',
        model_kwargs={
            'response_format': {
                'type': 'json_schema',
                'json_schema': {
                    'name': 'receipt',
                    'schema': {
                        'type': 'object',
                        'properties': {'total': {'type': 'number'}},
                    },
                },
            }
        },
    )
)

​

udf  embeddings()

# Signature 1:
@pxt.udf
embeddings(
    input: pxt.String,
    *,
    model: pxt.String,
    model_kwargs: pxt.Json | None
) -> pxt.Array[(None,), float32]

# Signature 2:
@pxt.udf
embeddings(
    input: pxt.Image,
    *,
    model: pxt.String,
    model_kwargs: pxt.Json | None
) -> pxt.Array[(None,), float32]

# Signature 3:
@pxt.udf
embeddings(
    input: pxt.Video,
    *,
    model: pxt.String,
    model_kwargs: pxt.Json | None
) -> pxt.Array[(None,), float32]

# Signature 4:
@pxt.udf
embeddings(
    input: pxt.Audio,
    *,
    model: pxt.String,
    model_kwargs: pxt.Json | None
) -> pxt.Array[(None,), float32]

# Signature 5:
@pxt.udf
embeddings(
    input: pxt.Document,
    *,
    model: pxt.String,
    model_kwargs: pxt.Json | None
) -> pxt.Array[(None,), float32]

• pip install openai

• input (String): The text to embed. Images (pxt.Image), and for models that accept them, videos (pxt.Video), audio (pxt.Audio) and PDF documents (pxt.Document) are also accepted.
• model (String): The embedding model to use: 'qwen/qwen3-vl-embedding-2b' (2048 dimensions; text, images and video) or 'google/embeddinggemma-2' (768 dimensions; text, images, video, audio and PDFs). For other models, pass dimensions in model_kwargs; it sets the column type and is also sent to the Gateway. An embedding index over 2048-dimensional vectors needs add_embedding_index()’s default precision='fp16', since fp32 indexes allow at most 2000 dimensions.
• model_kwargs (Json | None, default: Literal(None)): Additional keyword args for the Gateway embeddings API, such as dimensions (768, 512, 256 or 128 for 'google/embeddinggemma-2') and prompt_name. 'google/embeddinggemma-2' prefixes text with a task (prompt_name, default 'Classification'); for search indexes, use 'SentenceSimilarity', since the same function embeds both the query and the indexed values.

• pxt.Array[(None,), float32]: An array containing the embedding vector.

tbl.add_embedding_index(
    tbl.img,
    embedding=embeddings.using(model='qwen/qwen3-vl-embedding-2b'),
)
sim = tbl.img.similarity(string='a red car')
tbl.order_by(sim, asc=False).limit(5).select(tbl.img).collect()

​

udf  ocr()

# Signature 1:
@pxt.udf
ocr(
    document: pxt.Document,
    *,
    model: pxt.String,
    method: pxt.String | None,
    pages: pxt.Json[(Json, ...)] | None,
    dpi: pxt.Int | None,
    model_kwargs: pxt.Json | None
) -> pxt.Json

# Signature 2:
@pxt.udf
ocr(
    document: pxt.Image,
    *,
    model: pxt.String,
    method: pxt.String | None,
    model_kwargs: pxt.Json | None
) -> pxt.Json

• pip install openai

• document (Document): The PDF document to read. Other document formats raise an error. An image (pxt.Image) is also accepted.
• model (String, default: Literal('paddlepaddle/paddleocr-vl-1.6')): The OCR model to use, such as 'paddlepaddle/paddleocr-vl-1.6' (general purpose), 'paddleocr/pp-ocrv6' (text lines with bounding boxes) or 'nvidia/nemotron-3.5-parse' (layout boxes, with method='layout'). For the available models, see https://vlm.run/gateway/models.
• method (String | None, default: Literal(None)): The model entrypoint to run, such as 'markdown', 'ocr' or 'layout'. If omitted, the model’s default method is used. For each model’s methods, see https://docs.vlm.run/gateway/methods.
• pages (Json[(Json, ...)] | None, default: Literal(None)): The zero-based pages to read, as page indices and [start, stop] ranges that exclude stop, for example [0, [2, 4]] for pages 0, 2 and 3. Selected pages are renumbered from 0. Not available for images.
• dpi (Int | None, default: Literal(None)): The resolution at which PDF pages are rasterized (Gateway default: 96). Raise it, for example to 150, for small text or dense tables. Not available for images.
• model_kwargs (Json | None, default: Literal(None)): Additional Gateway fields, such as method_params or precision. For details, see https://docs.vlm.run/gateway/extra-kwargs.

• pxt.Json: The Gateway’s JSON result, unchanged. Its shape depends on the input and the method:
  • For a PDF, document_npages is the number of pages read and pages is a list of page records, each with page_id, page_width, page_height and content, where content is {'object': 'doc.page.blocks', 'items': [...]}. A page the Gateway could not read has 'status': 'error' and no content; the other pages are still returned.
  • For an image, content is a string for text methods such as 'markdown', or {'object': ..., 'items': [...]} for region methods such as 'ocr' on 'paddleocr/pp-ocrv6'.
  • 'vlm-run/ocr:auto' with method='markdown' returns the whole PDF as one content string.

tbl.add_computed_column(pages=ocr(tbl.doc))

tbl.add_computed_column(pages=ocr(tbl.doc, pages=[[0, 10]], dpi=150))

tbl.add_computed_column(
    lines=ocr(tbl.img, model='paddleocr/pp-ocrv6', method='ocr')
)

​

udf  transcriptions()

@pxt.udf
transcriptions(
    audio: pxt.Audio,
    *,
    model: pxt.String = 'nvidia/parakeet-tdt-0.6b-v3',
    model_kwargs: pxt.Json | None = None
) -> TranscriptionResponse

• pip install openai

• audio (pxt.Audio): The audio to transcribe.
• model (pxt.String): The transcription model to use. For the available models, see https://vlm.run/gateway/models.
• model_kwargs (pxt.Json | None): Additional keyword args for the Gateway audio/transcriptions API, such as language, response_format ('json', 'verbose_json', 'text', 'srt' or 'vtt') and timestamp_granularities (['segment'], ['word'] or both; applies to verbose_json only).

• TranscriptionResponse: A dictionary with the transcript in text, plus duration and usage (including the cost in USD in usage.cost). For verbose_json, it also has segments, and words when timestamp_granularities includes 'word'. For the text, srt and vtt formats, the dictionary contains only text.

tbl.add_computed_column(transcript=transcriptions(tbl.audio))