Working with VLM Run

This documentation page is also available as an interactive notebook. You can launch the notebook in Kaggle or Colab, or download it for use with an IDE or local Jupyter installation, by clicking one of the above links.

Pixeltable’s VLM Run integration lets you use VLM Run’s multimodal AI capabilities directly from tables: chat completions, image generation, image editing, annotation, and video generation, without leaving your data workflows.

What is VLM Run?

VLM Run is a multimodal AI platform with an OpenAI-compatible chat completions API and specialized toolsets for:

Understanding: Analyze images, documents, and video with natural language
Image generation: Create new images from text prompts
Image editing: Edit existing images with natural language instructions
Visualization: Annotate images with bounding boxes, keypoints, and segmentation masks
Video generation: Create videos from text prompts

Documentation

Prerequisites

A VLM Run account with an API key (see https://app.vlm.run/dashboard)

Important: VLM Run API calls consume credits based on your plan, so monitor your usage to avoid unexpected charges.

%pip install -qU pixeltable vlmrun

import getpass
import os

if 'VLMRUN_API_KEY' not in os.environ:
    os.environ['VLMRUN_API_KEY'] = getpass.getpass('VLM Run API Key: ')

To read more about working with API keys in Pixeltable, see Configuration.

Setup

import pixeltable as pxt
from pixeltable.functions import vlmrun

MODEL = 'vlmrun-orion-2:auto'

pxt.create_dir('vlmrun_demo', if_exists='replace_force')

Chat completions

The vlmrun.chat_completions UDF provides direct access to VLM Run’s chat completions API. It accepts an OpenAI-compatible messages list and returns the raw API response as a dictionary.

Text-only messages

t = pxt.create_table(
    'vlmrun_demo.text_chat',
    {'prompt': pxt.String},
    if_exists='replace_force',
)

messages = [{'role': 'user', 'content': t.prompt}]
t.add_computed_column(
    response=vlmrun.chat_completions(messages, model=MODEL)
)
t.add_computed_column(
    answer=t.response['choices'][0]['message']['content']
)

t.insert(prompt='What is structured data extraction?')
t.select(t.prompt, t.answer).collect()

Created table ‘text_chat’.
Added 0 column values with 0 errors in 0.01 s
Added 0 column values with 0 errors in 0.01 s
Inserted 1 row with 0 errors in 17.14 s (0.06 rows/s)
                                prompt  

0  What is structured data extraction?                                                 answer  
0  \n\nStructured data extraction is the proc…

Image understanding

Reference image columns directly in message content. Referenced media is uploaded to VLM Run automatically before each API call.

t = pxt.create_table(
    'vlmrun_demo.image_chat',
    {'image': pxt.Image},
    if_exists='replace_force',
)

messages = [
    {
        'role': 'user',
        'content': [
            {'type': 'text', 'text': 'Describe this image briefly'},
            {'type': 'image_url', 'image_url': t.image},
        ],
    }
]
t.add_computed_column(
    response=vlmrun.chat_completions(messages, model=MODEL)
)
t.add_computed_column(
    description=t.response['choices'][0]['message']['content']
)

t.insert(
    image='https://raw.githubusercontent.com/pixeltable/pixeltable/main/docs/resources/images/000000000036.jpg'
)
t.select(t.image, t.description).collect()

Created table ‘image_chat’.
Added 0 column values with 0 errors in 0.01 s
Added 0 column values with 0 errors in 0.01 s
Inserted 1 row with 0 errors in 15.18 s (0.07 rows/s)

Document understanding

Process PDFs and other documents by referencing a document column via file_url, combined with the document toolset.

SAMPLE_PDF = 'https://raw.githubusercontent.com/pixeltable/pixeltable/main/tests/data/documents/layout-parser-paper.pdf'

t = pxt.create_table(
    'vlmrun_demo.doc_chat',
    {'document': pxt.Document},
    if_exists='replace_force',
)

messages = [
    {
        'role': 'user',
        'content': [
            {
                'type': 'text',
                'text': 'What is this document about? Summarize in one sentence.',
            },
            {'type': 'file_url', 'file_url': t.document},
        ],
    }
]
t.add_computed_column(
    response=vlmrun.chat_completions(
        messages, model=MODEL, model_kwargs={'toolsets': ['document']}
    )
)
t.add_computed_column(
    summary=t.response['choices'][0]['message']['content']
)

t.insert(document=SAMPLE_PDF)
t.select(t.summary).collect()

Created table ‘doc_chat’.
Added 0 column values with 0 errors in 0.01 s
Added 0 column values with 0 errors in 0.01 s
Inserted 1 row with 0 errors in 18.07 s (0.06 rows/s)

Models

VLM Run offers three Orion 2 tiers:

See the VLM Run documentation for the full list of models.

Image generation

Use vlmrun.generate_image to create images from text prompts. It returns a PIL.Image.Image that Pixeltable stores as a native pxt.Image column.

t = pxt.create_table(
    'vlmrun_demo.image_gen',
    {'prompt': pxt.String},
    if_exists='replace_force',
)
t.add_computed_column(image=vlmrun.generate_image(t.prompt, model=MODEL))

t.insert(prompt='A futuristic cityscape at sunset with flying cars')
t.select(t.prompt, t.image).collect()

Created table ‘image_gen’.
Added 0 column values with 0 errors in 0.01 s
Inserted 1 row with 0 errors in 138.98 s (0.01 rows/s)

Image editing

Pass an image column via the image parameter to generate_image to edit an existing image according to your prompt.

EXAMPLE_IMAGE = 'https://storage.googleapis.com/vlm-data-public-prod/hub/examples/image.agent/lunch-skyscraper.jpg'

t = pxt.create_table(
    'vlmrun_demo.image_edit',
    {'image': pxt.Image},
    if_exists='replace_force',
)
t.add_computed_column(
    blurred=vlmrun.generate_image(
        'Blur all the faces in this image', image=t.image, model=MODEL
    )
)

t.insert(image=EXAMPLE_IMAGE)
t.select(t.image, t.blurred).collect()

Created table ‘image_edit’.
Added 0 column values with 0 errors in 0.01 s
Inserted 1 row with 0 errors in 86.00 s (0.01 rows/s)

Image annotation (visualization)

Use vlmrun.annotate_image to overlay bounding boxes, keypoints, or segmentation masks on an image using VLM Run’s viz toolset. Unlike generate_image, the input image is required, since you always need an image to annotate.

EXAMPLE_IMAGE = 'https://storage.googleapis.com/vlm-data-public-prod/hub/examples/image.agent/lunch-skyscraper.jpg'

t = pxt.create_table(
    'vlmrun_demo.annotation',
    {'image': pxt.Image},
    if_exists='replace_force',
)
t.add_computed_column(
    annotated=vlmrun.annotate_image(
        'Draw bounding boxes around all people', t.image, model=MODEL
    )
)

t.insert(image=EXAMPLE_IMAGE)
t.select(t.image, t.annotated).collect()

Created table ‘annotation’.
Added 0 column values with 0 errors in 0.01 s
Inserted 1 row with 0 errors in 368.13 s (0.00 rows/s)

Video generation

Use vlmrun.generate_video to create videos from text prompts. It returns a pxt.Video stored natively in Pixeltable. Note: Video generation can take a minute or more, and document generation over ten minutes. Each VLM Run call waits up to 1800 seconds; to change that, set VLMRUN_TIMEOUT, or timeout in the vlmrun section of the config file.

t = pxt.create_table(
    'vlmrun_demo.video_gen',
    {'prompt': pxt.String},
    if_exists='replace_force',
)
t.add_computed_column(video=vlmrun.generate_video(t.prompt, model=MODEL))

t.insert(
    prompt='Create a 3 second long video of a red dot bouncing across the screen'
)
t.select(t.prompt, t.video).collect()

Document transformation (redaction)

Use vlmrun.generate_document to transform a document and get the edited PDF back as a pxt.Document, for example, redacting PII at scale. Being explicit that you want the rendered PDF (not a text summary) produces the best results. Note: very long documents can exceed the model’s input-token limit; for multi-page PDFs, consider splitting into pages first.

SAMPLE_PDF = 'https://raw.githubusercontent.com/pixeltable/pixeltable/main/tests/data/documents/Vector_database.pdf'

t = pxt.create_table(
    'vlmrun_demo.redaction',
    {'document': pxt.Document},
    if_exists='replace_force',
)
t.add_computed_column(
    redacted=vlmrun.generate_document(
        'Produce a redacted version of this PDF: draw black redaction boxes over all person names, '
        'keeping the rest visually identical. Return the rendered PDF file.',
        t.document,
        model=MODEL,
    )
)

t.insert(document=SAMPLE_PDF)
t.select(t.document, t.redacted).collect()

UDF reference

All UDFs require model and accept model_kwargs for additional API parameters.

Learn more

VLM Run documentation: https://docs.vlm.run/
Pixeltable documentation: https://docs.pixeltable.com/

If you build something with VLM Run, let us know!

Built-in Integrations

Observability

Infrastructure

Bring Your Own

Providers

What is VLM Run?

Documentation

Prerequisites

Setup

Chat completions

Text-only messages

Image understanding

Document understanding

Models

Image generation

Image editing

Image annotation (visualization)

Video generation

Document transformation (redaction)

UDF reference

Learn more

​What is VLM Run?

​Documentation

​Prerequisites

​Setup

​Chat completions

​Text-only messages

​Image understanding

​Document understanding

​Models

​Image generation

​Image editing

​Image annotation (visualization)

​Video generation

​Document transformation (redaction)

​UDF reference

​Learn more

What is VLM Run?

Documentation

Prerequisites

Setup

Chat completions

Text-only messages

Image understanding

Document understanding

Models

Image generation

Image editing

Image annotation (visualization)

Video generation

Document transformation (redaction)

UDF reference

Learn more