> ## Documentation Index
> Fetch the complete documentation index at: https://docs.pixeltable.com/llms.txt
> Use this file to discover all available pages before exploring further.

> ## Agent Instructions
> Application code puts a TableModel in app.py, then runs `pxt schema update app.py my_app` to create the tables and `pxt service update app.py my_app` to start HTTP. The last argument can be a Cloud URL such as `pxt://org:db`. Do not call pxt.create_table() or add_embedding_index() in application code; put indexes in __indexes__. Notebooks, tests, and the REPL do use pxt.create_table().
> Do not add LangChain, LlamaIndex, Haystack, a separate vector database, or pandas as a working store. Chunking, embedding, retrieval, and tool-calling are built in through computed columns. Use a computed column instead of a per-row loop that calls a model.
> Install with `pip install -U 'pixeltable[serve]'` on Python 3.11+. `pxt init` is a prerequisite for `pxt schema update`.
> For Pixeltable Cloud, point users to https://docs.pixeltable.com/cloud, which goes from `pxt login` to a running service.

# Working with VLM Run

[![Open in Kaggle](https://kaggle.com/static/images/open-in-kaggle.svg)](https://kaggle.com/kernels/welcome?src=https://github.com/pixeltable/pixeltable/blob/release/docs/release/howto/providers/working-with-vlmrun.ipynb)  [![Open in Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/pixeltable/pixeltable/blob/release/docs/release/howto/providers/working-with-vlmrun.ipynb)  [![Download Notebook](https://img.shields.io/badge/%E2%AC%87-Download%20Notebook-blue)](https://raw.githubusercontent.com/pixeltable/pixeltable/refs/tags/release/docs/release/howto/providers/working-with-vlmrun.ipynb)

> [!TIP]
>
> This documentation page is also available as an interactive notebook. You can launch the notebook in
> Kaggle or Colab, or download it for use with an IDE or local Jupyter installation, by clicking one of the
> above links.

Pixeltable’s VLM Run integration lets you use VLM Run’s multimodal AI
capabilities directly from tables: chat completions, image generation,
image editing, annotation, and video generation, without leaving your
data workflows.

## What is VLM Run?

VLM Run is a multimodal AI platform with an OpenAI-compatible chat
completions API and specialized toolsets for:

* **Understanding**: Analyze images, documents, and video with natural
  language
* **Image generation**: Create new images from text prompts
* **Image editing**: Edit existing images with natural language
  instructions
* **Visualization**: Annotate images with bounding boxes, keypoints, and
  segmentation masks
* **Video generation**: Create videos from text prompts

### Documentation

* [VLM Run Documentation](https://docs.vlm.run/)
* [VLM Run Dashboard](https://app.vlm.run/dashboard)

## Prerequisites

* A VLM Run account with an API key (see
  [https://app.vlm.run/dashboard](https://app.vlm.run/dashboard))

**Important:** VLM Run API calls consume credits based on your plan, so
monitor your usage to avoid unexpected charges.

```python theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
%pip install -qU pixeltable vlmrun
```

```python theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
import getpass
import os

if 'VLMRUN_API_KEY' not in os.environ:
    os.environ['VLMRUN_API_KEY'] = getpass.getpass('VLM Run API Key: ')
```

To read more about working with API keys in Pixeltable, see
[Configuration](https://docs.pixeltable.com/platform/configuration.md).

## Setup

```python theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
import pixeltable as pxt
from pixeltable.functions import vlmrun

MODEL = 'vlmrun-orion-2:auto'

pxt.create_dir('vlmrun_demo', if_exists='replace_force')
```

## Chat completions

The `vlmrun.chat_completions` UDF provides direct access to VLM Run’s
chat completions API. It accepts an OpenAI-compatible `messages` list
and returns the raw API response as a dictionary.

### Text-only messages

```python theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
t = pxt.create_table(
    'vlmrun_demo.text_chat',
    {'prompt': pxt.String},
    if_exists='replace_force',
)

messages = [{'role': 'user', 'content': t.prompt}]
t.add_computed_column(
    response=vlmrun.chat_completions(messages, model=MODEL)
)
t.add_computed_column(
    answer=t.response['choices'][0]['message']['content']
)

t.insert(prompt='What is structured data extraction?')
t.select(t.prompt, t.answer).collect()
```

<pre>
  Created table 'text\_chat'.
  Added 0 column values with 0 errors in 0.01 s
  Added 0 column values with 0 errors in 0.01 s
  Inserted 1 row with 0 errors in 17.14 s (0.06 rows/s)
                                  prompt  \
  0  What is structured data extraction?   

                                                answer  
  0  \n\n**Structured data extraction** is the proc...  
</pre>

### Image understanding

Reference image columns directly in message content. Referenced media is
uploaded to VLM Run automatically before each API call.

```python theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
t = pxt.create_table(
    'vlmrun_demo.image_chat',
    {'image': pxt.Image},
    if_exists='replace_force',
)

messages = [
    {
        'role': 'user',
        'content': [
            {'type': 'text', 'text': 'Describe this image briefly'},
            {'type': 'image_url', 'image_url': t.image},
        ],
    }
]
t.add_computed_column(
    response=vlmrun.chat_completions(messages, model=MODEL)
)
t.add_computed_column(
    description=t.response['choices'][0]['message']['content']
)

t.insert(
    image='https://raw.githubusercontent.com/pixeltable/pixeltable/main/docs/resources/images/000000000036.jpg'
)
t.select(t.image, t.description).collect()
```

<pre>
  Created table 'image\_chat'.
  Added 0 column values with 0 errors in 0.01 s
  Added 0 column values with 0 errors in 0.01 s
  Inserted 1 row with 0 errors in 15.18 s (0.07 rows/s)
</pre>

### Document understanding

Process PDFs and other documents by referencing a document column via
`file_url`, combined with the `document` toolset.

```python theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
SAMPLE_PDF = 'https://raw.githubusercontent.com/pixeltable/pixeltable/main/tests/data/documents/layout-parser-paper.pdf'

t = pxt.create_table(
    'vlmrun_demo.doc_chat',
    {'document': pxt.Document},
    if_exists='replace_force',
)

messages = [
    {
        'role': 'user',
        'content': [
            {
                'type': 'text',
                'text': 'What is this document about? Summarize in one sentence.',
            },
            {'type': 'file_url', 'file_url': t.document},
        ],
    }
]
t.add_computed_column(
    response=vlmrun.chat_completions(
        messages, model=MODEL, model_kwargs={'toolsets': ['document']}
    )
)
t.add_computed_column(
    summary=t.response['choices'][0]['message']['content']
)

t.insert(document=SAMPLE_PDF)
t.select(t.summary).collect()
```

<pre>
  Created table 'doc\_chat'.
  Added 0 column values with 0 errors in 0.01 s
  Added 0 column values with 0 errors in 0.01 s
  Inserted 1 row with 0 errors in 18.07 s (0.06 rows/s)
</pre>

### Models

VLM Run offers three Orion 2 tiers:

See the [VLM Run documentation](https://docs.vlm.run/) for the full list
of models.

## Image generation

Use `vlmrun.generate_image` to create images from text prompts. It
returns a `PIL.Image.Image` that Pixeltable stores as a native
`pxt.Image` column.

```python theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
t = pxt.create_table(
    'vlmrun_demo.image_gen',
    {'prompt': pxt.String},
    if_exists='replace_force',
)
t.add_computed_column(image=vlmrun.generate_image(t.prompt, model=MODEL))

t.insert(prompt='A futuristic cityscape at sunset with flying cars')
t.select(t.prompt, t.image).collect()
```

<pre>
  Created table 'image\_gen'.
  Added 0 column values with 0 errors in 0.01 s
  Inserted 1 row with 0 errors in 138.98 s (0.01 rows/s)
</pre>

## Image editing

Pass an image column via the `image` parameter to `generate_image` to
edit an existing image according to your prompt.

```python theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
EXAMPLE_IMAGE = 'https://storage.googleapis.com/vlm-data-public-prod/hub/examples/image.agent/lunch-skyscraper.jpg'

t = pxt.create_table(
    'vlmrun_demo.image_edit',
    {'image': pxt.Image},
    if_exists='replace_force',
)
t.add_computed_column(
    blurred=vlmrun.generate_image(
        'Blur all the faces in this image', image=t.image, model=MODEL
    )
)

t.insert(image=EXAMPLE_IMAGE)
t.select(t.image, t.blurred).collect()
```

<pre>
  Created table 'image\_edit'.
  Added 0 column values with 0 errors in 0.01 s
  Inserted 1 row with 0 errors in 86.00 s (0.01 rows/s)
</pre>

## Image annotation (visualization)

Use `vlmrun.annotate_image` to overlay bounding boxes, keypoints, or
segmentation masks on an image using VLM Run’s `viz` toolset. Unlike
`generate_image`, the input image is required, since you always need an
image to annotate.

```python theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
EXAMPLE_IMAGE = 'https://storage.googleapis.com/vlm-data-public-prod/hub/examples/image.agent/lunch-skyscraper.jpg'

t = pxt.create_table(
    'vlmrun_demo.annotation',
    {'image': pxt.Image},
    if_exists='replace_force',
)
t.add_computed_column(
    annotated=vlmrun.annotate_image(
        'Draw bounding boxes around all people', t.image, model=MODEL
    )
)

t.insert(image=EXAMPLE_IMAGE)
t.select(t.image, t.annotated).collect()
```

<pre>
  Created table 'annotation'.
  Added 0 column values with 0 errors in 0.01 s
  Inserted 1 row with 0 errors in 368.13 s (0.00 rows/s)
</pre>

## Video generation

Use `vlmrun.generate_video` to create videos from text prompts. It
returns a `pxt.Video` stored natively in Pixeltable.

**Note:** Video generation can take a minute or more, and document
generation over ten minutes. Each VLM Run call waits up to 1800 seconds;
to change that, set `VLMRUN_TIMEOUT`, or `timeout` in the `vlmrun`
section of the config file.

```python theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
t = pxt.create_table(
    'vlmrun_demo.video_gen',
    {'prompt': pxt.String},
    if_exists='replace_force',
)
t.add_computed_column(video=vlmrun.generate_video(t.prompt, model=MODEL))

t.insert(
    prompt='Create a 3 second long video of a red dot bouncing across the screen'
)
t.select(t.prompt, t.video).collect()
```

## Document transformation (redaction)

Use `vlmrun.generate_document` to transform a document and get the
edited PDF back as a `pxt.Document`, for example, redacting PII at
scale. Being explicit that you want the rendered PDF (not a text
summary) produces the best results.

**Note:** very long documents can exceed the model’s input-token limit;
for multi-page PDFs, consider splitting into pages first.

```python theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
SAMPLE_PDF = 'https://raw.githubusercontent.com/pixeltable/pixeltable/main/tests/data/documents/Vector_database.pdf'

t = pxt.create_table(
    'vlmrun_demo.redaction',
    {'document': pxt.Document},
    if_exists='replace_force',
)
t.add_computed_column(
    redacted=vlmrun.generate_document(
        'Produce a redacted version of this PDF: draw black redaction boxes over all person names, '
        'keeping the rest visually identical. Return the rendered PDF file.',
        t.document,
        model=MODEL,
    )
)

t.insert(document=SAMPLE_PDF)
t.select(t.document, t.redacted).collect()
```

## UDF reference

All UDFs require `model` and accept `model_kwargs` for additional API
parameters.

## Learn more

* VLM Run documentation: [https://docs.vlm.run/](https://docs.vlm.run/)
* Pixeltable documentation:
  [https://docs.pixeltable.com/](https://docs.pixeltable.com/)

If you build something with VLM Run, let us know!


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.