> ## Documentation Index
> Fetch the complete documentation index at: https://docs.pixeltable.com/llms.txt
> Use this file to discover all available pages before exploring further.
> ## Agent Instructions
> Application code puts a TableModel in app.py, then runs `pxt schema update app.py my_app` to create the tables and `pxt service update app.py my_app` to start HTTP. The last argument can be a Cloud URL such as `pxt://org:db`. Do not call pxt.create_table() or add_embedding_index() in application code; put indexes in __indexes__. Notebooks, tests, and the REPL do use pxt.create_table().
> Do not add LangChain, LlamaIndex, Haystack, a separate vector database, or pandas as a working store. Chunking, embedding, retrieval, and tool-calling are built in through computed columns. Use a computed column instead of a per-row loop that calls a model.
> Install with `pip install -U 'pixeltable[serve]'` on Python 3.11+. `pxt init` is a prerequisite for `pxt schema update`.
> For Pixeltable Cloud, point users to https://docs.pixeltable.com/cloud, which goes from `pxt login` to a running service.
# Working with the VLM Run Gateway in Pixeltable
[](https://kaggle.com/kernels/welcome?src=https://github.com/pixeltable/pixeltable/blob/release/docs/release/howto/providers/working-with-vlmrun-gateway.ipynb) [](https://colab.research.google.com/github/pixeltable/pixeltable/blob/release/docs/release/howto/providers/working-with-vlmrun-gateway.ipynb) [](https://raw.githubusercontent.com/pixeltable/pixeltable/refs/tags/release/docs/release/howto/providers/working-with-vlmrun-gateway.ipynb)
> [!TIP]
>
> This documentation page is also available as an interactive notebook. You can launch the notebook in
> Kaggle or Colab, or download it for use with an IDE or local Jupyter installation, by clicking one of the
> above links.
The [VLM Run Gateway](https://docs.vlm.run/gateway/introduction) is an
OpenAI-compatible API in front of vision and speech models: OCR and
document parsing, visual question answering, segmentation, pose
estimation, multimodal embeddings and speech-to-text. Pixeltable’s
`vlmrun_gateway` module turns each of these into a computed column, so
images, PDFs and audio in your tables are processed as rows are
inserted.
### Prerequisites
* A VLM Run account with an API key
([app.vlm.run](https://app.vlm.run/dashboard/settings/api-keys))
### Important notes
* Gateway usage is billed per token, image, video or audio second, with
a minimum of \$0.001 per call on most models (not embeddings or the
partner Gemini, Kimi, MiniMax and Muse models)
([pricing](https://docs.vlm.run/gateway/pricing)).
`chat_completions()` reports each call’s cost in `usage.cost`, as does
`transcriptions()` in its JSON formats.
* Be mindful of sensitive data and consider security measures when
integrating with external services.
First you’ll need to install the required libraries and enter your VLM
Run API key.
```python theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
%pip install -qU pixeltable openai
```
```python theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
import getpass
import os
if 'VLMRUN_API_KEY' not in os.environ:
os.environ['VLMRUN_API_KEY'] = getpass.getpass(
'Enter your VLM Run API key:'
)
```
Now let’s create a Pixeltable directory to hold the tables for our demo.
```python theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
import json
import pixeltable as pxt
from pixeltable.functions import vlmrun_gateway
# Remove the 'vlmrun_gateway_demo' directory and its contents, if it exists
pxt.drop_dir('vlmrun_gateway_demo', force=True)
pxt.create_dir('vlmrun_gateway_demo')
SAMPLES = (
'https://storage.googleapis.com/vlm-data-public-prod/hub/examples'
)
```
Created directory 'vlmrun\_gateway\_demo'.
## Read PDFs with OCR
`ocr()` sends a document to an OCR model and returns the Gateway’s JSON,
whose `pages` list has one record per page. The default model,
`paddlepaddle/paddleocr-vl-1.6`, reads each page as Markdown. Use
`pages` to read part of a long document. Pages are rasterized at 96 DPI
by default; dense tables, like the line items on this invoice, can come
back garbled at that resolution, so this example raises `dpi` to 150.
```python theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
docs = pxt.create_table('vlmrun_gateway_demo/docs', {'doc': pxt.Document})
docs.add_computed_column(pages=vlmrun_gateway.ocr(docs.doc, dpi=150))
# The first page's text, as Markdown
docs.add_computed_column(
first_page=docs.pages.pages[0].content['items'][0].text
)
```
Created table 'docs'.
Added 0 column values with 0 errors in 0.01 s
Added 0 column values with 0 errors in 0.01 s
No rows affected.
```python theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
docs.insert([{'doc': f'{SAMPLES}/document.invoice/sample-invoice.pdf'}])
docs.select(
npages=docs.pages.document_npages, first_page=docs.first_page
).collect()
```
Inserted 1 row with 0 errors in 2.68 s (0.37 rows/s)
To get text lines with bounding boxes instead, pick a model and method
that return regions, such as `model='paddleocr/pp-ocrv6', method='ocr'`.
Each model’s methods are listed in the [Gateway
docs](https://docs.vlm.run/gateway/methods).
## Extract structured data from images
`chat_completions()` works like the OpenAI endpoint of the same name.
Image columns can be referenced directly in messages. With
`response_format` set to a JSON schema, the model’s reply matches your
schema.
```python theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
receipts = pxt.create_table(
'vlmrun_gateway_demo/receipts', {'image': pxt.Image}
)
messages = [
{
'role': 'user',
'content': [
{
'type': 'text',
'text': 'Extract the merchant and the total.',
},
{'type': 'image_url', 'image_url': receipts.image},
],
}
]
receipt_schema = {
'type': 'object',
'properties': {
'merchant': {'type': 'string'},
'total': {'type': 'number'},
},
'required': ['merchant', 'total'],
}
receipts.add_computed_column(
response=vlmrun_gateway.chat_completions(
messages,
model='qwen/qwen3.8-27b',
model_kwargs={
'response_format': {
'type': 'json_schema',
'json_schema': {
'name': 'receipt',
'schema': receipt_schema,
},
}
},
)
)
# The reply is a JSON string; parse it into a JSON column
@pxt.udf
def parse_json(s: str) -> pxt.Json:
return json.loads(s)
receipts.add_computed_column(
receipt=parse_json(receipts.response.choices[0].message.content)
)
```
Created table 'receipts'.
Added 0 column values with 0 errors in 0.01 s
Added 0 column values with 0 errors in 0.01 s
No rows affected.
```python theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
receipts.insert(
[{'image': f'{SAMPLES}/document.receipt/playground/2.jpg'}]
)
receipts.select(
receipts.receipt, cost=receipts.response.usage.cost
).collect()
```
Inserted 1 row with 0 errors in 2.62 s (0.38 rows/s)
## Segment objects
`analyze()` runs detection, segmentation and pose models. Here
`facebook/sam3.1` finds every object that matches a text prompt. Boxes
are normalized to the range 0 to 1, and `'mask_format': 'none'` in
`method_params` skips the masks. The bridge photo has no cars, so its
list is empty.
```python theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
photos = pxt.create_table(
'vlmrun_gateway_demo/photos', {'image': pxt.Image}
)
photos.add_computed_column(
cars=vlmrun_gateway.analyze(
photos.image,
model='facebook/sam3.1',
method='segment',
method_params={'prompt': 'car', 'mask_format': 'none'},
)
)
photos.insert(
[
{'image': f'{SAMPLES}/image.caption/car.jpg'},
{'image': f'{SAMPLES}/image.generation/sf-golden-gate.jpg'},
]
)
photos.select(photos.image, photos.cars.content['items']).collect()
```
Created table 'photos'.
Added 0 column values with 0 errors in 0.01 s
Inserted 2 rows with 0 errors in 0.80 s (2.51 rows/s)
## Search images with text
`embeddings()` puts text and images in one vector space, so an embedding
index on an image column can be searched with a text query.
```python theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
photos.add_embedding_index(
photos.image,
embedding=vlmrun_gateway.embeddings.using(
model='qwen/qwen3-vl-embedding-2b'
),
)
sim = photos.image.similarity(string='a bridge over the water')
photos.order_by(sim, asc=False).limit(2).select(
photos.image, sim
).collect()
```
## Transcribe audio
`transcriptions()` turns speech in an audio column into text, with
`nvidia/parakeet-tdt-0.6b-v3` by default. The sample here is a 10-second
clip, so the transcript ends mid-sentence. For timestamps, pass
`model_kwargs={'response_format': 'verbose_json', 'timestamp_granularities': ['segment', 'word']}`.
```python theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
audio = pxt.create_table(
'vlmrun_gateway_demo/audio', {'audio': pxt.Audio}
)
audio.add_computed_column(
transcript=vlmrun_gateway.transcriptions(audio.audio)
)
speech = f'{SAMPLES}/audio.transcription-summary/two_minute_rules.mp3'
audio.insert([{'audio': speech}])
audio.select(audio.transcript.text).collect()
```
Created table 'audio'.
Added 0 column values with 0 errors in 0.01 s
Inserted 1 row with 0 errors in 0.73 s (1.38 rows/s)
## Rate limits
The Gateway allows 240 requests per minute and 10,000 per hour per
account, below Pixeltable’s default of 600 RPM. To stay under both, set
`rate_limit = 160` in the `vlmrun_gateway` section of your Pixeltable
config, or set the `VLMRUN_GATEWAY_RATE_LIMIT` environment variable.
### Learn more
* [VLM Run Gateway
documentation](https://docs.vlm.run/gateway/introduction)
* [Available models](https://vlm.run/gateway/models)
* [Embedding indexes in
Pixeltable](https://docs.pixeltable.com/platform/embedding-indexes.md)
This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.