.rate_limits` section and list the rate limits for each model:
```toml theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
[openai]
api_key = 'my-openai-api-key'
[openai.rate_limits]
gpt-4o = 500
gpt-4o-mini = 1000
tts-1 = 50
dall-e-3 = 10
[gemini.rate_limits]
gemini-2.5-flash = 600
gemini-2.5-pro = 300
```
If no rate limit is configured, Pixeltable uses a default of 600 requests per minute.
## Configuration best practices
### Security considerations
When configuring API keys and sensitive information:
* Avoid hardcoding API keys in your code
* Use environment variables for temporary access
* Use the config file for persistent configuration
* Ensure your config.toml file has appropriate permissions (readable only by you)
### Performance tuning
* Adjust `file_cache_size_g` based on your available system memory
* For large datasets, increase the cache size to improve performance
* Set appropriate verbosity level based on your debugging needs
## Applying configuration changes
Configuration changes take effect when you restart your Python session.
Return to the installation guide for setup instructions
# Dashboard (local UI)
Source: https://docs.pixeltable.com/platform/dashboard
Browse tables, inspect pipelines, preview media, and debug computed columns from a local web UI served by the pxt daemon.
The Pixeltable dashboard is a local web UI for exploring your data. It's served by the pxt daemon at [http://localhost:22089](http://localhost:22089).
No extra install is needed. The dashboard ships inside the `pixeltable` package and uses the same daemon that backs the [`pxt` CLI](/platform/cli).
## Opening the dashboard
Run `pxt dashboard` from a terminal:
```bash theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
pxt dashboard
```
This auto-spawns the daemon if it isn't already running, prints the URL, and opens it in your default browser. The daemon survives across shells and stays warm; subsequent invocations just hand off the URL.
The dashboard binds to `127.0.0.1` only and is not accessible from other machines on the network. For remote access to tables and queries, see [HTTP Serving](/howto/deployment/serving).
## What you can do
The dashboard is a **read-only inspector**. You browse and debug your data visually; all mutations (inserts, schema changes, etc.) happen through the Python SDK or [CLI](/platform/cli).
### Browse the catalog
The left sidebar has a sticky **Local / Cloud** catalog switcher. You browse **one active catalog** at a time: the local catalog on this machine by default, or a hosted database you attach with a `pxt://org:db` URI. The tree under the switcher shows that catalog’s directories, tables, views, snapshots, and replicas. Each entry shows its kind (icon next to the name), version (`vN` when present), and error marks when relevant. Click a table to open it, or a directory for a summary and child list.
Large catalogs get a **Filter…** field under the switcher to narrow the tree by name. That is separate from global search: **Cmd/Ctrl + K** (search icon in the sidebar header) finds directories, tables, and columns across attached catalogs.
### Inspect table data
The main table view has three tabs:
* **Data** - paginated rows with typed columns, sortable by indexed columns, with configurable page sizes. Cells render inline previews for images, video, audio, and documents.
* **Lineage** - shows the table's position in the pipeline: its base table, derived views, and column-level dependency graphs.
* **History** - version list with change types (inserts, updates, deletes) and timestamps.
Above the data grid, a collapsible **COLUMNS** panel lists every column with its type, whether it is computed, stored vs. unstored, and any embedding indexes. Collapse it to a compact chip summary of the schema; expand for the full table. Click a computed expression to open a readable view (with a Raw toggle). The Copy button copies the original expression text, not the pretty-printed view.
### Filter and search rows
A filter panel lets you narrow down the current page:
* **Free-text search** across all columns (debounced)
* **Faceted filters**: text contains, numeric ranges, datetime ranges, boolean/enum checklists
* **Errors-only mode**: server-side filter that shows only rows with computation errors, useful for debugging failed computed columns
### Preview media
Images and videos render inline in the data grid. Click any media cell to open a **lightbox** with keyboard navigation (arrow keys for prev/next). For image-heavy tables, a **gallery view** displays thumbnails in a grid with a detail overlay per row.
### Explore the pipeline
The **Lineage** page (lineage icon in the sidebar header, next to Search) shows a full-instance pipeline graph built with React Flow. Every table and view appears as a node, with edges showing relationships (base table, iterator, query dependencies).
Click any node to open a detail drawer with:
* Row count and version info
* Column breakdown (insertable, computed, stored)
* Embedding index definitions
* Computed column expressions with dependency graphs
* Version history
A **Find table** search on the canvas lets you jump to any node and fit it into view.
### Export data
Download the current table as **CSV** (up to 100,000 rows) from the table view.
### Copy SDK snippets
The table view includes a **Copy** button that generates the Python SDK code for opening the current table, making it easy to switch from visual inspection to scripted work.
## Configuration
| Setting | Default | Environment variable | Description |
| - | - | - | - |
| Daemon port | `22089` | `PXT_PORT` | Port the daemon (and dashboard) listens on |
Set the port before invoking any `pxt` command:
```bash theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
PXT_PORT=3000 pxt dashboard
```
## Keyboard shortcuts
| Shortcut | Action |
| - | - |
| **Cmd/Ctrl + K** | Open global search |
| **Cmd/Ctrl + F** | Focus in-table row filter |
| **Esc** | Close search, modals, or lightbox |
| **Arrow keys** | Navigate media in lightbox |
## Troubleshooting
**Port already in use**: If another process occupies port 22089, set a different port via `PXT_PORT=3000`. The daemon refuses to start on a bound port and the CLI surfaces the bind error.
**Daemon not responding**: Check `pxt daemon status` for a quick health report. If the daemon is stuck, `pxt daemon stop` then `pxt daemon start` cycles it.
**Static files missing**: The React app ships as pre-built static files inside the `pixeltable` package. If you installed from source, ensure the dashboard was built (`npm run build` in the `dashboard/` directory).
# Embedding Indices
Source: https://docs.pixeltable.com/platform/embedding-indexes
![Open in Kaggle]()
![Open in Colab]()
![Download Notebook]()
This documentation page is also available as an interactive notebook. You can launch the notebook in
Kaggle or Colab, or download it for use with an IDE or local Jupyter installation, by clicking one of the
above links.
This notebook is about vector (embedding) indexes. B-tree indexes are
separate and opt-in: `pxt.BtreeIndex(col)` on a `TableModel`, or
`t.add_btree_index('col')` in a notebook.
This notebook uses `pxt.create_table()` and `add_embedding_index()` so
you can run cells without a project. In an app, list embedding indexes
on the model in `__indexes__` and create the tables with
`pxt schema update`.
* Indexing in Pixeltable is declarative
* you create an index on a column and supply the embedding functions
you want to use (for inserting data into the index as well as
lookups)
* Pixeltable maintains the index in response to any kind of update of
the indexed table (i.e., `insert()`/`update()`/`delete()`)
* Perform index lookups with the `similarity()` pseudo-function, in
combination with the `order_by()` and `limit()` clauses
To make this concrete, let’s create a table of images with the
[`create_table()`](/sdk/latest/pixeltable#func-create_table)
function. We’re also going to add some columns to demonstrate combining
similarity search with other predicates.
```python theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
%pip install -qU pixeltable transformers sentence-transformers
```
\[notice] A new release of pip is available: 25.3 -> 26.0.1
\[notice] To update, run: pip install --upgrade pip
Note: you may need to restart the kernel to use updated packages.
```python theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
import pixeltable as pxt
# Delete the `indices_demo` directory and its contents, if it exists
pxt.drop_dir('indices_demo', force=True)
# Create the directory and table to use for the demo
pxt.create_dir('indices_demo')
schema = {'id': pxt.Int, 'img': pxt.Image}
imgs = pxt.create_table('indices_demo/img_tbl', schema)
```
We start out by inserting 10 rows:
```python theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
img_urls = [
'https://raw.githubusercontent.com/pixeltable/pixeltable/release/docs/resources/images/000000000030.jpg',
'https://raw.githubusercontent.com/pixeltable/pixeltable/release/docs/resources/images/000000000034.jpg',
'https://raw.githubusercontent.com/pixeltable/pixeltable/release/docs/resources/images/000000000042.jpg',
'https://raw.githubusercontent.com/pixeltable/pixeltable/release/docs/resources/images/000000000049.jpg',
'https://raw.githubusercontent.com/pixeltable/pixeltable/release/docs/resources/images/000000000057.jpg',
'https://raw.githubusercontent.com/pixeltable/pixeltable/release/docs/resources/images/000000000061.jpg',
'https://raw.githubusercontent.com/pixeltable/pixeltable/release/docs/resources/images/000000000063.jpg',
'https://raw.githubusercontent.com/pixeltable/pixeltable/release/docs/resources/images/000000000064.jpg',
'https://raw.githubusercontent.com/pixeltable/pixeltable/release/docs/resources/images/000000000069.jpg',
'https://raw.githubusercontent.com/pixeltable/pixeltable/release/docs/resources/images/000000000071.jpg',
]
imgs.insert({'id': i, 'img': url} for i, url in enumerate(img_urls))
```
Inserted 10 rows with 0 errors in 2.53 s (3.96 rows/s)
10 rows inserted.
For the sake of convenience, we’re storing the images as external URLs,
which are cached transparently by Pixeltable. For details on working
with external media files, see [Working with External
Files](/platform/external-files).
## Creating an index
To create and populate an index, we call
[`Table.add_embedding_index()`](/sdk/latest/table#method-add_embedding_index)
and tell it which UDF or UDFs to use to create embeddings. That
definition is persisted as part of the table’s metadata, which allows
Pixeltable to maintain the index in response to updates to the table.
Any embedding UDF can be used for the index. For this example, we’re
going to use a
[CLIP](https://huggingface.co/docs/transformers/en/model_doc/clip)
model, which has built-in support in Pixeltable under the
[`pixeltable.functions.huggingface`](/sdk/latest/huggingface)
package. As an alternative, you could use an online service such as
OpenAI (see
[`pixeltable.functions.openai`](/sdk/latest/openai)),
or create your own embedding UDF with custom code (we’ll see how to do
this below).
Because we’re adding an index to an image column, the UDF we specify
*must* be able to handle images. In fact, CLIP models are multimodal:
they can handle both text and images, which is useful for doing lookups
against the index.
```python theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
import PIL.Image
from pixeltable.functions.huggingface import clip
# create embedding index on the 'img' column
imgs.add_embedding_index(
'img', embedding=clip.using(model_id='openai/clip-vit-base-patch32')
)
```
The first parameter of `add_embedding_index()` is the name of the column
being indexed; the `embed` parameter specifies the relevant embedding.
Notice the notation we used:
```python theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
clip.using(model_id='openai/clip-vit-base-patch32')
```
`clip` is a general-purpose UDF that can accept any CLIP model available
in the Hugging Face model repository. To define an embedding, however,
we need to provide a specific embedding function to
`add_embedding_index()`: a function that is *not* parameterized on
`model_id`. The `.using(model_id=...)` syntax tells Pixeltable to
specialize the `clip` UDF by fixing the `model_id` parameter to the
specific value `'openai/clip-vit-base-patch32'`.
If you’re familiar with functional programming
concepts, you might recognize .using() as a partial
function operator. It’s a general operator that can be applied to
any UDF (not just embedding functions), transforming a UDF with n
parameters into one with k parameters by fixing the values of
n-k of its arguments. Python has something similar in the
functools package: the
functools.partial()
operator.
`add_embedding_index()` provides a few other optional parameters:
* `idx_name`: optional name for the index, which needs to be unique for
the table; a default name is created if this isn’t provided explicitly
* `metric`: the metric to use to compute the similarity of two embedding
vectors; one of:
* `'cosine'`: cosine distance (default)
* `'ip'`: inner product
* `'l2'`: L2 distance
If desired, you can create multiple indexes on the same column, using
different embedding functions. This can be useful to evaluate the
effectiveness of different embedding functions side-by-side, or to use
embedding functions tailored to specific use cases. In that case, you
can provide explicit names for those indexes and then reference them
during queries. We’ll illustrate that later with an example.
## Using the index in queries
To take advantage of an embedding index when querying a table, we use
the `similarity()` pseudo-function, which is invoked as a method on the
indexed column, in combination with the
[`order_by()`](/sdk/latest/query#method-order_by)
and
[`limit()`](/sdk/latest/query#method-limit)
clauses. First, we’ll get a sample image from the table:
```python theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
# retrieve the 'img' column of some row as a PIL.Image.Image
sample_img = imgs.select(imgs.img).collect()[6]['img']
sample_img
```
![]()
We then call the `similarity()` pseudo-function as a method on the
indexed column and apply `order_by()` and `limit()`. We used the default
cosine distance when we created the index, so we’re going to order by
descending similarity (`order_by(..., asc=False)`):
```python theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
sim = imgs.img.similarity(image=sample_img)
res = (
imgs.order_by(sim, asc=False) # Order by descending similarity
.limit(2) # Limit number of results to 2
.select(imgs.id, imgs.img, sim)
.collect() # Retrieve results now
)
res
```
We can combine nearest-neighbor/similarity search with standard
predicates. Here’s the same query, but filtering out the selected
`sample_img` (which we already know has perfect similarity with itself):
```python theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
res = (
imgs.order_by(sim, asc=False)
.where(imgs.id != 6) # Additional clause
.limit(2)
.select(imgs.id, imgs.img, sim)
.collect()
)
res
```
## Index updates
In Pixeltable, each index is kept up-to-date automatically in response
to changes to the indexed table.
To illustrate this, let’s insert a few more rows:
```python theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
more_img_urls = [
'https://raw.githubusercontent.com/pixeltable/pixeltable/release/docs/resources/images/000000000080.jpg',
'https://raw.githubusercontent.com/pixeltable/pixeltable/release/docs/resources/images/000000000090.jpg',
'https://raw.githubusercontent.com/pixeltable/pixeltable/release/docs/resources/images/000000000106.jpg',
'https://raw.githubusercontent.com/pixeltable/pixeltable/release/docs/resources/images/000000000108.jpg',
'https://raw.githubusercontent.com/pixeltable/pixeltable/release/docs/resources/images/000000000139.jpg',
'https://raw.githubusercontent.com/pixeltable/pixeltable/release/docs/resources/images/000000000285.jpg',
'https://raw.githubusercontent.com/pixeltable/pixeltable/release/docs/resources/images/000000000632.jpg',
'https://raw.githubusercontent.com/pixeltable/pixeltable/release/docs/resources/images/000000000724.jpg',
'https://raw.githubusercontent.com/pixeltable/pixeltable/release/docs/resources/images/000000000776.jpg',
'https://raw.githubusercontent.com/pixeltable/pixeltable/release/docs/resources/images/000000000785.jpg',
]
imgs.insert(
{'id': 10 + i, 'img': url} for i, url in enumerate(more_img_urls)
)
```
Inserted 10 rows with 0 errors in 1.29 s (7.75 rows/s)
10 rows inserted.
When we now re-run the initial similarity query, we get a different
result:
```python theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
sim = imgs.img.similarity(image=sample_img)
res = (
imgs.order_by(sim, asc=False)
.limit(2)
.select(imgs.id, imgs.img, sim)
.collect()
)
res
```
## Similarity search on different types
Because CLIP models are multimodal, we can also do lookups by text.
```python theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
sim = imgs.img.similarity(string='train') # String lookup
res = (
imgs.order_by(sim, asc=False)
.limit(2)
.select(imgs.id, imgs.img, sim)
.collect()
)
res
```
## Creating multiple indexes on a single column
We can create multiple embedding indexes on the same column, utilizing
different embedding models. In order to use a specific index in a query,
we need to assign it a name and then use that name in the query.
To illustrate this, let’s create a table with text (taken from the
Wikipedia article on [Pablo
Picasso](https://en.wikipedia.org/wiki/Pablo_Picasso)):
```python theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
txts = pxt.create_table('indices_demo/text_tbl', {'text': pxt.String})
sentences = [
'Pablo Ruiz Picasso (25 October 1881 – 8 April 1973) was a Spanish painter, sculptor, printmaker, ceramicist, and theatre designer who spent most of his adult life in France.',
'One of the most influential artists of the 20th century, he is known for co-founding the Cubist movement, the invention of constructed sculpture,[8][9] the co-invention of collage, and for the wide variety of styles that he helped develop and explore.',
"Among his most famous works are the proto-Cubist Les Demoiselles d'Avignon (1907) and the anti-war painting Guernica (1937), a dramatic portrayal of the bombing of Guernica by German and Italian air forces during the Spanish Civil War.",
'Picasso demonstrated extraordinary artistic talent in his early years, painting in a naturalistic manner through his childhood and adolescence.',
'During the first decade of the 20th century, his style changed as he experimented with different theories, techniques, and ideas.',
'After 1906, the Fauvist work of the older artist Henri Matisse motivated Picasso to explore more radical styles, beginning a fruitful rivalry between the two artists, who subsequently were often paired by critics as the leaders of modern art.',
"Picasso's output, especially in his early career, is often periodized.",
'While the names of many of his later periods are debated, the most commonly accepted periods in his work are the Blue Period (1901–1904), the Rose Period (1904–1906), the African-influenced Period (1907–1909), Analytic Cubism (1909–1912), and Synthetic Cubism (1912–1919), also referred to as the Crystal period.',
"Much of Picasso's work of the late 1910s and early 1920s is in a neoclassical style, and his work in the mid-1920s often has characteristics of Surrealism.",
'His later work often combines elements of his earlier styles.',
]
txts.insert({'text': s} for s in sentences)
```
Inserted 10 rows with 0 errors in 0.03 s (301.25 rows/s)
10 rows inserted.
When calling
[`add_embedding_index()`](/sdk/latest/table#method-add_embedding_index),
we now specify the index name (`idx_name`) directly. If it is not
specified, Pixeltable will assign a name (such as `idx0`).
```python theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
from pixeltable.functions.huggingface import sentence_transformer
txts.add_embedding_index(
'text',
idx_name='minilm_idx',
embedding=sentence_transformer.using(
model_id='sentence-transformers/all-MiniLM-L12-v2'
),
)
txts.add_embedding_index(
'text',
idx_name='e5_idx',
embedding=sentence_transformer.using(model_id='intfloat/e5-large-v2'),
)
```
To do a similarity query, we now call `similarity()` with the `idx`
parameter:
```python theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
sim = txts.text.similarity('cubism', idx='minilm_idx')
res = (
txts.order_by(sim, asc=False)
.limit(2)
.select(txts.text, sim)
.collect()
)
res
```
## Using a UDF for a custom embedding
## Indexing precomputed embeddings (Array columns)
If your data already contains precomputed embedding vectors — for
example, embeddings exported from another system or generated by a
custom pipeline — you can index them directly without specifying an
embedding function.
`add_embedding_index` supports `Array` columns in addition to `String`,
`Image`, `Video`, and `Audio` columns. When the indexed column is an
`Array`, Pixeltable treats its values as precomputed embeddings and
builds the vector index directly from them.
An embedding function is **optional** for `Array` columns. If provided,
it will be used only to convert query values into embeddings during
similarity search.
```python theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
import numpy as np
import pixeltable as pxt
pxt.drop_table('indices_demo.precomputed', force=True)
# Table with a precomputed embedding column
precomputed = pxt.create_table(
'indices_demo.precomputed',
{'text': pxt.String, 'embedding': pxt.Array[(384,), pxt.Float]},
)
# Insert rows with precomputed vectors
rng = np.random.default_rng(42)
precomputed.insert(
[
{
'text': 'sample sentence',
'embedding': rng.random(384).tolist(),
},
{
'text': 'another sentence',
'embedding': rng.random(384).tolist(),
},
]
)
# Index the Array column directly — no embedding function needed
precomputed.add_embedding_index('embedding', metric='cosine')
```
To run a similarity search, pass a raw vector to the `similarity`
method:
```python theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
query_vector = rng.random(384)
sim = precomputed.embedding.similarity(vector=query_vector)
precomputed.order_by(sim, asc=False).limit(2).select(
precomputed.text, sim
).collect()
```
Error: similarity(vector=...): expected \`numpy.ndarray\`, or array \`Expr\`; got \`list\`
\[0;31m---------------------------------------------------------------------------\[0m
\[0;31mError\[0m Traceback (most recent call last)
Cell \[0;32mIn\[15], line 2\[0m
\[1;32m 1\[0m query\_vector \[38;5;241m=\[39m rng\[38;5;241m.\[39mrandom(\[38;5;241m384\[39m)\[38;5;241m.\[39mtolist()
\[0;32m----> 2\[0m sim \[38;5;241m=\[39m \[43mprecomputed\[49m\[38;5;241;43m.\[39;49m\[43membedding\[49m\[38;5;241;43m.\[39;49m\[43msimilarity\[49m\[43m(\[49m\[43mvector\[49m\[38;5;241;43m=\[39;49m\[43mquery\_vector\[49m\[43m)\[49m
\[1;32m 3\[0m precomputed\[38;5;241m.\[39morder\_by(sim, asc\[38;5;241m=\[39m\[38;5;28;01mFalse\[39;00m)\[38;5;241m.\[39mlimit(\[38;5;241m2\[39m)\[38;5;241m.\[39mselect(precomputed\[38;5;241m.\[39mtext, sim)\[38;5;241m.\[39mcollect()
File \[0;32m/Users/pierre/pixeltable/.venv/lib/python3.12/site-packages/pixeltable/exprs/column\_ref.py:313\[0m, in \[0;36mColumnRef.similarity\[0;34m(self, item, string, image, audio, video, document, vector, idx)\[0m
\[1;32m 311\[0m \[38;5;28;01melse\[39;00m:
\[1;32m 312\[0m \[38;5;28;01mif\[39;00m \[38;5;129;01mnot\[39;00m \[38;5;28misinstance\[39m(vector, np\[38;5;241m.\[39mndarray):
\[0;32m--> 313\[0m \[38;5;28;01mraise\[39;00m excs\[38;5;241m.\[39mError(
\[1;32m 314\[0m \[38;5;124mf\[39m\[38;5;124m'\[39m\[38;5;124msimilarity(vector=...): expected \`numpy.ndarray\`, or array \`Expr\`; \[39m\[38;5;124m'\[39m
\[1;32m 315\[0m \[38;5;124mf\[39m\[38;5;124m'\[39m\[38;5;124mgot \`\[39m\[38;5;132;01m\{\[39;00m\[38;5;28mtype\[39m(vector)\[38;5;241m.\[39m\[38;5;18m\_\_name\_\_\[39m\[38;5;132;01m}\[39;00m\[38;5;124m\`\[39m\[38;5;124m'\[39m
\[1;32m 316\[0m )
\[1;32m 317\[0m \[38;5;28;01mif\[39;00m vector\[38;5;241m.\[39mndim \[38;5;241m!=\[39m \[38;5;241m1\[39m:
\[1;32m 318\[0m \[38;5;28;01mraise\[39;00m excs\[38;5;241m.\[39mError(\[38;5;124mf\[39m\[38;5;124m'\[39m\[38;5;124msimilarity(vector=...): expected 1-dimensional array; got shape \[39m\[38;5;132;01m\{\[39;00mvector\[38;5;241m.\[39mshape\[38;5;132;01m}\[39;00m\[38;5;124m'\[39m)
\[0;31mError\[0m: similarity(vector=...): expected \`numpy.ndarray\`, or array \`Expr\`; got \`list\`
If you want to search by **text** (or another modality) rather than a
raw vector, provide an embedding function that converts the query value
into the same vector space. The function is only used at query time — it
does not re-embed the stored vectors.
```python theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
precomputed.add_embedding_index(
'embedding',
string_embed=my_text_embedding_fn, # converts query strings → vectors
metric='cosine',
)
sim = precomputed.embedding.similarity(string='search query')
```
The above examples show how to use any model in the Hugging Face `CLIP`
or `sentence_transformer` model families, and essentially the same
pattern can be used for any other embedding with built-in Pixeltable
support, such as OpenAI embeddings. But what if you want to adapt a new
model family that doesn’t have built-in support in Pixeltable? This can
be done by writing a custom Pixeltable UDF.
In the following example, we’ll write a simple UDF to use the
[BERT](https://www.kaggle.com/models/tensorflow/bert/tensorFlow2/en-uncased-preprocess/3)
model built on TensorFlow. First we install the necessary dependencies.
```python theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
%pip install -qU tensorflow tensorflow-hub tensorflow-text 'setuptools<82'
```
Text embedding UDFs must always take a string as input, and return a
1-dimensional numpy array of fixed dimension (512 in the case of
`small_bert`, the variant we’ll be using). If we were writing an image
embedding UDF, the `input` would have type `PIL.Image.Image` rather than
`str`. The UDF is straightforward, loading the model and evaluating it
against the input, with a minor data conversion on either side of the
model invocation.
```python theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
import pixeltable as pxt
import tensorflow as tf
import tensorflow_hub as hub
import tensorflow_text # Necessary to ensure BERT dependencies are loaded
@pxt.udf
def bert(input: str) -> pxt.Array[(512,), pxt.Float]:
"""Computes text embeddings using the small_bert model."""
preprocessor = hub.load(
'https://tfhub.dev/tensorflow/bert_en_uncased_preprocess/3'
)
bert_model = hub.load(
'https://tfhub.dev/tensorflow/small_bert/bert_en_uncased_L-4_H-512_A-8/2'
)
tensor = tf.constant([input]) # Convert the string to a tensor
result = bert_model(preprocessor(tensor))['pooled_output']
return result.numpy()[0, :]
```
```python theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
txts.add_embedding_index('text', idx_name='bert_idx', embedding=bert)
```
Here’s the output of our sample query run against `bert_idx`.
```python theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
sim = txts.text.similarity('cubism', idx='bert_idx')
res = (
txts.order_by(sim, asc=False)
.limit(2)
.select(txts.text, sim)
.collect()
)
res
```
Our example UDF is very simple, but it would perform poorly in a
production setting. To make our UDF production-ready, we’d want to do
two things:
* Cache the model: the current version calls `hub.load()` on every UDF
invocation. In a real application, we’d want to instantiate the model
just once, then reuse it on subsequent UDF calls.
* Batch our inputs: we’d use Pixeltable’s batching capability to ensure
we’re making efficient use of the model. Batched UDFs are described in
depth in the [User-Defined
Functions](/platform/udfs-in-pixeltable)
how-to guide.
You might have noticed that the updates to `bert_idx` seem sluggish;
that’s why!
## Deleting an index
To delete an index, call
[`Table.drop_embedding_index()`](/sdk/latest/table#method-drop_embedding_index):
* specify the `idx_name` parameter if you have multiple indices
* otherwise the `column_name` parameter is sufficient
Given that we have several embedding indices, we’ll specify which index
to drop:
```python theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
txts.drop_embedding_index(idx_name='e5_idx')
```
## Ship it as an app
This notebook used `add_embedding_index`. On a `TableModel`, indexes
live in `__indexes__`:
```python theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
import pixeltable as pxt
from pixeltable.functions.huggingface import clip
TableModel = pxt.model_base()
class ImgTbl(TableModel, name='img_tbl'):
img: pxt.Image
__indexes__ = [
pxt.EmbeddingIndex(img, embedding=clip.using(model_id='openai/clip-vit-base-patch32'), name='img_idx'),
]
```
```bash theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
pxt schema update app.py indices_demo
```
# External Files
Source: https://docs.pixeltable.com/platform/external-files
![Open in Kaggle]()
![Open in Colab]()
![Download Notebook]()
This documentation page is also available as an interactive notebook. You can launch the notebook in
Kaggle or Colab, or download it for use with an IDE or local Jupyter installation, by clicking one of the
above links.
In Pixeltable, all media data (videos, images, audio) resides in
external files, and Pixeltable stores references to those. The files can
be local or remote (e.g., in S3). For the latter, Pixeltable
automatically caches the files locally on access.
This notebook uses `pxt.create_table()` so you can run cells without a
project. An app puts the same columns on a `TableModel` in `app.py` and
creates the tables with `pxt schema update`.
When interacting with media data via Pixeltable, either through queries
or UDFs, the user sees the following Python types:
* `pxt.Image` (Python value: `PIL.Image.Image`)
* `pxt.Video` (Python value: a local path string)
* `pxt.Audio` (Python value: a local path string)
Let’s create a table and load some data to see what that looks like:
```python theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
%pip install -qU pixeltable
```
```python theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
import pixeltable as pxt
import random
import shutil
import tempfile
# First drop the `external_data` directory if it exists, to ensure
# a clean environment for the demo
pxt.drop_dir('external_data', force=True)
pxt.create_dir('external_data')
```
Connected to Pixeltable database at: postgresql+psycopg://postgres:@/pixeltable?host=/Users/asiegel/.pixeltable/pgdata
Created directory \`external\_data\`.
\
```python theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
v = pxt.create_table('external_data/videos', {'video': pxt.Video})
prefix = 's3://multimedia-commons/'
paths = [
'data/videos/mp4/ffe/ffb/ffeffbef41bbc269810b2a1a888de.mp4',
'data/videos/mp4/ffe/feb/ffefebb41485539f964760e6115fbc44.mp4',
'data/videos/mp4/ffe/f73/ffef7384d698b5f70d411c696247169.mp4',
]
v.insert({'video': prefix + p} for p in paths)
```
Created table \`videos\`.
Computing cells: 0%| | 0/6 \[00:00\
We just inserted 3 rows with video files residing in S3. When we now
query these, we are presented with their locally cached counterparts.
(Note: we don’t simply display the output of `collect()` here, because
that is formatted as an HTML table with a media player and so would
obscure the file path.)
```python theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
rows = list(v.select(v.video).collect())
rows[0]
```
\{'video': '/Users/asiegel/.pixeltable/file\_cache/682f022a704d4459adb2f29f7fe9577c\_0\_1fcfcb221263cff76a2853250fbbb2e90375dd495454c0007bc6ff4430c9a4a7.mp4'}
Let’s make a local copy of the first file and insert that separately.
First, the copy:
```python theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
local_path = tempfile.mktemp(suffix='.mp4')
shutil.copyfile(rows[0]['video'], local_path)
local_path
```
'/var/folders/hb/qd0dztsj43j\_mdb6hbl1gzyc0000gn/T/tmp1jo4a7ca.mp4'
Now the insert:
```python theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
v.insert([{'video': local_path}])
```
Computing cells: 0%| | 0/2 \[00:00\
When we query this again, we see that local paths are preserved:
```python theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
rows = list(v.select(v.video).collect())
rows
```
\[\{'video': '/Users/asiegel/.pixeltable/file\_cache/682f022a704d4459adb2f29f7fe9577c\_0\_1fcfcb221263cff76a2853250fbbb2e90375dd495454c0007bc6ff4430c9a4a7.mp4'},
\{'video': '/Users/asiegel/.pixeltable/file\_cache/682f022a704d4459adb2f29f7fe9577c\_0\_fc11428b32768ae782193a57ebcbad706f45bbd9fa13354471e0bcd798fee3ea.mp4'},
\{'video': '/Users/asiegel/.pixeltable/file\_cache/682f022a704d4459adb2f29f7fe9577c\_0\_b9fb0d9411bc9cd183a36866911baa7a8834f22f665bce47608566b38485c16a.mp4'},
\{'video': '/var/folders/hb/qd0dztsj43j\_mdb6hbl1gzyc0000gn/T/tmp1jo4a7ca.mp4'}]
UDFs also see local paths:
```python theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
@pxt.udf
def f(v: pxt.Video) -> int:
print(f'{type(v)}: {v}')
return 1
```
```python theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
v.select(f(v.video)).show()
```
\: /Users/asiegel/.pixeltable/file\_cache/682f022a704d4459adb2f29f7fe9577c\_0\_1fcfcb221263cff76a2853250fbbb2e90375dd495454c0007bc6ff4430c9a4a7.mp4
\: /Users/asiegel/.pixeltable/file\_cache/682f022a704d4459adb2f29f7fe9577c\_0\_fc11428b32768ae782193a57ebcbad706f45bbd9fa13354471e0bcd798fee3ea.mp4
\: /Users/asiegel/.pixeltable/file\_cache/682f022a704d4459adb2f29f7fe9577c\_0\_b9fb0d9411bc9cd183a36866911baa7a8834f22f665bce47608566b38485c16a.mp4
\: /var/folders/hb/qd0dztsj43j\_mdb6hbl1gzyc0000gn/T/tmp1jo4a7ca.mp4
## Dealing with errors
When interacting with media data in Pixeltable, the user can assume that
the underlying files exist, are local and are valid for their respective
data type. In other words, the user doesn’t need to consider error
conditions.
To that end, Pixeltable validates media data on ingest. The default
behavior is to reject invalid media files:
```python theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
v.insert([{'video': prefix + 'bad_path.mp4'}])
```
Computing cells: 0%| | 0/2 \[00:01\ 1\[0m \[43mv\[49m\[38;5;241;43m.\[39;49m\[43minsert\[49m\[43m(\[49m\[43mvideo\[49m\[38;5;241;43m=\[39;49m\[43mprefix\[49m\[43m \[49m\[38;5;241;43m+\[39;49m\[43m \[49m\[38;5;124;43m'\[39;49m\[38;5;124;43mbad\_path.mp4\[39;49m\[38;5;124;43m'\[39;49m\[43m)\[49m
File \[0;32m\~/Dropbox/workspace/pixeltable/pixeltable/pixeltable/catalog/insertable\_table.py:125\[0m, in \[0;36mInsertableTable.insert\[0;34m(self, rows, print\_stats, on\_error, \*\*kwargs)\[0m
\[1;32m 123\[0m \[38;5;28;01mraise\[39;00m excs\[38;5;241m.\[39mError(\[38;5;124m'\[39m\[38;5;124mrows must be a list of dictionaries\[39m\[38;5;124m'\[39m)
\[1;32m 124\[0m \[38;5;28mself\[39m\[38;5;241m.\[39m\_validate\_input\_rows(rows)
\[0;32m--> 125\[0m status \[38;5;241m=\[39m \[38;5;28;43mself\[39;49m\[38;5;241;43m.\[39;49m\[43m\_tbl\_version\[49m\[38;5;241;43m.\[39;49m\[43minsert\[49m\[43m(\[49m\[43mrows\[49m\[43m,\[49m\[43m \[49m\[38;5;28;43;01mNone\[39;49;00m\[43m,\[49m\[43m \[49m\[43mprint\_stats\[49m\[38;5;241;43m=\[39;49m\[43mprint\_stats\[49m\[43m,\[49m\[43m \[49m\[43mfail\_on\_exception\[49m\[38;5;241;43m=\[39;49m\[43mfail\_on\_exception\[49m\[43m)\[49m
\[1;32m 127\[0m \[38;5;28;01mif\[39;00m status\[38;5;241m.\[39mnum\_excs \[38;5;241m==\[39m \[38;5;241m0\[39m:
\[1;32m 128\[0m cols\_with\_excs\_str \[38;5;241m=\[39m \[38;5;124m'\[39m\[38;5;124m'\[39m
File \[0;32m\~/Dropbox/workspace/pixeltable/pixeltable/pixeltable/catalog/table\_version.py:723\[0m, in \[0;36mTableVersion.insert\[0;34m(self, rows, df, conn, print\_stats, fail\_on\_exception)\[0m
\[1;32m 721\[0m \[38;5;28;01mif\[39;00m conn \[38;5;129;01mis\[39;00m \[38;5;28;01mNone\[39;00m:
\[1;32m 722\[0m \[38;5;28;01mwith\[39;00m Env\[38;5;241m.\[39mget()\[38;5;241m.\[39mengine\[38;5;241m.\[39mbegin() \[38;5;28;01mas\[39;00m conn:
\[0;32m--> 723\[0m \[38;5;28;01mreturn\[39;00m \[38;5;28;43mself\[39;49m\[38;5;241;43m.\[39;49m\[43m\_insert\[49m\[43m(\[49m
\[1;32m 724\[0m \[43m \[49m\[43mplan\[49m\[43m,\[49m\[43m \[49m\[43mconn\[49m\[43m,\[49m\[43m \[49m\[43mtime\[49m\[38;5;241;43m.\[39;49m\[43mtime\[49m\[43m(\[49m\[43m)\[49m\[43m,\[49m\[43m \[49m\[43mprint\_stats\[49m\[38;5;241;43m=\[39;49m\[43mprint\_stats\[49m\[43m,\[49m\[43m \[49m\[43mrowids\[49m\[38;5;241;43m=\[39;49m\[43mrowids\[49m\[43m(\[49m\[43m)\[49m\[43m,\[49m\[43m \[49m\[43mabort\_on\_exc\[49m\[38;5;241;43m=\[39;49m\[43mfail\_on\_exception\[49m\[43m)\[49m
\[1;32m 725\[0m \[38;5;28;01melse\[39;00m:
\[1;32m 726\[0m \[38;5;28;01mreturn\[39;00m \[38;5;28mself\[39m\[38;5;241m.\[39m\_insert(
\[1;32m 727\[0m plan, conn, time\[38;5;241m.\[39mtime(), print\_stats\[38;5;241m=\[39mprint\_stats, rowids\[38;5;241m=\[39mrowids(), abort\_on\_exc\[38;5;241m=\[39mfail\_on\_exception)
File \[0;32m\~/Dropbox/workspace/pixeltable/pixeltable/pixeltable/catalog/table\_version.py:737\[0m, in \[0;36mTableVersion.\_insert\[0;34m(self, exec\_plan, conn, timestamp, rowids, print\_stats, abort\_on\_exc)\[0m
\[1;32m 735\[0m \[38;5;28mself\[39m\[38;5;241m.\[39mversion \[38;5;241m+\[39m\[38;5;241m=\[39m \[38;5;241m1\[39m
\[1;32m 736\[0m result \[38;5;241m=\[39m UpdateStatus()
\[0;32m--> 737\[0m num\_rows, num\_excs, cols\_with\_excs \[38;5;241m=\[39m \[38;5;28;43mself\[39;49m\[38;5;241;43m.\[39;49m\[43mstore\_tbl\[49m\[38;5;241;43m.\[39;49m\[43minsert\_rows\[49m\[43m(\[49m
\[1;32m 738\[0m \[43m \[49m\[43mexec\_plan\[49m\[43m,\[49m\[43m \[49m\[43mconn\[49m\[43m,\[49m\[43m \[49m\[43mv\_min\[49m\[38;5;241;43m=\[39;49m\[38;5;28;43mself\[39;49m\[38;5;241;43m.\[39;49m\[43mversion\[49m\[43m,\[49m\[43m \[49m\[43mrowids\[49m\[38;5;241;43m=\[39;49m\[43mrowids\[49m\[43m,\[49m\[43m \[49m\[43mabort\_on\_exc\[49m\[38;5;241;43m=\[39;49m\[43mabort\_on\_exc\[49m\[43m)\[49m
\[1;32m 739\[0m result\[38;5;241m.\[39mnum\_rows \[38;5;241m=\[39m num\_rows
\[1;32m 740\[0m result\[38;5;241m.\[39mnum\_excs \[38;5;241m=\[39m num\_excs
File \[0;32m\~/Dropbox/workspace/pixeltable/pixeltable/pixeltable/store.py:323\[0m, in \[0;36mStoreBase.insert\_rows\[0;34m(self, exec\_plan, conn, v\_min, show\_progress, rowids, abort\_on\_exc)\[0m
\[1;32m 321\[0m \[38;5;28;01mtry\[39;00m:
\[1;32m 322\[0m exec\_plan\[38;5;241m.\[39mopen()
\[0;32m--> 323\[0m \[38;5;28;01mfor\[39;00m row\_batch \[38;5;129;01min\[39;00m exec\_plan:
\[1;32m 324\[0m num\_rows \[38;5;241m+\[39m\[38;5;241m=\[39m \[38;5;28mlen\[39m(row\_batch)
\[1;32m 325\[0m \[38;5;28;01mfor\[39;00m batch\_start\_idx \[38;5;129;01min\[39;00m \[38;5;28mrange\[39m(\[38;5;241m0\[39m, \[38;5;28mlen\[39m(row\_batch), \[38;5;28mself\[39m\[38;5;241m.\[39m\_\_INSERT\_BATCH\_SIZE):
\[1;32m 326\[0m \[38;5;66;03m# compute batch of rows and convert them into table rows\[39;00m
File \[0;32m\~/Dropbox/workspace/pixeltable/pixeltable/pixeltable/exec/expr\_eval\_node.py:45\[0m, in \[0;36mExprEvalNode.\_\_next\_\_\[0;34m(self)\[0m
\[1;32m 44\[0m \[38;5;28;01mdef\[39;00m \[38;5;21m\_\_next\_\_\[39m(\[38;5;28mself\[39m) \[38;5;241m-\[39m\[38;5;241m>\[39m DataRowBatch:
\[0;32m---> 45\[0m input\_batch \[38;5;241m=\[39m \[38;5;28;43mnext\[39;49m\[43m(\[49m\[38;5;28;43mself\[39;49m\[38;5;241;43m.\[39;49m\[43minput\[49m\[43m)\[49m
\[1;32m 46\[0m \[38;5;66;03m# compute target exprs\[39;00m
\[1;32m 47\[0m \[38;5;28;01mfor\[39;00m cohort \[38;5;129;01min\[39;00m \[38;5;28mself\[39m\[38;5;241m.\[39mcohorts:
File \[0;32m\~/Dropbox/workspace/pixeltable/pixeltable/pixeltable/exec/cache\_prefetch\_node.py:71\[0m, in \[0;36mCachePrefetchNode.\_\_next\_\_\[0;34m(self)\[0m
\[1;32m 68\[0m futures\[executor\[38;5;241m.\[39msubmit(\[38;5;28mself\[39m\[38;5;241m.\[39m\_fetch\_url, row, info\[38;5;241m.\[39mslot\_idx)] \[38;5;241m=\[39m (row, info)
\[1;32m 69\[0m \[38;5;28;01mfor\[39;00m future \[38;5;129;01min\[39;00m concurrent\[38;5;241m.\[39mfutures\[38;5;241m.\[39mas\_completed(futures):
\[1;32m 70\[0m \[38;5;66;03m# TODO: does this need to deal with recoverable errors (such as retry after throttling)?\[39;00m
\[0;32m---> 71\[0m tmp\_path \[38;5;241m=\[39m \[43mfuture\[49m\[38;5;241;43m.\[39;49m\[43mresult\[49m\[43m(\[49m\[43m)\[49m
\[1;32m 72\[0m \[38;5;28;01mif\[39;00m tmp\_path \[38;5;129;01mis\[39;00m \[38;5;28;01mNone\[39;00m:
\[1;32m 73\[0m \[38;5;28;01mcontinue\[39;00m
File \[0;32m/opt/miniconda3/envs/pxt/lib/python3.9/concurrent/futures/\_base.py:439\[0m, in \[0;36mFuture.result\[0;34m(self, timeout)\[0m
\[1;32m 437\[0m \[38;5;28;01mraise\[39;00m CancelledError()
\[1;32m 438\[0m \[38;5;28;01melif\[39;00m \[38;5;28mself\[39m\[38;5;241m.\[39m\_state \[38;5;241m==\[39m FINISHED:
\[0;32m--> 439\[0m \[38;5;28;01mreturn\[39;00m \[38;5;28;43mself\[39;49m\[38;5;241;43m.\[39;49m\[43m\_\_get\_result\[49m\[43m(\[49m\[43m)\[49m
\[1;32m 441\[0m \[38;5;28mself\[39m\[38;5;241m.\[39m\_condition\[38;5;241m.\[39mwait(timeout)
\[1;32m 443\[0m \[38;5;28;01mif\[39;00m \[38;5;28mself\[39m\[38;5;241m.\[39m\_state \[38;5;129;01min\[39;00m \[CANCELLED, CANCELLED\_AND\_NOTIFIED]:
File \[0;32m/opt/miniconda3/envs/pxt/lib/python3.9/concurrent/futures/\_base.py:391\[0m, in \[0;36mFuture.\_\_get\_result\[0;34m(self)\[0m
\[1;32m 389\[0m \[38;5;28;01mif\[39;00m \[38;5;28mself\[39m\[38;5;241m.\[39m\_exception:
\[1;32m 390\[0m \[38;5;28;01mtry\[39;00m:
\[0;32m--> 391\[0m \[38;5;28;01mraise\[39;00m \[38;5;28mself\[39m\[38;5;241m.\[39m\_exception
\[1;32m 392\[0m \[38;5;28;01mfinally\[39;00m:
\[1;32m 393\[0m \[38;5;66;03m# Break a reference cycle with the exception in self.\_exception\[39;00m
\[1;32m 394\[0m \[38;5;28mself\[39m \[38;5;241m=\[39m \[38;5;28;01mNone\[39;00m
File \[0;32m/opt/miniconda3/envs/pxt/lib/python3.9/concurrent/futures/thread.py:58\[0m, in \[0;36m\_WorkItem.run\[0;34m(self)\[0m
\[1;32m 55\[0m \[38;5;28;01mreturn\[39;00m
\[1;32m 57\[0m \[38;5;28;01mtry\[39;00m:
\[0;32m---> 58\[0m result \[38;5;241m=\[39m \[38;5;28;43mself\[39;49m\[38;5;241;43m.\[39;49m\[43mfn\[49m\[43m(\[49m\[38;5;241;43m*\[39;49m\[38;5;28;43mself\[39;49m\[38;5;241;43m.\[39;49m\[43margs\[49m\[43m,\[49m\[43m \[49m\[38;5;241;43m*\[39;49m\[38;5;241;43m\*\[39;49m\[38;5;28;43mself\[39;49m\[38;5;241;43m.\[39;49m\[43mkwargs\[49m\[43m)\[49m
\[1;32m 59\[0m \[38;5;28;01mexcept\[39;00m \[38;5;167;01mBaseException\[39;00m \[38;5;28;01mas\[39;00m exc:
\[1;32m 60\[0m \[38;5;28mself\[39m\[38;5;241m.\[39mfuture\[38;5;241m.\[39mset\_exception(exc)
File \[0;32m\~/Dropbox/workspace/pixeltable/pixeltable/pixeltable/exec/cache\_prefetch\_node.py:115\[0m, in \[0;36mCachePrefetchNode.\_fetch\_url\[0;34m(self, row, slot\_idx)\[0m
\[1;32m 113\[0m \[38;5;28mself\[39m\[38;5;241m.\[39mrow\_builder\[38;5;241m.\[39mset\_exc(row, slot\_idx, exc)
\[1;32m 114\[0m \[38;5;28;01mif\[39;00m \[38;5;129;01mnot\[39;00m \[38;5;28mself\[39m\[38;5;241m.\[39mctx\[38;5;241m.\[39mignore\_errors:
\[0;32m--> 115\[0m \[38;5;28;01mraise\[39;00m exc \[38;5;28;01mfrom\[39;00m \[38;5;28;01mNone\[39;00m \[38;5;66;03m# suppress original exception\[39;00m
\[1;32m 116\[0m \[38;5;28;01mreturn\[39;00m \[38;5;28;01mNone\[39;00m
\[0;31mError\[0m: Failed to download s3://multimedia-commons/bad\_path.mp4: An error occurred (404) when calling the HeadObject operation: Not Found
The same happens for corrupted files:
```python theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
# create invalid .mp4
with tempfile.NamedTemporaryFile(
mode='wb', suffix='.mp4', delete=False
) as temp_file:
temp_file.write(random.randbytes(1024))
corrupted_path = temp_file.name
v.insert([{'video': corrupted_path}])
```
Computing cells: 100%|██████████████████████████████████████████| 2/2 \[00:00\<00:00, 1084.64 cells/s]
Error: Not a valid video: /var/folders/hb/qd0dztsj43j\_mdb6hbl1gzyc0000gn/T/tmp3djgfyjp.mp4
\[0;31m---------------------------------------------------------------------------\[0m
\[0;31mError\[0m Traceback (most recent call last)
Cell \[0;32mIn\[10], line 6\[0m
\[1;32m 3\[0m temp\_file\[38;5;241m.\[39mwrite(random\[38;5;241m.\[39mrandbytes(\[38;5;241m1024\[39m))
\[1;32m 4\[0m corrupted\_path \[38;5;241m=\[39m temp\_file\[38;5;241m.\[39mname
\[0;32m----> 6\[0m \[43mv\[49m\[38;5;241;43m.\[39;49m\[43minsert\[49m\[43m(\[49m\[43mvideo\[49m\[38;5;241;43m=\[39;49m\[43mcorrupted\_path\[49m\[43m)\[49m
File \[0;32m\~/Dropbox/workspace/pixeltable/pixeltable/pixeltable/catalog/insertable\_table.py:125\[0m, in \[0;36mInsertableTable.insert\[0;34m(self, rows, print\_stats, on\_error, \*\*kwargs)\[0m
\[1;32m 123\[0m \[38;5;28;01mraise\[39;00m excs\[38;5;241m.\[39mError(\[38;5;124m'\[39m\[38;5;124mrows must be a list of dictionaries\[39m\[38;5;124m'\[39m)
\[1;32m 124\[0m \[38;5;28mself\[39m\[38;5;241m.\[39m\_validate\_input\_rows(rows)
\[0;32m--> 125\[0m status \[38;5;241m=\[39m \[38;5;28;43mself\[39;49m\[38;5;241;43m.\[39;49m\[43m\_tbl\_version\[49m\[38;5;241;43m.\[39;49m\[43minsert\[49m\[43m(\[49m\[43mrows\[49m\[43m,\[49m\[43m \[49m\[38;5;28;43;01mNone\[39;49;00m\[43m,\[49m\[43m \[49m\[43mprint\_stats\[49m\[38;5;241;43m=\[39;49m\[43mprint\_stats\[49m\[43m,\[49m\[43m \[49m\[43mfail\_on\_exception\[49m\[38;5;241;43m=\[39;49m\[43mfail\_on\_exception\[49m\[43m)\[49m
\[1;32m 127\[0m \[38;5;28;01mif\[39;00m status\[38;5;241m.\[39mnum\_excs \[38;5;241m==\[39m \[38;5;241m0\[39m:
\[1;32m 128\[0m cols\_with\_excs\_str \[38;5;241m=\[39m \[38;5;124m'\[39m\[38;5;124m'\[39m
File \[0;32m\~/Dropbox/workspace/pixeltable/pixeltable/pixeltable/catalog/table\_version.py:723\[0m, in \[0;36mTableVersion.insert\[0;34m(self, rows, df, conn, print\_stats, fail\_on\_exception)\[0m
\[1;32m 721\[0m \[38;5;28;01mif\[39;00m conn \[38;5;129;01mis\[39;00m \[38;5;28;01mNone\[39;00m:
\[1;32m 722\[0m \[38;5;28;01mwith\[39;00m Env\[38;5;241m.\[39mget()\[38;5;241m.\[39mengine\[38;5;241m.\[39mbegin() \[38;5;28;01mas\[39;00m conn:
\[0;32m--> 723\[0m \[38;5;28;01mreturn\[39;00m \[38;5;28;43mself\[39;49m\[38;5;241;43m.\[39;49m\[43m\_insert\[49m\[43m(\[49m
\[1;32m 724\[0m \[43m \[49m\[43mplan\[49m\[43m,\[49m\[43m \[49m\[43mconn\[49m\[43m,\[49m\[43m \[49m\[43mtime\[49m\[38;5;241;43m.\[39;49m\[43mtime\[49m\[43m(\[49m\[43m)\[49m\[43m,\[49m\[43m \[49m\[43mprint\_stats\[49m\[38;5;241;43m=\[39;49m\[43mprint\_stats\[49m\[43m,\[49m\[43m \[49m\[43mrowids\[49m\[38;5;241;43m=\[39;49m\[43mrowids\[49m\[43m(\[49m\[43m)\[49m\[43m,\[49m\[43m \[49m\[43mabort\_on\_exc\[49m\[38;5;241;43m=\[39;49m\[43mfail\_on\_exception\[49m\[43m)\[49m
\[1;32m 725\[0m \[38;5;28;01melse\[39;00m:
\[1;32m 726\[0m \[38;5;28;01mreturn\[39;00m \[38;5;28mself\[39m\[38;5;241m.\[39m\_insert(
\[1;32m 727\[0m plan, conn, time\[38;5;241m.\[39mtime(), print\_stats\[38;5;241m=\[39mprint\_stats, rowids\[38;5;241m=\[39mrowids(), abort\_on\_exc\[38;5;241m=\[39mfail\_on\_exception)
File \[0;32m\~/Dropbox/workspace/pixeltable/pixeltable/pixeltable/catalog/table\_version.py:737\[0m, in \[0;36mTableVersion.\_insert\[0;34m(self, exec\_plan, conn, timestamp, rowids, print\_stats, abort\_on\_exc)\[0m
\[1;32m 735\[0m \[38;5;28mself\[39m\[38;5;241m.\[39mversion \[38;5;241m+\[39m\[38;5;241m=\[39m \[38;5;241m1\[39m
\[1;32m 736\[0m result \[38;5;241m=\[39m UpdateStatus()
\[0;32m--> 737\[0m num\_rows, num\_excs, cols\_with\_excs \[38;5;241m=\[39m \[38;5;28;43mself\[39;49m\[38;5;241;43m.\[39;49m\[43mstore\_tbl\[49m\[38;5;241;43m.\[39;49m\[43minsert\_rows\[49m\[43m(\[49m
\[1;32m 738\[0m \[43m \[49m\[43mexec\_plan\[49m\[43m,\[49m\[43m \[49m\[43mconn\[49m\[43m,\[49m\[43m \[49m\[43mv\_min\[49m\[38;5;241;43m=\[39;49m\[38;5;28;43mself\[39;49m\[38;5;241;43m.\[39;49m\[43mversion\[49m\[43m,\[49m\[43m \[49m\[43mrowids\[49m\[38;5;241;43m=\[39;49m\[43mrowids\[49m\[43m,\[49m\[43m \[49m\[43mabort\_on\_exc\[49m\[38;5;241;43m=\[39;49m\[43mabort\_on\_exc\[49m\[43m)\[49m
\[1;32m 739\[0m result\[38;5;241m.\[39mnum\_rows \[38;5;241m=\[39m num\_rows
\[1;32m 740\[0m result\[38;5;241m.\[39mnum\_excs \[38;5;241m=\[39m num\_excs
File \[0;32m\~/Dropbox/workspace/pixeltable/pixeltable/pixeltable/store.py:334\[0m, in \[0;36mStoreBase.insert\_rows\[0;34m(self, exec\_plan, conn, v\_min, show\_progress, rowids, abort\_on\_exc)\[0m
\[1;32m 332\[0m \[38;5;28;01mif\[39;00m abort\_on\_exc \[38;5;129;01mand\[39;00m row\[38;5;241m.\[39mhas\_exc():
\[1;32m 333\[0m exc \[38;5;241m=\[39m row\[38;5;241m.\[39mget\_first\_exc()
\[0;32m--> 334\[0m \[38;5;28;01mraise\[39;00m exc
\[1;32m 336\[0m rowid \[38;5;241m=\[39m (\[38;5;28mnext\[39m(rowids),) \[38;5;28;01mif\[39;00m rowids \[38;5;129;01mis\[39;00m \[38;5;129;01mnot\[39;00m \[38;5;28;01mNone\[39;00m \[38;5;28;01melse\[39;00m row\[38;5;241m.\[39mpk\[:\[38;5;241m-\[39m\[38;5;241m1\[39m]
\[1;32m 337\[0m pk \[38;5;241m=\[39m rowid \[38;5;241m+\[39m (v\_min,)
File \[0;32m\~/Dropbox/workspace/pixeltable/pixeltable/pixeltable/exprs/column\_ref.py:159\[0m, in \[0;36mColumnRef.eval\[0;34m(self, data\_row, row\_builder)\[0m
\[1;32m 156\[0m \[38;5;28;01mreturn\[39;00m
\[1;32m 158\[0m \[38;5;28;01mtry\[39;00m:
\[0;32m--> 159\[0m \[38;5;28;43mself\[39;49m\[38;5;241;43m.\[39;49m\[43mcol\[49m\[38;5;241;43m.\[39;49m\[43mcol\_type\[49m\[38;5;241;43m.\[39;49m\[43mvalidate\_media\[49m\[43m(\[49m\[43mdata\_row\[49m\[38;5;241;43m.\[39;49m\[43mfile\_paths\[49m\[43m\[\[49m\[43munvalidated\_slot\_idx\[49m\[43m]\[49m\[43m)\[49m
\[1;32m 160\[0m \[38;5;66;03m# access the value only after successful validation\[39;00m
\[1;32m 161\[0m val \[38;5;241m=\[39m data\_row\[unvalidated\_slot\_idx]
File \[0;32m\~/Dropbox/workspace/pixeltable/pixeltable/pixeltable/type\_system.py:906\[0m, in \[0;36mVideoType.validate\_media\[0;34m(self, val)\[0m
\[1;32m 904\[0m \[38;5;28;01mraise\[39;00m excs\[38;5;241m.\[39mError(\[38;5;124mf\[39m\[38;5;124m'\[39m\[38;5;124mNot a valid video: \[39m\[38;5;132;01m\{\[39;00mval\[38;5;132;01m}\[39;00m\[38;5;124m'\[39m)
\[1;32m 905\[0m \[38;5;28;01mexcept\[39;00m av\[38;5;241m.\[39mAVError:
\[0;32m--> 906\[0m \[38;5;28;01mraise\[39;00m excs\[38;5;241m.\[39mError(\[38;5;124mf\[39m\[38;5;124m'\[39m\[38;5;124mNot a valid video: \[39m\[38;5;132;01m\{\[39;00mval\[38;5;132;01m}\[39;00m\[38;5;124m'\[39m) \[38;5;28;01mfrom\[39;00m \[38;5;28;01mNone\[39;00m
\[0;31mError\[0m: Not a valid video: /var/folders/hb/qd0dztsj43j\_mdb6hbl1gzyc0000gn/T/tmp3djgfyjp.mp4
Alternatively, Pixeltable can also be instructed to record error
conditions and proceed with the ingest, via the `on_error` flag
(default: `'abort'`):
```python theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
v.insert(
[{'video': prefix + 'bad_path.mp4'}, {'video': corrupted_path}],
on_error='ignore',
)
```
Computing cells: 100%|████████████████████████████████████████████| 4/4 \[00:00\<00:00, 20.98 cells/s]
Inserting rows into \`videos\`: 2 rows \[00:00, 671.63 rows/s]
Computing cells: 100%|████████████████████████████████████████████| 4/4 \[00:00\<00:00, 20.13 cells/s]
Inserted 2 rows with 4 errors across 2 columns (videos.video, videos.None).
UpdateStatus(num\_rows=2, num\_computed\_values=4, num\_excs=4, updated\_cols=\[], cols\_with\_excs=\['videos.video', 'videos.None'])
Every media column has properties `errortype` and `errormsg` (both
containing `string` data) that indicate whether the column value is
valid. Invalid values show up as `None` and have non-null
`errortype`/`errormsg`:
```python theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
v.select(v.video == None, v.video.errortype, v.video.errormsg).collect()
```
Errors can now be inspected (and corrected) after the ingest:
```python theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
v.where(v.video.errortype != None).select(v.video.errormsg).collect()
```
## Accessing the original file paths
In some cases, it will be necessary to access file paths (not, say, the
`PIL.Image.Image`), and Pixeltable provides the column properties
`fileurl` and `localpath` for that purpose:
```python theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
v.select(v.video.fileurl, v.video.localpath).collect()
```
Note that for local media files, the `fileurl` property still returns a
parsable URL.
## Ship it as an app
Media columns on a `TableModel` are the same types as in this notebook:
```python theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
import pixeltable as pxt
TableModel = pxt.model_base()
class Videos(TableModel, name='videos'):
video: pxt.Video
```
```bash theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
pxt schema update app.py external_data
```
# Iterators
Source: https://docs.pixeltable.com/platform/iterators
Learn about iterators for processing documents, videos, audio, and images
## What are iterators?
Iterators in Pixeltable are specialized tools for processing and transforming media content. They efficiently break down large files into manageable chunks, enabling analysis at different granularities. Iterators work seamlessly with views to create virtual derived tables without duplicating storage.
In Pixeltable, iterators:
* Process media files incrementally to manage memory efficiently
* Transform single records into multiple output records
* Support various media types including documents, videos, images, and audio
* Integrate with the view system for automated processing pipelines
* Provide configurable parameters for fine-tuning output
Iterators are particularly useful when:
* Working with large media files that can't be processed at once
* Building retrieval systems that require chunked content
* Creating analysis pipelines for multimedia data
* Implementing feature extraction workflows
In an app file, set `iterator=` on the view's `TableModel`. In a notebook, pass `iterator=` to `pxt.create_view()`.
The examples below use `sentence` and `token_limit`. Install `spacy` and `tiktoken`, then download
spaCy's default English model:
```bash theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
pip install spacy tiktoken
python -m spacy download en_core_web_sm
```
```python theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
import pixeltable as pxt
import pixeltable.functions as pxtf
TableModel = pxt.model_base()
class Documents(TableModel, name='docs'):
document: pxt.Document
class Chunks(
TableModel,
name='chunks',
base=Documents,
iterator=pxtf.document.document_splitter(
Documents.document, separators='sentence,token_limit', limit=300
),
):
pass
```
In a notebook:
```python theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
import pixeltable as pxt
from pixeltable.functions.document import document_splitter
chunks = pxt.create_view(
'docs/chunks',
documents_table,
iterator=document_splitter(
document=documents_table.document,
separators='sentence,token_limit',
limit=300
)
)
```
## Core concepts
Split documents into chunks by headings, sentences, or token limits
Extract frames at specified intervals or counts
Divide images into overlapping or non-overlapping tiles
Split audio files into time-based chunks with configurable overlap
Iterators are powerful tools for processing large media files. They work seamlessly with Pixeltable's computed columns and versioning system.
## Available iterators
```python theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
from pixeltable.functions.document import document_splitter
# Create view with document chunks
chunks_view = pxt.create_view(
'docs/chunks',
docs_table,
iterator=document_splitter(
document=docs_table.document,
separators='sentence,token_limit',
limit=500,
metadata='title,heading'
)
)
```
### Parameters
* `separators`: Choose from 'heading', 'sentence', 'token\_limit', 'char\_limit', 'page'
* `limit`: Maximum tokens/characters per chunk
* `metadata`: Optional fields like 'title', 'heading', 'sourceline', 'page', 'bounding\_box'
* `overlap`: Optional overlap between chunks
```python theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
from pixeltable.functions.video import frame_iterator
# Extract frames at 1 FPS
frames_view = pxt.create_view(
'videos/frames',
videos_table,
iterator=frame_iterator(
video=videos_table.video,
fps=1.0
)
)
# Extract exact number of frames (evenly spaced)
frames_view = pxt.create_view(
'videos/sampled',
videos_table,
iterator=frame_iterator(
video=videos_table.video,
num_frames=10 # Extract 10 evenly-spaced frames
)
)
# Extract only keyframes (I-frames) for efficient processing
keyframes_view = pxt.create_view(
'videos/keyframes',
videos_table,
iterator=frame_iterator(
video=videos_table.video,
keyframes_only=True
)
)
```
### Parameters
* `fps`: Frames per second to extract (can be fractional)
* `num_frames`: Exact number of frames to extract
* `keyframes_only`: Extract only keyframes (I-frames) - efficient for quick video scanning
* Only one of `fps`, `num_frames`, or `keyframes_only` can be specified
```python theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
from pixeltable.functions.video import video_splitter
# Split video into 10-second segments
segments_view = pxt.create_view(
'videos/segments',
videos_table,
iterator=video_splitter(
video=videos_table.video,
duration=10.0,
min_segment_duration=1.0
)
)
```
### Parameters
* `duration`: Duration of each segment in seconds
* `overlap`: Overlap between segments in seconds
* `min_segment_duration`: Drop last segment if shorter than this value
### Returns
For each segment, yields:
* `segment_start`: Start time of the segment in seconds
* `segment_end`: End time of the segment in seconds
* `video_segment`: The video segment file
```python theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
from pixeltable.functions.string import string_splitter
# Split text into sentences
sentences_view = pxt.create_view(
'texts/sentences',
texts_table,
iterator=string_splitter(
text=texts_table.content,
separators='sentence'
)
)
```
### Parameters
* `separators`: Choose from 'sentence' (requires spacy)
### Returns
For each chunk, yields:
* `text`: The text chunk
```python theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
from pixeltable.functions.image import tile_iterator
# Create tiles with overlap
tiles_view = pxt.create_view(
'images/tiles',
images_table,
iterator=tile_iterator(
image=images_table.image,
tile_size=(224, 224), # Width, Height
overlap=(32, 32) # Horizontal, Vertical overlap
)
)
```
### Parameters
* `tile_size`: Tuple of (width, height) for each tile
* `overlap`: Optional tuple for overlap between tiles
```python theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
from pixeltable.functions.audio import audio_splitter
# Split by duration (exactly one of duration or max_size is required)
segments = pxt.create_view(
'audio/segments',
audio_table,
iterator=audio_splitter(
audio=audio_table.audio,
duration=30.0, # ~30-second segments
overlap=2.0, # 2-second overlap between segments
min_segment_duration=5.0, # Drop last segment if < 5 seconds
min_silence_len=0.3, # Snap cuts to quiet stretches
silence_thresh=-40.0,
trim_leading_silence=True,
),
)
# Or split by byte budget (e.g. API upload limits)
size_limited = pxt.create_view(
'audio/size_limited',
audio_table,
iterator=audio_splitter(
audio=audio_table.audio,
max_size=24 * 1024 * 1024, # at most 24 MB per segment
),
)
```
### Parameters
* `duration` (float | None): Segment length in seconds. Mutually exclusive with `max_size`.
* `max_size` (int | None): Maximum segment size in bytes. Mutually exclusive with `duration`.
* `overlap` (float, default: 0.0): Overlap between consecutive segments in seconds
* `min_segment_duration` (float, default: 0.0): Drop the last segment if shorter than this
* `min_silence_len` (float | None): If set, snap boundaries to quiet stretches of at least this length (seconds)
* `silence_thresh` (float, default: -40.0): dBFS level at or below which audio counts as silence
* `trim_leading_silence` (bool, default: False): Drop leading silence so each segment starts at audible content
### Returns
For each segment, yields:
* `segment_start`: Start time of the segment in seconds
* `segment_end`: End time of the segment in seconds
* `audio_segment`: The audio segment as `pxt.Audio`
### Notes
* Exactly one of `duration` or `max_size` must be specified
* With `max_size`, every emitted segment is guaranteed not to exceed the byte budget
* With `min_silence_len`, cuts land at the latest silence at or before the duration/`max_size` budget
* If the input contains no audio, no segments are yielded
* Supports various audio formats including MP3, AAC, Vorbis, Opus, FLAC
## Common use cases
Split documents for:
* RAG systems
* Text analysis
* Content extraction
Extract frames for:
* Object detection
* Scene classification
* Activity recognition
Create tiles for:
* High-resolution analysis
* Object detection
* Segmentation tasks
Split audio for:
* Speech recognition
* Sound classification
* Audio feature extraction
## Example workflows
```python theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
# Create document chunks
chunks = pxt.create_view(
'rag/chunks',
docs_table,
iterator=document_splitter(
document=docs_table.document,
separators='sentence,token_limit',
limit=500
)
)
# Add embeddings
chunks.add_embedding_index(
'text',
string_embed=sentence_transformer.using(
model_id='all-mpnet-base-v2'
)
)
```
```python theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
# Extract frames at 1 FPS
frames = pxt.create_view(
'detection/frames',
videos_table,
iterator=frame_iterator(
video=videos_table.video,
fps=1.0
)
)
# Add object detection
frames.add_computed_column(detections=detect_objects(frames.frame))
```
```python theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
# Split long audio files (silence-aware cuts avoid mid-word boundaries)
segments = pxt.create_view(
'audio/segments',
audio_table,
iterator=audio_splitter(
audio=audio_table.audio,
duration=30.0,
min_silence_len=0.3,
),
)
# Add transcription
segments.add_computed_column(text=whisper_transcribe(segments.audio_segment))
```
```python theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
from pixeltable.functions.video import make_video
# Extract frames at 1 FPS
frames = pxt.create_view(
'video/frames',
videos_table,
iterator=frame_iterator(
video=videos_table.video,
fps=1.0
)
)
# Process frames (e.g., spread pixels to soften the image)
frames.add_computed_column(processed=frames.frame.effect_spread(2))
# Create new videos from processed frames
processed_videos = frames.select(
frames.video_id,
make_video(frames.pos, frames.processed) # Default fps is 25
).group_by(frames.video_id).collect()
```
## Best practices
* Use appropriate chunk sizes
* Consider overlap requirements
* Monitor memory usage with large files
* Balance chunk size vs. processing time
* Use batch processing when possible
* Cache intermediate results
## Tips & tricks
When using `token_limit` with `document_splitter`, ensure the limit accounts for any model context windows in your pipeline.
## Custom iterators with `@pxt.iterator`
You can create your own iterators using the `@pxt.iterator` decorator on a Python generator function. This is the simplest way to define a custom iterator that splits one row into many.
```python theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
from typing import Iterator, TypedDict
import pixeltable as pxt
class WordRow(TypedDict):
word: str
position: int
@pxt.iterator
def word_iterator(text: str) -> Iterator[WordRow]:
for i, word in enumerate(text.split()):
yield WordRow(word=word, position=i)
# Use as a view iterator
words_view = pxt.create_view(
'text/words',
text_table,
iterator=word_iterator(text_table.content)
)
```
Use `unstored_cols` to mark columns that should not be persisted. An unstored column is recomputed
when a row is read, so the iterator must be able to jump straight to that row: `unstored_cols`
requires a `seek()` method, and only a class can supply one. A function-style `@pxt.iterator`
cannot use `unstored_cols`.
This skeleton shows the required methods, not a runnable frame extractor. Implement `__next__()`
to return a `FrameRow`, advance the position, and raise `StopIteration` at the end. After `seek(pos)`,
the next call to `__next__()` must return the row at `pos`. The cookbook below has a runnable example.
```python theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
from typing import TypedDict
import pixeltable as pxt
class FrameRow(TypedDict):
frame: pxt.Image
timestamp: float
@pxt.iterator(unstored_cols=['frame'])
class my_frame_extractor(pxt.PxtIterator[FrameRow]):
def __init__(self, video: pxt.Video, *, fps: float = 1.0):
self.video = video
self.fps = fps
self.pos = 0
def __next__(self) -> FrameRow:
# Custom frame extraction logic; raise StopIteration when done
...
# seek() receives the position of the row being read, plus the stored
# output columns of that row as keyword arguments.
def seek(self, pos: int, **kwargs) -> None:
self.pos = pos
```
Step-by-step guide to building custom iterators
## Additional resources
All built-in iterators
Chunk documents for RAG
Extract video frames
# Observability
Source: https://docs.pixeltable.com/platform/observability
![Open in Kaggle]()
![Open in Colab]()
![Download Notebook]()
This documentation page is also available as an interactive notebook. You can launch the notebook in
Kaggle or Colab, or download it for use with an IDE or local Jupyter installation, by clicking one of the
above links.
Pixeltable can export its internal telemetry as OpenTelemetry (OTel)
spans, so you can watch operations in any OTel backend. This notebook
wires the OTel bridge to **Grafana Cloud** and traces a single
`insert()` that exercises the full span set: source preparation, lock
acquisition, planning, external media downloads, UDF evaluation, media
persistence, the store write path, and view propagation.
The insert emits one nested span tree:
pixeltable.insert (operation span, root)
├─ pixeltable.data\_source.prepare (source resolution + schema validation)
├─ pixeltable.catalog.begin\_xact (connection + lock acquisition, one per attempt)
├─ pixeltable.plan.create (insert plan construction, DEBUG level)
├─ pixeltable.media.fetch (one per external file downloaded, DEBUG level)
├─ pixeltable.row (one per inserted row, DEBUG level)
│ └─ pixeltable.udf.\ (each UDF call, nested under its row)
├─ pixeltable.media.save (one per generated media file persisted, DEBUG level)
├─ pixeltable.store.build\_rows (row -> store-row conversion, DEBUG level)
├─ pixeltable.sa.insert\_rows (the SQL INSERT)
└─ pixeltable.view\_load (one per mutable view, containing its own row/store spans)
Traces land in Grafana Cloud Traces (Tempo).
```python theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
%pip install -qU 'pixeltable[otel]'
```
## Turn on the OpenTelemetry bridge
Telemetry is opt-in: call `pxt_otel.init()` once per process, before the
first table operation. It configures itself from the database’s
`[[pixeltable.database]]` entry in `pixeltable.toml`, from environment
variables, or from the `[otel]` section of `~/.pixeltable/config.toml`,
in that order of precedence.
**Option 1: the database entry in `pixeltable.toml`.** Put the endpoint
on the `[[pixeltable.database]]` entry of the database (the entry with
no `name` is the local database; see
[Configuration](../platform/configuration#per-database-settings)). The
headers are a secret and stay out of the file: keep
`OTEL_EXPORTER_OTLP_HEADERS` in the environment, and on a hosted
database set it with
`pxt secret set pxt://org:db OTEL_EXPORTER_OTLP_HEADERS=...`, then
`pxt db restart` so the running database sees it.
```toml theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
[[pixeltable.database]]
db_exporter_otlp_endpoint = 'https://otlp-gateway-.grafana.net/otlp'
```
**Option 2: environment variables.** Set these **before starting the
Jupyter kernel** (the kernel inherits them):
For Grafana Cloud, the OpenTelemetry connection page (Connections ->
Add new connection -> OpenTelemetry) shows your ``,
``, and lets you create a ``. Then
run these in the terminal, substituting your values, and start Jupyter
from that same terminal:
```bash theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
export OTEL_EXPORTER_OTLP_ENDPOINT='https://otlp-gateway-.grafana.net/otlp'
export OTEL_EXPORTER_OTLP_HEADERS="Authorization=Basic $(echo -n ':' | base64 | tr -d '\n')"
```
**Option 3: `~/.pixeltable/config.toml`**, shared by every database on
the installation:
```toml theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
[otel]
exporter_otlp_endpoint = 'https://otlp-gateway-.grafana.net/otlp'
```
`init()` runs **once per process**. If it ran without an endpoint
earlier in this kernel it can’t be reconfigured, so fix the
configuration, restart the kernel, and run this cell first.
```python theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
import opentelemetry.instrumentation.pixeltable as pxt_otel
from opentelemetry import trace
from opentelemetry.sdk.trace import TracerProvider
pxt_otel.init(
span_level='debug'
) # 'debug' emits per-row + per-UDF spans (default 'info' = operation span only)
# fail loud instead of silently dropping every span: a no-op ProxyTracerProvider means the SDK never came up
assert isinstance(trace.get_tracer_provider(), TracerProvider), (
'OpenTelemetry is INERT: no OTLP endpoint was configured. '
'Set the env vars (or config.toml) above, restart the kernel, and run this cell first.'
)
print('OTel bridge active')
```
OTel bridge active
## A pipeline that exercises every span type
One table with an image column and two computed columns, plus a view:
* inserting external image URLs triggers `pixeltable.media.fetch` (one
download span per file, on worker threads)
* the `add_one` UDF produces a `pixeltable.udf.add_one` span under each
`pixeltable.row`
* the stored `thumb` column generates thumbnails whose persistence
produces `pixeltable.media.save` spans
* the view is recomputed as part of the same insert, producing the
nested `pixeltable.view_load` subtree
`if_exists='replace_force'` also drops the dependent view on re-runs.
```python theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
import pixeltable as pxt
t = pxt.create_table(
'otel_demo1',
{'c': pxt.Int, 'img': pxt.Image},
if_exists='replace_force',
)
# Simple expression (span not reported since no UDF is invoked)
t.add_computed_column(inc=t.c + 1)
# Complex expression (span reported since UDF is invoked)
t.add_computed_column(thumb=t.img.resize((64, 64)))
v = pxt.create_view(
'otel_demo_view',
t,
additional_columns={'inc2': t.inc + 2},
if_exists='replace',
)
```
## Insert
This is the traced operation: 4 rows referencing external images (well
under the 100-span-per-operation cap, so every row gets a span). The
whole pipeline, downloads, UDFs, thumbnails, store writes, and the view
reload, lands under one `pixeltable.insert` trace.
Note: `pixeltable.media.fetch` spans only appear when the files are not
yet in the local file cache; on a re-run of this cell the downloads are
skipped.
```python theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
base = 'https://raw.githubusercontent.com/pixeltable/pixeltable/main/docs/resources/images'
images = [
'000000000001.jpg',
'000000000016.jpg',
'000000000019.jpg',
'000000000025.jpg',
]
status = t.insert(
[{'c': i, 'img': f'{base}/{name}'} for i, name in enumerate(images)]
)
status
```
Inserted 8 rows with 0 errors in 0.04 s (224.62 rows/s)
8 rows inserted.
## Wait a few seconds for export
`init()` sets up a `BatchSpanProcessor`, which ships queued spans
automatically about every 5 seconds while the kernel stays alive, so no
manual flush is needed. Give it a few seconds after the insert, then
check Grafana.
(In a short-lived script that exits immediately you would call
`trace.get_tracer_provider().force_flush()` before exit, but in a live
notebook kernel the timer handles it.)
## View it in Grafana
* **Traces**: Explore -> your Traces (Tempo) data source -> search
Service Name `pixeltable`, or span name `pixeltable.insert`. Expand
the trace to see the full hierarchy: `pixeltable.media.fetch`
downloads running in parallel, `pixeltable.row` ->
`pixeltable.udf.add_one`/`pixeltable.udf.resize` nesting,
`pixeltable.media.save` persistence, the
`pixeltable.store.build_rows`/`pixeltable.sa.insert_rows` write path,
and the `pixeltable.view_load` subtree with its own rows and store
writes.
For more detail use `span_level='trace'`; to reduce volume on a large
insert, use the default `'info'` (operation, `data_source.prepare`,
`begin_xact`, `sa.insert_rows`, and `view_load` spans only).
# Multimodal Type System
Source: https://docs.pixeltable.com/platform/type-system
Understanding Pixeltable types for structured data, media, and ML workflows
Pixeltable provides a rich type system designed for multimodal AI applications. Every column and expression has an associated type that determines what data it can hold and what operations are available.
This page leads with a `TableModel` in `app.py`. Create those tables with `pxt schema update`. Notebooks and tests can still call `pxt.create_table()`, `add_computed_column()`, and `add_embedding_index()`.
## Type overview
| Pixeltable Type | Python Type | Description |
| - | - | - |
| `pxt.String` | `str` | Text data |
| `pxt.Int` | `int` | Integer numbers |
| `pxt.Float` | `float` | Decimal numbers |
| `pxt.Bool` | `bool` | Boolean values |
| `pxt.Timestamp` | `datetime.datetime` | Timestamp values |
| `pxt.Date` | `datetime.date` | Date values |
| `pxt.UUID` | `uuid.UUID` | Unique identifiers |
| `pxt.Array` | `np.ndarray` | Numerical arrays (embeddings, tensors) |
| `pxt.Json` | `dict`, `list`, `str`, `int`, `float`, `bool` | Flexible JSON data |
| `pxt.Image` | `PIL.Image.Image` | Image data |
| `pxt.Video` | `str` (file path) | Video files |
| `pxt.Audio` | `str` (file path) | Audio files |
| `pxt.Document` | `str` (file path) | Documents (PDFs, markdown, html, etc.) |
A bare type is required. `title: pxt.String` rejects a missing value. Use `pxt.String | None` when the column may be empty. A column added to a table that already has rows must be nullable, so existing rows can stay empty.
Table, column, and database names match without regard to case.
`pxt.Audio`, `pxt.Video`, and `pxt.Document` return file paths when queried. Pixeltable automatically downloads and caches remote media locally. Use `.fileurl` to get the original URL.
## Basic types
```python theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
import pixeltable as pxt
TableModel = pxt.model_base()
class Example(TableModel, name='basic_types'):
text: pxt.String
count: pxt.Int
score: pxt.Float
active: pxt.Bool
created: pxt.Timestamp
```
In a notebook:
```python theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
table = pxt.create_table('example/basic_types', {
'text': pxt.String,
'count': pxt.Int,
'score': pxt.Float,
'active': pxt.Bool,
'created': pxt.Timestamp
})
```
### Auto-generated UUIDs
Use `uuid7()` to create columns that auto-generate unique identifiers:
```python theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
from pixeltable.functions.uuid import uuid7
# UUID as primary key - auto-generated for each row
products = pxt.create_table('example/products', {
'id': uuid7(), # Auto-generates UUID
'name': pxt.String,
'price': pxt.Float
}, primary_key=['id'])
# Insert without providing 'id' - it's generated automatically
products.insert([{'name': 'Laptop', 'price': 999.99}])
```
You can also add UUIDs to existing tables:
```python theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
# Add UUID column to existing table
orders.add_computed_column(order_id=uuid7())
```
By default, `stored=True` for all computed columns—values compute once and persist. For UUIDs, this ensures stable identifiers. Setting `stored=False` would regenerate UUIDs on every query (almost never what you want).
See the [UUID cookbook](/howto/cookbooks/core/workflow-uuid-identity) for more examples of working with unique identifiers.
## Media types
Pixeltable natively supports images, video, audio, and documents as first-class column types.
```python theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
import pixeltable as pxt
TableModel = pxt.model_base()
class Media(TableModel, name='media'):
image: pxt.Image
video: pxt.Video
audio: pxt.Audio
document: pxt.Document
```
In a notebook:
```python theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
media = pxt.create_table('example/media', {
'image': pxt.Image,
'video': pxt.Video,
'audio': pxt.Audio,
'document': pxt.Document
})
```
### Image specialization
Images can be constrained by resolution and/or color mode:
```python theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
# Constrain by resolution
thumbnails = pxt.create_table('example/thumbnails', {
'thumb': pxt.Image[(224, 224)] # Width 224, height 224
})
# Constrain by color mode
grayscale = pxt.create_table('example/grayscale', {
'img': pxt.Image['L'] # Grayscale (1-channel)
})
# Constrain both
rgb_fixed = pxt.create_table('example/rgb_fixed', {
'img': pxt.Image[(300, 200), 'RGB'] # 300x200 RGB images
})
```
See the [PIL Documentation](https://pillow.readthedocs.io/en/stable/handbook/concepts.html) for the full list of image modes (`'RGB'`, `'RGBA'`, `'L'`, etc.).
## Array types (embeddings & tensors)
Arrays are used for embeddings, feature vectors, and tensor data. They must always specify a shape and dtype.
```python theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
ml_data = pxt.create_table('example/ml_features', {
# Fixed-size embedding (e.g., from CLIP or OpenAI)
'embedding': pxt.Array[(768,), pxt.Float],
# Variable first dimension (batch of 512-dim vectors)
'features': pxt.Array[(None, 512), pxt.Float],
# 3D tensor with flexible dimensions
'tensor': pxt.Array[(None, None, 3), pxt.Float]
})
```
Array shapes follow NumPy conventions. Use `None` for unconstrained dimensions:
* `(512,)` — fixed 512-element vector
* `(None, 768)` — variable-length sequence of 768-dim vectors
* `(64, 64, 3)` — fixed 64×64×3 tensor
### Working with arrays
```python theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
# Arrays can be sliced like NumPy arrays
t.select(
t.embedding[0], # First element
t.embedding[5:10], # Slice
t.embedding[-3:] # Last 3 elements
).collect()
```
## JSON type
The `Json` type stores flexible structured data—dictionaries, lists, or primitives.
```python theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
logs = pxt.create_table('example/logs', {
'event': pxt.Json
})
logs.insert([
{'event': {'type': 'click', 'x': 100, 'y': 200}},
{'event': {'type': 'scroll', 'delta': 50}},
{'event': ['tag1', 'tag2', 'tag3']}
])
```
### JSON path access
Access nested data using dictionary or attribute syntax:
```python theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
# Dictionary syntax
t.select(t.event['type']).collect()
# Attribute syntax (JSONPath)
t.select(t.event.type).collect()
# List indexing
t.select(t.event.tags[0]).collect()
# Slicing
t.select(t.event.tags[:2]).collect()
```
Pixeltable handles missing keys gracefully—you'll get `None` instead of an exception.
### JSON schema validation
Validate JSON columns against a schema to ensure data integrity:
```python theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
# Define a JSON schema
movie_schema = {
'type': 'object',
'properties': {
'title': {'type': 'string'},
'year': {'type': 'integer'},
'rating': {'type': 'number'}
},
'required': ['title', 'year']
}
# Create table with validated JSON column
movies = pxt.create_table('example/validated_movies', {
'data': pxt.Json[movie_schema]
})
# Valid insert
movies.insert(data={'title': 'Inception', 'year': 2010, 'rating': 8.8})
# Invalid insert raises error (missing required 'year')
# movies.insert(data={'title': 'Movie'}) # Error!
```
### Using Pydantic models
```python theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
from pydantic import BaseModel
class Movie(BaseModel):
title: str
year: int
rating: float | None = None
# Use the model's JSON schema for validation
movies = pxt.create_table('example/pydantic_movies', {
'data': pxt.Json[Movie.model_json_schema()]
})
```
## Type conversion
Use `astype()` to convert string file paths or URLs to media types:
```python theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
# String file paths → Media types
media = pxt.create_table('media_table', {'path': pxt.String})
media.insert([{'path': '/path/to/image.jpg'}])
# Convert string path to Image
media.select(img=media.path.astype(pxt.Image)).collect()
```
**Primary use case:** Converting string columns containing file paths or URLs to media types (`Image`, `Video`, `Audio`, `Document`).
For other type conversions, use built-in functions from the [`string`](/sdk/latest/string), [`json`](/sdk/latest/json), or [`math`](/sdk/latest/math) modules. For example, use `string.len()` to get string length as an integer, or access JSON fields directly.
## Column properties
### Media column properties
Media columns (`Image`, `Video`, `Audio`, `Document`) have special properties:
```python theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
# Local file path (Pixeltable ensures this is on local filesystem)
t.select(t.image.localpath).collect()
# Original URL where the media resides
t.select(t.image.fileurl).collect()
```
### Error properties
Computed columns have `errortype` and `errormsg` properties for debugging:
```python theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
# Create a computed column that might fail
t.add_computed_column(
result=some_function(t.input),
on_error='ignore' # Continue on errors
)
# Query error information for failed rows
t.where(t.result == None).select(
t.input,
t.result.errortype, # Exception class name
t.result.errormsg # Error message
).collect()
```
## Best practices
Prefer `pxt.Image[(224,224), 'RGB']` over `pxt.Image` when you know the constraints. This enables optimizations and catches errors early.
Use JSON schema validation or Pydantic models for structured data to ensure consistency across your pipeline.
Always specify array shapes and dtypes. Use `None` for variable dimensions: `pxt.Array[(None, 768), pxt.Float]`.
Use `on_error='ignore'` in production pipelines, then query `.errortype` and `.errormsg` to debug failures.
## See also
Creating and managing tables
Transform data with computed columns
Complete type reference
# UDFs in Pixeltable
Source: https://docs.pixeltable.com/platform/udfs-in-pixeltable
![Open in Kaggle]()
![Open in Colab]()
![Download Notebook]()
This documentation page is also available as an interactive notebook. You can launch the notebook in
Kaggle or Colab, or download it for use with an IDE or local Jupyter installation, by clicking one of the
above links.
Pixeltable comes with a library of built-in functions and integrations,
but sooner or later, you’ll want to introduce some customized logic into
your workflow. This is where Pixeltable’s rich UDF (User-Defined
Function) capability comes in. Pixeltable UDFs let you write code in
Python, then directly insert your custom logic into Pixeltable
expressions and computed columns. In this how-to guide, we’ll show how
to define UDFs, extend their capabilities, and use them in computed
columns.
This notebook uses `pxt.create_table()` so you can run cells without a
project. An app puts the same columns on a `TableModel` in `app.py` and
creates the tables with `pxt schema update`.
To start, we’ll install the necessary dependencies, create a Pixeltable
directory and table to experiment with, and add some sample data.
```python theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
%pip install -qU pixeltable
```
```python theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
import pixeltable as pxt
# Create the directory and table
pxt.drop_dir('udf_demo', force=True) # Ensure a clean slate for the demo
pxt.create_dir('udf_demo')
t = pxt.create_table('udf_demo/strings', {'input': pxt.String})
# Add some sample data
t.insert(
[
{'input': 'Hello, world!'},
{'input': 'You can do a lot with Pixeltable UDFs.'},
]
)
t.show()
```
Connected to Pixeltable database at: postgresql+psycopg://postgres:@/pixeltable?host=/Users/asiegel/.pixeltable/pgdata
Created directory \`udf\_demo\`.
Created table \`strings\`.
Inserting rows into \`strings\`: 2 rows \[00:00, 763.99 rows/s]
Inserted 2 rows with 0 errors.
## What is a UDF?
A Pixeltable UDF is just a Python function that is marked with the
`@pxt.udf` decorator.
```python theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
@pxt.udf
def add_one(n: int) -> int:
return n + 1
```
It’s as simple as that! Without the decorator, `add_one` would be an
ordinary Python function that operates on integers. Adding `@pxt.udf`
converts it into a Pixeltable function that operates on *columns* of
integers. The decorated function can then be used directly to define
computed columns; Pixeltable will orchestrate its execution across all
the input data.
For our first working example, let’s do something slightly more
interesting: write a function to extract the longest word from a
sentence. (If there are ties for the longest word, we choose the first
word among those ties.) In Python, that might look something like this:
```python theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
import numpy as np
def longest_word(sentence: str, strip_punctuation: bool = False) -> str:
words = sentence.split()
if (
strip_punctuation
): # Remove non-alphanumeric characters from each word
words = [''.join(filter(str.isalnum, word)) for word in words]
i = np.argmax([len(word) for word in words])
return words[i]
```
```python theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
longest_word("Let's check that it works.", strip_punctuation=True)
```
'check'
The `longest_word` Python function isn’t a Pixeltable UDF (yet); it
operates on individual strings, not columns of strings. Adding the
decorator turns it into a UDF:
```python theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
@pxt.udf
def longest_word(sentence: str, strip_punctuation: bool = False) -> str:
words = sentence.split()
if (
strip_punctuation
): # Remove non-alphanumeric characters from each word
words = [''.join(filter(str.isalnum, word)) for word in words]
i = np.argmax([len(word) for word in words])
return words[i]
```
Now we can use it to create a computed column. Pixeltable orchestrates
the computation like it does with any other function, applying the UDF
in turn to each existing row of the table, then updating incrementally
each time a new row is added.
```python theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
t.add_computed_column(longest_word=longest_word(t.input))
t.show()
```
Computing cells: 100%|███████████████████████████████████████████| 2/2 \[00:00\<00:00, 138.28 cells/s]
Added 2 column values with 0 errors.
```python theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
t.insert([{'input': 'Pixeltable updates tables incrementally.'}])
t.show()
```
Computing cells: 0%| | 0/3 \[00:00\
Oops, those trailing punctuation marks are kind of annoying. Let’s add
another column, this time using the handy `strip_punctuation` parameter
from our UDF. (We could alternatively drop the first column before
adding the new one, but for purposes of this tutorial it’s convenient to
see how Pixeltable executes both variants side-by-side.) Note how
*columns* such as `t.input` and *constants* such as `True` can be freely
intermixed as arguments to the UDF.
```python theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
t.add_computed_column(
longest_word_2=longest_word(t.input, strip_punctuation=True)
)
t.show()
```
Computing cells: 100%|███████████████████████████████████████████| 3/3 \[00:00\<00:00, 252.91 cells/s]
Added 3 column values with 0 errors.
## Types in UDFs
You might have noticed that the `longest_word` UDF has *type hints* in
its signature.
```python theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
def longest_word(sentence: str, strip_punctuation: bool = False) -> str: ...
```
The `sentence` parameter, `strip_punctuation` parameter, and return
value all have explicit types (`str`, `bool`, and `str` respectively).
In general Python code, type hints are usually optional. But Pixeltable
is a database system: *everything* in Pixeltable must have a type. And
since Pixeltable is also an orchestrator - meaning it sets up workflows
and computed columns *before* executing them - these types need to be
known in advance. That’s the reasoning behind a fundamental principle of
Pixeltable UDFs:
* Type hints are *required*.
You can turn almost any Python function into a Pixeltable UDF, provided
that it has type hints, and provided that Pixeltable supports the types
that it uses. The most familiar types that you’ll use in UDFs are:
* `int`
* `float`
* `str`
* `list` (can optionally be parameterized, e.g., `list[str]`)
* `dict` (can optionally be parameterized, e.g., `dict[str, int]`)
* `PIL.Image.Image`
In addition to these standard Python types, Pixeltable also recognizes
various kinds of arrays, audio and video media, and documents.
## Local and module UDFs
The `longest_word` UDF that we defined above is a *local* UDF: it was
defined directly in our notebook, rather than in a module that we
imported. Many other UDFs, including all of Pixeltable’s built-in
functions, are defined in modules. We encountered a few of these in the
10-Minute Tour tutorial: the `huggingface.detr_for_object_detection` and
`openai.chat_completions` functions. (Although these are built-in
functions, they behave the same way as UDFs, and in fact they’re defined
the same way under the covers.)
There is an important difference between the two. When you add a module
UDF such as `openai.chat_completions` to a table, Pixeltable stores a
*reference* to the corresponding Python function in the module. If you
later restart your Python runtime and reload Pixeltable, then Pixeltable
will re-import the module UDF when it loads the computed column. This
means that any code changes made to the UDF will be picked up at that
time, and the new version of the UDF will be used in any future
execution.
Conversely, when you add a local UDF to a table, the *entire code* for
the UDF is serialized and stored in the table. This ensures that if you
restart your notebook kernel (say), or even delete the notebook
entirely, the UDF will continue to function. However, it also means that
if you modify the UDF code, the updated logic will not be reflected in
any existing Pixeltable columns.
To see how this works in practice, let’s modify our `longest_word` UDF
so that if `strip_punctuation` is `True`, then we remove only a single
punctuation mark from the *end* of each word.
```python theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
@pxt.udf
def longest_word(sentence: str, strip_punctuation: bool = False) -> str:
words = sentence.split()
if strip_punctuation:
words = [
word if word[-1].isalnum() else word[:-1] for word in words
]
i = np.argmax([len(word) for word in words])
return words[i]
```
Now we see that Pixeltable continues to use the *old* definition, even
as new rows are added to the table.
```python theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
t.insert([{'input': "Let's check that it still works."}])
t.show()
```
Computing cells: 0%| | 0/5 \[00:00\
But if we add a new *column* that references the `longest_word` UDF,
Pixeltable will use the updated version.
```python theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
t.add_computed_column(
longest_word_3=longest_word(t.input, strip_punctuation=True)
)
t.show()
```
Computing cells: 100%|███████████████████████████████████████████| 4/4 \[00:00\<00:00, 348.89 cells/s]
Added 4 column values with 0 errors.
The general rule is: changes to module UDFs will affect any future
execution; changes to local UDFs will only affect *new columns* that are
defined using the new version of the UDF.
## Batching
Pixeltable provides several ways to optimize UDFs for better
performance. One of the most common is *batching*, which is particularly
important for UDFs that involve GPU operations.
Ordinary UDFs process one row at a time, meaning the UDF will be invoked
exactly once per row processed. Conversely, a batched UDF processes
several rows at a time; the specific number is user-configurable. As an
example, let’s modify our `longest_word` UDF to take a batched
parameter. Here’s what it looks like:
```python theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
from pixeltable.func import Batch
@pxt.udf(batch_size=16)
def longest_word(
sentences: Batch[str], strip_punctuation: bool = False
) -> Batch[str]:
results = []
for sentence in sentences:
words = sentence.split()
if strip_punctuation:
words = [
word if word[-1].isalnum() else word[:-1]
for word in words
]
i = np.argmax([len(word) for word in words])
results.append(words[i])
return results
```
There are several changes:
* The parameter `batch_size=16` has been added to the `@pxt.udf`
decorator, specifying the batch size;
* The `sentences` parameter has changed from `str` to `Batch[str]`;
* The return type has also changed from `str` to `Batch[str]`; and
* Instead of processing a single sentence, the UDF is processing a
`Batch` of sentences and returning the result `Batch`.
What exactly is a `Batch[str]`? Functionally, it’s simply a `list[str]`,
and you can use it exactly like a `list[str]` in any Python code. The
only difference is in the type hint; a type hint of `Batch[str]` tells
Pixeltable, “My data consists of individual strings that I want you to
process in batches”. Conversely, a type hint of `list[str]` would mean,
“My data consists of *lists* of strings that I want you to process one
at a time”.
Notice that the `strip_punctuation` parameter is *not* wrapped in a
`Batch` type. This because `strip_punctuation` controls the behavior of
the UDF, rather than being part of the input data. When we use the
batched `longest_word` UDF, the `strip_punctuation` parameter will
always be a constant, not a column.
Let’s put the new, batched UDF to work.
```python theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
t.add_computed_column(
longest_word_3_batched=longest_word(t.input, strip_punctuation=True)
)
t.show()
```
Computing cells: 100%|███████████████████████████████████████████| 4/4 \[00:00\<00:00, 353.90 cells/s]
Added 4 column values with 0 errors.
As expected, the output of the `longest_word_3_batched` column is
identical to the `longest_word_3` column. Under the covers, though,
Pixeltable is orchestrating execution in batches of 16. That probably
won’t have much performance impact on our toy example, but for GPU-bound
computations such as text or image embeddings, it can make a substantial
difference.
## UDAs (aggregate UDFs)
Ordinary UDFs are always one-to-one on rows: each row of input generates
one UDF output value. Functions that aggregate data, conversely, are
many-to-one, and in Pixeltable they are represented by a related
abstraction, the UDA (User-Defined Aggregate).
Pixeltable has a number of built-in UDAs; if you’ve worked through the
Fundamentals tutorial, you’ll have already encountered a few of them,
such as `sum` and `count`. In this section, we’ll show how to define
your own custom UDAs. For demonstration purposes, let’s start by
creating a table containing all the integers from 0 to 49.
```python theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
import pixeltable as pxt
t = pxt.create_table('udf_demo/values', {'val': pxt.Int})
t.insert({'val': n} for n in range(50))
```
Created table \`values\`.
Inserting rows into \`values\`: 50 rows \[00:00, 9267.95 rows/s]
Inserted 50 rows with 0 errors.
UpdateStatus(num\_rows=50, num\_computed\_values=0, num\_excs=0, updated\_cols=\[], cols\_with\_excs=\[])
If we wanted to compute their sum using the built-in `sum` aggregate,
we’d do it like this:
```python theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
import pixeltable.functions as pxtf
t.select(pxtf.sum(t.val)).collect()
```
Or perhaps we want to group them by `n // 10` (corresponding to the tens
digit of each integer) and sum each group:
```python theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
t.group_by(t.val // 10).order_by(t.val // 10).select(
t.val // 10, pxtf.sum(t.val)
).collect()
```
Now let’s define a new aggregate to compute the sum of squares of a set
of numbers. To define an aggregate, we implement a subclass of the
`pxt.Aggregator` Python class and decorate it with the `@pxt.uda`
decorator, similar to what we did for UDFs. The subclass must implement
three methods:
* `__init__()` - initializes the aggregator; can be used to parameterize
aggregator behavior
* `update()` - updates the internal state of the aggregator with a new
value
* `value()` - retrieves the current value held by the aggregator
In our example, the class will have a single member `cur_sum`, which
holds a running total of the squares of all the values we’ve seen.
```python theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
@pxt.uda
class sum_of_squares(pxt.Aggregator):
def __init__(self):
# No data yet; initialize `cur_sum` to 0
self.cur_sum = 0
def update(self, val: int) -> None:
# Update the value of `cur_sum` with the new datapoint
self.cur_sum += val * val
def value(self) -> int:
# Retrieve the current value of `cur_sum`
return self.cur_sum
```
```python theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
t.select(sum_of_squares(t.val)).collect()
```
```python theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
t.group_by(t.val // 10).order_by(t.val // 10).select(
t.val // 10, sum_of_squares(t.val)
).collect()
```
## Ship it as an app
A UDF used as a column must be defined in this file (or another module
under the project root):
```python theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
import pixeltable as pxt
TableModel = pxt.model_base()
@pxt.udf
def reverse(s: str) -> str:
return s[::-1]
class Strings(TableModel, name='strings'):
input: pxt.String
reversed = reverse(input)
```
```bash theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
pxt schema update app.py udf_demo
```
# Version Control and Lineage
Source: https://docs.pixeltable.com/platform/version-control
Automatic versioning, time travel queries, and full data lineage tracking
Pixeltable automatically tracks every change to your tables—data insertions, updates, deletions, and schema modifications. Query any point in history, undo mistakes, and maintain full reproducibility without manual version management.
Put the table on a `TableModel` in `app.py` for an application. Inserts, updates, `history()`, and time travel below are runtime APIs and are the same in an app or a notebook.
## How it works
Every operation that modifies a table creates a new version:
```python theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
import pixeltable as pxt
TableModel = pxt.model_base()
class Products(TableModel, name='products'):
name: pxt.String
price: pxt.Float
```
```bash theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
pxt schema update app.py demo
```
```python theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
products = pxt.get_table('demo.products')
# Version 1: Data inserted
products.insert([
{'name': 'Widget', 'price': 9.99},
{'name': 'Gadget', 'price': 24.99}
])
# Version 2: Schema changed (notebook API; in app.py this is another computed column on the model)
products.add_computed_column(price_with_tax=products.price * 1.08)
# Version 3: Data updated
products.update({'price': 19.99}, where=products.name == 'Widget')
```
No configuration required—versioning is always on.
## Viewing history
### Human-readable history
```python theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
products.history()
```
Returns a DataFrame showing all versions with timestamps, change types, and row counts:
| version | created\_at | change\_type | inserts | updates | deletes | schema\_change |
| - | - | - | - | - | - | - |
| 3 | 2025-01-15 10:30:00 | data | 0 | 1 | 0 | None |
| 2 | 2025-01-15 10:29:00 | schema | 0 | 2 | 0 | Added: price\_with\_tax |
| 1 | 2025-01-15 10:28:00 | data | 2 | 0 | 0 | None |
| 0 | 2025-01-15 10:27:00 | schema | 0 | 0 | 0 | Initial Version |
### Programmatic access
```python theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
versions = products.get_versions() # List of dictionaries
latest = versions[0]
print(f"Version {latest['version']}: {latest['inserts']} inserts")
```
## Time travel queries
Query any historical version using the `table_name:version` syntax:
```python theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
# Get the table at version 1 (before computed column)
products_v1 = pxt.get_table('demo/products:1')
products_v1.collect() # Returns data as it was at version 1
# Compare with current state
products.collect() # Returns current data
```
Version handles are **read-only**—you cannot modify historical data.
### Use cases
* **Debugging**: Compare data before and after a problematic update
* **Auditing**: Track who changed what and when
* **Recovery**: Find and extract accidentally deleted or modified data
* **Reproducibility**: Query exact data used for a specific model training run
## Reverting changes
Undo the most recent change with `revert()`:
```python theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
# Oops, wrong update
products.update({'price': 0.00}, where=products.name == 'Widget')
# Undo it
products.revert() # Removes version N, table is now at version N-1
```
`revert()` permanently removes the latest version. This cannot be undone.
You can call `revert()` multiple times to go back further, but cannot revert past version 0 or past a version referenced by a snapshot.
## Snapshots
Create named, persistent point-in-time copies for long-term preservation:
```python theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
# Freeze current state before a major data update
baseline = pxt.create_snapshot('demo/products_baseline', products)
# Later: source table changes, but snapshot remains unchanged
products.insert([{'name': 'NewItem', 'price': 99.99}])
products.count() # 3 rows (updated)
baseline.count() # 2 rows (frozen)
```
**Snapshots vs Time Travel:**
* Time travel (`pxt.get_table('table:N')`) queries historical versions in place
* Snapshots create a named, independent copy that persists even if the source table is modified or deleted
## Data lineage
Pixeltable tracks the complete lineage of your data:
### Schema lineage
Every computed column records its dependencies:
```python theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
products.add_computed_column(
discounted=products.price * 0.9
)
products.add_computed_column(
discounted_with_tax=products.discounted * 1.08
)
# Pixeltable knows: discounted_with_tax → discounted → price
```
### View lineage
Views automatically track their source tables:
```python theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
expensive = pxt.create_view(
'demo/expensive_products',
products.where(products.price > 20)
)
# View lineage: expensive_products → products
```
### What's tracked
| Change Type | Tracked Information |
| - | - |
| `insert()` | Row count, timestamp, computed values generated |
| `update()` | Rows affected, old vs new values (via version comparison) |
| `delete()` | Row count removed |
| `add_column()` | Column name, type, dependencies |
| `add_computed_column()` | Column name, expression, dependencies |
| `drop_column()` | Column removed |
| `rename_column()` | Old name → new name |
## Best practices
Create snapshots before major data loads, model training runs, or production deployments.
Log table version numbers alongside model artifacts: `products.get_versions()[0]['version']`
Use `revert()` immediately after mistakes. For older issues, use time travel to identify the problem.
Use directories like `dev/products`, `staging/products` to isolate versioning across environments.
## Comparison with other systems
| Feature | Pixeltable | Git | Delta Lake |
| - | - | - | - |
| Automatic versioning | ✅ Every operation | Manual commits | ✅ Every operation |
| Time travel queries | ✅ `table:N` syntax | Checkout commit | ✅ `VERSION AS OF` |
| Schema versioning | ✅ Tracked | File-based | ✅ Schema evolution |
| Computed column lineage | ✅ Automatic | N/A | N/A |
| Revert | ✅ `revert()` | `git revert` | `RESTORE` |
| Named snapshots | ✅ `create_snapshot()` | Tags/branches | N/A |
## Next steps
Step-by-step cookbook with runnable examples
# Views
Source: https://docs.pixeltable.com/platform/views
Learn how to create and use virtual derived tables in Pixeltable through views
# When to Use Views
Views in Pixeltable are best used when you need to:
1. **Transform Data**: When you need to process or reshape data from a base table (e.g., splitting documents into chunks, extracting features from images)
2. **Filter Data**: When you frequently need to work with a specific subset of your data
3. **Create Virtual Tables**: When you want to avoid storing redundant data and automatically keep derived data in sync
4. **Build Data Workflows**: When you need to chain multiple data transformations together
5. **Save Storage**: When you want to compute data on demand rather than storing it permanently
Choose views over tables when your data is derived from other base tables and needs to stay synchronized with its source. Use regular tables when you need to store original data or when the computation cost of deriving data on demand is too high.
Production apps declare views as `TableModel` classes in a Python file and create them with `pxt schema update`. Notebooks and tests can still call `pxt.create_table()` / `pxt.create_view()`.
The `token_limit` separator below needs `tiktoken`, which the base install does not include:
`pip install tiktoken`.
## Phase 1: Define your base table and view structure
```python theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
import pixeltable as pxt
import pixeltable.functions as pxtf
TableModel = pxt.model_base()
class Documents(TableModel, name='collection'):
document: pxt.Document
class Chunks(
TableModel,
name='chunks',
base=Documents,
iterator=pxtf.document.document_splitter(Documents.document, separators='token_limit', limit=300),
):
pass
```
```bash theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
pxt schema update app.py documents
```
In a notebook:
```python theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
import pixeltable as pxt
from pixeltable.functions.document import document_splitter
pxt.drop_dir('documents', force=True)
pxt.create_dir('documents')
documents = pxt.create_table(
"documents/collection",
{"document": pxt.Document}
)
chunks = pxt.create_view(
'documents/chunks',
documents,
iterator=document_splitter(
document=documents.document,
separators='token_limit',
limit=300
)
)
```
## Phase 2: Use your application
```python theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
import pixeltable as pxt
# Connect to your base table and view
documents = pxt.get_table("documents/collection")
chunks = pxt.get_table("documents/chunks")
# Insert data into base table - view updates automatically
documents.insert([{
"document": "path/to/document.pdf"
}])
# Query the view
print(chunks.collect())
```
## View types
Views created using iterators to transform data:
```python theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
# Document splitting view
chunks = pxt.create_view(
'docs/chunks',
documents,
iterator=document_splitter(
document=documents.document
)
)
```
Views created from query operations:
```python theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
# Filtered view of high-budget movies
blockbusters = pxt.create_view(
'movies/blockbusters',
movies.where(movies.budget >= 100.0)
)
```
## View operations
Query views like regular tables:
```python theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
# Basic filtering on view
chunks.where(chunks.text.contains('specific topic')).collect()
# Select specific columns
chunks.select(chunks.text, chunks.pos).collect()
# Order results
chunks.order_by(chunks.pos).limit(5).collect()
```
Add computed columns to views:
```python theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
# Add embeddings to chunks
chunks.add_computed_column(
embedding=sentence_transformer.using(
model_id='intfloat/e5-large-v2'
)(chunks.text)
)
```
Create views based on other views:
```python theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
# Create a view of embedded chunks
embedded_chunks = pxt.create_view(
'docs/embedded_chunks',
chunks.where(chunks.text.len() > 100)
)
```
## Key features
Views automatically update when base tables change
Views compute data on demand, saving storage
Views can be part of larger data workflows
# AI Skills
Source: https://docs.pixeltable.com/resources/ai-skills
Install the Pixeltable skill. The coding agent writes TableModel classes in app.py.
One skill file teaches the coding agent to declare tables as `TableModel` classes in `app.py` and create them with `pxt schema update`.
The skill's workflow is: create tables from the file, start the endpoints with `pxt service update`, then try the app by inserting rows or opening the dashboard.
Install details: [AI coding agents](/overview/building-pixeltable-with-llms).
The skill is [pixeltable/pixeltable-skill](https://github.com/pixeltable/pixeltable-skill), and [docs.pixeltable.com/skill.md](https://docs.pixeltable.com/skill.md) points there. Install it:
```bash theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
npx skills add pixeltable/pixeltable-skill
```
If the agent writes `create_table` in app code, names the file `schema.py`, or writes the removed command `pxt serve`, the installed skill is stale: this page wins. Reinstall: `npx skills add pixeltable/pixeltable-skill`.
## What it teaches
| Do | Do not |
| - | - |
| `class Docs(TableModel, name='docs')` in `app.py` | LangChain / LlamaIndex for chunking and retrieval |
| Assignments as computed columns | pandas as the database that holds your app data |
| `__indexes__` on the model | A separate vector database |
| `pxt init`, then `pxt schema update`, then `pxt service update` | Per-row `for` loops that call a model |
| Use the same `app.py` with a `pxt://` Cloud URI | Call `add_embedding_index()` in application code |
| `from pixeltable.serving import FastAPIRouter` | Declare HTTP routes in TOML |
Scaffold: `uvx pixeltable-new myapp`, then `pxt schema update app.py agent`.
[Starter kit](/resources/starter-kit).
## How to invoke it
Compatible agents pick the skill up when the task is a Pixeltable app. To call it by name:
| Tool | Invoke |
| - | - |
| Cursor | `/pixeltable` |
| Claude Code | `/pixeltable` |
| GitHub Copilot | `/pixeltable` |
| Codex | `$pixeltable` |
Write `app.py`, then `pxt schema update` and `pxt service update`.
Install details and which file the agent should write.
# Starter kit
Source: https://docs.pixeltable.com/resources/starter-kit
uvx pixeltable-new copies a starter app. pxt schema update creates its tables. pxt service update starts the endpoints.
The [Pixeltable Starter Kit](https://github.com/pixeltable/pixeltable-starter-kit) is two apps.
`uvx pixeltable-new` copies one into a new directory.
First create the tables (`pxt schema update`), then start the endpoints (`pxt service update`), then try the app by inserting rows, calling `/ask`, or opening the dashboard.
A first run: [Quickstart](/overview/quick-start).
```bash theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
uvx pixeltable-new myapp
cd myapp
uv sync
pxt schema update app.py agent
pxt service update app.py agent
```
`agent` is the catalog Pixeltable creates, not a folder on disk. For the video app, run `uvx pixeltable-new myapp --video` and use `videointel` as that catalog.
The scaffold includes `pixeltable.toml`. If you copied files by hand, run `pxt init` first.
Routes live on a `FastAPIRouter` in the same file as the `TableModel` classes.
If you already have a FastAPI app, mount the kit's router with `app.include_router(...)`. [HTTP serving](/howto/deployment/serving).
To skip endpoints, see [Self-hosting](/howto/deployment/overview) for the insert-and-export path.
## Apps
The default app is the chat agent. You can insert documents without an API key; `/ask` needs `ANTHROPIC_API_KEY`.
```bash theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
uvx pixeltable-new myapp
pxt schema update app.py agent
```
Extracts video frames at 1 FPS, indexes them with CLIP, and accepts image uploads.
```bash theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
uvx pixeltable-new myapp --video
pxt schema update app.py videointel
```
To run the same starter app on Pixeltable Cloud, sign in with `pxt login`. `pxt org create` provisions `main`. Add this entry to the project file before `pxt db update`. Without it, the command errors. `pxt db update` uploads the project onto that database:
```toml theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
[[pixeltable.database]]
name = 'pxt://org:main'
```
```bash theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
pxt db update pxt://org:main
pxt schema update app.py pxt://org:main
pxt service update app.py pxt://org:main
```
[Pixeltable Cloud](/cloud). `pxt service run` starts a service in your terminal and cannot target Cloud; use `pxt service update` for hosted endpoints.
Teach an agent to write `TableModel` in `app.py`.
`chat-agent/` and `video-search/` in one repo.
# Support
Source: https://docs.pixeltable.com/resources/support
Get help from the Pixeltable community
Join the Pixeltable community for questions, feedback, and support.
Chat with the community and Pixeltable team
Ask questions and share ideas
Report bugs and request features
Schedule time with the team
# anthropic
Source: https://docs.pixeltable.com/sdk/latest/anthropic
# module pixeltable.functions.anthropic
Pixeltable UDFs
that wrap various endpoints from the Anthropic API. In order to use them, you must
first `pip install anthropic` and configure your Anthropic credentials, as described in
the [Working with Anthropic](https://docs.pixeltable.com/howto/providers/working-with-anthropic) tutorial.
## func invoke\_tools()
```python Signature theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
invoke_tools(
tools: pixeltable.func.tools.Tools,
response: pixeltable.exprs.expr.Expr
) -> pixeltable.exprs.inline_expr.InlineDict
```
Converts an Anthropic response dict to Pixeltable tool invocation format and calls `tools._invoke()`.
## udf messages()
```python Signature theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
@pxt.udf
messages(
messages: pxt.Json[(Json, ...)],
*,
model: pxt.String,
max_tokens: pxt.Int,
model_kwargs: pxt.Json | None = None,
tools: pxt.Json[(Json, ...)] | None = None,
tool_choice: pxt.Json | None = None
) -> pxt.Json
```
Create a Message.
Equivalent to the Anthropic `messages` API endpoint.
For additional details, see: [https://docs.anthropic.com/en/api/messages](https://docs.anthropic.com/en/api/messages)
Request throttling:
Uses the rate limit-related headers returned by the API to throttle requests adaptively, based on available
request and token capacity. No configuration is necessary.
**Requirements:**
* `pip install anthropic`
**Parameters:**
* **`messages`** (`pxt.Json[(Json`): Input messages.
* **`model`** (`Any`): The model that will complete your prompt.
* **`model_kwargs`** (`Any`): Additional keyword args for the Anthropic `messages` API.
For details on the available parameters, see: [https://docs.anthropic.com/en/api/messages](https://docs.anthropic.com/en/api/messages)
* **`tools`** (`Any`): An optional list of Pixeltable tools to use for the request.
* **`tool_choice`** (`Any`): An optional tool choice configuration.
**Returns:**
* `pxt.Json`: A dictionary containing the response and other metadata.
**Examples:**
Add a computed column that applies the model `claude-3-5-sonnet-20241022`
to an existing Pixeltable column `tbl.prompt` of the table `tbl`:
```python theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
msgs = [{'role': 'user', 'content': tbl.prompt}]
tbl.add_computed_column(
response=messages(msgs, model='claude-3-5-sonnet-20241022')
)
```
# array
Source: https://docs.pixeltable.com/sdk/latest/array
# module pixeltable.functions.array
Pixeltable UDFs for `ArrayType`.
Example:
```python theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
import pixeltable as pxt
t = pxt.get_table(...)
t.select(t.array_col.to_list()).collect()
```
## udf to\_list()
```python Signature theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
@pxt.udf
to_list(self: pxt.Array) -> pxt.Json
```
Convert an array to a nested Python list.
Equivalent to numpy's [`ndarray.tolist()`](https://numpy.org/doc/stable/reference/generated/numpy.ndarray.tolist.html).
Useful for exporting to systems that lack a fixed-shape tensor type (e.g. Iceberg).
**Examples:**
```python theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
t.select(t.array_col.to_list()).collect()
```
# audio
Source: https://docs.pixeltable.com/sdk/latest/audio
# module pixeltable.functions.audio
Pixeltable UDFs for `AudioType`.
## iterator audio\_splitter()
```python Signature theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
@pxt.iterator
audio_splitter(
audio: pxt.Audio,
*,
duration: pxt.Float | None = None,
max_size: pxt.Int | None = None,
overlap: pxt.Float = 0.0,
min_segment_duration: pxt.Float = 0.0,
min_silence_len: pxt.Float | None = None,
silence_thresh: pxt.Float = -40.0,
trim_leading_silence: pxt.Bool = False
)
```
Iterator over segments of an audio file. The audio file is split into smaller segments, sized either by
`duration` (seconds) or by `max_size` (bytes). Exactly one of the two must be specified.
* With `max_size`, every emitted segment is guaranteed not to exceed the given number of bytes.
* With `duration`, each segment is approximately that long (a segment may run over by up to one packet).
**Outputs**:
One row per audio segment, with the following columns:
* `segment_start` (`pxt.Float`): Start time of the audio segment in seconds
* `segment_end` (`pxt.Float`): End time of the audio segment in seconds
* `audio_segment` (`pxt.Audio | None`): The audio content of the segment
When `min_silence_len` is set, segment boundaries are snapped to silences so that segments do not end in the
middle of speech: each segment ends at the latest silence at or before its `duration` or `max_size` budget.
`trim_leading_silence` additionally drops silence from the start of every segment.
**Parameters:**
* **`duration`** (`pxt.Float | None`): Audio segment duration in seconds. Mutually exclusive with `max_size`.
* **`max_size`** (`pxt.Int | None`): Maximum audio segment size in bytes. Mutually exclusive with `duration`.
* **`overlap`** (`pxt.Float`): Overlap between consecutive segments in seconds
* **`min_segment_duration`** (`pxt.Float`): Drop the last segment if it is smaller than `min_segment_duration` (in seconds).
* **`min_silence_len`** (`pxt.Float | None`): If set, enables silence-aware boundaries; the minimum length in seconds of a quiet stretch
that counts as a usable cut point.
* **`silence_thresh`** (`pxt.Float`): Level in dBFS at or below which audio is considered silent. Used when `min_silence_len` is set
or `trim_leading_silence` is true.
* **`trim_leading_silence`** (`pxt.Bool`): If true, drop leading silence from each segment so it starts at audible content.
**Examples:**
This example assumes an existing table `tbl` with a column `audio` of type `pxt.Audio`.
Create a view that splits all audio files into segments of 30 seconds with 5 seconds overlap:
```python theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
pxt.create_view(
'audio_segments',
tbl,
iterator=audio_splitter(tbl.audio, duration=30.0, overlap=5.0),
)
```
Create a view that splits all audio files into segments of at most 24 MB, for transcription:
```python theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
pxt.create_view(
'audio_segments',
tbl,
iterator=audio_splitter(tbl.audio, max_size=24 * 1024 * 1024),
)
```
## udf encode\_audio()
```python Signature theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
@pxt.udf
encode_audio(
audio_data: pxt.Array[float32],
*,
input_sample_rate: pxt.Int,
format: pxt.String,
output_sample_rate: pxt.Int | None = None
) -> pxt.Audio
```
Encodes an audio clip represented as an array into a specified audio format.
**Parameters:**
* **`audio_data`** (`pxt.Array[float32]`): An array of sampled amplitudes. The accepted array shapes are `(N,)` or `(1, N)` for mono audio
or `(2, N)` for stereo.
* **`input_sample_rate`** (`pxt.Int`): The sample rate of the input audio data.
* **`format`** (`pxt.String`): The desired output audio format. The supported formats are 'wav', 'mp3', 'flac', and 'mp4'.
* **`output_sample_rate`** (`pxt.Int | None`): The desired sample rate for the output audio. Defaults to the input sample rate if
unspecified.
**Examples:**
Add a computed column with encoded FLAC audio files to a table with audio data (as arrays of floats) and sample
rates:
```python theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
t.add_computed_column(
audio_file=encode_audio(
t.audio_data, input_sample_rate=t.sample_rate, format='flac'
)
)
```
## udf get\_metadata()
```python Signature theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
@pxt.udf
get_metadata(audio: pxt.Audio) -> ContainerMetadata
```
Gets various metadata associated with an audio file and returns it as
a [`ContainerMetadata`](./containermetadata) dictionary.
**Parameters:**
* **`audio`** (`pxt.Audio`): The audio to get metadata for.
**Returns:**
* `ContainerMetadata`: A [`ContainerMetadata`](./containermetadata) with typical structure:
```json theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
{
'size': 2568827,
'streams': [
{
'type': 'audio',
'frames': 0,
'duration': 2646000,
'metadata': {},
'time_base': 2.2675736961451248e-05,
'codec_context': {
'name': 'flac',
'profile': None,
'channels': 1,
'codec_tag': '\x00\x00\x00\x00',
},
'duration_seconds': 60.0,
}
],
'bit_rate': 342510,
'metadata': {'encoder': 'Lavf61.1.100'},
'bit_exact': False,
}
```
**Examples:**
Extract metadata for files in the `audio_col` column of the table `tbl`:
```python theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
tbl.select(tbl.audio_col.get_metadata()).collect()
```
# bedrock
Source: https://docs.pixeltable.com/sdk/latest/bedrock
# module pixeltable.functions.bedrock
Pixeltable UDFs for AWS Bedrock AI models.
In order to use them, you must
first configure your AWS credentials, as described in
the [Working with Bedrock](https://docs.pixeltable.com/howto/providers/working-with-bedrock) tutorial.
## func invoke\_tools()
```python Signature theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
invoke_tools(
tools: pixeltable.func.tools.Tools,
response: pixeltable.exprs.expr.Expr
) -> pixeltable.exprs.inline_expr.InlineDict
```
Converts a Bedrock response dict to Pixeltable tool invocation format and calls `tools._invoke()`.
## udf converse()
```python Signature theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
@pxt.udf
converse(
messages: pxt.Json[(Json, ...)],
*,
model_id: pxt.String,
system: pxt.Json[(Json, ...)] | None = None,
inference_config: pxt.Json | None = None,
additional_model_request_fields: pxt.Json | None = None,
tool_config: pxt.Json[(Json, ...)] | None = None
) -> pxt.Json
```
Generate a conversation response.
Equivalent to the AWS Bedrock `converse` API endpoint.
For additional details, see:
[https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/bedrock-runtime/client/converse.html](https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/bedrock-runtime/client/converse.html)
PIL images and media file paths in `messages[*].content[*].(image|video|audio).source.bytes`
are converted to raw bytes automatically.
**Parameters:**
* **`messages`** (`pxt.Json[(Json`): Input messages.
* **`model_id`** (`Any`): The model that will complete your prompt.
* **`system`** (`Any`): An optional system prompt.
* **`inference_config`** (`Any`): Base inference parameters to use.
* **`additional_model_request_fields`** (`Any`): Additional inference parameters to use.
* **`tool_config`** (`Any`): An optional list of Pixeltable tools to use.
**Returns:**
* `pxt.Json`: A dictionary containing the response and other metadata.
**Examples:**
Add a computed column that applies the model `anthropic.claude-3-haiku-20240307-v1:0`
to an existing Pixeltable column `tbl.prompt` of the table `tbl`:
```python theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
msgs = [{'role': 'user', 'content': [{'text': tbl.prompt}]}]
tbl.add_computed_column(
response=converse(
msgs, model_id='anthropic.claude-3-haiku-20240307-v1:0'
)
)
```
Pass an image via the Converse API:
```python theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
msgs = [
{
'role': 'user',
'content': [
{'image': {'format': 'jpeg', 'source': {'bytes': tbl.image}}},
{'text': "What's in this image?"},
],
}
]
tbl.add_computed_column(
response=converse(msgs, model_id='amazon.nova-lite-v1:0')
)
```
## udf embed()
```python Signatures theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
# Signature 1:
@pxt.udf
embed(
text: pxt.String,
*,
model_id: pxt.String,
dimensions: pxt.Int | None
) -> pxt.Array[(None,), float32]
# Signature 2:
@pxt.udf
embed(
image: pxt.Image,
*,
model_id: pxt.String,
dimensions: pxt.Int | None
) -> pxt.Array[(None,), float32]
```
Generate text or image embeddings using Amazon Titan, Amazon Nova, or Cohere embedding models.
Calls the AWS Bedrock `invoke_model` API for embedding models.
For additional details, see:
[https://docs.aws.amazon.com/bedrock/latest/userguide/titan-embedding-models.html](https://docs.aws.amazon.com/bedrock/latest/userguide/titan-embedding-models.html)
[https://docs.aws.amazon.com/nova/latest/userguide/modality-embedding.html](https://docs.aws.amazon.com/nova/latest/userguide/modality-embedding.html)
[https://docs.aws.amazon.com/bedrock/latest/userguide/model-parameters-embed.html](https://docs.aws.amazon.com/bedrock/latest/userguide/model-parameters-embed.html)
**Parameters:**
* **`text`** (`String`): Input text to embed.
* **`model_id`** (`String`): The embedding model identifier. Supported models:
* `amazon.titan-embed-text-v1`
* `amazon.titan-embed-text-v2:0` (supports `dimensions`: 256, 512, 1024)
* `amazon.nova-2-multimodal-embeddings-v1:0` (supports `dimensions`: 256, 512, 1024, 3072)
* `cohere.embed-english-v3`
* `cohere.embed-multilingual-v3`
* `cohere.embed-v4:0` (supports `dimensions`: 256, 512, 1024, 1536)
* **`dimensions`** (`Int | None`, default: `Literal(None)`): Output embedding dimensions (model-dependent, optional).
**Returns:**
* `pxt.Array[(None,), float32]`: Embedding vector.
**Examples:**
Create an embedding index on a column `description` with Nova embeddings and custom dimensions:
```python theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
tbl.add_embedding_index(
tbl.description,
string_embed=embed.using(
model_id='amazon.nova-2-multimodal-embeddings-v1:0',
dimensions=1024,
),
)
```
## udf invoke\_model()
```python Signature theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
@pxt.udf
invoke_model(body: pxt.Json, *, model_id: pxt.String) -> pxt.Json
```
Invoke a Bedrock model.
Equivalent to the AWS Bedrock `invoke_model` API endpoint, with automatic routing to
`StartAsyncInvoke` for models that require it (e.g. video generation, audio/video embeddings).
For additional details, see:
[https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/bedrock-runtime/client/invoke\_model.html](https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/bedrock-runtime/client/invoke_model.html)
PIL images and media file paths anywhere in the request body are converted automatically
to the encoding expected by the target model. For image-generation models, base64-encoded
images in the response are automatically decoded into `PIL.Image` objects.
For models that require async invocation, `bedrock.temp_location` must be configured
(set environment variable `BEDROCK_TEMP_LOCATION` or add `temp_location` to the `[bedrock]` section of
your Pixeltable configuration file).
**Parameters:**
* **`body`** (`pxt.Json`): The prompt and inference parameters as a dictionary.
* **`model_id`** (`pxt.String`): The model identifier to invoke.
**Returns:**
* `pxt.Json`: A dictionary containing the model response. For image-generation models,
image fields are decoded to `PIL.Image` objects. For video-generation models,
returns a `pxt.Video` path.
**Examples:**
Invoke Amazon Titan text embeddings:
```python theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
body = {'inputText': tbl.text, 'dimensions': 512, 'normalize': True}
tbl.add_computed_column(
response=invoke_model(body, model_id='amazon.titan-embed-text-v2:0')
)
```
Invoke TwelveLabs Marengo with an image column (note that the image can be included directly
in the invoke body, and will be automatically base64-encoded by Pixeltable):
```python theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
body = {
'inputType': 'image',
'image': {'mediaSource': {'base64String': tbl.image}},
}
tbl.add_computed_column(
response=invoke_model(
body, model_id='twelvelabs.marengo-embed-3-0-v1:0'
)
)
```
Invoke TwelveLabs Marengo with audio (auto-routes to async via StartAsyncInvoke;
requires `bedrock.temp_location` to be configured):
```python theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
body = {
'inputType': 'audio',
'audio': {'mediaSource': {'base64String': tbl.audio}},
}
tbl.add_computed_column(
response=invoke_model(
body, model_id='twelvelabs.marengo-embed-3-0-v1:0'
)
)
```
Invoke Anthropic Claude with an image:
```python theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
body = {
'anthropic_version': 'bedrock-2023-05-31',
'max_tokens': 1024,
'messages': [
{
'role': 'user',
'content': [
{
'type': 'image',
'source': {
'type': 'base64',
'media_type': 'image/jpeg',
'data': tbl.image,
},
},
{'type': 'text', 'text': "What's in this image?"},
],
}
],
}
tbl.add_computed_column(
response=invoke_model(
body, model_id='anthropic.claude-3-haiku-20240307-v1:0'
)
)
```
Invoke Amazon Nova Lite with a video column:
```python theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
body = {
'messages': [
{
'role': 'user',
'content': [
{
'video': {
'format': 'mp4',
'source': {'bytes': tbl.video},
}
},
{'text': 'What happens in this video?'},
],
}
]
}
tbl.add_computed_column(
response=invoke_model(body, model_id='amazon.nova-lite-v1:0')
)
```
Invoke Stability AI for image generation:
```python theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
body = {
'prompt': tbl.prompt,
'mode': 'text-to-image',
'aspect_ratio': '1:1',
'output_format': 'jpeg',
}
tbl.add_computed_column(
response=invoke_model(body, model_id='stability.sd3-5-large-v1:0')
)
tbl.add_computed_column(image=tbl.response['images'][0])
```
Invoke Amazon Nova Reel for video generation (auto-routes to async; requires `bedrock.temp_location`):
```python theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
body = {
'taskType': 'TEXT_VIDEO',
'textToVideoParams': {'text': tbl.prompt},
'videoGenerationConfig': {'durationSeconds': 6, 'fps': 24},
}
tbl.add_computed_column(
video=invoke_model(body, model_id='amazon.nova-reel-v1:1')
)
```
# bfl
Source: https://docs.pixeltable.com/sdk/latest/bfl
# module pixeltable.functions.bfl
Pixeltable [UDFs](https://docs.pixeltable.com/platform/udfs-in-pixeltable) that wrap
[Black Forest Labs (BFL)](https://docs.bfl.ai/) FLUX image generation API. In order to use them,
the API key must be specified either with `BFL_API_KEY` environment variable, or as `api_key`
in the `bfl` section of the Pixeltable config file.
For more information on FLUX models, see the [BFL documentation](https://docs.bfl.ai/).
## udf edit()
```python Signature theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
@pxt.udf
edit(
prompt: pxt.String,
image: pxt.Image,
*,
model: pxt.String,
reference_images: pxt.Json[(Image, ...)] | None = None,
width: pxt.Int | None = None,
height: pxt.Int | None = None,
seed: pxt.Int | None = None,
safety_tolerance: pxt.Int | None = None,
output_format: pxt.String | None = None,
steps: pxt.Int | None = None,
guidance: pxt.Float | None = None
) -> pxt.Image
```
Edit an image using FLUX models with text prompts and optional reference images.
This UDF wraps the BFL FLUX image editing API. For more information, refer to the official
[API documentation](https://docs.bfl.ai/flux_2/flux2_image_editing).
**Parameters:**
* **`prompt`** (`pxt.String`): Text description of the edit to apply.
* **`image`** (`pxt.Image`): The base image to edit.
* **`model`** (`pxt.String`): The FLUX model to use for editing. See available models at
[https://docs.bfl.ai/](https://docs.bfl.ai/).
* **`reference_images`** (`pxt.Json[(Image`): Additional reference images (up to 7) for multi-reference editing.
* **`width`** (`Any`): Output width in pixels (multiple of 16). Matches input if not specified.
* **`height`** (`Any`): Output height in pixels (multiple of 16). Matches input if not specified.
* **`seed`** (`Any`): Random seed for reproducible results.
* **`safety_tolerance`** (`Any`): Moderation level from 0 (strict) to 6 (permissive). Default 2.
* **`output_format`** (`Any`): Image format, 'jpeg' or 'png'. Default 'jpeg'.
* **`steps`** (`Any`): Number of inference steps (flux-2-flex only, max 50).
* **`guidance`** (`Any`): Guidance scale 1.5-10 (flux-2-flex only). Default 4.5.
**Returns:**
* `pxt.Image`: An edited PIL Image.
**Examples:**
Edit an image to change its background:
```python theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
t.add_computed_column(
edited=bfl.edit(
'Change the background to a sunset beach',
t.original_image,
model='flux-2-pro',
)
)
```
Multi-reference editing with additional images:
```python theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
t.add_computed_column(
edited=bfl.edit(
'Combine the person from the first image with the background from the second',
t.person_image,
model='flux-kontext-pro',
reference_images=[t.background_image],
)
)
```
## udf expand()
```python Signature theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
@pxt.udf
expand(
prompt: pxt.String,
image: pxt.Image,
*,
model: pxt.String,
top: pxt.Int = 0,
bottom: pxt.Int = 0,
left: pxt.Int = 0,
right: pxt.Int = 0,
seed: pxt.Int | None = None,
safety_tolerance: pxt.Int | None = None,
output_format: pxt.String | None = None
) -> pxt.Image
```
Expand an image by adding pixels on any side using FLUX Expand models.
Outpaint an image by specifying how many pixels to add to each edge.
The expansion maintains context from the original image.
This UDF wraps the BFL FLUX Expand API. For more information, refer to the official
[API documentation](https://docs.bfl.ai/flux_tools/flux_1_expand).
**Parameters:**
* **`prompt`** (`pxt.String`): Text description to guide the expansion.
* **`image`** (`pxt.Image`): The base image to expand.
* **`model`** (`pxt.String`): The FLUX Expand model to use. See available models at
[https://docs.bfl.ai/](https://docs.bfl.ai/).
* **`top`** (`pxt.Int`): Pixels to add to the top edge.
* **`bottom`** (`pxt.Int`): Pixels to add to the bottom edge.
* **`left`** (`pxt.Int`): Pixels to add to the left edge.
* **`right`** (`pxt.Int`): Pixels to add to the right edge.
* **`seed`** (`pxt.Int | None`): Random seed for reproducible results.
* **`safety_tolerance`** (`pxt.Int | None`): Moderation level from 0 (strict) to 6 (permissive). Default 2.
* **`output_format`** (`pxt.String | None`): Image format, 'jpeg' or 'png'. Default 'jpeg'.
**Returns:**
* `pxt.Image`: An expanded PIL Image.
**Examples:**
Expand an image to create a wider landscape:
```python theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
t.add_computed_column(
wide=bfl.expand(
'Continue the landscape scenery',
t.original_image,
model='flux-pro-1.0-expand',
left=256,
right=256,
)
)
```
## udf fill()
```python Signature theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
@pxt.udf
fill(
prompt: pxt.String,
image: pxt.Image,
mask: pxt.Image,
*,
model: pxt.String,
steps: pxt.Int | None = None,
guidance: pxt.Float | None = None,
seed: pxt.Int | None = None,
safety_tolerance: pxt.Int | None = None,
output_format: pxt.String | None = None
) -> pxt.Image
```
Inpaint an image using FLUX Fill models.
Fill specified areas of an image based on a mask and text prompt. The mask can be
a separate image or applied to the alpha channel of the input image.
This UDF wraps the BFL FLUX Fill API. For more information, refer to the official
[API documentation](https://docs.bfl.ai/flux_tools/flux_1_fill).
**Parameters:**
* **`prompt`** (`pxt.String`): Text description of what to fill in the masked area.
* **`image`** (`pxt.Image`): The base image to inpaint.
* **`mask`** (`pxt.Image`): Mask image where white areas indicate regions to fill (black areas preserved).
* **`model`** (`pxt.String`): The FLUX Fill model to use. See available models at
[https://docs.bfl.ai/](https://docs.bfl.ai/).
* **`steps`** (`pxt.Int | None`): Number of inference steps (max 50). Default 50.
* **`guidance`** (`pxt.Float | None`): Guidance scale for generation. Default 30.
* **`seed`** (`pxt.Int | None`): Random seed for reproducible results.
* **`safety_tolerance`** (`pxt.Int | None`): Moderation level from 0 (strict) to 6 (permissive). Default 2.
* **`output_format`** (`pxt.String | None`): Image format, 'jpeg' or 'png'. Default 'jpeg'.
**Returns:**
* `pxt.Image`: An inpainted PIL Image.
**Examples:**
Fill a masked region with generated content:
```python theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
t.add_computed_column(
filled=bfl.fill(
'A beautiful garden with flowers',
t.original_image,
t.mask_image,
model='flux-pro-1.0-fill',
)
)
```
## udf generate()
```python Signature theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
@pxt.udf
generate(
prompt: pxt.String,
*,
model: pxt.String,
width: pxt.Int | None = None,
height: pxt.Int | None = None,
seed: pxt.Int | None = None,
safety_tolerance: pxt.Int | None = None,
output_format: pxt.String | None = None,
steps: pxt.Int | None = None,
guidance: pxt.Float | None = None
) -> pxt.Image
```
Generate an image from a text prompt using FLUX models.
This UDF wraps the BFL FLUX API endpoints. For more information, refer to the official
[API documentation](https://docs.bfl.ai/flux_2/flux2_text_to_image).
**Parameters:**
* **`prompt`** (`pxt.String`): Text description of the image to generate.
* **`model`** (`pxt.String`): The FLUX model to use. See available models at
[https://docs.bfl.ai/](https://docs.bfl.ai/).
* **`width`** (`pxt.Int | None`): Output width in pixels (multiple of 16). Default 1024.
* **`height`** (`pxt.Int | None`): Output height in pixels (multiple of 16). Default 1024.
* **`seed`** (`pxt.Int | None`): Random seed for reproducible results.
* **`safety_tolerance`** (`pxt.Int | None`): Moderation level from 0 (strict) to 6 (permissive). Default 2.
* **`output_format`** (`pxt.String | None`): Image format, 'jpeg' or 'png'. Default 'jpeg'.
* **`steps`** (`pxt.Int | None`): Number of inference steps (flux-2-flex only, max 50).
* **`guidance`** (`pxt.Float | None`): Guidance scale 1.5-10 (flux-2-flex only). Default 4.5.
**Returns:**
* `pxt.Image`: A generated PIL Image.
**Examples:**
Generate images using default dimensions:
```python theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
t.add_computed_column(image=bfl.generate(t.prompt, model='flux-2-pro'))
```
Generate with custom dimensions:
```python theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
t.add_computed_column(
image=bfl.generate(
t.prompt, model='flux-2-pro', width=1920, height=1080
)
)
```
Generate with specific seed for reproducibility:
```python theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
t.add_computed_column(
image=bfl.generate(t.prompt, model='flux-2-pro', seed=42)
)
```
# CellError
Source: https://docs.pixeltable.com/sdk/latest/cellerror
# class pixeltable.CellError
Error info for a cell.
## attr errormsg
```
errormsg: str
```
⚠️ **No documentation**
## attr errortype
```
errortype: str
```
⚠️ **No documentation**
# CodecContextMetadata
Source: https://docs.pixeltable.com/sdk/latest/codeccontextmetadata
# class pixeltable.functions.CodecContextMetadata
Metadata about a stream's codec.
## attr channels
```
channels: int | None
```
Number of audio channels. Present only for audio streams.
## attr codec\_tag
```
codec_tag: str
```
Four-character codec tag, unicode-escaped.
## attr name
```
name: str
```
Codec name (e.g. `'h264'`, `'aac'`).
## attr pix\_fmt
```
pix_fmt: str | None
```
Pixel format (e.g. `'yuv420p'`). Present only for video streams.
## attr profile
```
profile: str | None
```
Codec profile (e.g. `'High'`, `'LC'`), or `None` if unavailable.
# ColumnMetadata
Source: https://docs.pixeltable.com/sdk/latest/columnmetadata
# class pixeltable.ColumnMetadata
Metadata for a column of a Pixeltable table.
## attr comment
```
comment: str | None
```
User-provided column comment.
## attr computed\_with
```
computed_with: str | None
```
Expression used to compute this column; `None` if this is not a computed column.
## attr custom\_metadata
```
custom_metadata: Any
```
User-defined JSON metadata for this column, if any.
## attr defined\_in
```
defined_in: str | None
```
Name of the table where this column was originally defined.
If the current table is a view, then `defined_in` may differ from the current table name.
## attr depends\_on
```
depends_on: list[tuple[str, str]] | None
```
List of dependencies (table name, column name) if this is a computed column, else `None`.
## attr destination
```
destination: str | None
```
The object store destination defined for this column's computed files, if it defines one.
An instance-wide default destination is not reported here, because it is configuration rather than part of
the schema. A destination defined as a config var reads as `$` rather than as the location it
currently resolves to; see [`ConfigVar`](./configvar).
## attr is\_builtin
```
is_builtin: bool | None
```
If False, this computed column makes calls to custom UDFs; `None` if this is not a computed column.
## attr is\_computed
```
is_computed: bool
```
`True` if this column is a computed column.
## attr is\_iterator\_col
```
is_iterator_col: bool
```
`True` if this column is produced by an iterator (only applicable to views).
## attr is\_primary\_key
```
is_primary_key: bool
```
`True` if this column is part of the table's primary key.
## attr is\_stored
```
is_stored: bool
```
`True` if this is a stored column; `False` if it is dynamically computed.
## attr media\_validation
```
media_validation: Literal['on_read', 'on_write'] | None
```
The media validation policy for this column. `None` if the type of this column is not a media type.
## attr name
```
name: str
```
The name of the column.
## attr type\_
```
type_: str
```
The type specifier of the column.
## attr version\_added
```
version_added: int
```
The table version when this column was added.
# ColumnRef
Source: https://docs.pixeltable.com/sdk/latest/columnref
# class pixeltable.exprs.ColumnRef
A Pixeltable expression that references a column of a table. A `ColumnRef` is created by column access
on a [`Table`](./table), such as `t.col`.
Not thread-safe.
## method embedding()
```python Signature theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
embedding(*, idx: str | None = None) -> ColumnRef
```
Return a reference to the values of an embedding index on this column.
If an embedding index is defined on a column, the usual way to use that index is via a
[`similarity()`](./columnref#method-similarity) lookup. Sometimes it is also useful to directly access
the index values (i.e., the embedding vectors themselves). Calling `embedding()` returns a new `ColumnRef`
expression of type `pxt.Array[(dim,), prec]`, where `dim` and `prec` are the dimensionality and precision
of the column's embedding index.
If there is more than one embedding index defined on this column, then the `idx` parameter must be provided to
specify which index to reference. If there is only one index, then `idx` is optional.
Args:
idx: An optional embedding index name. *Required* if there is more than one embedding index defined on
this column.
Returns:
A new `ColumnRef` referencing the values of the specified embedding index on this column.
Raises:
`pxt.Error` if there is no embedding index defined on this column, if `idx` is not provided when there are
multiple embedding indices, or if `idx` does not match any embedding index defined on this column.
Examples:
All of these examples assume that `t` is a table with an image column `t.image`.
Add an embedding index to `t.image` using the `clip()`
embedding (this only needs to be done once):
```python theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
from pixeltable.functions.huggingface import clip
t.add_embedding_index(
t.image, clip.using(model_id='openai/clip-vit-base-patch32')
)
```
Reference the embedding index values directly:
```python theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
t.select(t.image, t.image.embedding())
```
## method similarity()
```python Signature theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
similarity(
item: Any = None,
*,
string: str | None = None,
image: str | PIL.Image.Image | None = None,
audio: str | None = None,
video: str | None = None,
document: str | None = None,
vector: np.ndarray | None = None,
idx: str | None = None,
_tbl_path: catalog.TablePath | None = None
) -> Expr
```
Return a new expression representing the similarity score between the values of this column and the given
(constant) item. In order for this to work, there must be an embedding index defined on this column that
supports the modality of the given item (string, image, audio, video, document). Similarity will be scored
according to the metric defined by the embedding index.
Exactly one of `string`, `image`, `audio`, `video`, `document`, or `vector` must be provided. The `item`
parameter is deprecated and exists for backward compatibility only.
If `string`, `image`, `audio`, `video`, or `document` is provided, then an embedding vector will be computed
for the given input as defined by the embedding index and used to determine similarity. If `vector` is
provided, then it must be a 1-dimensional array of the same dimensionality as the index, and similarity will
be determined directly against the vector.
The optional `idx` parameter specifies the name of the embedding index to use. If there is more than one
embedding index defined on this column, then `idx` *must* be provided.
**Parameters:**
* **`string`** (`str | None`): A string to compare against the values of this column.
* **`image`** (`str | PIL.Image.Image | None`): An image to compare against the values of this column (either a local file path, a URL, or an
in-memory `PIL.Image.Image`).
* **`audio`** (`str | None`): An audio file to compare against the values of this column (a local file path or a URL).
* **`video`** (`str | None`): A video file to compare against the values of this column (a local file path or a URL).
* **`document`** (`str | None`): A document file to compare against the values of this column (a local file path or a URL).
* **`vector`** (`np.ndarray | None`): A 1-dimensional NumPy array to compare against the values of this column.
* **`idx`** (`str | None`): An optional embedding index name. *Required* if there is more than one embedding index defined on
this column.
* **`item`** (`Any`): **Deprecated** as of version 0.5.7.
**Returns:**
* `Expr`: A new expression representing the similarity score between the values of this column and the given item.
**Examples:**
All of these examples assume that `t` is a table with an image column `t.image`.
Add an embedding index to `t.image` using the `clip()`
embedding (this only needs to be done once):
```python theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
from pixeltable.functions.huggingface import clip
t.add_embedding_index(
t.image, clip.using(model_id='openai/clip-vit-base-patch32')
)
```
Do a nearest neighbor search against a string (with `k=5`):
```python theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
sim = t.image.similarity(string='a photograph of a cat')
t.select(t.image, sim).order_by(sim, asc=False).head(5)
```
Do a nearest neighbor search against an image:
```python theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
sim = t.image.similarity(image='https://example.com/reference-cat.jpg')
t.select(t.image, sim).order_by(sim, asc=False).head(5)
```
# ColumnSpec
Source: https://docs.pixeltable.com/sdk/latest/columnspec
# class pixeltable.types.ColumnSpec
Column specification, a dictionary representation of a column's schema.
Exactly one of `type` or `value` must be included in the dictionary.
## attr comment
```
comment: str
```
Optional comment for the column. Displayed in .describe() output.
## attr custom\_metadata
```
custom_metadata: Any
```
User-defined metadata to associate with the column.
## attr destination
```
destination: str | Path | ConfigVar[URI]
```
Destination for storing computed output files. Only applicable for computed columns.
Can be:
* A local pathname (such as `path/to/outputs/`), or
* The URI of an object store (such as `s3://my-bucket/outputs/`).
## attr media\_validation
```
media_validation: Literal['on_read', 'on_write']
```
When to validate media; `'on_read'` or `'on_write'`.
## attr primary\_key
```
primary_key: bool
```
Whether this column is part of the primary key. Defaults to `False`.
## attr stored
```
stored: bool
```
Whether to store the column data. Defaults to `True`.
## attr type
```
type: TypeForm
```
The column type (e.g., `pxt.Image`, `str`). Required unless `value` is specified.
## attr value
```
value: 'exprs.Expr'
```
A Pixeltable expression for computed columns. Mutually exclusive with `type`.
# ConfigVar
Source: https://docs.pixeltable.com/sdk/latest/configvar
# class pixeltable.ConfigVar
A reference to a database variable, defined at module scope.
A definition names the variable; the target it is applied to supplies the value.
Define a variable and apply it to a column:
> > > MEDIA\_DEST = pxt.ConfigVar('media\_dest', pxt.URI)
> > > ...
> > > ...
> > > ... class Videos(TableModel, name='videos'):
> > > ... clip = pxt.Column(value=..., destination=MEDIA\_DEST)
Code that runs on the target reads the bound value with `value()`.
## method value()
```python Signature theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
value() -> ConfVarT
```
The bound value, converted to the declared type. Raises if the target has no binding for it.
**Examples:**
Read a var from a udf, which runs on the target:
```python theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
@pxt.udf
def summarize(text: str) -> str:
return _call(text, dest=MEDIA_DEST.value())
```
# ContainerMetadata
Source: https://docs.pixeltable.com/sdk/latest/containermetadata
# class pixeltable.functions.ContainerMetadata
Metadata for a media container, as returned by
[`audio.get_metadata()`](./audio#func-get_metadata)
or [`video.get_metadata()`](./video#func-get_metadata).
## attr bit\_exact
```
bit_exact: bool
```
Whether the container was opened in bit-exact mode.
## attr bit\_rate
```
bit_rate: int | None
```
Overall bit rate of the container in bits per second, or `None` if unknown.
## attr metadata
```
metadata: dict[str, str]
```
Additional container-level metadata tags (e.g. title, encoder).
## attr size
```
size: int | None
```
Size of the container in bytes, or `None` if unknown.
## attr streams
```
streams: list[StreamMetadata]
```
Per-stream metadata for each stream in the container.
# date
Source: https://docs.pixeltable.com/sdk/latest/date
# module pixeltable.functions.date
Pixeltable UDFs for `DateType`.
Usage example:
```python theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
import pixeltable as pxt
t = pxt.get_table(...)
t.select(t.date_col.year, t.date_col.weekday()).collect()
```
## udf add\_days()
```python Signature theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
@pxt.udf
add_days(self: pxt.Date, n: pxt.Int) -> pxt.Date
```
Add `n` days to the date.
Equivalent to [`date + timedelta(days=n)`](https://docs.python.org/3/library/datetime.html#datetime.timedelta).
## udf day()
```python Signature theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
@pxt.udf
day(self: pxt.Date) -> pxt.Int
```
Between 1 and the number of days in the given month of the given year.
Equivalent to [`date.day`](https://docs.python.org/3/library/datetime.html#datetime.date.day).
## udf isocalendar()
```python Signature theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
@pxt.udf
isocalendar(self: pxt.Date) -> IsoCalendar
```
Return a dictionary with three entries: `'year'`, `'week'`, and `'weekday'`.
Equivalent to
[`date.isocalendar()`](https://docs.python.org/3/library/datetime.html#datetime.date.isocalendar).
## udf isoformat()
```python Signature theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
@pxt.udf
isoformat(
self: pxt.Date,
sep: pxt.String = 'T',
timespec: pxt.String = 'auto'
) -> pxt.String
```
Return a string representing the date and time in ISO 8601 format.
Equivalent to [`date.isoformat()`](https://docs.python.org/3/library/datetime.html#datetime.date.isoformat).
**Parameters:**
* **`sep`** (`pxt.String`): Separator between date and time.
* **`timespec`** (`pxt.String`): The number of additional terms in the output. See the
[`date.isoformat()`](https://docs.python.org/3/library/datetime.html#datetime.date.isoformat)
documentation for more details.
## udf isoweekday()
```python Signature theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
@pxt.udf
isoweekday(self: pxt.Date) -> pxt.Int
```
Return the day of the week as an integer, where Monday is 1 and Sunday is 7.
Equivalent to [`date.isoweekday()`](https://docs.python.org/3/library/datetime.html#datetime.date.isoweekday).
## udf make\_date()
```python Signature theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
@pxt.udf
make_date(year: pxt.Int, month: pxt.Int, day: pxt.Int) -> pxt.Date
```
Create a date.
Equivalent to [`datetime()`](https://docs.python.org/3/library/datetime.html#datetime.date).
## udf month()
```python Signature theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
@pxt.udf
month(self: pxt.Date) -> pxt.Int
```
Between 1 and 12 inclusive.
Equivalent to [`date.month`](https://docs.python.org/3/library/datetime.html#datetime.date.month).
## udf strftime()
```python Signature theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
@pxt.udf
strftime(self: pxt.Date, format: pxt.String) -> pxt.String
```
Return a string representing the date and time, controlled by an explicit format string.
Equivalent to [`date.strftime()`](https://docs.python.org/3/library/datetime.html#datetime.date.strftime).
**Parameters:**
* **`format`** (`pxt.String`): The format string to control the output. For a complete list of formatting directives, see
[`strftime()` and `strptime()` Behavior](https://docs.python.org/3/library/datetime.html#strftime-and-strptime-behavior).
## udf toordinal()
```python Signature theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
@pxt.udf
toordinal(self: pxt.Date) -> pxt.Int
```
Return the proleptic Gregorian ordinal of the date, where January 1 of year 1 has ordinal 1.
Equivalent to [`date.toordinal()`](https://docs.python.org/3/library/datetime.html#datetime.date.toordinal).
## udf weekday()
```python Signature theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
@pxt.udf
weekday(self: pxt.Date) -> pxt.Int
```
Between 0 (Monday) and 6 (Sunday) inclusive.
Equivalent to [`date.weekday()`](https://docs.python.org/3/library/datetime.html#datetime.date.weekday).
## udf year()
```python Signature theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
@pxt.udf
year(self: pxt.Date) -> pxt.Int
```
Between 1 and 9999 inclusive.
(Between [`MINYEAR`](https://docs.python.org/3/library/datetime.html#datetime.MINYEAR) and
[`MAXYEAR`](https://docs.python.org/3/library/datetime.html#datetime.MAXYEAR) as defined by the Python `datetime`
library).
Equivalent to [`date.year`](https://docs.python.org/3/library/datetime.html#datetime.date.year).
# deepseek
Source: https://docs.pixeltable.com/sdk/latest/deepseek
# module pixeltable.functions.deepseek
Pixeltable UDFs for Deepseek AI models.
Provides integration with Deepseek's language models for chat completions
and other AI capabilities.
## udf chat\_completions()
```python Signature theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
@pxt.udf
chat_completions(
messages: pxt.Json[(Json, ...)],
*,
model: pxt.String,
model_kwargs: pxt.Json | None = None,
tools: pxt.Json[(Json, ...)] | None = None,
tool_choice: pxt.Json | None = None
) -> pxt.Json
```
Creates a model response for the given chat conversation.
Equivalent to the Deepseek `chat/completions` API endpoint.
For additional details, see: [https://api-docs.deepseek.com/api/create-chat-completion](https://api-docs.deepseek.com/api/create-chat-completion)
Deepseek uses the OpenAI SDK, so you will need to install the `openai` package to use this UDF.
Request throttling:
Applies the rate limit set in the config (section `deepseek`, key `rate_limit`). If no rate
limit is configured, uses a default of 600 RPM.
**Requirements:**
* `pip install openai`
**Parameters:**
* **`messages`** (`pxt.Json[(Json`): A list of messages to use for chat completion, as described in the Deepseek API documentation.
* **`model`** (`Any`): The model to use for chat completion.
* **`model_kwargs`** (`Any`): Additional keyword args for the Deepseek `chat/completions` API.
For details on the available parameters, see: [https://api-docs.deepseek.com/api/create-chat-completion](https://api-docs.deepseek.com/api/create-chat-completion)
* **`tools`** (`Any`): An optional list of Pixeltable tools to use for the request.
* **`tool_choice`** (`Any`): An optional tool choice configuration.
**Returns:**
* `pxt.Json`: A dictionary containing the response and other metadata.
**Examples:**
Add a computed column that applies the model `deepseek-v4-flash` to an existing Pixeltable column `tbl.prompt`
of the table `tbl`:
```python theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
messages = [
{'role': 'system', 'content': 'You are a helpful assistant.'},
{'role': 'user', 'content': tbl.prompt},
]
tbl.add_computed_column(
response=chat_completions(messages, model='deepseek-v4-flash')
)
```
# DirContents
Source: https://docs.pixeltable.com/sdk/latest/dircontents
# class pixeltable.DirContents
Represents the contents of a Pixeltable directory.
## attr dirs
```
dirs: list[str]
```
List of directory paths contained in this directory.
## attr tables
```
tables: list[str]
```
List of table paths contained in this directory.
# DirectoryNode
Source: https://docs.pixeltable.com/sdk/latest/directorynode
# class pixeltable.DirectoryNode
A directory entry in a [`TreeNode`](./treenode) tree.
## attr entries
```
entries: list['TreeNode']
```
⚠️ **No documentation**
## attr kind
```
kind: Literal['directory']
```
⚠️ **No documentation**
## attr name
```
name: str
```
⚠️ **No documentation**
## attr path
```
path: str
```
⚠️ **No documentation**
# document
Source: https://docs.pixeltable.com/sdk/latest/document
# module pixeltable.functions.document
Pixeltable UDFs for `DocumentType`.
## iterator document\_splitter()
```python Signature theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
@pxt.iterator
document_splitter(
document: pxt.Document,
separators: pxt.String,
*,
elements: pxt.Json[(String, ...)] | None = None,
limit: pxt.Int | None = None,
overlap: pxt.Int | None = None,
metadata: pxt.String = '',
skip_tags: pxt.Json[(String, ...)] | None = None,
spacy_model: pxt.String = 'en_core_web_sm',
tiktoken_encoding: pxt.String | None = 'cl100k_base',
tiktoken_target_model: pxt.String | None = None,
image_dpi: pxt.Int = 300,
image_format: pxt.String = 'png'
)
```
Iterator over chunks of a document. The document is chunked according to the specified `separators`.
Chunked text will be cleaned with `ftfy.fix_text` to fix up common problems with unicode sequences.
**Outputs**:
One row per chunk, with the following columns, depending on the specified `elements` and `metadata`:
* `text` (`pxt.String`): The text of the chunk. Present if `'text'` is specified in `elements`.
* `image` (`pxt.Image`): The image extracted from the chunk. Present if `'image'` is specified in `elements`.
* `title` (`pxt.String | None`): The document title. Present if `'title'` is specified in `metadata`.
* `heading` (`Heading | None`): The heading hierarchy at the start of the chunk, with the text of each
currently open heading level under the keys `h1` through `h6` (HTML and Markdown only).
Present if `'heading'` is specified in `metadata`.
* `sourceline` (`pxt.Int | None`): The source line number of the start of the chunk (HTML only).
Present if `'sourceline'` is specified in `metadata`.
* `page` (`pxt.Int | None`): The page number of the chunk (PDF only). Present if `'page'` is specified in
`metadata`.
* `bounding_box` (`BoundingBox | None`): The bounding box of the chunk on the page, with float entries
`x1`, `y1`, `x2`, `y2` (PDF only). Present if `'bounding_box'` is specified in `metadata`.
**Parameters:**
* **`separators`** (`pxt.String`): separators to use to chunk the document. Options are:
`'heading'`, `'paragraph'`, `'sentence'`, `'token_limit'`, `'char_limit'`, `'page'`.
This may be a comma-separated string, e.g., `'heading,token_limit'`.
* **`elements`** (`pxt.Json[(String`): list of elements to extract from the document. Options are:
`'text'`, `'image'`. Defaults to `['text']` if not specified. The `'image'` element is only supported
for the `'page'` separator on PDF documents.
* **`limit`** (`Any`): the maximum number of tokens or characters in each chunk, if `'token_limit'`
or `'char_limit'` is specified.
* **`metadata`** (`Any`): additional metadata fields to include in the output. Options are:
`'title'`, `'heading'` (HTML and Markdown), `'sourceline'` (HTML), `'page'` (PDF), `'bounding_box'`
(PDF). The input may be a comma-separated string, e.g., `'title,heading,sourceline'`.
* **`skip_tags`** (`Any`): list of HTML tags to skip when processing HTML documents.
* **`spacy_model`** (`Any`): Name of the spaCy model to use for sentence segmentation. This parameter is ignored unless
the `'sentence'` separator is specified.
* **`tiktoken_encoding`** (`Any`): Name of the tiktoken encoding to use when counting tokens. This parameter is ignored
unless the `'token_limit'` separator is specified.
* **`tiktoken_target_model`** (`Any`): Name of the target model to use when counting tokens with tiktoken. If specified,
this parameter overrides `tiktoken_encoding`. This parameter is ignored unless the `'token_limit'`
separator is specified.
* **`image_dpi`** (`Any`): DPI to use when extracting images from PDFs. Defaults to 300.
* **`image_format`** (`Any`): format to use when extracting images from PDFs. Defaults to 'png'.
**Examples:**
All these examples assume an existing table `tbl` with a column `doc` of type `pxt.Document`.
Create a view that splits all documents into chunks of up to 300 tokens:
```python theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
pxt.create_view(
'chunks',
tbl,
iterator=document_splitter(
tbl.doc, separators='token_limit', limit=300
),
)
```
Create a view that splits all documents along sentence boundaries, including title and heading metadata:
```python theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
pxt.create_view(
'sentence_chunks',
tbl,
iterator=document_splitter(
tbl.doc, separators='sentence', metadata='title,heading'
),
)
```
# Expr
Source: https://docs.pixeltable.com/sdk/latest/expr
# class pixeltable.exprs.Expr
A Pixeltable expression.
All Pixeltable expressions, including [column references](./columnref) (such as `t.col`),
UDF calls (`t.my_string.lower()`), and compound expressions (`t.col + 5`) are instances of this class.
Not thread-safe: Expr and subclasses contain execution state and are never thread-safe.
## method astype()
```python Signature theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
astype(new_type: ts.ColumnType | TypeForm) -> exprs.TypeCast
```
Return a new expression that casts this expression to a different type.
This represents a type *cast*, not a type *coercion*, so it will not mutate the underlying data; it simply
changes the static type given by the expression.
**Parameters:**
* **`new_type`** (`ts.ColumnType | TypeForm`): The type to cast to.
**Examples:**
Given an existing column `t.json_col` of type `pxt.Json`, cast that column to type `pxt.String`:
```python theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
t.json_col.astype(pxt.String)
```
This will assume that all values in `t.json_col` are *actually* strings, and it will result in an error
at runtime if there are values in `t.json_col` that are not strings. It will *not* convert those values to
a string representation. (For that, use the \[`dumps()`]\[pixeltable.functions.json.dumps] UDF instead.)
## method isin()
```python Signature theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
isin(value_set: Any) -> exprs.InPredicate
```
Return a new, boolean-valued expression that is `True` whenever this expression is in `value_set`.
**Parameters:**
* **`value_set`** (`Any`): Either another expression that evaluates to a set of values, or a constant collection of
values. If the latter, can be any `Iterable`.
**Examples:**
These examples assume that `t` is a table with a column `int_col` of type `pxt.Int`, and another column
`list_col` of type `pxt.Json`, containing lists of integers.
Select all rows where `int_col` is in the constant set `{1, 3, 22}`:
```python theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
t.where(t.int_col.isin({1, 3, 22})).select()
```
Select all rows where `int_col` is in the set of values in that row's `list_col`:
```python theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
t.where(t.int_col.isin(t.list_col)).select()
```
# fabric
Source: https://docs.pixeltable.com/sdk/latest/fabric
# module pixeltable.functions.fabric
Pixeltable UDFs that wrap Azure OpenAI endpoints via Microsoft Fabric.
These functions provide seamless access to Azure OpenAI models within Microsoft Fabric
notebook environments. Authentication and endpoint discovery are handled automatically
using Fabric's built-in service discovery and token utilities.
**Note:** These functions only work within Microsoft Fabric notebook environments.
For more information on Fabric AI services, see:
[https://learn.microsoft.com/en-us/fabric/data-science/ai-services/ai-services-overview](https://learn.microsoft.com/en-us/fabric/data-science/ai-services/ai-services-overview)
## udf chat\_completions()
```python Signature theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
@pxt.udf
chat_completions(
messages: pxt.Json[(Json, ...)],
*,
model: pxt.String,
api_version: pxt.String | None = None,
model_kwargs: pxt.Json | None = None
) -> pxt.Json
```
Creates a model response for the given chat conversation using Azure OpenAI in Fabric.
Equivalent to the Azure OpenAI `chat/completions` API endpoint.
For additional details, see: [https://learn.microsoft.com/en-us/azure/ai-services/openai/reference](https://learn.microsoft.com/en-us/azure/ai-services/openai/reference)
**Automatic authentication:** Authentication is handled automatically in Fabric notebooks using
token-based authentication. No API keys are required.
**Supported models in Fabric:**
* `gpt-5` (reasoning model)
* `gpt-4.1`
* `gpt-4.1-mini`
Request throttling:
Applies the rate limit set in the config (section `fabric.rate_limits`, key `chat`). If no rate
limit is configured, uses a default of 600 RPM.
**Requirements:**
* Microsoft Fabric notebook environment
* `synapse-ml-fabric` package (pre-installed in Fabric)
**Parameters:**
* **`messages`** (`pxt.Json[(Json`): A list of message dicts with 'role' and 'content' keys, as described in the
Azure OpenAI API documentation.
* **`model`** (`Any`): The deployment name to use (e.g., 'gpt-5', 'gpt-4.1', 'gpt-4.1-mini').
* **`api_version`** (`Any`): Optional API version override. If not specified, defaults to '2025-04-01-preview'
for reasoning models (gpt-5) and '2024-02-15-preview' for standard models.
* **`model_kwargs`** (`Any`): Additional keyword args for the Azure OpenAI `chat/completions` API.
For details on available parameters, see:
[https://learn.microsoft.com/en-us/azure/ai-services/openai/reference](https://learn.microsoft.com/en-us/azure/ai-services/openai/reference)
**Note:** Reasoning models (gpt-5) use `max_completion_tokens` instead of `max_tokens`
and do not support the `temperature` parameter.
**Returns:**
* `pxt.Json`: A dictionary containing the response and other metadata.
**Examples:**
Add a computed column that applies the model `gpt-4.1` to an existing Pixeltable column
`tbl.prompt` of the table `tbl`:
```python theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
from pixeltable.functions import fabric
messages = [
{'role': 'system', 'content': 'You are a helpful assistant.'},
{'role': 'user', 'content': tbl.prompt},
]
tbl.add_computed_column(
response=fabric.chat_completions(messages, model='gpt-4.1')
)
```
Using a reasoning model (gpt-5):
```python theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
tbl.add_computed_column(
reasoning_response=fabric.chat_completions(
messages,
model='gpt-5',
model_kwargs={'max_completion_tokens': 5000},
)
)
```
## udf embeddings()
```python Signature theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
@pxt.udf
embeddings(
input: pxt.String,
*,
model: pxt.String,
api_version: pxt.String = '2024-02-15-preview',
model_kwargs: pxt.Json | None = None
) -> pxt.Array[(None,), float32]
```
Creates an embedding vector representing the input text using Azure OpenAI in Fabric.
Equivalent to the Azure OpenAI `embeddings` API endpoint.
For additional details, see: [https://learn.microsoft.com/en-us/azure/ai-services/openai/reference](https://learn.microsoft.com/en-us/azure/ai-services/openai/reference)
**Automatic authentication:** Authentication is handled automatically in Fabric notebooks using
token-based authentication. No API keys are required.
**Supported models in Fabric:**
* `text-embedding-ada-002`
* `text-embedding-3-small`
* `text-embedding-3-large`
Request throttling:
Applies the rate limit set in the config (section `fabric.rate_limits`, key `embeddings`). If no rate
limit is configured, uses a default of 600 RPM. Batches up to 32 inputs per request for efficiency.
**Requirements:**
* Microsoft Fabric notebook environment
* `synapse-ml-fabric` package (pre-installed in Fabric)
**Parameters:**
* **`input`** (`pxt.String`): The text to embed (automatically batched).
* **`model`** (`pxt.String`): The embedding model deployment name
* **`api_version`** (`pxt.String`): The API version to use (default: '2024-02-15-preview').
* **`model_kwargs`** (`pxt.Json | None`): Additional keyword args for the Azure OpenAI `embeddings` API.
For details on available parameters, see:
[https://learn.microsoft.com/en-us/azure/ai-services/openai/reference](https://learn.microsoft.com/en-us/azure/ai-services/openai/reference)
**Returns:**
* `pxt.Array[(None,), float32]`: An array representing the embedding vector for the input text.
**Examples:**
Add a computed column that applies the model `text-embedding-3-small` to an existing
Pixeltable column `tbl.text` of the table `tbl`:
```python theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
from pixeltable.functions import fabric
tbl.add_computed_column(
embed=fabric.embeddings(tbl.text, model='text-embedding-3-small')
)
```
Add an embedding index to an existing column `text`:
```python theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
tbl.add_embedding_index(
'text',
embedding=fabric.embeddings.using(model='text-embedding-3-large'),
)
```
# fal
Source: https://docs.pixeltable.com/sdk/latest/fal
# module pixeltable.functions.fal
Pixeltable UDFs
that wrap various endpoints from the fal.ai API. In order to use them, you must
first `pip install fal-client` and configure your fal.ai credentials, as described in
the [Working with fal.ai](https://docs.pixeltable.com/howto/providers/working-with-fal) tutorial.
## udf run()
```python Signature theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
@pxt.udf
run(input: pxt.Json, *, app: pxt.String) -> pxt.Json
```
Run a model on fal.ai.
Uses fal's queue-based subscribe mechanism for reliable execution.
For additional details, see: [https://fal.ai/docs](https://fal.ai/docs)
Request throttling:
Applies the rate limit set in the config (section `fal`, key `rate_limit`). If no rate
limit is configured, uses a default of 600 RPM.
**Requirements:**
* `pip install fal-client`
**Parameters:**
* **`input`** (`pxt.Json`): The input parameters for the model.
* **`app`** (`pxt.String`): The name or ID of the fal.ai application to run (e.g., 'fal-ai/flux/schnell').
**Returns:**
* `pxt.Json`: The output of the model as a JSON object.
**Examples:**
Add a computed column that applies the model `fal-ai/flux/schnell`
to an existing Pixeltable column `tbl.prompt` of the table `tbl`:
```python theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
input = {'prompt': tbl.prompt}
tbl.add_computed_column(response=run(input, app='fal-ai/flux/schnell'))
```
Add a computed column that uses the model `fal-ai/fast-sdxl`
to generate images from an existing Pixeltable column `tbl.prompt`:
```python theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
input = {
'prompt': tbl.prompt,
'image_size': 'square',
'num_inference_steps': 25,
}
tbl.add_computed_column(response=run(input, app='fal-ai/fast-sdxl'))
tbl.add_computed_column(
image=tbl.response['images'][0]['url'].astype(pxt.Image)
)
```
# FastAPIRouter
Source: https://docs.pixeltable.com/sdk/latest/fastapirouter
# class pixeltable.serving.FastAPIRouter
A FastAPI `APIRouter` that exposes Pixeltable table operations as HTTP endpoints.
`FastAPIRouter` is for apps that already have a FastAPI server. If you do
not have one, use `pxt service` from the CLI; Pixeltable creates and runs the
FastAPI app for you. Learn more here: [HTTP Serving](https://docs.pixeltable.com/howto/deployment/serving).
## method add\_delete\_route()
```python Signature theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
add_delete_route(
t: RouteTarget,
*,
path: str,
match_columns: ColumnsParam | None = None,
background: bool = False
) -> None
```
Add a POST endpoint that deletes rows from `t` matching the given match column values.
The request body contains the match column values as JSON fields. The endpoint deletes every row
where each match column equals the provided value, and returns the number of rows affected.
**Parameters:**
* **`t`** (`RouteTarget`): The table to delete from, or the model that defines it.
* **`path`** (`str`): The URL path for the endpoint.
* **`match_columns`** (`ColumnsParam | None`): Columns to match on (AND-ed equality). Defaults to the table's primary key.
Must be non-empty.
* **`background`** (`bool`, default: `False`): If True, return immediately with `{"id": ..., "job_url": ...}` and run the
operation in a background thread. Poll `job_url` for the result.
**Examples:**
```python theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
router.add_delete_route(t, path='/delete')
```
```bash theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
curl -X POST http://localhost:8000/delete \
-H 'Content-Type: application/json' \
-d '{"id": 42}'
# {"num_rows": 1}
```
## method add\_insert\_route()
```python Signature theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
add_insert_route(
t: RouteTarget,
*,
path: str,
inputs: ColumnsParam | None = None,
uploadfile_inputs: ColumnsParam | None = None,
outputs: ColumnsParam | None = None,
return_fileresponse: bool = False,
export_sql: SqlExport | None = None,
background: bool = False
) -> None
```
Add a POST endpoint that inserts a single row into `t` and returns the resulting row.
The request body contains the input column values as JSON fields (or as
[multipart form data](https://fastapi.tiangolo.com/tutorial/request-files/) when
`uploadfile_inputs` is used). The response is a JSON object with the output column values,
or a [`FileResponse`](https://fastapi.tiangolo.com/advanced/custom-response/#fileresponse)
when `return_fileresponse=True`.
**Parameters:**
* **`t`** (`RouteTarget`): The table to insert into, or the model that defines it.
* **`path`** (`str`): The URL path for the endpoint.
* **`inputs`** (`ColumnsParam | None`): Columns to accept as request fields. Defaults to all non-computed columns. Together with
`uploadfile_inputs`, they must include every required (non-nullable, non-computed) column.
* **`uploadfile_inputs`** (`ColumnsParam | None`): Columns to accept as
[`UploadFile`](https://fastapi.tiangolo.com/tutorial/request-files/) fields
(must be media-typed). These are sent as multipart form data; all other inputs
become [`Form`](https://fastapi.tiangolo.com/tutorial/request-forms/) fields.
* **`outputs`** (`ColumnsParam | None`): Columns to include in the response. Defaults to all columns (including inputs).
* **`return_fileresponse`** (`bool`, default: `False`): If True, return the single media-typed output column as a
[`FileResponse`](https://fastapi.tiangolo.com/advanced/custom-response/#fileresponse).
Requires exactly one media-typed output column.
* **`export_sql`** (`SqlExport | None`): If set, export each inserted row into an external RDBMS table after the
Pixeltable insert succeeds. See [`SqlExport`](./sqlexport) for
the target specification and supported `method` values.
The row written is the response body: same columns as `outputs`, with media-typed
columns rendered as URL strings (so the corresponding target columns must be
string-typed).
Schema compatibility against the response columns is validated once at
registration time; the target table must already exist or registration fails.
Mutually exclusive with `return_fileresponse`. Compatible with `background=True`
(the SQL write runs in the worker thread).
Note: when paired with `method='update'`, a Pixeltable insert triggers a
target-side update -- this is intentional, supporting the append-only-source /
current-state-view pattern.
If the external write fails after the Pixeltable insert has already succeeded,
the request fails with HTTP 500; no rollback is performed.
* **`background`** (`bool`, default: `False`): If True, return immediately with `{"id": ..., "job_url": ...}` and run
the insert in a background thread. Poll `job_url` for the result. Mutually
exclusive with `return_fileresponse`.
**Examples:**
JSON request/response:
```python theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
router.add_insert_route(
t, path='/generate', inputs=['prompt'], outputs=['result']
)
```
```bash theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
curl -X POST http://localhost:8000/generate \
-H 'Content-Type: application/json' \
-d '{"prompt": "a sunset over the ocean"}'
# {"prompt": "a sunset over the ocean", "result": "..."}
```
File upload with `FileResponse`:
```python theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
router.add_insert_route(
t,
path='/resize',
inputs=['width', 'height'],
uploadfile_inputs=['image'],
outputs=['resized'],
return_fileresponse=True,
)
```
```bash theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
curl -X POST http://localhost:8000/resize \
-F image=@photo.jpg -F width=640 -F height=480 \
--output resized.jpg
# saves the resized image to resized.jpg
```
Export each inserted row into an external RDBMS table:
```python theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
router.add_insert_route(
t,
path='/generate',
inputs=['prompt'],
outputs=['prompt', 'result'],
export_sql=SqlExport(
db_connect='postgresql+psycopg://user:pw@host/analytics',
table='generations',
db_schema='public',
),
)
```
Each successful POST inserts a row into the Pixeltable table and then inserts the
same row (columns: `prompt`, `result`) to `public.generations` in the target database.
Background processing:
```python theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
router.add_insert_route(t, path='/slow', background=True)
```
```bash theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
# submit
curl -X POST http://localhost:8000/slow -d '{"prompt": "hello"}'
# {"id": "abc123", "job_url": "http://localhost:8000/jobs/abc123"}
```
```bash theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
# poll
curl http://localhost:8000/jobs/abc123
# {"status": "done", "result": {...}}
```
## method add\_query\_route()
```python Signature theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
add_query_route(
*,
path: str,
query: pxt.Function,
inputs: ColumnsParam | None = None,
uploadfile_inputs: ColumnsParam | None = None,
one_row: bool = False,
return_fileresponse: bool = False,
background: bool = False,
method: Literal['get', 'post'] = 'post'
) -> None
```
Add an endpoint that executes a `@pxt.query` or `pxt.retrieval_udf` and returns the results.
By default the endpoint accepts POST requests with a JSON
[`Body`](https://fastapi.tiangolo.com/tutorial/body/) and returns `{"rows": [{...}, ...]}`.
Use `method='get'` for
[`Query`](https://fastapi.tiangolo.com/tutorial/query-params/) parameters instead.
**Parameters:**
* **`path`** (`str`): The URL path for the endpoint.
* **`query`** (`pxt.Function`): The query to execute, created with `@pxt.query` or `pxt.retrieval_udf()`.
* **`inputs`** (`ColumnsParam | None`): Parameters to accept as request fields. Defaults to all query parameters.
* **`uploadfile_inputs`** (`ColumnsParam | None`): Parameters to accept as
[`UploadFile`](https://fastapi.tiangolo.com/tutorial/request-files/) fields
(must be media-typed).
* **`one_row`** (`bool`, default: `False`): If True, expect exactly one result row and return it as a plain JSON object
(not wrapped in `{"rows": [...]}`). 0 rows produces a 404, >1 rows a 409.
* **`return_fileresponse`** (`bool`, default: `False`): If True, return the single media-typed result column as a
[`FileResponse`](https://fastapi.tiangolo.com/advanced/custom-response/#fileresponse).
Requires `one_row` semantics (0 rows -> 404, >1 rows -> 409).
Mutually exclusive with `background`.
* **`background`** (`bool`, default: `False`): If True, return immediately with `{"id": ..., "job_url": ...}` and run
the query in a background thread. Poll `job_url` for the result. Mutually
exclusive with `return_fileresponse`.
* **`method`** (`Literal['get', 'post']`, default: `'post'`): HTTP method for the endpoint (`'get'` or `'post'`).
**Examples:**
Multi-row JSON response:
```python theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
router.add_query_route(path='/search', query=search_docs)
```
```bash theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
curl -X POST http://localhost:8000/search \
-H 'Content-Type: application/json' \
-d '{"query_text": "hello"}'
# {"rows": [{"id": 1, "text": "hello world", "score": 0.95}, ...]}
```
Single-row lookup:
```python theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
router.add_query_route(path='/lookup', query=lookup_by_id, one_row=True)
```
```bash theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
curl -X POST http://localhost:8000/lookup -d '{"id": 42}'
# {"id": 42, "name": "Alice", "email": "alice@example.com"}
```
GET with query-string parameters:
```python theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
router.add_query_route(path='/lookup', query=lookup_by_id, method='get')
```
```bash theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
curl 'http://localhost:8000/lookup?id=42'
# {"id": 42, "name": "Alice", "email": "alice@example.com"}
```
`FileResponse`:
```python theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
router.add_query_route(
path='/thumbnail', query=get_thumbnail, return_fileresponse=True
)
```
```bash theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
curl -X POST http://localhost:8000/thumbnail -d '{"id": 1}' --output thumb.jpg
# saves the thumbnail image to thumb.jpg
```
## method add\_update\_route()
```python Signature theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
add_update_route(
t: RouteTarget,
*,
path: str,
inputs: ColumnsParam | None = None,
outputs: ColumnsParam | None = None,
return_fileresponse: bool = False,
export_sql: SqlExport | None = None,
background: bool = False
) -> None
```
Add a POST endpoint that updates a single row in `t` and returns the updated row.
The row to update is identified by its primary key, which must be included in the
request body alongside the input column values. The update is performed via a
single-row [`batch_update()`](./table#method-batch_update) call, using the primary
key columns to identify the row and the columns referenced in `inputs` as the values
to set.
The request body contains values for the primary key columns plus the input columns
as JSON fields. The response is a JSON object with the output column values, or a
[`FileResponse`](https://fastapi.tiangolo.com/advanced/custom-response/#fileresponse)
when `return_fileresponse=True`.
Note: media-typed columns (image, video, audio, document) are excluded from `inputs`
and from the default input set.
**Parameters:**
* **`t`** (`RouteTarget`): The table to update, or the model that defines it.
* **`path`** (`str`): The URL path for the endpoint.
* **`inputs`** (`ColumnsParam | None`): Columns to accept as request fields, excluding primary key and media-typed
columns (which cannot be updated). Defaults to all non-computed, non-primary-key,
non-media columns.
* **`outputs`** (`ColumnsParam | None`): Columns to include in the response. Defaults to all columns (including
inputs).
* **`return_fileresponse`** (`bool`, default: `False`): If True, return the single media-typed output column as a
[`FileResponse`](https://fastapi.tiangolo.com/advanced/custom-response/#fileresponse).
Requires exactly one media-typed output column.
* **`export_sql`** (`SqlExport | None`): If set, export each updated row into an external RDBMS table after the
Pixeltable update succeeds. See [`SqlExport`](./sqlexport) for
the target specification and supported `method` values.
The row written is the response body: same columns as `outputs`, with media-typed
columns rendered as URL strings (so the corresponding target columns must be
string-typed).
Schema compatibility is validated once at registration time; the target table
must already exist or registration fails. Mutually exclusive with
`return_fileresponse`. Compatible with `background=True`.
Note: with `method='insert'` (the default), every update appends a new row to the
target table -- the target acts as an audit log, not a current-state view. Use
`method='update'` to keep the target as a current-state view keyed on the
target's primary key.
If the external write fails after the Pixeltable update has already succeeded,
the request fails with HTTP 500; no rollback is performed.
* **`background`** (`bool`, default: `False`): If True, return immediately with `{"id": ..., "job_url": ...}` and run
the update in a background thread. Poll `job_url` for the result. Mutually
exclusive with `return_fileresponse`.
**Examples:**
JSON request/response (table has primary key `id`):
```python theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
router.add_update_route(
t, path='/update', inputs=['prompt'], outputs=['prompt', 'result']
)
```
```bash theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
curl -X POST http://localhost:8000/update \
-H 'Content-Type: application/json' \
-d '{"id": 1, "prompt": "a sunset over the ocean"}'
# {"prompt": "a sunset over the ocean", "result": "..."}
```
Append every update to an external audit table:
```python theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
router.add_update_route(
t,
path='/update',
inputs=['prompt'],
outputs=['id', 'prompt', 'result'],
export_sql=SqlExport(
db_connect='postgresql+psycopg://user:pw@host/analytics',
table='update_log',
),
)
```
Background processing:
```python theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
router.add_update_route(t, path='/slow-update', background=True)
```
```bash theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
# submit
curl -X POST http://localhost:8000/slow-update \
-H 'Content-Type: application/json' \
-d '{"id": 1, "prompt": "hello"}'
# {"id": "abc123", "job_url": "http://localhost:8000/jobs/abc123"}
# poll
curl http://localhost:8000/jobs/abc123
# {"status": "done", "result": {...}}
```
## method insert\_route()
```python Signature theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
insert_route(
t: RouteTarget,
*,
path: str,
inputs: ColumnsParam | None = None,
uploadfile_inputs: ColumnsParam | None = None,
outputs: ColumnsParam | None = None,
export_sql: SqlExport | None = None,
background: bool = False
) -> Callable[[Callable[..., pydantic.BaseModel]], Callable[..., pydantic.BaseModel]]
```
Decorator that registers a POST endpoint performing a `Table.insert()` followed by user-defined post-processing.
The request body carries the input column values (JSON, or multipart form data when `uploadfile_inputs` is
used). After inserting the row, the decorated function is called with the requested output columns as
keyword arguments (parameter names and Pixeltable types must match `outputs`). Its return value must be a
Pydantic model and is returned as the HTTP response body.
Media-typed outputs (image, video, audio, document) are delivered to the function as `/media/` URL
strings -- annotate those parameters as `str` (or `str | None` if the column is nullable), not as
`pxt.Image` / `pxt.Video` / etc.
**Parameters:**
* **`t`** (`RouteTarget`): The table to insert into, or the model that defines it.
* **`path`** (`str`): The URL path for the endpoint.
* **`inputs`** (`ColumnsParam | None`): Columns to accept as request fields. Defaults to all non-computed columns. Together with
`uploadfile_inputs`, they must include every required (non-nullable, non-computed) column.
* **`uploadfile_inputs`** (`ColumnsParam | None`): Columns to accept as
[`UploadFile`](https://fastapi.tiangolo.com/tutorial/request-files/) fields
(must be media-typed). These are sent as multipart form data; all other inputs
become [`Form`](https://fastapi.tiangolo.com/tutorial/request-forms/) fields.
* **`outputs`** (`ColumnsParam | None`): Columns from the inserted row to pass to the decorated function as keyword
arguments. Defaults to all columns.
* **`export_sql`** (`SqlExport | None`): If set, export the decorated function's return value into an external
RDBMS table after the Pixeltable insert succeeds. See
[`SqlExport`](./sqlexport) for the target specification and
supported `method` values.
The row written is the user function's pydantic return value (its fields, not the
source columns), so the target table schema must match those fields. Media-typed
fields are modeled as strings (URL form).
Schema compatibility is validated once at registration time; the target table
must already exist or registration fails. Compatible with `background=True` (the
SQL write runs in the worker thread).
If the external write fails after the Pixeltable insert has already succeeded,
the request fails with HTTP 500; no rollback is performed.
* **`background`** (`bool`, default: `False`): If True, return immediately with `{"id": ..., "job_url": ...}` and run
the insert plus post-processing in a background thread. Poll `job_url` for the
result; the decorated function's return value is delivered as the job result.
**Examples:**
```python theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
class GenerateResponse(pydantic.BaseModel):
caption: str
score: float
@router.insert_route(
t,
path='/generate',
inputs=['prompt'],
outputs=['caption', 'score'],
background=False,
)
def format_response(*, caption: str, score: float) -> GenerateResponse:
return GenerateResponse(
caption=caption.strip(), score=round(score, 3)
)
```
```bash theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
curl -X POST http://localhost:8000/generate \
-H 'Content-Type: application/json' \
-d '{"prompt": "a sunset over the ocean"}'
# {"caption": "orange sky above calm water", "score": 0.932}
```
Export the post-processed response into an external RDBMS table:
```python theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
@router.insert_route(
t,
path='/generate',
inputs=['prompt'],
outputs=['caption', 'score'],
export_sql=SqlExport(
db_connect='postgresql+psycopg://user:pw@host/analytics',
table='captions',
),
)
def format_response(*, caption: str, score: float) -> GenerateResponse:
return GenerateResponse(
caption=caption.strip(), score=round(score, 3)
)
```
Each successful POST inserts a row into the Pixeltable table and then appends a row
with columns `caption`, `score` (the response model fields) to `captions`.
## method update\_route()
```python Signature theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
update_route(
t: RouteTarget,
*,
path: str,
inputs: ColumnsParam | None = None,
outputs: ColumnsParam | None = None,
export_sql: SqlExport | None = None,
background: bool = False
) -> Callable[[Callable[..., pydantic.BaseModel]], Callable[..., pydantic.BaseModel]]
```
Decorator that registers a POST endpoint performing a `Table.batch_update()` followed by
user-defined post-processing.
The request body carries values for the primary key columns (to identify the row) plus the
input column values as JSON fields. After updating the row, the decorated function is called
with the requested output columns as keyword arguments (parameter names and Pixeltable types
must match `outputs`). Its return value must be a Pydantic model and is returned as the HTTP
response body.
Media-typed outputs (image, video, audio, document) are delivered to the function as `/media/` URL
strings -- annotate those parameters as `str` (or `str | None` if the column is nullable), not as
`pxt.Image` / `pxt.Video` / etc.
If the row does not exist, the endpoint returns HTTP 404 without calling the decorated
function.
Note: media-typed columns (image, video, audio, document) and primary key columns cannot be
used as `inputs`. Primary key columns are always part of the request body for row
identification.
**Parameters:**
* **`t`** (`RouteTarget`): The table to update, or the model that defines it.
* **`path`** (`str`): The URL path for the endpoint.
* **`inputs`** (`ColumnsParam | None`): Columns to accept as update fields. Defaults to all non-computed, non-primary-key,
non-media columns.
* **`outputs`** (`ColumnsParam | None`): Columns from the updated row to pass to the decorated function as keyword
arguments. Defaults to all columns.
* **`export_sql`** (`SqlExport | None`): If set, export the decorated function's return value into an external
RDBMS table after the Pixeltable update succeeds. See
[`SqlExport`](./sqlexport) for the target specification and
supported `method` values.
The row written is the user function's pydantic return value (its fields, not the
source columns), so the target table schema must match those fields. Media-typed
fields are modeled as strings (URL form).
Schema compatibility is validated once at registration time; the target table
must already exist or registration fails. Compatible with `background=True`.
Note: with `method='insert'` (the default), every update appends a new row to the
target table -- the target acts as an audit log, not a current-state view. Use
`method='update'` to keep the target as a current-state view keyed on the
target's primary key.
If the external write fails after the Pixeltable update has already succeeded,
the request fails with HTTP 500; no rollback is performed.
* **`background`** (`bool`, default: `False`): If True, return immediately with `{"id": ..., "job_url": ...}` and run the
update plus post-processing in a background thread. Poll `job_url` for the result;
the decorated function's return value is delivered as the job result.
**Examples:**
```python theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
class ItemResponse(pydantic.BaseModel):
id: int
summary: str
score: float
@router.update_route(
t, path='/update', inputs=['text'], outputs=['id', 'text', 'score']
)
def format_response(*, id: int, text: str, score: float) -> ItemResponse:
return ItemResponse(id=id, summary=text[:100], score=round(score, 3))
```
```bash theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
curl -X POST http://localhost:8000/update \
-H 'Content-Type: application/json' \
-d '{"id": 42, "text": "new content"}'
# {"id": 42, "summary": "new content", "score": 0.871}
```
Append every post-processed update into an external audit table:
```python theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
@router.update_route(
t,
path='/update',
inputs=['text'],
outputs=['id', 'text', 'score'],
export_sql=SqlExport(
db_connect='postgresql+psycopg://user:pw@host/analytics',
table='item_log',
),
)
def format_response(*, id: int, text: str, score: float) -> ItemResponse:
return ItemResponse(id=id, summary=text[:100], score=round(score, 3))
```
Background processing:
```python theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
@router.update_route(t, path='/slow-update', background=True)
def format_response(*, id: int, result: str) -> MyResponse:
return MyResponse(id=id, result=result.strip())
```
```bash theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
# submit
curl -X POST http://localhost:8000/slow-update \
-H 'Content-Type: application/json' \
-d '{"id": 1, "text": "hello"}'
# {"id": "abc123", "job_url": "http://localhost:8000/jobs/abc123"}
# poll
curl http://localhost:8000/jobs/abc123
# {"status": "done", "result": {"id": 1, "result": "hello"}}
```
# fireworks
Source: https://docs.pixeltable.com/sdk/latest/fireworks
# module pixeltable.functions.fireworks
Pixeltable UDFs
that wrap various endpoints from the Fireworks AI API. In order to use them, you must
first `pip install fireworks-ai` and configure your Fireworks AI credentials, as described in
the [Working with Fireworks](https://docs.pixeltable.com/howto/providers/working-with-fireworks) tutorial.
## udf chat\_completions()
```python Signature theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
@pxt.udf
chat_completions(
messages: pxt.Json[(Json, ...)],
*,
model: pxt.String,
model_kwargs: pxt.Json | None = None
) -> pxt.Json
```
Creates a model response for the given chat conversation.
Equivalent to the Fireworks AI `chat/completions` API endpoint.
For additional details, see: [https://docs.fireworks.ai/api-reference/post-chatcompletions](https://docs.fireworks.ai/api-reference/post-chatcompletions)
Request throttling:
Applies the rate limit set in the config (section `fireworks`, key `rate_limit`). If no rate
limit is configured, uses a default of 600 RPM.
**Requirements:**
* `pip install fireworks-ai`
**Parameters:**
* **`messages`** (`pxt.Json[(Json`): A list of messages comprising the conversation so far.
* **`model`** (`Any`): The name of the model to use.
* **`model_kwargs`** (`Any`): Additional keyword args for the Fireworks `chat_completions` API. For details on the available
parameters, see: [https://docs.fireworks.ai/api-reference/post-chatcompletions](https://docs.fireworks.ai/api-reference/post-chatcompletions)
**Returns:**
* `pxt.Json`: A dictionary containing the response and other metadata.
**Examples:**
Add a computed column that applies the model `accounts/fireworks/models/nemotron-lightning-3p5-30b-a3b`
to an existing Pixeltable column `tbl.prompt` of the table `tbl`:
```python theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
messages = [{'role': 'user', 'content': tbl.prompt}]
tbl.add_computed_column(
response=chat_completions(
messages,
model='accounts/fireworks/models/nemotron-lightning-3p5-30b-a3b',
)
)
```
# functions
Source: https://docs.pixeltable.com/sdk/latest/functions
# module pixeltable.functions
General Pixeltable UDFs.
This parent module contains general-purpose UDFs that apply to multiple data types.
## func filter()
```python Signature theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
filter(
expr: pixeltable.exprs.expr.Expr,
predicate: Callable[[pixeltable.exprs.expr.Expr], Any]
) -> pixeltable.exprs.expr.Expr
```
Keeps the elements of a JSON array for which a predicate holds, producing a new array.
`filter()` is used like a UDF, for example in `select()` or `add_computed_column()`.
**Parameters:**
* **`expr`** (`pixeltable.exprs.expr.Expr`): The array to filter; an expression of type `pxt.Json` that resolves to a JSON array. Its element
type is preserved in the result.
* **`predicate`** (`typing.Callable[[pixeltable.exprs.expr.Expr], typing.Any]`): A Python function (typically a lambda) that decides which elements to keep. It receives `x`, a
stand-in for a single array element, and returns a boolean condition; the element is kept when it is
true. Operate on `x` exactly as you would on a column: comparisons, indexing (`x[0]`), field access
(`x.field`), and JSON methods all work. Combine multiple conditions with `&` and `|`, and negate with
`~` (not Python `and`/`or`/`not`). Example: `lambda x: (x > 0) & (x < 10)`.
**Returns:**
* `pixeltable.exprs.expr.Expr`: A new array containing the elements of `expr` for which `predicate` is true, unchanged. If `expr` is
`null` or does not resolve to a JSON array, the result is `null`.
**Examples:**
Given a table `tbl` with a `pxt.Json` column `data` holding lists of numbers, add a column that keeps only
the positive numbers:
```python theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
tbl.add_computed_column(
positives=pxt.functions.filter(tbl.data, lambda x: x > 0)
)
```
When `data` holds lists of objects such as `{'score': 0.9, 'label': 'cat'}`, keep only the high-confidence
ones:
```python theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
tbl.select(
confident=pxt.functions.filter(tbl.data, lambda x: x.score >= 0.9)
).collect()
```
## func map()
```python Signature theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
map(
expr: pixeltable.exprs.expr.Expr,
fn: Callable[[pixeltable.exprs.expr.Expr], Any]
) -> pixeltable.exprs.expr.Expr
```
Applies a function to each element of a JSON array, producing a new array.
`map()` is used like a UDF, for example in `select()` or `add_computed_column()`.
**Parameters:**
* **`expr`** (`pixeltable.exprs.expr.Expr`): The array to map over; an expression of type `pxt.Json` that resolves to a JSON array. If its
elements have a known type (e.g. the column is declared `pxt.Json[[int]]`), that type is available to
`fn` and carries through to the result.
* **`fn`** (`typing.Callable[[pixeltable.exprs.expr.Expr], typing.Any]`): A Python function (typically a lambda) applied to each element to produce its replacement. It receives
`x`, a stand-in for a single array element, and returns the value to store in its place. Operate on
`x` exactly as you would on a column: arithmetic, indexing (`x[0]`), field access (`x.field`), and
JSON methods (`x.len()`, `x.sum()`, etc.) all work.
**Returns:**
* `pixeltable.exprs.expr.Expr`: A new array holding `fn(x)` for each element `x` of `expr`. If `expr` is `null` or does not resolve to a
JSON array, the result is `null`.
**Examples:**
Given a table `tbl` with a `pxt.Json` column `data` holding lists of numbers, add a column that doubles
each number:
```python theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
tbl.add_computed_column(
doubled=pxt.functions.map(tbl.data, lambda x: x * 2)
)
```
When `data` holds lists of objects such as `{'score': 0.9, 'label': 'cat'}`, extract each score:
```python theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
tbl.select(
scores=pxt.functions.map(tbl.data, lambda x: x.score)
).collect()
```
## func sort()
```python Signature theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
sort(
expr: pixeltable.exprs.expr.Expr,
key: Optional[Callable[[pixeltable.exprs.expr.Expr], Any]] = None,
*,
asc: bool = True
) -> pixeltable.exprs.expr.Expr
```
Sorts the elements of a JSON array, producing a new array.
`sort()` is used like a UDF, for example in `select()` or `add_computed_column()`.
**Parameters:**
* **`expr`** (`pixeltable.exprs.expr.Expr`): The array to sort; an expression of type `pxt.Json` that resolves to a JSON array. Its element type
is preserved in the result.
* **`key`** (`typing.Optional[typing.Callable[[pixeltable.exprs.expr.Expr], typing.Any]]`): An optional Python function (typically a lambda) that produces the value each element is ordered by.
It receives `x`, a stand-in for a single array element, and returns the sort key. Operate on `x`
exactly as you would on a column: arithmetic, indexing (`x[0]`), field access (`x.field`), and JSON
methods all work. When `key` is omitted, the elements are ordered by their own natural ordering.
* **`asc`** (`bool`, default: `True`): Whether to sort in ascending (the default) or descending order.
**Returns:**
* `pixeltable.exprs.expr.Expr`: A new array with the elements of `expr` in sorted order. If `expr` is `null` or does not resolve to a JSON
array, the result is `null`. Sorting a list of scalars without a `key`, or by non-orderable keys, raises if
the values are not mutually comparable (matching Python's `sorted()`).
**Examples:**
Given a table `tbl` with a `pxt.Json` column `data` holding lists of numbers, add a column that sorts each
list in descending order:
```python theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
tbl.add_computed_column(ranked=pxt.functions.sort(tbl.data, asc=False))
```
When `data` holds lists of objects such as `{'score': 0.9, 'label': 'cat'}`, sort each list by score:
```python theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
tbl.select(
by_score=pxt.functions.sort(
tbl.data, key=lambda x: x.score, asc=False
)
).collect()
```
## uda count()
```python Signatures theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
# Signature 1:
@pxt.uda
count(val: pxt.String | None) -> pxt.Int
# Signature 2:
@pxt.uda
count(val: pxt.Bool | None) -> pxt.Int
# Signature 3:
@pxt.uda
count(val: pxt.Int | None) -> pxt.Int
# Signature 4:
@pxt.uda
count(val: pxt.Float | None) -> pxt.Int
# Signature 5:
@pxt.uda
count(val: pxt.Timestamp | None) -> pxt.Int
# Signature 6:
@pxt.uda
count(val: pxt.Json | None) -> pxt.Int
# Signature 7:
@pxt.uda
count(val: pxt.Array | None) -> pxt.Int
# Signature 8:
@pxt.uda
count(val: pxt.Image | None) -> pxt.Int
# Signature 9:
@pxt.uda
count(val: pxt.Video | None) -> pxt.Int
# Signature 10:
@pxt.uda
count(val: pxt.Audio | None) -> pxt.Int
# Signature 11:
@pxt.uda
count(val: pxt.Document | None) -> pxt.Int
# Signature 12:
@pxt.uda
count(val: pxt.Date | None) -> pxt.Int
# Signature 13:
@pxt.uda
count(val: pxt.UUID | None) -> pxt.Int
# Signature 14:
@pxt.uda
count(val: pxt.Binary | None) -> pxt.Int
```
Aggregate function that counts the number of non-null values in a column or grouping.
**Parameters:**
* **`val`** (`String | None`): The value to count.
**Returns:**
* `pxt.Int`: The count of non-null values.
**Examples:**
Count the number of non-null values in the `value` column of the table `tbl`:
```python theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
tbl.select(pxt.functions.count(tbl.value)).collect()
```
Group by the `category` column and compute the count of non-null values in the `value` column
for each category, assigning the name `'category_count'` to the new column:
```python theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
tbl.group_by(tbl.category).select(
tbl.category, category_count=pxt.functions.count(tbl.value)
).collect()
```
## uda max()
```python Signatures theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
# Signature 1:
@pxt.uda
max(val: pxt.String | None) -> pxt.String | None
# Signature 2:
@pxt.uda
max(val: pxt.Int | None) -> pxt.Int | None
# Signature 3:
@pxt.uda
max(val: pxt.Float | None) -> pxt.Float | None
# Signature 4:
@pxt.uda
max(val: pxt.Bool | None) -> pxt.Bool | None
# Signature 5:
@pxt.uda
max(val: pxt.Timestamp | None) -> pxt.Timestamp | None
```
Aggregate function that computes the maximum value in a column or grouping.
**Parameters:**
* **`val`** (`String | None`): The value to compare.
**Returns:**
* `pxt.String | None`: The maximum value, or `None` if there are no non-null values.
**Examples:**
Compute the maximum value in the `value` column of the table `tbl`:
```python theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
tbl.select(pxt.functions.max(tbl.value)).collect()
```
Group by the `category` column and compute the maximum value in the `value` column for each category,
assigning the name `'category_max'` to the new column:
```python theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
tbl.group_by(tbl.category).select(
tbl.category, category_max=pxt.functions.max(tbl.value)
).collect()
```
## uda mean()
```python Signatures theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
# Signature 1:
@pxt.uda
mean(val: pxt.Int | None) -> pxt.Float | None
# Signature 2:
@pxt.uda
mean(val: pxt.Float | None) -> pxt.Float | None
```
Aggregate function that computes the mean (average) of non-null values of a numeric column or grouping.
**Parameters:**
* **`val`** (`Int | None`): The numeric value to include in the mean.
**Returns:**
* `pxt.Float | None`: The mean of the non-null values, or `None` if there are no non-null values.
**Examples:**
Compute the mean of the values in the `value` column of the table `tbl`:
```python theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
tbl.select(pxt.functions.mean(tbl.value)).collect()
```
Group by the `category` column and compute the mean of the `value` column for each category,
assigning the name `'category_mean'` to the new column:
```python theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
tbl.group_by(tbl.category).select(
tbl.category, category_mean=pxt.functions.mean(tbl.value)
).collect()
```
## uda min()
```python Signatures theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
# Signature 1:
@pxt.uda
min(val: pxt.String | None) -> pxt.String | None
# Signature 2:
@pxt.uda
min(val: pxt.Int | None) -> pxt.Int | None
# Signature 3:
@pxt.uda
min(val: pxt.Float | None) -> pxt.Float | None
# Signature 4:
@pxt.uda
min(val: pxt.Bool | None) -> pxt.Bool | None
# Signature 5:
@pxt.uda
min(val: pxt.Timestamp | None) -> pxt.Timestamp | None
```
Aggregate function that computes the minimum value in a column or grouping.
**Parameters:**
* **`val`** (`String | None`): The value to compare.
**Returns:**
* `pxt.String | None`: The minimum value, or `None` if there are no non-null values.
**Examples:**
Compute the minimum value in the `value` column of the table `tbl`:
```python theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
tbl.select(pxt.functions.min(tbl.value)).collect()
```
Group by the `category` column and compute the minimum value in the `value` column for each category,
assigning the name `'category_min'` to the new column:
```python theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
tbl.group_by(tbl.category).select(
tbl.category, category_min=pxt.functions.min(tbl.value)
).collect()
```
## uda sum()
```python Signatures theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
# Signature 1:
@pxt.uda
sum(val: pxt.Int | None) -> pxt.Int | None
# Signature 2:
@pxt.uda
sum(val: pxt.Float | None) -> pxt.Float | None
```
Aggregate function that computes the sum of non-null values of a numeric column or grouping.
**Parameters:**
* **`val`** (`Int | None`): The numeric value to add to the sum.
**Returns:**
* `pxt.Int | None`: The sum of the non-null values, or `None` if there are no non-null values.
**Examples:**
Sum the values in the `value` column of the table `tbl`:
```python theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
tbl.select(pxt.functions.sum(tbl.value)).collect()
```
Group by the `category` column and compute the sum of the `value` column for each category,
assigning the name `'category_total'` to the new column:
```python theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
tbl.group_by(tbl.category).select(
tbl.category, category_total=pxt.functions.sum(tbl.value)
).collect()
```
# gemini
Source: https://docs.pixeltable.com/sdk/latest/gemini
# module pixeltable.functions.gemini
Pixeltable UDFs
that wrap various endpoints from the Google Gemini API. In order to use them, you must
first `pip install google-genai` and configure your Gemini credentials, as described in
the [Working with Gemini](https://docs.pixeltable.com/howto/providers/working-with-gemini) tutorial.
Supports two authentication methods:
* Google AI Studio: set `GOOGLE_API_KEY` or `GEMINI_API_KEY` (or put `api_key` in the `gemini` section of
the Pixeltable config file).
* Vertex AI: set `GOOGLE_GENAI_USE_VERTEXAI=true` and `GOOGLE_CLOUD_PROJECT` (and optionally
`GOOGLE_CLOUD_LOCATION`), then authenticate via Application Default Credentials
(e.g. `gcloud auth application-default login`).
## func invoke\_tools()
```python Signature theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
invoke_tools(
tools: pixeltable.func.tools.Tools,
response: pixeltable.exprs.expr.Expr
) -> pixeltable.exprs.inline_expr.InlineDict
```
Converts an OpenAI response dict to Pixeltable tool invocation format and calls `tools._invoke()`.
## udf embed\_content()
```python Signatures theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
# Signature 1:
@pxt.udf
embed_content(
contents: pxt.String,
*,
model: pxt.String,
config: pxt.Json | None
) -> pxt.Array[(None,), float32]
# Signature 2:
@pxt.udf
embed_content(
contents: pxt.Image,
*,
model: pxt.String,
config: pxt.Json | None
) -> pxt.Array[(None,), float32]
# Signature 3:
@pxt.udf
embed_content(
contents: pxt.Audio,
*,
model: pxt.String,
config: pxt.Json | None
) -> pxt.Array[(None,), float32]
# Signature 4:
@pxt.udf
embed_content(
contents: pxt.Video,
*,
model: pxt.String,
config: pxt.Json | None
) -> pxt.Array[(None,), float32]
# Signature 5:
@pxt.udf
embed_content(
contents: pxt.Document,
*,
model: pxt.String,
config: pxt.Json | None
) -> pxt.Array[(None,), float32]
```
Generate embeddings for text, images, video, and other content. For more information on Gemini embeddings API, see:
[https://ai.google.dev/gemini-api/docs/embeddings](https://ai.google.dev/gemini-api/docs/embeddings)
**Requirements:**
* `pip install google-genai`
**Parameters:**
* **`contents`** (`String`): The string, image, audio, video, or document to embed.
* **`model`** (`String`): The Gemini model to use.
* **`config`** (`Json | None`, default: `Literal(None)`): Configuration for embedding generation, corresponding to keyword arguments of
`genai.types.EmbedContentConfig`. For details on the parameters, see:
[https://googleapis.github.io/python-genai/genai.html#genai.types.EmbedContentConfig](https://googleapis.github.io/python-genai/genai.html#genai.types.EmbedContentConfig)
**Returns:**
* `pxt.Array[(None,), float32]`: The corresponding embedding vector.
**Examples:**
Add a computed column with embeddings to an existing table with a `text` column:
```python theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
t.add_computed_column(
embedding=embed_content(t.text, model='gemini-embedding-2')
)
```
Add an embedding index on `text` column:
```python theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
t.add_embedding_index(
t.text, embedding=embed_content.using(model='gemini-embedding-2')
)
```
## udf generate\_content()
```python Signature theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
@pxt.udf
generate_content(
contents: pxt.Json,
*,
model: pxt.String,
config: pxt.Json | None = None,
tools: pxt.Json[(Json, ...)] | None = None
) -> pxt.Json
```
Generate content from the specified model.
Request throttling:
Applies the rate limit set in the config (section `gemini.rate_limits`; use the model id as the key). If no rate
limit is configured, uses a default of 600 RPM.
**Requirements:**
* `pip install google-genai`
**Parameters:**
* **`contents`** (`pxt.Json`): The input content to generate from. Can be a prompt, or a list containing images and text
prompts, as described in: [https://ai.google.dev/gemini-api/docs/text-generation](https://ai.google.dev/gemini-api/docs/text-generation) and
[https://ai.google.dev/gemini-api/docs/image-generation](https://ai.google.dev/gemini-api/docs/image-generation) for image generation.
* **`model`** (`pxt.String`): The name of the model to use.
* **`config`** (`pxt.Json | None`): Configuration for generation, corresponding to keyword arguments of
`genai.types.GenerateContentConfig`. For details on the parameters, see:
[https://googleapis.github.io/python-genai/genai.html#genai.types.GenerateContentConfig](https://googleapis.github.io/python-genai/genai.html#genai.types.GenerateContentConfig)
* **`tools`** (`pxt.Json[(Json`): An optional list of Pixeltable tools to use. It is also possible to specify tools manually via the
`config['tools']` parameter, but at most one of `config['tools']` or `tools` may be used.
**Returns:**
* `pxt.Json`: A dictionary containing the response and other metadata.
**Examples:**
Add a computed column that applies the model `gemini-2.5-flash`
to an existing Pixeltable column `tbl.prompt` of the table `tbl`:
```python theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
tbl.add_computed_column(
response=generate_content(tbl.prompt, model='gemini-2.5-flash')
)
```
Generate an image with a Nano Banana model (Gemini image-generation models such as
`gemini-3.1-flash-image-preview`) and extract the PIL image from the response using JSON
subscripting. Image bytes in `inline_data.data` are decoded into PIL images
automatically. Pass `response_modalities=['IMAGE']` so the response contains a
single image part:
```python theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
tbl.add_computed_column(
response=generate_content(
tbl.prompt,
model='gemini-3.1-flash-image-preview',
config={'response_modalities': ['IMAGE']},
)
)
tbl.add_computed_column(
image=tbl.response.candidates[0].content.parts[0].inline_data.data
)
```
## udf generate\_images()
```python Signature theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
@pxt.udf
generate_images(
prompt: pxt.String,
*,
model: pxt.String,
config: pxt.Json | None = None
) -> pxt.Image
```
Generates images based on a text description and configuration. For additional details, see:
[https://ai.google.dev/gemini-api/docs/imagen](https://ai.google.dev/gemini-api/docs/imagen)
Note: This function is for Imagen models only. For Gemini image-generation models (Nano Banana,
e.g. `gemini-3.1-flash-image-preview`), use [`generate_content`](./gemini#func-generate_content)
instead. Imagen requires Vertex AI mode; `google-genai` 2.x no longer reaches it through the Developer API.
Request throttling:
Applies the rate limit set in the config (section `imagen.rate_limits`; use the model id as the key). If no rate
limit is configured, uses a default of 600 RPM.
**Requirements:**
* `pip install google-genai`
**Parameters:**
* **`prompt`** (`pxt.String`): A text description of the images to generate.
* **`model`** (`pxt.String`): The model to use.
* **`config`** (`pxt.Json | None`): Configuration for generation, corresponding to keyword arguments of
`genai.types.GenerateImagesConfig`. For details on the parameters, see:
[https://googleapis.github.io/python-genai/genai.html#genai.types.GenerateImagesConfig](https://googleapis.github.io/python-genai/genai.html#genai.types.GenerateImagesConfig)
**Returns:**
* `pxt.Image`: The generated image.
**Examples:**
Add a computed column that applies the model `imagen-4.0-generate-001`
to an existing Pixeltable column `tbl.prompt` of the table `tbl`:
```python theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
tbl.add_computed_column(
response=generate_images(tbl.prompt, model='imagen-4.0-generate-001')
)
```
## udf generate\_speech()
```python Signatures theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
# Signature 1:
@pxt.udf
generate_speech(
text: pxt.String,
*,
model: pxt.String,
voice: pxt.String,
config: pxt.Json | None
) -> pxt.Audio
# Signature 2:
@pxt.udf
generate_speech(
text: pxt.String,
*,
model: pxt.String,
voices: pxt.Json,
config: pxt.Json | None
) -> pxt.Audio
```
Generates speech audio from text using Gemini's text-to-speech capability. For additional details, see:
[https://ai.google.dev/gemini-api/docs/speech-generation](https://ai.google.dev/gemini-api/docs/speech-generation)
Request throttling:
Applies the rate limit set in the config (section `gemini.rate_limits`; use the model id as the key). If no rate
limit is configured, uses a default of 600 RPM.
**Requirements:**
* `pip install google-genai`
**Parameters:**
* **`text`** (`String`): The text to synthesize into speech.
* **`model`** (`String`): The model to use (e.g. `'gemini-2.5-flash-preview-tts'`).
* **`voice`** (`String`): The voice profile to use. Supported voices include `'Kore'`, `'Puck'`, `'Charon'`,
`'Fenrir'`, `'Aoede'`, `'Leda'`, `'Orus'`, `'Zephyr'`, and others. See the
[speech generation docs](https://ai.google.dev/gemini-api/docs/speech-generation) for the full list.
Mutually exclusive with `voices`.
* **`voices`** (`Json`): A mapping from speaker alias (as used in the text) to voice name. For example,
`{'Alice': 'Kore', 'Bob': 'Puck'}`. Mutually exclusive with `voice`.
* **`config`** (`Json | None`, default: `Literal(None)`): Additional configuration, corresponding to keyword arguments of
`genai.types.GenerateContentConfig`. Keys such as `response_modalities` and `speech_config`
are set automatically and should not be included.
**Returns:**
* `pxt.Audio`: An audio file (WAV, 24 kHz mono 16-bit) containing the synthesized speech.
**Examples:**
Add a computed column that generates speech from text:
```python theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
tbl.add_computed_column(
audio=generate_speech(
tbl.text, model='gemini-2.5-flash-preview-tts', voice='Kore'
)
)
```
## udf generate\_videos()
```python Signatures theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
# Signature 1:
@pxt.udf
generate_videos(
prompt: pxt.String | None,
image: pxt.Image | None,
*,
model: pxt.String,
config: pxt.Json | None
) -> pxt.Video
# Signature 2:
@pxt.udf
generate_videos(
prompt: pxt.String | None,
images: pxt.Json[(Image, ...)] | None,
*,
model: pxt.String,
config: pxt.Json | None,
reference_types: pxt.Json[(String, ...)] | None
) -> pxt.Video
```
Generates videos based on a text description and configuration. For additional details, see:
[https://ai.google.dev/gemini-api/docs/video](https://ai.google.dev/gemini-api/docs/video)
At least one of `prompt` or `image` must be provided. When `image` is a single image, it is used as the first
frame of the generated video. When `image` is a list of images, they are used as reference images to guide the
style or asset appearance throughout the video (Veo 3.1+). See the overloaded signature for details.
Request throttling:
Applies the rate limit set in the config (section `veo.rate_limits`; use the model id as the key). If no rate
limit is configured, uses a default of 600 RPM.
**Requirements:**
* `pip install google-genai`
**Parameters:**
* **`prompt`** (`String | None`, default: `Literal(None)`): A text description of the videos to generate.
* **`image`** (`Image | None`, default: `Literal(None)`): A single image to use as the first frame of the video, or as `images` a list of up to 3 reference images
for Veo 3.1 (see overloaded signature).
* **`model`** (`String`): The model to use.
* **`config`** (`Json | None`, default: `Literal(None)`): Configuration for generation, corresponding to keyword arguments of
`genai.types.GenerateVideosConfig`. For details on the parameters, see:
[https://googleapis.github.io/python-genai/genai.html#genai.types.GenerateVideosConfig](https://googleapis.github.io/python-genai/genai.html#genai.types.GenerateVideosConfig)
**Returns:**
* `pxt.Video`: The generated video.
**Examples:**
Add a computed column that applies the model `veo-3.0-generate-001`
to an existing Pixeltable column `tbl.prompt` of the table `tbl`:
```python theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
tbl.add_computed_column(
response=generate_videos(tbl.prompt, model='veo-3.0-generate-001')
)
```
Use reference images with Veo 3.1 to guide video generation:
```python theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
tbl.add_computed_column(
response=generate_videos(
tbl.prompt,
images=[tbl.ref_img1, tbl.ref_img2],
reference_types=['asset', 'asset'],
model='veo-3.1-generate-preview',
)
)
```
## udf transcribe()
```python Signature theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
@pxt.udf
transcribe(
audio: pxt.Audio,
*,
model: pxt.String,
prompt: pxt.String,
config: pxt.Json | None = None
) -> pxt.String
```
Transcribes audio to text using Gemini's audio understanding capability. For additional details, see:
[https://ai.google.dev/gemini-api/docs/audio](https://ai.google.dev/gemini-api/docs/audio)
Request throttling:
Applies the rate limit set in the config (section `gemini.rate_limits`; use the model id as the key). If no rate
limit is configured, uses a default of 600 RPM.
**Requirements:**
* `pip install google-genai`
**Parameters:**
* **`audio`** (`pxt.Audio`): The audio file to transcribe.
* **`model`** (`pxt.String`): The model to use (e.g. `'gemini-2.5-flash'`).
* **`prompt`** (`pxt.String`): The instruction prompt sent alongside the audio. For example,
`'Generate a transcript of the speech.'` or `'Summarize the audio content.'`.
* **`config`** (`pxt.Json | None`): Additional configuration, corresponding to keyword arguments of
`genai.types.GenerateContentConfig`.
**Returns:**
* `pxt.String`: The transcribed text.
**Examples:**
Add a computed column that transcribes audio:
```python theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
tbl.add_computed_column(
transcript=transcribe(tbl.audio, model='gemini-2.5-flash')
)
```
# groq
Source: https://docs.pixeltable.com/sdk/latest/groq
# module pixeltable.functions.groq
Pixeltable UDFs
that wrap various endpoints from the Groq API. In order to use them, you must
first `pip install groq` and configure your Groq credentials, as described in
the [Working with Groq](https://docs.pixeltable.com/howto/providers/working-with-groq) tutorial.
## func invoke\_tools()
```python Signature theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
invoke_tools(
tools: pixeltable.func.tools.Tools,
response: pixeltable.exprs.expr.Expr
) -> pixeltable.exprs.inline_expr.InlineDict
```
Converts an OpenAI response dict to Pixeltable tool invocation format and calls `tools._invoke()`.
## udf chat\_completions()
```python Signature theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
@pxt.udf
chat_completions(
messages: pxt.Json[(Json, ...)],
*,
model: pxt.String,
model_kwargs: pxt.Json | None = None,
tools: pxt.Json[(Json, ...)] | None = None,
tool_choice: pxt.Json | None = None
) -> pxt.Json
```
Chat Completion API.
Equivalent to the Groq `chat/completions` API endpoint.
For additional details, see: [https://console.groq.com/docs/api-reference#chat-create](https://console.groq.com/docs/api-reference#chat-create)
Request throttling:
Applies the rate limit set in the config (section `groq`, key `rate_limit`). If no rate
limit is configured, uses a default of 600 RPM.
**Requirements:**
* `pip install groq`
**Parameters:**
* **`messages`** (`pxt.Json[(Json`): A list of messages comprising the conversation so far.
* **`model`** (`Any`): ID of the model to use. (See overview here: [https://console.groq.com/docs/models](https://console.groq.com/docs/models))
* **`model_kwargs`** (`Any`): Additional keyword args for the Groq `chat/completions` API.
For details on the available parameters, see: [https://console.groq.com/docs/api-reference#chat-create](https://console.groq.com/docs/api-reference#chat-create)
**Returns:**
* `pxt.Json`: A dictionary containing the response and other metadata.
**Examples:**
Add a computed column that applies the model `openai/gpt-oss-20b`
to an existing Pixeltable column `tbl.prompt` of the table `tbl`:
```python theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
messages = [{'role': 'user', 'content': tbl.prompt}]
tbl.add_computed_column(
response=chat_completions(messages, model='openai/gpt-oss-20b')
)
```
# huggingface
Source: https://docs.pixeltable.com/sdk/latest/huggingface
# module pixeltable.functions.huggingface
Pixeltable UDFs
that wrap various models from the Hugging Face `transformers` package.
These UDFs will cause Pixeltable to invoke the relevant models locally. In order to use them, you must
first `pip install transformers` (or in some cases, `sentence-transformers`, as noted in the specific
UDFs).
## UDFs
## udf automatic\_speech\_recognition()
```python Signature theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
@pxt.udf
automatic_speech_recognition(
audio: pxt.Audio,
*,
model_id: pxt.String,
language: pxt.String | None = None,
chunk_length_s: pxt.Int | None = None,
return_timestamps: pxt.Bool = False
) -> pxt.String
```
Transcribes speech to text using a pretrained ASR model. `model_id` should be a reference to a
pretrained [automatic-speech-recognition model](https://huggingface.co/models?pipeline_tag=automatic-speech-recognition).
This is a **generic function** that works with many ASR model families. For production use with
specific models, consider specialized functions like `whisper.transcribe()` or
`speech2text_for_conditional_generation()`.
**Requirements:**
* `pip install torch transformers torchaudio torchcodec`
**Recommended Models:**
* **OpenAI Whisper**: `openai/whisper-tiny.en`, `openai/whisper-small`, `openai/whisper-base`
* **Facebook Wav2Vec2**: `facebook/wav2vec2-base-960h`, `facebook/wav2vec2-large-960h-lv60-self`
* **Microsoft SpeechT5**: `microsoft/speecht5_asr`
* **Meta MMS (Multilingual)**: `facebook/mms-1b-all`
**Parameters:**
* **`audio`** (`pxt.Audio`): The audio file(s) to transcribe.
* **`model_id`** (`pxt.String`): The pretrained ASR model to use.
* **`language`** (`pxt.String | None`): Language code for multilingual models (e.g., 'en', 'es', 'fr').
* **`chunk_length_s`** (`pxt.Int | None`): Maximum length of audio chunks in seconds for long audio processing.
* **`return_timestamps`** (`pxt.Bool`): Whether to return word-level timestamps (model dependent).
**Returns:**
* `pxt.String`: The transcribed text.
**Examples:**
Add a computed column that transcribes audio files:
```python theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
tbl.add_computed_column(
transcription=automatic_speech_recognition(
tbl.audio_file,
model_id='openai/whisper-tiny.en', # Recommended
)
)
```
Transcribe with language specification:
```python theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
tbl.add_computed_column(
transcription=automatic_speech_recognition(
tbl.audio_file, model_id='facebook/mms-1b-all', language='en'
)
)
```
## udf clip()
```python Signatures theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
# Signature 1:
@pxt.udf
clip(
text: pxt.String,
*,
model_id: pxt.String
) -> pxt.Array[(None,), float32]
# Signature 2:
@pxt.udf
clip(
image: pxt.Image,
*,
model_id: pxt.String
) -> pxt.Array[(None,), float32]
```
Computes a CLIP embedding for the specified text or image. `model_id` should be a reference to a pretrained
[CLIP Model](https://huggingface.co/docs/transformers/model_doc/clip).
**Requirements:**
* `pip install torch transformers`
**Parameters:**
* **`text`** (`String`): The string to embed.
* **`model_id`** (`String`): The pretrained model to use for the embedding.
**Returns:**
* `pxt.Array[(None,), float32]`: An array containing the output of the embedding model.
**Examples:**
Add a computed column that applies the model `openai/clip-vit-base-patch32` to an existing
Pixeltable column `tbl.text` of the table `tbl`:
```python theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
tbl.add_computed_column(
result=clip(tbl.text, model_id='openai/clip-vit-base-patch32')
)
```
The same would work with an image column `tbl.image` in place of `tbl.text`.
## udf cross\_encoder()
```python Signature theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
@pxt.udf
cross_encoder(
sentences1: pxt.String,
sentences2: pxt.String,
*,
model_id: pxt.String
) -> pxt.Float
```
Performs predicts on the given sentence pair.
`model_id` should be a pretrained Cross-Encoder model, as described in the
[Cross-Encoder Pretrained Models](https://www.sbert.net/docs/cross_encoder/pretrained_models.html)
documentation.
**Requirements:**
* `pip install torch sentence-transformers`
**Parameters:**
* **`sentences1`** (`pxt.String`): The first sentence to be paired.
* **`sentences2`** (`pxt.String`): The second sentence to be paired.
* **`model_id`** (`pxt.String`): The identifier of the cross-encoder model to use.
**Returns:**
* `pxt.Float`: The similarity score between the inputs.
**Examples:**
Add a computed column that applies the model `ms-marco-MiniLM-L-4-v2` to the sentences in
columns `tbl.sentence1` and `tbl.sentence2`:
```python theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
tbl.add_computed_column(
result=sentence_transformer(
tbl.sentence1, tbl.sentence2, model_id='ms-marco-MiniLM-L-4-v2'
)
)
```
## udf detr\_for\_object\_detection()
```python Signature theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
@pxt.udf
detr_for_object_detection(
image: pxt.Image,
*,
model_id: pxt.String,
threshold: pxt.Float = 0.5,
revision: pxt.String | None = None
) -> DetrForObjectDetectionResponse
```
Computes DETR object detections for the specified image. `model_id` should be a reference to a pretrained
[DETR Model](https://huggingface.co/docs/transformers/model_doc/detr).
**Requirements:**
* `pip install torch transformers`
**Parameters:**
* **`image`** (`pxt.Image`): The image to embed.
* **`model_id`** (`pxt.String`): The pretrained model to use for object detection.
* **`threshold`** (`pxt.Float`): Confidence threshold for filtering detections.
* **`revision`** (`pxt.String | None`): The specific model revision to use (e.g., a branch, tag, or git identifier). If not specified,
uses the default revision for the model (typically `'main'`).
**Returns:**
* `DetrForObjectDetectionResponse`: A dictionary containing the output of the object detection model, in the following format:
```python theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
{
# list of confidence scores for each detected object
'scores': [0.99, 0.999],
# list of COCO class labels for each detected object
'labels': [25, 25],
# corresponding text names of class labels
'label_text': ['giraffe', 'giraffe'],
# list of bounding boxes for each detected object, as [x1, y1, x2, y2]
'boxes': [
[51.942, 356.174, 181.481, 413.975],
[383.225, 58.66, 605.64, 361.346],
],
}
```
**Examples:**
Add a computed column that applies the model `facebook/detr-resnet-50` to an existing
Pixeltable column `image` of the table `tbl`:
```python theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
tbl.add_computed_column(
detections=detr_for_object_detection(
tbl.image, model_id='facebook/detr-resnet-50', threshold=0.8
)
)
```
## udf detr\_for\_segmentation()
```python Signature theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
@pxt.udf
detr_for_segmentation(
image: pxt.Image,
*,
model_id: pxt.String,
threshold: pxt.Float = 0.5
) -> DetrForSegmentationResponse
```
Computes DETR panoptic segmentation for the specified image. `model_id` should be a reference to a pretrained
[DETR Model](https://huggingface.co/docs/transformers/model_doc/detr) with a segmentation head.
**Requirements:**
* `pip install torch transformers timm`
**Parameters:**
* **`image`** (`pxt.Image`): The image to segment.
* **`model_id`** (`pxt.String`): The pretrained model to use for segmentation (e.g., 'facebook/detr-resnet-50-panoptic').
* **`threshold`** (`pxt.Float`): Confidence threshold for filtering segments.
**Returns:**
* `DetrForSegmentationResponse`: A dictionary containing the output of the segmentation model, in the following format:
```python theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
{
'segmentation': np.ndarray, # (H, W) array where each pixel value is a segment ID
'segments_info': [
{
'id': 1, # segment ID (matches pixel values in segmentation array)
'label_id': 0, # class label index
'label_text': 'person', # human-readable class name
'score': 0.98, # confidence score
'was_fused': False, # whether segment was fused from multiple instances
},
...,
],
}
```
**Examples:**
Add a computed column that applies the model `facebook/detr-resnet-50-panoptic` to an existing
Pixeltable column `image` of the table `tbl`:
```python theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
tbl.add_computed_column(
segmentation=detr_for_segmentation(
tbl.image,
model_id='facebook/detr-resnet-50-panoptic',
threshold=0.5,
)
)
```
## udf detr\_to\_coco()
```python Signature theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
@pxt.udf
detr_to_coco(image: pxt.Image, detr_info: pxt.Json) -> CocoResponse
```
Converts the output of a DETR object detection model to COCO format.
**Parameters:**
* **`image`** (`pxt.Image`): The image for which detections were computed.
* **`detr_info`** (`pxt.Json`): The output of a DETR object detection model, as returned by `detr_for_object_detection`.
**Returns:**
* `CocoResponse`: A dictionary containing the data from `detr_info`, converted to COCO format.
**Examples:**
Add a computed column that converts the output `tbl.detections` to COCO format, where `tbl.image`
is the image for which detections were computed:
```python theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
tbl.add_computed_column(
detections_coco=detr_to_coco(tbl.image, tbl.detections)
)
```
## udf image\_captioning()
```python Signature theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
@pxt.udf
image_captioning(
image: pxt.Image,
*,
model_id: pxt.String,
model_kwargs: pxt.Json | None = None
) -> pxt.String
```
Generates captions for images using a pretrained image captioning model. `model_id` should be a reference to a
pretrained [image-to-text model](https://huggingface.co/models?pipeline_tag=image-to-text) such as BLIP,
Git, or LLaVA.
**Requirements:**
* `pip install torch transformers`
**Parameters:**
* **`image`** (`pxt.Image`): The image to caption.
* **`model_id`** (`pxt.String`): The pretrained model to use for captioning.
* **`model_kwargs`** (`pxt.Json | None`): Additional keyword arguments to pass to the model's `generate` method, such as `max_length`.
**Returns:**
* `pxt.String`: The generated caption text.
**Examples:**
Add a computed column `caption` to an existing table `tbl` that generates captions using the
`Salesforce/blip-image-captioning-base` model:
```python theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
tbl.add_computed_column(
caption=image_captioning(
tbl.image,
model_id='Salesforce/blip-image-captioning-base',
model_kwargs={'max_length': 30},
)
)
```
## udf image\_to\_image()
```python Signature theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
@pxt.udf
image_to_image(
image: pxt.Image,
prompt: pxt.String,
*,
model_id: pxt.String,
seed: pxt.Int | None = None,
model_kwargs: pxt.Json | None = None
) -> pxt.Image
```
Transforms input images based on text prompts using a pretrained image-to-image model.
`model_id` should be a reference to a pretrained
[image-to-image model](https://huggingface.co/models?pipeline_tag=image-to-image) such as
Stable Diffusion.
**Requirements:**
* `pip install torch transformers diffusers accelerate`
**Parameters:**
* **`image`** (`pxt.Image`): The input image to transform.
* **`prompt`** (`pxt.String`): The text prompt describing the desired transformation.
* **`model_id`** (`pxt.String`): The pretrained image-to-image model to use.
* **`seed`** (`pxt.Int | None`): Random seed for reproducibility.
* **`model_kwargs`** (`pxt.Json | None`): Additional keyword arguments to pass to the model, such as `strength`,
`guidance_scale`, or `num_inference_steps`.
**Returns:**
* `pxt.Image`: The transformed image.
**Examples:**
Add a computed column that transforms images based on prompts:
```python theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
tbl.add_computed_column(
transformed=image_to_image(
tbl.source_image,
tbl.transformation_prompt,
model_id='stable-diffusion-v1-5/stable-diffusion-v1-5',
)
)
```
With custom transformation strength:
```python theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
tbl.add_computed_column(
transformed=image_to_image(
tbl.source_image,
tbl.transformation_prompt,
model_id='stable-diffusion-v1-5/stable-diffusion-v1-5',
model_kwargs={'strength': 0.75, 'num_inference_steps': 50},
)
)
```
## udf image\_to\_video()
```python Signature theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
@pxt.udf
image_to_video(
image: pxt.Image,
*,
model_id: pxt.String,
num_frames: pxt.Int = 25,
fps: pxt.Int = 6,
seed: pxt.Int | None = None,
model_kwargs: pxt.Json | None = None
) -> pxt.Video
```
Generates videos from input images using a pretrained image-to-video model.
`model_id` should be a reference to a pretrained
[image-to-video model](https://huggingface.co/models?pipeline_tag=image-to-video).
**Requirements:**
* `pip install torch transformers diffusers accelerate`
**Parameters:**
* **`image`** (`pxt.Image`): The input image to animate into a video.
* **`model_id`** (`pxt.String`): The pretrained image-to-video model to use.
* **`num_frames`** (`pxt.Int`): Number of video frames to generate.
* **`fps`** (`pxt.Int`): Frames per second for the output video.
* **`seed`** (`pxt.Int | None`): Random seed for reproducibility.
* **`model_kwargs`** (`pxt.Json | None`): Additional keyword arguments to pass to the model, such as `num_inference_steps`,
`motion_bucket_id`, or `guidance_scale`.
**Returns:**
* `pxt.Video`: The generated video file.
**Examples:**
Add a computed column that creates videos from images:
```python theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
tbl.add_computed_column(
video=image_to_video(
tbl.input_image,
model_id='stabilityai/stable-video-diffusion-img2vid-xt',
num_frames=25,
fps=7,
)
)
```
## udf question\_answering()
```python Signature theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
@pxt.udf
question_answering(
context: pxt.String,
question: pxt.String,
*,
model_id: pxt.String
) -> QuestionAnsweringResponse
```
Answers questions based on provided context using a pretrained QA model. `model_id` should be a reference to a
pretrained [question answering model](https://huggingface.co/models?pipeline_tag=question-answering) such as
BERT or RoBERTa.
**Requirements:**
* `pip install torch transformers`
**Parameters:**
* **`context`** (`pxt.String`): The context text containing the answer.
* **`question`** (`pxt.String`): The question to answer.
* **`model_id`** (`pxt.String`): The pretrained QA model to use.
**Returns:**
* `QuestionAnsweringResponse`: A dictionary containing the answer, confidence score, and start/end positions.
**Examples:**
Add a computed column that answers questions based on document context:
```python theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
tbl.add_computed_column(
answer=question_answering(
tbl.document_text,
tbl.question,
model_id='deepset/roberta-base-squad2',
)
)
```
## udf sam3\_for\_segmentation()
```python Signature theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
@pxt.udf
sam3_for_segmentation(
image: pxt.Image,
*,
model_id: pxt.String = 'facebook/sam3',
text: pxt.String | None = None,
input_boxes: pxt.Json[(Json[(Float, ...)], ...)] | None = None,
input_boxes_labels: pxt.Json[(Int, ...)] | None = None,
threshold: pxt.Float = 0.5,
mask_threshold: pxt.Float = 0.5,
revision: pxt.String | None = None
) -> SamForSegmentationResponse
```
Computes SAM 3 (Segment Anything Model 3) Promptable Concept Segmentation for the specified image.
`model_id` should be a reference to a pretrained
[SAM 3 Model](https://huggingface.co/docs/transformers/model_doc/sam3) such as `facebook/sam3`.
SAM 3 performs Promptable Concept Segmentation (PCS): given a concept prompt (a short text noun phrase,
a set of bounding boxes, or both), it detects and segments every object instance in the image that
matches the concept.
**Requirements:**
* `pip install torch torchvision transformers`
* `facebook/sam3` is a gated repository. Request access on its
[model page](https://huggingface.co/facebook/sam3), then authenticate with
`huggingface-cli login` (or set the `HF_TOKEN` environment variable) before calling this UDF.
**Parameters:**
* **`image`** (`pxt.Image`): The image to segment.
* **`model_id`** (`pxt.String`): The pretrained SAM 3 model to use.
* **`text`** (`pxt.String | None`): Optional concept prompt as a short noun phrase (e.g., `'yellow school bus'`).
* **`input_boxes`** (`pxt.Json[(Json[(Float`): Optional list of bounding boxes in `[x1, y1, x2, y2]` pixel coordinates that
constrain the concept (e.g., `[[100, 150, 500, 450]]`).
* **`input_boxes_labels`** (`Any`): Optional list of integer labels for each box: `1` for positive
(include this region), `0` for negative (exclude this region). If `input_boxes` is
provided and this is `None`, all boxes default to positive.
* **`threshold`** (`Any`): Confidence threshold for filtering detections.
* **`mask_threshold`** (`Any`): Threshold applied to mask logits to produce binary masks.
* **`revision`** (`Any`): The specific model revision to use (e.g., a branch, tag, or git identifier). If not
specified, uses the default revision for the model.
**Returns:**
* `SamForSegmentationResponse`: A `SamForSegmentationResponse` containing:
* `scores`: confidence score per detected instance, shape `(num_instances,)`.
* `boxes`: bounding box `[x1, y1, x2, y2]` per detected instance in absolute pixel coordinates, shape
`(num_instances, 4)`.
* `masks`: binary mask per detected instance, shape `(num_instances, H, W)`.
**Examples:**
Add a computed column that segments every "cat" in an existing Pixeltable column `image` of the
table `tbl`:
```python theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
tbl.add_computed_column(seg=sam3_for_segmentation(tbl.image, text='cat'))
```
Use a bounding-box prompt to segment a specific object:
```python theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
tbl.add_computed_column(
seg=sam3_for_segmentation(
tbl.image, input_boxes=[[100, 150, 500, 450]]
)
)
```
Combine a text prompt with a negative box to exclude a region:
```python theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
tbl.add_computed_column(
seg=sam3_for_segmentation(
tbl.image,
text='handle',
input_boxes=[[40, 183, 318, 204]],
input_boxes_labels=[0],
)
)
```
## udf sam\_automatic\_mask\_generation()
```python Signature theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
@pxt.udf
sam_automatic_mask_generation(
image: pxt.Image,
*,
model_id: pxt.String = 'facebook/sam3',
points_per_batch: pxt.Int = 64,
points_per_crop: pxt.Int = 32,
pred_iou_thresh: pxt.Float = 0.7,
stability_score_thresh: pxt.Float = 0.95,
crops_n_layers: pxt.Int = 0,
revision: pxt.String | None = None
) -> SamForSegmentationResponse
```
Segments every object in an image with SAM (Segment Anything Model), with no prompt. `model_id` should
be a reference to a pretrained
[SAM Model](https://huggingface.co/docs/transformers/model_doc/sam3) such as `facebook/sam3` or
`facebook/sam2-hiera-large`. This UDF is not tied to a specific SAM version; it works with any SAM model
that supports automatic mask generation.
Works by sampling a grid of points across the image and returns a binary mask for every distinct object it
finds (the "segment everything" mode of SAM 1 and SAM 2). Masks are filtered by predicted quality and
de-duplicated. The masks are class-agnostic: no labels or persistent object ids are produced, since the model
is not told what to look for.
**Requirements:**
* `pip install torch transformers`
* `facebook/sam3` is a gated repository. Request access on its
[model page](https://huggingface.co/facebook/sam3), then authenticate with
`huggingface-cli login` (or set the `HF_TOKEN` environment variable) before calling this UDF.
**Parameters:**
* **`image`** (`pxt.Image`): The image to segment.
* **`model_id`** (`pxt.String`): The pretrained SAM model to use.
* **`points_per_batch`** (`pxt.Int`): Number of grid points run through the model at once. Higher values are faster but
use more memory.
* **`points_per_crop`** (`pxt.Int`): Number of grid points sampled along each side of the image. Higher values find more
(and smaller) objects at the cost of speed.
* **`pred_iou_thresh`** (`pxt.Float`): Quality threshold on the model's predicted mask IoU; masks scoring below it are
discarded.
* **`stability_score_thresh`** (`pxt.Float`): Quality threshold on mask stability under mask-logit threshold changes; masks
scoring below it are discarded.
* **`crops_n_layers`** (`pxt.Int`): Number of additional image-crop layers to run. Values above `0` improve recall on
small objects at the cost of speed.
* **`revision`** (`pxt.String | None`): The specific model revision to use (e.g., a branch, tag, or git identifier). If not
specified, uses the default revision for the model.
**Returns:**
* `SamForSegmentationResponse`: A `SamForSegmentationResponse` containing:
* `scores`: predicted quality score per mask, shape `(num_masks,)`.
* `boxes`: bounding box `[x1, y1, x2, y2]` per mask in absolute pixel coordinates, shape
`(num_masks, 4)`.
* `masks`: binary mask per detected object, shape `(num_masks, H, W)`.
**Examples:**
Add a computed column that segments every object in an existing Pixeltable column `image` of the
table `tbl`:
```python theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
tbl.add_computed_column(seg=sam_automatic_mask_generation(tbl.image))
```
## udf sentence\_transformer()
```python Signature theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
@pxt.udf
sentence_transformer(
sentence: pxt.String,
*,
model_id: pxt.String,
normalize_embeddings: pxt.Bool = False
) -> pxt.Array[(None,), float32]
```
Computes sentence embeddings. `model_id` should be a pretrained Sentence Transformers model, as described
in the [Sentence Transformers Pretrained Models](https://sbert.net/docs/sentence_transformer/pretrained_models.html)
documentation.
**Requirements:**
* `pip install torch sentence-transformers`
**Parameters:**
* **`sentence`** (`pxt.String`): The sentence to embed.
* **`model_id`** (`pxt.String`): The pretrained model to use for the encoding.
* **`normalize_embeddings`** (`pxt.Bool`): If `True`, normalizes embeddings to length 1; see the
[Sentence Transformers API Docs](https://sbert.net/docs/package_reference/sentence_transformer/SentenceTransformer.html)
for more details
**Returns:**
* `pxt.Array[(None,), float32]`: An array containing the output of the embedding model.
**Examples:**
Add a computed column that applies the model `all-mpnet-base-2` to an existing Pixeltable column `tbl.sentence`
of the table `tbl`:
```python theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
tbl.add_computed_column(
result=sentence_transformer(
tbl.sentence, model_id='all-mpnet-base-v2'
)
)
```
## udf speech2text\_for\_conditional\_generation()
```python Signature theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
@pxt.udf
speech2text_for_conditional_generation(
audio: pxt.Audio,
*,
model_id: pxt.String,
language: pxt.String | None = None
) -> pxt.String
```
Transcribes or translates speech to text using a Speech2Text model. `model_id` should be a reference to a
pretrained [Speech2Text](https://huggingface.co/docs/transformers/en/model_doc/speech_to_text) model.
**Requirements:**
* `pip install torch torchaudio sentencepiece transformers`
**Parameters:**
* **`audio`** (`pxt.Audio`): The audio clip to transcribe or translate.
* **`model_id`** (`pxt.String`): The pretrained model to use for the transcription or translation.
* **`language`** (`pxt.String | None`): If using a multilingual translation model, the language code to translate to. If not provided,
the model's default language will be used. If the model is not translation model, is not a
multilingual model, or does not support the specified language, an error will be raised.
**Returns:**
* `pxt.String`: The transcribed or translated text.
**Examples:**
Add a computed column that applies the model `facebook/s2t-small-librispeech-asr` to an existing
Pixeltable column `audio` of the table `tbl`:
```python theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
tbl.add_computed_column(
transcription=speech2text_for_conditional_generation(
tbl.audio, model_id='facebook/s2t-small-librispeech-asr'
)
)
```
Add a computed column that applies the model `facebook/s2t-medium-mustc-multilingual-st` to an existing
Pixeltable column `audio` of the table `tbl`, translating the audio to French:
```python theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
tbl.add_computed_column(
translation=speech2text_for_conditional_generation(
tbl.audio,
model_id='facebook/s2t-medium-mustc-multilingual-st',
language='fr',
)
)
```
## udf summarization()
```python Signature theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
@pxt.udf
summarization(
text: pxt.String,
*,
model_id: pxt.String,
model_kwargs: pxt.Json | None = None
) -> pxt.String
```
Summarizes text using a pretrained summarization model. `model_id` should be a reference to a pretrained
[summarization model](https://huggingface.co/models?pipeline_tag=summarization) such as BART, T5, or Pegasus.
**Requirements:**
* `pip install torch transformers`
**Parameters:**
* **`text`** (`pxt.String`): The text to summarize.
* **`model_id`** (`pxt.String`): The pretrained model to use for summarization.
* **`model_kwargs`** (`pxt.Json | None`): Additional keyword arguments to pass to the model's `generate` method, such as `max_length`.
**Returns:**
* `pxt.String`: The generated summary text.
**Examples:**
Add a computed column that summarizes documents:
```python theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
tbl.add_computed_column(
summary=text_summarization(
tbl.document_text,
model_id='facebook/bart-large-cnn',
max_length=100,
)
)
```
## udf text\_classification()
```python Signature theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
@pxt.udf
text_classification(
text: pxt.String,
*,
model_id: pxt.String,
top_k: pxt.Int = 5
) -> pxt.Json[(Json[{'label': Int, 'label_text': String, 'score': Float}], ...)]
```
Classifies text using a pretrained classification model. `model_id` should be a reference to a pretrained
[text classification model](https://huggingface.co/models?pipeline_tag=text-classification)
such as BERT, RoBERTa, or DistilBERT.
**Requirements:**
* `pip install torch transformers`
**Parameters:**
* **`text`** (`pxt.String`): The text to classify.
* **`model_id`** (`pxt.String`): The pretrained model to use for classification.
* **`top_k`** (`pxt.Int`): The number of top predictions to return.
**Returns:**
* `pxt.Json[(Json[{'label': Int, 'label_text': String, 'score': Float}], ...)]`: A dictionary containing classification results with scores, labels, and label text.
**Examples:**
Add a computed column for sentiment analysis:
```python theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
tbl.add_computed_column(
sentiment=text_classification(
tbl.review_text,
model_id='cardiffnlp/twitter-roberta-base-sentiment-latest',
)
)
```
## udf text\_generation()
```python Signature theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
@pxt.udf
text_generation(
text: pxt.String,
*,
model_id: pxt.String,
model_kwargs: pxt.Json | None = None
) -> pxt.String
```
Generates text using a pretrained language model. `model_id` should be a reference to a pretrained
[text generation model](https://huggingface.co/models?pipeline_tag=text-generation).
**Requirements:**
* `pip install torch transformers`
**Parameters:**
* **`text`** (`pxt.String`): The input text to continue/complete.
* **`model_id`** (`pxt.String`): The pretrained model to use for text generation.
* **`model_kwargs`** (`pxt.Json | None`): Additional keyword arguments to pass to the model's `generate` method, such as `max_length`,
`temperature`, etc. See the
[Hugging Face text\_generation documentation](https://huggingface.co/docs/inference-providers/en/tasks/text-generation)
for details.
**Returns:**
* `pxt.String`: The generated text completion.
**Examples:**
Add a computed column that generates text completions using the `Qwen/Qwen3-0.6B` model:
```python theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
tbl.add_computed_column(
completion=text_generation(
tbl.prompt,
model_id='Qwen/Qwen3-0.6B',
model_kwargs={'temperature': 0.5, 'max_length': 150},
)
)
```
## udf text\_to\_image()
```python Signature theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
@pxt.udf
text_to_image(
prompt: pxt.String,
*,
model_id: pxt.String,
height: pxt.Int = 512,
width: pxt.Int = 512,
seed: pxt.Int | None = None,
model_kwargs: pxt.Json | None = None
) -> pxt.Image
```
Generates images from text prompts using a pretrained text-to-image model. `model_id` should be a reference to a
pretrained [text-to-image model](https://huggingface.co/models?pipeline_tag=text-to-image) such as
Stable Diffusion.
**Requirements:**
* `pip install torch transformers diffusers accelerate`
**Parameters:**
* **`prompt`** (`pxt.String`): The text prompt describing the desired image.
* **`model_id`** (`pxt.String`): The pretrained text-to-image model to use.
* **`height`** (`pxt.Int`): Height of the generated image in pixels.
* **`width`** (`pxt.Int`): Width of the generated image in pixels.
* **`seed`** (`pxt.Int | None`): Optional random seed for reproducibility.
* **`model_kwargs`** (`pxt.Json | None`): Additional keyword arguments to pass to the model, such as `num_inference_steps`,
`guidance_scale`, or `negative_prompt`.
**Returns:**
* `pxt.Image`: The generated Image.
**Examples:**
Add a computed column that generates images from text prompts:
```python theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
tbl.add_computed_column(
generated_image=text_to_image(
tbl.prompt,
model_id='stable-diffusion-v1.5/stable-diffusion-v1-5',
height=512,
width=512,
model_kwargs={'num_inference_steps': 25},
)
)
```
## udf text\_to\_speech()
```python Signature theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
@pxt.udf
text_to_speech(
text: pxt.String,
*,
model_id: pxt.String,
speaker_id: pxt.Int | None = None,
vocoder: pxt.String | None = None
) -> pxt.Audio
```
Converts text to speech using a pretrained TTS model. `model_id` should be a reference to a
pretrained [text-to-speech model](https://huggingface.co/models?pipeline_tag=text-to-speech).
**Requirements:**
* `pip install torch transformers datasets soundfile`
**Parameters:**
* **`text`** (`pxt.String`): The text to convert to speech.
* **`model_id`** (`pxt.String`): The pretrained TTS model to use.
* **`speaker_id`** (`pxt.Int | None`): Speaker ID for multi-speaker models.
* **`vocoder`** (`pxt.String | None`): Optional vocoder model for higher quality audio.
**Returns:**
* `pxt.Audio`: The generated audio file.
**Examples:**
Add a computed column that converts text to speech:
```python theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
tbl.add_computed_column(
audio=text_to_speech(
tbl.text_content, model_id='microsoft/speecht5_tts', speaker_id=0
)
)
```
## udf token\_classification()
```python Signature theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
@pxt.udf
token_classification(
text: pxt.String,
*,
model_id: pxt.String,
aggregation_strategy: pxt.String = 'simple'
) -> pxt.Json[(Json[{'word': String, 'entity_group': String, 'score': Float, 'start': Int, 'end': Int}], ...)]
```
Extracts named entities from text using a pretrained named entity recognition (NER) model.
`model_id` should be a reference to a pretrained
[token classification model](https://huggingface.co/models?pipeline_tag=token-classification) for NER.
**Requirements:**
* `pip install torch transformers`
**Parameters:**
* **`text`** (`pxt.String`): The text to analyze for named entities.
* **`model_id`** (`pxt.String`): The pretrained model to use.
* **`aggregation_strategy`** (`pxt.String`): Method used to aggregate tokens.
**Returns:**
* `pxt.Json[(Json[{'word': String, 'entity_group': String, 'score': Float, 'start': Int, 'end': Int}], ...)]`: A list of dictionaries containing entity information (text, label, confidence, start, end).
**Examples:**
Add a computed column that extracts named entities:
```python theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
tbl.add_computed_column(
entities=token_classification(
tbl.text,
model_id='dbmdz/bert-large-cased-finetuned-conll03-english',
)
)
```
## udf translation()
```python Signature theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
@pxt.udf
translation(
text: pxt.String,
*,
model_id: pxt.String,
src_lang: pxt.String | None = None,
target_lang: pxt.String | None = None
) -> pxt.String
```
Translates text using a pretrained translation model. `model_id` should be a reference to a pretrained
[translation model](https://huggingface.co/models?pipeline_tag=translation) such as MarianMT or T5.
**Requirements:**
* `pip install torch transformers sentencepiece`
**Parameters:**
* **`text`** (`pxt.String`): The text to translate.
* **`model_id`** (`pxt.String`): The pretrained translation model to use.
* **`src_lang`** (`pxt.String | None`): Source language code (optional, can be inferred from model).
* **`target_lang`** (`pxt.String | None`): Target language code (optional, can be inferred from model).
**Returns:**
* `pxt.String`: The translated text.
**Examples:**
Add a computed column that translates text:
```python theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
tbl.add_computed_column(
french_text=translation(
tbl.english_text,
model_id='Helsinki-NLP/opus-mt-en-fr',
src_lang='en',
target_lang='fr',
)
)
```
## udf vit\_for\_image\_classification()
```python Signature theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
@pxt.udf
vit_for_image_classification(
image: pxt.Image,
*,
model_id: pxt.String,
top_k: pxt.Int = 5
) -> VitForImageClassificationResponse
```
Computes image classifications for the specified image using a Vision Transformer (ViT) model.
`model_id` should be a reference to a pretrained [ViT Model](https://huggingface.co/docs/transformers/en/model_doc/vit).
**Note:** Be sure the model is a ViT model that is trained for image classification (that is, a model designed for
use with the
[ViTForImageClassification](https://huggingface.co/docs/transformers/en/model_doc/vit#transformers.ViTForImageClassification)
class), such as `google/vit-base-patch16-224`. General feature-extraction models such as
`google/vit-base-patch16-224-in21k` will not produce the desired results.
**Requirements:**
* `pip install torch transformers`
**Parameters:**
* **`image`** (`pxt.Image`): The image to classify.
* **`model_id`** (`pxt.String`): The pretrained model to use for the classification.
* **`top_k`** (`pxt.Int`): The number of classes to return.
**Returns:**
* `VitForImageClassificationResponse`: A dictionary containing the output of the image classification model, in the following format:
```python theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
{
# list of probabilities of the top-k most likely classes
'scores': [0.325, 0.198, 0.105],
# list of class IDs for the top-k most likely classes
'labels': [340, 353, 386],
# corresponding text names of the top-k most likely classes
'label_text': [
'zebra',
'gazelle',
'African elephant, Loxodonta africana',
],
}
```
**Examples:**
Add a computed column that applies the model `google/vit-base-patch16-224` to an existing
Pixeltable column `image` of the table `tbl`, returning the 10 most likely classes for each image:
```python theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
tbl.add_computed_column(
image_class=vit_for_image_classification(
tbl.image, model_id='google/vit-base-patch16-224', top_k=10
)
)
```
# image
Source: https://docs.pixeltable.com/sdk/latest/image
# module pixeltable.functions.image
Pixeltable UDFs for `ImageType`.
Example:
```python theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
import pixeltable as pxt
t = pxt.get_table(...)
t.select(t.img_col.convert('L')).collect()
```
## iterator tile\_iterator()
```python Signature theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
@pxt.iterator
tile_iterator(
image: pxt.Image,
tile_size: pxt.Json[(Int, Int)],
*,
overlap: pxt.Json[(Int, Int)] = (0, 0)
)
```
Iterator over tiles of an image. Each image will be divided into tiles of size `tile_size`, and the tiles will be
iterated over in row-major order (left-to-right, then top-to-bottom). An optional `overlap` parameter may be
specified. If the tiles do not exactly cover the image, then the rightmost and bottommost tiles will be padded with
blackspace, so that the output images all have the exact size `tile_size`.
**Outputs**:
One row per tile, with the following columns:
* `tile` (`pxt.Image`): The image tile
* `tile_coord` (`pxt.Json`): The (x, y) coordinates of the tile in the grid of tiles
* `tile_box` (`pxt.Json`): The (x1, y1, x2, y2) pixel coordinates of the tile in the original image
**Parameters:**
* **`image`** (`pxt.Image`): Image to split into tiles.
* **`tile_size`** (`pxt.Json[(Int`): Size of each tile, as a pair of integers `(width, height)`.
* **`overlap`** (`Any`): Amount of overlap between adjacent tiles, as a pair of integers `(width, height)`.
**Examples:**
This example assumes an existing table `tbl` with a column `img` of type `pxt.Image`.
Create a view that splits all images into 256x256 tiles with 32 pixels of overlap:
```python theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
pxt.create_view(
'image_tiles',
tbl,
iterator=tile_iterator(
tbl.img, tile_size=(256, 256), overlap=(32, 32)
),
)
```
## uda stitch\_tiles()
```python Signature theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
@pxt.uda
stitch_tiles(*args, **kwargs) -> pxt.Image | None
```
Reconstructs a single image from the tiles produced by [`tile_iterator`](./image#iterator-tile_iterator),
pasting each tile back at its original position. This is the inverse of
[`tile_iterator`](./image#iterator-tile_iterator): aggregate the tiles of an image (grouped by a column
that identifies the source image) to obtain the full image, optionally after transforming each tile first
(e.g., drawing a segmentation overlay on it).
The reconstruction is naive: tiles are pasted in the order they arrive, so where tiles overlap, later tiles
overwrite earlier ones. It composites pixels and does not merge or deduplicate detections across tile seams.
Call as `stitch_tiles(order_key, tile, tile_box, width, height)`. `order_by=` is invalid.
Args:
order\_key: Positional only. The expression that orders the tiles within a group, typically the tile
position emitted by `tile_iterator`.
tile: The image tile to paste, the same size as the tiles emitted by
[`tile_iterator`](./image#iterator-tile_iterator). All tiles in a group must have the same mode
(and palette, for palette images).
tile\_box: The `(x1, y1, x2, y2)` pixel coordinates of the tile in the original image, as emitted by
`tile_iterator`.
width: The width of the original image, in pixels.
height: The height of the original image, in pixels.
Returns:
The reconstructed image, of size `(width, height)` and in the mode of the tiles, or `None` if there
are no non-null tiles.
Examples:
Stitch per-tile overlays back into one full-resolution image per source image. Assumes a view `tiles`
created with [`tile_iterator`](./image#iterator-tile_iterator), with an `overlay` column
computed per tile:
```python theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
tiles.group_by(tiles.image).select(
stitched=stitch_tiles(
tiles.pos,
tiles.overlay,
tiles.tile_box,
tiles.image.width,
tiles.image.height,
)
).collect()
```
For the full pipeline, from tiling through segmentation to stitching, see the
[Promptable segmentation with SAM 3](https://docs.pixeltable.com/howto/cookbooks/images/img-promptable-segmentation)
cookbook.
## udf alpha\_composite()
```python Signature theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
@pxt.udf
alpha_composite(im1: pxt.Image, im2: pxt.Image) -> pxt.Image
```
Alpha composite `im2` over `im1`.
Equivalent to [`PIL.Image.alpha_composite()`](https://pillow.readthedocs.io/en/stable/reference/Image.html#PIL.Image.alpha_composite)
## udf b64\_encode()
```python Signature theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
@pxt.udf
b64_encode(
img: pxt.Image,
image_format: pxt.String = 'png'
) -> pxt.String
```
Convert image to a base64-encoded string.
**Parameters:**
* **`img`** (`pxt.Image`): image
* **`image_format`** (`pxt.String`): image format [supported by PIL](https://pillow.readthedocs.io/en/stable/handbook/image-file-formats.html#fully-supported-formats)
## udf blend()
```python Signature theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
@pxt.udf
blend(
im1: pxt.Image,
im2: pxt.Image,
alpha: pxt.Float
) -> pxt.Image
```
Return a new image by interpolating between two input images, using a constant alpha.
Equivalent to [`PIL.Image.blend()`](https://pillow.readthedocs.io/en/stable/reference/Image.html#PIL.Image.blend)
## udf composite()
```python Signature theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
@pxt.udf
composite(
image1: pxt.Image,
image2: pxt.Image,
mask: pxt.Image
) -> pxt.Image
```
Return a composite image by blending two images using a mask.
Equivalent to [`PIL.Image.composite()`](https://pillow.readthedocs.io/en/stable/reference/Image.html#PIL.Image.composite)
## udf convert()
```python Signature theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
@pxt.udf
convert(self: pxt.Image, mode: pxt.String) -> pxt.Image
```
Convert the image to a different mode.
Equivalent to
[`PIL.Image.Image.convert()`](https://pillow.readthedocs.io/en/stable/reference/Image.html#PIL.Image.Image.convert).
**Parameters:**
* **`mode`** (`pxt.String`): The mode to convert to. See the
[Pillow documentation](https://pillow.readthedocs.io/en/stable/handbook/concepts.html#concept-modes)
for a list of supported modes.
## udf crop()
```python Signature theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
@pxt.udf
crop(
self: pxt.Image,
box: pxt.Json[(Int, Int, Int, Int)]
) -> pxt.Image
```
Return a rectangular region from the image. The box is a 4-tuple defining the left, upper, right, and lower pixel
coordinates.
Equivalent to
[`PIL.Image.Image.crop()`](https://pillow.readthedocs.io/en/stable/reference/Image.html#PIL.Image.Image.crop)
## udf effect\_spread()
```python Signature theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
@pxt.udf
effect_spread(self: pxt.Image, distance: pxt.Int) -> pxt.Image
```
Randomly spread pixels in an image.
Equivalent to
[`PIL.Image.Image.effect_spread()`](https://pillow.readthedocs.io/en/stable/reference/Image.html#PIL.Image.Image.effect_spread)
**Parameters:**
* **`distance`** (`pxt.Int`): The distance to spread pixels.
## udf entropy()
```python Signature theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
@pxt.udf
entropy(
self: pxt.Image,
mask: pxt.Image | None = None,
extrema: pxt.Json[(Json, ...)] | None = None
) -> pxt.Float
```
Returns the entropy of the image, optionally using a mask and extrema.
Equivalent to
[`PIL.Image.Image.entropy()`](https://pillow.readthedocs.io/en/stable/reference/Image.html#PIL.Image.Image.entropy)
**Parameters:**
* **`mask`** (`pxt.Image | None`): An optional mask image.
* **`extrema`** (`pxt.Json[(Json`): An optional list of extrema.
## udf get\_metadata()
```python Signature theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
@pxt.udf
get_metadata(self: pxt.Image) -> ImageMetadata
```
Return metadata for the image.
## udf getbands()
```python Signature theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
@pxt.udf
getbands(self: pxt.Image) -> pxt.Json[(String, ...)]
```
Return a tuple containing the names of the image bands.
Equivalent to
[`PIL.Image.Image.getbands()`](https://pillow.readthedocs.io/en/stable/reference/Image.html#PIL.Image.Image.getbands)
## udf getbbox()
```python Signature theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
@pxt.udf
getbbox(
self: pxt.Image,
*,
alpha_only: pxt.Bool = True
) -> pxt.Json[(Int, Int, Int, Int)] | None
```
Return a bounding box for the non-zero regions of the image.
Equivalent to [`PIL.Image.Image.getbbox()`](https://pillow.readthedocs.io/en/stable/reference/Image.html#PIL.Image.Image.getbbox)
**Parameters:**
* **`alpha_only`** (`pxt.Bool`): If `True`, and the image has an alpha channel, trim transparent pixels. Otherwise,
trim pixels when all channels are zero.
## udf getchannel()
```python Signature theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
@pxt.udf
getchannel(self: pxt.Image, channel: pxt.Int) -> pxt.Image
```
Return an L-mode image containing a single channel of the original image.
Equivalent to
[`PIL.Image.Image.getchannel()`](https://pillow.readthedocs.io/en/stable/reference/Image.html#PIL.Image.Image.getchannel)
**Parameters:**
* **`channel`** (`pxt.Int`): The channel to extract. This is a 0-based index.
## udf getcolors()
```python Signature theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
@pxt.udf
getcolors(
self: pxt.Image,
maxcolors: pxt.Int = 256
) -> pxt.Json[(Json[(Int, Int)], ...)]
```
Return a list of colors used in the image, up to a maximum of `maxcolors`.
Equivalent to
[`PIL.Image.Image.getcolors()`](https://pillow.readthedocs.io/en/stable/reference/Image.html#PIL.Image.Image.getcolors)
**Parameters:**
* **`maxcolors`** (`pxt.Int`): The maximum number of colors to return.
## udf getextrema()
```python Signature theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
@pxt.udf
getextrema(self: pxt.Image) -> pxt.Json[(Int, Int)]
```
Return a 2-tuple containing the minimum and maximum pixel values of the image.
Equivalent to
[`PIL.Image.Image.getextrema()`](https://pillow.readthedocs.io/en/stable/reference/Image.html#PIL.Image.Image.getextrema)
## udf getpalette()
```python Signature theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
@pxt.udf
getpalette(
self: pxt.Image,
mode: pxt.String | None = None
) -> pxt.Json[(Int, ...)] | None
```
Return the palette of the image, optionally converting it to a different mode.
Equivalent to
[`PIL.Image.Image.getpalette()`](https://pillow.readthedocs.io/en/stable/reference/Image.html#PIL.Image.Image.getpalette)
**Parameters:**
* **`mode`** (`pxt.String | None`): The mode to convert the palette to.
## udf getpixel()
```python Signature theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
@pxt.udf
getpixel(
self: pxt.Image,
xy: pxt.Json[(Json, ...)]
) -> pxt.Json[(Int,)]
```
Return the pixel value at the given position. The position `xy` is a tuple containing the x and y coordinates.
Equivalent to
[`PIL.Image.Image.getpixel()`](https://pillow.readthedocs.io/en/stable/reference/Image.html#PIL.Image.Image.getpixel)
**Parameters:**
* **`xy`** (`pxt.Json[(Json`): The coordinates, given as (x, y).
## udf getprojection()
```python Signature theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
@pxt.udf
getprojection(self: pxt.Image) -> pxt.Json[(Json[(Int, ...)], Json[(Int, ...)])]
```
Return two sequences representing the horizontal and vertical projection of the image.
Equivalent to
[`PIL.Image.Image.getprojection()`](https://pillow.readthedocs.io/en/stable/reference/Image.html#PIL.Image.Image.getprojection)
## udf height()
```python Signature theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
@pxt.udf
height(self: pxt.Image) -> pxt.Int
```
Return the height of the image.
## udf histogram()
```python Signature theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
@pxt.udf
histogram(
self: pxt.Image,
mask: pxt.Image | None = None,
extrema: pxt.Json[(Json, ...)] | None = None
) -> pxt.Json[(Int, ...)]
```
Return a histogram for the image.
Equivalent to
[`PIL.Image.Image.histogram()`](https://pillow.readthedocs.io/en/stable/reference/Image.html#PIL.Image.Image.histogram)
**Parameters:**
* **`mask`** (`pxt.Image | None`): An optional mask image.
* **`extrema`** (`pxt.Json[(Json`): An optional list of extrema.
## udf mode()
```python Signature theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
@pxt.udf
mode(self: pxt.Image) -> pxt.String
```
Return the image mode.
## udf point()
```python Signature theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
@pxt.udf
point(
self: pxt.Image,
lut: pxt.Json[(Int, ...)],
mode: pxt.String | None = None
) -> pxt.Image
```
Map image pixels through a lookup table.
Equivalent to
[`PIL.Image.Image.point()`](https://pillow.readthedocs.io/en/stable/reference/Image.html#PIL.Image.Image.point)
**Parameters:**
* **`lut`** (`pxt.Json[(Int`): A lookup table.
## udf quantize()
```python Signature theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
@pxt.udf
quantize(
self: pxt.Image,
colors: pxt.Int = 256,
method: pxt.Int | None = None,
kmeans: pxt.Int = 0,
palette: pxt.Image | None = None,
dither: pxt.Int =
) -> pxt.Image
```
Convert the image to 'P' mode with the specified number of colors.
Equivalent to
[`PIL.Image.Image.quantize()`](https://pillow.readthedocs.io/en/stable/reference/Image.html#PIL.Image.Image.quantize)
**Parameters:**
* **`colors`** (`pxt.Int`): The number of colors to quantize to.
* **`method`** (`pxt.Int | None`): The quantization method. See the
[Pillow documentation](https://pillow.readthedocs.io/en/stable/reference/Image.html#PIL.Image.Image.quantize)
for a list of supported methods.
* **`kmeans`** (`pxt.Int`): The number of k-means clusters to use.
* **`palette`** (`pxt.Image | None`): The palette to use.
* **`dither`** (`pxt.Int`): The dithering method. See the
[Pillow documentation](https://pillow.readthedocs.io/en/stable/reference/Image.html#PIL.Image.Image.quantize)
for a list of supported methods.
## udf reduce()
```python Signature theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
@pxt.udf
reduce(
self: pxt.Image,
factor: pxt.Int,
box: pxt.Json[(Int, Int, Int, Int)] | None = None
) -> pxt.Image
```
Reduce the image by the given factor.
Equivalent to
[`PIL.Image.Image.reduce()`](https://pillow.readthedocs.io/en/stable/reference/Image.html#PIL.Image.Image.reduce)
**Parameters:**
* **`factor`** (`pxt.Int`): The reduction factor.
* **`box`** (`pxt.Json[(Int`): An optional 4-tuple of ints providing the source image region to be reduced. The values must be within
(0, 0, width, height) rectangle. If omitted or None, the entire source is used.
## udf resize()
```python Signature theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
@pxt.udf
resize(self: pxt.Image, size: pxt.Json[(Int, Int)]) -> pxt.Image
```
Return a resized copy of the image. The size parameter is a tuple containing the width and height of the new image.
Equivalent to
[`PIL.Image.Image.resize()`](https://pillow.readthedocs.io/en/stable/reference/Image.html#PIL.Image.Image.resize)
## udf rotate()
```python Signature theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
@pxt.udf
rotate(self: pxt.Image, angle: pxt.Int) -> pxt.Image
```
Return a copy of the image rotated by the given angle.
Equivalent to
[`PIL.Image.Image.rotate()`](https://pillow.readthedocs.io/en/stable/reference/Image.html#PIL.Image.Image.rotate)
**Parameters:**
* **`angle`** (`pxt.Int`): The angle to rotate the image, in degrees. Positive angles are counter-clockwise.
## udf thumbnail()
```python Signature theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
@pxt.udf
thumbnail(
self: pxt.Image,
size: pxt.Json[(Int, Int)],
resample: pxt.Int = ,
reducing_gap: pxt.Float | None = 2.0
) -> pxt.Image
```
Create a thumbnail of the image.
Equivalent to
[`PIL.Image.Image.thumbnail()`](https://pillow.readthedocs.io/en/stable/reference/Image.html#PIL.Image.Image.thumbnail)
**Parameters:**
* **`size`** (`pxt.Json[(Int`): The size of the thumbnail, as a tuple of (width, height).
* **`resample`** (`Any`): The resampling filter to use. See the
[Pillow documentation](https://pillow.readthedocs.io/en/stable/reference/Image.html#PIL.Image.Image.thumbnail)
for a list of supported filters.
* **`reducing_gap`** (`Any`): The reducing gap to use.
## udf to\_video()
```python Signature theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
@pxt.udf
to_video(
image: pxt.Image,
*,
duration: pxt.Float,
fps: pxt.Int = 24,
video_encoder: pxt.String | None = None,
video_encoder_args: pxt.Json | None = None
) -> pxt.Video
```
Convert a still image into a video of a specified duration with ffmpeg's `-loop` option.
**Requirements:**
* `ffmpeg` needs to be installed and in PATH
**Parameters:**
* **`image`** (`pxt.Image`): Input image to convert to video.
* **`duration`** (`pxt.Float`): Duration of the output video in seconds.
* **`fps`** (`pxt.Int`): Frames per second for the output video.
* **`video_encoder`** (`pxt.String | None`): Video encoder to use. If not specified, uses the default encoder.
* **`video_encoder_args`** (`pxt.Json | None`): Additional arguments to pass to the video encoder.
**Returns:**
* `pxt.Video`: A video displaying the input image for the specified duration.
**Examples:**
Create a 5-second video from an image:
```python theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
tbl.select(tbl.image.to_video(duration=5.0)).collect()
```
Create a 10-second video at 30 fps from a rotated image:
```python theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
tbl.select(
tbl.image.rotate(180).to_video(duration=10.0, fps=30)
).collect()
```
## udf transpose()
```python Signature theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
@pxt.udf
transpose(self: pxt.Image, method: pxt.Int) -> pxt.Image
```
Transpose the image.
Equivalent to
[`PIL.Image.Image.transpose()`](https://pillow.readthedocs.io/en/stable/reference/Image.html#PIL.Image.Image.transpose)
**Parameters:**
* **`method`** (`pxt.Int`): The transpose method. See the
[Pillow documentation](https://pillow.readthedocs.io/en/stable/reference/Image.html#PIL.Image.Image.transpose)
for a list of supported methods.
## udf width()
```python Signature theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
@pxt.udf
width(self: pxt.Image) -> pxt.Int
```
Return the width of the image.
# IndexMetadata
Source: https://docs.pixeltable.com/sdk/latest/indexmetadata
# class pixeltable.IndexMetadata
Metadata for an index on a Pixeltable table.
## attr columns
```
columns: list[str]
```
The table columns that are indexed.
## attr index\_type
```
index_type: Literal['embedding', 'btree']
```
The type of index. New types may be added in the future.
## attr name
```
name: str
```
The name of the index.
## attr parameters
```
parameters: EmbeddingIndexParams | None
```
Parameters specific to the index type. `None` for index types without parameters.
# io
Source: https://docs.pixeltable.com/sdk/latest/io
# module pixeltable.io
Functions for importing and exporting Pixeltable data.
## func export\_csv()
```python Signature theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
export_csv(
table_or_query: pxt.Table | pxt.Query,
file_path: str | Path,
*,
delimiter: str = ',',
quoting: int = 0
) -> None
```
Exports a query result or table to a CSV file.
Pixeltable column types are mapped to CSV values as follows:
* String, Int, Float, Bool: native CSV representation
* Timestamp, Date: ISO 8601 string representation
* UUID: string representation
* Json: JSON-encoded string
* Array: JSON-encoded string (via `tolist()`)
* Binary: excluded from export (not representable in CSV)
* Image, Video, Audio, Document: file path or URL string
**Parameters:**
* **`table_or_query`** (`pxt.Table | pxt.Query`): Table or Query to export.
* **`file_path`** (`str | Path`): Path to the output CSV file.
* **`delimiter`** (`str`, default: `','`): Field delimiter character. Default `','`.
* **`quoting`** (`int`, default: `0`): CSV quoting style (a `csv.QUOTE_*` constant). Default `csv.QUOTE_MINIMAL`.
## func export\_iceberg()
```python Signature theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
export_iceberg(
table_or_query: pxt.Table | pxt.Query,
catalog: Catalog,
table_name: str,
*,
batch_size_bytes: int = 134217728,
if_exists: Literal['error', 'replace', 'append'] = 'error',
schema_overrides: Mapping[str, pa.DataType] | None = None
) -> None
```
Exports a query result or table to an Apache Iceberg table.
Data is streamed into the Iceberg table via pyarrow
[`RecordBatches`](https://arrow.apache.org/docs/python/generated/pyarrow.RecordBatch.html), the size of
which can be controlled with the `batch_size_bytes` parameter.
**Requirements:**
* `pip install pyiceberg`
**Parameters:**
* **`table_or_query`** (`pxt.Table | pxt.Query`): Pixeltable `Table` or `Query` to export.
* **`catalog`** (`Catalog`): An Iceberg `Catalog` instance to write the table into.
* **`table_name`** (`str`): Fully-qualified Iceberg table identifier (e.g. `'my_namespace.my_table'`). If the namespace
does not exist, it will be created.
* **`batch_size_bytes`** (`int`, default: `134217728`): Maximum size in bytes for each in-memory pyarrow batch.
* **`if_exists`** (`Literal['error', 'replace', 'append']`, default: `'error'`): Determines the behavior if the table already exists. Must be one of the following:
* `'error'`: raise an error
* `'replace'`: drop the existing table and create a new one
* `'append'`: append to the existing table (source schema must be compatible)
* **`schema_overrides`** (`Mapping[str, pa.DataType] | None`): If specified, then for each (name, type) pair in `schema_overrides`, the column with
name `name` will be given type `type`, instead of being inferred from the Pixeltable schema. The keys in
`schema_overrides` should be the column names of the Pixeltable schema.
## func export\_images\_as\_fo\_dataset()
```python Signature theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
export_images_as_fo_dataset(
tbl: pxt.Table,
images: exprs.Expr,
image_format: str = 'webp',
classifications: exprs.Expr | list[exprs.Expr] | dict[str, exprs.Expr] | None = None,
detections: exprs.Expr | list[exprs.Expr] | dict[str, exprs.Expr] | None = None
) -> fo.Dataset
```
Export images from a Pixeltable table as a Voxel51 dataset. The data must consist of a single column
(or expression) containing image data, along with optional additional columns containing labels. Currently, only
classification and detection labels are supported.
The [Working with Voxel51 in Pixeltable](https://docs.pixeltable.com/howto/working-with-fiftyone)
tutorial contains a fully worked example showing how to export data from a Pixeltable table and
load it into Voxel51.
Images in the dataset that already exist on disk will be exported directly, in whatever format they
are stored in. Images that are not already on disk (such as frames extracted using a
[`frame_iterator`](./video#iterator-frame_iterator)) will first be written to disk in the specified
`image_format`.
The label parameters accept one or more sets of labels of each type. If a single `Expr` is provided, then it will
be exported as a single set of labels with a default name such as `classifications`.
(The single set of labels may still containing multiple individual labels; see below.)
If a list of `Expr`s is provided, then each one will be exported as a separate set of labels with a default name
such as `classifications`, `classifications_1`, etc. If a dictionary of `Expr`s is provided, then each entry will
be exported as a set of labels with the specified name.
**Requirements:**
* `pip install fiftyone`
**Parameters:**
* **`tbl`** (`pxt.Table`): The table from which to export data.
* **`images`** (`exprs.Expr`): A column or expression that contains the images to export.
* **`image_format`** (`str`, default: `'webp'`): The format to use when writing out images for export.
* **`classifications`** (`exprs.Expr | list[exprs.Expr] | dict[str, exprs.Expr] | None`): Optional image classification labels. If a single `Expr` is provided, it must be a table
column or an expression that evaluates to a list of dictionaries. Each dictionary in the list corresponds
to an image class and must have the following structure:
```python theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
{'label': 'zebra', 'confidence': 0.325}
```
If multiple `Expr`s are provided, each one must evaluate to a list of such dictionaries.
* **`detections`** (`exprs.Expr | list[exprs.Expr] | dict[str, exprs.Expr] | None`): Optional image detection labels. If a single `Expr` is provided, it must be a table column or an
expression that evaluates to a list of dictionaries. Each dictionary in the list corresponds to an image
detection, and must have the following structure:
```python theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
{
'label': 'giraffe',
'confidence': 0.99,
# [x, y, w, h], fractional coordinates
'bounding_box': [0.081, 0.836, 0.202, 0.136],
}
```
If multiple `Expr`s are provided, each one must evaluate to a list of such dictionaries.
**Returns:**
* `'fo.Dataset'`: A Voxel51 dataset.
**Examples:**
Export the images in the `image` column of the table `tbl` as a Voxel51 dataset, using classification
labels from `tbl.classifications`:
```python theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
export_images_as_fo_dataset(
tbl, tbl.image, classifications=tbl.classifications
)
```
See the [Working with Voxel51 in Pixeltable](https://docs.pixeltable.com/howto/working-with-fiftyone)
tutorial for a fully worked example.
## func export\_json()
```python Signature theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
export_json(
table_or_query: pxt.Table | pxt.Query,
file_path: str | Path
) -> None
```
Exports a query result or table to a JSONL file.
Pixeltable column types are mapped to JSON values as follows:
* String: string
* Int: number
* Float: number
* Bool: boolean
* Timestamp: ISO 8601 string
* Date: ISO 8601 string
* UUID: string
* Json: native JSON value (object, array, etc.)
* Array: nested JSON array (via `tolist()`)
* Binary: excluded from export (not representable in JSON)
* Image, Video, Audio, Document: file path or URL string
**Parameters:**
* **`table_or_query`** (`pxt.Table | pxt.Query`): Table or Query to export.
* **`file_path`** (`str | Path`): Path to the output JSONL file.
## func export\_lancedb()
```python Signature theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
export_lancedb(
table_or_query: pxt.Table | pxt.Query,
db_uri: Path,
table_name: str,
batch_size_bytes: int = 134217728,
if_exists: Literal['error', 'overwrite', 'append'] = 'error'
) -> None
```
Exports a Query's data to a LanceDB table.
This utilizes LanceDB's streaming interface for efficient table creation, via a sequence of in-memory pyarrow
`RecordBatches`, the size of which can be controlled with the `batch_size_bytes` parameter.
**Requirements:**
* `pip install lancedb pylance`
**Parameters:**
* **`table_or_query `** (`Any`): Table or Query to export.
* **`db_uri`** (`Path`): Local Path to the LanceDB database.
* **`table_name `** (`Any`): Name of the table in the LanceDB database.
* **`batch_size_bytes `** (`Any`): Maximum size in bytes for each batch.
* **`if_exists`** (`Literal['error', 'overwrite', 'append']`, default: `'error'`): Determines the behavior if the table already exists. Must be one of the following:
* `'error'`: raise an error
* `'overwrite'`: overwrite the existing table
* `'append'`: append to the existing table
## func export\_parquet()
```python Signature theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
export_parquet(
table_or_query: pxt.Table | pxt.Query,
parquet_path: Path,
partition_size_bytes: int = 100000000,
inline_images: bool = False,
_write_md: bool = False
) -> None
```
Exports a query result or table to one or more Parquet files. Requires pyarrow to be installed.
Pixeltable column types are mapped to Parquet types as follows:
* String: string
* Int: int64
* Float: float32
* Bool: bool
* Timestamp: timestamp\[us, tz=UTC]
* Date: date32
* UUID: uuid
* Binary: binary
* Image: binary (when `inline_images=True`)
* Audio, Video, Document: string (file paths)
* Array (requires shape to be known):
* fixed\_shape\_tensor for fixed-shape arrays
* list for ragged arrays (one or more dimensions are None)
* Json: struct
* Schema is inferred from data via `pyarrow.infer_type()`
* Fields that contain empty dicts cannot be mapped to a Parquet type and will result in an exception
**Parameters:**
* **`table_or_query `** (`Any`): Table or Query to export.
* **`parquet_path `** (`Any`): Path to directory to write the parquet files to.
* **`partition_size_bytes `** (`Any`): The maximum target size for each chunk. Default 100\_000\_000 bytes.
* **`inline_images `** (`Any`): If True, images are stored inline in the parquet file. This is useful
for small images, to be imported as pytorch dataset. But can be inefficient
for large images, and cannot be imported into pixeltable.
If False, will raise an error if the Query has any image column.
Default False.
## func export\_sql()
```python Signature theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
export_sql(
table_or_query: pixeltable.catalog.table.Table | pixeltable._query.Query,
target_table_name: str,
*,
db_connect_str: str,
target_schema_name: str | None = None,
if_exists: Literal['error', 'replace', 'insert'] = 'error',
if_not_exists: Literal['error', 'create'] = 'create'
) -> None
```
Exports a query result or table to an RDBMS table.
**Parameters:**
* **`table_or_query `** (`Any`): Table or Query to export.
* **`target_table_name `** (`Any`): Name of the target table.
* **`db_connect_str `** (`Any`): Connection string to the target database.
* **`target_schema_name `** (`Any`): Optional name of the target schema.
* **`if_exists `** (`Any`): What to do if the target table already exists.
* 'error': raise an error
* 'replace': drop the existing table and create a new one
* 'insert': insert new rows into the existing table
* **`if_not_exists `** (`Any`): What to do if the target table does not exist.
* 'error': raise an error
* 'create': create the table from the source schema
## func import\_csv()
```python Signature theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
import_csv(
tbl_name: str,
filepath_or_buffer: str | os.PathLike,
schema_overrides: dict[str, Any] | None = None,
primary_key: str | list[str] | None = None,
comment: str | None = None,
**kwargs: Any
) -> pxt.Table
```
Creates a new base table from a csv file. This is a convenience method and is equivalent
to calling `import_pandas(table_path, pd.read_csv(filepath_or_buffer, **kwargs), schema=schema)`.
See the Pandas documentation for [`read_csv`](https://pandas.pydata.org/docs/reference/api/pandas.read_csv.html)
for more details.
**Returns:**
* `pxt.Table`: A handle to the newly created [`Table`](./table).
## func import\_excel()
```python Signature theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
import_excel(
tbl_name: str,
io: str | os.PathLike,
*,
schema_overrides: dict[str, Any] | None = None,
primary_key: str | list[str] | None = None,
comment: str = '',
**kwargs: Any
) -> pxt.Table
```
Creates a new base table from an Excel (.xlsx) file. This is a convenience method and is
equivalent to calling `import_pandas(table_path, pd.read_excel(io, *args, **kwargs), schema=schema)`.
See the Pandas documentation for [`read_excel`](https://pandas.pydata.org/docs/reference/api/pandas.read_excel.html)
for more details.
**Returns:**
* `pxt.Table`: A handle to the newly created [`Table`](./table).
## func import\_huggingface\_dataset()
```python Signature theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
import_huggingface_dataset(
table_path: str,
dataset: datasets.Dataset | datasets.DatasetDict | datasets.IterableDataset | datasets.IterableDatasetDict,
*,
schema_overrides: dict[str, Any] | None = None,
primary_key: str | list[str] | None = None,
**kwargs: Any
) -> pxt.Table
```
Create a new base table from a Huggingface dataset, or dataset dict with multiple splits.
Requires `datasets` library to be installed.
HuggingFace feature types are mapped to Pixeltable column types as follows:
* `Value(bool)`: `Bool`
`Value(int*/uint*)`: `Int`
`Value(float*)`: `Float`
`Value(string/large_string)`: `String`
`Value(timestamp*)`: `Timestamp`
`Value(date*)`: `Date`
* `ClassLabel`: `String` (converted to label names)
* `Sequence`/`LargeList` of numeric types: `Array`
* `Sequence`/`LargeList` of string: `Json`
* `Sequence`/`LargeList` of dicts: `Json`
* `Array2D`-`Array5D`: `Array` (preserves shape)
* `Image`: `Image`
* `Audio`: `Audio`
* `Video`: `Video`
* `Translation`/`TranslationVariableLanguages`: `Json`
**Parameters:**
* **`table_path`** (`str`): Path to the table.
* **`dataset`** (`datasets.Dataset | datasets.DatasetDict | datasets.IterableDataset | datasets.IterableDatasetDict`): An instance of any of the Huggingface dataset classes:
[`datasets.Dataset`](https://huggingface.co/docs/datasets/en/package_reference/main_classes#datasets.Dataset),
[`datasets.DatasetDict`](https://huggingface.co/docs/datasets/en/package_reference/main_classes#datasets.DatasetDict),
[`datasets.IterableDataset`](https://huggingface.co/docs/datasets/en/package_reference/main_classes#datasets.IterableDataset),
[`datasets.IterableDatasetDict`](https://huggingface.co/docs/datasets/en/package_reference/main_classes#datasets.IterableDatasetDict)
* **`schema_overrides`** (`dict[str, Any] | None`): If specified, then for each (name, type) pair in `schema_overrides`, the column with
name `name` will be given type `type`, instead of being inferred from the `Dataset` or `DatasetDict`.
The keys in `schema_overrides` should be the column names of the `Dataset` or `DatasetDict` (whether or not
they are valid Pixeltable identifiers).
* **`primary_key`** (`str | list[str] | None`): The primary key of the table (see [`create_table()`](./pixeltable#func-create_table)).
* **`kwargs`** (`Any`): Additional arguments to pass to `create_table`.
An argument of `column_name_for_split` must be provided if the source is a DatasetDict.
This column name will contain the split information. If None, no split information will be stored.
**Returns:**
* `pxt.Table`: A handle to the newly created [`Table`](./table).
## func import\_json()
```python Signature theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
import_json(
tbl_path: str,
filepath_or_url: str,
*,
schema_overrides: dict[str, Any] | None = None,
primary_key: str | list[str] | None = None,
comment: str | None = None,
**kwargs: Any
) -> pxt.Table
```
Creates a new base table from a JSON file. This is a convenience method and is
equivalent to calling [`create_table()`](./pixeltable#func-create_table) with
`pxt.create_table(tbl_path, source=filepath_or_url, extra_args=kwargs, ...)`.
The contents of `filepath_or_url` are read and parsed as JSON internally (using `json.loads(**kwargs)`).
**Parameters:**
* **`tbl_path`** (`str`): The name of the table to create.
* **`filepath_or_url`** (`str`): The path or URL of the JSON file.
* **`schema_overrides`** (`dict[str, Any] | None`): If specified, then columns in `schema_overrides` will be given the specified types
(see [`import_rows()`](./io#func-import_rows)).
* **`primary_key`** (`str | list[str] | None`): The primary key of the table (see [`create_table()`](./pixeltable#func-create_table)).
* **`comment`** (`str | None`): A comment to attach to the table (see [`create_table()`](./pixeltable#func-create_table)).
* **`kwargs`** (`Any`): Additional keyword arguments to pass to `json.loads`.
**Returns:**
* `pxt.Table`: A handle to the newly created [`Table`](./table).
## func import\_pandas()
```python Signature theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
import_pandas(
tbl_name: str,
df: pd.DataFrame,
*,
schema_overrides: dict[str, Any] | None = None,
primary_key: str | list[str] | None = None,
comment: str = ''
) -> pxt.Table
```
Creates a new base table from a Pandas
[`DataFrame`](https://pandas.pydata.org/docs/reference/api/pandas.DataFrame.html), with the
specified name. The schema of the table will be inferred from the DataFrame.
The column names of the new table will be identical to those in the DataFrame, as long as they are valid
Pixeltable identifiers. If a column name is not a valid Pixeltable identifier, it will be normalized according to
the following procedure:
* first replace any non-alphanumeric characters with underscores;
* then, preface the result with the letter 'c' if it begins with a number or an underscore;
* then, if there are any duplicate column names, suffix the duplicates with '\_2', '\_3', etc., in column order.
**Parameters:**
* **`tbl_name`** (`str`): The name of the table to create.
* **`df`** (`pd.DataFrame`): The Pandas `DataFrame`.
* **`schema_overrides`** (`dict[str, Any] | None`): If specified, then for each (name, type) pair in `schema_overrides`, the column with
name `name` will be given type `type`, instead of being inferred from the `DataFrame`. The keys in
`schema_overrides` should be the column names of the `DataFrame` (whether or not they are valid
Pixeltable identifiers).
**Returns:**
* `pxt.Table`: A handle to the newly created [`Table`](./table).
## func import\_parquet()
```python Signature theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
import_parquet(
table: str,
*,
parquet_path: str,
schema_overrides: dict[str, Any] | None = None,
primary_key: str | list[str] | None = None,
**kwargs: Any
) -> pxt.Table
```
Creates a new base table from a Parquet file or set of files. Requires pyarrow to be installed.
**Parameters:**
* **`table`** (`str`): Fully qualified name of the table to import the data into.
* **`parquet_path`** (`str`): Path to an individual Parquet file or directory of Parquet files.
* **`schema_overrides`** (`dict[str, Any] | None`): If specified, then for each (name, type) pair in `schema_overrides`, the column with
name `name` will be given type `type`, instead of being inferred from the Parquet dataset. The keys in
`schema_overrides` should be the column names of the Parquet dataset (whether or not they are valid
Pixeltable identifiers).
* **`primary_key`** (`str | list[str] | None`): The primary key of the table (see [`create_table()`](./pixeltable#func-create_table)).
* **`kwargs`** (`Any`): Additional arguments to pass to `create_table`.
**Returns:**
* `pxt.Table`: A handle to the newly created table.
## func import\_rows()
```python Signature theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
import_rows(
tbl_path: str,
rows: list[dict[str, Any]],
*,
schema_overrides: dict[str, Any] | None = None,
primary_key: str | list[str] | None = None,
comment: str = ''
) -> pxt.Table
```
Creates a new base table from a list of dictionaries. The dictionaries must be of the
form `{column_name: value, ...}`. Pixeltable will attempt to infer the schema of the table from the
supplied data, using the most specific type that can represent all the values in a column.
If `schema_overrides` is specified, then for each entry `(column_name, type)` in `schema_overrides`,
Pixeltable will force the specified column to the specified type (and will not attempt any type inference
for that column).
All column types of the new table will be nullable unless explicitly specified as non-nullable in
`schema_overrides`.
**Parameters:**
* **`tbl_path`** (`str`): The qualified name of the table to create.
* **`rows`** (`list[dict[str, Any]]`): The list of dictionaries to import.
* **`schema_overrides`** (`dict[str, Any] | None`): If specified, then columns in `schema_overrides` will be given the specified types
as described above.
* **`primary_key`** (`str | list[str] | None`): The primary key of the table (see [`create_table()`](./pixeltable#func-create_table)).
* **`comment`** (`str`, default: `''`): A comment to attach to the table (see [`create_table()`](./pixeltable#func-create_table)).
**Returns:**
* `pxt.Table`: A handle to the newly created [`Table`](./table).
## func import\_sql()
```python Signature theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
import_sql(
selectable: sqlalchemy.sql.selectable.Selectable,
conn: sqlalchemy.engine.base.Engine | sqlalchemy.engine.base.Connection,
tbl_name: str,
*,
schema_overrides: dict[str, typing.Any] | None = None,
primary_key: str | list[str] | None = None,
comment: str | None = None,
custom_metadata: Any = None,
if_exists: Literal['error', 'append'] = 'error',
on_error: Literal['abort', 'ignore'] = 'abort',
send_connect_url: bool = False
) -> pixeltable.catalog.table.Table
```
Import a SQL source into a Pixeltable table.
Rows are streamed from the source via a server-side cursor and inserted in batches.
**Parameters:**
* **`selectable`** (`sqlalchemy.sql.selectable.Selectable`): A SQLAlchemy `Selectable` (a `Table`, a `select()` statement, or `text(...).columns(...)`)
describing the source rows.
* **`conn`** (`sqlalchemy.engine.base.Engine | sqlalchemy.engine.base.Connection`): A SQLAlchemy `Engine` or `Connection` to execute `selectable` against. If a `Connection` is
passed, it must remain open and untouched (no commits, rollbacks, or other statements) for the
duration of the import; rows are streamed directly from the server-side cursor.
* **`tbl_name`** (`str`): Pixeltable path of the destination table.
* **`schema_overrides`** (`dict[str, typing.Any] | None`): Optional per-column overrides applied on top of the inferred schema. Keys are column
names; values accept any Pixeltable type spec recognized by `pxt.create_table` (eg, `pxt.Image`,
`pxt.String | None`).
* **`primary_key`** (`str | list[str] | None`): An optional column name or list of column names to use as the primary key(s) of the
table. Only applies when a new table is created; ignored when appending to an existing table.
* **`comment`** (`str | None`): An optional comment; its meaning is user-defined. Only applies when a new table is created;
ignored when appending to an existing table.
* **`custom_metadata`** (`typing.Any`): Optional user-defined metadata to associate with the table. Must be a valid
JSON-serializable object \[str, int, float, bool, dict, list]. Only applies when a new table is
created; ignored when appending to an existing table.
* **`if_exists`** (`typing.Literal['error', 'append']`, default: `'error'`): How to handle the destination table.
* `'error'`: create the table; fail if it already exists.
* `'append'`: append into the table if it already exists (verifying the source schema is
compatible); otherwise create it.
* **`on_error`** (`typing.Literal['abort', 'ignore']`, default: `'abort'`): Determines the behavior if an error occurs while evaluating a computed column or detecting an
invalid media file (such as a corrupt image) for one of the inserted rows.
* If `on_error='abort'`, then an exception will be raised and the rows will not be inserted.
* If `on_error='ignore'`, then execution will continue and the rows will be inserted. Any cells
with errors will have a `None` value for that cell, with information about the error stored in the
corresponding `tbl.col_name.errortype` and `tbl.col_name.errormsg` fields.
* **`send_connect_url`** (`bool`, default: `False`): Determines where the query is run, and is only relevant when the destination table is hosted.
It has no effect on a local table.
* If `False`, the query is run locally and the result is sent to the server.
* If `True`, the server connects to the source database itself (using the source connection URL, including
credentials); this requires the server's host to be able to reach the database and is faster for large
imports.
**Returns:**
* `pixeltable.catalog.table.Table`: The destination `Table`.
# jina
Source: https://docs.pixeltable.com/sdk/latest/jina
# module pixeltable.functions.jina
Pixeltable [UDFs](https://docs.pixeltable.com/platform/udfs-in-pixeltable) that wrap [Jina AI](https://jina.ai/) APIs
for embeddings and reranking. In order to use them, the API key must be specified either with `JINA_API_KEY`
environment variable, or as `api_key` in the `jina` section of the Pixeltable config file.
## udf embeddings()
```python Signature theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
@pxt.udf
embeddings(
input: pxt.String,
*,
model: pxt.String,
task: pxt.String | None = None,
dimensions: pxt.Int | None = None,
late_chunking: pxt.Bool | None = None
) -> pxt.Array[(None,), float32]
```
Creates embedding vectors for the input text using Jina AI embedding models.
Equivalent to the Jina AI embeddings API endpoint.
For additional details, see: [https://jina.ai/embeddings/](https://jina.ai/embeddings/)
Request throttling:
Applies the rate limit set in the config (section `jina`, key `rate_limit`). If no rate
limit is configured, uses a default of 600 RPM.
**Parameters:**
* **`input`** (`pxt.String`): The text to embed.
* **`model`** (`pxt.String`): The Jina embedding model to use. See available models at
[https://jina.ai/embeddings/](https://jina.ai/embeddings/).
* **`task`** (`pxt.String | None`): Task-specific embedding optimization. Options:
* `retrieval.query`: For search queries
* `retrieval.passage`: For documents/passages to be searched
* `separation`: For clustering/separation tasks
* `classification`: For classification tasks
* `text-matching`: For semantic similarity
* **`dimensions`** (`pxt.Int | None`): Output embedding dimensions (optional). If not specified, uses
the model's default dimension.
* **`late_chunking`** (`pxt.Bool | None`): Enable late chunking for long documents.
**Returns:**
* `pxt.Array[(None,), float32]`: An array representing the embedding of `input`.
**Examples:**
Add a computed column that applies jina-embeddings-v3 to an existing column:
```python theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
tbl.add_computed_column(
embed=jina.embeddings(
tbl.text, model='jina-embeddings-v3', task='retrieval.passage'
)
)
```
Add an embedding index:
```python theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
tbl.add_embedding_index(
'text', string_embed=jina.embeddings.using(model='jina-embeddings-v3')
)
```
## udf rerank()
```python Signature theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
@pxt.udf
rerank(
query: pxt.String,
documents: pxt.Json[(String, ...)],
*,
model: pxt.String,
top_n: pxt.Int | None = None,
return_documents: pxt.Bool | None = None
) -> JinaRerankResponse
```
Reranks documents based on their relevance to a query using Jina AI reranker models.
Equivalent to the Jina AI rerank API endpoint.
For additional details, see: [https://jina.ai/reranker/](https://jina.ai/reranker/)
Request throttling:
Applies the rate limit set in the config (section `jina`, key `rate_limit`). If no rate
limit is configured, uses a default of 600 RPM.
**Parameters:**
* **`query`** (`pxt.String`): The query string to rank documents against.
* **`documents`** (`pxt.Json[(String`): The list of documents to rerank.
* **`model`** (`Any`): The Jina reranker model to use. See available models at
[https://jina.ai/reranker/](https://jina.ai/reranker/).
* **`top_n`** (`Any`): Number of top results to return. If not specified, returns all documents.
* **`return_documents`** (`Any`): Whether to include the original document text in results.
**Returns:**
* `JinaRerankResponse`: A dictionary containing:
* `results`: List of reranking results with `index` and `relevance_score`
(and `document` if `return_documents=True`)
* `usage`: Token usage information
**Examples:**
Rerank search results for better relevance:
```python theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
tbl.add_computed_column(
reranked=jina.rerank(
tbl.query,
tbl.candidate_docs,
model='jina-reranker-v2-base-multilingual',
top_n=5,
)
)
```
# json
Source: https://docs.pixeltable.com/sdk/latest/json
# module pixeltable.functions.json
Pixeltable UDFs for `JsonType`.
Example:
```python theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
import pixeltable as pxt
import pixeltable.functions as pxtf
t = pxt.get_table(...)
t.select(pxtf.json.make_list(t.json_col)).collect()
```
## iterator list\_iterator()
```python Signature theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
@pxt.iterator
list_iterator(
elements: pxt.Json[(Json, ...)] | None = None,
*,
mode: pxt.String = 'strict',
**kwargs
)
```
Iterator over elements of a list or lists. There are two distinct call patterns: either a single positional
argument; or one or more keyword arguments.
* If a single positional argument is specified, as in `list_iterator(t.col)`, then the elements of `t.col` must
contain lists of dictionaries with matching signatures (identical keys and compatible value types). The
iterator will yield one new column for each key in the dictionaries, and one output row per element in the
lists.
* If multiple keyword arguments are specified, as in `list_iterator(val_1=t.col_1, val_2=t.col_2)`, then the
elements of each input column must contain lists, but not necessarily lists of dictionaries. The iterator
will yield one new column for each keyword argument, zipping together the individual lists.
All of the inputs must be *typed* `Json` expressions. Untyped Json will be rejected (the type schema is
necessary in order for Pixeltable to determine the types of the output columns).
**Parameters:**
* **`elements`** (`pxt.Json[(Json`): A list of dictionaries to iterate over. The dictionary keys will be used as column names in the
output. Cannot be specified together with keyword arguments.
* **`mode`** (`Any`): Only applies when called with keyword arguments. Determines how to handle lists of different lengths:
* `'strict'`: Raises an error if the input lists have different lengths.
* `'truncated'`: Iterates until the shortest input list is exhausted, ignoring any remaining elements in
longer lists.
* `'padded'`: Iterates until the longest input list is exhausted, yielding `None` for any missing
elements from shorter lists.
* **`**kwargs`** (`Any`): One or more lists to iterate over. The kwarg names will be used as column names in the output.
Cannot be specified together with `elements`.
## uda make\_list()
```python Signature theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
@pxt.uda
make_list(*args, **kwargs) -> pxt.Json[(Json, ...)]
```
Collects arguments into a list.
## udf concat()
```python Signature theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
@pxt.udf
concat(self: pxt.Json | None, other: pxt.Json | None) -> pxt.Json
```
Concatenate two JSON arrays into a new array.
Returns `null` if either operand is `null`. Raises if either operand is a non-null, non-array value.
**Examples:**
```python theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
t.select(t.tags.concat(t.more_tags)).collect()
```
## udf contains()
```python Signature theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
@pxt.udf
contains(self: pxt.Json, value: pxt.Json) -> pxt.Bool
```
Return `True` if `value` is an element of a JSON array or a key of a JSON object; `False` otherwise.
**Examples:**
```python theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
t.where(t.detections.labels.contains('person')).collect()
```
## udf count()
```python Signature theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
@pxt.udf
count(self: pxt.Json, value: pxt.Json) -> pxt.Int
```
Return the number of times `value` occurs in a JSON array.
**Examples:**
```python theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
t.select(t.detections.labels.count('person')).collect()
```
## udf dumps()
```python Signature theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
@pxt.udf
dumps(obj: pxt.Json) -> pxt.String
```
Serialize a JSON object to a string.
Equivalent to [`json.dumps()`](https://docs.python.org/3/library/json.html#json.dumps).
**Parameters:**
* **`obj`** (`pxt.Json`): A JSON-serializable object (dict, list, or scalar).
**Returns:**
* `pxt.String`: A JSON-formatted string.
## udf flatten()
```python Signature theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
@pxt.udf
flatten(self: pxt.Json) -> pxt.Json
```
Concatenate the elements of a JSON array one level deep; non-array elements are kept as-is.
**Examples:**
```python theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
t.select(t.chunks.flatten()).collect()
```
## udf get()
```python Signature theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
@pxt.udf
get(
self: pxt.Json | None,
key: pxt.String,
default: pxt.Json | None = None
) -> pxt.Json
```
Return the value of `key` if the value is a JSON object containing it, otherwise `default`.
**Examples:**
```python theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
t.select(t.metadata.get('author', default='unknown')).collect()
```
## udf is\_empty()
```python Signature theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
@pxt.udf
is_empty(self: pxt.Json | None) -> pxt.Bool
```
Return `True` if the value is `null`, an empty array, an empty object, or an empty string; `False` otherwise
(including for numbers and booleans).
**Examples:**
```python theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
t.where(~t.detections.bboxes.is_empty()).collect()
```
## udf items()
```python Signature theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
@pxt.udf
items(self: pxt.Json) -> pxt.Json
```
Return the `[key, value]` pairs of a JSON object as a list. `keys()`, `values()`, and `items()` share the
same ordering.
**Examples:**
```python theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
t.select(t.metadata.items()).collect()
```
## udf keys()
```python Signature theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
@pxt.udf
keys(self: pxt.Json) -> pxt.Json[(String, ...)]
```
Return the keys of a JSON object. `keys()`, `values()`, and `items()` share the same ordering.
**Examples:**
```python theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
t.select(t.metadata.keys()).collect()
```
## udf len()
```python Signature theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
@pxt.udf
len(self: pxt.Json) -> pxt.Int
```
Return the number of elements in a JSON array, keys in a JSON object, or characters in a JSON string.
Not defined for numbers or booleans. A `null` value (or missing path) yields `null`.
**Examples:**
```python theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
t.select(t.detections.bboxes.len()).collect()
```
## udf max()
```python Signature theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
@pxt.udf
max(self: pxt.Json) -> pxt.Float | None
```
Return the largest number in a JSON array, or `null` if the array is empty.
**Examples:**
```python theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
t.select(t.detections.scores.max()).collect()
```
## udf mean()
```python Signature theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
@pxt.udf
mean(self: pxt.Json) -> pxt.Float | None
```
Return the arithmetic mean of the numbers in a JSON array, or `null` if the array is empty.
**Examples:**
```python theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
t.select(t.detections.scores.mean()).collect()
```
## udf merge()
```python Signature theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
@pxt.udf
merge(self: pxt.Json | None, other: pxt.Json | None) -> pxt.Json
```
Merge two JSON objects into a new object; on a key conflict the value from `other` wins.
Returns `null` if either operand is `null`. Raises if either operand is a non-null, non-object value.
**Examples:**
```python theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
t.select(t.defaults.merge(t.overrides)).collect()
```
## udf min()
```python Signature theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
@pxt.udf
min(self: pxt.Json) -> pxt.Float | None
```
Return the smallest number in a JSON array, or `null` if the array is empty.
**Examples:**
```python theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
t.select(t.detections.scores.min()).collect()
```
## udf sum()
```python Signature theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
@pxt.udf
sum(self: pxt.Json) -> pxt.Float
```
Return the sum of the numbers in a JSON array (0 for an empty array).
**Examples:**
```python theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
t.select(t.detections.scores.sum()).collect()
```
## udf values()
```python Signature theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
@pxt.udf
values(self: pxt.Json) -> pxt.Json
```
Return the values of a JSON object. `keys()`, `values()`, and `items()` share the same ordering.
**Examples:**
```python theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
t.select(t.metadata.values()).collect()
```
# llama_cpp
Source: https://docs.pixeltable.com/sdk/latest/llama_cpp
# module pixeltable.functions.llama\_cpp
Pixeltable UDFs for llama.cpp models.
Provides integration with llama.cpp for running quantized language models locally,
supporting chat completions and embeddings with GGUF format models.
## udf create\_chat\_completion()
```python Signature theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
@pxt.udf
create_chat_completion(
messages: pxt.Json[(Json, ...)],
*,
model_path: pxt.String | None = None,
repo_id: pxt.String | None = None,
repo_filename: pxt.String | None = None,
chat_format: pxt.String | None = None,
tools: pxt.Json[(Json, ...)] | None = None,
tool_choice: pxt.Json | None = None,
model_kwargs: pxt.Json | None = None
) -> pxt.Json
```
Generate a chat completion from a list of messages.
The model can be specified either as a local path, or as a repo\_id and repo\_filename that reference a pretrained
model on the Hugging Face model hub. Exactly one of `model_path` or `repo_id` must be provided; if `model_path`
is provided, then an optional `repo_filename` can also be specified.
For additional details, see the
[llama\_cpp create\_chat\_completion documentation](https://llama-cpp-python.readthedocs.io/en/latest/api-reference/#llama_cpp.Llama.create_chat_completion).
**Parameters:**
* **`messages`** (`pxt.Json[(Json`): A list of messages to generate a response for.
* **`model_path`** (`Any`): Path to the model (if using a local model).
* **`repo_id`** (`Any`): The Hugging Face model repo id (if using a pretrained model).
* **`repo_filename`** (`Any`): A filename or glob pattern to match the model file in the repo (optional, if using a
pretrained model).
* **`chat_format`** (`Any`): An optional string specifying the chat format to use with the model.
* **`tools`** (`Any`): An optional list of tools (functions) the model may call, specified as `pxt.func.tools.Tools`.
* **`tool_choice`** (`Any`): An optional `pxt.func.tools.ToolChoice` controlling which tool(s) the model should use.
* **`model_kwargs`** (`Any`): Additional keyword args for the llama\_cpp `create_chat_completion` API, such as `max_tokens`,
`temperature`, `top_p`, and `top_k`. For details, see the
[llama\_cpp create\_chat\_completion documentation](https://llama-cpp-python.readthedocs.io/en/latest/api-reference/#llama_cpp.Llama.create_chat_completion).
# math
Source: https://docs.pixeltable.com/sdk/latest/math
# module pixeltable.functions.math
Pixeltable UDFs for mathematical operations.
Example:
```python theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
import pixeltable as pxt
t = pxt.get_table(...)
t.select(t.float_col.floor()).collect()
```
## udf abs()
```python Signature theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
@pxt.udf
abs(self: pxt.Float) -> pxt.Float
```
Return the absolute value of the given number.
Equivalent to Python [`builtins.abs()`](https://docs.python.org/3/library/functions.html#abs).
## udf bitwise\_and()
```python Signature theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
@pxt.udf
bitwise_and(self: pxt.Int, other: pxt.Int) -> pxt.Int
```
Bitwise AND of two integers.
Equivalent to Python
[`self & other`](https://docs.python.org/3/library/stdtypes.html#bitwise-operations-on-integer-types).
## udf bitwise\_or()
```python Signature theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
@pxt.udf
bitwise_or(self: pxt.Int, other: pxt.Int) -> pxt.Int
```
Bitwise OR of two integers.
Equivalent to Python
[`self | other`](https://docs.python.org/3/library/stdtypes.html#bitwise-operations-on-integer-types).
## udf bitwise\_xor()
```python Signature theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
@pxt.udf
bitwise_xor(self: pxt.Int, other: pxt.Int) -> pxt.Int
```
Bitwise XOR of two integers.
Equivalent to Python
[`self ^ other`](https://docs.python.org/3/library/stdtypes.html#bitwise-operations-on-integer-types).
## udf ceil()
```python Signature theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
@pxt.udf
ceil(self: pxt.Float) -> pxt.Float
```
Return the ceiling of the given number.
Equivalent to Python [`float(math.ceil(self))`](https://docs.python.org/3/library/math.html#math.ceil) if `self`
is finite, or `self` itself if `self` is infinite. (This is slightly different from the default behavior of
`math.ceil(self)`, which always returns an `int` and raises an error if `self` is infinite. The behavior in
Pixeltable generalizes the Python operator and is chosen to align with the SQL standard.)
## udf floor()
```python Signature theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
@pxt.udf
floor(self: pxt.Float) -> pxt.Float
```
Return the ceiling of the given number.
Equivalent to Python [`float(math.floor(self))`](https://docs.python.org/3/library/math.html#math.ceil) if `self`
is finite, or `self` itself if `self` is infinite. (This is slightly different from the default behavior of
`math.floor(self)`, which always returns an `int` and raises an error if `self` is infinite. The behavior of
Pixeltable generalizes the Python operator and is chosen to align with the SQL standard.)
## udf pow()
```python Signature theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
@pxt.udf
pow(self: pxt.Float, other: pxt.Float) -> pxt.Float
```
Raise `self` to the power of `other`.
Equivalent to Python [`builtins.pow()`](https://docs.python.org/3/library/functions.html#pow).
## udf round()
```python Signature theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
@pxt.udf
round(
self: pxt.Float,
digits: pxt.Int | None = None
) -> pxt.Float
```
Round a number to a given precision in decimal digits.
Equivalent to Python [`builtins.round(self, digits or 0)`](https://docs.python.org/3/library/functions.html#round).
Note that if `digits` is not specified, the behavior matches `builtins.round(self, 0)` rather than
`builtins.round(self)`; this ensures that the return type is always `float` (as in SQL) rather than `int`.
# mistralai
Source: https://docs.pixeltable.com/sdk/latest/mistralai
# module pixeltable.functions.mistralai
Pixeltable UDFs
that wrap various endpoints from the Mistral AI API. In order to use them, you must
first `pip install mistralai` and configure your Mistral AI credentials, as described in
the [Working with Mistral AI](https://docs.pixeltable.com/howto/providers/working-with-mistralai) tutorial.
## udf chat\_completions()
```python Signature theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
@pxt.udf
chat_completions(
messages: pxt.Json[(Json, ...)],
*,
model: pxt.String,
model_kwargs: pxt.Json | None = None
) -> pxt.Json
```
Chat Completion API.
Equivalent to the Mistral AI `chat/completions` API endpoint.
For additional details, see: [https://docs.mistral.ai/api/#tag/chat](https://docs.mistral.ai/api/#tag/chat)
Request throttling:
Applies the rate limit set in the config (section `mistral`, key `rate_limit`). If no rate
limit is configured, uses a default of 600 RPM.
**Requirements:**
* `pip install mistralai`
**Parameters:**
* **`messages`** (`pxt.Json[(Json`): The prompt(s) to generate completions for.
* **`model`** (`Any`): ID of the model to use. (See overview here: [https://docs.mistral.ai/getting-started/models/](https://docs.mistral.ai/getting-started/models/))
* **`model_kwargs`** (`Any`): Additional keyword args for the Mistral `chat/completions` API.
For details on the available parameters, see: [https://docs.mistral.ai/api/#tag/chat](https://docs.mistral.ai/api/#tag/chat)
**Returns:**
* `pxt.Json`: A dictionary containing the response and other metadata.
**Examples:**
Add a computed column that applies the model `mistral-latest-small`
to an existing Pixeltable column `tbl.prompt` of the table `tbl`:
```python theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
messages = [{'role': 'user', 'content': tbl.prompt}]
tbl.add_computed_column(
response=completions(messages, model='mistral-latest-small')
)
```
## udf embeddings()
```python Signature theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
@pxt.udf
embeddings(
input: pxt.String,
*,
model: pxt.String
) -> pxt.Array[(None,), float32]
```
Embeddings API.
Equivalent to the Mistral AI `embeddings` API endpoint.
For additional details, see: [https://docs.mistral.ai/api/#tag/embeddings](https://docs.mistral.ai/api/#tag/embeddings)
Request throttling:
Applies the rate limit set in the config (section `mistral`, key `rate_limit`). If no rate
limit is configured, uses a default of 600 RPM.
**Requirements:**
* `pip install mistralai`
**Parameters:**
* **`input`** (`pxt.String`): Text to embed.
* **`model`** (`pxt.String`): ID of the model to use. (See overview here: [https://docs.mistral.ai/getting-started/models/](https://docs.mistral.ai/getting-started/models/))
**Returns:**
* `pxt.Array[(None,), float32]`: An array representing the application of the given embedding to `input`.
## udf fim\_completions()
```python Signature theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
@pxt.udf
fim_completions(
prompt: pxt.String,
*,
model: pxt.String,
model_kwargs: pxt.Json | None = None
) -> pxt.Json
```
Fill-in-the-middle Completion API.
Equivalent to the Mistral AI `fim/completions` API endpoint.
For additional details, see: [https://docs.mistral.ai/api/#tag/fim](https://docs.mistral.ai/api/#tag/fim)
Request throttling:
Applies the rate limit set in the config (section `mistral`, key `rate_limit`). If no rate
limit is configured, uses a default of 600 RPM.
**Requirements:**
* `pip install mistralai`
**Parameters:**
* **`prompt`** (`pxt.String`): The text/code to complete.
* **`model`** (`pxt.String`): ID of the model to use. (See overview here: [https://docs.mistral.ai/getting-started/models/](https://docs.mistral.ai/getting-started/models/))
* **`model_kwargs`** (`pxt.Json | None`): Additional keyword args for the Mistral `fim/completions` API.
For details on the available parameters, see: [https://docs.mistral.ai/api/#tag/fim](https://docs.mistral.ai/api/#tag/fim)
**Returns:**
* `pxt.Json`: A dictionary containing the response and other metadata.
**Examples:**
Add a computed column that applies the model `codestral-latest`
to an existing Pixeltable column `tbl.prompt` of the table `tbl`:
```python theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
tbl.add_computed_column(
response=completions(tbl.prompt, model='codestral-latest')
)
```
# nebius
Source: https://docs.pixeltable.com/sdk/latest/nebius
# module pixeltable.functions.nebius
Pixeltable UDFs for Nebius Token Factory models.
Provides integration with Nebius Token Factory's language and embedding models. In order to use
them, you must first `pip install openai` and configure your Nebius Token Factory API key, by
setting the `NEBIUS_API_KEY` environment variable or `nebius.api_key` in the Pixeltable config file.
## udf chat\_completions()
```python Signature theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
@pxt.udf
chat_completions(
messages: pxt.Json[(Json, ...)],
*,
model: pxt.String,
model_kwargs: pxt.Json | None = None,
tools: pxt.Json[(Json, ...)] | None = None,
tool_choice: pxt.Json | None = None
) -> pxt.Json
```
Creates a model response for the given chat conversation.
Equivalent to the Nebius Token Factory `chat/completions` API endpoint.
For additional details, see: [https://docs.tokenfactory.nebius.com/api-reference/inference/create-chat-completion](https://docs.tokenfactory.nebius.com/api-reference/inference/create-chat-completion)
Nebius Token Factory exposes an OpenAI-compatible API, so you will need to install the
`openai` package to use this UDF.
Request throttling:
Applies the rate limit set in the config (section `nebius`, key `rate_limit`). If no rate
limit is configured, a default rate limit is applied.
**Requirements:**
* `pip install openai`
**Parameters:**
* **`messages`** (`pxt.Json[(Json`): A list of messages to use for chat completion, as described in the Nebius API documentation.
* **`model`** (`Any`): The model to use for chat completion.
* **`model_kwargs`** (`Any`): Additional keyword args for the Nebius `chat/completions` API.
For details on the available parameters, see: [https://docs.tokenfactory.nebius.com/api-reference/inference/create-chat-completion](https://docs.tokenfactory.nebius.com/api-reference/inference/create-chat-completion)
* **`tools`** (`Any`): An optional list of Pixeltable tools to use for the request.
* **`tool_choice`** (`Any`): An optional tool choice configuration.
**Returns:**
* `pxt.Json`: A dictionary containing the response and other metadata.
**Examples:**
Add a computed column that applies the model `Qwen/Qwen3-30B-A3B-Instruct-2507`
to an existing Pixeltable column `tbl.prompt` of the table `tbl`:
```python theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
messages = [
{'role': 'system', 'content': 'You are a helpful assistant.'},
{'role': 'user', 'content': tbl.prompt},
]
tbl.add_computed_column(
response=chat_completions(
messages, model='Qwen/Qwen3-30B-A3B-Instruct-2507'
)
)
```
You can also include images in the messages list, for vision-capable models such as
`openbmb/MiniCPM-V-4_5`, by passing image data directly in the input dictionary, in
the `'image_url'` field of the message content, as in this example:
```python theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
messages = [
{
'role': 'user',
'content': [
{'type': 'text', 'text': "What's in this image?"},
{'type': 'image_url', 'image_url': tbl.image},
],
}
]
tbl.add_computed_column(
response=chat_completions(messages, model='openbmb/MiniCPM-V-4_5')
)
```
## udf embeddings()
```python Signature theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
@pxt.udf
embeddings(
input: pxt.String,
*,
model: pxt.String,
model_kwargs: pxt.Json | None = None
) -> pxt.Array[(None,), float32]
```
Query an embedding model for a given string of text.
Equivalent to the Nebius Token Factory `embeddings` API endpoint.
For additional details, see: [https://docs.tokenfactory.nebius.com/api-reference/inference/create-embeddings](https://docs.tokenfactory.nebius.com/api-reference/inference/create-embeddings)
Request throttling:
Applies the rate limit set in the config (section `nebius`, key `rate_limit`). If no rate
limit is configured, a default rate limit is applied.
**Requirements:**
* `pip install openai`
**Parameters:**
* **`input`** (`pxt.String`): A string providing the text for the model to embed.
* **`model`** (`pxt.String`): The name of the embedding model to use.
* **`model_kwargs`** (`pxt.Json | None`): Additional keyword args for the Nebius `embeddings` API, e.g. `dimensions` to
request a truncated embedding for models that support it (see the note below).
**Returns:**
* `pxt.Array[(None,), float32]`: An array representing the application of the given embedding to `input`.
**Examples:**
Add a computed column that applies the model `Qwen/Qwen3-Embedding-8B`
to an existing Pixeltable column `tbl.text` of the table `tbl`:
```python theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
tbl.add_computed_column(
response=embeddings(tbl.text, model='Qwen/Qwen3-Embedding-8B')
)
```
`Qwen/Qwen3-Embedding-8B` produces 4096-dimensional embeddings by default, which exceed
the maximum of 4000 dimensions supported by Pixeltable embedding indexes. Request a
smaller, indexable size via `model_kwargs`:
```python theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
tbl.add_embedding_index(
'text',
embedding=embeddings.using(
model='Qwen/Qwen3-Embedding-8B', model_kwargs={'dimensions': 1024}
),
)
```
# net
Source: https://docs.pixeltable.com/sdk/latest/net
# module pixeltable.functions.net
Pixeltable UDF for converting media file URIs to presigned HTTP URLs.
## udf presigned\_url()
```python Signature theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
@pxt.udf
presigned_url(uri: pxt.String, expiration_seconds: pxt.Int) -> pxt.String
```
Convert a blob storage URI to a presigned HTTP URL for direct access.
Generates a time-limited, publicly accessible URL from cloud storage URIs
(S3, GCS, Azure, etc.) that can be used to serve media files over HTTP.
Note:
This function uses presigned URLs from storage providers. Provider-specific
limitations apply:
* Google Cloud Storage: maximum 7-day expiration
* AWS S3: requires proper region configuration
* Azure: subject to storage account access policies
**Parameters:**
* **`uri`** (`pxt.String`): The media file URI (e.g., `s3://bucket/path`, `gs://bucket/path`, `azure://container/path`)
* **`expiration_seconds`** (`pxt.Int`): How long the URL remains valid
**Returns:**
* `pxt.String`: A presigned HTTP URL for accessing the file
**Examples:**
Generate a presigned URL for a video column with 1-hour expiration:
```python theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
tbl.select(
original_url=tbl.video.fileurl,
presigned_url=pxtf.net.presigned_url(tbl.video.fileurl, 3600),
).collect()
```
# ollama
Source: https://docs.pixeltable.com/sdk/latest/ollama
# module pixeltable.functions.ollama
Pixeltable UDFs for Ollama local models.
Provides integration with Ollama for running large language models locally,
including chat completions and embeddings.
## udf chat()
```python Signature theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
@pxt.udf
chat(
messages: pxt.Json[(Json, ...)],
*,
model: pxt.String,
tools: pxt.Json[(Json, ...)] | None = None,
format: pxt.String | None = None,
options: pxt.Json | None = None
) -> pxt.Json
```
Generate the next message in a chat with a provided model.
**Parameters:**
* **`messages`** (`pxt.Json[(Json`): The messages of the chat.
* **`model`** (`Any`): The model name.
* **`tools`** (`Any`): Tools for the model to use.
* **`format`** (`Any`): The format of the response; must be one of `'json'` or `None`.
* **`options`** (`Any`): Additional options to pass to the `chat` call, such as `max_tokens`, `temperature`, `top_p`, and
`top_k`. For details, see the
[Valid Parameters and Values](https://github.com/ollama/ollama/blob/main/docs/modelfile.mdx#valid-parameters-and-values)
section of the Ollama documentation.
## udf embed()
```python Signature theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
@pxt.udf
embed(
input: pxt.String,
*,
model: pxt.String,
truncate: pxt.Bool = True,
options: pxt.Json | None = None
) -> pxt.Array[(None,), float32]
```
Generate embeddings from a model.
**Parameters:**
* **`input`** (`pxt.String`): The input text to generate embeddings for.
* **`model`** (`pxt.String`): The model name.
* **`truncate`** (`pxt.Bool`): Truncates the end of each input to fit within context length.
Returns error if false and context length is exceeded.
* **`options`** (`pxt.Json | None`): Additional options to pass to the `embed` call.
For details, see the
[Valid Parameters and Values](https://github.com/ollama/ollama/blob/main/docs/modelfile.mdx#valid-parameters-and-values)
section of the Ollama documentation.
## udf generate()
```python Signature theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
@pxt.udf
generate(
prompt: pxt.String,
*,
model: pxt.String,
suffix: pxt.String = '',
system: pxt.String = '',
template: pxt.String = '',
context: pxt.Json[(Int, ...)] | None = None,
raw: pxt.Bool = False,
format: pxt.String | None = None,
options: pxt.Json | None = None
) -> pxt.Json
```
Generate a response for a given prompt with a provided model.
**Parameters:**
* **`prompt`** (`pxt.String`): The prompt to generate a response for.
* **`model`** (`pxt.String`): The model name.
* **`suffix`** (`pxt.String`): The text after the model response.
* **`format`** (`Any`): The format of the response; must be one of `'json'` or `None`.
* **`system`** (`pxt.String`): System message.
* **`template`** (`pxt.String`): Prompt template to use.
* **`context`** (`pxt.Json[(Int`): The context parameter returned from a previous call to `generate()`.
* **`raw`** (`Any`): If `True`, no formatting will be applied to the prompt.
* **`options`** (`Any`): Additional options for the Ollama `chat` call, such as `max_tokens`, `temperature`, `top_p`, and
`top_k`. For details, see the
[Valid Parameters and Values](https://github.com/ollama/ollama/blob/main/docs/modelfile.mdx#valid-parameters-and-values)
section of the Ollama documentation.
# openai
Source: https://docs.pixeltable.com/sdk/latest/openai
# module pixeltable.functions.openai
Pixeltable UDFs
that wrap various endpoints from the OpenAI API. In order to use them, you must
first `pip install openai` and configure your OpenAI credentials, as described in
the [Working with OpenAI](https://docs.pixeltable.com/howto/providers/working-with-openai) tutorial.
## func invoke\_tools()
```python Signature theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
invoke_tools(
tools: pixeltable.func.tools.Tools,
response: pixeltable.exprs.expr.Expr
) -> pixeltable.exprs.inline_expr.InlineDict
```
Converts an OpenAI response dict to Pixeltable tool invocation format and calls `tools._invoke()`.
## udf chat\_completions()
```python Signature theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
@pxt.udf
chat_completions(
messages: pxt.Json[(Json, ...)],
*,
model: pxt.String,
model_kwargs: pxt.Json | None = None,
tools: pxt.Json[(Json, ...)] | None = None,
tool_choice: pxt.Json | None = None
) -> pxt.Json
```
Creates a model response for the given chat conversation.
Equivalent to the OpenAI `chat/completions` API endpoint.
For additional details, see: [https://platform.openai.com/docs/guides/chat-completions](https://platform.openai.com/docs/guides/chat-completions)
Request throttling:
Uses the rate limit-related headers returned by the API to throttle requests adaptively, based on available
request and token capacity. No configuration is necessary.
**Requirements:**
* `pip install openai`
**Parameters:**
* **`messages`** (`pxt.Json[(Json`): A list of messages to use for chat completion, as described in the OpenAI API documentation.
* **`model`** (`Any`): The model to use for chat completion.
* **`model_kwargs`** (`Any`): Additional keyword args for the OpenAI `chat/completions` API. For details on the available
parameters, see: [https://platform.openai.com/docs/api-reference/chat/create](https://platform.openai.com/docs/api-reference/chat/create)
**Returns:**
* `pxt.Json`: A dictionary containing the response and other metadata.
**Examples:**
Add a computed column that applies the model `gpt-4o-mini` to an existing Pixeltable column `tbl.prompt`
of the table `tbl`:
```python theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
messages = [
{'role': 'system', 'content': 'You are a helpful assistant.'},
{'role': 'user', 'content': tbl.prompt},
]
tbl.add_computed_column(
response=chat_completions(messages, model='gpt-4o-mini')
)
```
You can also include images in the messages list by passing image data directly in the input dictionary, in
the `'image_url'` field of the message content, as in this example:
```python theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
messages = [
{
'role': 'user',
'content': [
{'type': 'text', 'text': "What's in this image?"},
{'type': 'image_url', 'image_url': tbl.image},
],
}
]
tbl.add_computed_column(
response=chat_completions(messages, model='gpt-4o-mini')
)
```
## udf embeddings()
```python Signature theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
@pxt.udf
embeddings(
input: pxt.String,
*,
model: pxt.String,
model_kwargs: pxt.Json | None = None
) -> pxt.Array[(None,), float32]
```
Creates an embedding vector representing the input text.
Equivalent to the OpenAI `embeddings` API endpoint.
For additional details, see: [https://platform.openai.com/docs/guides/embeddings](https://platform.openai.com/docs/guides/embeddings)
Request throttling:
Uses the rate limit-related headers returned by the API to throttle requests adaptively, based on available
request and token capacity. No configuration is necessary.
**Requirements:**
* `pip install openai`
**Parameters:**
* **`input`** (`pxt.String`): The text to embed.
* **`model`** (`pxt.String`): The model to use for the embedding.
* **`model_kwargs`** (`pxt.Json | None`): Additional keyword args for the OpenAI `embeddings` API. For details on the available
parameters, see: [https://platform.openai.com/docs/api-reference/embeddings](https://platform.openai.com/docs/api-reference/embeddings)
**Returns:**
* `pxt.Array[(None,), float32]`: An array representing the application of the given embedding to `input`.
**Examples:**
Add a computed column that applies the model `text-embedding-3-small` to an existing
Pixeltable column `tbl.text` of the table `tbl`:
```python theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
tbl.add_computed_column(
embed=embeddings(tbl.text, model='text-embedding-3-small')
)
```
Add an embedding index to an existing column `text`, using the model `text-embedding-3-small`:
```python theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
tbl.add_embedding_index(
embedding=embeddings.using(model='text-embedding-3-small')
)
```
## udf image\_edits()
```python Signature theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
@pxt.udf
image_edits(
image: pxt.Image,
*,
prompt: pxt.String,
model: pxt.String,
mask: pxt.Image | None = None,
model_kwargs: pxt.Json | None = None
) -> ImagesResponse
```
Creates an edited or extended image given a source image and a prompt.
Equivalent to the OpenAI `images/edits` API endpoint.
For additional details, see: [https://developers.openai.com/api/docs/guides/image-generation#edit-images](https://developers.openai.com/api/docs/guides/image-generation#edit-images)
Request throttling:
Applies the rate limit set in the config (section `openai.rate_limits`; use the model id as the key). If no rate
limit is configured, uses a default of 600 RPM.
**Requirements:**
* `pip install openai`
**Parameters:**
* **`image`** (`pxt.Image`): The source image to edit.
* **`prompt`** (`pxt.String`): A text description of the desired edit.
* **`model`** (`pxt.String`): The model to use for image editing.
* **`mask`** (`pxt.Image | None`): An optional mask image. See: [https://developers.openai.com/api/reference/resources/images/methods/edit](https://developers.openai.com/api/reference/resources/images/methods/edit)
* **`model_kwargs`** (`pxt.Json | None`): Additional keyword args for the OpenAI `images/edits` API. For details on the available
parameters, see: [https://developers.openai.com/api/reference/resources/images/methods/edit](https://developers.openai.com/api/reference/resources/images/methods/edit)
**Returns:**
* `ImagesResponse`: A dictionary containing the edited image data. Images will be deserialized into `PIL.Image.Image` objects,
and the result dictionary will have the following form:
```json theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
{
"created": 1234567890,
"data": [
PIL.Image.Image(...),
...
],
"usage":
}
```
**Examples:**
Edit an image with a text prompt:
```python theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
tbl.add_computed_column(
edited=image_edits(
tbl.source_image,
prompt='Add a sunset background',
model='gpt-image-2',
)
)
```
Edit an image with a mask to specify the edit area:
```python theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
tbl.add_computed_column(
edited=image_edits(
tbl.source_image,
mask=tbl.mask_image,
prompt='Replace with a beach scene',
model='gpt-image-2',
)
)
```
## udf image\_generations()
```python Signature theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
@pxt.udf
image_generations(
prompt: pxt.String,
*,
model: pxt.String,
model_kwargs: pxt.Json | None = None
) -> ImagesResponse
```
Creates an image given a prompt.
Equivalent to the OpenAI `images/generations` API endpoint.
For additional details, see: [https://platform.openai.com/docs/guides/images](https://platform.openai.com/docs/guides/images)
Request throttling:
Applies the rate limit set in the config (section `openai.rate_limits`; use the model id as the key). If no rate
limit is configured, uses a default of 600 RPM.
**Requirements:**
* `pip install openai`
**Parameters:**
* **`prompt`** (`pxt.String`): Prompt for the image.
* **`model`** (`pxt.String`): The model to use for the generations. See the OpenAI docs for supported models.
* **`model_kwargs`** (`pxt.Json | None`): Additional keyword args for the OpenAI `images/generations` API. For details on the available
parameters, see: [https://developers.openai.com/api/reference/resources/images/methods/generate](https://developers.openai.com/api/reference/resources/images/methods/generate)
**Returns:**
* `ImagesResponse`: A dictionary containing the generated image data. Images will be deserialized into `PIL.Image.Image` objects,
and the result dictionary will have the following form:
```json theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
{
"created": 1234567890,
"data": [
PIL.Image.Image(...),
PIL.Image.Image(...),
...
],
"usage":
}
```
**Examples:**
Add a computed column that applies the model `gpt-image-1.5` to an existing
Pixeltable column `tbl.text` of the table `tbl`:
```python theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
tbl.add_computed_column(
gen_image=image_generations(tbl.text, model='gpt-image-1.5')
)
```
Generate an image using the `gpt-image-2` model:
```python theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
tbl.add_computed_column(
gen_image=image_generations(tbl.text, model='gpt-image-2')
)
```
## udf image\_variations()
```python Signature theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
@pxt.udf
image_variations(
image: pxt.Image,
*,
model: pxt.String,
model_kwargs: pxt.Json | None = None
) -> ImagesResponse
```
Creates a variation of a given image.
Equivalent to the OpenAI `images/variations` API endpoint.
For additional details, see: [https://developers.openai.com/api/docs/guides/image-generation#image-variations](https://developers.openai.com/api/docs/guides/image-generation#image-variations)
Request throttling:
Applies the rate limit set in the config (section `openai.rate_limits`; use the model id as the key). If no rate
limit is configured, uses a default of 600 RPM.
**Requirements:**
* `pip install openai`
**Parameters:**
* **`image`** (`pxt.Image`): The source image to create a variation of.
* **`model`** (`pxt.String`): The model to use for creating variations.
* **`model_kwargs`** (`pxt.Json | None`): Additional keyword args for the OpenAI `images/variations` API. For details on the available
parameters, see: [https://developers.openai.com/api/reference/resources/images/methods/create\_variation](https://developers.openai.com/api/reference/resources/images/methods/create_variation)
**Returns:**
* `ImagesResponse`: A dictionary containing the variation image data. Images will be deserialized into `PIL.Image.Image` objects,
and the result dictionary will have the following form:
```json theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
{
"created": 1234567890,
"data": [
PIL.Image.Image(...),
...
]
}
```
**Examples:**
Generate a variation of an existing image:
```python theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
tbl.add_computed_column(
variation=image_variations(tbl.source_image, model='gpt-image-1.5')
)
```
## udf moderations()
```python Signature theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
@pxt.udf
moderations(
input: pxt.String,
*,
model: pxt.String = 'omni-moderation-latest'
) -> ModerationCreateResponse
```
Classifies if text is potentially harmful.
Equivalent to the OpenAI `moderations` API endpoint.
For additional details, see: [https://platform.openai.com/docs/guides/moderation](https://platform.openai.com/docs/guides/moderation)
Request throttling:
Applies the rate limit set in the config (section `openai.rate_limits`; use the model id as the key). If no rate
limit is configured, uses a default of 600 RPM.
**Requirements:**
* `pip install openai`
**Parameters:**
* **`input`** (`pxt.String`): Text to analyze with the moderations model.
* **`model`** (`pxt.String`): The model to use for moderations.
**Returns:**
* `ModerationCreateResponse`: Details of the moderations results.
**Examples:**
Add a computed column that applies the default moderation model to an existing Pixeltable column `tbl.input`
of the table `tbl`:
```python theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
tbl.add_computed_column(moderations=moderations(tbl.input))
```
## udf responses()
```python Signature theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
@pxt.udf
responses(
input: pxt.Json[(Json, ...)],
*,
model: pxt.String,
model_kwargs: pxt.Json | None = None,
tools: pxt.Json[(Json, ...)] | None = None,
tool_choice: pxt.Json | None = None
) -> pxt.Json
```
Creates a model response for the given input.
Equivalent to the OpenAI `responses` API endpoint.
For additional details, see: [https://developers.openai.com/api/docs/guides/migrate-to-responses](https://developers.openai.com/api/docs/guides/migrate-to-responses)
Request throttling:
Uses the rate limit-related headers returned by the API to throttle requests adaptively, based on available
request and token capacity. No configuration is necessary.
**Requirements:**
* `pip install openai`
**Parameters:**
* **`input`** (`pxt.Json[(Json`): A list of input items for the model, as described in the OpenAI API documentation.
* **`model`** (`Any`): The model to use for generating a response.
* **`model_kwargs`** (`Any`): Additional keyword args for the OpenAI `responses` API. For details on the available
parameters, see: [https://developers.openai.com/api/docs/api-reference/responses/create](https://developers.openai.com/api/docs/api-reference/responses/create)
Common options include `instructions`, `temperature`, `max_output_tokens`, `reasoning`, `text`
(for structured outputs), `previous_response_id` (for multi-turn), and `store`.
**Returns:**
* `pxt.Json`: A dictionary containing the response and other metadata. The response text is accessible via
the `output_text` field for simple cases, or through the `output` list for structured access.
**Examples:**
Add a computed column that applies the model `gpt-4o-mini` to an existing Pixeltable column `tbl.prompt`
of the table `tbl`:
```python theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
messages = [{'role': 'user', 'content': tbl.prompt}]
tbl.add_computed_column(
response=responses(
messages,
model='gpt-4o-mini',
model_kwargs={'instructions': 'You are a helpful assistant.'},
)
)
```
You can also include images in the input list by passing image data directly, using the Responses API
`input_image` content type:
```python theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
messages = [
{
'role': 'user',
'content': [
{'type': 'input_text', 'text': "What's in this image?"},
{'type': 'input_image', 'image_url': tbl.image},
],
}
]
tbl.add_computed_column(response=responses(messages, model='gpt-4o-mini'))
```
## udf speech()
```python Signature theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
@pxt.udf
speech(
input: pxt.String,
*,
model: pxt.String,
voice: pxt.String,
model_kwargs: pxt.Json | None = None
) -> pxt.Audio
```
Generates audio from the input text.
Equivalent to the OpenAI `audio/speech` API endpoint.
For additional details, see: [https://platform.openai.com/docs/guides/text-to-speech](https://platform.openai.com/docs/guides/text-to-speech)
Request throttling:
Applies the rate limit set in the config (section `openai.rate_limits`; use the model id as the key). If no rate
limit is configured, uses a default of 600 RPM.
**Requirements:**
* `pip install openai`
**Parameters:**
* **`input`** (`pxt.String`): The text to synthesize into speech.
* **`model`** (`pxt.String`): The model to use for speech synthesis.
* **`voice`** (`pxt.String`): The voice profile to use for speech synthesis. Supported options include:
`alloy`, `echo`, `fable`, `onyx`, `nova`, and `shimmer`.
* **`model_kwargs`** (`pxt.Json | None`): Additional keyword args for the OpenAI `audio/speech` API. For details on the available
parameters, see: [https://platform.openai.com/docs/api-reference/audio/createSpeech](https://platform.openai.com/docs/api-reference/audio/createSpeech)
**Returns:**
* `pxt.Audio`: An audio file containing the synthesized speech.
**Examples:**
Add a computed column that applies the model `tts-1` to an existing Pixeltable column `tbl.text`
of the table `tbl`:
```python theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
tbl.add_computed_column(
audio=speech(tbl.text, model='tts-1', voice='nova')
)
```
## udf transcriptions()
```python Signature theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
@pxt.udf
transcriptions(
audio: pxt.Audio,
*,
model: pxt.String,
model_kwargs: pxt.Json | None = None
) -> TranscriptionResponse
```
Transcribes audio into the input language.
Equivalent to the OpenAI `audio/transcriptions` API endpoint.
For additional details, see: [https://platform.openai.com/docs/guides/speech-to-text](https://platform.openai.com/docs/guides/speech-to-text)
Request throttling:
Applies the rate limit set in the config (section `openai.rate_limits`; use the model id as the key). If no rate
limit is configured, uses a default of 600 RPM.
**Requirements:**
* `pip install openai`
**Parameters:**
* **`audio`** (`pxt.Audio`): The audio to transcribe.
* **`model`** (`pxt.String`): The model to use for speech transcription.
* **`model_kwargs`** (`pxt.Json | None`): Additional keyword args for the OpenAI `audio/transcriptions` API. For details on the available
parameters, see: [https://platform.openai.com/docs/api-reference/audio/createTranscription](https://platform.openai.com/docs/api-reference/audio/createTranscription)
**Returns:**
* `TranscriptionResponse`: A dictionary containing the transcription and other metadata.
**Examples:**
Add a computed column that applies the model `whisper-1` to an existing Pixeltable column `tbl.audio`
of the table `tbl`:
```python theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
tbl.add_computed_column(
transcription=transcriptions(
tbl.audio, model='whisper-1', language='en'
)
)
```
## udf translations()
```python Signature theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
@pxt.udf
translations(
audio: pxt.Audio,
*,
model: pxt.String,
model_kwargs: pxt.Json | None = None
) -> TranslationResponse
```
Translates audio into English.
Equivalent to the OpenAI `audio/translations` API endpoint.
For additional details, see: [https://platform.openai.com/docs/guides/speech-to-text](https://platform.openai.com/docs/guides/speech-to-text)
Request throttling:
Applies the rate limit set in the config (section `openai.rate_limits`; use the model id as the key). If no rate
limit is configured, uses a default of 600 RPM.
**Requirements:**
* `pip install openai`
**Parameters:**
* **`audio`** (`pxt.Audio`): The audio to translate.
* **`model`** (`pxt.String`): The model to use for speech transcription and translation.
* **`model_kwargs`** (`pxt.Json | None`): Additional keyword args for the OpenAI `audio/translations` API. For details on the available
parameters, see: [https://platform.openai.com/docs/api-reference/audio/createTranslation](https://platform.openai.com/docs/api-reference/audio/createTranslation)
**Returns:**
* `TranslationResponse`: A dictionary containing the translation and other metadata.
**Examples:**
Add a computed column that applies the model `whisper-1` to an existing Pixeltable column `tbl.audio`
of the table `tbl`:
```python theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
tbl.add_computed_column(
translation=translations(tbl.audio, model='whisper-1', language='en')
)
```
# openrouter
Source: https://docs.pixeltable.com/sdk/latest/openrouter
# module pixeltable.functions.openrouter
Pixeltable UDFs that wrap the OpenRouter API.
OpenRouter provides a unified interface to multiple LLM providers. In order to use it,
you must first sign up at [https://openrouter.ai](https://openrouter.ai), create an API key, and configure it
as described in the Working with OpenRouter tutorial.
## udf chat\_completions()
```python Signature theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
@pxt.udf
chat_completions(
messages: pxt.Json[(Json, ...)],
*,
model: pxt.String,
model_kwargs: pxt.Json | None = None,
tools: pxt.Json[(Json, ...)] | None = None,
tool_choice: pxt.Json | None = None,
provider: pxt.Json | None = None,
transforms: pxt.Json[(String, ...)] | None = None
) -> pxt.Json
```
Chat Completion API via OpenRouter.
OpenRouter provides access to multiple LLM providers through a unified API.
For additional details, see: [https://openrouter.ai/docs](https://openrouter.ai/docs)
Supported models can be found at: [https://openrouter.ai/models](https://openrouter.ai/models)
Request throttling:
Applies the rate limit set in the config (section `openrouter`, key `rate_limit`). If no rate
limit is configured, uses a default of 600 RPM.
**Requirements:**
* `pip install openai`
**Parameters:**
* **`messages`** (`pxt.Json[(Json`): A list of messages comprising the conversation so far.
* **`model`** (`Any`): ID of the model to use (e.g., `'anthropic/claude-3.5-sonnet'`, `'openai/gpt-4'`).
* **`model_kwargs`** (`Any`): Additional OpenAI-compatible parameters.
* **`tools`** (`Any`): List of tools available to the model.
* **`tool_choice`** (`Any`): Controls which (if any) tool is called by the model.
* **`provider`** (`Any`): OpenRouter-specific provider preferences (e.g., `{'order': ['Anthropic', 'OpenAI']}`).
* **`transforms`** (`Any`): List of message transforms to apply (e.g., `['middle-out']`).
**Returns:**
* `pxt.Json`: A dictionary containing the response in OpenAI format.
**Examples:**
Basic chat completion:
```python theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
messages = [{'role': 'user', 'content': tbl.prompt}]
tbl.add_computed_column(
response=chat_completions(
messages, model='anthropic/claude-3.5-sonnet'
)
)
```
With provider routing:
```python theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
tbl.add_computed_column(
response=chat_completions(
messages,
model='anthropic/claude-3.5-sonnet',
provider={'require_parameters': True, 'order': ['Anthropic']},
)
)
```
With transforms:
```python theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
tbl.add_computed_column(
response=chat_completions(
messages,
model='openai/gpt-4',
transforms=['middle-out'], # Optimize for long contexts
)
)
```
# pixeltable
Source: https://docs.pixeltable.com/sdk/latest/pixeltable
# module opentelemetry.instrumentation.pixeltable
OpenTelemetry instrumentation for Pixeltable.
Translates Pixeltable's instrumentation hooks into OpenTelemetry spans and metrics. Instrumentation is
opt-in: either attach to an existing OpenTelemetry SDK with
[PixeltableInstrumentor](./pixeltableinstrumentor), or let
[init](./pixeltable#func-init) build providers from Pixeltable's `[otel]` config.
## func init()
```python Signature theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
init(
*,
endpoint: str | None = None,
protocol: str | None = None,
service_name: str | None = None,
headers: str | None = None,
span_level: Literal['info', 'debug', 'trace'] | None = None,
metrics: bool | None = None,
logs: bool | None = None,
tracer_provider: Any = None,
meter_provider: Any = None
) -> None
```
Configure pixeltable's OpenTelemetry instrumentation and start emitting telemetry.
Call once, before the first Pixeltable operation. Each argument overrides the matching `[otel]`
config setting and its standard `OTEL_*` environment variable; when an argument is left as None the
value resolves from the database's `[[pixeltable.database]]` entry where it sets one (the endpoint and
protocol, as `db_`), then that env var, then the `[otel]` config section. Pass
`tracer_provider`/`meter_provider` to instrument against an SDK your application owns instead.
**Parameters:**
* **`endpoint`** (`str | None`): OTLP collector endpoint (eg `http://localhost:4318`); resolves from the database entry's
`db_exporter_otlp_endpoint`, then `OTEL_EXPORTER_OTLP_ENDPOINT`, then `otel.exporter_otlp_endpoint`.
With no endpoint configured and no application-owned provider, instrumentation stays inert and
exports nothing.
* **`protocol`** (`str | None`): OTLP transport, `http/protobuf` (default) or `grpc`; resolves from the database entry's
`db_exporter_otlp_protocol`, then `OTEL_EXPORTER_OTLP_PROTOCOL`, then `otel.exporter_otlp_protocol`.
* **`service_name`** (`str | None`): `service.name` resource attribute (default `pixeltable`); resolves from
`otel.service_name` / `OTEL_SERVICE_NAME`.
* **`headers`** (`str | None`): OTLP headers as comma-separated `key=value` pairs; resolves from
`otel.exporter_otlp_headers` / `OTEL_EXPORTER_OTLP_HEADERS`.
* **`span_level`** (`Literal['info', 'debug', 'trace'] | None`): span emission threshold: `info` (default; operation-level spans only), `debug`
(adds per-row and per-UDF spans), or `trace`.
* **`metrics`** (`bool | None`): force metric export on/off (by default metrics are exported only when an OTLP endpoint
is configured).
* **`logs`** (`bool | None`): force log export on/off (default off; must be explicitly enabled).
* **`tracer_provider`** (`Any`): an existing TracerProvider to instrument against.
* **`meter_provider`** (`Any`): an existing MeterProvider to instrument against.
**Examples:**
```python theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
import opentelemetry.instrumentation.pixeltable as pxt_otel
pxt_otel.init(endpoint='http://localhost:4318')
```
# PixeltableInstrumentor
Source: https://docs.pixeltable.com/sdk/latest/pixeltableinstrumentor
# class opentelemetry.instrumentation.pixeltable.PixeltableInstrumentor
Enables OpenTelemetry instrumentation of pixeltable against the given (or global) providers.
`instrument()` accepts `tracer_provider`, `meter_provider`, and `span_level` (span emission
threshold: `info` (default), `debug`, or `trace`).
## method instrument()
```python Signature theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
instrument(**kwargs: Any)
```
Instrument the library
This method will be called without any optional arguments by the
`opentelemetry-instrument` command.
This means that calling this method directly without passing any
optional values should do the very same thing that the
`opentelemetry-instrument` command does.
## method uninstrument()
```python Signature theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
uninstrument(**kwargs: Any)
```
Uninstrument the library
See `BaseInstrumentor.instrument` for more information regarding the
usage of `kwargs`.
# Query
Source: https://docs.pixeltable.com/sdk/latest/query
# class pixeltable.Query
A query over tables in a catalog: it can be executed, and its rows updated or deleted.
Thread-safe.
## method collect()
```python Signature theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
collect() -> ResultSet
```
## method cursor()
```python Signature theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
cursor() -> ResultCursor
```
Return a [`ResultCursor`](./resultcursor) that iterates over the query results row by row.
See [`ResultCursor`](./resultcursor) for usage examples and lifecycle details.
## method distinct()
```python Signature theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
distinct() -> Self
```
Remove duplicate rows from this Query.
Note that grouping will be applied to the rows based on the select clause of this Query.
In the absence of a select clause, by default, all columns are selected in the grouping.
**Examples:**
Select unique addresses from table `addresses`.
```python theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
results = addresses.distinct()
```
Select unique cities in table `addresses`
```python theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
results = addresses.select(addresses.city).distinct()
```
Select unique locations (street, city) in the state of `CA`
```python theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
results = (
addresses.select(addresses.street, addresses.city)
.where(addresses.state == 'CA')
.distinct()
)
```
## method group\_by()
```python Signatures theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
# Signature 1:
group_by(grouping_tbl: catalog.Table, /) -> Self
# Signature 2:
group_by(*grouping_items: exprs.Expr) -> Self
```
Add a group-by clause to this Query.
Variants:
* group\_by(base\_tbl): group a component view by their respective base table rows
* group\_by(expr1, expr2, expr3): group by the given expressions
Note that grouping will be applied to the rows and take effect when
used with an aggregation function like sum(), count() etc.
**Parameters:**
* **`grouping_items`** (`exprs.Expr | catalog.Table`): expressions to group by
**Returns:**
* `Self`: A new query with the specified group-by clause.
**Examples:**
Given the Query book from a table t with all its columns and rows:
```python theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
book = t.select()
```
Group the above Query book by the 'genre' column (referenced in table t):
```python theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
query = book.group_by(t.genre)
```
Use the above Query grouped by genre to count the number of
books for each 'genre':
```python theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
query = (
book.group_by(t.genre).select(t.genre, count=count(t.genre)).show()
)
```
Use the above Query grouped by genre to the total price of
books for each 'genre':
```python theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
query = book.group_by(t.genre).select(t.genre, total=sum(t.price)).show()
```
## method head()
```python Signature theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
head(n: int = 10) -> ResultSet
```
Return the first n rows of the Query, in insertion order of the underlying Table.
head() is not supported for joins.
**Parameters:**
* **`n`** (`int`, default: `10`): Number of rows to select. Default is 10.
**Returns:**
* `ResultSet`: A ResultSet with the first n rows of the Query.
## method join()
```python Signature theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
join(
other: catalog.Table,
on: exprs.Expr | Sequence[exprs.ColumnRef] | None = None,
how: JoinType.LiteralType = 'inner'
) -> Self
```
Join this Query with a table.
**Parameters:**
* **`other`** (`catalog.Table`): the table to join with
* **`on`** (`exprs.Expr | Sequence[exprs.ColumnRef] | None`): the join condition, which can be either a) references to one or more columns or b) a boolean
expression.
* column references: implies an equality predicate that matches columns in both this
Query and `other` by name.
* column in `other`: A column with that same name must be present in this Query, and **it must
be unique** (otherwise the join is ambiguous).
* column in this Query: A column with that same name must be present in `other`.
* boolean expression: The expressions must be valid in the context of the joined tables.
* **`how`** (`JoinType.LiteralType`, default: `'inner'`): the type of join to perform.
* `'inner'`: only keep rows that have a match in both
* `'left'`: keep all rows from this Query and only matching rows from the other table
* `'right'`: keep all rows from the other table and only matching rows from this Query
* `'full_outer'`: keep all rows from both this Query and the other table
* `'cross'`: Cartesian product; no `on` condition allowed
**Returns:**
* `Self`: A new query.
**Examples:**
Perform an inner join between t1 and t2 on the column id:
```python theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
join1 = t1.join(t2, on=t2.id)
```
Perform a left outer join of join1 with t3, also on id (note that we can't specify `on=t3.id` here,
because that would be ambiguous, since both t1 and t2 have a column named id):
```python theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
join2 = join1.join(t3, on=t2.id, how='left')
```
Do the same, but now with an explicit join predicate:
```python theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
join2 = join1.join(t3, on=t2.id == t3.id, how='left')
```
Join t with d, which has a composite primary key (columns pk1 and pk2, with corresponding foreign
key columns d1 and d2 in t):
```python theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
query = t.join(d, on=(t.d1 == d.pk1) & (t.d2 == d.pk2), how='left')
```
## method limit()
```python Signature theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
limit(n: int, offset: int | None = None) -> Self
```
Limit the number of rows in the Query, optionally skipping rows for pagination.
**Parameters:**
* **`n`** (`int`): Number of rows to select.
* **`offset`** (`int | None`): Number of rows to skip before returning results. Default is None (no offset).
**Returns:**
* `Self`: A new query with the specified limited rows.
**Examples:**
```python theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
query = t.select()
```
Get the first 10 rows:
```python theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
query.limit(10).collect()
```
Get rows 21-30 (skip first 20, return next 10):
```python theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
query.limit(10, offset=20).collect()
```
## method order\_by()
```python Signature theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
order_by(*expr_list: exprs.Expr, asc: bool = True) -> Self
```
Add an order-by clause to this Query.
**Parameters:**
* **`expr_list`** (`exprs.Expr`): expressions to order by
* **`asc`** (`bool`, default: `True`): whether to order in ascending order (True) or descending order (False).
Default is True.
**Returns:**
* `Self`: A new query with the specified order-by clause.
**Examples:**
Given the Query book from a table t with all its columns and rows:
```python theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
book = t.select()
```
Order the above Query book by two columns (price, pages) in descending order:
```python theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
query = book.order_by(t.price, t.pages, asc=False)
```
Order the above Query book by price in descending order, but order the pages
in ascending order:
```python theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
query = book.order_by(t.price, asc=False).order_by(t.pages)
```
## method sample()
```python Signature theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
sample(
n: int | None = None,
n_per_stratum: int | None = None,
fraction: float | None = None,
seed: int | None = None,
stratify_by: Any = None
) -> Self
```
Return a new query specifying a sample of rows from this query, considered in a shuffled order.
The size of the sample can be specified in three ways:
* `n`: the total number of rows to produce as a sample
* `n_per_stratum`: the number of rows to produce per stratum as a sample
* `fraction`: the fraction of available rows to produce as a sample
The sample can be stratified by one or more columns, which means that the sample will
be selected from each stratum separately.
The data is shuffled before creating the sample.
**Parameters:**
* **`n`** (`int | None`): Total number of rows to produce as a sample.
* **`n_per_stratum`** (`int | None`): Number of rows to produce per stratum as a sample. This parameter is only valid if
`stratify_by` is specified. Only one of `n` or `n_per_stratum` can be specified.
* **`fraction`** (`float | None`): Fraction of available rows to produce as a sample. This parameter is not usable with `n` or
`n_per_stratum`. The fraction must be between 0.0 and 1.0.
* **`seed`** (`int | None`): Random seed for reproducible shuffling
* **`stratify_by`** (`Any`): If specified, the sample will be stratified by these values.
**Returns:**
* `Self`: A new query which specifies the sampled rows
**Examples:**
Given the Table `person` containing the field 'age', we can create samples of the table in various ways:
Sample 100 rows from the above Table:
```python theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
query = person.sample(n=100)
```
Sample 10% of the rows from the above Table:
```python theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
query = person.sample(fraction=0.1)
```
Sample 10% of the rows from the above Table, stratified by the column 'age':
```python theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
query = person.sample(fraction=0.1, stratify_by=t.age)
```
Equal allocation sampling: Sample 2 rows from each age present in the above Table:
```python theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
query = person.sample(n_per_stratum=2, stratify_by=t.age)
```
Sampling is compatible with the where clause, so we can also sample from a filtered Query:
```python theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
query = person.where(t.age > 30).sample(n=100)
```
## method select()
```python Signature theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
select(*items: Any, **named_items: Any) -> Self
```
Select columns or expressions from the Query.
**Parameters:**
* **`items`** (`Any`): expressions to be selected
* **`named_items`** (`Any`): named expressions to be selected
**Returns:**
* `Self`: A new query with the specified select list.
**Examples:**
Given the Query person from a table t with all its columns and rows:
```python theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
person = t.select()
```
Select the columns 'name' and 'age' (referenced in table t) from the Query person:
```python theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
query = person.select(t.name, t.age)
```
Select the columns 'name' (referenced in table t) from the Query person,
and a named column 'is\_adult' from the expression `age >= 18` where 'age' is
another column in table t:
```python theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
query = person.select(t.name, is_adult=(t.age >= 18))
```
## method show()
```python Signature theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
show(n: int = 20) -> ResultSet
```
## method tail()
```python Signature theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
tail(n: int = 10) -> ResultSet
```
Return the last n rows of the Query, in insertion order of the underlying Table.
tail() is not supported for joins.
**Parameters:**
* **`n`** (`int`, default: `10`): Number of rows to select. Default is 10.
**Returns:**
* `ResultSet`: A ResultSet with the last n rows of the Query.
## method to\_coco\_dataset()
```python Signature theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
to_coco_dataset() -> Path
```
Convert the Query to a COCO dataset.
This Query must return a single json-typed output column in the following format:
```python theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
{
'image': PIL.Image.Image,
'annotations': [
{
'bbox': [x: int, y: int, w: int, h: int],
'category': str | int,
},
...
],
}
```
**Returns:**
* `Path`: Path to the COCO dataset file.
## method to\_pytorch\_dataset()
```python Signature theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
to_pytorch_dataset(image_format: str = 'pt') -> torch.utils.data.IterableDataset
```
Convert the Query to a pytorch IterableDataset suitable for parallel loading
with torch.utils.data.DataLoader.
This method requires pyarrow >= 13, torch and torchvision to work.
This method serializes data so it can be read from disk efficiently and repeatedly without
re-executing the query. This data is cached to disk for future re-use.
**Parameters:**
* **`image_format`** (`str`, default: `'pt'`): format of the images. Can be 'pt' (pytorch tensor) or 'np' (numpy array).
'np' means image columns return as an RGB uint8 array of shape HxWxC.
'pt' means image columns return as a CxHxW tensor with values in \[0,1] and type torch.float32.
(the format output by torchvision.transforms.ToTensor())
**Returns:**
* `'torch.utils.data.IterableDataset'`: A pytorch IterableDataset: Columns become fields of the dataset, where rows are returned as a dictionary
compatible with torch.utils.data.DataLoader default collation.
## method where()
```python Signature theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
where(pred: exprs.Expr) -> Self
```
Filter rows based on a predicate.
**Parameters:**
* **`pred`** (`exprs.Expr`): the predicate to filter rows
**Returns:**
* `Self`: A new query with the specified predicates replacing the where-clause.
**Examples:**
Given the Query person from a table t with all its columns and rows:
```python theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
person = t.select()
```
Filter the above Query person to only include rows where the column 'age'
(referenced in table t) is greater than 30:
```python theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
query = person.where(t.age > 30)
```
# replicate
Source: https://docs.pixeltable.com/sdk/latest/replicate
# module pixeltable.functions.replicate
Pixeltable UDFs
that wrap various endpoints from the Replicate API. In order to use them, you must
first `pip install replicate` and configure your Replicate credentials, as described in
the [Working with Replicate](https://docs.pixeltable.com/howto/providers/working-with-replicate) tutorial.
## udf run()
```python Signature theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
@pxt.udf
run(input: pxt.Json, *, ref: pxt.String) -> pxt.Json
```
Run a model on Replicate.
For additional details, see: [https://replicate.com/docs/topics/models/run-a-model](https://replicate.com/docs/topics/models/run-a-model)
Request throttling:
Applies the rate limit set in the config (section `replicate`, key `rate_limit`). If no rate
limit is configured, uses a default of 600 RPM.
**Requirements:**
* `pip install replicate`
**Parameters:**
* **`input`** (`pxt.Json`): The input parameters for the model.
* **`ref`** (`pxt.String`): The name of the model to run.
**Returns:**
* `pxt.Json`: The output of the model.
**Examples:**
Add a computed column that applies the model `meta/meta-llama-3-8b-instruct`
to an existing Pixeltable column `tbl.prompt` of the table `tbl`:
```python theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
input = {
'system_prompt': 'You are a helpful assistant.',
'prompt': tbl.prompt,
}
tbl.add_computed_column(
response=run(input, ref='meta/meta-llama-3-8b-instruct')
)
```
Add a computed column that uses the model `black-forest-labs/flux-schnell`
to generate images from an existing Pixeltable column `tbl.prompt`:
```python theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
input = {'prompt': tbl.prompt, 'go_fast': True, 'megapixels': '1'}
tbl.add_computed_column(
response=run(input, ref='black-forest-labs/flux-schnell')
)
tbl.add_computed_column(image=tbl.response.output[0].astype(pxt.Image))
```
# ResultCursor
Source: https://docs.pixeltable.com/sdk/latest/resultcursor
# class pixeltable.ResultCursor
Cursor that iterates over query results.
Wraps a Query and yields Row objects one at a time,
avoiding materializing all results into memory.
A cursor transitions through three states: pending (created but not yet started), open (actively
iterating), and closed (resources released). Iteration auto-opens and auto-closes the cursor, or you can
use it as a context manager for explicit lifecycle control.
## method close()
```python Signature theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
close() -> None
```
Release the underlying database transaction and query resources.
Safe to call multiple times. Once closed, the cursor cannot be reopened.
Also called automatically via the context manager protocol and on garbage collection.
## method open()
```python Signature theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
open() -> None
```
Start the underlying query and prepare the cursor for iteration.
Raises an error if the cursor is already open or has been closed.
Called automatically when iterating if not already open.
# ResultSet
Source: https://docs.pixeltable.com/sdk/latest/resultset
# class pixeltable.ResultSet
A dataset obtained by executing a [`Query`](./query). Returned by
[`Query.collect()`](./query#method-collect), [`Query.head()`](./query#method-head),
[`Query.tail()`](./query#method-tail), and the equivalent methods on class [`Table`](./table).
A `ResultSet` is structured as a table with rows (indexed by integers) and columns (indexed by strings).
The column names correspond to the expressions in the query's select list. The values in a `ResultSet` can
be accessed in various ways:
* `len(result)` returns the number of rows
* `result[i]` returns the `i`th row as a `dict` mapping column names to values
* `result['col']` returns a `list` of all values in the column named `'col'`
* `result[i, 'col']` returns the specific value in the `i`th row and column `'col'`
`ResultSet` implements the Sequence protocol, so it can be iterated over and converted to other sequence
types in the usual fashion; for example:
* `for row in result` (iterates over rows)
* `list(result)` (converts to a list of rows)
## method to\_pandas()
```python Signature theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
to_pandas() -> pd.DataFrame
```
Convert the `ResultSet` to a Pandas `DataFrame`.
**Returns:**
* `pd.DataFrame`: A `DataFrame` with one column per column in the `ResultSet`.
## method to\_pydantic()
```python Signature theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
to_pydantic(model: type[BaseModelT]) -> Iterator[BaseModelT]
```
Convert the `ResultSet` to Pydantic model instances.
**Parameters:**
* **`model`** (`type[BaseModelT]`): A Pydantic model class.
**Returns:**
* `Iterator[BaseModelT]`: An iterator over Pydantic model instances, one for each row in the result set.
# Row
Source: https://docs.pixeltable.com/sdk/latest/row
# class pixeltable.Row
A dict-like wrapper over a single result row.
Supports key access (`row['col']`), membership (`'col' in row`), iteration over keys, and the standard `get()`,
`keys()`, `values()`, and `items()` methods.
The `errors` property holds error info (`{'errortype': ..., 'errormsg': ...}`) for each cell whose
evaluation failed; the `index_values` property holds the values of embedding indexes defined on the
row's table. Both are keyed by column or index name.
Keys are case-sensitive: every name is the folded (lower-case) spelling stored in the catalog.
## method to\_json()
```python Signature theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
to_json() -> dict[str, Any]
```
Return a JSON-serializable dict of this row's values.
* `None`: preserved as `None`
* Timestamp, Date: ISO 8601 string
* UUID: string
* Array: Python list (via `tolist()`)
* Json: validated for serializability, kept as native Python
* Binary: omitted (not representable in JSON)
* All others: unchanged
# RowBatch
Source: https://docs.pixeltable.com/sdk/latest/rowbatch
# class pixeltable.RowBatch
A sequence of [`Row`](./row) instances that share a common schema.
Supports indexing (`batch[0]`), iteration, `len()`.
## method to\_json()
```python Signature theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
to_json() -> list[dict[str, Any]]
```
Return a JSON-serializable list of row dicts (see [`Row.to_json()`](./row#method-to_json)).
# runwayml
Source: https://docs.pixeltable.com/sdk/latest/runwayml
# module pixeltable.functions.runwayml
Pixeltable UDFs
that wrap various endpoints from the RunwayML API. In order to use them, you must
first `pip install runwayml` and configure your RunwayML credentials by setting the `RUNWAYML_API_SECRET` environment
variable.
## udf image\_to\_video()
```python Signature theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
@pxt.udf
image_to_video(
prompt_image: pxt.Image,
model: pxt.String,
ratio: pxt.String,
*,
prompt_text: pxt.String | None = None,
duration: pxt.Int | None = None,
seed: pxt.Int | None = None,
audio: pxt.Bool | None = None,
model_kwargs: pxt.Json | None = None
) -> pxt.Json
```
Generate videos from images.
For additional details, see: [Image to video](https://docs.dev.runwayml.com/api/#tag/Start-generating/paths/~1v1~1image_to_video/post)
**Requirements:**
* `pip install runwayml`
**Parameters:**
* **`prompt_image`** (`pxt.Image`): Input image to use as the first frame.
* **`model`** (`pxt.String`): The model to use.
* **`ratio`** (`pxt.String`): Aspect ratio of the generated video.
* **`prompt_text`** (`pxt.String | None`): Text description to guide generation.
* **`duration`** (`pxt.Int | None`): Duration in seconds.
* **`seed`** (`pxt.Int | None`): Seed for reproducibility.
* **`audio`** (`pxt.Bool | None`): Whether to generate audio.
* **`model_kwargs`** (`pxt.Json | None`): Additional API parameters.
**Returns:**
* `pxt.Json`: A dictionary containing the response and metadata.
**Examples:**
Add a computed column that generates videos from images:
```python theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
tbl.add_computed_column(
response=image_to_video(
tbl.image,
model='gen4_turbo',
ratio='16:9',
prompt_text='Slow motion',
duration=5,
)
)
tbl.add_computed_column(video=tbl.response['output'].astype(pxt.Video))
```
## udf text\_to\_image()
```python Signature theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
@pxt.udf
text_to_image(
prompt_text: pxt.String,
reference_images: pxt.Json[(Image, ...)],
model: pxt.String,
ratio: pxt.String,
*,
seed: pxt.Int | None = None,
model_kwargs: pxt.Json | None = None
) -> pxt.Json
```
Generate images from text prompts and reference images.
For additional details, see: [Text/Image to Image](https://docs.dev.runwayml.com/api/#tag/Start-generating/paths/~1v1~1text_to_image/post)
**Requirements:**
* `pip install runwayml`
**Parameters:**
* **`prompt_text`** (`pxt.String`): Text description of the image to generate.
* **`reference_images`** (`pxt.Json[(Image`): List of 1-3 reference images.
* **`model`** (`Any`): The model to use.
* **`ratio`** (`Any`): Aspect ratio of the generated image.
* **`seed`** (`Any`): Seed for reproducibility.
* **`model_kwargs`** (`Any`): Additional API parameters.
**Returns:**
* `pxt.Json`: A dictionary containing the response and metadata.
**Examples:**
Add a computed column that generates images from prompts:
```python theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
tbl.add_computed_column(
response=text_to_image(
tbl.prompt, [tbl.ref_image], model='gen4_image', ratio='16:9'
)
)
tbl.add_computed_column(image=tbl.response['output'][0].astype(pxt.Image))
```
## udf text\_to\_video()
```python Signature theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
@pxt.udf
text_to_video(
prompt_text: pxt.String,
model: pxt.String,
ratio: pxt.String,
*,
duration: pxt.Int | None = None,
audio: pxt.Bool | None = None,
model_kwargs: pxt.Json | None = None
) -> pxt.Json
```
Generate videos from text prompts.
For additional details, see: [Text to video](https://docs.dev.runwayml.com/api/#tag/Start-generating/paths/~1v1~1text_to_video/post)
**Requirements:**
* `pip install runwayml`
**Parameters:**
* **`prompt_text`** (`pxt.String`): Text description of the video to generate.
* **`model`** (`pxt.String`): The model to use.
* **`ratio`** (`pxt.String`): Aspect ratio of the generated video.
* **`duration`** (`pxt.Int | None`): Duration in seconds.
* **`audio`** (`pxt.Bool | None`): Whether to generate audio.
* **`model_kwargs`** (`pxt.Json | None`): Additional API parameters.
**Returns:**
* `pxt.Json`: A dictionary containing the response and metadata.
**Examples:**
Add a computed column that generates videos from prompts:
```python theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
tbl.add_computed_column(
response=text_to_video(
tbl.prompt, model='veo3.1', ratio='16:9', duration=4
)
)
tbl.add_computed_column(video=tbl.response['output'].astype(pxt.Video))
```
## udf video\_to\_video()
```python Signature theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
@pxt.udf
video_to_video(
video_uri: pxt.String,
prompt_text: pxt.String,
model: pxt.String,
ratio: pxt.String,
*,
seed: pxt.Int | None = None,
model_kwargs: pxt.Json | None = None
) -> pxt.Json
```
Transform videos with text guidance.
For additional details, see: [Video to video](https://docs.dev.runwayml.com/api/#tag/Start-generating/paths/~1v1~1video_to_video/post)
**Requirements:**
* `pip install runwayml`
**Parameters:**
* **`video_uri`** (`pxt.String`): HTTPS URL to the input video.
* **`prompt_text`** (`pxt.String`): Text description of the transformation.
* **`model`** (`pxt.String`): The model to use.
* **`ratio`** (`pxt.String`): Aspect ratio of the output video.
* **`seed`** (`pxt.Int | None`): Seed for reproducibility.
* **`model_kwargs`** (`pxt.Json | None`): Additional API parameters.
**Returns:**
* `pxt.Json`: A dictionary containing the response and metadata.
**Examples:**
Add a computed column that transforms videos:
```python theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
tbl.add_computed_column(
response=video_to_video(
tbl.video_url, 'Anime style', model='aleph2', ratio='16:9'
)
)
tbl.add_computed_column(video=tbl.response['output'].astype(pxt.Video))
```
# SceneInfo
Source: https://docs.pixeltable.com/sdk/latest/sceneinfo
# class pixeltable.functions.video.SceneInfo
One scene detected within a video: where it starts and how long it lasts.
## attr duration
```
duration: float
```
Scene duration, in seconds.
## attr start\_pts
```
start_pts: int
```
Scene start, as a presentation timestamp (in the video stream's time base).
## attr start\_time
```
start_time: float
```
Scene start, in seconds from the start of the video.
# Secret
Source: https://docs.pixeltable.com/sdk/latest/secret
## ⚠️ Error Loading Class
Failed to load class `Secret`:
```
Class Secret not found in pixeltable
```
# serving
Source: https://docs.pixeltable.com/sdk/latest/serving
# module pixeltable.serving
Adapters for web serving frameworks.
# SqlExport
Source: https://docs.pixeltable.com/sdk/latest/sqlexport
# class pixeltable.serving.SqlExport
Specification of an external RDBMS target for SQL export.
**Parameters:**
* **`db_connect`** (`Any`): SQLAlchemy connection string for the target database (e.g.
`'postgresql+psycopg://user:pw@host/db'`, `'sqlite:///path/to.db'`).
* **`table`** (`Any`): Name of the target table. It must already exist; resolution fails
if the table is missing.
* **`db_schema`** (`Any`): Optional database schema qualifier (e.g. `'analytics'`); leave `None` to
use the connection's default schema.
* **`method`** (`Any`): How to write each row into the target table.
* `'insert'`: append the row via `INSERT ... VALUES`.
* `'update'`: update the row by primary-key match
(`UPDATE ... SET ... WHERE pk=...`). Requires that the target table has a
primary key whose metadata is exposed by the dialect. The exported columns
must include all primary-key columns of the target plus at least one non-PK
column to set. This is a strict update, **not** an upsert: if the WHERE
clause matches zero rows, the export fails. Useful when the source is
append-only but the target is a deduplicated current-state view.
* `'merge'`: upsert via the target table's primary key.
**Currently not supported.**
# StreamMetadata
Source: https://docs.pixeltable.com/sdk/latest/streammetadata
# class pixeltable.functions.StreamMetadata
Metadata for a stream within a media container.
## attr average\_rate
```
average_rate: float | None
```
Average frame rate in FPS (frames per second). Present only for video streams.
## attr base\_rate
```
base_rate: float | None
```
Base (constant) frame rate in FPS. Present only for video streams.
## attr codec\_context
```
codec_context: CodecContextMetadata
```
Codec information for this stream.
## attr duration
```
duration: int | None
```
Stream duration in `time_base` units, or `None` if unknown.
## attr duration\_seconds
```
duration_seconds: float | None
```
Stream duration in seconds, computed from `duration` and `time_base`.
## attr frames
```
frames: int
```
Number of frames in the stream (may be 0 if unknown).
## attr guessed\_rate
```
guessed_rate: float | None
```
Guessed frame rate in FPS. Present only for video streams.
## attr height
```
height: int
```
Frame height in pixels. Present only for video streams.
## attr metadata
```
metadata: dict[str, str]
```
Additional stream-specific metadata tags (e.g. language, title).
## attr time\_base
```
time_base: float | None
```
Time base of the stream as a float (seconds per tick), or `None` if unknown.
## attr type
```
type: str
```
Stream type: typically `'audio'` or `'video'`. Other stream types (e.g. subtitles) will have a
`StreamMetadata` entry, but with no metadata other than `type`.
## attr width
```
width: int
```
Frame width in pixels. Present only for video streams.
# string
Source: https://docs.pixeltable.com/sdk/latest/string
# module pixeltable.functions.string
Pixeltable UDFs for `StringType`.
It closely follows the Pandas `pandas.Series.str` API.
Example:
```python theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
import pixeltable as pxt
t = pxt.get_table(...)
t.select(t.str_col.capitalize()).collect()
```
## iterator string\_splitter()
```python Signature theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
@pxt.iterator
string_splitter(
text: pxt.String,
separators: pxt.String,
*,
spacy_model: pxt.String = 'en_core_web_sm'
)
```
Iterator over chunks of a string. The string is chunked according to the specified `separators`.
**Outputs**:
One row per chunk, with the following columns:
* `text` (`pxt.String`): The text of the chunk.
**Parameters:**
* **`separators`** (`pxt.String`): Separators to use to chunk the document. Currently the only supported option is `'sentence'`.
* **`spacy_model`** (`pxt.String`): Name of the spaCy model to use for sentence segmentation.
**Examples:**
This example assumes an existing table `tbl` with a column `text` of type `pxt.String`.
Create a view that splits all strings on sentence boundaries:
```python theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
pxt.create_view(
'sentence_chunks',
tbl,
iterator=string_splitter(tbl.text, separators='sentence'),
)
```
## udf capitalize()
```python Signature theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
@pxt.udf
capitalize(self: pxt.String) -> pxt.String
```
Return string with its first character capitalized and the rest lowercased.
Equivalent to [`str.capitalize()`](https://docs.python.org/3/library/stdtypes.html#str.capitalize).
## udf casefold()
```python Signature theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
@pxt.udf
casefold(self: pxt.String) -> pxt.String
```
Return a casefolded copy of string.
Equivalent to [`str.casefold()`](https://docs.python.org/3/library/stdtypes.html#str.casefold).
## udf center()
```python Signature theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
@pxt.udf
center(
self: pxt.String,
width: pxt.Int,
fillchar: pxt.String = ' '
) -> pxt.String
```
Return a centered string of length `width`.
Equivalent to [`str.center()`](https://docs.python.org/3/library/stdtypes.html#str.center).
**Parameters:**
* **`width`** (`pxt.Int`): Total width of the resulting string.
* **`fillchar`** (`pxt.String`): Character used for padding.
## udf contains()
```python Signature theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
@pxt.udf
contains(
self: pxt.String,
substr: pxt.String,
case: pxt.Bool = True
) -> pxt.Bool
```
Test if string contains a substring.
**Parameters:**
* **`substr`** (`pxt.String`): string literal or regular expression
* **`case`** (`pxt.Bool`): if False, ignore case
## udf contains\_re()
```python Signature theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
@pxt.udf
contains_re(
self: pxt.String,
pattern: pxt.String,
flags: pxt.Int = 0
) -> pxt.Bool
```
Test if string contains a regular expression pattern.
**Parameters:**
* **`pattern`** (`pxt.String`): regular expression pattern
* **`flags`** (`pxt.Int`): [flags](https://docs.python.org/3/library/re.html#flags) for the `re` module
## udf count()
```python Signature theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
@pxt.udf
count(
self: pxt.String,
pattern: pxt.String,
flags: pxt.Int = 0
) -> pxt.Int
```
Count occurrences of pattern or regex.
**Parameters:**
* **`pattern`** (`pxt.String`): string literal or regular expression
* **`flags`** (`pxt.Int`): [flags](https://docs.python.org/3/library/re.html#flags) for the `re` module
## udf endswith()
```python Signature theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
@pxt.udf
endswith(self: pxt.String, substr: pxt.String) -> pxt.Bool
```
Return `True` if the string ends with the specified suffix, otherwise return `False`.
Equivalent to [`str.endswith()`](https://docs.python.org/3/library/stdtypes.html#str.endswith).
**Parameters:**
* **`substr`** (`pxt.String`): string literal
## udf fill()
```python Signature theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
@pxt.udf
fill(self: pxt.String, width: pxt.Int, **kwargs) -> pxt.String
```
Wraps the single paragraph in string, and returns a single string containing the wrapped paragraph.
Equivalent to [`textwrap.fill()`](https://docs.python.org/3/library/textwrap.html#textwrap.fill).
**Parameters:**
* **`width`** (`pxt.Int`): Maximum line width.
* **`kwargs`** (`Any`): Additional keyword arguments to pass to `textwrap.fill()`.
## udf find()
```python Signature theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
@pxt.udf
find(
self: pxt.String,
substr: pxt.String,
start: pxt.Int = 0,
end: pxt.Int | None = None
) -> pxt.Int
```
Return the lowest index in string where `substr` is found within the slice `s[start:end]`.
Equivalent to [`str.find()`](https://docs.python.org/3/library/stdtypes.html#str.find).
**Parameters:**
* **`substr`** (`pxt.String`): substring to search for
* **`start`** (`pxt.Int`): slice start
* **`end`** (`pxt.Int | None`): slice end
## udf findall()
```python Signature theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
@pxt.udf
findall(
self: pxt.String,
pattern: pxt.String,
flags: pxt.Int = 0
) -> pxt.Json[(Json, ...)]
```
Find all occurrences of a regular expression pattern in string.
Equivalent to [`re.findall()`](https://docs.python.org/3/library/re.html#re.findall).
**Parameters:**
* **`pattern`** (`pxt.String`): regular expression pattern
* **`flags`** (`pxt.Int`): [flags](https://docs.python.org/3/library/re.html#flags) for the `re` module
## udf format()
```python Signature theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
@pxt.udf
format(self: pxt.String, *args, **kwargs) -> pxt.String
```
Perform string formatting.
Equivalent to [`str.format()`](https://docs.python.org/3/library/stdtypes.html#str.format).
## udf fullmatch()
```python Signature theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
@pxt.udf
fullmatch(
self: pxt.String,
pattern: pxt.String,
case: pxt.Bool = True,
flags: pxt.Int = 0
) -> pxt.Bool
```
Determine if string fully matches a regular expression.
Equivalent to [`re.fullmatch()`](https://docs.python.org/3/library/re.html#re.fullmatch).
**Parameters:**
* **`pattern`** (`pxt.String`): regular expression pattern
* **`case`** (`pxt.Bool`): if False, ignore case
* **`flags`** (`pxt.Int`): [flags](https://docs.python.org/3/library/re.html#flags) for the `re` module
## udf index()
```python Signature theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
@pxt.udf
index(
self: pxt.String,
substr: pxt.String,
start: pxt.Int = 0,
end: pxt.Int | None = None
) -> pxt.Int
```
Return the lowest index in string where `substr` is found within the slice `[start:end]`.
Raises ValueError if `substr` is not found.
Equivalent to [`str.index()`](https://docs.python.org/3/library/stdtypes.html#str.index).
**Parameters:**
* **`substr`** (`pxt.String`): substring to search for
* **`start`** (`pxt.Int`): slice start
* **`end`** (`pxt.Int | None`): slice end
## udf isalnum()
```python Signature theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
@pxt.udf
isalnum(self: pxt.String) -> pxt.Bool
```
Return `True` if all characters in the string are alphanumeric and there is at least one character, `False`
otherwise.
Equivalent to \[`str.isalnum()`]\([https://docs.python.org/3/library/stdtypes.html#str.isalnum](https://docs.python.org/3/library/stdtypes.html#str.isalnum)
## udf isalpha()
```python Signature theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
@pxt.udf
isalpha(self: pxt.String) -> pxt.Bool
```
Return `True` if all characters in the string are alphabetic and there is at least one character, `False` otherwise.
Equivalent to [`str.isalpha()`](https://docs.python.org/3/library/stdtypes.html#str.isalpha).
## udf isascii()
```python Signature theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
@pxt.udf
isascii(self: pxt.String) -> pxt.Bool
```
Return `True` if the string is empty or all characters in the string are ASCII, `False` otherwise.
Equivalent to [`str.isascii()`](https://docs.python.org/3/library/stdtypes.html#str.isascii).
## udf isdecimal()
```python Signature theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
@pxt.udf
isdecimal(self: pxt.String) -> pxt.Bool
```
Return `True` if all characters in the string are decimal characters and there is at least one character, `False`
otherwise.
Equivalent to [`str.isdecimal()`](https://docs.python.org/3/library/stdtypes.html#str.isdecimal).
## udf isdigit()
```python Signature theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
@pxt.udf
isdigit(self: pxt.String) -> pxt.Bool
```
Return `True` if all characters in the string are digits and there is at least one character, `False` otherwise.
Equivalent to [`str.isdigit()`](https://docs.python.org/3/library/stdtypes.html#str.isdigit).
## udf isidentifier()
```python Signature theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
@pxt.udf
isidentifier(self: pxt.String) -> pxt.Bool
```
Return `True` if the string is a valid identifier according to the language definition, `False` otherwise.
Equivalent to [`str.isidentifier()`](https://docs.python.org/3/library/stdtypes.html#str.isidentifier)
## udf islower()
```python Signature theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
@pxt.udf
islower(self: pxt.String) -> pxt.Bool
```
Return `True` if all cased characters in the string are lowercase and there is at least one cased character,
`False` otherwise.
Equivalent to [`str.islower()`](https://docs.python.org/3/library/stdtypes.html#str.islower)
## udf isnumeric()
```python Signature theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
@pxt.udf
isnumeric(self: pxt.String) -> pxt.Bool
```
Return `True` if all characters in the string are numeric characters, `False` otherwise.
Equivalent to [`str.isnumeric()`](https://docs.python.org/3/library/stdtypes.html#str.isnumeric)
## udf isspace()
```python Signature theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
@pxt.udf
isspace(self: pxt.String) -> pxt.Bool
```
Return `True` if there are only whitespace characters in the string and there is at least one character,
`False` otherwise.
Equivalent to [`str.isspace()`](https://docs.python.org/3/library/stdtypes.html#str.isspace)
## udf istitle()
```python Signature theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
@pxt.udf
istitle(self: pxt.String) -> pxt.Bool
```
Return `True` if the string is a titlecased string and there is at least one character, `False` otherwise.
Equivalent to [`str.istitle()`](https://docs.python.org/3/library/stdtypes.html#str.istitle)
## udf isupper()
```python Signature theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
@pxt.udf
isupper(self: pxt.String) -> pxt.Bool
```
Return `True` if all cased characters in the string are uppercase and there is at least one cased character,
`False` otherwise.
Equivalent to [`str.isupper()`](https://docs.python.org/3/library/stdtypes.html#str.isupper)
## udf join()
```python Signature theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
@pxt.udf
join(
sep: pxt.String,
elements: pxt.Json[(Json, ...)]
) -> pxt.String
```
Return a string which is the concatenation of the strings in `elements`.
Equivalent to [`str.join()`](https://docs.python.org/3/library/stdtypes.html#str.join)
## udf len()
```python Signature theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
@pxt.udf
len(self: pxt.String) -> pxt.Int
```
Return the number of characters in the string.
Equivalent to [`len(str)`](https://docs.python.org/3/library/functions.html#len)
## udf ljust()
```python Signature theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
@pxt.udf
ljust(
self: pxt.String,
width: pxt.Int,
fillchar: pxt.String = ' '
) -> pxt.String
```
Return the string left-justified in a string of length `width`.
Equivalent to [`str.ljust()`](https://docs.python.org/3/library/stdtypes.html#str.ljust)
**Parameters:**
* **`width`** (`pxt.Int`): Minimum width of resulting string; additional characters will be filled with character defined in
`fillchar`.
* **`fillchar`** (`pxt.String`): Additional character for filling.
## udf lower()
```python Signature theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
@pxt.udf
lower(self: pxt.String) -> pxt.String
```
Return a copy of the string with all the cased characters converted to lowercase.
Equivalent to [`str.lower()`](https://docs.python.org/3/library/stdtypes.html#str.lower)
## udf lstrip()
```python Signature theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
@pxt.udf
lstrip(
self: pxt.String,
chars: pxt.String | None = None
) -> pxt.String
```
Return a copy of the string with leading characters removed. The `chars` argument is a string specifying the set of
characters to be removed. If omitted or `None`, whitespace characters are removed.
Equivalent to [`str.lstrip()`](https://docs.python.org/3/library/stdtypes.html#str.lstrip)
**Parameters:**
* **`chars`** (`pxt.String | None`): The set of characters to be removed.
## udf match()
```python Signature theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
@pxt.udf
match(
self: pxt.String,
pattern: pxt.String,
case: pxt.Bool = True,
flags: pxt.Int = 0
) -> pxt.Bool
```
Determine if string starts with a match of a regular expression
**Parameters:**
* **`pattern`** (`pxt.String`): regular expression pattern
* **`case`** (`pxt.Bool`): if False, ignore case
* **`flags`** (`pxt.Int`): [flags](https://docs.python.org/3/library/re.html#flags) for the `re` module
## udf normalize()
```python Signature theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
@pxt.udf
normalize(self: pxt.String, form: pxt.String) -> pxt.String
```
Return the Unicode normal form.
Equivalent to [`unicodedata.normalize()`](https://docs.python.org/3/library/unicodedata.html#unicodedata.normalize)
**Parameters:**
* **`form`** (`pxt.String`): Unicode normal form (`'NFC'`, `'NFKC'`, `'NFD'`, `'NFKD'`)
## udf pad()
```python Signature theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
@pxt.udf
pad(
self: pxt.String,
width: pxt.Int,
side: pxt.String = 'left',
fillchar: pxt.String = ' '
) -> pxt.String
```
Pad string up to width
**Parameters:**
* **`width`** (`pxt.Int`): Minimum width of resulting string; additional characters will be filled with character defined in
`fillchar`.
* **`side`** (`pxt.String`): Side from which to fill resulting string (`'left'`, `'right'`, `'both'`)
* **`fillchar`** (`pxt.String`): Additional character for filling
## udf partition()
```python Signature theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
@pxt.udf
partition(
self: pxt.String,
sep: pxt.String = ' '
) -> pxt.Json[(String, String, String)]
```
Splits string at the first occurrence of `sep`, and returns 3 elements containing the part before the
separator, the separator itself, and the part after the separator. If the separator is not found, return 3 elements
containing string itself, followed by two empty strings.
## udf removeprefix()
```python Signature theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
@pxt.udf
removeprefix(self: pxt.String, prefix: pxt.String) -> pxt.String
```
Remove prefix. If the prefix is not present, returns string.
## udf removesuffix()
```python Signature theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
@pxt.udf
removesuffix(self: pxt.String, suffix: pxt.String) -> pxt.String
```
Remove suffix. If the suffix is not present, returns string.
## udf repeat()
```python Signature theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
@pxt.udf
repeat(self: pxt.String, n: pxt.Int) -> pxt.String
```
Repeat string `n` times.
## udf replace()
```python Signature theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
@pxt.udf
replace(
self: pxt.String,
substr: pxt.String,
repl: pxt.String,
n: pxt.Int | None = None
) -> pxt.String
```
Replace occurrences of `substr` with `repl`.
Equivalent to [`str.replace()`](https://docs.python.org/3/library/stdtypes.html#str.replace).
**Parameters:**
* **`substr`** (`pxt.String`): string literal
* **`repl`** (`pxt.String`): replacement string
* **`n`** (`pxt.Int | None`): number of replacements to make (if `None`, replace all occurrences)
## udf replace\_re()
```python Signature theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
@pxt.udf
replace_re(
self: pxt.String,
pattern: pxt.String,
repl: pxt.String,
n: pxt.Int | None = None,
flags: pxt.Int = 0
) -> pxt.String
```
Replace occurrences of a regular expression pattern with `repl`.
Equivalent to [`re.sub()`](https://docs.python.org/3/library/re.html#re.sub).
**Parameters:**
* **`pattern`** (`pxt.String`): regular expression pattern
* **`repl`** (`pxt.String`): replacement string
* **`n`** (`pxt.Int | None`): number of replacements to make (if `None`, replace all occurrences)
* **`flags`** (`pxt.Int`): [flags](https://docs.python.org/3/library/re.html#flags) for the `re` module
## udf reverse()
```python Signature theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
@pxt.udf
reverse(self: pxt.String) -> pxt.String
```
Return a reversed copy of the string.
Equivalent to `str[::-1]`.
## udf rfind()
```python Signature theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
@pxt.udf
rfind(
self: pxt.String,
substr: pxt.String,
start: pxt.Int | None = 0,
end: pxt.Int | None = None
) -> pxt.Int
```
Return the highest index where `substr` is found, such that `substr` is contained within `[start:end]`.
Equivalent to [`str.rfind()`](https://docs.python.org/3/library/stdtypes.html#str.rfind).
**Parameters:**
* **`substr`** (`pxt.String`): substring to search for
* **`start`** (`pxt.Int | None`): slice start
* **`end`** (`pxt.Int | None`): slice end
## udf rindex()
```python Signature theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
@pxt.udf
rindex(
self: pxt.String,
substr: pxt.String,
start: pxt.Int | None = 0,
end: pxt.Int | None = None
) -> pxt.Int
```
Return the highest index where `substr` is found, such that `substr` is contained within `[start:end]`.
Raises ValueError if `substr` is not found.
Equivalent to [`str.rindex()`](https://docs.python.org/3/library/stdtypes.html#str.rindex).
## udf rjust()
```python Signature theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
@pxt.udf
rjust(
self: pxt.String,
width: pxt.Int,
fillchar: pxt.String = ' '
) -> pxt.String
```
Return the string right-justified in a string of length `width`.
Equivalent to [`str.rjust()`](https://docs.python.org/3/library/stdtypes.html#str.rjust).
**Parameters:**
* **`width`** (`pxt.Int`): Minimum width of resulting string.
* **`fillchar`** (`pxt.String`): Additional character for filling.
## udf rpartition()
```python Signature theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
@pxt.udf
rpartition(
self: pxt.String,
sep: pxt.String = ' '
) -> pxt.Json[(String, String, String)]
```
This method splits string at the last occurrence of `sep`, and returns a list containing the part before the
separator, the separator itself, and the part after the separator.
## udf rstrip()
```python Signature theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
@pxt.udf
rstrip(
self: pxt.String,
chars: pxt.String | None = None
) -> pxt.String
```
Return a copy of string with trailing characters removed.
Equivalent to [`str.rstrip()`](https://docs.python.org/3/library/stdtypes.html#str.rstrip).
**Parameters:**
* **`chars`** (`pxt.String | None`): The set of characters to be removed. If omitted or `None`, whitespace characters are removed.
## udf slice()
```python Signature theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
@pxt.udf
slice(
self: pxt.String,
start: pxt.Int | None = None,
stop: pxt.Int | None = None,
step: pxt.Int | None = None
) -> pxt.String
```
Return a slice.
**Parameters:**
* **`start`** (`pxt.Int | None`): slice start
* **`stop`** (`pxt.Int | None`): slice end
* **`step`** (`pxt.Int | None`): slice step
## udf slice\_replace()
```python Signature theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
@pxt.udf
slice_replace(
self: pxt.String,
start: pxt.Int | None = None,
stop: pxt.Int | None = None,
repl: pxt.String | None = None
) -> pxt.String
```
Replace a positional slice with another value.
**Parameters:**
* **`start`** (`pxt.Int | None`): slice start
* **`stop`** (`pxt.Int | None`): slice end
* **`repl`** (`pxt.String | None`): replacement value
## udf startswith()
```python Signature theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
@pxt.udf
startswith(self: pxt.String, substr: pxt.String) -> pxt.Bool
```
Return `True` if string starts with `substr`, otherwise return `False`.
Equivalent to [`str.startswith()`](https://docs.python.org/3/library/stdtypes.html#str.startswith).
**Parameters:**
* **`substr`** (`pxt.String`): string literal
## udf strip()
```python Signature theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
@pxt.udf
strip(
self: pxt.String,
chars: pxt.String | None = None
) -> pxt.String
```
Return a copy of string with leading and trailing characters removed.
Equivalent to [`str.strip()`](https://docs.python.org/3/library/stdtypes.html#str.strip).
**Parameters:**
* **`chars`** (`pxt.String | None`): The set of characters to be removed. If omitted or `None`, whitespace characters are removed.
## udf swapcase()
```python Signature theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
@pxt.udf
swapcase(self: pxt.String) -> pxt.String
```
Return a copy of string with uppercase characters converted to lowercase and vice versa.
Equivalent to [`str.swapcase()`](https://docs.python.org/3/library/stdtypes.html#str.swapcase).
## udf title()
```python Signature theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
@pxt.udf
title(self: pxt.String) -> pxt.String
```
Return a titlecased version of string, i.e. words start with uppercase characters, all remaining cased characters
are lowercase.
Equivalent to [`str.title()`](https://docs.python.org/3/library/stdtypes.html#str.title).
## udf upper()
```python Signature theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
@pxt.udf
upper(self: pxt.String) -> pxt.String
```
Return a copy of string converted to uppercase.
Equivalent to [`str.upper()`](https://docs.python.org/3/library/stdtypes.html#str.upper).
## udf wrap()
```python Signature theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
@pxt.udf
wrap(
self: pxt.String,
width: pxt.Int,
**kwargs
) -> pxt.Json[(String, ...)]
```
Wraps the single paragraph in string so every line is at most `width` characters long.
Returns a list of output lines, without final newlines.
Equivalent to [`textwrap.fill()`](https://docs.python.org/3/library/textwrap.html#textwrap.fill).
**Parameters:**
* **`width`** (`pxt.Int`): Maximum line width.
* **`kwargs`** (`Any`): Additional keyword arguments to pass to `textwrap.fill()`.
## udf zfill()
```python Signature theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
@pxt.udf
zfill(self: pxt.String, width: pxt.Int) -> pxt.String
```
Pad a numeric string with ASCII `0` on the left to a total length of `width`.
Equivalent to [`str.zfill()`](https://docs.python.org/3/library/stdtypes.html#str.zfill).
**Parameters:**
* **`width`** (`pxt.Int`): Minimum width of resulting string.
# Table
Source: https://docs.pixeltable.com/sdk/latest/table
# class pixeltable.Table
A handle to a table, view, or snapshot. This class is the primary interface through which table operations
(queries, insertions, updates, etc.) are performed in Pixeltable.
Thread-safe.
## method add\_btree\_index()
```python Signature theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
add_btree_index(
column: str | ColumnRef,
*,
idx_name: str | None = None,
if_exists: Literal['error', 'ignore'] = 'error'
) -> None
```
Add a B-tree index to the table. Once the index is created, it will be automatically kept up-to-date as new
rows are inserted into and existing rows are updated in the table.
A B-tree index accelerates equality and range comparisons (used in `where` clauses) and `order_by` on the
indexed column. Any non-boolean scalar column (`String`, `Int`, `Float`, `Timestamp`, `Date`) or non-computed
media column (`Image`, `Video`, `Audio`, `Document`) is supported; the index on a media column is over the
file URL. The column must be stored and, if a media column, must not be produced by a view iterator.
Only tables created with `has_default_idxs=False` (the default) accept explicit B-tree indexes. A table
created with `has_default_idxs=True` indexes its eligible columns automatically, and its B-tree indexes
cannot be managed separately from their respective columns.
**Parameters:**
* **`column`** (`str | ColumnRef`): The name of, or reference to, the column to be indexed; must be an indexable scalar or media
column (see above).
* **`idx_name`** (`str | None`): An optional name for the index. If not specified, a unique name will be generated automatically.
If specified, the name must be unique for this table and a valid pixeltable column name.
* **`if_exists`** (`Literal['error', 'ignore']`, default: `'error'`): Directive for handling an existing B-tree index on the same column, or an existing index with
the same name. Must be one of `'error'`, `'ignore'`.
**Examples:**
Add an index to the `name` column of the table `my_table`:
```python theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
tbl = pxt.get_table('my_table')
tbl.add_btree_index('name')
```
The column may also be specified by reference, and the index may be given an explicit name:
```python theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
tbl.add_btree_index(tbl.name, idx_name='name_idx')
```
## method add\_column()
```python Signature theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
add_column(
*,
if_exists: Literal['error', 'ignore', 'replace', 'replace_force'] = 'error',
**kwargs: TypeForm | ColumnSpec
) -> UpdateStatus
```
Adds an ordinary (non-computed) column to the table.
**Parameters:**
* **`kwargs`** (`TypeForm | ColumnSpec`): Exactly one keyword argument of the form `col_name=type` or `col_name=col_spec_dict`,
where `col_spec_dict` is a [`ColumnSpec`](./columnspec) dict.
* **`if_exists`** (`Literal['error', 'ignore', 'replace', 'replace_force']`, default: `'error'`): Determines the behavior if the column already exists. Must be one of the following:
* `'error'`: an exception will be raised.
* `'ignore'`: do nothing and return.
* `'replace'` or `'replace_force'`: drop the existing column and add the new column, if it has
no dependents.
**Returns:**
* `UpdateStatus`: Information about the execution status of the operation.
**Examples:**
Add an int column:
```python theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
tbl.add_column(new_col=pxt.Int | None)
```
Add a column with column metadata using a dict:
```python theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
tbl.add_column(
img_col={
'type': pxt.Image | None,
'stored': True,
'media_validation': 'on_write',
}
)
```
Alternatively, adding a column can also be expressed using `add_columns`:
```python theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
tbl.add_columns({'new_col': pxt.Int | None})
```
As well as with column metadata:
```python theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
tbl.add_columns(
{
'img_col': {
'type': pxt.Image | None,
'stored': True,
'media_validation': 'on_write',
}
}
)
```
## method add\_columns()
```python Signature theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
add_columns(
schema: Mapping[str, TypeForm | ColumnSpec],
if_exists: Literal['error', 'ignore', 'replace', 'replace_force'] = 'error'
) -> UpdateStatus
```
Adds multiple columns to the table. The columns must be concrete (non-computed) columns; to add computed
columns, use [`add_computed_column()`](./table#method-add_computed_column) instead.
The format of the `schema` argument is a dict mapping column names to their types. A bare type such as
`pxt.Int` defines a non-nullable column; use `pxt.Int | None` to allow nulls. Note that a column added
to a table that already has rows must be nullable.
**Parameters:**
* **`schema`** (`Mapping[str, TypeForm | ColumnSpec]`): A dictionary mapping column names to a `type` or a [`ColumnSpec`](./columnspec) dict.
* **`if_exists`** (`Literal['error', 'ignore', 'replace', 'replace_force']`, default: `'error'`): Determines the behavior if a column already exists. Must be one of the following:
* `'error'`: an exception will be raised.
* `'ignore'`: do nothing and return.
* `'replace' or 'replace_force'`: drop the existing column and add the new column, if it has no
dependents.
Note that the `if_exists` parameter is applied to all columns in the schema.
To apply different behaviors to different columns, please use
[`add_column()`](./table#method-add_column) for each column.
**Returns:**
* `UpdateStatus`: Information about the execution status of the operation.
**Examples:**
Add multiple columns to the table `my_table`:
```python theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
tbl = pxt.get_table('my_table')
schema = {'new_col_1': pxt.Int | None, 'new_col_2': pxt.String | None}
tbl.add_columns(schema)
```
It is also possible to specify column metadata using a dict:
```python theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
tbl = pxt.get_table('my_table')
schema = {
'new_col_1': {
'type': pxt.Image | None,
'stored': True,
'media_validation': 'on_write',
},
'new_col_2': pxt.String | None,
}
tbl.add_columns(schema)
```
## method add\_computed\_column()
```python Signature theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
add_computed_column(
*,
stored: bool | None = None,
destination: str | Path | None = None,
custom_metadata: Any = None,
comment: str = '',
print_stats: bool = False,
on_error: Literal['abort', 'ignore'] = 'abort',
if_exists: Literal['error', 'ignore', 'replace'] = 'error',
**kwargs: exprs.Expr
) -> UpdateStatus
```
Adds a computed column to the table.
**Parameters:**
* **`kwargs`** (`exprs.Expr`): Exactly one keyword argument of the form `col_name=expression`.
* **`stored`** (`bool | None`): Whether the column is materialized and stored or computed on demand.
* **`destination`** (`str | Path | None`): An object store reference for persisting computed files.
* **`custom_metadata`** (`Any`): Optional user-defined metadata to associate with the column. Must be a valid
JSON-serializable object.
* **`comment`** (`str`, default: `''`): An optional comment; its meaning is user-defined.
* **`print_stats`** (`bool`, default: `False`): If `True`, print execution metrics during evaluation.
* **`on_error`** (`Literal['abort', 'ignore']`, default: `'abort'`): Determines the behavior if an error occurs while evaluating the column expression for at least one
row.
* `'abort'`: an exception will be raised and the column will not be added.
* `'ignore'`: execution will continue and the column will be added. Any rows
with errors will have a `None` value for the column, with information about the error stored in the
corresponding `tbl.col_name.errormsg` and `tbl.col_name.errortype` fields.
* **`if_exists`** (`Literal['error', 'ignore', 'replace']`, default: `'error'`): Determines the behavior if the column already exists. Must be one of the following:
* `'error'`: an exception will be raised.
* `'ignore'`: do nothing and return.
* `'replace' or 'replace_force'`: drop the existing column and add the new column, iff it has
no dependents.
**Returns:**
* `UpdateStatus`: Information about the execution status of the operation.
**Examples:**
For a table with an image column `frame`, add an image column `rotated` that rotates the image by
90 degrees:
```python theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
tbl.add_computed_column(rotated=tbl.frame.rotate(90))
```
Do the same, but now the column is unstored:
```python theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
tbl.add_computed_column(rotated=tbl.frame.rotate(90), stored=False)
```
## method add\_embedding\_index()
```python Signature theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
add_embedding_index(
column: str | ColumnRef,
*,
idx_name: str | None = None,
embedding: Function | None = None,
string_embed: Function | None = None,
image_embed: Function | None = None,
audio_embed: Function | None = None,
video_embed: Function | None = None,
document_embed: Function | None = None,
metric: Literal['cosine', 'ip', 'l2'] = 'cosine',
precision: Literal['fp16', 'fp32'] = 'fp16',
if_exists: Literal['error', 'ignore', 'replace', 'replace_force'] = 'error'
) -> None
```
Add an embedding index to the table. Once the index is created, it will be automatically kept up-to-date as new
rows are inserted into the table.
To add an embedding index, specify the column to be indexed and, if the column is not an `Array` column, an
embedding UDF. `String`, `Image`, `Audio`, `Video`, `Document`, and `Array` columns are currently supported.
Multimodal embeddings can be specified in one of two ways: via a single `embedding` argument with a
multi-signature UDF (one signature per modality), or via separate modality-specific arguments (`string_embed`,
`image_embed`, etc.). If both are provided, the modality-specific arguments will supersede the corresponding
signatures of the `embedding` UDF.
For `Array` columns, which are assumed to contain precomputed embeddings, an embedding function is optional;
if provided, it will be used to convert query values into embeddings for similarity search.
**Parameters:**
* **`column`** (`str | ColumnRef`): The name of, or reference to, the column to be indexed; must be a `String`, `Image`, `Audio`,
`Video`, `Document`, or `Array` column.
* **`idx_name`** (`str | None`): An optional name for the index. If not specified, a unique name will be generated automatically.
If specified, the name must be unique for this table and a valid pixeltable column name.
When `idx_name` is omitted, duplicates are detected by the index definition (the embedding
function(s), `metric`, and `precision`) on the column: re-adding an index with an identical
definition is governed by `if_exists`.
* **`embedding`** (`Function | None`): The UDF to use for the embedding. Must be a UDF that accepts a single argument of type `String`
or `Image` (as appropriate for the column being indexed) and returns a fixed-size 1-dimensional
array of floats. If omitted, then at least one of the modality-specific `*_embed` arguments must be
supplied.
* **`string_embed`** (`Function | None`): An optional UDF to use for the string embedding component of this index.
* **`image_embed`** (`Function | None`): An optional UDF to use for the image embedding component of this index.
* **`audio_embed`** (`Function | None`): An optional UDF to use for the audio embedding component of this index.
* **`video_embed`** (`Function | None`): An optional UDF to use for the video embedding component of this index.
* **`document_embed`** (`Function | None`): An optional UDF to use for the document embedding component of this index.
* **`metric`** (`Literal['cosine', 'ip', 'l2']`, default: `'cosine'`): Distance metric to use for the index; one of `'cosine'`, `'ip'`, or `'l2'`.
The default is `'cosine'`.
* **`precision`** (`Literal['fp16', 'fp32']`, default: `'fp16'`): level of precision for the embeddings; one of `'fp16'` or `'fp32'`.
* **`if_exists`** (`Literal['error', 'ignore', 'replace', 'replace_force']`, default: `'error'`): Directive for handling an existing index. The existing index is the one with the same name
(if `idx_name` is given), or one with an identical definition on the same column (if `idx_name` is
omitted). Must be one of the following:
* `'error'`: raise an error if such an index already exists.
* `'ignore'`: do nothing if such an index already exists.
* `'replace'` or `'replace_force'`: replace the existing index with the new one.
**Examples:**
Add an index to the `img` column of the table `my_table`:
```python theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
from pixeltable.functions.huggingface import clip
tbl = pxt.get_table('my_table')
embedding_fn = clip.using(model_id='openai/clip-vit-base-patch32')
tbl.add_embedding_index(tbl.img, embedding=embedding_fn)
```
Alternatively, the `img` column may be specified by name:
```python theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
tbl.add_embedding_index('img', embedding=embedding_fn)
```
Once the index is created, similarity lookups can be performed using the `similarity` pseudo-function:
```python theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
sim = tbl.img.similarity(
image='/path/to/my-image.jpg' # can also be a URL or a PIL image
)
tbl.select(tbl.img, sim).order_by(sim, asc=False).limit(5)
```
If the embedding UDF is a multimodal embedding (supporting more than one data type), then lookups may be
performed using any of its supported modalities. In our example, CLIP supports both text and images, so we
can also search for images using a text description:
```python theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
sim = tbl.img.similarity(string='a picture of a train')
tbl.select(tbl.img, sim).order_by(sim, asc=False).limit(5)
```
Audio and video lookups would look like this:
```python theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
sim = tbl.img.similarity(audio='/path/to/audio.flac')
sim = tbl.img.similarity(video='/path/to/video.mp4')
```
Multiple indexes can be defined on each column. Add a second index to the `img` column, using the inner
product as the distance metric, and with a specific name:
```python theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
tbl.add_embedding_index(
tbl.img, idx_name='ip_idx', embedding=embedding_fn, metric='ip'
)
```
Add an index using separately specified string and image embeddings:
```python theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
tbl.add_embedding_index(
tbl.img,
string_embed=string_embedding_fn,
image_embed=image_embedding_fn,
)
```
## method alter\_computed\_column()
```python Signature theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
alter_computed_column(
*,
recompute: bool = True,
cascade: bool = True,
**kwargs: exprs.Expr
) -> UpdateStatus
```
Change the value expression of a computed column.
The new expression must have the same column type as the current one. It can reference any column of this
table or of one of its ancestors, as long as there are no dependency cycles.
Args:
kwargs: Exactly one keyword argument of the form `col_name=expression`.
recompute: If `True`, the stored values of the column are recomputed. If `False`, they remain those
produced by the previous expression, and are stale until the column is recomputed with
[`recompute_columns()`](./table#method-recompute_columns).
cascade: If `True`, computed columns and views that transitively depend on this column are recomputed
as well; if `False`, they retain their current values. Only applies if `recompute` is `True`.
Returns:
An [`UpdateStatus`](./updatestatus) describing the recompute; rows that raise during
evaluation are recorded as cell errors and counted in `num_excs`.
Raises:
Error: If the column does not exist, is not a computed column, belongs to a base table, or if the new
expression has a different column type or creates a dependency cycle.
Examples:
Change a computed column to use a different scaling factor:
```python theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
tbl = pxt.create_table('my_table', {'n': pxt.Int})
tbl.add_computed_column(scaled=tbl.n * 2)
tbl.alter_computed_column(scaled=tbl.n * 3)
```
Update the definition without recomputing the existing rows:
```python theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
tbl.alter_computed_column(scaled=tbl.n * 4, recompute=False)
```
## method batch\_update()
```python Signature theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
batch_update(
rows: Iterable[dict[str, Any]],
cascade: bool = True,
if_not_exists: Literal['error', 'ignore', 'insert'] = 'error',
return_rows: bool = False
) -> UpdateStatus
```
Update rows in this table.
**Parameters:**
* **`rows`** (`Iterable[dict[str, Any]]`): an Iterable of dictionaries containing values for the updated columns plus values for the primary key
columns.
* **`cascade`** (`bool`, default: `True`): if True, also update all computed columns that transitively depend on the updated columns.
* **`if_not_exists`** (`Literal['error', 'ignore', 'insert']`, default: `'error'`): Specifies the behavior if a row to update does not exist:
* `'error'`: Raise an error.
* `'ignore'`: Skip the row silently.
* `'insert'`: Insert the row.
* **`return_rows`** (`bool`, default: `False`): If `True`, populate `UpdateStatus.rows` with one dict per affected row, mapping column
names to their new stored values. Rows newly inserted via `if_not_exists='insert'` are included.
If `False` (default), `UpdateStatus.rows` is `None`.
**Examples:**
Update the `name` and `age` columns for the rows with ids 1 and 2 (assuming `id` is the primary key).
If either row does not exist, this raises an error:
```python theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
tbl.batch_update(
[
{'id': 1, 'name': 'Alice', 'age': 30},
{'id': 2, 'name': 'Bob', 'age': 40},
]
)
```
Update the `name` and `age` columns for the row with `id` 1 (assuming `id` is the primary key) and insert
the row with new `id` 3 (assuming this key does not exist):
```python theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
tbl.batch_update(
[
{'id': 1, 'name': 'Alice', 'age': 30},
{'id': 3, 'name': 'Bob', 'age': 40},
],
if_not_exists='insert',
)
```
## method collect()
```python Signature theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
collect() -> ResultSet
```
Return rows from this table.
## method columns()
```python Signature theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
columns() -> list[str]
```
Return the names of the columns in this table.
## method compute()
```python Signature theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
compute(
source: Sequence[dict[str, Any]] | Sequence[pydantic.BaseModel],
/,
*,
outputs: Sequence[str | ColumnRef] | None = None,
on_error: Literal['abort', 'ignore'] = 'abort'
) -> RowBatch
```
Materialize the computed columns of this table for the given input rows and return the resulting rows
without persisting them.
If `outputs` is specified, will compute only those columns and the dependent columns that they use. An input
row then needs to supply only the stored required columns used by `outputs`.
If this table is a view, the input rows are applied to the view's insertable base table (i.e., the root of the
view hierarchy) and the output rows are the resulting rows of the view, as if the input had been inserted into
the base:
* rows that don't satisfy the view's filter are dropped
* an iterator view can produce multiple output rows per input row
**Parameters:**
* **`source`** (`Sequence[dict[str, Any]] | Sequence[pydantic.BaseModel]`): Rows to compute, as a sequence of dictionaries or Pydantic model instances. Rows contain
values for the base table's columns (for a view) or this table's columns; each row must supply
values for every required (non-nullable, non-computed) column used by `outputs`; otherwise the same
rules as [`insert()`](./table#method-insert) apply.
* **`outputs`** (`Sequence[str | ColumnRef] | None`): The columns to compute and return, as names or column references. Defaults to all columns
of the table.
* **`on_error`** (`Literal['abort', 'ignore']`, default: `'abort'`): Determines the behavior if an error occurs while evaluating a computed column or detecting an
invalid media file (such as a corrupt image).
* If `on_error='abort'`, an exception will be raised.
* If `on_error='ignore'`, execution will continue and the (possibly partially) completed rows will be
returned. Any cells with errors will have a `None` value for that cell, with information about the
error recorded in the row's [`errors`](./row). A row whose view filter fails to
evaluate is dropped.
**Returns:**
* `RowBatch`: A [`RowBatch`](./rowbatch) of output rows, in input row order (with an iterator's output
rows in iteration order). Each [`Row`](./row) contains a value for every column in `outputs`.
[`Row.errors`](./row) holds `{'errortype': ..., 'errormsg': ...}` for each cell that raised,
keyed by column or index name (only with `on_error='ignore'`).
**Examples:**
Compute output rows for a table with int columns `a`, `b` and a computed column `c = a + b`:
```python theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
tbl = pxt.get_table('my_table')
rows = tbl.compute([{'a': 1, 'b': 1}, {'a': 2, 'b': 2}])
# rows == [{'a': 1, 'b': 1, 'c': 2}, {'a': 2, 'b': 2, 'c': 4}]
```
Compute and return only `c`:
```python theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
rows = tbl.compute([{'a': 1, 'b': 1}], outputs=['c'])
# rows == [{'c': 2}]
```
Same with Pydantic model inputs:
```python theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
class MyModel(pydantic.BaseModel):
a: int
b: int
rows = tbl.compute([MyModel(a=1, b=2), MyModel(a=3, b=4)])
```
Continue past per-row failures and inspect the per-cell error info:
```python theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
rows = tbl.compute(
[{'a': 0, 'b': 1}, {'a': 2, 'b': 2}], on_error='ignore'
)
# If `c` raised on row 0, rows[0]['c'] is None and rows[0].errors['c']
# contains {'errortype': ..., 'errormsg': ...}.
```
Compute view rows from base table input, for a view `my_view` defined over `my_table` with a
filter `a > 0`:
```python theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
v = pxt.get_table('my_view')
rows = v.compute([{'a': 0, 'b': 1}, {'a': 2, 'b': 2}])
# only the second input row satisfies the filter; rows contains its view row
```
## method count()
```python Signature theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
count() -> int
```
Return the number of rows in this table.
## method cursor()
```python Signature theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
cursor() -> ResultCursor
```
Return a [`ResultCursor`](./resultcursor) that iterates over this table's rows.
See [`ResultCursor`](./resultcursor) for usage examples and lifecycle details.
## method delete()
```python Signature theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
delete(where: exprs.Expr | None = None) -> UpdateStatus
```
Delete rows in this table.
**Parameters:**
* **`where`** (`'exprs.Expr' | None`): a predicate to filter rows to delete.
**Examples:**
Delete all rows in a table:
```python theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
tbl.delete()
```
Delete all rows in a table where column `a` is greater than 5:
```python theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
tbl.delete(tbl.a > 5)
```
## method describe()
```python Signature theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
describe() -> None
```
Print the table schema.
## method distinct()
```python Signature theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
distinct() -> Query
```
Remove duplicate rows from table.
## method drop\_column()
```python Signature theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
drop_column(
column: str | ColumnRef,
if_not_exists: Literal['error', 'ignore'] = 'error'
) -> None
```
Drop a column from the table.
**Parameters:**
* **`column`** (`str | ColumnRef`): The name or reference of the column to drop.
* **`if_not_exists`** (`Literal['error', 'ignore']`, default: `'error'`): Directive for handling a non-existent column. Must be one of the following:
* `'error'`: raise an error if the column does not exist.
* `'ignore'`: do nothing if the column does not exist.
**Examples:**
Drop the column `col` from the table `my_table` by column name:
```python theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
tbl = pxt.get_table('my_table')
tbl.drop_column('col')
```
Drop the column `col` from the table `my_table` by column reference:
```python theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
tbl = pxt.get_table('my_table')
tbl.drop_column(tbl.col)
```
Drop the column `col` from the table `my_table` if it exists, otherwise do nothing:
```python theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
tbl = pxt.get_table('my_table')
tbl.drop_col(tbl.col, if_not_exists='ignore')
```
## method drop\_embedding\_index()
```python Signature theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
drop_embedding_index(
*,
column: str | ColumnRef | None = None,
idx_name: str | None = None,
if_not_exists: Literal['error', 'ignore'] = 'error'
) -> None
```
Drop an embedding index from the table. Either a column name or an index name (but not both) must be
specified. If a column name or reference is specified, it must be a column containing exactly one
embedding index; otherwise the specific index name must be provided instead.
**Parameters:**
* **`column`** (`str | ColumnRef | None`): The name of, or reference to, the column from which to drop the index.
The column must have only one embedding index.
* **`idx_name`** (`str | None`): The name of the index to drop.
* **`if_not_exists`** (`Literal['error', 'ignore']`, default: `'error'`): Directive for handling a non-existent index. Must be one of the following:
* `'error'`: raise an error if the index does not exist.
* `'ignore'`: do nothing if the index does not exist.
Note that `if_not_exists` parameter is only applicable when an `idx_name` is specified
and it does not exist, or when `column` is specified and it has no index.
`if_not_exists` does not apply to non-exisitng column.
**Examples:**
Drop the embedding index on the `img` column of the table `my_table` by column name:
```python theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
tbl = pxt.get_table('my_table')
tbl.drop_embedding_index(column='img')
```
Drop the embedding index on the `img` column of the table `my_table` by column reference:
```python theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
tbl = pxt.get_table('my_table')
tbl.drop_embedding_index(column=tbl.img)
```
Drop the embedding index `idx1` of the table `my_table` by index name:
```python theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
tbl = pxt.get_table('my_table')
tbl.drop_embedding_index(idx_name='idx1')
```
Drop the embedding index `idx1` of the table `my_table` by index name, if it exists, otherwise do nothing:
```python theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
tbl = pxt.get_table('my_table')
tbl.drop_embedding_index(idx_name='idx1', if_not_exists='ignore')
```
## method drop\_index()
```python Signature theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
drop_index(
*,
column: str | ColumnRef | None = None,
idx_name: str | None = None,
if_not_exists: Literal['error', 'ignore'] = 'error'
) -> None
```
Drop an index from the table. Either a column name or an index name (but not both) must be
specified. If a column name or reference is specified, it must be a column containing exactly one index;
otherwise the specific index name must be provided instead.
The B-tree indexes of a table created with `has_default_idxs=True` cannot be dropped: they are managed
automatically, together with the columns they index.
**Parameters:**
* **`column`** (`str | ColumnRef | None`): The name of, or reference to, the column from which to drop the index.
The column must have only one embedding index.
* **`idx_name`** (`str | None`): The name of the index to drop.
* **`if_not_exists`** (`Literal['error', 'ignore']`, default: `'error'`): Directive for handling a non-existent index. Must be one of the following:
* `'error'`: raise an error if the index does not exist.
* `'ignore'`: do nothing if the index does not exist.
Note that `if_not_exists` parameter is only applicable when an `idx_name` is specified
and it does not exist, or when `column` is specified and it has no index.
`if_not_exists` does not apply to non-exisitng column.
**Examples:**
Drop the index on the `img` column of the table `my_table` by column name:
```python theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
tbl = pxt.get_table('my_table')
tbl.drop_index(column_name='img')
```
Drop the index on the `img` column of the table `my_table` by column reference:
```python theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
tbl = pxt.get_table('my_table')
tbl.drop_index(tbl.img)
```
Drop the index `idx1` of the table `my_table` by index name:
```python theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
tbl = pxt.get_table('my_table')
tbl.drop_index(idx_name='idx1')
```
Drop the index `idx1` of the table `my_table` by index name, if it exists, otherwise do nothing:
```python theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
tbl = pxt.get_table('my_table')
tbl.drop_index(idx_name='idx1', if_not_exists='ignore')
```
## method get\_metadata()
```python Signature theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
get_metadata() -> TableMetadata
```
Retrieves metadata associated with this table.
**Returns:**
* `'TableMetadata'`: A [TableMetadata](./tablemetadata) instance containing this table's metadata.
## method get\_versions()
```python Signature theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
get_versions(n: int | None = None) -> list[VersionMetadata]
```
Returns information about versions of this table, most recent first.
`get_versions()` is intended for programmatic access to version metadata; for human-readable
output, use [`history()`](./table#method-history) instead.
**Parameters:**
* **`n`** (`int | None`): if specified, will return at most `n` versions
**Returns:**
* `list[VersionMetadata]`: A list of [VersionMetadata](./versionmetadata) dictionaries, one per version retrieved, most
recent first.
**Examples:**
Retrieve metadata about all versions of the table `tbl`:
```python theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
tbl.get_versions()
```
Retrieve metadata about the most recent 5 versions of the table `tbl`:
```python theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
tbl.get_versions(n=5)
```
## method group\_by()
```python Signatures theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
# Signature 1:
group_by(grouping_tbl: Table, /) -> Query
# Signature 2:
group_by(*grouping_items: exprs.Expr) -> Query
```
Group the rows of this table based on the expression.
See [`Query.group_by`](./query#method-group_by) for more details.
## method head()
```python Signature theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
head(n: int = 10) -> ResultSet
```
Return the first n rows inserted into this table.
## method history()
```python Signature theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
history(n: int | None = None) -> pd.DataFrame
```
Returns a human-readable report about versions of this table.
`history()` is intended for human-readable output of version metadata; for programmatic access,
use [`get_versions()`](./table#method-get_versions) instead.
**Parameters:**
* **`n`** (`int | None`): if specified, will return at most `n` versions
**Returns:**
* `pd.DataFrame`: A report with information about each version, one per row, most recent first.
**Examples:**
Report all versions of the table:
```python theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
tbl.history()
```
Report only the most recent 5 changes to the table:
```python theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
tbl.history(n=5)
```
## method insert()
```python Signatures theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
# Signature 1:
insert(
source: TableDataSource,
/,
*,
source_format: Literal['csv', 'excel', 'parquet', 'json'] | None = None,
schema_overrides: dict[str, ts.ColumnType] | None = None,
on_error: Literal['abort', 'ignore'] = 'abort',
print_stats: bool = False,
return_rows: bool = False,
**kwargs: Any
) -> UpdateStatus
# Signature 2:
insert(
*,
on_error: Literal['abort', 'ignore'] = 'abort',
print_stats: bool = False,
return_rows: bool = False,
**kwargs: Any
) -> UpdateStatus
```
Inserts rows into this table.
You can insert rows directly by providing a list of dictionaries as the `source`.
```python theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
tbl.insert([{'col1': 1, 'col2': 'egg'}, {'col1': 2, 'col2': 'fish'}])
```
You can also insert data from any recognized data source by providing a file path or URL.
```python theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
tbl.insert('path/to/file.csv')
tbl.insert('https://example.com/data.xlsx')
tbl.insert('s3://my-bucket/data.parquet')
```
Pixeltable will attempt to infer the format of the source data, unless the optional `source_format`
parameter is specified. Pixeltable will also attempt to infer the schema of the source data; you can
override the inferred schema by providing a `schema_overrides` dictionary (which may include all
columns or just a subset of columns).
The `source` can also be another table or a [`Query`](./query):
```python theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
tbl.insert(
other_tbl.select(
col1=other_tbl.other_col, col2=other_tbl.yet_another_col
)
)
```
For inserting just a single row, there is a convenient shorthand key/value syntax:
```python theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
tbl.insert(col1=1, col2='egg')
```
**Parameters:**
* **`source`** (`TableDataSource | None`): A data source from which data can be imported. Can be any of the following:
* A list of dictionaries
* A list of Pydantic model instances
* A file path or URI of a recognized data source
* A Pandas `DataFrame`
* Another Pixeltable table or a `Query`
* A Hugging Face dataset
* **`kwargs`** (`Any`): (if inserting a single row) Keyword-argument pairs representing column names and values.
(if inserting multiple rows) Additional keyword arguments are passed to the data source.
* **`source_format`** (`Literal['csv', 'excel', 'parquet', 'json'] | None`): A hint about the format of the source data. If not specified, Pixeltable will attempt
to infer the format.
* **`schema_overrides`** (`dict[str, ts.ColumnType] | None`): If specified, then columns in `schema_overrides` will be given the specified types.
Any columns not included in `schema_overrides` will have their types inferred as usual.
* **`on_error`** (`Literal['abort', 'ignore']`, default: `'abort'`): Determines the behavior if an error occurs while evaluating a computed column or detecting an
invalid media file (such as a corrupt image) for one of the inserted rows.
* If `on_error='abort'`, then an exception will be raised and the rows will not be inserted.
* If `on_error='ignore'`, then execution will continue and the rows will be inserted. Any cells
with errors will have a `None` value for that cell, with information about the error stored in the
corresponding `tbl.col_name.errortype` and `tbl.col_name.errormsg` fields.
* **`print_stats`** (`bool`, default: `False`): If `True`, print statistics about the cost of computed columns.
* **`return_rows`** (`bool`, default: `False`): If `True`, populate `UpdateStatus.rows` with one dict per inserted row, mapping column names
to their stored or computed values. If `False` (default), `UpdateStatus.rows` is `None`.
**Returns:**
* `UpdateStatus`: An [`UpdateStatus`](./updatestatus) object containing information about the update.
**Examples:**
Insert two rows into the table `my_table` with three int columns `a`, `b`, and `c`.
Column `c` is nullable:
```python theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
tbl = pxt.get_table('my_table')
tbl.insert([{'a': 1, 'b': 1, 'c': 1}, {'a': 2, 'b': 2}])
```
Insert a single row using the alternative syntax:
```python theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
tbl.insert(a=3, b=3, c=3)
```
Insert rows from a CSV file:
```python theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
tbl.insert('path/to/file.csv')
```
Insert Pydantic model instances into a table with two `pxt.Int` columns `a` and `b`:
```python theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
class MyModel(pydantic.BaseModel):
a: int
b: int
models = [MyModel(a=1, b=2), MyModel(a=3, b=4)]
tbl.insert(models)
```
## method join()
```python Signature theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
join(
other: Table,
*,
on: exprs.Expr | None = None,
how: JoinType.LiteralType = 'inner'
) -> Query
```
Join this table with another table.
## method limit()
```python Signature theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
limit(n: int, offset: int | None = None) -> Query
```
Select a limited number of rows from the Table, optionally skipping rows for pagination.
**Parameters:**
* **`n`** (`int`): Number of rows to select.
* **`offset`** (`int | None`): Number of rows to skip before returning results. Default is None (no offset).
**Returns:**
* `'Query'`: A Query with the specified limited rows.
**Examples:**
Get the first 10 rows:
```python theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
t.limit(10).collect()
```
Get rows 21-30 (skip first 20, return next 10):
```python theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
t.limit(10, offset=20).collect()
```
## method list\_views()
```python Signature theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
list_views(*, recursive: bool = True) -> list[str]
```
Returns a list of all views and snapshots of this `Table`.
**Parameters:**
* **`recursive`** (`bool`, default: `True`): If `False`, returns only the immediate successor views of this `Table`. If `True`, returns
all sub-views (including views of views, etc.)
**Returns:**
* `list[str]`: A list of view paths.
## method order\_by()
```python Signature theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
order_by(*items: exprs.Expr, asc: bool = True) -> Query
```
Order the rows of this table based on the expression.
See [`Query.order_by`](./query#method-order_by) for more details.
## method recompute\_columns()
```python Signature theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
recompute_columns(
*columns: str | ColumnRef,
where: exprs.Expr | None = None,
errors_only: bool = False,
cascade: bool = True
) -> UpdateStatus
```
Recompute the values in one or more computed columns of this table.
**Parameters:**
* **`columns`** (`str | ColumnRef`): The names or references of the computed columns to recompute.
* **`where`** (`'exprs.Expr' | None`): A predicate to filter rows to recompute.
* **`errors_only`** (`bool`, default: `False`): If True, only run the recomputation for rows that have errors in the column (ie, the column's
`errortype` property indicates that an error occurred). Only allowed for recomputing a single column.
* **`cascade`** (`bool`, default: `True`): if True, also update all computed columns that transitively depend on the recomputed columns.
**Examples:**
Recompute computed columns `c1` and `c2` for all rows in this table, and everything that transitively
depends on them:
```python theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
tbl.recompute_columns('c1', 'c2')
```
Recompute computed column `c1` for all rows in this table, but don't recompute other columns that depend on
it:
```python theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
tbl.recompute_columns(tbl.c1, tbl.c2, cascade=False)
```
Recompute column `c1` and its dependents, but only for rows with `c2` == 0:
```python theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
tbl.recompute_columns('c1', where=tbl.c2 == 0)
```
Recompute column `c1` and its dependents, but only for rows that have errors in it:
```python theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
tbl.recompute_columns('c1', errors_only=True)
```
## method rename\_column()
```python Signature theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
rename_column(old_name: str, new_name: str) -> None
```
Rename a column.
**Parameters:**
* **`old_name`** (`str`): The current name of the column.
* **`new_name`** (`str`): The new name of the column.
**Examples:**
Rename the column `col1` to `col2` of the table `my_table`:
```python theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
tbl = pxt.get_table('my_table')
tbl.rename_column('col1', 'col2')
```
## method revert()
```python Signature theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
revert() -> None
```
Reverts the table to the previous version.
.. warning::
This operation is irreversible.
## method sample()
```python Signature theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
sample(
n: int | None = None,
n_per_stratum: int | None = None,
fraction: float | None = None,
seed: int | None = None,
stratify_by: Any = None
) -> Query
```
Choose a shuffled sample of rows
See [`Query.sample`](./query#method-sample) for more details.
## method select()
```python Signature theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
select(*items: Any, **named_items: Any) -> Query
```
Select columns or expressions from this table.
See [`Query.select`](./query#method-select) for more details.
## method show()
```python Signature theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
show(n: int = 20) -> ResultSet
```
Return the first n rows from this table.
## method tail()
```python Signature theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
tail(n: int = 10) -> ResultSet
```
Return the last n rows inserted into this table.
## method update()
```python Signature theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
update(
value_spec: dict[str, Any],
where: exprs.Expr | None = None,
cascade: bool = True,
return_rows: bool = False
) -> UpdateStatus
```
Update rows in this table.
**Parameters:**
* **`value_spec`** (`dict[str, Any]`): a dictionary mapping column names to literal values or Pixeltable expressions.
* **`where`** (`'exprs.Expr' | None`): a predicate to filter rows to update.
* **`cascade`** (`bool`, default: `True`): if True, also update all computed columns that transitively depend on the updated columns.
* **`return_rows`** (`bool`, default: `False`): If `True`, populate `UpdateStatus.rows` with one dict per updated row, mapping column
names to their new stored values. If `False` (default), `UpdateStatus.rows` is `None`.
**Returns:**
* `UpdateStatus`: An [`UpdateStatus`](./updatestatus) object containing information about the update.
**Examples:**
Set column `int_col` to 1 for all rows:
```python theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
tbl.update({'int_col': 1})
```
Set column `int_col` to 1 for all rows where `int_col` is 0:
```python theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
tbl.update({'int_col': 1}, where=tbl.int_col == 0)
```
Set `int_col` to the value of `other_int_col` + 1:
```python theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
tbl.update({'int_col': tbl.other_int_col + 1})
```
Increment `int_col` by 1 for all rows where `int_col` is 0:
```python theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
tbl.update({'int_col': tbl.int_col + 1}, where=tbl.int_col == 0)
```
## method where()
```python Signature theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
where(pred: exprs.Expr) -> Query
```
Filter rows from this table based on the expression.
See [`Query.where`](./query#method-where) for more details.
# TableMetadata
Source: https://docs.pixeltable.com/sdk/latest/tablemetadata
# class pixeltable.TableMetadata
Metadata for a Pixeltable table.
## attr base
```
base: str | None
```
If this table is a view or snapshot, the full path of its base table; otherwise `None`.
## attr columns
```
columns: dict[str, ColumnMetadata]
```
Column metadata for all of the visible columns of the table.
## attr comment
```
comment: str | None
```
User-provided table comment, if one exists.
## attr custom\_metadata
```
custom_metadata: Any
```
User-defined JSON metadata for this table, if any.
## attr has\_default\_idxs
```
has_default_idxs: bool
```
`True` if eligible columns of this table get a default B-tree index.
## attr id
```
id: uuid.UUID
```
The stable UUID of the table. Useful for detecting drop-and-recreate across time.
## attr indexes
```
indexes: dict[str, IndexMetadata]
```
Index metadata for all of the indexes of the table.
## attr is\_data\_versioned
```
is_data_versioned: bool
```
`True` if this is a data-versioned table.
## attr is\_snapshot
```
is_snapshot: bool
```
`True` if this table is a snapshot.
## attr is\_view
```
is_view: bool
```
`True` if this table is a view.
## attr iterator\_call
```
iterator_call: str | None
```
The iterator call for views that use an iterator; otherwise `None`.
## attr kind
```
kind: Literal['table', 'view', 'snapshot']
```
The kind of table: `'table'`, `'view'`, or `'snapshot'`.
## attr media\_validation
```
media_validation: Literal['on_read', 'on_write']
```
The media validation policy for this table.
## attr name
```
name: str
```
The name of the table (ex: `'my_table'`).
## attr path
```
path: str
```
The full path of the table (ex: `'my_dir.my_subdir.my_table'`).
## attr primary\_key
```
primary_key: list[str] | None
```
List of primary key column names, or `None` if this table has no primary key.
## attr schema\_version
```
schema_version: int
```
The current schema version of the table.
## attr version
```
version: int
```
The current version of the table.
## attr version\_created
```
version_created: datetime.datetime
```
The timestamp when this table version was created.
## attr view\_filter
```
view_filter: str | None
```
A display string of a view's filter (`where`) predicate; `None` if this is not a view or has no filter.
## attr view\_sample
```
view_sample: str | None
```
A display string of a view's `sample` clause; `None` if this is not a view or has no sample clause.
# TableNode
Source: https://docs.pixeltable.com/sdk/latest/tablenode
# class pixeltable.TableNode
A table/view/snapshot entry in a [`TreeNode`](./treenode) tree.
## attr base
```
base: str | None
```
Path of the immediate base table for views/snapshots; None for plain tables.
## attr error\_count
```
error_count: int
```
Cumulative error count as recorded in table's history.
## attr kind
```
kind: TableKind
```
⚠️ **No documentation**
## attr name
```
name: str
```
⚠️ **No documentation**
## attr path
```
path: str
```
⚠️ **No documentation**
## attr version
```
version: int | None
```
⚠️ **No documentation**
# telemetry
Source: https://docs.pixeltable.com/sdk/latest/telemetry
# module pixeltable.telemetry
Dependency-free instrumentation hooks.
Core pixeltable reports spans (timed, nested units of work) and discrete events through this module;
subscribers (e.g. the bridge in `opentelemetry-instrumentation-pixeltable`) translate them into a
telemetry backend. With no subscribers registered every call is a near-free no-op, but hot loops should
still guard with `if telemetry.active():` before building attribute dicts (or pass `attrs` as a
callable).
Spans should cover contiguous units of real computation (a UDF call, a DB insert batch, model loading) or
serve as structural containers (operation and exec-node spans). A CPU work span must not contain a
`yield`/`await`, which would let it cover unrelated interleaved work; awaiting an external call inside a
span is fine (the request really is in flight).
## func emit()
```python Signature theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
emit(name: str, attrs: HookAttrs = None) -> None
```
Report a discrete event (e.g. a counter increment) to all subscribers.
## func span()
```python Signature theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
span(
name: str,
*,
level: int = 20,
parent: SpanHandle | None = None,
set_current: bool = False,
**attrs: Any
) -> Iterator[SpanHandle | None]
```
Context-manager sugar for a lexical-block span.
Keyword attrs get a 'pxt.' prefix; None values are skipped.
## func span\_end()
```python Signature theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
span_end(
handle: SpanHandle | None,
*,
exc: BaseException | None = None,
attrs: HookAttrs = None
) -> None
```
End a span started with span\_start(); a no-op for None handles.
## func span\_start()
```python Signature theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
span_start(
name: str,
*,
level: int = 20,
parent: SpanHandle | None = None,
set_current: bool = False,
attrs: HookAttrs = None
) -> SpanHandle | None
```
Start a span and return its handle; None if no subscribers are registered or the span is suppressed.
Spans below the level threshold are suppressed, as are non-set\_current spans with no ambient ancestor;
descendants of a suppressed span parent to the ambient span. set\_current=True makes the span the
ambient parent (a ContextVar, so it propagates into asyncio tasks) and requires that span\_end() runs
on the same thread/context; use capture\_context()/restore\_context()/exit\_context() for explicit
thread handoffs.
# timestamp
Source: https://docs.pixeltable.com/sdk/latest/timestamp
# module pixeltable.functions.timestamp
Pixeltable UDFs for `TimestampType`.
Usage example:
```python theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
import pixeltable as pxt
t = pxt.get_table(...)
t.select(t.timestamp_col.year, t.timestamp_col.weekday()).collect()
```
## udf astimezone()
```python Signature theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
@pxt.udf
astimezone(self: pxt.Timestamp, tz: pxt.String) -> pxt.Timestamp
```
Convert the datetime to the given time zone.
**Parameters:**
* **`tz`** (`pxt.String`): The time zone to convert to. Must be a valid time zone name from the
[IANA Time Zone Database](https://en.wikipedia.org/wiki/List_of_tz_database_time_zones).
## udf day()
```python Signature theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
@pxt.udf
day(self: pxt.Timestamp) -> pxt.Int
```
Between 1 and the number of days in the given month of the given year.
Equivalent to [`datetime.day`](https://docs.python.org/3/library/datetime.html#datetime.datetime.day).
## udf hour()
```python Signature theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
@pxt.udf
hour(self: pxt.Timestamp) -> pxt.Int
```
Between 0 and 23 inclusive.
Equivalent to [`datetime.hour`](https://docs.python.org/3/library/datetime.html#datetime.datetime.hour).
## udf isocalendar()
```python Signature theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
@pxt.udf
isocalendar(self: pxt.Timestamp) -> IsoCalendar
```
Return a dictionary with three entries: `'year'`, `'week'`, and `'weekday'`.
Equivalent to
[`datetime.isocalendar()`](https://docs.python.org/3/library/datetime.html#datetime.datetime.isocalendar).
## udf isoformat()
```python Signature theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
@pxt.udf
isoformat(
self: pxt.Timestamp,
sep: pxt.String = 'T',
timespec: pxt.String = 'auto'
) -> pxt.String
```
Return a string representing the date and time in ISO 8601 format.
Equivalent to [`datetime.isoformat()`](https://docs.python.org/3/library/datetime.html#datetime.datetime.isoformat).
**Parameters:**
* **`sep`** (`pxt.String`): Separator between date and time.
* **`timespec`** (`pxt.String`): The number of additional terms in the output. See the
[`datetime.isoformat()`](https://docs.python.org/3/library/datetime.html#datetime.datetime.isoformat)
documentation for more details.
## udf isoweekday()
```python Signature theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
@pxt.udf
isoweekday(self: pxt.Timestamp) -> pxt.Int
```
Return the day of the week as an integer, where Monday is 1 and Sunday is 7.
Equivalent to [`datetime.isoweekday()`](https://docs.python.org/3/library/datetime.html#datetime.datetime.isoweekday).
## udf make\_timestamp()
```python Signature theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
@pxt.udf
make_timestamp(
year: pxt.Int,
month: pxt.Int,
day: pxt.Int,
hour: pxt.Int = 0,
minute: pxt.Int = 0,
second: pxt.Int = 0,
microsecond: pxt.Int = 0
) -> pxt.Timestamp
```
Create a timestamp.
Equivalent to [`datetime()`](https://docs.python.org/3/library/datetime.html#datetime.datetime).
## udf microsecond()
```python Signature theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
@pxt.udf
microsecond(self: pxt.Timestamp) -> pxt.Int
```
Between 0 and 999999 inclusive.
Equivalent to [`datetime.microsecond`](https://docs.python.org/3/library/datetime.html#datetime.datetime.microsecond).
## udf minute()
```python Signature theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
@pxt.udf
minute(self: pxt.Timestamp) -> pxt.Int
```
Between 0 and 59 inclusive.
Equivalent to [`datetime.minute`](https://docs.python.org/3/library/datetime.html#datetime.datetime.minute).
## udf month()
```python Signature theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
@pxt.udf
month(self: pxt.Timestamp) -> pxt.Int
```
Between 1 and 12 inclusive.
Equivalent to [`datetime.month`](https://docs.python.org/3/library/datetime.html#datetime.datetime.month).
## udf posix\_timestamp()
```python Signature theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
@pxt.udf
posix_timestamp(self: pxt.Timestamp) -> pxt.Float
```
Return POSIX timestamp corresponding to the datetime instance.
Equivalent to [`datetime.timestamp()`](https://docs.python.org/3/library/datetime.html#datetime.datetime.timestamp).
## udf replace()
```python Signature theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
@pxt.udf
replace(
self: pxt.Timestamp,
year: pxt.Int | None = None,
month: pxt.Int | None = None,
day: pxt.Int | None = None,
hour: pxt.Int | None = None,
minute: pxt.Int | None = None,
second: pxt.Int | None = None,
microsecond: pxt.Int | None = None
) -> pxt.Timestamp
```
Return a datetime with the same attributes, except for those attributes given new values by whichever keyword
arguments are specified.
Equivalent to [`datetime.replace()`](https://docs.python.org/3/library/datetime.html#datetime.datetime.replace).
## udf second()
```python Signature theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
@pxt.udf
second(self: pxt.Timestamp) -> pxt.Int
```
Between 0 and 59 inclusive.
Equivalent to [`datetime.second`](https://docs.python.org/3/library/datetime.html#datetime.datetime.second).
## udf strftime()
```python Signature theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
@pxt.udf
strftime(self: pxt.Timestamp, format: pxt.String) -> pxt.String
```
Return a string representing the date and time, controlled by an explicit format string.
Equivalent to [`datetime.strftime()`](https://docs.python.org/3/library/datetime.html#datetime.datetime.strftime).
**Parameters:**
* **`format`** (`pxt.String`): The format string to control the output. For a complete list of formatting directives, see
[`strftime()` and `strptime()` Behavior](https://docs.python.org/3/library/datetime.html#strftime-and-strptime-behavior).
## udf strip\_timezone()
```python Signature theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
@pxt.udf
strip_timezone(self: pxt.Timestamp) -> pxt.Timestamp
```
Return a naive datetime with the same wall-clock as `self` and no `tzinfo`.
Equivalent to [`datetime.replace(tzinfo=None)`](https://docs.python.org/3/library/datetime.html#datetime.datetime.replace).
## udf toordinal()
```python Signature theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
@pxt.udf
toordinal(self: pxt.Timestamp) -> pxt.Int
```
Return the proleptic Gregorian ordinal of the date, where January 1 of year 1 has ordinal 1.
Equivalent to [`datetime.toordinal()`](https://docs.python.org/3/library/datetime.html#datetime.datetime.toordinal).
## udf weekday()
```python Signature theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
@pxt.udf
weekday(self: pxt.Timestamp) -> pxt.Int
```
Between 0 (Monday) and 6 (Sunday) inclusive.
Equivalent to [`datetime.weekday()`](https://docs.python.org/3/library/datetime.html#datetime.datetime.weekday).
## udf year()
```python Signature theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
@pxt.udf
year(self: pxt.Timestamp) -> pxt.Int
```
Between [`MINYEAR`](https://docs.python.org/3/library/datetime.html#datetime.MINYEAR) and
[`MAXYEAR`](https://docs.python.org/3/library/datetime.html#datetime.MAXYEAR) inclusive.
Equivalent to [`datetime.year`](https://docs.python.org/3/library/datetime.html#datetime.datetime.year).
# together
Source: https://docs.pixeltable.com/sdk/latest/together
# module pixeltable.functions.together
Pixeltable UDFs
that wrap various endpoints from the Together AI API. In order to use them, you must
first `pip install together` and configure your Together AI credentials, as described in
the [Working with Together AI](https://docs.pixeltable.com/howto/providers/working-with-together) tutorial.
## udf chat\_completions()
```python Signature theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
@pxt.udf
chat_completions(
messages: pxt.Json[(Json, ...)],
*,
model: pxt.String,
model_kwargs: pxt.Json | None = None
) -> pxt.Json
```
Generate chat completions based on a given prompt using a specified model.
Equivalent to the Together AI `chat/completions` API endpoint.
For additional details, see: [https://docs.together.ai/reference/chat-completions-1](https://docs.together.ai/reference/chat-completions-1)
Request throttling:
Applies the rate limit set in the config (section `together.rate_limits`, key `chat`). If no rate
limit is configured, uses a default of 600 RPM.
**Requirements:**
* `pip install together`
**Parameters:**
* **`messages`** (`pxt.Json[(Json`): A list of messages comprising the conversation so far.
* **`model`** (`Any`): The name of the model to query.
* **`model_kwargs`** (`Any`): Additional keyword arguments for the Together `chat/completions` API.
For details on the available parameters, see: [https://docs.together.ai/reference/chat-completions-1](https://docs.together.ai/reference/chat-completions-1)
**Returns:**
* `pxt.Json`: A dictionary containing the response and other metadata.
**Examples:**
Add a computed column that applies the model `openai/gpt-oss-20b` to an existing Pixeltable column
`tbl.prompt` of the table `tbl`:
```python theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
messages = [{'role': 'user', 'content': tbl.prompt}]
tbl.add_computed_column(
response=chat_completions(messages, model='openai/gpt-oss-20b')
)
```
## udf completions()
```python Signature theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
@pxt.udf
completions(
prompt: pxt.String,
*,
model: pxt.String,
model_kwargs: pxt.Json | None = None
) -> pxt.Json
```
Generate completions based on a given prompt using a specified model.
Equivalent to the Together AI `completions` API endpoint.
For additional details, see: [https://docs.together.ai/reference/completions-1](https://docs.together.ai/reference/completions-1)
Request throttling:
Applies the rate limit set in the config (section `together.rate_limits`, key `chat`). If no rate
limit is configured, uses a default of 600 RPM.
**Requirements:**
* `pip install together`
**Parameters:**
* **`prompt`** (`pxt.String`): A string providing context for the model to complete.
* **`model`** (`pxt.String`): The name of the model to query.
* **`model_kwargs`** (`pxt.Json | None`): Additional keyword arguments for the Together `completions` API.
For details on the available parameters, see: [https://docs.together.ai/reference/completions-1](https://docs.together.ai/reference/completions-1)
**Returns:**
* `pxt.Json`: A dictionary containing the response and other metadata.
**Examples:**
Add a computed column that applies the model `Qwen/Qwen3.5-9B` to an existing Pixeltable column
`tbl.prompt` of the table `tbl`:
```python theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
tbl.add_computed_column(
response=completions(tbl.prompt, model='Qwen/Qwen3.5-9B')
)
```
## udf embeddings()
```python Signature theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
@pxt.udf
embeddings(
input: pxt.String,
*,
model: pxt.String
) -> pxt.Array[(None,), float32]
```
Query an embedding model for a given string of text.
Equivalent to the Together AI `embeddings` API endpoint.
For additional details, see: [https://docs.together.ai/reference/embeddings-2](https://docs.together.ai/reference/embeddings-2)
Request throttling:
Applies the rate limit set in the config (section `together.rate_limits`, key `embeddings`). If no rate
limit is configured, uses a default of 600 RPM.
**Requirements:**
* `pip install together`
**Parameters:**
* **`input`** (`pxt.String`): A string providing the text for the model to embed.
* **`model`** (`pxt.String`): The name of the embedding model to use.
**Returns:**
* `pxt.Array[(None,), float32]`: An array representing the application of the given embedding to `input`.
**Examples:**
Add a computed column that applies the model `intfloat/multilingual-e5-large-instruct`
to an existing Pixeltable column `tbl.text` of the table `tbl`:
```python theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
tbl.add_computed_column(
response=embeddings(
tbl.text, model='intfloat/multilingual-e5-large-instruct'
)
)
```
## udf image\_generations()
```python Signature theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
@pxt.udf
image_generations(
prompt: pxt.String,
*,
model: pxt.String,
model_kwargs: pxt.Json | None = None
) -> pxt.Image
```
Generate images based on a given prompt using a specified model.
Equivalent to the Together AI `images/generations` API endpoint.
For additional details, see: [https://docs.together.ai/reference/post\_images-generations](https://docs.together.ai/reference/post_images-generations)
Request throttling:
Applies the rate limit set in the config (section `together.rate_limits`, key `images`). If no rate
limit is configured, uses a default of 600 RPM.
**Requirements:**
* `pip install together`
**Parameters:**
* **`prompt`** (`pxt.String`): A description of the desired images.
* **`model`** (`pxt.String`): The model to use for image generation.
* **`model_kwargs`** (`pxt.Json | None`): Additional keyword args for the Together `images/generations` API.
For details on the available parameters, see: [https://docs.together.ai/reference/post\_images-generations](https://docs.together.ai/reference/post_images-generations)
**Returns:**
* `pxt.Image`: The generated image.
**Examples:**
Add a computed column that applies the model `Rundiffusion/Juggernaut-Lightning-Flux`
to an existing Pixeltable column `tbl.prompt` of the table `tbl`:
```python theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
tbl.add_computed_column(
response=image_generations(
tbl.prompt, model='Rundiffusion/Juggernaut-Lightning-Flux'
)
)
```
# TreeNode
Source: https://docs.pixeltable.com/sdk/latest/treenode
## ⚠️ Error Loading Class
Failed to load class `TreeNode`:
```
TreeNode is not a class
```
# twelvelabs
Source: https://docs.pixeltable.com/sdk/latest/twelvelabs
# module pixeltable.functions.twelvelabs
Pixeltable UDFs
that wrap various endpoints from the TwelveLabs API. In order to use them, you must
first `pip install twelvelabs` and configure your TwelveLabs credentials, as described in
the [Working with TwelveLabs](https://docs.pixeltable.com/howto/providers/working-with-twelvelabs) tutorial.
## udf embed()
```python Signatures theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
# Signature 1:
@pxt.udf
embed(
text: pxt.String,
image: pxt.Image | None,
*,
model_name: pxt.String
) -> pxt.Array[float32] | None
# Signature 2:
@pxt.udf
embed(
image: pxt.Image,
*,
model_name: pxt.String
) -> pxt.Array[float32] | None
# Signature 3:
@pxt.udf
embed(
audio: pxt.Audio,
*,
model_name: pxt.String,
start_sec: pxt.Float | None,
end_sec: pxt.Float | None,
embedding_option: pxt.Json[(String, ...)] | None
) -> pxt.Array[float32] | None
# Signature 4:
@pxt.udf
embed(
video: pxt.Video,
*,
model_name: pxt.String,
start_sec: pxt.Float | None,
end_sec: pxt.Float | None,
embedding_option: pxt.Json[(String, ...)] | None
) -> pxt.Array[float32] | None
```
Creates an embedding vector for the given text, audio, image, or video input.
Each UDF signature corresponds to one of the four supported input types. If text is specified, it is possible to
specify an image as well, corresponding to the `text_image` embedding type in the TwelveLabs API. This is
(currently) the only way to include more than one input type at a time.
Equivalent to the TwelveLabs Embed API:
[https://docs.twelvelabs.io/v1.3/docs/guides/create-embeddings](https://docs.twelvelabs.io/v1.3/docs/guides/create-embeddings)
Request throttling:
Applies the rate limit set in the config (section `twelvelabs`, key `rate_limit`). If no rate
limit is configured, uses a default of 600 RPM.
**Requirements:**
* `pip install twelvelabs`
**Parameters:**
* **`model_name`** (`String`): The name of the model to use. Check
[the TwelveLabs documentation](https://docs.twelvelabs.io/v1.3/sdk-reference/python/create-embeddings-v-1/create-text-image-and-audio-embeddings)
for available models.
* **`text`** (`String`): The text to embed.
* **`image`** (`Image | None`, default: `Literal(None)`): If specified, the embedding will be created from both the text and the image.
**Returns:**
* `pxt.Array[float32] | None`: The embedding.
**Examples:**
Add a computed column `embed` for an embedding of a string column `input`:
```python theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
tbl.add_computed_column(
embed=embed(model_name='marengo3.0', text=tbl.input)
)
```
# UpdateStatus
Source: https://docs.pixeltable.com/sdk/latest/updatestatus
# class pixeltable.UpdateStatus
Information about changes to table data or table schema
## attr num\_computed\_values
```
num_computed_values: int
```
Total number of computed values affected (including cascaded changes).
## attr num\_excs
```
num_excs: int
```
Total number of exceptions encountered (including cascaded changes).
## attr num\_rows
```
num_rows: int
```
Total number of rows affected (including cascaded changes).
## attr pxt\_rows\_updated
```
pxt_rows_updated: int
```
Returns the number of Pixeltable rows that were updated as a result of the operation.
# URI
Source: https://docs.pixeltable.com/sdk/latest/uri
# class pixeltable.URI
A storage destination: an object-store URI, or a local filesystem path. Validates at construction.
# uuid
Source: https://docs.pixeltable.com/sdk/latest/uuid
# module pixeltable.functions.uuid
Pixeltable UDFs for `UUID`.
## udf to\_string()
```python Signature theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
@pxt.udf
to_string(u: pxt.UUID) -> pxt.String
```
Convert a UUID to its string representation.
**Parameters:**
* **`u`** (`pxt.UUID`): The UUID to convert.
**Returns:**
* `pxt.String`: The string representation of the UUID, in the form `xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx`.
**Examples:**
Convert the UUID column `id` in an existing table `tbl` to a string:
```python theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
tbl.add_computed_column(id_string=to_string(tbl.id))
```
## udf uuid4()
```python Signature theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
@pxt.udf
uuid4() -> pxt.UUID
```
Generate a random UUID (version 4).
Equivalent to [`uuid.uuid4()`](https://docs.python.org/3/library/uuid.html#uuid.uuid4).
## udf uuid7()
```python Signature theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
@pxt.udf
uuid7() -> pxt.UUID
```
Generate a time-based UUID.
Equivalent to [`uuid.uuid7()`](https://docs.python.org/3/library/uuid.html#uuid.uuid7).
# VersionMetadata
Source: https://docs.pixeltable.com/sdk/latest/versionmetadata
# class pixeltable.VersionMetadata
Metadata for a specific version of a Pixeltable table.
## attr change\_type
```
change_type: Literal['data', 'schema']
```
The type of table transformation that this version represents (`'data'` or `'schema'`).
## attr created\_at
```
created_at: datetime.datetime
```
The timestamp when this version was created.
## attr deletes
```
deletes: int
```
The number of rows deleted in this version.
## attr errors
```
errors: int
```
The number of errors encountered during this version.
## attr inserts
```
inserts: int
```
The number of rows inserted in this version.
## attr schema\_change
```
schema_change: str | None
```
A description of the schema change that occurred in this version, if any.
## attr updates
```
updates: int
```
The number of rows updated in this version.
## attr user
```
user: str | None
```
The user who created this version, if defined.
## attr version
```
version: int
```
The version number.
# video
Source: https://docs.pixeltable.com/sdk/latest/video
# module pixeltable.functions.video
Pixeltable UDFs for VideoType.
## iterator frame\_iterator()
```python Signature theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
@pxt.iterator
frame_iterator(
video: pxt.Video,
*,
fps: pxt.Float | None = None,
num_frames: pxt.Int | None = None,
keyframes_only: pxt.Bool = False
)
```
Iterator over frames of a video. At most one of `fps`, `num_frames` or `keyframes_only` may be specified. If `fps`
is specified, then frames will be extracted at the specified rate (frames per second). If `num_frames` is specified,
then the exact number of frames will be extracted. If neither is specified, then all frames will be extracted.
**Outputs:**
One row per extracted frame, with the following columns:
* `frame` (`pxt.Image`): The extracted video frame
* `frame_attrs` (`pxt.Json`): A dictionary containing the following attributes (for more information,
see `pyav`'s documentation on
[VideoFrame](https://pyav.org/docs/develop/api/video.html#module-av.video.frame) and
[Frame](https://pyav.org/docs/develop/api/frame.html)):
* `index` (`int`): The index of the frame in the video stream
* `pts` (`int | None`): The presentation timestamp of the frame
* `dts` (`int | None`): The decoding timestamp of the frame
* `time` (`float | None`): The timestamp of the frame in seconds
* `is_corrupt` (`bool`): `True` if the frame is corrupt
* `key_frame` (`bool`): `True` if the frame is a keyframe
* `pict_type` (`int`): The picture type of the frame
* `interlaced_frame` (`bool`): `True` if the frame is interlaced
**Parameters:**
* **`fps`** (`pxt.Float | None`): Number of frames to extract per second of video. This may be a fractional value, such as 0.5.
If omitted, or if greater than the native framerate of the video,
then the framerate of the video will be used (all frames will be extracted).
* **`num_frames`** (`pxt.Int | None`): Exact number of frames to extract. The frames will be spaced as evenly as possible. If
`num_frames` is greater than the number of frames in the video, all frames will be extracted.
* **`keyframes_only`** (`pxt.Bool`): If True, only extract keyframes.
**Examples:**
All these examples assume an existing table `tbl` with a column `video` of type `pxt.Video`.
Create a view that extracts all frames from all videos:
```python theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
pxt.create_view('all_frames', tbl, iterator=frame_iterator(tbl.video))
```
Create a view that extracts only keyframes from all videos:
```python theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
pxt.create_view(
'keyframes',
tbl,
iterator=frame_iterator(tbl.video, keyframes_only=True),
)
```
Create a view that extracts frames from all videos at a rate of 1 frame per second:
```python theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
pxt.create_view(
'one_fps_frames', tbl, iterator=frame_iterator(tbl.video, fps=1.0)
)
```
Create a view that extracts exactly 10 frames from each video:
```python theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
pxt.create_view(
'ten_frames', tbl, iterator=frame_iterator(tbl.video, num_frames=10)
)
```
## iterator video\_splitter()
```python Signature theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
@pxt.iterator
video_splitter(
video: pxt.Video,
*,
duration: pxt.Float | None = None,
overlap: pxt.Float | None = None,
min_segment_duration: pxt.Float | None = None,
segment_times: pxt.Json[(Float, ...)] | None = None,
mode: pxt.String = 'accurate',
video_encoder: pxt.String | None = None,
video_encoder_args: pxt.Json | None = None
)
```
Iterator over segments of a video file, which is split into segments. The segments are specified either via a
fixed duration or a list of split points.
**Parameters:**
* **`duration`** (`pxt.Float | None`): Video segment duration in seconds
* **`overlap`** (`pxt.Float | None`): Overlap between consecutive segments in seconds. Only available for `mode='fast'`.
* **`min_segment_duration`** (`pxt.Float | None`): Drop the last segment if it is smaller than min\_segment\_duration.
* **`segment_times`** (`pxt.Json[(Float`): List of timestamps (in seconds) in video where segments should be split. Note that these are not
segment durations. If all segment times are less than the duration of the video, produces exactly
`len(segment_times) + 1` segments. An argument of `[]` will produce a single segment containing the
entire video.
* **`mode`** (`Any`): Segmentation mode:
* `'fast'`: Quick segmentation using stream copy (splits only at keyframes, approximate durations)
* `'accurate'`: Precise segmentation with re-encoding (exact durations, slower)
* **`video_encoder`** (`Any`): Video encoder to use. If not specified, uses the default encoder for the current platform.
Only available for `mode='accurate'`.
* **`video_encoder_args`** (`Any`): Additional arguments to pass to the video encoder. Only available for `mode='accurate'`.
**Examples:**
All these examples assume an existing table `tbl` with a column `video` of type `pxt.Video`.
Create a view that splits each video into 10-second segments:
```python theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
pxt.create_view(
'ten_second_segments',
tbl,
iterator=video_splitter(tbl.video, duration=10.0),
)
```
Create a view that splits each video into segments at specified fixed times:
```python theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
split_times = [5.0, 15.0, 30.0]
pxt.create_view(
'custom_segments',
tbl,
iterator=video_splitter(tbl.video, segment_times=split_times),
)
```
Create a view that splits each video into segments at times specified by a column `split_times` of type
`pxt.Json`, containing a list of timestamps in seconds:
```python theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
pxt.create_view(
'custom_segments',
tbl,
iterator=video_splitter(tbl.video, segment_times=tbl.split_times),
)
```
## uda concat\_videos\_agg()
```python Signature theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
@pxt.uda
concat_videos_agg(*args, **kwargs) -> pxt.Video | None
```
Aggregate function that merges videos into a single video.
**Requirements:**
* `ffmpeg` needs to be installed and in PATH
* All videos must have the same resolution
**Parameters:**
* **`video_encoder`** (`pxt.String | None`): Video encoder to use. If not specified, uses the default encoder for the current platform.
* **`video_encoder_args`** (`pxt.Json | None`): Additional arguments to pass to the video encoder.
**Returns:**
* `pxt.Video | None`: A new video containing all input videos concatenated in order, or None if all inputs are None.
**Examples:**
Concatenate all videos in a table, ordered by timestamp:
```python theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
tbl.select(concat_videos_agg(tbl.timestamp, tbl.video)).collect()
```
## uda make\_video()
```python Signature theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
@pxt.uda
make_video(*args, **kwargs) -> pxt.Video
```
Aggregate function that creates a video from a sequence of images, using the default video encoder and
yuv420p pixel format.
Call as `make_video(order_key, frame, *, fps=25)`. `order_key` is a frame index or timestamp.
`order_by=` is invalid.
**Parameters:**
* **`fps`** (`pxt.Int`): Frames per second for the output video.
**Returns:**
* `pxt.Video`: The video obtained by combining the input frames at the specified `fps`.
**Examples:**
Combine the frames in `tbl.frame`, ordered by `tbl.pos`:
```python theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
tbl.select(make_video(tbl.pos, tbl.frame, fps=30)).collect()
```
Combine a sequence of rotated frames:
```python theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
tbl.select(make_video(tbl.pos, tbl.frame.rotate(45), fps=30)).collect()
```
For a more extensive example, see the
[Object Detection in Videos](https://docs.pixeltable.com/howto/use-cases/object-detection-in-videos)
cookbook.
## udf clip()
```python Signature theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
@pxt.udf
clip(
video: pxt.Video,
*,
start_time: pxt.Float,
end_time: pxt.Float | None = None,
duration: pxt.Float | None = None,
mode: pxt.String = 'accurate',
video_encoder: pxt.String | None = None,
video_encoder_args: pxt.Json | None = None
) -> pxt.Video | None
```
Extract a clip from a video, specified by `start_time` and either `end_time` or `duration` (in seconds).
If `start_time` is beyond the end of the video, returns None. Can only specify one of `end_time` and `duration`.
If both `end_time` and `duration` are None, the clip goes to the end of the video.
**Requirements:**
* `ffmpeg` needs to be installed and in PATH
**Parameters:**
* **`video`** (`pxt.Video`): Input video file
* **`start_time`** (`pxt.Float`): Start time in seconds
* **`end_time`** (`pxt.Float | None`): End time in seconds
* **`duration`** (`pxt.Float | None`): Duration of the clip in seconds
* **`mode`** (`pxt.String`): Clip mode:
* `'fast'`: avoids re-encoding but starts the clip at the nearest keyframes and as a result, the clip
duration will be slightly longer than requested
* `'accurate'`: extracts a frame-accurate clip, but requires re-encoding
* **`video_encoder`** (`pxt.String | None`): Video encoder to use. If not specified, uses the default encoder for the current platform.
Only available for `mode='accurate'`.
* **`video_encoder_args`** (`pxt.Json | None`): Additional arguments to pass to the video encoder. Only available for `mode='accurate'`.
**Returns:**
* `pxt.Video | None`: New video containing only the specified time range or None if start\_time is beyond the end of the video.
## udf concat\_videos()
```python Signature theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
@pxt.udf
concat_videos(
videos: pxt.Json[(Video, ...)],
*,
video_encoder: pxt.String | None = None,
video_encoder_args: pxt.Json | None = None
) -> pxt.Video | None
```
Merge multiple videos into a single video.
**Requirements:**
* `ffmpeg` needs to be installed and in PATH
**Parameters:**
* **`videos`** (`pxt.Json[(Video`): List of videos to merge.
* **`video_encoder`** (`Any`): Video encoder to use. If not specified, uses the default encoder for the current platform.
* **`video_encoder_args`** (`Any`): Additional arguments to pass to the video encoder.
**Returns:**
* `pxt.Video | None`: A new video containing the merged videos, or None if the input list is empty.
## udf crop()
```python Signature theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
@pxt.udf
crop(
video: pxt.Video,
bbox: pxt.Json[(Int, ...)],
*,
bbox_format: pxt.String = 'xywh',
video_encoder: pxt.String | None = None,
video_encoder_args: pxt.Json | None = None
) -> pxt.Video
```
Crop a rectangular region from a video using ffmpeg's crop filter.
**Requirements:**
* `ffmpeg` needs to be installed and in PATH
**Parameters:**
* **`video`** (`pxt.Video`): Input video.
* **`bbox`** (`pxt.Json[(Int`): Crop region as a list of 4 integers.
* **`bbox_format`** (`Any`): Format of the `bbox` coordinates:
* `'xyxy'`: `[x1, y1, x2, y2]` where (x1, y1) is top-left and (x2, y2) is bottom-right
* `'xywh'`: `[x, y, width, height]` where (x, y) is top-left corner
* `'cxcywh'`: `[cx, cy, width, height]` where (cx, cy) is the center
* **`video_encoder`** (`Any`): Video encoder to use. If not specified, uses the default encoder.
* **`video_encoder_args`** (`Any`): Additional arguments to pass to the video encoder.
**Returns:**
* `pxt.Video`: Video containing the cropped region.
**Examples:**
Crop using default xywh format:
```python theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
tbl.select(tbl.video.crop([100, 50, 320, 240])).collect()
```
Crop using xyxy format (common in object detection):
```python theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
tbl.select(
tbl.video.crop([100, 50, 420, 290], bbox_format='xyxy')
).collect()
```
Crop using center format:
```python theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
tbl.select(
tbl.video.crop([260, 170, 320, 240], bbox_format='cxcywh')
).collect()
```
Use with yolox object detection output:
```python theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
tbl.add_computed_column(
cropped=tbl.video.crop(tbl.detections.bboxes[0], bbox_format='xyxy')
)
```
## udf extract\_audio()
```python Signature theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
@pxt.udf
extract_audio(
video_path: pxt.Video,
stream_idx: pxt.Int = 0,
format: pxt.String = 'wav',
codec: pxt.String | None = None
) -> pxt.Audio | None
```
Extract an audio stream from a video.
**Parameters:**
* **`stream_idx`** (`pxt.Int`): Index of the audio stream to extract.
* **`format`** (`pxt.String`): The target audio format. (`'wav'`, `'mp3'`, `'flac'`).
* **`codec`** (`pxt.String | None`): The codec to use for the audio stream. If not provided, a default codec will be used.
**Returns:**
* `pxt.Audio | None`: The extracted audio, or None if the video has no audio stream at `stream_idx`.
**Examples:**
Add a computed column to a table `tbl` that extracts audio from an existing column `video_col`:
```python theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
tbl.add_computed_column(
extracted_audio=tbl.video_col.extract_audio(format='flac')
)
```
## udf extract\_frame()
```python Signature theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
@pxt.udf
extract_frame(
video: pxt.Video,
*,
timestamp: pxt.Float
) -> pxt.Image | None
```
Extract a single frame from a video at a specific timestamp.
**Parameters:**
* **`video`** (`pxt.Video`): The video from which to extract the frame.
* **`timestamp`** (`pxt.Float`): Extract frame at this timestamp (in seconds).
**Returns:**
* `pxt.Image | None`: The extracted frame as a PIL Image, or None if the timestamp is beyond the video duration.
**Examples:**
Extract the first frame from each video in the `video` column of the table `tbl`:
```python theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
tbl.select(tbl.video.extract_frame(0.0)).collect()
```
Extract a frame close to the end of each video in the `video` column of the table `tbl`:
```python theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
tbl.select(
tbl.video.extract_frame(
tbl.video.get_metadata().streams[0].duration_seconds - 0.1
)
).collect()
```
## udf fade\_in()
```python Signature theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
@pxt.udf
fade_in(
video: pxt.Video,
*,
duration: pxt.Float = 1.0,
color: pxt.String = 'black',
video_encoder: pxt.String | None = None,
video_encoder_args: pxt.Json | None = None
) -> pxt.Video
```
Apply a fade-in effect from a solid color at the start of a video using ffmpeg's fade filter.
The video transitions from a solid `color` to the full video content over `duration` seconds.
**Requirements:**
* `ffmpeg` needs to be installed and in PATH
**Parameters:**
* **`video`** (`pxt.Video`): Input video.
* **`duration`** (`pxt.Float`): Duration of the fade-in effect in seconds.
* **`color`** (`pxt.String`): Color to fade from (e.g., `'black'`, `'white'`, `'#FF0000'`).
* **`video_encoder`** (`pxt.String | None`): Video encoder to use. If not specified, uses the default encoder for the current platform.
* **`video_encoder_args`** (`pxt.Json | None`): Additional arguments to pass to the video encoder.
**Returns:**
* `pxt.Video`: A new video with the fade-in effect applied.
**Examples:**
Apply a 1-second fade from black (default):
```python theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
tbl.select(tbl.video.fade_in()).collect()
```
Apply a 2-second fade from white:
```python theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
tbl.select(tbl.video.fade_in(duration=2.0, color='white')).collect()
```
## udf fade\_out()
```python Signature theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
@pxt.udf
fade_out(
video: pxt.Video,
*,
duration: pxt.Float = 1.0,
color: pxt.String = 'black',
video_encoder: pxt.String | None = None,
video_encoder_args: pxt.Json | None = None
) -> pxt.Video
```
Apply a fade-out effect to a solid color at the end of a video using ffmpeg's fade filter.
The video transitions from the full video content to a solid `color` over the final `duration` seconds.
**Requirements:**
* `ffmpeg` needs to be installed and in PATH
**Parameters:**
* **`video`** (`pxt.Video`): Input video.
* **`duration`** (`pxt.Float`): Duration of the fade-out effect in seconds.
* **`color`** (`pxt.String`): Color to fade to (e.g., `'black'`, `'white'`, `'#FF0000'`).
* **`video_encoder`** (`pxt.String | None`): Video encoder to use. If not specified, uses the default encoder for the current platform.
* **`video_encoder_args`** (`pxt.Json | None`): Additional arguments to pass to the video encoder.
**Returns:**
* `pxt.Video`: A new video with the fade-out effect applied.
**Examples:**
Apply a 1-second fade to black (default):
```python theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
tbl.select(tbl.video.fade_out()).collect()
```
Apply a 3-second fade to white:
```python theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
tbl.select(tbl.video.fade_out(duration=3.0, color='white')).collect()
```
## udf get\_duration()
```python Signature theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
@pxt.udf
get_duration(video: pxt.Video) -> pxt.Float | None
```
Get video duration in seconds.
**Parameters:**
* **`video`** (`pxt.Video`): The video for which to get the duration.
**Returns:**
* `pxt.Float | None`: The duration in seconds, or None if the duration cannot be determined.
## udf get\_metadata()
```python Signature theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
@pxt.udf
get_metadata(video: pxt.Video) -> ContainerMetadata
```
Gets various metadata associated with a video file and returns it as
a [`ContainerMetadata`](./containermetadata) dictionary.
**Parameters:**
* **`video`** (`pxt.Video`): The video for which to get metadata.
**Returns:**
* `ContainerMetadata`: A [`ContainerMetadata`](./containermetadata) with typical structure:
```json theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
{
'bit_exact': False,
'bit_rate': 967260,
'size': 2234371,
'metadata': {
'encoder': 'Lavf60.16.100',
'major_brand': 'isom',
'minor_version': '512',
'compatible_brands': 'isomiso2avc1mp41',
},
'streams': [
{
'type': 'video',
'width': 640,
'height': 360,
'frames': 462,
'time_base': 1.0 / 12800,
'duration': 236544,
'duration_seconds': 236544.0 / 12800,
'average_rate': 25.0,
'base_rate': 25.0,
'guessed_rate': 25.0,
'metadata': {
'language': 'und',
'handler_name': 'L-SMASH Video Handler',
'vendor_id': '[0][0][0][0]',
'encoder': 'Lavc60.31.102 libx264',
},
'codec_context': {'name': 'h264', 'codec_tag': 'avc1', 'profile': 'High', 'pix_fmt': 'yuv420p'},
}
],
}
```
**Examples:**
Extract metadata for files in the `video_col` column of the table `tbl`:
```python theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
tbl.select(tbl.video_col.get_metadata()).collect()
```
## udf grayscale()
```python Signature theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
@pxt.udf
grayscale(
video: pxt.Video,
*,
video_encoder: pxt.String | None = None,
video_encoder_args: pxt.Json | None = None
) -> pxt.Video
```
Convert a video to grayscale
**Requirements:**
* `ffmpeg` needs to be installed and in PATH
**Parameters:**
* **`video`** (`pxt.Video`): Input video.
* **`video_encoder`** (`pxt.String | None`): Video encoder to use. If not specified, uses the default encoder for the current platform.
* **`video_encoder_args`** (`pxt.Json | None`): Additional arguments to pass to the video encoder.
**Returns:**
* `pxt.Video`: A grayscale version of the video.
**Examples:**
```python theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
tbl.select(tbl.video.grayscale()).collect()
```
## udf mirror\_x()
```python Signature theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
@pxt.udf
mirror_x(
video: pxt.Video,
*,
video_encoder: pxt.String | None = None,
video_encoder_args: pxt.Json | None = None
) -> pxt.Video
```
Flip a video horizontally using ffmpeg's hflip filter.
**Requirements:**
* `ffmpeg` needs to be installed and in PATH
**Parameters:**
* **`video`** (`pxt.Video`): Input video.
* **`video_encoder`** (`pxt.String | None`): Video encoder to use. If not specified, uses the default encoder for the current platform.
* **`video_encoder_args`** (`pxt.Json | None`): Additional arguments to pass to the video encoder.
**Returns:**
* `pxt.Video`: A horizontally flipped video.
**Examples:**
```python theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
tbl.select(tbl.video.mirror_x()).collect()
```
## udf mirror\_y()
```python Signature theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
@pxt.udf
mirror_y(
video: pxt.Video,
*,
video_encoder: pxt.String | None = None,
video_encoder_args: pxt.Json | None = None
) -> pxt.Video
```
Flip a video vertically using ffmpeg's vflip filter.
**Requirements:**
* `ffmpeg` needs to be installed and in PATH
**Parameters:**
* **`video`** (`pxt.Video`): Input video.
* **`video_encoder`** (`pxt.String | None`): Video encoder to use. If not specified, uses the default encoder for the current platform.
* **`video_encoder_args`** (`pxt.Json | None`): Additional arguments to pass to the video encoder.
**Returns:**
* `pxt.Video`: A vertically flipped video.
**Examples:**
```python theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
tbl.select(tbl.video.mirror_y()).collect()
```
## udf mix\_audio()
```python Signature theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
@pxt.udf
mix_audio(
video: pxt.Video,
audio: pxt.Audio,
*,
audio_volume: pxt.Float = 1.0,
original_volume: pxt.Float = 1.0,
audio_start_time: pxt.Float = 0.0,
mix_duration: pxt.String = 'longest',
normalize: pxt.Bool = False,
dropout_transition: pxt.Float = 2.0,
align_to_video: pxt.String = 'trim',
video_encoder: pxt.String | None = None,
video_encoder_args: pxt.Json | None = None
) -> pxt.Video
```
Mix an audio track into a video's existing audio, blending both tracks together. Volume levels for each track can
be controlled independently.
**Requirements:**
* `ffmpeg` needs to be installed and in PATH
**Parameters:**
* **`video`** (`pxt.Video`): Input video (must have an existing audio stream).
* **`audio`** (`pxt.Audio`): Audio track to mix in.
* **`audio_volume`** (`pxt.Float`): Volume multiplier for the added audio track. 1.0 is original volume.
* **`original_volume`** (`pxt.Float`): Volume multiplier for the video's existing audio track. 1.0 is original volume.
* **`audio_start_time`** (`pxt.Float`): Time in seconds at which the added audio begins playing in the output.
* **`mix_duration`** (`pxt.String`): Controls which input determines the length of the mixed audio stream.
* `"longest"`: the mix runs until the longer of the two audio inputs ends. Use
this when the added audio (e.g. a music bed) is longer than the video's original audio.
* `"first"`: the mix ends when the video's original audio track ends. Useful when the
original audio and video streams are the same length.
* `"shortest"`: the mix ends when the shorter input ends, truncating whichever track is longer.
* **`normalize`** (`pxt.Bool`): If `True`, ffmpeg scales the mixed output to prevent clipping by dividing each track's
contribution by the number of inputs. Defaults to `False` so that `audio_volume` and
`original_volume` mean what they say; flip on if you are not setting volumes explicitly and
want automatic clip protection.
* **`dropout_transition`** (`pxt.Float`): Duration in seconds over which a track's contribution fades to zero after
it ends, preventing audible clicks at hard boundaries. Defaults to 2.0 seconds, matching
ffmpeg's own default. Set to 0.0 to disable. Relevant whenever one input ends before
the mixed output ends, regardless of which `mix_duration` mode is selected.
* **`align_to_video`** (`pxt.String`): Post-mix adjustment to align the output audio stream with the video stream duration.
Applied after `amix`, so it is independent of `mix_duration`.
* `"trim"`: if the mixed audio is longer than the video stream, truncate it to
match. Pairs naturally with `mix_duration="longest"` for music-bed workflows.
* `"none"`: no adjustment; output audio duration is whatever `amix` produces.
* `"pad"`: if the mixed audio is shorter than the video stream, extend it with silence. Also
trims to the video duration if the mix runs long.
* **`video_encoder`** (`pxt.String | None`): Video encoder to use. If not specified, uses the default encoder.
* **`video_encoder_args`** (`pxt.Json | None`): Additional arguments to pass to the video encoder.
**Returns:**
* `pxt.Video`: A new video with both audio tracks mixed together.
**Examples:**
Add background music at 30% volume. With the defaults, this lays the music as a bed under the
video: `mix_duration="longest"` keeps the music playing past the end of the original audio,
`align_to_video="trim"` caps the result at the video stream duration, and `normalize=False`
means the `audio_volume=0.3` setting is taken at face value rather than halved by `amix`:
```python theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
tbl.select(tbl.video.mix_audio(tbl.music, audio_volume=0.3)).collect()
```
Mix audio starting at second 5, with the original audio reduced:
```python theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
tbl.select(
tbl.video.mix_audio(
tbl.music,
audio_volume=0.5,
original_volume=0.7,
audio_start_time=5.0,
)
).collect()
```
Pad a short ambient track with silence so the audio stream matches the full video length:
```python theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
tbl.select(
tbl.video.mix_audio(
tbl.ambient, audio_volume=0.6, align_to_video='pad'
)
).collect()
```
## udf overlay\_image()
```python Signature theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
@pxt.udf
overlay_image(
video: pxt.Video,
image: pxt.Image,
*,
horizontal_align: pxt.String = 'center',
horizontal_margin: pxt.Int = 0,
vertical_align: pxt.String = 'center',
vertical_margin: pxt.Int = 0,
scale: pxt.Float | None = None,
opacity: pxt.Float = 1.0,
start_time: pxt.Float | None = None,
end_time: pxt.Float | None = None,
video_encoder: pxt.String | None = None,
video_encoder_args: pxt.Json | None = None
) -> pxt.Video
```
Overlay an image on a video with customizable positioning, scaling, opacity, and timing.
**Requirements:**
* `ffmpeg` needs to be installed and in PATH
**Parameters:**
* **`video`** (`pxt.Video`): Input video to overlay the image on.
* **`image`** (`pxt.Image`): Image to overlay.
* **`horizontal_align`** (`pxt.String`): Horizontal alignment of the overlay (`'left'`, `'center'`, `'right'`).
* **`horizontal_margin`** (`pxt.Int`): Horizontal margin in pixels from the alignment edge.
* **`vertical_align`** (`pxt.String`): Vertical alignment of the overlay (`'top'`, `'center'`, `'bottom'`).
* **`vertical_margin`** (`pxt.Int`): Vertical margin in pixels from the alignment edge.
* **`scale`** (`pxt.Float | None`): Scale factor for the overlay image relative to the video height. For example, 0.1 scales the
image to 10% of the video height while preserving aspect ratio. If None, uses the original size.
* **`opacity`** (`pxt.Float`): Overlay opacity from 0.0 (transparent) to 1.0 (opaque).
* **`start_time`** (`pxt.Float | None`): Time in seconds when the overlay appears. If None, the overlay is visible from the start.
* **`end_time`** (`pxt.Float | None`): Time in seconds when the overlay disappears. If None, the overlay is visible until the end.
* **`video_encoder`** (`pxt.String | None`): Video encoder to use. If not specified, uses the default encoder.
* **`video_encoder_args`** (`pxt.Json | None`): Additional arguments to pass to the video encoder.
**Returns:**
* `pxt.Video`: A new video with the image overlay applied.
**Examples:**
Add a logo to the top-right corner:
```python theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
tbl.select(
tbl.video.overlay_image(
tbl.logo_img, horizontal_align='right', vertical_align='top'
)
).collect()
```
Add a watermark at 50% opacity, scaled to 10% of video height:
```python theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
tbl.select(
tbl.video.overlay_image(tbl.watermark_img, scale=0.1, opacity=0.5)
).collect()
```
Show an image only between seconds 2 and 8:
```python theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
tbl.select(
tbl.video.overlay_image(
tbl.img, start_time=2.0, end_time=8.0, horizontal_align='right'
)
).collect()
```
## udf overlay\_text()
```python Signature theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
@pxt.udf
overlay_text(
video: pxt.Video,
text: pxt.String,
*,
font: pxt.String | None = None,
font_size: pxt.Int = 24,
line_spacing: pxt.Int = 0,
color: pxt.String = 'white',
opacity: pxt.Float = 1.0,
horizontal_align: pxt.String = 'center',
horizontal_margin: pxt.Int = 0,
vertical_align: pxt.String = 'center',
vertical_margin: pxt.Int = 0,
box: pxt.Bool = False,
box_color: pxt.String = 'black',
box_opacity: pxt.Float = 1.0,
box_border: pxt.Json[(Int, ...)] | None = None,
start_time: pxt.Float | None = None,
end_time: pxt.Float | None = None,
video_encoder: pxt.String | None = None,
video_encoder_args: pxt.Json | None = None
) -> pxt.Video
```
Overlay text on a video with customizable positioning and styling.
**Requirements:**
* `ffmpeg` needs to be installed and in PATH
Args:
video: Input video to overlay text on.
text: The text string to overlay on the video.
font: Font family or path to font file. If None, uses the system default.
font\_size: Size of the text in points.
line\_spacing: Pixels of vertical space added between lines of multi-line text (text containing
`
`). Defaults to 0. Negative values pull lines closer together for tighter packing.
Ignored for single-line text.
color: Text color (e.g., `'white'`, `'red'`, `'#FF0000'`).
opacity: Text opacity from 0.0 (transparent) to 1.0 (opaque).
horizontal\_align: Horizontal text alignment (`'left'`, `'center'`, `'right'`).
horizontal\_margin: Horizontal margin in pixels from the alignment edge.
vertical\_align: Vertical text alignment (`'top'`, `'center'`, `'bottom'`).
vertical\_margin: Vertical margin in pixels from the alignment edge.
box: Whether to draw a background box behind the text.
box\_color: Background box color as a string.
box\_opacity: Background box opacity from 0.0 to 1.0.
box\_border: Padding around text in the box in pixels.
* `[10]`: 10 pixels on all sides
* `[10, 20]`: 10 pixels on top/bottom, 20 on left/right
* `[10, 20, 30]`: 10 pixels on top, 20 on left/right, 30 on bottom
* `[10, 20, 30, 40]`: 10 pixels on top, 20 on right, 30 on bottom, 40 on left
start\_time: Time in seconds when the text appears. If None, the text is visible from the start.
end\_time: Time in seconds when the text disappears. If None, the text is visible until the end.
video\_encoder: Video encoder to use. If not specified, uses the default encoder.
video\_encoder\_args: Additional arguments to pass to the video encoder.
Returns:
A new video with the text overlay applied.
Examples:
Add a simple text overlay to videos in a table:
```python theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
tbl.select(tbl.video.overlay_text('Sample Text')).collect()
```
Add a YouTube-style caption:
```python theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
tbl.select(
tbl.video.overlay_text(
'Caption text',
font_size=32,
color='white',
opacity=1.0,
box=True,
box_color='black',
box_opacity=0.8,
box_border=[6, 14],
horizontal_margin=10,
vertical_align='bottom',
vertical_margin=70,
)
).collect()
```
Add text with a semi-transparent background box:
```python theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
tbl.select(
tbl.video.overlay_text(
'Important Message',
font_size=32,
color='yellow',
box=True,
box_color='black',
box_opacity=0.6,
box_border=[20, 10],
)
).collect()
```
## udf pan()
```python Signature theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
@pxt.udf
pan(*args, **kwargs) -> pxt.Video | None
```
A parameterized expression from which an executable Expr is created with a function call.
## udf resize()
```python Signature theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
@pxt.udf
resize(
video: pxt.Video,
*,
width: pxt.Int | None = None,
height: pxt.Int | None = None,
scale: pxt.Float | None = None,
video_encoder: pxt.String | None = None,
video_encoder_args: pxt.Json | None = None
) -> pxt.Video
```
Resize a video using ffmpeg's scale filter.
**Requirements:**
* `ffmpeg` needs to be installed and in PATH
**Parameters:**
* **`video`** (`pxt.Video`): Input video.
* **`width`** (`pxt.Int | None`): Width of the output video. Maintains the existing aspect ratio if no `height` is provided.
* **`height`** (`pxt.Int | None`): Height of the output video. Maintains the existing aspect ratio if no `width` is provided.
* **`scale`** (`pxt.Float | None`): Scale factor. Mutually exclusive with `width` and `height`.
* **`video_encoder`** (`pxt.String | None`): Video encoder to use. If not specified, uses the default encoder for the current platform.
* **`video_encoder_args`** (`pxt.Json | None`): Additional arguments to pass to the video encoder.
**Returns:**
* `pxt.Video`: The resized video.
**Examples:**
Resize to a specific width, preserving aspect ratio:
```python theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
tbl.select(tbl.video.resize(width=640)).collect()
```
Resize to exact dimensions:
```python theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
tbl.select(tbl.video.resize(width=1280, height=720)).collect()
```
Scale down by half:
```python theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
tbl.select(tbl.video.resize(scale=0.5)).collect()
```
## udf reverse()
```python Signature theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
@pxt.udf
reverse(
video: pxt.Video,
audio: pxt.String = 'drop',
*,
video_encoder: pxt.String | None = None,
video_encoder_args: pxt.Json | None = None
) -> pxt.Video
```
Reverse a video using ffmpeg's reverse filter.
**Requirements:**
* `ffmpeg` needs to be installed and in PATH
**Parameters:**
* **`video`** (`pxt.Video`): Input video.
* **`audio`** (`pxt.String`): Specifies what to do with audio streams
* `'drop'`: drop the audio streams
* `'reverse'`: also reverse the audio streams
* `'keep'`: keep the audio streams
* **`video_encoder`** (`pxt.String | None`): Video encoder to use. If not specified, uses the default encoder for the current platform.
* **`video_encoder_args`** (`pxt.Json | None`): Additional arguments to pass to the video encoder.
**Returns:**
* `pxt.Video`: The reversed video.
**Examples:**
Reverse a video, dropping audio:
```python theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
tbl.select(tbl.video.reverse()).collect()
```
Reverse a video along with its audio:
```python theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
tbl.select(tbl.video.reverse(audio='reverse')).collect()
```
## udf rotate()
```python Signature theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
@pxt.udf
rotate(
video: pxt.Video,
*,
angle: pxt.Float,
unit: pxt.String = 'deg',
expand: pxt.Bool = False,
video_encoder: pxt.String | None = None,
video_encoder_args: pxt.Json | None = None
) -> pxt.Video
```
Rotate a video by a fixed angle using ffmpeg's rotate filter.
**Requirements:**
* `ffmpeg` needs to be installed and in PATH
**Parameters:**
* **`video`** (`pxt.Video`): Input video.
* **`angle`** (`pxt.Float`): Rotation angle. Positive values rotate counter-clockwise.
* **`unit`** (`pxt.String`): Unit of the angle: `'deg'` for degrees or `'rad'` for radians.
* **`expand`** (`pxt.Bool`): If True, the output frame is enlarged to contain the entire rotated frame (no cropping).
If False (default), the output frame keeps the original dimensions, cropping corners.
* **`video_encoder`** (`pxt.String | None`): Video encoder to use. If not specified, uses the default encoder for the current platform.
* **`video_encoder_args`** (`pxt.Json | None`): Additional arguments to pass to the video encoder.
**Returns:**
* `pxt.Video`: A new video rotated by the specified angle.
**Examples:**
Rotate 90 degrees counter-clockwise:
```python theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
tbl.select(tbl.video.rotate(angle=90)).collect()
```
Rotate 45 degrees with frame expansion to avoid cropping:
```python theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
tbl.select(tbl.video.rotate(angle=45, expand=True)).collect()
```
Rotate by pi/2 radians:
```python theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
tbl.select(tbl.video.rotate(angle=1.5708, unit='rad')).collect()
```
## udf scene\_detect\_adaptive()
```python Signature theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
@pxt.udf
scene_detect_adaptive(
video: pxt.Video,
*,
fps: pxt.Float | None = None,
adaptive_threshold: pxt.Float = 3.0,
min_scene_len: pxt.Int = 15,
window_width: pxt.Int = 2,
min_content_val: pxt.Float = 15.0,
delta_hue: pxt.Float = 1.0,
delta_sat: pxt.Float = 1.0,
delta_lum: pxt.Float = 1.0,
delta_edges: pxt.Float = 0.0,
luma_only: pxt.Bool = False,
kernel_size: pxt.Int | None = None
) -> pxt.Json[(Json[{'start_time': Float, 'start_pts': Int, 'duration': Float}], ...)]
```
Detect scene cuts in a video using PySceneDetect's
[AdaptiveDetector](https://www.scenedetect.com/docs/latest/api/detectors.html#scenedetect.detectors.adaptive_detector.AdaptiveDetector).
**Requirements:**
* `pip install scenedetect`
**Parameters:**
* **`video`** (`pxt.Video`): The video to analyze for scene cuts.
* **`fps`** (`pxt.Float | None`): Number of frames to extract per second for analysis. If None or 0, analyzes all frames.
Lower values process faster but may miss exact scene cuts.
* **`adaptive_threshold`** (`pxt.Float`): Threshold that the score ratio must exceed to trigger a new scene cut.
Lower values will detect more scenes (more sensitive), higher values will detect fewer scenes.
* **`min_scene_len`** (`pxt.Int`): Once a cut is detected, this many frames must pass before a new one can be added to the scene
list.
* **`window_width`** (`pxt.Int`): Size of window (number of frames) before and after each frame to average together in order to
detect deviations from the mean. Must be at least 1.
* **`min_content_val`** (`pxt.Float`): Minimum threshold (float) that the content\_val must exceed in order to register as a new scene.
This is calculated the same way that `scene_detect_content()` calculates frame
score based on weights/luma\_only/kernel\_size.
* **`delta_hue`** (`pxt.Float`): Weight for hue component changes. Higher values make hue changes more important.
* **`delta_sat`** (`pxt.Float`): Weight for saturation component changes. Higher values make saturation changes more important.
* **`delta_lum`** (`pxt.Float`): Weight for luminance component changes. Higher values make brightness changes more important.
* **`delta_edges`** (`pxt.Float`): Weight for edge detection changes. Higher values make edge changes more important.
Edge detection can help detect cuts in scenes with similar colors but different content.
* **`luma_only`** (`pxt.Bool`): If True, only analyzes changes in the luminance (brightness) channel of the video,
ignoring color information. This can be faster and may work better for grayscale content.
* **`kernel_size`** (`pxt.Int | None`): Size of kernel to use for post edge detection filtering. If None, automatically set based on video
resolution.
**Returns:**
* `pxt.Json[(Json[{'start_time': Float, 'start_pts': Int, 'duration': Float}], ...)]`: A list of dictionaries, one for each detected scene, with the following keys:
* `start_time` (float): The start time of the scene in seconds.
* `start_pts` (int): The pts of the start of the scene.
* `duration` (float): The duration of the scene in seconds.
The list is ordered chronologically. Returns the full duration of the video if no scenes are detected.
**Examples:**
Detect scene cuts with default parameters:
```python theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
tbl.select(tbl.video.scene_detect_adaptive()).collect()
```
Detect more scenes by lowering the threshold:
```python theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
tbl.select(
tbl.video.scene_detect_adaptive(adaptive_threshold=1.5)
).collect()
```
Use luminance-only detection with a longer minimum scene length:
```python theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
tbl.select(
tbl.video.scene_detect_adaptive(luma_only=True, min_scene_len=30)
).collect()
```
Add scene cuts as a computed column:
```python theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
tbl.add_computed_column(
scene_cuts=tbl.video.scene_detect_adaptive(adaptive_threshold=2.0)
)
```
Analyze at a lower frame rate for faster processing:
```python theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
tbl.select(tbl.video.scene_detect_adaptive(fps=2.0)).collect()
```
## udf scene\_detect\_content()
```python Signature theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
@pxt.udf
scene_detect_content(
video: pxt.Video,
*,
fps: pxt.Float | None = None,
threshold: pxt.Float = 27.0,
min_scene_len: pxt.Int = 15,
delta_hue: pxt.Float = 1.0,
delta_sat: pxt.Float = 1.0,
delta_lum: pxt.Float = 1.0,
delta_edges: pxt.Float = 0.0,
luma_only: pxt.Bool = False,
kernel_size: pxt.Int | None = None,
filter_mode: pxt.String = 'merge'
) -> pxt.Json[(Json[{'start_time': Float, 'start_pts': Int, 'duration': Float}], ...)]
```
Detect scene cuts in a video using PySceneDetect's
[ContentDetector](https://www.scenedetect.com/docs/latest/api/detectors.html#scenedetect.detectors.content_detector.ContentDetector).
**Requirements:**
* `pip install scenedetect`
**Parameters:**
* **`video`** (`pxt.Video`): The video to analyze for scene cuts.
* **`fps`** (`pxt.Float | None`): Number of frames to extract per second for analysis. If None, analyzes all frames.
Lower values process faster but may miss exact scene cuts.
* **`threshold`** (`pxt.Float`): Threshold that the weighted sum of component changes must exceed to trigger a scene cut.
Lower values detect more scenes (more sensitive), higher values detect fewer scenes.
* **`min_scene_len`** (`pxt.Int`): Once a cut is detected, this many frames must pass before a new one can be added to the scene
list.
* **`delta_hue`** (`pxt.Float`): Weight for hue component changes. Higher values make hue changes more important.
* **`delta_sat`** (`pxt.Float`): Weight for saturation component changes. Higher values make saturation changes more important.
* **`delta_lum`** (`pxt.Float`): Weight for luminance component changes. Higher values make brightness changes more important.
* **`delta_edges`** (`pxt.Float`): Weight for edge detection changes. Higher values make edge changes more important.
Edge detection can help detect cuts in scenes with similar colors but different content.
* **`luma_only`** (`pxt.Bool`): If True, only analyzes changes in the luminance (brightness) channel,
ignoring color information. This can be faster and may work better for grayscale content.
* **`kernel_size`** (`pxt.Int | None`): Size of kernel for expanding detected edges. Must be odd integer greater than or equal to 3. If
None, automatically set using video resolution.
* **`filter_mode`** (`pxt.String`): How to handle fast cuts/flashes. 'merge' combines quick cuts, 'suppress' filters them out.
**Returns:**
* `pxt.Json[(Json[{'start_time': Float, 'start_pts': Int, 'duration': Float}], ...)]`: A list of dictionaries, one for each detected scene, with the following keys:
* `start_time` (float): The start time of the scene in seconds.
* `start_pts` (int): The pts of the start of the scene.
* `duration` (float): The duration of the scene in seconds.
The list is ordered chronologically. Returns the full duration of the video if no scenes are detected.
**Examples:**
Detect scene cuts with default parameters:
```python theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
tbl.select(tbl.video.scene_detect_content()).collect()
```
Detect more scenes by lowering the threshold:
```python theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
tbl.select(tbl.video.scene_detect_content(threshold=15.0)).collect()
```
Use luminance-only detection:
```python theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
tbl.select(tbl.video.scene_detect_content(luma_only=True)).collect()
```
Emphasize edge detection for scenes with similar colors:
```python theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
tbl.select(
tbl.video.scene_detect_content(
delta_edges=1.0, delta_hue=0.5, delta_sat=0.5
)
).collect()
```
Add scene cuts as a computed column:
```python theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
tbl.add_computed_column(
scene_cuts=tbl.video.scene_detect_content(threshold=20.0)
)
```
## udf scene\_detect\_hash()
```python Signature theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
@pxt.udf
scene_detect_hash(
video: pxt.Video,
*,
fps: pxt.Float | None = None,
threshold: pxt.Float = 0.395,
size: pxt.Int = 16,
lowpass: pxt.Int = 2,
min_scene_len: pxt.Int = 15
) -> pxt.Json[(Json[{'start_time': Float, 'start_pts': Int, 'duration': Float}], ...)]
```
Detect scene cuts in a video using PySceneDetect's
[HashDetector](https://www.scenedetect.com/docs/latest/api/detectors.html#scenedetect.detectors.hash_detector.HashDetector).
HashDetector uses perceptual hashing for very fast scene detection. It computes a hash of each
frame at reduced resolution and compares hash distances.
**Requirements:**
* `pip install scenedetect`
**Parameters:**
* **`video`** (`pxt.Video`): The video to analyze for scene cuts.
* **`fps`** (`pxt.Float | None`): Number of frames to extract per second for analysis. If None, analyzes all frames.
Lower values process faster but may miss exact scene cuts.
* **`threshold`** (`pxt.Float`): Value from 0.0 and 1.0 representing the relative hamming distance between the perceptual hashes of
adjacent frames. A distance of 0 means the image is the same, and 1 means no correlation. Smaller threshold
values thus require more correlation, making the detector more sensitive. The Hamming distance is divided
by size x size before comparing to threshold for normalization.
Lower values detect more scenes (more sensitive), higher values detect fewer scenes.
* **`size`** (`pxt.Int`): Size of square of low frequency data to use for the DCT. Larger values are more precise but slower.
Common values are 8, 16, or 32.
* **`lowpass`** (`pxt.Int`): How much high frequency information to filter from the DCT. A value of 2 means keep lower 1/2 of the
frequency data, 4 means only keep 1/4, etc. Larger values make the
detector less sensitive to high-frequency details and noise.
* **`min_scene_len`** (`pxt.Int`): Once a cut is detected, this many frames must pass before a new one can be added to the scene
list.
**Returns:**
* `pxt.Json[(Json[{'start_time': Float, 'start_pts': Int, 'duration': Float}], ...)]`: A list of dictionaries, one for each detected scene, with the following keys:
* `start_time` (float): The start time of the scene in seconds.
* `start_pts` (int): The pts of the start of the scene.
* `duration` (float): The duration of the scene in seconds.
The list is ordered chronologically. Returns the full duration of the video if no scenes are detected.
**Examples:**
Detect scene cuts with default parameters:
```python theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
tbl.select(tbl.video.scene_detect_hash()).collect()
```
Detect more scenes by lowering the threshold:
```python theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
tbl.select(tbl.video.scene_detect_hash(threshold=0.3)).collect()
```
Use larger hash size for more precision:
```python theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
tbl.select(tbl.video.scene_detect_hash(size=32)).collect()
```
Use for fast processing with lower frame rate:
```python theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
tbl.select(tbl.video.scene_detect_hash(fps=1.0, threshold=0.4)).collect()
```
Add scene cuts as a computed column:
```python theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
tbl.add_computed_column(scene_cuts=tbl.video.scene_detect_hash())
```
## udf scene\_detect\_histogram()
```python Signature theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
@pxt.udf
scene_detect_histogram(
video: pxt.Video,
*,
fps: pxt.Float | None = None,
threshold: pxt.Float = 0.05,
bins: pxt.Int = 256,
min_scene_len: pxt.Int = 15
) -> pxt.Json[(Json[{'start_time': Float, 'start_pts': Int, 'duration': Float}], ...)]
```
Detect scene cuts in a video using PySceneDetect's
[HistogramDetector](https://www.scenedetect.com/docs/latest/api/detectors.html#scenedetect.detectors.histogram_detector.HistogramDetector).
HistogramDetector compares frame histograms on the Y (luminance) channel after YUV conversion.
It detects scenes based on relative histogram differences and is more robust to gradual lighting
changes than content-based detection.
**Requirements:**
* `pip install scenedetect`
**Parameters:**
* **`video`** (`pxt.Video`): The video to analyze for scene cuts.
* **`fps`** (`pxt.Float | None`): Number of frames to extract per second for analysis. If None or 0, analyzes all frames.
Lower values process faster but may miss exact scene cuts.
* **`threshold`** (`pxt.Float`): Maximum relative difference between 0.0 and 1.0 that the histograms can differ. Histograms are
calculated on the Y channel after converting the frame to YUV, and normalized based on the number of bins.
Higher differences imply greater change in content, so larger threshold values are less sensitive to cuts.
Lower values detect more scenes (more sensitive), higher values detect fewer scenes.
* **`bins`** (`pxt.Int`): Number of bins to use for histogram calculation (typically 16-256). More bins provide
finer granularity but may be more sensitive to noise.
* **`min_scene_len`** (`pxt.Int`): Once a cut is detected, this many frames must pass before a new one can be added to the scene
list.
**Returns:**
* `pxt.Json[(Json[{'start_time': Float, 'start_pts': Int, 'duration': Float}], ...)]`: A list of dictionaries, one for each detected scene, with the following keys:
* `start_time` (float): The start time of the scene in seconds.
* `start_pts` (int): The pts of the start of the scene.
* `duration` (float): The duration of the scene in seconds.
The list is ordered chronologically. Returns the full duration of the video if no scenes are detected.
**Examples:**
Detect scene cuts with default parameters:
```python theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
tbl.select(tbl.video.scene_detect_histogram()).collect()
```
Detect more scenes by lowering the threshold:
```python theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
tbl.select(tbl.video.scene_detect_histogram(threshold=0.03)).collect()
```
Use fewer bins for faster processing:
```python theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
tbl.select(tbl.video.scene_detect_histogram(bins=64)).collect()
```
Use with a longer minimum scene length:
```python theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
tbl.select(tbl.video.scene_detect_histogram(min_scene_len=30)).collect()
```
Add scene cuts as a computed column:
```python theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
tbl.add_computed_column(
scene_cuts=tbl.video.scene_detect_histogram(threshold=0.04)
)
```
## udf scene\_detect\_threshold()
```python Signature theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
@pxt.udf
scene_detect_threshold(
video: pxt.Video,
*,
fps: pxt.Float | None = None,
threshold: pxt.Float = 12.0,
min_scene_len: pxt.Int = 15,
fade_bias: pxt.Float = 0.0,
add_final_scene: pxt.Bool = False,
method: pxt.String = 'floor'
) -> pxt.Json[(Json[{'start_time': Float, 'start_pts': Int, 'duration': Float}], ...)]
```
Detect fade-in and fade-out transitions in a video using PySceneDetect's
[ThresholdDetector](https://www.scenedetect.com/docs/latest/api/detectors.html#scenedetect.detectors.threshold_detector.ThresholdDetector).
ThresholdDetector identifies scenes by detecting when pixel brightness falls below or rises above
a threshold value, suitable for detecting fade-to-black, fade-to-white, and similar transitions.
**Requirements:**
* `pip install scenedetect`
**Parameters:**
* **`video`** (`pxt.Video`): The video to analyze for fade transitions.
* **`fps`** (`pxt.Float | None`): Number of frames to extract per second for analysis. If None or 0, analyzes all frames.
Lower values process faster but may miss exact transition points.
* **`threshold`** (`pxt.Float`): 8-bit intensity value that each pixel value (R, G, and B) must be less than or equal to in order
to trigger a fade in/out.
* **`min_scene_len`** (`pxt.Int`): Once a cut is detected, this many frames must pass before a new one can be added to the scene
list.
* **`fade_bias`** (`pxt.Float`): Float between -1.0 and +1.0 representing the percentage of timecode skew for the start of a scene
(-1.0 causing a cut at the fade-to-black, 0.0 in the middle, and +1.0 causing the cut to be right at the
position where the threshold is passed).
* **`add_final_scene`** (`pxt.Bool`): Boolean indicating if the video ends on a fade-out to generate an additional scene at this
timecode.
* **`method`** (`pxt.String`): How to treat threshold when detecting fade events
* 'ceiling': Fade out happens when frame brightness rises above threshold.
* 'floor': Fade out happens when frame brightness falls below threshold.
**Returns:**
* `pxt.Json[(Json[{'start_time': Float, 'start_pts': Int, 'duration': Float}], ...)]`: A list of dictionaries, one for each detected scene, with the following keys:
* `start_time` (float): The start time of the scene in seconds.
* `start_pts` (int): The pts of the start of the scene.
* `duration` (float): The duration of the scene in seconds.
The list is ordered chronologically. Returns the full duration of the video if no scenes are detected.
**Examples:**
Detect fade-to-black transitions with default parameters:
```python theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
tbl.select(tbl.video.scene_detect_threshold()).collect()
```
Use a lower threshold to detect darker fades:
```python theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
tbl.select(tbl.video.scene_detect_threshold(threshold=8.0)).collect()
```
Detect fade-to-white transitions using the 'ceiling' method:
```python theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
tbl.select(tbl.video.scene_detect_threshold(method='ceiling')).collect()
```
Add final scene boundary:
```python theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
tbl.select(
tbl.video.scene_detect_threshold(add_final_scene=True)
).collect()
```
Add fade transitions as a computed column:
```python theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
tbl.add_computed_column(
fade_cuts=tbl.video.scene_detect_threshold(threshold=15.0)
)
```
## udf scroll()
```python Signature theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
@pxt.udf
scroll(
video: pxt.Video,
*,
w: pxt.Int | None = None,
h: pxt.Int | None = None,
x_speed: pxt.Float = 0,
y_speed: pxt.Float = 0,
x_start: pxt.Int = 0,
y_start: pxt.Int = 0,
video_encoder: pxt.String | None = None,
video_encoder_args: pxt.Json | None = None
) -> pxt.Video
```
Apply a scrolling viewport effect to a video using ffmpeg's crop filter.
Extracts a viewport of size `w` x `h` from each frame, starting at position (`x_start`, `y_start`) and moving
at (`x_speed`, `y_speed`) pixels per second. The viewport clamps at the frame edges: once it reaches a boundary,
it stops moving and the remaining frames show a static crop.
At least one of `w` or `h` must be smaller than the input dimensions for the effect to be visible.
The clip duration is unchanged. To pan across the full available range, set
`x_speed = (input_width - w) / duration`.
**Requirements:**
* `ffmpeg` needs to be installed and in PATH
**Parameters:**
* **`video`** (`pxt.Video`): Input video.
* **`w`** (`pxt.Int | None`): Width of the output viewport in pixels. If None, uses the input width.
* **`h`** (`pxt.Int | None`): Height of the output viewport in pixels. If None, uses the input height.
* **`x_speed`** (`pxt.Float`): Horizontal scroll speed in pixels per second. Positive values scroll rightward (the viewport moves
right, revealing content to the right). Negative values scroll leftward.
* **`y_speed`** (`pxt.Float`): Vertical scroll speed in pixels per second. Positive values scroll downward. Negative values scroll
upward.
* **`x_start`** (`pxt.Int`): Initial horizontal offset of the viewport in pixels.
* **`y_start`** (`pxt.Int`): Initial vertical offset of the viewport in pixels.
* **`video_encoder`** (`pxt.String | None`): Video encoder to use. If not specified, uses the default encoder for the current platform.
* **`video_encoder_args`** (`pxt.Json | None`): Additional arguments to pass to the video encoder.
**Returns:**
* `pxt.Video`: A new video with the scrolling effect applied. Output dimensions are `w` x `h`.
**Examples:**
Pan rightward across a 1920x1080 video using a 1280-pixel-wide viewport, scrolling at 50 px/s:
```python theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
tbl.select(tbl.video.scroll(w=1280, x_speed=50)).collect()
```
Pan rightward across the full range of a 1920x1080 video in exactly its duration. The viewport is
1280 px wide, so the pan range is 1920 - 1280 = 640 px. For a 10-second video, set
`x_speed = 640 / 10 = 64`:
```python theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
tbl.select(tbl.video.scroll(w=1280, x_speed=64)).collect()
```
Pan leftward across a 1920x1080 video, starting from the right edge:
```python theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
tbl.select(tbl.video.scroll(w=1280, x_start=640, x_speed=-64)).collect()
```
## udf segment\_video()
```python Signature theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
@pxt.udf
segment_video(
video: pxt.Video,
*,
duration: pxt.Float | None = None,
segment_times: pxt.Json[(Float, ...)] | None = None,
mode: pxt.String = 'accurate',
video_encoder: pxt.String | None = None,
video_encoder_args: pxt.Json | None = None
) -> pxt.Json[(Json[{'segment_start': Float | None, 'segment_start_pts': Int | None, 'segment_end': Float | None, 'segment_end_pts': Int | None, 'video_segment': Video}], ...)]
```
Split a video into segments.
**Requirements:**
* `ffmpeg` needs to be installed and in PATH
**Parameters:**
* **`video`** (`pxt.Video`): Input video file to segment
* **`duration`** (`pxt.Float | None`): Duration of each segment (in seconds). For `mode='fast'`, this is approximate;
for `mode='accurate'`, segments will have exact durations. Cannot be specified together with
`segment_times`.
* **`segment_times`** (`pxt.Json[(Float`): List of timestamps (in seconds) in video where segments should be split. Note that these are not
segment durations. If all segment times are less than the duration of the video, produces exactly
`len(segment_times) + 1` segments. Cannot be empty or be specified together with `duration`.
* **`mode`** (`Any`): Segmentation mode:
* `'fast'`: Quick segmentation using stream copy (splits only at keyframes, approximate durations)
* `'accurate'`: Precise segmentation with re-encoding (exact durations, slower)
* **`video_encoder`** (`Any`): Video encoder to use. If not specified, uses the default encoder for the current platform.
Only available for `mode='accurate'`.
* **`video_encoder_args`** (`Any`): Additional arguments to pass to the video encoder. Only available for `mode='accurate'`.
**Returns:**
* `pxt.Json[(Json[{'segment_start': Float | None, 'segment_start_pts': Int | None, 'segment_end': Float | None, 'segment_end_pts': Int | None, 'video_segment': Video}], ...)]`: A list of [`VideoSegment`](./videosegment) records, one per generated segment.
**Examples:**
Split a video at 1 minute intervals using fast mode:
```python theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
tbl.select(
segments=tbl.video.segment_video(duration=60, mode='fast')
).collect()
```
Split video into exact 10-second segments with default accurate mode, using the libx264 encoder with a CRF of 23
and slow preset (for smaller output files):
```python theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
tbl.select(
segments=tbl.video.segment_video(
duration=10,
video_encoder='libx264',
video_encoder_args={'crf': 23, 'preset': 'slow'},
)
).collect()
```
Split video into two parts at the midpoint:
```python theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
duration = tbl.video.get_duration()
tbl.select(
segments=tbl.video.segment_video(segment_times=[duration / 2])
).collect()
```
## udf speed()
```python Signature theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
@pxt.udf
speed(
video: pxt.Video,
*,
factor: pxt.Float,
video_encoder: pxt.String | None = None,
video_encoder_args: pxt.Json | None = None
) -> pxt.Video
```
Change the playback speed of a video using ffmpeg's setpts filter.
A factor of 2.0 doubles the speed (halves the duration); a factor of 0.5 halves the speed (doubles the duration).
Audio pitch is preserved using ffmpeg's `atempo` filter.
**Requirements:**
* `ffmpeg` needs to be installed and in PATH
**Parameters:**
* **`video`** (`pxt.Video`): Input video.
* **`factor`** (`pxt.Float`): Speed multiplier. Must be positive. Values > 1.0 speed up, values \< 1.0 slow down.
* **`video_encoder`** (`pxt.String | None`): Video encoder to use. If not specified, uses the default encoder for the current platform.
* **`video_encoder_args`** (`pxt.Json | None`): Additional arguments to pass to the video encoder.
**Returns:**
* `pxt.Video`: A new video with the adjusted playback speed.
**Examples:**
Double the speed:
```python theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
tbl.select(tbl.video.speed(factor=2.0)).collect()
```
Half speed (slow motion):
```python theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
tbl.select(tbl.video.speed(factor=0.5)).collect()
```
## udf with\_audio()
```python Signature theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
@pxt.udf
with_audio(
video: pxt.Video,
audio: pxt.Audio,
*,
video_start_time: pxt.Float = 0.0,
video_duration: pxt.Float | None = None,
audio_start_time: pxt.Float = 0.0,
audio_duration: pxt.Float | None = None
) -> pxt.Video
```
Creates a new video that combines the video stream from `video` and the audio stream from `audio`.
The `start_time` and `duration` parameters can be used to select a specific time range from each input.
If the audio input (or selected time range) is longer than the video, the audio will be truncated.
**Requirements:**
* `ffmpeg` needs to be installed and in PATH
**Parameters:**
* **`video`** (`pxt.Video`): Input video.
* **`audio`** (`pxt.Audio`): Input audio.
* **`video_start_time`** (`pxt.Float`): Start time in the video input (in seconds).
* **`video_duration`** (`pxt.Float | None`): Duration of video segment (in seconds). If None, uses the remainder of the video after
`video_start_time`. `video_duration` determines the duration of the output video.
* **`audio_start_time`** (`pxt.Float`): Start time in the audio input (in seconds).
* **`audio_duration`** (`pxt.Float | None`): Duration of audio segment (in seconds). If None, uses the remainder of the audio after
`audio_start_time`. If the audio is longer than the output video, it will be truncated.
**Returns:**
* `pxt.Video`: A new video file with the audio track added.
**Examples:**
Add background music to a video:
```python theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
tbl.select(tbl.video.with_audio(tbl.music_track)).collect()
```
Add audio starting 5 seconds into both files:
```python theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
tbl.select(
tbl.video.with_audio(
tbl.music_track, video_start_time=5.0, audio_start_time=5.0
)
).collect()
```
Use a 10-second clip from the middle of both files:
```python theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
tbl.select(
tbl.video.with_audio(
tbl.music_track,
video_start_time=30.0,
video_duration=10.0,
audio_start_time=15.0,
audio_duration=10.0,
)
).collect()
```
## udf zoom()
```python Signature theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
@pxt.udf
zoom(
video: pxt.Video,
*,
start_scale: pxt.Float = 1.0,
end_scale: pxt.Float = 1.3,
center: pxt.Json[(Float, ...)] | None = None,
video_encoder: pxt.String | None = None,
video_encoder_args: pxt.Json | None = None
) -> pxt.Video
```
Apply a smooth zoom effect over the duration of a video using ffmpeg's zoompan filter.
The zoom factor interpolates linearly from `start_scale` to `end_scale`. The effect works by computing a crop
region at each frame (centered on `center`) and scaling it back to the original resolution. Output dimensions
match the input.
* `start_scale < end_scale`: zoom in (frame progressively tightens)
* `start_scale > end_scale`: zoom out (frame progressively widens)
* `start_scale == end_scale`: static zoom (constant crop, no animation)
**Requirements:**
* `ffmpeg` needs to be installed and in PATH
**Parameters:**
* **`video`** (`pxt.Video`): Input video.
* **`start_scale`** (`pxt.Float`): Zoom factor at the start of the video. Must be >= 1.0.
* **`end_scale`** (`pxt.Float`): Zoom factor at the end of the video. Must be >= 1.0.
* **`center`** (`pxt.Json[(Float`): Zoom center as `[x, y]` in normalized coordinates (0.0 to 1.0), where `[0.5, 0.5]` is the frame
center. If None, defaults to `[0.5, 0.5]`.
* **`video_encoder`** (`Any`): Video encoder to use. If not specified, uses the default encoder for the current platform.
* **`video_encoder_args`** (`Any`): Additional arguments to pass to the video encoder.
**Returns:**
* `pxt.Video`: A new video with the zoom effect applied. Output resolution matches the input.
**Examples:**
Zoom in (default, 1.0x to 1.3x centered):
```python theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
tbl.select(tbl.video.zoom()).collect()
```
Zoom out from 2x to 1x:
```python theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
tbl.select(tbl.video.zoom(start_scale=2.0, end_scale=1.0)).collect()
```
Zoom in toward the upper-left quadrant:
```python theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
tbl.select(tbl.video.zoom(end_scale=1.5, center=[0.25, 0.25])).collect()
```
Static 1.5x zoom (no animation):
```python theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
tbl.select(tbl.video.zoom(start_scale=1.5, end_scale=1.5)).collect()
```
# VideoSegment
Source: https://docs.pixeltable.com/sdk/latest/videosegment
# class pixeltable.functions.video.VideoSegment
One segment of a video: the segment's video plus where it falls in the source video.
## attr segment\_end
```
segment_end: float | None
```
Segment end, in seconds from the start of the source video; None if the source stream does not report it.
## attr segment\_end\_pts
```
segment_end_pts: int | None
```
Segment end, as a presentation timestamp (in the source stream's time base); None if unavailable.
## attr segment\_start
```
segment_start: float | None
```
Segment start, in seconds from the start of the source video; None if the source stream does not report it.
## attr segment\_start\_pts
```
segment_start_pts: int | None
```
Segment start, as a presentation timestamp (in the source stream's time base); None if unavailable.
## attr video\_segment
```
video_segment: pxt.Video
```
The segment's video.
# vision
Source: https://docs.pixeltable.com/sdk/latest/vision
# module pixeltable.functions.vision
Pixeltable UDFs for Computer Vision.
Example:
```python theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
import pixeltable as pxt
from pixeltable.functions import vision as pxtv
t = pxt.get_table(...)
t.select(
pxtv.bboxes_draw(t.img, boxes=t.boxes, labels=t.labels)
).collect()
```
## udf bboxes\_clip\_to\_canvas()
```python Signature theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
@pxt.udf
bboxes_clip_to_canvas(
bboxes: pxt.Json[(Json, ...)],
format: pxt.String,
*,
width: pxt.Int | None = None,
height: pxt.Int | None = None,
min_visibility: pxt.Float = 0.0,
min_area: pxt.Float = 0.0
) -> pxt.Json[(Json, ...)]
```
Clip a list of bounding boxes to a canvas of specified size.
**Parameters:**
* **`bboxes`** (`pxt.Json[(Json`): List of bounding boxes, specified with absolute pixel coordinates if `width`/`height` are given,
with relative coordinates otherwise. All-integer boxes are always treated as absolute and require
`width`/`height`.
* **`format`** (`Any`): Format of the bounding box coordinates, one of 'xyxy', 'xywh', 'cxcywh'.
* **`width`** (`Any`): Canvas width in absolute pixels. Required for absolute coordinates, omit for relative.
* **`height`** (`Any`): Canvas height in absolute pixels. Required for absolute coordinates, omit for relative.
* **`min_visibility`** (`Any`): Minimum fraction of the bounding box that must be visible after clipping. If the visibility
is less than this value, returns None.
* **`min_area`** (`Any`): Minimum area of the bounding box after clipping. If the area is less than this value, returns None.
**Returns:**
* `pxt.Json[(Json, ...)]`: List of clipped bounding boxes in the same format as the input. Boxes that don't meet the
min\_visibility or min\_area thresholds are replaced with None.
## udf bboxes\_convert()
```python Signature theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
@pxt.udf
bboxes_convert(
bboxes: pxt.Json[(Json, ...)],
*,
src_format: pxt.String,
dst_format: pxt.String
) -> pxt.Json[(Json, ...)]
```
Convert a list of bounding boxes from src\_format to dst\_format.
**Parameters:**
* **`bboxes`** (`pxt.Json[(Json`): List of bounding boxes, each with absolute pixel coordinates or relative coordinates in \[0, 1].
Coordinates may be int or float; relative vs absolute is inferred (relative only when every coordinate
is within \[0, 1]), so absolute float coordinates such as 10.5 are supported.
* **`src_format`** (`Any`): Source format, one of 'xyxy', 'xywh', 'cxcywh'.
* **`dst_format`** (`Any`): Destination format, one of 'xyxy', 'xywh', 'cxcywh'.
**Returns:**
* `pxt.Json[(Json, ...)]`: List of bounding boxes in dst\_format.
## udf bboxes\_crop\_canvas()
```python Signature theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
@pxt.udf
bboxes_crop_canvas(
bboxes: pxt.Json[(Json, ...)],
format: pxt.String,
*,
canvas_region: pxt.Json[(Json, ...)],
canvas_region_format: pxt.String,
canvas_width: pxt.Int | None = None,
canvas_height: pxt.Int | None = None
) -> pxt.Json[(Json, ...)]
```
Adjust a list of bounding boxes to account for a canvas crop.
**Parameters:**
* **`bboxes`** (`pxt.Json[(Json`): List of bounding boxes, specified with absolute pixel coordinates if `canvas_width`/`canvas_height`
are given, with relative coordinates otherwise. All-integer boxes are always treated as absolute and
require `canvas_width`/`canvas_height`.
* **`format`** (`Any`): Format of the bounding box coordinates, one of 'xyxy', 'xywh', 'cxcywh'.
* **`canvas_width`** (`Any`): Canvas width in absolute pixels. Required for absolute coordinates, omit for relative.
* **`canvas_height`** (`Any`): Canvas height in absolute pixels. Required for absolute coordinates, omit for relative.
* **`canvas_region`** (`Any`): Canvas region that was cropped, either specified with absolute pixel coordinates or relative
coordinates, in the format specified by `canvas_region_format`.
* **`canvas_region_format`** (`Any`): Format of the `canvas_region` coordinates, one of 'xyxy', 'xywh', 'cxcywh'.
**Returns:**
* `pxt.Json[(Json, ...)]`: List of adjusted bounding boxes in the same format as the input. They can extend beyond the canvas boundaries.
## udf bboxes\_draw()
```python Signature theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
@pxt.udf
bboxes_draw(
img: pxt.Image,
boxes: pxt.Json[(Json[(Int, ...)], ...)],
*,
labels: pxt.Json[(Json, ...)] | None = None,
color: pxt.String | None = None,
box_colors: pxt.Json[(String, ...)] | None = None,
alpha: pxt.Float | None = None,
fill: pxt.Bool = False,
fill_alpha: pxt.Float | None = None,
width: pxt.Int = 1,
font: pxt.String | None = None,
font_size: pxt.Int | None = None
) -> pxt.Image
```
Draws bounding boxes on the given image.
Labels can be any type that supports `str()` and is hashable (e.g., strings, ints, etc.).
Colors can be specified as common HTML color names (e.g., 'red') supported by PIL's
[`ImageColor`](https://pillow.readthedocs.io/en/stable/reference/ImageColor.html#imagecolor-module) module or as
RGB/RGBA hex codes (e.g., '#FF0000', '#FF0000FF'). If opacity isn't specified in the color string and
`alpha`/`fill_alpha` is `None`, defaults to 1.0 for box borders and 0.5 for filled boxes.
If no colors are specified, this function randomly assigns each label a specific color based on a hash of the label.
**Parameters:**
* **`img`** (`pxt.Image`): The image on which to draw the bounding boxes.
* **`boxes`** (`pxt.Json[(Json[(Int`): List of bounding boxes, each represented as \[xmin, ymin, xmax, ymax].
* **`labels`** (`Any`): List of labels for each bounding box.
* **`color`** (`Any`): Single color to be used for all bounding boxes and labels.
* **`box_colors`** (`Any`): List of colors, one per bounding box.
* **`alpha`** (`Any`): Opacity (0-1) of the bounding box borders and labels. If non-`None`, overrides any alpha in
`color`/`box_colors`.
* **`fill`** (`Any`): Whether to fill the bounding boxes with color.
* **`fill_alpha`** (`Any`): Opacity (0-1) of the bounding box fill. If non-`None`, overrides any alpha in
`color`/`box_colors`.
* **`width`** (`Any`): Width of the bounding box borders.
* **`font`** (`Any`): Name of a system font or path to a TrueType font file, as required by
[`PIL.ImageFont.truetype()`](https://pillow.readthedocs.io/en/stable/reference/ImageFont.html#PIL.ImageFont.truetype).
If `None`, uses the default provided by
[`PIL.ImageFont.load_default()`](https://pillow.readthedocs.io/en/stable/reference/ImageFont.html#PIL.ImageFont.load_default).
* **`font_size`** (`Any`): Size of the font used for labels in points. Only used in conjunction with non-`None` `font` argument.
**Returns:**
* `pxt.Image`: The image with bounding boxes drawn on it.
## udf bboxes\_pad()
```python Signature theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
@pxt.udf
bboxes_pad(
bboxes: pxt.Json[(Json, ...)],
format: pxt.String,
*,
top: pxt.Int | None = None,
bottom: pxt.Int | None = None,
left: pxt.Int | None = None,
right: pxt.Int | None = None,
x: pxt.Int | None = None,
y: pxt.Int | None = None
) -> pxt.Json[(Json, ...)]
```
Pad a list of bounding boxes.
**Parameters:**
* **`bboxes`** (`pxt.Json[(Json`): List of bounding boxes in absolute pixel coordinates.
* **`format`** (`Any`): Format of the bounding box coordinates, one of 'xyxy', 'xywh', 'cxcywh'.
* **`top`** (`Any`): Amount to pad at the top, in absolute pixels.
* **`bottom`** (`Any`): Amount to pad at the bottom, in absolute pixels.
* **`left`** (`Any`): Amount to pad at the left, in absolute pixels.
* **`right`** (`Any`): Amount to pad at the right, in absolute pixels.
* **`x`** (`Any`): Amount to pad at the left and right, in absolute pixels.
* **`y`** (`Any`): Amount to pad at the top and bottom, in absolute pixels.
**Returns:**
* `pxt.Json[(Json, ...)]`: List of padded bounding boxes in the same format as the input.
## udf bboxes\_resize()
```python Signatures theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
# Signature 1:
@pxt.udf
bboxes_resize(
bboxes: pxt.Json[(Json, ...)],
format: pxt.String,
*,
width: pxt.Int | None,
height: pxt.Int | None,
aspect: pxt.String | None,
aspect_mode: pxt.String | None
) -> pxt.Json[(Json, ...)]
# Signature 2:
@pxt.udf
bboxes_resize(
bboxes: pxt.Json[(Json, ...)],
format: pxt.String,
*,
width: pxt.Float | None,
height: pxt.Float | None,
aspect: pxt.Float | None,
aspect_mode: pxt.String | None
) -> pxt.Json[(Json, ...)]
```
Resize a list of bounding boxes (center-anchored):
* to a specified width or height (the other dimension is scaled to maintain the aspect ratio)
* to a specified aspect ratio
Only one of `width`, `height`, or `aspect` can be specified.
**Parameters:**
* **`bboxes`** (`Json[(Json, ...)]`): List of bounding boxes, each with absolute pixel coordinates or relative coordinates in \[0, 1].
Coordinates may be int or float; relative vs absolute is inferred (relative only when every coordinate
is within \[0, 1]), so absolute float coordinates such as 10.5 are supported.
* **`format`** (`String`): Format of the bounding box coordinates, one of 'xyxy', 'xywh', 'cxcywh'.
* **`width`** (`Int | None`, default: `Literal(None)`): Target width. Pass an `int` for absolute pixels or a `float` for relative coordinates.
* **`height`** (`Int | None`, default: `Literal(None)`): Target height. Pass an `int` for absolute pixels or a `float` for relative coordinates.
* **`aspect`** (`String | None`, default: `Literal(None)`): Target aspect ratio. Pass a `str` like '16:9' or a `float` like 1.78.
* **`aspect`** (`String | None`, default: `Literal(None)`): Target aspect ratio as a string 'W:H' (e.g., '16:9') or a `float`. Resizes either the width
or height to match the specified aspect ratio, maintaining the other dimension. Requires `aspect_mode`.
* **`aspect_mode`** (`String | None`, default: `Literal(None)`): Either 'crop' or 'pad'. Required when `aspect` is specified. If `crop`, reduces the oversized
dimension to match the aspect ratio. If `pad`, extends the undersized dimension to match the aspect ratio.
**Returns:**
* `pxt.Json[(Json, ...)]`: List of resized bounding boxes in the same format as the input.
## udf bboxes\_resize\_canvas()
```python Signature theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
@pxt.udf
bboxes_resize_canvas(
bboxes: pxt.Json[(Json, ...)],
format: pxt.String,
*,
canvas_width: pxt.Int | None = None,
canvas_height: pxt.Int | None = None,
new_canvas_width: pxt.Int | None = None,
new_canvas_height: pxt.Int | None = None,
canvas_scale: pxt.Float | None = None,
canvas_scale_x: pxt.Float | None = None,
canvas_scale_y: pxt.Float | None = None
) -> pxt.Json[(Json, ...)]
```
Adjust a list of bounding boxes to account for a canvas resize. The resize operation can be expressed
* as absolute pixel dimensions (requires canvas\_width, canvas\_height, new\_canvas\_width, new\_canvas\_height)
* as relative dimensions (requires at least one of canvas\_scale, canvas\_scale\_x, canvas\_scale\_y)
**Parameters:**
* **`bboxes`** (`pxt.Json[(Json`): List of bounding boxes in absolute pixel coordinates.
* **`format`** (`Any`): Format of the bounding box coordinates, one of 'xyxy', 'xywh', 'cxcywh'.
* **`canvas_width`** (`Any`): Original canvas width in absolute pixels.
* **`canvas_height`** (`Any`): Original canvas height in absolute pixels.
* **`new_canvas_width`** (`Any`): New canvas width in absolute pixels. Requires canvas\_width/canvas\_height to be specified.
* **`new_canvas_height`** (`Any`): New canvas height in absolute pixels. Requires canvas\_width/canvas\_height to be specified.
* **`canvas_scale`** (`Any`): Scale factor to apply to both canvas dimensions.
* **`canvas_scale_x`** (`Any`): Scale factor to apply to the canvas width.
* **`canvas_scale_y`** (`Any`): Scale factor to apply to the canvas height.
**Returns:**
* `pxt.Json[(Json, ...)]`: List of adjusted bounding boxes in the same format as the input.
## udf bboxes\_scale()
```python Signature theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
@pxt.udf
bboxes_scale(
bboxes: pxt.Json[(Json, ...)],
format: pxt.String,
*,
factor: pxt.Float | None = None,
x_factor: pxt.Float | None = None,
y_factor: pxt.Float | None = None
) -> pxt.Json[(Json, ...)]
```
Re-scale a list of bounding boxes (center-anchored).
**Parameters:**
* **`bboxes`** (`pxt.Json[(Json`): List of bounding boxes, each with absolute pixel coordinates or relative coordinates in \[0, 1].
Coordinates may be int or float; relative vs absolute is inferred (relative only when every coordinate
is within \[0, 1]), so absolute float coordinates such as 10.5 are supported.
* **`format`** (`Any`): Format of the bounding box coordinates, one of 'xyxy', 'xywh', 'cxcywh'.
* **`factor`** (`Any`): Scale factor to apply to both box dimensions.
* **`x_factor`** (`Any`): Scale factor to apply to the box width.
* **`y_factor`** (`Any`): Scale factor to apply to the box height.
**Returns:**
* `pxt.Json[(Json, ...)]`: List of scaled bounding boxes in the same format as the input.
## udf eval\_detections()
```python Signature theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
@pxt.udf
eval_detections(
pred_bboxes: pxt.Json[(Json[(Int, ...)], ...)],
pred_labels: pxt.Json[(Int, ...)],
pred_scores: pxt.Json[(Float, ...)],
gt_bboxes: pxt.Json[(Json[(Int, ...)], ...)],
gt_labels: pxt.Json[(Int, ...)],
min_iou: pxt.Float = 0.5
) -> pxt.Json[(Json[{'min_iou': Float, 'category': Int, 'tp': Json[(Int, ...)], 'fp': Json[(Int, ...)], 'scores': Json[(Float, ...)], 'num_gts': Int}], ...)]
```
Evaluates the performance of a set of predicted bounding boxes against a set of ground truth bounding boxes.
**Parameters:**
* **`pred_bboxes`** (`pxt.Json[(Json[(Int`): List of predicted bounding boxes, each represented as \[xmin, ymin, xmax, ymax].
* **`pred_labels`** (`Any`): List of predicted labels.
* **`pred_scores`** (`Any`): List of predicted scores.
* **`gt_bboxes`** (`Any`): List of ground truth bounding boxes, each represented as \[xmin, ymin, xmax, ymax].
* **`gt_labels`** (`Any`): List of ground truth labels.
* **`min_iou`** (`Any`): Minimum intersection-over-union (IoU) threshold for a predicted bounding box to be
considered a true positive.
**Returns:**
* `pxt.Json[(Json[{'min_iou': Float, 'category': Int, 'tp': Json[(Int, ...)], 'fp': Json[(Int, ...)], 'scores': Json[(Float, ...)], 'num_gts': Int}], ...)]`: A list of dictionaries, one per label class, with the following entries:
* `min_iou` (`float`): the value of `min_iou` used for the detections
* `category` (`int`): the label class
* `tp` (`list[int]`): 1 or 0 for each predicted bounding box of this class, indicating a true positive
* `fp` (`list[int]`): 1 or 0 for each predicted bounding box of this class; `fp[n] == 1 - tp[n]`
* `scores` (`list[float]`): the predicted scores for each bounding box of this class
* `num_gts` (`int`): the number of ground truth bounding boxes of this class
## udf overlay\_segmentation()
```python Signatures theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
# Signature 1:
@pxt.udf
overlay_segmentation(
img: pxt.Image,
segmentation: pxt.Array[(None, None), int32],
*,
alpha: pxt.Float,
background: pxt.Int,
segment_colors: pxt.Json[(String, ...)] | None,
draw_contours: pxt.Bool,
contour_thickness: pxt.Int
) -> pxt.Image
# Signature 2:
@pxt.udf
overlay_segmentation(
img: pxt.Image,
segmentation: pxt.Array[(None, None, None), bool],
*,
ids: pxt.Array[(None,), int64] | None,
alpha: pxt.Float,
segment_colors: pxt.Json[(String, ...)] | None,
draw_contours: pxt.Bool,
contour_thickness: pxt.Int
) -> pxt.Image
```
Overlays a colored segmentation map on an image.
Colors can be specified as common HTML color names (e.g., 'red') supported by PIL's
[`ImageColor`](https://pillow.readthedocs.io/en/stable/reference/ImageColor.html#imagecolor-module) module or as
RGB hex codes (e.g., '#FF0000').
If no colors are specified, this function randomly assigns each segment a specific color based on a hash of its id.
Two input formats are supported:
* A 2D `(H, W)` int32 id-map where each pixel value is a segment id. This matches the panoptic
output of [`detr_for_segmentation()`](./huggingface#func-detr_for_segmentation).
* A 3D `(num_instances, H, W)` boolean stack with one binary mask per instance. This matches the
per-instance mask output of [`sam_for_segmentation()`](./huggingface#func-sam_for_segmentation).
Instances are assigned ids `1..num_instances` in order; where masks overlap the highest id wins.
**Parameters:**
* **`img`** (`Image`): Input image.
* **`segmentation`** (`Array[(None, None), int32]`): Either a 2D `(H, W)` int32 array of segment ids, or a 3D `(num_instances, H, W)`
boolean array of per-instance masks.
* **`alpha`** (`Float`, default: `Literal(0.5)`): Blend factor for the overlay (0.0 = only original image, 1.0 = only segmentation colors).
* **`background`** (`Int`, default: `Literal(0)`): Segment id to treat as background (not overlaid with color, showing the original
image through). Only meaningful for the 2D id-map form.
* **`segment_colors`** (`Json[(String, ...)] | None`, default: `Literal(None)`): List of colors, one per segment id. If the list is shorter than the number of segments, the
remaining segments will be assigned colors automatically.
* **`draw_contours`** (`Bool`, default: `Literal(True)`): If True, draw contours around each segment with full opacity.
* **`contour_thickness`** (`Int`, default: `Literal(1)`): Thickness of the contour lines in pixels.
**Returns:**
* `pxt.Image`: The image with the colored segmentation overlay.
# vllm
Source: https://docs.pixeltable.com/sdk/latest/vllm
# module pixeltable.functions.vllm
Pixeltable UDFs for vLLM models.
Provides integration with vLLM for high-throughput inference with large language models,
supporting chat completions and text generation with HuggingFace models.
## udf chat\_completions()
```python Signature theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
@pxt.udf
chat_completions(
messages: pxt.Json[(Json, ...)],
*,
model: pxt.String,
engine_args: pxt.Json | None = None,
sampling_params: pxt.Json | None = None
) -> VllmRequestOutput
```
Generate a chat completion from a list of messages using vLLM.
For additional details, see the
[vLLM documentation](https://docs.vllm.ai/en/stable/).
**Requirements:**
* `pip install vllm`
**Parameters:**
* **`messages`** (`pxt.Json[(Json`): A list of messages to generate a response for. Each message should be a dict
with `role` and `content` keys, following the OpenAI chat format.
* **`model`** (`Any`): The HuggingFace model identifier (e.g., `'Qwen/Qwen2.5-0.5B-Instruct'`).
* **`engine_args`** (`Any`): Additional keyword args for the vLLM `LLM` constructor, such as `dtype`,
`max_model_len`, `gpu_memory_utilization`, `tensor_parallel_size`. For details, see the
[vLLM engine args documentation](https://docs.vllm.ai/en/stable/configuration/engine_args/).
* **`sampling_params`** (`Any`): Keyword args for vLLM `SamplingParams`, such as `max_tokens`,
`temperature`, `top_p`, `top_k`. For details, see the
[vLLM sampling params documentation](https://docs.vllm.ai/en/stable/api/vllm/sampling_params.html).
**Returns:**
* `VllmRequestOutput`: A dict containing the vLLM `RequestOutput` in its native format.
**Examples:**
Add a computed column that generates chat completions:
```python theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
t.add_computed_column(
result=chat_completions(
t.messages, model='Qwen/Qwen2.5-0.5B-Instruct'
)
)
```
With custom sampling parameters:
```python theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
t.add_computed_column(
result=chat_completions(
t.messages,
model='Qwen/Qwen2.5-0.5B-Instruct',
sampling_params={'max_tokens': 256, 'temperature': 0.7},
)
)
```
## udf generate()
```python Signature theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
@pxt.udf
generate(
prompt: pxt.String,
*,
model: pxt.String,
engine_args: pxt.Json | None = None,
sampling_params: pxt.Json | None = None
) -> VllmRequestOutput
```
Generate text completion for a given prompt using vLLM.
Uses vLLM's high-throughput inference engine for efficient local LLM serving.
Models are loaded from HuggingFace and cached for reuse across calls.
For additional details, see the
[vLLM documentation](https://docs.vllm.ai/en/stable/).
**Requirements:**
* `pip install vllm`
**Parameters:**
* **`prompt`** (`pxt.String`): The text prompt to generate a completion for.
* **`model`** (`pxt.String`): The HuggingFace model identifier (e.g., `'Qwen/Qwen2.5-0.5B-Instruct'`).
* **`engine_args`** (`pxt.Json | None`): Additional keyword args for the vLLM `LLM` constructor, such as `dtype`,
`max_model_len`, `gpu_memory_utilization`, `tensor_parallel_size`. For details, see the
[vLLM engine args documentation](https://docs.vllm.ai/en/stable/configuration/engine_args/).
* **`sampling_params`** (`pxt.Json | None`): Keyword args for vLLM `SamplingParams`, such as `max_tokens`,
`temperature`, `top_p`, `top_k`. For details, see the
[vLLM sampling params documentation](https://docs.vllm.ai/en/stable/api/vllm/sampling_params.html).
**Returns:**
* `VllmRequestOutput`: A dict containing the vLLM `RequestOutput` in its native format.
**Examples:**
Add a computed column that generates text completions:
```python theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
t.add_computed_column(
result=generate(t.prompt, model='Qwen/Qwen2.5-0.5B-Instruct')
)
```
# voyageai
Source: https://docs.pixeltable.com/sdk/latest/voyageai
# module pixeltable.functions.voyageai
Pixeltable UDFs
that wrap various endpoints from the Voyage AI API. In order to use them, you must
first `pip install voyageai` and configure your Voyage AI credentials, as described in
the [Working with Voyage AI](https://docs.pixeltable.com/howto/providers/working-with-voyageai) tutorial.
## udf embeddings()
```python Signature theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
@pxt.udf
embeddings(
input: pxt.String,
*,
model: pxt.String,
input_type: pxt.String | None = None,
truncation: pxt.Bool | None = None,
output_dimension: pxt.Int | None = None,
output_dtype: pxt.String | None = None
) -> pxt.Array[(None,), float32]
```
Creates an embedding vector representing the input text.
Equivalent to the Voyage AI `embeddings` API endpoint.
For additional details, see: [https://docs.voyageai.com/docs/embeddings](https://docs.voyageai.com/docs/embeddings)
Request throttling:
Applies the rate limit set in the config (section `voyageai`, key `rate_limit`). If no rate
limit is configured, uses a default of 600 RPM.
**Requirements:**
* `pip install voyageai`
**Parameters:**
* **`input`** (`pxt.String`): The text to embed.
* **`model`** (`pxt.String`): The model to use for the embedding. Recommended options: `voyage-3-large`, `voyage-3.5`,
`voyage-3.5-lite`, `voyage-code-3`, `voyage-finance-2`, `voyage-law-2`.
* **`input_type`** (`pxt.String | None`): Type of the input text. Options: `None`, `query`, `document`.
When `input_type` is `None`, the embedding model directly converts the inputs into numerical vectors.
For retrieval/search purposes, we recommend setting this to `query` or `document` as appropriate.
* **`truncation`** (`pxt.Bool | None`): Whether to truncate the input texts to fit within the context length. Defaults to `True`.
* **`output_dimension`** (`pxt.Int | None`): The number of dimensions for resulting output embeddings.
Most models only support a single default dimension. Models `voyage-3-large`, `voyage-3.5`,
`voyage-3.5-lite`, and `voyage-code-3` support: 256, 512, 1024 (default), and 2048.
* **`output_dtype`** (`pxt.String | None`): The data type for the embeddings to be returned. Options: `float`, `int8`, `uint8`,
`binary`, `ubinary`. Only `float` is currently supported in Pixeltable.
**Returns:**
* `pxt.Array[(None,), float32]`: An array representing the application of the given embedding to `input`.
**Examples:**
Add a computed column that applies the model `voyage-3.5` to an existing
Pixeltable column `tbl.text` of the table `tbl`:
```python theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
tbl.add_computed_column(
embed=embeddings(tbl.text, model='voyage-3.5', input_type='document')
)
```
Add an embedding index to an existing column `text`, using the model `voyage-3.5`:
```python theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
tbl.add_embedding_index(
'text', string_embed=embeddings.using(model='voyage-3.5')
)
```
## udf multimodal\_embed()
```python Signatures theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
# Signature 1:
@pxt.udf
multimodal_embed(
text: pxt.String,
*,
model: pxt.String,
input_type: pxt.String | None,
truncation: pxt.Bool
) -> pxt.Array[(1024,), float32]
# Signature 2:
@pxt.udf
multimodal_embed(
image: pxt.Image,
*,
model: pxt.String,
input_type: pxt.String | None,
truncation: pxt.Bool
) -> pxt.Array[(1024,), float32]
# Signature 3:
@pxt.udf
multimodal_embed(
video: pxt.Video,
*,
model: pxt.String,
input_type: pxt.String | None,
truncation: pxt.Bool
) -> pxt.Array[(1024,), float32]
```
Creates an embedding vector for text, images, or video using Voyage AI's multimodal model.
Equivalent to the Voyage AI `multimodal_embed` API endpoint.
For additional details, see: [https://docs.voyageai.com/docs/multimodal-embeddings](https://docs.voyageai.com/docs/multimodal-embeddings)
Request throttling:
Applies the rate limit set in the config (section `voyageai`, key `rate_limit`). If no rate
limit is configured, uses a default of 600 RPM.
**Requirements:**
* `pip install voyageai`
**Parameters:**
* **`text`** (`String`): The text to embed.
* **`image`** (`Image`): The image to embed.
* **`video`** (`Video`): The video to embed.
* **`model`** (`String`): The model to use. Currently only `voyage-multimodal-3` is supported.
* **`input_type`** (`String | None`, default: `Literal(None)`): Type of the input. Options: `None`, `query`, `document`.
For retrieval/search, set to `query` or `document` as appropriate.
* **`truncation`** (`Bool`, default: `Literal(True)`): Whether to truncate inputs to fit within context length. Defaults to `True`.
**Returns:**
* `pxt.Array[(1024,), float32]`: An array of 1024 floats representing the embedding.
**Examples:**
Embed a text column `description`:
```python theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
tbl.add_computed_column(
embed=multimodal_embed(tbl.description, input_type='document')
)
```
Add an embedding index for column `description`:
```python theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
tbl.add_embedding_index(
'description',
embed=multimodal_embed.using(model='voyage-multimodal-3'),
)
```
## udf rerank()
```python Signature theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
@pxt.udf
rerank(
query: pxt.String,
documents: pxt.Json[(String, ...)],
*,
model: pxt.String,
top_k: pxt.Int | None = None,
truncation: pxt.Bool = True
) -> VoyageRerankResponse
```
Reranks documents based on their relevance to a query.
Equivalent to the Voyage AI `rerank` API endpoint.
For additional details, see: [https://docs.voyageai.com/docs/reranker](https://docs.voyageai.com/docs/reranker)
Request throttling:
Applies the rate limit set in the config (section `voyageai`, key `rate_limit`). If no rate
limit is configured, uses a default of 600 RPM.
**Requirements:**
* `pip install voyageai`
**Parameters:**
* **`query`** (`pxt.String`): The query as a string.
* **`documents`** (`pxt.Json[(String`): The documents to be reranked as a list of strings.
* **`model`** (`Any`): The model to use for reranking. Recommended options: `rerank-2.5`, `rerank-2.5-lite`.
* **`top_k`** (`Any`): The number of most relevant documents to return. If not specified, all documents
will be reranked and returned.
* **`truncation`** (`Any`): Whether to truncate the input to satisfy context length limits. Defaults to `True`.
**Returns:**
* `VoyageRerankResponse`: A dictionary containing:
* `results`: List of reranking results with `index`, `document`, and `relevance_score`
* `total_tokens`: The total number of tokens used
**Examples:**
Rerank similarity search results for better relevance. First, create a table with
an embedding index, then use a query function to retrieve candidates and rerank them:
```python theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
docs = pxt.create_table('docs', {'text': pxt.String})
docs.add_computed_column(embed=embeddings(docs.text, model='voyage-3.5'))
docs.add_embedding_index('text', embed=docs.embed)
@pxt.query
def get_candidates(query_text: str):
sim = docs.text.similarity(
query_text, embed=embeddings.using(model='voyage-3.5')
)
return docs.order_by(sim, asc=False).limit(20).select(docs.text)
queries = pxt.create_table('queries', {'query': pxt.String})
queries.add_computed_column(candidates=get_candidates(queries.query))
queries.add_computed_column(
reranked=rerank(
queries.query,
queries.candidates.text,
model='rerank-2.5',
top_k=5,
)
)
```
# whisper
Source: https://docs.pixeltable.com/sdk/latest/whisper
# module pixeltable.functions.whisper
Pixeltable UDFs
that wraps the OpenAI Whisper library.
This UDF will cause Pixeltable to invoke the relevant model locally. In order to use it, you must
first `pip install openai-whisper`.
## udf transcribe()
```python Signature theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
@pxt.udf
transcribe(
audio: pxt.Audio,
*,
model: pxt.String,
temperature: pxt.Json[(Float, ...)] | None = (0.0, 0.2, 0.4, 0.6, 0.8, 1.0),
compression_ratio_threshold: pxt.Float | None = 2.4,
logprob_threshold: pxt.Float | None = -1.0,
no_speech_threshold: pxt.Float | None = 0.6,
condition_on_previous_text: pxt.Bool = True,
initial_prompt: pxt.String | None = None,
word_timestamps: pxt.Bool = False,
prepend_punctuations: pxt.String = '"\'“¿([{-',
append_punctuations: pxt.String = '"\'.。,，!！?？:：”)]}、',
decode_options: pxt.Json | None = None
) -> WhisperTranscription
```
Transcribe an audio file using Whisper.
This UDF runs a transcription model *locally* using the Whisper library,
equivalent to the Whisper `transcribe` function, as described in the
[Whisper library documentation](https://github.com/openai/whisper).
**Requirements:**
* `pip install openai-whisper`
**Parameters:**
* **`audio`** (`pxt.Audio`): The audio file to transcribe.
* **`model`** (`pxt.String`): The name of the model to use for transcription.
**Returns:**
* `WhisperTranscription`: A [`WhisperTranscription`](./whispertranscription) dictionary with the
transcription's text, segments, and language.
**Examples:**
Add a computed column that applies the model `base.en` to an existing Pixeltable column `tbl.audio`
of the table `tbl`:
```python theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
tbl.add_computed_column(result=transcribe(tbl.audio, model='base.en'))
```
Add a `String` column with the transcription's text:
```python theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
tbl.add_computed_column(text=tbl.result.text)
```
# WhisperSegment
Source: https://docs.pixeltable.com/sdk/latest/whispersegment
# class pixeltable.functions.whisper.WhisperSegment
One segment of a transcription, with its timing and decoding statistics.
## attr avg\_logprob
```
avg_logprob: float
```
Average log probability of the tokens in the segment's decoding window.
## attr compression\_ratio
```
compression_ratio: float
```
Compression ratio of the text in the segment's decoding window; a high value indicates repetitive output.
## attr end
```
end: float
```
Segment end, in seconds from the start of the audio.
## attr id
```
id: int
```
Index of the segment in the transcription's `segments` list.
## attr no\_speech\_prob
```
no_speech_prob: float
```
Probability that the segment's decoding window contains no speech.
## attr seek
```
seek: int
```
Start of the segment's decoding window, in 10 ms frames.
## attr start
```
start: float
```
Segment start, in seconds from the start of the audio.
## attr temperature
```
temperature: float
```
Sampling temperature of the segment's decoding window.
## attr text
```
text: str
```
The segment's text.
## attr tokens
```
tokens: list[int]
```
Token IDs of the segment; can include timestamp tokens.
## attr words
```
words: NotRequired[list[WhisperWord]]
```
Timings of the segment's words; empty or absent unless `word_timestamps=True`.
# WhisperTranscription
Source: https://docs.pixeltable.com/sdk/latest/whispertranscription
# class pixeltable.functions.whisper.WhisperTranscription
Output of [`transcribe()`](./whisper#func-transcribe).
## attr language
```
language: str
```
Code of the transcription's language, such as `en`.
## attr segments
```
segments: list[WhisperSegment]
```
The transcription's segments, in order.
## attr text
```
text: str
```
The full transcription.
# WhisperWord
Source: https://docs.pixeltable.com/sdk/latest/whisperword
# class pixeltable.functions.whisper.WhisperWord
One word of a transcription segment, with its timing.
## attr end
```
end: float
```
Word end, in seconds from the start of the audio.
## attr probability
```
probability: float
```
Mean probability of the word's tokens.
## attr start
```
start: float
```
Word start, in seconds from the start of the audio.
## attr word
```
word: str
```
The word's text, including any leading space and attached punctuation.
# whisperx
Source: https://docs.pixeltable.com/sdk/latest/whisperx
# module pixeltable.functions.whisperx
WhisperX audio transcription and diarization functions.
## udf transcribe()
```python Signature theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
@pxt.udf
transcribe(
audio: pxt.Audio,
*,
model: pxt.String,
diarize: pxt.Bool = False,
compute_type: pxt.String | None = None,
language: pxt.String | None = None,
task: pxt.String | None = None,
chunk_size: pxt.Int | None = None,
alignment_model_name: pxt.String | None = None,
interpolate_method: pxt.String | None = None,
return_char_alignments: pxt.Bool | None = None,
diarization_model_name: pxt.String | None = None,
num_speakers: pxt.Int | None = None,
min_speakers: pxt.Int | None = None,
max_speakers: pxt.Int | None = None
) -> pxt.Json
```
Transcribe an audio file using WhisperX.
This UDF runs a transcription model *locally* using the WhisperX library,
equivalent to the WhisperX `transcribe` function, as described in the
[WhisperX library documentation](https://github.com/m-bain/whisperX).
If `diarize=True`, then speaker diarization will also be performed. Several of the UDF parameters are only valid if
`diarize=True`, as documented in the parameters list below.
**Requirements:**
* `pip install whisperx`
**Parameters:**
* **`audio`** (`pxt.Audio`): The audio file to transcribe.
* **`model`** (`pxt.String`): The name of the model to use for transcription.
* **`diarize`** (`pxt.Bool`): Whether to perform speaker diarization.
* **`compute_type`** (`pxt.String | None`): The compute type to use for the model (e.g., `'int8'`, `'float16'`). If `None`,
defaults to `'float16'` on CUDA devices and `'int8'` otherwise.
* **`language`** (`pxt.String | None`): The language code for the transcription (e.g., `'en'` for English).
* **`task`** (`pxt.String | None`): The task to perform (e.g., `'transcribe'` or `'translate'`). Defaults to `'transcribe'`.
* **`chunk_size`** (`pxt.Int | None`): The size of the audio chunks to process, in seconds. Defaults to `30`.
* **`alignment_model_name`** (`pxt.String | None`): The name of the alignment model to use. If `None`, uses the default model for the given
language. Only valid if `diarize=True`.
* **`interpolate_method`** (`pxt.String | None`): The method to use for interpolation of the alignment results. If not specified, uses the
WhisperX default (`'nearest'`). Only valid if `diarize=True`.
* **`return_char_alignments`** (`pxt.Bool | None`): Whether to return character-level alignments. Defaults to `False`.
Only valid if `diarize=True`.
* **`diarization_model_name`** (`pxt.String | None`): The name of the diarization model to use. Defaults to
`pyannote/speaker-diarization-3.1`. Only valid if `diarize=True`.
* **`num_speakers`** (`pxt.Int | None`): The number of speakers to expect in the audio. By default, the model with try to detect the
number of speakers. Only valid if `diarize=True`.
* **`min_speakers`** (`pxt.Int | None`): If specified, the minimum number of speakers to expect in the audio.
Only valid if `diarize=True`.
* **`max_speakers`** (`pxt.Int | None`): If specified, the maximum number of speakers to expect in the audio.
Only valid if `diarize=True`.
**Returns:**
* `pxt.Json`: A dictionary containing the audio transcription, diarization (if enabled), and various other metadata.
**Examples:**
Add a computed column that applies the model `tiny.en` to an existing Pixeltable column `tbl.audio`
of the table `tbl`:
```python theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
tbl.add_computed_column(result=transcribe(tbl.audio, model='tiny.en'))
```
Add a computed column that applies the model `tiny.en` to an existing Pixeltable column `tbl.audio`
of the table `tbl`, with speaker diarization enabled, expecting at least 2 speakers:
```python theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
tbl.add_computed_column(
result=transcribe(
tbl.audio, model='tiny.en', diarize=True, min_speakers=2
)
)
```
# yolox
Source: https://docs.pixeltable.com/sdk/latest/yolox
# module pixeltable.functions.yolox
YOLOX object detection functions.
## udf yolo\_to\_coco()
```python Signature theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
@pxt.udf
yolo_to_coco(detections: pxt.Json) -> pxt.Json[(Json[{'bbox': Json[(Int, ...)], 'category': Int}], ...)]
```
Converts the output of a YOLOX object detection model to COCO format.
**Parameters:**
* **`detections`** (`pxt.Json`): The output of a YOLOX object detection model, as returned by `yolox`.
**Returns:**
* `pxt.Json[(Json[{'bbox': Json[(Int, ...)], 'category': Int}], ...)]`: A dictionary containing the data from `detections`, converted to COCO format.
**Examples:**
Add a computed column that converts the output `tbl.detections` to COCO format, where `tbl.image`
is the image for which detections were computed:
```python theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
tbl.add_computed_column(
detections=yolox(tbl.image, model_id='yolox_m', threshold=0.8)
)
tbl.add_computed_column(detections_coco=yolo_to_coco(tbl.detections))
```
## udf yolox()
```python Signature theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
@pxt.udf
yolox(
images: pxt.Image,
*,
model_id: pxt.String,
threshold: pxt.Float = 0.5
) -> YoloxResponse
```
Computes YOLOX object detections for the specified image. `model_id` should reference one of the models
defined in the [YOLOX documentation](https://github.com/Megvii-BaseDetection/YOLOX).
**Requirements**:
* `pip install pixeltable-yolox`
**Parameters:**
* **`model_id`** (`pxt.String`): one of: `yolox_nano`, `yolox_tiny`, `yolox_s`, `yolox_m`, `yolox_l`, `yolox_x`
* **`threshold`** (`pxt.Float`): the threshold for object detection
**Returns:**
* `YoloxResponse`: A dictionary containing the output of the object detection model.
**Examples:**
Add a computed column that applies the model `yolox_m` to an existing
Pixeltable column `tbl.image` of the table `tbl`:
```python theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
tbl.add_computed_column(
detections=yolox(tbl.image, model_id='yolox_m', threshold=0.8)
)
```
# Computed Columns
Source: https://docs.pixeltable.com/tutorials/computed-columns
![Open in Kaggle]()
![Open in Colab]()
![Download Notebook]()
This documentation page is also available as an interactive notebook. You can launch the notebook in
Kaggle or Colab, or download it for use with an IDE or local Jupyter installation, by clicking one of the
above links.
This notebook uses `pxt.create_table()` and `add_computed_column()` so
you can run cells without a project. An app puts the same columns on a
`TableModel` in `app.py` and creates the tables with
`pxt schema update`.
This guide introduces one of Pixeltable’s most essential and powerful
concepts: computed columns. You’ll learn how to:
* Add computed columns to a table
* Use computed columns for complex operations such as image processing
and model inference
## Prerequisites
This guide assumes you’re familiar with:
* Creating and managing tables
* Inserting and querying data
* Basic table operations
If you’re new to Pixeltable, start with the [Tables and Data
Operations](/tutorials/tables-and-data-operations)
guide.
First, let’s ensure the Pixeltable library is installed in your
environment, along with the Huggingface `transformers` library.
```python theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
%pip install -qU pixeltable torch transformers timm
```
### Computed Columns
Let’s start with a simple example that illustrates the basic concepts
behind computed columns. We’ll use a table of world population data for
our example. Remember that you can import datasets into a Pixeltable
table by using `pxt.create_table()` with the `source` parameter.
```python theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
import pixeltable as pxt
pxt.create_dir('fundamentals', if_exists='ignore')
pop_t = pxt.create_table(
'fundamentals/population',
source='https://github.com/pixeltable/pixeltable/raw/release/docs/resources/world-population-data.csv',
if_exists='replace',
)
```
Connected to Pixeltable database at: postgresql+psycopg://postgres:@/pixeltable?host=/Users/asiegel/.pixeltable/pgdata
Created directory 'fundamentals'.
Created table 'population'.
Inserting rows into \`population\`: 234 rows \[00:00, 6850.71 rows/s]
Inserted 234 rows with 0 errors.
Also recall that `pop_t.head()` returns the first few rows of a table,
and typing the table name `pop_t` by itself gives the schema.
```python theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
pop_t.head(5)
```
```python theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
pop_t
```
Now let’s suppose we want to add a new column for the year-over-year
population change from 2022 to 2023. You can `select()` such a quantity
into a Pixeltable `Query`, giving it the name `yoy_change`
(year-over-year change):
```python theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
pop_t.select(
pop_t.country, yoy_change=(pop_t.pop_2023 - pop_t.pop_2022)
).head(5)
```
A **computed column** is a way of turning such a selection into a new,
permanent column of the table. Here’s how it works:
```python theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
pop_t.add_computed_column(yoy_change=(pop_t.pop_2023 - pop_t.pop_2022))
```
Added 234 column values with 0 errors.
234 rows updated, 468 values computed.
As soon as the column is added, Pixeltable will (by default)
automatically compute its value for all rows in the table, storing the
results in the new column. If we now inspect the schema of `pop_t`, we
see the new column and its definition.
```python theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
pop_t
```
The new column can be queried in the usual manner.
```python theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
pop_t.select(pop_t.country, pop_t.yoy_change).head(5)
```
The output is identical to the previous example, but now we’re
retrieving the computed output from the database, instead of computing
it on-the-fly.
Computed columns can be “chained” with other computed columns. Here’s an
example that expresses population change as a percentage:
```python theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
pop_t.add_computed_column(
yoy_percent_change=(100 * pop_t.yoy_change / pop_t.pop_2022)
)
```
Added 234 column values with 0 errors.
234 rows updated, 468 values computed.
```python theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
pop_t
```
```python theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
pop_t.select(
pop_t.country, pop_t.yoy_change, pop_t.yoy_percent_change
).head(5)
```
Although computed columns appear superficially similar to Queries, there
is a key difference. Because computed columns are a permanent part of
the table, they will be automatically updated any time new data is added
to the table. These updates will propagate through any other computed
columns that are “downstream” of the new data, ensuring that the state
of the entire data is kept up-to-date.
In traditional data workflows, it is commonplace
to recompute entire pipelines when the input dataset is changed or
enlarged. In Pixeltable, by contrast, all updates are applied
incrementally. When new data appear in a table or existing data are
altered, Pixeltable will recompute only those rows that are dependent on
the changed data.
Let’s see how this works in practice. For purposes of illustration,
we’ll add an entry for California to the table, as if it were a country.
```python theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
pop_t.insert(country='California', pop_2023=39110000, pop_2022=39030000)
```
Inserting rows into \`population\`: 1 rows \[00:00, 228.35 rows/s]
Inserted 1 row with 0 errors.
1 row inserted, 5 values computed.
Observe that the computed columns `yoy_growth` and `yoy_percent_growth`
have been automatically updated in response to the new data.
```python theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
pop_t.tail(5)
```
Remember that all tables in Pixeltable are
persistent. This includes computed columns: when you create a
computed column, its definition is stored in the database. You can think
of computed columns as setting up a persistent compute workflow: if you
close your notebook or restart your Python instance, computed columns
(along with the relationships between them, and any data contained in
them) will be preserved.
### Recomputing Columns
From time to time you might need to recompute the data in an existing
computed column. Perhaps the *code* for one of your UDFs has changed,
and you want to recompute a column that uses that UDF in order to pick
up the new logic. Or perhaps you want to re-run a nondeterministic
computation such as model inference. The command to do this is
`recompute_columns()`. It won’t do much in the current example, because
all our computations are simple and deterministic, but for demonstration
purposes here’s what it looks like:
```python theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
pop_t.recompute_columns(pop_t.yoy_change, pop_t.yoy_percent_change)
```
Inserting rows into \`population\`: 235 rows \[00:00, 8795.92 rows/s]
235 rows updated, 940 values computed.
```python theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
pop_t.tail(5)
```
As expected, it looks the same.
If you modify the data that a computed
column depends on, Pixeltable will recompute automatically; so
recompute\_columns() is primarily useful when the input data
remains the same, but your UDF business logic changes.
### A More Complex Example: Image Processing
Pixeltable supports media data such as images alongside traditional
structured data. Let’s explore an example that uses computed columns for
image processing operations.
In this example, we’ll create the table directly by providing a schema,
rather than importing it from a CSV.
```python theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
t = pxt.create_table('fundamentals/image_ops', {'source': pxt.Image})
```
Created table 'image\_ops'.
```python theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
url_prefix = 'https://github.com/pixeltable/pixeltable/raw/release/docs/resources/images'
images = ['000000000139.jpg', '000000000632.jpg', '000000000872.jpg']
t.insert({'source': f'{url_prefix}/{image}'} for image in images)
```
Inserting rows into \`image\_ops\`: 3 rows \[00:00, 1133.39 rows/s]
Inserted 3 rows with 0 errors.
3 rows inserted, 6 values computed.
```python theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
t.collect()
```
What are some things we might want to do with these images? A fairly
basic one is to extract metadata. Pixeltable provides the built-in UDF
`get_metadata()`, which returns a dictionary with various metadata about
the image. Let’s go ahead and make this a computed column.
“UDF” is standard terminology in databases,
meaning “User-Defined Function”. Technically speaking, the
get\_metadata() function isn’t user-defined, it’s built in
to the Pixeltable library. But we’ll consistently refer to Pixeltable
functions as “UDFs” in order to clearly distinguish them from ordinary
Python functions. Later in this guide, we’ll see how to turn (almost)
any Python function into a Pixeltable UDF.
```python theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
t.add_computed_column(metadata=t.source.get_metadata())
t.collect()
```
Added 3 column values with 0 errors.
Image operations, of course, can also return new images.
```python theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
t.add_computed_column(rotated=t.source.rotate(10))
```
Added 3 column values with 0 errors.
3 rows updated, 3 values computed.
```python theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
t.collect()
```
Or, perhaps we want to rotate our images and fill them in with a
transparent background rather than black. We can do this by chaining
image operations, adding a transparency layer before doing the rotation.
```python theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
t.add_computed_column(
rotated_transparent=t.source.convert('RGBA').rotate(10)
)
t.collect()
```
Added 3 column values with 0 errors.
In addition to get\_metadata(),
convert(), and rotate(), Pixeltable has a
sizable library of other common image operations that can be used as
UDFs in computed columns. For the most part, the image UDFs are analogs
of the operations provided by the
Pillow library
(in fact, Pixeltable is just using Pillow under the covers). You can
read more about the provided image (and other) UDFs in the
Pixeltable SDK
Documentation.
Let’s have a look at our table schema.
```python theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
t
```
### Image Detection
In addition to simple operations like `rotate()` and `convert()`, the
Pixeltable API includes UDFs for various off-the-shelf image models.
Let’s look at one example: object detection using the ResNet-50 model.
Model inference is a UDF too, and it can be inserted into a computed
column like any other.
This one may take a little more time to compute, since it involves first
downloading the ResNet-50 model (if it isn’t already cached), then
running inference on the images in our table.
```python theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
from pixeltable.functions.huggingface import detr_for_object_detection
t.add_computed_column(
detections=detr_for_object_detection(
t.source, model_id='facebook/detr-resnet-50', threshold=0.8
)
)
```
Added 3 column values with 0 errors.
3 rows updated, 3 values computed.
```python theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
t.select(t.source, t.detections).collect()
```
It’s great that the DETR model gave us so much information about the
images, but it’s not exactly in human-readable form. Those are JSON
structures that encode bounding boxes, confidence scores, and categories
for each detected object. Let’s do something more useful with them:
we’ll use Pixeltable’s `bboxes_draw()` API to superimpose bounding boxes
on the images, using different colors to distinguish different object
categories.
```python theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
from pixeltable.functions.vision import bboxes_draw
t.add_computed_column(
image_with_bb=bboxes_draw(
t.source,
t.detections.boxes,
labels=t.detections.label_text,
fill=True,
)
)
t.select(t.source, t.image_with_bb).collect()
```
Added 3 column values with 0 errors.
It can be a little hard to see what’s going on, so let’s zoom in on just
one image. If you select a single image in a notebook, Pixeltable will
enlarge its display:
```python theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
t.select(t.image_with_bb).head(1)
```
Let’s check in on our schema. We now have five computed columns, all
derived from the single source column.
```python theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
t
```
And as always, when we add new data to the table, its computed columns
are updated automatically. Let’s try this on a few more images.
```python theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
more_images = ['000000000108.jpg', '000000000885.jpg']
t.insert({'source': f'{url_prefix}/{image}'} for image in more_images)
```
Inserting rows into \`image\_ops\`: 2 rows \[00:00, 944.77 rows/s]
Inserted 2 rows with 0 errors.
2 rows inserted, 14 values computed.
```python theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
t.select(
t.source, t.image_with_bb, t.detections.label_text, t.metadata
).tail(2)
```
It bears repeating that Pixeltable is
persistent! Anything you put into a table, including computed
columns, will be saved in persistent storage. This includes inference
outputs such as t.detections, as well as generated images
such as t.image\_with\_bb. (Later we’ll see how to tune this
behavior in cases where it might be undesirable to store
everything, but the default behavior is that computed column
output is always persisted.)
### Expressions
Let’s have a closer look at that call to `bboxes_draw()` in the last
example.
```python theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
bboxes_draw(t.source, t.detections.boxes, labels=t.detections.label_text, fill=True)
```
There are a couple of things going on. `bboxes_draw()` is, of course, a
UDF, and its first argument is a column reference of the sort we’ve used
many times now: `t.source`, the source image. The other two arguments
are more than simple column references, though: they’re compound
expressions that include the column reference `t.detections` along with
a suffix (`.boxes` or `.label_text`) that tells Pixeltable to look
inside the dictionary stored in `t.detections`.
These are all examples of Pixeltable expressions. In fact, we’ve
seen other types of Pixeltable expressions as well, without explicitly
calling them out:
* Calls to a UDF are expressions, such as `t.source.rotate(10)`, or the
`bboxes_draw()` example above;
* Arithmetic operations are expressions, such as year-over-year
calculation in our first example:
`100 * pop_t.yoy_change / pop_t.pop_2022`.
## Next Steps
Learn more about working with Pixeltable:
* [Queries and
Expressions](/tutorials/queries-and-expressions)
* [Tables and Data
Operations](/tutorials/tables-and-data-operations)
## Ship it as an app
This notebook used `create_table` and `add_computed_column`. On a
`TableModel`, a computed column is an `=` on the class, not
`add_computed_column()`:
```python theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
import pixeltable as pxt
TableModel = pxt.model_base()
class Population(TableModel, name='population'):
country: pxt.String
year_2022: pxt.Int
year_2023: pxt.Int
yoy_change = year_2023 - year_2022
```
```bash theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
pxt schema update app.py fundamentals
```
# Queries and Expressions
Source: https://docs.pixeltable.com/tutorials/queries-and-expressions
![Open in Kaggle]()
![Open in Colab]()
![Download Notebook]()
This documentation page is also available as an interactive notebook. You can launch the notebook in
Kaggle or Colab, or download it for use with an IDE or local Jupyter installation, by clicking one of the
above links.
This notebook uses `pxt.create_table()` so you can run cells without a
project. An app puts the same columns on a `TableModel` in `app.py` and
creates the tables with `pxt schema update`. Once the table exists,
`select()` is the same in an app.
Expressions are the basic building blocks of Pixeltable. This guide
explores how to use queries and expressions, including:
* Different types of Pixeltable expressions
* Column references and arithmetic operations
* Function calls and media operations
* The Pixeltable type system
## Prerequisites
This guide assumes you’re familiar with:
* Creating and managing tables
* Basic table operations and queries
* Computed columns
If you’re new to these concepts, start with:
* [Tables and Data
Operations](/tutorials/tables-and-data-operations)
* [Computed
Columns](/tutorials/computed-columns)
## Understanding Expressions
You can use Pixeltable expressions in queries:
```python theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
pop_t.select(yoy_change=(pop_t.pop_2023 - pop_t.pop_2022)).collect()
```
Or as computed columns that update automatically:
```python theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
pop_t.add_column(yoy_change=(pop_t.pop_2023 - pop_t.pop_2022))
```
Both examples use the expression `pop_t.pop_2023 - pop_t.pop_2022`. You
can also chain operations:
```python theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
t.source.convert('RGBA').rotate(10)
```
Or invoke models:
```python theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
detr_for_object_detection(
t.source,
model_id='facebook/detr-resnet-50',
threshold=0.8
)
```
You can include an expression in a `select()` statement to evaluate it
dynamically, or in an `add_column()` statement to add it to the table
schema as a computed column.
To get started, let’s import the necessary libraries and set up a demo
directory.
```python theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
%pip install -qU pixeltable datasets torch transformers
```
```python theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
import pixeltable as pxt
pxt.drop_dir('demo', force=True)
pxt.create_dir('demo')
```
Connected to Pixeltable database at: postgresql+psycopg://postgres:@/pixeltable?host=/Users/asiegel/.pixeltable/pgdata
Created directory 'demo'.
In this guide we’ll work with a subset of the MNIST dataset, a classic
reference database of hand-drawn digits. A copy of the MNIST dataset is
hosted on the Hugging Face datasets repository, so we can use
`create_table()` with the `source` parameter to load it into a
Pixeltable table.
```python theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
import datasets
# Download the first 50 images of the MNIST dataset
ds = datasets.load_dataset('ylecun/mnist', split='train[:50]')
# Import them into a Pixeltable table
t = pxt.create_table('demo/mnist', source=ds)
```
Created table 'mnist'.
Inserting rows into \`mnist\`: 50 rows \[00:00, 7516.67 rows/s]
Inserted 50 rows with 0 errors.
```python theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
t.head(5)
```
### Column References
The most basic type of expression is a **column reference**: that’s what
you get when you type, say, `t.image`. An expression such as `t.image`
by itself is just a Python object; it doesn’t contain any actual data,
and no data will be loaded until you use the expression in a `select()`
query or `add_column()` statement. Here’s what we get if we type
`t.image` by itself:
```python theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
t.image
```
This is true of all Pixeltable expressions: we can freely create them
and manipulate them in various ways, but no actual data will be loaded
until we use them in a query.
### JSON Collections (Dicts and Lists)
Data is commonly presented in JSON format: for example, API responses
and model output often take the shape of JSON dictionaries or lists of
dictionaries. Pixeltable has native support for JSON accessors. To
demonstrate this, let’s add a computed column that runs an image
classification model against the images in our dataset.
```python theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
from pixeltable.functions.huggingface import vit_for_image_classification
t.add_computed_column(
classification=vit_for_image_classification(
t.image, model_id='farleyknight-org-username/vit-base-mnist'
)
)
```
Added 50 column values with 0 errors.
50 rows updated, 50 values computed.
```python theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
t.select(t.image, t.classification).head(3)
```
We see that the output is returned as a dict containing three lists: the
five most likely labels (classes) for the image, the corresponding text
labels (in this case, just the string form of the class number), and the
scores (confidences) of each prediction. The Pixeltable type of the
`classification` column is `pxt.Json`:
```python theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
t
```
Pixeltable provides a range of operators on `Json`-typed output that
behave just as you’d expect. To look up a key in a dictionary, use the
syntax `t.classification['labels']`:
```python theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
t.select(t.classification['labels']).head(3)
```
You can also use a convenient “attribute” syntax for dictionary lookups.
This follows the standard
[JSONPath](https://en.wikipedia.org/wiki/JSONPath) expression syntax.
```python theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
t.select(t.classification.labels).head(3)
```
The “attribute” syntax isn’t fully general (it won’t work for dictionary
keys that are not valid Python identifiers), but it’s handy when it
works.
`t.classification.labels` is another Pixeltable expression; you can
think of it as saying, “do the `'labels'` lookup from every dictionary
in the column `t.classification`, and return the result as a new
column.” As before, the expression by itself contains no data; it’s the
query that does the actual work of retrieving data. Here’s what we see
if we just give the expression by itself, without a query:
```python theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
t.classification.labels
```
classification.labels
Similarly, one can pull out a specific item in a list (for this model,
we’re probably mostly interested in the first item anyway):
```python theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
t.select(t.classification.labels[0]).head(3)
```
Or slice a list in the usual manner:
```python theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
t.select(t.classification.labels[:2]).head(3)
```
Pixeltable is resilient against out-of-bounds indices. If an index
doesn’t exist for a particular row, you’ll get a `None` output for that
row.
```python theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
t.select(t.classification.labels[10]).head(3)
```
As always, any expression can be used to create a computed column.
```python theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
# Use label_text to be consistent with t.label, which was given
# to us as a string
t.add_computed_column(pred_label=t.classification.label_text[0])
t
```
Added 50 column values with 0 errors.
Finally, just as it’s possible to extract items from lists and
dictionaries using Pixeltable expressions, you can also construct new
lists and dictionaries: just package them up in the usual way.
```python theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
custom_dict = {
# Keys must be strings; values can be any expressions
'ground_truth': t.label,
'prediction': t.pred_label,
'is_correct': t.label == t.pred_label,
# You can also use constants as values
'engine': 'pixeltable',
}
t.select(t.image, custom_dict).head(5)
```
### UDF Calls
UDF calls are another common type of expression. For example, we used
one earlier when we added a model invocation to our workload:
```python theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
vit_for_image_classification(
t.image,
model_id='farleyknight-org-username/vit-base-mnist'
)
```
This calls the `vit_for_image_classification` UDF in the
`pxt.functions.huggingface` module. Note that
`vit_for_image_classification` is a Pixeltable UDF, not an ordinary
Python function. You can think of a Pixeltable UDF as a function that
operates on columns of data, iteratively applying an underlying
operation to each row in the column (or columns). In this case,
`vit_for_image_classification` operates on `t.image`, running the model
against every image in the column.
Notice that in addition to the column `t.image`, this call to
`vit_for_image_classification` also takes a constant argument specifying
the `model_id`. Any UDF call argument may be a constant, and the
constant value simply means “use this value for every row being
evaluated”.
You can always compose Pixeltable expressions to form more complicated
ones; here’s an example that runs the model against a 90-degree rotation
of every image in the sample and extracts the label. Not surprisingly,
the model doesn’t perform as well on the rotated images.
```python theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
rot_model_result = vit_for_image_classification(
t.image.rotate(90),
model_id='farleyknight-org-username/vit-base-mnist',
)
t.select(t.image, rot_label=rot_model_result.labels[0]).head(5)
```
Note that we employed a useful trick here: we
assigned an expression to the variable rot\_model\_result for
later reuse. Every Pixeltable expression is a Python object, so you can
freely assign them to variables, reuse them, compose them, and so on.
Remember that nothing actually happens until the expression is used in a
query - so in this example, setting the variable
rot\_model\_result doesn’t itself result in any data being
retrieved; that only happens later, when we actually use it in the
select() query.
There are a large number of built-in UDFs that ship with Pixeltable; you
can always refer back to the [SDK
Documentation](/sdk/latest/) for details.
### Method Calls
Many built-in UDFs allow a convenient alternate syntax. The following
two expressions are exactly equivalent:
```python theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
a = t.image.rotate(90)
b = pxt.functions.image.rotate(t.image, 90)
```
`a` and `b` can always be used interchangeably in queries, with
identical results. Just like in standard Python classes, whenever
Pixeltable sees the **method call** `t.image.rotate(90)`, it interprets
it as a **function call** `pxt.functions.image.rotate(self, 90)`, with
(in this case) `self` equal to `t.image`.
Any method call can also be written as a function call, but (just like
in standard Python) not every function call can be written as a method
call. For example, the following won’t work:
```python theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
t.image.vit_for_image_classification(
model_id='farleyknight-org-username/vit-base-mnist'
)
```
That’s because `vit_for_image_classification` is part of the
`pxt.functions.huggingface` module, not the core module
`pxt.functions.image`. Most Pixeltable types have a corresponding **core
module** of UDFs that can be used as method calls (`pxt.functions.image`
for `Image`; `pxt.functions.string` for `String`; and so on), described
fully in the [SDK
Documentation](/sdk/latest/).
### Arithmetic and Boolean Operations
Expressions can also be combined using standard arithmetic and boolean
operators. As with everything else, arithmetic and boolean expressions
are operations on columns that (when used in a query) are applied to
every row.
```python theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
t.select(t.image, t.label, t.label == '4', t.label < '5').head(5)
```
When you use a `where` clause in a query, you’re giving it a Pixeltable
expression, too (a boolean-valued one).
```python theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
t.where(t.label == '4').select(t.image).show()
```
The following example shows how boolean expressions can be assigned to
variables and used to form more complex expressions.
```python theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
# Reuse `rot_model_result` from above, extracting
# the dominant label as a new expression
rot_label = rot_model_result.label_text[0]
# Select all the rows where the ground truth label is '5',
# and the "rotated" version of the model got it wrong
# (by returning something other than a '5')
t.where((t.label == '5') & (rot_label != '5')).select(
t.image, t.label, rot_label=rot_label
).show()
```
Notice that to form a logical “and”, we wrote
```python theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
(t.label == '5') & (rot_label != '5')
```
using the operator `&` rather than `and`. Likewise, to form a logical
“or”, we’d use `|` rather than `or`:
```python theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
(t.label == '5') | (rot_label != '5')
```
For logical negation:
```python theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
~(t.label == '5')
```
This follows the convention used by other popular data-manipulation
frameworks such as Pandas, and it’s necessary because the Python
language does not allow the meanings of `and`, `or`, and `not` to be
customized. There is one more instance of this to be aware of: to check
whether an expression is `None`, it’s necessary to write (say)
```python theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
t.label == None
```
rather than `t.label is None`, for the same reason.
### Arrays
In addition to lists and dicts, Pixeltable also has built-in support for
numerical arrays. A typical place where arrays show up is as the output
of an embedding.
```python theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
from pixeltable.functions.huggingface import clip
# Add a computed column that computes a CLIP embedding for each image
t.add_computed_column(
clip=clip(t.image, model_id='openai/clip-vit-base-patch32')
)
t.select(t.image, t.clip).head(5)
```
Added 50 column values with 0 errors.
The underlying Python type of `pxt.Array` is an ordinary NumPy array
(`np.ndarray`), so that an array-typed column is a column of NumPy
arrays (in this example, representing the embedding output of each image
in the table). As with lists, arrays can be sliced in all the usual
ways.
```python theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
t.select(t.clip[0], t.clip[5:10], t.clip[-3:]).head(5)
```
### Ad hoc UDFs with `apply`
We’ve now seen the most commonly encountered Pixeltable expression
types. There are a few other less commonly encountered expressions that
are occasionally useful.
You can use `apply` to map any Python function onto a column of data.
You can think of `apply` as a quick way of constructing an “on-the-fly”
UDF for one-off use.
```python theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
import numpy as np
t.select(t.clip.apply(np.ndarray.dumps, col_type=pxt.String)).head(2)
```
Note, however, that if the function you’re `apply`ing doesn’t have type
hints (as in the example here), you’ll need to specify the output column
type explicitly.
### Type Conversion with `astype`
Sometimes it’s useful to transform an expression of one type into a
different type. For example, you can use `astype` to turn an expression
of type `pxt.Json` into one of type `pxt.String`. This assumes that the
value being converted is actually a string; otherwise, you’ll get an
exception. Here’s an example:
```python theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
# Select the text in position 0 of `t.classification.label_text`; since
# `t.classification.label_text` has type `pxt.Json`, so does
# `t.classification.label_text[0]`
t.classification.label_text[0].col_type
```
Optional\[Json]
```python theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
# Select the text in position 0 of `t.classification.label_text`, this time
# cast as a `pxt.String`
t.classification.label_text[0].astype(pxt.String).col_type
```
Optional\[String]
### Column Properties
Some `ColumnRef` expressions have additional useful properties. A media
column (image, video, audio, or document) has the following two
properties:
* `localpath`: the media location on the local filesystem
* `fileurl`: the original URL where the media resides (could be the same
as `localpath`)
```python theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
t.select(t.image, t.image.localpath).head(5)
```
Any computed column will have two additional properties, `errortype` and
`errormsg`. These properties will usually be `None`. However, if the
computed column was created with `on_error='ignore'` and an exception
was encountered during column execution, then the properties will
contain additional information about the exception.
To demonstrate this feature, we’re going to deliberately trigger an
exception in a computed column. The images in our example table are
black and white, meaning they have only one color channel. If we try to
extract a channel other than channel number `0`, we’ll get an exception.
Ordinarily when we call `add_computed_column`, the exception is raised
and the `add_computed_column` operation is aborted.
```python theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
t.add_computed_column(channel=t.image.getchannel(1))
```
Error: Error while evaluating computed column 'channel':
band index out of range
\[0;31m---------------------------------------------------------------------------\[0m
\[0;31mValueError\[0m Traceback (most recent call last)
File \[0;32m\~/Dropbox/workspace/pixeltable/pixeltable/pixeltable/exec/expr\_eval/evaluators.py:225\[0m, in \[0;36mFnCallEvaluator.eval\[0;34m(self, call\_args\_batch)\[0m
\[1;32m 224\[0m \[38;5;28;01mtry\[39;00m:
\[0;32m--> 225\[0m item\[38;5;241m.\[39mrow\[\[38;5;28mself\[39m\[38;5;241m.\[39mfn\_call\[38;5;241m.\[39mslot\_idx] \[38;5;241m=\[39m \[38;5;28;43mself\[39;49m\[38;5;241;43m.\[39;49m\[43mscalar\_py\_fn\[49m\[43m(\[49m\[38;5;241;43m*\[39;49m\[43mitem\[49m\[38;5;241;43m.\[39;49m\[43margs\[49m\[43m,\[49m\[43m \[49m\[38;5;241;43m*\[39;49m\[38;5;241;43m\*\[39;49m\[43mitem\[49m\[38;5;241;43m.\[39;49m\[43mkwargs\[49m\[43m)\[49m
\[1;32m 226\[0m \[38;5;28;01mexcept\[39;00m \[38;5;167;01mException\[39;00m \[38;5;28;01mas\[39;00m exc:
File \[0;32m/opt/miniconda3/envs/pxt/lib/python3.10/site-packages/PIL/Image.py:2682\[0m, in \[0;36mImage.getchannel\[0;34m(self, channel)\[0m
\[1;32m 2680\[0m \[38;5;28;01mraise\[39;00m \[38;5;167;01mValueError\[39;00m(msg) \[38;5;28;01mfrom\[39;00m \[38;5;21;01me\[39;00m
\[0;32m-> 2682\[0m \[38;5;28;01mreturn\[39;00m \[38;5;28mself\[39m\[38;5;241m.\[39m\_new(\[38;5;28;43mself\[39;49m\[38;5;241;43m.\[39;49m\[43mim\[49m\[38;5;241;43m.\[39;49m\[43mgetband\[49m\[43m(\[49m\[43mchannel\[49m\[43m)\[49m)
\[0;31mValueError\[0m: band index out of range
The above exception was the direct cause of the following exception:
\[0;31mError\[0m Traceback (most recent call last)
Cell \[0;32mIn\[27], line 1\[0m
\[0;32m----> 1\[0m \[43mt\[49m\[38;5;241;43m.\[39;49m\[43madd\_computed\_column\[49m\[43m(\[49m\[43mchannel\[49m\[38;5;241;43m=\[39;49m\[43mt\[49m\[38;5;241;43m.\[39;49m\[43mimage\[49m\[38;5;241;43m.\[39;49m\[43mgetchannel\[49m\[43m(\[49m\[38;5;241;43m1\[39;49m\[43m)\[49m\[43m)\[49m
File \[0;32m\~/Dropbox/workspace/pixeltable/pixeltable/pixeltable/catalog/table.py:697\[0m, in \[0;36mTable.add\_computed\_column\[0;34m(self, stored, destination, print\_stats, on\_error, if\_exists, \*\*kwargs)\[0m
\[1;32m 695\[0m \[38;5;28mself\[39m\[38;5;241m.\[39m\_verify\_column(new\_col)
\[1;32m 696\[0m \[38;5;28;01massert\[39;00m \[38;5;28mself\[39m\[38;5;241m.\[39m\_tbl\_version \[38;5;129;01mis\[39;00m \[38;5;129;01mnot\[39;00m \[38;5;28;01mNone\[39;00m
\[0;32m--> 697\[0m result \[38;5;241m+\[39m\[38;5;241m=\[39m \[38;5;28;43mself\[39;49m\[38;5;241;43m.\[39;49m\[43m\_tbl\_version\[49m\[38;5;241;43m.\[39;49m\[43mget\[49m\[43m(\[49m\[43m)\[49m\[38;5;241;43m.\[39;49m\[43madd\_columns\[49m\[43m(\[49m\[43m\[\[49m\[43mnew\_col\[49m\[43m]\[49m\[43m,\[49m\[43m \[49m\[43mprint\_stats\[49m\[38;5;241;43m=\[39;49m\[43mprint\_stats\[49m\[43m,\[49m\[43m \[49m\[43mon\_error\[49m\[38;5;241;43m=\[39;49m\[43mon\_error\[49m\[43m)\[49m
\[1;32m 698\[0m FileCache\[38;5;241m.\[39mget()\[38;5;241m.\[39memit\_eviction\_warnings()
\[1;32m 699\[0m \[38;5;28;01mreturn\[39;00m result
File \[0;32m\~/Dropbox/workspace/pixeltable/pixeltable/pixeltable/catalog/table\_version.py:666\[0m, in \[0;36mTableVersion.add\_columns\[0;34m(self, cols, print\_stats, on\_error)\[0m
\[1;32m 664\[0m all\_cols\[38;5;241m.\[39mappend(undo\_col)
\[1;32m 665\[0m \[38;5;66;03m# Add all columns\[39;00m
\[0;32m--> 666\[0m status \[38;5;241m=\[39m \[38;5;28;43mself\[39;49m\[38;5;241;43m.\[39;49m\[43m\_add\_columns\[49m\[43m(\[49m\[43mall\_cols\[49m\[43m,\[49m\[43m \[49m\[43mprint\_stats\[49m\[38;5;241;43m=\[39;49m\[43mprint\_stats\[49m\[43m,\[49m\[43m \[49m\[43mon\_error\[49m\[38;5;241;43m=\[39;49m\[43mon\_error\[49m\[43m)\[49m
\[1;32m 667\[0m \[38;5;66;03m# Create indices and their md records\[39;00m
\[1;32m 668\[0m \[38;5;28;01mfor\[39;00m col, (idx, val\_col, undo\_col) \[38;5;129;01min\[39;00m index\_cols\[38;5;241m.\[39mitems():
File \[0;32m\~/Dropbox/workspace/pixeltable/pixeltable/pixeltable/catalog/table\_version.py:732\[0m, in \[0;36mTableVersion.\_add\_columns\[0;34m(self, cols, print\_stats, on\_error)\[0m
\[1;32m 730\[0m plan\[38;5;241m.\[39mopen()
\[1;32m 731\[0m \[38;5;28;01mtry\[39;00m:
\[0;32m--> 732\[0m excs\_per\_col \[38;5;241m=\[39m \[38;5;28;43mself\[39;49m\[38;5;241;43m.\[39;49m\[43mstore\_tbl\[49m\[38;5;241;43m.\[39;49m\[43mload\_column\[49m\[43m(\[49m\[43mcol\[49m\[43m,\[49m\[43m \[49m\[43mplan\[49m\[43m,\[49m\[43m \[49m\[43mon\_error\[49m\[43m \[49m\[38;5;241;43m==\[39;49m\[43m \[49m\[38;5;124;43m'\[39;49m\[38;5;124;43mabort\[39;49m\[38;5;124;43m'\[39;49m\[43m)\[49m
\[1;32m 733\[0m \[38;5;28;01mexcept\[39;00m sql\_exc\[38;5;241m.\[39mDBAPIError \[38;5;28;01mas\[39;00m exc:
\[1;32m 734\[0m Catalog\[38;5;241m.\[39mget()\[38;5;241m.\[39mconvert\_sql\_exc(exc, \[38;5;28mself\[39m\[38;5;241m.\[39mid, \[38;5;28mself\[39m\[38;5;241m.\[39mhandle, convert\_db\_excs\[38;5;241m=\[39m\[38;5;28;01mTrue\[39;00m)
File \[0;32m\~/Dropbox/workspace/pixeltable/pixeltable/pixeltable/store.py:247\[0m, in \[0;36mStoreBase.load\_column\[0;34m(self, col, exec\_plan, abort\_on\_exc)\[0m
\[1;32m 245\[0m \[38;5;28;01mif\[39;00m abort\_on\_exc \[38;5;129;01mand\[39;00m row\[38;5;241m.\[39mhas\_exc():
\[1;32m 246\[0m exc \[38;5;241m=\[39m row\[38;5;241m.\[39mget\_first\_exc()
\[0;32m--> 247\[0m \[38;5;28;01mraise\[39;00m excs\[38;5;241m.\[39mError(\[38;5;124mf\[39m\[38;5;124m'\[39m\[38;5;124mError while evaluating computed column \[39m\[38;5;132;01m\{\[39;00mcol\[38;5;241m.\[39mname\[38;5;132;01m!r}\[39;00m\[38;5;124m:\[39m\[38;5;130;01m\n\[39;00m\[38;5;132;01m\{\[39;00mexc\[38;5;132;01m}\[39;00m\[38;5;124m'\[39m) \[38;5;28;01mfrom\[39;00m \[38;5;21;01mexc\[39;00m
\[1;32m 248\[0m table\_row, num\_row\_exc \[38;5;241m=\[39m row\_builder\[38;5;241m.\[39mcreate\_store\_table\_row(row, \[38;5;28;01mNone\[39;00m, row\[38;5;241m.\[39mpk)
\[1;32m 249\[0m num\_excs \[38;5;241m+\[39m\[38;5;241m=\[39m num\_row\_exc
\[0;31mError\[0m: Error while evaluating computed column 'channel':
band index out of range
But if we use `on_error='ignore'`, the exception will be logged in the
column properties instead.
```python theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
t.add_computed_column(channel=t.image.getchannel(1), on_error='ignore')
```
Added 50 column values with 50 errors.
50 rows updated, 50 values computed, 50 exceptions.
Notice that the update status informs us that there were 50 errors. If
we query the table, we see that the column contains only `None` values,
but the `errortype` and `errormsg` fields contain details of the error.
```python theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
t.select(
t.image, t.channel, t.channel.errortype, t.channel.errormsg
).head(5)
```
More details on Pixeltable’s error handling can be found in the
[External Files](/platform/external-files)
guide.
## The Pixeltable Type System
We’ve seen that every column and every expression in Pixeltable has an
associated **Pixeltable type**. In this section, we’ll briefly survey
the various Pixeltable types and their uses.
Here are all the supported types and their corresponding Python types:
The Python type is what you’ll get back if you query an expression of
the given Pixeltable type. For `pxt.Json`, it can be any of `str`,
`int`, `float`, `bool`, `list`, or `dict`.
pxt.Audio, pxt.Video,
and pxt.Document all correspond to the Python type
str. This is because those types are represented by file
paths that reference the media in question. When you query for, say,
t.select(t.video\_col), you’re guaranteed to get a file path
on the local filesystem (Pixeltable will download and cache a
local copy of the video if necessary to ensure this). If you want the
original URL, use t.video\_col.fileurl instead.
Several types can be **specialized** to constrain the allowable data in
a column.
* `pxt.Image` can be specialized with a resolution and/or an image mode:
* `pxt.Image[(300,200)]` - images with width 300 and height 200
* `pxt.Image['RGB']` - images with mode `'RGB'`; see the [PIL
Documentation](https://pillow.readthedocs.io/en/stable/handbook/concepts.html)
for the full list
* `pxt.Image[(300,200), 'RGB']` - combines the above constraints
* `pxt.Array` can be specialized with a shape and/or a dtype:
* `pxt.Array[pxt.Float]` - arrays with dtype `pxt.Float`
* `pxt.Array[(64,64,3), pxt.Float]` - 3-dimensional arrays with dtype
`pxt.Float` and 64x64x3 shape
If we look at the structure of our table now, we see examples of
specialized image and array types.
```python theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
t
```
`t.clip` has type `pxt.Array[(512,), pxt.Float]`, since the output of
the embedding is always a 1x512 array. `t.channel` has type
`Image['L']`, since it’s always an `'L'` mode (1-channel) image.
You can freely use pxt.Image by
itself to mean “any image, without constraints”, but numerical arrays
must always specify a shape and a dtype; pxt.Array by
itself will raise an error.
Array shapes follow standard numpy conventions: a
shape is a tuple of integers, such as (512,) or
(64,64,3). A None may be used in place of an
integer to indicate an unconstrained size for that dimension, as in
(None,None,3) (3-dimensional array with two unconstrained
dimensions), or simply (None,) (unconstrained 1-dimensional
array).
## Ship it as an app
Once the table exists, `select()` is the same in an app. To create this
table from a file, put the columns on a `TableModel` (CSV/HF import
stays `create_table(..., source=)` in notebooks):
```python theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
import pixeltable as pxt
TableModel = pxt.model_base()
class Mnist(TableModel, name='mnist'):
image: pxt.Image
label: pxt.Int
```
```bash theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
pxt schema update app.py demo
```
# Tables and Data Operations
Source: https://docs.pixeltable.com/tutorials/tables-and-data-operations
![Open in Kaggle]()
![Open in Colab]()
![Download Notebook]()
This documentation page is also available as an interactive notebook. You can launch the notebook in
Kaggle or Colab, or download it for use with an IDE or local Jupyter installation, by clicking one of the
above links.
This notebook uses `pxt.create_table()` so you can run cells without a
project. An app puts the same columns on a `TableModel` in `app.py` and
creates the tables with `pxt schema update`.
This guide shows you how to:
* Create and manage tables: Understand Pixeltable’s table structure,
create and modify tables, and work with table schemas
* Manipulate data: Insert, update, and delete data within tables, and
retrieve data from tables into Python variables
* Filter and select data: Use `where()`, `select()`, and `order_by()` to
query for specific rows and columns
* Import data from CSV files and other file types
First, let’s ensure the Pixeltable library is installed in your
environment.
```python theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
%pip install -qU pixeltable
```
### Tables
All data in Pixeltable is stored in tables. At a high level, a
Pixeltable table behaves similarly to an ordinary SQL database table,
but with many additional capabilities to support complex AI workflows.
We’ll introduce those advanced capabilities gradually throughout this
tutorial; in this section, the focus is on basic table and data
operations.
Tables in Pixeltable are grouped into **directories**, which are simply
user-defined namespaces. The following command creates a new directory,
`fundamentals`, which we’ll use to store the tables in our tutorial.
```python theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
import pixeltable as pxt
# First we delete the `fundamentals` directory and all its contents (if
# it exists), in order to ensure a clean environment for the tutorial.
pxt.drop_dir('fundamentals', force=True)
# Now we create the directory.
pxt.create_dir('fundamentals')
```
Connected to Pixeltable database at: postgresql+psycopg://postgres:@/pixeltable?host=/Users/asiegel/.pixeltable/pgdata
Created directory 'fundamentals'.
Now let’s create our first table. To create a table, we must give it a
name and a **schema** that describes the table structure. Note that
prefacing the name with `fundamentals` causes it to be placed in our
newly-created directory.
```python theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
films_t = pxt.create_table(
'fundamentals/films',
{'film_name': pxt.String, 'year': pxt.Int, 'revenue': pxt.Float},
)
```
Created table 'films'.
To insert data into a table, we use the `insert()` method, passing it a
list of Python dicts.
```python theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
films_t.insert(
[
{'film_name': 'Jurassic Park', 'year': 1993, 'revenue': 1037.5},
{'film_name': 'Titanic', 'year': 1997, 'revenue': 2257.8},
{
'film_name': 'Avengers: Endgame',
'year': 2019,
'revenue': 2797.5,
},
]
)
```
Inserting rows into \`films\`: 3 rows \[00:00, 572.84 rows/s]
Inserted 3 rows with 0 errors.
3 rows inserted, 3 values computed.
If you’re inserting just a single row, you can use an alternate syntax
that is sometimes more convenient.
```python theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
films_t.insert(
[{'film_name': 'Inside Out 2', 'year': 2024, 'revenue': 1462.7}]
)
```
Inserting rows into \`films\`: 1 rows \[00:00, 318.76 rows/s]
Inserted 1 row with 0 errors.
1 row inserted, 1 value computed.
We can peek at the data in our table with the `collect()` method, which
retrieves all the rows in the table.
```python theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
films_t.collect()
```
Pixeltable also provides `update()` and `delete()` methods for modifying
and removing data from a table; we’ll see examples of them shortly.
### Filtering and Selecting Data
Often you want to select only certain rows and/or certain columns in a
table. You can do this with the `where()` and `select()` methods.
```python theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
films_t.where(films_t.revenue >= 2000.0).collect()
```
```python theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
films_t.select(films_t.film_name, films_t.year).collect()
```
Note the expressions that appear inside the calls to `where()` and
`select()`, such as `films_t.year`. These are **column references** that
point to specific columns within a table. In place of `films_t.year`,
you can also use dictionary syntax and type `films_t['year']`, which
means exactly the same thing but is sometimes more convenient.
```python theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
films_t.select(films_t['film_name'], films_t['year']).collect()
```
In addition to selecting columns directly, you can use column references
inside various kinds of expressions. For example, our `revenue` numbers
are given in millions of dollars. Let’s say we wanted to select revenue
in thousands of dollars instead; we could do that as follows:
```python theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
films_t.select(films_t.film_name, films_t.revenue * 1000).collect()
```
Note that since we selected an abstract expression rather than a
specific column, Pixeltable gave it the generic name `col_1`. You can
assign it a more informative name with Python keyword syntax:
```python theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
films_t.select(
films_t.film_name, revenue_thousands=films_t.revenue * 1000
).collect()
```
### Tables are Persistent
This is a good time to mention a few key differences between Pixeltable
tables and other familiar datastructures, such as Python dicts or Pandas
dataframes.
First, **Pixeltable is persistent. Unlike in-memory Python libraries
such as Pandas, Pixeltable is a database**. When you reset a notebook
kernel or start a new Python session, you’ll have access to all the data
you’ve stored previously in Pixeltable. Let’s demonstrate this by using
the IPython `%reset -f` command to clear out all our notebook variables,
so that `films_t` is no longer defined.
```python theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
%reset -f
films_t.collect() # Throws an exception now
```
NameError: name 'films\_t' is not defined
\[0;31m---------------------------------------------------------------------------\[0m
\[0;31mNameError\[0m Traceback (most recent call last)
Cell \[0;32mIn\[11], line 2\[0m
\[1;32m 1\[0m get\_ipython()\[38;5;241m.\[39mrun\_line\_magic(\[38;5;124m'\[39m\[38;5;124mreset\[39m\[38;5;124m'\[39m, \[38;5;124m'\[39m\[38;5;124m-f\[39m\[38;5;124m'\[39m)
\[0;32m----> 2\[0m \[43mfilms\_t\[49m\[38;5;241m.\[39mcollect() \[38;5;66;03m# Throws an exception now\[39;00m
\[0;31mNameError\[0m: name 'films\_t' is not defined
The `films_t` variable (along with all other variables in our Python
session) has been cleared out - but that’s ok, because it wasn’t the
source of record for our data. The `films_t` variable is just a
reference to the underlying database table. We can recover it with the
`get_table` command, referencing the `films` table by name.
```python theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
import pixeltable as pxt
films_t = pxt.get_table('fundamentals/films')
films_t.collect()
```
You can always get a list of existing tables with the Pixeltable
`pxt.ls()` command. Let’s use it to see the contents of the
`fundamentals` directory.
```python theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
pxt.ls(path='fundamentals')
```
Note that if you’re running Pixeltable on colab
or kaggle, the database will persist only for as long as your
colab/kaggle session remains active. If you’re running it locally or on
your own server, then your database will persist indefinitely (until you
actively delete it).
### Tables are Typed
The second major difference is that **Pixeltable is strongly typed**.
Because Pixeltable is a database, every column has a data type: that’s
why we specified `String`, `Int`, and `Float` for the three columns when
we created the table. These **type specifiers** are *mandatory* when
creating tables, and they become part of the table schema. You can
always see the table schema with the `describe()` method.
```python theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
films_t.describe()
```
In a notebook, you can also just type `films_t` to see the schema; its
output is identical to `films_t.describe()`.
```python theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
films_t
```
In addition to String,
Int, and Float, Pixeltable provides several
additional data types:
Bool, whose values are True or
False; Array for
numerical arrays; Json, for
lists or dicts that correspond to valid JSON structures;
and The media types
Image, Video, Audio, and
Document.
We’ll see examples of each of these types later in this guide.
Besides the column names and types, there’s a third element to the
schema, `Computed With`. To learn more about this, see the [Computed
Columns](/tutorials/computed-columns) guide.
All of the methods we’ve discussed so far, such as `insert()` and
`get_table()`, are documented in the [Pixeltable
SDK](/sdk/latest/) Documentation. The
following pages are particularly relevant:
* [pixeltable](/sdk/latest/pixeltable)
package reference
* [pxt.Table](/sdk/latest/table) class
reference
### A Real-World Example: Earthquake Data
Now let’s dive a little deeper into Pixeltable’s data operations. To
showcase all the features, it’ll be helpful to have a real-world
dataset, rather than our toy dataset with four movies. The dataset we’ll
be using consists of Earthquake data drawn from the US Geological
Survey: all recorded Earthquakes that occurred within 100 km of Seattle,
Washington, between January 1, 2023 and June 30, 2024.
The dataset is in CSV format, and we can load it into Pixeltable by
using `create_table()` with the `source` parameter, which creates a new
Pixeltable table from the contents of a CSV file.
```python theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
eq_t = pxt.create_table(
'fundamentals/earthquakes', # Name for the new table
source='https://raw.githubusercontent.com/pixeltable/pixeltable/release/docs/resources/earthquakes.csv',
primary_key='id', # Column 'id' is the primary key
schema_overrides={
'timestamp': pxt.Timestamp
}, # Interpret column 3 as a timestamp
)
```
Created table 'earthquakes'.
Inserting rows into \`earthquakes\`: 1823 rows \[00:00, 19554.24 rows/s]
Inserted 1823 rows with 0 errors.
In Pixeltable, you can always import external
data by giving a URL instead of a local file path. This applies to CSV
datasets, media files (such images and video), and other types of
content. The URL will often be an http\:// URL, but it can
also be an s3:// URL referencing an S3 bucket.
Pixeltable’s create\_table() function
with the source parameter can import data from various
formats including CSV, Excel, and Hugging Face datasets. You can also
use source to import from a Pandas dataframe. For more
details, see the
pixeltable.io
package reference.
Let’s have a peek at our new dataset. The dataset contains 1823 rows,
and we probably don’t want to display them all at once. We can limit our
query to fewer rows with the `limit()` method.
```python theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
eq_t.limit(5).collect()
```
A different way of achieving something similar is to use the `head()`
and `tail()` methods. Pixeltable keeps track of the insertion order of
all its data, and `head()` and `tail()` will always return the *earliest
inserted* and *most recently inserted* rows in a table, respectively.
```python theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
eq_t.head(5)
```
```python theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
eq_t.tail(5)
```
head(n) and
limit(n).collect() appear similar in this example. But
head() always returns the earliest rows in a table,
whereas limit() makes no promises about the ordering of its
results (unless you specify an order\_by() clause - more on
this below).
Let’s also peek at the schema:
```python theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
eq_t.describe()
```
Note that while specifying a schema is mandatory when *creating* a
table, it’s not always required when *importing* data. This is because
Pixeltable uses the structure of the imported data to infer the column
types, when feasible. You can always override the inferred column types
with the `schema_overrides` parameter of `import_csv()`.
The following examples showcase some common data operations.
```python theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
eq_t.count() # Number of rows in the table
```
1823
```python theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
# 5 highest-magnitude earthquakes
eq_t.order_by(eq_t.magnitude, asc=False).limit(5).collect()
```
```python theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
from datetime import datetime
# 5 highest-magnitude earthquakes in Q3 2023
eq_t.where(
(eq_t.timestamp >= datetime(2023, 6, 1))
& (eq_t.timestamp < datetime(2023, 10, 1))
).order_by(eq_t.magnitude, asc=False).limit(5).collect()
```
Note that Pixeltable uses Pandas-like operators for filtering data: the
expression
```python theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
(eq_t.timestamp >= datetime(2023, 6, 1)) & (eq_t.timestamp < datetime(2023, 10, 1))
```
means *both* conditions must be true; similarly (say),
```python theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
(eq_t.timestamp < datetime(2023, 6, 1)) | (eq_t.timestamp >= datetime(2023, 10, 1))
```
would mean *either* condition must be true.
You can also use the special `isin` operator to select just those values
that appear within a particular list:
```python theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
# Earthquakes with specific ids
eq_t.where(eq_t.id.isin([123, 456, 789])).collect()
```
In addition to basic operators like `>=` and `isin`, a Pixeltable
`where` clause can also contain more complex operations. For example,
the `location` column in our dataset is a string that contains a lot of
information, but in a relatively unstructured way. Suppose we wanted to
see all Earthquakes in the vicinity of Rainier, Washington; one way to
do this is with the `contains()` method:
```python theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
# All earthquakes in the vicinity of Rainier
eq_t.where(eq_t.location.contains('Rainier')).collect()
```
Pixeltable also supports various **aggregators**; here’s an example
showcasing two fairly simple ones, `max()` and `min()`:
```python theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
# Min and max ids
eq_t.select(
min=pxt.functions.min(eq_t.id), max=pxt.functions.max(eq_t.id)
).collect()
```
To learn more about Pixeltable functions and expressions, see the
[Computed
Columns](/tutorials/computed-columns) guide.
They’re also exhaustively documented in the [Pixeltable SDK
Documentation](/sdk/latest).
### Extracting Data from Tables into Python/Pandas
Sometimes it’s handy to pull out data from a table into a Python object.
We’ve actually already done this; the call to `collect()` returns an
in-memory result set, which we can then dereference in various ways. For
example:
```python theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
result = eq_t.limit(5).collect()
result[0] # Get the first row of the results as a dict
```
\{'id': 0,
'magnitude': 1.15,
'location': '10 km NW of Belfair, Washington',
'timestamp': datetime.datetime(2023, 1, 1, 8, 10, 37, 50000, tzinfo=zoneinfo.ZoneInfo(key='America/Los\_Angeles')),
'longitude': -122.93,
'latitude': 47.51}
```python theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
result[
'timestamp'
] # Get a list of the `timestamp` field of all the rows that were queried
```
\[datetime.datetime(2023, 1, 1, 8, 10, 37, 50000, tzinfo=zoneinfo.ZoneInfo(key='America/Los\_Angeles')),
datetime.datetime(2023, 1, 2, 1, 2, 43, 950000, tzinfo=zoneinfo.ZoneInfo(key='America/Los\_Angeles')),
datetime.datetime(2023, 1, 2, 12, 5, 1, 420000, tzinfo=zoneinfo.ZoneInfo(key='America/Los\_Angeles')),
datetime.datetime(2023, 1, 2, 12, 45, 14, 220000, tzinfo=zoneinfo.ZoneInfo(key='America/Los\_Angeles')),
datetime.datetime(2023, 1, 2, 13, 19, 27, 200000, tzinfo=zoneinfo.ZoneInfo(key='America/Los\_Angeles'))]
```python theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
df = result.to_pandas() # Convert the result set into a Pandas dataframe
df['magnitude'].describe()
```
count 5.000000
mean 0.744000
std 0.587988
min 0.200000
25% 0.290000
50% 0.520000
75% 1.150000
max 1.560000
Name: magnitude, dtype: float64
`collect()` without a preceding `limit()` returns the entire contents of
a query or table. Be careful! For very large tables, this could result
in out-of-memory errors. In this example, the 1823 rows in the table fit
comfortably into a dataframe.
```python theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
df = eq_t.collect().to_pandas()
df['magnitude'].describe()
```
count 1823.000000
mean 0.900378
std 0.625492
min -0.830000
25% 0.420000
50% 0.850000
75% 1.310000
max 4.300000
Name: magnitude, dtype: float64
### Adding Columns
Like other database tables, Pixeltable tables aren’t fixed entities:
they’re meant to evolve over time. Suppose we want to add a new column
to hold user-specified comments about particular earthquake events. We
can do this with the `add_column()` method:
```python theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
eq_t.add_column(note=(pxt.String | None))
```
Added 1823 column values with 0 errors.
1823 rows updated, 1823 values computed.
Here, `note` is the column name, and `pxt.String | None` specifies the
type of the new column. Note that the new column *must* be nullable
(`| None`) so that Pixeltable can fill in empty values for existing
rows.
```python theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
eq_t.add_column(contact_email=(pxt.String | None))
```
Added 1823 column values with 0 errors.
1823 rows updated, 1823 values computed.
Let’s have a look at the revised schema.
```python theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
eq_t.describe()
```
### Updating Rows in a Table
Table rows can be modified and deleted with the SQL-like `update()` and
`delete()` commands.
```python theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
# Add a comment to records with IDs 123 and 127
(
eq_t.where(eq_t.id.isin([121, 123])).update(
{
'note': 'Still investigating.',
'contact_email': 'contact@pixeltable.com',
}
)
)
```
Inserting rows into \`earthquakes\`: 2 rows \[00:00, 366.84 rows/s]
2 rows updated, 4 values computed.
```python theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
eq_t.where(eq_t.id >= 120).select(
eq_t.id, eq_t.magnitude, eq_t.note, eq_t.contact_email
).head(5)
```
`update()` can also accept an expression, rather than a constant value.
For example, suppose we wanted to shorten the location strings by
replacing every occurrence of `Washington` with `WA`. One way to do this
is with an `update()` clause, using a Pixeltable expression with the
`replace()` method.
```python theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
eq_t.update({'location': eq_t.location.replace('Washington', 'WA')})
```
Inserting rows into \`earthquakes\`: 1823 rows \[00:00, 21494.07 rows/s]
1823 rows updated, 1823 values computed.
```python theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
eq_t.head(5)
```
Notice that in all cases, the `update()` clause takes a Python
dictionary, but its values can be either constants such as
`'contact@pixeltable.com'`, or more complex expressions such as
`eq_t.location.replace('Washington', 'WA')`. Also notice that if
`update()` appears without a `where()` clause, then every row in the
table will be updated, as in the preceding example.
### Batch Updates
The `batch_update()` method provides an alternative way to update
multiple rows with different values. With a `batch_update()`, the
contents of each row are specified by individual `dict`s, rather than
according to a formula. Here’s a toy example that shows `batch_update()`
in action.
```python theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
updates = [
{'id': 500, 'note': 'This is an example note.'},
{'id': 501, 'note': 'This is a different note.'},
{'id': 502, 'note': 'A third note, unrelated to the others.'},
]
eq_t.batch_update(updates)
```
Inserting rows into \`earthquakes\`: 3 rows \[00:00, 984.58 rows/s]
3 rows updated, 3 values computed.
```python theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
eq_t.where(eq_t.id >= 500).select(
eq_t.id, eq_t.magnitude, eq_t.note, eq_t.contact_email
).head(5)
```
### Deleting Rows
To delete rows from a table, use the `delete()` method.
```python theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
# Delete all rows in 2024
eq_t.where(eq_t.timestamp >= datetime(2024, 1, 1)).delete()
```
587 rows deleted.
```python theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
eq_t.count() # How many are left after deleting?
```
1236
Don’t forget to specify a `where()` clause when using `delete()`! If you
run `delete()` without a `where()` clause, the entire contents of the
table will be deleted.
```python theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
eq_t.delete()
```
1236 rows deleted.
```python theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
eq_t.count()
```
0
### Table Versioning
Every table in Pixeltable is versioned: some or all of its modification
history is preserved. We’ve seen a reference to this already; `pxt.ls()`
will show the most recent version along with each table it lists.
```python theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
pxt.ls('fundamentals')
```
To see the version history of a particular table:
```python theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
eq_t.history()
```
If you ever make a mistake, you can always call `revert()` to undo the
most recent change to a table and roll back to the previous version.
Let’s try it out: we’ll use it to revert the successive `delete()` calls
that we just executed.
```python theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
eq_t.revert()
```
```python theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
eq_t.count()
```
1236
```python theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
eq_t.revert()
```
```python theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
eq_t.count()
```
1823
Be aware: calling revert() cannot
be undone!
### Multimodal Data
In addition to the structured data we’ve been exploring so far,
Pixeltable has native support for **media types**: images, video, audio,
and unstructured documents such as pdfs. Media support is one of
Pixeltable’s core capabilities. Here’s an example showing how media data
lives side-by-side with structured data in Pixeltable.
```python theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
# Add a new column of type `Image`
eq_t.add_column(map_image=(pxt.Image | None))
eq_t.describe()
```
Added 1823 column values with 0 errors.
```python theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
# Update the row with id == 1002, adding an image to the `map_image` column
eq_t.where(eq_t.id == 1002).update(
{
'map_image': 'https://raw.githubusercontent.com/pixeltable/pixeltable/release/docs/resources/port-townsend-map.jpeg'
}
)
```
Inserting rows into \`earthquakes\`: 1 rows \[00:00, 192.79 rows/s]
1 row updated, 1 value computed.
Note that in Pixeltable, you can always insert
images into a table by giving the file path or URL of the image (as a
string). It’s not necessary to load the image first; Pixeltable will
manage the loading and caching of images in the background. The same
applies to other media data such as documents and videos.
Pixeltable will also embed image thumbnails in your notebook when you do
a query:
```python theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
eq_t.where(eq_t.id >= 1000).select(
eq_t.id, eq_t.magnitude, eq_t.location, eq_t.map_image
).head(5)
```
### Directory Hierarchies
So far we’ve only seen an example of a single directory with a table
inside it, but one can also put directories inside other directories, in
whatever fashion makes the most sense for a given application.
```python theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
pxt.create_dir('fundamentals/subdir')
pxt.create_dir('fundamentals/subdir/subsubdir')
pxt.create_table(
'fundamentals/subdir/subsubdir/my_table', {'my_col': pxt.String}
)
```
Created directory 'fundamentals/subdir'.
Created directory 'fundamentals/subdir/subsubdir'.
Created table 'my\_table'.
### Deleting Columns, Tables, and Directories
`drop_column()`, `drop_table()`, and `drop_dir()` are used to delete
columns, tables, and directories, respectively.
```python theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
# Delete the `contact_email` column
eq_t.drop_column('contact_email')
```
```python theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
eq_t.describe()
```
```python theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
# Delete the entire table (cannot be reverted!)
pxt.drop_table('fundamentals/earthquakes')
```
```python theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
# Delete the entire directory and all its contents, including any nested
# subdirectories (cannot be reverted)
pxt.drop_dir('fundamentals', force=True)
```
## Next Steps
Learn more about working with Pixeltable:
* [Computed
Columns](/tutorials/computed-columns)
* [Queries and
Expressions](/tutorials/queries-and-expressions)
## Ship it as an app
This notebook used `create_table` so you can run cells without a
project. Columns you added later with `add_column` belong on the class
only after you write those fields. Starting schema in `app.py`:
```python theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
import pixeltable as pxt
TableModel = pxt.model_base()
class Films(TableModel, name='films'):
film_name: pxt.String
year: pxt.Int
revenue: pxt.Float
```
```bash theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
pxt schema update app.py fundamentals
```
# Agentic workflows
Source: https://docs.pixeltable.com/use-cases/agentic-workflows
Insert a message. Tool calls run as columns.
Insert a message. The model picks a tool, `invoke_tools` runs it, and both results are columns on that row.
Create the tables with `pxt schema update app.py my_app`, then open them with `t = pxt.get_table('my_app.assistant')`. Cookbooks on this topic use `pxt.create_table()` so you can run cells without a project. An app puts the same columns on a `TableModel` in `app.py` and creates the tables with `pxt schema update`.
```python theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
import pixeltable as pxt
import pixeltable.functions as pxtf
from pixeltable.serving import FastAPIRouter
TableModel = pxt.model_base()
@pxt.udf
def get_weather(city: str) -> str:
return f'Weather in {city}: 72°F, sunny'
tools = pxt.tools(get_weather)
class Assistant(TableModel, name='assistant'):
message: pxt.String
response = pxtf.openai.chat_completions(
messages=[{'role': 'user', 'content': message}],
model='gpt-4o-mini',
tools=tools,
)
tool_output = pxtf.openai.invoke_tools(tools, response)
ask = FastAPIRouter(name='ask')
ask.add_insert_route(
Assistant, path='/ask', inputs=[Assistant.message], outputs=[Assistant.tool_output]
)
```
```bash theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
pxt schema update app.py my_app
pxt service update app.py my_app
```
```python theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
assistant = pxt.get_table('my_app.assistant')
assistant.insert([{'message': 'Weather in Paris?'}])
assistant.select(assistant.message, assistant.tool_output).collect()
```
`pxt service list` prints the URL. POST `/ask` with `{"message": "..."}`. Chat history is another table with an embedding index: [Agent memory](/howto/cookbooks/agents/pattern-agent-memory). MCP tools load with `pxt.mcp_udfs(...)` and go into the same `pxt.tools()` list.
A full chat app: [`uvx pixeltable-new`](https://github.com/pixeltable/pixeltable-new) copies the [starter kit](/resources/starter-kit) agent. Pass `agent` as the last argument to `pxt schema update`. `/ask` needs `ANTHROPIC_API_KEY`.
On Cloud, sign in with `pxt login`, or set `PIXELTABLE_API_KEY`. `pxt org create` provisions `main`. Then run `pxt db update`, `pxt schema update`, and `pxt service update` against `pxt://org:main`. [Pixeltable Cloud](/cloud).
Search over chunks instead of tools: [RAG and live APIs](/use-cases/multimodal-backend). Coming from LangGraph: [Migrate](/howto/coming-from#langgraph).
Register tools and run `invoke_tools` as a column.
Patterns for agents as tables.
Search over chunks instead of tools.
POST `/ask` from the same file.
# Media processing
Source: https://docs.pixeltable.com/use-cases/media-processing
Insert a video. Frames are rows. Export when the dataset is ready.
Insert media. Iterators explode it into rows. Computed columns run on the new rows. Export when you are ready to train.
Create the tables with `pxt schema update app.py my_app`, then open them with `t = pxt.get_table('my_app.videos')`. Cookbooks on this topic use `pxt.create_table()` so you can run cells without a project. An app puts the same columns on a `TableModel` in `app.py` and creates the tables with `pxt schema update`.
```python theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
import pixeltable as pxt
import pixeltable.functions as pxtf
TableModel = pxt.model_base()
class Videos(TableModel, name='videos'):
video: pxt.Video
class Frames(
TableModel,
name='frames',
base=Videos,
iterator=pxtf.video.frame_iterator(Videos.video, fps=1),
):
thumb = frame.resize((256, 256)) # type: ignore[name-defined] # iterator output
```
```bash theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
pxt schema update app.py my_app
```
```python theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
videos = pxt.get_table('my_app.videos')
videos.insert([{'video': 'demo.mp4'}])
frames = pxt.get_table('my_app.frames')
frames.select(frames.thumb).head(10)
```
Add Whisper, YOLOX, and other models from the cookbooks as computed columns on these tables. When the rows look ready for training, export with `to_pytorch_dataset()`. Skip `pxt service update` on this path. [Self-hosting](/howto/deployment/overview).
A video search app with endpoints: `uvx pixeltable-new myapp --video`, then pass `videointel` to `pxt schema update`. Live search over chunks: [RAG and live APIs](/use-cases/multimodal-backend).
Frames as rows from a `pxt.Video` column.
Transcripts as computed columns.
`to_pytorch_dataset()` when the rows are ready.
# Multimodal backend
Source: https://docs.pixeltable.com/use-cases/multimodal-backend
Insert a document. Chunks are rows. POST search.
Insert a document. The iterator explodes it into chunks. The embedding index stays current. `pxt service update` starts the insert and search endpoints.
Create the tables with `pxt schema update app.py my_app`, then open them with `t = pxt.get_table('my_app.docs')`. Cookbooks on this topic use `pxt.create_table()` so you can run cells without a project. An app puts the same columns on a `TableModel` in `app.py` and creates the tables with `pxt schema update`.
```python theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
import pixeltable as pxt
import pixeltable.functions as pxtf
from pixeltable.serving import FastAPIRouter
TableModel = pxt.model_base()
embed = pxtf.huggingface.sentence_transformer.using(
model_id='sentence-transformers/all-MiniLM-L6-v2'
)
class Docs(TableModel, name='docs'):
document: pxt.Document
class Chunks(
TableModel,
name='chunks',
base=Docs,
iterator=pxtf.document.document_splitter(
Docs.document, separators='token_limit', limit=300
),
):
__indexes__ = [
pxt.EmbeddingIndex(text, embedding=embed, name='chunks_embed')
]
@pxt.query
def search_docs(query_text: str):
sim = Chunks.text.similarity(string=query_text)
return Chunks.order_by(sim, asc=False).limit(10).select(Chunks.text, sim)
api = FastAPIRouter(name='rag')
api.add_insert_route(
Docs,
path='/docs',
uploadfile_inputs=[Docs.document],
outputs=[Docs.document],
)
api.add_query_route(path='/search', query=search_docs, method='post')
```
```bash theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
pxt schema update app.py my_app
pxt service update app.py my_app
```
```python theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
docs = pxt.get_table('my_app.docs')
docs.insert([{'document': 'report.pdf'}])
```
`pxt service list` prints the URL. POST `/docs` to insert. POST `/search` with `{"query_text": "..."}` to rank chunks.
On Cloud, sign in with `pxt login`, or set `PIXELTABLE_API_KEY`. `pxt org create` provisions `main`. Then run `pxt db update`, `pxt schema update`, and `pxt service update` against `pxt://org:main`. [Pixeltable Cloud](/cloud).
Export those tables for training: [Datasets from media](/use-cases/media-processing). Tool calls as columns: [Agents](/use-cases/agentic-workflows).
POST `/docs` to insert. Chunks stay current.
Rank chunks with `{"query_text": "..."}`.
The same index pattern on images.
Insert and search routes from the same file.