Skip to main content
Open in Kaggle  Open in Colab  Download Notebook
This documentation page is also available as an interactive notebook. You can launch the notebook in Kaggle or Colab, or download it for use with an IDE or local Jupyter installation, by clicking one of the above links.
Main takeaways:
  • Indexing in Pixeltable is declarative
    • you create an index on a column and supply the embedding functions you want to use (for inserting data into the index as well as lookups)
    • Pixeltable maintains the index in response to any kind of update of the indexed table (i.e., insert()/update()/delete())
  • Perform index lookups with the similarity() pseudo-function, in combination with the order_by() and limit() clauses
To make this concrete, let’s create a table of images with the create_table() function. We’re also going to add some columns to demonstrate combining similarity search with other predicates. 
[notice] A new release of pip is available: 25.3 -> 26.0.1
[notice] To update, run: pip install —upgrade pip
Note: you may need to restart the kernel to use updated packages.
We start out by inserting 10 rows: 
Inserted 10 rows with 0 errors in 2.53 s (3.96 rows/s)
10 rows inserted.
For the sake of convenience, we’re storing the images as external URLs, which are cached transparently by Pixeltable. For details on working with external media files, see Working with External Files.

​
Creating an index

To create and populate an index, we call Table.add_embedding_index() and tell it which UDF or UDFs to use to create embeddings. That definition is persisted as part of the table’s metadata, which allows Pixeltable to maintain the index in response to updates to the table. Any embedding UDF can be used for the index. For this example, we’re going to use a CLIP model, which has built-in support in Pixeltable under the pixeltable.functions.huggingface package. As an alternative, you could use an online service such as OpenAI (see pixeltable.functions.openai), or create your own embedding UDF with custom code (we’ll see how to do this below). Because we’re adding an index to an image column, the UDF we specify must be able to handle images. In fact, CLIP models are multimodal: they can handle both text and images, which is useful for doing lookups against the index.
The first parameter of add_embedding_index() is the name of the column being indexed; the embed parameter specifies the relevant embedding. Notice the notation we used:
clip is a general-purpose UDF that can accept any CLIP model available in the Hugging Face model repository. To define an embedding, however, we need to provide a specific embedding function to add_embedding_index(): a function that is not parameterized on model_id. The .using(model_id=...) syntax tells Pixeltable to specialize the clip UDF by fixing the model_id parameter to the specific value 'openai/clip-vit-base-patch32'.
If you’re familiar with functional programming concepts, you might recognize .using() as a partial function operator. It’s a general operator that can be applied to any UDF (not just embedding functions), transforming a UDF with n parameters into one with k parameters by fixing the values of n-k of its arguments. Python has something similar in the functools package: the functools.partial() operator.
add_embedding_index() provides a few other optional parameters:
  • idx_name: optional name for the index, which needs to be unique for the table; a default name is created if this isn’t provided explicitly
  • metric: the metric to use to compute the similarity of two embedding vectors; one of:
    • 'cosine': cosine distance (default)
    • 'ip': inner product
    • 'l2': L2 distance
If desired, you can create multiple indexes on the same column, using different embedding functions. This can be useful to evaluate the effectiveness of different embedding functions side-by-side, or to use embedding functions tailored to specific use cases. In that case, you can provide explicit names for those indexes and then reference them during queries. We’ll illustrate that later with an example.

​
Using the index in queries

To take advantage of an embedding index when querying a table, we use the similarity() pseudo-function, which is invoked as a method on the indexed column, in combination with the order_by() and limit() clauses. First, we’ll get a sample image from the table:
We then call the similarity() pseudo-function as a method on the indexed column and apply order_by() and limit(). We used the default cosine distance when we created the index, so we’re going to order by descending similarity (order_by(..., asc=False)):
We can combine nearest-neighbor/similarity search with standard predicates. Here’s the same query, but filtering out the selected sample_img (which we already know has perfect similarity with itself):

​
Index updates

In Pixeltable, each index is kept up-to-date automatically in response to changes to the indexed table. To illustrate this, let’s insert a few more rows: 
Inserted 10 rows with 0 errors in 1.29 s (7.75 rows/s)
10 rows inserted.
When we now re-run the initial similarity query, we get a different result:

​
Similarity search on different types

Because CLIP models are multimodal, we can also do lookups by text.

​
Creating multiple indexes on a single column

We can create multiple embedding indexes on the same column, utilizing different embedding models. In order to use a specific index in a query, we need to assign it a name and then use that name in the query. To illustrate this, let’s create a table with text (taken from the Wikipedia article on Pablo Picasso): 
Inserted 10 rows with 0 errors in 0.03 s (301.25 rows/s)
10 rows inserted.
When calling add_embedding_index(), we now specify the index name (idx_name) directly. If it is not specified, Pixeltable will assign a name (such as idx0).
To do a similarity query, we now call similarity() with the idx parameter:

​
Using a UDF for a custom embedding

​
Indexing precomputed embeddings (Array columns)

If your data already contains precomputed embedding vectors — for example, embeddings exported from another system or generated by a custom pipeline — you can index them directly without specifying an embedding function. add_embedding_index supports Array columns in addition to String, Image, Video, and Audio columns. When the indexed column is an Array, Pixeltable treats its values as precomputed embeddings and builds the vector index directly from them. An embedding function is optional for Array columns. If provided, it will be used only to convert query values into embeddings during similarity search.
To run a similarity search, pass a raw vector to the similarity method: 
Error: similarity(vector=…): expected `numpy.ndarray`, or array `Expr`; got `list`
---------------------------------------------------------------------------
Error                                     Traceback (most recent call last)
Cell In[15], line 2
      1 query_vector = rng.random(384).tolist()
----> 2 sim = precomputed.embedding.similarity(vector=query_vector)
      3 precomputed.order_by(sim, asc=False).limit(2).select(precomputed.text, sim).collect()File /Users/pierre/pixeltable/.venv/lib/python3.12/site-packages/pixeltable/exprs/column_ref.py:313, in ColumnRef.similarity(self, item, string, image, audio, video, document, vector, idx)
    311 else:
    312     if not isinstance(vector, np.ndarray):
—> 313         raise excs.Error(
    314             f’similarity(vector=…): expected `numpy.ndarray`, or array `Expr`; ’
    315             f’got `{type(vector).__name__}`’
    316         )
    317     if vector.ndim != 1:
    318         raise excs.Error(f’similarity(vector=…): expected 1-dimensional array; got shape {vector.shape}’)Error: similarity(vector=…): expected `numpy.ndarray`, or array `Expr`; got `list`
If you want to search by text (or another modality) rather than a raw vector, provide an embedding function that converts the query value into the same vector space. The function is only used at query time — it does not re-embed the stored vectors.
The above examples show how to use any model in the Hugging Face CLIP or sentence_transformer model families, and essentially the same pattern can be used for any other embedding with built-in Pixeltable support, such as OpenAI embeddings. But what if you want to adapt a new model family that doesn’t have built-in support in Pixeltable? This can be done by writing a custom Pixeltable UDF. In the following example, we’ll write a simple UDF to use the BERT model built on TensorFlow. First we install the necessary dependencies.
Text embedding UDFs must always take a string as input, and return a 1-dimensional numpy array of fixed dimension (512 in the case of small_bert, the variant we’ll be using). If we were writing an image embedding UDF, the input would have type PIL.Image.Image rather than str. The UDF is straightforward, loading the model and evaluating it against the input, with a minor data conversion on either side of the model invocation.
Here’s the output of our sample query run against bert_idx.
Our example UDF is very simple, but it would perform poorly in a production setting. To make our UDF production-ready, we’d want to do two things:
  • Cache the model: the current version calls hub.load() on every UDF invocation. In a real application, we’d want to instantiate the model just once, then reuse it on subsequent UDF calls.
  • Batch our inputs: we’d use Pixeltable’s batching capability to ensure we’re making efficient use of the model. Batched UDFs are described in depth in the User-Defined Functions how-to guide.
You might have noticed that the updates to bert_idx seem sluggish; that’s why!

​
Deleting an index

To delete an index, call Table.drop_embedding_index():
  • specify the idx_name parameter if you have multiple indices
  • otherwise the column_name parameter is sufficient
Given that we have several embedding indices, we’ll specify which index to drop:
Last modified on June 24, 2026