Observability

This documentation page is also available as an interactive notebook. You can launch the notebook in Kaggle or Colab, or download it for use with an IDE or local Jupyter installation, by clicking one of the above links.

Pixeltable can export its internal telemetry as OpenTelemetry (OTel) spans and metrics, so you can watch operations in any OTel backend. This notebook wires the OTel bridge to Grafana Cloud and walks a small image table through its lifecycle: directory and table creation, computed columns, an insert with external media, an embedding index, schema evolution, DML, a version revert, and teardown. Every table operation roots its own trace, and the same operations feed a set of metric instruments (row/cell/UDF counters, a UDF latency histogram, media byte counters) covered in the Metrics section below. The richest trace is the insert; its span tree looks like this:

pixeltable.insert                        (operation span, root)
├─ pixeltable.data_source.prepare        (source resolution)
├─ pixeltable.catalog.begin_xact         (connection + lock acquisition, one per attempt)
├─ pixeltable.plan.create                (insert plan construction, DEBUG level)
├─ pixeltable.store.insert_rows          (plan execution and store writes)
│  ├─ pixeltable.media.fetch             (one per external file downloaded, DEBUG level)
│  ├─ pixeltable.row                     (one per inserted row, DEBUG level)
│  │  ├─ pixeltable.udf.resize           (each UDF call, nested under its row)
│  │  ├─ pixeltable.udf.convert
│  │  └─ pixeltable.udf.get_metadata
│  ├─ pixeltable.media.encode            (one per generated media file written to a temp file, DEBUG level)
│  ├─ pixeltable.media.save              (one per generated media file persisted, DEBUG level)
│  ├─ pixeltable.store.build_rows        (row -> store-row conversion, DEBUG level)
│  └─ pixeltable.sa.insert_rows          (the SQL INSERT)
├─ pixeltable.view_load                  (one per mutable view: its own pipeline, rows, store writes)
└─ pixeltable.catalog.write_tbl_md       (metadata commit)

Traces land in Grafana Cloud Traces (Tempo); metrics land in Grafana Cloud Metrics (Prometheus).

%pip install -qU 'pixeltable[otel]' sentence-transformers

Turn on the OpenTelemetry bridge

Telemetry is opt-in: call pxt_otel.init() once per process, before the first table operation. It configures itself from the database’s [[pixeltable.database]] entry in pixeltable.toml, from environment variables, or from the [otel] section of ~/.pixeltable/config.toml, in that order of precedence. Option 1: the database entry in pixeltable.toml. Put the endpoint on the [[pixeltable.database]] entry of the database (the entry with no name is the local database; see Configuration). The headers are a secret and stay out of the file: keep OTEL_EXPORTER_OTLP_HEADERS in the environment. On a hosted database, the collector in each pod sends the telemetry to this endpoint (see Telemetry from a hosted database). After you change the entry, run pxt db update and then pxt service update so that the pods receive the new endpoint. Set the headers there with pxt secret set pxt://org:db OTEL_EXPORTER_OTLP_HEADERS=..., then run pxt db restart and pxt service restart so that the running pods read them.

[[pixeltable.database]]
db_exporter_otlp_endpoint = 'https://otlp-gateway-<zone>.grafana.net/otlp'

Option 2: environment variables. Set these before starting the Jupyter kernel (the kernel inherits them):

For Grafana Cloud, the OpenTelemetry connection page (Connections -> Add new connection -> OpenTelemetry) shows your <zone>, <instance-id>, and lets you create a <service-account-token>. Then run these in the terminal, substituting your values, and start Jupyter from that same terminal:

export OTEL_EXPORTER_OTLP_ENDPOINT='https://otlp-gateway-<zone>.grafana.net/otlp'
export OTEL_EXPORTER_OTLP_HEADERS="Authorization=Basic $(echo -n '<instance-id>:<service-account-token>' | base64 | tr -d '\n')"

Option 3: ~/.pixeltable/config.toml, shared by every database on the installation:

[otel]
exporter_otlp_endpoint = 'https://otlp-gateway-<zone>.grafana.net/otlp'

init() runs once per process. If it ran without an endpoint earlier in this kernel it can’t be reconfigured, so fix the configuration, restart the kernel, and run this cell first.

import opentelemetry.instrumentation.pixeltable as pxt_otel
from opentelemetry import trace
from opentelemetry.sdk.trace import TracerProvider

pxt_otel.init(
    span_level='debug'
)  # 'debug' adds per-row, per-UDF, and per-file spans to the default 'info' level


# fail loud instead of silently dropping every span: a no-op ProxyTracerProvider means the SDK never came up
assert isinstance(trace.get_tracer_provider(), TracerProvider), (
    'OpenTelemetry is INERT: no OTLP endpoint was configured. '
    'Set the env vars (or config.toml) above, restart the kernel, and run this cell first.'
)
print('OTel bridge active')

Connected to Pixeltable database at: postgresql+psycopg://postgres:@/pixeltable?host=/private/tmp/pxt_otel_demo/pgdata
OTel bridge active

Build the pipeline

Everything here is traced: create_dir and create_table produce DDL traces, each add_computed_column call is its own trace, and create_view contains a pixeltable.op.LoadViewOp subtree that loads the (still empty) view. The computed columns are real work so the insert has something to trace: a stored thumbnail (persisted -> pixeltable.media.save spans), a grayscale conversion, and image metadata extraction. The view derives a rotated thumbnail from the base table. if_exists='replace_force' on the directory drops everything from a previous run.

import pixeltable as pxt

pxt.create_dir('otel_demo', if_exists='replace_force')

t = pxt.create_table(
    'otel_demo.images', {'img': pxt.Image, 'caption': pxt.String}
)
t.add_computed_column(thumb=t.img.resize((64, 64)))
t.add_computed_column(gray=t.img.convert('L'))
t.add_computed_column(meta=t.img.get_metadata())

v = pxt.create_view(
    'otel_demo.thumbs',
    t,
    additional_columns={'thumb_rot': t.thumb.rotate(90)},
)

Created directory ‘otel_demo’.
Created table ‘images’.
Added 0 column values with 0 errors in 0.00 s
Added 0 column values with 0 errors in 0.00 s
Added 0 column values with 0 errors in 0.00 s
Created view ‘otel_demo/thumbs’.

Insert

4 rows referencing external images (well under the 100-span-per-operation cap, so every row gets a span). Downloads, UDF evaluation, thumbnail persistence, the store write path, and the view reload all land under one pixeltable.insert trace, following the tree from the introduction. Note: pixeltable.media.fetch spans only appear when the files are not yet in the local file cache; on a re-run of this cell the downloads are skipped.

base = 'https://raw.githubusercontent.com/pixeltable/pixeltable/main/docs/resources/images'
rows = [
    {
        'img': f'{base}/000000000001.jpg',
        'caption': 'a green and cream taxi parked at the curb',
    },
    {
        'img': f'{base}/000000000016.jpg',
        'caption': 'a baseball player mid-swing',
    },
    {
        'img': f'{base}/000000000019.jpg',
        'caption': 'two cows lying in a grassy pasture',
    },
    {
        'img': f'{base}/000000000025.jpg',
        'caption': 'giraffes browsing leaves from a tree',
    },
]
t.insert(rows)

Inserted 8 rows with 0 errors in 0.49 s (16.27 rows/s)
8 rows inserted.

Embedding index

add_embedding_index backfills embeddings for the existing rows and builds the vector index, all in one trace: pixeltable.model.load (first use of the model in this process), batched pixeltable.udf.sentence_transformer calls, the backfill write (pixeltable.store.write_column), and pixeltable.store.create_index. Queries are traced too: collect() (along with head, tail, and count) roots its own trace, a pixeltable.collect span carrying the table names and row count, with a nested pixeltable.result_cursor.yield_rows span and, at 'debug', the per-row and per-UDF spans below it. The similarity search below produces its own trace alongside the index build.

from pixeltable.functions.huggingface import sentence_transformer

t.add_embedding_index(
    'caption',
    embedding=sentence_transformer.using(model_id='all-MiniLM-L6-v2'),
)
t.order_by(t.caption.similarity(string='a farm animal'), asc=False).limit(
    1
).select(t.caption).collect()

Evolve, mutate, roll back

Three more operations, one trace each:

add_computed_column on a table that already has data shows the backfill: a pixeltable.store.write_column span containing the evaluation pipeline.
update recomputes dependents of the changed column inside the same trace, including the embedding index (watch for another pixeltable.udf.sentence_transformer span).
revert rolls the update back; version cleanup shows up as pixeltable.catalog.revert_version and pixeltable.media.delete spans.

Other operations trace the same way (delete, rename_column, recompute_columns, batch_update, snapshots, drops, …); this notebook is a sample, not the full list.

t.add_computed_column(entropy=t.gray.entropy())
t.update({'caption': t.caption.upper()})
t.revert()
t.select(t.caption, t.entropy).collect()

Added 4 column values with 0 errors in 0.02 s (184.28 rows/s)

Metrics

The operations above didn’t just produce traces: the same bridge exports Pixeltable’s metric instruments through the OTLP endpoint. No setup is needed beyond init(); a PeriodicExportingMetricReader ships them every 60 seconds by default (set OTEL_METRIC_EXPORT_INTERVAL, in milliseconds, to change that).

This notebook’s lifecycle populated most of them: rows.written for the images table and the thumbs view, cells.computed, udf.calls and udf.latency for resize, convert, get_metadata, rotate, entropy, and the batched sentence_transformer, and both media byte counters (the insert downloaded four images and persisted the generated thumbnails). The error, retry, and token metrics are absent here because instruments are created lazily on first record; they appear when a computed column raises (with on_error='ignore'), when a rate-limited provider call is retried, or when a token-instrumented provider UDF (OpenAI chat_completions/responses, Anthropic messages, Gemini generate_content, or Ollama generate/chat) returns a response carrying usage information. Viewing in Grafana. Explore -> your Prometheus (Metrics) data source. OTLP metric names arrive translated to Prometheus conventions: dots become underscores, counters get a _total suffix, and time/byte units are spelled out, so pixeltable.udf.calls becomes pixeltable_udf_calls_total and pixeltable.udf.latency becomes pixeltable_udf_latency_seconds_bucket/_sum/_count. Dimensions arrive as labels (pxt_table, pxt_udf). Some queries to try:

# UDF invocations by function, last 15 minutes
increase(pixeltable_udf_calls_total[15m])

# p95 UDF latency by function
histogram_quantile(0.95, sum by (le, pxt_udf) (rate(pixeltable_udf_latency_seconds_bucket[15m])))

# rows written to the demo table
increase(pixeltable_rows_written_total{pxt_table="images"}[15m])

Counter panels and short-lived processes. These are cumulative counters, and rate()/increase() need at least two samples per series with growth between them. A process that runs one operation and exits exports a single sample, which those functions treat as the baseline: the panel reads 0 even though the data arrived. In a live kernel like this one the periodic exporter keeps shipping samples, so rate panels work after a couple of export intervals; for one-shot scripts, query the raw values instead, e.g. last_over_time(pixeltable_udf_calls_total[1h]).

Wait a moment for export

init() sets up a BatchSpanProcessor, which ships queued spans automatically about every 5 seconds while the kernel stays alive, and a PeriodicExportingMetricReader, which ships metrics every 60 seconds by default. Give it a minute after the last operation, then check Grafana. (In a short-lived script that exits immediately you would call trace.get_tracer_provider().force_flush() and metrics.get_meter_provider().force_flush() before exit, but in a live notebook kernel the timers handle it.)

View it in Grafana

Explore -> your Traces (Tempo) data source -> search Service Name pixeltable. This notebook produced one trace per operation: pixeltable.create_dir, pixeltable.create_table, pixeltable.add_computed_column (x4), pixeltable.create_view, pixeltable.insert, pixeltable.add_embedding_index, pixeltable.update, pixeltable.revert, pixeltable.drop_table (x2), and pixeltable.drop_dir, plus a pixeltable.collect trace for each of the two queries (the similarity search and the final select). The two showcase traces:

pixeltable.insert: the full tree from the introduction - parallel media.fetch downloads (on a cold cache), pixeltable.row spans with nested udf.resize/udf.convert/udf.get_metadata calls, media.save persistence, the store.build_rows/sa.insert_rows write path, and the view_load subtree with its own pipeline and rows.
pixeltable.add_embedding_index: model.load, batched udf.sentence_transformer calls, the backfill write, and store.create_index.

For more detail use span_level='trace'; to reduce volume on a large insert, use the default 'info', which keeps the operation spans and the INFO-level work spans (catalog.begin_xact, sa.insert_rows, sa.select, store.insert_rows, store.write_column, view_load, metadata writes) and drops the DEBUG spans (per-row, per-UDF, per-file, store.build_rows, and the finer-grained catalog and store spans).

Other backends

The same bridge exports to Langfuse, Phoenix, Logfire, SigNoz, and Opik. Each page points the exporter at that product. The Langfuse page also covers the Langfuse SDK and per-call LLM generations.

Teardown

Drops are traced too: each pixeltable.drop_table trace contains the pixeltable.op.* sequence (media cleanup, store table drop, metadata deletion) that executes the drop, and pixeltable.drop_dir closes out the demo. The view must drop before its base table.

pxt.drop_table('otel_demo.thumbs')
pxt.drop_table('otel_demo.images')
pxt.drop_dir('otel_demo')

Getting started

Tools

How to

Resources

Turn on the OpenTelemetry bridge

Build the pipeline

Insert

Embedding index

Evolve, mutate, roll back

Metrics

Wait a moment for export

View it in Grafana

Other backends

Teardown

​Turn on the OpenTelemetry bridge

​Build the pipeline

​Insert

​Embedding index

​Evolve, mutate, roll back

​Metrics

​Wait a moment for export

​View it in Grafana

​Other backends

​Teardown

Turn on the OpenTelemetry bridge

Build the pipeline

Insert

Embedding index

Evolve, mutate, roll back

Metrics

Wait a moment for export

View it in Grafana

Other backends

Teardown