This documentation page is also available as an interactive notebook. You can launch the notebook in
Kaggle or Colab, or download it for use with an IDE or local Jupyter installation, by clicking one of the
above links.
pixeltable.insert (operation span, root)
├─ pixeltable.data_source.prepare (source resolution + schema validation)
├─ pixeltable.catalog.begin_xact (connection + lock acquisition, one per attempt)
├─ pixeltable.plan.create (insert plan construction, DEBUG level)
├─ pixeltable.store.insert_rows (plan execution and store writes)
│ ├─ pixeltable.media.fetch (one per external file downloaded, DEBUG level)
│ ├─ pixeltable.row (one per inserted row, DEBUG level)
│ │ ├─ pixeltable.udf.resize (each UDF call, nested under its row)
│ │ ├─ pixeltable.udf.convert
│ │ └─ pixeltable.udf.get_metadata
│ ├─ pixeltable.media.encode (one per generated media file written to a temp file, DEBUG level)
│ ├─ pixeltable.media.save (one per generated media file persisted, DEBUG level)
│ ├─ pixeltable.store.build_rows (row -> store-row conversion, DEBUG level)
│ └─ pixeltable.sa.insert_rows (the SQL INSERT)
├─ pixeltable.view_load (one per mutable view: its own pipeline, rows, store writes)
└─ pixeltable.catalog.write_tbl_md (metadata commit)
Traces land in Grafana Cloud Traces (Tempo); metrics land in Grafana
Cloud Metrics (Prometheus).
Turn on the OpenTelemetry bridge
Telemetry is opt-in: call pxt_otel.init() once per process, before the
first table operation. It configures itself from the database’s
[[pixeltable.database]] entry in pixeltable.toml, from environment
variables, or from the [otel] section of ~/.pixeltable/config.toml,
in that order of precedence.
Option 1: the database entry in pixeltable.toml. Put the endpoint
on the [[pixeltable.database]] entry of the database (the entry with
no name is the local database; see
Configuration). The
headers are a secret and stay out of the file: keep
OTEL_EXPORTER_OTLP_HEADERS in the environment, and on a hosted
database set it with
pxt secret set pxt://org:db OTEL_EXPORTER_OTLP_HEADERS=..., then
pxt db restart so the running database sees it.
Option 2: environment variables. Set these before starting the
Jupyter kernel (the kernel inherits them):
For Grafana Cloud, the OpenTelemetry connection page (Connections ->
Add new connection -> OpenTelemetry) shows your <zone>,
<instance-id>, and lets you create a <service-account-token>. Then
run these in the terminal, substituting your values, and start Jupyter
from that same terminal:
Option 3: ~/.pixeltable/config.toml, shared by every database on
the installation:
init() runs once per process. If it ran without an endpoint
earlier in this kernel it can’t be reconfigured, so fix the
configuration, restart the kernel, and run this cell first.
Connected to Pixeltable database at: postgresql+psycopg://postgres:@/pixeltable?host=/private/tmp/pxt_otel_demo/pgdata
OTel bridge active
Build the pipeline
Everything here is traced: create_dir and create_table produce DDL
traces, each add_computed_column call is its own trace, and
create_view contains a pixeltable.op.LoadViewOp subtree that loads
the (still empty) view.
The computed columns are real work so the insert has something to trace:
a stored thumbnail (persisted -> pixeltable.media.save spans), a
grayscale conversion, and image metadata extraction. The view derives a
rotated thumbnail from the base table.
if_exists='replace_force' on the directory drops everything from a
previous run.
Created directory ‘otel_demo’.
Created table ‘images’.
Added 0 column values with 0 errors in 0.00 s
Added 0 column values with 0 errors in 0.00 s
Added 0 column values with 0 errors in 0.00 s
Created view ‘otel_demo/thumbs’.
Insert
4 rows referencing external images (well under the
100-span-per-operation cap, so every row gets a span). Downloads, UDF
evaluation, thumbnail persistence, the store write path, and the view
reload all land under one pixeltable.insert trace, following the tree
from the introduction.
Note: pixeltable.media.fetch spans only appear when the files are not
yet in the local file cache; on a re-run of this cell the downloads are
skipped.
Inserted 8 rows with 0 errors in 0.49 s (16.27 rows/s)
8 rows inserted.
Embedding index
add_embedding_index backfills embeddings for the existing rows and
builds the vector index, all in one trace: pixeltable.model.load
(first use of the model in this process), batched
pixeltable.udf.sentence_transformer calls, the backfill write
(pixeltable.store.write_column), and pixeltable.store.create_index.
Queries are traced too: collect() (along with head, tail, and
count) roots its own trace, a pixeltable.collect span carrying the
table names and row count, with a nested
pixeltable.result_cursor.yield_rows span and, at 'debug', the
per-row and per-UDF spans below it. The similarity search below produces
its own trace alongside the index build.
Evolve, mutate, roll back
Three more operations, one trace each:
add_computed_column on a table that already has data shows the
backfill: a pixeltable.store.write_column span containing the
evaluation pipeline.update recomputes dependents of the changed column inside the same
trace, including the embedding index (watch for another
pixeltable.udf.sentence_transformer span).revert rolls the update back; version cleanup shows up as
pixeltable.catalog.revert_version and pixeltable.media.delete
spans.
Other operations trace the same way (delete, rename_column,
recompute_columns, batch_update, snapshots, drops, …); this notebook
is a sample, not the full list.
Added 4 column values with 0 errors in 0.02 s (184.28 rows/s)
Metrics
The operations above didn’t just produce traces: the same bridge exports
Pixeltable’s metric instruments through the OTLP endpoint. No setup is
needed beyond init(); a PeriodicExportingMetricReader ships them
every 60 seconds by default (set OTEL_METRIC_EXPORT_INTERVAL, in
milliseconds, to change that).
This notebook’s lifecycle populated most of them: rows.written for the
images table and the thumbs view, cells.computed, udf.calls and
udf.latency for resize, convert, get_metadata, rotate,
entropy, and the batched sentence_transformer, and both media byte
counters (the insert downloaded four images and persisted the generated
thumbnails). The error, retry, and token metrics are absent here because
instruments are created lazily on first record; they appear when a
computed column raises (with on_error='ignore'), when a rate-limited
provider call is retried, or when a token-instrumented provider UDF
(OpenAI chat_completions/responses, Anthropic messages, Gemini
generate_content, or Ollama generate/chat) returns a response
carrying usage information.
Viewing in Grafana. Explore -> your Prometheus (Metrics) data
source. OTLP metric names arrive translated to Prometheus conventions:
dots become underscores, counters get a _total suffix, and time/byte
units are spelled out, so pixeltable.udf.calls becomes
pixeltable_udf_calls_total and pixeltable.udf.latency becomes
pixeltable_udf_latency_seconds_bucket/_sum/_count. Dimensions
arrive as labels (pxt_table, pxt_udf). Some queries to try:
Counter panels and short-lived processes. These are cumulative
counters, and rate()/increase() need at least two samples per series
with growth between them. A process that runs one operation and exits
exports a single sample, which those functions treat as the baseline:
the panel reads 0 even though the data arrived. In a live kernel like
this one the periodic exporter keeps shipping samples, so rate panels
work after a couple of export intervals; for one-shot scripts, query the
raw values instead,
e.g. last_over_time(pixeltable_udf_calls_total[1h]).
Wait a moment for export
init() sets up a BatchSpanProcessor, which ships queued spans
automatically about every 5 seconds while the kernel stays alive, and a
PeriodicExportingMetricReader, which ships metrics every 60 seconds by
default. Give it a minute after the last operation, then check Grafana.
(In a short-lived script that exits immediately you would call
trace.get_tracer_provider().force_flush() and
metrics.get_meter_provider().force_flush() before exit, but in a live
notebook kernel the timers handle it.)
View it in Grafana
Explore -> your Traces (Tempo) data source -> search Service Name
pixeltable. This notebook produced one trace per operation:
pixeltable.create_dir, pixeltable.create_table,
pixeltable.add_computed_column (x4), pixeltable.create_view,
pixeltable.insert, pixeltable.add_embedding_index,
pixeltable.update, pixeltable.revert, pixeltable.drop_table (x2),
and pixeltable.drop_dir, plus a pixeltable.collect trace for each of
the two queries (the similarity search and the final select).
The two showcase traces:
pixeltable.insert: the full tree from the introduction -
parallel media.fetch downloads (on a cold cache), pixeltable.row
spans with nested udf.resize/udf.convert/udf.get_metadata calls,
media.save persistence, the store.build_rows/sa.insert_rows
write path, and the view_load subtree with its own pipeline and
rows.pixeltable.add_embedding_index: model.load, batched
udf.sentence_transformer calls, the backfill write, and
store.create_index.
For more detail use span_level='trace'; to reduce volume on a large
insert, use the default 'info', which keeps the operation spans and
the INFO-level work spans (catalog.begin_xact, sa.insert_rows,
sa.select, store.insert_rows, store.write_column, view_load,
metadata writes) and drops the DEBUG spans (per-row, per-UDF, per-file,
store.build_rows, and the finer-grained catalog and store spans).
Teardown
Drops are traced too: each pixeltable.drop_table trace contains the
pixeltable.op.* sequence (media cleanup, store table drop, metadata
deletion) that executes the drop, and pixeltable.drop_dir closes out
the demo. The view must drop before its base table.
