> ## Documentation Index
> Fetch the complete documentation index at: https://docs.pixeltable.com/llms.txt
> Use this file to discover all available pages before exploring further.
# Observability with OpenTelemetry
This documentation page is also available as an interactive notebook. You can launch the notebook in
Kaggle or Colab, or download it for use with an IDE or local Jupyter installation, by clicking one of the
above links.
export const quartoRawHtml = [`
| Environment variable |
Value |
OTEL_EXPORTER_OTLP_ENDPOINT |
Your OTLP endpoint,
e.g. https://otlp-gateway-<zone>.grafana.net/otlp for
Grafana Cloud |
OTEL_EXPORTER_OTLP_HEADERS |
Authorization=Basic <token>, where
<token> is the base64 encoding of
<instance-id>:<service-account-token> |
`];
Pixeltable can export its internal telemetry as OpenTelemetry (OTel)
spans, so you can watch operations in any OTel backend. This notebook
wires the OTel bridge to **Grafana Cloud** and traces a single
`insert()` that exercises the full span set: source preparation, lock
acquisition, planning, external media downloads, UDF evaluation, media
persistence, the store write path, and view propagation.
The insert emits one nested span tree:
pixeltable.insert (operation span, root)
├─ pixeltable.data\_source.prepare (source resolution + schema validation)
├─ pixeltable.catalog.begin\_xact (connection + lock acquisition, one per attempt)
├─ pixeltable.plan.create (insert plan construction, DEBUG level)
├─ pixeltable.media.fetch (one per external file downloaded, DEBUG level)
├─ pixeltable.row (one per inserted row, DEBUG level)
│ └─ pixeltable.udf.\ (each UDF call, nested under its row)
├─ pixeltable.media.save (one per generated media file persisted, DEBUG level)
├─ pixeltable.store.build\_rows (row -> store-row conversion, DEBUG level)
├─ pixeltable.sa.insert\_rows (the SQL INSERT)
└─ pixeltable.view\_load (one per mutable view, containing its own row/store spans)
Traces land in Grafana Cloud Traces (Tempo).
```python theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
%pip install -qU 'pixeltable[otel]'
```
## Turn on the OpenTelemetry bridge
Telemetry is opt-in: call `pxt_otel.init()` once per process, before the
first table operation. It configures itself from environment variables
or from the `[otel]` section of `~/.pixeltable/config.toml` (env vars
take precedence).
**Option 1: environment variables.** Set these **before starting the
Jupyter kernel** (the kernel inherits them):
For Grafana Cloud, the OpenTelemetry connection page (Connections ->
Add new connection -> OpenTelemetry) shows your ``,
``, and lets you create a ``. Then
run these in the terminal, substituting your values, and start Jupyter
from that same terminal:
```bash theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
export OTEL_EXPORTER_OTLP_ENDPOINT='https://otlp-gateway-.grafana.net/otlp'
export OTEL_EXPORTER_OTLP_HEADERS="Authorization=Basic $(echo -n ':' | base64 | tr -d '\n')"
```
**Option 2: `~/.pixeltable/config.toml`.** Generate the token first
(`echo -n ':' | base64 | tr -d '\n'`),
then add:
```toml theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
[otel]
exporter_otlp_endpoint = 'https://otlp-gateway-.grafana.net/otlp'
exporter_otlp_headers = 'Authorization=Basic '
```
`init()` runs **once per process**. If it ran without an endpoint
earlier in this kernel it can’t be reconfigured, so fix the
configuration, restart the kernel, and run this cell first.
```python theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
import opentelemetry.instrumentation.pixeltable as pxt_otel
from opentelemetry import trace
from opentelemetry.sdk.trace import TracerProvider
pxt_otel.init(
span_level='debug'
) # 'debug' emits per-row + per-UDF spans (default 'info' = operation span only)
# fail loud instead of silently dropping every span: a no-op ProxyTracerProvider means the SDK never came up
assert isinstance(trace.get_tracer_provider(), TracerProvider), (
'OpenTelemetry is INERT: no OTLP endpoint was configured. '
'Set the env vars (or config.toml) above, restart the kernel, and run this cell first.'
)
print('OTel bridge active')
```
OTel bridge active
## A pipeline that exercises every span type
One table with an image column and two computed columns, plus a view:
* inserting external image URLs triggers `pixeltable.media.fetch` (one
download span per file, on worker threads)
* the `add_one` UDF produces a `pixeltable.udf.add_one` span under each
`pixeltable.row`
* the stored `thumb` column generates thumbnails whose persistence
produces `pixeltable.media.save` spans
* the view is recomputed as part of the same insert, producing the
nested `pixeltable.view_load` subtree
`if_exists='replace_force'` also drops the dependent view on re-runs.
```python theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
import pixeltable as pxt
t = pxt.create_table(
'otel_demo1',
{'c': pxt.Int, 'img': pxt.Image},
if_exists='replace_force',
)
# Simple expression (span not reported since no UDF is invoked)
t.add_computed_column(inc=t.c + 1)
# Complex expression (span reported since UDF is invoked)
t.add_computed_column(thumb=t.img.resize((64, 64)))
v = pxt.create_view(
'otel_demo_view',
t,
additional_columns={'inc2': t.inc + 2},
if_exists='replace',
)
```
## Insert
This is the traced operation: 4 rows referencing external images (well
under the 100-span-per-operation cap, so every row gets a span). The
whole pipeline, downloads, UDFs, thumbnails, store writes, and the view
reload, lands under one `pixeltable.insert` trace.
Note: `pixeltable.media.fetch` spans only appear when the files are not
yet in the local file cache; on a re-run of this cell the downloads are
skipped.
```python theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
base = 'https://raw.githubusercontent.com/pixeltable/pixeltable/main/docs/resources/images'
images = [
'000000000001.jpg',
'000000000016.jpg',
'000000000019.jpg',
'000000000025.jpg',
]
status = t.insert(
[{'c': i, 'img': f'{base}/{name}'} for i, name in enumerate(images)]
)
status
```
Inserted 8 rows with 0 errors in 0.04 s (224.62 rows/s)
8 rows inserted.
## Wait a few seconds for export
`init()` sets up a `BatchSpanProcessor`, which ships queued spans
automatically about every 5 seconds while the kernel stays alive, so no
manual flush is needed. Give it a few seconds after the insert, then
check Grafana.
(In a short-lived script that exits immediately you would call
`trace.get_tracer_provider().force_flush()` before exit, but in a live
notebook kernel the timer handles it.)
## View it in Grafana
* **Traces**: Explore -> your Traces (Tempo) data source -> search
Service Name `pixeltable`, or span name `pixeltable.insert`. Expand
the trace to see the full hierarchy: `pixeltable.media.fetch`
downloads running in parallel, `pixeltable.row` ->
`pixeltable.udf.add_one`/`pixeltable.udf.resize` nesting,
`pixeltable.media.save` persistence, the
`pixeltable.store.build_rows`/`pixeltable.sa.insert_rows` write path,
and the `pixeltable.view_load` subtree with its own rows and store
writes.
For more detail use `span_level='trace'`; to reduce volume on a large
insert, use the default `'info'` (operation, `data_source.prepare`,
`begin_xact`, `sa.insert_rows`, and `view_load` spans only).