Working with Voxel51 for Visualization in Pixeltable

This documentation page is also available as an interactive notebook. You can launch the notebook in Kaggle or Colab, or download it for use with an IDE or local Jupyter installation, by clicking one of the above links.

Pixeltable can export data directly from tables and views to the popular Voxel51 frontend, providing a way to visualize and explore image and video datasets. In this tutorial, we’ll learn how to:

Export data from Pixeltable to Voxel51
Apply labels from image classification and object detection models to exported data

We begin by installing the necessary libraries for this tutorial.

%pip install -qU pixeltable fiftyone torch transformers timm

Example 1: An Image Dataset

import fiftyone as fo
import pixeltable as pxt

# Create a Pixeltable directory for the demo. We first drop the directory if it
# exists, in order to ensure a clean environment.

pxt.drop_dir('fo_demo', force=True)
pxt.create_dir('fo_demo')

Connected to Pixeltable database at: postgresql+psycopg://postgres:@/pixeltable?host=/Users/pjlb/.pixeltable/pgdata
Created directory ‘fo_demo’.
<pixeltable.catalog.dir.Dir at 0x31423b350>

# Create a Pixeltable table for our dataset and insert some sample images.

url_prefix = 'https://raw.githubusercontent.com/pixeltable/pixeltable/main/docs/resources/images'

urls = [
    'https://raw.githubusercontent.com/pixeltable/pixeltable/main/docs/resources/images/000000000019.jpg',
    'https://raw.githubusercontent.com/pixeltable/pixeltable/main/docs/resources/images/000000000025.jpg',
    'https://raw.githubusercontent.com/pixeltable/pixeltable/main/docs/resources/images/000000000030.jpg',
    'https://raw.githubusercontent.com/pixeltable/pixeltable/main/docs/resources/images/000000000034.jpg',
]

t = pxt.create_table('fo_demo/images', {'image': pxt.Image})
t.insert({'image': url} for url in urls)
t.head()

Created table ‘images’.
Inserted 4 rows with 0 errors in 0.71 s (5.60 rows/s)

Now we export our new table to a Voxel51 dataset and load it into a new Voxel51 session within our demo notebook. Once it’s been loaded, the images can be interactively navigated as with any other Voxel51 dataset.

fo_dataset = pxt.io.export_images_as_fo_dataset(t, t.image)
session = fo.launch_app(fo_dataset)

You are running the oldest supported major version of MongoDB. Please refer to https://deprecation.voxel51.com&nbsp;for&nbsp;deprecation&nbsp;notices.&nbsp;You&nbsp;can&nbsp;suppress&nbsp;this&nbsp;exception&nbsp;by&nbsp;setting&nbsp;your&nbsp;\`database_validation\`&nbsp;config&nbsp;parameter&nbsp;to&nbsp;\`False\`.&nbsp;See&nbsp;https://docs.voxel51.com/user_guide/config.html#configuring-a-mongodb-connection&nbsp;for&nbsp;more&nbsp;information
 28 [31.4ms elapsed, ? remaining, 890.5 samples/s]

Adding Labels

We’ll now show how Voxel51 labels can be attached to the exported dataset. Currently, Pixeltable supports only classification and detection labels; other Voxel51 label types may be added in the future. First, let’s generate some labels by applying two models from the Huggingface transformers library: A ViT model for image classification and a DETR model for object detection.

from pixeltable.functions.huggingface import (
    detr_for_object_detection,
    vit_for_image_classification,
)

t.add_computed_column(
    classifications=vit_for_image_classification(
        t.image, model_id='google/vit-base-patch16-224'
    )
)
t.add_computed_column(
    detections=detr_for_object_detection(
        t.image, model_id='facebook/detr-resnet-50'
    )
)

Added 4 column values with 0 errors in 4.17 s (0.96 rows/s)
Added 4 column values with 0 errors in 2.72 s (1.47 rows/s)
4 rows updated.

Both models output JSON containing the model results. Let’s peek at the contents of our table now:

t.head()

Now we need to transform our model data into the format the Voxel51 API expects (see the Pixeltable documentation for pxt.io.export_images_as_fo_dataset for details). We’ll use Pixeltable UDFs to do the appropriate conversions.

@pxt.udf
def vit_to_fo(vit_labels: list) -> list:
    return [
        {'label': label, 'confidence': score}
        for label, score in zip(
            vit_labels['label_text'], vit_labels['scores']
        )
    ]


@pxt.udf
def detr_to_fo(img: pxt.Image, detr_labels: dict) -> list:
    result = []
    for label, box, score in zip(
        detr_labels['label_text'],
        detr_labels['boxes'],
        detr_labels['scores'],
    ):
        # DETR gives us bounding boxes in (x1,y1,x2,y2) absolute (pixel) coordinates.
        # Voxel51 expects (x,y,w,h) relative (fractional) coordinates.
        # So we need to do a conversion.
        fo_box = [
            box[0] / img.width,
            box[1] / img.height,
            (box[2] - box[0]) / img.width,
            (box[3] - box[1]) / img.height,
        ]
        result.append(
            {'label': label, 'bounding_box': fo_box, 'confidence': score}
        )
    return result

We can test that our UDFs are working as expected with a select() statement.

t.select(
    t.image,
    t.classifications,
    vit_to_fo(t.classifications),
    t.detections,
    detr_to_fo(t.image, t.detections),
).head()

Now we pass the modified structures to export_images_as_fo_dataset.

fo_dataset = pxt.io.export_images_as_fo_dataset(
    t,
    t.image,
    classifications=vit_to_fo(t.classifications),
    detections=detr_to_fo(t.image, t.detections),
)
session = fo.launch_app(fo_dataset)

 28 [41.8ms elapsed, ? remaining, 669.2 samples/s]

Adding Multiple Label Sets

You can include multiple label sets of the same type in the same dataset by passing a list or dict of expressions to the classifications and/or detections parameters. If a list is specified, default names will be assigned to the label sets; if a dict is specified, the label sets will be named according to its keys. As an example, let’s try recomputing our detections using the more powerful DETR model ResNet-101, and then load them into the same Voxel51 dataset as the earlier detections in order to compare them side-by-side.

t.add_computed_column(
    detections_101=detr_for_object_detection(
        t.image, model_id='facebook/detr-resnet-101'
    )
)

Added 4 column values with 0 errors in 21.91 s (0.18 rows/s)
4 rows updated.

fo_dataset = pxt.io.export_images_as_fo_dataset(
    t,
    t.image,
    classifications=vit_to_fo(t.classifications),
    detections={
        'detections_50': detr_to_fo(t.image, t.detections),
        'detections_101': detr_to_fo(t.image, t.detections_101),
    },
)
session = fo.launch_app(fo_dataset)

 28 [44.2ms elapsed, ? remaining, 633.4 samples/s]

Exploring the resulting images, we can see that the results are not much different between the two models, at least on our small sample dataset.

Built-in Integrations

Observability

Infrastructure

Bring Your Own

Providers

Working with Voxel51 for Visualization in Pixeltable

Example 1: An Image Dataset

Adding Labels

Adding Multiple Label Sets

​Example 1: An Image Dataset

​Adding Labels

​Adding Multiple Label Sets

Example 1: An Image Dataset

Adding Labels

Adding Multiple Label Sets