module pixeltable.functions.vision

Pixeltable UDFs for Computer Vision. Example:

import pixeltable as pxt
from pixeltable.functions import vision as pxtv

t = pxt.get_table(...)
t.select(
    pxtv.bboxes_draw(t.img, boxes=t.boxes, labels=t.labels)
).collect()

udf bboxes_clip_to_canvas()

Signature

@pxt.udf
bboxes_clip_to_canvas(bboxes: pxt.Json[(Json, format: pxt.String, *, width: pxt.Int | None = None, height: pxt.Int | None = None, min_visibility: pxt.Float = 0.0, min_area: pxt.Float = 0.0) -> pxt.Json[(Json, ...) ]

Clip a list of bounding boxes to a canvas of specified size. Parameters:

bboxes (pxt.Json[(Json): List of bounding boxes, each either specified with absolute pixel coordinates (int) or relative coordinates (float).
format (Any): Format of the bounding box coordinates, one of ‘xyxy’, ‘xywh’, ‘cxcywh’.
width (Any): Canvas width in absolute pixels. Required for absolute coordinates, must not be specified for relative.
height (Any): Canvas height in absolute pixels. Required for absolute coordinates, must not be specified for relative.
min_visibility (Any): Minimum fraction of the bounding box that must be visible after clipping. If the visibility is less than this value, returns None.
min_area (Any): Minimum area of the bounding box after clipping. If the area is less than this value, returns None.

Returns:

pxt.Json[(Json, ...)]: List of clipped bounding boxes in the same format as the input. Boxes that don’t meet the min_visibility or min_area thresholds are replaced with None.

udf bboxes_convert()

Signature

@pxt.udf
bboxes_convert(bboxes: pxt.Json[(Json, *, src_format: pxt.String, dst_format: pxt.String) -> pxt.Json[(Json, ...) ]

Convert a list of bounding boxes from src_format to dst_format. Parameters:

bboxes (pxt.Json[(Json): List of bounding boxes, each either specified with absolute pixel coordinates or relative coordinates in [0, 1].
src_format (Any): Source format, one of ‘xyxy’, ‘xywh’, ‘cxcywh’.
dst_format (Any): Destination format, one of ‘xyxy’, ‘xywh’, ‘cxcywh’.

Returns:

pxt.Json[(Json, ...)]: List of bounding boxes in dst_format.

udf bboxes_crop_canvas()

Signature

@pxt.udf
bboxes_crop_canvas(bboxes: pxt.Json[(Json, format: pxt.String, *, canvas_region: pxt.Json[(Json, canvas_region_format: pxt.String, canvas_width: pxt.Int | None = None, canvas_height: pxt.Int | None = None) -> pxt.Json[(Json, ...) ]

Adjust a list of bounding boxes to account for a canvas crop. Parameters:

bboxes (pxt.Json[(Json): List of bounding boxes, each either specified with absolute pixel coordinates or relative coordinates.
format (Any): Format of the bounding box coordinates, one of ‘xyxy’, ‘xywh’, ‘cxcywh’.
canvas_width (Any): Canvas width.
canvas_height (Any): Canvas height.
canvas_region (Any): Canvas region that was cropped, either specified with absolute pixel coordinates or relative coordinates, in the format specified by canvas_region_format.
canvas_region_format (Any): Format of the canvas_region coordinates, one of ‘xyxy’, ‘xywh’, ‘cxcywh’.

Returns:

pxt.Json[(Json, ...)]: List of adjusted bounding boxes in the same format as the input. They can extend beyond the canvas boundaries.

udf bboxes_draw()

Signature

@pxt.udf
bboxes_draw(
    img: pxt.Image,
    boxes: pxt.Json[(Json[(Int, *, labels: pxt.Json[(Json = None, color: pxt.String | None = None, box_colors: pxt.Json[(String = None, alpha: pxt.Float | None = None, fill: pxt.Bool = False, fill_alpha: pxt.Float | None = None, width: pxt.Int = 1, font: pxt.String | None = None, font_size: pxt.Int | None = None
) -> pxt.Image

Draws bounding boxes on the given image. Labels can be any type that supports str() and is hashable (e.g., strings, ints, etc.). Colors can be specified as common HTML color names (e.g., ‘red’) supported by PIL’s ImageColor module or as RGB/RGBA hex codes (e.g., ‘#FF0000’, ‘#FF0000FF’). If opacity isn’t specified in the color string and alpha/fill_alpha is None, defaults to 1.0 for box borders and 0.5 for filled boxes. If no colors are specified, this function randomly assigns each label a specific color based on a hash of the label. Parameters:

img (pxt.Image): The image on which to draw the bounding boxes.
boxes (pxt.Json[(Json[(Int): List of bounding boxes, each represented as [xmin, ymin, xmax, ymax].
labels (Any): List of labels for each bounding box.
color (Any): Single color to be used for all bounding boxes and labels.
box_colors (Any): List of colors, one per bounding box.
alpha (Any): Opacity (0-1) of the bounding box borders and labels. If non-None, overrides any alpha in color/box_colors.
fill (Any): Whether to fill the bounding boxes with color.
fill_alpha (Any): Opacity (0-1) of the bounding box fill. If non-None, overrides any alpha in color/box_colors.
width (Any): Width of the bounding box borders.
font (Any): Name of a system font or path to a TrueType font file, as required by PIL.ImageFont.truetype(). If None, uses the default provided by PIL.ImageFont.load_default().
font_size (Any): Size of the font used for labels in points. Only used in conjunction with non-None font argument.

Returns:

pxt.Image: The image with bounding boxes drawn on it.

udf bboxes_pad()

Signature

@pxt.udf
bboxes_pad(bboxes: pxt.Json[(Json, format: pxt.String, *, top: pxt.Int | None = None, bottom: pxt.Int | None = None, left: pxt.Int | None = None, right: pxt.Int | None = None, x: pxt.Int | None = None, y: pxt.Int | None = None) -> pxt.Json[(Json, ...) ]

Pad a list of bounding boxes. Parameters:

bboxes (pxt.Json[(Json): List of bounding boxes in absolute pixel coordinates.
format (Any): Format of the bounding box coordinates, one of ‘xyxy’, ‘xywh’, ‘cxcywh’.
top (Any): Amount to pad at the top, in absolute pixels.
bottom (Any): Amount to pad at the bottom, in absolute pixels.
left (Any): Amount to pad at the left, in absolute pixels.
right (Any): Amount to pad at the right, in absolute pixels.
x (Any): Amount to pad at the left and right, in absolute pixels.
y (Any): Amount to pad at the top and bottom, in absolute pixels.

Returns:

pxt.Json[(Json, ...)]: List of padded bounding boxes in the same format as the input.

udf bboxes_resize()

Signatures

# Signature 1:
@pxt.udf
bboxes_resize(
    bboxes: pxt.Json[(Json, ...)],
    format: pxt.String,
    width: pxt.Int | None,
    height: pxt.Int | None,
    aspect: pxt.String | None,
    aspect_mode: pxt.String | None
) -> pxt.Json[(Json, ...)]

# Signature 2:
@pxt.udf
bboxes_resize(
    bboxes: pxt.Json[(Json, ...)],
    format: pxt.String,
    width: pxt.Float | None,
    height: pxt.Float | None,
    aspect: pxt.Float | None,
    aspect_mode: pxt.String | None
) -> pxt.Json[(Json, ...)]

Resize a list of bounding boxes (center-anchored):

to a specified width or height (the other dimension is scaled to maintain the aspect ratio)
to a specified aspect ratio

Only one of width, height, or aspect can be specified. Parameters:

bboxes (Json[(Json, ...)]): List of bounding boxes, each either specified with absolute pixel coordinates or relative coordinates in [0, 1].
format (String): Format of the bounding box coordinates, one of ‘xyxy’, ‘xywh’, ‘cxcywh’.
width (Int | None, default: Literal(None)): Target width. Pass an int for absolute pixels or a float for relative coordinates.
height (Int | None, default: Literal(None)): Target height. Pass an int for absolute pixels or a float for relative coordinates.
aspect (String | None, default: Literal(None)): Target aspect ratio. Pass a str like ‘16:9’ or a float like 1.78.
aspect (String | None, default: Literal(None)): Target aspect ratio as a string ‘W:H’ (e.g., ‘16:9’) or a float. Resizes either the width or height to match the specified aspect ratio, maintaining the other dimension. Requires aspect_mode.
aspect_mode (String | None, default: Literal(None)): Either ‘crop’ or ‘pad’. Required when aspect is specified. If crop, reduces the oversized dimension to match the aspect ratio. If pad, extends the undersized dimension to match the aspect ratio.

Returns:

pxt.Json[(Json, ...)]: List of resized bounding boxes in the same format as the input.

udf bboxes_resize_canvas()

Signature

@pxt.udf
bboxes_resize_canvas(bboxes: pxt.Json[(Json, format: pxt.String, *, canvas_width: pxt.Int | None = None, canvas_height: pxt.Int | None = None, new_canvas_width: pxt.Int | None = None, new_canvas_height: pxt.Int | None = None, canvas_scale: pxt.Float | None = None, canvas_scale_x: pxt.Float | None = None, canvas_scale_y: pxt.Float | None = None) -> pxt.Json[(Json, ...) ]

Adjust a list of bounding boxes to account for a canvas resize. The resize operation can be expressed

as absolute pixel dimensions (requires canvas_width, canvas_height, new_canvas_width, new_canvas_height)
as relative dimensions (requires at least one of canvas_scale, canvas_scale_x, canvas_scale_y)

Parameters:

bboxes (pxt.Json[(Json): List of bounding boxes in absolute pixel coordinates.
format (Any): Format of the bounding box coordinates, one of ‘xyxy’, ‘xywh’, ‘cxcywh’.
canvas_width (Any): Original canvas width in absolute pixels.
canvas_height (Any): Original canvas height in absolute pixels.
new_canvas_width (Any): New canvas width in absolute pixels. Requires canvas_width/canvas_height to be specified.
new_canvas_height (Any): New canvas height in absolute pixels. Requires canvas_width/canvas_height to be specified.
canvas_scale (Any): Scale factor to apply to both canvas dimensions.
canvas_scale_x (Any): Scale factor to apply to the canvas width.
canvas_scale_y (Any): Scale factor to apply to the canvas height.

Returns:

pxt.Json[(Json, ...)]: List of adjusted bounding boxes in the same format as the input.

udf bboxes_scale()

Signature

@pxt.udf
bboxes_scale(bboxes: pxt.Json[(Json, format: pxt.String, *, factor: pxt.Float | None = None, x_factor: pxt.Float | None = None, y_factor: pxt.Float | None = None) -> pxt.Json[(Json, ...) ]

Re-scale a list of bounding boxes (center-anchored). Parameters:

bboxes (pxt.Json[(Json): List of bounding boxes, each either specified with absolute pixel coordinates or relative coordinates in [0, 1].
format (Any): Format of the bounding box coordinates, one of ‘xyxy’, ‘xywh’, ‘cxcywh’.
factor (Any): Scale factor to apply to both box dimensions.
x_factor (Any): Scale factor to apply to the box width.
y_factor (Any): Scale factor to apply to the box height.

Returns:

pxt.Json[(Json, ...)]: List of scaled bounding boxes in the same format as the input.

udf eval_detections()

Signature

@pxt.udf
eval_detections(pred_bboxes: pxt.Json[(Json[(Int, pred_labels: pxt.Json[(Int, pred_scores: pxt.Json[(Float, gt_bboxes: pxt.Json[(Json[(Int, gt_labels: pxt.Json[(Int, min_iou: pxt.Float = 0.5) -> pxt.Json[(Json, ...) ]

Evaluates the performance of a set of predicted bounding boxes against a set of ground truth bounding boxes. Parameters:

pred_bboxes (pxt.Json[(Json[(Int): List of predicted bounding boxes, each represented as [xmin, ymin, xmax, ymax].
pred_labels (Any): List of predicted labels.
pred_scores (Any): List of predicted scores.
gt_bboxes (Any): List of ground truth bounding boxes, each represented as [xmin, ymin, xmax, ymax].
gt_labels (Any): List of ground truth labels.
min_iou (Any): Minimum intersection-over-union (IoU) threshold for a predicted bounding box to be considered a true positive.

Returns:

pxt.Json[(Json, ...)]: A list of dictionaries, one per label class, with the following structure:

{
    'min_iou': float,  # The value of `min_iou` used for the detections
    'class': int,  # The label class
    # List of 1's and 0's indicating true positives for each
    # predicted bounding box of this class
    'tp': list[int],
    # List of 1's and 0's indicating false positives for each
    # predicted bounding box of this class; `fp[n] == 1 - tp[n]`
    'fp': list[int],
    # List of predicted scores for each bounding box of this class
    'scores': list[float],
    'num_gts': int,  # Number of ground truth bounding boxes of this class
}

udf overlay_segmentation()

Signature

@pxt.udf
overlay_segmentation(
    img: pxt.Image,
    segmentation: pxt.Array[(None, *, alpha: pxt.Float = 0.5, background: pxt.Int = 0, segment_colors: pxt.Json[(String = None, draw_contours: pxt.Bool = True, contour_thickness: pxt.Int = 1
) -> pxt.Image

Overlays a colored segmentation map on an image. Colors can be specified as common HTML color names (e.g., ‘red’) supported by PIL’s ImageColor module or as RGB hex codes (e.g., ‘#FF0000’). If no colors are specified, this function randomly assigns each segment a specific color based on a hash of its id. Parameters:

img (pxt.Image): Input image.
segmentation (pxt.Array[(None): 2D array of the same shape as img where each pixel value is a segment id.
alpha (Any): Blend factor for the overlay (0.0 = only original image, 1.0 = only segmentation colors).
background (Any): Segment id to treat as background (not overlaid with color, showing the original image through).
segment_colors (Any): List of colors, one per segment id. If the list is shorter than the number of segments, the remaining segments will be assigned colors automatically.
draw_contours (Any): If True, draw contours around each segment with full opacity.
contour_thickness (Any): Thickness of the contour lines in pixels.

Returns:

pxt.Image: The image with the colored segmentation overlay.

SDK Reference

​module pixeltable.functions.vision

​udf bboxes_clip_to_canvas()

​udf bboxes_convert()

​udf bboxes_crop_canvas()

​udf bboxes_draw()

​udf bboxes_pad()

​udf bboxes_resize()

​udf bboxes_resize_canvas()

​udf bboxes_scale()

​udf eval_detections()

​udf overlay_segmentation()

module pixeltable.functions.vision

udf bboxes_clip_to_canvas()

udf bboxes_convert()

udf bboxes_crop_canvas()

udf bboxes_draw()

udf bboxes_pad()

udf bboxes_resize()

udf bboxes_resize_canvas()

udf bboxes_scale()

udf eval_detections()

udf overlay_segmentation()