| view 'detection_demo/frames' (of 'detection_demo/videos') |
| Column Name | Type | Computed With |
|---|---|---|
| pos | Required[Int] | |
| frame_idx | Required[Int] | |
| pos_msec | Required[Float] | |
| pos_frame | Required[Int] | |
| frame | Required[Image] | |
| video | Video |
| video |
|---|
| pos | frame | width | height |
|---|---|---|---|
| 0 | 
|
1280 | 720 |
| 1 | 
|
1280 | 720 |
| 2 | 
|
1280 | 720 |
| 3 | 
|
1280 | 720 |
| 4 | 
|
1280 | 720 |
| frame | yolox |
|---|---|
|
|
{"bboxes": [[338.1894836425781, 345.59979248046875, 433.25408935546875, 402.1943359375], [101.51329040527344, 419.788330078125, 259.72821044921875, 512.6688842773438], [-0.2787722051143646, 555.6809692382812, 96.86801147460938, 675.8363037109375], [478.0632629394531, 290.70819091796875, 541.0510864257812, 333.1061096191406], [317.3228759765625, 488.96636962890625, 571.7535400390625, 640.4901733398438], [561.7608032226562, 282.067138671875, 609.9308471679688, 318.0382995605469], [582.622802734375, 409.7627258300781, 675.9083862304688, 518.007568359375], [884.5014038085938, 341.38031005859375, 994.5653076171875, 413.2419128417969], [40.29541778564453, 447.2200622558594, 98.4399642944336, 512.966064453125], [261.9676513671875, 626.4561767578125, 395.9239807128906, 716.9423217773438], [483.1141662597656, 574.5985717773438, 597.12109375, 686.2103271484375], [881.5115356445312, 340.0090026855469, 997.948974609375, 415.1181945800781]], "scores": [0.83, 0.812, 0.797, 0.763, 0.755, 0.688, 0.606, 0.541, 0.514, 0.503, 0.501, 0.5], "labels": [2, 2, 2, 2, 2, 2, 2, 2, 3, 3, 2, 7]} |
|
|
{"bboxes": [[-0.6882242560386658, 552.1439208984375, 99.95340728759766, 674.5260620117188], [341.227294921875, 343.3525695800781, 436.5254821777344, 401.6473083496094], [105.3248062133789, 417.0894775390625, 261.7541198730469, 508.3920593261719], [317.1871337890625, 489.1314392089844, 571.6358642578125, 640.3237915039062], [478.8114318847656, 288.74725341796875, 539.8158569335938, 329.8291931152344], [261.3262939453125, 623.7770385742188, 396.0945129394531, 717.0194702148438], [563.2278442382812, 280.2528991699219, 607.8761596679688, 317.1873474121094], [583.1015625, 409.9834899902344, 675.8267211914062, 516.9342041015625]], "scores": [0.823, 0.816, 0.767, 0.764, 0.76, 0.636, 0.633, 0.586], "labels": [2, 2, 2, 2, 2, 3, 2, 2]} |
|
|
{"bboxes": [[-0.2689302861690521, 550.2958374023438, 105.7685546875, 674.0948486328125], [342.9891052246094, 343.6980285644531, 436.35302734375, 400.53289794921875], [317.1612548828125, 489.2161865234375, 571.1923217773438, 640.4578247070312], [484.0317077636719, 290.1268005371094, 541.7188720703125, 330.689697265625], [103.90556335449219, 415.7806091308594, 265.3205261230469, 503.9178161621094], [563.6137084960938, 280.443359375, 609.4480590820312, 316.4215087890625], [582.57763671875, 410.0472412109375, 675.9314575195312, 516.9766235351562], [886.6967163085938, 344.71942138671875, 1004.0186767578125, 418.2158203125], [829.179931640625, 309.6842346191406, 875.6467895507812, 347.74945068359375], [833.5235595703125, 257.552490234375, 880.591064453125, 302.02337646484375], [1039.58984375, 366.77166748046875, 1095.6329345703125, 410.4030456542969], [481.5090026855469, 576.0469360351562, 596.9996337890625, 686.8409423828125], [885.1788330078125, 343.3632507324219, 1004.2406005859375, 417.9952392578125]], "scores": [0.847, 0.831, 0.767, 0.73, 0.722, 0.695, 0.623, 0.553, 0.546, 0.545, 0.543, 0.531, 0.502], "labels": [2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 7]} |
| view 'detection_demo/frames' (of 'detection_demo/videos') |
| Column Name | Type | Computed With |
|---|---|---|
| pos | Required[Int] | |
| frame_idx | Required[Int] | |
| pos_msec | Required[Float] | |
| pos_frame | Required[Int] | |
| frame | Required[Image] | |
| detections_tiny | Required[Json] | yolox(frame, model_id='yolox_tiny', threshold=0.25) |
| video | Video |
| frame | detections_tiny |
|---|---|
|
|
{"bboxes": [[338.189, 345.6, 433.254, 402.194], [101.513, 419.788, 259.728, 512.669], [-0.279, 555.681, 96.868, 675.836], [478.063, 290.708, 541.051, 333.106], [317.323, 488.966, 571.754, 640.49], [561.761, 282.067, 609.931, 318.038], ..., [796.496, 271.656, 833.955, 303.78], [444.182, 301.092, 475.788, 339.79], [539.506, 238.815, 584.09, 286.23], [1091.483, 384.693, 1122.668, 419.25], [455.287, 542.279, 596.589, 672.937], [829.069, 290.945, 877.77, 330.228]], "labels": [2, 2, 2, 2, 2, 2, ..., 2, 2, 2, 2, 2, 2], "scores": [0.83, 0.812, 0.797, 0.763, 0.755, 0.688, ..., 0.391, 0.361, 0.359, 0.262, 0.252, 0.251]} |
|
|
{"bboxes": [[-0.688, 552.144, 99.953, 674.526], [341.227, 343.353, 436.525, 401.647], [105.325, 417.09, 261.754, 508.392], [317.187, 489.131, 571.636, 640.324], [478.811, 288.747, 539.816, 329.829], [261.326, 623.777, 396.095, 717.019], ..., [486.762, 243.492, 535.403, 283.84], [830.206, 271.019, 881.183, 316.182], [830.853, 248.258, 877.105, 291.472], [455.334, 541.789, 596.258, 667.598], [804.526, 209.59, 833.616, 239.673], [1091.282, 384.247, 1125.306, 419.068]], "labels": [2, 2, 2, 2, 2, 3, ..., 2, 2, 2, 2, 2, 2], "scores": [0.823, 0.816, 0.767, 0.764, 0.76, 0.636, ..., 0.311, 0.291, 0.289, 0.264, 0.263, 0.256]} |
|
|
{"bboxes": [[-0.269, 550.296, 105.769, 674.095], [342.989, 343.698, 436.353, 400.533], [317.161, 489.216, 571.192, 640.458], [484.032, 290.127, 541.719, 330.69], [103.906, 415.781, 265.321, 503.918], [563.614, 280.443, 609.448, 316.422], ..., [796.171, 270.344, 832.105, 302.89], [1081.594, 381.512, 1122.797, 417.943], [1100.1, 386.188, 1132.332, 421.028], [455.572, 541.766, 596.424, 668.731], [55.976, 437.112, 109.67, 507.464], [520.065, 575.432, 596.629, 665.397]], "labels": [2, 2, 2, 2, 2, 2, ..., 2, 2, 2, 2, 3, 2], "scores": [0.847, 0.831, 0.767, 0.73, 0.722, 0.695, ..., 0.29, 0.289, 0.284, 0.275, 0.272, 0.256]} |
| frame | bboxes_draw |
|---|---|
|
|
| make_video |
|---|
| make_video | make_video_1 |
|---|---|
| view 'detection_demo/frames' (of 'detection_demo/videos') |
| Column Name | Type | Computed With |
|---|---|---|
| pos | Required[Int] | |
| frame_idx | Required[Int] | |
| pos_msec | Required[Float] | |
| pos_frame | Required[Int] | |
| frame | Required[Image] | |
| detections_tiny | Required[Json] | yolox(frame, model_id='yolox_tiny', threshold=0.25) |
| detections_m | Required[Json] | yolox(frame, model_id='yolox_m', threshold=0.25) |
| detections_x | Required[Json] | yolox(frame, model_id='yolox_x', threshold=0.25) |
| video | Video |
| eval_yolox_tiny | eval_yolox_m |
|---|---|
| [{"fp": [], "tp": [], "class": 0, "scores": [], "min_iou": 0.5, "num_gts": 5}, {"fp": [0, 0, 0, 0, 0, 0, ..., 1, 1, 0, 0, 1, 0], "tp": [1, 1, 1, 1, 1, 1, ..., 0, 0, 1, 1, 0, 1], "class": 2, "scores": [0.868, 0.776, 0.769, 0.756, 0.753, 0.719, ..., 0.416, 0.401, 0.356, 0.342, 0.265, 0.263], "min_iou": 0.5, "num_gts": 26}, {"fp": [0], "tp": [1], "class": 3, "scores": [0.692], "min_iou": 0.5, "num_gts": 4}, {"fp": [1], "tp": [0], "class": 5, "scores": [0.434], "min_iou": 0.5, "num_gts": 0}, {"fp": [1, 0, 0], "tp": [0, 1, 1], "class": 7, "scores": [0.47, 0.305, 0.303], "min_iou": 0.5, "num_gts": 4}, {"fp": [], "tp": [], "class": 62, "scores": [], "min_iou": 0.5, "num_gts": 1}] | [{"fp": [0, 0, 0], "tp": [1, 1, 1], "class": 0, "scores": [0.454, 0.399, 0.317], "min_iou": 0.5, "num_gts": 5}, {"fp": [0, 0, 0, 0, 0, 0, ..., 0, 0, 1, 1, 0, 0], "tp": [1, 1, 1, 1, 1, 1, ..., 1, 1, 0, 0, 1, 1], "class": 2, "scores": [0.935, 0.902, 0.902, 0.883, 0.875, 0.861, ..., 0.45, 0.444, 0.394, 0.337, 0.33, 0.269], "min_iou": 0.5, "num_gts": 26}, {"fp": [0, 0, 0], "tp": [1, 1, 1], "class": 3, "scores": [0.864, 0.697, 0.583], "min_iou": 0.5, "num_gts": 4}, {"fp": [0, 0, 0], "tp": [1, 1, 1], "class": 7, "scores": [0.738, 0.501, 0.481], "min_iou": 0.5, "num_gts": 4}, {"fp": [], "tp": [], "class": 62, "scores": [], "min_iou": 0.5, "num_gts": 1}] |
| mean_ap | mean_ap_1 |
|---|---|
| {0: 0.101, 2: 0.622, 3: 0.278, 5: 0.01, 7: 0.098, 62: 0., 58: 0., 9: 0., 8: 0., 1: 0., 24: 0.} | {0: 0.564, 2: 0.911, 3: 0.723, 7: 0.53, 62: 0., 4: 0., 58: 0.032, 1: 0., 9: 0., 5: 0.483, 25: 0., 8: 0., 24: 0.} |
Connected to Pixeltable database at: postgresql+psycopg://postgres:@/pixeltable?host=/Users/asiegel/.pixeltable/pgdata Created directory 'detection\_demo'. Created table 'videos'.In order to interact with the frames, we take advantage of Pixeltable’s component view concept: we create a “view” of our video table that contains one row for each frame of each video in the table. Pixeltable provides the built-in `frame_iterator` for this. ```python theme={null} from pixeltable.functions.video import frame_iterator frames_view = pxt.create_view( 'detection_demo/frames', videos_table, iterator=frame_iterator(videos_table.video), ) ``` You’ll see that neither the `videos` table nor the `frames` view has any actual data yet, because we haven’t yet added any videos to the table. However, the `frames` view is now configured to automatically track the `videos` table as new data shows up. The new view is automatically configured with six columns: * `pos` - a system column that is part of every component view * `video` - the column inherited from our base table (all base table columns are visible in any of its views) * `frame_idx`, `pos_msec`, `pos_frame`, `frame` - these four columns are created by the `frame_iterator`. Let’s have a look at the new view: ```python theme={null} frames_view ``` We’ll now insert a single row into the videos table, containing a video of a busy intersection in Bangkok. ```python theme={null} videos_table.insert( [ { 'video': 'https://raw.githubusercontent.com/pixeltable/pixeltable/release/docs/resources/bangkok.mp4' } ] ) ```
Inserted 462 rows with 0 errors in 4.35 s (106.25 rows/s) 462 rows inserted.Notice that both the `videos` table and `frames` view were automatically updated, expanding the single video into 461 rows in the view. Let’s have a look at `videos` first. ```python theme={null} videos_table.show() ``` Now let’s peek at the first five rows of `frames`: ```python theme={null} frames_view.select( frames_view.pos, frames_view.frame, frames_view.frame.width, frames_view.frame.height, ).show(5) ``` One advantage of using Pixeltable’s component view mechanism is that Pixeltable does not physically store the frames. Instead, Pixeltable re-extracts the frames on retrieval using the frame index, which can be done very efficiently and avoids any storage overhead (which can be quite substantial for video frames). ## Object Detection with Pixeltable Now let’s apply an object detection model to our frames. Pixeltable includes built-in support for a number of models; we’re going to use the YOLOX family of models, which are lightweight models with solid performance. We first import the `yolox` Pixeltable function. ```python theme={null} from pixeltable.functions.yolox import yolox ``` Pixeltable functions operate on columns and expressions using standard Python function call syntax. Here’s an example that shows how we might experiment with applying one of the YOLOX models to the first few frames in our video, using Pixeltable’s powerful `select` comprehension. ```python theme={null} # Show the results of applying the `yolox_tiny` model # to the first few frames in the table. frames_view.select( frames_view.frame, yolox(frames_view.frame, model_id='yolox_tiny') ).head(3) ``` It may appear that we just ran the YOLOX inference over the entire view of 461 frames, but remember that Pixeltable evaluates expressions lazily: in this case, it only ran inference over the 3 frames that we actually displayed. The inference output looks like what we’d expect, so let’s add a *computed column* that runs inference over the entire view (computed columns are discussed in detail in the [Computed Columns](https://github.com/pixeltable/pixeltable/blob/release/docs/tutorials/computed-columns.ipynb) tutorial). Remember that once a computed column is created, Pixeltable will update it incrementally any time new rows are added to the view. This is a convenient way to incorporate inference (and other operations) into data workflows. This *will* cause Pixeltable to run inference over all 461 frames, so please be patient. ```python theme={null} # Create a computed column to compute detections using the `yolox_tiny` # model. # We'll adjust the confidence threshold down a bit (the default is 0.5) # to pick up even more bounding boxes. frames_view.add_computed_column( detections_tiny=yolox( frames_view.frame, model_id='yolox_tiny', threshold=0.25 ) ) ```
Added 461 column values with 0 errors in 15.09 s (30.55 rows/s) 461 rows updated.The new column is now part of the schema of the `frames` view: ```python theme={null} frames_view ``` The data in the computed column is now stored for fast retrieval. ```python theme={null} frames_view.select(frames_view.frame, frames_view.detections_tiny).show(3) ``` Now let’s create a new set of images, in which we superimpose the detected bounding boxes on top of the original images. We’ll use the handy built-in `bboxes_draw` UDF for this. We could create a new computed column to hold the superimposed images, but we don’t have to; sometimes it’s easier just to use a `select` comprehension, as we did when we were first experimenting with the detection model. ```python theme={null} import pixeltable.functions as pxtf frames_view.select( frames_view.frame, pxtf.vision.bboxes_draw( frames_view.frame, frames_view.detections_tiny.bboxes, width=4 ), ).show(1) ``` Our `select` comprehension ranged over the entire table, but just as before, Pixeltable computes the output lazily: image operations are performed at retrieval time, so in this case, Pixeltable drew the annotations just for the one frame that we actually displayed. Looking at individual frames gives us some idea of how well our detection algorithm works, but it would be more instructive to turn the visualization output back into a video. We do that with the built-in function `make_video()`, which is an aggregation function that takes a frame index (actually: any expression that can be used to order the frames; a timestamp would also work) and an image, and then assembles the sequence of images into a video. ```python theme={null} frames_view.group_by(videos_table).select( pxt.functions.video.make_video( frames_view.pos, pxtf.vision.bboxes_draw( frames_view.frame, frames_view.detections_tiny.bboxes, width=4 ), ) ).show(1) ``` ## Comparing Object Detection Models The detections that we get out of `yolox_tiny` are passable, but a little choppy. Suppose we want to experiment with a more powerful object detection model, to see if there is any improvement in detection quality. We can create an additional column to hold the new inferences. The larger model takes longer to download and run, so please be patient. ```python theme={null} # Here we use the larger `yolox_m` (medium) model. frames_view.add_computed_column( detections_m=yolox( frames_view.frame, model_id='yolox_m', threshold=0.25 ) ) ```
Added 461 column values with 0 errors in 65.94 s (6.99 rows/s) 461 rows updated.Let’s see the results of the two models side-by-side. ```python theme={null} frames_view.group_by(videos_table).select( pxt.functions.video.make_video( frames_view.pos, pxtf.vision.bboxes_draw( frames_view.frame, frames_view.detections_tiny.bboxes, width=4 ), ), pxt.functions.video.make_video( frames_view.pos, pxtf.vision.bboxes_draw( frames_view.frame, frames_view.detections_m.bboxes, width=4 ), ), ).show(1) ``` Running the videos side-by-side, we can see that the larger model is higher in quality: less flickering, with more stable boxes from frame to frame. ## Evaluating Models Against a Ground Truth In order to do a quantitative evaluation of model performance, we need a ground truth to compare them against. Let’s generate some (synthetic) “ground truth” data by running against the largest YOLOX model available. It will take even longer to cache and evaluate this model. ```python theme={null} frames_view.add_computed_column( detections_x=yolox( frames_view.frame, model_id='yolox_x', threshold=0.25 ) ) ```
Added 461 column values with 0 errors in 156.55 s (2.94 rows/s) 461 rows updated.Let’s have a look at our enlarged view, now with three `detections` columns. ```python theme={null} frames_view ``` We’re going to be evaluating the generated detections with the commonly-used [mean average precision](https://learnopencv.com/mean-average-precision-map-object-detection-model-evaluation-metric/) metric (mAP). The mAP metric is based on per-frame metrics, such as true and false positives per detected class, which are then aggregated into a single (per-class) number. In Pixeltable, functionality is available via the `eval_detections()` and `mean_ap()` built-in functions. ```python theme={null} from pixeltable.functions.vision import eval_detections, mean_ap frames_view.add_computed_column( eval_yolox_tiny=eval_detections( pred_bboxes=frames_view.detections_tiny.bboxes, pred_labels=frames_view.detections_tiny.labels, pred_scores=frames_view.detections_tiny.scores, gt_bboxes=frames_view.detections_x.bboxes, gt_labels=frames_view.detections_x.labels, ) ) frames_view.add_computed_column( eval_yolox_m=eval_detections( pred_bboxes=frames_view.detections_m.bboxes, pred_labels=frames_view.detections_m.labels, pred_scores=frames_view.detections_m.scores, gt_bboxes=frames_view.detections_x.bboxes, gt_labels=frames_view.detections_x.labels, ) ) ```
Added 461 column values with 0 errors in 0.29 s (1589.38 rows/s) Added 461 column values with 0 errors in 0.31 s (1475.98 rows/s) 461 rows updated.Let’s take a look at the output. ```python theme={null} frames_view.select( frames_view.eval_yolox_tiny, frames_view.eval_yolox_m ).show(1) ``` The computation of the mAP metric is now simply a query over the evaluation output, aggregated with the `mean_ap()` function. ```python theme={null} frames_view.select( mean_ap(frames_view.eval_yolox_tiny), mean_ap(frames_view.eval_yolox_m), ).show() ``` This two-step process allows you to compute mAP at every granularity: over your entire dataset, only for specific videos, only for videos that pass a certain filter, etc. Moreover, you can compute this metric any time, not just during training, and use it to guide your understanding of your dataset and how it affects the quality of your models. Built with [Mintlify](https://mintlify.com).