Iterators

​

What are Iterators?

Iterators in Pixeltable are specialized tools for processing and transforming media content. They efficiently break down large files into manageable chunks, enabling analysis at different granularities. Iterators work seamlessly with views to create virtual derived tables without duplicating storage.

In Pixeltable, iterators:

• Process media files incrementally to manage memory efficiently
• Transform single records into multiple output records
• Support various media types including documents, videos, images, and audio
• Integrate with the view system for automated processing pipelines
• Provide configurable parameters for fine-tuning output

Iterators are particularly useful when:

• Working with large media files that can’t be processed at once
• Building retrieval systems that require chunked content
• Creating analysis pipelines for multimedia data
• Implementing feature extraction workflows

import pixeltable as pxt
from pixeltable.iterators import DocumentSplitter

# Create a view using an iterator
chunks = pxt.create_view(
    'docs.chunks',
    documents_table,
    iterator=DocumentSplitter.create(
        document=documents_table.document,
        separators='paragraph'
    )
)

​

Core Concepts

​

Available Iterators

from pixeltable.iterators import DocumentSplitter

# Create view with document chunks
chunks_view = pxt.create_view(
    'docs.chunks',
    docs_table,
    iterator=DocumentSplitter.create(
        document=docs_table.document,
        separators='paragraph,token_limit',
        limit=500,
        metadata='title,heading'
    )
)

​

Common Use Cases

​

Example Workflows

# Create document chunks
chunks = pxt.create_view(
    'rag.chunks',
    docs_table,
    iterator=DocumentSplitter.create(
        document=docs_table.document,
        separators='paragraph',
        limit=500
    )
)

# Add embeddings
chunks.add_embedding_index(
    'text',
    string_embed=sentence_transformer.using(
        model_id='all-mpnet-base-v2'
    )
)

# Extract frames at 1 FPS
frames = pxt.create_view(
    'detection.frames',
    videos_table,
    iterator=FrameIterator.create(
        video=videos_table.video,
        fps=1.0
    )
)

# Add object detection
frames.add_computed_column(detections=detect_objects(frames.frame))

# Split long audio files
chunks = pxt.create_view(
    'audio.chunks',
    audio_table,
    iterator=AudioSplitter.create(
        audio=audio_table.audio,
        chunk_duration_sec=30.0
    )
)

# Add transcription
chunks.add_computed_column(text=whisper_transcribe(chunks.audio_chunk))

​

Best Practices

​

Tips & Tricks

​

Additional Resources