Ecosystem - Pixeltable Documentation

From language models to computer vision frameworks, Pixeltable integrates with the entire ecosystem. All integrations are available out-of-the-box with Pixeltable installation. No additional setup required unless specified.

Missing an integration? Build your own with Pixeltable UDFs, or suggest one on GitHub Discussions.

Cloud LLM providers

Anthropic Claude

Integrate Claude models for advanced language understanding and generation with multimodal capabilities

Google Gemini

Access Google’s Gemini models via Google AI Studio or Vertex AI for state-of-the-art multimodal AI capabilities

OpenAI

Leverage GPT models for text generation, embeddings, and image analysis

Azure OpenAI

Use OpenAI models via Azure with enterprise security and compliance

Mistral AI

Use Mistral’s efficient language models for various NLP tasks

DeepSeek

Leverage DeepSeek’s powerful language and code models for text and code generation

Groq

Access Groq’s models for text generation

Model hubs

Platforms that host or route many models from a shared catalog.

Hugging Face Hub

Access thousands of pre-trained models across vision, text, and audio domains

Replicate

Deploy and run ML models through Replicate’s cloud infrastructure

Together AI

Access a variety of open-source models through Together AI’s platform

Fireworks

Use Fireworks.ai’s optimized model inference infrastructure

OpenRouter

Unified access to 100+ LLMs from various providers through a single API

AWS Bedrock

Access a variety of AI models through AWS Bedrock’s unified API

Nebius

Nebius Token Factory language and embedding models via an OpenAI-compatible API

Embeddings & Reranking

Voyage AI

High-quality embeddings and reranking for text, images, and video

Jina AI

Embeddings and reranking optimized for search and RAG pipelines

Twelve Labs

Multimodal embeddings for text, image, audio, and video via the TwelveLabs Embed API

Media Generation

BFL (FLUX)

Image generation, editing, fill, and expansion with FLUX models from Black Forest Labs

fal.ai

Fast image generation with Flux, Stable Diffusion, and other models

Reve

AI-powered video generation and editing capabilities

RunwayML

AI video generation with Gen-4 and other Runway models

Local LLM runtimes

Llama.cpp

High-performance C++ implementation for running LLMs on CPU and GPU

Ollama

Easy-to-use toolkit for running and managing open-source models locally

vLLM

High-throughput local LLM inference with PagedAttention and continuous batching

Hugging Face models

Pixeltable provides seamless integration with Hugging Face’s transformers library through built-in UDFs. These functions allow you to use state-of-the-art models directly in your data workflows.

Requirements: Install required dependencies with pip install transformers. Some models may require additional packages like sentence-transformers or torch.

CLIP models

from pixeltable.functions.huggingface import clip

# For text embedding
t.add_computed_column(
    text_embedding=clip(
        t.text_column,
        model_id='openai/clip-vit-base-patch32'
    )
)

# For image embedding
t.add_computed_column(
    image_embedding=clip(
        t.image_column,
        model_id='openai/clip-vit-base-patch32'
    )
)

Perfect for multimodal applications combining text and image understanding.

Cross-encoders

from pixeltable.functions.huggingface import cross_encoder

t.add_computed_column(
    similarity_score=cross_encoder(
        t.sentence1,
        t.sentence2,
        model_id='cross-encoder/ms-marco-MiniLM-L-4-v2'
    )
)

Ideal for semantic similarity tasks and sentence pair classification.

DETR object detection

from pixeltable.functions.huggingface import detr_for_object_detection

t.add_computed_column(
    detections=detr_for_object_detection(
        t.image,
        model_id='facebook/detr-resnet-50',
        threshold=0.8
    )
)

# Convert to COCO format if needed
t.add_computed_column(
    coco_format=detr_to_coco(t.image, t.detections)
)

Powerful object detection with end-to-end transformer architecture.

Sentence transformers

from pixeltable.functions.huggingface import sentence_transformer

t.add_computed_column(
    embeddings=sentence_transformer(
        t.text,
        model_id='sentence-transformers/all-mpnet-base-v2',
        normalize_embeddings=True
    )
)

State-of-the-art sentence and document embeddings for semantic search and similarity.

Speech2Text models

from pixeltable.functions.huggingface import speech2text_for_conditional_generation

# Basic transcription
t.add_computed_column(
    transcript=speech2text_for_conditional_generation(
        t.audio,
        model_id='facebook/s2t-small-librispeech-asr'
    )
)

# Multilingual translation
t.add_computed_column(
    translation=speech2text_for_conditional_generation(
        t.audio,
        model_id='facebook/s2t-medium-mustc-multilingual-st',
        language='fr'
    )
)

Support for both transcription and translation of audio content.

Vision Transformer (ViT)

from pixeltable.functions.huggingface import vit_for_image_classification

t.add_computed_column(
    classifications=vit_for_image_classification(
        t.image,
        model_id='google/vit-base-patch16-224',
        top_k=5
    )
)

Modern image classification using transformer architecture.

Model selection guide

Choose Task

Select the appropriate model family based on your task:

Text/Image Similarity → CLIP
Object Detection → DETR
Text Embeddings → Sentence Transformers
Speech Processing → Speech2Text
Image Classification → ViT

Check Requirements

Install necessary dependencies:

pip install transformers torch sentence-transformers

Setup Integration

Import and use the model in your Pixeltable workflow:

from pixeltable.functions.huggingface import clip, sentence_transformer

Computer vision

YOLOX

State-of-the-art object detection with YOLOX models

Voxel51

Advanced video and image dataset management with Voxel51

Audio processing

Whisper/WhisperX

High-quality speech recognition and transcription using OpenAI’s Whisper models

Enterprise Platforms

Microsoft Fabric

Azure OpenAI integration through Microsoft Fabric for enterprise AI workloads

Data Wrangling

Pandas

Import and export from and to Pandas DataFrames

Usage examples

LLM Integration

import pixeltable as pxt
from pixeltable.functions import openai

# Create a table with computed column for OpenAI completion
table = pxt.create_table('responses', {'prompt': pxt.String})

table.add_computed_column(
    response=openai.chat_completions(
        messages=[{'role': 'user', 'content': table.prompt}],
        model='gpt-4'
    )
)

Computer Vision

from pixeltable.functions.yolox import yolox

# Add object detection to video frames
frames_view.add_computed_column(
    detections=yolox(
        frames_view.frame,
        model_id='yolox_l'
    )
)

Audio Processing

from pixeltable.functions import openai

# Transcribe audio files
audio_table.add_computed_column(
    transcription=openai.transcriptions(
        audio=audio_table.file,
        model='whisper-1'
    )
)

Model Output Format

# Object Detection Output
{
    'scores': [0.99, 0.98],  # confidence scores
    'labels': [25, 30],      # class labels
    'label_text': ['cat', 'dog'], # human-readable labels
    'boxes': [[x1, y1, x2, y2], ...] # bounding boxes
}

# Image Classification Output
{
    'scores': [0.8, 0.15],   # class probabilities
    'labels': [340, 353],    # class IDs
    'label_text': ['zebra', 'gazelle'] # class names
}

Integration features

Easy Setup

Most integrations work out-of-the-box with simple API configuration

Computed Columns

Use integrations directly in computed columns for automated processing

Batch Processing

Efficient handling of batch operations with automatic optimization

Check the provider notebooks for detailed usage instructions for each integration.

​Cloud LLM providers

Anthropic Claude

Google Gemini

OpenAI

Azure OpenAI

Mistral AI

DeepSeek

Groq

​Model hubs

Hugging Face Hub

Replicate

Together AI

Fireworks

OpenRouter

AWS Bedrock

Nebius

​Embeddings & Reranking

Voyage AI

Jina AI

Twelve Labs

​Media Generation

BFL (FLUX)

fal.ai

Reve

RunwayML

​Local LLM runtimes

Llama.cpp

Ollama

vLLM

​Hugging Face models

​CLIP models

​Cross-encoders

​DETR object detection

​Sentence transformers

​Speech2Text models

​Vision Transformer (ViT)

​Model selection guide

​Computer vision

YOLOX

Voxel51

​Audio processing

Whisper/WhisperX

​Enterprise Platforms

Microsoft Fabric

​Data Wrangling

Pandas

​Usage examples

​Integration features

Cloud LLM providers

Model hubs

Embeddings & Reranking

Media Generation

Local LLM runtimes

Hugging Face models

CLIP models

Cross-encoders

DETR object detection

Sentence transformers

Speech2Text models

Vision Transformer (ViT)

Model selection guide

Computer vision

Audio processing

Enterprise Platforms

Data Wrangling

Usage examples

Integration features