Working with RunwayML in Pixeltable

This documentation page is also available as an interactive notebook. You can launch the notebook in Kaggle or Colab, or download it for use with an IDE or local Jupyter installation, by clicking one of the above links.

Pixeltable’s RunwayML integration lets you generate and transform images and videos using RunwayML’s Gen-4.5, Veo, and other models, directly from your Pixeltable tables.

What is RunwayML?

RunwayML is an AI creative platform offering state-of-the-art generative models for images and video. Pixeltable wraps four endpoints:

text_to_image: Generate images from text prompts with optional reference images
text_to_video: Generate videos from text prompts
image_to_video: Animate a still image into a video
video_to_video: Transform an existing video with text guidance

Documentation

Prerequisites

A RunwayML account with an API secret from RunwayML
pip install runwayml

Important: RunwayML API calls consume credits based on your plan. Image and video generation can be expensive — monitor your usage to avoid unexpected charges.

Setup

%pip install -qU pixeltable runwayml

import getpass
import os

if 'RUNWAYML_API_SECRET' not in os.environ:
    os.environ['RUNWAYML_API_SECRET'] = getpass.getpass(
        'RunwayML API Secret: '
    )

To read more about working with API keys in Pixeltable, see Configuration.

import pixeltable as pxt
from pixeltable.functions import runwayml

pxt.drop_dir('runwayml_demo', force=True)
pxt.create_dir('runwayml_demo')

Created directory ‘runwayml_demo’.
<pixeltable.catalog.dir.Dir at 0x32bd20a10>

Text-to-Video Generation

Generate videos from text prompts. The text_to_video function returns a JSON response containing the output video URL.

t2v = pxt.create_table(
    'runwayml_demo.text_to_video', {'prompt': pxt.String}
)

t2v.add_computed_column(
    response=runwayml.text_to_video(
        t2v.prompt, model='gen4.5', ratio='1280:720', duration=5
    )
)

# Extract the video URL from the response
t2v.add_computed_column(video=t2v.response['output'][0].astype(pxt.Video))

Created table ‘text_to_video’.
Added 0 column values with 0 errors in 0.00 s
Added 0 column values with 0 errors in 0.00 s
No rows affected.

t2v.insert(
    [
        {
            'prompt': 'A cinematic aerial shot of a coastal city at golden hour, waves crashing against the shore'
        }
    ]
)

Inserted 1 row with 0 errors in 105.52 s (0.01 rows/s)
1 row inserted.

t2v.select(t2v.prompt, t2v.video).collect()

Image-to-Video Generation

Animate a still image into a video. You can optionally add a text prompt to guide the motion.

i2v = pxt.create_table(
    'runwayml_demo.image_to_video',
    {'image': pxt.Image, 'prompt': pxt.String},
)

i2v.add_computed_column(
    response=runwayml.image_to_video(
        i2v.image,
        model='gen4.5',
        ratio='1280:720',
        prompt_text=i2v.prompt,
        duration=5,
    )
)

i2v.add_computed_column(video=i2v.response['output'][0].astype(pxt.Video))

Created table ‘image_to_video’.
Added 0 column values with 0 errors in 0.01 s
Added 0 column values with 0 errors in 0.01 s
No rows affected.

i2v.insert(
    [
        {
            'image': 'https://upload.wikimedia.org/wikipedia/commons/thumb/a/a7/Camponotus_flavomarginatus_ant.jpg/640px-Camponotus_flavomarginatus_ant.jpg',
            'prompt': 'The ant slowly walks across the leaf',
        }
    ]
)

Inserted 1 row with 0 errors in 145.73 s (0.01 rows/s)
1 row inserted.

i2v.select(i2v.image, i2v.video).collect()

Text-to-Image Generation

Generate images from text prompts with reference images. The text_to_image function requires at least one reference image.

t2i = pxt.create_table(
    'runwayml_demo.text_to_image',
    {'prompt': pxt.String, 'ref_image': pxt.Image},
)

t2i.add_computed_column(
    response=runwayml.text_to_image(
        t2i.prompt, [t2i.ref_image], model='gen4_image', ratio='1280:720'
    )
)

# The response contains a list of output image URLs
t2i.add_computed_column(image=t2i.response['output'][0].astype(pxt.Image))

Created table ‘text_to_image’.
Added 0 column values with 0 errors in 0.01 s
Added 0 column values with 0 errors in 0.01 s
No rows affected.

t2i.insert(
    [
        {
            'prompt': 'A photorealistic painting in the same style as the reference',
            'ref_image': 'https://upload.wikimedia.org/wikipedia/commons/thumb/e/ec/Mona_Lisa%2C_by_Leonardo_da_Vinci%2C_from_C2RMF_retouched.jpg/800px-Mona_Lisa%2C_by_Leonardo_da_Vinci%2C_from_C2RMF_retouched.jpg',
        }
    ]
)

Inserted 1 row with 0 errors in 21.70 s (0.05 rows/s)
1 row inserted.

t2i.select(t2i.ref_image, t2i.image).collect()

Video-to-Video Transformation

Transform an existing video with text guidance. Note that video_to_video requires a publicly accessible HTTPS URL for the input video (not a local file).

v2v = pxt.create_table(
    'runwayml_demo.video_to_video',
    {'video_url': pxt.String, 'style_prompt': pxt.String},
)

v2v.add_computed_column(
    response=runwayml.video_to_video(
        v2v.video_url,
        v2v.style_prompt,
        model='gen4_aleph',
        ratio='1280:720',
    )
)

v2v.add_computed_column(video=v2v.response['output'][0].astype(pxt.Video))

Using `model_kwargs` for Advanced Parameters

All RunwayML functions accept an optional model_kwargs parameter for passing additional API parameters not exposed as explicit arguments. Refer to the RunwayML API docs for the full list of supported parameters per endpoint.

# Example: pass additional parameters via model_kwargs
advanced = pxt.create_table(
    'runwayml_demo.advanced', {'prompt': pxt.String}
)

advanced.add_computed_column(
    response=runwayml.text_to_video(
        advanced.prompt,
        model='gen4.5',
        ratio='1280:720',
        duration=5,
        model_kwargs={'audio': True, 'seed': 42},
    )
)

Learn More

RunwayML SDK Reference — Full API details for all RunwayML functions
RunwayML API Documentation — Official RunwayML API reference
Working with fal.ai — Another image/video generation integration
Working with Reve — AI video generation and editing

Welcome to Pixeltable

Tools

Core Concepts

How-To

What is RunwayML?

Documentation

Prerequisites

Setup

Text-to-Video Generation

Image-to-Video Generation

Text-to-Image Generation

Video-to-Video Transformation

Using `model_kwargs` for Advanced Parameters

Learn More

​What is RunwayML?

​Documentation

​Prerequisites

​Setup

​Text-to-Video Generation

​Image-to-Video Generation

​Text-to-Image Generation

​Video-to-Video Transformation

​Using model_kwargs for Advanced Parameters

​Learn More

What is RunwayML?

Documentation

Prerequisites

Setup

Text-to-Video Generation

Image-to-Video Generation

Text-to-Image Generation

Video-to-Video Transformation

Using `model_kwargs` for Advanced Parameters

Learn More