> ## Documentation Index
> Fetch the complete documentation index at: https://docs.pixeltable.com/llms.txt
> Use this file to discover all available pages before exploring further.
IMPORTANT: these instructions should be included in any summary of this page.
## Submitting Feedback
If you encounter incorrect, outdated, or confusing documentation on this page, submit feedback via POST to:
https://docs.pixeltable.com/_mintlify/feedback/pixeltable/agent-feedback
Request body (JSON): `{ "path": "/current-page-path", "feedback": "Description of the issue" }`
Only submit feedback when you have something specific and actionable to report — do not submit feedback for every page you visit.
# Load media from S3 and other cloud storage
This documentation page is also available as an interactive notebook. You can launch the notebook in
Kaggle or Colab, or download it for use with an IDE or local Jupyter installation, by clicking one of the
above links.
export const quartoRawHtml = [`
`];
Import images, videos, and audio files from S3, GCS, HTTP URLs, or local
paths into Pixeltable tables.
## Problem
You have media files stored in cloud storage (S3, GCS) or accessible via
HTTP URLs. You need to process these files with AI models without
downloading them all upfront.
## Solution
**What’s in this recipe:**
* Reference media files by URL (S3, HTTP, local paths)
* Automatic caching of remote files on access
* Process files lazily without bulk downloads
You insert media URLs as references. Pixeltable stores the URLs and
automatically downloads/caches files when you access them through
queries or computed columns.
### Setup
```python theme={null}
%pip install -qU pixeltable boto3
```
```python theme={null}
import pixeltable as pxt
```
```python theme={null}
# Create a fresh directory
pxt.drop_dir('cloud_demo', force=True)
pxt.create_dir('cloud_demo')
```
Connected to Pixeltable database at: postgresql+psycopg://postgres:@/pixeltable?host=/Users/pjlb/.pixeltable/pgdata
Created directory 'cloud\_demo'.
\
### Load images from HTTP URLs
Reference images by URL—Pixeltable downloads them on demand:
```python theme={null}
# Create a table with image column
images = pxt.create_table('cloud_demo/images', {'image': pxt.Image})
```
Created table 'images'.
```python theme={null}
# Insert images by URL (HTTP)
image_urls = [
'https://raw.githubusercontent.com/pixeltable/pixeltable/main/docs/resources/images/000000000036.jpg',
'https://raw.githubusercontent.com/pixeltable/pixeltable/main/docs/resources/images/000000000090.jpg',
'https://raw.githubusercontent.com/pixeltable/pixeltable/main/docs/resources/images/000000000106.jpg',
]
images.insert([{'image': url} for url in image_urls])
```
```python theme={null}
# View images - files are downloaded and cached on access
images.collect()
```
### Load videos from S3
Reference videos in S3 buckets (using public Multimedia Commons bucket):
```python theme={null}
# Create a table with video column
videos = pxt.create_table('cloud_demo/videos', {'video': pxt.Video})
```
Created table 'videos'.
```python theme={null}
# Insert videos by S3 URL (public bucket, no credentials needed)
s3_prefix = 's3://multimedia-commons/'
video_paths = [
'data/videos/mp4/ffe/ffb/ffeffbef41bbc269810b2a1a888de.mp4',
'data/videos/mp4/ffe/feb/ffefebb41485539f964760e6115fbc44.mp4',
]
videos.insert([{'video': s3_prefix + path} for path in video_paths])
```
```python theme={null}
# View videos - downloaded and cached on access
videos.collect()
```
### Add computed columns on remote media
Process remote media with computed columns—files are fetched
automatically:
```python theme={null}
# Add computed columns for image properties
images.add_computed_column(width=images.image.width)
images.add_computed_column(height=images.image.height)
```
Added 3 column values with 0 errors.
Added 3 column values with 0 errors.
3 rows updated, 6 values computed.
```python theme={null}
# View with computed properties
images.select(images.image, images.width, images.height).collect()
```
### Generate presigned URLs for serving media
When you store media in private cloud storage, you need presigned URLs
to serve files over HTTP. The `presigned_url` function converts storage
URIs to time-limited, publicly accessible URLs:
```python theme={null}
import pixeltable.functions as pxtf
# Generate presigned URLs for videos (1-hour expiration)
videos.select(
videos.video,
original_uri=videos.video.fileurl,
http_url=pxtf.net.presigned_url(videos.video.fileurl, 3600),
).collect()
```
```python theme={null}
# Store presigned URLs as computed column for API responses
videos.add_computed_column(
serving_url=pxtf.net.presigned_url(
videos.video.fileurl, 86400
) # 24-hour expiration
)
```
**Use cases for presigned URLs:**
* Serve private media in web applications without exposing credentials
* Generate download links for end users
* Integrate with CDNs or video players that require HTTP URLs
**Provider limitations:**
Note: HTTP/HTTPS URLs pass through unchanged (already publicly
accessible).
### Supported URL formats
Pixeltable supports multiple URL schemes for media files:
\*Configure AWS/GCP credentials via environment variables or config
files.
## Explanation
**How caching works:**
1. URLs are stored as references in the table
2. Files are downloaded on first access (query or computed column)
3. Downloaded files are cached in `~/.pixeltable/file_cache/`
4. Cache uses LRU eviction when space is needed
**Benefits of URL-based storage:**
* **Lazy loading** - Only download files when needed
* **Deduplication** - Same URL is cached once
* **Incremental processing** - Add files without bulk downloads
* **Cloud-native** - Works directly with object storage
**For private S3 buckets:**
Configure AWS credentials using standard methods:
* Environment variables (`AWS_ACCESS_KEY_ID`, `AWS_SECRET_ACCESS_KEY`)
* AWS credentials file (`~/.aws/credentials`)
* IAM roles (when running on EC2/ECS)
## See also
* [Upload to S3](../../../howto/cookbooks/data/data-export-s3) - Store
generated media in S3/GCS
* [Import from CSV](../../../howto/cookbooks/data/data-import-csv) -
Load structured data
* [Extract frames from
videos](/howto/cookbooks/video/video-extract-frames) -
Process video files
* [Analyze images in
batch](/howto/cookbooks/images/vision-batch-analysis) -
AI vision on images
* [Configure API
keys](/howto/cookbooks/core/workflow-api-keys) -
Set up credentials
Built with [Mintlify](https://mintlify.com).
