module pixeltable.io

Functions for importing and exporting Pixeltable data.

func export_csv()

Signature

export_csv(
    table_or_query: pxt.Table | pxt.Query,
    file_path: str | Path,
    *,
    delimiter: str = ',',
    quoting: int = 0
) -> None

Exports a query result or table to a CSV file. Pixeltable column types are mapped to CSV values as follows:

String, Int, Float, Bool: native CSV representation
Timestamp, Date: ISO 8601 string representation
UUID: string representation
Json: JSON-encoded string
Array: JSON-encoded string (via tolist())
Binary: excluded from export (not representable in CSV)
Image, Video, Audio, Document: file path or URL string

Parameters:

table_or_query (pxt.Table | pxt.Query): Table or Query to export.
file_path (str | Path): Path to the output CSV file.
delimiter (str, default: ','): Field delimiter character. Default ','.
quoting (int, default: 0): CSV quoting style (a csv.QUOTE_* constant). Default csv.QUOTE_MINIMAL.

func export_iceberg()

Signature

export_iceberg(
    table_or_query: pxt.Table | pxt.Query,
    catalog: Catalog,
    table_name: str,
    *,
    batch_size_bytes: int = 134217728,
    if_exists: Literal['error', 'replace', 'append'] = 'error',
    schema_overrides: Mapping[str, pa.DataType] | None = None
) -> None

Exports a query result or table to an Apache Iceberg table. Data is streamed into the Iceberg table via pyarrow RecordBatches, the size of which can be controlled with the batch_size_bytes parameter. Requirements:

pip install pyiceberg

Parameters:

table_or_query (pxt.Table | pxt.Query): Pixeltable Table or Query to export.
catalog (Catalog): An Iceberg Catalog instance to write the table into.
table_name (str): Fully-qualified Iceberg table identifier (e.g. 'my_namespace.my_table'). If the namespace does not exist, it will be created.
batch_size_bytes (int, default: 134217728): Maximum size in bytes for each in-memory pyarrow batch.
if_exists (Literal['error', 'replace', 'append'], default: 'error'): Determines the behavior if the table already exists. Must be one of the following:
- 'error': raise an error
- 'replace': drop the existing table and create a new one
- 'append': append to the existing table (source schema must be compatible)
schema_overrides (Mapping[str, pa.DataType] | None): If specified, then for each (name, type) pair in schema_overrides, the column with name name will be given type type, instead of being inferred from the Pixeltable schema. The keys in schema_overrides should be the column names of the Pixeltable schema.

func export_images_as_fo_dataset()

Signature

export_images_as_fo_dataset(
    tbl: pxt.Table,
    images: exprs.Expr,
    image_format: str = 'webp',
    classifications: exprs.Expr | list[exprs.Expr] | dict[str, exprs.Expr] | None = None,
    detections: exprs.Expr | list[exprs.Expr] | dict[str, exprs.Expr] | None = None
) -> fo.Dataset

Export images from a Pixeltable table as a Voxel51 dataset. The data must consist of a single column (or expression) containing image data, along with optional additional columns containing labels. Currently, only classification and detection labels are supported. The Working with Voxel51 in Pixeltable tutorial contains a fully worked example showing how to export data from a Pixeltable table and load it into Voxel51. Images in the dataset that already exist on disk will be exported directly, in whatever format they are stored in. Images that are not already on disk (such as frames extracted using a frame_iterator) will first be written to disk in the specified image_format. The label parameters accept one or more sets of labels of each type. If a single Expr is provided, then it will be exported as a single set of labels with a default name such as classifications. (The single set of labels may still containing multiple individual labels; see below.) If a list of Exprs is provided, then each one will be exported as a separate set of labels with a default name such as classifications, classifications_1, etc. If a dictionary of Exprs is provided, then each entry will be exported as a set of labels with the specified name. Requirements:

pip install fiftyone

Parameters:

tbl (pxt.Table): The table from which to export data.
images (exprs.Expr): A column or expression that contains the images to export.
image_format (str, default: 'webp'): The format to use when writing out images for export.
classifications (exprs.Expr | list[exprs.Expr] | dict[str, exprs.Expr] | None): Optional image classification labels. If a single Expr is provided, it must be a table column or an expression that evaluates to a list of dictionaries. Each dictionary in the list corresponds to an image class and must have the following structure:
{'label': 'zebra', 'confidence': 0.325}
If multiple Exprs are provided, each one must evaluate to a list of such dictionaries.
detections (exprs.Expr | list[exprs.Expr] | dict[str, exprs.Expr] | None): Optional image detection labels. If a single Expr is provided, it must be a table column or an expression that evaluates to a list of dictionaries. Each dictionary in the list corresponds to an image detection, and must have the following structure:
{ 'label': 'giraffe', 'confidence': 0.99, # [x, y, w, h], fractional coordinates 'bounding_box': [0.081, 0.836, 0.202, 0.136], }
If multiple Exprs are provided, each one must evaluate to a list of such dictionaries.

Returns:

'fo.Dataset': A Voxel51 dataset.

Examples: Export the images in the image column of the table tbl as a Voxel51 dataset, using classification labels from tbl.classifications:

export_images_as_fo_dataset(
    tbl, tbl.image, classifications=tbl.classifications
)

See the Working with Voxel51 in Pixeltable tutorial for a fully worked example.

func export_json()

Signature

export_json(
    table_or_query: pxt.Table | pxt.Query,
    file_path: str | Path
) -> None

Exports a query result or table to a JSONL file. Pixeltable column types are mapped to JSON values as follows:

String: string
Int: number
Float: number
Bool: boolean
Timestamp: ISO 8601 string
Date: ISO 8601 string
UUID: string
Json: native JSON value (object, array, etc.)
Array: nested JSON array (via tolist())
Binary: excluded from export (not representable in JSON)
Image, Video, Audio, Document: file path or URL string

Parameters:

table_or_query (pxt.Table | pxt.Query): Table or Query to export.
file_path (str | Path): Path to the output JSONL file.

func export_lancedb()

Signature

export_lancedb(
    table_or_query: pxt.Table | pxt.Query,
    db_uri: Path,
    table_name: str,
    batch_size_bytes: int = 134217728,
    if_exists: Literal['error', 'overwrite', 'append'] = 'error'
) -> None

Exports a Query’s data to a LanceDB table. This utilizes LanceDB’s streaming interface for efficient table creation, via a sequence of in-memory pyarrow RecordBatches, the size of which can be controlled with the batch_size_bytes parameter. Requirements:

pip install lancedb pylance

Parameters:

table_or_query (Any): Table or Query to export.
db_uri (Path): Local Path to the LanceDB database.
table_name (Any): Name of the table in the LanceDB database.
batch_size_bytes (Any): Maximum size in bytes for each batch.
if_exists (Literal['error', 'overwrite', 'append'], default: 'error'): Determines the behavior if the table already exists. Must be one of the following:
- 'error': raise an error
- 'overwrite': overwrite the existing table
- 'append': append to the existing table

func export_parquet()

Signature

export_parquet(
    table_or_query: pxt.Table | pxt.Query,
    parquet_path: Path,
    partition_size_bytes: int = 100000000,
    inline_images: bool = False,
    _write_md: bool = False
) -> None

Exports a query result or table to one or more Parquet files. Requires pyarrow to be installed. Pixeltable column types are mapped to Parquet types as follows:

String: string
Int: int64
Float: float32
Bool: bool
Timestamp: timestamp[us, tz=UTC]
Date: date32
UUID: uuid
Binary: binary
Image: binary (when inline_images=True)
Audio, Video, Document: string (file paths)
Array (requires shape to be known):
- fixed_shape_tensor for fixed-shape arrays
- list for ragged arrays (one or more dimensions are None)
Json: struct
- Schema is inferred from data via pyarrow.infer_type()
- Fields that contain empty dicts cannot be mapped to a Parquet type and will result in an exception

Parameters:

table_or_query (Any): Table or Query to export.
parquet_path (Any): Path to directory to write the parquet files to.
partition_size_bytes (Any): The maximum target size for each chunk. Default 100_000_000 bytes.
inline_images (Any): If True, images are stored inline in the parquet file. This is useful for small images, to be imported as pytorch dataset. But can be inefficient for large images, and cannot be imported into pixeltable. If False, will raise an error if the Query has any image column. Default False.

func export_sql()

Signature

export_sql(
    table_or_query: pixeltable.catalog.table.Table | pixeltable._query.Query,
    target_table_name: str,
    *,
    db_connect_str: str,
    target_schema_name: str | None = None,
    if_exists: Literal['error', 'replace', 'insert'] = 'error',
    if_not_exists: Literal['error', 'create'] = 'create'
) -> None

Exports a query result or table to an RDBMS table. Parameters:

table_or_query (Any): Table or Query to export.
target_table_name (Any): Name of the target table.
db_connect_str (Any): Connection string to the target database.
target_schema_name (Any): Optional name of the target schema.
if_exists (Any): What to do if the target table already exists.
- ‘error’: raise an error
- ‘replace’: drop the existing table and create a new one
- ‘insert’: insert new rows into the existing table
if_not_exists (Any): What to do if the target table does not exist.
- ‘error’: raise an error
- ‘create’: create the table from the source schema

func import_csv()

Signature

import_csv(
    tbl_name: str,
    filepath_or_buffer: str | os.PathLike,
    schema_overrides: dict[str, Any] | None = None,
    primary_key: str | list[str] | None = None,
    comment: str | None = None,
    **kwargs: Any
) -> pxt.Table

Creates a new base table from a csv file. This is a convenience method and is equivalent to calling import_pandas(table_path, pd.read_csv(filepath_or_buffer, **kwargs), schema=schema). See the Pandas documentation for read_csv for more details. Returns:

pxt.Table: A handle to the newly created Table.

func import_excel()

Signature

import_excel(
    tbl_name: str,
    io: str | os.PathLike,
    *,
    schema_overrides: dict[str, Any] | None = None,
    primary_key: str | list[str] | None = None,
    comment: str = '',
    **kwargs: Any
) -> pxt.Table

Creates a new base table from an Excel (.xlsx) file. This is a convenience method and is equivalent to calling import_pandas(table_path, pd.read_excel(io, *args, **kwargs), schema=schema). See the Pandas documentation for read_excel for more details. Returns:

pxt.Table: A handle to the newly created Table.

func import_huggingface_dataset()

Signature

import_huggingface_dataset(
    table_path: str,
    dataset: datasets.Dataset | datasets.DatasetDict | datasets.IterableDataset | datasets.IterableDatasetDict,
    *,
    schema_overrides: dict[str, Any] | None = None,
    primary_key: str | list[str] | None = None,
    **kwargs: Any
) -> pxt.Table

Create a new base table from a Huggingface dataset, or dataset dict with multiple splits. Requires datasets library to be installed. HuggingFace feature types are mapped to Pixeltable column types as follows:

Value(bool): Bool
Value(int*/uint*): Int
Value(float*): Float
Value(string/large_string): String
Value(timestamp*): Timestamp
Value(date*): Date
ClassLabel: String (converted to label names)
Sequence/LargeList of numeric types: Array
Sequence/LargeList of string: Json
Sequence/LargeList of dicts: Json
Array2D-Array5D: Array (preserves shape)
Image: Image
Audio: Audio
Video: Video
Translation/TranslationVariableLanguages: Json

Parameters:

table_path (str): Path to the table.
dataset (datasets.Dataset | datasets.DatasetDict | datasets.IterableDataset | datasets.IterableDatasetDict): An instance of any of the Huggingface dataset classes: datasets.Dataset, datasets.DatasetDict, datasets.IterableDataset, datasets.IterableDatasetDict
schema_overrides (dict[str, Any] | None): If specified, then for each (name, type) pair in schema_overrides, the column with name name will be given type type, instead of being inferred from the Dataset or DatasetDict. The keys in schema_overrides should be the column names of the Dataset or DatasetDict (whether or not they are valid Pixeltable identifiers).
primary_key (str | list[str] | None): The primary key of the table (see create_table()).
kwargs (Any): Additional arguments to pass to create_table. An argument of column_name_for_split must be provided if the source is a DatasetDict. This column name will contain the split information. If None, no split information will be stored.

Returns:

pxt.Table: A handle to the newly created Table.

func import_json()

Signature

import_json(
    tbl_path: str,
    filepath_or_url: str,
    *,
    schema_overrides: dict[str, Any] | None = None,
    primary_key: str | list[str] | None = None,
    comment: str | None = None,
    **kwargs: Any
) -> pxt.Table

Creates a new base table from a JSON file. This is a convenience method and is equivalent to calling create_table() with pxt.create_table(tbl_path, source=filepath_or_url, extra_args=kwargs, ...). The contents of filepath_or_url are read and parsed as JSON internally (using json.loads(**kwargs)). Parameters:

tbl_path (str): The name of the table to create.
filepath_or_url (str): The path or URL of the JSON file.
schema_overrides (dict[str, Any] | None): If specified, then columns in schema_overrides will be given the specified types (see import_rows()).
primary_key (str | list[str] | None): The primary key of the table (see create_table()).
comment (str | None): A comment to attach to the table (see create_table()).
kwargs (Any): Additional keyword arguments to pass to json.loads.

Returns:

pxt.Table: A handle to the newly created Table.

func import_pandas()

Signature

import_pandas(
    tbl_name: str,
    df: pd.DataFrame,
    *,
    schema_overrides: dict[str, Any] | None = None,
    primary_key: str | list[str] | None = None,
    comment: str = ''
) -> pxt.Table

Creates a new base table from a Pandas DataFrame, with the specified name. The schema of the table will be inferred from the DataFrame. The column names of the new table will be identical to those in the DataFrame, as long as they are valid Pixeltable identifiers. If a column name is not a valid Pixeltable identifier, it will be normalized according to the following procedure:

first replace any non-alphanumeric characters with underscores;
then, preface the result with the letter ‘c’ if it begins with a number or an underscore;
then, if there are any duplicate column names, suffix the duplicates with ‘_2’, ‘_3’, etc., in column order.

Parameters:

tbl_name (str): The name of the table to create.
df (pd.DataFrame): The Pandas DataFrame.
schema_overrides (dict[str, Any] | None): If specified, then for each (name, type) pair in schema_overrides, the column with name name will be given type type, instead of being inferred from the DataFrame. The keys in schema_overrides should be the column names of the DataFrame (whether or not they are valid Pixeltable identifiers).

Returns:

pxt.Table: A handle to the newly created Table.

func import_parquet()

Signature

import_parquet(
    table: str,
    *,
    parquet_path: str,
    schema_overrides: dict[str, Any] | None = None,
    primary_key: str | list[str] | None = None,
    **kwargs: Any
) -> pxt.Table

Creates a new base table from a Parquet file or set of files. Requires pyarrow to be installed. Parameters:

table (str): Fully qualified name of the table to import the data into.
parquet_path (str): Path to an individual Parquet file or directory of Parquet files.
schema_overrides (dict[str, Any] | None): If specified, then for each (name, type) pair in schema_overrides, the column with name name will be given type type, instead of being inferred from the Parquet dataset. The keys in schema_overrides should be the column names of the Parquet dataset (whether or not they are valid Pixeltable identifiers).
primary_key (str | list[str] | None): The primary key of the table (see create_table()).
kwargs (Any): Additional arguments to pass to create_table.

Returns:

pxt.Table: A handle to the newly created table.

func import_rows()

Signature

import_rows(
    tbl_path: str,
    rows: list[dict[str, Any]],
    *,
    schema_overrides: dict[str, Any] | None = None,
    primary_key: str | list[str] | None = None,
    comment: str = ''
) -> pxt.Table

Creates a new base table from a list of dictionaries. The dictionaries must be of the form {column_name: value, ...}. Pixeltable will attempt to infer the schema of the table from the supplied data, using the most specific type that can represent all the values in a column. If schema_overrides is specified, then for each entry (column_name, type) in schema_overrides, Pixeltable will force the specified column to the specified type (and will not attempt any type inference for that column). All column types of the new table will be nullable unless explicitly specified as non-nullable in schema_overrides. Parameters:

tbl_path (str): The qualified name of the table to create.
rows (list[dict[str, Any]]): The list of dictionaries to import.
schema_overrides (dict[str, Any] | None): If specified, then columns in schema_overrides will be given the specified types as described above.
primary_key (str | list[str] | None): The primary key of the table (see create_table()).
comment (str, default: ''): A comment to attach to the table (see create_table()).

Returns:

pxt.Table: A handle to the newly created Table.

func import_sql()

Signature

import_sql(
    selectable: sqlalchemy.sql.selectable.Selectable,
    conn: sqlalchemy.engine.base.Engine | sqlalchemy.engine.base.Connection,
    tbl_name: str,
    *,
    schema_overrides: dict[str, typing.Any] | None = None,
    primary_key: str | list[str] | None = None,
    comment: str | None = None,
    custom_metadata: Any = None,
    if_exists: Literal['error', 'append'] = 'error',
    on_error: Literal['abort', 'ignore'] = 'abort',
    send_connect_url: bool = False
) -> pixeltable.catalog.table.Table

Import a SQL source into a Pixeltable table. Rows are streamed from the source via a server-side cursor and inserted in batches. Parameters:

selectable (sqlalchemy.sql.selectable.Selectable): A SQLAlchemy Selectable (a Table, a select() statement, or text(...).columns(...)) describing the source rows.
conn (sqlalchemy.engine.base.Engine | sqlalchemy.engine.base.Connection): A SQLAlchemy Engine or Connection to execute selectable against. If a Connection is passed, it must remain open and untouched (no commits, rollbacks, or other statements) for the duration of the import; rows are streamed directly from the server-side cursor.
tbl_name (str): Pixeltable path of the destination table.
schema_overrides (dict[str, typing.Any] | None): Optional per-column overrides applied on top of the inferred schema. Keys are column names; values accept any Pixeltable type spec recognized by pxt.create_table (eg, pxt.Image, pxt.Required[pxt.String]).
primary_key (str | list[str] | None): An optional column name or list of column names to use as the primary key(s) of the table. Only applies when a new table is created; ignored when appending to an existing table.
comment (str | None): An optional comment; its meaning is user-defined. Only applies when a new table is created; ignored when appending to an existing table.
custom_metadata (typing.Any): Optional user-defined metadata to associate with the table. Must be a valid JSON-serializable object [str, int, float, bool, dict, list]. Only applies when a new table is created; ignored when appending to an existing table.
if_exists (typing.Literal['error', 'append'], default: 'error'): How to handle the destination table.
- 'error': create the table; fail if it already exists.
- 'append': append into the table if it already exists (verifying the source schema is compatible); otherwise create it.
on_error (typing.Literal['abort', 'ignore'], default: 'abort'): Determines the behavior if an error occurs while evaluating a computed column or detecting an invalid media file (such as a corrupt image) for one of the inserted rows.
- If on_error='abort', then an exception will be raised and the rows will not be inserted.
- If on_error='ignore', then execution will continue and the rows will be inserted. Any cells with errors will have a None value for that cell, with information about the error stored in the corresponding tbl.col_name.errortype and tbl.col_name.errormsg fields.
send_connect_url (bool, default: False): Determines where the query is run, and is only relevant when the destination table is hosted. It has no effect on a local table.
- If False, the query is run locally and the result is sent to the server.
- If True, the server connects to the source database itself (using the source connection URL, including credentials); this requires the server’s host to be able to reach the database and is faster for large imports.

Returns:

pixeltable.catalog.table.Table: The destination Table.

​module pixeltable.io

​func export_csv()

​func export_iceberg()

​func export_images_as_fo_dataset()

​func export_json()

​func export_lancedb()

​func export_parquet()

​func export_sql()

​func import_csv()

​func import_excel()

​func import_huggingface_dataset()

​func import_json()

​func import_pandas()

​func import_parquet()

​func import_rows()

​func import_sql()

module pixeltable.io

func export_csv()

func export_iceberg()

func export_images_as_fo_dataset()

func export_json()

func export_lancedb()

func export_parquet()

func export_sql()

func import_csv()

func import_excel()

func import_huggingface_dataset()

func import_json()

func import_pandas()

func import_parquet()

func import_rows()

func import_sql()