module pixeltable.functions.openai

Pixeltable UDFs that wrap various endpoints from the OpenAI API. In order to use them, you must first pip install openai and configure your OpenAI credentials, as described in the Working with OpenAI tutorial.

func invoke_tools()

invoke_tools(
    tools: pixeltable.func.tools.Tools,
    response: pixeltable.exprs.expr.Expr
) -> pixeltable.exprs.inline_expr.InlineDict

Converts an OpenAI response dict to Pixeltable tool invocation format and calls tools._invoke().

udf chat_completions()

chat_completions(
    messages: Json,
    *,
    model: String,
    model_kwargs: Json | None = None,
    tools: Json | None = None,
    tool_choice: Json | None = None
) -> Json

Creates a model response for the given chat conversation. Equivalent to the OpenAI chat/completions API endpoint. For additional details, see: https://platform.openai.com/docs/guides/chat-completions Request throttling: Uses the rate limit-related headers returned by the API to throttle requests adaptively, based on available request and token capacity. No configuration is necessary. Requirements:

pip install openai

Parameters:

messages (Json): A list of messages to use for chat completion, as described in the OpenAI API documentation.
model (String): The model to use for chat completion.
model_kwargs (Json | None): Additional keyword args for the OpenAI chat/completions API. For details on the available parameters, see: https://platform.openai.com/docs/api-reference/chat/create

Returns:

Json: A dictionary containing the response and other metadata.

Examples: Add a computed column that applies the model gpt-4o-mini to an existing Pixeltable column tbl.prompt of the table tbl:

messages = [
    {'role': 'system', 'content': 'You are a helpful assistant.'},
    {'role': 'user', 'content': tbl.prompt},
]
tbl.add_computed_column(
    response=chat_completions(messages, model='gpt-4o-mini')
)

udf embeddings()

embeddings(
    input: String,
    *,
    model: String,
    model_kwargs: Json | None = None
) -> Array[(None,), Float]

Creates an embedding vector representing the input text. Equivalent to the OpenAI embeddings API endpoint. For additional details, see: https://platform.openai.com/docs/guides/embeddings Request throttling: Uses the rate limit-related headers returned by the API to throttle requests adaptively, based on available request and token capacity. No configuration is necessary. Requirements:

pip install openai

Parameters:

input (String): The text to embed.
model (String): The model to use for the embedding.
model_kwargs (Json | None): Additional keyword args for the OpenAI embeddings API. For details on the available parameters, see: https://platform.openai.com/docs/api-reference/embeddings

Returns:

Array[(None,), Float]: An array representing the application of the given embedding to input.

Examples: Add a computed column that applies the model text-embedding-3-small to an existing Pixeltable column tbl.text of the table tbl:

tbl.add_computed_column(
    embed=embeddings(tbl.text, model='text-embedding-3-small')
)

Add an embedding index to an existing column text, using the model text-embedding-3-small:

tbl.add_embedding_index(
    embedding=embeddings.using(model='text-embedding-3-small')
)

udf image_generations()

image_generations(
    prompt: String,
    *,
    model: String = 'dall-e-2',
    model_kwargs: Json | None = None
) -> Image

Creates an image given a prompt. Equivalent to the OpenAI images/generations API endpoint. For additional details, see: https://platform.openai.com/docs/guides/images Request throttling: Applies the rate limit set in the config (section openai.rate_limits; use the model id as the key). If no rate limit is configured, uses a default of 600 RPM. Requirements:

pip install openai

Parameters:

prompt (String): Prompt for the image.
model (String): The model to use for the generations.
model_kwargs (Json | None): Additional keyword args for the OpenAI images/generations API. For details on the available parameters, see: https://platform.openai.com/docs/api-reference/images/create

Returns:

Image: The generated image.

Examples: Add a computed column that applies the model dall-e-2 to an existing Pixeltable column tbl.text of the table tbl:

tbl.add_computed_column(
    gen_image=image_generations(tbl.text, model='dall-e-2')
)

udf moderations()

moderations(
    input: String,
    *,
    model: String = 'omni-moderation-latest'
) -> Json

Classifies if text is potentially harmful. Equivalent to the OpenAI moderations API endpoint. For additional details, see: https://platform.openai.com/docs/guides/moderation Request throttling: Applies the rate limit set in the config (section openai.rate_limits; use the model id as the key). If no rate limit is configured, uses a default of 600 RPM. Requirements:

pip install openai

Parameters:

input (String): Text to analyze with the moderations model.
model (String): The model to use for moderations.

Returns:

Json: Details of the moderations results.

Examples: Add a computed column that applies the model text-moderation-stable to an existing Pixeltable column tbl.input of the table tbl:

tbl.add_computed_column(
    moderations=moderations(tbl.text, model='text-moderation-stable')
)

udf speech()

speech(
    input: String,
    *,
    model: String,
    voice: String,
    model_kwargs: Json | None = None
) -> Audio

Generates audio from the input text. Equivalent to the OpenAI audio/speech API endpoint. For additional details, see: https://platform.openai.com/docs/guides/text-to-speech Request throttling: Applies the rate limit set in the config (section openai.rate_limits; use the model id as the key). If no rate limit is configured, uses a default of 600 RPM. Requirements:

pip install openai

Parameters:

input (String): The text to synthesize into speech.
model (String): The model to use for speech synthesis.
voice (String): The voice profile to use for speech synthesis. Supported options include: alloy, echo, fable, onyx, nova, and shimmer.
model_kwargs (Json | None): Additional keyword args for the OpenAI audio/speech API. For details on the available parameters, see: https://platform.openai.com/docs/api-reference/audio/createSpeech

Returns:

Audio: An audio file containing the synthesized speech.

Examples: Add a computed column that applies the model tts-1 to an existing Pixeltable column tbl.text of the table tbl:

tbl.add_computed_column(
    audio=speech(tbl.text, model='tts-1', voice='nova')
)

udf transcriptions()

transcriptions(
    audio: Audio,
    *,
    model: String,
    model_kwargs: Json | None = None
) -> Json

Transcribes audio into the input language. Equivalent to the OpenAI audio/transcriptions API endpoint. For additional details, see: https://platform.openai.com/docs/guides/speech-to-text Request throttling: Applies the rate limit set in the config (section openai.rate_limits; use the model id as the key). If no rate limit is configured, uses a default of 600 RPM. Requirements:

pip install openai

Parameters:

audio (Audio): The audio to transcribe.
model (String): The model to use for speech transcription.
model_kwargs (Json | None): Additional keyword args for the OpenAI audio/transcriptions API. For details on the available parameters, see: https://platform.openai.com/docs/api-reference/audio/createTranscription

Returns:

Json: A dictionary containing the transcription and other metadata.

Examples: Add a computed column that applies the model whisper-1 to an existing Pixeltable column tbl.audio of the table tbl:

tbl.add_computed_column(
    transcription=transcriptions(
        tbl.audio, model='whisper-1', language='en'
    )
)

udf translations()

translations(
    audio: Audio,
    *,
    model: String,
    model_kwargs: Json | None = None
) -> Json

Translates audio into English. Equivalent to the OpenAI audio/translations API endpoint. For additional details, see: https://platform.openai.com/docs/guides/speech-to-text Request throttling: Applies the rate limit set in the config (section openai.rate_limits; use the model id as the key). If no rate limit is configured, uses a default of 600 RPM. Requirements:

pip install openai

Parameters:

audio (Audio): The audio to translate.
model (String): The model to use for speech transcription and translation.
model_kwargs (Json | None): Additional keyword args for the OpenAI audio/translations API. For details on the available parameters, see: https://platform.openai.com/docs/api-reference/audio/createTranslation

Returns:

Json: A dictionary containing the translation and other metadata.

Examples: Add a computed column that applies the model whisper-1 to an existing Pixeltable column tbl.audio of the table tbl:

tbl.add_computed_column(
    translation=translations(tbl.audio, model='whisper-1', language='en')
)

udf vision()

vision(
    prompt: String,
    image: Image,
    *,
    model: String,
    model_kwargs: Json | None = None
) -> String

Analyzes an image with the OpenAI vision capability. This is a convenience function that takes an image and prompt, and constructs a chat completion request that utilizes OpenAI vision. For additional details, see: https://platform.openai.com/docs/guides/vision Request throttling: Uses the rate limit-related headers returned by the API to throttle requests adaptively, based on available request and token capacity. No configuration is necessary. Requirements:

pip install openai

Parameters:

prompt (String): A prompt for the OpenAI vision request.
image (Image): The image to analyze.
model (String): The model to use for OpenAI vision.

Returns:

String: The response from the OpenAI vision API.

Examples: Add a computed column that applies the model gpt-4o-mini to an existing Pixeltable column tbl.image of the table tbl:

tbl.add_computed_column(
    response=vision(
        "What's in this image?", tbl.image, model='gpt-4o-mini'
    )
)

​module pixeltable.functions.openai

​func invoke_tools()

​udf chat_completions()

​udf embeddings()

​udf image_generations()

​udf moderations()

​udf speech()

​udf transcriptions()

​udf translations()

​udf vision()

module pixeltable.functions.openai

func invoke_tools()

udf chat_completions()

udf embeddings()

udf image_generations()

udf moderations()

udf speech()

udf transcriptions()

udf translations()

udf vision()