Working with vLLM

This documentation page is also available as an interactive notebook. You can launch the notebook in Kaggle or Colab, or download it for use with an IDE or local Jupyter installation, by clicking one of the above links.

This tutorial demonstrates how to use Pixeltable’s built-in vLLM integration to run local LLMs with high-throughput inference.

Important notes

vLLM provides high-throughput inference with techniques like PagedAttention and continuous batching
Models are loaded from HuggingFace and cached in memory for reuse
vLLM currently requires a Linux environment with GPU support for best performance
Consider GPU memory when choosing model sizes

Set up environment

First, let’s install Pixeltable with vLLM support:

%pip install -qU pixeltable vllm

Create a table for chat completions

Now let’s create a table that will contain our inputs and responses.

import pixeltable as pxt
from pixeltable.functions import vllm

pxt.drop_dir('vllm_demo', force=True)
pxt.create_dir('vllm_demo')

t = pxt.create_table('vllm_demo/chat', {'input': pxt.String})

Created directory ‘vllm_demo’.
Created table ‘chat’.

Next, we add a computed column that calls the Pixeltable chat_completions UDF, which uses vLLM’s high-throughput inference engine under the hood. We specify a HuggingFace model identifier, and vLLM will download and cache the model automatically. (If this is your first time using Pixeltable, the Pixeltable Fundamentals tutorial contains more details about table creation, computed columns, and UDFs.) For this demo we’ll use Qwen2.5-0.5B-Instruct, a very small (0.5-billion parameter) model that still produces decent results.

# Add a computed column that uses vLLM for chat completion
# against the input.

messages = [
    {'role': 'system', 'content': 'You are a helpful assistant.'},
    {'role': 'user', 'content': t.input},
]

t.add_computed_column(
    result=vllm.chat_completions(
        messages, model='Qwen/Qwen2.5-0.5B-Instruct'
    )
)

# Extract the output content from the native vLLM response.

t.add_computed_column(output=t.result.outputs[0].text)

Added 0 column values with 0 errors in 0.01 s
Added 0 column values with 0 errors in 0.00 s
No rows affected.

Test chat completion

Let’s try a few queries:

# Test with a few questions
t.insert(
    [
        {'input': 'What is the capital of France?'},
        {'input': 'What are some edible species of fish?'},
        {'input': 'Who are the most prominent classical composers?'},
    ]
)

Inserted 3 rows with 0 errors in 1.74 s (1.72 rows/s)
3 rows inserted.

t.select(t.input, t.output).collect()

Comparing models

vLLM makes it easy to compare the output of different models. Let’s try comparing the output from Qwen2.5-0.5B against a somewhat larger model, Qwen2.5-1.5B-Instruct. As always, when we add a new computed column to our table, it’s automatically evaluated against the existing table rows.

t.add_computed_column(
    result_qwen15=vllm.chat_completions(
        messages, model='Qwen/Qwen2.5-1.5B-Instruct'
    )
)

t.add_computed_column(output_qwen15=t.result_qwen15.outputs[0].text)

t.select(t.input, t.output, t.output_qwen15).collect()

Added 3 column values with 0 errors in 3.45 s (0.87 rows/s)
Added 3 column values with 0 errors in 0.01 s (225.06 rows/s)

Using sampling parameters

vLLM supports fine-grained control over generation through sampling_params. Parameters like max_tokens, temperature, top_p, and top_k control the decoding behavior. Engine-level settings (such as max_model_len) can be passed separately via engine_args. Let’s try running with a different system prompt and custom sampling settings.

messages_teacher = [
    {
        'role': 'system',
        'content': 'You are a patient school teacher. Explain concepts simply and clearly.',
    },
    {'role': 'user', 'content': t.input},
]

t.add_computed_column(
    result_teacher=vllm.chat_completions(
        messages_teacher,
        model='Qwen/Qwen2.5-0.5B-Instruct',
        sampling_params={
            'max_tokens': 256,
            'temperature': 0.7,
            'top_p': 0.9,
        },
    )
)

t.add_computed_column(output_teacher=t.result_teacher.outputs[0].text)

t.select(t.input, t.output_teacher).collect()

Added 3 column values with 0 errors in 14.16 s (0.21 rows/s)
Added 3 column values with 0 errors in 0.01 s (271.08 rows/s)

Text generation

In addition to chat completions, vLLM also supports direct text generation with the generate UDF.

gen_t = pxt.create_table('vllm_demo/generation', {'prompt': pxt.String})

gen_t.add_computed_column(
    result=vllm.generate(
        gen_t.prompt,
        model='Qwen/Qwen2.5-0.5B-Instruct',
        sampling_params={'max_tokens': 100},
    )
)

gen_t.add_computed_column(output=gen_t.result.outputs[0].text)

gen_t.insert(
    [
        {'prompt': 'The capital of France is'},
        {'prompt': 'Once upon a time, there was a'},
    ]
)

gen_t.select(gen_t.prompt, gen_t.output).collect()

Created table ‘generation’.
Added 0 column values with 0 errors in 0.00 s
Added 0 column values with 0 errors in 0.00 s
Inserted 2 rows with 0 errors in 5.88 s (0.34 rows/s)

Built-in Integrations

Observability

Infrastructure

Bring Your Own

Providers

Important notes

Set up environment

Create a table for chat completions

Test chat completion

Comparing models

Using sampling parameters

Text generation

Additional Resources

​Important notes

​Set up environment

​Create a table for chat completions

​Test chat completion

​Comparing models

​Using sampling parameters

​Text generation

​Additional Resources

Important notes

Set up environment

Create a table for chat completions

Test chat completion

Comparing models

Using sampling parameters

Text generation

Additional Resources