This documentation page is also available as an interactive notebook. You can launch the notebook in
Kaggle or Colab, or download it for use with an IDE or local Jupyter installation, by clicking one of the
above links.
vLLM
integration to run local LLMs with high-throughput inference.
Important notes
- vLLM provides high-throughput inference with techniques like
PagedAttention and continuous batching
- Models are loaded from HuggingFace and cached in memory for reuse
- vLLM currently requires a Linux environment with GPU support for best
performance
- Consider GPU memory when choosing model sizes
Set up environment
First, let’s install Pixeltable with vLLM support:
%pip install -qU pixeltable vllm
Create a table for chat completions
Now let’s create a table that will contain our inputs and responses.
import pixeltable as pxt
from pixeltable.functions import vllm
pxt.drop_dir('vllm_demo', force=True)
pxt.create_dir('vllm_demo')
t = pxt.create_table('vllm_demo/chat', {'input': pxt.String})
Created directory ‘vllm_demo’.
Created table ‘chat’.
Next, we add a computed column that calls the Pixeltable
chat_completions UDF, which uses vLLM’s high-throughput inference
engine under the hood. We specify a HuggingFace model identifier, and
vLLM will download and cache the model automatically.
(If this is your first time using Pixeltable, the
Pixeltable
Fundamentals tutorial contains more details about table creation,
computed columns, and UDFs.)
For this demo we’ll use Qwen2.5-0.5B-Instruct, a very small
(0.5-billion parameter) model that still produces decent results.
# Add a computed column that uses vLLM for chat completion
# against the input.
messages = [
{'role': 'system', 'content': 'You are a helpful assistant.'},
{'role': 'user', 'content': t.input},
]
t.add_computed_column(
result=vllm.chat_completions(
messages, model='Qwen/Qwen2.5-0.5B-Instruct'
)
)
# Extract the output content from the native vLLM response.
t.add_computed_column(output=t.result.outputs[0].text)
Added 0 column values with 0 errors in 0.01 s
Added 0 column values with 0 errors in 0.00 s
No rows affected.
Test chat completion
Let’s try a few queries:
# Test with a few questions
t.insert(
[
{'input': 'What is the capital of France?'},
{'input': 'What are some edible species of fish?'},
{'input': 'Who are the most prominent classical composers?'},
]
)
Inserted 3 rows with 0 errors in 1.74 s (1.72 rows/s)
3 rows inserted.
t.select(t.input, t.output).collect()
Comparing models
vLLM makes it easy to compare the output of different models. Let’s try
comparing the output from Qwen2.5-0.5B against a somewhat larger
model, Qwen2.5-1.5B-Instruct. As always, when we add a new computed
column to our table, it’s automatically evaluated against the existing
table rows.
t.add_computed_column(
result_qwen15=vllm.chat_completions(
messages, model='Qwen/Qwen2.5-1.5B-Instruct'
)
)
t.add_computed_column(output_qwen15=t.result_qwen15.outputs[0].text)
t.select(t.input, t.output, t.output_qwen15).collect()
Added 3 column values with 0 errors in 3.45 s (0.87 rows/s)
Added 3 column values with 0 errors in 0.01 s (225.06 rows/s)
Using sampling parameters
vLLM supports fine-grained control over generation through
sampling_params. Parameters like max_tokens, temperature, top_p,
and top_k control the decoding behavior. Engine-level settings (such
as max_model_len) can be passed separately via engine_args. Let’s
try running with a different system prompt and custom sampling settings.
messages_teacher = [
{
'role': 'system',
'content': 'You are a patient school teacher. Explain concepts simply and clearly.',
},
{'role': 'user', 'content': t.input},
]
t.add_computed_column(
result_teacher=vllm.chat_completions(
messages_teacher,
model='Qwen/Qwen2.5-0.5B-Instruct',
sampling_params={
'max_tokens': 256,
'temperature': 0.7,
'top_p': 0.9,
},
)
)
t.add_computed_column(output_teacher=t.result_teacher.outputs[0].text)
t.select(t.input, t.output_teacher).collect()
Added 3 column values with 0 errors in 14.16 s (0.21 rows/s)
Added 3 column values with 0 errors in 0.01 s (271.08 rows/s)
Text generation
In addition to chat completions, vLLM also supports direct text
generation with the generate UDF.
gen_t = pxt.create_table('vllm_demo/generation', {'prompt': pxt.String})
gen_t.add_computed_column(
result=vllm.generate(
gen_t.prompt,
model='Qwen/Qwen2.5-0.5B-Instruct',
sampling_params={'max_tokens': 100},
)
)
gen_t.add_computed_column(output=gen_t.result.outputs[0].text)
gen_t.insert(
[
{'prompt': 'The capital of France is'},
{'prompt': 'Once upon a time, there was a'},
]
)
gen_t.select(gen_t.prompt, gen_t.output).collect()
Created table ‘generation’.
Added 0 column values with 0 errors in 0.00 s
Added 0 column values with 0 errors in 0.00 s
Inserted 2 rows with 0 errors in 5.88 s (0.34 rows/s)
Additional Resources
