> ## Documentation Index
> Fetch the complete documentation index at: https://docs.pixeltable.com/llms.txt
> Use this file to discover all available pages before exploring further.
# Working with vLLM
This documentation page is also available as an interactive notebook. You can launch the notebook in
Kaggle or Colab, or download it for use with an IDE or local Jupyter installation, by clicking one of the
above links.
export const quartoRawHtml = [`
| input |
output |
| What is the capital of France? |
The capital of France is Paris. |
| What are some edible species of fish? |
Some edible species of fish include: 1. Salmon 2. Trout 3 |
| Who are the most prominent classical composers? |
There have been many notable classical composers throughout history,
and their contributions to music have |
`, `
| input |
output |
output_qwen15 |
| What is the capital of France? |
The capital of France is Paris. |
The capital of France is Paris. |
| What are some edible species of fish? |
Some edible species of fish include: 1. Salmon 2. Trout 3 |
Some edible species of fish include salmon, trout, cod,
halibut, |
| Who are the most prominent classical composers? |
There have been many notable classical composers throughout history,
and their contributions to music have |
There have been many influential classical composers throughout
history, but some of the most prominent |
`, `
| input |
output_teacher |
| What is the capital of France? |
The capital of France is Paris. |
| What are some edible species of fish? |
Edible species of fish are a fascinating group of aquatic creatures
that can be enjoyed for their various flavors and nutritional benefits.
Here are some examples: 1. **Salmon**: These are a type of fish that is
often caught for their rich, oily flesh. They are popular in many
cuisines around the world. 2. **Haddock**: A species of large, flat fish
that can be used in many recipes, from fish tacos to fish puree. 3.
**Cod**: Another type of fish, cod is known for its rich, buttery flavor
a ...... particularly in Canada and the United States. It is known for
its mild flavor and can be used in soups, stews, and as a base for
sauces. 5. **Eel**: Eels are a type of fish that can be found in
saltwater and freshwater environments. They are often smoked or boiled
for a unique flavor. 6. **Pufferfish**: Pufferfish is a species of fish
that can be quite toxic. They are sometimes used in certain dishes for
their unique flavor. 7. **Flounder**: Flounders are a type of fish that
can be caught |
| Who are the most prominent classical composers? |
The most prominent classical composers include: 1. Mozart: He was a
master of the symphonic form, known for his piano sonatas and operas. 2.
Beethoven: He was a pivotal figure in the development of symphonic
structure and helped shape the classical era. 3. Bach: He composed in a
variety of styles, including cantatas and operas, and is considered the
father of classical music. 4. Haydn: He was a prolific composer who is
often referred to as the "father of the classical era." 5. Beethoven's
son: Ludwig von Beethoven, who lived to be 56, was a prodigious composer
who also wrote piano sonatas and other works. These composers, along
with others, have had a profound influence on the development of
classical music and have been celebrated for their mastery of the form
and their contributions to the art of music. |
`, `
| prompt |
output |
| The capital of France is |
Paris, the center of political, economic, cultural and social life.
It is also found at the foot of the Montmartre hill under the
exaggerated view of sculptor, Eugène Balzac. What are the enlargements
where you could observe Paris's views? A: B: C: D: To determine which
city has the capital, we need to understand the concept of capital. The
capital of a country is a defined location where that country's
government is located. Therefore, |
| Once upon a time, there was a |
very large tree. It had a very wide root, leaving it can not reach
any further, and it had a fantastic friend named Frog who could keep the
tree tree safe. One sunny day, a fox came to visit the tree. When it put
down its cane, it hurt the tree root. The root was strong since it had
been there for a very long time, so when the owner's friend Frog got
mad, he gave the fox a test. The next day, when the fox tried |
`];
This tutorial demonstrates how to use Pixeltable’s built-in `vLLM`
integration to run local LLMs with high-throughput inference.
### Important notes
* vLLM provides high-throughput inference with techniques like
PagedAttention and continuous batching
* Models are loaded from HuggingFace and cached in memory for reuse
* vLLM currently requires a Linux environment with GPU support for best
performance
* Consider GPU memory when choosing model sizes
## Set up environment
First, let’s install Pixeltable with vLLM support:
```python theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
%pip install -qU pixeltable vllm
```
## Create a table for chat completions
Now let’s create a table that will contain our inputs and responses.
```python theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
import pixeltable as pxt
from pixeltable.functions import vllm
pxt.drop_dir('vllm_demo', force=True)
pxt.create_dir('vllm_demo')
t = pxt.create_table('vllm_demo/chat', {'input': pxt.String})
```
Created directory 'vllm\_demo'.
Created table 'chat'.
Next, we add a computed column that calls the Pixeltable
`chat_completions` UDF, which uses vLLM’s high-throughput inference
engine under the hood. We specify a HuggingFace model identifier, and
vLLM will download and cache the model automatically.
(If this is your first time using Pixeltable, the
Pixeltable
Fundamentals tutorial contains more details about table creation,
computed columns, and UDFs.)
For this demo we’ll use `Qwen2.5-0.5B-Instruct`, a very small
(0.5-billion parameter) model that still produces decent results.
```python theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
# Add a computed column that uses vLLM for chat completion
# against the input.
messages = [
{'role': 'system', 'content': 'You are a helpful assistant.'},
{'role': 'user', 'content': t.input},
]
t.add_computed_column(
result=vllm.chat_completions(
messages, model='Qwen/Qwen2.5-0.5B-Instruct'
)
)
# Extract the output content from the native vLLM response.
t.add_computed_column(output=t.result.outputs[0].text)
```
Added 0 column values with 0 errors in 0.01 s
Added 0 column values with 0 errors in 0.00 s
No rows affected.
## Test chat completion
Let’s try a few queries:
```python theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
# Test with a few questions
t.insert(
[
{'input': 'What is the capital of France?'},
{'input': 'What are some edible species of fish?'},
{'input': 'Who are the most prominent classical composers?'},
]
)
```
Inserted 3 rows with 0 errors in 1.74 s (1.72 rows/s)
3 rows inserted.
```python theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
t.select(t.input, t.output).collect()
```
## Comparing models
vLLM makes it easy to compare the output of different models. Let’s try
comparing the output from `Qwen2.5-0.5B` against a somewhat larger
model, `Qwen2.5-1.5B-Instruct`. As always, when we add a new computed
column to our table, it’s automatically evaluated against the existing
table rows.
```python theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
t.add_computed_column(
result_qwen15=vllm.chat_completions(
messages, model='Qwen/Qwen2.5-1.5B-Instruct'
)
)
t.add_computed_column(output_qwen15=t.result_qwen15.outputs[0].text)
t.select(t.input, t.output, t.output_qwen15).collect()
```
Added 3 column values with 0 errors in 3.45 s (0.87 rows/s)
Added 3 column values with 0 errors in 0.01 s (225.06 rows/s)
## Using sampling parameters
vLLM supports fine-grained control over generation through
`sampling_params`. Parameters like `max_tokens`, `temperature`, `top_p`,
and `top_k` control the decoding behavior. Engine-level settings (such
as `max_model_len`) can be passed separately via `engine_args`. Let’s
try running with a different system prompt and custom sampling settings.
```python theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
messages_teacher = [
{
'role': 'system',
'content': 'You are a patient school teacher. Explain concepts simply and clearly.',
},
{'role': 'user', 'content': t.input},
]
t.add_computed_column(
result_teacher=vllm.chat_completions(
messages_teacher,
model='Qwen/Qwen2.5-0.5B-Instruct',
sampling_params={
'max_tokens': 256,
'temperature': 0.7,
'top_p': 0.9,
},
)
)
t.add_computed_column(output_teacher=t.result_teacher.outputs[0].text)
t.select(t.input, t.output_teacher).collect()
```
Added 3 column values with 0 errors in 14.16 s (0.21 rows/s)
Added 3 column values with 0 errors in 0.01 s (271.08 rows/s)
## Text generation
In addition to chat completions, vLLM also supports direct text
generation with the `generate` UDF.
```python theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
gen_t = pxt.create_table('vllm_demo/generation', {'prompt': pxt.String})
gen_t.add_computed_column(
result=vllm.generate(
gen_t.prompt,
model='Qwen/Qwen2.5-0.5B-Instruct',
sampling_params={'max_tokens': 100},
)
)
gen_t.add_computed_column(output=gen_t.result.outputs[0].text)
gen_t.insert(
[
{'prompt': 'The capital of France is'},
{'prompt': 'Once upon a time, there was a'},
]
)
gen_t.select(gen_t.prompt, gen_t.output).collect()
```
Created table 'generation'.
Added 0 column values with 0 errors in 0.00 s
Added 0 column values with 0 errors in 0.00 s
Inserted 2 rows with 0 errors in 5.88 s (0.34 rows/s)
## Additional Resources
* [Pixeltable Documentation](/)
* [vLLM Documentation](https://docs.vllm.ai/)
* [vLLM GitHub](https://github.com/vllm-project/vllm)