Skip to main content
This tutorial demonstrates how to use Pixeltable’s built-in llama.cpp integration to run local LLMs efficiently. If you are running this tutorial in Colab: In order to make the tutorial run a bit snappier, let’s switch to a GPU-equipped instance for this Colab session. To do that, click on the Runtime -> Change runtime type menu item at the top, then select the GPU radio button and click on Save.

​
Important Notes

  • Models are automatically downloaded from Hugging Face and cached locally
  • Different quantization levels are available for performance/quality tradeoffs
  • Consider memory usage when choosing models and quantizations

​
Set Up Environment

First, let’s install Pixeltable with llama.cpp support: 
%pip install -qU pixeltable llama-cpp-python huggingface-hub

​
Create a Table for Chat Completions

Now let’s create a table that will contain our inputs and responses. 
import pixeltable as pxt
from pixeltable.functions import llama_cpp

pxt.drop_dir('llama_demo', force=True)
pxt.create_dir('llama_demo')

t = pxt.create_table('llama_demo.chat', {'input': pxt.String})

Connected to Pixeltable database at: postgresql+psycopg://postgres:@/pixeltable?host=/Users/asiegel/.pixeltable/pgdata
Created directory `llama_demo`.
Created table `chat`.
Next, we add a computed column that calls the Pixeltable create_chat_completion UDF, which adapts the corresponding llama.cpp API call. In our examples, we’ll use pretrained models from the Hugging Face repository. llama.cpp makes it easy to do this by specifying a repo_id (from the URL of the model) and filename from the model repo; the model will then be downloaded and cached automatically. (If this is your first time using Pixeltable, the Pixeltable Fundamentals tutorial contains more details about table creation, computed columns, and UDFs.) For this demo we’ll use Qwen2.5-0.5B, a very small (0.5-billion parameter) model that still produces decent results. We’ll use Q5_K_M (5-bit) quantization, which gives an excellent balance of quality and efficiency. 
# Add a computed column that uses llama.cpp for chat completion
# against the input.

messages = [
    {'role': 'system', 'content': 'You are a helpful assistant.'},
    {'role': 'user', 'content': t.input}
]

t.add_computed_column(result=llama_cpp.create_chat_completion(
    messages,
    repo_id='Qwen/Qwen2.5-0.5B-Instruct-GGUF',
    repo_filename='*q5_k_m.gguf'
))

# Extract the output content from the JSON structure returned
# by llama_cpp.

t.add_computed_column(output=t.result.choices[0].message.content)

Added 0 column values with 0 errors.
Added 0 column values with 0 errors.

UpdateStatus(num_rows=0, num_computed_values=0, num_excs=0, updated_cols=[], cols_with_excs=[])

​
Test Chat Completion

Let’s try a simple query: 
# Test with a simple question
t.insert([
    {'input': 'What is the capital of France?'},
    {'input': 'What are some edible species of fish?'},
    {'input': 'Who are the most prominent classical composers?'}
])

Computing cells: 100%|████████████████████████████████████████████| 9/9 [00:03<00:00,  2.25 cells/s]
Inserting rows into `chat`: 3 rows [00:00, 1112.74 rows/s]
Computing cells: 100%|████████████████████████████████████████████| 9/9 [00:03<00:00,  2.25 cells/s]
Inserted 3 rows with 0 errors.

UpdateStatus(num_rows=3, num_computed_values=9, num_excs=0, updated_cols=[], cols_with_excs=[])

t.select(t.input, t.output).collect()
input output
What is the capital of France? Paris is the capital of France.
What are some edible species of fish? Here are some edible species of fish that you may consider: 1. Tilapia 2. Cod 3. Salmon 4. Trout 5. Mahi-mahi (Mako) 6. Bluefish 7. Eel 8. Sardines 9. Shrimps 10. Scallops Remember to check the environmental impact of fishing and take steps to protect wildlife and the ocean.
Who are the most prominent classical composers? The most prominent classical composers in the world are often referred to as "The Great Composers" or "The Four Great Composers" in Western classical music, and include the following composers: 1. Beethoven - The most famous and influential classical composer of all time, with over 2, 000 concertos, sonatas, and symphonies. Beethoven's works have influenced generations of musicians, composers, and listeners. 2. Bach - Another famous classical composer of the 17th and 18th centuries. His wo ...... and prolific classical composer of the late 18th and early 19th centuries. His works include the opera "Don Giovanni," the symphonies, the string quartets, and the concerto "Piano Concerto No. 20." 4. Chopin - A Polish classical composer of the 19th century, known for his compositions of the Chopin Suite No. 1, the Ballade No. 2 in A minor, and the Waltz "Tante Marie." These composers are still celebrated today and are widely regarded as some of the greatest classical composers in history.

​
Comparing Models

Local model frameworks like llama.cpp make it easy to compare the output of different models. Let’s try comparing the output from Qwen against a somewhat larger model, Llama-3.2-1B. As always, when we add a new computed column to our table, it’s automatically evaluated against the existing table rows. 
t.add_computed_column(result_l3=llama_cpp.create_chat_completion(
    messages,
    repo_id='bartowski/Llama-3.2-1B-Instruct-GGUF',
    repo_filename='*Q5_K_M.gguf'
))

t.add_computed_column(output_l3=t.result_l3.choices[0].message.content)

t.select(t.input, t.output, t.output_l3).collect()

Computing cells: 100%|████████████████████████████████████████████| 3/3 [00:08<00:00,  2.74s/ cells]
Added 3 column values with 0 errors.
Computing cells: 100%|███████████████████████████████████████████| 3/3 [00:00<00:00, 349.89 cells/s]
Added 3 column values with 0 errors.
input output output_l3
What is the capital of France? Paris is the capital of France. The capital of France is Paris.
What are some edible species of fish? Here are some edible species of fish that you may consider: 1. Tilapia 2. Cod 3. Salmon 4. Trout 5. Mahi-mahi (Mako) 6. Bluefish 7. Eel 8. Sardines 9. Shrimps 10. Scallops Remember to check the environmental impact of fishing and take steps to protect wildlife and the ocean. Here are some popular edible species of fish: **Warm-Weather Fish:** 1. Grouper (Epinephelus spp.): found in tropical and subtropical waters, these fish are rich in protein and low in fat. 2. Mahi-Mahi (Coryphaena hippurus): a popular game fish with a flavor similar to swordfish. 3. Snapper (Lutjanus spp.): found in warm waters, these fish are known for their mild flavor and firm texture. 4. Wahoo (Acanthurus triangularis): a fast-swimming fish with a rich, buttery flavor. 5. Sailfish (Ist ...... fish with a meaty texture and rich flavor. 3. Marlin (Makaira mazara): a fast-swimming fish with a rich, buttery flavor. 4. Bluefin Tuna (Thunnus thynnus): a highly prized fish with a rich, savory flavor. 5. Sharks (various species): while not typically considered a "fish," sharks are an important food source for many cultures. **Wild-Caught Fish:** 1. Sablefish (Dicentrarchus labrax): a fatty fish with a rich, buttery flavor. 2. Eel (Anguilla anguilla): a soft, sweet fish with a delicate
Who are the most prominent classical composers? The most prominent classical composers in the world are often referred to as "The Great Composers" or "The Four Great Composers" in Western classical music, and include the following composers: 1. Beethoven - The most famous and influential classical composer of all time, with over 2, 000 concertos, sonatas, and symphonies. Beethoven's works have influenced generations of musicians, composers, and listeners. 2. Bach - Another famous classical composer of the 17th and 18th centuries. His wo ...... and prolific classical composer of the late 18th and early 19th centuries. His works include the opera "Don Giovanni," the symphonies, the string quartets, and the concerto "Piano Concerto No. 20." 4. Chopin - A Polish classical composer of the 19th century, known for his compositions of the Chopin Suite No. 1, the Ballade No. 2 in A minor, and the Waltz "Tante Marie." These composers are still celebrated today and are widely regarded as some of the greatest classical composers in history. Here's a list of some of the most prominent classical composers: 1. **Claude Debussy (1862-1918)**: French composer known for his Impressionist and Symbolist music, particularly for the piano and orchestral works. 2. **Ludwig van Beethoven (1770-1827)**: German composer and pianist who revolutionized classical music with his symphonies, piano sonatas, and chamber music. 3. **Wolfgang Amadeus Mozart (1756-1791)**: Austrian composer of Mozart and his brother, widely regarded as one of the gre ...... or who pioneered the art of opera seria, particularly with his epic opera, Das Rheingold. 9. **Antonín Dvořák (1841-1904)**: Czech composer known for his symphonies, opera, and chamber music, which reflect the spirit of his homeland. 10. **George Frideric Handel (1685-1759)**: German-born British composer and organist famous for his operas, orchestral suites, and church music. 11. **Carl Maria von Weber (1786-1826)**: German composer who was a key figure in the development of Romantic music,
Just for fun, let’s try running against a different system prompt with a different persona. 
messages_teacher = [
    {'role': 'system',
     'content': 'You are a patient school teacher. '
                'Explain concepts simply and clearly.'},
    {'role': 'user', 'content': t.input}
]

t.add_computed_column(result_teacher=llama_cpp.create_chat_completion(
    messages_teacher,
    repo_id='bartowski/Llama-3.2-1B-Instruct-GGUF',
    repo_filename='*Q5_K_M.gguf'
))

t.add_computed_column(output_teacher=t.result_teacher.choices[0].message.content)

t.select(t.input, t.output_teacher).collect()

Computing cells: 100%|████████████████████████████████████████████| 3/3 [00:06<00:00,  2.30s/ cells]
Added 3 column values with 0 errors.
Computing cells: 100%|███████████████████████████████████████████| 3/3 [00:00<00:00, 605.33 cells/s]
Added 3 column values with 0 errors.
input output_teacher
What is the capital of France? Bonjour! Don't worry if you didn't know that already. The capital of France is actually called Paris. But just to clarify, I'd like to break it down in a way that's easy to understand. In simple terms, the capital of a country is the city where it's located. France, like many other countries, has a big city called Paris, which is its capital. If we think of it like this: * The country (France) = where it's located * The city (Paris) = where the government, or the capital, is located Now, the capital is a very special city, and it's home to a lot of important buildings, museums, and places where people go to make important decisions. Does that make sense? Do you have any other questions about France or its capital?
What are some edible species of fish? Hello there, young learner. I'm excited to share some fascinating and delicious edible species of fish with you. Here are some common edible fish species, categorized by their preferred cooking methods: **Freshwater Fish:** 1. Trout: A popular choice for pan-frying or baking, with a delicate flavor and firm texture. 2. Bass: Similar to trout, but with a slightly sweeter taste and firmer flesh. 3. Channel Catfish: A mild-flavored fish with a soft texture, great for breaded and fried dishe ...... 6. Grouper: A mild-flavored fish with a firm texture, often used in breaded and fried dishes. **Other Edible Fish:** 1. Pollock: A mild-flavored fish with a soft texture, often used in breaded and fried dishes. 2. Halibut: A firm-textured fish with a slightly sweet flavor, often used in baking or grilling. Remember, always check local fishing regulations and cooking guidelines for specific fish species to ensure food safety and sustainability. Do you have any other questions about fish?
Who are the most prominent classical composers? As a teacher, I'd be delighted to introduce you to some of the most prominent classical composers. These are individuals who made significant contributions to the world of classical music, often leaving an indelible mark on the genre. Here are a few of the most famous ones: 1. **Wolfgang Amadeus Mozart** (1756-1791): A child prodigy, Mozart is considered one of the most influential composers in history. He wrote over 600 works, including symphonies, operas, and chamber music. Example: "Ei ...... and symphonies. Example: "Swan Lake" - a mesmerizing ballet with beautiful music 5. **Johann Sebastian Bach** (1685-1750): A German composer and organist, Bach is widely regarded as one of the greatest composers in history. He's known for his complex and intricate compositions, including orchestral suites, cantatas, and organ music. Example: "Air on the G String" - a beautiful, haunting melody 6. **Georg Friedrich Handel** (1685-1759): A German composer, Handel is famous for his operas,

​
Additional Resources