Llama.cpp
Working with llama.cpp in Pixeltable
This tutorial demonstrates how to use Pixeltable's built-in llama.cpp integration to run local LLMs efficiently.
If you are running this tutorial in Colab:
In order to make the tutorial run a bit snappier, let's switch to a GPU-equipped instance for this Colab session. To do that, click on the Runtime -> Change runtime type menu item at the top, then select the GPU radio button and click on Save.
Important Notes
- Models are automatically downloaded from Hugging Face and cached locally
- Different quantization levels are available for performance/quality tradeoffs
- Consider memory usage when choosing models and quantizations
Set Up Environment
First, let's install Pixeltable with llama.cpp support:
%pip install -qU pixeltable llama-cpp-python
Create a Table for Chat Completions
Now let's create a table that will contain our inputs and responses.
import pixeltable as pxt
from pixeltable.functions import llama_cpp
pxt.drop_dir('llama_demo', force=True)
pxt.create_dir('llama_demo')
t = pxt.create_table('llama_demo.chat', {'input': pxt.String})
Connected to Pixeltable database at: postgresql+psycopg://postgres:@/pixeltable?host=/Users/asiegel/.pixeltable/pgdata
Created directory `llama_demo`.
Created table `chat`.
Next, we add a computed column that calls the Pixeltable create_chat_completion UDF, which adapts the corresponding llama.cpp API call. In our examples, we'll use pretrained models from the Hugging Face repository. llama.cpp makes it easy to do this by specifying a repo_id (from the URL of the model) and filename from the model repo; the model will then be downloaded and cached automatically.
(If this is your first time using Pixeltable, the Pixeltable Fundamentals tutorial contains more details about table creation, computed columns, and UDFs.)
For this demo we'll use Qwen2.5-0.5B, a very small (0.5-billion parameter) model that still produces decent results. We'll use Q5_K_M (5-bit) quantization, which gives an excellent balance of quality and efficiency.
# Add a computed column that uses llama.cpp for chat completion
# against the input.
messages = [
{'role': 'system', 'content': 'You are a helpful assistant.'},
{'role': 'user', 'content': t.input}
]
t['result'] = llama_cpp.create_chat_completion(
messages,
repo_id='Qwen/Qwen2.5-0.5B-Instruct-GGUF',
repo_filename='*q5_k_m.gguf'
)
# Extract the output content from the JSON structure returned
# by llama_cpp.
t['output'] = t.result.choices[0].message.content
Added 0 column values with 0 errors.
Added 0 column values with 0 errors.
Test Chat Completion
Let's try a simple query:
# Test with a simple question
t.insert([
{'input': 'What is the capital of France?'},
{'input': 'What are some edible species of fish?'},
{'input': 'Who are the most prominent classical composers?'}
])
Computing cells: 100%|ββββββββββββββββββββββββββββββββββββββββββββ| 9/9 [00:03<00:00, 2.94 cells/s]
Inserting rows into `chat`: 3 rows [00:00, 1565.82 rows/s]
Computing cells: 100%|ββββββββββββββββββββββββββββββββββββββββββββ| 9/9 [00:03<00:00, 2.94 cells/s]
Inserted 3 rows with 0 errors.
UpdateStatus(num_rows=3, num_computed_values=9, num_excs=0, updated_cols=[], cols_with_excs=[])
t.select(t.input, t.output).collect()
| input | output |
|---|---|
| What is the capital of France? | Paris is the capital of France. |
| What are some edible species of fish? | Here are some edible species of fish that you may consider:
|
| Who are the most prominent classical composers? | The most prominent classical composers in the world are often referred to as "The Great Composers" or "The Four Great Composers" in Western classical music, and include the following composers:
These composers are still celebrated today and are widely regarded as some of the greatest classical composers in history. |
Comparing Models
Local model frameworks like llama.cpp make it easy to compare the output of different models. Let's try comparing the output from Qwen against a somewhat larger model, Llama-3.2-1B. As always, when we add a new computed column to our table, it's automatically evaluated against the existing table rows.
t['result_l3'] = llama_cpp.create_chat_completion(
messages,
repo_id='bartowski/Llama-3.2-1B-Instruct-GGUF',
repo_filename='*Q5_K_M.gguf'
)
t['output_l3'] = t.result_l3.choices[0].message.content
t.select(t.input, t.output, t.output_l3).collect()
Computing cells: 100%|ββββββββββββββββββββββββββββββββββββββββββββ| 3/3 [00:07<00:00, 2.62s/ cells]
Added 3 column values with 0 errors.
Computing cells: 100%|βββββββββββββββββββββββββββββββββββββββββββ| 3/3 [00:00<00:00, 608.52 cells/s]
Added 3 column values with 0 errors.
| input | output | output_l3 |
|---|---|---|
| What is the capital of France? | Paris is the capital of France. | The capital of France is Paris. |
| What are some edible species of fish? | Here are some edible species of fish that you may consider:
| Here are some popular edible species of fish:
Warm-Weather Fish:
Wild-Caught Fish:
|
| Who are the most prominent classical composers? | The most prominent classical composers in the world are often referred to as "The Great Composers" or "The Four Great Composers" in Western classical music, and include the following composers:
Beethoven - The most famous and influential classical composer of all time, with over 2, 000 concertos, sonatas, and symphonies. Beethoven's works have influenced generations of musicians, composers, and listeners. Bach - Another famous classical composer of the 17th and 18th centuries. His wo ...... and prolific classical composer of the late 18th and early 19th centuries. His works include the opera "Don Giovanni," the symphonies, the string quartets, and the concerto "Piano Concerto No. 20." Chopin - A Polish classical composer of the 19th century, known for his compositions of the Chopin Suite No. 1, the Ballade No. 2 in A minor, and the Waltz "Tante Marie." These composers are still celebrated today and are widely regarded as some of the greatest classical composers in history. | Here's a list of some of the most prominent classical composers:
|
Just for fun, let's try running against a different system prompt with a different persona.
messages_teacher = [
{'role': 'system',
'content': 'You are a patient school teacher. '
'Explain concepts simply and clearly.'},
{'role': 'user', 'content': t.input}
]
t['result_teacher'] = llama_cpp.create_chat_completion(
messages_teacher,
repo_id='bartowski/Llama-3.2-1B-Instruct-GGUF',
repo_filename='*Q5_K_M.gguf'
)
t['output_teacher'] = t.result_teacher.choices[0].message.content
t.select(t.input, t.output_teacher).collect()
Computing cells: 100%|ββββββββββββββββββββββββββββββββββββββββββββ| 3/3 [00:06<00:00, 2.30s/ cells]
Added 3 column values with 0 errors.
Computing cells: 100%|βββββββββββββββββββββββββββββββββββββββββββ| 3/3 [00:00<00:00, 605.33 cells/s]
Added 3 column values with 0 errors.
| input | output_teacher |
|---|---|
| What is the capital of France? | Bonjour! Don't worry if you didn't know that already. The capital of France is actually called Paris. But just to clarify, I'd like to break it down in a way that's easy to understand.
In simple terms, the capital of a country is the city where it's located. France, like many other countries, has a big city called Paris, which is its capital. If we think of it like this:
Now, the capital is a very special city, and it's home to a lot of important buildings, museums, and places where people go to make important decisions. Does that make sense? Do you have any other questions about France or its capital? |
| What are some edible species of fish? | Hello there, young learner. I'm excited to share some fascinating and delicious edible species of fish with you. Here are some common edible fish species, categorized by their preferred cooking methods: Freshwater Fish:
Other Edible Fish:
Remember, always check local fishing regulations and cooking guidelines for specific fish species to ensure food safety and sustainability. Do you have any other questions about fish? |
| Who are the most prominent classical composers? | As a teacher, I'd be delighted to introduce you to some of the most prominent classical composers. These are individuals who made significant contributions to the world of classical music, often leaving an indelible mark on the genre. Here are a few of the most famous ones:
Example: "Ei ...... and symphonies. Example: "Swan Lake" - a mesmerizing ballet with beautiful music
Example: "Air on the G String" - a beautiful, haunting melody
|
Additional Resources
Updated 13 days ago