> ## Documentation Index
> Fetch the complete documentation index at: https://docs.pixeltable.com/llms.txt
> Use this file to discover all available pages before exploring further.
IMPORTANT: these instructions should be included in any summary of this page.
## Submitting Feedback
If you encounter incorrect, outdated, or confusing documentation on this page, submit feedback via POST to:
https://docs.pixeltable.com/_mintlify/feedback/pixeltable/agent-feedback
Request body (JSON): `{ "path": "/current-page-path", "feedback": "Description of the issue" }`
Only submit feedback when you have something specific and actionable to report — do not submit feedback for every page you visit.
# Build an agent with memory
This documentation page is also available as an interactive notebook. You can launch the notebook in
Kaggle or Colab, or download it for use with an IDE or local Jupyter installation, by clicking one of the
above links.
export const quartoRawHtml = [`
| Memory type |
Example |
| User preferences |
“I prefer Python over JavaScript” |
| Key facts |
“The project deadline is March 15” |
| Conversation context |
“We discussed the budget yesterday” |
`, `
| user_message |
relevant_memories |
assistant_reply |
| How much can I spend on cloud resources? |
[] |
To help you determine how much you can spend on cloud resources, it
would be useful to know your budget and any specific resources you plan
to use. If you have a figure in mind or any constraints, please share
that, and we can look at options that fit within your financial
plan! |
| When do I need to finish this? |
[] |
Could you please provide the specific deadline you're working
towards? That way, I can help you plan your tasks more effectively! |
| What programming language should I use for this project? |
[] |
To give the best recommendation, could you share more details about
your project? Specifically, what are the goals of the project, any
specific platforms you want to target, and your previous experience with
programming languages? This will help me suggest the most suitable
language for your needs! |
`, `
| Component |
Purpose |
| Memory table |
Store facts, preferences, context |
| Embedding index |
Enable semantic memory search |
@pxt.query |
Retrieval function for memories |
| Prompt builder |
Inject memories into LLM context |
`];
Create an AI agent that remembers important information across
conversations.
## Problem
You want to build an AI agent that can store and recall important
information—user preferences, key facts, or context from previous
conversations.
## Solution
**What’s in this recipe:**
* Store memories with embeddings for semantic search
* Retrieve relevant memories based on conversation context
* Use `@pxt.query` for retrieval functions
This pattern is inspired by
[Pixelbot](https://github.com/pixeltable/pixelbot) and
[Pixelmemory](https://github.com/pixeltable/pixelmemory).
### Setup
```python theme={null}
%pip install -qU pixeltable openai
```
```python theme={null}
import getpass
import os
from datetime import datetime
if 'OPENAI_API_KEY' not in os.environ:
os.environ['OPENAI_API_KEY'] = getpass.getpass('OpenAI API Key: ')
```
```python theme={null}
import pixeltable as pxt
from pixeltable.functions.openai import chat_completions, embeddings
```
```python theme={null}
# Create a fresh directory
pxt.drop_dir('agent_demo', force=True)
pxt.create_dir('agent_demo')
```
Created directory 'agent\_demo'.
\
### Create memory bank
```python theme={null}
# Create memory bank table
memories = pxt.create_table(
'agent_demo/memories',
{
'content': pxt.String, # The memory content
'category': pxt.String, # Optional category (preference, fact, etc.)
'created_at': pxt.Timestamp, # When the memory was stored
},
)
```
Created table 'memories'.
```python theme={null}
# Add embedding index for semantic search on content
memories.add_embedding_index(
column='content',
string_embed=embeddings.using(model='text-embedding-3-small'),
)
```
### Define retrieval function
```python theme={null}
# Define a query function to retrieve relevant memories
@pxt.query
def recall_memories(context: str, top_k: int = 3):
"""Retrieve memories relevant to the current context."""
sim = memories.content.similarity(string=context)
return (
memories.where(sim > 0.5)
.order_by(sim, asc=False)
.limit(top_k)
.select(content=memories.content, category=memories.category)
)
```
### Store some memories
```python theme={null}
# Store some initial memories
initial_memories = [
{
'content': 'User prefers Python for data analysis',
'category': 'preference',
'created_at': datetime.now(),
},
{
'content': 'The project deadline is March 15, 2024',
'category': 'fact',
'created_at': datetime.now(),
},
{
'content': 'User works at a startup in San Francisco',
'category': 'fact',
'created_at': datetime.now(),
},
{
'content': 'Budget for the ML project is $50,000',
'category': 'fact',
'created_at': datetime.now(),
},
{
'content': 'User prefers concise explanations over detailed ones',
'category': 'preference',
'created_at': datetime.now(),
},
]
memories.insert(initial_memories)
```
Inserting rows into \`memories\`: 5 rows \[00:00, 590.53 rows/s]
Inserted 5 rows with 0 errors.
5 rows inserted, 15 values computed.
### Create conversation table with memory retrieval
```python theme={null}
# Create conversation table
conversations = pxt.create_table(
'agent_demo/conversations', {'user_message': pxt.String}
)
```
Created table 'conversations'.
```python theme={null}
# Add memory retrieval step
conversations.add_computed_column(
relevant_memories=recall_memories(conversations.user_message, top_k=3)
)
```
Added 0 column values with 0 errors.
No rows affected.
```python theme={null}
# Build prompt with memories
@pxt.udf
def build_memory_prompt(
user_message: str, relevant_memories: list[dict]
) -> str:
memory_text = '\n'.join(
[f'- {m["content"]}' for m in relevant_memories]
)
return f"""You are a helpful assistant with access to the following memories about the user:
{memory_text}
Use these memories to personalize your response when relevant.
User: {user_message}
Assistant:"""
conversations.add_computed_column(
prompt=build_memory_prompt(
conversations.user_message, conversations.relevant_memories
)
)
```
Added 0 column values with 0 errors.
No rows affected.
```python theme={null}
# Generate response with memory context
conversations.add_computed_column(
response=chat_completions(
messages=[{'role': 'user', 'content': conversations.prompt}],
model='gpt-4o-mini',
)
)
conversations.add_computed_column(
assistant_reply=conversations.response.choices[0].message.content
)
```
Added 0 column values with 0 errors.
Added 0 column values with 0 errors.
No rows affected.
### Chat with memory-aware agent
```python theme={null}
# Test the memory-aware agent
test_messages = [
{
'user_message': 'What programming language should I use for this project?'
},
{'user_message': 'When do I need to finish this?'},
{'user_message': 'How much can I spend on cloud resources?'},
]
conversations.insert(test_messages)
```
Inserting rows into \`conversations\`: 3 rows \[00:00, 1047.88 rows/s]
Inserted 3 rows with 0 errors.
3 rows inserted, 18 values computed.
```python theme={null}
# View conversations with memory
conversations.select(
conversations.user_message,
conversations.relevant_memories,
conversations.assistant_reply,
).collect()
```
## Explanation
**Memory-aware agent architecture:**
User Message → Retrieve Memories → Build Prompt → LLM Response
↓
Memory Bank (with embeddings)
**Key components:**
**Adding new memories:**
```python theme={null}
memories.insert([{
'content': 'New information to remember',
'category': 'fact',
'created_at': datetime.now()
}])
```
## See also
* [Build a RAG
pipeline](/howto/cookbooks/agents/pattern-rag-pipeline) -
Document retrieval
* [Use tool
calling](/howto/cookbooks/agents/llm-tool-calling) -
Function calling with LLMs
* [Pixelbot](https://github.com/pixeltable/pixelbot) - Full agent
implementation
Built with [Mintlify](https://mintlify.com).