| table 'transcription_demo/video_table' |
| Column Name | Type | Computed With |
|---|---|---|
| video | Video |
| video |
|---|
| video | audio |
|---|---|
| table 'transcription_demo/video_table' |
| Column Name | Type | Computed With |
|---|---|---|
| video | Video | |
| audio | Audio | extract_audio(video, format='mp3') |
| video | audio | metadata |
|---|---|---|
| {"size": 959084, "streams": [{"type": "audio", "frames": 0, "duration": 845703936, "metadata": {"encoder": "Lavf"}, "time_base": 7.086e-08, "codec_context": {"name": "mp3float", "profile": null, "channels": 2, "codec_tag": "\\x00\\x00\\x00\\x00"}, "duration_seconds": 59.928}], "bit_rate": 128031, "metadata": {"encoder": "Lavf61.7.100"}, "bit_exact": false} | ||
| {"size": 959084, "streams": [{"type": "audio", "frames": 0, "duration": 845703936, "metadata": {"encoder": "Lavf"}, "time_base": 7.086e-08, "codec_context": {"name": "mp3float", "profile": null, "channels": 2, "codec_tag": "\\x00\\x00\\x00\\x00"}, "duration_seconds": 59.928}], "bit_rate": 128031, "metadata": {"encoder": "Lavf61.7.100"}, "bit_exact": false} |
| video | transcription_text |
|---|---|
| of experiencing self versus remembering self. I was hoping you can give a simple answer of how we should live life. Based on the fact that our memories could be a source of happiness or could be the primary source of happiness, that an event when experienced bears its fruits the most when it's remembered over and over and over and over. And maybe there is some wisdom in the fact that we can control to some degree how we remember how we evolve our memory of it, such that it can maximize the long-term happiness of that repeated experience. Oh, well, first I'll say I wish I could take you on the road with me. That was such a great description. Can I be your opening answer? Oh my God, no, I'm gonna open for you, dude. Otherwise it's like, you know, everybody leaves after you're done. | |
| worse, the young adults had episodic memory. And I always say, would say, God, that's so weird. Why would we have this period of time that's so short when we're perfect, right? Or optimal. And I like to use that word optimal now because there's such a culture of optimization right now. And it's like, I realize I have to redefine what optimal is because for most of the human condition, I think we had a series of stages of life where you have basically adults saying, okay, young adults saying, I've got a child and, you know, I'm part of this village and I have to hunt and forage and get things done. I need a prefrontal cortex so I can stay focused on the big picture and the long haul goals. Now, I'm a child. I'm in this village. I'm kind of wandering around. And I've got some safety and I need to learn about this culture because I know so little. What's the best way to do that? Let's explore. I don't want to be constrained by goals as much. I want to really be |
| pos | text |
|---|---|
| 0 | of experiencing self versus remembering self. |
| 1 | I was hoping you can give a simple answer of how we should live life. |
| 2 | Based on the fact that our memories could be a source of happiness or could be the primary source of happiness, that an event when experienced bears its fruits the most when it's remembered over and over and over and over. |
| 3 | And maybe there is some wisdom in the fact that we can control to some degree how we remember how we evolve our memory of it, such that it can maximize the long-term happiness of that repeated experience. |
| 4 | Oh, well, first I'll say I wish I could take you on the road with me. |
| 5 | That was such a great description. |
| 6 | Can I be your opening answer? |
| 7 | Oh my God, no, I'm gonna open for you, dude. |
| text | similarity |
|---|---|
| Can I be your opening answer? | 0.804 |
| Or optimal. | 0.79 |
| I was hoping you can give a simple answer of how we should live life. | 0.788 |
| of experiencing self versus remembering self. | 0.779 |
| What's the best way to do that? | 0.777 |
| Why would we have this period of time that's so short when we're perfect, right? | 0.773 |
| I need a prefrontal cortex so I can stay focused on the big picture and the long haul goals. | 0.769 |
| I don't want to be constrained by goals as much. | 0.768 |
| Based on the fact that our memories could be a source of happiness or could be the primary source of happiness, that an event when experienced bears its fruits the most when it's remembered over and over and over and over. | 0.766 |
| I'm kind of wandering around. | 0.761 |
| video | metadata | transcription_text |
|---|---|---|
| {"size": 959084, "streams": [{"type": "audio", "frames": 0, "duration": 845703936, "metadata": {"encoder": "Lavf"}, "time_base": 7.086e-08, "codec_context": {"name": "mp3float", "profile": null, "channels": 2, "codec_tag": "\\x00\\x00\\x00\\x00"}, "duration_seconds": 59.928}], "bit_rate": 128031, "metadata": {"encoder": "Lavf61.7.100"}, "bit_exact": false} | of experiencing self versus remembering self. I was hoping you can give a simple answer of how we should live life. Based on the fact that our memories could be a source of happiness or could be the primary source of happiness, that an event when experienced bears its fruits the most when it's remembered over and over and over and over. And maybe there is some wisdom in the fact that we can control to some degree how we remember how we evolve our memory of it, such that it can maximize the long-term happiness of that repeated experience. Oh, well, first I'll say I wish I could take you on the road with me. That was such a great description. Can I be your opening answer? Oh my God, no, I'm gonna open for you, dude. Otherwise it's like, you know, everybody leaves after you're done. | |
| {"size": 959084, "streams": [{"type": "audio", "frames": 0, "duration": 845703936, "metadata": {"encoder": "Lavf"}, "time_base": 7.086e-08, "codec_context": {"name": "mp3float", "profile": null, "channels": 2, "codec_tag": "\\x00\\x00\\x00\\x00"}, "duration_seconds": 59.928}], "bit_rate": 128031, "metadata": {"encoder": "Lavf61.7.100"}, "bit_exact": false} | worse, the young adults had episodic memory. And I always say, would say, God, that's so weird. Why would we have this period of time that's so short when we're perfect, right? Or optimal. And I like to use that word optimal now because there's such a culture of optimization right now. And it's like, I realize I have to redefine what optimal is because for most of the human condition, I think we had a series of stages of life where you have basically adults saying, okay, young adults saying, I've got a child and, you know, I'm part of this village and I have to hunt and forage and get things done. I need a prefrontal cortex so I can stay focused on the big picture and the long haul goals. Now, I'm a child. I'm in this village. I'm kind of wandering around. And I've got some safety and I need to learn about this culture because I know so little. What's the best way to do that? Let's explore. I don't want to be constrained by goals as much. I want to really be | |
| {"size": 959084, "streams": [{"type": "audio", "frames": 0, "duration": 845703936, "metadata": {"encoder": "Lavf"}, "time_base": 7.086e-08, "codec_context": {"name": "mp3float", "profile": null, "channels": 2, "codec_tag": "\\x00\\x00\\x00\\x00"}, "duration_seconds": 59.928}], "bit_rate": 128031, "metadata": {"encoder": "Lavf61.7.100"}, "bit_exact": false} | reusing information and making the most of what we already have. And so that's why basically again, what you see biologically is, you know, neuromodulators, for instance, these chemicals in the brain, like norepinephrine, dopamine, serotonin. These are chemicals that are released during moments that tend to be biologically significant, surprise, fear, stress, etc. And so these chemicals promote lasting plasticity, right? Essentially, some mechanisms for which the brain can say prioritize the information that you carry with you into the future. Attention is a big factor as well, our ability to focus our attention on what's important. And so there's different schools of thought on training attention, for instance. So one of my colleagues, Amishi Jia, she wrote a book called Peak Mind and talks about mindfulness as a method for improving attention. And |
| text | similarity |
|---|---|
| Can I be your opening answer? | 0.804 |
| Attention is a big factor as well, our ability to focus our attention on what's important. | 0.795 |
| Or optimal. | 0.79 |
| I was hoping you can give a simple answer of how we should live life. | 0.788 |
| These are chemicals that are released during moments that tend to be biologically significant, surprise, fear, stress, etc. | 0.781 |
| of experiencing self versus remembering self. | 0.779 |
| And | 0.779 |
| What's the best way to do that? | 0.777 |
| Essentially, some mechanisms for which the brain can say prioritize the information that you carry with you into the future. | 0.775 |
| Why would we have this period of time that's so short when we're perfect, right? | 0.773 |
| I need a prefrontal cortex so I can stay focused on the big picture and the long haul goals. | 0.769 |
| I don't want to be constrained by goals as much. | 0.768 |
| Based on the fact that our memories could be a source of happiness or could be the primary source of happiness, that an event when experienced bears its fruits the most when it's remembered over and over and over and over. | 0.766 |
| I'm kind of wandering around. | 0.761 |
| reusing information and making the most of what we already have. | 0.76 |
| And maybe there is some wisdom in the fact that we can control to some degree how we remember how we evolve our memory of it, such that it can maximize the long-term happiness of that repeated experience. | 0.76 |
| And so these chemicals promote lasting plasticity, right? | 0.756 |
| And so there's different schools of thought on training attention, for instance. | 0.755 |
| That was such a great description. | 0.754 |
| So one of my colleagues, Amishi Jia, she wrote a book called Peak Mind and talks about mindfulness as a method for improving attention. | 0.754 |
| video | transcription_text | transcriptionfromapi_text |
|---|---|---|
| of experiencing self versus remembering self. I was hoping you can give a simple answer of how we should live life. Based on the fact that our memories could be a source of happiness or could be the primary source of happiness, that an event when experienced bears its fruits the most when it's remembered over and over and over and over. And maybe there is some wisdom in the fact that we can control to some degree how we remember how we evolve our memory of it, such that it can maximize the long-term happiness of that repeated experience. Oh, well, first I'll say I wish I could take you on the road with me. That was such a great description. Can I be your opening answer? Oh my God, no, I'm gonna open for you, dude. Otherwise it's like, you know, everybody leaves after you're done. | of experiencing self versus remembering self, I was hoping you can give a simple answer of how we should live life, based on the fact that our memories could be a source of happiness, or could be the primary source of happiness, that an event, when experienced, bears its fruits the most when it's remembered over and over and over and over, and maybe there is some wisdom in the fact that we can control, to some degree, how we remember it, how we evolve our memory of it, such that it can maximize the long-term happiness of that repeated experience. Okay, well, first I'll say, I wish I could take you on the road with me, because that was such a great description. Can I be your opening excerpt? Oh my God, no, I'm gonna open for you, dude. Otherwise, it's like, you know, everybody leaves after you're done. Ha ha ha ha! | |
| worse, the young adults had episodic memory. And I always say, would say, God, that's so weird. Why would we have this period of time that's so short when we're perfect, right? Or optimal. And I like to use that word optimal now because there's such a culture of optimization right now. And it's like, I realize I have to redefine what optimal is because for most of the human condition, I think we had a series of stages of life where you have basically adults saying, okay, young adults saying, I've got a child and, you know, I'm part of this village and I have to hunt and forage and get things done. I need a prefrontal cortex so I can stay focused on the big picture and the long haul goals. Now, I'm a child. I'm in this village. I'm kind of wandering around. And I've got some safety and I need to learn about this culture because I know so little. What's the best way to do that? Let's explore. I don't want to be constrained by goals as much. I want to really be | or worse, the young adults at episodic memory. And I always would say, God, this is so weird. Why would we have this period of time that's so short when we're perfect, right? Or optimal. And I like to use that word optimal now because there's such a culture of optimization right now. And it's like, I realized I have to redefine what optimal is because for most of the human condition, I think we had a series of stages of life where you have basically adults saying, okay, young adults saying, I've got a child and I'm part of this village and I have to hunt and forage and get things done. I need a prefrontal cortex so I can stay focused on the big picture and long haul goals. Now I'm a child, I'm in this village, I'm kind of wandering around and I've got some safety and I need to learn about this culture because I know so little. What's the best way to do that? Let's explore. I don't want to be constrained by goals as much. I want to really be free. | |
| reusing information and making the most of what we already have. And so that's why basically again, what you see biologically is, you know, neuromodulators, for instance, these chemicals in the brain, like norepinephrine, dopamine, serotonin. These are chemicals that are released during moments that tend to be biologically significant, surprise, fear, stress, etc. And so these chemicals promote lasting plasticity, right? Essentially, some mechanisms for which the brain can say prioritize the information that you carry with you into the future. Attention is a big factor as well, our ability to focus our attention on what's important. And so there's different schools of thought on training attention, for instance. So one of my colleagues, Amishi Jia, she wrote a book called Peak Mind and talks about mindfulness as a method for improving attention. And | about reusing information and making the most of what we already have. And so that's why basically, again, what you see biologically is neuromodulators, for instance, these chemicals in the brain like norepinephrine, dopamine, serotonin. These are chemicals that are released during moments that tend to be biologically significant, surprise, fear, stress, et cetera. And so these chemicals promote lasting plasticity, right? Essentially some mechanisms by which the brain can prioritize the information that you carry with you into the future. Attention is a big factor as well, our ability to focus our attention on what's important. And so there's different schools of thought on training attention, for instance. So one of my colleagues, Amishi Jha, she wrote a book called Peak Mind and talks about mindfulness as a method for improving attention and focus. |
| transcription | transcription_from_api |
|---|---|
| {"text": " of experiencing self versus remembering self. I was hoping you can give a simple answer of how we should live life. Based on the fact that our me ...... ion. Can I be your opening answer? Oh my God, no, I'm gonna open for you, dude. Otherwise it's like, you know, everybody leaves after you're done.", "language": "en", "segments": [{"id": 0, "end": 5., "seek": 0, "text": " of experiencing self versus remembering self.", "start": 0., "tokens": [50363, 286, 13456, 2116, 9051, 24865, 2116, 13, 50613], "avg_logprob": -0.282, "temperature": 0., "no_speech_prob": 0.213, "compression_ratio": 1.632}, {"id": 1, "end": 8.68, "seek": 0, "text": " I was hoping you can give a simple answer", "start": 6., "tokens": [50663, 314, 373, 7725, 345, 460, 1577, 257, 2829, 3280, 50797], "avg_logprob": -0.282, "temperature": 0., "no_speech_prob": 0.213, "compression_ratio": 1.632}, {"id": 2, "end": 10.2, "seek": 0, "text": " of how we should live life.", "start": 8.68, "tokens": [50797, 286, 703, 356, 815, 2107, 1204, 13, 50873], "avg_logprob": -0.282, "temperature": 0., "no_speech_prob": 0.213, "compression_ratio": 1.632}, {"id": 3, "end": 16.04, "seek": 0, "text": " Based on the fact that our memories", "start": 12.24, "tokens": [50975, 13403, 319, 262, 1109, 326, 674, 9846, 51165], "avg_logprob": -0.282, "temperature": 0., "no_speech_prob": 0.213, "compression_ratio": 1.632}, {"id": 4, "end": 17.84, "seek": 0, "text": " could be a source of happiness", "start": 16.04, "tokens": [51165, 714, 307, 257, 2723, 286, 12157, 51255], "avg_logprob": -0.282, "temperature": 0., "no_speech_prob": 0.213, "compression_ratio": 1.632}, {"id": 5, "end": 20.52, "seek": 0, "text": " or could be the primary source of happiness,", "start": 17.84, "tokens": [51255, 393, 714, 307, 262, 4165, 2723, 286, 12157, 11, 51389], "avg_logprob": -0.282, "temperature": 0., "no_speech_prob": 0.213, "compression_ratio": 1.632}, ..., {"id": 14, "end": 48.52, "seek": 2552, "text": " on the road with me.", "start": 47.52, "tokens": [51463, 319, 262, 2975, 351, 502, 13, 51513], "avg_logprob": -0.286, "temperature": 0., "no_speech_prob": 0.001, "compression_ratio": 1.611}, {"id": 15, "end": 50.52, "seek": 2552, "text": " That was such a great description.", "start": 48.52, "tokens": [51513, 1320, 373, 884, 257, 1049, 6764, 13, 51613], "avg_logprob": -0.286, "temperature": 0., "no_speech_prob": 0.001, "compression_ratio": 1.611}, {"id": 16, "end": 52.88, "seek": 2552, "text": " Can I be your opening answer?", "start": 51.52, "tokens": [51663, 1680, 314, 307, 534, 4756, 3280, 30, 51731], "avg_logprob": -0.286, "temperature": 0., "no_speech_prob": 0.001, "compression_ratio": 1.611}, {"id": 17, "end": 56.08, "seek": 5288, "text": " Oh my God, no, I'm gonna open for you, dude.", "start": 52.88, "tokens": [50363, 3966, 616, 1793, 11, 645, ..., 329, 345, 11, 18396, 13, 50523], "avg_logprob": -0.336, "temperature": 0., "no_speech_prob": 0.012, "compression_ratio": 1.121}, {"id": 18, "end": 57.28, "seek": 5288, "text": " Otherwise it's like, you know,", "start": 56.08, "tokens": [50523, 15323, 340, 338, 588, 11, 345, 760, 11, 50583], "avg_logprob": -0.336, "temperature": 0., "no_speech_prob": 0.012, "compression_ratio": 1.121}, {"id": 19, "end": 58.88, "seek": 5288, "text": " everybody leaves after you're done.", "start": 57.28, "tokens": [50583, 7288, 5667, 706, 345, 821, 1760, 13, 50663], "avg_logprob": -0.336, "temperature": 0., "no_speech_prob": 0.012, "compression_ratio": 1.121}]} | {"text": "of experiencing self versus remembering self, I was hoping you can give a simple answer of how we should live life, based on the fact that our mem ...... our opening excerpt? Oh my God, no, I'm gonna open for you, dude. Otherwise, it's like, you know, everybody leaves after you're done. Ha ha ha ha!", "usage": {"type": "duration", "seconds": 60}, "logprobs": null} |
Connected to Pixeltable database at: postgresql+psycopg://postgres:@/pixeltable?host=/Users/asiegel/.pixeltable/pgdata Created directory 'transcription\_demo'. Created table 'video\_table'.Next let’s insert some video files into the table. In this demo, we’ll be using one-minute excerpts from a Lex Fridman podcast. We’ll begin by inserting two of them into our new table. In this demo, our videos are given as `https` links, but Pixeltable also accepts local files and S3 URLs as input. ```python theme={null} videos = [ 'https://github.com/pixeltable/pixeltable/raw/release/docs/resources/audio-transcription-demo/' f'Lex-Fridman-Podcast-430-Excerpt-{n}.mp4' for n in range(3) ] video_table.insert({'video': video} for video in videos[:2]) video_table.show() ```
Inserted 2 rows with 0 errors in 2.04 s (0.98 rows/s)Now we’ll add another column to hold extracted audio from our videos. The new column is an example of a *computed column*: it’s updated automatically based on the contents of another column (or columns). In this case, the value of the `audio` column is defined to be the audio track extracted from whatever’s in the `video` column. ```python theme={null} from pixeltable.functions.video import extract_audio video_table.add_computed_column( audio=extract_audio(video_table.video, format='mp3') ) video_table.show() ```
Added 2 column values with 0 errors in 0.91 s (2.19 rows/s)If we look at the structure of the video table, we see that the new column is a computed column. ```python theme={null} video_table ``` We can also add another computed column to extract metadata from the audio streams. ```python theme={null} from pixeltable.functions.audio import get_metadata video_table.add_computed_column(metadata=get_metadata(video_table.audio)) video_table.show() ```
Added 2 column values with 0 errors in 0.02 s (95.47 rows/s)## Create Transcriptions Now we’ll add a step to create transcriptions of our videos. As mentioned above, we’re going to use the Whisper library for this, running locally. Pixeltable has a built-in function, `whisper.transcribe`, that serves as an adapter for the Whisper library’s transcription capability. All we have to do is add a computed column that calls this function: ```python theme={null} from pixeltable.functions import whisper video_table.add_computed_column( transcription=whisper.transcribe( audio=video_table.audio, model='base.en' ) ) video_table.select( video_table.video, video_table.transcription.text ).show() ```
Added 2 column values with 0 errors in 4.63 s (0.43 rows/s)In order to index the transcriptions, we’ll first need to split them into sentences. We can do this using Pixeltable’s built-in `string_splitter` iterator. ```python theme={null} from pixeltable.functions.string import string_splitter sentences_view = pxt.create_view( 'transcription_demo/sentences_view', video_table, iterator=string_splitter( video_table.transcription.text, separators='sentence' ), ) ``` The `string_splitter` creates a new view, with the audio transcriptions broken into individual, one-sentence chunks. ```python theme={null} sentences_view.select(sentences_view.pos, sentences_view.text).show(8) ``` ## Add an Embedding Index Next, let’s use the Huggingface `sentence_transformers` library to create an embedding index of our sentences, attaching it to the `text` column of our `sentences_view`. ```python theme={null} from pixeltable.functions.huggingface import sentence_transformer sentences_view.add_embedding_index( 'text', embedding=sentence_transformer.using(model_id='intfloat/e5-large-v2'), ) ```
modules.json: 0%| | 0.00/387 \[00:00\
We can do a simple lookup to test our new index. The following snippet
returns the results of a nearest-neighbor search on the input “What is
happiness?”
```python theme={null}
sim = sentences_view.text.similarity(string='What is happiness?')
(
sentences_view.order_by(sim, asc=False)
.limit(10)
.select(sentences_view.text, similarity=sim)
.collect()
)
```
## Incremental Updates
*Incremental updates* are a key feature of Pixeltable. Whenever a new
video is added to the original table, all of its downstream computed
columns are updated automatically. Let’s demonstrate this by adding a
third video to the table and seeing how the updates propagate through to
the index.
```python theme={null}
video_table.insert([{'video': videos[2]}])
```
Inserted 10 rows with 0 errors in 4.20 s (2.38 rows/s)
10 rows inserted.
```python theme={null}
video_table.select(
video_table.video,
video_table.metadata,
video_table.transcription.text,
).show()
```
```python theme={null}
sim = sentences_view.text.similarity(string='What is happiness?')
(
sentences_view.order_by(sim, asc=False)
.limit(20)
.select(sentences_view.text, similarity=sim)
.collect()
)
```
We can see the new results showing up in `sentences_view`.
## Using the OpenAI API
This concludes our tutorial using the locally installed Whisper library.
Sometimes, it may be preferable to use the OpenAI API rather than a
locally installed library. In this section we’ll show how this can be
done in Pixeltable, simply by using a different function to construct
our computed columns.
Since this section relies on calling out to the OpenAI API, you’ll need
to have an API key, which you can enter below.
```python theme={null}
import getpass
import os
if 'OPENAI_API_KEY' not in os.environ:
os.environ['OPENAI_API_KEY'] = getpass.getpass('OpenAI API Key:')
```
```python theme={null}
from pixeltable.functions import openai
video_table.add_computed_column(
transcription_from_api=openai.transcriptions(
video_table.audio, model='whisper-1'
)
)
```
Added 3 column values with 0 errors in 6.49 s (0.46 rows/s)
3 rows updated.
Now let’s compare the results from the local model and the API
side-by-side.
```python theme={null}
video_table.select(
video_table.video,
video_table.transcription.text,
video_table.transcription_from_api.text,
).show()
```
They look pretty similar, which isn’t surprising, since the OpenAI
transcriptions endpoint runs on Whisper.
One difference is that the local library spits out a lot more
information about the internal behavior of the model. Note that we’ve
been selecting `video_table.transcription.text` in the preceding
queries, which pulls out just the `text` field of the transcription
results. The actual results are a sizable JSON structure that includes a
lot of metadata. To see the full output, we can select
`video_table.transcription` instead, to get the full JSON struct. Here’s
what it looks like (we’ll select just one row, since it’s a lot of
output):
```python theme={null}
video_table.select(
video_table.transcription, video_table.transcription_from_api
).show(1)
```
Built with [Mintlify](https://mintlify.com).