> ## Documentation Index
> Fetch the complete documentation index at: https://docs.pixeltable.com/llms.txt
> Use this file to discover all available pages before exploring further.

# document

> <a href="https://github.com/pixeltable/pixeltable/blob/main/pixeltable/functions/document.py#L0" id="viewSource" target="_blank" rel="noopener noreferrer"><img src="https://img.shields.io/badge/View%20Source%20on%20Github-blue?logo=github&labelColor=gray" alt="View Source on GitHub" style={{ display: 'inline', margin: '0px' }} noZoom /></a>

# <span style={{ 'color': 'gray' }}>module</span>  pixeltable.functions.document

Pixeltable UDFs for `DocumentType`.

## <span style={{ 'color': 'gray' }}>iterator</span>  document\_splitter()

```python Signature theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
document_splitter(
    document: Any,
    separators: str,
    *,
    elements: list[typing.Literal['text', 'image']] | None = None,
    limit: int | None = None,
    overlap: int | None = None,
    metadata: str = '',
    skip_tags: list[str] | None = None,
    spacy_model: str = 'en_core_web_sm',
    tiktoken_encoding: str | None = 'cl100k_base',
    tiktoken_target_model: str | None = None,
    image_dpi: int = 300,
    image_format: str = 'png'
) 
```

Iterator over chunks of a document. The document is chunked according to the specified `separators`.

The iterator yields a `text` field containing the text of the chunk, and it may also
include additional metadata fields if specified in the `metadata` parameter, as explained below.

Chunked text will be cleaned with `ftfy.fix_text` to fix up common problems with unicode sequences.

**Parameters:**

* **`separators`** (`str`): separators to use to chunk the document. Options are:
  `'heading'`, `'paragraph'`, `'sentence'`, `'token_limit'`, `'char_limit'`, `'page'`.
  This may be a comma-separated string, e.g., `'heading,token_limit'`.
* **`elements`** (`list[typing.Literal['text', 'image']] | None`): list of elements to extract from the document. Options are:
  `'text'`, `'image'`. Defaults to `['text']` if not specified. The `'image'` element is only supported
  for the `'page'` separator on PDF documents.
* **`limit`** (`int | None`): the maximum number of tokens or characters in each chunk, if `'token_limit'`
  or `'char_limit'` is specified.
* **`metadata`** (`str`, default: `''`): additional metadata fields to include in the output. Options are:
  `'title'`, `'heading'` (HTML and Markdown), `'sourceline'` (HTML), `'page'` (PDF), `'bounding_box'`
  (PDF). The input may be a comma-separated string, e.g., `'title,heading,sourceline'`.
* **`spacy_model`** (`str`, default: `'en_core_web_sm'`): Name of the spaCy model to use for sentence segmentation. This parameter is ignored unless
  the `'sentence'` separator is specified.
* **`image_dpi`** (`int`, default: `300`): DPI to use when extracting images from PDFs. Defaults to 300.
* **`image_format`** (`str`, default: `'png'`): format to use when extracting images from PDFs. Defaults to 'png'.

**Examples:**

All these examples assume an existing table `tbl` with a column `doc` of type `pxt.Document`. Create a view that splits all documents into chunks of up to 300 tokens:

```python theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
pxt.create_view(
    'chunks',
    tbl,
    iterator=document_splitter(
        tbl.doc, separators='token_limit', limit=300
    ),
)
```

Create a view that splits all documents along sentence boundaries, including title and heading metadata:

```python theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
pxt.create_view(
    'sentence_chunks',
    tbl,
    iterator=document_splitter(
        tbl.doc, separators='sentence', metadata='title,heading'
    ),
)
```