Ingestion

Ingesting files with R2R.

Document Ingestion and Management

Ingest Files

Ingest files or directories into your R2R system:

const files = [
  { path: 'path/to/file1.txt', name: 'file1.txt' },
  { path: 'path/to/file2.txt', name: 'file2.txt' }
];
const metadatas = [{ key1: 'value1' }, { key2: 'value2' }];

const ingestResponse = await client.ingestFiles(files, {
  metadatas,
  user_ids: ['user-id-1', 'user-id-2'],
});

Response

response

object

The response from the R2R system after ingesting the files.

[{'message': 'Ingestion task queued successfully.', 'task_id': '6e27dfca-606d-422d-b73f-2d9e138661b4', 'document_id': 'c3291abf-8a4e-5d9d-80fd-232ef6fd8526'}, ...]

files

Array<string | File | { path: string; name: string }>

required

An array of file paths, File objects, or objects with path and name properties to ingest.

options

object

metadatas

Record<string, any>

An optional array of metadata objects corresponding to each file.

document_ids

Array<string>

An optional array of document IDs to assign to the ingested files.

user_ids

Array<string | null>

An optional array of user IDs associated with the ingested files.

ingestion_config

Optional[Union[dict, ChunkingConfig]]

The ingestion config override parameter enables developers to customize their R2R chunking strategy at runtime.

Show properties

provider

str

default:

"unstructured_local"

Which chunking provider to use. Options are “r2r”, “unstructured_local”, or “unstructured_api”.

method

str

default:

"by_title"

Which chunking method to apply. Options are “by_title”, “basic”, “recursive”, or “character”.

chunk_size

int

default:

"512"

The average size of chunks, in tokens.

chunk_overlap

int

default:

"20"

The default overlap between chunks.

max_chunk_size

Optional[int]

default:

"None"

Sets a maximum size on output chunks.

combine_under_n_chars

Optional[int]

Combine chunks smaller than this number of characters.

max_characters

Optional[int]

Maximum number of characters per chunk.

coordinates

bool

default:

"False"

Whether to include coordinates in the output.

encoding

Optional[str]

Encoding to use for text files.

extract_image_block_types

Optional[list[str]]

Types of image blocks to extract.

gz_uncompressed_content_type

Optional[str]

Content type for uncompressed gzip files.

hi_res_model_name

Optional[str]

Name of the high-resolution model to use.

include_orig_elements

Optional[bool]

default:

"False"

Whether to include original elements in the output.

include_page_breaks

bool

Whether to include page breaks in the output.

languages

Optional[list[str]]

List of languages to consider for text processing.

multipage_sections

bool

default:

"True"

Whether to allow sections to span multiple pages.

new_after_n_chars

Optional[int]

Start a new chunk after this many characters.

ocr_languages

Optional[list[str]]

Languages to use for OCR.

output_format

str

default:

"application/json"

Format of the output.

overlap

int

default:

"0"

Number of characters to overlap between chunks.

overlap_all

bool

default:

"False"

Whether to overlap all chunks.

pdf_infer_table_structure

bool

default:

"True"

Whether to infer table structure in PDFs.

similarity_threshold

Optional[float]

Threshold for considering chunks similar.

skip_infer_table_types

Optional[list[str]]

Types of tables to skip inferring.

split_pdf_concurrency_level

int

default:

"5"

Concurrency level for splitting PDFs.

split_pdf_page

bool

default:

"True"

Whether to split PDFs by page.

starting_page_number

Optional[int]

Page number to start processing from.

strategy

str

default:

"auto"

Strategy for processing. Options are “auto”, “fast”, or “hi_res”.

chunking_strategy

Optional[str]

default:

"by_title"

Strategy for chunking. Options are “by_title” or “basic”.

unique_element_ids

bool

default:

"False"

Whether to generate unique IDs for elements.

xml_keep_tags

bool

default:

"False"

Whether to keep XML tags in the output.

Update Files

Update existing documents:

const files = [
  { path: '/path/to/updated_file1.txt', name: 'updated_file1.txt' }
];
const document_ids = ['document-id-1'];
const updateResponse = await client.updateFiles(files, {
  document_ids,
  metadatas: [{ key: 'updated_value' }] // to overwrite the existing metadata
});

Response

response

object

The response from the R2R system after updating the files.

  {'results': {'processed_documents': [{'id': '9f375ce9-efe9-5b57-8bf2-a63dee5f3621', 'title': 'aristotle_v2.txt'}], 'failed_documents': [], 'skipped_documents': []}}

files

Array<File | { path: string; name: string }>

required

An array of File objects or objects with path and name properties to update.

options

object

required

document_ids

Array<string>

required

An array of document IDs corresponding to the files being updated.

metadatas

Array<Record<string, any>>

An optional array of metadata objects for the updated files.

ingestion_config

Record<string, any>

The ingestion config override parameter enables developers to customize their R2R chunking strategy at runtime.

Show properties

provider

str

default:

"r2r"

Which chunking provider to use, r2r or unstructured. Selecting unstructured is generally recommended when parsing with unstructured or unstructured_api.

method

str

default:

"recursive"

Which chunking method to apply? When using unstructured, by_title or basic are supported.

chunk_size

int

default:

"512"

The average size of chunks, in tokens.

chunk_overlap

int

default:

"20"

The default overlap between chunks.

max_chunk_size

Optional[int]

default:

"None"

Sets a maximum size on output chunks.

Documents Overview

Retrieve high-level document information, restricted to user files, except when called by a superuser where it will then return results from over all users:

const documentsOverview = await client.documentsOverview();

Response

response

Array<object>

An array of objects containing document information.

[
  {
    'document_id': '9fbe403b-c11c-5aae-8ade-ef22980c3ad1',
    'version': 'v1',
    'size_in_bytes': 73353,
    'metadata': {},
    'status': 'success',
    'user_id': '2acb499e-8428-543b-bd85-0d9098718220',
    'title': 'aristotle.txt',
    'created_at': '2024-07-21T20:09:14.218741Z',
    'updated_at': '2024-07-21T20:09:14.218741Z',
    'metadata': {'x': 'y'}
  },
  ...
]

document_ids

Array<string>

An optional array of document IDs to filter the overview.

Document Chunks

Fetch chunks for a particular document:

const documentId = '9fbe403b-c11c-5aae-8ade-ef22980c3ad1';
const chunks = await client.documentChunks(documentId);

Response

response

Array<object>

An array of objects containing chunk information.

[{
  'text': 'Aristotle[A] (Greek: Ἀριστοτέλης Aristotélēs, pronounced [aristotélɛːs]; 384–322 BC) was an Ancient Greek philosopher and polymath...',
  'user_id': '2acb499e-8428-543b-bd85-0d9098718220',
  'document_id': '9fbe403b-c11c-5aae-8ade-ef22980c3ad1',
  'extraction_id': 'aeba6400-1bd0-5ee9-8925-04732d675434',
  'fragment_id': 'f48bcdad-4155-52a4-8c9d-8ba06e996ba3'
  'metadata': {'title': 'aristotle.txt', 'version': 'v0', 'chunk_order': 0}}
},
...]

document_id

string

required

The ID of the document to retrieve chunks for.

Delete Documents

Delete a document by its ID:

const deleteResponse = await client.delete({ document_id: "91662726-7271-51a5-a0ae-34818509e1fd" });

Response

response

object

The response from the R2R system after successfully deleting the documents.

{'results': {}}

filters

{ [key: string]: string | string[] }

required

A list of logical filters to perform over input documents fields which identifies the unique set of documents to delete (e.g., {"document_id": {"$eq": "9fbe403b-c11c-5aae-8ade-ef22980c3ad1"}}). Logical operations might include variables such as "user_id" or "title" and filters like neq, gte, etc.