# Table of Contents - [Chonkie Documentation](#chonkie-documentation) - [Chonkie Documentation](#chonkie-documentation) - [Chonkie Documentation](#chonkie-documentation) - [Unknown](#unknown) - [Chonkie Documentation](#chonkie-documentation) - [Chonkie Documentation](#chonkie-documentation) - [Chonkie Documentation](#chonkie-documentation) - [Chonkie Documentation](#chonkie-documentation) - [Chonkie Documentation](#chonkie-documentation) - [Chonkie Documentation](#chonkie-documentation) - [Chonkie Documentation](#chonkie-documentation) - [Chonkie Documentation](#chonkie-documentation) - [Chonkie Documentation](#chonkie-documentation) - [Chonkie Documentation](#chonkie-documentation) - [Chonkie Documentation](#chonkie-documentation) - [Chonkie Documentation](#chonkie-documentation) - [Chonkie Documentation](#chonkie-documentation) - [Chonkie Documentation](#chonkie-documentation) - [Chonkie Documentation](#chonkie-documentation) - [Chonkie Documentation](#chonkie-documentation) - [Chonkie Documentation](#chonkie-documentation) - [Chonkie Documentation](#chonkie-documentation) - [Chonkie Documentation](#chonkie-documentation) - [Chonkie Documentation](#chonkie-documentation) - [Chonkie Documentation](#chonkie-documentation) - [Chonkie Documentation](#chonkie-documentation) - [Chonkie Documentation](#chonkie-documentation) - [Chonkie Documentation](#chonkie-documentation) - [Chonkie Documentation](#chonkie-documentation) - [Chonkie Documentation](#chonkie-documentation) - [Chonkie Documentation](#chonkie-documentation) - [Chonkie Documentation](#chonkie-documentation) - [Chonkie Documentation](#chonkie-documentation) - [Chonkie Documentation](#chonkie-documentation) - [Chonkie Documentation](#chonkie-documentation) - [Chonkie Documentation](#chonkie-documentation) - [Chonkie Documentation](#chonkie-documentation) - [Chonkie Documentation](#chonkie-documentation) - [Chonkie Documentation](#chonkie-documentation) - [Chonkie Documentation](#chonkie-documentation) - [Chonkie Documentation](#chonkie-documentation) - [Chonkie Documentation](#chonkie-documentation) - [Chonkie Documentation](#chonkie-documentation) - [Chonkie Documentation](#chonkie-documentation) - [Chonkie Documentation](#chonkie-documentation) - [Chonkie Documentation](#chonkie-documentation) - [Chonkie Documentation](#chonkie-documentation) - [Chonkie Documentation](#chonkie-documentation) - [Chonkie Documentation](#chonkie-documentation) - [Chonkie Documentation](#chonkie-documentation) - [Chonkie Documentation](#chonkie-documentation) - [Chonkie Documentation](#chonkie-documentation) - [Chonkie Documentation](#chonkie-documentation) - [Chonkie Documentation](#chonkie-documentation) - [Chonkie Documentation](#chonkie-documentation) - [Chonkie Documentation](#chonkie-documentation) - [Chonkie Documentation](#chonkie-documentation) - [Chonkie Documentation](#chonkie-documentation) - [Chonkie Documentation](#chonkie-documentation) - [Chonkie Documentation](#chonkie-documentation) - [Chonkie Documentation](#chonkie-documentation) - [Chonkie Documentation](#chonkie-documentation) - [Chonkie Documentation](#chonkie-documentation) - [Chonkie Documentation](#chonkie-documentation) - [Chonkie Documentation](#chonkie-documentation) - [Chonkie Documentation](#chonkie-documentation) - [Chonkie Documentation](#chonkie-documentation) - [Page Not Found](#page-not-found) - [Page Not Found](#page-not-found) - [Page Not Found](#page-not-found) - [Page Not Found](#page-not-found) - [Page Not Found](#page-not-found) - [Page Not Found](#page-not-found) - [Page Not Found](#page-not-found) - [Page Not Found](#page-not-found) - [Page Not Found](#page-not-found) - [Page Not Found](#page-not-found) - [Page Not Found](#page-not-found) - [Page Not Found](#page-not-found) - [Page Not Found](#page-not-found) --- # Chonkie Documentation > Documentation Index > ------------------- > > Fetch the complete documentation index at: [/llms.txt](https://docs.chonkie.ai/llms.txt) > > Use this file to discover all available pages before exploring further. [Skip to main content](https://docs.chonkie.ai/common/welcome#content-area) ![Chonkie Logo Light](https://mintcdn.com/chonkie/z5jcWEi822NzVea9/assets/logo/chonkie_logo_br_transparent_bg.png?w=2500&fit=max&auto=format&n=z5jcWEi822NzVea9&q=85&s=738337032f40c20a66f03fab4f01246a)![Chonkie Logo Dark](https://mintcdn.com/chonkie/z5jcWEi822NzVea9/assets/logo/chonkie_logo_br_transparent_bg.png?w=2500&fit=max&auto=format&n=z5jcWEi822NzVea9&q=85&s=738337032f40c20a66f03fab4f01246a) Ever found yourself making a RAG pipeline yet again (your 2,342,148th one), only to realize you’re stuck having to write your ingestion logic with bloated software library X or the painfully feature-less library Y? _WHY CAN’T THIS JUST BE SIMPLE, UGH?_ Well, look no further than Chonkie! (chonkie boi is a gud boi 🦛) Feature-rich ------------ Clean, CHONK, Embed, Refine and Store your data - all from one library! Easy to use ----------- Install, Import, CHONK - it’s that simple! Lightning Fast -------------- CHONK at the speed of light! zooooooooom Wide Support ------------ Supports your favorite tokenizers, chunkers, embeddings and vector DBs Versatile --------- CHONK in Python, JavaScript or via our API. Chonkie is there wherever you need it Cute Mascot ----------- psst it’s a pygmy hippo btw! Moto Moto approved AI Agent Skills --------------- Install via `npx skills add chonkie-inc/skills` or browse on skills.sh — works with Claude Code, Cursor, Copilot, and 20+ agents * * * Was this page helpful? YesNo [Concepts\ \ Next](https://docs.chonkie.ai/common/concepts) ⌘I --- # Chonkie Documentation > Documentation Index > ------------------- > > Fetch the complete documentation index at: [/llms.txt](https://docs.chonkie.ai/llms.txt) > > Use this file to discover all available pages before exploring further. [Skip to main content](https://docs.chonkie.ai/common/concepts#content-area) This page outlines some common concepts of Chonkie, that will help you understand how to use Chonkie effectively. [​](https://docs.chonkie.ai/common/concepts#what-are-chonkie%E2%80%99s-core-values) What are Chonkie’s core values? ---------------------------------------------------------------------------------------------------------------------- Chonkie is a very opinionated library, and it all stems from innate human mortality. We are all going to die one day, and we have no reason to waste time figuring out how to chunk documents. Just use Chonkie. Chonkie needs to be and always adheres to be: * **Simple**: We care about how simple it is to use Chonkie. No brainer. * **Fast**: We care about your latency. No time to waste. * **Lightweight**: We care about your memory. No space to waste. * **Flexible**: We care about your customization needs. Hassle free. Chonkie just works. It’s that simple. [​](https://docs.chonkie.ai/common/concepts#what-is-chunking-what-is-an-ideal-chunk-and-chunker) What is chunking? What is an ideal chunk and chunker? --------------------------------------------------------------------------------------------------------------------------------------------------------- Chunking is the process of breaking down a text into smaller, more manageable pieces, that can be used for RAG applications. An ideal chunk is one that is: * **Reconstructable**: A chunk should be part of the whole text, such that combining chunks should give you the original text back. * **Independent**: It should be a standalone unit tackling only one idea, i.e., removing it from the chunk should not remove important information from the original text. * **Sufficient**: It should be long enough to be meaningful, i.e., it should contain enough information to be useful. As a consequence, an (ideal) chunker is one that: * Breaks down the text into chunks that are reconstructable, independent and sufficient. * Is deterministic, i.e., given the same text, it should always return the same chunks. * Is efficient, i.e., it should be fast and lightweight. This is how Chonkie’s chunkers are designed to be. Understanding this will help you understand why Chonkie divides the chunking process into multiple stages: Pre-processing, chunking and post-processing. [​](https://docs.chonkie.ai/common/concepts#is-chunking-necessary-can%E2%80%99t-i-just-use-the-entire-document-as-a-chunk) Is chunking necessary? Can’t I just use the entire document as a chunk? ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Nope. Here’s why chunking is absolutely essential: #### [​](https://docs.chonkie.ai/common/concepts#1-limited-context-windows) 1\. **Limited Context Windows** All models have a limit on how much text they can process at once. This is referred to as their “context window”. Chunking breaks down large documents into manageable pieces that fit within these limits. #### [​](https://docs.chonkie.ai/common/concepts#2-computational-efficiency) 2\. **Computational Efficiency** Processing a 100GB document every time you make a query? Bad idea. Attention mechanisms, even when optimized, are computationally expensive (`O(n)`). Chunking keeps things efficient and memory-friendly. #### [​](https://docs.chonkie.ai/common/concepts#3-better-representation) 3\. **Better Representation** As mentioned earlier, chunks represent each idea as an independent entity. Not chunking your document will likely cause your model to conflate concepts and get confused. Representation models use lossy compression, so keeping chunks concise ensures the model understands the context better. #### [​](https://docs.chonkie.ai/common/concepts#4-reduced-hallucination) 4\. **Reduced Hallucination** Feeding too much context at once makes models hallucinate. They start using irrelevant information to answer queries, and that’s a **big** no-no. Smaller, focused chunks reduce this risk. All of this makes chunking a **must-have** for RAG applications. Don’t get caught using your whole document as a single chunk! [​](https://docs.chonkie.ai/common/concepts#how-is-chonkie-so-fast-what-is-the-secret-sauce) How is Chonkie so fast? What is the secret sauce? ------------------------------------------------------------------------------------------------------------------------------------------------- Chonkie is fast because it cares about your latency. Chonkie is raised with love, care, and strong beliefs that the speed of light should be the only limit to your RAG applications. Of course, we do a lot of optimizations under the hood to make sure that Chonkie is as fast as it gets. Here are some of the things we do: * **Pipelining**: We use a pipelining approach to process the document, so as to make stronger heuristics for chunking. This allows to have a faster chunking process, without compromising on the quality of chunks. * **Caching and Pre-computation**: We cache the results of the chunking process, so as to avoid re-computation. This allows to have a faster chunking process, without compromising on the quality of chunks. * **Smart Token Estimate-Validate feedback Loops**: We use a token estimate-validate feedback loops to make sure that we have near optimal chunk sizes, while bypassing some of the inefficiencies of the tokenizers. * **Faster Tokenizers**: We use a faster tokenizer, [tiktoken](https://github.com/openai/tiktoken) , which is faster and more efficient than the default tokenizer. Tiktoken by default does not support all model types, so we use a wrapper around it, AutoTikTokenizer which adds support for all HF models. * **Ultra-fast embedding**: By default, Chonkie uses Static Embeddings from Model2Vec, which are ultra-fast and lightweight. Static Embeddings are pre-computed and stored in a lookup table, so as to avoid the overhead of running an embedding model at query time. * **Parallel Processing**: We use parallel processing to process the document in parallel, so as to make better use of the available resources. This allows to have a faster chunking process, without compromising on the quality of chunks. All these optimizations allow Chonkie to process documents at the speed of light, without compromising on the quality of chunks. So, the next time you want to process a large document, remember to use Chonkie! Was this page helpful? YesNo [🦛 Chonkie ✨\ \ Previous](https://docs.chonkie.ai/common/welcome) [Open Source\ \ Next](https://docs.chonkie.ai/common/open-source) ⌘I --- # Chonkie Documentation > Documentation Index > ------------------- > > Fetch the complete documentation index at: [/llms.txt](https://docs.chonkie.ai/llms.txt) > > Use this file to discover all available pages before exploring further. [Skip to main content](https://docs.chonkie.ai/common/open-source#content-area) ![Chonkie Logo](https://mintcdn.com/chonkie/z5jcWEi822NzVea9/assets/logo/chonkie_xray.png?w=2500&fit=max&auto=format&n=z5jcWEi822NzVea9&q=85&s=3833417da633e1df79fff35f86f64a28)_✨Look Inside! We’re Open Source!✨_ **Chonkie’s Open Source library** provides lightweight, and high-performance features for building modern RAG applications. Install it locally, run anywhere, and keep full control over your chunking pipeline. [​](https://docs.chonkie.ai/common/open-source#why-chonkie-oss) Why Chonkie OSS? ----------------------------------------------------------------------------------- Completely Free --------------- Released under the MIT license. Use however you like. Privacy First ------------- All processing happens locally. Your data never leaves your infrastructure. Production Ready ---------------- Battle-tested algorithms used by thousands of developers. Optimized for speed and reliability. Lightning Fast -------------- Optimized with caching, parallel processing, and fast tokenizers. Process millions of chunks efficiently. [​](https://docs.chonkie.ai/common/open-source#core-capabilities) Core Capabilities -------------------------------------------------------------------------------------- ### [​](https://docs.chonkie.ai/common/open-source#advanced-chunkers) Advanced Chunkers Chonkie OSS includes a comprehensive suite of chunking algorithms, each designed for specific document types and use cases: TokenChunker **Best for**: General-purpose chunking, most use casesSplits text into fixed-size token chunks with configurable overlap. The most straightforward and reliable chunking strategy.Available in: Python, JavaScript SentenceChunker **Best for**: Q&A systems, maintaining complete thoughtsChunks at sentence boundaries while respecting token limits. Ensures sentences are never split mid-thought.Available in: Python, JavaScript RecursiveChunker **Best for**: Markdown, structured documents, hierarchical contentHierarchically chunks using multiple delimiters—paragraphs, then sentences, then words. Preserves document structure naturally.Available in: Python, JavaScript FastChunker **Best for**: High-throughput pipelines, large-scale document processingSIMD-accelerated chunking with 100+ GB/s throughput. Uses byte-size limits for extreme performance without tokenization overhead.Available in: Python, JavaScript TableChunker **Best for**: Markdown tables, tabular dataSplits large tables into manageable chunks by rows while preserving headers. Perfect for data-heavy documents.Available in: Python, JavaScript SemanticChunker **Best for**: Multi-topic documents, maintaining topical coherenceUses embeddings to identify natural topic boundaries. Creates chunks based on semantic similarity, not just structure. Includes Savitzky-Golay filtering and skip-window merging for advanced boundary detection.Available in: Python, JavaScript LateChunker **Best for**: Retrieval optimization, higher recall RAG systemsImplements the Late Chunking algorithm from research. Generates document-level embeddings first, then derives chunk embeddings for richer contextual representation.Available in: Python CodeChunker **Best for**: Source code, API documentation, technical contentLanguage-aware chunking using Abstract Syntax Trees (AST). Preserves function and class boundaries for better code understanding.Available in: Python, JavaScript NeuralChunker **Best for**: Maximum quality, complex documents with subtle topic shiftsUses a fine-tuned BERT model to detect semantic shifts in text. ML-powered boundary detection for topic-coherent chunks.Available in: Python SlumberChunker **Best for**: Books, research papers, when quality matters mostAgentic chunking powered by LLMs via the Genie interface. Uses generative models (Gemini, OpenAI, etc.) to intelligently determine optimal chunk boundaries.Available in: Python ### [​](https://docs.chonkie.ai/common/open-source#embedding-providers) Embedding Providers Flexible embedding support for semantic chunking and refineries: * **AutoEmbeddings** - Automatically select the best embeddings for your use case * **Model2VecEmbeddings** - Ultra-fast static embeddings (default for semantic chunking) * **SentenceTransformerEmbeddings** - Hugging Face Sentence Transformers models * **OpenAIEmbeddings** - OpenAI’s text-embedding models * **AzureOpenAIEmbeddings** - Azure-hosted OpenAI embeddings * **CohereEmbeddings** - Cohere’s embedding models * **JinaEmbeddings** - Jina AI embeddings * **GeminiEmbeddings** - Google Gemini embeddings * **VoyageAIEmbeddings** - Voyage AI embeddings * **Custom Embeddings** - Bring your own embedding model All embeddings follow a consistent interface and can be swapped seamlessly. ### [​](https://docs.chonkie.ai/common/open-source#refineries) Refineries Enhance your chunks with additional context and embeddings: OverlapRefinery --------------- Adds contextual overlap between chunks to prevent information loss at boundaries. Configurable overlap sizes for optimal retrieval. EmbeddingsRefinery ------------------ Generates and attaches vector embeddings to your chunks. Supports all major embedding providers with automatic dimension detection. ### [​](https://docs.chonkie.ai/common/open-source#database-handshakes) Database Handshakes Seamlessly connect Chonkie to your favorite database: ChromaDB -------- Ephemeral or persistent ChromaDB instances Qdrant ------ High-performance vector search with Qdrant Weaviate -------- Knowledge graph + vector search with Weaviate Turbopuffer ----------- Serverless vector database by Turbopuffer Pinecone -------- Managed vector database with Pinecone pgvector -------- PostgreSQL with pgvector extension MongoDB ------- MongoDB Atlas Vector Search Elastic ------- Elasticsearch vector search Each handshake provides a simple interface to embed chunks and write them directly to your database. ### [​](https://docs.chonkie.ai/common/open-source#chefs) Chefs Chefs automatically prepare raw data for chunking: * **TableChef** - Extracts tables from markdown text * **TextChef** - Processes plain text files into structured Documents * **MarkdownChef** - Parses markdown with tables, code blocks, and images ### [​](https://docs.chonkie.ai/common/open-source#porters) Porters Export chunks to common formats: * **JSONPorter** - Export chunks to JSON for storage or processing * **DatasetsPorter** - Export to Hugging Face Datasets format ### [​](https://docs.chonkie.ai/common/open-source#utils) Utils * **Visualizer** - Rich text visualization of chunks with color-coded boundaries * **Hubbie** - Hugging Face Hub integration for sharing and loading chunkers [​](https://docs.chonkie.ai/common/open-source#language-support) Language Support ------------------------------------------------------------------------------------ * Python * JavaScript/TypeScript **Full Feature Set**All chunkers, embedding providers, refineries, handshakes, chefs, and porters available. Choose from minimal to full installations based on your needs. * Default install: Token, Sentence, Recursive, Table chunkers * Semantic install: + SemanticChunker, LateChunker, NeuralChunker with Model2Vec * All install: Every feature available **Core Chunking**JavaScript support includes the most commonly used chunkers: * TokenChunker * SentenceChunker * RecursiveChunker * FastChunker * TableChunker * SemanticChunker * CodeChunker Available via `@chonkiejs/core` package with full TypeScript support. Other chunkers available through the Chonkie Cloud API via `@chonkiejs/cloud`. To use custom tokenizers with the chunkers, install `@chonkiejs/token` [​](https://docs.chonkie.ai/common/open-source#performance-characteristics) Performance Characteristics ---------------------------------------------------------------------------------------------------------- Chonkie OSS is optimized for speed: * **Pipelining** - Efficient multi-stage processing * **Caching** - Smart caching to avoid recomputation * **Fast Tokenizers** - TikToken and AutoTikTokenizer for speed * **Parallel Processing** - Multi-threaded batch operations * **Ultra-fast Embeddings** - Model2Vec static embeddings (default) * **Token Estimate-Validate** - Efficient feedback loops for optimal chunk sizes Process thousands of documents per second on commodity hardware. [​](https://docs.chonkie.ai/common/open-source#next-steps) Next Steps ------------------------------------------------------------------------ Ready to get started with Chonkie OSS? Quick Start ----------- Install and create your first chunk in under 2 minutes Installation Guide ------------------ Detailed installation options for all features Chunkers Overview ----------------- Explore all chunking algorithms in detail GitHub Repository ----------------- Star the repo and contribute to the project * * * Need hosted chunking with zero setup? Check out our [Chunking API](https://docs.chonkie.ai/common/chunking-api) for a managed solution. Was this page helpful? YesNo [Concepts\ \ Previous](https://docs.chonkie.ai/common/concepts) ⌘I --- # Unknown \# Chonkie ## Docs - \[Concepts\](https://docs.chonkie.ai/common/concepts.md): Common concepts of Chonkie - \[Open Source\](https://docs.chonkie.ai/common/open-source.md): The Open Source Library For RAG - \[🦛 Chonkie ✨\](https://docs.chonkie.ai/common/welcome.md): The lightweight ingestion library for fast, efficient and robust RAG pipelines - \[Docker\](https://docs.chonkie.ai/oss/api/docker.md): Deploy the Chonkie API server with Docker and docker-compose - \[Endpoints\](https://docs.chonkie.ai/oss/api/endpoints.md): API reference for all Chonkie chunkers and refineries - \[API Server\](https://docs.chonkie.ai/oss/api/overview.md): Self-host Chonkie as a REST API for language-agnostic text chunking - \[Pipelines\](https://docs.chonkie.ai/oss/api/pipelines.md): Store and manage reusable chunking pipeline configurations - \[Quick Start\](https://docs.chonkie.ai/oss/api/quickstart.md): Get the Chonkie API server running in under a minute - \[Changelog\](https://docs.chonkie.ai/oss/changelog.md): Chonkie's Release Notes and Updates 🦛✨ - \[MarkdownChef\](https://docs.chonkie.ai/oss/chefs/markdownchef.md): Process markdown files, extracting tables, code blocks, and images. - \[MistralOCR\](https://docs.chonkie.ai/oss/chefs/mistral-ocr.md): Extract text from images and PDFs using Mistral's OCR API. - \[Chefs Overview\](https://docs.chonkie.ai/oss/chefs/overview.md): Overview of the different chefs available in Chonkie - \[TableChef\](https://docs.chonkie.ai/oss/chefs/tablechef.md): Extract tables from markdown text (including HTML tables) and prepare them for future usage. - \[TextChef\](https://docs.chonkie.ai/oss/chefs/textchef.md): Process plain text files into Document objects. - \[Code Chunker\](https://docs.chonkie.ai/oss/chunkers/code-chunker.md): Split code into chunks based on code structure - \[Fast Chunker\](https://docs.chonkie.ai/oss/chunkers/fast-chunker.md): SIMD-accelerated text chunking at 100+ GB/s throughput - \[Late Chunker\](https://docs.chonkie.ai/oss/chunkers/late-chunker.md): Split text into chunks based on a late-bound token count - \[Neural Chunker\](https://docs.chonkie.ai/oss/chunkers/neural-chunker.md): Split text using a fine-tuned BERT model to detect semantic shifts - \[Chunkers Overview\](https://docs.chonkie.ai/oss/chunkers/overview.md): Overview of the different chunkers available in Chonkie - \[Recursive Chunker\](https://docs.chonkie.ai/oss/chunkers/recursive-chunker.md): Recursively chunk documents into smaller chunks. - \[SDPM Chunker (Legacy)\](https://docs.chonkie.ai/oss/chunkers/sdpm-chunker.md): Semantic Double-Pass Merging chunker - now integrated into SemanticChunker - \[Semantic Chunker\](https://docs.chonkie.ai/oss/chunkers/semantic-chunker.md): Split text into chunks based on semantic similarity with advanced features - \[Sentence Chunker\](https://docs.chonkie.ai/oss/chunkers/sentence-chunker.md): Split text into chunks while preserving sentence boundaries - \[Slumber Chunker\](https://docs.chonkie.ai/oss/chunkers/slumber-chunker.md): Agentic chunking powered by generative models via the Genie interface - \[Table Chunker\](https://docs.chonkie.ai/oss/chunkers/table-chunker.md): Split markdown or HTML tables into manageable chunks by row, preserving headers. - \[TeraflopAI Chunker\](https://docs.chonkie.ai/oss/chunkers/teraflopai-chunker.md): Segment text using the TeraflopAI Segmentation API - \[Token Chunker\](https://docs.chonkie.ai/oss/chunkers/token-chunker.md): Split text into fixed-size token chunks with configurable overlap - \[AutoEmbeddings\](https://docs.chonkie.ai/oss/embeddings/auto-embeddings.md): Automatically select the best embeddings handler for your use case - \[AzureOpenAIEmbeddings\](https://docs.chonkie.ai/oss/embeddings/azure-embeddings.md): Embed text using Azure OpenAI embeddings - \[CohereEmbeddings\](https://docs.chonkie.ai/oss/embeddings/cohere-embeddings.md): Embed text using Cohere embeddings - \[Create your own custom embeddings handler\](https://docs.chonkie.ai/oss/embeddings/custom-embeddings.md) - \[GeminiEmbeddings\](https://docs.chonkie.ai/oss/embeddings/gemini-embeddings.md): Embed text using Google Gemini embeddings via GenAI API - \[JinaEmbeddings\](https://docs.chonkie.ai/oss/embeddings/jina-embeddings.md): JinaEmbeddings is a utility for embedding chunks. - \[Model2VecEmbeddings\](https://docs.chonkie.ai/oss/embeddings/model2vec-embeddings.md): Embed text using Model2Vec embeddings - \[OpenAIEmbeddings\](https://docs.chonkie.ai/oss/embeddings/openai-embeddings.md): Embed text using OpenAI embeddings - \[Embeddings Overview\](https://docs.chonkie.ai/oss/embeddings/overview.md): Overview of the different embeddings available in Chonkie - \[SentenceTransformerEmbeddings\](https://docs.chonkie.ai/oss/embeddings/sentence-transformer-embeddings.md): Embed text using SentenceTransformer embedding models - \[VoyageAIEmbeddings\](https://docs.chonkie.ai/oss/embeddings/voyageai-embeddings.md): Embed text using VoyageAI embeddings - \[CLI\](https://docs.chonkie.ai/oss/experimental/chonkie-cli.md): Chonkie Command Line Interface - \[Code Chunker\](https://docs.chonkie.ai/oss/experimental/code-chunker.md): Advanced AST-based code chunking with intelligent semantic preservation - \[Overview\](https://docs.chonkie.ai/oss/experimental/overview.md): Explore cutting-edge chunking capabilities with Chonkie's experimental features - \[FileFetcher\](https://docs.chonkie.ai/oss/fetchers/file-fetcher.md): Fetch files from local filesystem for pipeline processing - \[Fetchers Overview\](https://docs.chonkie.ai/oss/fetchers/overview.md): Overview of the different fetchers available in Chonkie - \[Chroma Handshake\](https://docs.chonkie.ai/oss/handshakes/chroma-handshake.md): Export Chonkie's Chunks into a Chroma collection. - \[Elasticsearch Handshake\](https://docs.chonkie.ai/oss/handshakes/elastic-handshake.md): Export Chonkie's Chunks into an Elasticsearch index. - \[LanceDB Handshake\](https://docs.chonkie.ai/oss/handshakes/lancedb-handshake.md): Export Chonkie's Chunks into a LanceDB table. - \[Milvus Handshake\](https://docs.chonkie.ai/oss/handshakes/milvus-handshake.md): Export Chonkie's Chunks into a Milvus collection. - \[MongoDB Handshake\](https://docs.chonkie.ai/oss/handshakes/mongodb-handshake.md): Export Chonkie's Chunks into a MongoDB collection. - \[Handshakes Overview\](https://docs.chonkie.ai/oss/handshakes/overview.md): Overview of the different handshakes available in Chonkie - \[Pgvector Handshake\](https://docs.chonkie.ai/oss/handshakes/pgvector-handshake.md): Export Chonkie's Chunks into a PostgreSQL database with pgvector. - \[Pinecone Handshake\](https://docs.chonkie.ai/oss/handshakes/pinecone-handshake.md): Export Chonkie's Chunks into a Pinecone index. - \[Qdrant Handshake\](https://docs.chonkie.ai/oss/handshakes/qdrant-handshake.md): Export Chonkie's Chunks into a Qdrant collection. - \[Turbopuffer Handshake\](https://docs.chonkie.ai/oss/handshakes/turbopuffer-handshake.md): Export Chonkie's Chunks into a Turbopuffer database. - \[Weaviate Handshake\](https://docs.chonkie.ai/oss/handshakes/weaviate-handshake.md): Export Chonkie's Chunks into a Weaviate collection. - \[Installation\](https://docs.chonkie.ai/oss/installation.md): Installing Chonkie and its various components - \[Building Pipelines\](https://docs.chonkie.ai/oss/pipelines.md): Build powerful text processing workflows with Chonkie's Pipeline API - \[DatasetsPorter\](https://docs.chonkie.ai/oss/porters/datasets-porter.md): Export Chonkie's Chunks into a Hugging Face Dataset. - \[JSONPorter\](https://docs.chonkie.ai/oss/porters/json-porter.md): Export Chonkie's Chunks into a JSON file. - \[Porters Overview\](https://docs.chonkie.ai/oss/porters/overview.md): Overview of the different porters available in Chonkie - \[Get Started with Chonkie\](https://docs.chonkie.ai/oss/quick-start.md): Get started with Chonkie - \[Embeddings Refinery\](https://docs.chonkie.ai/oss/refinery/embeddings-refinery.md): Embed Chunked Texts - \[Overlap Refinery\](https://docs.chonkie.ai/oss/refinery/overlap-refinery.md): Refine chunks by adding overlapping context from adjacent chunks. - \[Refinery Overview\](https://docs.chonkie.ai/oss/refinery/overview.md): Overview of the different refinery available in Chonkie - \[Hubbie\](https://docs.chonkie.ai/oss/utils/hubbie.md): Hubbie is a utility for accessing Chonkie's saved recipes. - \[Logging\](https://docs.chonkie.ai/oss/utils/logging.md): Control Chonkie's log output - \[Visualizer\](https://docs.chonkie.ai/oss/utils/visualizer.md): Visualize your chunks and embeddings ## Optional - \[Discord\](https://discord.gg/Q6zkP8w6ur) - \[Contact\](mailto:support@chonkie.ai) --- # Chonkie Documentation > Documentation Index > ------------------- > > Fetch the complete documentation index at: [/llms.txt](https://docs.chonkie.ai/llms.txt) > > Use this file to discover all available pages before exploring further. [Skip to main content](https://docs.chonkie.ai/oss/api/docker#content-area) [​](https://docs.chonkie.ai/oss/api/docker#quick-start) Quick Start ---------------------------------------------------------------------- docker compose up The API is available at `http://localhost:8000`. Visit `/docs` for the interactive Swagger UI. [​](https://docs.chonkie.ai/oss/api/docker#docker-compose-yml) docker-compose.yml ------------------------------------------------------------------------------------ The repository ships with a ready-to-use `docker-compose.yml`: services: chonkie-api: build: context: . dockerfile: Dockerfile image: chonkie-oss-api:latest container_name: chonkie-api ports: - "8000:8000" volumes: - ./data:/app/data environment: LOG_LEVEL: "${LOG_LEVEL:-INFO}" CORS_ORIGINS: "${CORS_ORIGINS:-*}" DATABASE_URL: "sqlite+aiosqlite:////app/data/chonkie.db" OPENAI_API_KEY: "${OPENAI_API_KEY:-}" COHERE_API_KEY: "${COHERE_API_KEY:-}" VOYAGE_API_KEY: "${VOYAGE_API_KEY:-}" MISTRAL_API_KEY: "${MISTRAL_API_KEY:-}" restart: unless-stopped healthcheck: test: ["CMD", "python", "-c",\ "import urllib.request; urllib.request.urlopen('http://localhost:8000/health')"] interval: 30s timeout: 10s retries: 3 start_period: 15s The `./data` volume mount persists the SQLite database (`chonkie.db`) across container restarts. [​](https://docs.chonkie.ai/oss/api/docker#environment-variables) Environment Variables ------------------------------------------------------------------------------------------ | Variable | Default | Description | | --- | --- | --- | | `LOG_LEVEL` | `INFO` | Log verbosity: `DEBUG`, `INFO`, `WARNING`, `ERROR` | | `CORS_ORIGINS` | `*` | Comma-separated allowed origins. Use `*` to allow all. | | `DATABASE_URL` | `sqlite+aiosqlite:///./data/chonkie.db` | SQLite database path. Override for custom locations. | | `OPENAI_API_KEY` | _(empty)_ | For OpenAI embeddings (`text-embedding-3-small`, etc.) | | `COHERE_API_KEY` | _(empty)_ | For Cohere embeddings (`embed-english-v3.0`, etc.) | | `VOYAGE_API_KEY` | _(empty)_ | For Voyage AI embeddings (`voyage-large-2`, etc.) | | `MISTRAL_API_KEY` | _(empty)_ | For Mistral embeddings (`mistral-embed`) | Pass them inline: LOG_LEVEL=DEBUG CORS_ORIGINS=https://myapp.com docker compose up Or create a `.env` file in the project root: LOG_LEVEL=INFO CORS_ORIGINS=https://myapp.com,https://api.myapp.com # Set your preferred embedding provider key: OPENAI_API_KEY=sk-... # COHERE_API_KEY=... # VOYAGE_API_KEY=... [​](https://docs.chonkie.ai/oss/api/docker#build-and-run-without-compose) Build and Run Without Compose ---------------------------------------------------------------------------------------------------------- # Build the image docker build -t chonkie-oss-api . # Run the container docker run -p 8000:8000 chonkie-oss-api # With environment variables docker run -p 8000:8000 \ -e LOG_LEVEL=DEBUG \ -e OPENAI_API_KEY=sk-... \ chonkie-oss-api [​](https://docs.chonkie.ai/oss/api/docker#image-details) Image Details -------------------------------------------------------------------------- The Dockerfile uses a multi-stage build to keep the final image lean: * **Builder stage** — installs `chonkie[api,semantic,code,openai]` into a virtual environment * **Runtime stage** — copies only the venv; runs as a non-root `chonkie` user * **Exposed port** — `8000` * **Health check** — HTTP GET to `/health` every 30 seconds [​](https://docs.chonkie.ai/oss/api/docker#production-tips) Production Tips ------------------------------------------------------------------------------ **Restrict CORS** — in production, replace `*` with your actual domains: CORS_ORIGINS=https://myapp.com,https://admin.myapp.com docker compose up **Add a reverse proxy** — put Nginx or Caddy in front for TLS termination and rate limiting: server { listen 443 ssl; server_name api.myapp.com; location / { proxy_pass http://chonkie-api:8000; proxy_set_header Host $host; proxy_set_header X-Real-IP $remote_addr; proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for; } } **Scale horizontally** — run multiple replicas behind a load balancer: services: chonkie-api: image: chonkie-oss-api:latest deploy: replicas: 3 ports: - "8000:8000" The `SemanticChunker` loads its embedding model on first use. Send a warm-up request after startup to avoid cold-start latency on the first real request in production. Was this page helpful? YesNo [Pipelines\ \ Previous](https://docs.chonkie.ai/oss/api/pipelines) [Chefs Overview\ \ Next](https://docs.chonkie.ai/oss/chefs/overview) ⌘I --- # Chonkie Documentation > Documentation Index > ------------------- > > Fetch the complete documentation index at: [/llms.txt](https://docs.chonkie.ai/llms.txt) > > Use this file to discover all available pages before exploring further. [Skip to main content](https://docs.chonkie.ai/oss/api/endpoints#content-area) Start the server and visit `http://localhost:8000/docs` for an interactive Swagger UI where you can try every endpoint directly in your browser. [​](https://docs.chonkie.ai/oss/api/endpoints#response-format) Response Format --------------------------------------------------------------------------------- All chunking endpoints return a list of chunk objects: [\ {\ "text": "chunk content",\ "start_index": 0,\ "end_index": 42,\ "token_count": 8\ }\ ] Submit a **list of strings** instead of a single string to get back a **list of lists** — one inner list per input document. * * * [​](https://docs.chonkie.ai/oss/api/endpoints#chunkers) Chunkers ------------------------------------------------------------------- ### [​](https://docs.chonkie.ai/oss/api/endpoints#token-chunker) Token Chunker `POST /v1/chunk/token` Splits text into fixed-size token windows. The fastest and most predictable chunker. curl -X POST http://localhost:8000/v1/chunk/token \ -H "Content-Type: application/json" \ -d '{ "text": "Your text here...", "chunk_size": 512, "chunk_overlap": 50 }' Parameters [​](https://docs.chonkie.ai/oss/api/endpoints#param-text) text string | string\[\] required Text or list of texts to chunk. [​](https://docs.chonkie.ai/oss/api/endpoints#param-tokenizer) tokenizer string default:"character" Tokenizer to use. Options: `"character"`, `"gpt2"`, `"cl100k_base"`, or any HuggingFace tokenizer name. [​](https://docs.chonkie.ai/oss/api/endpoints#param-chunk-size) chunk\_size integer default:"512" Maximum tokens per chunk. [​](https://docs.chonkie.ai/oss/api/endpoints#param-chunk-overlap) chunk\_overlap integer default:"0" Token overlap between consecutive chunks. * * * ### [​](https://docs.chonkie.ai/oss/api/endpoints#sentence-chunker) Sentence Chunker `POST /v1/chunk/sentence` Groups sentences into chunks while respecting a token-size limit. Preserves sentence boundaries — no mid-sentence splits. curl -X POST http://localhost:8000/v1/chunk/sentence \ -H "Content-Type: application/json" \ -d '{ "text": "First sentence. Second sentence. Third sentence.", "chunk_size": 256, "min_sentences_per_chunk": 2 }' Parameters [​](https://docs.chonkie.ai/oss/api/endpoints#param-text-1) text string | string\[\] required Text or list of texts to chunk. [​](https://docs.chonkie.ai/oss/api/endpoints#param-tokenizer-1) tokenizer string default:"character" Tokenizer to use. [​](https://docs.chonkie.ai/oss/api/endpoints#param-chunk-size-1) chunk\_size integer default:"512" Maximum tokens per chunk. [​](https://docs.chonkie.ai/oss/api/endpoints#param-chunk-overlap-1) chunk\_overlap integer default:"0" Token overlap between chunks. [​](https://docs.chonkie.ai/oss/api/endpoints#param-min-sentences-per-chunk) min\_sentences\_per\_chunk integer default:"1" Minimum sentences to include in each chunk. [​](https://docs.chonkie.ai/oss/api/endpoints#param-min-characters-per-sentence) min\_characters\_per\_sentence integer default:"12" Minimum characters required to count as a sentence. [​](https://docs.chonkie.ai/oss/api/endpoints#param-approximate) approximate boolean default:"false" Use approximate token counting for faster processing. [​](https://docs.chonkie.ai/oss/api/endpoints#param-delim) delim string | string\[\] default:"\[\\"\\\\n\\", \\". \\", \\"! \\", \\"? \\"\]" Sentence delimiter(s). [​](https://docs.chonkie.ai/oss/api/endpoints#param-include-delim) include\_delim "prev" | "next" default:"\\"prev\\"" Attach the delimiter to the previous (`"prev"`) or next (`"next"`) sentence. * * * ### [​](https://docs.chonkie.ai/oss/api/endpoints#recursive-chunker) Recursive Chunker `POST /v1/chunk/recursive` Splits text using a hierarchy of separators defined by a named recipe. Great for structured text like Markdown or code. Chunker instances are cached per `(recipe, lang, tokenizer)` for speed. curl -X POST http://localhost:8000/v1/chunk/recursive \ -H "Content-Type: application/json" \ -d '{ "text": "# Heading\n\nParagraph one.\n\nParagraph two.", "chunk_size": 256, "recipe": "markdown" }' Parameters [​](https://docs.chonkie.ai/oss/api/endpoints#param-text-2) text string | string\[\] required Text or list of texts to chunk. [​](https://docs.chonkie.ai/oss/api/endpoints#param-tokenizer-2) tokenizer string default:"character" Tokenizer to use. [​](https://docs.chonkie.ai/oss/api/endpoints#param-chunk-size-2) chunk\_size integer default:"512" Maximum tokens per chunk. [​](https://docs.chonkie.ai/oss/api/endpoints#param-recipe) recipe string default:"\\"default\\"" Named splitting recipe. Options: `"default"` (paragraph → sentence → word), `"markdown"`, `"python"`, `"js"`. [​](https://docs.chonkie.ai/oss/api/endpoints#param-lang) lang string default:"\\"en\\"" Language hint for the recipe. [​](https://docs.chonkie.ai/oss/api/endpoints#param-min-characters-per-chunk) min\_characters\_per\_chunk integer default:"24" Minimum characters to include in a chunk. * * * ### [​](https://docs.chonkie.ai/oss/api/endpoints#semantic-chunker) Semantic Chunker `POST /v1/chunk/semantic` Splits where semantic similarity between adjacent sentences drops below a threshold. Produces topically coherent chunks. Requires the `semantic` extra. curl -X POST http://localhost:8000/v1/chunk/semantic \ -H "Content-Type: application/json" \ -d '{ "text": "Dogs are loyal and friendly pets. Cats are independent animals. Quantum physics studies subatomic particles.", "embedding_model": "minishlab/potion-base-8M", "threshold": 0.5 }' Parameters [​](https://docs.chonkie.ai/oss/api/endpoints#param-text-3) text string | string\[\] required Text or list of texts to chunk. [​](https://docs.chonkie.ai/oss/api/endpoints#param-embedding-model) embedding\_model string default:"\\"minishlab/potion-base-8M\\"" Sentence-embedding model for computing similarity. Any model compatible with `sentence-transformers` works. [​](https://docs.chonkie.ai/oss/api/endpoints#param-threshold) threshold float default:"0.5" Cosine-similarity threshold for splitting (0.0–1.0). Lower values produce larger, fewer chunks. [​](https://docs.chonkie.ai/oss/api/endpoints#param-chunk-size-3) chunk\_size integer default:"512" Maximum tokens per chunk. [​](https://docs.chonkie.ai/oss/api/endpoints#param-similarity-window) similarity\_window integer default:"3" Number of surrounding sentences to consider when computing similarity. [​](https://docs.chonkie.ai/oss/api/endpoints#param-min-sentences-per-chunk-1) min\_sentences\_per\_chunk integer default:"1" Minimum sentences per chunk. [​](https://docs.chonkie.ai/oss/api/endpoints#param-min-characters-per-sentence-1) min\_characters\_per\_sentence integer default:"12" Minimum characters per sentence. * * * ### [​](https://docs.chonkie.ai/oss/api/endpoints#code-chunker) Code Chunker `POST /v1/chunk/code` Splits source code at syntactic boundaries using AST parsing. Never breaks inside a function or class. Requires the `code` extra. curl -X POST http://localhost:8000/v1/chunk/code \ -H "Content-Type: application/json" \ -d '{ "text": "def hello():\n print(\"Hello\")\n\ndef world():\n print(\"World\")", "language": "python", "chunk_size": 100 }' Parameters [​](https://docs.chonkie.ai/oss/api/endpoints#param-text-4) text string | string\[\] required Source code or list of source code snippets to chunk. [​](https://docs.chonkie.ai/oss/api/endpoints#param-tokenizer-3) tokenizer string default:"character" Tokenizer to use. [​](https://docs.chonkie.ai/oss/api/endpoints#param-chunk-size-4) chunk\_size integer default:"512" Maximum tokens per chunk. [​](https://docs.chonkie.ai/oss/api/endpoints#param-language) language string default:"\\"python\\"" Programming language. Supported: `"python"`, `"javascript"`, `"typescript"`, `"java"`, `"go"`, `"rust"`, `"c"`, `"cpp"`, and more. [​](https://docs.chonkie.ai/oss/api/endpoints#param-include-nodes) include\_nodes boolean default:"false" Include AST node metadata (node type, line numbers) in the chunk output. * * * [​](https://docs.chonkie.ai/oss/api/endpoints#refineries) Refineries ----------------------------------------------------------------------- Refineries enrich an existing list of chunks. Pass the output of any chunker endpoint directly into a refinery. ### [​](https://docs.chonkie.ai/oss/api/endpoints#overlap-refinery) Overlap Refinery `POST /v1/refine/overlap` Appends or prepends overlapping context from neighbouring chunks. Useful when downstream consumers need continuity across chunk boundaries. curl -X POST http://localhost:8000/v1/refine/overlap \ -H "Content-Type: application/json" \ -d '{ "chunks": [\ {"text": "First chunk.", "start_index": 0, "end_index": 12, "token_count": 3},\ {"text": "Second chunk.", "start_index": 13, "end_index": 26, "token_count": 3}\ ], "context_size": 0.25, "method": "suffix" }' Parameters [​](https://docs.chonkie.ai/oss/api/endpoints#param-chunks) chunks Chunk\[\] required List of chunk objects from any chunker endpoint. Each must contain `text`, `start_index`, `end_index`, and `token_count`. [​](https://docs.chonkie.ai/oss/api/endpoints#param-tokenizer-4) tokenizer string default:"character" Tokenizer to use. [​](https://docs.chonkie.ai/oss/api/endpoints#param-context-size) context\_size float | integer default:"0.25" Size of the overlap context. A float (0–1) is treated as a fraction of the chunk size; an integer is an absolute token count. [​](https://docs.chonkie.ai/oss/api/endpoints#param-mode) mode "token" | "recursive" default:"\\"token\\"" Strategy used to create the overlap window. [​](https://docs.chonkie.ai/oss/api/endpoints#param-method) method "suffix" | "prefix" | "justified" default:"\\"suffix\\"" `"suffix"` appends context from the next chunk; `"prefix"` prepends context from the previous chunk; `"justified"` adds context from both sides. [​](https://docs.chonkie.ai/oss/api/endpoints#param-merge) merge boolean default:"true" Merge the overlap context into the chunk text field. * * * ### [​](https://docs.chonkie.ai/oss/api/endpoints#embeddings-refinery) Embeddings Refinery `POST /v1/refine/embeddings` Computes and attaches embeddings to each chunk via Chonkie’s `AutoEmbeddings`. Each chunk in the response gains an `embedding` field containing a list of floats. **Local models** (e.g. `minishlab/potion-base-8M`) run entirely on-device and require no API key. **API-based models** require the appropriate environment variable for your provider. # Local model (no API key required) curl -X POST http://localhost:8000/v1/refine/embeddings \ -H "Content-Type: application/json" \ -d '{ "chunks": [\ {"text": "First chunk.", "start_index": 0, "end_index": 12, "token_count": 3},\ {"text": "Second chunk.", "start_index": 13, "end_index": 26, "token_count": 3}\ ], "embedding_model": "minishlab/potion-base-8M" }' # OpenAI (requires OPENAI_API_KEY) curl -X POST http://localhost:8000/v1/refine/embeddings \ -H "Content-Type: application/json" \ -d '{ "chunks": [\ {"text": "First chunk.", "start_index": 0, "end_index": 12, "token_count": 3}\ ], "embedding_model": "text-embedding-3-small" }' [​](https://docs.chonkie.ai/oss/api/endpoints#embeddings-providers) Embeddings Providers ------------------------------------------------------------------------------------------- | Type | Example Model | Requirement | | --- | --- | --- | | Local (model2vec) | `minishlab/potion-base-8M`, `minishlab/potion-retrieval-32M` | None | | OpenAI | `text-embedding-3-small`, `text-embedding-3-large` | `OPENAI_API_KEY` | | Cohere | `embed-english-v3.0`, `embed-multilingual-v3.0` | `COHERE_API_KEY` | | Voyage AI | `voyage-large-2`, `voyage-code-2` | `VOYAGE_API_KEY` | Parameters [​](https://docs.chonkie.ai/oss/api/endpoints#param-chunks-1) chunks Chunk\[\] required List of chunk objects to embed. [​](https://docs.chonkie.ai/oss/api/endpoints#param-embedding-model-1) embedding\_model string default:"\\"minishlab/potion-retrieval-32M\\"" Embedding model name. Local model2vec models (e.g. `minishlab/potion-base-8M`) require no API key. For API-based models, set the appropriate environment variable for your provider. * * * [​](https://docs.chonkie.ai/oss/api/endpoints#batch-processing) Batch Processing ----------------------------------------------------------------------------------- Send a list of strings to process multiple documents in one request: curl -X POST http://localhost:8000/v1/chunk/token \ -H "Content-Type: application/json" \ -d '{ "text": ["First document.", "Second document.", "Third document."], "chunk_size": 512 }' The response is a **list of lists** — one inner list of chunks per input document: [\ [{"text": "First document.", "start_index": 0, "end_index": 15, "token_count": 3}],\ [{"text": "Second document.", "start_index": 0, "end_index": 16, "token_count": 3}],\ [{"text": "Third document.", "start_index": 0, "end_index": 15, "token_count": 3}]\ ] * * * [​](https://docs.chonkie.ai/oss/api/endpoints#chaining-chunkers-and-refineries) Chaining Chunkers and Refineries ------------------------------------------------------------------------------------------------------------------- Pipeline example — chunk semantically, then add overlap context: import requests BASE = "http://localhost:8000" # Step 1: chunk chunks = requests.post(f"{BASE}/v1/chunk/semantic", json={ "text": "Your long document here...", "threshold": 0.5, }).json() # Step 2: add overlap enriched = requests.post(f"{BASE}/v1/refine/overlap", json={ "chunks": chunks, "context_size": 0.2, }).json() # Step 3: embed (requires OPENAI_API_KEY) embedded = requests.post(f"{BASE}/v1/refine/embeddings", json={ "chunks": enriched, "embedding_model": "text-embedding-3-small", }).json() * * * [​](https://docs.chonkie.ai/oss/api/endpoints#error-handling) Error Handling ------------------------------------------------------------------------------- | Status | Meaning | | --- | --- | | `200` | Success | | `400` | Invalid request parameters or chunk format | | `500` | Internal error (missing extras, model loading failure, etc.) | Error responses follow FastAPI’s standard format: { "detail": "SemanticChunker requires the 'semantic' extra. Install it with: pip install 'chonkie[semantic]'" } * * * [​](https://docs.chonkie.ai/oss/api/endpoints#health-&-info) Health & Info ----------------------------------------------------------------------------- # Health check (used by load balancers and container orchestrators) curl http://localhost:8000/health # {"status": "ok"} # API info curl http://localhost:8000/ # {"name": "Chonkie OSS API", "version": "...", "docs": "/docs", ...} Was this page helpful? YesNo [Quick Start\ \ Previous](https://docs.chonkie.ai/oss/api/quickstart) [Pipelines\ \ Next](https://docs.chonkie.ai/oss/api/pipelines) ⌘I --- # Chonkie Documentation > Documentation Index > ------------------- > > Fetch the complete documentation index at: [/llms.txt](https://docs.chonkie.ai/llms.txt) > > Use this file to discover all available pages before exploring further. [Skip to main content](https://docs.chonkie.ai/oss/api/overview#content-area) Run Chonkie as a self-hosted REST API and call any chunker or refinery from any language, framework, or HTTP client — no auth, no billing, no data leaving your infra. Quick Start ----------- Up and running in under a minute Endpoints --------- All chunkers and refineries Pipelines --------- Save and execute reusable chunking workflows Docker ------ Container and production deployment [​](https://docs.chonkie.ai/oss/api/overview#why-use-the-api) Why Use the API? --------------------------------------------------------------------------------- * **Language-agnostic** — call from JavaScript, Go, Ruby, or any HTTP client * **Self-hosted** — your data never leaves your infrastructure * **Full feature parity** — all Chonkie chunkers and refineries, over HTTP * **Batch support** — chunk multiple documents in a single request * **No auth required** — just run it and chunk away [​](https://docs.chonkie.ai/oss/api/overview#available-endpoints) Available Endpoints ---------------------------------------------------------------------------------------- | Endpoint | Description | | --- | --- | | `POST /v1/chunk/token` | Fixed-size token windows | | `POST /v1/chunk/sentence` | Sentence-boundary splitting | | `POST /v1/chunk/recursive` | Structural/hierarchical splitting | | `POST /v1/chunk/semantic` | Embedding-based semantic splitting | | `POST /v1/chunk/code` | AST-aware code splitting | | `POST /v1/refine/overlap` | Add overlap context to chunks | | `POST /v1/refine/embeddings` | Attach embeddings to chunks | | `POST /v1/pipelines` | Create a reusable pipeline | | `GET /v1/pipelines` | List all pipelines | | `GET /v1/pipelines/{id}` | Get a pipeline by ID | | `PUT /v1/pipelines/{id}` | Update a pipeline | | `DELETE /v1/pipelines/{id}` | Delete a pipeline | | `POST /v1/pipelines/{id}/execute` | Execute a pipeline on text | | `GET /health` | Health check | | `GET /` | API info and available endpoints | Was this page helpful? YesNo [Building Pipelines\ \ Previous](https://docs.chonkie.ai/oss/pipelines) [Quick Start\ \ Next](https://docs.chonkie.ai/oss/api/quickstart) ⌘I --- # Chonkie Documentation > Documentation Index > ------------------- > > Fetch the complete documentation index at: [/llms.txt](https://docs.chonkie.ai/llms.txt) > > Use this file to discover all available pages before exploring further. [Skip to main content](https://docs.chonkie.ai/oss/chefs/overview#content-area) Chefs are simple classes that automatically prepare data for future usage. They are designed to make preprocessing and data transformation easy and reusable. Chefs are available only in Python TableChef --------- Extracts tables from markdown text and prepares them for future usage. TextChef -------- Processes plain text files and returns structured Document objects. MarkdownChef ------------ Processes markdown files, extracting tables, code blocks, and images into a MarkdownDocument. MistralOCR ---------- Extracts text from images and PDFs using Mistral’s OCR API. Was this page helpful? YesNo [Docker\ \ Previous](https://docs.chonkie.ai/oss/api/docker) [TableChef\ \ Next](https://docs.chonkie.ai/oss/chefs/tablechef) ⌘I --- # Chonkie Documentation > Documentation Index > ------------------- > > Fetch the complete documentation index at: [/llms.txt](https://docs.chonkie.ai/llms.txt) > > Use this file to discover all available pages before exploring further. [Skip to main content](https://docs.chonkie.ai/oss/api/quickstart#content-area) [​](https://docs.chonkie.ai/oss/api/quickstart#1-install) 1\. Install ------------------------------------------------------------------------ pip install "chonkie[api,semantic,code,openai]" The `api` extra includes FastAPI and uvicorn. Add `semantic` for the semantic chunker and `code` for the code chunker. The embeddings refinery works out of the box with local models (e.g. `minishlab/potion-base-8M`); add the relevant extra (e.g. `openai`) only if you plan to use API-based embedding providers. [​](https://docs.chonkie.ai/oss/api/quickstart#2-start-the-server) 2\. Start the Server ------------------------------------------------------------------------------------------ Default Custom Port Debug Logging Direct Uvicorn chonkie serve # 🦛 Starting Chonkie API server on http://0.0.0.0:8000 # 📚 API docs available at http://0.0.0.0:8000/docs # 🔍 Log level: info # # Press CTRL+C to stop the server Visit `http://localhost:8000/docs` for the interactive Swagger UI, or `http://localhost:8000/redoc` for ReDoc. [​](https://docs.chonkie.ai/oss/api/quickstart#3-make-your-first-request) 3\. Make Your First Request -------------------------------------------------------------------------------------------------------- curl Python JavaScript curl -X POST http://localhost:8000/v1/chunk/token \ -H "Content-Type: application/json" \ -d '{ "text": "Chonkie makes chunking easy. It splits text into manageable pieces for RAG pipelines.", "chunk_size": 20 }' **Response:** [\ {\ "text": "Chonkie makes chunking easy.",\ "start_index": 0,\ "end_index": 28,\ "token_count": 5\ },\ {\ "text": "It splits text into manageable pieces for RAG pipelines.",\ "start_index": 29,\ "end_index": 85,\ "token_count": 10\ }\ ] [​](https://docs.chonkie.ai/oss/api/quickstart#or-use-docker) Or Use Docker ------------------------------------------------------------------------------ docker compose up The server starts on port `8000`. See the [Docker guide](https://docs.chonkie.ai/oss/api/docker) for the full `docker-compose.yml` and production setup. [​](https://docs.chonkie.ai/oss/api/quickstart#server-options) Server Options -------------------------------------------------------------------------------- | Flag | Default | Description | | --- | --- | --- | | `--host` | `0.0.0.0` | Bind address | | `--port` | `8000` | Port number | | `--reload` | `false` | Auto-reload on file changes (development only) | | `--log-level` | `info` | Log verbosity: `debug`, `info`, `warning`, `error` | [​](https://docs.chonkie.ai/oss/api/quickstart#next-steps) Next Steps ------------------------------------------------------------------------ All Endpoints ------------- Token, Sentence, Recursive, Semantic, Code, and refineries Pipelines --------- Save and execute reusable chunking workflows Docker Deployment ----------------- Production-ready Docker setup with env vars Was this page helpful? YesNo [API Server\ \ Previous](https://docs.chonkie.ai/oss/api/overview) [Endpoints\ \ Next](https://docs.chonkie.ai/oss/api/endpoints) ⌘I --- # Chonkie Documentation > Documentation Index > ------------------- > > Fetch the complete documentation index at: [/llms.txt](https://docs.chonkie.ai/llms.txt) > > Use this file to discover all available pages before exploring further. [Skip to main content](https://docs.chonkie.ai/oss/chefs/markdownchef#content-area) The `MarkdownChef` processes markdown files and strings, extracting tables, code blocks, and images into a structured `MarkdownDocument`. It intelligently parses markdown content and separates it into distinct components while preserving their positions in the original text. [​](https://docs.chonkie.ai/oss/chefs/markdownchef#installation) Installation -------------------------------------------------------------------------------- MarkdownChef is included in the base installation of Chonkie. No additional dependencies are required. For installation instructions, see the [Installation Guide](https://docs.chonkie.ai/oss/installation) . [​](https://docs.chonkie.ai/oss/chefs/markdownchef#initialization) Initialization ------------------------------------------------------------------------------------ from chonkie import MarkdownChef # Basic initialization with default tokenizer chef = MarkdownChef() # Initialize with a specific tokenizer chef = MarkdownChef(tokenizer="gpt2") # Or use a custom tokenizer instance from transformers import AutoTokenizer custom_tokenizer = AutoTokenizer.from_pretrained("bert-base-uncased") chef = MarkdownChef(tokenizer=custom_tokenizer) [​](https://docs.chonkie.ai/oss/chefs/markdownchef#parameters) Parameters ---------------------------------------------------------------------------- [​](https://docs.chonkie.ai/oss/chefs/markdownchef#param-tokenizer) tokenizer Union\[TokenizerProtocol, str\] default:"character" Tokenizer to use for counting tokens in text chunks. Can be a string identifier (“character”, “gpt2”, etc.) or a tokenizer instance that follows the TokenizerProtocol. [​](https://docs.chonkie.ai/oss/chefs/markdownchef#methods) Methods ---------------------------------------------------------------------- ### [​](https://docs.chonkie.ai/oss/chefs/markdownchef#process) process() Process a markdown file. #### [​](https://docs.chonkie.ai/oss/chefs/markdownchef#parameters-2) Parameters [​](https://docs.chonkie.ai/oss/chefs/markdownchef#param-path) path Union\[str, Path\] required Path to the markdown file (string or Path object) #### [​](https://docs.chonkie.ai/oss/chefs/markdownchef#returns) Returns `MarkdownDocument` containing parsed content with extracted tables, code, images, and text chunks ### [​](https://docs.chonkie.ai/oss/chefs/markdownchef#process-batch) process\_batch() Process multiple markdown files at once. #### [​](https://docs.chonkie.ai/oss/chefs/markdownchef#parameters-3) Parameters [​](https://docs.chonkie.ai/oss/chefs/markdownchef#param-paths) paths list\[Union\[str, Path\]\] required List of file paths to process #### [​](https://docs.chonkie.ai/oss/chefs/markdownchef#returns-2) Returns List of `MarkdownDocument` objects [​](https://docs.chonkie.ai/oss/chefs/markdownchef#basic-usage) Basic Usage ------------------------------------------------------------------------------ from chonkie import MarkdownChef # Initialize the chef chef = MarkdownChef() # Process a markdown file doc = chef.process("example.md") # Access the extracted components print(f"Found {len(doc.tables)} tables") print(f"Found {len(doc.code)} code blocks") print(f"Found {len(doc.images)} images") print(f"Found {len(doc.chunks)} text chunks") [​](https://docs.chonkie.ai/oss/chefs/markdownchef#return-type) Return Type ------------------------------------------------------------------------------ MarkdownChef returns a `MarkdownDocument` object, which extends the base `Document` class with additional fields: @dataclass class MarkdownTable: content: str # The table content start_index: int # Starting position in original text end_index: int # Ending position in original text @dataclass class MarkdownCode: content: str # The code content language: Optional[str] # Programming language (if specified) start_index: int # Starting position in original text end_index: int # Ending position in original text @dataclass class MarkdownImage: alias: str # Alt text or filename content: str # Image path or data URL start_index: int # Starting position in original text end_index: int # Ending position in original text link: Optional[str] # Link URL (if image is clickable) @dataclass class MarkdownDocument(Document): id: str # Unique document ID content: str # Full markdown content tables: list[MarkdownTable] # Extracted tables code: list[MarkdownCode] # Extracted code blocks images: list[MarkdownImage] # Extracted images chunks: list[Chunk] # Remaining text chunks metadata: dict[str, Any] # Additional metadata Was this page helpful? YesNo [TextChef\ \ Previous](https://docs.chonkie.ai/oss/chefs/textchef) [MistralOCR\ \ Next](https://docs.chonkie.ai/oss/chefs/mistral-ocr) ⌘I --- # Chonkie Documentation > Documentation Index > ------------------- > > Fetch the complete documentation index at: [/llms.txt](https://docs.chonkie.ai/llms.txt) > > Use this file to discover all available pages before exploring further. [Skip to main content](https://docs.chonkie.ai/oss/chefs/mistral-ocr#content-area) The `MistralOCR` chef extracts text from images and PDF files using Mistral’s OCR API, returning structured `MarkdownDocument` objects for further processing. [​](https://docs.chonkie.ai/oss/chefs/mistral-ocr#installation) Installation ------------------------------------------------------------------------------- pip install chonkie[mistral] You need a Mistral API key. Set the `MISTRAL_API_KEY` environment variable or pass it directly. [​](https://docs.chonkie.ai/oss/chefs/mistral-ocr#initialization) Initialization ----------------------------------------------------------------------------------- from chonkie import MistralOCR # Default initialization (uses MISTRAL_API_KEY env var) ocr = MistralOCR() # Custom model and explicit API key ocr = MistralOCR(model="mistral-ocr-2505", api_key="sk-...") ### [​](https://docs.chonkie.ai/oss/chefs/mistral-ocr#parameters) Parameters [​](https://docs.chonkie.ai/oss/chefs/mistral-ocr#param-model) model str default:"mistral-ocr-latest" The Mistral OCR model to use. [​](https://docs.chonkie.ai/oss/chefs/mistral-ocr#param-api-key) api\_key Optional\[str\] default:"None" Mistral API key. Falls back to the `MISTRAL_API_KEY` environment variable. [​](https://docs.chonkie.ai/oss/chefs/mistral-ocr#methods) Methods --------------------------------------------------------------------- ### [​](https://docs.chonkie.ai/oss/chefs/mistral-ocr#process) process() Process an image or PDF file and return a `MarkdownDocument`. #### [​](https://docs.chonkie.ai/oss/chefs/mistral-ocr#parameters-2) Parameters [​](https://docs.chonkie.ai/oss/chefs/mistral-ocr#param-path) path Union\[str, Path\] required Path to the image or PDF file. #### [​](https://docs.chonkie.ai/oss/chefs/mistral-ocr#returns) Returns `MarkdownDocument` containing the extracted text as markdown content. ### [​](https://docs.chonkie.ai/oss/chefs/mistral-ocr#process_batch) process\_batch() Process multiple image or PDF files at once. #### [​](https://docs.chonkie.ai/oss/chefs/mistral-ocr#parameters-3) Parameters [​](https://docs.chonkie.ai/oss/chefs/mistral-ocr#param-paths) paths list\[Union\[str, Path\]\] required List of file paths to process. #### [​](https://docs.chonkie.ai/oss/chefs/mistral-ocr#returns-2) Returns `list[MarkdownDocument]` where each document contains extracted text from a file. ### [​](https://docs.chonkie.ai/oss/chefs/mistral-ocr#parse) parse() Parse raw text into a `Document` (wraps text as-is, since OCR operates on files). #### [​](https://docs.chonkie.ai/oss/chefs/mistral-ocr#parameters-4) Parameters [​](https://docs.chonkie.ai/oss/chefs/mistral-ocr#param-text) text str required Raw text to wrap into a Document. #### [​](https://docs.chonkie.ai/oss/chefs/mistral-ocr#returns-3) Returns `Document` containing the provided text. [​](https://docs.chonkie.ai/oss/chefs/mistral-ocr#supported-file-types) Supported File Types ----------------------------------------------------------------------------------------------- | Type | Extensions | | --- | --- | | Images | `.png`, `.jpg`, `.jpeg`, `.gif`, `.bmp`, `.webp`, `.tiff`, `.tif` | | Documents | `.pdf` | [​](https://docs.chonkie.ai/oss/chefs/mistral-ocr#usage) Usage ----------------------------------------------------------------- ### [​](https://docs.chonkie.ai/oss/chefs/mistral-ocr#standalone) Standalone from chonkie import MistralOCR ocr = MistralOCR() # Single file doc = ocr.process("research_paper.pdf") print(doc.content) print(f"Source: {doc.metadata['filename']}") # Multiple files docs = ocr.process_batch(["page1.png", "page2.png"]) # Async import asyncio doc = asyncio.run(ocr.aprocess("document.pdf")) ### [​](https://docs.chonkie.ai/oss/chefs/mistral-ocr#pipeline) Pipeline Use `.process_with("mistral")` to add OCR to a pipeline: from chonkie import Pipeline # Process a PDF with OCR and chunk it doc = (Pipeline() .fetch_from("file", path="document.pdf") .process_with("mistral") .chunk_with("recursive", chunk_size=512) .run()) print(f"Extracted {len(doc.chunks)} chunks from PDF") ### [​](https://docs.chonkie.ai/oss/chefs/mistral-ocr#ocr-+-rag-pipeline) OCR + RAG Pipeline Build a complete pipeline from scanned documents to vector database: from chonkie import Pipeline docs = (Pipeline() .fetch_from("file", dir="./scanned_docs", ext=[".pdf", ".png"]) .process_with("mistral") .chunk_with("recursive", chunk_size=1024) .refine_with("overlap", context_size=100) .store_in("qdrant", collection_name="scanned_documents") .run()) print(f"Ingested {len(docs)} documents") ### [​](https://docs.chonkie.ai/oss/chefs/mistral-ocr#ocr-+-semantic-chunking) OCR + Semantic Chunking Use semantic chunking on OCR output for intelligent retrieval boundaries: from chonkie import Pipeline doc = (Pipeline() .fetch_from("file", path="textbook_chapter.pdf") .process_with("mistral") .chunk_with("semantic", threshold=0.8, chunk_size=1024) .refine_with("embedding", model="text-embedding-3-small") .export_with("json", file="textbook_chunks.json") .run()) [​](https://docs.chonkie.ai/oss/chefs/mistral-ocr#integration-with-chunkers) Integration with Chunkers --------------------------------------------------------------------------------------------------------- MistralOCR returns a `MarkdownDocument`, making it compatible with any chunker: from chonkie import MistralOCR, RecursiveChunker # Step 1: Extract text from PDF ocr = MistralOCR() doc = ocr.process("report.pdf") # Step 2: Chunk the extracted content chunker = RecursiveChunker(chunk_size=512) chunks = chunker.chunk(doc.content) # Step 3: Store chunks in the document doc.chunks = chunks print(f"Document: {doc.metadata['filename']}") print(f" Content: {len(doc.content)} characters") print(f" Chunks: {len(doc.chunks)}") [​](https://docs.chonkie.ai/oss/chefs/mistral-ocr#notes) Notes ----------------------------------------------------------------- * OCR quality depends on image resolution and clarity * Large PDFs are processed page-by-page and concatenated with double newlines * The extracted text is returned as markdown, preserving structure from the source document * API calls are synchronous by default; use `aprocess()` for async execution Was this page helpful? YesNo [MarkdownChef\ \ Previous](https://docs.chonkie.ai/oss/chefs/markdownchef) [Fetchers Overview\ \ Next](https://docs.chonkie.ai/oss/fetchers/overview) ⌘I --- # Chonkie Documentation > Documentation Index > ------------------- > > Fetch the complete documentation index at: [/llms.txt](https://docs.chonkie.ai/llms.txt) > > Use this file to discover all available pages before exploring further. [Skip to main content](https://docs.chonkie.ai/oss/api/pipelines#content-area) [​](https://docs.chonkie.ai/oss/api/pipelines#what-are-pipelines) What Are Pipelines? ---------------------------------------------------------------------------------------- A pipeline is a named, reusable configuration that describes a sequence of chunking and refinement steps. Instead of passing the same configuration on every request, you define it once and reference it by ID. A pipeline step is either: * **`chunk`** — runs a chunker (e.g. `"semantic"`, `"token"`, `"recursive"`) * **`refine`** — runs a refinery (e.g. `"embeddings"`, `"overlap"`) [​](https://docs.chonkie.ai/oss/api/pipelines#create-a-pipeline) Create a Pipeline ------------------------------------------------------------------------------------- `POST /v1/pipelines` curl -X POST http://localhost:8000/v1/pipelines \ -H "Content-Type: application/json" \ -d '{ "name": "rag-chunker", "description": "Semantic chunking with embeddings for RAG", "steps": [\ {\ "type": "chunk",\ "chunker": "semantic",\ "config": {"chunk_size": 512, "threshold": 0.5}\ },\ {\ "type": "refine",\ "refinery": "embeddings",\ "config": {"embedding_model": "text-embedding-3-small"}\ }\ ] }' **Response (201 Created):** { "id": "550e8400-e29b-41d4-a716-446655440000", "name": "rag-chunker", "description": "Semantic chunking with embeddings for RAG", "config": { "steps": [\ {"type": "chunk", "chunker": "semantic", "refinery": null, "config": {"chunk_size": 512, "threshold": 0.5}},\ {"type": "refine", "chunker": null, "refinery": "embeddings", "config": {"embedding_model": "text-embedding-3-small"}}\ ] }, "created_at": "2026-02-20T10:00:00.000000", "updated_at": "2026-02-20T10:00:00.000000" } Request Parameters [​](https://docs.chonkie.ai/oss/api/pipelines#param-name) name string required Unique pipeline name. Used as a human-readable identifier. [​](https://docs.chonkie.ai/oss/api/pipelines#param-description) description string Optional description of what this pipeline does. [​](https://docs.chonkie.ai/oss/api/pipelines#param-steps) steps PipelineStep\[\] required Ordered list of steps to execute. Each step has: * `type`: `"chunk"` or `"refine"` * `chunker`: chunker name (for `chunk` steps, e.g. `"semantic"`, `"token"`) * `refinery`: refinery name (for `refine` steps, e.g. `"embeddings"`, `"overlap"`) * `config`: step-specific parameters (same fields as the individual endpoints) * * * [​](https://docs.chonkie.ai/oss/api/pipelines#list-pipelines) List Pipelines ------------------------------------------------------------------------------- `GET /v1/pipelines` curl http://localhost:8000/v1/pipelines Returns all pipelines ordered by creation date (newest first). * * * [​](https://docs.chonkie.ai/oss/api/pipelines#get-a-pipeline) Get a Pipeline ------------------------------------------------------------------------------- `GET /v1/pipelines/{pipeline_id}` curl http://localhost:8000/v1/pipelines/550e8400-e29b-41d4-a716-446655440000 * * * [​](https://docs.chonkie.ai/oss/api/pipelines#update-a-pipeline) Update a Pipeline ------------------------------------------------------------------------------------- `PUT /v1/pipelines/{pipeline_id}` You can update `name`, `description`, or `steps` independently: curl -X PUT http://localhost:8000/v1/pipelines/550e8400-e29b-41d4-a716-446655440000 \ -H "Content-Type: application/json" \ -d '{ "description": "Updated description", "steps": [\ {\ "type": "chunk",\ "chunker": "recursive",\ "config": {"chunk_size": 1024, "recipe": "markdown"}\ }\ ] }' * * * [​](https://docs.chonkie.ai/oss/api/pipelines#delete-a-pipeline) Delete a Pipeline ------------------------------------------------------------------------------------- `DELETE /v1/pipelines/{pipeline_id}` curl -X DELETE http://localhost:8000/v1/pipelines/550e8400-e29b-41d4-a716-446655440000 Returns `204 No Content` on success. * * * [​](https://docs.chonkie.ai/oss/api/pipelines#pipeline-examples) Pipeline Examples ------------------------------------------------------------------------------------- ### [​](https://docs.chonkie.ai/oss/api/pipelines#basic-token-chunking) Basic Token Chunking { "name": "token-basic", "steps": [\ {"type": "chunk", "chunker": "token", "config": {"chunk_size": 512}}\ ] } ### [​](https://docs.chonkie.ai/oss/api/pipelines#markdown-documents-with-overlap) Markdown Documents with Overlap { "name": "markdown-with-overlap", "description": "Recursive markdown chunking with overlap context", "steps": [\ {\ "type": "chunk",\ "chunker": "recursive",\ "config": {"chunk_size": 512, "recipe": "markdown"}\ },\ {\ "type": "refine",\ "refinery": "overlap",\ "config": {"context_size": 0.2, "method": "suffix"}\ }\ ] } ### [​](https://docs.chonkie.ai/oss/api/pipelines#full-rag-pipeline) Full RAG Pipeline { "name": "full-rag", "description": "Semantic chunking + overlap + embeddings", "steps": [\ {\ "type": "chunk",\ "chunker": "semantic",\ "config": {"chunk_size": 512, "threshold": 0.5}\ },\ {\ "type": "refine",\ "refinery": "overlap",\ "config": {"context_size": 0.1}\ },\ {\ "type": "refine",\ "refinery": "embeddings",\ "config": {"embedding_model": "voyage-large-2"}\ }\ ] } * * * [​](https://docs.chonkie.ai/oss/api/pipelines#storage) Storage ----------------------------------------------------------------- Pipelines are stored in a local SQLite database (`data/chonkie.db`). The database is created automatically on first startup. When using Docker, mount `./data:/app/data` to persist the database across container restarts. * * * [​](https://docs.chonkie.ai/oss/api/pipelines#execute-a-pipeline) Execute a Pipeline --------------------------------------------------------------------------------------- `POST /v1/pipelines/{pipeline_id}/execute` Runs the pipeline steps sequentially on the provided text. Each `chunk` step produces chunks; each `refine` step enriches them. Returns the final list of chunks. curl -X POST http://localhost:8000/v1/pipelines/550e8400-e29b-41d4-a716-446655440000/execute \ -H "Content-Type: application/json" \ -d '{"text": "Your document text goes here. It will be chunked and refined."}' **Response:** [\ {\ "id": "chnk_abc123",\ "text": "Your document text goes here.",\ "start_index": 0,\ "end_index": 29,\ "token_count": 29,\ "context": null,\ "embedding": null\ },\ {\ "id": "chnk_def456",\ "text": "It will be chunked and refined.",\ "start_index": 30,\ "end_index": 61,\ "token_count": 31,\ "context": null,\ "embedding": null\ }\ ] ### [​](https://docs.chonkie.ai/oss/api/pipelines#batch-execution) Batch Execution Submit a list of strings to process multiple documents in one request. The response is a list of lists — one inner list per input document. curl -X POST http://localhost:8000/v1/pipelines/550e8400-e29b-41d4-a716-446655440000/execute \ -H "Content-Type: application/json" \ -d '{"text": ["First document.", "Second document.", "Third document."]}' Request Parameters [​](https://docs.chonkie.ai/oss/api/pipelines#param-text) text string | string\[\] required Text or list of texts to process through the pipeline. ### [​](https://docs.chonkie.ai/oss/api/pipelines#error-responses) Error Responses | Status | Cause | | --- | --- | | `404` | Pipeline ID not found | | `400` | Pipeline has no steps, a `refine` step appears before any `chunk` step, or a step is missing required fields | | `500` | A step failed at runtime (e.g. missing extra, model error) | Was this page helpful? YesNo [Endpoints\ \ Previous](https://docs.chonkie.ai/oss/api/endpoints) [Docker\ \ Next](https://docs.chonkie.ai/oss/api/docker) ⌘I --- # Chonkie Documentation > Documentation Index > ------------------- > > Fetch the complete documentation index at: [/llms.txt](https://docs.chonkie.ai/llms.txt) > > Use this file to discover all available pages before exploring further. [Skip to main content](https://docs.chonkie.ai/oss/changelog#content-area) [​](https://docs.chonkie.ai/oss/changelog#v1-5-4) v1.5.4 [​](https://docs.chonkie.ai/oss/changelog#v1-5-4-release-highlights-) v1.5.4 Release Highlights ✨ ==================================================================================================== * **New `GroqGenie`**: Fast inference on Groq hardware! Use Llama models with blazing speed via Groq’s infrastructure. pip install "chonkie[groq]" from chonkie import GroqGenie genie = GroqGenie(model="llama-3.3-70b-versatile") response = genie.generate("Hello!") * **New `CerebrasGenie`**: Fastest inference on Cerebras hardware! Experience ultra-fast LLM inference. pip install "chonkie[cerebras]" from chonkie import CerebrasGenie genie = CerebrasGenie(model="llama-3.3-70b") response = genie.generate("Hello!") Both new Genies support `generate()` for text generation and `generate_json()` for structured JSON output, following the same interface as existing Genies.**Full Changelog**: [https://github.com/chonkie-inc/chonkie/compare/v1.5.3…v1.5.4](https://github.com/chonkie-inc/chonkie/compare/v1.5.3...v1.5.4) [​](https://docs.chonkie.ai/oss/changelog#v1-3-0) v1.3.0 [​](https://docs.chonkie.ai/oss/changelog#v1-3-0-release-highlights-) v1.3.0 Release Highlights ✨ ==================================================================================================== [​](https://docs.chonkie.ai/oss/changelog#breaking-changes) Breaking Changes ------------------------------------------------------------------------------- * **Unified Chunk Type**: All chunkers now return the base `Chunk` type instead of specialized types. The specialized chunk types (`SentenceChunk`, `RecursiveChunk`, `SemanticChunk`, `CodeChunk`, and `LateChunk`) have been removed entirely. This simplifies the API and improves interoperability between different chunkers and refineries. * **Unified Sentence Type**: The `SemanticSentence` type has been removed. The base `Sentence` type now includes an optional `embedding` attribute, providing the same functionality with a simpler API. * **New `embedding` Attribute**: Both the base `Chunk` and `Sentence` types now include an optional `embedding` attribute that can store embedding vectors (as lists or numpy arrays). This is automatically populated by `EmbeddingsRefinery` and certain chunkers like `LateChunker`. [​](https://docs.chonkie.ai/oss/changelog#migration-guide) Migration Guide ----------------------------------------------------------------------------- If you were relying on specialized chunk attributes: * `SentenceChunk.sentences` → No longer available in base Chunk * `SemanticChunk.sentences` → No longer available in base Chunk * `CodeChunk.nodes` → No longer available in base Chunk * `RecursiveChunk.level` → No longer available in base Chunk * `LateChunk` → Use base `Chunk` (embedding is now part of base type) * `SemanticSentence` → Use base `Sentence` (embedding is now part of base type) All chunkers now consistently return `Chunk` objects with: @dataclass class Chunk: text: str start_index: int end_index: int token_count: int context: Optional[Context] = None embedding: Union[list[float], "np.ndarray", None] = None # NEW! [​](https://docs.chonkie.ai/oss/changelog#import-changes) Import Changes --------------------------------------------------------------------------- When importing the Chunk type, use: from chonkie.types import Chunk The specialized types are deprecated but remain available for backward compatibility in the legacy module. [​](https://docs.chonkie.ai/oss/changelog#v1-0-6) v1.0.6 [​](https://docs.chonkie.ai/oss/changelog#v1-0-6-release-highlights-) v1.0.6 Release Highlights ✨ ==================================================================================================== * **New `SlumberChunker`**: Welcome Chonkie’s very own agentic chunker! Requires the `genie` optional install and a `GEMINI_API_KEY`. It leverages `Genie`, Chonkie’s interface for generative models. pip install "chonkie[genie]" # Import from chonkie import SlumberChunker # Initialize chunker = SlumberChunker(verbose=True) # set verbose to True, since it takes a while~ # CHONK! chunker(text) * **New `NeuralChunker`**: Introducing a fully neural approach to chunking! Requires the `neural` optional install. This uses a fine-tuned BERT-like model for fast, high-quality chunking. pip install "chonkie[neural]" # import from chonkie import NeuralChunker # initialize chunker = NeuralChunker() # CHONK! chunks = chunker(text) * **`auto` Language Detection for `CodeChunker`**: `CodeChunker` can now automatically detect the programming language. Specify the language manually if performance is critical. # Import from chonkie import CodeChunker # Initialize the "auto" CodeChunker chunker = CodeChunker() # No need to specify, "auto" by default # CHONK! chunks = chunker(code) * **Introducing `Genie`s**: Added `Genie` to power `SlumberChunker` and future generative features. `Genie`s are Chonkie’s way to handle multiple generative APIs and model interfaces. The first is `GeminiGenie`, requiring the `genie` optional install. pip install "chonkie[genie]" # Import from chonkie import GeminiGenie # Init genie = GeminiGenie(api_key=YOUR_API_KEY) # generate genie.generate("Hi!") # generate JSON genie.generate_json("Hi", JSON_SCHEMA) **Full Changelog**: [https://github.com/chonkie-inc/chonkie/compare/v1.0.5…v1.0.6](https://github.com/chonkie-inc/chonkie/compare/v1.0.5...v1.0.6) [​](https://docs.chonkie.ai/oss/changelog#v1-0-5) v1.0.5 [​](https://docs.chonkie.ai/oss/changelog#v1-0-5-release-highlights-) v1.0.5 Release Highlights ✨ ==================================================================================================== This is a quick patch release to include `CodeChunker` in the `__init__.py` for `chonkie` so it can be properly accessed via `from chonkie import CodeChunker`.**Full Changelog**: [https://github.com/chonkie-inc/chonkie/compare/v1.0.4…v1.0.5](https://github.com/chonkie-inc/chonkie/compare/v1.0.4...v1.0.5) [​](https://docs.chonkie.ai/oss/changelog#v1-0-4) v1.0.4 [​](https://docs.chonkie.ai/oss/changelog#v1-0-4-release-highlights-) v1.0.4 Release Highlights ✨ ==================================================================================================== * **New `CodeChunker`**: Introducing the `CodeChunker`, specialized for handling code files across 100+ programming languages. It understands code structure to provide more meaningful chunks. pip install "chonkie[code]" # Initialize the code chunker chunker = CodeChunker(language="python") # Chunk the code code = ... # Your code string # CHONK! chunks = chunker(code) * **`JinaAI` Embeddings Support**: Added `JinaEmbeddings`, enabling their use with `SemanticChunker` and `SDPMChunker`. Just install the `jina` optional install to use it! pip install "chonkie[jina]" # Initialize the Jina embeddings from chonkie import JinaEmbeddings, SemanticChunker # Initialize the Jina embeddings embeddings = JinaEmbeddings() # Initialize the semantic chunker chunker = SemanticChunker(embeddings) # Chunk the text text = ... # Your text string # CHONK! chunks = chunker(text) * **`OverlapRefinery`**: Enhance your chunks by adding overlapping context using the new `OverlapRefinery`. It’s included in the default install and works seamlessly with any chunker. # Initialize the recursive chunker from chonkie import RecursiveChunker, OverlapRefinery chunker = RecursiveChunker() # Initialize the overlap refinery refinery = OverlapRefinery() # Or OverlapRefinery("gpt2") # Chunk the text text = ... # Your text string # CHONK! chunks = chunker(text) # Refine the chunks chunks = refinery(chunks) * **`EmbeddingsRefinery`**: Compute and attach embeddings directly to your chunks using the `EmbeddingsRefinery`. Streamline the process of loading chunks into vector databases. from chonkie import RecursiveChunker, EmbeddingsRefinery, JinaEmbeddings # Initialize the recursive chunker chunker = RecursiveChunker() # Initialize the embeddings model # Here we use Jina embeddings for this example, but you can use any other embeddings model embeddings = JinaEmbeddings() # Initialize the embeddings refinery refinery = EmbeddingsRefinery(embeddings) # Chunk the text text = ... # Your text string chunks = chunker(text) chunks = refinery(chunks) # Each chunk now has a .embedding attribute **Full Changelog**: [https://github.com/chonkie-inc/chonkie/compare/v1.0.3…v1.0.4](https://github.com/chonkie-inc/chonkie/compare/v1.0.3...v1.0.4) [​](https://docs.chonkie.ai/oss/changelog#v1-0-3) v1.0.3 [​](https://docs.chonkie.ai/oss/changelog#v1-0-3-release-highlights-) v1.0.3 Release Highlights ✨ ==================================================================================================== * **Chonkie `Visualizer`**: Visualize and debug chunks easily via terminal printouts or HTML saves. Understand chunk quality and debug your chunker with visual feedback~ Use the `print` method to print rich text on your terminal or use the `save` method to save a highlighted `html` on your device! It’s very simple to use, just pass in your chunks~ from chonkie import Visualizer viz = Visualizer() # Print the chunks on the terminal with .print or directly call the Visualizer object too viz.print(chunks) # Save the HTML file viz.save("chonkie.html", chunks) ![Chonkie Visualizer Example](https://github.com/user-attachments/assets/ea38959b-01bd-4be9-90de-6442542e98a0) * **Recipes**: Chonkie now adds support for `Recipes` which allow you to use multilingual chunking out-of-the-box, as well as document specific chunking methods. Initial support starts with: `en`, `hi`, `zh`, `jp` and `ko`, while document type `markdown` is supported too. Use it via the `from_recipe` class method with any chunker that takes delimiters or `RecursiveRules`. from chonkie import RecursiveChunker # Initialize the recursive chunker to chunk Markdown chunker = RecursiveChunker.from_recipe("markdown", lang="en") # Initialize the recursive chunker to chunk Hindi texts chunker = RecursiveChunker.from_recipe(lang="hi") * Performance enhancements in `RecursiveChunker`, `SentenceChunker`, and `WordTokenizer`. **Full Changelog**: [https://github.com/chonkie-inc/chonkie/compare/v1.0.2…v1.0.3](https://github.com/chonkie-inc/chonkie/compare/v1.0.2...v1.0.3) Was this page helpful? YesNo [SDPM Chunker (Legacy)\ \ Previous](https://docs.chonkie.ai/oss/chunkers/sdpm-chunker) ⌘I --- # Chonkie Documentation > Documentation Index > ------------------- > > Fetch the complete documentation index at: [/llms.txt](https://docs.chonkie.ai/llms.txt) > > Use this file to discover all available pages before exploring further. [Skip to main content](https://docs.chonkie.ai/oss/porters/overview#content-area) Porters allow you to easily Port your chunks to any format or destination. JSONPorter ---------- Port your chunks to a JSON file. DatasetsPorter -------------- Port your chunks to a Hugging Face Datasets. Was this page helpful? YesNo [Weaviate Handshake\ \ Previous](https://docs.chonkie.ai/oss/handshakes/weaviate-handshake) [JSONPorter\ \ Next](https://docs.chonkie.ai/oss/porters/json-porter) ⌘I --- # Chonkie Documentation > Documentation Index > ------------------- > > Fetch the complete documentation index at: [/llms.txt](https://docs.chonkie.ai/llms.txt) > > Use this file to discover all available pages before exploring further. [Skip to main content](https://docs.chonkie.ai/oss/chefs/textchef#content-area) The `TextChef` processes plain text files and returns structured `Document` objects for further processing. [​](https://docs.chonkie.ai/oss/chefs/textchef#installation) Installation ---------------------------------------------------------------------------- TextChef is included in the base installation of Chonkie. No additional dependencies are required. For installation instructions, see the [Installation Guide](https://docs.chonkie.ai/oss/installation) . [​](https://docs.chonkie.ai/oss/chefs/textchef#initialization) Initialization -------------------------------------------------------------------------------- from chonkie import TextChef # Simple initialization - no parameters required chef = TextChef() [​](https://docs.chonkie.ai/oss/chefs/textchef#methods) Methods ------------------------------------------------------------------ ### [​](https://docs.chonkie.ai/oss/chefs/textchef#process) process() Process a text file and return a `Document` object. #### [​](https://docs.chonkie.ai/oss/chefs/textchef#parameters) Parameters [​](https://docs.chonkie.ai/oss/chefs/textchef#param-path) path Union\[str, Path\] required Path to the text file (string or Path object) #### [​](https://docs.chonkie.ai/oss/chefs/textchef#returns) Returns `Document` object containing the file content ### [​](https://docs.chonkie.ai/oss/chefs/textchef#process-batch) process\_batch() Process multiple text files at once. #### [​](https://docs.chonkie.ai/oss/chefs/textchef#parameters-2) Parameters [​](https://docs.chonkie.ai/oss/chefs/textchef#param-paths) paths list\[Union\[str, Path\]\] required List of file paths to process #### [​](https://docs.chonkie.ai/oss/chefs/textchef#returns-2) Returns `list[Document]` where each `Document` contains a file’s contents. [​](https://docs.chonkie.ai/oss/chefs/textchef#usage) Usage -------------------------------------------------------------- from chonkie import TextChef # Initialize the chef chef = TextChef() # Process a text file doc = chef.process("example.txt") # Access the content print(doc.content) print(f"Document ID: {doc.id}") [​](https://docs.chonkie.ai/oss/chefs/textchef#integration-with-chunkers) Integration with Chunkers ------------------------------------------------------------------------------------------------------ TextChef is designed to work seamlessly with Chonkie’s chunkers: from chonkie import TextChef, TokenChunker # Step 1: Load text file chef = TextChef() doc = chef.process("article.txt") # Step 2: Chunk the content chunker = TokenChunker(chunk_size=512, chunk_overlap=50) chunks = chunker.chunk(doc.content) # Step 3: Store chunks back in the document doc.chunks = chunks # Now your document has both content and chunks print(f"Document {doc.id}:") print(f" Content: {len(doc.content)} characters") print(f" Chunks: {len(doc.chunks)}") [​](https://docs.chonkie.ai/oss/chefs/textchef#encoding) Encoding -------------------------------------------------------------------- TextChef reads files with UTF-8 encoding by default, ensuring proper handling of: * Unicode characters * International text * Special symbols * Emoji and other non-ASCII characters All text is read as strings and preserved exactly as it appears in the source file. Was this page helpful? YesNo [TableChef\ \ Previous](https://docs.chonkie.ai/oss/chefs/tablechef) [MarkdownChef\ \ Next](https://docs.chonkie.ai/oss/chefs/markdownchef) ⌘I --- # Chonkie Documentation > Documentation Index > ------------------- > > Fetch the complete documentation index at: [/llms.txt](https://docs.chonkie.ai/llms.txt) > > Use this file to discover all available pages before exploring further. [Skip to main content](https://docs.chonkie.ai/oss/chefs/tablechef#content-area) The `TableChef` is a versatile chef that extracts and processes tables from multiple sources. It can read CSV and Excel files, convert them to markdown format, or extract tables from markdown text. The parsed tables are returned in a structured format ready for downstream processing. [​](https://docs.chonkie.ai/oss/chefs/tablechef#installation) Installation ----------------------------------------------------------------------------- TableChef requires the `pandas` library for processing CSV and Excel files. pip install "chonkie[table]" For more installation options, see the [Installation Guide](https://docs.chonkie.ai/oss/installation) . [​](https://docs.chonkie.ai/oss/chefs/tablechef#initialization) Initialization --------------------------------------------------------------------------------- from chonkie import TableChef chef = TableChef() [​](https://docs.chonkie.ai/oss/chefs/tablechef#methods) Methods ------------------------------------------------------------------- ### [​](https://docs.chonkie.ai/oss/chefs/tablechef#process) process() Process a file or markdown string to extract tables. #### [​](https://docs.chonkie.ai/oss/chefs/tablechef#parameters) Parameters [​](https://docs.chonkie.ai/oss/chefs/tablechef#param-path) path Union\[str, Path\] required Can be a file path (CSV/Excel) or a markdown string containing tables #### [​](https://docs.chonkie.ai/oss/chefs/tablechef#returns) Returns list\[MarkdownTable\] | None A list of `MarkdownTable` objects. `None` if no tables are found ### [​](https://docs.chonkie.ai/oss/chefs/tablechef#process_batch) process\_batch() Process multiple files or markdown strings at once. #### [​](https://docs.chonkie.ai/oss/chefs/tablechef#parameters-2) Parameters [​](https://docs.chonkie.ai/oss/chefs/tablechef#param-paths) paths Union\[list\[str\], list\[Path\]\] required List of file paths or markdown strings to process #### [​](https://docs.chonkie.ai/oss/chefs/tablechef#returns-2) Returns list\[MarkdownTable\] | None A list of `MarkdownTable` objects. `None` if no tables are found [​](https://docs.chonkie.ai/oss/chefs/tablechef#usage) Usage --------------------------------------------------------------- Files (CSV/Excel) Markdown Tables HTML Tables from chonkie import TableChef # Initialize the chef chef = TableChef() # Process a CSV file doc = chef.process("data.csv") print(f"Extracted {len(doc.tables)} table from CSV") # Process an Excel file (all sheets) doc = chef.process("spreadsheet.xlsx") print(f"Extracted {len(doc.tables)} tables from Excel") [​](https://docs.chonkie.ai/oss/chefs/tablechef#supported-file-formats) Supported File Formats ------------------------------------------------------------------------------------------------- * **CSV files** (`.csv`) - Comma-separated values * **Excel files** (`.xls`, `.xlsx`) - Microsoft Excel spreadsheets * **Markdown strings** - Text containing pipe-separated tables or HTML tables (``) Was this page helpful? YesNo [Chefs Overview\ \ Previous](https://docs.chonkie.ai/oss/chefs/overview) [TextChef\ \ Next](https://docs.chonkie.ai/oss/chefs/textchef) ⌘I --- # Chonkie Documentation > Documentation Index > ------------------- > > Fetch the complete documentation index at: [/llms.txt](https://docs.chonkie.ai/llms.txt) > > Use this file to discover all available pages before exploring further. [Skip to main content](https://docs.chonkie.ai/oss/embeddings/sentence-transformer-embeddings#content-area) Embeddings are handled by the `SentenceTransformer` class, which is a wrapper around the `sentence-transformers` library. [​](https://docs.chonkie.ai/oss/embeddings/sentence-transformer-embeddings#installation) Installation -------------------------------------------------------------------------------------------------------- Embeddings require the `sentence-transformers` library. See the [Installation Guide](https://docs.chonkie.ai/oss/installation) for more information. [​](https://docs.chonkie.ai/oss/embeddings/sentence-transformer-embeddings#usage) Usage ------------------------------------------------------------------------------------------ from chonkie import SentenceTransformerEmbeddings embeddings = SentenceTransformerEmbeddings("all-MiniLM-L6-v2") Was this page helpful? YesNo [CohereEmbeddings\ \ Previous](https://docs.chonkie.ai/oss/embeddings/cohere-embeddings) [OpenAIEmbeddings\ \ Next](https://docs.chonkie.ai/oss/embeddings/openai-embeddings) ⌘I --- # Chonkie Documentation > Documentation Index > ------------------- > > Fetch the complete documentation index at: [/llms.txt](https://docs.chonkie.ai/llms.txt) > > Use this file to discover all available pages before exploring further. [Skip to main content](https://docs.chonkie.ai/oss/embeddings/openai-embeddings#content-area) Embeddings are handled by the `OpenAIEmbeddings` class, which is a wrapper around the `openai` library. [​](https://docs.chonkie.ai/oss/embeddings/openai-embeddings#installation) Installation ------------------------------------------------------------------------------------------ Embeddings require the `openai` library. See the [Installation Guide](https://docs.chonkie.ai/oss/installation) for more information. [​](https://docs.chonkie.ai/oss/embeddings/openai-embeddings#usage) Usage ---------------------------------------------------------------------------- from chonkie import OpenAIEmbeddings embeddings = OpenAIEmbeddings() Was this page helpful? YesNo [SentenceTransformerEmbeddings\ \ Previous](https://docs.chonkie.ai/oss/embeddings/sentence-transformer-embeddings) [AzureOpenAIEmbeddings\ \ Next](https://docs.chonkie.ai/oss/embeddings/azure-embeddings) ⌘I --- # Chonkie Documentation > Documentation Index > ------------------- > > Fetch the complete documentation index at: [/llms.txt](https://docs.chonkie.ai/llms.txt) > > Use this file to discover all available pages before exploring further. [Skip to main content](https://docs.chonkie.ai/oss/refinery/overview#content-area) Refinery is a module in Chonkie that allows you to refine chunks. Refineries help you to add additional `context` to your chunks which are useful to improve the quality of your embeddings and keyword indexing. OverlapRefinery --------------- Refines chunks by adding overlapping chunks to the original chunk. EmbeddingsRefinery ------------------ Refines chunks by adding embeddings to the original chunk. Was this page helpful? YesNo [Create your own custom embeddings handler\ \ Previous](https://docs.chonkie.ai/oss/embeddings/custom-embeddings) [Overlap Refinery\ \ Next](https://docs.chonkie.ai/oss/refinery/overlap-refinery) ⌘I --- # Chonkie Documentation > Documentation Index > ------------------- > > Fetch the complete documentation index at: [/llms.txt](https://docs.chonkie.ai/llms.txt) > > Use this file to discover all available pages before exploring further. [Skip to main content](https://docs.chonkie.ai/oss/embeddings/jina-embeddings#content-area) JinaEmbeddings is a utility class to use JinaAI’s API for Chonkie’s semantic chunking. [​](https://docs.chonkie.ai/oss/embeddings/jina-embeddings#installation) Installation ---------------------------------------------------------------------------------------- Embeddings require the `jina` library. See the [Installation Guide](https://docs.chonkie.ai/oss/installation) for more information. pip install "chonkie[jina]" [​](https://docs.chonkie.ai/oss/embeddings/jina-embeddings#usage) Usage -------------------------------------------------------------------------- from chonkie import JinaEmbeddings # Initialize the Jina embeddings embeddings = JinaEmbeddings() # Initialize the semantic chunker chunker = SemanticChunker(embeddings) # Chunk the text text = ... # Your text string # CHONK! chunks = chunker(text) Was this page helpful? YesNo [Model2VecEmbeddings\ \ Previous](https://docs.chonkie.ai/oss/embeddings/model2vec-embeddings) [GeminiEmbeddings\ \ Next](https://docs.chonkie.ai/oss/embeddings/gemini-embeddings) ⌘I --- # Chonkie Documentation > Documentation Index > ------------------- > > Fetch the complete documentation index at: [/llms.txt](https://docs.chonkie.ai/llms.txt) > > Use this file to discover all available pages before exploring further. [Skip to main content](https://docs.chonkie.ai/oss/embeddings/model2vec-embeddings#content-area) Embeddings are handled by the `Model2VecEmbeddings` class, which is a wrapper around the `model2vec` library. [​](https://docs.chonkie.ai/oss/embeddings/model2vec-embeddings#installation) Installation --------------------------------------------------------------------------------------------- Embeddings require the `model2vec` library. See the [Installation Guide](https://docs.chonkie.ai/oss/installation) for more information. [​](https://docs.chonkie.ai/oss/embeddings/model2vec-embeddings#usage) Usage ------------------------------------------------------------------------------- from chonkie import Model2VecEmbeddings embeddings = Model2VecEmbeddings() Was this page helpful? YesNo [AzureOpenAIEmbeddings\ \ Previous](https://docs.chonkie.ai/oss/embeddings/azure-embeddings) [JinaEmbeddings\ \ Next](https://docs.chonkie.ai/oss/embeddings/jina-embeddings) ⌘I --- # Chonkie Documentation > Documentation Index > ------------------- > > Fetch the complete documentation index at: [/llms.txt](https://docs.chonkie.ai/llms.txt) > > Use this file to discover all available pages before exploring further. [Skip to main content](https://docs.chonkie.ai/oss/embeddings/cohere-embeddings#content-area) Embeddings are handled by the `CohereEmbeddings` class, which is a wrapper around the Cohere API. [​](https://docs.chonkie.ai/oss/embeddings/cohere-embeddings#installation) Installation ------------------------------------------------------------------------------------------ Embeddings require the `cohere` library. See the [Installation Guide](https://docs.chonkie.ai/oss/installation) for more information. [​](https://docs.chonkie.ai/oss/embeddings/cohere-embeddings#usage) Usage ---------------------------------------------------------------------------- from chonkie import CohereEmbeddings # Initialize Cohere embeddings embeddings = CohereEmbeddings() # Specify model and API key embeddings = CohereEmbeddings(model="embed-english-light-v3.0", api_key="YOUR_API_KEY") [​](https://docs.chonkie.ai/oss/embeddings/cohere-embeddings#example) Example -------------------------------------------------------------------------------- embeddings = CohereEmbeddings() vectors = embeddings.embed("your text here") # Or you can vectors = embeddings.embed_batch(["text1", "text2"]) Was this page helpful? YesNo [AutoEmbeddings\ \ Previous](https://docs.chonkie.ai/oss/embeddings/auto-embeddings) [SentenceTransformerEmbeddings\ \ Next](https://docs.chonkie.ai/oss/embeddings/sentence-transformer-embeddings) ⌘I --- # Chonkie Documentation > Documentation Index > ------------------- > > Fetch the complete documentation index at: [/llms.txt](https://docs.chonkie.ai/llms.txt) > > Use this file to discover all available pages before exploring further. [Skip to main content](https://docs.chonkie.ai/oss/embeddings/azure-embeddings#content-area) Embeddings are handled by the `AzureOpenAIEmbeddings` class, which wraps the Azure OpenAI service. [​](https://docs.chonkie.ai/oss/embeddings/azure-embeddings#installation) Installation ----------------------------------------------------------------------------------------- Embeddings require the `openai`, `azure-identity`, `numpy`, and `tiktoken` libraries. See the [Installation Guide](https://docs.chonkie.ai/oss/installation) for more information. [​](https://docs.chonkie.ai/oss/embeddings/azure-embeddings#usage) Usage --------------------------------------------------------------------------- from chonkie import AzureOpenAIEmbeddings # Initialize Azure OpenAI embeddings embeddings = AzureOpenAIEmbeddings( azure_endpoint="https://.openai.azure.com/", azure_api_key="", model="text-embedding-3-small", # or other supported model deployment="" ) # Single embedding emb = embeddings.embed("your text here") # Batch embedding embs = embeddings.embed_batch(["text1", "text2"]) [​](https://docs.chonkie.ai/oss/embeddings/azure-embeddings#example) Example ------------------------------------------------------------------------------- embeddings = AzureOpenAIEmbeddings( azure_endpoint="https://my-resource.openai.azure.com/", azure_api_key="my-key", model="text-embedding-3-small", deployment="embedding-deployment" ) Was this page helpful? YesNo [OpenAIEmbeddings\ \ Previous](https://docs.chonkie.ai/oss/embeddings/openai-embeddings) [Model2VecEmbeddings\ \ Next](https://docs.chonkie.ai/oss/embeddings/model2vec-embeddings) ⌘I --- # Chonkie Documentation > Documentation Index > ------------------- > > Fetch the complete documentation index at: [/llms.txt](https://docs.chonkie.ai/llms.txt) > > Use this file to discover all available pages before exploring further. [Skip to main content](https://docs.chonkie.ai/oss/embeddings/overview#content-area) Chonkie provides a variety of embeddings handlers to handle different embedding models in a consistent manner. Embeddings handlers are used in conjunction with chunkers to embed chunks of text. Only few chunkers require embeddings, see the [Chunkers Overview](https://docs.chonkie.ai/oss/chunkers/overview) for more information. [​](https://docs.chonkie.ai/oss/embeddings/overview#installation) Installation --------------------------------------------------------------------------------- Embeddings handlers require additional dependencies. See the [Installation Guide](https://docs.chonkie.ai/oss/installation) for more information. By default, Chonkie `semantic` installation includes `Model2VecEmbeddings`, which is the current default embeddings handler [​](https://docs.chonkie.ai/oss/embeddings/overview#available-embeddings) Available Embeddings ------------------------------------------------------------------------------------------------- AutoEmbeddings -------------- Automatically select the best embeddings handler for your use case. CohereEmbeddings ---------------- Embed text using Cohere embeddings (requires `cohere`). SentenceTransformerEmbeddings ----------------------------- Embed text using SentenceTransformer embeddings (requires `sentence-transformers`). OpenAIEmbeddings ---------------- Embed text using OpenAI embeddings (requires `openai`). Model2VecEmbeddings ------------------- Embed text using Model2Vec embeddings (requires `model2vec`). GeminiEmbeddings ---------------- Embed text using Google Gemini embeddings (requires `google-genai`). JinaEmbeddings -------------- Embed text using JinaAI embeddings (requires `jina`). AzureOpenAIEmbeddings --------------------- Embed text using Azure OpenAI embeddings (requires `openai`, `azure-identity`). VoyageAIEmbeddings ------------------ Embed text using VoyageAI embeddings (requires `voyageai`). [​](https://docs.chonkie.ai/oss/embeddings/overview#common-interface) Common Interface ----------------------------------------------------------------------------------------- All embeddings handlers share a consistent interface: # Single text embedding emb = embeddings.embed(text) # Batch processing emb = embeddings.embed_batch(texts) # Direct calling emb = embeddings(text) # or embeddings([text1, text2]) Was this page helpful? YesNo [Token Chunker\ \ Previous](https://docs.chonkie.ai/oss/chunkers/token-chunker) [AutoEmbeddings\ \ Next](https://docs.chonkie.ai/oss/embeddings/auto-embeddings) ⌘I --- # Chonkie Documentation > Documentation Index > ------------------- > > Fetch the complete documentation index at: [/llms.txt](https://docs.chonkie.ai/llms.txt) > > Use this file to discover all available pages before exploring further. [Skip to main content](https://docs.chonkie.ai/oss/embeddings/gemini-embeddings#content-area) Embeddings are handled by the `GeminiEmbeddings` class, which is a wrapper around the Google GenAI API. [​](https://docs.chonkie.ai/oss/embeddings/gemini-embeddings#installation) Installation ------------------------------------------------------------------------------------------ Gemini embeddings require the `google-genai` and `numpy` libraries. See the [Installation Guide](https://docs.chonkie.ai/oss/installation) for more information. pip install "chonkie[gemini]" [​](https://docs.chonkie.ai/oss/embeddings/gemini-embeddings#usage) Usage ---------------------------------------------------------------------------- from chonkie import GeminiEmbeddings # Initialize Gemini embeddings embeddings = GeminiEmbeddings( model="gemini-embedding-exp-03-07", # Optional: specify model api_key="YOUR_GEMINI_API_KEY", # Optional: or set GEMINI_API_KEY env var task_type="SEMANTIC_SIMILARITY", # Optional: task type ) # Embed a single text vector = embeddings.embed("Your text here") [​](https://docs.chonkie.ai/oss/embeddings/gemini-embeddings#example) Example -------------------------------------------------------------------------------- texts = ["Hello world", "Goodbye world"] embeddings = GeminiEmbeddings() vectors = embeddings.embed_batch(texts) Was this page helpful? YesNo [JinaEmbeddings\ \ Previous](https://docs.chonkie.ai/oss/embeddings/jina-embeddings) [VoyageAIEmbeddings\ \ Next](https://docs.chonkie.ai/oss/embeddings/voyageai-embeddings) ⌘I --- # Chonkie Documentation > Documentation Index > ------------------- > > Fetch the complete documentation index at: [/llms.txt](https://docs.chonkie.ai/llms.txt) > > Use this file to discover all available pages before exploring further. [Skip to main content](https://docs.chonkie.ai/oss/embeddings/voyageai-embeddings#content-area) Embeddings are handled by the `VoyageAIEmbeddings` class, which is a wrapper around the VoyageAI API. [​](https://docs.chonkie.ai/oss/embeddings/voyageai-embeddings#installation) Installation -------------------------------------------------------------------------------------------- Embeddings require the `voyageai`, `numpy`, and `tokenizers` libraries. See the [Installation Guide](https://docs.chonkie.ai/oss/installation) for more information. pip install "chonkie[voyageai]" [​](https://docs.chonkie.ai/oss/embeddings/voyageai-embeddings#usage) Usage ------------------------------------------------------------------------------ from chonkie import VoyageAIEmbeddings # Initialize VoyageAI embeddings embeddings = VoyageAIEmbeddings( model="voyage-3-large", # Optional: specify model api_key="YOUR_VOYAGE_API_KEY", # Optional: or set VOYAGE_API_KEY env var output_dimension=1024, # Optional: set output dimension batch_size=64 # Optional: set batch size ) # Embed a single text vector = embeddings.embed("Your text here") # Embed a batch of texts vectors = embeddings.embed_batch(["Text 1", "Text 2"]) [​](https://docs.chonkie.ai/oss/embeddings/voyageai-embeddings#example) Example ---------------------------------------------------------------------------------- embeddings = VoyageAIEmbeddings() vectors = embeddings.embed("your text here") # or you can vectors = embeddings.embed_batch(["text1", "text2"]) Was this page helpful? YesNo [GeminiEmbeddings\ \ Previous](https://docs.chonkie.ai/oss/embeddings/gemini-embeddings) [Create your own custom embeddings handler\ \ Next](https://docs.chonkie.ai/oss/embeddings/custom-embeddings) ⌘I --- # Chonkie Documentation > Documentation Index > ------------------- > > Fetch the complete documentation index at: [/llms.txt](https://docs.chonkie.ai/llms.txt) > > Use this file to discover all available pages before exploring further. [Skip to main content](https://docs.chonkie.ai/oss/fetchers/overview#content-area) Fetchers connect different data sources to Chonkie’s pipeline system, enabling seamless data ingestion from various sources. [​](https://docs.chonkie.ai/oss/fetchers/overview#what-are-fetchers) What are Fetchers? ------------------------------------------------------------------------------------------ Fetchers are the first step in the CHOMP pipeline (CHef -> CHunker -> Refinery -> Porter/Handshake). They retrieve data from different sources and pass it to the next pipeline stage for processing. Fetchers make it easy to: * Load files from local storage * Fetch documents from cloud storage (coming soon) * Retrieve data from databases (coming soon) * Connect to APIs and web sources (coming soon) [​](https://docs.chonkie.ai/oss/fetchers/overview#installation) Installation ------------------------------------------------------------------------------- Fetchers are included with the base Chonkie installation: pip install chonkie [​](https://docs.chonkie.ai/oss/fetchers/overview#using-fetchers-in-pipelines) Using Fetchers in Pipelines ------------------------------------------------------------------------------------------------------------- Fetchers integrate seamlessly with the Pipeline API: from chonkie.pipeline import Pipeline # Single file doc = (Pipeline() .fetch_from("file", path="document.txt") .process_with("text") .chunk_with("recursive", chunk_size=512) .run()) # Directory with multiple files docs = (Pipeline() .fetch_from("file", dir="./docs", ext=[".txt", ".md"]) .process_with("text") .chunk_with("recursive", chunk_size=512) .run()) [​](https://docs.chonkie.ai/oss/fetchers/overview#available-fetchers) Available Fetchers ------------------------------------------------------------------------------------------- FileFetcher ----------- Fetch files from local filesystem - single files or entire directories. More fetchers are coming soon! We’re working on cloud storage, database, and API fetchers. Was this page helpful? YesNo [MistralOCR\ \ Previous](https://docs.chonkie.ai/oss/chefs/mistral-ocr) [FileFetcher\ \ Next](https://docs.chonkie.ai/oss/fetchers/file-fetcher) ⌘I --- # Chonkie Documentation > Documentation Index > ------------------- > > Fetch the complete documentation index at: [/llms.txt](https://docs.chonkie.ai/llms.txt) > > Use this file to discover all available pages before exploring further. [Skip to main content](https://docs.chonkie.ai/oss/utils/hubbie#content-area) Hubbie is a utility for accessing Chonkie’s saved recipes. Recipes are pre-defined chunking rules for different languages and document types. When initializing a chunker with `from_recipe`, you can pass in the recipe name and language to use the recipe. [​](https://docs.chonkie.ai/oss/utils/hubbie#installation) Installation -------------------------------------------------------------------------- To make use of Hubbie, you’ll need to install the `hub` optional install. pip install "chonkie[hub]" [​](https://docs.chonkie.ai/oss/utils/hubbie#usage) Usage ------------------------------------------------------------ from chonkie import RecursiveChunker # Initialize the recursive chunker with the recipe name and language chunker = RecursiveChunker.from_recipe("markdown", lang="en") # Chunk the text text = ... # Your text string # CHONK! chunks = chunker(text) [​](https://docs.chonkie.ai/oss/utils/hubbie#recipes) Recipes ---------------------------------------------------------------- You can access Chonkie’s saved recipes on [Hugging Face](https://huggingface.co/datasets/chonkie-ai/recipes) . Was this page helpful? YesNo [Visualizer\ \ Previous](https://docs.chonkie.ai/oss/utils/visualizer) [Logging\ \ Next](https://docs.chonkie.ai/oss/utils/logging) ⌘I --- # Chonkie Documentation > Documentation Index > ------------------- > > Fetch the complete documentation index at: [/llms.txt](https://docs.chonkie.ai/llms.txt) > > Use this file to discover all available pages before exploring further. [Skip to main content](https://docs.chonkie.ai/oss/experimental/overview#content-area) [​](https://docs.chonkie.ai/oss/experimental/overview#experimental-features) Experimental Features ===================================================================================================== Welcome to Chonkie’s experimental features! This section contains advanced, cutting-edge functionality that’s currently in development and testing phases. **Experimental Notice**: Features in this section are experimental and may change significantly between versions. They are provided for early testing and feedback. Use with caution in production environments. [​](https://docs.chonkie.ai/oss/experimental/overview#what%E2%80%99s-experimental) What’s Experimental? ---------------------------------------------------------------------------------------------------------- Experimental features in Chonkie represent: * **Advanced algorithms** that are still being refined * **New chunking strategies** that may not be fully optimized * **Innovative approaches** to text processing that need real-world validation * **Features with evolving APIs** that may change based on user feedback * **CLI improvements** allowing for directory processing and pipeline execution from the terminal [​](https://docs.chonkie.ai/oss/experimental/overview#getting-started) Getting Started ----------------------------------------------------------------------------------------- To use experimental features, import them from the `chonkie.experimental` module: from chonkie.experimental import CodeChunker # Create an experimental chunker chunker = CodeChunker(language="python", chunk_size=2048) [​](https://docs.chonkie.ai/oss/experimental/overview#providing-feedback) Providing Feedback ----------------------------------------------------------------------------------------------- Your feedback is crucial for graduating experimental features to stable status. If you encounter issues or have suggestions: 1. **Open an issue** on our [GitHub repository](https://github.com/chonkie-inc/chonkie) 2. **Join our Discord** to discuss with the community 3. **Share your use cases** to help us understand real-world applications [​](https://docs.chonkie.ai/oss/experimental/overview#migration-to-stable) Migration to Stable ------------------------------------------------------------------------------------------------- When experimental features become stable, they will: * Move to the main Chonkie namespace * Receive API stability guarantees * Include comprehensive documentation and examples * Be covered by semantic versioning promises We recommend using experimental features in development and testing environments first, and carefully evaluating their performance before production use. Was this page helpful? YesNo [Logging\ \ Previous](https://docs.chonkie.ai/oss/utils/logging) [Code Chunker\ \ Next](https://docs.chonkie.ai/oss/experimental/code-chunker) ⌘I --- # Chonkie Documentation > Documentation Index > ------------------- > > Fetch the complete documentation index at: [/llms.txt](https://docs.chonkie.ai/llms.txt) > > Use this file to discover all available pages before exploring further. [Skip to main content](https://docs.chonkie.ai/oss/embeddings/custom-embeddings#content-area) Chonkie allows you to use your own embeddings handler by creating a child class of the `BaseEmbeddings` class, and implementing the necessary methods. It’s quite simple! [​](https://docs.chonkie.ai/oss/embeddings/custom-embeddings#example) Example -------------------------------------------------------------------------------- First, we create a child class of the `BaseEmbeddings` class, and implement the necessary methods. from chonkie.embeddings import BaseEmbeddings class CustomEmbeddings(BaseEmbeddings): @property def dimension(self) -> int: ... def embed(self, text: str) -> "np.ndarray": ... def embed_batch(self, texts: list[str]) -> list["np.ndarray"]: ... def count_tokens(self, text: str) -> int: ... def count_tokens_batch(self, texts: list[str]) -> list[int]: ... def get_tokenizer(self): ... @classmethod def is_available(cls) -> bool: ... def __repr__(self) -> str: ... At this point, we have a custom embeddings handler, we can use it like this: embeddings = CustomEmbeddings() But let’s say we want to use this together with the `AutoEmbeddings` class, for the sake of convenience. We can do this by registering it with the `EmbeddingsRegistry`. from chonkie.embeddings import EmbeddingsRegistry # Register with the embeddings registry EmbeddingsRegistry.register( "custom", CustomEmbeddings, pattern=r"^custom/|^model-name", valid_types=["CustomEmbeddings"] ) Now we can use our custom embeddings handler with the `AutoEmbeddings` class. embeddings = AutoEmbeddings.get_embeddings("custom/my-custom-embeddings") Finally, we can use our custom embeddings handler in the same way we would use any other embeddings handler. chunker = SemanticChunker(embedding_model=embeddings, threshold=0.7) chunks = chunker(text) Was this page helpful? YesNo [VoyageAIEmbeddings\ \ Previous](https://docs.chonkie.ai/oss/embeddings/voyageai-embeddings) [Refinery Overview\ \ Next](https://docs.chonkie.ai/oss/refinery/overview) ⌘I --- # Chonkie Documentation > Documentation Index > ------------------- > > Fetch the complete documentation index at: [/llms.txt](https://docs.chonkie.ai/llms.txt) > > Use this file to discover all available pages before exploring further. [Skip to main content](https://docs.chonkie.ai/oss/embeddings/auto-embeddings#content-area) AutoEmbeddings is a class that automatically selects the appropriate embeddings handler for you, based on the model name you provide. [​](https://docs.chonkie.ai/oss/embeddings/auto-embeddings#installation) Installation ---------------------------------------------------------------------------------------- Embeddings require the appropriate library to be installed. See the [Installation Guide](https://docs.chonkie.ai/oss/installation) for more information. [​](https://docs.chonkie.ai/oss/embeddings/auto-embeddings#usage) Usage -------------------------------------------------------------------------- Load the embeddings handler for the model you want to use. from chonkie import AutoEmbeddings # Get the embeddings handler for SentenceTransformer embeddings = AutoEmbeddings.get_embeddings("all-MiniLM-L6-v2") # Get the embeddings handler for OpenAI embeddings = AutoEmbeddings.get_embeddings("text-embedding-3-large") # Get the embeddings handler for Model2Vec embeddings = AutoEmbeddings.get_embeddings("minishlab/potion-base-32M") After loading the embeddings handler, you can use it in the same way you would use any other embeddings handler. from chonkie import SemanticChunker chunker = SemanticChunker(embedding_model=embeddings, threshold=0.7) # Chunk the text chunks = chunker(text) SemanticChunkers interally call upon the AutoEmbeddings class to get the embeddings handler. So you can directly pass in a string to the `embeddings` parameter as well, as long as it matches one of the models supported by AutoEmbeddings, and its dependencies are installed. [​](https://docs.chonkie.ai/oss/embeddings/auto-embeddings#method-get_embeddings) Method: `get_embeddings` ------------------------------------------------------------------------------------------------------------- The `get_embeddings` method is a factory method that returns an instance of the appropriate embeddings handler. [​](https://docs.chonkie.ai/oss/embeddings/auto-embeddings#param-model-name) model\_name str The name of the embeddings model to use. Was this page helpful? YesNo [Embeddings Overview\ \ Previous](https://docs.chonkie.ai/oss/embeddings/overview) [CohereEmbeddings\ \ Next](https://docs.chonkie.ai/oss/embeddings/cohere-embeddings) ⌘I --- # Chonkie Documentation > Documentation Index > ------------------- > > Fetch the complete documentation index at: [/llms.txt](https://docs.chonkie.ai/llms.txt) > > Use this file to discover all available pages before exploring further. [Skip to main content](https://docs.chonkie.ai/oss/utils/visualizer#content-area) The `Visualizer` helps you visualize your chunks properly and compare different chunkers and settings with ease, either on the terminal or via HTML. ![Visualizer Example](https://mintcdn.com/chonkie/z5jcWEi822NzVea9/assets/viz/viz-terminal.png?w=2500&fit=max&auto=format&n=z5jcWEi822NzVea9&q=85&s=77787d8c1fec6ddf87f65ef69fe47cea) [​](https://docs.chonkie.ai/oss/utils/visualizer#installation) Installation ------------------------------------------------------------------------------ To make use of the `Visualizer`, you’ll need to install the `viz` optional install. pip install "chonkie[viz]" [​](https://docs.chonkie.ai/oss/utils/visualizer#usage) Usage ---------------------------------------------------------------- To use the `Visualizer`, you’ll need to import it from `chonkie.utils`. # Import the Visualizer from chonkie import Visualizer # Initialize the Visualizer viz = Visualizer() The `Visualizer` has two main methods: `print` and `save`. The `print` method will print the chunks to the terminal. It accepts a list of `Chunk` objects or plain strings. viz.print(chunks) # Or you can directly call the Visualizer object viz(chunks) # You can also pass a list of plain strings viz(["First text segment", "Second text segment", "Third text segment"]) The `save` method will save the chunks to a HTML file. Like `print`, it accepts `Chunk` objects or strings. viz.save("chonkie.html", chunks) ![Visualizer Example](https://mintcdn.com/chonkie/z5jcWEi822NzVea9/assets/viz/viz-html.png?w=2500&fit=max&auto=format&n=z5jcWEi822NzVea9&q=85&s=20de3ea375c2320ca1180a9e93a2147b) Was this page helpful? YesNo [DatasetsPorter\ \ Previous](https://docs.chonkie.ai/oss/porters/datasets-porter) [Hubbie\ \ Next](https://docs.chonkie.ai/oss/utils/hubbie) ⌘I --- # Chonkie Documentation > Documentation Index > ------------------- > > Fetch the complete documentation index at: [/llms.txt](https://docs.chonkie.ai/llms.txt) > > Use this file to discover all available pages before exploring further. [Skip to main content](https://docs.chonkie.ai/oss/handshakes/overview#content-area) Handshakes allow you to easily connect Chonkie to a vector database of your choice. Embed your chunks and write them to your database in just a few lines of code. ![https://mintcdn.com/chonkie/E9etFi5RI4FiegDM/assets/handshakes/chroma-icon.svg?fit=max&auto=format&n=E9etFi5RI4FiegDM&q=85&s=ed44eaf8fbf93039bbaedc08ea434219](https://mintcdn.com/chonkie/E9etFi5RI4FiegDM/assets/handshakes/chroma-icon.svg?fit=max&auto=format&n=E9etFi5RI4FiegDM&q=85&s=ed44eaf8fbf93039bbaedc08ea434219) ChromaDB Handshake ------------------ Connect Chonkie to your ephemeral or persistent ChromaDB instance ![https://mintcdn.com/chonkie/Qk-SRm0DEtc-6k0k/assets/handshakes/elastic%20search.png?fit=max&auto=format&n=Qk-SRm0DEtc-6k0k&q=85&s=b06c2f2950d1c9b08681bff2020f1740](https://mintcdn.com/chonkie/Qk-SRm0DEtc-6k0k/assets/handshakes/elastic%20search.png?fit=max&auto=format&n=Qk-SRm0DEtc-6k0k&q=85&s=b06c2f2950d1c9b08681bff2020f1740) Elasticsearch Handshake ----------------------- Connect Chonkie to your Elasticsearch index ![https://mintcdn.com/chonkie/u4KTZ1_nKVBf-8Xj/assets/handshakes/lancedb-logo.png?fit=max&auto=format&n=u4KTZ1_nKVBf-8Xj&q=85&s=8a7f3663affad67fa583c7e775e59902](https://mintcdn.com/chonkie/u4KTZ1_nKVBf-8Xj/assets/handshakes/lancedb-logo.png?fit=max&auto=format&n=u4KTZ1_nKVBf-8Xj&q=85&s=8a7f3663affad67fa583c7e775e59902) LanceDB Handshake ----------------- Connect Chonkie to your local or cloud LanceDB table ![https://mintcdn.com/chonkie/Qk-SRm0DEtc-6k0k/assets/handshakes/milvus.png?fit=max&auto=format&n=Qk-SRm0DEtc-6k0k&q=85&s=4f0303108cb5bc849f7fca1375bf5742](https://mintcdn.com/chonkie/Qk-SRm0DEtc-6k0k/assets/handshakes/milvus.png?fit=max&auto=format&n=Qk-SRm0DEtc-6k0k&q=85&s=4f0303108cb5bc849f7fca1375bf5742) Milvus Handshake ---------------- Connect Chonkie to your Milvus collection ![https://mintcdn.com/chonkie/E9etFi5RI4FiegDM/assets/handshakes/mongodb.svg?fit=max&auto=format&n=E9etFi5RI4FiegDM&q=85&s=b8fc79a35d32529782c560385446bc70](https://mintcdn.com/chonkie/E9etFi5RI4FiegDM/assets/handshakes/mongodb.svg?fit=max&auto=format&n=E9etFi5RI4FiegDM&q=85&s=b8fc79a35d32529782c560385446bc70) MongoDB Handshake ----------------- Connect Chonkie to your MongoDB collection ![https://mintcdn.com/chonkie/z5jcWEi822NzVea9/assets/handshakes/pgvector.png?fit=max&auto=format&n=z5jcWEi822NzVea9&q=85&s=97db6878144bf5375f2522e95b0b675a](https://mintcdn.com/chonkie/z5jcWEi822NzVea9/assets/handshakes/pgvector.png?fit=max&auto=format&n=z5jcWEi822NzVea9&q=85&s=97db6878144bf5375f2522e95b0b675a) Pgvector Handshake ------------------ Connect Chonkie to your Pgvector database ![https://mintcdn.com/chonkie/z5jcWEi822NzVea9/assets/handshakes/pinecone.png?fit=max&auto=format&n=z5jcWEi822NzVea9&q=85&s=4dc1f12d40712725e51eb2386366fcbb](https://mintcdn.com/chonkie/z5jcWEi822NzVea9/assets/handshakes/pinecone.png?fit=max&auto=format&n=z5jcWEi822NzVea9&q=85&s=4dc1f12d40712725e51eb2386366fcbb) Pinecone Handshake ------------------ Connect Chonkie to your Pinecone index ![https://mintcdn.com/chonkie/E9etFi5RI4FiegDM/assets/handshakes/qdrant-logo.svg?fit=max&auto=format&n=E9etFi5RI4FiegDM&q=85&s=11941c5b8a7a928f6718dabe5a18c655](https://mintcdn.com/chonkie/E9etFi5RI4FiegDM/assets/handshakes/qdrant-logo.svg?fit=max&auto=format&n=E9etFi5RI4FiegDM&q=85&s=11941c5b8a7a928f6718dabe5a18c655) Qdrant Handshake ---------------- Connect Chonkie to your Qdrant database ![https://mintcdn.com/chonkie/E9etFi5RI4FiegDM/assets/handshakes/turbopuffer.svg?fit=max&auto=format&n=E9etFi5RI4FiegDM&q=85&s=376dff092804d8ca045e58224207f555](https://mintcdn.com/chonkie/E9etFi5RI4FiegDM/assets/handshakes/turbopuffer.svg?fit=max&auto=format&n=E9etFi5RI4FiegDM&q=85&s=376dff092804d8ca045e58224207f555) Turbopuffer Handshake --------------------- Connect Chonkie to your Turbopuffer database ![https://mintcdn.com/chonkie/z5jcWEi822NzVea9/assets/handshakes/weaviate.png?fit=max&auto=format&n=z5jcWEi822NzVea9&q=85&s=bb2a61062045e2d1c6a19d4695bdb28f](https://mintcdn.com/chonkie/z5jcWEi822NzVea9/assets/handshakes/weaviate.png?fit=max&auto=format&n=z5jcWEi822NzVea9&q=85&s=bb2a61062045e2d1c6a19d4695bdb28f) Weaviate Handshake ------------------ Connect Chonkie to your Weaviate database Was this page helpful? YesNo [Embeddings Refinery\ \ Previous](https://docs.chonkie.ai/oss/refinery/embeddings-refinery) [Chroma Handshake\ \ Next](https://docs.chonkie.ai/oss/handshakes/chroma-handshake) ⌘I --- # Chonkie Documentation > Documentation Index > ------------------- > > Fetch the complete documentation index at: [/llms.txt](https://docs.chonkie.ai/llms.txt) > > Use this file to discover all available pages before exploring further. [Skip to main content](https://docs.chonkie.ai/oss/porters/json-porter#content-area) Port your chunks to a JSON file with the `JSONPorter`. This is useful for exporting your chunked data for use in other applications or for archiving. [​](https://docs.chonkie.ai/oss/porters/json-porter#initialization) Initialization ------------------------------------------------------------------------------------- from chonkie import JSONPorter from chonkie.types.base import Chunk chunks = [\ Chunk(\ id="chunk1",\ text="This is the first chunk.",\ metadata={"source": "document1.txt"}\ ),\ Chunk(\ id="chunk2",\ text="This is the second chunk.",\ metadata={"source": "document2.txt"}\ )\ ] porter = JSONPorter() porter.export(chunks) Was this page helpful? YesNo [Porters Overview\ \ Previous](https://docs.chonkie.ai/oss/porters/overview) [DatasetsPorter\ \ Next](https://docs.chonkie.ai/oss/porters/datasets-porter) ⌘I --- # Chonkie Documentation > Documentation Index > ------------------- > > Fetch the complete documentation index at: [/llms.txt](https://docs.chonkie.ai/llms.txt) > > Use this file to discover all available pages before exploring further. [Skip to main content](https://docs.chonkie.ai/oss/handshakes/milvus-handshake#content-area) The `MilvusHandshake` class provides seamless integration between Chonkie’s chunking system and Milvus, a powerful, open-source vector database. Embed and store your Chonkie chunks in a Milvus collection, with automatic schema and index creation, without ever leaving the Chonkie SDK. [​](https://docs.chonkie.ai/oss/handshakes/milvus-handshake#installation) Installation ----------------------------------------------------------------------------------------- Before using the Milvus handshake, make sure to install the required dependencies: pip install chonkie[milvus] [​](https://docs.chonkie.ai/oss/handshakes/milvus-handshake#basic-usage) Basic Usage --------------------------------------------------------------------------------------- ### [​](https://docs.chonkie.ai/oss/handshakes/milvus-handshake#initialization) Initialization Initialize for a Local Instance Initialize using a URI from chonkie import MilvusHandshake # Connects to Milvus at http://localhost:19530 by default handshake = MilvusHandshake() ### [​](https://docs.chonkie.ai/oss/handshakes/milvus-handshake#parameters) Parameters [​](https://docs.chonkie.ai/oss/handshakes/milvus-handshake#param-collection-name) collection\_name Union\[str, Literal\['random'\]\] default:"random" The name of the Milvus collection to use. If “random”, a unique name is generated. [​](https://docs.chonkie.ai/oss/handshakes/milvus-handshake#param-embedding-model) embedding\_model Union\[str, BaseEmbeddings\] default:"minishlab/potion-retrieval-32M" The embedding model to use for creating vectors. [​](https://docs.chonkie.ai/oss/handshakes/milvus-handshake#param-uri) uri Optional\[str\] default:"None" The full URI to connect to Milvus. This is the preferred method for specifying connection details. [​](https://docs.chonkie.ai/oss/handshakes/milvus-handshake#param-host) host str default:"localhost" The host of the Milvus instance. Used if `uri` is not provided. [​](https://docs.chonkie.ai/oss/handshakes/milvus-handshake#param-port) port str default:"19530" The port of the Milvus instance. Used if `uri` is not provided. [​](https://docs.chonkie.ai/oss/handshakes/milvus-handshake#param-alias) alias str default:"default" The connection alias to use for this Milvus connection. ### [​](https://docs.chonkie.ai/oss/handshakes/milvus-handshake#writing-chunks-to-milvus) Writing Chunks to Milvus from chonkie import MilvusHandshake, SentenceChunker # Initialize the handshake for your deployment handshake = MilvusHandshake( uri="http://localhost:19530", collection_name="my_documents", ) # Create some chunks chunker = SentenceChunker() chunks = chunker.chunk("Milvus stores data in collections. Chonkie makes ingestion easy!") # Write chunks to the Milvus collection handshake.write(chunks) ### [​](https://docs.chonkie.ai/oss/handshakes/milvus-handshake#searching-chunks-in-milvus) Searching Chunks in Milvus You can retrieve the most similar chunks from your Milvus collection using the `search` method. Search using a Text Query Search using an Embedding Vector from chonkie import MilvusHandshake # Initialize the handshake to connect to your collection handshake = MilvusHandshake( uri="http://localhost:19530", collection_name="my_documents", ) results = handshake.search(query="easy data ingestion", limit=2) Was this page helpful? YesNo [LanceDB Handshake\ \ Previous](https://docs.chonkie.ai/oss/handshakes/lancedb-handshake) [MongoDB Handshake\ \ Next](https://docs.chonkie.ai/oss/handshakes/mongodb-handshake) ⌘I --- # Chonkie Documentation > Documentation Index > ------------------- > > Fetch the complete documentation index at: [/llms.txt](https://docs.chonkie.ai/llms.txt) > > Use this file to discover all available pages before exploring further. [Skip to main content](https://docs.chonkie.ai/oss/utils/logging#content-area) By default, Chonkie logs warnings and errors. You can control what gets logged using the `CHONKIE_LOG` environment variable or configure it programmatically. [​](https://docs.chonkie.ai/oss/utils/logging#quick-setup) Quick Setup ------------------------------------------------------------------------- Set the `CHONKIE_LOG` environment variable before importing chonkie: export CHONKIE_LOG=debug # Show everything export CHONKIE_LOG=info # Show info, warnings, and errors export CHONKIE_LOG=warning # Show warnings and errors (default) export CHONKIE_LOG=error # Show only errors export CHONKIE_LOG=off # Disable all logging [​](https://docs.chonkie.ai/oss/utils/logging#programmatic-configuration) Programmatic Configuration ------------------------------------------------------------------------------------------------------- Configure logging from your Python code: import chonkie # Enable debug logging chonkie.logger.configure("debug") # Disable logging completely chonkie.logger.configure("off") # Back to warnings and errors chonkie.logger.configure("warning") [​](https://docs.chonkie.ai/oss/utils/logging#log-levels) Log Levels ----------------------------------------------------------------------- * `debug` - Everything (chunker initialization, text processing, chunk counts) * `info` - High-level operations (chunk creation, file operations) * `warning` - Potential issues (default) * `error` - Only errors * `off` - Silent [​](https://docs.chonkie.ai/oss/utils/logging#custom-format) Custom Format ----------------------------------------------------------------------------- Change the log format to suit your needs: * Default * Simple * Minimal * Detailed * Structured import chonkie chonkie.logger.configure("info") Output: 2025-11-13 14:18:58,714 | INFO | chonkie.chunker.token:chunk:143 - Created 5 chunks import chonkie chonkie.logger.configure("info", format="%(levelname)s - %(message)s") Output: INFO - Created 5 chunks import chonkie chonkie.logger.configure("info", format="%(message)s") Output: Created 5 chunks import chonkie chonkie.logger.configure("info", format="[%(name)s] %(levelname)s: %(message)s") Output: [chonkie.chunker.token] INFO: Created 5 chunks import chonkie import logging # Custom formatter that includes extra fields class StructuredFormatter(logging.Formatter): def format(self, record): msg = super().format(record) # Add any extra fields extras = [] for key in ['chunk_count', 'tokens']: if hasattr(record, key): extras.append(f"{key}={getattr(record, key)}") if extras: msg += f" [{', '.join(extras)}]" return msg # Apply custom formatter logger = logging.getLogger("chonkie") handler = logging.StreamHandler() handler.setFormatter(StructuredFormatter("%(levelname)s - %(message)s")) logger.addHandler(handler) logger.setLevel(logging.INFO) Output: INFO - Created chunks [chunk_count=5, tokens=128] That’s it. Logging is simple and stays out of your way. Was this page helpful? YesNo [Hubbie\ \ Previous](https://docs.chonkie.ai/oss/utils/hubbie) [Overview\ \ Next](https://docs.chonkie.ai/oss/experimental/overview) ⌘I --- # Chonkie Documentation > Documentation Index > ------------------- > > Fetch the complete documentation index at: [/llms.txt](https://docs.chonkie.ai/llms.txt) > > Use this file to discover all available pages before exploring further. [Skip to main content](https://docs.chonkie.ai/oss/chunkers/teraflopai-chunker#content-area) The `TeraflopAIChunker` uses the [TeraflopAI](https://www.teraflopai.com/) Segmentation API to split text into semantically meaningful segments. It is especially useful for domain-specific segmentation such as legal documents. [​](https://docs.chonkie.ai/oss/chunkers/teraflopai-chunker#installation) Installation ----------------------------------------------------------------------------------------- TeraflopAI Chunker requires the `teraflopai` Python package: pip install "chonkie[teraflopai]" For general installation instructions, see the [Installation Guide](https://docs.chonkie.ai/oss/installation) . [​](https://docs.chonkie.ai/oss/chunkers/teraflopai-chunker#initialization) Initialization --------------------------------------------------------------------------------------------- using api\_key using custom\_url using external client from chonkie import TeraflopAIChunker # Using an API key (or set the TERAFLOPAI_API_KEY environment variable) chunker = TeraflopAIChunker(api_key="your_api_key_here") [​](https://docs.chonkie.ai/oss/chunkers/teraflopai-chunker#parameters) Parameters ------------------------------------------------------------------------------------- [​](https://docs.chonkie.ai/oss/chunkers/teraflopai-chunker#param-client) client Optional\[TeraflopAI\] default:"None" An existing TeraflopAI client instance. If provided, `url` and `api_key` are ignored. [​](https://docs.chonkie.ai/oss/chunkers/teraflopai-chunker#param-url) url str The URL for the TeraflopAI segmentation API endpoint. [​](https://docs.chonkie.ai/oss/chunkers/teraflopai-chunker#param-api-key) api\_key Optional\[str\] default:"None" The API key for authentication. If not provided, it will be read from the `TERAFLOPAI_API_KEY` environment variable. [​](https://docs.chonkie.ai/oss/chunkers/teraflopai-chunker#param-tokenizer) tokenizer Union\[str, TokenizerProtocol\] default:"character" The tokenizer used to compute token counts for returned chunks. [​](https://docs.chonkie.ai/oss/chunkers/teraflopai-chunker#usage) Usage --------------------------------------------------------------------------- ### [​](https://docs.chonkie.ai/oss/chunkers/teraflopai-chunker#single-text-chunking) Single Text Chunking text = """ Global warming refers to the long-term increase in Earth’s average surface temperature due to human activities, primarily the emission of greenhouse gases such as carbon dioxide and methane. These gases trap heat in the atmosphere, leading to significant changes in climate patterns across the globe. Scientists have observed rising temperatures, melting polar ice caps, and increasing sea levels, all of which pose serious risks to ecosystems and human societies. Extreme weather events such as hurricanes, droughts, and heatwaves are becoming more frequent and intense as a result of these changes. Governments and organizations around the world are working to reduce emissions, transition to renewable energy sources, and promote sustainable practices. However, global cooperation and immediate action are essential to mitigate the long-term impacts and protect future generations from the most severe consequences of climate change. Public awareness and individual responsibility also play a crucial role in addressing global warming. Simple actions like reducing energy consumption, minimizing waste, and supporting environmentally friendly initiatives can collectively make a meaningful difference in slowing down this global crisis. """ chunks = chunker.chunk(text) for chunk in chunks: print(f"Chunk text: {chunk.text}") print(f"Token count: {chunk.token_count}") print(f"Start index: {chunk.start_index}") print(f"End index: {chunk.end_index}") ### [​](https://docs.chonkie.ai/oss/chunkers/teraflopai-chunker#batch-chunking) Batch Chunking texts = [\ "First document to segment.",\ "Second document with more content to segment.",\ ] batch_results = chunker(texts) for i, chunks in enumerate(batch_results): print(f"Document {i}: {len(chunks)} chunks") ### [​](https://docs.chonkie.ai/oss/chunkers/teraflopai-chunker#using-with-environment-variable) Using with Environment Variable export TERAFLOPAI_API_KEY="your_api_key_here" from chonkie import TeraflopAIChunker # No need to pass api_key — it will be read from the environment chunker = TeraflopAIChunker() chunks = chunker.chunk("Your text here.") [​](https://docs.chonkie.ai/oss/chunkers/teraflopai-chunker#how-it-works) How It Works ----------------------------------------------------------------------------------------- 1. The text is sent to the TeraflopAI Segmentation API endpoint. 2. The API returns a list of text segments. 3. Each segment is converted into a Chonkie `Chunk` object with proper `start_index`, `end_index`, and `token_count` fields. The TeraflopAI Segmentation API performs the segmentation on the server side. This chunker requires an active internet connection and a valid API key. Was this page helpful? YesNo [Table Chunker\ \ Previous](https://docs.chonkie.ai/oss/chunkers/table-chunker) [Token Chunker\ \ Next](https://docs.chonkie.ai/oss/chunkers/token-chunker) ⌘I --- # Chonkie Documentation > Documentation Index > ------------------- > > Fetch the complete documentation index at: [/llms.txt](https://docs.chonkie.ai/llms.txt) > > Use this file to discover all available pages before exploring further. [Skip to main content](https://docs.chonkie.ai/oss/handshakes/chroma-handshake#content-area) The `ChromaHandshake` class provides seamless integration between Chonkie’s chunking system and ChromaDB, a popular vector database. Embed and store your Chonkie chunks in ChromaDB without ever leaving the Chonkie SDK. [​](https://docs.chonkie.ai/oss/handshakes/chroma-handshake#installation) Installation ----------------------------------------------------------------------------------------- Before using the Chroma handshake, make sure to install the required dependencies: pip install chonkie[chroma] [​](https://docs.chonkie.ai/oss/handshakes/chroma-handshake#basic-usage) Basic Usage --------------------------------------------------------------------------------------- ### [​](https://docs.chonkie.ai/oss/handshakes/chroma-handshake#initialization) Initialization from chonkie import ChromaHandshake # Initialize with default settings (in-memory ChromaDB) handshake = ChromaHandshake() # Or specify a persistent storage path handshake = ChromaHandshake(path="./chroma_db") # Or use an existing Chroma client import chromadb client = chromadb.Client() handshake = ChromaHandshake(client=client, collection_name="my_collection") ### [​](https://docs.chonkie.ai/oss/handshakes/chroma-handshake#writing-chunks-to-chromadb) Writing Chunks to ChromaDB from chonkie import ChromaHandshake, SemanticChunker handshake = ChromaHandshake() # Initializes a new Chroma client chunker = SemanticChunker() chunks = chunker("Chonkie is the best chonker ever!") handshake.write(chunks) ### [​](https://docs.chonkie.ai/oss/handshakes/chroma-handshake#searching-chunks-in-chromadb) Searching Chunks in ChromaDB You can retrieve the most similar chunks from your ChromaDB collection using the `search` method: search using a query search using embedding search using chonkie chunks from chonkie import ChromaHandshake # Initialize the handshake handshake = ChromaHandshake(collection_name="my_documents") results = handshake.search(query="best chonker", limit=2) for result in results: print(result["score"], result["text"]) [​](https://docs.chonkie.ai/oss/handshakes/chroma-handshake#parameters) Parameters ------------------------------------------------------------------------------------- [​](https://docs.chonkie.ai/oss/handshakes/chroma-handshake#param-client) client Optional\[chromadb.Client\] default:"None" Chroma client instance. If not provided, a new client will be created. [​](https://docs.chonkie.ai/oss/handshakes/chroma-handshake#param-collection-name) collection\_name Union\[str, Literal\['random'\]\] default:"random" Name of the collection to use. If “random”, a random name will be generated. [​](https://docs.chonkie.ai/oss/handshakes/chroma-handshake#param-embedding-model) embedding\_model Union\[str, BaseEmbeddings\] default:"minishlab/potion-retrieval-32M" Embedding model to use. [​](https://docs.chonkie.ai/oss/handshakes/chroma-handshake#param-path) path Optional\[str\] default:"None" If provided, creates a persistent Chroma client at the specified path. Was this page helpful? YesNo [Handshakes Overview\ \ Previous](https://docs.chonkie.ai/oss/handshakes/overview) [Elasticsearch Handshake\ \ Next](https://docs.chonkie.ai/oss/handshakes/elastic-handshake) ⌘I --- # Chonkie Documentation > Documentation Index > ------------------- > > Fetch the complete documentation index at: [/llms.txt](https://docs.chonkie.ai/llms.txt) > > Use this file to discover all available pages before exploring further. [Skip to main content](https://docs.chonkie.ai/oss/handshakes/lancedb-handshake#content-area) The `LanceDBHandshake` class provides seamless integration between Chonkie’s chunking system and LanceDB, a serverless vector database built on Apache Arrow. Embed and store your Chonkie chunks in LanceDB — locally or in the cloud — without ever leaving the Chonkie SDK. [​](https://docs.chonkie.ai/oss/handshakes/lancedb-handshake#installation) Installation ------------------------------------------------------------------------------------------ Before using the LanceDB handshake, make sure to install the required dependencies: pip install chonkie[lancedb] [​](https://docs.chonkie.ai/oss/handshakes/lancedb-handshake#basic-usage) Basic Usage ---------------------------------------------------------------------------------------- ### [​](https://docs.chonkie.ai/oss/handshakes/lancedb-handshake#initialization) Initialization in-memory (ephemeral) local persistent storage existing connection LanceDB Cloud from chonkie import LanceDBHandshake # Default: in-memory LanceDB, auto-generated table name handshake = LanceDBHandshake() ### [​](https://docs.chonkie.ai/oss/handshakes/lancedb-handshake#writing-chunks-to-lancedb) Writing Chunks to LanceDB from chonkie import LanceDBHandshake, SemanticChunker # Initialize the handshake handshake = LanceDBHandshake(uri="./my_lancedb", table_name="my_documents") # Create some chunks chunker = SemanticChunker() chunks = chunker("Chonkie loves to chonk your texts!") # Write chunks to LanceDB handshake.write(chunks) ### [​](https://docs.chonkie.ai/oss/handshakes/lancedb-handshake#searching-chunks-in-lancedb) Searching Chunks in LanceDB You can retrieve the most similar chunks from your LanceDB table using the `search` method: search using a query search using an embedding from chonkie import LanceDBHandshake handshake = LanceDBHandshake(uri="./my_lancedb", table_name="my_documents") results = handshake.search(query="chonk your texts", limit=5) for result in results: print(result["score"], result["text"]) [​](https://docs.chonkie.ai/oss/handshakes/lancedb-handshake#parameters) Parameters -------------------------------------------------------------------------------------- [​](https://docs.chonkie.ai/oss/handshakes/lancedb-handshake#param-connection) connection Optional\[lancedb.DBConnection\] default:"None" An existing LanceDB connection. If not provided, a new connection is created using `uri`. [​](https://docs.chonkie.ai/oss/handshakes/lancedb-handshake#param-uri) uri Union\[str, os.PathLike\] default:"memory://" URI of the LanceDB database. Use `"memory://"` for an ephemeral in-memory database, a local directory path for persistent storage, or a `db://` URI for LanceDB Cloud. [​](https://docs.chonkie.ai/oss/handshakes/lancedb-handshake#param-table-name) table\_name Union\[str, Literal\['random'\]\] default:"random" Name of the table to write chunks to. If `"random"`, a unique name is auto-generated. [​](https://docs.chonkie.ai/oss/handshakes/lancedb-handshake#param-embedding-model) embedding\_model Union\[str, BaseEmbeddings\] default:"minishlab/potion-retrieval-32M" Embedding model to use. Can be a model name string or a `BaseEmbeddings` instance. Was this page helpful? YesNo [Elasticsearch Handshake\ \ Previous](https://docs.chonkie.ai/oss/handshakes/elastic-handshake) [Milvus Handshake\ \ Next](https://docs.chonkie.ai/oss/handshakes/milvus-handshake) ⌘I --- # Chonkie Documentation > Documentation Index > ------------------- > > Fetch the complete documentation index at: [/llms.txt](https://docs.chonkie.ai/llms.txt) > > Use this file to discover all available pages before exploring further. [Skip to main content](https://docs.chonkie.ai/oss/chunkers/sdpm-chunker#content-area) **Deprecated as of v1.2.0**The SDPM (Semantic Double-Pass Merging) functionality has been integrated into the main `SemanticChunker`.**Recommended Migration:** # Old way (deprecated) from chonkie.legacy import SDPMChunker chunker = SDPMChunker(skip_window=1) # New way (recommended) from chonkie import SemanticChunker chunker = SemanticChunker(skip_window=1) The new SemanticChunker provides all SDPM capabilities plus additional improvements like Savitzky-Golay filtering for better boundary detection. The `SDPMChunker` extends semantic chunking by using a double-pass merging approach. It first groups content by semantic similarity, then merges similar groups within a skip window, allowing it to connect related content that may not be consecutive in the text. [​](https://docs.chonkie.ai/oss/chunkers/sdpm-chunker#why-use-the-new-semanticchunker-instead) Why Use the New SemanticChunker Instead? ------------------------------------------------------------------------------------------------------------------------------------------ The new `SemanticChunker` includes all SDPM functionality plus: * **Better performance**: Optimized C extensions for faster processing * **Smoother boundaries**: Savitzky-Golay filtering for noise reduction * **Cleaner API**: Simplified parameter names and improved defaults * **Active development**: Ongoing improvements and bug fixes [​](https://docs.chonkie.ai/oss/chunkers/sdpm-chunker#legacy-installation) Legacy Installation ------------------------------------------------------------------------------------------------- If you need to use the legacy version for compatibility: pip install "chonkie[semantic]" Then import from the legacy module: from chonkie.legacy import SDPMChunker [​](https://docs.chonkie.ai/oss/chunkers/sdpm-chunker#legacy-usage) Legacy Usage ----------------------------------------------------------------------------------- This documentation is preserved for users who need to maintain existing code using SDPMChunker. For new projects, please use the main [SemanticChunker](https://docs.chonkie.ai/oss/chunkers/semantic-chunker) . ### [​](https://docs.chonkie.ai/oss/chunkers/sdpm-chunker#basic-initialization) Basic Initialization from chonkie.legacy import SDPMChunker # Legacy initialization chunker = SDPMChunker( embedding_model="minishlab/potion-base-32M", threshold=0.5, chunk_size=2048, min_sentences=1, skip_window=1 ) ### [​](https://docs.chonkie.ai/oss/chunkers/sdpm-chunker#legacy-parameters) Legacy Parameters The legacy SDPMChunker uses these parameters (many now renamed in the new SemanticChunker): * `embedding_model`: Model identifier or embedding instance * `mode`: “cumulative” or “window” (removed in new version) * `threshold`: Similarity threshold (0-1) or “auto” * `chunk_size`: Maximum tokens per chunk * `similarity_window`: Sentences for threshold calculation * `min_sentences`: Minimum sentences per chunk (now `min_sentences_per_chunk`) * `min_chunk_size`: Minimum tokens per chunk (removed in new version) * `min_characters_per_sentence`: Minimum characters per sentence * `threshold_step`: Step size for threshold calculation (removed in new version) * `skip_window`: Number of chunks to skip when merging ### [​](https://docs.chonkie.ai/oss/chunkers/sdpm-chunker#example-migration) Example Migration #### [​](https://docs.chonkie.ai/oss/chunkers/sdpm-chunker#old-code-legacy) Old Code (Legacy) from chonkie.legacy import SDPMChunker chunker = SDPMChunker( embedding_model="minishlab/potion-base-32M", mode="window", threshold="auto", chunk_size=512, min_sentences=1, min_chunk_size=2, skip_window=1 ) chunks = chunker.chunk(text) for chunk in chunks: print(f"Sentences: {len(chunk.sentences)}") #### [​](https://docs.chonkie.ai/oss/chunkers/sdpm-chunker#new-code-recommended) New Code (Recommended) from chonkie import SemanticChunker chunker = SemanticChunker( embedding_model="minishlab/potion-base-32M", threshold=0.7, # Explicit threshold instead of "auto" chunk_size=512, min_sentences_per_chunk=1, # Renamed parameter skip_window=1 # Same functionality ) chunks = chunker.chunk(text) for chunk in chunks: print(f"Token count: {chunk.token_count}") [​](https://docs.chonkie.ai/oss/chunkers/sdpm-chunker#return-type-changes) Return Type Changes ------------------------------------------------------------------------------------------------- ### [​](https://docs.chonkie.ai/oss/chunkers/sdpm-chunker#legacy-return-type) Legacy Return Type The legacy SDPMChunker returns `SemanticChunk` objects with sentence details: @dataclass class SemanticChunk: text: str start_index: int end_index: int token_count: int sentences: list[SemanticSentence] # Detailed sentence information ### [​](https://docs.chonkie.ai/oss/chunkers/sdpm-chunker#new-return-type) New Return Type The new SemanticChunker returns simpler `Chunk` objects: @dataclass class Chunk: text: str start_index: int end_index: int token_count: int # No sentence details - cleaner and more efficient [​](https://docs.chonkie.ai/oss/chunkers/sdpm-chunker#full-legacy-documentation) Full Legacy Documentation ------------------------------------------------------------------------------------------------------------- For users who must use the legacy version, the complete original functionality remains available: from chonkie.legacy import SDPMChunker # All original parameters still work chunker = SDPMChunker( embedding_model="minishlab/potion-base-32M", mode="window", threshold="auto", chunk_size=2048, similarity_window=1, min_sentences=1, min_chunk_size=2, min_characters_per_sentence=12, threshold_step=0.01, delim=[". ", "! ", "? ", "\n"], include_delim="prev", skip_window=1 ) # Original methods preserved chunks = chunker.chunk(text) batch_chunks = chunker.chunk_batch(texts) [​](https://docs.chonkie.ai/oss/chunkers/sdpm-chunker#support) Support ------------------------------------------------------------------------- While the legacy SDPMChunker remains available for backward compatibility, it is no longer actively developed. Please consider migrating to the new SemanticChunker for: * Better performance * Active bug fixes * New features * Ongoing support For migration assistance, see the [SemanticChunker documentation](https://docs.chonkie.ai/oss/chunkers/semantic-chunker) or open an issue on our [GitHub repository](https://github.com/chonkie-ai/chonkie) . Was this page helpful? YesNo [CLI\ \ Previous](https://docs.chonkie.ai/oss/experimental/chonkie-cli) [Changelog\ \ Next](https://docs.chonkie.ai/oss/changelog) ⌘I --- # Chonkie Documentation > Documentation Index > ------------------- > > Fetch the complete documentation index at: [/llms.txt](https://docs.chonkie.ai/llms.txt) > > Use this file to discover all available pages before exploring further. [Skip to main content](https://docs.chonkie.ai/oss/handshakes/pinecone-handshake#content-area) The `PineconeHandshake` class provides seamless integration between Chonkie’s chunking system and Pinecone, a managed vector database. Embed and store your Chonkie chunks in Pinecone directly from the Chonkie SDK. [​](https://docs.chonkie.ai/oss/handshakes/pinecone-handshake#installation) Installation ------------------------------------------------------------------------------------------- Before using the Pinecone handshake, make sure to install the required dependencies: pip install chonkie[pinecone] [​](https://docs.chonkie.ai/oss/handshakes/pinecone-handshake#initialization) Initialization ----------------------------------------------------------------------------------------------- Initialize using chonkie initialize using the client specify embedding model from chonkie import PineconeHandshake handshake = PineconeHandshake(api_key="YOUR_API_KEY") ### [​](https://docs.chonkie.ai/oss/handshakes/pinecone-handshake#parameters) Parameters [​](https://docs.chonkie.ai/oss/handshakes/pinecone-handshake#param-client) client Optional\[pinecone.Pinecone\] default:"None" Pinecone client instance. If not provided, a new client will be created based on other parameters. [​](https://docs.chonkie.ai/oss/handshakes/pinecone-handshake#param-api-key) api\_key Optional\[str\] default:"None" Pinecone API key for authentication. [​](https://docs.chonkie.ai/oss/handshakes/pinecone-handshake#param-index-name) index\_name Union\[str, Literal\['random'\]\] default:"random" Name of the index to use. If “random”, a unique name will be generated. [​](https://docs.chonkie.ai/oss/handshakes/pinecone-handshake#param-embedding-model) embedding\_model Union\[str, BaseEmbeddings\] default:"minishlab/potion-retrieval-32M" Embedding model to use. Can be a model name or a BaseEmbeddings instance. [​](https://docs.chonkie.ai/oss/handshakes/pinecone-handshake#param-dimension) dimension Optional\[int\] default:"None" Dimension of the embeddings. If not provided, will be inferred from the embedding model. [​](https://docs.chonkie.ai/oss/handshakes/pinecone-handshake#param-kwargs) \*\*kwargs dict\[str, Any\] default:"{}" Additional keyword arguments to pass to the Pinecone client or index creation. [​](https://docs.chonkie.ai/oss/handshakes/pinecone-handshake#writing-chunks-to-pinecone) Writing Chunks to Pinecone ----------------------------------------------------------------------------------------------------------------------- from chonkie import PineconeHandshake, SemanticChunker # Initialize the handshake handshake = PineconeHandshake(api_key="YOUR_API_KEY", index_name="my_documents") # Create some chunks chunker = SemanticChunker() chunks = chunker.chunk("Chonkie loves to chonk your texts!") # Write chunks to Pinecone handshake.write(chunks) [​](https://docs.chonkie.ai/oss/handshakes/pinecone-handshake#searching-chunks-in-pinecone) Searching Chunks in Pinecone --------------------------------------------------------------------------------------------------------------------------- You can retrieve the most similar chunks from your Pinecone index using the `search` method: search using a query search using embedding search using chonkie chunks from chonkie import PineconeHandshake # Initialize the handshake handshake = PineconeHandshake(api_key="YOUR_API_KEY", index_name="my_documents") results = handshake.search(query="chonk your texts", limit=2) for result in results: print(result["score"], result["text"]) Was this page helpful? YesNo [Pgvector Handshake\ \ Previous](https://docs.chonkie.ai/oss/handshakes/pgvector-handshake) [Qdrant Handshake\ \ Next](https://docs.chonkie.ai/oss/handshakes/qdrant-handshake) ⌘I --- # Chonkie Documentation > Documentation Index > ------------------- > > Fetch the complete documentation index at: [/llms.txt](https://docs.chonkie.ai/llms.txt) > > Use this file to discover all available pages before exploring further. [Skip to main content](https://docs.chonkie.ai/oss/handshakes/turbopuffer-handshake#content-area) The `TurbopufferHandshake` class provides seamless integration between Chonkie’s chunking system and Turbopuffer, a high-performance vector database. Embed and store your Chonkie chunks in Turbopuffer without ever leaving the Chonkie SDK. The Turbopuffer Handshake requires a Turbopuffer API key. You can get one by signing up for a [Turbopuffer account](https://turbopuffer.com/) . [​](https://docs.chonkie.ai/oss/handshakes/turbopuffer-handshake#installation) Installation ---------------------------------------------------------------------------------------------- Before using the Turbopuffer handshake, make sure to install the required dependencies: pip install chonkie[turbopuffer] [​](https://docs.chonkie.ai/oss/handshakes/turbopuffer-handshake#basic-usage) Basic Usage -------------------------------------------------------------------------------------------- ### [​](https://docs.chonkie.ai/oss/handshakes/turbopuffer-handshake#initialization) Initialization from chonkie import TurbopufferHandshake # Initialize with default settings (requires TURBOPUFFER_API_KEY environment variable) handshake = TurbopufferHandshake() # Or provide an API key directly handshake = TurbopufferHandshake(api_key="your_turbopuffer_api_key") # Use a specific namespace handshake = TurbopufferHandshake(namespace_name="my_documents") # Or use an existing Turbopuffer namespace import turbopuffer as tpuf ns = tpuf.Namespace("existing_namespace") handshake = TurbopufferHandshake(namespace=ns) ### [​](https://docs.chonkie.ai/oss/handshakes/turbopuffer-handshake#writing-chunks-to-turbopuffer) Writing Chunks to Turbopuffer from chonkie import TurbopufferHandshake, SemanticChunker handshake = TurbopufferHandshake(namespace_name="my_documents") chunker = SemanticChunker() chunks = chunker("Chonkie chunks, turbopuffer puffs!") handshake.write(chunks) [​](https://docs.chonkie.ai/oss/handshakes/turbopuffer-handshake#parameters) Parameters ------------------------------------------------------------------------------------------ [​](https://docs.chonkie.ai/oss/handshakes/turbopuffer-handshake#param-namespace) namespace Optional\[tpuf.Namespace\] default:"None" An existing Turbopuffer Namespace instance to use. If not provided, a new namespace will be created. [​](https://docs.chonkie.ai/oss/handshakes/turbopuffer-handshake#param-namespace-name) namespace\_name Union\[str, Literal\['random'\]\] default:"random" Name of the namespace to use. If “random”, a unique name will be generated. Only used if `namespace` parameter is not provided. [​](https://docs.chonkie.ai/oss/handshakes/turbopuffer-handshake#param-embedding-model) embedding\_model Union\[str, BaseEmbeddings\] default:"minishlab/potion-retrieval-32M" Embedding model to use. Can be a model name or a BaseEmbeddings instance. [​](https://docs.chonkie.ai/oss/handshakes/turbopuffer-handshake#param-api-key) api\_key Optional\[str\] default:"None" Turbopuffer API key. If not provided, will look for TURBOPUFFER\_API\_KEY environment variable. [​](https://docs.chonkie.ai/oss/handshakes/turbopuffer-handshake#authentication) Authentication -------------------------------------------------------------------------------------------------- You can authenticate with Turbopuffer in one of two ways: 1. **Environment Variable** (Recommended for development): export TURBOPUFFER_API_KEY='your-api-key-here' 2. **Directly in code** (Not recommended for production): handshake = TurbopufferHandshake(api_key="your-api-key-here") For production environments, it’s recommended to use environment variables or a secure secret management system to handle your API keys. Was this page helpful? YesNo [Qdrant Handshake\ \ Previous](https://docs.chonkie.ai/oss/handshakes/qdrant-handshake) [Weaviate Handshake\ \ Next](https://docs.chonkie.ai/oss/handshakes/weaviate-handshake) ⌘I --- # Chonkie Documentation > Documentation Index > ------------------- > > Fetch the complete documentation index at: [/llms.txt](https://docs.chonkie.ai/llms.txt) > > Use this file to discover all available pages before exploring further. [Skip to main content](https://docs.chonkie.ai/oss/fetchers/file-fetcher#content-area) The `FileFetcher` retrieves files from your local filesystem. It supports two modes: fetching a single file or fetching multiple files from a directory with optional extension filtering. [​](https://docs.chonkie.ai/oss/fetchers/file-fetcher#installation) Installation ----------------------------------------------------------------------------------- FileFetcher is included with the base Chonkie installation: pip install chonkie [​](https://docs.chonkie.ai/oss/fetchers/file-fetcher#usage) Usage --------------------------------------------------------------------- ### [​](https://docs.chonkie.ai/oss/fetchers/file-fetcher#single-file-mode) Single File Mode Fetch a single file by providing the `path` parameter: from chonkie.pipeline import Pipeline # Fetch and process a single file doc = (Pipeline() .fetch_from("file", path="document.txt") .process_with("text") .chunk_with("recursive", chunk_size=512) .run()) print(f"Chunked into {len(doc.chunks)} chunks") ### [​](https://docs.chonkie.ai/oss/fetchers/file-fetcher#directory-mode) Directory Mode Fetch multiple files from a directory using the `dir` parameter: # Fetch all files from a directory docs = (Pipeline() .fetch_from("file", dir="./documents") .process_with("text") .chunk_with("recursive", chunk_size=512) .run()) print(f"Processed {len(docs)} documents") for doc in docs: print(f" - {len(doc.chunks)} chunks") ### [​](https://docs.chonkie.ai/oss/fetchers/file-fetcher#extension-filtering) Extension Filtering Filter files by extension when using directory mode: # Fetch only .txt and .md files docs = (Pipeline() .fetch_from("file", dir="./documents", ext=[".txt", ".md"]) .process_with("text") .chunk_with("recursive", chunk_size=512) .run()) [​](https://docs.chonkie.ai/oss/fetchers/file-fetcher#parameters) Parameters ------------------------------------------------------------------------------- [​](https://docs.chonkie.ai/oss/fetchers/file-fetcher#param-path) path str Path to a single file. Cannot be used with `dir`. [​](https://docs.chonkie.ai/oss/fetchers/file-fetcher#param-dir) dir str Directory to fetch files from. Cannot be used with `path`. [​](https://docs.chonkie.ai/oss/fetchers/file-fetcher#param-ext) ext list\[str\] List of file extensions to filter (e.g., `[".txt", ".md"]`). Only used with `dir` parameter. [​](https://docs.chonkie.ai/oss/fetchers/file-fetcher#return-values) Return Values ------------------------------------------------------------------------------------- * **Single file mode** (`path` provided): Returns a single `Path` object * **Directory mode** (`dir` provided): Returns `list[Path]` containing all matching files [​](https://docs.chonkie.ai/oss/fetchers/file-fetcher#standalone-usage) Standalone Usage ------------------------------------------------------------------------------------------- You can also use FileFetcher directly without the pipeline: from chonkie import FileFetcher fetcher = FileFetcher() # Single file file_path = fetcher.fetch(path="document.txt") print(file_path) # PosixPath('document.txt') # Directory with extension filter files = fetcher.fetch(dir="./docs", ext=[".txt", ".md"]) for file in files: print(file) [​](https://docs.chonkie.ai/oss/fetchers/file-fetcher#error-handling) Error Handling --------------------------------------------------------------------------------------- FileFetcher validates inputs and provides clear error messages: # FileNotFoundError if file doesn't exist fetcher.fetch(path="nonexistent.txt") # Raises FileNotFoundError # ValueError if both path and dir are provided fetcher.fetch(path="file.txt", dir="./docs") # Raises ValueError # ValueError if neither is provided fetcher.fetch() # Raises ValueError [​](https://docs.chonkie.ai/oss/fetchers/file-fetcher#best-practices) Best Practices --------------------------------------------------------------------------------------- Use extension filtering for large directories When working with directories containing many files, always specify `ext` to avoid processing unwanted files: # Good - only processes markdown files .fetch_from("file", dir="./docs", ext=[".md"]) # Potentially slow - processes ALL files .fetch_from("file", dir="./docs") Use absolute paths for clarity While relative paths work, absolute paths make your pipeline more portable: from pathlib import Path docs_dir = Path(__file__).parent / "documents" .fetch_from("file", dir=str(docs_dir), ext=[".txt"]) [​](https://docs.chonkie.ai/oss/fetchers/file-fetcher#what%E2%80%99s-next) What’s Next? ------------------------------------------------------------------------------------------ After fetching files, you’ll typically want to: 1. **Process** them with a [Chef](https://docs.chonkie.ai/oss/chefs/overview) to parse content 2. **Chunk** them with a [Chunker](https://docs.chonkie.ai/oss/chunkers/overview) to split into manageable pieces 3. **Refine** chunks with [Refineries](https://docs.chonkie.ai/oss/refinery/overview) for better quality See the [Pipeline Guide](https://docs.chonkie.ai/oss/pipelines) for complete examples. Was this page helpful? YesNo [Fetchers Overview\ \ Previous](https://docs.chonkie.ai/oss/fetchers/overview) [Chunkers Overview\ \ Next](https://docs.chonkie.ai/oss/chunkers/overview) ⌘I --- # Chonkie Documentation > Documentation Index > ------------------- > > Fetch the complete documentation index at: [/llms.txt](https://docs.chonkie.ai/llms.txt) > > Use this file to discover all available pages before exploring further. [Skip to main content](https://docs.chonkie.ai/oss/handshakes/mongodb-handshake#content-area) The `MongoDBHandshake` class provides integration between Chonkie’s chunking system and MongoDB, a popular NoSQL database. Embed and store your Chonkie chunks in MongoDB directly from the Chonkie SDK. [​](https://docs.chonkie.ai/oss/handshakes/mongodb-handshake#installation) Installation ------------------------------------------------------------------------------------------ Before using the MongoDB handshake, make sure to install the required dependencies: pip install chonkie[mongodb] [​](https://docs.chonkie.ai/oss/handshakes/mongodb-handshake#initialization) Initialization ---------------------------------------------------------------------------------------------- Initialize using chonkie initialize using the client specify connection and embedding from chonkie import MongoDBHandshake # Initialize with default settings (auto-generated database/collection) handshake = MongoDBHandshake(uri="mongodb://localhost:27017") ### [​](https://docs.chonkie.ai/oss/handshakes/mongodb-handshake#parameters) Parameters [​](https://docs.chonkie.ai/oss/handshakes/mongodb-handshake#param-client) client Optional\[pymongo.MongoClient\] default:"None" MongoDB client instance. If not provided, a new client will be created based on other parameters. [​](https://docs.chonkie.ai/oss/handshakes/mongodb-handshake#param-uri) uri Optional\[str\] default:"None" MongoDB connection URI. [​](https://docs.chonkie.ai/oss/handshakes/mongodb-handshake#param-username) username Optional\[str\] default:"None" MongoDB username for authentication. [​](https://docs.chonkie.ai/oss/handshakes/mongodb-handshake#param-password) password Optional\[str\] default:"None" MongoDB password for authentication. [​](https://docs.chonkie.ai/oss/handshakes/mongodb-handshake#param-hostname) hostname Optional\[str\] default:"None" MongoDB host address. [​](https://docs.chonkie.ai/oss/handshakes/mongodb-handshake#param-port) port Optional\[Union\[int, str\]\] default:"None" MongoDB port number. [​](https://docs.chonkie.ai/oss/handshakes/mongodb-handshake#param-db-name) db\_name Union\[str, Literal\['random'\]\] default:"random" Name of the database to use. If “random”, a unique name will be generated. [​](https://docs.chonkie.ai/oss/handshakes/mongodb-handshake#param-collection-name) collection\_name Union\[str, Literal\['random'\]\] default:"random" Name of the collection to use. If “random”, a unique name will be generated. [​](https://docs.chonkie.ai/oss/handshakes/mongodb-handshake#param-embedding-model) embedding\_model Union\[str, BaseEmbeddings\] default:"minishlab/potion-retrieval-32M" Embedding model to use. Can be a model name or a BaseEmbeddings instance. [​](https://docs.chonkie.ai/oss/handshakes/mongodb-handshake#param-kwargs) \*\*kwargs dict\[str, Any\] default:"{}" Additional keyword arguments to pass to the MongoDB client or collection creation. [​](https://docs.chonkie.ai/oss/handshakes/mongodb-handshake#writing-chunks-to-mongodb) Writing Chunks to MongoDB -------------------------------------------------------------------------------------------------------------------- from chonkie import MongoDBHandshake, SemanticChunker # Initialize the handshake handshake = MongoDBHandshake( uri="mongodb://localhost:27017", db_name="my_documents", collection_name="my_collection" ) # Create some chunks chunker = SemanticChunker() chunks = chunker.chunk("Chonkie loves to chonk your texts!") # Write chunks to MongoDB handshake.write(chunks) [​](https://docs.chonkie.ai/oss/handshakes/mongodb-handshake#searching-chunks-in-mongodb) Searching Chunks in MongoDB ------------------------------------------------------------------------------------------------------------------------ You can retrieve the most similar chunks from your MongoDB collection using the `search` method: search using a query search using embedding search using chonkie chunks from chonkie import MongoDBHandshake # Initialize the handshake handshake = MongoDBHandshake( uri="mongodb://localhost:27017", db_name="my_documents", collection_name="my_collection" ) results = handshake.search(query="chonk your texts", limit=2) for result in results: print(result["score"], result["text"]) Was this page helpful? YesNo [Milvus Handshake\ \ Previous](https://docs.chonkie.ai/oss/handshakes/milvus-handshake) [Pgvector Handshake\ \ Next](https://docs.chonkie.ai/oss/handshakes/pgvector-handshake) ⌘I --- # Chonkie Documentation > Documentation Index > ------------------- > > Fetch the complete documentation index at: [/llms.txt](https://docs.chonkie.ai/llms.txt) > > Use this file to discover all available pages before exploring further. [Skip to main content](https://docs.chonkie.ai/oss/porters/datasets-porter#content-area) The `DatasetsPorter` exports a list of `Chunk` objects into a Hugging Face `Dataset` object. This is particularly useful for saving your processed chunks in a standardized format for training models, sharing, or archiving. [​](https://docs.chonkie.ai/oss/porters/datasets-porter#installation) Installation ------------------------------------------------------------------------------------- The `DatasetsPorter` requires the `datasets` library. You can install it with: pip install "chonkie[datasets]" For general installation instructions, see the [Installation Guide](https://docs.chonkie.ai/oss/installation) . [​](https://docs.chonkie.ai/oss/porters/datasets-porter#initialization) Initialization ----------------------------------------------------------------------------------------- To get started, simply import and initialize the porter. from chonkie import DatasetsPorter porter = DatasetsPorter() [​](https://docs.chonkie.ai/oss/porters/datasets-porter#parameters) Parameters --------------------------------------------------------------------------------- [​](https://docs.chonkie.ai/oss/porters/datasets-porter#param-chunks) chunks list\[Chunk\] required The list of `Chunk` objects to be exported. [​](https://docs.chonkie.ai/oss/porters/datasets-porter#param-save-to-disk) save\_to\_disk bool default:"True" If `True`, the dataset will be saved to the location specified in the `path` parameter. [​](https://docs.chonkie.ai/oss/porters/datasets-porter#param-path) path str default:"chunks" The local directory path where the dataset should be saved. This is only used if `save_to_disk` is `True`. [​](https://docs.chonkie.ai/oss/porters/datasets-porter#param-kwargs) \*\*kwargs Any Additional keyword arguments to be passed directly to the `datasets.Dataset.save_to_disk` method. This allows you to control aspects like the number of shards or processes. [​](https://docs.chonkie.ai/oss/porters/datasets-porter#usage) Usage ----------------------------------------------------------------------- The `DatasetsPorter` can either return a `Dataset` object directly for in-memory use or save it to disk. ### [​](https://docs.chonkie.ai/oss/porters/datasets-porter#return-a-dataset-object) Return a Dataset Object By default, the porter returns a `Dataset` object without writing any files. from chonkie import Chunk chunks = [\ Chunk(text="This is the first chunk.", start_index=0, end_index=25, token_count=5),\ Chunk(text="This is the second chunk.", start_index=26, end_index=52, token_count=5),\ ] # Get the dataset in memory dataset = porter.export(chunks) print(dataset) # Expected output: # Dataset({ # features: ['text', 'start_index', 'end_index', 'token_count', 'context'], # num_rows: 2 # }) ### [​](https://docs.chonkie.ai/oss/porters/datasets-porter#save-a-dataset-to-disk) Save a Dataset to Disk To save the dataset, set `save_to_disk=True` and provide a `path`. The method will still return the `Dataset` object. # Save the dataset to a directory named "my_exported_chunks" dataset = porter.export(chunks, save_to_disk=True, path="my_exported_chunks") # You can now find the dataset files in the "my_exported_chunks" directory ### [​](https://docs.chonkie.ai/oss/porters/datasets-porter#using-as-a-callable) Using as a Callable The porter can also be used as a callable, which is an alias for the `export` method. # Get the dataset in memory dataset = porter(chunks) # Save the dataset to disk porter(chunks, save_to_disk=True, path="my_exported_chunks") [​](https://docs.chonkie.ai/oss/porters/datasets-porter#return-type) Return Type ----------------------------------------------------------------------------------- The `export` method (and the `__call__` method) will always return a `datasets.Dataset` object, regardless of whether it is saved to disk. This allows you to immediately work with the dataset after exporting. Was this page helpful? YesNo [JSONPorter\ \ Previous](https://docs.chonkie.ai/oss/porters/json-porter) [Visualizer\ \ Next](https://docs.chonkie.ai/oss/utils/visualizer) ⌘I --- # Chonkie Documentation > Documentation Index > ------------------- > > Fetch the complete documentation index at: [/llms.txt](https://docs.chonkie.ai/llms.txt) > > Use this file to discover all available pages before exploring further. [Skip to main content](https://docs.chonkie.ai/oss/handshakes/pgvector-handshake#content-area) The `PgvectorHandshake` class provides seamless integration between Chonkie’s chunking system and PostgreSQL with pgvector. It uses the vecs client library from Supabase underneath to provide a higher-level API with automatic indexing, metadata filtering, and simplified connection management. Store your Chonkie chunks in PostgreSQL with vector embeddings and perform semantic search without ever leaving the Chonkie SDK. [​](https://docs.chonkie.ai/oss/handshakes/pgvector-handshake#installation) Installation ------------------------------------------------------------------------------------------- Before using the Pgvector handshake, make sure to install the required dependencies: pip install chonkie[pgvector] You’ll also need PostgreSQL with the pgvector extension installed: -- Connect to your database and enable pgvector CREATE EXTENSION IF NOT EXISTS vector; [​](https://docs.chonkie.ai/oss/handshakes/pgvector-handshake#initialization) Initialization ----------------------------------------------------------------------------------------------- from chonkie import PgvectorHandshake # Initialize with individual connection parameters handshake = PgvectorHandshake( host="localhost", port=5432, database="your_database", user="your_user", password="your_password", collection_name="chonkie_chunks" ) # Or use a connection string handshake = PgvectorHandshake( connection_string="postgresql://user:password@localhost:5432/database" ) # Or use an existing vecs client import vecs client = vecs.create_client("postgresql://user:password@localhost:5432/database") handshake = PgvectorHandshake(client=client, collection_name="my_collection") [​](https://docs.chonkie.ai/oss/handshakes/pgvector-handshake#usage) Usage ----------------------------------------------------------------------------- Writing Chunks Store your chunked text in PostgreSQL with vector embeddings from chonkie import PgvectorHandshake, RecursiveChunker # Initialize the handshake handshake = PgvectorHandshake( host="localhost", database="my_database", user="my_user", password="my_password" ) # Create some chunks chunker = RecursiveChunker(chunk_size=2048) chunks = chunker.chunk("Chonkie makes PostgreSQL vector search easy!") # Write chunks to PostgreSQL handshake.write(chunks) Searching Chunks Find similar chunks using vector similarity search # Search for similar chunks results = handshake.search( query="PostgreSQL vector search", limit=5 ) for result in results: print(f"Text: {result['text']}") print(f"Similarity: {result['similarity']:.3f}") print("---") Creating Indexes Optimize search performance with vector indexes # Create an HNSW index for better performance handshake.create_index(method="hnsw") # Or create with custom parameters handshake.create_index( method="hnsw", m=16, ef_construction=64 ) [​](https://docs.chonkie.ai/oss/handshakes/pgvector-handshake#parameters) Parameters --------------------------------------------------------------------------------------- [​](https://docs.chonkie.ai/oss/handshakes/pgvector-handshake#param-client) client Optional\[vecs.Client\] default:"None" An existing vecs.Client instance. If provided, other connection parameters are ignored. [​](https://docs.chonkie.ai/oss/handshakes/pgvector-handshake#param-host) host str default:"localhost" PostgreSQL host address. [​](https://docs.chonkie.ai/oss/handshakes/pgvector-handshake#param-port) port int default:"5432" PostgreSQL port number. [​](https://docs.chonkie.ai/oss/handshakes/pgvector-handshake#param-database) database str default:"postgres" PostgreSQL database name. [​](https://docs.chonkie.ai/oss/handshakes/pgvector-handshake#param-user) user str default:"postgres" PostgreSQL username. [​](https://docs.chonkie.ai/oss/handshakes/pgvector-handshake#param-password) password str default:"postgres" PostgreSQL password. [​](https://docs.chonkie.ai/oss/handshakes/pgvector-handshake#param-connection-string) connection\_string Optional\[str\] default:"None" Full PostgreSQL connection string. If provided, individual connection parameters are ignored. [​](https://docs.chonkie.ai/oss/handshakes/pgvector-handshake#param-collection-name) collection\_name str default:"chonkie\_chunks" Name of the collection to store chunks in. [​](https://docs.chonkie.ai/oss/handshakes/pgvector-handshake#param-embedding-model) embedding\_model Union\[str, BaseEmbeddings\] default:"minishlab/potion-retrieval-32M" Embedding model to use. Can be a model name or a BaseEmbeddings instance. [​](https://docs.chonkie.ai/oss/handshakes/pgvector-handshake#param-vector-dimensions) vector\_dimensions Optional\[int\] default:"None" Number of dimensions for the vector embeddings. If not provided, will be inferred from the embedding model. Was this page helpful? YesNo [MongoDB Handshake\ \ Previous](https://docs.chonkie.ai/oss/handshakes/mongodb-handshake) [Pinecone Handshake\ \ Next](https://docs.chonkie.ai/oss/handshakes/pinecone-handshake) ⌘I --- # Chonkie Documentation > Documentation Index > ------------------- > > Fetch the complete documentation index at: [/llms.txt](https://docs.chonkie.ai/llms.txt) > > Use this file to discover all available pages before exploring further. [Skip to main content](https://docs.chonkie.ai/oss/chunkers/overview#content-area) Chonkie provides multiple chunking strategies to handle different text processing needs. Each chunker in Chonkie is designed to follow the same core principles outlined in the [concepts](https://docs.chonkie.ai/common/concepts) page. CodeChunker ----------- Splits code based on its structure using ASTs. Ideal for chunking source code files. FastChunker ----------- SIMD-accelerated byte-based chunking at 100+ GB/s. Best for high-throughput pipelines where byte size limits are acceptable. LateChunker ----------- Chunks using Late Chunking algorithm, best for higher recall in your RAG applications. NeuralChunker ------------- Uses a fine-tuned BERT model to split text based on semantic shifts. Great for topic-coherent chunks. RecursiveChunker ---------------- Recursively chunks documents into smaller chunks. Best for long documents with well-defined structure. SemanticChunker --------------- Groups content based on semantic similarity. Best for preserving context and topical coherence. SentenceChunker --------------- Splits text at sentence boundaries. Perfect for maintaining semantic completeness at the sentence level. SlumberChunker -------------- Agentic chunking using generative models (LLMs) via the Genie interface for S-tier chunk quality. 🦛🧞 TableChunker ------------ Splits large markdown tables into smaller, manageable chunks by row, preserving headers. Great for tabular data in RAG and LLM pipelines. TeraflopAIChunker ----------------- Segments text using the TeraflopAI Segmentation API. Ideal for domain-specific segmentation such as legal documents. TokenChunker ------------ Splits text into fixed-size token chunks. Best for maintaining consistent chunk sizes and working with token-based models. [​](https://docs.chonkie.ai/oss/chunkers/overview#availability) Availability ------------------------------------------------------------------------------- Different chunkers are available depending on your installation: | Chunker | Default | embeddings | `"chonkie[all]"` | Chonkie JS | API Chunking | | --- | --- | --- | --- | --- | --- | | TokenChunker | | | | | | | FastChunker | | | | | | | SentenceChunker | | | | | | | RecursiveChunker | | | | | | | TableChunker | | | | | | | CodeChunker | | | | | | | SemanticChunker | | | | | | | LateChunker | | | | | | | NeuralChunker | | | | | | | SlumberChunker | | | | | | [​](https://docs.chonkie.ai/oss/chunkers/overview#common-interface) Common Interface --------------------------------------------------------------------------------------- All chunkers share a consistent interface: Python JavaScript # Single text chunking chunks = chunker.chunk(text) # Batch processing chunks = chunker.chunk_batch(texts) # Direct calling chunks = chunker(text) # or chunker([text1, text2]) # Async variants (all chunkers support these) chunks = await chunker.achunk(text) chunks = await chunker.achunk_batch(texts) [​](https://docs.chonkie.ai/oss/chunkers/overview#async-support) Async Support --------------------------------------------------------------------------------- Every chunker supports async out of the box — no extra setup required. | Method | Async Equivalent | Description | | --- | --- | --- | | `chunk(text)` | `achunk(text)` | Chunk a single text | | `chunk_batch(texts)` | `achunk_batch(texts)` | Chunk a list of texts | | `chunk_document(doc)` | `achunk_document(doc)` | Chunk a `Document` object | ### [​](https://docs.chonkie.ai/oss/chunkers/overview#basic-usage) Basic Usage import asyncio from chonkie import RecursiveChunker async def main(): chunker = RecursiveChunker(chunk_size=512) chunks = await chunker.achunk("Your document text here...") all_chunks = await chunker.achunk_batch([\ "First document...",\ "Second document...",\ "Third document...",\ ]) asyncio.run(main()) ### [​](https://docs.chonkie.ai/oss/chunkers/overview#concurrent-chunking) Concurrent Chunking Use `asyncio.gather` to chunk multiple texts concurrently: import asyncio from chonkie import SemanticChunker async def process_documents(texts: list[str]): chunker = SemanticChunker(chunk_size=512) results = await asyncio.gather( *[chunker.achunk(text) for text in texts] ) return results ### [​](https://docs.chonkie.ai/oss/chunkers/overview#how-it-works) How It Works * **`achunk` and `achunk_batch`** run the synchronous methods in a thread pool via `asyncio.to_thread`, so CPU-bound chunking does not block your event loop. * **`achunk_document`** goes further: if the document has pre-existing chunks, it dispatches a concurrent `asyncio.gather` over all of them. Because `achunk` and `achunk_batch` use `asyncio.to_thread`, they are safe to use in async web frameworks (FastAPI, Starlette, aiohttp, etc.) without blocking the event loop. [​](https://docs.chonkie.ai/oss/chunkers/overview#f-a-q) F.A.Q. ------------------------------------------------------------------ Are all the chunkers thread-safe? Yes, all the chunkers are thread-safe. Though, the performance might vary since some chunkers use threading under the hood. So, monitor your performance accordingly. Do I need to install anything extra for async support? No. Async support is built into every chunker via `BaseChunker`. Any chunker you import from `chonkie` already has `achunk`, `achunk_batch`, and `achunk_document` available. Does async chunking improve throughput? Yes, especially when chunking many texts concurrently. `achunk` offloads work to a thread pool, so multiple coroutines can chunk in parallel without blocking the event loop. For single-text chunking the overhead is minimal. Is it safe to share a chunker instance across async tasks? Yes. All chunkers are thread-safe, so sharing a single instance across concurrent `asyncio.gather` calls is fine and avoids redundant initialization costs. Which async frameworks are supported? Any framework that uses `asyncio` — FastAPI, Starlette, aiohttp, Sanic, Litestar, and others. The async methods use standard `asyncio` primitives with no framework-specific dependencies. Was this page helpful? YesNo [FileFetcher\ \ Previous](https://docs.chonkie.ai/oss/fetchers/file-fetcher) [Code Chunker\ \ Next](https://docs.chonkie.ai/oss/chunkers/code-chunker) ⌘I --- # Chonkie Documentation > Documentation Index > ------------------- > > Fetch the complete documentation index at: [/llms.txt](https://docs.chonkie.ai/llms.txt) > > Use this file to discover all available pages before exploring further. [Skip to main content](https://docs.chonkie.ai/oss/handshakes/weaviate-handshake#content-area) The `WeaviateHandshake` class provides seamless integration between Chonkie’s chunking system and [Weaviate](https://weaviate.io/) , a powerful vector database. Embed and store your Chonkie chunks in Weaviate without ever leaving the Chonkie SDK. [​](https://docs.chonkie.ai/oss/handshakes/weaviate-handshake#installation) Installation ------------------------------------------------------------------------------------------- Before using the Weaviate handshake, make sure to install the required dependencies: pip install chonkie[weaviate] [​](https://docs.chonkie.ai/oss/handshakes/weaviate-handshake#basic-usage) Basic Usage ----------------------------------------------------------------------------------------- ### [​](https://docs.chonkie.ai/oss/handshakes/weaviate-handshake#initialization) Initialization Initialize using chonkie initialize using the client weaviate cloud initialization from chonkie import WeaviateHandshake # Initialize with default settings (local Weaviate) handshake = WeaviateHandshake() # Or connect to a Weaviate server handshake = WeaviateHandshake(url="http://localhost:8080", api_key= "YOUR_API_KEY") [​](https://docs.chonkie.ai/oss/handshakes/weaviate-handshake#parameters) Parameters --------------------------------------------------------------------------------------- [​](https://docs.chonkie.ai/oss/handshakes/weaviate-handshake#param-client) client Optional\[weaviate.Client\] default:"None" Weaviate client instance. If not provided, a new client will be created based on other parameters. [​](https://docs.chonkie.ai/oss/handshakes/weaviate-handshake#param-collection-name) collection\_name Union\[str, Literal\['random'\]\] default:"random" Name of the collection to use. If “random”, a unique name will be generated. [​](https://docs.chonkie.ai/oss/handshakes/weaviate-handshake#param-embedding-model) embedding\_model Union\[str, BaseEmbeddings\] default:"minishlab/potion-retrieval-32M" Embedding model to use. Can be a model name or a BaseEmbeddings instance. [​](https://docs.chonkie.ai/oss/handshakes/weaviate-handshake#param-url) url Optional\[str\] default:"None" URL of the Weaviate server. If provided, will connect to this server. [​](https://docs.chonkie.ai/oss/handshakes/weaviate-handshake#param-api-key) api\_key Optional\[str\] default:"None" API key for Weaviate Cloud authentication. [​](https://docs.chonkie.ai/oss/handshakes/weaviate-handshake#param-auth-config) auth\_config Optional\[dict\[str, Any\]\] default:"None" OAuth configuration for authentication (optional). [​](https://docs.chonkie.ai/oss/handshakes/weaviate-handshake#param-batch-size) batch\_size int default:"100" Batch size for batch operations. [​](https://docs.chonkie.ai/oss/handshakes/weaviate-handshake#param-batch-dynamic) batch\_dynamic bool default:"True" Whether to use dynamic batching. [​](https://docs.chonkie.ai/oss/handshakes/weaviate-handshake#param-batch-timeout-retries) batch\_timeout\_retries int default:"3" Number of retries for batch timeouts. [​](https://docs.chonkie.ai/oss/handshakes/weaviate-handshake#param-additional-headers) additional\_headers Optional\[dict\[str, str\]\] default:"None" Additional headers for the Weaviate client. [​](https://docs.chonkie.ai/oss/handshakes/weaviate-handshake#writing-chunks-to-weaviate) Writing Chunks to Weaviate ----------------------------------------------------------------------------------------------------------------------- from chonkie import WeaviateHandshake, SemanticChunker # Initialize the handshake handshake = WeaviateHandshake( url="YOUR_CLOUD_URL", api_key="YOUR_API_KEY", collection_name="my_documents" ) # Create some chunks chunker = SemanticChunker() chunks = chunker.chunk("Chonkie loves to chonk your texts!") # Write chunks to Weaviate handshake.write(chunks) ### [​](https://docs.chonkie.ai/oss/handshakes/weaviate-handshake#searching-chunks-in-weaviate) Searching Chunks in Weaviate You can retrieve the most similar chunks from your Weaviate collection using the `search` method: search using a query search using embedding search using chonkie chunks from chonkie import WeaviateHandshake # Initialize the handshake handshake = WeaviateHandshake( url="YOUR_CLOUD_URL", api_key="YOUR_API_KEY", collection_name="my_documents" ) results = handshake.search(query="chonk your texts", limit=2) for result in results: print(result["score"], result["text"]) Was this page helpful? YesNo [Turbopuffer Handshake\ \ Previous](https://docs.chonkie.ai/oss/handshakes/turbopuffer-handshake) [Porters Overview\ \ Next](https://docs.chonkie.ai/oss/porters/overview) ⌘I --- # Chonkie Documentation > Documentation Index > ------------------- > > Fetch the complete documentation index at: [/llms.txt](https://docs.chonkie.ai/llms.txt) > > Use this file to discover all available pages before exploring further. [Skip to main content](https://docs.chonkie.ai/oss/handshakes/elastic-handshake#content-area) The `ElasticHandshake` class provides seamless integration between Chonkie’s chunking system and Elasticsearch, allowing you to leverage its powerful vector search capabilities. Embed and store your Chonkie chunks in an Elasticsearch index without ever leaving the Chonkie SDK. The handshake automatically handles index creation and the necessary vector field mapping. [​](https://docs.chonkie.ai/oss/handshakes/elastic-handshake#installation) Installation ------------------------------------------------------------------------------------------ Before using the Elasticsearch handshake, make sure to install the required dependencies: pip install chonkie[elastic] [​](https://docs.chonkie.ai/oss/handshakes/elastic-handshake#basic-usage) Basic Usage ---------------------------------------------------------------------------------------- ### [​](https://docs.chonkie.ai/oss/handshakes/elastic-handshake#initialization) Initialization Initialize for a Local Instance Initialize for Elastic Cloud Initialize with an Existing Client from chonkie import ElasticHandshake # Connects to http://localhost:9200 by default handshake = ElasticHandshake() ### [​](https://docs.chonkie.ai/oss/handshakes/elastic-handshake#parameters) Parameters [​](https://docs.chonkie.ai/oss/handshakes/elastic-handshake#param-client) client Optional\[Elasticsearch\] default:"None" An existing `elasticsearch.Elasticsearch` client instance. If not provided, a new client will be created based on other parameters. [​](https://docs.chonkie.ai/oss/handshakes/elastic-handshake#param-index-name) index\_name Union\[str, Literal\['random'\]\] default:"random" Name of the Elasticsearch index to use. If “random”, a unique name will be generated. [​](https://docs.chonkie.ai/oss/handshakes/elastic-handshake#param-embedding-model) embedding\_model Union\[str, BaseEmbeddings\] default:"minishlab/potion-retrieval-32M" The embedding model to use for creating vectors. Can be a model name from Hugging Face or a `BaseEmbeddings` instance. [​](https://docs.chonkie.ai/oss/handshakes/elastic-handshake#param-hosts) hosts Optional\[Union\[str, list\[str\]\]\] default:"None" The URL(s) of the Elasticsearch instance(s) to connect to. [​](https://docs.chonkie.ai/oss/handshakes/elastic-handshake#param-cloud-id) cloud\_id Optional\[str\] default:"None" The Cloud ID for connecting to an Elastic Cloud deployment. [​](https://docs.chonkie.ai/oss/handshakes/elastic-handshake#param-api-key) api\_key Optional\[str\] default:"None" The API key for authenticating with Elasticsearch, commonly used for Elastic Cloud. ### [​](https://docs.chonkie.ai/oss/handshakes/elastic-handshake#writing-chunks-to-elasticsearch) Writing Chunks to Elasticsearch from chonkie import ElasticHandshake, SentenceChunker # Initialize the handshake for your deployment handshake = ElasticHandshake( cloud_id="YOUR_CLOUD_ID", api_key="YOUR_API_KEY", index_name="my_documents", ) # Create some chunks chunker = SentenceChunker() chunks = chunker.chunk("Chonkie uses the bulk API for efficient indexing. It's fast and reliable!") # Write chunks to Elasticsearch handshake.write(chunks) ### [​](https://docs.chonkie.ai/oss/handshakes/elastic-handshake#searching-chunks-in-elasticsearch) Searching Chunks in Elasticsearch You can retrieve the most similar chunks from your Elasticsearch index using the `search` method, which performs a k-Nearest Neighbor (kNN) vector search. Search using a Text Query Search using an Embedding Vector from chonkie import ElasticHandshake # Initialize the handshake to connect to your index handshake = ElasticHandshake( hosts="YOUR_CLOUD_ID", api_key="YOUR_API_KEY", index_name="my_documents", ) results = handshake.search(query="fast and efficient indexing", limit=2) Was this page helpful? YesNo [Chroma Handshake\ \ Previous](https://docs.chonkie.ai/oss/handshakes/chroma-handshake) [LanceDB Handshake\ \ Next](https://docs.chonkie.ai/oss/handshakes/lancedb-handshake) ⌘I --- # Chonkie Documentation > Documentation Index > ------------------- > > Fetch the complete documentation index at: [/llms.txt](https://docs.chonkie.ai/llms.txt) > > Use this file to discover all available pages before exploring further. [Skip to main content](https://docs.chonkie.ai/oss/handshakes/qdrant-handshake#content-area) The `QdrantHandshake` class provides seamless integration between Chonkie’s chunking system and Qdrant, a high-performance vector database. Embed and store your Chonkie chunks in Qdrant without ever leaving the Chonkie SDK. [​](https://docs.chonkie.ai/oss/handshakes/qdrant-handshake#installation) Installation ----------------------------------------------------------------------------------------- Before using the Qdrant handshake, make sure to install the required dependencies: pip install chonkie[qdrant] [​](https://docs.chonkie.ai/oss/handshakes/qdrant-handshake#basic-usage) Basic Usage --------------------------------------------------------------------------------------- ### [​](https://docs.chonkie.ai/oss/handshakes/qdrant-handshake#initialization) Initialization Initialize using chonkie initialize using the client qdrant cloud Initialization from chonkie import QdrantHandshake handshake = QdrantHandshake(url="http://localhost:6333") [​](https://docs.chonkie.ai/oss/handshakes/qdrant-handshake#parameters) Parameters ===================================================================================== [​](https://docs.chonkie.ai/oss/handshakes/qdrant-handshake#param-client) client Optional\[qdrant\_client.QdrantClient\] default:"None" Qdrant client instance. If not provided, a new client will be created based on other parameters. [​](https://docs.chonkie.ai/oss/handshakes/qdrant-handshake#param-collection-name) collection\_name Union\[str, Literal\['random'\]\] default:"random" Name of the collection to use. If “random”, a unique name will be generated. [​](https://docs.chonkie.ai/oss/handshakes/qdrant-handshake#param-embedding-model) embedding\_model Union\[str, BaseEmbeddings\] default:"minishlab/potion-retrieval-32M" Embedding model to use. Can be a model name or a BaseEmbeddings instance. [​](https://docs.chonkie.ai/oss/handshakes/qdrant-handshake#param-url) url Optional\[str\] default:"None" URL of the Qdrant server. If provided, will connect to this server. [​](https://docs.chonkie.ai/oss/handshakes/qdrant-handshake#param-path) path Optional\[str\] default:"None" If provided, creates a persistent Qdrant client at the specified path. [​](https://docs.chonkie.ai/oss/handshakes/qdrant-handshake#param-api-key) api\_key Optional\[str\] default:"None" API key for Qdrant Cloud authentication. ### [​](https://docs.chonkie.ai/oss/handshakes/qdrant-handshake#writing-chunks-to-qdrant) Writing Chunks to Qdrant from chonkie import QdrantHandshake, SemanticChunker # Initialize the handshake handshake = QdrantHandshake( url="YOUR_CLOUD_URL", api_key="YOUR_API_KEY", collection_name="my_documents", ) # Create some chunks chunker = SemanticChunker() chunks = chunker.chunk("Chonkie loves to chonk your texts!") # Write chunks to Qdrant handshake.write(chunks) ### [​](https://docs.chonkie.ai/oss/handshakes/qdrant-handshake#searching-chunks-in-qdrant) Searching Chunks in Qdrant You can retrieve the most similar chunks from your Qdrant collection using the `search` method: search using a query search using embedding search using chonkie chunks from chonkie import QdrantHandshake # Initialize the handshake handshake = QdrantHandshake( url="YOUR_CLOUD_URL", api_key="YOUR_API_KEY", collection_name="my_documents", ) results = handshake.search(query="chonk your texts", limit=2) for result in results: print(result["score"], result["text"]) Was this page helpful? YesNo [Pinecone Handshake\ \ Previous](https://docs.chonkie.ai/oss/handshakes/pinecone-handshake) [Turbopuffer Handshake\ \ Next](https://docs.chonkie.ai/oss/handshakes/turbopuffer-handshake) ⌘I --- # Chonkie Documentation > Documentation Index > ------------------- > > Fetch the complete documentation index at: [/llms.txt](https://docs.chonkie.ai/llms.txt) > > Use this file to discover all available pages before exploring further. [Skip to main content](https://docs.chonkie.ai/oss/refinery/embeddings-refinery#content-area) The `EmbeddingsRefinery` allows you to add more more information to your chunks by adding embeddings to them. This is useful for downstream tasks like semantic search, clustering, or vector database insertions. [​](https://docs.chonkie.ai/oss/refinery/embeddings-refinery#api-reference) API Reference -------------------------------------------------------------------------------------------- To use the `EmbeddingsRefinery` via the API, check out the [API reference documentation](https://docs.chonkie.ai/api/refineries/embeddings) . [​](https://docs.chonkie.ai/oss/refinery/embeddings-refinery#initialization) Initialization ---------------------------------------------------------------------------------------------- To use the `EmbeddingsRefinery`, you need to initialize it with an embedding model. from chonkie import EmbeddingsRefinery # Initialize with string model identifier # or an embedding model instance em_refinery = EmbeddingsRefinery( embedding_model="minishlab/potion-base-32M", # Required ) [​](https://docs.chonkie.ai/oss/refinery/embeddings-refinery#usage) Usage ---------------------------------------------------------------------------- Use the `EmbeddingsRefinery` object as a callable or the `refine` method to add embeddings to your chunks. from chonkie import TokenChunker, EmbeddingsRefinery test_string = "This is a test string. It will be chunked and embedded." chunker = TokenChunker() chunks = chunker(test_string) # Add embeddings to the chunks em_refinery = EmbeddingsRefinery( embedding_model="minishlab/potion-base-32M", # Model string or BaseEmbeddings instance ) chunks_with_embeddings = em_refinery(chunks) [​](https://docs.chonkie.ai/oss/refinery/embeddings-refinery#parameters) Parameters -------------------------------------------------------------------------------------- [​](https://docs.chonkie.ai/oss/refinery/embeddings-refinery#param-embedding-model) embedding\_model Union\[str, BaseEmbeddings\] Model identifier or embedding model instance Was this page helpful? YesNo [Overlap Refinery\ \ Previous](https://docs.chonkie.ai/oss/refinery/overlap-refinery) [Handshakes Overview\ \ Next](https://docs.chonkie.ai/oss/handshakes/overview) ⌘I --- # Chonkie Documentation > Documentation Index > ------------------- > > Fetch the complete documentation index at: [/llms.txt](https://docs.chonkie.ai/llms.txt) > > Use this file to discover all available pages before exploring further. [Skip to main content](https://docs.chonkie.ai/oss/chunkers/neural-chunker#content-area) The `NeuralChunker` leverages the power of deep learning! It uses a fine-tuned BERT model specifically trained to identify semantic shifts within text, allowing it to split documents at points where the topic or context changes significantly. This provides highly coherent chunks ideal for RAG. [​](https://docs.chonkie.ai/oss/chunkers/neural-chunker#api-reference) API Reference --------------------------------------------------------------------------------------- To use the `NeuralChunker` via the API, check out the [API reference documentation](https://docs.chonkie.ai/api/chunkers/neural-chunker) . [​](https://docs.chonkie.ai/oss/chunkers/neural-chunker#installation) Installation ------------------------------------------------------------------------------------- NeuralChunker requires specific dependencies for its deep learning model. You can install it with: pip install "chonkie[neural]" For general installation instructions, see the [Installation Guide](https://docs.chonkie.ai/oss/installation) . [​](https://docs.chonkie.ai/oss/chunkers/neural-chunker#initialization) Initialization ----------------------------------------------------------------------------------------- from chonkie import NeuralChunker # Basic initialization with default parameters chunker = NeuralChunker( model="mirth/chonky_modernbert_base_1", # Default model device_map="cpu", # Device to run the model on ('cpu', 'cuda', etc.) min_characters_per_chunk=10, # Minimum characters for a chunk ) # Specify a different model or device chunker = NeuralChunker( model="path/to/your/model", device_map="cuda:0" # Use GPU if available ) [​](https://docs.chonkie.ai/oss/chunkers/neural-chunker#parameters) Parameters --------------------------------------------------------------------------------- [​](https://docs.chonkie.ai/oss/chunkers/neural-chunker#param-model) model str default:"mirth/chonky\_modernbert\_base\_1" The identifier or path to the fine-tuned BERT model used for detecting semantic shifts. [​](https://docs.chonkie.ai/oss/chunkers/neural-chunker#param-tokenizer) tokenizer Optional\[Union\[str, Any\]\] default:"None" The tokenizer to use for the chunker [​](https://docs.chonkie.ai/oss/chunkers/neural-chunker#param-device-map) device\_map str default:"cpu" The device to run the inference on (e.g., “cpu”, “cuda”, “mps”). Chonkie will try to auto-detect the best available device if not specified. [​](https://docs.chonkie.ai/oss/chunkers/neural-chunker#param-min-characters-per-chunk) min\_characters\_per\_chunk int default:"10" The minimum number of characters required for a text segment to be considered a valid chunk. [​](https://docs.chonkie.ai/oss/chunkers/neural-chunker#param-stride) stride Optional\[int\] default:"None" Stride to use for the chunker. Will automatically select appropriate stride for the model if not specified. [​](https://docs.chonkie.ai/oss/chunkers/neural-chunker#usage) Usage ----------------------------------------------------------------------- ### [​](https://docs.chonkie.ai/oss/chunkers/neural-chunker#single-text-chunking) Single Text Chunking text = """Topic one starts here and continues for a bit. Suddenly, the context shifts to topic two, which is quite different. Topic two carries on, discussing various aspects. Then topic one briefly returns. Finally, we conclude with topic three.""" chunks = chunker.chunk(text) for chunk in chunks: print(f"Chunk text: {chunk.text}") print(f"Token count: {chunk.token_count}") # Note: token_count might be added post-hoc or not available depending on implementation print(f"Start index: {chunk.start_index}") print(f"End index: {chunk.end_index}") ### [​](https://docs.chonkie.ai/oss/chunkers/neural-chunker#batch-chunking) Batch Chunking texts = [\ "Document 1 discussing AI ethics. Then shifts to model training techniques.",\ "Document 2 about pygmy hippos. Their habitat and diet. Then conservation efforts."\ ] batch_chunks = chunker.chunk_batch(texts) for doc_chunks in batch_chunks: for chunk in doc_chunks: print(f"Chunk: {chunk.text}") ### [​](https://docs.chonkie.ai/oss/chunkers/neural-chunker#using-as-a-callable) Using as a Callable # Single text chunks = chunker("Text discussing topic A... then topic B...") # Multiple texts batch_chunks = chunker(["Text 1...", "Text 2..."]) [​](https://docs.chonkie.ai/oss/chunkers/neural-chunker#return-type) Return Type ----------------------------------------------------------------------------------- NeuralChunker returns chunks as `Chunk` objects. from dataclasses import dataclass from typing import Optional, Union @dataclass class Chunk: text: str # The chunk text start_index: int # Starting position in original text end_index: int # Ending position in original text token_count: int # Number of tokens in chunk context: Optional[str] = None # Optional overlap context text embedding: Union[list[float], "np.ndarray", None] = None # Optional embedding vector Was this page helpful? YesNo [Late Chunker\ \ Previous](https://docs.chonkie.ai/oss/chunkers/late-chunker) [Recursive Chunker\ \ Next](https://docs.chonkie.ai/oss/chunkers/recursive-chunker) ⌘I --- # Chonkie Documentation > Documentation Index > ------------------- > > Fetch the complete documentation index at: [/llms.txt](https://docs.chonkie.ai/llms.txt) > > Use this file to discover all available pages before exploring further. [Skip to main content](https://docs.chonkie.ai/oss/experimental/chonkie-cli#content-area) [​](https://docs.chonkie.ai/oss/experimental/chonkie-cli#chonkie-cli) Chonkie CLI ==================================================================================== Chonkie provides a powerful Command Line Interface (CLI) to perform chunking and run pipelines directly from your terminal. [​](https://docs.chonkie.ai/oss/experimental/chonkie-cli#installation) Installation -------------------------------------------------------------------------------------- The CLI is included with the default `chonkie` installation: pip install chonkie [​](https://docs.chonkie.ai/oss/experimental/chonkie-cli#basic-usage) Basic Usage ------------------------------------------------------------------------------------ The CLI provides a single `chonkie` command with two primary subcommands: 1. **`chunk`** – Quickly chunk text or files. 2. **`pipeline`** – Run full Chonkie pipelines (fetch → chef → chunk → refine → handbook). To see available options and usage details, use the help flags: main chunk pipeline chonkie --help # Usage: chonkie [OPTIONS] COMMAND [ARGS]... # # > 🦛 CHONK your texts with Chonkie # # ╭─ Options ──────────────────────────────────────────────────────────────────────────────────────────────────────╮ # │ --install-completion Install completion for the current shell. │ # │ --show-completion Show completion for the current shell, to copy it or customize the installation. │ # │ --help Show this message and exit. │ # ╰────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯ # # ╭─ Commands ─────────────────────────────────────────────────────────────────────────────────────────────────────╮ # │ chunk Chunk text using a specified chunker and optionally store it. │ # │ pipeline Run a processing pipeline on text or files. │ # ╰────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯ ### [​](https://docs.chonkie.ai/oss/experimental/chonkie-cli#chunking-texts-or-files) Chunking Texts or Files Use the `chunk` command to quickly chunk text or a single file. **Syntax**: chonkie chunk [TEXT_OR_PATH] [OPTIONS] **Options**: * `--chunker`: The chunking method to use (default: `semantic`). Options: `semantic`, `token`, `sentence`, `recursive`, etc. * `--chunk-size`: Maximum number of tokens per chunk (e.g., `512`, `1024`). * `--chunk-overlap`: Number of tokens to overlap between chunks (e.g., `50`, `100`). * `--threshold`: Threshold for semantic similarity (0-1), used by semantic chunkers. * `--chunker-params`: Additional chunker parameters as `key=value` pairs. Can be used multiple times. * `--handshaker`: Optional storage backend to export chunks. **Examples**: # Chunk raw text with default settings chonkie chunk "This is a long text that needs chunking..." --chunker token # Chunk with explicit chunk size chonkie chunk "Long text..." --chunker recursive --chunk-size 512 # Chunk with overlap chonkie chunk document.txt --chunker token --chunk-size 1024 --chunk-overlap 100 # Chunk with semantic threshold chonkie chunk document.txt --chunker semantic --threshold 0.8 # Chunk with additional parameters using key=value pairs chonkie chunk document.txt \ --chunker recursive \ --chunk-size 512 \ --chunker-params min_characters_per_chunk=50 \ --chunker-params tokenizer=gpt2 # Chunk and store in a vector DB (e.g., Chroma) chonkie chunk document.txt --handshaker chroma * * * ### [​](https://docs.chonkie.ai/oss/experimental/chonkie-cli#running-pipelines) Running Pipelines The `pipeline` command is more powerful and supports processing directories, applying chefs/refiners, and exporting data. **Syntax**: chonkie pipeline [TEXT_OR_PATH] [OPTIONS] **Core Options**: * `--d`: Directory to process (mutually exclusive with text/file argument). * `--ext`: File extensions to include when processing a directory (e.g., `.md`, `.txt`). Can be used multiple times. * `--chef`: Preprocessor to use (e.g., `text`, `markdown`). * `--chef-params`: Parameters for the chef as `key=value` pairs. Can be used multiple times. * `--chunker`: Chunking method (default: `semantic`). * `--chunk-size`: Maximum number of tokens per chunk. * `--chunk-overlap`: Number of tokens to overlap between chunks. * `--threshold`: Threshold for semantic similarity (0-1). * `--chunker-params`: Additional chunker parameters as `key=value` pairs. Can be used multiple times. * `--refiner`: Optional refinement strategy (e.g., `overlap`). * `--refiner-params`: Parameters for the refiner as `key=value` pairs. Can be used multiple times. * `--handshaker`: Optional destination storage. * `--handshaker-params`: Parameters for the handshaker as `key=value` pairs. Can be used multiple times. **Examples**: #### [​](https://docs.chonkie.ai/oss/experimental/chonkie-cli#1-process-a-directory) 1\. Process a Directory Process all markdown and text files in the `docs` directory: chonkie pipeline --d docs --ext .md --ext .txt --chunker recursive #### [​](https://docs.chonkie.ai/oss/experimental/chonkie-cli#2-process-a-single-file) 2\. Process a Single File Run a pipeline on a single file: chonkie pipeline README.md --chunker token --chef text #### [​](https://docs.chonkie.ai/oss/experimental/chonkie-cli#3-pipeline-with-custom-chunking-parameters) 3\. Pipeline with Custom Chunking Parameters Use explicit parameters and additional chunker options: chonkie pipeline document.txt \ --chunker recursive \ --chunk-size 512 \ --chunker-params min_characters_per_chunk=50 #### [​](https://docs.chonkie.ai/oss/experimental/chonkie-cli#4-pipeline-with-multiple-component-parameters) 4\. Pipeline with Multiple Component Parameters Configure chef, chunker, and refiner with custom parameters: chonkie pipeline document.txt \ --chef text \ --chunker token \ --chunk-size 1024 \ --chunk-overlap 100 \ --refiner overlap \ --refiner-params context_size=50 #### [​](https://docs.chonkie.ai/oss/experimental/chonkie-cli#5-full-rag-pipeline) 5\. Full RAG Pipeline Run a full RAG pipeline: fetch from directory -> process markdown -> chunk recursively -> export to ChromaDB. chonkie pipeline \ --d ./knowledge_base \ --ext .md \ --chef markdown \ --chunker recursive \ --chunk-size 512 \ --handshaker chroma \ --handshaker-params collection_name=documents [​](https://docs.chonkie.ai/oss/experimental/chonkie-cli#parameter-configuration) Parameter Configuration ------------------------------------------------------------------------------------------------------------ ### [​](https://docs.chonkie.ai/oss/experimental/chonkie-cli#explicit-parameters) Explicit Parameters For commonly used parameters, you can use dedicated options: * `--chunk-size`: Set the maximum tokens per chunk * `--chunk-overlap`: Set overlap between chunks * `--threshold`: Set semantic similarity threshold ### [​](https://docs.chonkie.ai/oss/experimental/chonkie-cli#key-value-parameters) Key-Value Parameters For additional or component-specific parameters, use the `*_params` options with `key=value` syntax: # Single parameter --chunker-params tokenizer=gpt2 # Multiple parameters (repeat the option) --chunker-params tokenizer=gpt2 --chunker-params min_characters_per_chunk=50 # Boolean parameters --chunker-params verbose=true # Numeric parameters (automatically converted) --chunker-params chunk_size=512 --chunker-params threshold=0.8 **Type Conversion**: Parameters are automatically converted: * `true`/`false` → boolean * `none`/`null` → None * Numeric strings → int or float * Other strings → string **Parameter Precedence**: Explicit options (like `--chunk-size`) override values in `--chunker-params` if both are provided. [​](https://docs.chonkie.ai/oss/experimental/chonkie-cli#tips) Tips ---------------------------------------------------------------------- * Use `--help` on any command to see full options: `chonkie pipeline --help`. * Directory processing recursively walks subdirectories. * Output is printed to stdout by default unless a handshaker is specified. * Combine explicit parameters with `*_params` for maximum flexibility. * Check component documentation for available parameters for each chunker, chef, refiner, or handshaker. Was this page helpful? YesNo [Code Chunker\ \ Previous](https://docs.chonkie.ai/oss/experimental/code-chunker) [SDPM Chunker (Legacy)\ \ Next](https://docs.chonkie.ai/oss/chunkers/sdpm-chunker) ⌘I --- # Chonkie Documentation > Documentation Index > ------------------- > > Fetch the complete documentation index at: [/llms.txt](https://docs.chonkie.ai/llms.txt) > > Use this file to discover all available pages before exploring further. [Skip to main content](https://docs.chonkie.ai/oss/chunkers/table-chunker#content-area) The `TableChunker` splits large markdown or HTML tables into smaller, manageable chunks by row, always preserving the header. This is especially useful for processing, indexing, or embedding tabular data in LLM and RAG pipelines. [​](https://docs.chonkie.ai/oss/chunkers/table-chunker#api-reference) API Reference -------------------------------------------------------------------------------------- Use the `recursive` endpoint to access table chunking functionality. On the API, the table chunker operates as part of the recursive chunker, allowing you to process documents containing inline tables while ensuring that table structures remain intact across chunk boundaries. [​](https://docs.chonkie.ai/oss/chunkers/table-chunker#installation) Installation ------------------------------------------------------------------------------------ TableChunker is included in the base installation of Chonkie. No additional dependencies are required. For installation instructions, see the [Installation Guide](https://docs.chonkie.ai/oss/installation) . [​](https://docs.chonkie.ai/oss/chunkers/table-chunker#initialization) Initialization ---------------------------------------------------------------------------------------- * Python * JavaScript row chunker token chunker from chonkie import TableChunker # Basic initialization custom parameters chunker = TableChunker( tokenizer="row", # Chunk by rows, valid only for TableChunker chunk_size=3 # Maximum number of rows per chunk (not including header) ) row chunker token chunker import { TableChunker } from "@chonkiejs/core"; // Basic initialization with custom parameters const chunker = await TableChunker.create({ tokenizer: "row", // Chunk by rows, valid only for TableChunker chunkSize: 3 // Maximum number of rows per chunk (not including header) }); [​](https://docs.chonkie.ai/oss/chunkers/table-chunker#parameters) Parameters -------------------------------------------------------------------------------- [​](https://docs.chonkie.ai/oss/chunkers/table-chunker#param-tokenizer) tokenizer Union\[ Literal\["row", "character"\], str, Callable\[\[str\], int\], Any\] default:"row" Tokenizer to use. Default is “row”. Can be a string identifier (“row”, “character”, “word”, “gpt2”, “byte”, etc.) or a tokenizer instance. [​](https://docs.chonkie.ai/oss/chunkers/table-chunker#param-chunk-size) chunk\_size int default:"3" Maximum number of rows (if tokenizer=“row”) or tokens/characters per chunk. [​](https://docs.chonkie.ai/oss/chunkers/table-chunker#usage) Usage ---------------------------------------------------------------------- * Python * JavaScript Markdown (Row-Based) Markdown (Token-Based) HTML Tables from chonkie import TableChunker table = """ | Name | Age | City | |--------|-----|----------| | Alice | 30 | New York | | Bob | 25 | London | | Carol | 28 | Paris | | Dave | 35 | Berlin | """ chunker = TableChunker(tokenizer="row", chunk_size=3) chunks = chunker.chunk(table) for chunk in chunks: print(chunk.text) # Each chunk is a valid markdown table segment, always including the header. For the example above and `chunk_size=3`, you might get: # >>> # | Name | Age | City | # |--------|-----|----------| # | Alice | 30 | New York | # | Bob | 25 | London | # | Carol | 28 | Paris | # | Name | Age | City | # |--------|-----|----------| # | Dave | 35 | Berlin | Markdown (Row-Based) Markdown (Token-Based) HTML Tables import { TableChunker } from "@chonkiejs/core"; const table = ` | Name | Age | City | |--------|-----|----------| | Alice | 30 | New York | | Bob | 25 | London | | Carol | 28 | Paris | | Dave | 35 | Berlin | `; const chunker = await TableChunker.create({ tokenizer: "row", chunkSize: 3 }); const chunks = await chunker.chunk(table); for (const chunk of chunks) { console.log(chunk.text); } [​](https://docs.chonkie.ai/oss/chunkers/table-chunker#methods) Methods -------------------------------------------------------------------------- * `chunk(table: str) -> list[Chunk]`: Chunk a markdown table string. * `chunk_document(document: Document) -> Document`: Chunk all tables in a `MarkdownDocument`. [​](https://docs.chonkie.ai/oss/chunkers/table-chunker#notes) Notes ---------------------------------------------------------------------- * Supports both standard Markdown pipe tables and HTML `
` elements. * Requires at least a header, separator, and one data row (for Markdown) or at least one `` data row for HTML tables (with optional `` and `` structure). * If the table fits within the chunk size, it is returned as a single chunk. * For advanced use, pass a custom tokenizer for token-based chunking. * * * See also: [Chunkers Overview](https://docs.chonkie.ai/oss/chunkers/overview) Was this page helpful? YesNo [Slumber Chunker\ \ Previous](https://docs.chonkie.ai/oss/chunkers/slumber-chunker) [TeraflopAI Chunker\ \ Next](https://docs.chonkie.ai/oss/chunkers/teraflopai-chunker) ⌘I --- # Chonkie Documentation > Documentation Index > ------------------- > > Fetch the complete documentation index at: [/llms.txt](https://docs.chonkie.ai/llms.txt) > > Use this file to discover all available pages before exploring further. [Skip to main content](https://docs.chonkie.ai/oss/installation#content-area) Chonkie follows a modular approach to dependencies, keeping the base installation lightweight while allowing you to add extra features as needed. [​](https://docs.chonkie.ai/oss/installation#basic-installation) Basic Installation -------------------------------------------------------------------------------------- For basic token and sentence chunking capabilities. ### [​](https://docs.chonkie.ai/oss/installation#python) Python pip uv pip install chonkie This installs our basic chunkers, plus the Python API SDK. To use advanced features locally, skip ahead to [Installation Options](https://docs.chonkie.ai/oss/installation#installation-options) ### [​](https://docs.chonkie.ai/oss/installation#javascript) JavaScript Install the core package for local chunking npm pnpm bun yarn npm install @chonkiejs/core To use custom tokenizers, install the `@chonkiejs/token` package npm pnpm bun yarn npm install @chonkiejs/token To use the API, install the `@chonkiejs/cloud` package npm pnpm bun yarn npm install @chonkiejs/cloud [​](https://docs.chonkie.ai/oss/installation#installation-options) Installation Options ------------------------------------------------------------------------------------------ Chonkie provides several installation options to match your specific needs: Python JavaScript # Basic installation (TokenChunker, SentenceChunker, RecursiveChunker) pip install chonkie # For Hugging Face Hub support pip install "chonkie[hub]" # For visualization support (e.g., rich text output) pip install "chonkie[viz]" # For the default semantic provider support (includes Model2Vec) pip install "chonkie[semantic]" # For OpenAI embeddings support pip install "chonkie[openai]" # For Cohere embeddings support pip install "chonkie[cohere]" # For Jina embeddings support pip install "chonkie[jina]" # For SentenceTransformer embeddings support (required by LateChunker) pip install "chonkie[st]" # For CodeChunker support pip install "chonkie[code]" # For NeuralChunker support (BERT-based) pip install "chonkie[neural]" # For SlumberChunker support (Genie/LLM interface) pip install "chonkie[genie]" # For Groq Genie support (fast inference) pip install "chonkie[groq]" # For Cerebras Genie support (fastest inference) pip install "chonkie[cerebras]" # For installing multiple features together pip install "chonkie[st, code, genie]" # For all features pip install "chonkie[all]" [​](https://docs.chonkie.ai/oss/installation#chunker-availability) Chunker Availability ------------------------------------------------------------------------------------------ The following table shows which chunkers are available with different installation options: | Chunker | Default | embeddings | ’all’ | Chonkie JS | API | | --- | --- | --- | --- | --- | --- | | TokenChunker | | | | | | | FastChunker | | | | | | | RecursiveChunker | | | | | | | SentenceChunker | | | | | | | TableChunker | | | | | | | SemanticChunker | | | | | | | LateChunker | | | | | | | CodeChunker | | | | | | | NeuralChunker | | | | | | | SlumberChunker | | | | | | [​](https://docs.chonkie.ai/oss/installation#embeddings-availability) Embeddings Availability ------------------------------------------------------------------------------------------------ Different embedding providers are available with different installation options: | Embeddings Provider | Default | ’model2vec' | 'st' | 'openai' | 'semantic' | 'all’ | | --- | --- | --- | --- | --- | --- | --- | | Model2VecEmbeddings | | | | | | | | SentenceTransformerEmbeddings | | | | | | | | OpenAIEmbeddings | | | | | | | [​](https://docs.chonkie.ai/oss/installation#dependencies) Dependencies -------------------------------------------------------------------------- Here’s what each installation option adds: | Installation Option | Additional Dependencies | | --- | --- | | Default | tqdm, numpy, chonkie-core, tenacity | | ’hub’ | \+ huggingface-hub, jsonschema | | ’viz’ | \+ rich | | ’model2vec’ | \+ tokenizers, model2vec, numpy | | ’st’ | \+ tokenizers, sentence-transformers, accelerate | | ’openai’ | \+ openai, tiktoken, pydantic | | ’cohere’ | \+ tokenizers, cohere | | ’jina’ | \+ tokenizers | | ’semantic’ | \+ tokenizers, model2vec | | ’code’ | \+ tree-sitter, tree-sitter-language-pack, magika | | ’neural’ | \+ transformers, torch | | ’genie’ | \+ pydantic, google-genai | | ’groq’ | \+ pydantic, groq | | ’cerebras’ | \+ pydantic, cerebras-cloud-sdk | | ’litellm’ | \+ litellm, tiktoken, tokenizers | | ’all’ | all above dependencies | [​](https://docs.chonkie.ai/oss/installation#important-notes) Important Notes -------------------------------------------------------------------------------- * We provide separate `semantic` and `all` installs pre-packaged that might match other installation options breeding redundancy. This redundancy is intentional to provide users with the best experience and freedom to choose their preferred means. * The `semantic` and `all` optional installs may change in future versions, so what you download today may not be the same for tomorrow. * Installing either ‘semantic’ or ‘openai’ extras will enable SemanticChunker, as it can work with any embeddings provider. The difference is in which embedding providers are available for use with this chunker. [​](https://docs.chonkie.ai/oss/installation#logging) Logging ---------------------------------------------------------------- Chonkie logs warnings and errors by default. Control logging with the `CHONKIE_LOG` environment variable: export CHONKIE_LOG=off # Disable logging export CHONKIE_LOG=warning # Warnings and errors (default) export CHONKIE_LOG=info # More verbose export CHONKIE_LOG=debug # Everything See [Logging](https://docs.chonkie.ai/oss/utils/logging) for more details. Was this page helpful? YesNo [Get Started with Chonkie\ \ Previous](https://docs.chonkie.ai/oss/quick-start) [Building Pipelines\ \ Next](https://docs.chonkie.ai/oss/pipelines) ⌘I --- # Chonkie Documentation > Documentation Index > ------------------- > > Fetch the complete documentation index at: [/llms.txt](https://docs.chonkie.ai/llms.txt) > > Use this file to discover all available pages before exploring further. [Skip to main content](https://docs.chonkie.ai/oss/chunkers/late-chunker#content-area) The **LateChunker** implements the late chunking strategy described in the [Late Chunking](https://arxiv.org/abs/2409.04701) paper. It builds on top of the `RecursiveChunker` and uses document-level embeddings to create more semantically rich chunk representations. Instead of generating embeddings for each chunk independently, the LateChunker first encodes the entire text into a single embedding. It then splits the text using recursive rules and derives each chunk’s embedding by averaging relevant parts of the full document embedding. This allows each chunk to carry broader contextual information, improving retrieval performance in RAG systems. [​](https://docs.chonkie.ai/oss/chunkers/late-chunker#api-reference) API Reference ------------------------------------------------------------------------------------- To use the `LateChunker` via the API, check out the [API reference documentation](https://docs.chonkie.ai/api/chunkers/late-chunker) . [​](https://docs.chonkie.ai/oss/chunkers/late-chunker#installation) Installation ----------------------------------------------------------------------------------- LateChunker requires the `sentence-transformers` library to be installed, and currently only supports SentenceTransformer models. You can install it with: The LateChunker uses `RecursiveRules` to determine how to chunk the text. The rules are a list of `RecursiveLevel` objects, which define the delimiters and whitespace rules for each level of the recursive tree. Find more information about the rules in the [Additional Information](https://docs.chonkie.ai/oss/chunkers/late-chunker#additional-information) section. pip install "chonkie[st]" For installation instructions, see the [Installation Guide](https://docs.chonkie.ai/oss/installation) . [​](https://docs.chonkie.ai/oss/chunkers/late-chunker#initialization) Initialization --------------------------------------------------------------------------------------- from chonkie import LateChunker chunker = LateChunker( embedding_model="nomic-ai/modernbert-embed-base", chunk_size=2048, rules=RecursiveRules(), min_characters_per_chunk=24, ) You can also initialize the LateChunker using a recipe. Recipes are pre-defined rules for common chunking tasks. Find all available recipes on our Hugging Face Hub [here](https://huggingface.co/datasets/chonkie-ai/recipes) . from chonkie import LateChunker # Initialize the late chunker to chunk Markdown chunker = LateChunker.from_recipe("markdown", lang="en") # Initialize the late chunker to chunk Hindi texts chunker = LateChunker.from_recipe(lang="hi") [​](https://docs.chonkie.ai/oss/chunkers/late-chunker#parameters) Parameters ------------------------------------------------------------------------------- [​](https://docs.chonkie.ai/oss/chunkers/late-chunker#param-embedding-model) embedding\_model str default:"nomic-ai/modernbert-embed-base" SentenceTransformer model to use for embedding [​](https://docs.chonkie.ai/oss/chunkers/late-chunker#param-chunk-size) chunk\_size int default:"2048" Maximum number of tokens per chunk [​](https://docs.chonkie.ai/oss/chunkers/late-chunker#param-rules) rules RecursiveRules default:"RecursiveRules()" Rules to use for chunking [​](https://docs.chonkie.ai/oss/chunkers/late-chunker#param-min-characters-per-chunk) min\_characters\_per\_chunk int default:"24" Minimum number of characters per sentence [​](https://docs.chonkie.ai/oss/chunkers/late-chunker#usage) Usage --------------------------------------------------------------------- ### [​](https://docs.chonkie.ai/oss/chunkers/late-chunker#single-text-chunking) Single Text Chunking text = """First paragraph about a specific topic. Second paragraph continuing the same topic. Third paragraph switching to a different topic. Fourth paragraph expanding on the new topic.""" chunks = chunker(text) for chunk in chunks: print(f"Chunk text: {chunk.text}") print(f"Token count: {chunk.token_count}") print(f"Embedding shape: {chunk.embedding.shape}") ### [​](https://docs.chonkie.ai/oss/chunkers/late-chunker#batch-chunking) Batch Chunking texts = [\ "First document about topic A...",\ "Second document about topic B..."\ ] batch_chunks = chunker(texts) for chunk in batch_chunks: print(f"Chunk text: {chunk.text}") print(f"Token count: {chunk.token_count}") print(f"Embedding shape: {chunk.embedding.shape}") [​](https://docs.chonkie.ai/oss/chunkers/late-chunker#return-type) Return Type --------------------------------------------------------------------------------- LateChunker returns chunks as `Chunk` objects: @dataclass class Chunk: text: str # The chunk text start_index: int # Starting position in original text end_index: int # Ending position in original text token_count: int # Number of tokens in chunk context: Optional[Context] = None # Optional context metadata embedding: Union[list[float], "np.ndarray", None] = None # Optional embedding vector As of version 1.3.0, LateChunker returns the base `Chunk` type instead of the specialized `LateChunk` type. The embedding is automatically populated by the LateChunker during the chunking process. [​](https://docs.chonkie.ai/oss/chunkers/late-chunker#additional-information) Additional Information ------------------------------------------------------------------------------------------------------- LateChunker uses the `RecursiveRules` class to determine the chunking rules. The rules are a list of `RecursiveLevel` objects, which define the delimiters and whitespace rules for each level of the recursive tree. @dataclass class RecursiveRules: rules: list[RecursiveLevel] @dataclass class RecursiveLevel: delimiters: Union[None, str, list[str]] whitespace: bool = False include_delim: Optional[Literal["prev", "next"]] # Whether to include the delimiter in the previous chunk or the next chunk. You can pass in custom rules to the LateChunker, or use the default ones. Default rules are designed to be a good starting point for most documents, but you can customize them to your needs. `RecursiveLevel` expects the list of custom delimiters to **not** include whitespace. If whitespace as a delimiter is required, you can set the `whitespace` parameter in the `RecursiveLevel` class to True. Note that if `whitespace = True`, you cannot pass a list of custom delimiters. Was this page helpful? YesNo [Fast Chunker\ \ Previous](https://docs.chonkie.ai/oss/chunkers/fast-chunker) [Neural Chunker\ \ Next](https://docs.chonkie.ai/oss/chunkers/neural-chunker) ⌘I --- # Chonkie Documentation > Documentation Index > ------------------- > > Fetch the complete documentation index at: [/llms.txt](https://docs.chonkie.ai/llms.txt) > > Use this file to discover all available pages before exploring further. [Skip to main content](https://docs.chonkie.ai/oss/quick-start#content-area) Using Chonkie takes two simple steps: First, install the package. Next, start chunking! This page covers Chonkie Open Source. To get started with our API, visit the [API Reference](https://docs.chonkie.ai/api/common/introduction) . [​](https://docs.chonkie.ai/oss/quick-start#installation) Installation ------------------------------------------------------------------------- ### [​](https://docs.chonkie.ai/oss/quick-start#python) Python pip uv pip install chonkie Want more features? Install everything with `pip install "chonkie[all]"`. See [Installation](https://docs.chonkie.ai/oss/installation) for more options. ### [​](https://docs.chonkie.ai/oss/quick-start#javascript) JavaScript Install the core package for local chunking npm pnpm bun yarn npm install @chonkiejs/core Chonkie JS provides local support for TokenChunker, SentenceChunker, RecursiveChunker, FastChunker, TableChunker, SemanticChunker, and CodeChunker. Other chunkers are available through the API. [​](https://docs.chonkie.ai/oss/quick-start#chonk-) CHONK! 🦛✨ ----------------------------------------------------------------- Python JavaScript # First import the chunker you want from Chonkie from chonkie import TokenChunker # Initialize the chunker chunker = TokenChunker() # defaults to using character tokenizer # Here's some text to chunk text = """Woah! Chonkie, the chunking library is so cool!""" # Chunk some text chunks = chunker(text) # Access chunks for chunk in chunks: print(f"Chunk: {chunk.text}") print(f"Tokens: {chunk.token_count}") Was this page helpful? YesNo [Installation\ \ Next](https://docs.chonkie.ai/oss/installation) ⌘I --- # Chonkie Documentation > Documentation Index > ------------------- > > Fetch the complete documentation index at: [/llms.txt](https://docs.chonkie.ai/llms.txt) > > Use this file to discover all available pages before exploring further. [Skip to main content](https://docs.chonkie.ai/oss/chunkers/slumber-chunker#content-area) Meet the `SlumberChunker` – Chonkie’s first **agentic chunker**! This isn’t your average chunker; it uses the reasoning power of large generative models (LLMs) to understand your text deeply and create truly S-tier chunks. [​](https://docs.chonkie.ai/oss/chunkers/slumber-chunker#api-reference) API Reference ---------------------------------------------------------------------------------------- To use the `SlumberChunker` via the API, check out the [API reference documentation](https://docs.chonkie.ai/api/chunkers/slumber-chunker) . [​](https://docs.chonkie.ai/oss/chunkers/slumber-chunker#introducing-genie-) Introducing Genie! 🧞 ----------------------------------------------------------------------------------------------------- The magic behind `SlumberChunker` is **Genie**, Chonkie’s interface for integrating generative models and APIs. Genie allows `SlumberChunker` to intelligently analyze text structure, identify optimal split points, and even summarize or rephrase content for the best possible chunk quality. **Available Genies:** * `GeminiGenie` - Google Gemini APIs * `OpenAIGenie` - OpenAI APIs (also works with OpenAI-compatible providers) * `AzureOpenAIGenie` - Azure OpenAI APIs * `GroqGenie` - Fast inference on Groq hardware * `CerebrasGenie` - Fastest inference on Cerebras hardware Requires \[genie\] Install -------------------------- To unleash the power of SlumberChunker and Genie, you need the `[genie]` optional install. This includes the necessary libraries to connect to various generative model APIs. pip install "chonkie[genie]" [​](https://docs.chonkie.ai/oss/chunkers/slumber-chunker#installation) Installation -------------------------------------------------------------------------------------- As mentioned, SlumberChunker requires the `[genie]` optional install: pip install "chonkie[genie]" For general installation instructions, see the [Installation Guide](https://docs.chonkie.ai/oss/installation) . [​](https://docs.chonkie.ai/oss/chunkers/slumber-chunker#initialization) Initialization ------------------------------------------------------------------------------------------ from chonkie import SlumberChunker from chonkie.genie import GeminiGenie # Optional: Initialize Genie genie = GeminiGenie("gemini-3-pro-preview") # Basic initialization chunker = SlumberChunker( genie=genie, # Genie interface to use tokenizer="character", # Default tokenizer (or use "gpt2", etc.) chunk_size=1024, # Maximum chunk size candidate_size=128, # How many tokens Genie looks at for potential splits min_characters_per_chunk=24, # Minimum number of characters per chunk verbose=True # See the progress bar for the chunking process ) # You can also rely on default Genie setup if configured globally # chunker = SlumberChunker() # Uses default Genie if available [​](https://docs.chonkie.ai/oss/chunkers/slumber-chunker#parameters) Parameters ---------------------------------------------------------------------------------- [​](https://docs.chonkie.ai/oss/chunkers/slumber-chunker#param-genie) genie Optional\[BaseGenie\] default:"None" An instance of a Genie interface (e.g., `GeminiGenie`). If `None`, tries to load a default Genie configuration, which is `GeminiGenie("gemini-3-pro-preview")` [​](https://docs.chonkie.ai/oss/chunkers/slumber-chunker#param-tokenizer) tokenizer Union\[str, Callable, Any\] default:"character" Tokenizer or token counting function used for initial splitting and size estimation. [​](https://docs.chonkie.ai/oss/chunkers/slumber-chunker#param-chunk-size) chunk\_size int default:"1024" The target maximum number of tokens per chunk. Genie will try to adhere to this. [​](https://docs.chonkie.ai/oss/chunkers/slumber-chunker#param-rules) rules RecursiveRules default:"RecursiveRules()" Initial recursive rules used to generate candidate split points before Genie refines them. See [RecursiveChunker](https://docs.chonkie.ai/oss/chunkers/recursive-chunker#additional-information) for details. [​](https://docs.chonkie.ai/oss/chunkers/slumber-chunker#param-candidate-size) candidate\_size int default:"128" The number of tokens around a potential split point that Genie examines to make its decision. [​](https://docs.chonkie.ai/oss/chunkers/slumber-chunker#param-min-characters-per-chunk) min\_characters\_per\_chunk int default:"24" Minimum number of characters required for a chunk to be considered valid. [​](https://docs.chonkie.ai/oss/chunkers/slumber-chunker#param-verbose) verbose bool default:"True" If `True`, prints detailed information about Genie’s decision-making process during chunking. Useful for debugging! [​](https://docs.chonkie.ai/oss/chunkers/slumber-chunker#usage) Usage ------------------------------------------------------------------------ ### [​](https://docs.chonkie.ai/oss/chunkers/slumber-chunker#single-text-chunking) Single Text Chunking text = """Complex document with interwoven ideas. Section 1 introduces concept A. Section 2 discusses concept B, but references A frequently. Section 3 concludes by merging A and B. Traditional chunkers might struggle here.""" # Assuming 'chunker' is initialized as shown above chunks = chunker.chunk(text) for chunk in chunks: print(f"Chunk text: {chunk.text}") print(f"Token count: {chunk.token_count}") print(f"Start index: {chunk.start_index}") print(f"End index: {chunk.end_index}") # SlumberChunk might have additional metadata from Genie ### [​](https://docs.chonkie.ai/oss/chunkers/slumber-chunker#batch-chunking) Batch Chunking texts = [\ "First document requiring nuanced splitting...",\ "Second document where agentic understanding helps..."\ ] batch_chunks = chunker.chunk_batch(texts) # Note: Batch processing might be slower due to LLM calls for doc_chunks in batch_chunks: for chunk in doc_chunks: print(f"Chunk: {chunk.text}") ### [​](https://docs.chonkie.ai/oss/chunkers/slumber-chunker#using-as-a-callable) Using as a Callable # Single text chunks = chunker("Let Genie decide the best way to CHONK this...") # Multiple texts batch_chunks = chunker(["Text 1...", "Text 2..."]) [​](https://docs.chonkie.ai/oss/chunkers/slumber-chunker#return-type) Return Type ------------------------------------------------------------------------------------ SlumberChunker returns chunks as `Chunk` objects. from dataclasses import dataclass from typing import Optional, Union @dataclass class Chunk: text: str # The chunk text start_index: int # Starting position in original text end_index: int # Ending position in original text token_count: int # Number of tokens in chunk context: Optional[str] = None # Optional overlap context text embedding: Union[list[float], "np.ndarray", None] = None # Optional embedding vector Was this page helpful? YesNo [Sentence Chunker\ \ Previous](https://docs.chonkie.ai/oss/chunkers/sentence-chunker) [Table Chunker\ \ Next](https://docs.chonkie.ai/oss/chunkers/table-chunker) ⌘I --- # Chonkie Documentation > Documentation Index > ------------------- > > Fetch the complete documentation index at: [/llms.txt](https://docs.chonkie.ai/llms.txt) > > Use this file to discover all available pages before exploring further. [Skip to main content](https://docs.chonkie.ai/oss/refinery/overlap-refinery#content-area) The `OverlapRefinery` enhances chunks by incorporating context from neighboring chunks. This is useful for tasks where maintaining contextual continuity between chunks is important, such as question answering or summarization over long documents. It can add context as a prefix (from the preceding chunk) or a suffix (from the next chunk). [​](https://docs.chonkie.ai/oss/refinery/overlap-refinery#api-reference) API Reference ----------------------------------------------------------------------------------------- To use the `OverlapRefinery` via the API, check out the [API reference documentation](https://docs.chonkie.ai/api/refineries/overlap) . [​](https://docs.chonkie.ai/oss/refinery/overlap-refinery#initialization) Initialization ------------------------------------------------------------------------------------------- To use the `OverlapRefinery`, initialize it with the desired parameters. You can specify a tokenizer, context size, overlap mode, method, and other options. from chonkie import OverlapRefinery # Initialize with default character-level overlap (25% context size) overlap_refinery = OverlapRefinery() # Initialize with a specific tokenizer and context size overlap_refinery_token = OverlapRefinery( tokenizer="character", # Default tokenizer (or use "gpt2", etc.) context_size=0.25, # The size of the context to add to the chunks. method="prefix", # Add context from the previous chunk merge=True # Merge context directly into chunk text ) # Initialize with justified method (context from both sides) overlap_refinery_justified = OverlapRefinery( tokenizer="character", context_size=0.25, method="justified", # Add context from both previous and next chunks merge=True ) # Initialize for recursive overlap based on rules from chonkie import RecursiveRules, RecursiveLevel rules = RecursiveRules( levels=[\ RecursiveLevel(delimiters=["\n\n"], include_delim="prev"),\ RecursiveLevel(delimiters=["."], include_delim="prev"),\ RecursiveLevel(whitespace=True)\ ] ) overlap_refinery_recursive = OverlapRefinery( tokenizer="character", context_size=0.25, mode="recursive", rules=rules, method="suffix" ) [​](https://docs.chonkie.ai/oss/refinery/overlap-refinery#usage) Usage ------------------------------------------------------------------------- Use the `OverlapRefinery` object as a callable or use the `refine` method to add overlapping context to your chunks. from chonkie import TokenChunker, OverlapRefinery test_string = "This is the first sentence. This is the second sentence, providing context. This is the third sentence, which needs context from the second." chunker = TokenChunker() chunks = chunker(test_string) # Initialize refinery to add suffix overlap overlap_refinery = OverlapRefinery( tokenizer="character", context_size=0.5, method="suffix", merge=True ) refined_chunks = overlap_refinery(chunks) [​](https://docs.chonkie.ai/oss/refinery/overlap-refinery#parameters) Parameters ----------------------------------------------------------------------------------- [​](https://docs.chonkie.ai/oss/refinery/overlap-refinery#param-tokenizer) tokenizer Union\[str, Callable, Any\] default:"\\"character\\"" The tokenizer to use for calculating overlap size. Can be a string identifier (e.g., “character”, “word”, “gpt2”), a callable, or a `chonkie.Tokenizer` instance. Defaults to “character”. [​](https://docs.chonkie.ai/oss/refinery/overlap-refinery#param-context-size) context\_size Union\[int, float\] default:"0.25" The size of the overlap context. If an `int`, it’s the absolute number of tokens. If a `float` (between 0 and 1), it’s the fraction of the maximum chunk token count. [​](https://docs.chonkie.ai/oss/refinery/overlap-refinery#param-mode) mode Literal\["token", "recursive"\] default:"\\"token\\"" The mode for calculating overlap. `"token"` uses the tokenizer directly. `"recursive"` uses hierarchical splitting based on `rules`. [​](https://docs.chonkie.ai/oss/refinery/overlap-refinery#param-method) method Literal\["suffix", "prefix", "justified"\] default:"\\"suffix\\"" The method for adding context. `"suffix"` adds context from the _next_ chunk to the end of the current chunk. `"prefix"` adds context from the _previous_ chunk to the beginning of the current chunk. `"justified"` adds context from both the previous and next chunks - for middle chunks it includes context from both sides, while first and last chunks get context from the single adjacent chunk. [​](https://docs.chonkie.ai/oss/refinery/overlap-refinery#param-rules) rules RecursiveRules default:"RecursiveRules()" The rules used for splitting text when `mode` is `"recursive"`. Defines delimiters and behavior at different hierarchical levels. See `chonkie.types.RecursiveRules`. [​](https://docs.chonkie.ai/oss/refinery/overlap-refinery#param-merge) merge bool default:"True" If `True`, the calculated context is directly prepended (for `prefix`) or appended (for `suffix`) to the `chunk.text`. If `False`, the context is stored in `chunk.context` attribute without modifying `chunk.text`. [​](https://docs.chonkie.ai/oss/refinery/overlap-refinery#param-inplace) inplace bool default:"True" If `True`, modifies the input list of chunks directly. If `False`, returns a new list of modified chunks. Was this page helpful? YesNo [Refinery Overview\ \ Previous](https://docs.chonkie.ai/oss/refinery/overview) [Embeddings Refinery\ \ Next](https://docs.chonkie.ai/oss/refinery/embeddings-refinery) ⌘I --- # Chonkie Documentation > Documentation Index > ------------------- > > Fetch the complete documentation index at: [/llms.txt](https://docs.chonkie.ai/llms.txt) > > Use this file to discover all available pages before exploring further. [Skip to main content](https://docs.chonkie.ai/oss/chunkers/code-chunker#content-area) The `CodeChunker` splits code into chunks based on its structure, leveraging Abstract Syntax Trees (ASTs) to create contextually relevant segments. [​](https://docs.chonkie.ai/oss/chunkers/code-chunker#overview) Overview --------------------------------------------------------------------------- * Supports 165+ languages * Powered by [tree-sitter-language-pack](https://github.com/Goldziher/tree-sitter-language-pack) * Auto language detection via [Magika](https://github.com/google/magika) , a language detection library made by Google [​](https://docs.chonkie.ai/oss/chunkers/code-chunker#supported-languages) Supported Languages ------------------------------------------------------------------------------------------------- Show all supported languages Each language is identified by the **key** used with `get_language(key)` and `get_parser(key)`. ### [​](https://docs.chonkie.ai/oss/chunkers/code-chunker#general-purpose-programming-languages) General-Purpose Programming Languages | Language | Key | License | | --- | --- | --- | | ActionScript | `actionscript` | MIT | | Ada | `ada` | MIT | | Agda | `agda` | MIT | | C | `c` | MIT | | C++ | `cpp` | MIT | | C# | `csharp` | MIT | | Dart | `dart` | MIT | | Go | `go` | MIT | | Java | `java` | MIT | | JavaScript | `javascript` | MIT | | Julia | `julia` | MIT | | Kotlin | `kotlin` | MIT | | Nim | `nim` | MPL-2.0 | | OCaml | `ocaml/ocaml_interface` | MIT | | Perl | `perl` | Artistic-2.0 | | Python | `python` | MIT | | R | `r` | MIT | | Ruby | `ruby` | MIT | | Rust | `rust` | MIT | | Scala | `scala` | MIT | | Swift | `swift` | MIT | | TypeScript | `typescript` | MIT | | Zig | `zig` | MIT | ### [​](https://docs.chonkie.ai/oss/chunkers/code-chunker#web-ui-&-markup) Web, UI & Markup | Language | Key | License | | --- | --- | --- | | HTML | `html` | MIT | | CSS | `css` | MIT | | SCSS | `scss` | MIT | | Astro | `astro` | MIT | | Vue | `vue` | MIT | | Svelte | `svelte` | MIT | | TSX | `tsx` | MIT | | Markdown | `markdown` | MIT | | Markdown Inline | `markdown_inline` | MIT | | Mermaid | `mermaid` | MIT | | XML | `xml` | MIT | | YAML | `yaml` | MIT | ### [​](https://docs.chonkie.ai/oss/chunkers/code-chunker#config-build-&-devops) Config, Build & DevOps | Language | Key | License | | --- | --- | --- | | Bash | `bash` | MIT | | Dockerfile | `dockerfile` | MIT | | Git Ignore | `gitignore` | MIT | | Git Commit | `gitcommit` | WTFPL | | Make | `make` | MIT | | Ninja | `ninja` | MIT | | Meson | `meson` | MIT | | Prisma | `prisma` | MIT | | Requirements | `requirements` | MIT | ### [​](https://docs.chonkie.ai/oss/chunkers/code-chunker#systems-gpu-&-low-level) Systems, GPU & Low-level | Language | Key | License | | --- | --- | --- | | ASM | `asm` | MIT | | CUDA | `cuda` | MIT | | GLSL | `glsl` | MIT | | HLSL | `hlsl` | MIT | | LLVM | `llvm` | MIT | | Verilog | `verilog` | MIT | | VHDL | `vhdl` | MIT | | WGSL | `wgsl` | MIT | | WAST / WAT | `wasm` | Apache-2.0 + LLVM | [​](https://docs.chonkie.ai/oss/chunkers/code-chunker#api-reference) API Reference ------------------------------------------------------------------------------------- To use the `CodeChunker` via the API, check out the [API reference documentation](https://docs.chonkie.ai/api/chunkers/code-chunker) . [​](https://docs.chonkie.ai/oss/chunkers/code-chunker#installation) Installation ----------------------------------------------------------------------------------- CodeChunker requires additional dependencies for code parsing. You can install it with: pip install "chonkie[code]" For installation instructions, see the [Installation Guide](https://docs.chonkie.ai/oss/installation) . [​](https://docs.chonkie.ai/oss/chunkers/code-chunker#initialization) Initialization --------------------------------------------------------------------------------------- Basic initialization Auto Custom tokenizer from chonkie import CodeChunker chunker = CodeChunker( language="python", # Specify the programming language tokenizer="character", # Default tokenizer (or use "gpt2", etc.) chunk_size=2048, # Maximum tokens per chunk include_nodes=False # Optionally include AST nodes in output ) Automatic language detection using Magika can impact performance. For best results, specify the language directly via the `language` parameter. [​](https://docs.chonkie.ai/oss/chunkers/code-chunker#parameters) Parameters ------------------------------------------------------------------------------- [​](https://docs.chonkie.ai/oss/chunkers/code-chunker#param-language) language str required The programming language of the code. Accepts languages supported by `tree-sitter-language-pack`. [​](https://docs.chonkie.ai/oss/chunkers/code-chunker#param-tokenizer) tokenizer Union\[str, Callable, Any\] default:"character" Tokenizer or token counting function to use for measuring chunk size. [​](https://docs.chonkie.ai/oss/chunkers/code-chunker#param-chunk-size) chunk\_size int default:"2048" Maximum number of tokens per chunk. [​](https://docs.chonkie.ai/oss/chunkers/code-chunker#param-include-nodes) include\_nodes bool default:"False" Whether to include AST node information (Note: with the base Chunk type, node information is not stored). [​](https://docs.chonkie.ai/oss/chunkers/code-chunker#usage) Usage --------------------------------------------------------------------- ### [​](https://docs.chonkie.ai/oss/chunkers/code-chunker#single-code-chunking) Single Code Chunking * Python * JavaScript code = """ def hello_world(): print("Hello, Chonkie!") class MyClass: def __init__(self): self.value = 42 """ chunks = chunker.chunk(code) for chunk in chunks: print(f"Chunk text: {chunk.text}") print(f"Token count: {chunk.token_count}") const code = ` def hello_world(): print("Hello, Chonkie!") class MyClass: def __init__(self): self.value = 42 `; const chunks = await chunker.chunk(code); for (const chunk of chunks) { console.log(`Chunk text: ${chunk.text}`); console.log(`Token count: ${chunk.tokenCount}`); } ### [​](https://docs.chonkie.ai/oss/chunkers/code-chunker#batch-chunking) Batch Chunking * Python * JavaScript codes = [\ "def func1():\n pass",\ "const x = 10;\nfunction add(a, b) { return a + b; }"\ ] batch_chunks = chunker.chunk_batch(codes) for doc_chunks in batch_chunks: for chunk in doc_chunks: print(f"Chunk: {chunk.text}") const codes = [\ "def func1():\n pass",\ "const x = 10;\nfunction add(a, b) { return a + b; }"\ ]; const batchChunks = await chunker.chunkBatch(codes); for (const docChunks of batchChunks) { for (const chunk of docChunks) { console.log(`Chunk: ${chunk.text}`); } } ### [​](https://docs.chonkie.ai/oss/chunkers/code-chunker#using-as-a-callable) Using as a Callable # Single code string chunks = chunker("def greet(name):\n print(f'Hello, {name}')") # Multiple code strings batch_chunks = chunker(["int main() { return 0; }", "package main\nimport \"fmt\"\nfunc main() { fmt.Println(\"Hi\") }"]) [​](https://docs.chonkie.ai/oss/chunkers/code-chunker#return-type) Return Type --------------------------------------------------------------------------------- CodeChunker returns chunks as `Chunk` objects: @dataclass class Chunk: text: str # The chunk text (code snippet) start_index: int # Starting position in original code end_index: int # Ending position in original code token_count: int # Number of tokens in chunk context: Optional[Context] = None # Optional context metadata embedding: Union[list[float], "np.ndarray", None] = None # Optional embedding vector As of version 1.3.0, CodeChunker returns the base `Chunk` type instead of the specialized `CodeChunk` type. This simplifies integration with other chunkers and refineries. Was this page helpful? YesNo [Chunkers Overview\ \ Previous](https://docs.chonkie.ai/oss/chunkers/overview) [Fast Chunker\ \ Next](https://docs.chonkie.ai/oss/chunkers/fast-chunker) ⌘I --- # Chonkie Documentation > Documentation Index > ------------------- > > Fetch the complete documentation index at: [/llms.txt](https://docs.chonkie.ai/llms.txt) > > Use this file to discover all available pages before exploring further. [Skip to main content](https://docs.chonkie.ai/oss/experimental/code-chunker#content-area) The experimental CodeChunker provides advanced AST-based code parsing that goes beyond simple line-based splitting to understand and preserve code structure and semantics. **Experimental Feature**: This CodeChunker is experimental and may change significantly between versions. Use with caution in production environments. [​](https://docs.chonkie.ai/oss/experimental/code-chunker#key-features) Key Features --------------------------------------------------------------------------------------- * **AST-based parsing** using tree-sitter for accurate code understanding * **Automatic language detection** using Magika for seamless multi-language handling * **Language-specific rules** for optimal chunking based on programming language * **Intelligent grouping** of related code elements (imports, comments, classes) * **Semantic preservation** prioritizes code coherence over strict size limits * **Multi-language support** for popular programming languages * **Recursive splitting** for large code constructs when chunk size is specified [​](https://docs.chonkie.ai/oss/experimental/code-chunker#installation) Installation --------------------------------------------------------------------------------------- To use the experimental CodeChunker, you need the code dependencies: pip install chonkie[code] [​](https://docs.chonkie.ai/oss/experimental/code-chunker#supported-languages) Supported Languages ----------------------------------------------------------------------------------------------------- The experimental CodeChunker supports the following programming languages: * **Python** - Classes, functions, imports, docstrings * **TypeScript** - Functions, classes, interfaces, modules * **JavaScript** - Functions, classes, modules, JSX * **Rust** - Functions, structs, modules, traits * **Go** - Functions, structs, packages, interfaces * **Java** - Classes, methods, packages, interfaces * **C** - Functions, structs, headers * **C++** - Functions, classes, namespaces, structs * **C#** - Classes, methods, namespaces, properties * **HTML** - Tags, elements, attributes * **CSS** - Rules, selectors, properties * **Markdown** - Headers, sections, code blocks [​](https://docs.chonkie.ai/oss/experimental/code-chunker#basic-usage) Basic Usage ------------------------------------------------------------------------------------- from chonkie.experimental import CodeChunker # Create a code chunker for Python chunker = CodeChunker(language="python") # Chunk some Python code code = ''' import os from typing import List def process_files(directory: str) -> list[str]: """Process all files in a directory.""" files = [] for filename in os.listdir(directory): if filename.endswith('.py'): files.append(filename) return files class FileProcessor: def __init__(self, base_dir: str): self.base_dir = base_dir self.processed_count = 0 def process(self, filename: str) -> bool: """Process a single file.""" # Processing logic here self.processed_count += 1 return True ''' chunks = chunker.chunk(code) for i, chunk in enumerate(chunks): print(f"Chunk {i+1}:") print(chunk.text) print("---") [​](https://docs.chonkie.ai/oss/experimental/code-chunker#advanced-configuration) Advanced Configuration ----------------------------------------------------------------------------------------------------------- ### [​](https://docs.chonkie.ai/oss/experimental/code-chunker#with-chunk-size-limit) With Chunk Size Limit # Set a chunk size limit (chunks may exceed this to preserve semantics) chunker = CodeChunker( language="python", chunk_size=2048, # Target chunk size in characters tokenizer="character" ) ### [​](https://docs.chonkie.ai/oss/experimental/code-chunker#language-auto-detection) Language Auto-Detection The experimental CodeChunker can automatically detect the programming language using Magika, Google’s deep learning-based language detection model: # Let the chunker detect the language automatically chunker = CodeChunker(language="auto") # Chunk different types of code - language is detected automatically python_code = ''' def fibonacci(n): if n <= 1: return n return fibonacci(n-1) + fibonacci(n-2) ''' javascript_code = ''' function fibonacci(n) { if (n <= 1) return n; return fibonacci(n-1) + fibonacci(n-2); } ''' rust_code = ''' fn fibonacci(n: u32) -> u32 { if n <= 1 { n } else { fibonacci(n-1) + fibonacci(n-2) } } ''' # All will be chunked with appropriate language-specific rules python_chunks = chunker.chunk(python_code) # Detected as Python js_chunks = chunker.chunk(javascript_code) # Detected as JavaScript rust_chunks = chunker.chunk(rust_code) # Detected as Rust **Performance Consideration**: When using `language="auto"`, the chunker will show a warning that auto-detection may affect performance. For better performance in production, specify the language explicitly when known. ### [​](https://docs.chonkie.ai/oss/experimental/code-chunker#split-context-control) Split Context Control # Control whether to add split context information chunker = CodeChunker( language="typescript", add_split_context=True # Include context about split locations ) [​](https://docs.chonkie.ai/oss/experimental/code-chunker#understanding-chunk-behavior) Understanding Chunk Behavior ----------------------------------------------------------------------------------------------------------------------- ### [​](https://docs.chonkie.ai/oss/experimental/code-chunker#semantic-preservation) Semantic Preservation The experimental CodeChunker prioritizes semantic coherence over strict size limits: chunker = CodeChunker(language="python", chunk_size=100) # This class will likely stay together even if it exceeds 100 characters code = ''' class SmallButImportant: def __init__(self): self.value = "important" def get_value(self): return self.value ''' chunks = chunker.chunk(code) # The class will typically be kept as one chunk for semantic coherence ### [​](https://docs.chonkie.ai/oss/experimental/code-chunker#language-specific-grouping) Language-Specific Grouping Different languages have different grouping behaviors: Python Example Classes, functions, and imports are intelligently grouped # Python code is grouped by logical units python_code = ''' import numpy as np import pandas as pd def data_processor(): """Process data using pandas.""" return pd.DataFrame() class DataAnalyzer: def analyze(self, data): return np.mean(data) ''' # Likely chunks: # 1. Import statements together # 2. Function definition # 3. Class definition JavaScript Example Modules, functions, and classes are preserved // JavaScript/TypeScript grouping const code = ` import { Component } from 'react'; import { useState } from 'react'; export const MyComponent = () => { const [state, setState] = useState(null); return
{state}
; }; export class DataService { async fetchData() { return fetch('/api/data'); } } `; // Likely chunks: // 1. Import statements // 2. Component definition // 3. Class definition Rust Example Modules, structs, and implementations are grouped // Rust code grouping let rust_code = r#" use std::collections::HashMap; use serde::{Deserialize, Serialize}; #[derive(Debug, Serialize, Deserialize)] pub struct User { id: u32, name: String, } impl User { pub fn new(id: u32, name: String) -> Self { Self { id, name } } } "#; // Likely chunks: // 1. Use statements // 2. Struct definition with derives // 3. Implementation block [​](https://docs.chonkie.ai/oss/experimental/code-chunker#best-practices) Best Practices ------------------------------------------------------------------------------------------- ### [​](https://docs.chonkie.ai/oss/experimental/code-chunker#choose-appropriate-chunk-sizes) Choose Appropriate Chunk Sizes # For code analysis tasks chunker = CodeChunker(language="python", chunk_size=1024) # For embedding generation (smaller chunks often work better) chunker = CodeChunker(language="python", chunk_size=2048) # No size limit (preserve all semantic units) chunker = CodeChunker(language="python", chunk_size=None) ### [​](https://docs.chonkie.ai/oss/experimental/code-chunker#language-specific-considerations) Language-Specific Considerations # For web development files with mixed content html_chunker = CodeChunker(language="html", chunk_size=800) # For documentation with code examples md_chunker = CodeChunker(language="markdown", chunk_size=600) # For system-level code that needs precise structure c_chunker = CodeChunker(language="c", chunk_size=1200) [​](https://docs.chonkie.ai/oss/experimental/code-chunker#output-format) Output Format ----------------------------------------------------------------------------------------- Each chunk contains detailed metadata about the code structure: chunks = chunker.chunk(code) for chunk in chunks: print(f"Text: {chunk.text}") print(f"Start: {chunk.start_index}") print(f"End: {chunk.end_index}") print(f"Token count: {chunk.token_count}") [​](https://docs.chonkie.ai/oss/experimental/code-chunker#limitations) Limitations ------------------------------------------------------------------------------------- **Current Limitations**: * **Experimental status**: APIs may change between versions * **Performance**: AST parsing may be slower than simple text splitting * **Language support**: Not all programming languages are supported yet * **Size flexibility**: Chunks may significantly exceed specified size limits * **Dependencies**: Requires tree-sitter and language packs [​](https://docs.chonkie.ai/oss/experimental/code-chunker#migration-from-stable-codechunker) Migration from Stable CodeChunker --------------------------------------------------------------------------------------------------------------------------------- If migrating from the stable CodeChunker to the experimental version: # Old stable version from chonkie import CodeChunker # New experimental version from chonkie.experimental import CodeChunker # The API is similar but with enhanced capabilities chunker = CodeChunker(language="python", chunk_size=2048) [​](https://docs.chonkie.ai/oss/experimental/code-chunker#feedback-and-support) Feedback and Support ------------------------------------------------------------------------------------------------------- Since this is an experimental feature, your feedback is valuable: * **Report issues** on [GitHub](https://github.com/chonkie-inc/chonkie) * **Share use cases** to help improve the chunker * **Test with your code** and let us know what works well or needs improvement The experimental CodeChunker will eventually replace or supplement the stable CodeChunker based on community feedback and testing results. Was this page helpful? YesNo [Overview\ \ Previous](https://docs.chonkie.ai/oss/experimental/overview) [CLI\ \ Next](https://docs.chonkie.ai/oss/experimental/chonkie-cli) ⌘I --- # Chonkie Documentation > Documentation Index > ------------------- > > Fetch the complete documentation index at: [/llms.txt](https://docs.chonkie.ai/llms.txt) > > Use this file to discover all available pages before exploring further. [Skip to main content](https://docs.chonkie.ai/oss/chunkers/sentence-chunker#content-area) The `SentenceChunker` splits text into chunks while preserving complete sentences, ensuring that each chunk maintains proper sentence boundaries and context. [​](https://docs.chonkie.ai/oss/chunkers/sentence-chunker#api-reference) API Reference ----------------------------------------------------------------------------------------- To use the `SentenceChunker` via the API, check out the [API reference documentation](https://docs.chonkie.ai/api/chunkers/sentence-chunker) . [​](https://docs.chonkie.ai/oss/chunkers/sentence-chunker#installation) Installation --------------------------------------------------------------------------------------- SentenceChunker is included in the base installation of Chonkie. No additional dependencies are required. For installation instructions, see the [Installation Guide](https://docs.chonkie.ai/oss/installation) . [​](https://docs.chonkie.ai/oss/chunkers/sentence-chunker#initialization) Initialization ------------------------------------------------------------------------------------------- Python JavaScript from chonkie import SentenceChunker # Basic initialization with default parameters chunker = SentenceChunker( tokenizer="character", # Default tokenizer (or use "gpt2", etc.) chunk_size=2048, # Maximum tokens per chunk chunk_overlap=128, # Overlap between chunks min_sentences_per_chunk=1 # Minimum sentences in each chunk ) [​](https://docs.chonkie.ai/oss/chunkers/sentence-chunker#parameters) Parameters ----------------------------------------------------------------------------------- [​](https://docs.chonkie.ai/oss/chunkers/sentence-chunker#param-tokenizer) tokenizer Union\[str, Callable, Any\] default:"character" Tokenizer to use. Can be a string identifier (“character”, “word”, “byte”, “gpt2”, etc.) or a tokenizer instance [​](https://docs.chonkie.ai/oss/chunkers/sentence-chunker#param-chunk-size) chunk\_size int default:"2048" Maximum number of tokens per chunk [​](https://docs.chonkie.ai/oss/chunkers/sentence-chunker#param-chunk-overlap) chunk\_overlap int default:"0" Number of overlapping tokens between chunks [​](https://docs.chonkie.ai/oss/chunkers/sentence-chunker#param-min-sentences-per-chunk) min\_sentences\_per\_chunk int default:"1" Minimum number of sentences to include in each chunk [​](https://docs.chonkie.ai/oss/chunkers/sentence-chunker#param-min-characters-per-sentence) min\_characters\_per\_sentence int default:"12" Minimum number of characters per sentence [​](https://docs.chonkie.ai/oss/chunkers/sentence-chunker#param-approximate) approximate bool default:"False" Use approximate token counting for faster processing. This field is deprecated and will be removed in future versions. [​](https://docs.chonkie.ai/oss/chunkers/sentence-chunker#param-delim) delim Union\[str, list\[str\]\] default:"\['.', '!', '?', '\\\\n'\]" Delimiters to split sentences on [​](https://docs.chonkie.ai/oss/chunkers/sentence-chunker#param-include-delim) include\_delim Optional\[Literal\["prev", "next"\]\] default:"prev" Specify whether to include the delimiter with the previous or next chunk. [​](https://docs.chonkie.ai/oss/chunkers/sentence-chunker#usage) Usage ------------------------------------------------------------------------- ### [​](https://docs.chonkie.ai/oss/chunkers/sentence-chunker#single-text-chunking) Single Text Chunking Python JavaScript text = """This is the first sentence. This is the second sentence. And here's a third one with some additional context.""" chunks = chunker.chunk(text) for chunk in chunks: print(f"Chunk text: {chunk.text}") print(f"Token count: {chunk.token_count}") ### [​](https://docs.chonkie.ai/oss/chunkers/sentence-chunker#batch-chunking) Batch Chunking Python JavaScript texts = [\ "First document. With multiple sentences.",\ "Second document. Also with sentences. And more context."\ ] batch_chunks = chunker.chunk_batch(texts) for doc_chunks in batch_chunks: for chunk in doc_chunks: print(f"Chunk: {chunk.text}") ### [​](https://docs.chonkie.ai/oss/chunkers/sentence-chunker#using-as-a-callable) Using as a Callable # Single text chunks = chunker("First sentence. Second sentence.") # Multiple texts batch_chunks = chunker(["Text 1. More text.", "Text 2. More."]) [​](https://docs.chonkie.ai/oss/chunkers/sentence-chunker#supported-tokenizers) Supported Tokenizers ------------------------------------------------------------------------------------------------------- SentenceChunker supports multiple tokenizer backends: * **TikToken** (Recommended) import tiktoken tokenizer = tiktoken.get_encoding("gpt2") * **AutoTikTokenizer** from autotiktokenizer import AutoTikTokenizer tokenizer = AutoTikTokenizer.from_pretrained("gpt2") * **Hugging Face Tokenizers** from tokenizers import Tokenizer tokenizer = Tokenizer.from_pretrained("gpt2") * **Transformers** from transformers import AutoTokenizer tokenizer = AutoTokenizer.from_pretrained("gpt2") [​](https://docs.chonkie.ai/oss/chunkers/sentence-chunker#return-type) Return Type ------------------------------------------------------------------------------------- SentenceChunker returns chunks as `Chunk` objects: @dataclass class Chunk: text: str # The chunk text start_index: int # Starting position in original text end_index: int # Ending position in original text token_count: int # Number of tokens in chunk context: Optional[str] = None # Optional overlap context text embedding: Union[list[float], "np.ndarray", None] = None # Optional embedding vector Was this page helpful? YesNo [Semantic Chunker\ \ Previous](https://docs.chonkie.ai/oss/chunkers/semantic-chunker) [Slumber Chunker\ \ Next](https://docs.chonkie.ai/oss/chunkers/slumber-chunker) ⌘I --- # Chonkie Documentation > Documentation Index > ------------------- > > Fetch the complete documentation index at: [/llms.txt](https://docs.chonkie.ai/llms.txt) > > Use this file to discover all available pages before exploring further. [Skip to main content](https://docs.chonkie.ai/oss/chunkers/recursive-chunker#content-area) The RecursiveChunker is a chunker that recursively chunks documents into smaller chunks. It is a good choice for documents that are long but well structured, for example, a book or a research paper. [​](https://docs.chonkie.ai/oss/chunkers/recursive-chunker#api-reference) API Reference ------------------------------------------------------------------------------------------ To use the `RecursiveChunker` via the API, check out the [API reference documentation](https://docs.chonkie.ai/api/chunkers/recursive-chunker) . [​](https://docs.chonkie.ai/oss/chunkers/recursive-chunker#installation) Installation ---------------------------------------------------------------------------------------- The RecursiveChunker is included in the base installation of Chonkie. No additional dependencies are required. If you would like to use custom tokenizers in JavaScript, please install the `@chonkiejs/token` library [​](https://docs.chonkie.ai/oss/chunkers/recursive-chunker#initialization) Initialization -------------------------------------------------------------------------------------------- The RecursiveChunker uses `RecursiveRules` to determine how to chunk the text. The rules are a list of `RecursiveLevel` objects, which define the delimiters and whitespace rules for each level of the recursive tree. Find more information about the rules in the [Additional Information](https://docs.chonkie.ai/oss/chunkers/recursive-chunker#additional-information) section. Python JavaScript from chonkie import RecursiveChunker, RecursiveRules chunker = RecursiveChunker( tokenizer: Union[str, Callable, Any] = "character", chunk_size: int = 2048, rules: RecursiveRules = RecursiveRules(), min_characters_per_chunk: int = 24, ) You can also initialize the RecursiveChunker using a recipe. Recipes are pre-defined rules for common chunking tasks. Find all available recipes on our Hugging Face Hub [here](https://huggingface.co/datasets/chonkie-ai/recipes) . Recipes are supported on Python only from chonkie import RecursiveChunker # Initialize the recursive chunker to chunk Markdown chunker = RecursiveChunker.from_recipe("markdown", lang="en") # Initialize the recursive chunker to chunk Hindi texts chunker = RecursiveChunker.from_recipe(lang="hi") [​](https://docs.chonkie.ai/oss/chunkers/recursive-chunker#parameters) Parameters ------------------------------------------------------------------------------------ [​](https://docs.chonkie.ai/oss/chunkers/recursive-chunker#param-tokenizer) tokenizer Union\[str, Callable, Any\] default:"character" Tokenizer to use. Can be a string identifier or a tokenizer instance [​](https://docs.chonkie.ai/oss/chunkers/recursive-chunker#param-chunk-size-chunk-size) chunk\_size / chunkSize int default:"2048" Maximum number of tokens per chunk [​](https://docs.chonkie.ai/oss/chunkers/recursive-chunker#param-rules) rules RecursiveRules default:"RecursiveRules()" Rules to use for chunking. [​](https://docs.chonkie.ai/oss/chunkers/recursive-chunker#param-min-characters-per-chunk-min-characters-per-chunk) min\_characters\_per\_chunk / minCharactersPerChunk int default:"12" Minimum number of characters per chunk [​](https://docs.chonkie.ai/oss/chunkers/recursive-chunker#usage) Usage -------------------------------------------------------------------------- ### [​](https://docs.chonkie.ai/oss/chunkers/recursive-chunker#single-text-chunking) Single Text Chunking Python JavaScript text = """This is the first sentence. This is the second sentence. And here's a third one with some additional context.""" chunks = chunker.chunk(text) for chunk in chunks: print(f"Chunk text: {chunk.text}") print(f"Token count: {chunk.token_count}") ### [​](https://docs.chonkie.ai/oss/chunkers/recursive-chunker#batch-chunking) Batch Chunking texts = [\ "This is the first sentence. This is the second sentence.\ And here's a third one with some additional context.",\ "This is the first sentence. This is the second sentence.\ And here's a third one with some additional context.",\ ] chunks = chunker.chunk_batch(texts) for chunk in chunks: print(f"Chunk text: {chunk.text}") print(f"Token count: {chunk.token_count}") ### [​](https://docs.chonkie.ai/oss/chunkers/recursive-chunker#using-as-a-callable) Using as a Callable # Single text chunks = chunker("This is the first sentence. This is the second sentence.") # Multiple texts batch_chunks = chunker(["Text 1. More text.", "Text 2. More."]) [​](https://docs.chonkie.ai/oss/chunkers/recursive-chunker#return-type) Return Type -------------------------------------------------------------------------------------- The RecursiveChunker returns chunks as `Chunk` objects: Python JavaScript @dataclass class Chunk: text: str # The chunk text start_index: int # Starting position in original text end_index: int # Ending position in original text token_count: int # Number of tokens in chunk context: Optional[str] = None # Optional overlap context text embedding: Union[list[float], "np.ndarray", None] = None # Optional embedding vector [​](https://docs.chonkie.ai/oss/chunkers/recursive-chunker#additional-information) Additional Information ------------------------------------------------------------------------------------------------------------ The RecursiveChunker uses the `RecursiveRules` class to determine the chunking rules. The rules are a list of `RecursiveLevel` objects, which define the delimiters and whitespace rules for each level of the recursive tree. Python JavaScript @dataclass class RecursiveRules: rules: list[RecursiveLevel] @dataclass class RecursiveLevel: delimiters: Optional[Union[str, list[str]]] whitespace: bool = False include_delim: Optional[Literal["prev", "next"]]) # Whether to include the delimiter in the previous chunk or the next chunk. You can pass in custom rules to the RecursiveChunker, or use the default rules. The default rules are designed to be a good starting point for most documents, but you can customize them to your needs. `RecursiveLevel` expects the list of custom delimiters to **not** include whitespace. If whitespace as a delimiter is required, you can set the `whitespace` parameter in the `RecursiveLevel` class to True. Note that if `whitespace = True`, you cannot pass a list of custom delimiters. Was this page helpful? YesNo [Neural Chunker\ \ Previous](https://docs.chonkie.ai/oss/chunkers/neural-chunker) [Semantic Chunker\ \ Next](https://docs.chonkie.ai/oss/chunkers/semantic-chunker) ⌘I --- # Chonkie Documentation > Documentation Index > ------------------- > > Fetch the complete documentation index at: [/llms.txt](https://docs.chonkie.ai/llms.txt) > > Use this file to discover all available pages before exploring further. [Skip to main content](https://docs.chonkie.ai/oss/chunkers/fast-chunker#content-area) The `FastChunker` uses [chonkie-core](https://github.com/chonkie-inc/chunk) for SIMD-accelerated boundary detection, enabling chunking speeds of 100+ GB/s. Unlike other chunkers, FastChunker uses **byte size** limits instead of token counts. This tradeoff enables extreme performance for high-throughput pipelines. [​](https://docs.chonkie.ai/oss/chunkers/fast-chunker#initialization) Initialization --------------------------------------------------------------------------------------- * Python * JavaScript Basic initialization with default parameters Split at paragraph boundaries Pattern-based splitting (e.g., for SentencePiece tokenizers) from chonkie import FastChunker chunker = FastChunker( chunk_size=4096, # Target size in BYTES (not tokens) delimiters="\n.?", # Split at newlines, periods, question marks ) Basic initialization with default parameters Split at paragraph boundaries Pattern-based splitting (e.g., for SentencePiece tokenizers) import { FastChunker } from "@chonkiejs/core"; let chunker = await FastChunker.create({ chunkSize: 4096, // Target size in BYTES (not tokens) delimiters: "\n.?", // Split at newlines, periods, question marks }); [​](https://docs.chonkie.ai/oss/chunkers/fast-chunker#parameters) Parameters ------------------------------------------------------------------------------- [​](https://docs.chonkie.ai/oss/chunkers/fast-chunker#param-chunk-size) chunk\_size int default:"4096" Target chunk size in **bytes** (not tokens) [​](https://docs.chonkie.ai/oss/chunkers/fast-chunker#param-delimiters) delimiters str default:"\\\\n.?" Single-byte delimiter characters to split on [​](https://docs.chonkie.ai/oss/chunkers/fast-chunker#param-pattern) pattern str default:"None" Multi-byte pattern to split on (overrides delimiters if set) [​](https://docs.chonkie.ai/oss/chunkers/fast-chunker#param-prefix) prefix bool default:"False" If True, keep the delimiter/pattern at the start of the next chunk instead of the end of the current chunk [​](https://docs.chonkie.ai/oss/chunkers/fast-chunker#param-consecutive) consecutive bool default:"False" If True, split at the START of consecutive delimiter runs instead of the middle [​](https://docs.chonkie.ai/oss/chunkers/fast-chunker#param-forward-fallback) forward\_fallback bool default:"False" If True, search forward for a delimiter when none is found in the backward search window [​](https://docs.chonkie.ai/oss/chunkers/fast-chunker#basic-usage) Basic Usage --------------------------------------------------------------------------------- * Python * JavaScript from chonkie import FastChunker # Initialize the chunker chunker = FastChunker( chunk_size=1024, delimiters=". \n", ) # Chunk your text text = "Your long document text here..." chunks = chunker.chunk(text) # Access chunk information for chunk in chunks: print(f"Chunk: {chunk.text[:50]}...") print(f"Bytes: {len(chunk.text)}") print(f"Position: {chunk.start_index}-{chunk.end_index}") import { FastChunker } from "@chonkiejs/core"; // Initialize the chunker const chunker = await FastChunker.create({ chunkSize: 1024, delimiters: ". \n", }); // Chunk your text const text = "Your long document text here..."; const chunks = await chunker.chunk(text); // Access chunk information for (const chunk of chunks) { console.log(`Chunk: ${chunk.text.slice(0, 50)}...`); console.log(`Bytes: ${chunk.text.length}`); console.log(`Position: ${chunk.startIndex}-${chunk.endIndex}`); } [​](https://docs.chonkie.ai/oss/chunkers/fast-chunker#examples) Examples --------------------------------------------------------------------------- Sentence-Based Chunking from chonkie import FastChunker # Split at sentence boundaries chunker = FastChunker( chunk_size=70, delimiters=".!?\n", ) text = """Machine learning has transformed technology. It enables computers to learn from data. Neural networks power many modern applications. The field continues to evolve rapidly.""" chunks = chunker.chunk(text) for i, chunk in enumerate(chunks): print(f"\n--- Chunk {i+1} ---") print(f"Text: {chunk.text}") print(f"Bytes: {len(chunk.text)}") Pattern-Based Chunking (SentencePiece) from chonkie import FastChunker # Split at metaspace boundaries (common in SentencePiece tokenizers) chunker = FastChunker( chunk_size=10, pattern="▁", # Metaspace character prefix=True, # Keep ▁ at start of next chunk ) text = "Hello▁World▁this▁is▁a▁test▁sentence" chunks = chunker.chunk(text) for chunk in chunks: print(f"Chunk: {chunk.text}") Handling Consecutive Delimiters from chonkie import FastChunker # Split at START of consecutive whitespace runs chunker = FastChunker( chunk_size=10, pattern=" ", consecutive=True, ) text = """First paragraph with multiple sentences. This is still the first paragraph. Second paragraph starts here. More content in the second paragraph.""" # Multiple spaces between words chunks = chunker.chunk(text) # Without consecutive=True: might split in middle of " " # With consecutive=True: splits at START of " " for chunk in chunks: print(f"Chunk: '{chunk.text}'") Forward Fallback Search from chonkie import FastChunker # Search forward if no delimiter found in backward window chunker = FastChunker( chunk_size=10, pattern=" ", forward_fallback=True, ) text = "verylongword short" chunks = chunker.chunk(text) # Without forward_fallback: hard split at byte 10 # With forward_fallback: finds space after "verylongword" for chunk in chunks: print(f"Chunk: '{chunk.text}'") Batch Processing from chonkie import FastChunker chunker = FastChunker(chunk_size=2048) documents = [\ "First document content here...",\ "Second document with different content...",\ "Third document for processing...",\ ] # Process all documents batch_results = chunker.chunk_batch(documents) for doc_idx, doc_chunks in enumerate(batch_results): print(f"\nDocument {doc_idx + 1}: {len(doc_chunks)} chunks") for chunk in doc_chunks: print(f" - {chunk.text[:30]}... ({len(chunk.text)} bytes)") High-Throughput Pipeline from chonkie import FastChunker import time # Configure for maximum throughput chunker = FastChunker( chunk_size=8192, delimiters="\n", ) # Read a large file with open("large_file.txt", "r") as f: large_text = f.read() # Benchmark chunking speed start = time.perf_counter() chunks = chunker.chunk(large_text) elapsed = time.perf_counter() - start mb_size = len(large_text) / (1024 * 1024) throughput = mb_size / elapsed print(f"Processed {mb_size:.1f} MB in {elapsed*1000:.1f}ms") print(f"Throughput: {throughput:.1f} MB/s") print(f"Chunks: {len(chunks)}") [​](https://docs.chonkie.ai/oss/chunkers/fast-chunker#comparison-with-other-chunkers) Comparison with Other Chunkers ----------------------------------------------------------------------------------------------------------------------- | Feature | FastChunker | TokenChunker | SentenceChunker | | --- | --- | --- | --- | | Size unit | Bytes | Tokens | Tokens | | Tokenizer required | No | Yes | Yes | | `token_count` | Always 0 | Computed | Computed | | Speed | ~100+ GB/s | Tokenizer-bound | Tokenizer-bound | | Best for | High-throughput pipelines | Token-precise chunking | Semantic boundaries | [​](https://docs.chonkie.ai/oss/chunkers/fast-chunker#when-to-use-fastchunker) When to Use FastChunker --------------------------------------------------------------------------------------------------------- **Use FastChunker when:** * Processing large volumes of text (>100KB documents) * Building high-throughput pipelines * Byte-level precision is acceptable * You don’t need exact token counts **Use other chunkers when:** * You need precise token counts for LLM context limits * Working with small documents (< 1KB) * Complex semantic boundaries are required [​](https://docs.chonkie.ai/oss/chunkers/fast-chunker#return-type) Return Type --------------------------------------------------------------------------------- FastChunker returns chunks as `Chunk` objects: @dataclass class Chunk: text: str # The chunk text start_index: int # Starting character position in original text end_index: int # Ending character position in original text token_count: int # Always 0 (not computed for speed) context: Optional[str] = None # Optional overlap context text embedding: Union[list[float], "np.ndarray", None] = None # Optional embedding vector The `token_count` field is always 0 in FastChunker output. If you need token counts, use the tokenizer separately or choose a different chunker. Was this page helpful? YesNo [Code Chunker\ \ Previous](https://docs.chonkie.ai/oss/chunkers/code-chunker) [Late Chunker\ \ Next](https://docs.chonkie.ai/oss/chunkers/late-chunker) ⌘I --- # Chonkie Documentation > Documentation Index > ------------------- > > Fetch the complete documentation index at: [/llms.txt](https://docs.chonkie.ai/llms.txt) > > Use this file to discover all available pages before exploring further. [Skip to main content](https://docs.chonkie.ai/oss/pipelines#content-area) Chonkie’s Pipeline API provides a fluent, chainable interface for building text processing workflows. Pipelines follow the **CHOMP architecture**, automatically orchestrating components in the correct order. [​](https://docs.chonkie.ai/oss/pipelines#what-is-chomp) What is CHOMP? -------------------------------------------------------------------------- CHOMP (CHOnkie’s Multi-step Pipeline) is our standardized architecture for document processing: Fetcher → Chef → Chunker → Refinery → Porter/Handshake 1 [](https://docs.chonkie.ai/oss/pipelines#) Fetcher Retrieve raw data from files, APIs, or databases 2 [](https://docs.chonkie.ai/oss/pipelines#) Chef Preprocess and transform raw data into Documents 3 [](https://docs.chonkie.ai/oss/pipelines#) Chunker Split documents into manageable chunks 4 [](https://docs.chonkie.ai/oss/pipelines#) Refinery (Optional) Post-process and enhance chunks 5 [](https://docs.chonkie.ai/oss/pipelines#) Porter/Handshake (Optional) Export or store chunks Pipelines automatically reorder components to follow CHOMP, so you can add them in any order. [​](https://docs.chonkie.ai/oss/pipelines#quick-start) Quick Start --------------------------------------------------------------------- ### [​](https://docs.chonkie.ai/oss/pipelines#single-file-processing) Single File Processing from chonkie import Pipeline # Build and execute pipeline doc = (Pipeline() .fetch_from("file", path="document.txt") .process_with("text") .chunk_with("recursive", chunk_size=512) .run()) # Access chunks print(f"Created {len(doc.chunks)} chunks") for chunk in doc.chunks: print(f"Chunk: {chunk.text[:50]}...") ### [​](https://docs.chonkie.ai/oss/pipelines#directory-processing) Directory Processing Process multiple files at once: # Process all markdown files in a directory docs = (Pipeline() .fetch_from("file", dir="./documents", ext=[".md", ".txt"]) .process_with("text") .chunk_with("recursive", chunk_size=512) .run()) # Process each document for doc in docs: print(f"Document has {len(doc.chunks)} chunks") ### [​](https://docs.chonkie.ai/oss/pipelines#direct-text-input) Direct Text Input Skip the fetcher and provide text directly: # No fetcher needed doc = (Pipeline() .process_with("text") .chunk_with("semantic", threshold=0.8) .run(texts="Your text here")) # Multiple texts docs = (Pipeline() .chunk_with("recursive", chunk_size=512) .run(texts=["Text 1", "Text 2", "Text 3"])) ### [​](https://docs.chonkie.ai/oss/pipelines#asynchronous-execution) Asynchronous Execution For high-throughput applications (e.g., web servers, batch processing), use `arun()`: import asyncio async def process_docs(): pipe = Pipeline().chunk_with("recursive") # Run pipeline asynchronously doc = await pipe.arun(texts="Async processing is fast!") # Process multiple concurrently docs = await pipe.arun(texts=["Doc 1", "Doc 2"]) return docs [​](https://docs.chonkie.ai/oss/pipelines#pipeline-methods) Pipeline Methods ------------------------------------------------------------------------------- ### [​](https://docs.chonkie.ai/oss/pipelines#fetch_from) fetch\_from() Fetch data from a source: # Single file .fetch_from("file", path="document.txt") # Directory with extension filter .fetch_from("file", dir="./docs", ext=[".txt", ".md"]) ### [​](https://docs.chonkie.ai/oss/pipelines#process_with) process\_with() Process data with a chef: # Text processing .process_with("text") # Markdown processing .process_with("markdown") # Table processing .process_with("table") ### [​](https://docs.chonkie.ai/oss/pipelines#chunk_with) chunk\_with() Chunk documents (required): # Recursive chunking .chunk_with("recursive", chunk_size=512) # Semantic chunking .chunk_with("semantic", threshold=0.8, chunk_size=1024) # Code chunking .chunk_with("code", chunk_size=512) ### [​](https://docs.chonkie.ai/oss/pipelines#refine_with) refine\_with() Refine chunks (optional, can chain multiple): # Add overlap context .refine_with("overlap", context_size=100, method="prefix") # Add embeddings .refine_with("embedding", model="text-embedding-3-small") ### [​](https://docs.chonkie.ai/oss/pipelines#export_with) export\_with() Export chunks to formats (optional): # Export to JSON .export_with("json", file="chunks.json") # Export to Hugging Face Datasets .export_with("datasets", name="my-dataset") ### [​](https://docs.chonkie.ai/oss/pipelines#store_in) store\_in() Store in vector databases (optional): # Store in Chroma .store_in("chroma", collection_name="documents") # Store in Qdrant .store_in("qdrant", collection_name="docs", url="http://localhost:6333") [​](https://docs.chonkie.ai/oss/pipelines#advanced-examples) Advanced Examples --------------------------------------------------------------------------------- ### [​](https://docs.chonkie.ai/oss/pipelines#rag-knowledge-base) RAG Knowledge Base Build a complete RAG ingestion pipeline: # Ingest documents into vector database docs = (Pipeline() .fetch_from("file", dir="./knowledge_base", ext=[".txt", ".md"]) .process_with("text") .chunk_with("semantic", threshold=0.8, chunk_size=1024) .refine_with("overlap", context_size=100) .store_in("qdrant", collection_name="knowledge", url="http://localhost:6333") .run()) print(f"Ingested {len(docs)} documents") ### [​](https://docs.chonkie.ai/oss/pipelines#semantic-search-pipeline) Semantic Search Pipeline Process documents with embeddings for search: # Chunk with embeddings doc = (Pipeline() .fetch_from("file", path="research_paper.txt") .process_with("text") .chunk_with("semantic", threshold=0.8, chunk_size=1024, similarity_window=3) .refine_with("overlap", context_size=100) .refine_with("embedding", model="minishlab/potion-base-32M") .run()) # All chunks now have embeddings for chunk in doc.chunks: if chunk.embedding is not None: print(f"Chunk: {chunk.text[:30]}... | Embedding shape: {chunk.embedding.shape}") ### [​](https://docs.chonkie.ai/oss/pipelines#code-documentation) Code Documentation Process code with specialized chunking: # Chunk Python files docs = (Pipeline() .fetch_from("file", dir="./src", ext=[".py"]) .chunk_with("code", chunk_size=512) .export_with("json", file="code_chunks.json") .run()) print(f"Processed {len(docs)} Python files") ### [​](https://docs.chonkie.ai/oss/pipelines#markdown-processing) Markdown Processing Handle markdown with table and code awareness: # Process markdown documentation doc = (Pipeline() .fetch_from("file", path="README.md") .process_with("markdown") .chunk_with("recursive", chunk_size=512) .run()) # Access markdown metadata print(f"Found {len(doc.tables)} tables") print(f"Found {len(doc.code)} code blocks") print(f"Created {len(doc.chunks)} chunks") [​](https://docs.chonkie.ai/oss/pipelines#recipe-based-pipelines) Recipe-Based Pipelines ------------------------------------------------------------------------------------------- Load pre-configured pipelines from the Chonkie Hub: # Load markdown processing recipe pipeline = Pipeline.from_recipe("markdown") # Run with your content doc = pipeline.run(texts="# My Markdown\n\nContent here") # Load custom local recipe pipeline = Pipeline.from_recipe("custom", path="./my_recipe.json") Recipes are stored in the [chonkie-ai/recipes](https://huggingface.co/datasets/chonkie-ai/recipes) repository. [​](https://docs.chonkie.ai/oss/pipelines#best-practices) Best Practices --------------------------------------------------------------------------- Always specify chunk\_size Explicitly set `chunk_size` for predictable behavior: # Good - explicit size .chunk_with("recursive", chunk_size=512) # Avoid - uses defaults that may change .chunk_with("recursive") Match chunkers to content type Choose chunkers appropriate for your content: # Code files → Code chunker .chunk_with("code") # Need semantic similarity → Semantic chunker .chunk_with("semantic", threshold=0.8) # General text → Recursive chunker .chunk_with("recursive") Use refineries for RAG applications Add overlap refineries for better retrieval context: .chunk_with("recursive", chunk_size=512) .refine_with("overlap", context_size=100) Filter extensions in directory mode Always specify file extensions to avoid unwanted files: # Good - filtered .fetch_from("file", dir="./docs", ext=[".txt", ".md"]) # Bad - processes everything including binaries .fetch_from("file", dir="./docs") Chain refineries for complex processing Multiple refineries can be chained: .chunk_with("recursive", chunk_size=512) .refine_with("overlap", context_size=50) .refine_with("embedding", model="text-embedding-3-small") [​](https://docs.chonkie.ai/oss/pipelines#pipeline-validation) Pipeline Validation ------------------------------------------------------------------------------------- Pipelines validate configuration before execution: ✅ **Must have**: At least one chunker ✅ **Must have**: Fetcher OR text input via `run(texts=...)` ❌ **Cannot have**: Multiple chefs (only one allowed) # ❌ Invalid - no chunker Pipeline().fetch_from("file", path="doc.txt").run() # ❌ Invalid - multiple chefs Pipeline() .process_with("text") .process_with("markdown") # Error! .chunk_with("recursive") # ✅ Valid - has chunker and input source Pipeline() .fetch_from("file", path="doc.txt") .chunk_with("recursive", chunk_size=512) .run() # ✅ Valid - text input, no fetcher needed Pipeline() .chunk_with("recursive") .run(texts="Hello world") [​](https://docs.chonkie.ai/oss/pipelines#return-values) Return Values ------------------------------------------------------------------------- Pipeline behavior depends on input: * **Single file/text**: Returns `Document` * **Multiple files/texts**: Returns `list[Document]` # Single file → Document doc = Pipeline().fetch_from("file", path="doc.txt").chunk_with("recursive").run() assert isinstance(doc, Document) # Directory → list[Document] docs = Pipeline().fetch_from("file", dir="./docs").chunk_with("recursive").run() assert isinstance(docs, list) # Multiple texts → list[Document] docs = Pipeline().chunk_with("recursive").run(texts=["t1", "t2"]) assert isinstance(docs, list) [​](https://docs.chonkie.ai/oss/pipelines#error-handling) Error Handling --------------------------------------------------------------------------- Pipelines provide clear error messages: from pathlib import Path try: doc = Pipeline() .fetch_from("file", path="missing.txt") .chunk_with("recursive") .run() except FileNotFoundError as e: print(f"File not found: {e}") except ValueError as e: print(f"Configuration error: {e}") except RuntimeError as e: print(f"Pipeline execution failed: {e}") [​](https://docs.chonkie.ai/oss/pipelines#component-overview) Component Overview ----------------------------------------------------------------------------------- ### [​](https://docs.chonkie.ai/oss/pipelines#available-components) Available Components Explore each component type: Fetchers -------- Connect to data sources (files, APIs, databases) Chefs ----- Preprocess text, markdown, tables, etc. Chunkers -------- Split text with various strategies Refineries ---------- Add overlap, embeddings, and more Porters ------- Export to JSON, Datasets, etc. Handshakes ---------- Store in Chroma, Qdrant, Pinecone, etc. [​](https://docs.chonkie.ai/oss/pipelines#what%E2%80%99s-next) What’s Next? ------------------------------------------------------------------------------ 1 [](https://docs.chonkie.ai/oss/pipelines#) Explore Fetchers Learn how to connect different data sources in [Fetchers](https://docs.chonkie.ai/oss/fetchers/overview) 2 [](https://docs.chonkie.ai/oss/pipelines#) Choose Your Chunker Find the right chunking strategy in [Chunkers](https://docs.chonkie.ai/oss/chunkers/overview) 3 [](https://docs.chonkie.ai/oss/pipelines#) Enhance with Refineries Improve chunk quality in [Refineries](https://docs.chonkie.ai/oss/refinery/overview) 4 [](https://docs.chonkie.ai/oss/pipelines#) Store Your Chunks Ingest into vector databases with [Handshakes](https://docs.chonkie.ai/oss/handshakes/overview) Was this page helpful? YesNo [Installation\ \ Previous](https://docs.chonkie.ai/oss/installation) [API Server\ \ Next](https://docs.chonkie.ai/oss/api/overview) ⌘I --- # Chonkie Documentation > Documentation Index > ------------------- > > Fetch the complete documentation index at: [/llms.txt](https://docs.chonkie.ai/llms.txt) > > Use this file to discover all available pages before exploring further. [Skip to main content](https://docs.chonkie.ai/oss/chunkers/token-chunker#content-area) The `TokenChunker` splits text into chunks based on token count, ensuring each chunk stays within specified token limits. [​](https://docs.chonkie.ai/oss/chunkers/token-chunker#api-reference) API Reference -------------------------------------------------------------------------------------- To use the `TokenChunker` via the API, check out the [API reference documentation](https://docs.chonkie.ai/api/chunkers/token-chunker) . [​](https://docs.chonkie.ai/oss/chunkers/token-chunker#installation) Installation ------------------------------------------------------------------------------------ TokenChunker is included in the base installation of Chonkie. If you would like to use custom tokenizers in JavaScript, please install the `@chonkiejs/token` library [​](https://docs.chonkie.ai/oss/chunkers/token-chunker#initialization) Initialization ---------------------------------------------------------------------------------------- Python JavaScript from chonkie import TokenChunker # Basic initialization with default parameters chunker = TokenChunker( tokenizer="character", # Default tokenizer (or use "gpt2", etc.) chunk_size=2048, # Maximum tokens per chunk chunk_overlap=128 # Overlap between chunks ) # Using a custom tokenizer from tokenizers import Tokenizer custom_tokenizer = Tokenizer.from_pretrained("your-tokenizer") chunker = TokenChunker( tokenizer=custom_tokenizer, chunk_size=2048, chunk_overlap=128 ) [​](https://docs.chonkie.ai/oss/chunkers/token-chunker#parameters) Parameters -------------------------------------------------------------------------------- [​](https://docs.chonkie.ai/oss/chunkers/token-chunker#param-tokenizer) tokenizer Union\[str, Any\] default:"character" Tokenizer to use. Can be a string identifier (“character”, “word”, “byte”, “gpt2”, etc.) or a tokenizer instance [​](https://docs.chonkie.ai/oss/chunkers/token-chunker#param-chunk-size-chunk-size) chunk\_size / chunkSize int default:"2048" Maximum number of tokens per chunk [​](https://docs.chonkie.ai/oss/chunkers/token-chunker#param-chunk-overlap-chunk-overlap) chunk\_overlap / chunkOverlap Union\[int, float\] default:"0" Number or percentage of overlapping tokens between chunks [​](https://docs.chonkie.ai/oss/chunkers/token-chunker#basic-usage) Basic Usage ---------------------------------------------------------------------------------- Python JavaScript from chonkie import TokenChunker # Initialize the chunker chunker = TokenChunker( tokenizer="gpt2", chunk_size=512, chunk_overlap=50 ) # Chunk your text text = "Your long document text here..." chunks = chunker.chunk(text) # Access chunk information for chunk in chunks: print(f"Chunk: {chunk.text[:50]}...") print(f"Tokens: {chunk.token_count}") [​](https://docs.chonkie.ai/oss/chunkers/token-chunker#examples) Examples ---------------------------------------------------------------------------- Single Text Chunking Python JavaScript from chonkie import TokenChunker # Create a chunker with specific parameters chunker = TokenChunker( tokenizer="gpt2", chunk_size=1024, chunk_overlap=128 ) text = """Natural language processing has revolutionized how we interact with computers. Machine learning models can now understand context, generate text, and even translate between languages with remarkable accuracy. This transformation has enabled applications ranging from virtual assistants to automated content generation.""" # Chunk the text chunks = chunker.chunk(text) # Process each chunk for i, chunk in enumerate(chunks): print(f"\n--- Chunk {i+1} ---") print(f"Text: {chunk.text}") print(f"Token count: {chunk.token_count}") print(f"Start index: {chunk.start_index}") print(f"End index: {chunk.end_index}") Batch Processing Batch processing is only supported in Python from chonkie import TokenChunker # Initialize chunker for batch processing chunker = TokenChunker( tokenizer="gpt2", chunk_size=512, chunk_overlap=50 ) # Multiple documents to process documents = [\ "First document about machine learning fundamentals...",\ "Second document discussing neural networks...",\ "Third document on natural language processing..."\ ] # Process all documents at once batch_chunks = chunker.chunk_batch(documents) # Iterate through results for doc_idx, doc_chunks in enumerate(batch_chunks): print(f"\nDocument {doc_idx + 1}: {len(doc_chunks)} chunks") for chunk in doc_chunks: print(f" - Chunk: {chunk.text[:50]}... ({chunk.token_count} tokens)") Using Custom Tokenizers Custom tokenizers are only supported in Python. See the Installation section for JavaScript tokenizer support. from chonkie import TokenChunker import tiktoken # Using TikToken with a specific model encoding tokenizer = tiktoken.get_encoding("cl100k_base") # GPT-4 encoding chunker = TokenChunker( tokenizer=tokenizer, chunk_size=2048, chunk_overlap=200 ) # Or using Hugging Face tokenizers from transformers import AutoTokenizer hf_tokenizer = AutoTokenizer.from_pretrained("bert-base-uncased") chunker = TokenChunker( tokenizer=hf_tokenizer, chunk_size=512, chunk_overlap=50 ) text = "Your text to chunk with custom tokenizer..." chunks = chunker.chunk(text) Callable Interface The callable interface is only supported in Python from chonkie import TokenChunker # Initialize once chunker = TokenChunker( tokenizer="gpt2", chunk_size=1024, chunk_overlap=100 ) # Use as a callable for single text single_text = "This is a document that needs chunking..." chunks = chunker(single_text) print(f"Single text produced {len(chunks)} chunks") # Use as a callable for multiple texts multiple_texts = [\ "First document text...",\ "Second document text...",\ "Third document text..."\ ] batch_results = chunker(multiple_texts) print(f"Processed {len(batch_results)} documents") Overlap Configuration Python JavaScript from chonkie import TokenChunker # Fixed token overlap chunker_fixed = TokenChunker( tokenizer="gpt2", chunk_size=1000, chunk_overlap=100 # Exactly 100 tokens overlap ) # Percentage-based overlap chunker_percent = TokenChunker( tokenizer="gpt2", chunk_size=1000, chunk_overlap=0.1 # 10% overlap (100 tokens for 1000 token chunks) ) text = "Long document text that will be chunked with overlap..." # Compare the results fixed_chunks = chunker_fixed.chunk(text) percent_chunks = chunker_percent.chunk(text) print(f"Fixed overlap: {len(fixed_chunks)} chunks") print(f"Percentage overlap: {len(percent_chunks)} chunks") Processing Large Documents Python JavaScript from chonkie import TokenChunker # Configure for large documents chunker = TokenChunker( tokenizer="gpt2", chunk_size=4096, # Larger chunks for efficiency chunk_overlap=512 # Maintain context between chunks ) # Read a large document with open("large_document.txt", "r") as f: large_text = f.read() # Process efficiently chunks = chunker.chunk(large_text) print(f"Document statistics:") print(f" Original length: {len(large_text)} characters") print(f" Number of chunks: {len(chunks)}") print(f" Average chunk size: {sum(c.token_count for c in chunks) / len(chunks):.1f} tokens") # Save chunks for further processing for i, chunk in enumerate(chunks): with open(f"chunk_{i:03d}.txt", "w") as f: f.write(chunk.text) [​](https://docs.chonkie.ai/oss/chunkers/token-chunker#supported-tokenizers) Supported Tokenizers ---------------------------------------------------------------------------------------------------- Changing tokenizer backend is only supported on Python TokenChunker supports multiple tokenizer backends: * **TikToken** (Recommended) import tiktoken tokenizer = tiktoken.get_encoding("gpt2") * **AutoTikTokenizer** from autotiktokenizer import AutoTikTokenizer tokenizer = AutoTikTokenizer.from_pretrained("gpt2") * **Hugging Face Tokenizers** from tokenizers import Tokenizer tokenizer = Tokenizer.from_pretrained("gpt2") * **Transformers** from transformers import AutoTokenizer tokenizer = AutoTokenizer.from_pretrained("gpt2") [​](https://docs.chonkie.ai/oss/chunkers/token-chunker#return-type) Return Type ---------------------------------------------------------------------------------- TokenChunker returns chunks as `Chunk` objects. Python JavaScript @dataclass class Chunk: text: str # The chunk text start_index: int # Starting position in original text end_index: int # Ending position in original text token_count: int # Number of tokens in chunk context: Optional[str] = None # Optional overlap context text embedding: Union[list[float], "np.ndarray", None] = None # Optional embedding vector Was this page helpful? YesNo [TeraflopAI Chunker\ \ Previous](https://docs.chonkie.ai/oss/chunkers/teraflopai-chunker) [Embeddings Overview\ \ Next](https://docs.chonkie.ai/oss/embeddings/overview) ⌘I --- # Chonkie Documentation > Documentation Index > ------------------- > > Fetch the complete documentation index at: [/llms.txt](https://docs.chonkie.ai/llms.txt) > > Use this file to discover all available pages before exploring further. [Skip to main content](https://docs.chonkie.ai/oss/chunkers/semantic-chunker#content-area) The `SemanticChunker` splits text into chunks based on semantic similarity, ensuring that related content stays together in the same chunk. This chunker now includes advanced features like Savitzky-Golay filtering for smoother boundary detection and skip-window merging for connecting related content that may not be consecutive. This chunker is inspired by the work of [Greg Kamradt](https://github.com/gkamradt) . [​](https://docs.chonkie.ai/oss/chunkers/semantic-chunker#api-reference) API Reference ----------------------------------------------------------------------------------------- To use the `SemanticChunker` via the API, check out the [API reference documentation](https://docs.chonkie.ai/api/chunkers/semantic-chunker) . [​](https://docs.chonkie.ai/oss/chunkers/semantic-chunker#installation) Installation --------------------------------------------------------------------------------------- SemanticChunker requires additional dependencies for semantic capabilities. You can install it with: Python JavaScript pip install "chonkie[semantic]" For installation instructions, see the [Installation Guide](https://docs.chonkie.ai/oss/installation) . [​](https://docs.chonkie.ai/oss/chunkers/semantic-chunker#initialization) Initialization ------------------------------------------------------------------------------------------- * Python * JavaScript from chonkie import SemanticChunker # Basic initialization with default parameters chunker = SemanticChunker( embedding_model="minishlab/potion-base-32M", # Default model threshold=0.8, # Similarity threshold (0-1) chunk_size=2048, # Maximum tokens per chunk similarity_window=3, # Window for similarity calculation skip_window=0 # Skip-and-merge window (0=disabled) ) # With skip-and-merge enabled (similar to legacy SDPM behavior) chunker = SemanticChunker( embedding_model="minishlab/potion-base-32M", threshold=0.7, chunk_size=2048, skip_window=1 # Enable merging of similar non-consecutive groups ) import { SemanticChunker } from "@chonkiejs/core"; // Basic initialization with custom embedding function const embedFn = async (texts) => { // Your embedding logic here // Return array of embeddings for each text }; const chunker = await SemanticChunker.create({ embedFunction: embedFn, // Custom embedding function threshold: 0.8, // Similarity threshold (0-1) chunkSize: 2048, // Maximum tokens per chunk similarityWindow: 3, // Window for similarity calculation skipWindow: 0 // Skip-and-merge window (0=disabled) }); [​](https://docs.chonkie.ai/oss/chunkers/semantic-chunker#parameters) Parameters ----------------------------------------------------------------------------------- [​](https://docs.chonkie.ai/oss/chunkers/semantic-chunker#param-embedding-model) embedding\_model Union\[str, BaseEmbeddings\] default:"minishlab/potion-base-32M" Model identifier or embedding model instance [​](https://docs.chonkie.ai/oss/chunkers/semantic-chunker#param-threshold) threshold float default:"0.8" Similarity threshold for grouping sentences (0-1). Lower values create larger groups. [​](https://docs.chonkie.ai/oss/chunkers/semantic-chunker#param-chunk-size) chunk\_size int default:"2048" Maximum tokens per chunk [​](https://docs.chonkie.ai/oss/chunkers/semantic-chunker#param-similarity-window) similarity\_window int default:"3" Number of sentences to consider for similarity calculation [​](https://docs.chonkie.ai/oss/chunkers/semantic-chunker#param-min-sentences-per-chunk) min\_sentences\_per\_chunk int default:"1" Minimum number of sentences per chunk [​](https://docs.chonkie.ai/oss/chunkers/semantic-chunker#param-min-characters-per-sentence) min\_characters\_per\_sentence int default:"24" Minimum number of characters per sentence [​](https://docs.chonkie.ai/oss/chunkers/semantic-chunker#param-skip-window) skip\_window int default:"0" Number of groups to skip when looking for similar content to merge. * `0` (default): No skip-and-merge, uses standard semantic grouping * `1` or higher: Enables merging of semantically similar groups within the skip window This feature allows the chunker to connect related content that may not be consecutive in the text. [​](https://docs.chonkie.ai/oss/chunkers/semantic-chunker#param-filter-window) filter\_window int default:"5" Window length for the Savitzky-Golay filter used in boundary detection [​](https://docs.chonkie.ai/oss/chunkers/semantic-chunker#param-filter-polyorder) filter\_polyorder int default:"3" Polynomial order for the Savitzky-Golay filter [​](https://docs.chonkie.ai/oss/chunkers/semantic-chunker#param-filter-tolerance) filter\_tolerance float default:"0.2" Tolerance for the Savitzky-Golay filter boundary detection [​](https://docs.chonkie.ai/oss/chunkers/semantic-chunker#param-delim) delim Union\[str, list\[str\]\] default:"\[\\". \\", \\"! \\", \\"? \\", \\"\\\\n\\"\]" Delimiters to split sentences on [​](https://docs.chonkie.ai/oss/chunkers/semantic-chunker#param-include-delim) include\_delim Optional\[Literal\["prev", "next"\]\] default:"prev" Include delimiters in the chunk text. Specify whether to include with the previous or next sentence. [​](https://docs.chonkie.ai/oss/chunkers/semantic-chunker#basic-usage) Basic Usage ------------------------------------------------------------------------------------- * Python * JavaScript from chonkie import SemanticChunker # Initialize with semantic similarity grouping chunker = SemanticChunker( embedding_model="minishlab/potion-base-32M", threshold=0.7, # Similarity threshold chunk_size=512 ) text = """Your document text with multiple topics and themes...""" chunks = chunker.chunk(text) # Process chunks for chunk in chunks: print(f"Chunk: {chunk.text[:50]}...") print(f"Tokens: {chunk.token_count}") import { SemanticChunker } from "@chonkiejs/core"; // Define custom embedding function const embedFn = async (texts) => { // Your embedding logic here (e.g., call to an API) // Return array of embeddings for each text }; // Initialize with semantic similarity grouping const chunker = await SemanticChunker.create({ embedFunction: embedFn, threshold: 0.7, // Similarity threshold chunkSize: 512 }); const text = "Your document text with multiple topics and themes..."; const chunks = await chunker.chunk(text); // Process chunks for (const chunk of chunks) { console.log(`Chunk: ${chunk.text.slice(0, 50)}...`); console.log(`Tokens: ${chunk.tokenCount}`); } [​](https://docs.chonkie.ai/oss/chunkers/semantic-chunker#examples) Examples ------------------------------------------------------------------------------- Basic Semantic Chunking from chonkie import SemanticChunker text = """Artificial intelligence is transforming industries worldwide. Machine learning algorithms can now process vast amounts of data efficiently. Deep learning models have achieved remarkable accuracy in complex tasks. Climate change poses significant challenges to our planet. Rising temperatures affect ecosystems and biodiversity globally. Sustainable practices are essential for environmental preservation. Quantum computing represents a paradigm shift in computation. These systems leverage quantum mechanical phenomena for processing. Potential applications include cryptography and drug discovery.""" # Create semantic chunker chunker = SemanticChunker( embedding_model="minishlab/potion-base-32M", threshold=0.75, # Higher threshold = more similar content grouped chunk_size=1024 ) chunks = chunker.chunk(text) # Analyze semantic groupings for i, chunk in enumerate(chunks): print(f"\n--- Semantic Group {i+1} ---") print(f"Content: {chunk.text[:100]}...") print(f"Token count: {chunk.token_count}") print(f"Theme: {chunk.text.split('.')[0]}") # First sentence as theme indicator Skip-Window Merging from chonkie import SemanticChunker # Text with alternating topics text = """Neural networks process information through interconnected nodes. The stock market experienced significant volatility this quarter. Deep learning models require substantial training data for optimization. Economic indicators point to potential recession risks ahead. GPU acceleration has revolutionized machine learning computations. Federal reserve policies impact global financial markets. Transformer architectures dominate modern NLP applications. Cryptocurrency markets show correlation with traditional assets.""" # Enable skip-window to merge non-consecutive similar content chunker = SemanticChunker( embedding_model="minishlab/potion-base-32M", threshold=0.65, chunk_size=512, skip_window=2 # Look ahead 2 groups for similar content ) chunks = chunker.chunk(text) # AI-related content will be grouped together # Financial content will be grouped separately for i, chunk in enumerate(chunks): print(f"\nGroup {i+1}: {len(chunk.text.split('.'))} sentences") print(f"Preview: {chunk.text[:80]}...") Fine-tuned Similarity Control from chonkie import SemanticChunker text = """Your comprehensive document with various topics...""" # Experiment with different thresholds thresholds = [0.5, 0.7, 0.9] for threshold in thresholds: chunker = SemanticChunker( embedding_model="minishlab/potion-base-32M", threshold=threshold, chunk_size=512, similarity_window=3 # Consider 3 sentences for similarity ) chunks = chunker.chunk(text) print(f"\nThreshold {threshold}: {len(chunks)} chunks created") # Lower threshold = larger, more diverse chunks # Higher threshold = smaller, more focused chunks avg_size = sum(c.token_count for c in chunks) / len(chunks) print(f"Average chunk size: {avg_size:.1f} tokens") Batch Document Processing from chonkie import SemanticChunker # Initialize chunker once chunker = SemanticChunker( embedding_model="minishlab/potion-base-32M", threshold=0.7, chunk_size=1024, min_sentences_per_chunk=2 # Ensure meaningful chunks ) # Multiple documents with different topics documents = [\ """Document about artificial intelligence and machine learning...""",\ """Document about climate change and environmental science...""",\ """Document about quantum computing and physics..."""\ ] # Process all documents batch_results = chunker.chunk_batch(documents) # Analyze results for doc_idx, chunks in enumerate(batch_results): print(f"\nDocument {doc_idx + 1}:") print(f" Total chunks: {len(chunks)}") print(f" Total tokens: {sum(c.token_count for c in chunks)}") # Show semantic boundaries for i, chunk in enumerate(chunks): first_sentence = chunk.text.split('.')[0] print(f" Chunk {i+1}: {first_sentence[:50]}...") Custom Embeddings Integration from chonkie import SemanticChunker from chonkie.embeddings import AutoEmbeddings # Use AutoEmbeddings for automatic model selection embeddings = AutoEmbeddings.get_embeddings( model="sentence-transformers/all-MiniLM-L6-v2" ) chunker = SemanticChunker( embedding_model=embeddings, threshold=0.8, chunk_size=512 ) # Or use specific embedding providers from chonkie.embeddings import OpenAIEmbeddings openai_embeddings = OpenAIEmbeddings( model="text-embedding-ada-002" ) chunker = SemanticChunker( embedding_model=openai_embeddings, threshold=0.75, chunk_size=1024 ) text = "Your text to chunk with custom embeddings..." chunks = chunker.chunk(text) Advanced Filtering Options from chonkie import SemanticChunker # Configure Savitzky-Golay filter for smoother boundaries chunker = SemanticChunker( embedding_model="minishlab/potion-base-32M", threshold=0.7, chunk_size=512, filter_window=7, # Larger window for smoother filtering filter_polyorder=4, # Higher order polynomial filter_tolerance=0.15 # Stricter boundary detection ) text = """Complex document with subtle topic transitions...""" chunks = chunker.chunk(text) # The filtering helps identify more natural semantic boundaries # especially in documents with gradual topic shifts for chunk in chunks: print(f"Smooth boundary chunk: {chunk.text[:60]}...") Sentence Configuration from chonkie import SemanticChunker # Customize sentence detection chunker = SemanticChunker( embedding_model="minishlab/potion-base-32M", threshold=0.7, chunk_size=1024, min_sentences_per_chunk=3, # At least 3 sentences per chunk min_characters_per_sentence=30, # Filter out short fragments delim=[". ", "! ", "? ", "\n\n"], # Custom sentence delimiters include_delim="prev" # Include delimiter with previous sentence ) # Text with various sentence structures text = """Short sentence. This is a much longer sentence with more detail. Question here? Exclamation point! New paragraph starts here. Another paragraph with different content...""" chunks = chunker.chunk(text) for chunk in chunks: sentences = chunk.text.split('. ') print(f"Chunk with {len(sentences)} sentences") RAG Pipeline Integration from chonkie import SemanticChunker from chonkie.refinery import OverlapRefinery, EmbeddingsRefinery # Create semantic chunker chunker = SemanticChunker( embedding_model="minishlab/potion-base-32M", threshold=0.7, chunk_size=512 ) # Add refineries for RAG optimization overlap_refinery = OverlapRefinery(overlap_size=50) embeddings_refinery = EmbeddingsRefinery( embedding_model="minishlab/potion-base-32M" ) # Process document text = """Your document for RAG system...""" chunks = chunker.chunk(text) # Apply refinements chunks = overlap_refinery.refine(chunks) chunks = embeddings_refinery.refine(chunks) # Add embeddings # Ready for vector database for chunk in chunks: print(f"Chunk ready for indexing: {chunk.text[:50]}...") if chunk.embedding is not None: print(f" Embedding shape: {chunk.embedding.shape}") [​](https://docs.chonkie.ai/oss/chunkers/semantic-chunker#advanced-features) Advanced Features ------------------------------------------------------------------------------------------------- ### [​](https://docs.chonkie.ai/oss/chunkers/semantic-chunker#savitzky-golay-filtering) Savitzky-Golay Filtering The SemanticChunker uses Savitzky-Golay filtering for smoother boundary detection in similarity curves. This reduces noise in the semantic similarity signal and provides more stable chunk boundaries. ### [​](https://docs.chonkie.ai/oss/chunkers/semantic-chunker#skip-window-merging) Skip-Window Merging When `skip_window > 0`, the chunker can merge semantically similar groups that are not consecutive. This is useful for: * Documents with alternating topics * Content with recurring themes * Technical documents with distributed related sections [​](https://docs.chonkie.ai/oss/chunkers/semantic-chunker#supported-embeddings) Supported Embeddings ------------------------------------------------------------------------------------------------------- SemanticChunker supports multiple embedding providers through Chonkie’s embedding system. See the [Embeddings Overview](https://docs.chonkie.ai/python-sdk/embeddings/overview) for more information. [​](https://docs.chonkie.ai/oss/chunkers/semantic-chunker#return-type) Return Type ------------------------------------------------------------------------------------- SemanticChunker returns `Chunk` objects: @dataclass class Chunk: text: str start_index: int end_index: int token_count: int Was this page helpful? YesNo [Recursive Chunker\ \ Previous](https://docs.chonkie.ai/oss/chunkers/recursive-chunker) [Sentence Chunker\ \ Next](https://docs.chonkie.ai/oss/chunkers/sentence-chunker) ⌘I --- # Page Not Found > Documentation Index > ------------------- > > Fetch the complete documentation index at: [/llms.txt](https://docs.chonkie.ai/llms.txt) > > Use this file to discover all available pages before exploring further. [Skip to main content](https://docs.chonkie.ai/common/chunking-api#content-area) 404 Page Not Found ============== We couldn't find the page. Maybe you were looking for one of these pages below? [Late Chunker](https://docs.chonkie.ai/oss/chunkers/late-chunker#initialization) [Fast Chunker](https://docs.chonkie.ai/oss/chunkers/fast-chunker#pattern-based-chunking-sentencepiece) [Recursive Chunker](https://docs.chonkie.ai/oss/chunkers/recursive-chunker#initialization) Ctrl+I --- # Page Not Found > Documentation Index > ------------------- > > Fetch the complete documentation index at: [/llms.txt](https://docs.chonkie.ai/llms.txt) > > Use this file to discover all available pages before exploring further. [Skip to main content](https://docs.chonkie.ai/api/chunkers/semantic-chunker#content-area) 404 Page Not Found ============== We couldn't find the page. Maybe you were looking for one of these pages below? [Semantic Chunker](https://docs.chonkie.ai/oss/chunkers/semantic-chunker#api-reference) [SDPM Chunker (Legacy)](https://docs.chonkie.ai/oss/chunkers/sdpm-chunker#why-use-the-new-semanticchunker-instead) [Endpoints](https://docs.chonkie.ai/oss/api/endpoints#semantic-chunker-8) Ctrl+I --- # Page Not Found > Documentation Index > ------------------- > > Fetch the complete documentation index at: [/llms.txt](https://docs.chonkie.ai/llms.txt) > > Use this file to discover all available pages before exploring further. [Skip to main content](https://docs.chonkie.ai/api/chunkers/recursive-chunker#content-area) 404 Page Not Found ============== We couldn't find the page. Maybe you were looking for one of these pages below? [Recursive Chunker](https://docs.chonkie.ai/oss/chunkers/recursive-chunker#api-reference) [Endpoints](https://docs.chonkie.ai/oss/api/endpoints#recursive-chunker-7) [Table Chunker](https://docs.chonkie.ai/oss/chunkers/table-chunker#api-reference) Ctrl+I --- # Page Not Found > Documentation Index > ------------------- > > Fetch the complete documentation index at: [/llms.txt](https://docs.chonkie.ai/llms.txt) > > Use this file to discover all available pages before exploring further. [Skip to main content](https://docs.chonkie.ai/api/refineries/overlap#content-area) 404 Page Not Found ============== We couldn't find the page. Maybe you were looking for one of these pages below? [Overlap Refinery](https://docs.chonkie.ai/oss/refinery/overlap-refinery#api-reference) [Endpoints](https://docs.chonkie.ai/oss/api/endpoints#overlap-refinery-7) [Refinery Overview](https://docs.chonkie.ai/oss/refinery/overview) Ctrl+I --- # Page Not Found > Documentation Index > ------------------- > > Fetch the complete documentation index at: [/llms.txt](https://docs.chonkie.ai/llms.txt) > > Use this file to discover all available pages before exploring further. [Skip to main content](https://docs.chonkie.ai/python-sdk/embeddings/overview#content-area) 404 Page Not Found ============== We couldn't find the page. Maybe you were looking for one of these pages below? [Embeddings Overview](https://docs.chonkie.ai/oss/embeddings/overview) [Chefs Overview](https://docs.chonkie.ai/oss/chefs/overview) [Overview](https://docs.chonkie.ai/oss/experimental/overview) Ctrl+I --- # Page Not Found > Documentation Index > ------------------- > > Fetch the complete documentation index at: [/llms.txt](https://docs.chonkie.ai/llms.txt) > > Use this file to discover all available pages before exploring further. [Skip to main content](https://docs.chonkie.ai/api/refineries/embeddings#content-area) 404 Page Not Found ============== We couldn't find the page. Maybe you were looking for one of these pages below? [Embeddings Refinery](https://docs.chonkie.ai/oss/refinery/embeddings-refinery#api-reference) [Endpoints](https://docs.chonkie.ai/oss/api/endpoints#embeddings-refinery) [Refinery Overview](https://docs.chonkie.ai/oss/refinery/overview) Ctrl+I --- # Page Not Found > Documentation Index > ------------------- > > Fetch the complete documentation index at: [/llms.txt](https://docs.chonkie.ai/llms.txt) > > Use this file to discover all available pages before exploring further. [Skip to main content](https://docs.chonkie.ai/api/chunkers/neural-chunker#content-area) 404 Page Not Found ============== We couldn't find the page. Maybe you were looking for one of these pages below? [Neural Chunker](https://docs.chonkie.ai/oss/chunkers/neural-chunker#api-reference) [Chunkers Overview](https://docs.chonkie.ai/oss/chunkers/overview#availability) [Changelog](https://docs.chonkie.ai/oss/changelog#v1-0-6) Ctrl+I --- # Page Not Found > Documentation Index > ------------------- > > Fetch the complete documentation index at: [/llms.txt](https://docs.chonkie.ai/llms.txt) > > Use this file to discover all available pages before exploring further. [Skip to main content](https://docs.chonkie.ai/api/chunkers/slumber-chunker#content-area) 404 Page Not Found ============== We couldn't find the page. Maybe you were looking for one of these pages below? [Slumber Chunker](https://docs.chonkie.ai/oss/chunkers/slumber-chunker#api-reference) [Open Source](https://docs.chonkie.ai/common/open-source#slumberchunker) [Chunkers Overview](https://docs.chonkie.ai/oss/chunkers/overview#availability) Ctrl+I --- # Page Not Found > Documentation Index > ------------------- > > Fetch the complete documentation index at: [/llms.txt](https://docs.chonkie.ai/llms.txt) > > Use this file to discover all available pages before exploring further. [Skip to main content](https://docs.chonkie.ai/api/chunkers/late-chunker#content-area) 404 Page Not Found ============== We couldn't find the page. Maybe you were looking for one of these pages below? [Late Chunker](https://docs.chonkie.ai/oss/chunkers/late-chunker#api-reference) [Chunkers Overview](https://docs.chonkie.ai/oss/chunkers/overview#availability) [Changelog](https://docs.chonkie.ai/oss/changelog#breaking-changes) Ctrl+I --- # Page Not Found > Documentation Index > ------------------- > > Fetch the complete documentation index at: [/llms.txt](https://docs.chonkie.ai/llms.txt) > > Use this file to discover all available pages before exploring further. [Skip to main content](https://docs.chonkie.ai/api/common/introduction#content-area) 404 Page Not Found ============== We couldn't find the page. Maybe you were looking for one of these pages below? [Concepts](https://docs.chonkie.ai/common/concepts) [Embeddings Overview](https://docs.chonkie.ai/oss/embeddings/overview#common-interface) [Chunkers Overview](https://docs.chonkie.ai/oss/chunkers/overview#common-interface) Ctrl+I --- # Page Not Found > Documentation Index > ------------------- > > Fetch the complete documentation index at: [/llms.txt](https://docs.chonkie.ai/llms.txt) > > Use this file to discover all available pages before exploring further. [Skip to main content](https://docs.chonkie.ai/api/chunkers/code-chunker#content-area) 404 Page Not Found ============== We couldn't find the page. Maybe you were looking for one of these pages below? [Code Chunker](https://docs.chonkie.ai/oss/chunkers/code-chunker#api-reference) [Endpoints](https://docs.chonkie.ai/oss/api/endpoints#code-chunker-6) [Open Source](https://docs.chonkie.ai/common/open-source#codechunker) Ctrl+I --- # Page Not Found > Documentation Index > ------------------- > > Fetch the complete documentation index at: [/llms.txt](https://docs.chonkie.ai/llms.txt) > > Use this file to discover all available pages before exploring further. [Skip to main content](https://docs.chonkie.ai/api/chunkers/token-chunker#content-area) 404 Page Not Found ============== We couldn't find the page. Maybe you were looking for one of these pages below? [Token Chunker](https://docs.chonkie.ai/oss/chunkers/token-chunker#api-reference) [Endpoints](https://docs.chonkie.ai/oss/api/endpoints#token-chunker-5) [TeraflopAI Chunker](https://docs.chonkie.ai/oss/chunkers/teraflopai-chunker#how-it-works) Ctrl+I --- # Page Not Found > Documentation Index > ------------------- > > Fetch the complete documentation index at: [/llms.txt](https://docs.chonkie.ai/llms.txt) > > Use this file to discover all available pages before exploring further. [Skip to main content](https://docs.chonkie.ai/api/chunkers/sentence-chunker#content-area) 404 Page Not Found ============== We couldn't find the page. Maybe you were looking for one of these pages below? [Sentence Chunker](https://docs.chonkie.ai/oss/chunkers/sentence-chunker#api-reference) [Endpoints](https://docs.chonkie.ai/oss/api/endpoints#sentence-chunker-10) [Chunkers Overview](https://docs.chonkie.ai/oss/chunkers/overview#availability) Ctrl+I ---