Aulafy
Courses/Advanced and secure RAG/Ingestion, cleaning, and chunking

Ingestion, cleaning, and chunking

Chunking is not about cutting text every thousand characters. It is about preserving meaning, headings, tables, dates, permissions, and source so retrieval can find useful context.

  • Prepare documents before converting them into vectors.
  • Choose a chunking strategy based on document type.
  • Store metadata for permissions, citations, and auditing.

Minimum metadata

TerminalCode
{
  "document_id": "contrato-2026-001",
  "title": "Contrato proveedor",
  "source": "drive/legal/contrato.pdf",
  "page": 12,
  "section": "penalizaciones",
  "owner": "legal",
  "visibility": "internal",
  "updated_at": "2026-07-02"
}

Chunking strategies

  • By headings: manuals, policies, technical documentation.
  • By page: contracts, case files, PDFs with page-level citations.
  • By table: invoices, catalogs, inventories.
  • With overlap: narrative text where an idea spans paragraphs.