Ingestion Pipeline: Turning Arabic Site Content Into Citation-Ready RAG Chunks

Retrieval quality starts before the first embedding is created. Ask Zico's ingestion contract is designed so a result can be found flexibly but cited exactly: normalize for search, preserve readable evidence, and give every chunk one deterministic identity within a corpus build.

The previous article explains why the system owns this pipeline.

The core record

Each chunk carries two different text fields:

• text is the extractor-emitted readable source excerpt used for evidence and hydration. It is stored unchanged after extraction, which does not imply byte-for-byte preservation of the source page.
• search_text is a normalized representation used only for retrieval: lexical matching and the embedding input. It is never substituted into an answer citation or displayed as source evidence.

For example, a visitor may type فين اتولد السيد المسيح؟ while a source uses وُلِدَ السيد المسيح في بيت لحم. Search normalization can reduce diacritics and punctuation differences without degrading the Arabic shown in the answer context.

flowchart TD
    source["Public source document"] --> extract["Extract readable text and metadata"]
    extract --> chunk["Create bounded chunks"]
    chunk --> identity["Assign deterministic chunk ID"]
    identity --> original["Store extractor-emitted text unchanged"]
    identity --> normalized["Build normalized search_text"]
    original --> artifacts["JSONL and KV lookup"]
    normalized --> artifacts

This separation prevents an easy mistake: optimizing the only copy of the text for retrieval until it is no longer suitable evidence.

Here is a deliberately fabricated, sanitized record. It shows the contract, not a production source or identifier:

{
  "doc_id": "bible-summary:birth-of-christ",
  "chunk_id": "bible-summary:birth-of-christ:003",
  "text": "وُلِدَ السيد المسيح في بيت لحم.",
  "search_text": "ولد السيد المسيح في بيت لحم",
  "title": "ميلاد السيد المسيح",
  "url": "/bible-summary/birth-of-christ",
  "chunk_index": 3,
  "domains": ["bible"]
}

doc_id identifies the source document; chunk_id identifies one deterministic chunk position within that corpus build. The record may hold additional source-specific metadata, but the retrieval contract stays small and inspectable.

Deterministic identity within a corpus build

A doc_id is derived from source identity, and a chunk_id is derived from that document identity plus its chunk position—not from a provider upload response. This makes IDs deterministic for identical source inputs and chunking rules within one corpus build. It does not make position-derived IDs durable across content edits: inserting text or changing chunk boundaries can shift later IDs.

Within one build, the same chunk ID appears in:

• generated JSONL;
• the Vectorize vector ID;
• lexical-domain shards;
• the KV hydration record;
• expected-source eval fixtures;
• answer citations and operational events.

Vectorize, lexical shards, and hydration records therefore need coordinated replacement, or an explicit corpus version that prevents artifacts from different builds being mixed at runtime. If a retriever returns an ID that cannot be hydrated in the active corpus version, the pipeline has found an invariant failure, not a harmless missing field.

Chunking is product work

Chunk boundaries determine what can be retrieved, cited, and understood together. The chunker therefore follows source structure where available, bounds passage size, and is reviewed against real questions rather than treated as a one-time preprocessing default. A split that separates a title from its answer, joins unrelated page sections, or pulls navigation chrome into a passage is a retrieval defect.

Changes to chunking are accepted only after manual samples remain readable and focused plus the relevant focused and regression fixture sets show no retrieval regression. This makes chunking policy, source parsing, and evaluation one product decision.

Choosing too small chunks is bad because the answer might overlap different chunks, and choosing too large chunks makes it difficult and more costly to search. In our case, a chunk of 7k characters was good enough to fit an individual article into a single chunk.

Artifact flow

flowchart TD
    mdx["Curated content sources"] --> generator["Library-specific extraction"]
    generator --> jsonl["Chunk JSONL"]
    jsonl --> inspect["Manual UTF-8 and metadata inspection"]
    inspect --> embeddings["Embedding preparation"]
    embeddings --> vectorize["Vectorize index"]
    jsonl --> lexical["KV: lexical/domain shards"]
    jsonl --> chunks["KV: original chunk records"]
    question["Runtime question"] --> vectorize
    question --> lexical
    vectorize --> rank["Fuse and rank candidate IDs"]
    lexical --> rank
    rank --> chunks
    chunks --> context["Hydrated evidence context"]

The artifact set is intentionally redundant. Vectorize is optimized for semantic candidates. KV holds two logical artifact families: lean lexical and domain shards read before fusion and ranking, and original chunk records hydrated only after ranking. Both families use the same KV storage system; the separation describes their runtime roles, not separate stores. This avoids putting full source text into every vector result or lexical scan.

At a high level, the ingestion tooling has five inspectable stages:

1. source parsers extract main content and metadata while excluding site navigation, repeated chrome, and other non-evidence text;
2. a JSONL generator creates deterministic document and chunk records for the corpus build;
3. a chunk-lookup build produces hydration records keyed by chunk_id;
4. a lexical-preparation build produces compact domain shards from search_text; and
5. manual sample inspection checks UTF-8, boundaries, IDs, metadata, and lookup coverage before upload.

A lookup artifact is intentionally boring and easy to inspect:

chunk_id=bible-summary:birth-of-christ:003 → title, url, text, domains

The lexical artifact is separate and contains retrieval-oriented fields, for example domain=bible, chunk_id=…:003, and normalized tokens. Neither artifact replaces the original text used for citation.

Metadata that survives retrieval

Useful metadata includes the public URL, title, library, semantic domains, chunk position, and source-specific fields needed for filtering. Date-sensitive Seneksar material, for example, needs structured date information that the runtime can compare with the question. A title string alone is not enough.

Library support and library availability are also different claims. The ingestion system can produce artifacts for a collection while runtime feature flags decide whether that collection is enabled in a particular environment. This permits staged corpus expansion without implying that every supported library is live everywhere.

Arabic normalization is a retrieval tool

The search surface uses two classes of transformation. Conservative character and formatting normalization reduces superficial variation. More lossy, recall-oriented aliases are isolated and additive rather than substituted into text. The extractor-emitted Arabic, Coptic, and transliterated strings stored in text remain unchanged after extraction.

Even useful normalization can change meaning or create collisions. I accept a rule only when a focused retrieval case demonstrates its value and collision-regression checks do not expose unacceptable ambiguity; otherwise I narrow or reject it.

flowchart TD
    raw["Extractor-emitted readable text"] --> keep["Store unchanged after extraction"]
    raw --> conservative["Apply conservative search normalization"]
    conservative --> aliases["Add isolated recall-oriented aliases where needed"]
    aliases --> test["Run focused and collision-regression checks"]
    test -->|"Accepted"| keepRule["Keep the rule"]
    test -->|"Ambiguous or regressive"| rejectRule["Reject or narrow the rule"]

Manual inspection remains a gate

Automation catches schemas; humans catch damaged language. Before upload, representative records are inspected for:

• readable Arabic with no mojibake or escaped prose;
• sensible chunk boundaries;
• no navigation, footer, breadcrumb, or repeated page chrome in the evidence text;
• correct public URLs and titles;
• deterministic IDs across reruns with identical inputs and build settings;
• valid library and domain metadata;
• lookup coverage for every Vectorize upload record and vector ID.

This is especially important when the source mix includes Markdown, generated pages, Coptic, transliterated names, and content-specific metadata.

Published build checkpoints

The following public rows are milestone-labeled summaries of historical ingestion-and-retrieval checkpoints, not production SLAs. “Top 5” means the expected source appeared among the first five retrieved results for the stated fixture set. Exact run dates, build identifiers, and detailed reports remain private.

The value of these numbers is the comparison they enabled: corpus expansion could be accepted only when the new library worked and older libraries did not silently regress.

The implementation scripts and detailed reports remain in the private Ask Zico source repository. The figures published here are aggregate checkpoints with stated fixture sizes and limitations, not independently reproducible public reports.

What transfers to other RAG systems

Preserve readable evidence, normalize into a separate field, own deterministic IDs, scope them to an explicit corpus version, and replace related artifacts coherently. Tie every ingestion change to focused plus regression retrieval tests. The specific Arabic rules and library metadata remain product-specific.

Next: the runtime hybrid retrieval pipeline.

References

• Cloudflare Vectorize metadata and filtering
• Cloudflare Workers KV documentation
• Arabic Natural Language Processing: Challenges and Solutions
• Arabic Diacritics in the Wild: Exploiting Opportunities for Improved Diacritization
• QCRI Arabic Normalizer
• Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks
• Comparative Evaluation of Advanced Chunking for Retrieval-Augmented Generation