Evaluation: Making An Arabic RAG Assistant Earn Its Citations

A single “accuracy” score hides the failures that matter in citation-bound RAG. Ask Zico evaluates discovery, evidence use, citation validity, refusal, and operational completion as separate gates so a fluent answer cannot conceal a broken retriever.

The answer-generation article defines the contract under test.

Research note: RAGAS treats retrieval relevance, faithfulness to context, and generation quality as separate dimensions rather than one accuracy score.

The evaluation stack

flowchart TD
    fixtures["Curated Arabic questions and expected sources"] --> retrieval["Retrieval evaluation"]
    retrieval --> context["Context-selection evaluation"]
    context --> answer["Answer and grounding evaluation"]
    answer --> citations["Citation precision and completeness evaluation"]
    citations --> refusal["Refusal and fallback evaluation"]
    refusal --> operational["Resource and provider completion"]

Each layer answers a different question:

• Retrieval: did the expected source appear in the defined top-k?
• Context: did selection keep the evidence needed to answer?
• Answer: is the response correct and supported by supplied text?
• Citations: are cited IDs allowed, relevant, and attached to supported claims? Do material claims have coverage?
• Refusal: did the product avoid generation when evidence was insufficient?
• Operations: did CPU, storage, provider, or quota behavior prevent the intended path?

Manual Arabic review is the public-exposure gate

Before a new retrieval path, model configuration, or corpus becomes publicly reachable, a reviewer manually checks curated Arabic fixtures. The review covers dialect and spelling variation where the source material permits it, source selection, answer support, citation placement, refusal behavior, and whether each returned public citation URL resolves to the intended source.

Automated checks can catch malformed IDs and broken links; they cannot by themselves decide whether a claim is supported by the nearby evidence. A passing retrieval run is therefore necessary but not sufficient for public exposure. ALCE likewise evaluates correctness and citation quality independently rather than treating the presence of a citation as proof of support.

Retrieval suites are focused and mixed

Every library has focused fixtures that expose its vocabulary and metadata. Mixed-domain fixtures test routing and cross-library regressions. When a corpus expands, both must pass: a new collection is not successful if it silently damages an older one.

The regression matrix keeps Wa3zat, Bible Summary, and mixed-domain questions in scope today, and adds an equivalent focused suite plus the full existing regression suite for every future library.

flowchart LR
    change["Corpus or retrieval change"] --> focused["New-library focused suite"]
    change --> regression["Existing-library regressions"]
    change --> mixed["Mixed-domain routing suite"]
    focused --> gate{"All required gates pass?"}
    regression --> gate
    mixed --> gate
    gate -->|"Yes"| accept["Accept checkpoint"]
    gate -->|"No"| classify["Classify and fix failure"]

Fixture records include the question, expected source identity, applicable library or domain, top-k definition, and notes about acceptable alternatives. Stable chunk IDs make failures reproducible across local artifacts and the Worker.

Dated checkpoints, not production promises

The real fixture sets remain private because they may reveal internal test coverage or operational details. The public contract is intentionally small and representative:

{
  "id": "wa3zat-017",
  "question": "[redacted Arabic evaluation question]",
  "library": "wa3zat",
  "expectedChunkIds": ["wa3zat:sermon:017:003"],
  "acceptableChunkIds": [],
  "topK": 5,
  "expects": {
    "answer": "grounded",
    "citationPrecision": "all-supported",
    "citationCompleteness": "all-material-claims-covered",
    "refusal": false,
    "publicLinksValid": true
  }
}

An out-of-corpus fixture instead sets answer to refuse, makes the expected source list empty, and checks that the fallback does not invent a citation.

These historical build records show how decisions were gated. They do not measure live traffic or establish a production SLA.

Sample sizes are intentionally visible. “39/40” is useful engineering evidence; “97.5% accurate” would incorrectly imply a broad, stable population estimate and conflate retrieval with answer quality.

Answers and citations have separate gates

An answer can be factually plausible while citing the wrong chunk. It can also cite a relevant chunk without fully answering the question, or leave material claims uncited. Evaluation therefore records correctness, support, citation precision, and citation completeness independently.

A citation check first verifies deterministic properties: schema validity and whether every ID came from the supplied context. Human or rubric-based review then asks whether the cited evidence actually supports the nearby claim and whether material claims have sufficient citation coverage.

Refusal cases are equally important. The expected behavior for an out-of-corpus or weak-evidence question is a safe fallback, not a creative answer. These fixtures protect the conservative product boundary during model and prompt changes.

The deterministic check also resolves each accepted citation to its server-owned public URL and validates that the link reaches the intended source. Human or rubric-based review still decides whether the nearby evidence supports the claim.

Resource failures count

An eval that times out, exceeds Worker CPU, cannot reserve provider capacity, or loses a hydration record has found a system failure even if its completed examples look good. The run records completion counts and failure categories alongside quality results.

Compact context must earn its token savings

Compaction is compared with the same fixed answer-evaluation fixtures, corpus version, model/provider configuration, and completion policy as the un-compacted baseline. The comparison records retrieved and selected chunk counts, prompt-context tokens, output tokens, completion count, answer correctness, support/groundedness, citation precision and completeness, refusal correctness, link validity, latency, and any invariant failure.

This is an experiment design, not an unpublished result. It prevents a lower prompt-token count from being mistaken for success when a necessary passage was removed. CRAG similarly treats retrieved-document quality as a condition that should affect downstream behavior.

This changed the testing method. Larger suites are run in resource-safe batches, then aggregated with the batch definition preserved. Batch execution is not allowed to erase partial failures.

Production feedback becomes a regression case

The launched interface attaches lightweight feedback to the returned message ID. Operational metadata can show what routed, retrieved, and failed without publishing the visitor prompt or identity.

Private-beta feedback becomes future evaluation data

During private beta, message-linked feedback is future evaluation data rather than a public quality claim. After privacy review, a reproducible redacted question, an expected source or refusal, and a clear failure category turn it into a regression fixture.

flowchart TD
    response["Production response"] --> feedback["Message-linked feedback"]
    feedback --> triage["Private operational triage"]
    triage --> reproduce["Reproduce with public source material"]
    reproduce --> fixture["Redacted regression fixture"]
    fixture --> suites["Run retrieval and answer suites"]
    suites --> change["Ship only after gates pass"]

A downvote is not automatically an eval fixture. It first needs privacy review, a reproducible question that does not expose user data, an expected source or refusal behavior, and a clear failure category.

Reading the evidence honestly

The useful unit is a dated tuple: corpus version, fixture set, sample size, top-k, model/provider configuration, completion count, result, and limitation. Changing any part creates a new checkpoint rather than silently overwriting history.

This discipline keeps the public evidence modest and the internal feedback loop actionable. The goal is not to produce the most attractive number; it is to know which layer to fix.

Next: how the launched Arabic interface communicates these confidence boundaries.

References

• RAGAS: Automated Evaluation of Retrieval Augmented Generation
• ARES: An Automated Evaluation Framework for Retrieval-Augmented Generation Systems
• ALCE: Enabling Large Language Models to Generate Text with Citations
• Google Responsible AI practices