Grounded Answer Generation: Turning Retrieved Arabic Evidence Into Safe Answers

Retrieval answers “what evidence might matter?” Answer generation has a stricter job: produce useful Arabic prose without extending beyond that evidence, or decline to generate when the contract cannot be met.

Read the retrieval pipeline first for how evidence reaches this layer.

A small answer contract

The model receives a compact set of original source chunks labeled with server-issued citation IDs. It returns a deliberately small structured payload: answer text and the IDs it used. It does not invent URLs, titles, or source objects.

Sanitized illustrative contract

This is a fabricated teaching excerpt, not a production prompt or a copy of internal policy:

Use only the supplied evidence chunks. Cite the supplied evidence IDs for each material claim.
Do not create URLs, titles, sources, dates, or liturgical details that the chunks do not support.
If the evidence is missing, conflicting, or too weak, return the refusal state so the server can show sources or search instead.

The contract asks for answer text and evidence IDs, not trusted source metadata. The model receives no URLs, canonical titles, or internal policy details; the Worker retains those until validation succeeds.

Religious and liturgical claims fail closed

Questions about rites, fasts, sacramental practice, hymn timing, or liturgical dates have a higher cost of being plausible but wrong. Ask Zico synthesizes these claims only when the supplied chunks directly support the answer. Missing, conflicting, or ambiguous evidence is not resolved by general model knowledge, a weaker matching chunk, or a second provider: the response becomes source cards or site search. This is a product safety boundary, not a claim that a generic benchmark can certify a tradition-specific answer.

flowchart TD
    retrieved["Hydrated candidates"] --> select["Compact context selection"]
    select --> prompt["Evidence-bound prompt"]
    prompt --> model["Model returns answer and IDs"]
    model --> validate["Server validates schema and citation IDs"]
    validate -->|"Valid and supported"| answer["Answer with source cards"]
    validate -->|"Invalid or unsupported"| fallback["Sources or search fallback"]

The Worker resolves accepted IDs back to trusted metadata. This makes a citation a server-validated relationship between the answer and supplied evidence, not decorative model output. Citation quality needs independent measurement: ALCE evaluates correctness, completeness, and style separately, rather than treating the presence of a citation as proof of support.

Compact context is enabled

The launched configuration retrieves broadly and answers narrowly. Compact context selection removes redundant chunks, trims each retained chunk to its query-focused excerpt, and limits prompt size while preserving the strongest coverage. The prompt uses short server-issued evidence IDs beside those excerpts; URLs, titles, and source cards remain server-side until after validation. This reduces token use and makes it harder for a weaker chunk to distract the model. Research on adaptive retrieval likewise finds that irrelevant or indiscriminately fixed context can make generation less useful; Self-RAG treats retrieval relevance as a separate decision.

Compaction is not assumed safe. It is evaluated against the un-compacted path because less context can also remove a needed supporting passage. Retrieval success and context sufficiency therefore remain separate measurements.

Progressive context is implemented as a guarded option but is disabled in the current production configuration. The system does not automatically expand through successive model calls. That avoids presenting an experimental cost and latency strategy as shipped behavior.

Reasoning and output budgets are measured settings

Reasoning effort and maximum output tokens are configuration inputs, not universal quality settings. Ask Zico compares candidate settings against the same dated fixture and model ladder, recording answer correctness, chunk support, citation completeness, refusal precision, Arabic readability, latency, and cost. A larger reasoning or output budget is retained only when its measured quality gain justifies its operational cost; otherwise the more compact setting wins.

Citation provenance validation is deterministic

flowchart TD
    output["Structured model output"] --> schema{"Valid schema?"}
    schema -->|"No"| fail["Fallback"]
    schema -->|"Yes"| ids{"Every cited ID was supplied?"}
    ids -->|"No"| fail
    ids -->|"Yes"| publish["Resolve metadata and return"]

The prompt asks the model to stay inside evidence, but the prompt is only one guardrail. Parsing and allowed-ID checks are deterministic application controls; support is a separate threshold, rubric, or human-evaluation question. A prompt cannot enforce an invariant that the server never verifies. This matters because RAG can still produce unsupported or contradictory claims even when retrieval has supplied documents, as documented by RAGTruth.

Refusal and fallback are different useful states

Ask Zico can return:

• a grounded answer with validated citations;
• source cards when retrieval is useful but generation is unavailable or unsafe;
• a site-search fallback when the system lacks enough evidence;
• a quota state that explains why generation is temporarily unavailable.

These states preserve user agency. The product can still help a visitor reach material without manufacturing a direct answer.

Appropriate abstention is evaluated as a useful outcome, not as a blank failure. Research on knowledge gaps similarly treats the ability to decline unanswerable questions as a measurable capability (Don't Hallucinate, Abstain).

Provider fallback does not repair knowledge

As of July 21, 2026, the Worker reserves capacity across a configured Gemini model ladder and can try OpenRouter when Gemini has a qualifying provider-layer failure or no reserved capacity. The provider and model list can evolve.

flowchart TD
    context["Validated evidence context"] --> reserve["Reserve provider capacity"]
    reserve --> primary["Gemini model ladder"]
    primary -->|"Provider succeeds"| validate["Validate grounded output"]
    primary -->|"Provider failure or exhaustion"| secondary["OpenRouter route"]
    secondary --> validate
    validate -->|"Grounding fails"| safe["Do not try providers for a better story"]
    validate -->|"Grounding passes"| response["Return answer"]

This boundary is important: provider fallback solves availability, not unsupported knowledge. If a model saw the evidence and still produced an invalid or unsupported answer, another fluent answer is not automatically safer.

Follow-ups retrieve again

A follow-up can depend on prior context, but the previous generated answer is not evidence. The Worker can hydrate previously cited chunks to preserve the subject, then runs fresh retrieval for the new question. The final context is again bounded and citation-validatable.

This avoids two common chat failures: losing the topic entirely and accumulating unsupported claims across turns.

What answer evaluation checks

A generated response is evaluated independently on:

• answer correctness against the reference material;
• support by the supplied chunks;
• citation precision and completeness, including whether every ID is allowed and relevant and whether material claims have supporting citations;
• appropriate refusal when evidence is insufficient;
• schema and Arabic readability;
• provider and resource failures as operational findings.

Historical answer-eval results are engineering checkpoints tied to their fixture, model, configuration, and date. They are not promises that every production question will receive a correct generated answer. This separation follows the evaluation shape described by RAGAS: retrieval focus, answer faithfulness, and response quality can fail independently.

The durable lesson

Grounding is a system property. It comes from stable evidence identity, constrained context, structured output, deterministic validation, and safe response states. Model instructions help, but they cannot substitute for those controls.

Next: the evaluation system that tests retrieval, answers, citations, and refusals separately.

References

• Google Gemini API structured output
• OpenRouter provider routing
• ALCE: Enabling Large Language Models to Generate Text with Citations
• Self-RAG: Learning to Retrieve, Generate, and Critique through Self-Reflection
• RAGAS: Automated Evaluation of Retrieval Augmented Generation
• RAGTruth: A Hallucination Corpus for Developing Trustworthy Retrieval-Augmented Language Models
• Don't Hallucinate, Abstain: Identifying LLM Knowledge Gaps via Multi-LLM Collaboration