Design Alternatives: Why Ask Zico Uses Controlled Retrieval
Start with the production architecture if you have not read it yet. That article maps the deployed components. This one explains why the retrieval path keeps control in the application instead of delegating the whole decision to one managed search product.
The constraint: relevance is not enough
Ask Zico answers from mixed Arabic, Coptic, transliterated, and legacy-site material. In that setting, a result is useful only when the system can explain what it is, load its original text, and cite the same record later. A plausible snippet with an opaque ID is not sufficient evidence.
Decision 1: search systems propose candidates; the application chooses evidence
Constraint: Arabic concepts, exact names, Coptic and Arabized terms, spelling variants, and transliterations are different retrieval problems. No single score consistently describes all of them.
Alternatives: vector-only retrieval, lexical-only retrieval, or a controlled hybrid pipeline.
What I shipped: Vectorize supplies semantic candidates. Domain-scoped lexical and metadata lookups supply exact-term candidates when they are needed. The Worker applies eligibility and exclusion filters, joins candidates by stable chunk_id, combines rank and deterministic ranking signals, then hydrates the original text from KV. The model receives hydrated evidence, not a raw vector result.
flowchart TD
q["Normalized question"] --> route["Configurable semantic routing"]
route --> v["Vectorize: semantic candidates"]
route --> l["Lexical and metadata candidates"]
v --> eligibility["Apply eligibility and exclusion filters"]
l --> eligibility
eligibility --> join["Join and deduplicate by chunk_id"]
join --> rank["Controlled fusion and deterministic ranking"]
rank --> kv["KV: hydrate original text"]
kv --> context["Compact answer context"]
Dense retrieval is good at semantic similarity; lexical signals protect exact titles, names, and uncommon transliterated terms. Their scores are not interchangeable, so the Worker fuses ranks rather than pretending that one numeric scale is universally correct.
The offline controlled Wa3zat checkpoint supported continuing the controlled pipeline. The evaluation article reports the historical retrieval result with its fixture definition and limitations; it is not a permanent SLA or an ablation proving that every component caused the result.
Reranking is controlled, not a second black box
Direct vector retrieval is a candidate generator, not the final evidence order. The production shape first applies eligibility and exclusion policy, then uses controlled fusion and ranking signals such as title, structured metadata, and date-sensitive rules where relevant, followed by hydration by stable IDs. That makes a bad rank diagnosable: I can see which signal introduced a candidate, which policy changed its position, and whether the source record actually hydrated.
Decision 2: Cloudflare AI Search was not the right primary control layer
I evaluated Cloudflare AI Search against the same 40-question Wa3zat core set used for the controlled path. The test covered the serialization, managed chunking, retrieval modes, thresholds, and candidate counts configured for this project. Because some runs used different top-k settings, the observations were diagnostics rather than apples-to-apples product benchmarks.
AI Search was attractive because it bundled ingestion, embeddings, hybrid search, reranking, and generation. The internal experiment did not meet Ask Zico's requirements for identity continuity, original-text hydration, and complementary lexical recall:
| Observed in the internal experiment | Why it mattered here |
|---|---|
Returned IDs were Cloudflare-generated opaque IDs, not application chunk_id values. | They did not directly preserve the existing deterministic identity contract; hydration, citation, feedback joins, and failure reproduction would require an additional mapping or parsing layer. |
| In the tested keyword path, no candidates remained after the configured retrieval settings. | The experiment did not establish the complementary exact-term recall required for Arabic, Coptic, titles, and transliterated terms. |
| Hybrid diagnostics also showed no keyword contribution. | The experiment therefore did not demonstrate that hybrid mode supplied the required complementary lexical recall. |
| In the tested hybrid and vector configurations, no candidates remained after the configured retrieval settings for multiple questions. | With different top-k settings across runs, these observations identified a project-fit problem but did not support a universal comparison between modes. |
Managed chunking of the tested large-JSON-array representation could separate text from search_text across generated chunks. | That ingestion representation did not reliably preserve Ask Zico's two-text-surface contract. |
These are internal project observations, not a general limit of AI Search. Atomic Markdown improved this experiment, but not by much. Likewise, opaque IDs did not make hydration or citation inherently impossible; they would have required mapping or parsing rather than directly preserving the existing identity contract.
The resulting design keeps Vectorize as one controlled semantic signal and keeps chunking, lexical retrieval, fusion, hydration, and citation meaning under application control.
Decision 3: semantic domains are routing labels, not user-facing filters
The application uses configurable internal labels such as bible, taqs, al7an, saints, and ta3lim to decide where to look first. These labels can evolve with the corpus and describe retrieval meaning, not a menu the user must understand or a public claim that every question belongs to exactly one library. Citation and source identity remain stable within the deployed corpus contract even as routing labels change.
flowchart TD
q["Natural-language question"] --> score["Score possible semantic domains"]
score --> one["One relevant domain"]
score --> many["Several relevant domains"]
one --> retrieve["Search eligible sources"]
many --> retrieve
retrieve --> filters["Apply source-library flags and request policy"]
- A user asks normally; they do not have to select a corpus before asking.
- Ambiguous questions can search more than one domain instead of being forced into a false single bucket.
- A semantic domain can guide retrieval while feature flags and source-library metadata decide what is enabled in a deployment.
In the Worker, the router scores candidate domains, returns a default multi-domain route for weak signals, and intentionally expands broad questions across relevant areas. The label vocabulary can evolve without changing URLs, citations, or the user's mental model. The durable design principle is that routing is a reversible retrieval hint, not user-facing taxonomy.
Shipped tradeoffs
| Decision | Benefit in production | Cost accepted |
|---|---|---|
| Controlled hybrid retrieval | Complementary recall and explainable evidence selection | More application code and evaluation work |
| Stable application IDs and hydration | Reproducible citations, feedback joins, and debugging | Ingestion must preserve the identity contract |
| Application-owned fusion and ranking | Signals and policy are inspectable and tunable | Ranking needs continued regression evaluation |
| Internal multi-domain routing | Users ask naturally while retrieval stays focused | Routing rules require maintenance as libraries grow |
Control has a maintenance cost. It was justified here because every incident needs an answerable chain: what routed, which candidates appeared, what was fused, what hydrated, what the model saw, and what it cited.
Next: building stable, citation-ready ingestion artifacts.