Observability: Operating A Citation-Bound Arabic RAG Assistant
For a citation-bound RAG system, “the request failed” is not enough. Operators need to know whether the failure began in routing, retrieval, hydration, context selection, provider capacity, answer validation, quota policy, or the user interface.
The UX article shows how those failures appear to visitors.
The current operating loop
Ask Zico records structured query events and feedback in D1. Protected operational summaries expose aggregates for triage; they are not public analytics endpoints. Worker logs remain useful for short-lived debugging, while D1 connects product behavior across requests and deployments.
flowchart TD
request["Assistant request"] --> event["Structured D1 query event"]
response["Returned message ID"] --> feedback["Message-linked feedback"]
event --> join["Private operational join"]
feedback --> join
join --> summary["Protected aggregate summary"]
summary --> triage["Investigate failure category"]
triage --> eval["Redacted regression fixture"]
The event stream exists to answer operational questions, not to reconstruct a visitor profile.
What needs to be observable
A useful event records bounded metadata such as:
- request class, response kind (answer mode), and explicit failure reason;
- routed domains and retrieval mode;
- candidate and hydrated-result counts;
- selected chunk IDs or source identities needed for debugging;
- vector and lexical contribution at an aggregate or ID level;
- provider/model attempts, fallback reason, and bounded input/output-token estimates;
- citation count and validation outcome;
- end-to-end latency and Worker CPU phase timings;
- quota decision and whether capacity was consumed;
- message ID for feedback joins.
The schema should make invalid states visible. A vector candidate with no KV record, a model response with a foreign citation ID, and a quota reservation that cannot be persisted are different incidents and should not collapse into error=true.
Trace the decision path
flowchart TD
received["Request accepted"] --> routed["Routing metadata"]
routed --> retrieved["Vector and lexical metadata"]
retrieved --> hydrated["Hydration counts and invariants"]
hydrated --> generated["Provider attempts and CPU phase"]
generated --> validated["Citation and grounding outcome"]
validated --> returned["Response kind and quota result"]
This is enough to reconstruct why the system chose an answer, sources-only response, search fallback, or quota state without storing a screenshot of the conversation.
Failure reasons beat averages
Average latency can look healthy while a small content library always falls back. Overall answer rate can improve while citation failures rise. Triage starts with categorical outcomes:
- access or abuse-control denial;
- quota or reservation failure;
- routing miss;
- broad or underspecified query;
- retrieval miss or weak retrieval;
- hydration invariant failure;
- content gap or insufficient evidence;
- unsupported answer;
- wrong citation, schema, grounding, or model-formatting failure;
- refusal too strict;
- provider failure or exhaustion;
- successful answer, source fallback, or search fallback.
flowchart TD
signal["Downvote or fallback spike"] --> segment["Segment by reason, library, and deployment"]
segment --> inspect["Inspect bounded retrieval and provider metadata"]
inspect --> reproduce["Reproduce with public evidence"]
reproduce --> fix["Change corpus, retrieval, prompt, or policy"]
fix --> verify["Run focused and regression evals"]
The category determines the owner. A routing miss belongs in retrieval logic; a provider exhaustion pattern belongs in capacity policy; an unsupported answer belongs in answer validation or context selection. A broad query may need a clarification path, while a repeated content gap is evidence for corpus work rather than a prompt change.
Turn traces into future evaluations
A trace or downvote is a lead, not an automatic benchmark. After privacy review, an operator reduces it to a reproducible redacted question, expected source or refusal behavior, and one failure category. That fixture enters the focused regression set, where a corpus, retrieval, prompt, or policy change must improve the case without regressing established citation and refusal checks. This closes the loop from production signal to an auditable future evaluation without retaining raw visitor details.
CPU phases are quality signals
Worker CPU is not only a billing metric. A larger lexical shard, more candidate fusion, or additional context processing can push otherwise correct requests toward runtime limits. Recording phase timings helps distinguish network wait from application work and catches corpus growth that changes the feasibility of a retrieval strategy.
This is why resource-limit failures remain valid eval and production findings. A pipeline that is accurate only when it completes is not operationally accurate.
Privacy boundary
The public portfolio intentionally contains no raw prompts, visitor identifiers, traffic counts, production thresholds, private URLs, dashboard captures, or D1 rows. Operational access is protected and the PHP boundary derives privacy-preserving quota identities rather than forwarding raw browser identifiers as authority.
Where exact screenshots would expose private data, the correct public evidence is a diagram or a dated aggregate with its definition. Redaction is part of the system design, not a publishing cleanup step.
Why a dedicated trace platform is deferred
A specialist LLM tracing platform such as Braintrust could add richer prompt comparison, span visualization, and experiment management. Ask Zico currently favors the smaller D1 event model until the thin-beta trace-and-feedback loop is proven, because it matches the product's immediate questions and keeps data ownership explicit.
That is a tradeoff, not a claim that bespoke telemetry is always better. The trigger to revisit it is concrete: when cross-step analysis or experiment volume becomes more expensive to maintain than the additional platform and privacy surface.
Next: how the same events, quotas, and fallback states control cost.