Confluence Coastal AI

Numeric Accuracy measures whether any generated number lands within ±15% of empirical ground truth across 32 field evaluations (4 fields × 8 regimes). The ±15% tolerance window (with a minimum floor of ±0.5 units) was chosen as an honest, pragmatic engineering benchmarking heuristic rather than derived from a single published regulatory standard. (Published hardware standards like WMO-No. 8 target tight ±0.2°C calibration targets for physical instruments; applying that to natural-language LLM outputs would artificially fail models over conversational rounding and microclimatic drift, while a ±25–30% window would credit pure guesses). When an ungrounded model answers, it predictably produces broad textbook ranges typical of Indian tropical coastlines (e.g., "temperatures 30–35°C, waves 0.5–1.5m, winds 10–15 km/h"). Because normal tropical weather frequently lands inside these wide seasonal envelopes, 23 of 32 checks matched within tolerance purely by statistical luck.

Hallucination Rate measures epistemic grounding: did the model have access to live sensor telemetry, or did it invent live readings from parametric weights? The ungrounded LLM has zero sensor access, yet asserts its guesses as verified real-time operational facts. In safety-critical maritime operations, a guess that lands on 31°C on a normal afternoon is still an ungrounded hallucination—and when critical anomalies hit (Regimes 2, 4, 5, 7), ungrounded guessing fails catastrophically.

Both RAG baselines (Dense MiniLM 65.6% [21/32], Sparse TF-IDF 56.2% [18/32]) have a 0.0% hallucination rate because they are strictly constrained to cite authentic retrieved documents (2023–2024 CPCB/INCOIS bulletins). They do not invent numbers. However, because those genuine historical records are 200 to 1,100+ days old, their numbers diverge from today's live conditions. RAG misses numeric accuracy due to temporal document staleness, not fabrication.

Actionability Scoring Rubric: Graded on a 3-tiered scale (100 = issued explicit, regime-mandated directives matching the active hazard [N95 masks, small craft recall, hydration]; 50 = vague, generic caution; 0 = omitted hazard or suppressed alert). Actionability is evaluated via tiered lexical matching, which reveals two distinct phenomena:

• Skeptical Read (Boilerplate Artifact): Ungrounded models emit sprawling, generic precautionary boilerplate ("stay hydrated, wear masks, check sea conditions"), triggering lexical credit across multiple common hazards without knowing which hazard is currently active.
• Substantive Finding (Stale RAG Alert Suppression): In contrast, RAG models are strictly anchored to retrieved text. When RAG retrieved a 2023 paper stating Kochi air was pristine, it explicitly told workers that "respiratory issues are unlikely"—scoring 0 and actively suppressing vital protection. Stale grounding proved demonstrably more dangerous than ungrounded boilerplate.