Qaf · Retrieval · 22 Oct 2026Cohere Rerank 5 vs our current reranker
280 real production searches, replayed through prod retrieval on our Azure Foundry resource and graded passage-by-passage by a 3-model judge panel.
Switch to Rerank 5 Pro. It ranks as well as v4 Pro (no measurable difference) at ~70% lower cost: about $2.6k/month instead of $8.6k at today's ~114k searches/day. It adds ~250 ms per search.
Don't use Rerank 5 Fast. It is cheaper still ($1.0k/month) but delivers measurably worse passages than v4 Pro, and it isn't faster than v4 Pro either.
Quality of the passages the model actually receives
Each search scored at the passage count prod delivered (5, 10 or 15 by chat tier). 245 searches from the current config.
| Reranker | nDCG | Relevant share | Δ vs v4 Pro | Wins / losses |
|---|---|---|---|---|
| v4 Pro (today) | 0.847 | 83.0% | — | — |
| v5 Pro | 0.851 | 82.4% | +0.004 (±0.014, n.s.) | 100 / 92 |
| v5 Fast | 0.834 | 81.4% | −0.013 | 91 / 101 |
| v4 Fast | 0.817 | 80.0% | −0.030 | 69 / 118 |
| No reranker | 0.712 | 70.1% | −0.136 | 36 / 185 |
Relevant share = delivered passages graded ≥2 of 3 (clearly relevant). v5 Fast's lower average grade vs v4 Pro is significant (−0.037, p=0.009); v5 Pro vs v4 Pro is a tie on every metric. On the full 280-search sample, each of the three judges on its own also puts v5 Pro at or above v4 Pro.
Cost and latency
Monthly cost at ~114k reranks/day (Azure Monitor, Oct 14–21).
| Model | Billing | Per 1K searches | p50 latency | p95 latency |
|---|---|---|---|---|
| v4 Pro | $2.50 / 1K searches | $2.50 | ~620 ms | ~1.1–1.7 s |
| v5 Pro | $0.05 / 1M input tokens | $0.77 | ~850–950 ms | ~1.5 s |
| v5 Fast | $0.02 / 1M input tokens | $0.30 | ~750–800 ms | ~1.4 s |
- v5 bills by tokens, not searches. Our pools average ~46 passages ≈ 15.3k tokens per search, so cost scales with candidate depth and chunk length. v4 bills a flat unit per search.
- Latency is from 3 passes (510 calls per model) on day 2 after launch. v5 Fast is not faster than v4 Pro on our pools, despite Cohere's "3× faster than Pro" claim. Re-check in a few weeks.
- This spend draws on Azure credits, not cash: v5 deployed on our sponsorship subscription, which only allows first-party models. Azure hasn't published a v5 price meter yet, so the $0.05 / $0.02 rates are Cohere's list prices. Confirm on the first day's invoice.
Worth trying next
v5's scores are calibrated; v4's aren't. Passages v5 Pro scores ≥0.8 are 96% relevant, and those below 0.1 are 2% relevant. Dropping delivered passages below Cohere's 0.5 cutoff would cut ~12% of passages at k=10 while losing ~6% of the relevant ones. That trims tokens off the far larger LLM bill, and needs an answer-level test before shipping.
Method and caveats
- Queries: 280 distinct production
searchcalls from Langfuse, 40 per day for Oct 15–21. Filtered searches were excluded. All are model-written Arabic queries. - Retrieval: prod's own hybrid search (24 semantic + 24 BM25, RRF). Each model reranked the identical pool, so one set of grades scores every config. v4 Pro's replayed top-k matches what prod actually delivered 94% of the time.
- Judging: GPT-6.1 Sol, Gemini 3.1 Pro and Gemini 3.7 Flash graded all 12.7k passages 0–3 (mean of three). Passages were shuffled, and 4 unrelated passages were planted per query as a control: 99%+ graded 0, none graded ≥2. Judge agreement (weighted κ) was 0.75–0.86.
- Verification: three independent re-analyses. They caught that Oct 15 searches ran under the retired 20-passage config, and that fixed nDCG@10 overstated v5 Pro (+0.014). Both are corrected above. On the full sample, all configs keep the same order.
- Not measured: answer quality. In Aug, a +0.15 retrieval gain moved answers only slightly, so a tie here means no answer-level change is expected. Prod's session de-duplication was not replayed, which mainly affects follow-up searches.