[Feature] Span-level context confidence scores in multi-hop RAG retrieval — per-span quality signals for observability

[Ruthwik-Data]

Background

Phoenix is excellent for LLM/RAG tracing and observability. I've been using it for my finrag-eval project, which evaluates multi-hop RAG pipelines over financial documents.

Phoenix traces capture the full span tree for retrieval chains, but the span-level data doesn't expose context quality signals — only the final retrieved context and output.

The Gap

For multi-hop RAG pipelines, the most important observability signal isn't the final output quality — it's which retrieval span degraded the chain.

Currently, Phoenix shows:

Span 1: retrieval (0.8s) ✓
Span 2: reranking (0.3s) ✓
Span 3: synthesis (1.2s) ✓
Final answer: [text]

What's missing:

Span 1: retrieval — context_confidence: 0.92, retrieved_chunks: 5
Span 2: reranking — context_confidence: 0.65, chunks_retained: 2  ← DEGRADATION HERE
Span 3: synthesis — context_confidence: 0.65 (inherited from upstream)
Final answer quality: 0.61 (directly caused by Span 2 degradation)

Feature Request

Add span-level context confidence as a first-class attribute in Phoenix's RAG span schema:

# Proposed: Phoenix span attributes for RAG spans
span.set_attribute("retrieval.context_confidence", 0.92)  # Per-span quality score
span.set_attribute("retrieval.chunks_retrieved", 5)
span.set_attribute("retrieval.chunks_above_threshold", 5)
span.set_attribute("reranking.context_confidence", 0.65)  # Post-rerank quality
span.set_attribute("reranking.confidence_delta", -0.27)   # Signal: quality dropped here

This would enable Phoenix's existing evaluation framework to attribute final answer quality degradation back to specific retrieval spans — closing the loop between traces and evals.

Why This Matters

For production RAG in financial/regulated domains, debugging "why did this answer fail?" requires span-level attribution, not just final output quality. Phoenix already has the trace infrastructure — adding context_confidence as a standard span attribute would make it the best tool for retrieval chain debugging.

Minimal Implementation Path

1. Add context_confidence to the OpenInference RAG span spec
2. Expose it in Phoenix's trace detail view as a per-span quality indicator
3. Allow filtering/sorting spans by context_confidence in the UI
4. Add confidence_delta between parent/child retrieval spans — surfaces degradation points automatically

Related Work

I explored this problem in my finrag-eval project and had to instrument span confidence manually using custom attributes. Standardizing this in Phoenix/OpenInference would make it available to the whole ecosystem.

[cschanhniem]

This is a solid proposal. The key design question is whether context_confidence should be a generic span attribute (like input.value or output.value) or a semantic convention with evaluator integration.

The generic approach (treat it as a numeric span attribute the user sets) is simpler to implement but misses the observability loop you described — Phoenix's eval framework would need explicit wiring to read arbitrary span attributes for attribution.

The semantic approach would add context_confidence as a first-class field on RAG-related span kinds in the OpenInference spec, which lets Phoenix's evaluator framework auto-attribute answer_relevance degradation to the lowest-confidence retrieval span without per-project configuration. This is the harder path but the one that actually closes the loop.

A pragmatic middle ground: define context_confidence as a standard annotatable span attribute (not a span kind field) with a confidence_delta computed in the UI by comparing parent/child spans sharing the same annotated key. This way:

1. Any span can carry the signal (generic)
2. The UI can compute delta automatically (useful)
3. Eval attribution can be added incrementally later without changing the data model

I'd also suggest the name retrieval.quality_score rather than context_confidence — it's more discoverable alongside the existing OpenInference retrieval.documents convention, and "confidence" has overloaded meaning in ML (model confidence scores vs retrieval quality).

On the agent observability side— this kind of per-span quality signal is exactly what governance frameworks like AGENTOWNERS (github.com/cschanhniem/AGENTOWNERS) need to surface: when an agent makes decisions based on degraded retrieval, you need trace-level attribution to determine whether it was a model error or a data-quality failure. Happy to discuss the spec direction further.

[axiomofjoy]

transferring this issue to the openinference repo

[Necmttn]

I would attach confidence as an evaluator receipt, not only a numeric attribute.

For each retrieval/rerank span, keep score, score bucket, evaluator name/version, query hash, corpus or index snapshot, retained document ids, and upstream span link. Then the final answer evaluator can point to the span where confidence dropped without treating the score as a universal truth.

That matters for multi-hop RAG: a 0.65 score from one evaluator and corpus snapshot should be comparable inside that run, but not silently comparable across every app.

---

Generated with ax.