TeleRadShift — Worst-Group Conformal Risk Control for Safe VLM Triage

Overview

Abstract

Chest radiographs often reach clinicians in low-resource telemedicine settings as display-recaptured smartphone images rather than native digital studies. These images introduce structured acquisition artifacts that create clinically meaningful distribution shift for radiology vision-language models (VLMs). Standard Conformal Risk Control (CRC) enforces average-risk guarantees, but it can still violate worst-group safety on harder corruption subgroups — a failure that remains invisible under aggregate evaluation.

We trace this behavior to cross-modal decoupling: when visual evidence is degraded, the language decoder remains fluent and confident, producing disagreement across stochastic samples without correspondingly high token entropy. Motivated by this effect, we introduce Findings-Disagreement-Variance (FDV), an uncertainty score combining entropy and cross-sample disagreement, together with Worst-Group Conformal Risk Control (WG-CRC), a post-hoc calibration framework targeting subgroup-level safety without retraining.

On TeleRadShift, a structured 15-group benchmark, WG-CRC achieves 0/15 violations at 23.9% coverage, an 8.9× improvement over Bonferroni at matched safety. The method transfers without recalibration to 1,800 real smartphone captures (0/9 violations, worst-group risk 0.492) and remains stable across 30 random splits (worst-group risk 0.506 ± 0.022).

Code: github.com/gen-ai-research/teleradshift

Summary

Key Contributions

(1) Failure Mode Analysis

Standard CRC satisfies global risk guarantees while silently violating worst-group safety under structured telemedicine acquisition shift — an undetected deployment hazard invisible to conventional evaluation.

(2) FDV + WG-CRC Framework

Post-hoc calibration combining disagreement-based uncertainty with group-conditional conformal bounds. Achieves 0/15 group violations at 23.9% coverage — an 8.9× improvement over Bonferroni at matched safety.

(3) TeleRadShift Benchmark

15 synthetic corruption groups (5 families × 3 severities) plus 1,800 real smartphone captures for systematic evaluation of VLM reliability under realistic telemedicine conditions.

Technical Approach

Method

Figure 1 — TeleRadShift Overview

A telemedicine chest X-ray undergoes structured acquisition corruptions (blur, brightness, JPEG, occlusion, screen). N=6 stochastic VLM samples are drawn, each parsed into binary findings. FDV uncertainty (entropy H and cross-modal disagreement D) is computed and used to calibrate a per-group abstention threshold τ̂_g. At inference, FDV(x) < τ̂_g delivers a risk-controlled report; otherwise the case is deferred to a radiologist.

Corruption Simulation

15 groups: 5 families (Gaussian blur, brightness shift, JPEG compression, screen re-photography, occlusion) × 3 severity levels applied to 11,733 CheXpert chest X-rays.

Stochastic VLM Sampling

CheXagent-8B generates N=6 stochastic reports per image at T=0.8 with nucleus sampling (top-p=0.9). Each report is parsed into binary pathology predictions ŷ ∈ {0,1}^L.

FDV Uncertainty Score

Combines normalized entropy H̃(x) and pairwise disagreement D(x). Disagreement achieves Spearman ρ=0.606 — nearly 3× stronger than entropy alone (ρ=0.197).

FDV(x) = 0.5 · H̃(x) + 0.5 · D(x)

Worst-Group Conformal Calibration

Per-group thresholds calibrated at ε_cal = ε − Δ*, where Δ* adds base margin Δ and Hoeffding's finite-sample correction. For TeleRadShift: Δ=0.04, correction≈0.104, ε_cal=0.356.

Δ* = Δ + √( log(2G/δ') / 2n_min )

Cross-Modal Decoupling. Under acquisition corruption the visual encoder weakens while the language decoder stays confident, reproducing fluent outputs for findings it cannot visually resolve. Repeated sampling of the same corrupted image yields contradictory conclusions — entropy stays low but disagreement rises, exposing the failure mode.

Threshold Sensitivity — τ vs. Coverage and Worst-Group Risk

As threshold τ increases, both worst-group risk and coverage rise. The safe zone (below ε=0.50) is entered at τ≈0.38. WG-CRC selects the coverage-maximising τ̂_g per group calibrated at ε_cal=0.356.

Figure 3 — Uncertainty Signal Quality by Corruption Type

Disagreement D dominates entropy H across all five corruption families (D: 0.57–0.67, H: 0.16–0.23). FDV combines both signals for calibration stability and coverage.

Experiments

Results

§5.1 — WG-CRC Achieves Worst-Group Safety

Evaluated on 5,333 test images across 15 corruption groups (ε=0.50). WG-CRC is the only method achieving zero violations with clinically meaningful coverage. Per-Group CRC without the buffer Δ fails 11/15 groups — showing Δ is structurally necessary, not a post-hoc tuning device. Bonferroni is safe but certifies only 2.7%; WG-CRC certifies 23.9% — an 8.9× improvement.

Table 1 — Main Results on TeleRadShift (n=5,333, 15 corruption groups, ε=0.50)

Method	Avg. Risk ↓	Worst-Group Risk ↓	Coverage ↑	Violations	Safe?
No Abstention	0.635	0.696	100.0%	15/15	✗
Self-Consistency (H only)	0.548	0.670	16.6%	14/15	✗
Per-Group CRC (no Δ)	0.521	0.588	39.1%	11/15	✗
Bonferroni (Hoeffding)	0.304	0.467	2.7%	0/15	✓
Worst-Group Tuned Global	0.387	0.436	12.9%	0/15	✓
Minimax (cov ≥ 23.9%)	0.464	0.516	30.6%	3/15	✗
Vanilla CRC	0.472	0.528	33.6%	3/15	✗
WG-CRC + FDV (ours)	0.450	0.500	23.9%	0/15	✓

Figure 2 — Coverage vs. Worst-Group Risk Frontier

WG-CRC is the only method below ε=0.50 with clinically meaningful coverage. Points above the dashed line violate safety; lower-left methods are safe but unusable. The gap between Vanilla CRC and WG-CRC reveals the hidden failure of average-risk calibration.

§5.2 — Disagreement Dominates Entropy Under Acquisition Shift

Disagreement achieves Spearman ρ=0.606 with diagnostic risk (n=5,000, p < 10⁻¹⁰⁰), while entropy achieves only ρ=0.197 — nearly three times weaker. FDV achieves ρ=0.492, capturing disagreement's predictive power while gaining coverage stability from entropy.

Table 2 — Spearman ρ between Uncertainty Signals and Diagnostic Risk

Signal	ρ (corrupted) ↑	ρ (clean)	Range
H (entropy)	0.197	0.121	0.16–0.23
D (disagreement)	0.606	0.244	0.57–0.67
FDV (α=0.5)	0.492	0.171	0.47–0.51

FDV Score vs. Diagnostic Risk — by Corruption Type

Consistent positive correlation (ρ=0.47–0.51) across blur, brightness, JPEG, occlusion, and screen. This family-wise consistency confirms FDV as a reliable risk proxy under all tested acquisition artifacts.

§5.3 — Real-World Validation: Smartphone Captures

200 real chest X-rays photographed on three consumer smartphones (iPhone 13 Pro Max, Samsung Galaxy S23, Google Pixel 7) under three lighting conditions → 1,800 total captures across G'=9 acquisition groups. WG-CRC thresholds applied directly from synthetic CheXpert calibration — no phone-specific tuning.

Figure 4 — Real Telemedicine Capture Examples

Naturalistic artifacts absent from synthetic calibration: (a) occlusion from hand placement, (b) display moiré from sensor-monitor interference, (c) perspective distortion from off-axis capture, (d) low-light sensor noise.

FDV achieves Spearman ρ=0.492 on real captures, consistent with the synthetic test-set range (0.471–0.514). WG-CRC maintains worst-group risk 0.492 ≤ ε=0.50 with 24.3% coverage; Vanilla CRC violates safety in 1/9 groups.

Table 3 — WG-CRC on Real Smartphone Captures (1,800 images, G'=9 groups)

Method	Coverage ↑	Violations ↓	Worst Risk ↓
Vanilla CRC	21.2%	1/9	0.531
WG-CRC (ours)	24.3%	0/9	0.492

§5.4 — Robustness Analysis

Figure 5 — 30-Seed Stability

WG-CRC: mean 0.506±0.022, 17/30 seeds achieve zero violations, 27/30 at most one. Vanilla CRC violates ε=0.50 in all 30 runs, averaging 4.8 group violations per run.

Multi-seed stability. 17/30 seeds produce zero violations; 27/30 at most one. Four seeds exceed ε+Δ*=0.54, within the expected range under a 5% per-split failure probability (Proposition 1). α=0.5 is chosen over α=0.2 for a 2.1× reliability improvement despite modest coverage reduction.

Held-out corruption generalization. With screen family withheld from calibration, the global fallback τ̂_min achieves worst-group risk 0.470 and 31.7% coverage — matching Vanilla CRC and only 0.8% below oracle assignment.

Table 4 — Held-Out Screen Corruption Family

Calibration	Worst Risk ↓	Coverage ↑	Safe?
Vanilla CRC (4 families)	0.470	31.7%	✓
WG-CRC (4f, fallback)	0.470	31.7%	✓
WG-CRC (5f, oracle)	0.464	30.9%	✓

Cross-dataset transfer. On NIH ChestX-ray14 (zero-shot, n=15,000), FDV achieves Spearman ρ=0.513 — consistent across all five families (range 0.511–0.520). The disagreement signal generalizes across institutions, scanner types, and label schemas.

Figure 6 — FDV–Risk: CheXpert (calibration) vs. NIH ChestX-ray14 (zero-shot)

On NIH, disagreement D and entropy H achieve similar correlations (both ρ≈0.51) — contrasting with CheXpert where D substantially outperformed H. FDV's combined signal remains robust across institutions.

Design Choices

Ablation Studies

FDV Mixing Weight α

Using D alone (α=0) satisfies the constraint but yields only 12.7% coverage. α=0.2 gives higher single-split coverage (27.6%) but only 8/30 zero-violation seeds (27%) vs. 17/30 (57%) for α=0.5 — a 2.1× reliability improvement. Values α>0.5 progressively degrade safety.

Table 5 — WG-CRC Across α

α	Worst Risk ↓	Coverage ↑	Violations
0.0 (D only)	0.577	12.7%	1/15
0.2	0.498	27.6%	0/15
0.5 (proposed)	0.500	23.9%	0/15
0.8	0.549	8.6%	2/15
1.0 (H only)	0.670	16.6%	14/15

Calibration Buffer Δ

Zero violations hold for all Δ ∈ [0.04, 0.08] — confirming Δ=0.04 is the coverage-optimal value within a stable feasible range, not a cherry-picked optimum. Values Δ≤0.03 produce 1–3/15 violations; Δ≥0.05 reduces coverage unnecessarily.

Table 6 — WG-CRC Across Δ Values

Δ	ε_cal	Violations	Worst Risk	Coverage
0.00	0.50	3/15	0.519	30.7%
0.01	0.49	3/15	0.519	27.7%
0.02	0.48	1/15	0.516	25.7%
0.03	0.47	1/15	0.505	25.3%
0.04	0.46	0/15	0.500	23.9%
0.05	0.45	0/15	0.471	16.2%
0.06	0.44	0/15	0.460	13.4%
0.07	0.43	0/15	0.423	11.9%
0.08	0.42	0/15	0.427	10.9%

Backbone Comparison — Precondition for FDV Deployability

WG-CRC effectiveness tracks ρ(D, risk). CheXagent (ρ=0.606) achieves 0/15 violations. LLaVA-Rad and MAIRA-2 (ρ≈0.49) also achieve 0/15 violations per minimum single-split result, but show higher multi-seed variability. Practical deployability criterion: estimate ρ(D, risk) on held-out calibration data before deployment.

Table 7 — Backbone Deployability Analysis (ε=0.50, δ=0.05, G=15)

Backbone	Method	Coverage ↑	Violations	Worst Risk ↓	Multi-seed
CheXagent	Vanilla CRC	33.6%	3/15	0.528	0.551±0.013
CheXagent	WG-CRC + FDV	23.9%	0/15	0.500	0.506±0.022
LLaVA-Rad	Vanilla CRC	33.6%	2/15	0.534	0.568±0.037
LLaVA-Rad	WG-CRC + FDV	20.5%	0/15	0.515	0.510±0.027
MAIRA-2	Vanilla CRC	27.2%	3/15	0.549	0.571±0.022
MAIRA-2	WG-CRC + FDV	21.5%	0/15	0.517	0.550±0.140

Backbone	ρ(D, r)	ρ(FDV, r)	ρ(H, r)	Δ*	Min. viol.
CheXagent	0.606	0.492	0.197	0.04	0/15
LLaVA-Rad	0.497	0.413	0.197	0.04	0/15
MAIRA-2	0.489	0.397	0.157	0.05	0/15

TeleRadShift: Worst-Group Conformal Risk Control for Safe VLM Triage Under Telemedicine Acquisition Shift