SDR Citation Evaluation Diagnostic

Worker analysis, 2026-03-11

Summary

Benchmark_2 results: citation accuracy = 0% (0/1 evaluated), 2 of 20 questions missing.

Root Cause Analysis

1. Citation 0% — Quality Issue, NOT Pipeline Bug

SDR generated a report with completely wrong references:
- [1] Scaled sparse linear regression (doi.org/10.1093/biomet/ass043)
- [2] Regularized M-estimators (openalex.org/W2616050959)
- [3] Fast global convergence (doi.org/10.1214/12-aos1032)

Required citation: Beck & Teboulle (2009), “A fast iterative shrinkage-thresholding algorithm…” (SIAM J. Imaging Sci.)

→ SDR chose wrong papers during research. The evaluation correctly marked citation as incorrect.

2. Missing Questions Q4 and Q15

Benchmark has 20 questions; only 18 evaluated. Q4 (boolean) and Q15 (citation) are absent from results.

Probable cause: Parallel evaluation (parallel_questions=5) hit rate limit errors on batches containing Q4 (batch 1: Q1-5) and Q15 (batch 3: Q11-15). If run with --skip-failed-questions CLI flag, failed questions are silently dropped with just a log warning.

Evidence: skip_failed_questions: bool = False in config.py default, BUT CLI override is possible. The fact that 18/20 questions ARE present (not a hard abort) suggests soft failure mode.

Impact: True citation count_by_type shows 1 citation evaluated instead of 2 (Q15 also maps to Beck & Teboulle 2009 — near-duplicate of Q19).

3. Benchmark Design Issue

Q15: “Работа, в которой FISTA был впервые предложен для задач со спаршенностью”
Q19: “Укажите оригинальную работу, в которой был предложен алгоритм ISTA/FISTA”

Both have identical ground truth (Beck & Teboulle 2009). Near-duplicate questions in benchmark_2 — should be consolidated.

4. Judge Hallucination (llama-3.1-8b-instant)

For Q19 citation, judge output was incoherent:

{
  "evaluated_answer": "true",         // should be citation text, not "true"
  "answer_quote": "Мы рассмотрим...", // unrelated quote
  "explanation": "Авторы совпадают (Beck и Teboulle), год совпадает (2009)...", // hallucinated
  "verdict": "false",                  // contradicts explanation
  "citation_accuracy": 0.35,           // inconsistent with confidence: 0.98
  "confidence": 0.98
}

The 8B model fails at structured citation evaluation:
- Returns “true” as evaluated_answer instead of the citation text
- Hallucinated confirmation of authors/year that aren’t in the report
- Explanation contradicts verdict

Recommendations (Priority Order)

P1: Fix Judge for Citations

• Option A: Use a better judge model for citation type questions
  (llama-70b-versatile or claude-haiku instead of 8b-instant)
• Option B: Add citation-specific evaluation logic (regex-based + LLM for ambiguous cases)

P2: Fix Missing Questions Bug

• Add explicit logging when questions are skipped in parallel batch
• Check exit code / question count after evaluation; fail loudly if questions < expected
• Consider: assert len(evaluated_answers) == len(task.questions), f"Missing {len(task.questions) - len(evaluated_answers)} questions"

P3: Fix Benchmark Design

• Deduplicate Q15 and Q19 in benchmark_2.yaml
• Add unique questions covering: LassoPath, Elastic Net, coordinate descent

P4: Fix SDR Citation Quality

• Root cause: SDR’s search finds wrong papers when topic is well-known algorithms
• Fix: Add OpenAlex/Semantic Scholar exact paper lookup by title keywords
• Or: Add “cite original paper” instruction to deep_research_task

Current Benchmarks Status

Note: benchmark_2 citation 0% inflates weakness — SDR correctly handles proximal methods (boolean+mcq at 80-100%) but fails at bibliographic accuracy. This is fixable with better paper search integration.

Next Steps

1. --skip-failed-questions=False (verify default is respected when running benchmark)
2. Merge eval branch → main (SDR clarifier fix cabcc5b + all eval improvements)
3. Upgrade judge model for citation questions (need API key)
4. Fix benchmark_2: deduplicate Q15+Q19, add 3 new unique citation questions