DeepResearchGym — Feasibility Study for SDR Evaluation

Date: 2026-02-27
Status: Research complete, integration feasible

Key Repos

• Evaluation Protocol: https://github.com/cxcscmu/deepresearch_benchmarking
• Search API: https://github.com/cxcscmu/large-scale-embeddings
• Dataset: https://huggingface.co/datasets/cx-cmu/deepresearchgym-agentic-search-logs

Quick Facts

• NOT pip-installable — clone repos, run scripts
• Free public APIs: FineWeb + ClueWeb22 (HTTP endpoints, no API key needed)
• Evaluation needs OpenAI API key (LLM-as-a-judge), ~$0.01-0.10/report
• 1,000 Researchy Questions benchmark (complex research queries)
• Metrics: Citation Precision/Recall, Key Point Recall (KPR), Info Alignment, Report Quality

Integration Path for SDR

1. Generate reports on Researchy Questions queries using SDR (Tavily+OpenAlex search as-is)
2. Feed reports into deepresearch_benchmarking eval scripts
3. LLM judge scores automatically → baseline metrics
4. Optional: add DeepResearchGym search APIs as additional search backend

Recommended Approach

1. ✅ Clone cxcscmu/deepresearch_benchmarking → /root/Strategy/research/deepresearch_benchmarking/
2. Run SDR on 10-50 sample queries from Researchy Questions
3. Evaluate with their scripts (needs OpenAI key)
4. Compare vs published baselines

Setup Status (2026-02-27 07:00)

• Repo cloned ✅ → /root/Strategy/research/deepresearch_benchmarking/
• Queries available: 1000 (full), 100 (sample), 10 (small test) — all in JSONL format
• Query format: {"id": "123", "query": "..."}
• Expected output format: {id}.a (answer), {id}.q (query) in /data/group_data/.../reports/{system}/

Important Observations

• Domain mismatch: Researchy Questions are general web queries (COVID safety, economics, politics)
  SDR is optimized for scientific paper research → may score lower than GPT Researcher on this benchmark
• Alternative: Run on 5 custom scientific queries for SDR-relevant demo
• Eval cost: ~$0.01-0.05/report with gpt-4.1-mini → ~$0.50-$5 for 100 reports
• Needs OpenAI key: eval scripts use openai library — must add to keys.env

Next Steps to Complete

1. Get OpenAI API key → add to keys.env in deepresearch_benchmarking/
2. Create local reports dir: mkdir -p /root/Strategy/research/sdr_reports/SDR/
3. Modify eval scripts: change /data/group_data/... to /root/Strategy/research/sdr_reports/
4. Run SDR on 10 sample queries → save as {id}.a + {id}.q files
5. Run python eval_quality_async.py --subfolder SDR --open_ai_model gpt-4.1-mini

Script: Run SDR on Sample Queries

See: /root/Strategy/research/sdr_benchmark_runner.py (to be created)

Alternative Benchmarks

• DeepResearch Bench (100 PhD tasks, simpler setup): https://github.com/Ayanami0730/deep_research_bench
• DEER (hierarchical, expert rubrics)
• LiveDRBench (scientific + public events, 100 tasks)

Next Step

Clone deepresearch_benchmarking, review eval scripts, run SDR on 10 test queries → get baseline score. Effort: ~2-3h.