papers_storage_exec_summary
Scientific Papers Storage System — Executive Summary
Goal: Build a searchable repository for 100K–1M academic papers with semantic search. Prototype in 4 weeks.
The Approach (One Slide)
AWS S3 (PDFs) ← PyMuPDF (extract) → PostgreSQL (metadata) + Qdrant (vectors) → Hybrid Search API
↑
Claude/LLM (metadata)
Data: Title, authors, DOI, abstract, FWCI (citation impact), keywords
Search: Full-text (PostgreSQL) + semantic (Qdrant vectors) combined
Scale: 100K papers = 300GB; 1M papers = 3TB
Why This Stack?
| Component | Why |
|---|---|
| AWS S3 | Standard (proof-of-concept); $0.023/GB/mo. Switch to MinIO at 20TB+ if frequent access. |
| PostgreSQL | Metadata + FTS (full-text search). Proven, no vendor lock. |
| Qdrant | 41 QPS at 99% recall @ 50M vectors. Open-source, easy self-hosting. |
| PyMuPDF | 2–3× faster than PDFMiner for PDF text extraction. |
| FastAPI | Python async, auto-docs, fast deployment. |
Costs (Monthly)
100K Papers (Startup Phase)
- S3 storage + requests: $47
- PostgreSQL (t3.medium): $40
- Qdrant (t3.large): $60
- Egress/networking: $45
- **Total: $192/mo** (~$1,150 for 6-month pilot)
1M Papers (Growth Phase)
- S3 Intelligent-Tiering: $75
- PostgreSQL (r6i.xlarge): $200
- Qdrant cluster (3×t3.xlarge): $450
- Egress: $75
- Total: $800/mo
- Breakeven with MinIO: ~7 months (then $120/mo on self-hosted)
Timeline (4 Weeks)
| Week | Deliverable |
|---|---|
| 1 | Infrastructure (AWS S3 + RDS + Qdrant); schema design |
| 2 | Ingestion pipeline (PDF → metadata); async queue |
| 3 | Search API (FTS + semantic + hybrid); caching |
| 4 | Testing; Docker deployment; cost dashboard |
MVP Scope: 10K papers → sub-500ms search → deployable
Quick Decisions Needed
- Paper Source? (arXiv dump, CrossRef, institutional)
- Metadata Enrichment Budget? (use paid APIs or heuristics-only)
- OCR for Scanned PDFs? (adds ~$0.02/page but 15% more coverage)
- Team Assignment? (1 backend engineer + 1 DevOps)
Risk Mitigation
| Risk | Solution |
|---|---|
| PDF parsing fails | Add Tesseract OCR; manual review queue |
| S3 egress explodes | Implement CloudFront CDN; use Glacier for cold |
| Qdrant crashes @ scale | Shard by discipline; pgvector backup |
| Metadata inaccurate | CrossRef API + human review; >90% target |
Success Criteria
- ✅ Ingest 100K papers in <4 hours
- ✅ Search latency <500ms (p99)
- ✅ Monthly cost <$200
- ✅ Deployment via Docker Compose (reproducible)
- ✅ Semantic search recall >85% (validation set)
Recommendation
Start AWS (proof-of-concept), plan MinIO migration at 20TB+
AWS pros: 5-min setup, pay-as-you-go, no ops burden
MinIO pros: No egress charges, control, cost breakeven at 7 months
Next Steps
- Approve budget: $1,500 (6-month AWS pilot)
- Assign team: 1 backend eng + 1 DevOps
- Schedule data source confirmation (arXiv/CrossRef/internal)
- Weekly cost tracking + performance dashboard
Questions? See full technical plan: scientific_papers_storage_plan.md