Strategy/projects/files/papers_db/PAPERS_README.md
+

PAPERS_README

Scientific Papers Storage System — Project Index

Overview

Build a production-ready semantic search system for 100K–1M academic papers. 4-week prototype targeting <500ms search latency on 100K papers, costs ~$200/mo.

Documents (Read in Order)

1. Executive Summary (5 min read)

📄 papers_storage_exec_summary.md
- One-page overview for managers
- Cost breakdown (100K vs 1M papers)
- 4-week timeline
- Key decisions needed

2. Technical Plan (15 min read)

📄 scientific_papers_storage_plan.md
- Full architecture (with ASCII diagrams)
- Cost estimates vs AWS/Yandex/MinIO
- Metadata schema (fields, indices)
- Tech stack recommendations
- 4-week detailed timeline
- Risk mitigation
- Success criteria

3. Tech Stack & Architecture (20 min read)

📄 papers_tech_stack.md
- Detailed layer-by-layer breakdown
- Database schema (PostgreSQL with JSONB + FTS)
- Vector DB config (Qdrant with HNSW)
- Ingestion pipeline (6 steps: PDF → embedding → store)
- Search logic (hybrid FTS + semantic)
- Technology decision matrix

4. Implementation Checklist (10 min read)

📄 papers_implementation_checklist.md
- Week-by-week tasks
- File structure to create
- Key decision points
- Success metrics
- Quick cost reference

Quick Reference

Stack at a Glance

┌─────────────────────────────────────────┐
│ FastAPI (Python)                        │
├─────────────────────────────────────────┤
│ Hybrid Search (FTS + Semantic)          │
├─────────────────────────────────────────┤
│ PostgreSQL 15 + Qdrant (Vector DB)      │
├─────────────────────────────────────────┤
│ AWS S3 + PyMuPDF Ingestion              │
└─────────────────────────────────────────┘

Data Flow

PDF Upload → PyMuPDF Extract → LLM Enrich → Embed → Store (S3 + PG + Qdrant) → Search

Costs (6 months, 100K papers)

  • AWS: $47/mo storage → **$1,150 total**
  • MinIO: $120/mo self-hosted → **$720 total** (plus hardware)

Timeline

  • Week 1: Infrastructure + schema
  • Week 2: Ingestion pipeline
  • Week 3: Search API
  • Week 4: Testing + deployment

Key Decisions (For Manager Approval)

  1. Paper Source? (arXiv, CrossRef, institutional repo)
    - Affects: initial data load, metadata enrichment strategy

  2. Metadata Enrichment Budget?
    - LLM extraction (+$0.002/paper) vs heuristics-only
    - Affects: search quality, metadata accuracy

  3. OCR for Scanned PDFs?
    - Yes: +$0.02/page, +15% coverage
    - Affects: total cost, ingestion time

  4. Team Assignment?
    - Need: 1 backend engineer (Python) + 1 DevOps
    - Affects: timeline, handoff readiness

Recommendation

Start AWS S3 + PostgreSQL + Qdrant (proof-of-concept)
- Fast setup (5 min)
- Pay-as-you-go pricing
- Migrate to MinIO at 20TB+ if access patterns justify

Success Metrics:
- Ingest 100K papers in <4 hours ✓
- Search latency <500ms (p99) ✓
- Monthly cost <$200 ✓
- Deployment via Docker Compose ✓

File Locations

Strategy/projects/
├── PAPERS_README.md                         (this file)
├── papers_storage_exec_summary.md          (1-pager for Макс)
├── scientific_papers_storage_plan.md       (full technical plan)
├── papers_tech_stack.md                    (detailed architecture)
└── papers_implementation_checklist.md      (week-by-week tasks)

Code Repo (To Create):
papers-storage/
├── docker-compose.yml
├── requirements.txt
├── ingest/
│   ├── extractor.py          (PyMuPDF + metadata)
│   ├── embedder.py           (sentence-transformers)
│   └── schema.py
├── api/
│   ├── main.py               (FastAPI app)
│   ├── routes.py             (endpoints)
│   └── models.py             (ORM models)
├── db/
│   └── init.sql              (PostgreSQL schema)
└── scripts/
    ├── ingest_batch.py       (batch import)
    └── health_check.py       (monitoring)

Contact & Next Steps

  1. Review: papers_storage_exec_summary.md (5 min)
  2. Discuss: Key decisions above with team
  3. Approve: Budget ($1,500 for 6-month AWS pilot)
  4. Assign: 1 backend eng + 1 DevOps
  5. Start: Week 1 tasks (infrastructure provisioning)

References

Last Updated: 2026-02-25
Version: 1.0 (MVP Planning)

Choose icon