rag-ops-console

Online RAG Ops console with Chroma VectorDB and LangGraph 5 nodes — monitor retrieval quality with RAGAS and Sheets feedback loop in production.

Overview

Online RAG operations console — Chroma VectorDB retrieval, LangGraph orchestration (retrieve → grade → rewrite → generate → human review), Deterministic RAG metrics + optional LLM-backed RAGAS evaluation, and customer feedback → Google Sheets monitoring in production.

  • Python 3.11, Chroma persistent, LangGraph conditional routing
  • Deterministic metrics by default; ragas.evaluate() when credentials present
  • Feedback classification and Top-5 curated questions via gspread

README — Failure Analysis — Limits — Reproducibility

Install

git clone https://github.com/ianlyoo/rag-ops-console.git
cd rag-ops-console
python -m venv .venv
# Windows: .venv\Scripts\activate
pip install -r requirements.txt
pytest -q
ruff check .
python -c "from rag.vector_store import ingest; print(ingest(reset=False))"
python -m graph.run --query "환불 정책" --log out/ops_rag.log
python -m eval.ragas_eval --output out/ragas_result.json

Evaluation — Deterministic RAG metrics

Committed Deterministic metrics (not LLM-backed RAGAS): 0.42 / 0.627 / 0.9953 (context precision / recall / faithfulness) from out/ragas_result.json (50 samples, token overlap, no answer_relevancy).

Limitations adjacent to benchmark: Token-overlap proxy (no LLM evaluator); 50-sample offline run; optional LLM-backed RAGAS via ragas.evaluate() requires credentials and is not committed here; retrieval faithfulness alone can be 1.0 on wrong document — pipeline tracks relevance separately (see docs/evaluation_failure_analysis.md).

Pipeline — VectorDB → LangGraph → RAGAS → Sheets

  • VectorDB rag/vector_store.py — Chroma persistent embedding and rerank
  • LangGraph graph/nodes.py — retrieve → grade → rewrite → generate → human review gate
  • Evaluation eval/ragas_eval.py — deterministic metrics + optional ragas.evaluate()
  • Feedback monitor/feedback_classifier.py — positive/negative/bug/improvement + root-cause
  • Ops monitor/feedback_to_sheet.py — Sheets load + Top-5 question curation

License and ownership

Owner: ianlyoo — License: MIT — Version: 0.1.0 — Language: Python