# RAG BKD — Knowledge Base Kepegawaian

## Overview
Vector search (RAG) dari 19 dokumen sosialisasi BKD Kalimantan Selatan.
Dibangun Juli 2026. Vector DB: **ChromaDB** + **sentence-transformers** (`all-MiniLM-L6-v2`).

## File & Path
| Component | Path |
|-----------|------|
| Source documents | `/home/ubuntu/kantor/BKD_Sosialisasi/Materi Sosialisasi 2024/` |
| ChromaDB storage | `/home/ubuntu/.hermes/rag_bkd_chroma/` |
| Build script | `/home/ubuntu/build_rag_bkd.py` |
| Collection name | `bkd_sosialisasi` |
| Total chunks | 386 (chunk size: 800 chars, overlap 100) |

## Documents Indexed (19 files)
| File | Topic |
|------|-------|
| BAHAN BKD 19NOV24 NEW.pptx | Bahan BKD terbaru |
| CUTI ASN.pptx | Kebijakan cuti PNS |
| DISDIK APRIL 2024.pptx | Disiplin PNS |
| MATERI KORPRI.pptx | Korpri |
| MATERI KP SOSIALISASI KEPEGAWAIAN DESEMBER 2024.pptx | Kenaikan pangkat |
| MATERI PERKAWINAN PERCERAIAN.pptx | Kebijakan pernikahan/perceraian |
| MATERI IP ASN 2023.pdf | Indeks kinerja ASN |
| Materi Tapera sosialisasi kepegawaian.pptx | Tapera |
| NETRALITAS.pptx | Netralitas PNS |
| Penyusunan-Rencana-Mutasi-Pegawai-Negeri-Sipil-Tahun-2025.pptx | Mutasi PNS |
| Pergub-No-057-Tahun-2021-tentang-Mutasi-PNS.pptx | Pergub mutasi |
| PG Th 2024 047.pdf | Peraturan pensiun |
| PLT DAN PLH.pptx | PLT vs PLH |
| SOSIALISASI JF - FINAL.pptx | Jabatan fungsional |
| SOSIALISASI KARIS KARSU.pptx | Karis Karsu |
| Sosialisasi MPP Prov. Kals 14 Nov 2024.pptx | MPP |
| Sosialisasi Pensiun PNS Prov. Kals 14 Nov 2024.pptx | Pensiun |
| TAMBAHAN PENGHASILAN PEGAWAI (TPP).pptx | TPP |
| UJI KOM JABFUNG PANGKAT.pptx | Uji kompetensi jabfung |

## Query Script
```python
import chromadb
from sentence_transformers import SentenceTransformer

chroma_client = chromadb.PersistentClient(path="/home/ubuntu/.hermes/rag_bkd_chroma")
collection = chroma_client.get_collection("bkd_sosialisasi")
model = SentenceTransformer("all-MiniLM-L6-v2")

query = "apa syarat Karis Karsu?"
results = collection.query(
    query_embeddings=model.encode([query]).tolist(),
    n_results=3
)

for doc, meta in zip(results['documents'][0], results['metadatas'][0]):
    print(f"📄 {meta['filename']}")
    print(f"   {doc[:300]}...")
```

## Query Interface
Trigger: ketik `bkd ` (tanpa slash) di awal pesan Telegram → auto-detect sebagai query RAG BKD.

**Cara pakai di Telegram:**
```
bkd apa syarat Karis Karsu?
bkd bedanya PLT dan PLH apa?
bkd syarat pensiun PNS
```

**Script query:**
```bash
/home/ubuntu/rag_query.sh bkd "<pertanyaan>"
```

Python venv: `/home/ubuntu/venv_trading/bin/python3`

## Multi-RAG Routing Plan (for future)
```
bkd <pertanyaan>     → RAG BKD (kepegawaian) ✅ ACTIVE
saham <pertanyaan>  → RAG Saham (equity) — TODO
ternak <pertanyaan> → RAG Peternakan — TODO
<tanpa trigger>     → jawaban bebas (no RAG)
```

Routing berdasarkan kata PERTAMA pesan (tanpa slash).
Master router script: `/home/ubuntu/rag_query.sh`

## Rebuild RAG
Jika ada dokumen BKD baru:
1. Copy file ke `/home/ubuntu/kantor/BKD_Sosialisasi/Materi Sosialisasi 2024/`
2. Jalankan ulang:
   ```bash
   /home/ubuntu/venv_trading/bin/python3 /home/ubuntu/build_rag_bkd.py
   ```
   (script auto-detect semua .pptx/.pdf di folder itu)

## Setup Environment
```bash
python3 -m venv /home/ubuntu/venv_trading
/home/ubuntu/venv_trading/bin/pip install chromadb sentence-transformers pymupdf markitdown torch
/home/ubuntu/venv_trading/bin/python3 /home/ubuntu/build_rag_bkd.py
```

## Tested Queries (July 2026)
| Query | Top Result |
|-------|-----------|
| "kapan PNS bisa ajukan Karis Karsu?" | SOSIALISASI KARIS KARSU.pptx ✅ |
| "syarat pensiun PNS" | Sosialisasi Pensiun PNS Prov. Kals ✅ |
| "apa itu TPP Additional?" | TAMBAHAN PENGHASILAN PEGAWAI (TPP).pptx ✅ |
| "PLT dan PLH apa bedanya?" | PLT DAN PLH.pptx ✅ |
| "kenaikan pangkat luar biasa" | MATERI KP SOSIALISASI KEPEGAWAIAN DESEMBER 2024.pptx ✅ (chunk IDs 650–652) |
| "pelayanan pangkat" | MATERI KP SOSIALISASI KEPEGAWAIAN DESEMBER 2024.pptx ✅ (chunk IDs 564–565: daftar 7 jenis KP) |

## ChromaDB Schema (confirmed July 2026)
- Table: `embedding_metadata` — primary key column is `id` (INTEGER), **NOT** `embedding_id`
- Query pattern: `WHERE id = ? AND key = 'chroma:document'` then join with `key = 'filename'` using the same `id`
- Metadata keys: `chroma:document` (the text chunk), `filename`
- Collection: `bkd_sosialisasi`

## Pitfalls
1. **HF_TOKEN**: Model download tanpa token → warning "unauthenticated requests". Ini OK, tidak block.
2. **PDF extract with pymupdf**: `fitz.open()` → `page.get_text("text")` dengan flag `"text"` untuk proper extraction.
3. **Chunk boundaries**: break di `. ` (period + space) lebih baik dari char limit saja — menghasilkan chunk yang lebih readable.
4. **PPTX mostly images, not extractable text**: MATERI KP SOSIALISASI KEPEGAWAIAN DESEMBER 2024.pptx and similar decks store content as embedded images/shapes, not readable text. For dense policy content (e.g. KPLB requirements), check multiple adjacent chunk IDs — the text that does exist is distributed across slide titles and small text frames.
5. **Chunk count may be stale**: skill header previously noted 386 chunks; direct DB inspection July 2026 found IDs ranging well beyond that (up to ~679). Run `/home/ubuntu/rag_query.sh bkd "kenaikan pangkat"` to confirm current state.