# Learning Model — What It Is & Where Data Lives

## 1. What is the “learning model”?

In this project it is a **RAG learning model**, not a fine-tuned neural net you train for days.

```
Source text (brand, posts, context)
        ↓
   Chunk into documents
        ↓
   Embed → vector (list of numbers)
        ↓
   Save vectors  ← "learning data"
        ↓
   Query → find similar vectors → send chunks to GPT-4o mini / Claude
```

Code:

| File | Role |
|------|------|
| `model/types.mjs` | Chunk / hit shapes |
| `model/embedder.mjs` | Text → vector (local hash embed, or OpenAI if key set) |
| `model/store.mjs` | Load/save `data/vectors/chunks.json` |
| `model/ingest.mjs` | Read PostSync data → sources → vectors |
| `model/retrieve.mjs` | Cosine similarity top-k search |
| `scripts/build-vectors.mjs` | CLI: build learning data |
| `scripts/query-vectors.mjs` | CLI: test learning retrieval |

---

## 2. Where do we save learning data?

### A. File store (active now) — **recommended for this workspace**

| Path | Contents |
|------|----------|
| `learning/data/sources/documents.json` | Normalized text docs before embedding |
| `learning/data/vectors/chunks.json` | **Vector learning data** (id, text, metadata, `embedding[]`) |
| `learning/data/vectors/manifest.json` | Counts, model name, built-at timestamp |

You can open `chunks.json` in the editor and **see** the vectors.

### B. Database (optional later)

SQL: `learning/sql/009_rag_chunks.sql` → table `rag_chunks`.

Use DB when:

- Multiple Node/PM2 instances must share one index  
- You want SQL filters with vectors  
- File JSON gets too large  

**You do not need DB to start.** Build files first; migrate embeddings into MariaDB when ready.

---

## 3. Do I need to save vector data in the database?

| Choice | When |
|--------|------|
| **Files only** | Local/dev, single server, learning workspace (current) |
| **DB** | Production multi-instance, large corpus |
| **Both** | Build files → sync job → `rag_chunks` |

Answer: **No, not required for v1.** Yes for production scale. This workspace creates **file vectors now**.

---

## 4. Embedding modes

| Mode | Env | Behavior |
|------|-----|----------|
| `local` (default) | none | Deterministic 384-dim hash embedding — free, offline, good for learning/demo |
| `openai` | `OPENAI_API_KEY` + `RAG_EMBEDDING_MODE=openai` | `text-embedding-3-small` — better quality |

Rebuild after switching mode (vectors are not compatible across modes).

---

## 5. How this ties to ChatGPT / Claude

1. `retrieve(question)` → top chunks from `chunks.json`  
2. Build grounded prompt with those chunks  
3. Call **GPT-4o mini** or **Claude** (wired later in `src/lib/rag/query.ts` / Next API)

Retrieval works **today** via CLI without calling an LLM.

---

## 6. Rebuild commands

```bash
# From repo root
node learning/scripts/build-vectors.mjs

# Test retrieval
node learning/scripts/query-vectors.mjs "What is the B2BEFAX brand voice?"
node learning/scripts/query-vectors.mjs "Q3 product launch"
```

---

## 7. Security

- Never embed OAuth tokens or `.env` secrets  
- Sources come from captions, brand text, context snapshots only