# 01 — RAG Data Sources (Current Data)

What exists **today** that can be ingested into a RAG index. Platform KB vectors are built under `data/knowledge/`; project-scoped ingest (generated posts, website cache) remains optional future work.

---

## 1. Principle (current product)

**User-driven setup (no predefined sample project):**

1. **My Project** — user creates project name, website, industry, timezone, target countries  
2. **Brand Profile** — company name, tagline, logo, contacts (from their business)  
3. **Posting Context** — sequence beats built from brand + website scrape (Refresh from brand)  
4. **Generate** — caption, image, video use project + brand + platform RAG only  

Generation uses **project-specific context** (website scrape + brand + creative brief) **enhanced by platform RAG** when `RAG_ENABLED=true`.

| Layer | Path | Status |
|-------|------|--------|
| Platform KB sources | `knowledge_base/{platform}_post_details/*.md` | Built |
| Vector stores | `data/knowledge/{platform}/chunks.json` | Built (`npm run learning:build-knowledge`) |
| Runtime retrieval | `src/lib/rag/` → `run-generation.ts` | **Live** (caption, image, video prompts) |
| RAG chat API | `POST /api/rag/query` | **Live** |
| Reindex API | `POST /api/rag/reindex` | **Live** |

RAG **enhances** project grounding — it does not replace brand/website facts or relevance validators.

---

## 1b. Legacy note (pre-Aug 2026)

Earlier docs stated there was no retrieval in the live path. That is **superseded** by `src/lib/rag/` wiring documented above.

---

## 2. Domain objects (TypeScript)

Source: `src/types/workflow.ts`, `src/lib/project/defaults.ts`

| Document type | Key fields | RAG value |
|---------------|------------|-----------|
| **Project** | name, website, industry, timezone, countries, status | Tenant scope + market facts |
| **BrandProfile** | companyName, tagline, industry, brandVoice, logo, contacts | Voice & identity chunks |
| **PostingContext / sequence** | name, posts[{title, summary, tag, content}] (≤5) | Primary narrative beats |
| **GeneratedPost** | caption, image/video prompts, contextSnapshot, status, platforms, QA | Historical content library |
| **ConnectedAccount** | platform, label, externalId, status | **Do not embed tokens** — metadata only |
| **Schedule / Automation** | postsPerDay, rules | Weak RAG value (ops, not content) |

---

## 3. Website intelligence

Source: `src/lib/context/website-intelligence.ts`  
Cache: `data/website-cache/` (≈24h TTL)

| Field group | Use in RAG |
|-------------|------------|
| title, description, tagline, companyName | Identity |
| primaryService, services[], benefits[] | Offerings |
| keywords[], headings[] | Retrieval keys |
| analysisMarkdown | Long-form chunk source |
| colors / productVisualHint | Image briefs (optional) |

---

## 4. File / JSON stores

| Path | Contents | Ingest? |
|------|----------|---------|
| `data/users/<userId>/projects.json` | Projects + scoped brand/context/schedule | Yes |
| `data/generated-posts.json` (or DB) | All generated posts | Yes (prefer published/approved) |
| `data/website-cache/*` | Scraped site intel | Yes |
| `data/social-accounts/**` | OAuth tokens | **No** (secrets) |
| `data/provider-post-stats.json` | Success/fail counters | Optional analytics only |
| `data/schedule-fires.json` | Slot claims | No |
| `logs/publish-*.log`, `logs/auto-post-*.log` | Ops logs | Optional ops RAG only |

---

## 5. MariaDB tables (relevant)

From `extras/db/postsync_pro_backup.sql` + migrations `005`–`008`:

| Table | RAG relevance |
|-------|----------------|
| `projects` | Scope + metadata |
| `brand_profiles` | Brand kit (+ JSON audiences/products/keywords in legacy columns) |
| `posting_contexts` | Sequence `posts_json` |
| `generated_posts` | Captions, QA JSON, media refs |
| `website_pages` | Crawlable page URLs/titles (legacy crawl model) |
| `content_items` / `platform_variants` | Legacy content model — migrate or ignore |
| `ai_usage_logs` | Cost analytics, not content |
| `connected_accounts` | Destination metadata only — **never embed tokens** |
| `articles` | If populated, good RAG corpus |
| `media_assets` | Metadata + captions; binary via URL not vectors |

---

## 6. Suggested document ID scheme

```
proj:{projectId}:brand
proj:{projectId}:sequence:{postId}
proj:{projectId}:website:{hash}
proj:{projectId}:generated:{postId}
proj:{projectId}:page:{websitePageId}
```

Always include metadata: `{ userId, projectId, docType, updatedAt, status }`.

---

## 7. Chunking guidance

| Source | Chunk strategy |
|--------|----------------|
| Brand profile | Single chunk (small) |
| Sequence item | One chunk per beat (title+summary+content) |
| Website analysisMarkdown | 400–800 tokens, 10–15% overlap |
| Generated caption | One chunk per post; attach imagePrompt as sibling metadata |
| Headings/services lists | Bullet chunks with keyword tags |

---

## 8. Refresh triggers (ingest events)

| Event | Action |
|-------|--------|
| Brand / context saved | Re-index project brand + sequence docs |
| Website refresh | Re-scrape → replace website chunks |
| Post published / approved | Upsert generated-post chunk |
| Project paused/deleted | Soft-delete or filter out of retrieval |
| Manual media upload (future) | Index caption + asset description |

---

## 9. Out of scope for v1 RAG

- Raw OAuth tokens / refresh tokens  
- Binary image/video blobs (store URLs + alt text / captions only)  
- Cross-project retrieval (no tenant bleed)  
- Public web search (keep grounding internal unless product asks)