Knowledge Bases

A Knowledge Base (KB) is a named container of Documents that have been chunked, embedded, and indexed for semantic retrieval. Agents call search_knowledge_base(query, kb_id?) to surface the most relevant chunks for whatever they’re working on.

KBs ship with three concrete tables (KnowledgeBase, Document, DocumentChunk) plus a KnowledgeEdge layer that materializes the kNN graph for traversal queries (Apache AGE-compatible).

What a KB stores

┌──────────────────────────────────────────────────────────────────┐
│ KnowledgeBase                                                    │
│   - kb_id, name, description                                     │
│   - scope: user | account | public                               │
│   - source_type: upload | gdrive | creative_ingest               │
│   - client_id, agency_id, owner                                  │
└──────────────────────────────────────────────────────────────────┘
                           │ has many
                           ▼
┌──────────────────────────────────────────────────────────────────┐
│ Document                                                         │
│   - document_id, title, source_uri, mime_type, sha256, file_size │
│   - status: pending | chunking | embedding | ready | failed      │
└──────────────────────────────────────────────────────────────────┘
                           │ has many
                           ▼
┌──────────────────────────────────────────────────────────────────┐
│ DocumentChunk                                                    │
│   - chunk_id, document, idx, text                                │
│   - embedding: vector(3072) — text-embedding-3-large             │
│   - HNSW pgvector index                                          │
└──────────────────────────────────────────────────────────────────┘
                           │ kNN edges materialized to
                           ▼
┌──────────────────────────────────────────────────────────────────┐
│ KnowledgeEdge                                                    │
│   - from_chunk → to_chunk, relation, weight (cosine distance)    │
│   - Used for graph traversal (recursive CTE, or AGE Cypher)      │
└──────────────────────────────────────────────────────────────────┘

Three KB scopes

Scope	Visibility	Typical use
`user`	Only the owner.	Personal scratchpad — research notes, docs only the user cares about.
`account`	All users in the same `client_id`.	Team / client-shared knowledge. The default for most KBs.
`public`	All authenticated users.	System-wide reference (taxonomy guides, attribution methodology, etc.).

The “Creative Insights” KB is a special account-scoped KB auto-built per tenant by the creative-analysis pipeline (one Document per analyzed creative).

Ingestion paths

Direct upload

POST /api/knowledge_base/<kb_id>/documents/ (multipart). Accepts PDF, DOCX, TXT, MD, HTML, JSON. Triggers ingest_document.delay(...):

Fetch + parse → plain text.
Chunk (default 800 tokens, 80-token overlap).
Embed each chunk via text-embedding-3-large (3072-dim).
Update Document.status from chunking → embedding → ready.

Re-uploading a file with the same SHA256 is a no-op (idempotent).

Google Drive

OAuth-based ingestion of an entire Drive folder.

User connects their Drive via the OAuth flow at /api/knowledge_base/google-drive/oauth/start/.
Picks a folder. We watch it for changes via push notifications.
New / updated files are downloaded, parsed, and ingested as Documents.
Removed files mark their Documents as archived (kept for audit, excluded from retrieval).

The OAuth flow + watcher logic lives in knowledge_base/google_drive_oauth.py + knowledge_base/google_drive.py. Setup notes in knowledge_base/GOOGLE_DRIVE_SETUP.md.

Creative ingest

Auto-built per tenant by the creative-analysis pipeline. Each CreativeAnalysis row materializes a Document in the tenant’s “Creative Insights” KB with the prose synthesis as the body. This means agents can search_knowledge_base("ugc problem-solution") and get hits across the user’s analyzed creatives without going through the bespoke creative-search endpoint.See Creative Analytics overview.

Automation `add_to_kb` action

Any automation can write a Document into a KB via the add_to_kb action — useful for archiving Slack threads, capturing meeting notes via webhook, etc.

Retrieval

Agents call search_knowledge_base(query, kb_id=None, top_k=5):

Embed the query with text-embedding-3-large.
Cosine kNN over DocumentChunk.embedding, filtered by accessible KBs (scope rules).
Return top-K chunks with { document_title, chunk_text, similarity, document_id, chunk_idx }.

If kb_id is omitted, search runs across all KBs the user can see (user-scope + account-scope for their tenant + public).

Graph traversal

KnowledgeEdge materializes the kNN graph. For each chunk we emit edges to its top-N nearest neighbors with cosine distance as edge weight. This enables:

Recursive CTE traversal — “find all chunks within 2 hops of X” via Postgres recursive CTE, no AGE required.
Apache AGE Cypher — when AGE is installed, the same graph is mirrored into AGE label/edge tables (managed by tasks.build_graph_edges). Agents can run Cypher for richer queries.

The dual-write pattern means KB graph queries work whether or not AGE is enabled — AGE is a performance accelerator, not a correctness dependency.

Costs

Per OpenAI pricing (May 2026):

Embedding: ~$0.13 per 1M tokens (text-embedding-3-large).
Average chunk: ~800 tokens. So ~$0.00010 per chunk.
A 100-page PDF (~50k tokens) → ~63 chunks → ~$0.0065 to ingest.

Document.cost_cents (planned) will track per-doc ingest cost so a per-KB rollup is trivial.

Endpoints

List KBs the caller can see (filtered by scope rules).

Create a new KB. Body: { name, description, scope, source_type }.

List documents in a KB. Paginated.

Upload a document (multipart file field) or post a URL to ingest.

Search across accessible KBs. Body: { query, kb_id?, top_k? }.

Begin the Google Drive OAuth flow.

OAuth callback — completes the connection and lets the user pick a folder.

All gated by shared.permissions.AgentSurfaceAllowed for write paths; reads honor the per-KB scope rules so a public KB is searchable by anyone authenticated.

Bob tool

@tool
def search_knowledge_base(
    query: str,
    config: Annotated[RunnableConfig, InjectedToolArg],
    kb_id: str = '',
    top_k: int = 5,
) -> str:
    """Semantic search over the user's accessible knowledge bases."""

Returns a JSON list of chunks with their similarity scores and source document titles. Houston, Space Bob, and Social Sally all have this tool — the same query routes to the same KBs from any agent.

Operations

Re-ingest a stale document

POST /api/knowledge_base/documents/<doc_id>/reingest/ resets the document to pending and re-runs the chunk+embed pipeline. Useful when chunking parameters change.

Rebuild the kNN graph

python manage.py shell -c "from knowledge_base.tasks import build_graph_edges; build_graph_edges('<kb_id>')" recomputes KnowledgeEdge rows for one KB. Run after a large ingest to reflect the new chunks in graph queries.

Enable pgvector

python manage.py enable_pgvector runs CREATE EXTENSION IF NOT EXISTS vector;. The migration also includes a defensive RunSQL so this is rarely needed manually — but useful on managed Postgres where the migration’s RunSQL is rolled back.

System checks

python manage.py check surfaces:

agent_surface.W001 — OPENAI_API_KEY not set (KB ingest stores chunks without embeddings).
agent_surface.W002 — pgvector extension not enabled (search returns 503).

Where the code lives

Concern	Path
Models	`knowledge_base/models.py` (`KnowledgeBase`, `Document`, `DocumentChunk`, `KnowledgeEdge`)
Ingest pipeline	`knowledge_base/tasks.py` (`ingest_document`, `chunk_document`, `embed_chunks`, `build_graph_edges`)
Google Drive	`knowledge_base/google_drive.py`, `knowledge_base/google_drive_oauth.py`
API views	`knowledge_base/views.py`
Search tool	`ai_chat/kb_tools.py:search_knowledge_base`
pgvector status check	`knowledge_base/checks.py`, `knowledge_base/pgvector_status.py`
Frontend KB management	`l5ui/src/app/components/knowledge-base/`

Getting Started

Onboarding

Platform

Configuration

AI Agents

Agent Surface

Creative Analytics

Development

Lunar Interns

Performance Reports

Attribution & Reporting

Notes / PMT

What a KB stores

Three KB scopes

Ingestion paths

Retrieval

Graph traversal

Costs

Endpoints

Bob tool

Operations

Where the code lives

​What a KB stores

​Three KB scopes

​Ingestion paths

​Retrieval

​Graph traversal

​Costs

​Endpoints

​Bob tool

​Operations

​Where the code lives

What a KB stores

Three KB scopes

Ingestion paths

Retrieval

Graph traversal

Costs

Endpoints

Bob tool

Operations

Where the code lives