Skip to main content
A Knowledge Base (KB) is a named container of Documents that have been chunked, embedded, and indexed for semantic retrieval. Agents call search_knowledge_base(query, kb_id?) to surface the most relevant chunks for whatever they’re working on.
KBs ship with three concrete tables (KnowledgeBase, Document, DocumentChunk) plus a KnowledgeEdge layer that materializes the kNN graph for traversal queries (Apache AGE-compatible).

​
What a KB stores

┌──────────────────────────────────────────────────────────────────┐
│ KnowledgeBase                                                    │
│   - kb_id, name, description                                     │
│   - scope: user | account | public                               │
│   - source_type: upload | gdrive | creative_ingest               │
│   - client_id, agency_id, owner                                  │
└──────────────────────────────────────────────────────────────────┘
                           │ has many
                           ▼
┌──────────────────────────────────────────────────────────────────┐
│ Document                                                         │
│   - document_id, title, source_uri, mime_type, sha256, file_size │
│   - status: pending | chunking | embedding | ready | failed      │
└──────────────────────────────────────────────────────────────────┘
                           │ has many
                           ▼
┌──────────────────────────────────────────────────────────────────┐
│ DocumentChunk                                                    │
│   - chunk_id, document, idx, text                                │
│   - embedding: vector(3072) — text-embedding-3-large             │
│   - HNSW pgvector index                                          │
└──────────────────────────────────────────────────────────────────┘
                           │ kNN edges materialized to
                           ▼
┌──────────────────────────────────────────────────────────────────┐
│ KnowledgeEdge                                                    │
│   - from_chunk → to_chunk, relation, weight (cosine distance)    │
│   - Used for graph traversal (recursive CTE, or AGE Cypher)      │
└──────────────────────────────────────────────────────────────────┘

​
Three KB scopes

ScopeVisibilityTypical use
userOnly the owner.Personal scratchpad — research notes, docs only the user cares about.
accountAll users in the same client_id.Team / client-shared knowledge. The default for most KBs.
publicAll authenticated users.System-wide reference (taxonomy guides, attribution methodology, etc.).
The “Creative Insights” KB is a special account-scoped KB auto-built per tenant by the creative-analysis pipeline (one Document per analyzed creative).

​
Ingestion paths

POST /api/knowledge_base/<kb_id>/documents/ (multipart). Accepts PDF, DOCX, TXT, MD, HTML, JSON. Triggers ingest_document.delay(...):
  1. Fetch + parse → plain text.
  2. Chunk (default 800 tokens, 80-token overlap).
  3. Embed each chunk via text-embedding-3-large (3072-dim).
  4. Update Document.status from chunking → embedding → ready.
Re-uploading a file with the same SHA256 is a no-op (idempotent).
OAuth-based ingestion of an entire Drive folder.
  1. User connects their Drive via the OAuth flow at /api/knowledge_base/google-drive/oauth/start/.
  2. Picks a folder. We watch it for changes via push notifications.
  3. New / updated files are downloaded, parsed, and ingested as Documents.
  4. Removed files mark their Documents as archived (kept for audit, excluded from retrieval).
The OAuth flow + watcher logic lives in knowledge_base/google_drive_oauth.py + knowledge_base/google_drive.py. Setup notes in knowledge_base/GOOGLE_DRIVE_SETUP.md.
Auto-built per tenant by the creative-analysis pipeline. Each CreativeAnalysis row materializes a Document in the tenant’s “Creative Insights” KB with the prose synthesis as the body. This means agents can search_knowledge_base("ugc problem-solution") and get hits across the user’s analyzed creatives without going through the bespoke creative-search endpoint.See Creative Analytics overview.
Any automation can write a Document into a KB via the add_to_kb action — useful for archiving Slack threads, capturing meeting notes via webhook, etc.

​
Retrieval

Agents call search_knowledge_base(query, kb_id=None, top_k=5):
  1. Embed the query with text-embedding-3-large.
  2. Cosine kNN over DocumentChunk.embedding, filtered by accessible KBs (scope rules).
  3. Return top-K chunks with { document_title, chunk_text, similarity, document_id, chunk_idx }.
If kb_id is omitted, search runs across all KBs the user can see (user-scope + account-scope for their tenant + public).

​
Graph traversal

KnowledgeEdge materializes the kNN graph. For each chunk we emit edges to its top-N nearest neighbors with cosine distance as edge weight. This enables:
  • Recursive CTE traversal — “find all chunks within 2 hops of X” via Postgres recursive CTE, no AGE required.
  • Apache AGE Cypher — when AGE is installed, the same graph is mirrored into AGE label/edge tables (managed by tasks.build_graph_edges). Agents can run Cypher for richer queries.
The dual-write pattern means KB graph queries work whether or not AGE is enabled — AGE is a performance accelerator, not a correctness dependency.

​
Costs

Per OpenAI pricing (May 2026):
  • Embedding: ~$0.13 per 1M tokens (text-embedding-3-large).
  • Average chunk: ~800 tokens. So ~$0.00010 per chunk.
  • A 100-page PDF (~50k tokens) → ~63 chunks → ~$0.0065 to ingest.
Document.cost_cents (planned) will track per-doc ingest cost so a per-KB rollup is trivial.

​
Endpoints

​
List KBs the caller can see (filtered by scope rules).
​
POST /api/knowledge_base/
Create a new KB. Body: { name, description, scope, source_type }.
​
List documents in a KB. Paginated.
​
POST /api/knowledge_base/<kb_id>/documents/
Upload a document (multipart file field) or post a URL to ingest.
​
POST /api/knowledge_base/search/
Search across accessible KBs. Body: { query, kb_id?, top_k? }.
​
Begin the Google Drive OAuth flow.
​
OAuth callback — completes the connection and lets the user pick a folder.
All gated by shared.permissions.AgentSurfaceAllowed for write paths; reads honor the per-KB scope rules so a public KB is searchable by anyone authenticated.

​
Bob tool

@tool
def search_knowledge_base(
    query: str,
    config: Annotated[RunnableConfig, InjectedToolArg],
    kb_id: str = '',
    top_k: int = 5,
) -> str:
    """Semantic search over the user's accessible knowledge bases."""
Returns a JSON list of chunks with their similarity scores and source document titles. Houston, Space Bob, and Social Sally all have this tool — the same query routes to the same KBs from any agent.

​
Operations

POST /api/knowledge_base/documents/<doc_id>/reingest/ resets the document to pending and re-runs the chunk+embed pipeline. Useful when chunking parameters change.
python manage.py shell -c "from knowledge_base.tasks import build_graph_edges; build_graph_edges('<kb_id>')" recomputes KnowledgeEdge rows for one KB. Run after a large ingest to reflect the new chunks in graph queries.
python manage.py enable_pgvector runs CREATE EXTENSION IF NOT EXISTS vector;. The migration also includes a defensive RunSQL so this is rarely needed manually — but useful on managed Postgres where the migration’s RunSQL is rolled back.
python manage.py check surfaces:
  • agent_surface.W001 — OPENAI_API_KEY not set (KB ingest stores chunks without embeddings).
  • agent_surface.W002 — pgvector extension not enabled (search returns 503).

​
Where the code lives

ConcernPath
Modelsknowledge_base/models.py (KnowledgeBase, Document, DocumentChunk, KnowledgeEdge)
Ingest pipelineknowledge_base/tasks.py (ingest_document, chunk_document, embed_chunks, build_graph_edges)
Google Driveknowledge_base/google_drive.py, knowledge_base/google_drive_oauth.py
API viewsknowledge_base/views.py
Search toolai_chat/kb_tools.py:search_knowledge_base
pgvector status checkknowledge_base/checks.py, knowledge_base/pgvector_status.py
Frontend KB managementl5ui/src/app/components/knowledge-base/