Skip to main content
The MetaCogna RAG worker is a Cloudflare Worker that handles all backend functionality including authentication, document ingestion, vector search, and knowledge graph extraction.

Architecture

Environment Bindings

Request Routing

The worker uses a simple path-based router:

Core Functions

Semantic Chunking

Text is split into semantic chunks for embedding:
Parameters:
  • maxSize: Maximum chunk size in characters (default: 512)
  • overlap: Overlap size in characters (default: 50)

Rate Limiting

KV-based rate limiting per user and endpoint:
Rate Limits:
  • Chat: 10 requests per minute
  • Search: 20 requests per minute

Ingestion Pipeline

The ingestion process runs through 4 stages:

1. Chunking (0-29%)

  • Text split into semantic chunks (~500 chars)
  • Sentence boundary preservation
  • Overlap between chunks for context

2. Embedding (30-59%)

  • Generate 768-dimensional embeddings using Workers AI
  • Model: @cf/baai/bge-base-en-v1.5
  • Store in Vectorize index

3. Graph Extraction (60-89%)

  • Extract entities and relationships using LLM
  • Model: @cf/meta/llama-3-8b-instruct
  • Store nodes and edges in D1 database

4. Finalizing (90-100%)

  • Update document status
  • Store metadata
  • Mark ingestion complete

Database Schema

Tables

users
  • User authentication and profile data
  • Fields: id, username, email, passwordHash, goals, isAdmin
documents
  • Document metadata
  • Fields: id, userId, title, content (preview), r2Key, metadata, status
graph_nodes
  • Knowledge graph entities
  • Fields: id, label, type, summary, documentId
graph_edges
  • Knowledge graph relationships
  • Fields: id, source, target, relation, documentId
See Database Schema for complete schema.

Storage Strategy

R2 Storage

Full document content is stored in R2:
  • Path format: users/{userId}/documents/{docId}-{filename}
  • Metadata: Stored as R2 object metadata
  • Content types: PDF, Markdown, Text

D1 Database

Metadata and previews stored in D1:
  • Preview: First 500 characters of document
  • Metadata: JSON string with custom metadata
  • Status: Processing state tracking

Vectorize

Vector embeddings stored in Vectorize:
  • Dimensions: 768
  • Metric: Cosine similarity
  • Metadata: Document ID, title, chunk index

Error Handling

All endpoints use consistent error responses:

CORS

CORS headers for all responses:

Configuration

wrangler.toml

Deployment

Or use the deployment script: