AI Architecture
A deep dive into how Masst Docs uses AI — from document ingestion and vector embeddings to real-time RAG-powered chat and deep research.
Hover over the annotated chips below each diagram to see exact numbers, limitations, and tradeoffs.
High-Level System Overview
The AI system in Masst Docs is a Retrieval-Augmented Generation (RAG) pipeline. Instead of relying solely on a language model's training data, every response is grounded in the actual documentation content stored in a vector database.
Document Ingestion Pipeline
The ingestion pipeline runs offline via scripts/generate-embeddings.ts. It processes all MDX documentation files into searchable vector embeddings.
Text Chunking
Documents are split into 1000-character chunks with 200-character overlap to preserve context across boundaries.
Embedding Model
HuggingFace's all-MiniLM-L6-v2 generates 384-dimensional dense vectors optimized for semantic similarity.
Vector Storage
Pinecone stores vectors with metadata (title, URL, category, content) for fast approximate nearest-neighbor search.
Chat API — Real-Time RAG
When a user sends a message, the chat API retrieves relevant documentation context and augments the LLM prompt with it — this is the "retrieval" in Retrieval-Augmented Generation.
Key Design Decisions
- Streaming SSE — Responses are streamed token-by-token via Server-Sent Events for instant perceived latency.
- Rate Limiting — IP-based rate limiting at 30 requests/minute prevents abuse without requiring authentication.
- Input Guardrails — Sanitization and validation filter unsafe or off-topic queries before hitting the LLM.
- Context Window — Top 5 most relevant chunks are injected into the system prompt, keeping the context focused.
Deep Research Agent
The research agent handles complex, multi-faceted questions by decomposing them into sub-queries, searching independently, and synthesizing a comprehensive answer with citations.
Technology Stack
Groq
LLM Inference
Runs Llama 3.3 70B with ultra-low latency. Used for both chat responses and query decomposition in the research agent.
Pinecone
Vector Database
Stores 384-dimensional embeddings with metadata. Supports fast approximate nearest-neighbor search for semantic retrieval.
HuggingFace
Embedding Generation
all-MiniLM-L6-v2 model generates dense embeddings for both documents (offline) and queries (real-time).
Next.js
API Routes & SSE
App Router API routes handle streaming responses via ReadableStream, enabling real-time token delivery to the client.
Cost & Economics
The entire AI pipeline runs on free tiers. Here are the exact numbers and where the cost cliffs are.
| Service | Current Cost | Tier | Limit |
|---|---|---|---|
| Vercel | $0 | Hobby (free) | 100GB bandwidth, 60s function timeout |
| Groq | $0 | Free tier | ~30 RPM, ~14.4K req/day |
| HuggingFace | $0 | Free Inference API | Rate limited, shared infra |
| Pinecone | $0 | Starter (free) | 100K vectors, 1 index |
| Domain | ~$1/mo | Custom domain | docs.masst.dev |
Evaluation & Monitoring
How the RAG pipeline quality is measured — and where the gaps are.
End-to-End Data Flow
Built with Next.js 15, Groq, Pinecone, HuggingFace, and Mermaid.
← Back to Home