Website Knowledge Base
Three implementations of the same problem — crawl a website, make it answerable — built on AWS Bedrock Knowledge Bases, on Firecrawl with MongoDB vectors, and as a Bedrock web agent over Google Search.
Overview
Point a system at a URL and let people ask questions about what’s there. It sounds like one feature; it’s really a stack of decisions about crawling, chunking, embedding, storage, and freshness. I built it three separate ways to find out which of those decisions actually matter — one managed on AWS, one assembled from parts, and one that skipped storage entirely in favour of live search.
The three implementations
1. AWS Bedrock Knowledge Bases (managed)
A Python service that provisions and drives Bedrock end to end: creating knowledge bases, attaching web-crawler data sources, re-pointing an existing data source at a new URL, and kicking off ingestion jobs — with client-token generation for idempotent creation. Embeddings via Cohere embed-english-v3, vectors stored in OpenSearch Serverless with Bedrock’s standard field mapping, and retrieval-and-generation through the bedrock-agent-runtime client.
What it bought: ingestion, chunking, embedding, and index management handled by the platform. What it cost: everything is a provisioning call, so iteration is slow and the pipeline is opaque when retrieval quality disappoints.
2. Firecrawl + MongoDB vectors (assembled)
A Flask service that crawls with Firecrawl, embeds through an OpenAI-compatible endpoint, and stores vectors in MongoDB. Crawls are per-user and idempotent by construction — re-crawling a site deletes that user’s prior documents before writing new ones, so the knowledge base reflects the site as it is now rather than accumulating stale copies. Long crawls run through a Celery task queue rather than blocking the request.
What it bought: full control over chunking and freshness semantics, and a much shorter feedback loop. What it cost: you own the operational surface — queue, workers, index lifecycle.
3. Bedrock + Google Custom Search (no storage at all)
A web agent that skips the vector store completely: query Google Custom Search, scrape the top results with BeautifulSoup, and feed the extracted text straight to a Bedrock model as context. No embedding, no index, nothing to keep fresh.
What it bought: always-current answers and zero storage cost. What it cost: latency per query, and total dependence on whatever the search ranking hands back.
Technologies Used
- AWS Bedrock (
bedrock-runtime,bedrock-agent-runtime) with Knowledge Bases and web-crawler data sources - Amazon OpenSearch Serverless for vector storage; Cohere embed-english-v3 for embeddings
- Firecrawl for JS-aware crawling; MongoDB as vector store; OpenAI embeddings and completions
- Flask + Celery for the API and background crawl jobs
- Google Custom Search API + BeautifulSoup for the live-retrieval variant
- boto3, Python 3.12
Challenges and Learnings
The interesting result was that the retrieval mechanism mattered less than the freshness policy. A managed pipeline with excellent embeddings still gives wrong answers if the crawl is a month stale, while a crude live-search approach stays right by construction. That pushed me toward treating “when does this data expire, and what happens then” as the first question in a RAG design rather than an operational afterthought — the delete-then-recrawl semantics in the Firecrawl version came directly out of getting this wrong first.
The managed-versus-assembled tradeoff resolved the way it usually does: Bedrock Knowledge Bases are the right call when the pipeline is boring and you want it to stay that way, and the wrong call the moment retrieval quality becomes the thing you’re iterating on.
Outcome
Three working systems and a clear opinion about when to reach for each. The freshness lessons and the chunk-metadata instincts from this work fed directly into the retrieval systems I build now.