Systems / Personal system / Private repository
Web context gateway
A private search and crawl service my machines and AI agents share, with hard crawl budgets and per-client tokens.
Overview
What it is
My AI workflows depended on separate hosted search and scraping services, each with its own schema, limits, and outages. The gateway gives every trusted machine and agent one API for search, page-to-Markdown, site maps, and bounded crawls. It runs on a small always-on Linux box that is never exposed to the internet.
Scope
What it includes
- 01
Web search through self-hosted SearXNG
- 02
Scrape, map, and crawl through self-hosted Firecrawl
- 03
REST, MCP, and a CLI over one gateway
- 04
A separate revocable token for each client
Role
My role
I built it alone in July 2026: the Go gateway, the CLI, the MCP adapter, the deployment, and the qualification suites.
Build
Technical notes
- Gateway
- A small Go service owns auth, policy, normalized responses, REST, and MCP. SearXNG and Firecrawl sit on an internal container network with no host ports.
- Capacity
- The host has two cores and 8 GB of RAM, so the gateway runs one crawl and one browser page at a time and sheds load past that.
- Safety
- Every submitted or discovered URL is treated as hostile input, and crawled content is only kept when a request opts in.
- Qualification
- Unit, integration, race, live REST and MCP, security, load-shedding, restart, and restore suites all pass.
- Go
- SearXNG
- Firecrawl
- Docker
- Tailscale
- MCP
Selected work
Development work
- 01 Architecture
- 02 Go Gateway
- 03 CLI
- 04 MCP Adapter
- 05 Deployment
- 06 Qualification
Results
Results
- Qualified for three clients, each with its own revocable token: the CLI, Codex, and Claude Code.