A type-safe crawler API built to explore crawling, extraction, Markdown conversion, caching, rate limiting, and pipeline-oriented backend design.
Context
QuickCrawl is a type-safe crawler API built as a learning-led infrastructure project. It was inspired by products like Firecrawl, but the goal was understanding the shape of the problem — not cloning an existing product.
Problem
Crawling sounds simple until the system has to fetch pages reliably, extract metadata, convert HTML into Markdown, cache expensive work, rate-limit requests, and expose everything through an API that stays easy to extend.
Product
QuickCrawl exposes a small but coherent crawler API: page fetching, metadata extraction, HTML-to-Markdown conversion, request validation, caching, and rate limiting — all behind type-safe request and response contracts.
System Architecture
QuickCrawl is organised as a pipeline-oriented system with distinct responsibilities:
Loading diagram…
QuickCrawl API — the Hono-based HTTP surface with validated request/response contracts. The entry point for clients; stays thin so the pipeline can evolve independently.
Pipeline Engine — the workflow stages: fetch, extract, convert, validate. Each stage is a discrete step rather than a tangled utility layer, which makes the system easier to reason about, test, and extend.
Cache Layer — caches expensive crawl operations so repeated requests do not re-fetch and re-process identical work.
Built with Bun and Hono for a fast, lightweight runtime. Architecture notes stay visible through documentation and decision records so the reasoning does not disappear behind the code.
Current State
QuickCrawl is a focused developer tool and architecture study. The natural next steps are deeper extraction controls, better crawl queues, and richer structured output formats.
