llm.txt generator
Crawl docs. Compose concise llms.txt.
Politely crawls documentation, extracts clean Markdown, and composes an optimized `llms.txt` for LLM context windows — fast, deterministic, and robots-aware.
• Robots-aware
• Optimized size budget
• Live progress
Status
Shortcuts: Ctrl+Enter generate • / search • Ctrl+C copy • Ctrl+D download
Status will appear here after you start a job.
How it works
1) Paste a docs URL
We validate and normalize the URL, then check robots.txt and discover sitemaps (EN‑only).
2) Crawl and extract
Polite, rate‑limited crawl. Markdown‑first extraction that strips navigation and boilerplate; preserves code.
3) Compose and download
Prioritize high‑signal sections, apply a size budget, and produce llms.txt (optional llms‑full.txt).
Why you need this
Stop copying and pasting docs into ChatGPT.
Documentation sites are massive, messy, and built for humans — not LLMs. We crawl, clean, and compress them into context‑ready files.
Your LLM context windows are precious.
2M tokens go fast when you paste entire sites. Get clean, prioritized content that fits your budget.
Respect the robots.
We follow robots.txt, throttle politely, and never overwhelm servers. Professional crawling for professional developers.