Web & Research
17HTTP clients, scraping, crawling, search, and DNS. How an agent reaches out to the internet and pulls structured data back.
Scraping & crawling
- Firecrawlturn any site into clean LLM-ready markdown; crawl, scrape, and extract via CLI/API. Built for agents; handles JS rendering and returns parseable content.
- crawl4aiopen-source LLM-friendly web crawler with a
crwlCLI; outputs markdown and extracted structured data. - monolithsave a complete web page (CSS, JS, images inlined) as one self-contained HTML file for later parsing/archiving.
- pupparse HTML on the command line with CSS selectors; pull structured data out of fetched pages into text/JSON.
- crawleyunix-way web crawler that emits links/data to stdout for pipelines.
- hgetrender a web page as plain text in the terminal; cheap page-to-text.
- deadlinkfind dead/broken links in files and pages; scriptable link validation.
- Playwrightdrive a real browser (click, type, screenshot, wait) from code/CLI when a page needs full JS execution.
HTTP clients
DNS & network lookup
Search & translation APIs
- Exaembeddings-based web search API built for LLMs; returns clean content, not just links.
- Tavilysearch API optimized for agent RAG with source-ranked results.
- Jina Readerprepend
r.jina.ai/to any URL to get LLM-ready markdown of the page. - translate-shelltranslate text via Google/Bing/Yandex from the CLI; scriptable, reads args/stdin.