GitHub repo leaderboard by stars, growth rate and activity.
The context API to search, scrape, and interact with the web at scale. 🔥
🕷️ An adaptive Web Scraping framework that handles everything from a single request to a full-scale crawl!
Scrapy, a fast high-level web crawling & scraping framework for Python.
🕵️♂️ Collect a dossier on a person by username from 3000+ sites
Crawlee—A web scraping and browser automation library for Node.js to build reliable crawlers. In JavaScript and TypeScript. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and other files from websites. Works with Puppeteer, Playwright, Cheerio, JSDOM, and raw HTTP. Both headful and headless mode. With proxy rotation.
Custom Selenium Chromedriver | Zero-Config | Passes ALL bot mitigation systems (like Distil / Imperva/ Datadadome / CloudFlare IUAM)
A scalable web crawler framework for Java.
| # | Repo | Language | Stars | 30-day trend | Last updated |
|---|---|---|---|---|---|
| 1 | The context API to search, scrape, and interact with the web at scale. 🔥 | TypeScript | 178,457 | last pushed Yesterday | |
| 2 | 🕷️ An adaptive Web Scraping framework that handles everything from a single request to a full-scale crawl! | Python | 79,739 | last pushed 6 days ago | |
| 3 | Scrapy, a fast high-level web crawling & scraping framework for Python. | Python | 64,268 | last pushed 2 days ago | |
| 4 | 🕵️♂️ Collect a dossier on a person by username from 3000+ sites | Python | 37,412 | last pushed Yesterday | |
| 5 | Python scraper based on AI | Python | 30,783 | last pushed 3 days ago | |
| 6 | Crawlee—A web scraping and browser automation library for Node.js to build reliable crawlers. In JavaScript and TypeScript. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and other files from websites. Works with Puppeteer, Playwright, Cheerio, JSDOM, and raw HTTP. Both headful and headless mode. With proxy rotation. | TypeScript | 25,713 | last pushed Yesterday | |
| 7 | Elegant Scraper and Crawler Framework for Golang | Go | 25,504 | last pushed 1 week ago | |
| 8 | Custom Selenium Chromedriver | Zero-Config | Passes ALL bot mitigation systems (like Distil / Imperva/ Datadadome / CloudFlare IUAM) | Python | 12,827 | last pushed 1 year ago | |
| 9 | 🦊 Anti-detect browser | C++ | 11,782 | last pushed 3 days ago | |
| 10 | A scalable web crawler framework for Java. | Java | 11,678 | last pushed 9 months ago |
All · 11,474