Web scraping - 精选合集

Scrapy, a fast high-level web crawling & scraping framework for Python.

Crawlee—A web scraping and browser automation library for Node.js to build reliable crawlers. In JavaScript and TypeScript. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, an…

Crawlee—A web scraping and browser automation library for Python to build reliable crawlers. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and other files from websites. Wo…

Elegant Scraper and Crawler Framework for Golang

🕷️ A Web Scraping framework that handles everything from a single request to a full-scale crawl! Don't be shy, join here: https://discord.gg/EMgGbDceNQ and follow here for daily tips and tricks: ht…

Foundational low latency web data collecting in Rust

A next-generation crawling and spidering framework.

Supercharge your AI agents with data from the web and beyond. Building the library for superintelligence. 🔥

Open-source web crawler and scraper for LLMs and AI agents: any website into clean, LLM-ready Markdown. Run it yourself, or use Crawl4AI Cloud with one key. Join our Discord https://discord.gg/Vuxz…

Python scraper based on AI

Playwright is a framework for Web Testing and Automation. It allows testing Chromium, Firefox and WebKit with a single API.

JavaScript API for Chrome and Firefox

A browser automation framework and ecosystem.

📊 Python APIs for Web Automation, E2E Testing, and Data Scraping. CDP Mode bypasses bot-detection and handles CAPTCHAs. Includes a Stealth mode for Playwright.

Lightpanda: the headless browser designed for AI and automation

Deploy headless browsers in Docker. Run on our cloud or bring your own. Free for non-commercial uses.

The fast, flexible, and elegant library for parsing and manipulating HTML and XML.

jsoup: the Java HTML parser, built for HTML editing, cleaning, scraping, and XSS safety.

Nokogiri (鋸) makes it easy and painless to work with XML and HTML from Ruby.

A little like that j-thing, only in Go.