crawl4ai

Turns any website into clean, LLM-ready data

★ 86k · updated 2 days ago

View unclecode/crawl4ai on GitHub

A crawler: respect each site's terms and robots.txt.

Skills can run commands on your computer. Review a skill before installing.

Listed from public GitHub data. Not affiliated with or endorsed by the authors unless marked Official. Names and logos belong to their owners and only identify the repositories. Sources and removal requests

What it does

An open-source Python crawler that turns web pages into clean Markdown or structured data for language models. It runs a real browser for pages that need JavaScript and can crawl many pages at once.

Read the README on GitHub

Who it’s for

Developers building RAG pipelines, research agents or data collection that starts from websites.

Install

Install it with:

pip install -U crawl4ai
crawl4ai-setup

Frameworks usually need some setup: follow the README for its configuration.

Activity

Last commit
2 days ago
Open issues and pull requests
282
Forks
8,962

From GitHub. Stars and commits are refreshed daily; contributors, languages and releases weekly.

At a glance

Licence: Apache-2.0. The licence is the authors’, as GitHub reports it. Check the repository before you rely on it.

Type
Framework
Languages
Python
Maintainer
unclecode
Stars on GitHub
★ 86k
Badges
Top, Dual-use

Is this your repository? Choose “Request removal” on this page, or email legal@antspace.ai: we act on removal requests within 48 hours.

Similar skills

More in Browser, web & scraping · All AI skills