# Table of Contents - [Home - Crawl4AI Documentation](#home-crawl4ai-documentation) - [Crawl Dispatcher - Crawl4AI Documentation](#crawl-dispatcher-crawl4ai-documentation) - [File Downloading - Crawl4AI Documentation](#file-downloading-crawl4ai-documentation) - [Identity Based Crawling - Crawl4AI Documentation](#identity-based-crawling-crawl4ai-documentation) - [Proxy & Security - Crawl4AI Documentation](#proxy-security-crawl4ai-documentation) - [SSL Certificate - Crawl4AI Documentation](#ssl-certificate-crawl4ai-documentation) - [Hooks & Auth - Crawl4AI Documentation](#hooks-auth-crawl4ai-documentation) - [Multi-URL Crawling - Crawl4AI Documentation](#multi-url-crawling-crawl4ai-documentation) - [Lazy Loading - Crawl4AI Documentation](#lazy-loading-crawl4ai-documentation) - [Dockerize hooks - Crawl4AI Documentation](#dockerize-hooks-crawl4ai-documentation) - [Session Management - Crawl4AI Documentation](#session-management-crawl4ai-documentation) - [AsyncWebCrawler - Crawl4AI Documentation](#asyncwebcrawler-crawl4ai-documentation) - [Release Summary for Version 0.4.0 (December 1, 2024) - Crawl4AI Documentation](#release-summary-for-version-0-4-0-december-1-2024-crawl4ai-documentation) - [Blog Home - Crawl4AI Documentation](#blog-home-crawl4ai-documentation) - [Release Summary for Version 0.4.1 (December 8, 2024): Major Efficiency Boosts with New Features! - Crawl4AI Documentation](#release-summary-for-version-0-4-1-december-8-2024-major-efficiency-boosts-with-new-features-crawl4ai-documentation) - [CrawlResult - Crawl4AI Documentation](#crawlresult-crawl4ai-documentation) - [Cache Modes - Crawl4AI Documentation](#cache-modes-crawl4ai-documentation) - [Installation - Crawl4AI Documentation](#installation-crawl4ai-documentation) - [arun() - Crawl4AI Documentation](#arun-crawl4ai-documentation) - [0.4.2 - Crawl4AI Documentation](#0-4-2-crawl4ai-documentation) - [Strategies - Crawl4AI Documentation](#strategies-crawl4ai-documentation) - [Browser & Crawler Config - Crawl4AI Documentation](#browser-crawler-config-crawl4ai-documentation) - [Browser & Crawler Config - Crawl4AI Documentation](#browser-crawler-config-crawl4ai-documentation) - [Crawler Result - Crawl4AI Documentation](#crawler-result-crawl4ai-documentation) - [Docker Deployment - Crawl4AI Documentation](#docker-deployment-crawl4ai-documentation) - [Markdown Generation - Crawl4AI Documentation](#markdown-generation-crawl4ai-documentation) - [Overview - Crawl4AI Documentation](#overview-crawl4ai-documentation) - [Fit Markdown - Crawl4AI Documentation](#fit-markdown-crawl4ai-documentation) - [Quick Start - Crawl4AI Documentation](#quick-start-crawl4ai-documentation) - [Content Selection - Crawl4AI Documentation](#content-selection-crawl4ai-documentation) - [Link & Media - Crawl4AI Documentation](#link-media-crawl4ai-documentation) - [Local Files & Raw HTML - Crawl4AI Documentation](#local-files-raw-html-crawl4ai-documentation) - [Page Interaction - Crawl4AI Documentation](#page-interaction-crawl4ai-documentation) - [Simple Crawling - Crawl4AI Documentation](#simple-crawling-crawl4ai-documentation) - [Chunking - Crawl4AI Documentation](#chunking-crawl4ai-documentation) - [Clustering Strategies - Crawl4AI Documentation](#clustering-strategies-crawl4ai-documentation) - [LLM-Free Strategies - Crawl4AI Documentation](#llm-free-strategies-crawl4ai-documentation) - [Home - Crawl4AI Documentation](#home-crawl4ai-documentation) - [LLM Strategies - Crawl4AI Documentation](#llm-strategies-crawl4ai-documentation) - [Quick Start - Crawl4AI Documentation](#quick-start-crawl4ai-documentation) - [Docker Deployment - Crawl4AI Documentation](#docker-deployment-crawl4ai-documentation) - [Blog Home - Crawl4AI Documentation](#blog-home-crawl4ai-documentation) - [Installation - Crawl4AI Documentation](#installation-crawl4ai-documentation) - [Simple Crawling - Crawl4AI Documentation](#simple-crawling-crawl4ai-documentation) - [Crawler Result - Crawl4AI Documentation](#crawler-result-crawl4ai-documentation) - [Browser & Crawler Config - Crawl4AI Documentation](#browser-crawler-config-crawl4ai-documentation) - [Markdown Generation - Crawl4AI Documentation](#markdown-generation-crawl4ai-documentation) - [Fit Markdown - Crawl4AI Documentation](#fit-markdown-crawl4ai-documentation) - [File Downloading - Crawl4AI Documentation](#file-downloading-crawl4ai-documentation) - [Page Interaction - Crawl4AI Documentation](#page-interaction-crawl4ai-documentation) - [Cache Modes - Crawl4AI Documentation](#cache-modes-crawl4ai-documentation) - [Content Selection - Crawl4AI Documentation](#content-selection-crawl4ai-documentation) - [Link & Media - Crawl4AI Documentation](#link-media-crawl4ai-documentation) - [Session Management - Crawl4AI Documentation](#session-management-crawl4ai-documentation) - [Proxy & Security - Crawl4AI Documentation](#proxy-security-crawl4ai-documentation) - [Lazy Loading - Crawl4AI Documentation](#lazy-loading-crawl4ai-documentation) - [Overview - Crawl4AI Documentation](#overview-crawl4ai-documentation) - [Local Files & Raw HTML - Crawl4AI Documentation](#local-files-raw-html-crawl4ai-documentation) - [Hooks & Auth - Crawl4AI Documentation](#hooks-auth-crawl4ai-documentation) - [Multi-URL Crawling - Crawl4AI Documentation](#multi-url-crawling-crawl4ai-documentation) - [Crawl Dispatcher - Crawl4AI Documentation](#crawl-dispatcher-crawl4ai-documentation) - [Identity Based Crawling - Crawl4AI Documentation](#identity-based-crawling-crawl4ai-documentation) - [SSL Certificate - Crawl4AI Documentation](#ssl-certificate-crawl4ai-documentation) - [Clustering Strategies - Crawl4AI Documentation](#clustering-strategies-crawl4ai-documentation) - [Chunking - Crawl4AI Documentation](#chunking-crawl4ai-documentation) - [0.4.2 - Crawl4AI Documentation](#0-4-2-crawl4ai-documentation) - [Release Summary for Version 0.4.1 (December 8, 2024): Major Efficiency Boosts with New Features! - Crawl4AI Documentation](#release-summary-for-version-0-4-1-december-8-2024-major-efficiency-boosts-with-new-features-crawl4ai-documentation) - [Release Summary for Version 0.4.0 (December 1, 2024) - Crawl4AI Documentation](#release-summary-for-version-0-4-0-december-1-2024-crawl4ai-documentation) - [LLM-Free Strategies - Crawl4AI Documentation](#llm-free-strategies-crawl4ai-documentation) - [AsyncWebCrawler - Crawl4AI Documentation](#asyncwebcrawler-crawl4ai-documentation) - [arun() - Crawl4AI Documentation](#arun-crawl4ai-documentation) - [LLM Strategies - Crawl4AI Documentation](#llm-strategies-crawl4ai-documentation) - [Strategies - Crawl4AI Documentation](#strategies-crawl4ai-documentation) - [CrawlResult - Crawl4AI Documentation](#crawlresult-crawl4ai-documentation) - [Browser & Crawler Config - Crawl4AI Documentation](#browser-crawler-config-crawl4ai-documentation) --- # Home - Crawl4AI Documentation 🚀🤖 Crawl4AI: Open-Source LLM-Friendly Web Crawler & Scraper ============================================================= [![unclecode%2Fcrawl4ai | Trendshift](https://trendshift.io/api/badge/repositories/11716)](https://trendshift.io/repositories/11716) [![GitHub Stars](https://img.shields.io/github/stars/unclecode/crawl4ai?style=social)](https://github.com/unclecode/crawl4ai/stargazers) [![GitHub Forks](https://img.shields.io/github/forks/unclecode/crawl4ai?style=social)](https://github.com/unclecode/crawl4ai/network/members) [![PyPI version](https://badge.fury.io/py/crawl4ai.svg)](https://badge.fury.io/py/crawl4ai) [![Python Version](https://img.shields.io/pypi/pyversions/crawl4ai)](https://pypi.org/project/crawl4ai/) [![Downloads](https://static.pepy.tech/badge/crawl4ai/month)](https://pepy.tech/project/crawl4ai) [![License](https://img.shields.io/github/license/unclecode/crawl4ai)](https://github.com/unclecode/crawl4ai/blob/main/LICENSE) Crawl4AI is the #1 trending GitHub repository, actively maintained by a vibrant community. It delivers blazing-fast, AI-ready web crawling tailored for large language models, AI agents, and data pipelines. Fully open source, flexible, and built for real-time performance, **Crawl4AI** empowers developers with unmatched speed, precision, and deployment ease. > **Note**: If you're looking for the old documentation, you can access it [here](https://old.docs.crawl4ai.com) > . Quick Start ----------- Here's a quick example to show you how easy it is to use Crawl4AI with its asynchronous capabilities: `[](#__codelineno-0-1) import asyncio [](#__codelineno-0-2) from crawl4ai import AsyncWebCrawler [](#__codelineno-0-3) [](#__codelineno-0-4) async def main(): [](#__codelineno-0-5) # Create an instance of AsyncWebCrawler [](#__codelineno-0-6) async with AsyncWebCrawler() as crawler: [](#__codelineno-0-7) # Run the crawler on a URL [](#__codelineno-0-8) result = await crawler.arun(url="https://crawl4ai.com") [](#__codelineno-0-9) [](#__codelineno-0-10) # Print the extracted content [](#__codelineno-0-11) print(result.markdown) [](#__codelineno-0-12) [](#__codelineno-0-13) # Run the async main function [](#__codelineno-0-14) asyncio.run(main())` * * * What Does Crawl4AI Do? ---------------------- Crawl4AI is a feature-rich crawler and scraper that aims to: 1. **Generate Clean Markdown**: Perfect for RAG pipelines or direct ingestion into LLMs. 2. **Structured Extraction**: Parse repeated patterns with CSS, XPath, or LLM-based extraction. 3. **Advanced Browser Control**: Hooks, proxies, stealth modes, session re-use—fine-grained control. 4. **High Performance**: Parallel crawling, chunk-based extraction, real-time use cases. 5. **Open Source**: No forced API keys, no paywalls—everyone can access their data. **Core Philosophies**: - **Democratize Data**: Free to use, transparent, and highly configurable. \- **LLM Friendly**: Minimally processed, well-structured text, images, and metadata, so AI models can easily consume it. * * * Documentation Structure ----------------------- To help you get started, we’ve organized our docs into clear sections: * **Setup & Installation** Basic instructions to install Crawl4AI via pip or Docker. * **Quick Start** A hands-on introduction showing how to do your first crawl, generate Markdown, and do a simple extraction. * **Core** Deeper guides on single-page crawling, advanced browser/crawler parameters, content filtering, and caching. * **Advanced** Explore link & media handling, lazy loading, hooking & authentication, proxies, session management, and more. * **Extraction** Detailed references for no-LLM (CSS, XPath) vs. LLM-based strategies, chunking, and clustering approaches. * **API Reference** Find the technical specifics of each class and method, including `AsyncWebCrawler`, `arun()`, and `CrawlResult`. Throughout these sections, you’ll find code samples you can **copy-paste** into your environment. If something is missing or unclear, raise an issue or PR. * * * How You Can Support ------------------- * **Star & Fork**: If you find Crawl4AI helpful, star the repo on GitHub or fork it to add your own features. * **File Issues**: Encounter a bug or missing feature? Let us know by filing an issue, so we can improve. * **Pull Requests**: Whether it’s a small fix, a big feature, or better docs—contributions are always welcome. * **Join Discord**: Come chat about web scraping, crawling tips, or AI workflows with the community. * **Spread the Word**: Mention Crawl4AI in your blog posts, talks, or on social media. **Our mission**: to empower everyone—students, researchers, entrepreneurs, data scientists—to access, parse, and shape the world’s data with speed, cost-efficiency, and creative freedom. * * * Quick Links ----------- * **[GitHub Repo](https://github.com/unclecode/crawl4ai) ** * **[Installation Guide](core/installation/) ** * **[Quick Start](core/quickstart/) ** * **[API Reference](api/async-webcrawler/) ** * **[Changelog](https://github.com/unclecode/crawl4ai/blob/main/CHANGELOG.md) ** Thank you for joining me on this journey. Let’s keep building an **open, democratic** approach to data extraction and AI together. Happy Crawling! — _Unclecde, Founder & Maintainer of Crawl4AI_ * * * --- # Crawl Dispatcher - Crawl4AI Documentation Crawl Dispatcher ================ We’re excited to announce a **Crawl Dispatcher** module that can handle **thousands** of crawling tasks simultaneously. By efficiently managing system resources (memory, CPU, network), this dispatcher ensures high-performance data extraction at scale. It also provides **real-time monitoring** of each crawler’s status, memory usage, and overall progress. Stay tuned—this feature is **coming soon** in an upcoming release of Crawl4AI! For the latest news, keep an eye on our changelogs and follow [@unclecode](https://twitter.com/unclecode) on X. Below is a **sample** of how the dispatcher’s performance monitor might look in action: ![Crawl Dispatcher Performance Monitor](../../assets/images/dispatcher.png) We can’t wait to bring you this streamlined, **scalable** approach to multi-URL crawling—**watch this space** for updates! * * * --- # File Downloading - Crawl4AI Documentation Download Handling in Crawl4AI ============================= This guide explains how to use Crawl4AI to handle file downloads during crawling. You'll learn how to trigger downloads, specify download locations, and access downloaded files. Enabling Downloads ------------------ To enable downloads, set the `accept_downloads` parameter in the `BrowserConfig` object and pass it to the crawler. `[](#__codelineno-0-1) from crawl4ai.async_configs import BrowserConfig, AsyncWebCrawler [](#__codelineno-0-2) [](#__codelineno-0-3) async def main(): [](#__codelineno-0-4) config = BrowserConfig(accept_downloads=True) # Enable downloads globally [](#__codelineno-0-5) async with AsyncWebCrawler(config=config) as crawler: [](#__codelineno-0-6) # ... your crawling logic ... [](#__codelineno-0-7) [](#__codelineno-0-8) asyncio.run(main())` Specifying Download Location ---------------------------- Specify the download directory using the `downloads_path` attribute in the `BrowserConfig` object. If not provided, Crawl4AI defaults to creating a "downloads" directory inside the `.crawl4ai` folder in your home directory. `[](#__codelineno-1-1) from crawl4ai.async_configs import BrowserConfig [](#__codelineno-1-2) import os [](#__codelineno-1-3) [](#__codelineno-1-4) downloads_path = os.path.join(os.getcwd(), "my_downloads") # Custom download path [](#__codelineno-1-5) os.makedirs(downloads_path, exist_ok=True) [](#__codelineno-1-6) [](#__codelineno-1-7) config = BrowserConfig(accept_downloads=True, downloads_path=downloads_path) [](#__codelineno-1-8) [](#__codelineno-1-9) async def main(): [](#__codelineno-1-10) async with AsyncWebCrawler(config=config) as crawler: [](#__codelineno-1-11) result = await crawler.arun(url="https://example.com") [](#__codelineno-1-12) # ...` Triggering Downloads -------------------- Downloads are typically triggered by user interactions on a web page, such as clicking a download button. Use `js_code` in `CrawlerRunConfig` to simulate these actions and `wait_for` to allow sufficient time for downloads to start. `[](#__codelineno-2-1) from crawl4ai.async_configs import CrawlerRunConfig [](#__codelineno-2-2) [](#__codelineno-2-3) config = CrawlerRunConfig( [](#__codelineno-2-4) js_code=""" [](#__codelineno-2-5) const downloadLink = document.querySelector('a[href$=".exe"]'); [](#__codelineno-2-6) if (downloadLink) { [](#__codelineno-2-7) downloadLink.click(); [](#__codelineno-2-8) } [](#__codelineno-2-9) """, [](#__codelineno-2-10) wait_for=5 # Wait 5 seconds for the download to start [](#__codelineno-2-11) ) [](#__codelineno-2-12) [](#__codelineno-2-13) result = await crawler.arun(url="https://www.python.org/downloads/", config=config)` Accessing Downloaded Files -------------------------- The `downloaded_files` attribute of the `CrawlResult` object contains paths to downloaded files. `[](#__codelineno-3-1) if result.downloaded_files: [](#__codelineno-3-2) print("Downloaded files:") [](#__codelineno-3-3) for file_path in result.downloaded_files: [](#__codelineno-3-4) print(f"- {file_path}") [](#__codelineno-3-5) file_size = os.path.getsize(file_path) [](#__codelineno-3-6) print(f"- File size: {file_size} bytes") [](#__codelineno-3-7) else: [](#__codelineno-3-8) print("No files downloaded.")` Example: Downloading Multiple Files ----------------------------------- `[](#__codelineno-4-1) from crawl4ai.async_configs import BrowserConfig, CrawlerRunConfig [](#__codelineno-4-2) import os [](#__codelineno-4-3) from pathlib import Path [](#__codelineno-4-4) [](#__codelineno-4-5) async def download_multiple_files(url: str, download_path: str): [](#__codelineno-4-6) config = BrowserConfig(accept_downloads=True, downloads_path=download_path) [](#__codelineno-4-7) async with AsyncWebCrawler(config=config) as crawler: [](#__codelineno-4-8) run_config = CrawlerRunConfig( [](#__codelineno-4-9) js_code=""" [](#__codelineno-4-10) const downloadLinks = document.querySelectorAll('a[download]'); [](#__codelineno-4-11) for (const link of downloadLinks) { [](#__codelineno-4-12) link.click(); [](#__codelineno-4-13) // Delay between clicks [](#__codelineno-4-14) await new Promise(r => setTimeout(r, 2000)); [](#__codelineno-4-15) } [](#__codelineno-4-16) """, [](#__codelineno-4-17) wait_for=10 # Wait for all downloads to start [](#__codelineno-4-18) ) [](#__codelineno-4-19) result = await crawler.arun(url=url, config=run_config) [](#__codelineno-4-20) [](#__codelineno-4-21) if result.downloaded_files: [](#__codelineno-4-22) print("Downloaded files:") [](#__codelineno-4-23) for file in result.downloaded_files: [](#__codelineno-4-24) print(f"- {file}") [](#__codelineno-4-25) else: [](#__codelineno-4-26) print("No files downloaded.") [](#__codelineno-4-27) [](#__codelineno-4-28) # Usage [](#__codelineno-4-29) download_path = os.path.join(Path.home(), ".crawl4ai", "downloads") [](#__codelineno-4-30) os.makedirs(download_path, exist_ok=True) [](#__codelineno-4-31) [](#__codelineno-4-32) asyncio.run(download_multiple_files("https://www.python.org/downloads/windows/", download_path))` Important Considerations ------------------------ * **Browser Context:** Downloads are managed within the browser context. Ensure `js_code` correctly targets the download triggers on the webpage. * **Timing:** Use `wait_for` in `CrawlerRunConfig` to manage download timing. * **Error Handling:** Handle errors to manage failed downloads or incorrect paths gracefully. * **Security:** Scan downloaded files for potential security threats before use. This revised guide ensures consistency with the `Crawl4AI` codebase by using `BrowserConfig` and `CrawlerRunConfig` for all download-related configurations. Let me know if further adjustments are needed! * * * --- # Identity Based Crawling - Crawl4AI Documentation Preserve Your Identity with Crawl4AI ==================================== Crawl4AI empowers you to navigate and interact with the web using your **authentic digital identity**, ensuring you’re recognized as a human and not mistaken for a bot. This tutorial covers: 1. **Managed Browsers** – The recommended approach for persistent profiles and identity-based crawling. 2. **Magic Mode** – A simplified fallback solution for quick automation without persistent identity. * * * 1\. Managed Browsers: Your Digital Identity Solution ---------------------------------------------------- **Managed Browsers** let developers create and use **persistent browser profiles**. These profiles store local storage, cookies, and other session data, letting you browse as your **real self**—complete with logins, preferences, and cookies. ### Key Benefits * **Authentic Browsing Experience**: Retain session data and browser fingerprints as though you’re a normal user. * **Effortless Configuration**: Once you log in or solve CAPTCHAs in your chosen data directory, you can re-run crawls without repeating those steps. * **Empowered Data Access**: If you can see the data in your own browser, you can automate its retrieval with your genuine identity. * * * Below is a **partial update** to your **Managed Browsers** tutorial, specifically the section about **creating a user-data directory** using **Playwright’s Chromium** binary rather than a system-wide Chrome/Edge. We’ll show how to **locate** that binary and launch it with a `--user-data-dir` argument to set up your profile. You can then point `BrowserConfig.user_data_dir` to that folder for subsequent crawls. * * * ### Creating a User Data Directory (Command-Line Approach via Playwright) If you installed Crawl4AI (which installs Playwright under the hood), you already have a Playwright-managed Chromium on your system. Follow these steps to launch that **Chromium** from your command line, specifying a **custom** data directory: 1. **Find** the Playwright Chromium binary: - On most systems, installed browsers go under a `~/.cache/ms-playwright/` folder or similar path. \- To see an overview of installed browsers, run: `[](#__codelineno-0-1) python -m playwright install --dry-run` or `[](#__codelineno-1-1) playwright install --dry-run` (depending on your environment). This shows where Playwright keeps Chromium. * For instance, you might see a path like: `[](#__codelineno-2-1) ~/.cache/ms-playwright/chromium-1234/chrome-linux/chrome` on Linux, or a corresponding folder on macOS/Windows. 2. **Launch** the Playwright Chromium binary with a **custom** user-data directory: `[](#__codelineno-3-1) # Linux example [](#__codelineno-3-2) ~/.cache/ms-playwright/chromium-1234/chrome-linux/chrome \ [](#__codelineno-3-3) --user-data-dir=/home//my_chrome_profile` `[](#__codelineno-4-1) # macOS example (Playwright’s internal binary) [](#__codelineno-4-2) ~/Library/Caches/ms-playwright/chromium-1234/chrome-mac/Chromium.app/Contents/MacOS/Chromium \ [](#__codelineno-4-3) --user-data-dir=/Users//my_chrome_profile` `[](#__codelineno-5-1) # Windows example (PowerShell/cmd) [](#__codelineno-5-2) "C:\Users\\AppData\Local\ms-playwright\chromium-1234\chrome-win\chrome.exe" ^ [](#__codelineno-5-3) --user-data-dir="C:\Users\\my_chrome_profile"` **Replace** the path with the actual subfolder indicated in your `ms-playwright` cache structure. \- This **opens** a fresh Chromium with your new or existing data folder. \- **Log into** any sites or configure your browser the way you want. \- **Close** when done—your profile data is saved in that folder. 3. **Use** that folder in **`BrowserConfig.user_data_dir`**: `[](#__codelineno-6-1) from crawl4ai import AsyncWebCrawler, BrowserConfig, CrawlerRunConfig [](#__codelineno-6-2) [](#__codelineno-6-3) browser_config = BrowserConfig( [](#__codelineno-6-4) headless=True, [](#__codelineno-6-5) use_managed_browser=True, [](#__codelineno-6-6) user_data_dir="/home//my_chrome_profile", [](#__codelineno-6-7) browser_type="chromium" [](#__codelineno-6-8) )` \- Next time you run your code, it reuses that folder—**preserving** your session data, cookies, local storage, etc. * * * 3\. Using Managed Browsers in Crawl4AI -------------------------------------- Once you have a data directory with your session data, pass it to **`BrowserConfig`**: `[](#__codelineno-7-1) import asyncio [](#__codelineno-7-2) from crawl4ai import AsyncWebCrawler, BrowserConfig, CrawlerRunConfig [](#__codelineno-7-3) [](#__codelineno-7-4) async def main(): [](#__codelineno-7-5) # 1) Reference your persistent data directory [](#__codelineno-7-6) browser_config = BrowserConfig( [](#__codelineno-7-7) headless=True, # 'True' for automated runs [](#__codelineno-7-8) verbose=True, [](#__codelineno-7-9) use_managed_browser=True, # Enables persistent browser strategy [](#__codelineno-7-10) browser_type="chromium", [](#__codelineno-7-11) user_data_dir="/path/to/my-chrome-profile" [](#__codelineno-7-12) ) [](#__codelineno-7-13) [](#__codelineno-7-14) # 2) Standard crawl config [](#__codelineno-7-15) crawl_config = CrawlerRunConfig( [](#__codelineno-7-16) wait_for="css:.logged-in-content" [](#__codelineno-7-17) ) [](#__codelineno-7-18) [](#__codelineno-7-19) async with AsyncWebCrawler(config=browser_config) as crawler: [](#__codelineno-7-20) result = await crawler.arun(url="https://example.com/private", config=crawl_config) [](#__codelineno-7-21) if result.success: [](#__codelineno-7-22) print("Successfully accessed private data with your identity!") [](#__codelineno-7-23) else: [](#__codelineno-7-24) print("Error:", result.error_message) [](#__codelineno-7-25) [](#__codelineno-7-26) if __name__ == "__main__": [](#__codelineno-7-27) asyncio.run(main())` ### Workflow 1. **Login** externally (via CLI or your normal Chrome with `--user-data-dir=...`). 2. **Close** that browser. 3. **Use** the same folder in `user_data_dir=` in Crawl4AI. 4. **Crawl** – The site sees your identity as if you’re the same user who just logged in. * * * 4\. Magic Mode: Simplified Automation ------------------------------------- If you **don’t** need a persistent profile or identity-based approach, **Magic Mode** offers a quick way to simulate human-like browsing without storing long-term data. `[](#__codelineno-8-1) from crawl4ai import AsyncWebCrawler, CrawlerRunConfig [](#__codelineno-8-2) [](#__codelineno-8-3) async with AsyncWebCrawler() as crawler: [](#__codelineno-8-4) result = await crawler.arun( [](#__codelineno-8-5) url="https://example.com", [](#__codelineno-8-6) config=CrawlerRunConfig( [](#__codelineno-8-7) magic=True, # Simplifies a lot of interaction [](#__codelineno-8-8) remove_overlay_elements=True, [](#__codelineno-8-9) page_timeout=60000 [](#__codelineno-8-10) ) [](#__codelineno-8-11) )` **Magic Mode**: * Simulates a user-like experience * Randomizes user agent & navigator * Randomizes interactions & timings * Masks automation signals * Attempts pop-up handling **But** it’s no substitute for **true** user-based sessions if you want a fully legitimate identity-based solution. * * * 5\. Comparing Managed Browsers vs. Magic Mode --------------------------------------------- | Feature | **Managed Browsers** | **Magic Mode** | | --- | --- | --- | | **Session Persistence** | Full localStorage/cookies retained in user\_data\_dir | No persistent data (fresh each run) | | **Genuine Identity** | Real user profile with full rights & preferences | Emulated user-like patterns, but no actual identity | | **Complex Sites** | Best for login-gated sites or heavy config | Simple tasks, minimal login or config needed | | **Setup** | External creation of user\_data\_dir, then use in Crawl4AI | Single-line approach (`magic=True`) | | **Reliability** | Extremely consistent (same data across runs) | Good for smaller tasks, can be less stable | * * * 6\. Summary ----------- * **Create** your user-data directory by launching Chrome/Chromium externally with `--user-data-dir=/some/path`. * **Log in** or configure sites as needed, then close the browser. * **Reference** that folder in `BrowserConfig(user_data_dir="...")` + `use_managed_browser=True`. * Enjoy **persistent** sessions that reflect your real identity. * If you only need quick, ephemeral automation, **Magic Mode** might suffice. **Recommended**: Always prefer a **Managed Browser** for robust, identity-based crawling and simpler interactions with complex sites. Use **Magic Mode** for quick tasks or prototypes where persistent data is unnecessary. With these approaches, you preserve your **authentic** browsing environment, ensuring the site sees you exactly as a normal user—no repeated logins or wasted time. * * * --- # Proxy & Security - Crawl4AI Documentation Proxy ===== Basic Proxy Setup ----------------- Simple proxy configuration with `BrowserConfig`: `[](#__codelineno-0-1) from crawl4ai.async_configs import BrowserConfig [](#__codelineno-0-2) [](#__codelineno-0-3) # Using proxy URL [](#__codelineno-0-4) browser_config = BrowserConfig(proxy="http://proxy.example.com:8080") [](#__codelineno-0-5) async with AsyncWebCrawler(config=browser_config) as crawler: [](#__codelineno-0-6) result = await crawler.arun(url="https://example.com") [](#__codelineno-0-7) [](#__codelineno-0-8) # Using SOCKS proxy [](#__codelineno-0-9) browser_config = BrowserConfig(proxy="socks5://proxy.example.com:1080") [](#__codelineno-0-10) async with AsyncWebCrawler(config=browser_config) as crawler: [](#__codelineno-0-11) result = await crawler.arun(url="https://example.com")` Authenticated Proxy ------------------- Use an authenticated proxy with `BrowserConfig`: `[](#__codelineno-1-1) from crawl4ai.async_configs import BrowserConfig [](#__codelineno-1-2) [](#__codelineno-1-3) proxy_config = { [](#__codelineno-1-4) "server": "http://proxy.example.com:8080", [](#__codelineno-1-5) "username": "user", [](#__codelineno-1-6) "password": "pass" [](#__codelineno-1-7) } [](#__codelineno-1-8) [](#__codelineno-1-9) browser_config = BrowserConfig(proxy_config=proxy_config) [](#__codelineno-1-10) async with AsyncWebCrawler(config=browser_config) as crawler: [](#__codelineno-1-11) result = await crawler.arun(url="https://example.com")` Rotating Proxies ---------------- Example using a proxy rotation service and updating `BrowserConfig` dynamically: `[](#__codelineno-2-1) from crawl4ai.async_configs import BrowserConfig [](#__codelineno-2-2) [](#__codelineno-2-3) async def get_next_proxy(): [](#__codelineno-2-4) # Your proxy rotation logic here [](#__codelineno-2-5) return {"server": "http://next.proxy.com:8080"} [](#__codelineno-2-6) [](#__codelineno-2-7) browser_config = BrowserConfig() [](#__codelineno-2-8) async with AsyncWebCrawler(config=browser_config) as crawler: [](#__codelineno-2-9) # Update proxy for each request [](#__codelineno-2-10) for url in urls: [](#__codelineno-2-11) proxy = await get_next_proxy() [](#__codelineno-2-12) browser_config.proxy_config = proxy [](#__codelineno-2-13) result = await crawler.arun(url=url, config=browser_config)` * * * --- # SSL Certificate - Crawl4AI Documentation `SSLCertificate` Reference ========================== The **`SSLCertificate`** class encapsulates an SSL certificate’s data and allows exporting it in various formats (PEM, DER, JSON, or text). It’s used within **Crawl4AI** whenever you set **`fetch_ssl_certificate=True`** in your **`CrawlerRunConfig`**. 1\. Overview ------------ **Location**: `crawl4ai/ssl_certificate.py` `[](#__codelineno-0-1) class SSLCertificate: [](#__codelineno-0-2) """ [](#__codelineno-0-3) Represents an SSL certificate with methods to export in various formats. [](#__codelineno-0-4) [](#__codelineno-0-5) Main Methods: [](#__codelineno-0-6) - from_url(url, timeout=10) [](#__codelineno-0-7) - from_file(file_path) [](#__codelineno-0-8) - from_binary(binary_data) [](#__codelineno-0-9) - to_json(filepath=None) [](#__codelineno-0-10) - to_pem(filepath=None) [](#__codelineno-0-11) - to_der(filepath=None) [](#__codelineno-0-12) ... [](#__codelineno-0-13) [](#__codelineno-0-14) Common Properties: [](#__codelineno-0-15) - issuer [](#__codelineno-0-16) - subject [](#__codelineno-0-17) - valid_from [](#__codelineno-0-18) - valid_until [](#__codelineno-0-19) - fingerprint [](#__codelineno-0-20) """` ### Typical Use Case 1. You **enable** certificate fetching in your crawl by: `[](#__codelineno-1-1) CrawlerRunConfig(fetch_ssl_certificate=True, ...)` 2. After `arun()`, if `result.ssl_certificate` is present, it’s an instance of **`SSLCertificate`**. 3. You can **read** basic properties (issuer, subject, validity) or **export** them in multiple formats. * * * 2\. Construction & Fetching --------------------------- ### 2.1 **`from_url(url, timeout=10)`** Manually load an SSL certificate from a given URL (port 443). Typically used internally, but you can call it directly if you want: `[](#__codelineno-2-1) cert = SSLCertificate.from_url("https://example.com") [](#__codelineno-2-2) if cert: [](#__codelineno-2-3) print("Fingerprint:", cert.fingerprint)` ### 2.2 **`from_file(file_path)`** Load from a file containing certificate data in ASN.1 or DER. Rarely needed unless you have local cert files: `[](#__codelineno-3-1) cert = SSLCertificate.from_file("/path/to/cert.der")` ### 2.3 **`from_binary(binary_data)`** Initialize from raw binary. E.g., if you captured it from a socket or another source: `[](#__codelineno-4-1) cert = SSLCertificate.from_binary(raw_bytes)` * * * 3\. Common Properties --------------------- After obtaining a **`SSLCertificate`** instance (e.g. `result.ssl_certificate` from a crawl), you can read: 1. **`issuer`** _(dict)_ \- E.g. `{"CN": "My Root CA", "O": "..."}` 2. **`subject`** _(dict)_ \- E.g. `{"CN": "example.com", "O": "ExampleOrg"}` 3. **`valid_from`** _(str)_ \- NotBefore date/time. Often in ASN.1/UTC format. 4. **`valid_until`** _(str)_ \- NotAfter date/time. 5. **`fingerprint`** _(str)_ \- The SHA-256 digest (lowercase hex). \- E.g. `"d14d2e..."` * * * 4\. Export Methods ------------------ Once you have a **`SSLCertificate`** object, you can **export** or **inspect** it: ### 4.1 **`to_json(filepath=None)` → `Optional[str]`** * Returns a JSON string containing the parsed certificate fields. * If `filepath` is provided, saves it to disk instead, returning `None`. **Usage**: `[](#__codelineno-5-1) json_data = cert.to_json() # returns JSON string [](#__codelineno-5-2) cert.to_json("certificate.json") # writes file, returns None` ### 4.2 **`to_pem(filepath=None)` → `Optional[str]`** * Returns a PEM-encoded string (common for web servers). * If `filepath` is provided, saves it to disk instead. `[](#__codelineno-6-1) pem_str = cert.to_pem() # in-memory PEM string [](#__codelineno-6-2) cert.to_pem("/path/to/cert.pem") # saved to file` ### 4.3 **`to_der(filepath=None)` → `Optional[bytes]`** * Returns the original DER (binary ASN.1) bytes. * If `filepath` is specified, writes the bytes there instead. `[](#__codelineno-7-1) der_bytes = cert.to_der() [](#__codelineno-7-2) cert.to_der("certificate.der")` ### 4.4 (Optional) **`export_as_text()`** * If you see a method like `export_as_text()`, it typically returns an OpenSSL-style textual representation. * Not always needed, but can help for debugging or manual inspection. * * * 5\. Example Usage in Crawl4AI ----------------------------- Below is a minimal sample showing how the crawler obtains an SSL cert from a site, then reads or exports it. The code snippet: `[](#__codelineno-8-1) import asyncio [](#__codelineno-8-2) import os [](#__codelineno-8-3) from crawl4ai import AsyncWebCrawler, CrawlerRunConfig, CacheMode [](#__codelineno-8-4) [](#__codelineno-8-5) async def main(): [](#__codelineno-8-6) tmp_dir = "tmp" [](#__codelineno-8-7) os.makedirs(tmp_dir, exist_ok=True) [](#__codelineno-8-8) [](#__codelineno-8-9) config = CrawlerRunConfig( [](#__codelineno-8-10) fetch_ssl_certificate=True, [](#__codelineno-8-11) cache_mode=CacheMode.BYPASS [](#__codelineno-8-12) ) [](#__codelineno-8-13) [](#__codelineno-8-14) async with AsyncWebCrawler() as crawler: [](#__codelineno-8-15) result = await crawler.arun("https://example.com", config=config) [](#__codelineno-8-16) if result.success and result.ssl_certificate: [](#__codelineno-8-17) cert = result.ssl_certificate [](#__codelineno-8-18) # 1. Basic Info [](#__codelineno-8-19) print("Issuer CN:", cert.issuer.get("CN", "")) [](#__codelineno-8-20) print("Valid until:", cert.valid_until) [](#__codelineno-8-21) print("Fingerprint:", cert.fingerprint) [](#__codelineno-8-22) [](#__codelineno-8-23) # 2. Export [](#__codelineno-8-24) cert.to_json(os.path.join(tmp_dir, "certificate.json")) [](#__codelineno-8-25) cert.to_pem(os.path.join(tmp_dir, "certificate.pem")) [](#__codelineno-8-26) cert.to_der(os.path.join(tmp_dir, "certificate.der")) [](#__codelineno-8-27) [](#__codelineno-8-28) if __name__ == "__main__": [](#__codelineno-8-29) asyncio.run(main())` * * * 6\. Notes & Best Practices -------------------------- 1. **Timeout**: `SSLCertificate.from_url` internally uses a default **10s** socket connect and wraps SSL. 2. **Binary Form**: The certificate is loaded in ASN.1 (DER) form, then re-parsed by `OpenSSL.crypto`. 3. **Validation**: This does **not** validate the certificate chain or trust store. It only fetches and parses. 4. **Integration**: Within Crawl4AI, you typically just set `fetch_ssl_certificate=True` in `CrawlerRunConfig`; the final result’s `ssl_certificate` is automatically built. 5. **Export**: If you need to store or analyze a cert, the `to_json` and `to_pem` are quite universal. * * * ### Summary * **`SSLCertificate`** is a convenience class for capturing and exporting the **TLS certificate** from your crawled site(s). * Common usage is in the **`CrawlResult.ssl_certificate`** field, accessible after setting `fetch_ssl_certificate=True`. * Offers quick access to essential certificate details (`issuer`, `subject`, `fingerprint`) and is easy to export (PEM, DER, JSON) for further analysis or server usage. Use it whenever you need **insight** into a site’s certificate or require some form of cryptographic or compliance check. * * * --- # Hooks & Auth - Crawl4AI Documentation Hooks & Auth in AsyncWebCrawler =============================== Crawl4AI’s **hooks** let you customize the crawler at specific points in the pipeline: 1. **`on_browser_created`** – After browser creation. 2. **`on_page_context_created`** – After a new context & page are created. 3. **`before_goto`** – Just before navigating to a page. 4. **`after_goto`** – Right after navigation completes. 5. **`on_user_agent_updated`** – Whenever the user agent changes. 6. **`on_execution_started`** – Once custom JavaScript execution begins. 7. **`before_retrieve_html`** – Just before the crawler retrieves final HTML. 8. **`before_return_html`** – Right before returning the HTML content. **Important**: Avoid heavy tasks in `on_browser_created` since you don’t yet have a page context. If you need to _log in_, do so in **`on_page_context_created`**. > note "Important Hook Usage Warning" **Avoid Misusing Hooks**: Do not manipulate page objects in the wrong hook or at the wrong time, as it can crash the pipeline or produce incorrect results. A common mistake is attempting to handle authentication prematurely—such as creating or closing pages in `on_browser_created`. > > **Use the Right Hook for Auth**: If you need to log in or set tokens, use `on_page_context_created`. This ensures you have a valid page/context to work with, without disrupting the main crawling flow. > > **Identity-Based Crawling**: For robust auth, consider identity-based crawling (or passing a session ID) to preserve state. Run your initial login steps in a separate, well-defined process, then feed that session to your main crawl—rather than shoehorning complex authentication into early hooks. Check out [Identity-Based Crawling](../identity-based-crawling/) > for more details. > > **Be Cautious**: Overwriting or removing elements in the wrong hook can compromise the final crawl. Keep hooks focused on smaller tasks (like route filters, custom headers), and let your main logic (crawling, data extraction) proceed normally. Below is an example demonstration. * * * Example: Using Hooks in AsyncWebCrawler --------------------------------------- `[](#__codelineno-0-1) import asyncio [](#__codelineno-0-2) import json [](#__codelineno-0-3) from crawl4ai import AsyncWebCrawler, BrowserConfig, CrawlerRunConfig, CacheMode [](#__codelineno-0-4) from playwright.async_api import Page, BrowserContext [](#__codelineno-0-5) [](#__codelineno-0-6) async def main(): [](#__codelineno-0-7) print("🔗 Hooks Example: Demonstrating recommended usage") [](#__codelineno-0-8) [](#__codelineno-0-9) # 1) Configure the browser [](#__codelineno-0-10) browser_config = BrowserConfig( [](#__codelineno-0-11) headless=True, [](#__codelineno-0-12) verbose=True [](#__codelineno-0-13) ) [](#__codelineno-0-14) [](#__codelineno-0-15) # 2) Configure the crawler run [](#__codelineno-0-16) crawler_run_config = CrawlerRunConfig( [](#__codelineno-0-17) js_code="window.scrollTo(0, document.body.scrollHeight);", [](#__codelineno-0-18) wait_for="body", [](#__codelineno-0-19) cache_mode=CacheMode.BYPASS [](#__codelineno-0-20) ) [](#__codelineno-0-21) [](#__codelineno-0-22) # 3) Create the crawler instance [](#__codelineno-0-23) crawler = AsyncWebCrawler(config=browser_config) [](#__codelineno-0-24) [](#__codelineno-0-25) # [](#__codelineno-0-26) # Define Hook Functions [](#__codelineno-0-27) # [](#__codelineno-0-28) [](#__codelineno-0-29) async def on_browser_created(browser, **kwargs): [](#__codelineno-0-30) # Called once the browser instance is created (but no pages or contexts yet) [](#__codelineno-0-31) print("[HOOK] on_browser_created - Browser created successfully!") [](#__codelineno-0-32) # Typically, do minimal setup here if needed [](#__codelineno-0-33) return browser [](#__codelineno-0-34) [](#__codelineno-0-35) async def on_page_context_created(page: Page, context: BrowserContext, **kwargs): [](#__codelineno-0-36) # Called right after a new page + context are created (ideal for auth or route config). [](#__codelineno-0-37) print("[HOOK] on_page_context_created - Setting up page & context.") [](#__codelineno-0-38) [](#__codelineno-0-39) # Example 1: Route filtering (e.g., block images) [](#__codelineno-0-40) async def route_filter(route): [](#__codelineno-0-41) if route.request.resource_type == "image": [](#__codelineno-0-42) print(f"[HOOK] Blocking image request: {route.request.url}") [](#__codelineno-0-43) await route.abort() [](#__codelineno-0-44) else: [](#__codelineno-0-45) await route.continue_() [](#__codelineno-0-46) [](#__codelineno-0-47) await context.route("**", route_filter) [](#__codelineno-0-48) [](#__codelineno-0-49) # Example 2: (Optional) Simulate a login scenario [](#__codelineno-0-50) # (We do NOT create or close pages here, just do quick steps if needed) [](#__codelineno-0-51) # e.g., await page.goto("https://example.com/login") [](#__codelineno-0-52) # e.g., await page.fill("input[name='username']", "testuser") [](#__codelineno-0-53) # e.g., await page.fill("input[name='password']", "password123") [](#__codelineno-0-54) # e.g., await page.click("button[type='submit']") [](#__codelineno-0-55) # e.g., await page.wait_for_selector("#welcome") [](#__codelineno-0-56) # e.g., await context.add_cookies([...]) [](#__codelineno-0-57) # Then continue [](#__codelineno-0-58) [](#__codelineno-0-59) # Example 3: Adjust the viewport [](#__codelineno-0-60) await page.set_viewport_size({"width": 1080, "height": 600}) [](#__codelineno-0-61) return page [](#__codelineno-0-62) [](#__codelineno-0-63) async def before_goto( [](#__codelineno-0-64) page: Page, context: BrowserContext, url: str, **kwargs [](#__codelineno-0-65) ): [](#__codelineno-0-66) # Called before navigating to each URL. [](#__codelineno-0-67) print(f"[HOOK] before_goto - About to navigate: {url}") [](#__codelineno-0-68) # e.g., inject custom headers [](#__codelineno-0-69) await page.set_extra_http_headers({ [](#__codelineno-0-70) "Custom-Header": "my-value" [](#__codelineno-0-71) }) [](#__codelineno-0-72) return page [](#__codelineno-0-73) [](#__codelineno-0-74) async def after_goto( [](#__codelineno-0-75) page: Page, context: BrowserContext, [](#__codelineno-0-76) url: str, response, **kwargs [](#__codelineno-0-77) ): [](#__codelineno-0-78) # Called after navigation completes. [](#__codelineno-0-79) print(f"[HOOK] after_goto - Successfully loaded: {url}") [](#__codelineno-0-80) # e.g., wait for a certain element if we want to verify [](#__codelineno-0-81) try: [](#__codelineno-0-82) await page.wait_for_selector('.content', timeout=1000) [](#__codelineno-0-83) print("[HOOK] Found .content element!") [](#__codelineno-0-84) except: [](#__codelineno-0-85) print("[HOOK] .content not found, continuing anyway.") [](#__codelineno-0-86) return page [](#__codelineno-0-87) [](#__codelineno-0-88) async def on_user_agent_updated( [](#__codelineno-0-89) page: Page, context: BrowserContext, [](#__codelineno-0-90) user_agent: str, **kwargs [](#__codelineno-0-91) ): [](#__codelineno-0-92) # Called whenever the user agent updates. [](#__codelineno-0-93) print(f"[HOOK] on_user_agent_updated - New user agent: {user_agent}") [](#__codelineno-0-94) return page [](#__codelineno-0-95) [](#__codelineno-0-96) async def on_execution_started(page: Page, context: BrowserContext, **kwargs): [](#__codelineno-0-97) # Called after custom JavaScript execution begins. [](#__codelineno-0-98) print("[HOOK] on_execution_started - JS code is running!") [](#__codelineno-0-99) return page [](#__codelineno-0-100) [](#__codelineno-0-101) async def before_retrieve_html(page: Page, context: BrowserContext, **kwargs): [](#__codelineno-0-102) # Called before final HTML retrieval. [](#__codelineno-0-103) print("[HOOK] before_retrieve_html - We can do final actions") [](#__codelineno-0-104) # Example: Scroll again [](#__codelineno-0-105) await page.evaluate("window.scrollTo(0, document.body.scrollHeight);") [](#__codelineno-0-106) return page [](#__codelineno-0-107) [](#__codelineno-0-108) async def before_return_html( [](#__codelineno-0-109) page: Page, context: BrowserContext, html: str, **kwargs [](#__codelineno-0-110) ): [](#__codelineno-0-111) # Called just before returning the HTML in the result. [](#__codelineno-0-112) print(f"[HOOK] before_return_html - HTML length: {len(html)}") [](#__codelineno-0-113) return page [](#__codelineno-0-114) [](#__codelineno-0-115) # [](#__codelineno-0-116) # Attach Hooks [](#__codelineno-0-117) # [](#__codelineno-0-118) [](#__codelineno-0-119) crawler.crawler_strategy.set_hook("on_browser_created", on_browser_created) [](#__codelineno-0-120) crawler.crawler_strategy.set_hook( [](#__codelineno-0-121) "on_page_context_created", on_page_context_created [](#__codelineno-0-122) ) [](#__codelineno-0-123) crawler.crawler_strategy.set_hook("before_goto", before_goto) [](#__codelineno-0-124) crawler.crawler_strategy.set_hook("after_goto", after_goto) [](#__codelineno-0-125) crawler.crawler_strategy.set_hook( [](#__codelineno-0-126) "on_user_agent_updated", on_user_agent_updated [](#__codelineno-0-127) ) [](#__codelineno-0-128) crawler.crawler_strategy.set_hook( [](#__codelineno-0-129) "on_execution_started", on_execution_started [](#__codelineno-0-130) ) [](#__codelineno-0-131) crawler.crawler_strategy.set_hook( [](#__codelineno-0-132) "before_retrieve_html", before_retrieve_html [](#__codelineno-0-133) ) [](#__codelineno-0-134) crawler.crawler_strategy.set_hook( [](#__codelineno-0-135) "before_return_html", before_return_html [](#__codelineno-0-136) ) [](#__codelineno-0-137) [](#__codelineno-0-138) await crawler.start() [](#__codelineno-0-139) [](#__codelineno-0-140) # 4) Run the crawler on an example page [](#__codelineno-0-141) url = "https://example.com" [](#__codelineno-0-142) result = await crawler.arun(url, config=crawler_run_config) [](#__codelineno-0-143) [](#__codelineno-0-144) if result.success: [](#__codelineno-0-145) print("\nCrawled URL:", result.url) [](#__codelineno-0-146) print("HTML length:", len(result.html)) [](#__codelineno-0-147) else: [](#__codelineno-0-148) print("Error:", result.error_message) [](#__codelineno-0-149) [](#__codelineno-0-150) await crawler.close() [](#__codelineno-0-151) [](#__codelineno-0-152) if __name__ == "__main__": [](#__codelineno-0-153) asyncio.run(main())` * * * Hook Lifecycle Summary ---------------------- 1. **`on_browser_created`**: \- Browser is up, but **no** pages or contexts yet. \- Light setup only—don’t try to open or close pages here (that belongs in `on_page_context_created`). 2. **`on_page_context_created`**: \- Perfect for advanced **auth** or route blocking. \- You have a **page** + **context** ready but haven’t navigated to the target URL yet. 3. **`before_goto`**: \- Right before navigation. Typically used for setting **custom headers** or logging the target URL. 4. **`after_goto`**: \- After page navigation is done. Good place for verifying content or waiting on essential elements. 5. **`on_user_agent_updated`**: \- Whenever the user agent changes (for stealth or different UA modes). 6. **`on_execution_started`**: \- If you set `js_code` or run custom scripts, this runs once your JS is about to start. 7. **`before_retrieve_html`**: \- Just before the final HTML snapshot is taken. Often you do a final scroll or lazy-load triggers here. 8. **`before_return_html`**: \- The last hook before returning HTML to the `CrawlResult`. Good for logging HTML length or minor modifications. * * * When to Handle Authentication ----------------------------- **Recommended**: Use **`on_page_context_created`** if you need to: * Navigate to a login page or fill forms * Set cookies or localStorage tokens * Block resource routes to avoid ads This ensures the newly created context is under your control **before** `arun()` navigates to the main URL. * * * Additional Considerations ------------------------- * **Session Management**: If you want multiple `arun()` calls to reuse a single session, pass `session_id=` in your `CrawlerRunConfig`. Hooks remain the same. * **Performance**: Hooks can slow down crawling if they do heavy tasks. Keep them concise. * **Error Handling**: If a hook fails, the overall crawl might fail. Catch exceptions or handle them gracefully. * **Concurrency**: If you run `arun_many()`, each URL triggers these hooks in parallel. Ensure your hooks are thread/async-safe. * * * Conclusion ---------- Hooks provide **fine-grained** control over: * **Browser** creation (light tasks only) * **Page** and **context** creation (auth, route blocking) * **Navigation** phases * **Final HTML** retrieval Follow the recommended usage: - **Login** or advanced tasks in `on_page_context_created` \- **Custom headers** or logs in `before_goto` / `after_goto` \- **Scrolling** or final checks in `before_retrieve_html` / `before_return_html` * * * --- # Multi-URL Crawling - Crawl4AI Documentation Optimized Multi-URL Crawling ============================ > **Note**: We’re developing a new **executor module** that uses a sophisticated algorithm to dynamically manage multi-URL crawling, optimizing for speed and memory usage. The approaches in this document remain fully valid, but keep an eye on **Crawl4AI**’s upcoming releases for this powerful feature! Follow [@unclecode](https://twitter.com/unclecode) > on X and check the changelogs to stay updated. Crawl4AI’s **AsyncWebCrawler** can handle multiple URLs in a single run, which can greatly reduce overhead and speed up crawling. This guide shows how to: 1. **Sequentially** crawl a list of URLs using the **same** session, avoiding repeated browser creation. 2. **Parallel**\-crawl subsets of URLs in batches, again reusing the same browser. When the entire process finishes, you close the browser once—**minimizing** memory and resource usage. * * * 1\. Why Avoid Simple Loops per URL? ----------------------------------- If you naively do: `[](#__codelineno-0-1) for url in urls: [](#__codelineno-0-2) async with AsyncWebCrawler() as crawler: [](#__codelineno-0-3) result = await crawler.arun(url)` You end up: 1. Spinning up a **new** browser for each URL 2. Closing it immediately after the single crawl 3. Potentially using a lot of CPU/memory for short-living browsers 4. Missing out on session reusability if you have login or ongoing states **Better** approaches ensure you **create** the browser once, then crawl multiple URLs with minimal overhead. * * * 2\. Sequential Crawling with Session Reuse ------------------------------------------ ### 2.1 Overview 1. **One** `AsyncWebCrawler` instance for **all** URLs. 2. **One** session (via `session_id`) so we can preserve local storage or cookies across URLs if needed. 3\. The crawler is only closed at the **end**. **This** is the simplest pattern if your workload is moderate (dozens to a few hundred URLs). ### 2.2 Example Code `[](#__codelineno-1-1) import asyncio [](#__codelineno-1-2) from typing import List [](#__codelineno-1-3) from crawl4ai import AsyncWebCrawler, BrowserConfig, CrawlerRunConfig [](#__codelineno-1-4) from crawl4ai.markdown_generation_strategy import DefaultMarkdownGenerator [](#__codelineno-1-5) [](#__codelineno-1-6) async def crawl_sequential(urls: List[str]): [](#__codelineno-1-7) print("\n=== Sequential Crawling with Session Reuse ===") [](#__codelineno-1-8) [](#__codelineno-1-9) browser_config = BrowserConfig( [](#__codelineno-1-10) headless=True, [](#__codelineno-1-11) # For better performance in Docker or low-memory environments: [](#__codelineno-1-12) extra_args=["--disable-gpu", "--disable-dev-shm-usage", "--no-sandbox"], [](#__codelineno-1-13) ) [](#__codelineno-1-14) [](#__codelineno-1-15) crawl_config = CrawlerRunConfig( [](#__codelineno-1-16) markdown_generator=DefaultMarkdownGenerator() [](#__codelineno-1-17) ) [](#__codelineno-1-18) [](#__codelineno-1-19) # Create the crawler (opens the browser) [](#__codelineno-1-20) crawler = AsyncWebCrawler(config=browser_config) [](#__codelineno-1-21) await crawler.start() [](#__codelineno-1-22) [](#__codelineno-1-23) try: [](#__codelineno-1-24) session_id = "session1" # Reuse the same session across all URLs [](#__codelineno-1-25) for url in urls: [](#__codelineno-1-26) result = await crawler.arun( [](#__codelineno-1-27) url=url, [](#__codelineno-1-28) config=crawl_config, [](#__codelineno-1-29) session_id=session_id [](#__codelineno-1-30) ) [](#__codelineno-1-31) if result.success: [](#__codelineno-1-32) print(f"Successfully crawled: {url}") [](#__codelineno-1-33) # E.g. check markdown length [](#__codelineno-1-34) print(f"Markdown length: {len(result.markdown_v2.raw_markdown)}") [](#__codelineno-1-35) else: [](#__codelineno-1-36) print(f"Failed: {url} - Error: {result.error_message}") [](#__codelineno-1-37) finally: [](#__codelineno-1-38) # After all URLs are done, close the crawler (and the browser) [](#__codelineno-1-39) await crawler.close() [](#__codelineno-1-40) [](#__codelineno-1-41) async def main(): [](#__codelineno-1-42) urls = [ [](#__codelineno-1-43) "https://example.com/page1", [](#__codelineno-1-44) "https://example.com/page2", [](#__codelineno-1-45) "https://example.com/page3" [](#__codelineno-1-46) ] [](#__codelineno-1-47) await crawl_sequential(urls) [](#__codelineno-1-48) [](#__codelineno-1-49) if __name__ == "__main__": [](#__codelineno-1-50) asyncio.run(main())` **Why It’s Good**: * **One** browser launch. * Minimal memory usage. * If the site requires login, you can log in once in `session_id` context and preserve auth across all URLs. * * * 3\. Parallel Crawling with Browser Reuse ---------------------------------------- ### 3.1 Overview To speed up crawling further, you can crawl multiple URLs in **parallel** (batches or a concurrency limit). The crawler still uses **one** browser, but spawns different sessions (or the same, depending on your logic) for each task. ### 3.2 Example Code For this example make sure to install the [psutil](https://pypi.org/project/psutil/) package. `[](#__codelineno-2-1) pip install psutil` Then you can run the following code: `[](#__codelineno-3-1) import os [](#__codelineno-3-2) import sys [](#__codelineno-3-3) import psutil [](#__codelineno-3-4) import asyncio [](#__codelineno-3-5) [](#__codelineno-3-6) __location__ = os.path.dirname(os.path.abspath(__file__)) [](#__codelineno-3-7) __output__ = os.path.join(__location__, "output") [](#__codelineno-3-8) [](#__codelineno-3-9) # Append parent directory to system path [](#__codelineno-3-10) parent_dir = os.path.dirname(os.path.dirname(os.path.abspath(__file__))) [](#__codelineno-3-11) sys.path.append(parent_dir) [](#__codelineno-3-12) [](#__codelineno-3-13) from typing import List [](#__codelineno-3-14) from crawl4ai import AsyncWebCrawler, BrowserConfig, CrawlerRunConfig, CacheMode [](#__codelineno-3-15) [](#__codelineno-3-16) async def crawl_parallel(urls: List[str], max_concurrent: int = 3): [](#__codelineno-3-17) print("\n=== Parallel Crawling with Browser Reuse + Memory Check ===") [](#__codelineno-3-18) [](#__codelineno-3-19) # We'll keep track of peak memory usage across all tasks [](#__codelineno-3-20) peak_memory = 0 [](#__codelineno-3-21) process = psutil.Process(os.getpid()) [](#__codelineno-3-22) [](#__codelineno-3-23) def log_memory(prefix: str = ""): [](#__codelineno-3-24) nonlocal peak_memory [](#__codelineno-3-25) current_mem = process.memory_info().rss # in bytes [](#__codelineno-3-26) if current_mem > peak_memory: [](#__codelineno-3-27) peak_memory = current_mem [](#__codelineno-3-28) print(f"{prefix} Current Memory: {current_mem // (1024 * 1024)} MB, Peak: {peak_memory // (1024 * 1024)} MB") [](#__codelineno-3-29) [](#__codelineno-3-30) # Minimal browser config [](#__codelineno-3-31) browser_config = BrowserConfig( [](#__codelineno-3-32) headless=True, [](#__codelineno-3-33) verbose=False, # corrected from 'verbos=False' [](#__codelineno-3-34) extra_args=["--disable-gpu", "--disable-dev-shm-usage", "--no-sandbox"], [](#__codelineno-3-35) ) [](#__codelineno-3-36) crawl_config = CrawlerRunConfig(cache_mode=CacheMode.BYPASS) [](#__codelineno-3-37) [](#__codelineno-3-38) # Create the crawler instance [](#__codelineno-3-39) crawler = AsyncWebCrawler(config=browser_config) [](#__codelineno-3-40) await crawler.start() [](#__codelineno-3-41) [](#__codelineno-3-42) try: [](#__codelineno-3-43) # We'll chunk the URLs in batches of 'max_concurrent' [](#__codelineno-3-44) success_count = 0 [](#__codelineno-3-45) fail_count = 0 [](#__codelineno-3-46) for i in range(0, len(urls), max_concurrent): [](#__codelineno-3-47) batch = urls[i : i + max_concurrent] [](#__codelineno-3-48) tasks = [] [](#__codelineno-3-49) [](#__codelineno-3-50) for j, url in enumerate(batch): [](#__codelineno-3-51) # Unique session_id per concurrent sub-task [](#__codelineno-3-52) session_id = f"parallel_session_{i + j}" [](#__codelineno-3-53) task = crawler.arun(url=url, config=crawl_config, session_id=session_id) [](#__codelineno-3-54) tasks.append(task) [](#__codelineno-3-55) [](#__codelineno-3-56) # Check memory usage prior to launching tasks [](#__codelineno-3-57) log_memory(prefix=f"Before batch {i//max_concurrent + 1}: ") [](#__codelineno-3-58) [](#__codelineno-3-59) # Gather results [](#__codelineno-3-60) results = await asyncio.gather(*tasks, return_exceptions=True) [](#__codelineno-3-61) [](#__codelineno-3-62) # Check memory usage after tasks complete [](#__codelineno-3-63) log_memory(prefix=f"After batch {i//max_concurrent + 1}: ") [](#__codelineno-3-64) [](#__codelineno-3-65) # Evaluate results [](#__codelineno-3-66) for url, result in zip(batch, results): [](#__codelineno-3-67) if isinstance(result, Exception): [](#__codelineno-3-68) print(f"Error crawling {url}: {result}") [](#__codelineno-3-69) fail_count += 1 [](#__codelineno-3-70) elif result.success: [](#__codelineno-3-71) success_count += 1 [](#__codelineno-3-72) else: [](#__codelineno-3-73) fail_count += 1 [](#__codelineno-3-74) [](#__codelineno-3-75) print(f"\nSummary:") [](#__codelineno-3-76) print(f" - Successfully crawled: {success_count}") [](#__codelineno-3-77) print(f" - Failed: {fail_count}") [](#__codelineno-3-78) [](#__codelineno-3-79) finally: [](#__codelineno-3-80) print("\nClosing crawler...") [](#__codelineno-3-81) await crawler.close() [](#__codelineno-3-82) # Final memory log [](#__codelineno-3-83) log_memory(prefix="Final: ") [](#__codelineno-3-84) print(f"\nPeak memory usage (MB): {peak_memory // (1024 * 1024)}") [](#__codelineno-3-85) [](#__codelineno-3-86) async def main(): [](#__codelineno-3-87) urls = [ [](#__codelineno-3-88) "https://example.com/page1", [](#__codelineno-3-89) "https://example.com/page2", [](#__codelineno-3-90) "https://example.com/page3", [](#__codelineno-3-91) "https://example.com/page4" [](#__codelineno-3-92) ] [](#__codelineno-3-93) await crawl_parallel(urls, max_concurrent=2) [](#__codelineno-3-94) [](#__codelineno-3-95) if __name__ == "__main__": [](#__codelineno-3-96) asyncio.run(main())` **Notes**: * We **reuse** the same `AsyncWebCrawler` instance for all parallel tasks, launching **one** browser. * Each parallel sub-task might get its own `session_id` so they don’t share cookies/localStorage (unless that’s desired). * We limit concurrency to `max_concurrent=2` or 3 to avoid saturating CPU/memory. * * * 4\. Performance Tips -------------------- 1. **Extra Browser Args** \- `--disable-gpu`, `--no-sandbox` can help in Docker or restricted environments. \- `--disable-dev-shm-usage` avoids using `/dev/shm` which can be small on some systems. 2. **Session Reuse** \- If your site requires a login or you want to maintain local data across URLs, share the **same** `session_id`. \- If you want isolation (each URL fresh), create unique sessions. 3. **Batching** \- If you have **many** URLs (like thousands), you can do parallel crawling in chunks (like `max_concurrent=5`). \- Use `arun_many()` for a built-in approach if you prefer, but the example above is often more flexible. 4. **Cache** \- If your pages share many resources or you’re re-crawling the same domain repeatedly, consider setting `cache_mode=CacheMode.ENABLED` in `CrawlerRunConfig`. \- If you need fresh data each time, keep `cache_mode=CacheMode.BYPASS`. 5. **Hooks** \- You can set up global hooks for each crawler (like to block images) or per-run if you want. \- Keep them consistent if you’re reusing sessions. * * * 5\. Summary ----------- * **One** `AsyncWebCrawler` + multiple calls to `.arun()` is far more efficient than launching a new crawler per URL. * **Sequential** approach with a shared session is simple and memory-friendly for moderate sets of URLs. * **Parallel** approach can speed up large crawls by concurrency, but keep concurrency balanced to avoid overhead. * Close the crawler once at the end, ensuring the browser is only opened/closed once. For even more advanced memory optimizations or dynamic concurrency patterns, see future sections on hooking or distributed crawling. The patterns above suffice for the majority of multi-URL scenarios—**giving you speed, simplicity, and minimal resource usage**. Enjoy your optimized crawling! * * * --- # Lazy Loading - Crawl4AI Documentation Handling Lazy-Loaded Images --------------------------- Many websites now load images **lazily** as you scroll. If you need to ensure they appear in your final crawl (and in `result.media`), consider: 1. **`wait_for_images=True`** – Wait for images to fully load. 2. **`scan_full_page`** – Force the crawler to scroll the entire page, triggering lazy loads. 3. **`scroll_delay`** – Add small delays between scroll steps. **Note**: If the site requires multiple “Load More” triggers or complex interactions, see the [Page Interaction docs](../../core/page-interaction/) . ### Example: Ensuring Lazy Images Appear `[](#__codelineno-0-1) import asyncio [](#__codelineno-0-2) from crawl4ai import AsyncWebCrawler, CrawlerRunConfig, BrowserConfig [](#__codelineno-0-3) from crawl4ai.async_configs import CacheMode [](#__codelineno-0-4) [](#__codelineno-0-5) async def main(): [](#__codelineno-0-6) config = CrawlerRunConfig( [](#__codelineno-0-7) # Force the crawler to wait until images are fully loaded [](#__codelineno-0-8) wait_for_images=True, [](#__codelineno-0-9) [](#__codelineno-0-10) # Option 1: If you want to automatically scroll the page to load images [](#__codelineno-0-11) scan_full_page=True, # Tells the crawler to try scrolling the entire page [](#__codelineno-0-12) scroll_delay=0.5, # Delay (seconds) between scroll steps [](#__codelineno-0-13) [](#__codelineno-0-14) # Option 2: If the site uses a 'Load More' or JS triggers for images, [](#__codelineno-0-15) # you can also specify js_code or wait_for logic here. [](#__codelineno-0-16) [](#__codelineno-0-17) cache_mode=CacheMode.BYPASS, [](#__codelineno-0-18) verbose=True [](#__codelineno-0-19) ) [](#__codelineno-0-20) [](#__codelineno-0-21) async with AsyncWebCrawler(config=BrowserConfig(headless=True)) as crawler: [](#__codelineno-0-22) result = await crawler.arun("https://www.example.com/gallery", config=config) [](#__codelineno-0-23) [](#__codelineno-0-24) if result.success: [](#__codelineno-0-25) images = result.media.get("images", []) [](#__codelineno-0-26) print("Images found:", len(images)) [](#__codelineno-0-27) for i, img in enumerate(images[:5]): [](#__codelineno-0-28) print(f"[Image {i}] URL: {img['src']}, Score: {img.get('score','N/A')}") [](#__codelineno-0-29) else: [](#__codelineno-0-30) print("Error:", result.error_message) [](#__codelineno-0-31) [](#__codelineno-0-32) if __name__ == "__main__": [](#__codelineno-0-33) asyncio.run(main())` **Explanation**: * **`wait_for_images=True`** The crawler tries to ensure images have finished loading before finalizing the HTML. * **`scan_full_page=True`** Tells the crawler to attempt scrolling from top to bottom. Each scroll step helps trigger lazy loading. * **`scroll_delay=0.5`** Pause half a second between each scroll step. Helps the site load images before continuing. **When to Use**: * **Lazy-Loading**: If images appear only when the user scrolls into view, `scan_full_page` + `scroll_delay` helps the crawler see them. * **Heavier Pages**: If a page is extremely long, be mindful that scanning the entire page can be slow. Adjust `scroll_delay` or the max scroll steps as needed. * * * Combining with Other Link & Media Filters ----------------------------------------- You can still combine **lazy-load** logic with the usual **exclude\_external\_images**, **exclude\_domains**, or link filtration: `[](#__codelineno-1-1) config = CrawlerRunConfig( [](#__codelineno-1-2) wait_for_images=True, [](#__codelineno-1-3) scan_full_page=True, [](#__codelineno-1-4) scroll_delay=0.5, [](#__codelineno-1-5) [](#__codelineno-1-6) # Filter out external images if you only want local ones [](#__codelineno-1-7) exclude_external_images=True, [](#__codelineno-1-8) [](#__codelineno-1-9) # Exclude certain domains for links [](#__codelineno-1-10) exclude_domains=["spammycdn.com"], [](#__codelineno-1-11) )` This approach ensures you see **all** images from the main domain while ignoring external ones, and the crawler physically scrolls the entire page so that lazy-loading triggers. * * * Tips & Troubleshooting ---------------------- 1. **Long Pages** \- Setting `scan_full_page=True` on extremely long or infinite-scroll pages can be resource-intensive. \- Consider using [hooks](../../core/page-interaction/) or specialized logic to load specific sections or “Load More” triggers repeatedly. 2. **Mixed Image Behavior** \- Some sites load images in batches as you scroll. If you’re missing images, increase your `scroll_delay` or call multiple partial scrolls in a loop with JS code or hooks. 3. **Combining with Dynamic Wait** \- If the site has a placeholder that only changes to a real image after a certain event, you might do `wait_for="css:img.loaded"` or a custom JS `wait_for`. 4. **Caching** \- If `cache_mode` is enabled, repeated crawls might skip some network fetches. If you suspect caching is missing new images, set `cache_mode=CacheMode.BYPASS` for fresh fetches. * * * With **lazy-loading** support, **wait\_for\_images**, and **scan\_full\_page** settings, you can capture the entire gallery or feed of images you expect—even if the site only loads them as the user scrolls. Combine these with the standard media filtering and domain exclusion for a complete link & media handling strategy. * * * --- # Dockerize hooks - Crawl4AI Documentation Introducing Event Streams and Interactive Hooks in Crawl4AI ----------------------------------------------------------- ![event-driven-crawl](https://res.cloudinary.com/kidocode/image/upload/t_400x400/v1734344008/15bb8bbb-83ac-43ac-962d-3feb3e0c3bbf_2_tjmr4n.webp) In the near future, I’m planning to enhance Crawl4AI’s capabilities by introducing an event stream mechanism that will give clients deeper, real-time insights into the crawling process. Today, hooks are a powerful feature at the code level—they let developers define custom logic at key points in the crawl. However, when using Crawl4AI as a service (e.g., through a Dockerized API), there isn’t an easy way to interact with these hooks at runtime. **What’s Changing?** I’m working on a solution that will allow the crawler to emit a continuous stream of events, updating clients on the current crawling stage, encountered pages, and any decision points. This event stream could be exposed over a standardized protocol like Server-Sent Events (SSE) or WebSockets, enabling clients to “subscribe” and listen as the crawler works. **Interactivity Through Process IDs** A key part of this new design is the concept of a unique process ID for each crawl session. Imagine you’re listening to an event stream that informs you: - The crawler just hit a certain page \- It triggered a hook and is now pausing for instructions With the event stream in place, you can send a follow-up request back to the server—referencing the unique process ID—to provide extra data, instructions, or parameters. This might include selecting which links to follow next, adjusting extraction strategies, or providing authentication tokens for a protected API. Once the crawler receives these instructions, it resumes execution with the updated context. `[](#__codelineno-0-1) sequenceDiagram [](#__codelineno-0-2) participant Client [](#__codelineno-0-3) participant Server [](#__codelineno-0-4) participant Crawler [](#__codelineno-0-5) [](#__codelineno-0-6) Client->>Server: Start crawl request [](#__codelineno-0-7) Server->>Crawler: Initiate crawl with Process ID [](#__codelineno-0-8) Crawler-->>Server: Event: Page hit [](#__codelineno-0-9) Server-->>Client: Stream: Page hit event [](#__codelineno-0-10) Client->>Server: Instruction for Process ID [](#__codelineno-0-11) Server->>Crawler: Update crawl with new instructions [](#__codelineno-0-12) Crawler-->>Server: Event: Crawl completed [](#__codelineno-0-13) Server-->>Client: Stream: Crawl completed` **Benefits for Developers and Users** 1. **Fine-Grained Control**: Instead of predefining all logic upfront, you can dynamically guide the crawler in response to actual data and conditions encountered mid-crawl. 2. **Real-Time Insights**: Monitor progress, errors, or network bottlenecks as they happen, without waiting for the entire crawl to finish. 3. **Enhanced Collaboration**: Different team members or automated systems can watch the same crawl events and provide input, making the crawling process more adaptive and intelligent. **Next Steps** I’m currently exploring the best APIs, technologies, and patterns to make this vision a reality. My goal is to deliver a seamless developer experience—one that integrates with existing Crawl4AI workflows while offering new flexibility and power. Stay tuned for more updates as I continue building this feature out. In the meantime, I’d love to hear any feedback or suggestions you might have to help shape this interactive, event-driven future of web crawling with Crawl4AI. * * * --- # Session Management - Crawl4AI Documentation Session Management ================== Session management in Crawl4AI is a powerful feature that allows you to maintain state across multiple requests, making it particularly suitable for handling complex multi-step crawling tasks. It enables you to reuse the same browser tab (or page object) across sequential actions and crawls, which is beneficial for: * **Performing JavaScript actions before and after crawling.** * **Executing multiple sequential crawls faster** without needing to reopen tabs or allocate memory repeatedly. **Note:** This feature is designed for sequential workflows and is not suitable for parallel operations. * * * #### Basic Session Usage Use `BrowserConfig` and `CrawlerRunConfig` to maintain state with a `session_id`: `[](#__codelineno-0-1) from crawl4ai.async_configs import BrowserConfig, CrawlerRunConfig [](#__codelineno-0-2) [](#__codelineno-0-3) async with AsyncWebCrawler() as crawler: [](#__codelineno-0-4) session_id = "my_session" [](#__codelineno-0-5) [](#__codelineno-0-6) # Define configurations [](#__codelineno-0-7) config1 = CrawlerRunConfig( [](#__codelineno-0-8) url="https://example.com/page1", session_id=session_id [](#__codelineno-0-9) ) [](#__codelineno-0-10) config2 = CrawlerRunConfig( [](#__codelineno-0-11) url="https://example.com/page2", session_id=session_id [](#__codelineno-0-12) ) [](#__codelineno-0-13) [](#__codelineno-0-14) # First request [](#__codelineno-0-15) result1 = await crawler.arun(config=config1) [](#__codelineno-0-16) [](#__codelineno-0-17) # Subsequent request using the same session [](#__codelineno-0-18) result2 = await crawler.arun(config=config2) [](#__codelineno-0-19) [](#__codelineno-0-20) # Clean up when done [](#__codelineno-0-21) await crawler.crawler_strategy.kill_session(session_id)` * * * #### Dynamic Content with Sessions Here's an example of crawling GitHub commits across multiple pages while preserving session state: `[](#__codelineno-1-1) from crawl4ai.async_configs import CrawlerRunConfig [](#__codelineno-1-2) from crawl4ai.extraction_strategy import JsonCssExtractionStrategy [](#__codelineno-1-3) from crawl4ai.cache_context import CacheMode [](#__codelineno-1-4) [](#__codelineno-1-5) async def crawl_dynamic_content(): [](#__codelineno-1-6) async with AsyncWebCrawler() as crawler: [](#__codelineno-1-7) session_id = "github_commits_session" [](#__codelineno-1-8) url = "https://github.com/microsoft/TypeScript/commits/main" [](#__codelineno-1-9) all_commits = [] [](#__codelineno-1-10) [](#__codelineno-1-11) # Define extraction schema [](#__codelineno-1-12) schema = { [](#__codelineno-1-13) "name": "Commit Extractor", [](#__codelineno-1-14) "baseSelector": "li.Box-sc-g0xbh4-0", [](#__codelineno-1-15) "fields": [{ [](#__codelineno-1-16) "name": "title", "selector": "h4.markdown-title", "type": "text" [](#__codelineno-1-17) }], [](#__codelineno-1-18) } [](#__codelineno-1-19) extraction_strategy = JsonCssExtractionStrategy(schema) [](#__codelineno-1-20) [](#__codelineno-1-21) # JavaScript and wait configurations [](#__codelineno-1-22) js_next_page = """document.querySelector('a[data-testid="pagination-next-button"]').click();""" [](#__codelineno-1-23) wait_for = """() => document.querySelectorAll('li.Box-sc-g0xbh4-0').length > 0""" [](#__codelineno-1-24) [](#__codelineno-1-25) # Crawl multiple pages [](#__codelineno-1-26) for page in range(3): [](#__codelineno-1-27) config = CrawlerRunConfig( [](#__codelineno-1-28) url=url, [](#__codelineno-1-29) session_id=session_id, [](#__codelineno-1-30) extraction_strategy=extraction_strategy, [](#__codelineno-1-31) js_code=js_next_page if page > 0 else None, [](#__codelineno-1-32) wait_for=wait_for if page > 0 else None, [](#__codelineno-1-33) js_only=page > 0, [](#__codelineno-1-34) cache_mode=CacheMode.BYPASS [](#__codelineno-1-35) ) [](#__codelineno-1-36) [](#__codelineno-1-37) result = await crawler.arun(config=config) [](#__codelineno-1-38) if result.success: [](#__codelineno-1-39) commits = json.loads(result.extracted_content) [](#__codelineno-1-40) all_commits.extend(commits) [](#__codelineno-1-41) print(f"Page {page + 1}: Found {len(commits)} commits") [](#__codelineno-1-42) [](#__codelineno-1-43) # Clean up session [](#__codelineno-1-44) await crawler.crawler_strategy.kill_session(session_id) [](#__codelineno-1-45) return all_commits` * * * Example 1: Basic Session-Based Crawling --------------------------------------- A simple example using session-based crawling: `[](#__codelineno-2-1) import asyncio [](#__codelineno-2-2) from crawl4ai.async_configs import BrowserConfig, CrawlerRunConfig [](#__codelineno-2-3) from crawl4ai.cache_context import CacheMode [](#__codelineno-2-4) [](#__codelineno-2-5) async def basic_session_crawl(): [](#__codelineno-2-6) async with AsyncWebCrawler() as crawler: [](#__codelineno-2-7) session_id = "dynamic_content_session" [](#__codelineno-2-8) url = "https://example.com/dynamic-content" [](#__codelineno-2-9) [](#__codelineno-2-10) for page in range(3): [](#__codelineno-2-11) config = CrawlerRunConfig( [](#__codelineno-2-12) url=url, [](#__codelineno-2-13) session_id=session_id, [](#__codelineno-2-14) js_code="document.querySelector('.load-more-button').click();" if page > 0 else None, [](#__codelineno-2-15) css_selector=".content-item", [](#__codelineno-2-16) cache_mode=CacheMode.BYPASS [](#__codelineno-2-17) ) [](#__codelineno-2-18) [](#__codelineno-2-19) result = await crawler.arun(config=config) [](#__codelineno-2-20) print(f"Page {page + 1}: Found {result.extracted_content.count('.content-item')} items") [](#__codelineno-2-21) [](#__codelineno-2-22) await crawler.crawler_strategy.kill_session(session_id) [](#__codelineno-2-23) [](#__codelineno-2-24) asyncio.run(basic_session_crawl())` This example shows: 1. Reusing the same `session_id` across multiple requests. 2. Executing JavaScript to load more content dynamically. 3. Properly closing the session to free resources. * * * Advanced Technique 1: Custom Execution Hooks -------------------------------------------- > Warning: You might feel confused by the end of the next few examples 😅, so make sure you are comfortable with the order of the parts before you start this. Use custom hooks to handle complex scenarios, such as waiting for content to load dynamically: `[](#__codelineno-3-1) async def advanced_session_crawl_with_hooks(): [](#__codelineno-3-2) first_commit = "" [](#__codelineno-3-3) [](#__codelineno-3-4) async def on_execution_started(page): [](#__codelineno-3-5) nonlocal first_commit [](#__codelineno-3-6) try: [](#__codelineno-3-7) while True: [](#__codelineno-3-8) await page.wait_for_selector("li.commit-item h4") [](#__codelineno-3-9) commit = await page.query_selector("li.commit-item h4") [](#__codelineno-3-10) commit = await commit.evaluate("(element) => element.textContent").strip() [](#__codelineno-3-11) if commit and commit != first_commit: [](#__codelineno-3-12) first_commit = commit [](#__codelineno-3-13) break [](#__codelineno-3-14) await asyncio.sleep(0.5) [](#__codelineno-3-15) except Exception as e: [](#__codelineno-3-16) print(f"Warning: New content didn't appear: {e}") [](#__codelineno-3-17) [](#__codelineno-3-18) async with AsyncWebCrawler() as crawler: [](#__codelineno-3-19) session_id = "commit_session" [](#__codelineno-3-20) url = "https://github.com/example/repo/commits/main" [](#__codelineno-3-21) crawler.crawler_strategy.set_hook("on_execution_started", on_execution_started) [](#__codelineno-3-22) [](#__codelineno-3-23) js_next_page = """document.querySelector('a.pagination-next').click();""" [](#__codelineno-3-24) [](#__codelineno-3-25) for page in range(3): [](#__codelineno-3-26) config = CrawlerRunConfig( [](#__codelineno-3-27) url=url, [](#__codelineno-3-28) session_id=session_id, [](#__codelineno-3-29) js_code=js_next_page if page > 0 else None, [](#__codelineno-3-30) css_selector="li.commit-item", [](#__codelineno-3-31) js_only=page > 0, [](#__codelineno-3-32) cache_mode=CacheMode.BYPASS [](#__codelineno-3-33) ) [](#__codelineno-3-34) [](#__codelineno-3-35) result = await crawler.arun(config=config) [](#__codelineno-3-36) print(f"Page {page + 1}: Found {len(result.extracted_content)} commits") [](#__codelineno-3-37) [](#__codelineno-3-38) await crawler.crawler_strategy.kill_session(session_id) [](#__codelineno-3-39) [](#__codelineno-3-40) asyncio.run(advanced_session_crawl_with_hooks())` This technique ensures new content loads before the next action. * * * Advanced Technique 2: Integrated JavaScript Execution and Waiting ----------------------------------------------------------------- Combine JavaScript execution and waiting logic for concise handling of dynamic content: `[](#__codelineno-4-1) async def integrated_js_and_wait_crawl(): [](#__codelineno-4-2) async with AsyncWebCrawler() as crawler: [](#__codelineno-4-3) session_id = "integrated_session" [](#__codelineno-4-4) url = "https://github.com/example/repo/commits/main" [](#__codelineno-4-5) [](#__codelineno-4-6) js_next_page_and_wait = """ [](#__codelineno-4-7) (async () => { [](#__codelineno-4-8) const getCurrentCommit = () => document.querySelector('li.commit-item h4').textContent.trim(); [](#__codelineno-4-9) const initialCommit = getCurrentCommit(); [](#__codelineno-4-10) document.querySelector('a.pagination-next').click(); [](#__codelineno-4-11) while (getCurrentCommit() === initialCommit) { [](#__codelineno-4-12) await new Promise(resolve => setTimeout(resolve, 100)); [](#__codelineno-4-13) } [](#__codelineno-4-14) })(); [](#__codelineno-4-15) """ [](#__codelineno-4-16) [](#__codelineno-4-17) for page in range(3): [](#__codelineno-4-18) config = CrawlerRunConfig( [](#__codelineno-4-19) url=url, [](#__codelineno-4-20) session_id=session_id, [](#__codelineno-4-21) js_code=js_next_page_and_wait if page > 0 else None, [](#__codelineno-4-22) css_selector="li.commit-item", [](#__codelineno-4-23) js_only=page > 0, [](#__codelineno-4-24) cache_mode=CacheMode.BYPASS [](#__codelineno-4-25) ) [](#__codelineno-4-26) [](#__codelineno-4-27) result = await crawler.arun(config=config) [](#__codelineno-4-28) print(f"Page {page + 1}: Found {len(result.extracted_content)} commits") [](#__codelineno-4-29) [](#__codelineno-4-30) await crawler.crawler_strategy.kill_session(session_id) [](#__codelineno-4-31) [](#__codelineno-4-32) asyncio.run(integrated_js_and_wait_crawl())` * * * #### Common Use Cases for Sessions 1. **Authentication Flows**: Login and interact with secured pages. 2. **Pagination Handling**: Navigate through multiple pages. 3. **Form Submissions**: Fill forms, submit, and process results. 4. **Multi-step Processes**: Complete workflows that span multiple actions. 5. **Dynamic Content Navigation**: Handle JavaScript-rendered or event-triggered content. * * * --- # AsyncWebCrawler - Crawl4AI Documentation AsyncWebCrawler =============== The **`AsyncWebCrawler`** is the core class for asynchronous web crawling in Crawl4AI. You typically create it **once**, optionally customize it with a **`BrowserConfig`** (e.g., headless, user agent), then **run** multiple **`arun()`** calls with different **`CrawlerRunConfig`** objects. **Recommended usage**: 1. **Create** a `BrowserConfig` for global browser settings. 2. **Instantiate** `AsyncWebCrawler(config=browser_config)`. 3. **Use** the crawler in an async context manager (`async with`) or manage start/close manually. 4. **Call** `arun(url, config=crawler_run_config)` for each page you want. * * * 1\. Constructor Overview ------------------------ `[](#__codelineno-0-1) class AsyncWebCrawler: [](#__codelineno-0-2) def __init__( [](#__codelineno-0-3) self, [](#__codelineno-0-4) crawler_strategy: Optional[AsyncCrawlerStrategy] = None, [](#__codelineno-0-5) config: Optional[BrowserConfig] = None, [](#__codelineno-0-6) always_bypass_cache: bool = False, # deprecated [](#__codelineno-0-7) always_by_pass_cache: Optional[bool] = None, # also deprecated [](#__codelineno-0-8) base_directory: str = ..., [](#__codelineno-0-9) thread_safe: bool = False, [](#__codelineno-0-10) **kwargs, [](#__codelineno-0-11) ): [](#__codelineno-0-12) """ [](#__codelineno-0-13) Create an AsyncWebCrawler instance. [](#__codelineno-0-14) [](#__codelineno-0-15) Args: [](#__codelineno-0-16) crawler_strategy: [](#__codelineno-0-17) (Advanced) Provide a custom crawler strategy if needed. [](#__codelineno-0-18) config: [](#__codelineno-0-19) A BrowserConfig object specifying how the browser is set up. [](#__codelineno-0-20) always_bypass_cache: [](#__codelineno-0-21) (Deprecated) Use CrawlerRunConfig.cache_mode instead. [](#__codelineno-0-22) base_directory: [](#__codelineno-0-23) Folder for storing caches/logs (if relevant). [](#__codelineno-0-24) thread_safe: [](#__codelineno-0-25) If True, attempts some concurrency safeguards. Usually False. [](#__codelineno-0-26) **kwargs: [](#__codelineno-0-27) Additional legacy or debugging parameters. [](#__codelineno-0-28) """ [](#__codelineno-0-29) ) [](#__codelineno-0-30) [](#__codelineno-0-31) ### Typical Initialization [](#__codelineno-0-32) [](#__codelineno-0-33) ```python [](#__codelineno-0-34) from crawl4ai import AsyncWebCrawler, BrowserConfig [](#__codelineno-0-35) [](#__codelineno-0-36) browser_cfg = BrowserConfig( [](#__codelineno-0-37) browser_type="chromium", [](#__codelineno-0-38) headless=True, [](#__codelineno-0-39) verbose=True [](#__codelineno-0-40) ) [](#__codelineno-0-41) [](#__codelineno-0-42) crawler = AsyncWebCrawler(config=browser_cfg)` **Notes**: - **Legacy** parameters like `always_bypass_cache` remain for backward compatibility, but prefer to set **caching** in `CrawlerRunConfig`. * * * 2\. Lifecycle: Start/Close or Context Manager --------------------------------------------- ### 2.1 Context Manager (Recommended) `[](#__codelineno-1-1) async with AsyncWebCrawler(config=browser_cfg) as crawler: [](#__codelineno-1-2) result = await crawler.arun("https://example.com") [](#__codelineno-1-3) # The crawler automatically starts/closes resources` When the `async with` block ends, the crawler cleans up (closes the browser, etc.). ### 2.2 Manual Start & Close `[](#__codelineno-2-1) crawler = AsyncWebCrawler(config=browser_cfg) [](#__codelineno-2-2) await crawler.start() [](#__codelineno-2-3) [](#__codelineno-2-4) result1 = await crawler.arun("https://example.com") [](#__codelineno-2-5) result2 = await crawler.arun("https://another.com") [](#__codelineno-2-6) [](#__codelineno-2-7) await crawler.close()` Use this style if you have a **long-running** application or need full control of the crawler’s lifecycle. * * * 3\. Primary Method: `arun()` ---------------------------- `[](#__codelineno-3-1) async def arun( [](#__codelineno-3-2) self, [](#__codelineno-3-3) url: str, [](#__codelineno-3-4) config: Optional[CrawlerRunConfig] = None, [](#__codelineno-3-5) # Legacy parameters for backward compatibility... [](#__codelineno-3-6) ) -> CrawlResult: [](#__codelineno-3-7) ...` ### 3.1 New Approach You pass a `CrawlerRunConfig` object that sets up everything about a crawl—content filtering, caching, session reuse, JS code, screenshots, etc. `[](#__codelineno-4-1) import asyncio [](#__codelineno-4-2) from crawl4ai import CrawlerRunConfig, CacheMode [](#__codelineno-4-3) [](#__codelineno-4-4) run_cfg = CrawlerRunConfig( [](#__codelineno-4-5) cache_mode=CacheMode.BYPASS, [](#__codelineno-4-6) css_selector="main.article", [](#__codelineno-4-7) word_count_threshold=10, [](#__codelineno-4-8) screenshot=True [](#__codelineno-4-9) ) [](#__codelineno-4-10) [](#__codelineno-4-11) async with AsyncWebCrawler(config=browser_cfg) as crawler: [](#__codelineno-4-12) result = await crawler.arun("https://example.com/news", config=run_cfg) [](#__codelineno-4-13) print("Crawled HTML length:", len(result.cleaned_html)) [](#__codelineno-4-14) if result.screenshot: [](#__codelineno-4-15) print("Screenshot base64 length:", len(result.screenshot))` ### 3.2 Legacy Parameters Still Accepted For **backward** compatibility, `arun()` can still accept direct arguments like `css_selector=...`, `word_count_threshold=...`, etc., but we strongly advise migrating them into a **`CrawlerRunConfig`**. * * * 4\. Helper Methods ------------------ ### 4.1 `arun_many()` `[](#__codelineno-5-1) async def arun_many( [](#__codelineno-5-2) self, [](#__codelineno-5-3) urls: List[str], [](#__codelineno-5-4) config: Optional[CrawlerRunConfig] = None, [](#__codelineno-5-5) # Legacy parameters... [](#__codelineno-5-6) ) -> List[CrawlResult]: [](#__codelineno-5-7) ...` Crawls multiple URLs in concurrency. Accepts the same style `CrawlerRunConfig`. Example: `[](#__codelineno-6-1) run_cfg = CrawlerRunConfig( [](#__codelineno-6-2) # e.g., concurrency, wait_for, caching, extraction, etc. [](#__codelineno-6-3) semaphore_count=5 [](#__codelineno-6-4) ) [](#__codelineno-6-5) [](#__codelineno-6-6) async with AsyncWebCrawler(config=browser_cfg) as crawler: [](#__codelineno-6-7) results = await crawler.arun_many( [](#__codelineno-6-8) urls=["https://example.com", "https://another.com"], [](#__codelineno-6-9) config=run_cfg [](#__codelineno-6-10) ) [](#__codelineno-6-11) for r in results: [](#__codelineno-6-12) print(r.url, ":", len(r.cleaned_html))` ### 4.2 `start()` & `close()` Allows manual lifecycle usage instead of context manager: `[](#__codelineno-7-1) crawler = AsyncWebCrawler(config=browser_cfg) [](#__codelineno-7-2) await crawler.start() [](#__codelineno-7-3) [](#__codelineno-7-4) # Perform multiple operations [](#__codelineno-7-5) resultA = await crawler.arun("https://exampleA.com", config=run_cfg) [](#__codelineno-7-6) resultB = await crawler.arun("https://exampleB.com", config=run_cfg) [](#__codelineno-7-7) [](#__codelineno-7-8) await crawler.close()` * * * 5\. `CrawlResult` Output ------------------------ Each `arun()` returns a **`CrawlResult`** containing: * `url`: Final URL (if redirected). * `html`: Original HTML. * `cleaned_html`: Sanitized HTML. * `markdown_v2` (or future `markdown`): Markdown outputs (raw, fit, etc.). * `extracted_content`: If an extraction strategy was used (JSON for CSS/LLM strategies). * `screenshot`, `pdf`: If screenshots/PDF requested. * `media`, `links`: Information about discovered images/links. * `success`, `error_message`: Status info. For details, see [CrawlResult doc](../crawl-result/) . * * * 6\. Quick Example ----------------- Below is an example hooking it all together: `[](#__codelineno-8-1) import asyncio [](#__codelineno-8-2) from crawl4ai import AsyncWebCrawler, BrowserConfig, CrawlerRunConfig, CacheMode [](#__codelineno-8-3) from crawl4ai.extraction_strategy import JsonCssExtractionStrategy [](#__codelineno-8-4) import json [](#__codelineno-8-5) [](#__codelineno-8-6) async def main(): [](#__codelineno-8-7) # 1. Browser config [](#__codelineno-8-8) browser_cfg = BrowserConfig( [](#__codelineno-8-9) browser_type="firefox", [](#__codelineno-8-10) headless=False, [](#__codelineno-8-11) verbose=True [](#__codelineno-8-12) ) [](#__codelineno-8-13) [](#__codelineno-8-14) # 2. Run config [](#__codelineno-8-15) schema = { [](#__codelineno-8-16) "name": "Articles", [](#__codelineno-8-17) "baseSelector": "article.post", [](#__codelineno-8-18) "fields": [ [](#__codelineno-8-19) { [](#__codelineno-8-20) "name": "title", [](#__codelineno-8-21) "selector": "h2", [](#__codelineno-8-22) "type": "text" [](#__codelineno-8-23) }, [](#__codelineno-8-24) { [](#__codelineno-8-25) "name": "url", [](#__codelineno-8-26) "selector": "a", [](#__codelineno-8-27) "type": "attribute", [](#__codelineno-8-28) "attribute": "href" [](#__codelineno-8-29) } [](#__codelineno-8-30) ] [](#__codelineno-8-31) } [](#__codelineno-8-32) [](#__codelineno-8-33) run_cfg = CrawlerRunConfig( [](#__codelineno-8-34) cache_mode=CacheMode.BYPASS, [](#__codelineno-8-35) extraction_strategy=JsonCssExtractionStrategy(schema), [](#__codelineno-8-36) word_count_threshold=15, [](#__codelineno-8-37) remove_overlay_elements=True, [](#__codelineno-8-38) wait_for="css:.post" # Wait for posts to appear [](#__codelineno-8-39) ) [](#__codelineno-8-40) [](#__codelineno-8-41) async with AsyncWebCrawler(config=browser_cfg) as crawler: [](#__codelineno-8-42) result = await crawler.arun( [](#__codelineno-8-43) url="https://example.com/blog", [](#__codelineno-8-44) config=run_cfg [](#__codelineno-8-45) ) [](#__codelineno-8-46) [](#__codelineno-8-47) if result.success: [](#__codelineno-8-48) print("Cleaned HTML length:", len(result.cleaned_html)) [](#__codelineno-8-49) if result.extracted_content: [](#__codelineno-8-50) articles = json.loads(result.extracted_content) [](#__codelineno-8-51) print("Extracted articles:", articles[:2]) [](#__codelineno-8-52) else: [](#__codelineno-8-53) print("Error:", result.error_message) [](#__codelineno-8-54) [](#__codelineno-8-55) asyncio.run(main())` **Explanation**: - We define a **`BrowserConfig`** with Firefox, no headless, and `verbose=True`. \- We define a **`CrawlerRunConfig`** that **bypasses cache**, uses a **CSS** extraction schema, has a `word_count_threshold=15`, etc. \- We pass them to `AsyncWebCrawler(config=...)` and `arun(url=..., config=...)`. * * * 7\. Best Practices & Migration Notes ------------------------------------ 1. **Use** `BrowserConfig` for **global** settings about the browser’s environment. 2. **Use** `CrawlerRunConfig` for **per-crawl** logic (caching, content filtering, extraction strategies, wait conditions). 3. **Avoid** legacy parameters like `css_selector` or `word_count_threshold` directly in `arun()`. Instead: `[](#__codelineno-9-1) run_cfg = CrawlerRunConfig(css_selector=".main-content", word_count_threshold=20) [](#__codelineno-9-2) result = await crawler.arun(url="...", config=run_cfg)` 4. **Context Manager** usage is simplest unless you want a persistent crawler across many calls. * * * 8\. Summary ----------- **AsyncWebCrawler** is your entry point to asynchronous crawling: * **Constructor** accepts **`BrowserConfig`** (or defaults). * **`arun(url, config=CrawlerRunConfig)`** is the main method for single-page crawls. * **`arun_many(urls, config=CrawlerRunConfig)`** handles concurrency across multiple URLs. * For advanced lifecycle control, use `start()` and `close()` explicitly. **Migration**: \- If you used `AsyncWebCrawler(browser_type="chromium", css_selector="...")`, move browser settings to `BrowserConfig(...)` and content/crawl logic to `CrawlerRunConfig(...)`. This modular approach ensures your code is **clean**, **scalable**, and **easy to maintain**. For any advanced or rarely used parameters, see the [BrowserConfig docs](../parameters/) . * * * --- # Release Summary for Version 0.4.0 (December 1, 2024) - Crawl4AI Documentation Release Summary for Version 0.4.0 (December 1, 2024) ==================================================== Overview -------- The 0.4.0 release introduces significant improvements to content filtering, multi-threaded environment handling, user-agent generation, and test coverage. Key highlights include the introduction of the PruningContentFilter, designed to automatically identify and extract the most valuable parts of an HTML document, as well as enhancements to the BM25ContentFilter to extend its versatility and effectiveness. Major Features and Enhancements ------------------------------- ### 1\. PruningContentFilter * Introduced a new unsupervised content filtering strategy that scores and prunes less relevant nodes in an HTML document based on metrics like text and link density. * Focuses on retaining the most valuable parts of the content, making it highly effective for extracting relevant information from complex web pages. * Fully documented with updated README and expanded user guides. ### 2\. User-Agent Generator * Added a user-agent generator utility that resolves compatibility issues and supports customizable user-agent strings. * By default, the generator randomizes user agents for each request, adding diversity, but users can customize it for tailored scenarios. ### 3\. Enhanced Thread Safety * Improved handling of multi-threaded environments by adding better thread locks for parallel processing, ensuring consistency and stability when running multiple threads. ### 4\. Extended Content Filtering Strategies * Users now have access to both the PruningContentFilter for unsupervised extraction and the BM25ContentFilter for supervised filtering based on user queries. * Enhanced BM25ContentFilter with improved capabilities to process page titles, meta tags, and descriptions, allowing for more effective classification and clustering of text chunks. ### 5\. Documentation Updates * Updated examples and tutorials to promote the use of the PruningContentFilter alongside the BM25ContentFilter, providing clear instructions for selecting the appropriate filter for each use case. ### 6\. Unit Test Enhancements * Added unit tests for PruningContentFilter to ensure accuracy and reliability. * Enhanced BM25ContentFilter tests to cover additional edge cases and performance metrics, particularly for malformed HTML inputs. Revised Change Logs for Version 0.4.0 ------------------------------------- ### PruningContentFilter (Dec 01, 2024) * Introduced the PruningContentFilter to optimize content extraction by pruning less relevant HTML nodes. * **Affected Files:** * **crawl4ai/content\_filter\_strategy.py**: Added a scoring-based pruning algorithm. * **README.md**: Updated to include PruningContentFilter usage. * **docs/md\_v2/basic/content\_filtering.md**: Expanded user documentation, detailing the use and benefits of PruningContentFilter. ### Unit Tests for PruningContentFilter (Dec 01, 2024) * Added comprehensive unit tests for PruningContentFilter to ensure correctness and efficiency. * **Affected Files:** * **tests/async/test\_content\_filter\_prune.py**: Created tests covering different pruning scenarios to ensure stability and correctness. ### Enhanced BM25ContentFilter Tests (Dec 01, 2024) * Expanded tests to cover additional extraction scenarios and performance metrics, improving robustness. * **Affected Files:** * **tests/async/test\_content\_filter\_bm25.py**: Added tests for edge cases, including malformed HTML inputs. ### Documentation and Example Updates (Dec 01, 2024) * Revised examples to illustrate the use of PruningContentFilter alongside existing content filtering methods. * **Affected Files:** * **docs/examples/quickstart\_async.py**: Enhanced example clarity and usability for new users. Experimental Features --------------------- * The PruningContentFilter is still under experimental development, and we continue to gather feedback for further refinements. Conclusion ---------- This release significantly enhances the content extraction capabilities of Crawl4ai with the introduction of the PruningContentFilter, improved supervised filtering with BM25ContentFilter, and robust multi-threaded handling. Additionally, the user-agent generator provides much-needed versatility, resolving compatibility issues faced by many users. Users are encouraged to experiment with the new content filtering methods to determine which best suits their needs. * * * --- # Blog Home - Crawl4AI Documentation Crawl4AI Blog ============= Welcome to the Crawl4AI blog! Here you'll find detailed release notes, technical insights, and updates about the project. Whether you're looking for the latest improvements or want to dive deep into web crawling techniques, this is the place. Latest Release -------------- ### [0.4.2 - Configurable Crawlers, Session Management, and Smarter Screenshots](releases/0.4.2/) _December 12, 2024_ The 0.4.2 update brings massive improvements to configuration, making crawlers and browsers easier to manage with dedicated objects. You can now import/export local storage for seamless session management. Plus, long-page screenshots are faster and cleaner, and full-page PDF exports are now possible. Check out all the new features to make your crawling experience even smoother. [Read full release notes →](releases/0.4.2/) * * * ### [0.4.1 - Smarter Crawling with Lazy-Load Handling, Text-Only Mode, and More](releases/0.4.1/) _December 8, 2024_ This release brings major improvements to handling lazy-loaded images, a blazing-fast Text-Only Mode, full-page scanning for infinite scrolls, dynamic viewport adjustments, and session reuse for efficient crawling. If you're looking to improve speed, reliability, or handle dynamic content with ease, this update has you covered. [Read full release notes →](releases/0.4.1/) * * * ### [0.4.0 - Major Content Filtering Update](releases/0.4.0/) _December 1, 2024_ Introduced significant improvements to content filtering, multi-threaded environment handling, and user-agent generation. This release features the new PruningContentFilter, enhanced thread safety, and improved test coverage. [Read full release notes →](releases/0.4.0/) Project History --------------- Curious about how Crawl4AI has evolved? Check out our [complete changelog](https://github.com/unclecode/crawl4ai/blob/main/CHANGELOG.md) for a detailed history of all versions and updates. Stay Updated ------------ * Star us on [GitHub](https://github.com/unclecode/crawl4ai) * Follow [@unclecode](https://twitter.com/unclecode) on Twitter * Join our community discussions on GitHub * * * --- # Release Summary for Version 0.4.1 (December 8, 2024): Major Efficiency Boosts with New Features! - Crawl4AI Documentation Release Summary for Version 0.4.1 (December 8, 2024): Major Efficiency Boosts with New Features! ================================================================================================ _This post was generated with the help of ChatGPT, take everything with a grain of salt. 🧂_ Hi everyone, I just finished putting together version 0.4.1 of Crawl4AI, and there are a few changes in here that I think you’ll find really helpful. I’ll explain what’s new, why it matters, and exactly how you can use these features (with the code to back it up). Let’s get into it. * * * ### Handling Lazy Loading Better (Images Included) One thing that always bugged me with crawlers is how often they miss lazy-loaded content, especially images. In this version, I made sure Crawl4AI **waits for all images to load** before moving forward. This is useful because many modern websites only load images when they’re in the viewport or after some JavaScript executes. Here’s how to enable it: `[](#__codelineno-0-1) await crawler.crawl( [](#__codelineno-0-2) url="https://example.com", [](#__codelineno-0-3) wait_for_images=True # Add this argument to ensure images are fully loaded [](#__codelineno-0-4) )` What this does is: 1. Waits for the page to reach a "network idle" state. 2. Ensures all images on the page have been completely loaded. This single change handles the majority of lazy-loading cases you’re likely to encounter. * * * ### Text-Only Mode (Fast, Lightweight Crawling) Sometimes, you don’t need to download images or process JavaScript at all. For example, if you’re crawling to extract text data, you can enable **text-only mode** to speed things up. By disabling images, JavaScript, and other heavy resources, this mode makes crawling **3-4 times faster** in most cases. Here’s how to turn it on: `[](#__codelineno-1-1) crawler = AsyncPlaywrightCrawlerStrategy( [](#__codelineno-1-2) text_mode=True # Set this to True to enable text-only crawling [](#__codelineno-1-3) )` When `text_mode=True`, the crawler automatically: - Disables GPU processing. - Blocks image and JavaScript resources. - Reduces the viewport size to 800x600 (you can override this with `viewport_width` and `viewport_height`). If you need to crawl thousands of pages where you only care about text, this mode will save you a ton of time and resources. * * * ### Adjusting the Viewport Dynamically Another useful addition is the ability to **dynamically adjust the viewport size** to match the content on the page. This is particularly helpful when you’re working with responsive layouts or want to ensure all parts of the page load properly. Here’s how it works: 1. The crawler calculates the page’s width and height after it loads. 2. It adjusts the viewport to fit the content dimensions. 3. (Optional) It uses Chrome DevTools Protocol (CDP) to simulate zooming out so everything fits in the viewport. To enable this, use: `[](#__codelineno-2-1) await crawler.crawl( [](#__codelineno-2-2) url="https://example.com", [](#__codelineno-2-3) adjust_viewport_to_content=True # Dynamically adjusts the viewport [](#__codelineno-2-4) )` This approach makes sure the entire page gets loaded into the viewport, especially for layouts that load content based on visibility. * * * ### Simulating Full-Page Scrolling Some websites load data dynamically as you scroll down the page. To handle these cases, I added support for **full-page scanning**. It simulates scrolling to the bottom of the page, checking for new content, and capturing it all. Here’s an example: `[](#__codelineno-3-1) await crawler.crawl( [](#__codelineno-3-2) url="https://example.com", [](#__codelineno-3-3) scan_full_page=True, # Enables scrolling [](#__codelineno-3-4) scroll_delay=0.2 # Waits 200ms between scrolls (optional) [](#__codelineno-3-5) )` What happens here: 1. The crawler scrolls down in increments, waiting for content to load after each scroll. 2. It stops when no new content appears (i.e., dynamic elements stop loading). 3. It scrolls back to the top before finishing (if necessary). If you’ve ever had to deal with infinite scroll pages, this is going to save you a lot of headaches. * * * ### Reusing Browser Sessions (Save Time on Setup) By default, every time you crawl a page, a new browser context (or tab) is created. That’s fine for small crawls, but if you’re working on a large dataset, it’s more efficient to reuse the same session. I added a method called `create_session` for this: `[](#__codelineno-4-1) session_id = await crawler.create_session() [](#__codelineno-4-2) [](#__codelineno-4-3) # Use the same session for multiple crawls [](#__codelineno-4-4) await crawler.crawl( [](#__codelineno-4-5) url="https://example.com/page1", [](#__codelineno-4-6) session_id=session_id # Reuse the session [](#__codelineno-4-7) ) [](#__codelineno-4-8) await crawler.crawl( [](#__codelineno-4-9) url="https://example.com/page2", [](#__codelineno-4-10) session_id=session_id [](#__codelineno-4-11) )` This avoids creating a new tab for every page, speeding up the crawl and reducing memory usage. * * * ### Other Updates Here are a few smaller updates I’ve made: - **Light Mode**: Use `light_mode=True` to disable background processes, extensions, and other unnecessary features, making the browser more efficient. - **Logging**: Improved logs to make debugging easier. - **Defaults**: Added sensible defaults for things like `delay_before_return_html` (now set to 0.1 seconds). * * * ### How to Get the Update You can install or upgrade to version `0.4.1` like this: `[](#__codelineno-5-1) pip install crawl4ai --upgrade` As always, I’d love to hear your thoughts. If there’s something you think could be improved or if you have suggestions for future versions, let me know! Enjoy the new features, and happy crawling! 🕷️ * * * * * * --- # CrawlResult - Crawl4AI Documentation `CrawlResult` Reference ======================= The **`CrawlResult`** class encapsulates everything returned after a single crawl operation. It provides the **raw or processed content**, details on links and media, plus optional metadata (like screenshots, PDFs, or extracted JSON). **Location**: `crawl4ai/crawler/models.py` (for reference) `[](#__codelineno-0-1) class CrawlResult(BaseModel): [](#__codelineno-0-2) url: str [](#__codelineno-0-3) html: str [](#__codelineno-0-4) success: bool [](#__codelineno-0-5) cleaned_html: Optional[str] = None [](#__codelineno-0-6) media: Dict[str, List[Dict]] = {} [](#__codelineno-0-7) links: Dict[str, List[Dict]] = {} [](#__codelineno-0-8) downloaded_files: Optional[List[str]] = None [](#__codelineno-0-9) screenshot: Optional[str] = None [](#__codelineno-0-10) pdf : Optional[bytes] = None [](#__codelineno-0-11) markdown: Optional[Union[str, MarkdownGenerationResult]] = None [](#__codelineno-0-12) markdown_v2: Optional[MarkdownGenerationResult] = None [](#__codelineno-0-13) fit_markdown: Optional[str] = None [](#__codelineno-0-14) fit_html: Optional[str] = None [](#__codelineno-0-15) extracted_content: Optional[str] = None [](#__codelineno-0-16) metadata: Optional[dict] = None [](#__codelineno-0-17) error_message: Optional[str] = None [](#__codelineno-0-18) session_id: Optional[str] = None [](#__codelineno-0-19) response_headers: Optional[dict] = None [](#__codelineno-0-20) status_code: Optional[int] = None [](#__codelineno-0-21) ssl_certificate: Optional[SSLCertificate] = None [](#__codelineno-0-22) ...` Below is a **field-by-field** explanation and possible usage patterns. * * * 1\. Basic Crawl Info -------------------- ### 1.1 **`url`** _(str)_ **What**: The final crawled URL (after any redirects). **Usage**: `[](#__codelineno-1-1) print(result.url) # e.g., "https://example.com/"` ### 1.2 **`success`** _(bool)_ **What**: `True` if the crawl pipeline ended without major errors; `False` otherwise. **Usage**: `[](#__codelineno-2-1) if not result.success: [](#__codelineno-2-2) print(f"Crawl failed: {result.error_message}")` ### 1.3 **`status_code`** _(Optional\[int\])_ **What**: The page’s HTTP status code (e.g., 200, 404). **Usage**: `[](#__codelineno-3-1) if result.status_code == 404: [](#__codelineno-3-2) print("Page not found!")` ### 1.4 **`error_message`** _(Optional\[str\])_ **What**: If `success=False`, a textual description of the failure. **Usage**: `[](#__codelineno-4-1) if not result.success: [](#__codelineno-4-2) print("Error:", result.error_message)` ### 1.5 **`session_id`** _(Optional\[str\])_ **What**: The ID used for reusing a browser context across multiple calls. **Usage**: `[](#__codelineno-5-1) # If you used session_id="login_session" in CrawlerRunConfig, see it here: [](#__codelineno-5-2) print("Session:", result.session_id)` ### 1.6 **`response_headers`** _(Optional\[dict\])_ **What**: Final HTTP response headers. **Usage**: `[](#__codelineno-6-1) if result.response_headers: [](#__codelineno-6-2) print("Server:", result.response_headers.get("Server", "Unknown"))` ### 1.7 **`ssl_certificate`** _(Optional\[SSLCertificate\])_ **What**: If `fetch_ssl_certificate=True` in your CrawlerRunConfig, **`result.ssl_certificate`** contains a [**`SSLCertificate`**](../../advanced/ssl-certificate/) object describing the site’s certificate. You can export the cert in multiple formats (PEM/DER/JSON) or access its properties like `issuer`, `subject`, `valid_from`, `valid_until`, etc. **Usage**: `[](#__codelineno-7-1) if result.ssl_certificate: [](#__codelineno-7-2) print("Issuer:", result.ssl_certificate.issuer)` * * * 2\. Raw / Cleaned Content ------------------------- ### 2.1 **`html`** _(str)_ **What**: The **original** unmodified HTML from the final page load. **Usage**: `[](#__codelineno-8-1) # Possibly large [](#__codelineno-8-2) print(len(result.html))` ### 2.2 **`cleaned_html`** _(Optional\[str\])_ **What**: A sanitized HTML version—scripts, styles, or excluded tags are removed based on your `CrawlerRunConfig`. **Usage**: `[](#__codelineno-9-1) print(result.cleaned_html[:500]) # Show a snippet` ### 2.3 **`fit_html`** _(Optional\[str\])_ **What**: If a **content filter** or heuristic (e.g., Pruning/BM25) modifies the HTML, the “fit” or post-filter version. **When**: This is **only** present if your `markdown_generator` or `content_filter` produces it. **Usage**: `[](#__codelineno-10-1) if result.fit_html: [](#__codelineno-10-2) print("High-value HTML content:", result.fit_html[:300])` * * * 3\. Markdown Fields ------------------- ### 3.1 The Markdown Generation Approach Crawl4AI can convert HTML→Markdown, optionally including: * **Raw** markdown * **Links as citations** (with a references section) * **Fit** markdown if a **content filter** is used (like Pruning or BM25) ### 3.2 **`markdown_v2`** _(Optional\[MarkdownGenerationResult\])_ **What**: The **structured** object holding multiple markdown variants. Soon to be consolidated into `markdown`. **`MarkdownGenerationResult`** includes: - **`raw_markdown`** _(str)_: The full HTML→Markdown conversion. \- **`markdown_with_citations`** _(str)_: Same markdown, but with link references as academic-style citations. \- **`references_markdown`** _(str)_: The reference list or footnotes at the end. \- **`fit_markdown`** _(Optional\[str\])_: If content filtering (Pruning/BM25) was applied, the filtered “fit” text. \- **`fit_html`** _(Optional\[str\])_: The HTML that led to `fit_markdown`. **Usage**: `[](#__codelineno-11-1) if result.markdown_v2: [](#__codelineno-11-2) md_res = result.markdown_v2 [](#__codelineno-11-3) print("Raw MD:", md_res.raw_markdown[:300]) [](#__codelineno-11-4) print("Citations MD:", md_res.markdown_with_citations[:300]) [](#__codelineno-11-5) print("References:", md_res.references_markdown) [](#__codelineno-11-6) if md_res.fit_markdown: [](#__codelineno-11-7) print("Pruned text:", md_res.fit_markdown[:300])` ### 3.3 **`markdown`** _(Optional\[Union\[str, MarkdownGenerationResult\]\])_ **What**: In future versions, `markdown` will fully replace `markdown_v2`. Right now, it might be a `str` or a `MarkdownGenerationResult`. **Usage**: `[](#__codelineno-12-1) # Soon, you might see: [](#__codelineno-12-2) if isinstance(result.markdown, MarkdownGenerationResult): [](#__codelineno-12-3) print(result.markdown.raw_markdown[:200]) [](#__codelineno-12-4) else: [](#__codelineno-12-5) print(result.markdown)` ### 3.4 **`fit_markdown`** _(Optional\[str\])_ **What**: A direct reference to the final filtered markdown (legacy approach). **When**: This is set if a filter or content strategy explicitly writes there. Usually overshadowed by `markdown_v2.fit_markdown`. **Usage**: `[](#__codelineno-13-1) print(result.fit_markdown) # Legacy field, prefer result.markdown_v2.fit_markdown` **Important**: “Fit” content (in `fit_markdown`/`fit_html`) only exists if you used a **filter** (like **PruningContentFilter** or **BM25ContentFilter**) within a `MarkdownGenerationStrategy`. * * * 4\. Media & Links ----------------- ### 4.1 **`media`** _(Dict\[str, List\[Dict\]\])_ **What**: Contains info about discovered images, videos, or audio. Typically keys: `"images"`, `"videos"`, `"audios"`. **Common Fields** in each item: * `src` _(str)_: Media URL * `alt` or `title` _(str)_: Descriptive text * `score` _(float)_: Relevance score if the crawler’s heuristic found it “important” * `desc` or `description` _(Optional\[str\])_: Additional context extracted from surrounding text **Usage**: `[](#__codelineno-14-1) images = result.media.get("images", []) [](#__codelineno-14-2) for img in images: [](#__codelineno-14-3) if img.get("score", 0) > 5: [](#__codelineno-14-4) print("High-value image:", img["src"])` ### 4.2 **`links`** _(Dict\[str, List\[Dict\]\])_ **What**: Holds internal and external link data. Usually two keys: `"internal"` and `"external"`. **Common Fields**: * `href` _(str)_: The link target * `text` _(str)_: Link text * `title` _(str)_: Title attribute * `context` _(str)_: Surrounding text snippet * `domain` _(str)_: If external, the domain **Usage**: `[](#__codelineno-15-1) for link in result.links["internal"]: [](#__codelineno-15-2) print(f"Internal link to {link['href']} with text {link['text']}")` * * * 5\. Additional Fields --------------------- ### 5.1 **`extracted_content`** _(Optional\[str\])_ **What**: If you used **`extraction_strategy`** (CSS, LLM, etc.), the structured output (JSON). **Usage**: `[](#__codelineno-16-1) if result.extracted_content: [](#__codelineno-16-2) data = json.loads(result.extracted_content) [](#__codelineno-16-3) print(data)` ### 5.2 **`downloaded_files`** _(Optional\[List\[str\]\])_ **What**: If `accept_downloads=True` in your `BrowserConfig` + `downloads_path`, lists local file paths for downloaded items. **Usage**: `[](#__codelineno-17-1) if result.downloaded_files: [](#__codelineno-17-2) for file_path in result.downloaded_files: [](#__codelineno-17-3) print("Downloaded:", file_path)` ### 5.3 **`screenshot`** _(Optional\[str\])_ **What**: Base64-encoded screenshot if `screenshot=True` in `CrawlerRunConfig`. **Usage**: `[](#__codelineno-18-1) import base64 [](#__codelineno-18-2) if result.screenshot: [](#__codelineno-18-3) with open("page.png", "wb") as f: [](#__codelineno-18-4) f.write(base64.b64decode(result.screenshot))` ### 5.4 **`pdf`** _(Optional\[bytes\])_ **What**: Raw PDF bytes if `pdf=True` in `CrawlerRunConfig`. **Usage**: `[](#__codelineno-19-1) if result.pdf: [](#__codelineno-19-2) with open("page.pdf", "wb") as f: [](#__codelineno-19-3) f.write(result.pdf)` ### 5.5 **`metadata`** _(Optional\[dict\])_ **What**: Page-level metadata if discovered (title, description, OG data, etc.). **Usage**: `[](#__codelineno-20-1) if result.metadata: [](#__codelineno-20-2) print("Title:", result.metadata.get("title")) [](#__codelineno-20-3) print("Author:", result.metadata.get("author"))` * * * 6\. Example: Accessing Everything --------------------------------- `[](#__codelineno-21-1) async def handle_result(result: CrawlResult): [](#__codelineno-21-2) if not result.success: [](#__codelineno-21-3) print("Crawl error:", result.error_message) [](#__codelineno-21-4) return [](#__codelineno-21-5) [](#__codelineno-21-6) # Basic info [](#__codelineno-21-7) print("Crawled URL:", result.url) [](#__codelineno-21-8) print("Status code:", result.status_code) [](#__codelineno-21-9) [](#__codelineno-21-10) # HTML [](#__codelineno-21-11) print("Original HTML size:", len(result.html)) [](#__codelineno-21-12) print("Cleaned HTML size:", len(result.cleaned_html or "")) [](#__codelineno-21-13) [](#__codelineno-21-14) # Markdown output [](#__codelineno-21-15) if result.markdown_v2: [](#__codelineno-21-16) print("Raw Markdown:", result.markdown_v2.raw_markdown[:300]) [](#__codelineno-21-17) print("Citations Markdown:", result.markdown_v2.markdown_with_citations[:300]) [](#__codelineno-21-18) if result.markdown_v2.fit_markdown: [](#__codelineno-21-19) print("Fit Markdown:", result.markdown_v2.fit_markdown[:200]) [](#__codelineno-21-20) else: [](#__codelineno-21-21) print("Raw Markdown (legacy):", result.markdown[:200] if result.markdown else "N/A") [](#__codelineno-21-22) [](#__codelineno-21-23) # Media & Links [](#__codelineno-21-24) if "images" in result.media: [](#__codelineno-21-25) print("Image count:", len(result.media["images"])) [](#__codelineno-21-26) if "internal" in result.links: [](#__codelineno-21-27) print("Internal link count:", len(result.links["internal"])) [](#__codelineno-21-28) [](#__codelineno-21-29) # Extraction strategy result [](#__codelineno-21-30) if result.extracted_content: [](#__codelineno-21-31) print("Structured data:", result.extracted_content) [](#__codelineno-21-32) [](#__codelineno-21-33) # Screenshot/PDF [](#__codelineno-21-34) if result.screenshot: [](#__codelineno-21-35) print("Screenshot length:", len(result.screenshot)) [](#__codelineno-21-36) if result.pdf: [](#__codelineno-21-37) print("PDF bytes length:", len(result.pdf))` * * * 7\. Key Points & Future ----------------------- 1. **`markdown_v2` vs `markdown`** \- Right now, `markdown_v2` is the more robust container (`MarkdownGenerationResult`), providing **raw\_markdown**, **markdown\_with\_citations**, references, plus possible **fit\_markdown**. \- In future versions, everything will unify under **`markdown`**. If you rely on advanced features (citations, fit content), check `markdown_v2`. 2. **Fit Content** \- **`fit_markdown`** and **`fit_html`** appear only if you used a content filter (like **PruningContentFilter** or **BM25ContentFilter**) inside your **MarkdownGenerationStrategy** or set them directly. \- If no filter is used, they remain `None`. 3. **References & Citations** \- If you enable link citations in your `DefaultMarkdownGenerator` (`options={"citations": True}`), you’ll see `markdown_with_citations` plus a **`references_markdown`** block. This helps large language models or academic-like referencing. 4. **Links & Media** \- `links["internal"]` and `links["external"]` group discovered anchors by domain. \- `media["images"]` / `["videos"]` / `["audios"]` store extracted media elements with optional scoring or context. 5. **Error Cases** \- If `success=False`, check `error_message` (e.g., timeouts, invalid URLs). \- `status_code` might be `None` if we failed before an HTTP response. Use **`CrawlResult`** to glean all final outputs and feed them into your data pipelines, AI models, or archives. With the synergy of a properly configured **BrowserConfig** and **CrawlerRunConfig**, the crawler can produce robust, structured results here in **`CrawlResult`**. * * * --- # Cache Modes - Crawl4AI Documentation Crawl4AI Cache System and Migration Guide ========================================= Overview -------- Starting from version 0.5.0, Crawl4AI introduces a new caching system that replaces the old boolean flags with a more intuitive `CacheMode` enum. This change simplifies cache control and makes the behavior more predictable. Old vs New Approach ------------------- ### Old Way (Deprecated) The old system used multiple boolean flags: - `bypass_cache`: Skip cache entirely - `disable_cache`: Disable all caching - `no_cache_read`: Don't read from cache - `no_cache_write`: Don't write to cache ### New Way (Recommended) The new system uses a single `CacheMode` enum: - `CacheMode.ENABLED`: Normal caching (read/write) - `CacheMode.DISABLED`: No caching at all - `CacheMode.READ_ONLY`: Only read from cache - `CacheMode.WRITE_ONLY`: Only write to cache - `CacheMode.BYPASS`: Skip cache for this operation Migration Example ----------------- ### Old Code (Deprecated) `[](#__codelineno-0-1) import asyncio [](#__codelineno-0-2) from crawl4ai import AsyncWebCrawler [](#__codelineno-0-3) [](#__codelineno-0-4) async def use_proxy(): [](#__codelineno-0-5) async with AsyncWebCrawler(verbose=True) as crawler: [](#__codelineno-0-6) result = await crawler.arun( [](#__codelineno-0-7) url="https://www.nbcnews.com/business", [](#__codelineno-0-8) bypass_cache=True # Old way [](#__codelineno-0-9) ) [](#__codelineno-0-10) print(len(result.markdown)) [](#__codelineno-0-11) [](#__codelineno-0-12) async def main(): [](#__codelineno-0-13) await use_proxy() [](#__codelineno-0-14) [](#__codelineno-0-15) if __name__ == "__main__": [](#__codelineno-0-16) asyncio.run(main())` ### New Code (Recommended) `[](#__codelineno-1-1) import asyncio [](#__codelineno-1-2) from crawl4ai import AsyncWebCrawler, CacheMode [](#__codelineno-1-3) from crawl4ai.async_configs import CrawlerRunConfig [](#__codelineno-1-4) [](#__codelineno-1-5) async def use_proxy(): [](#__codelineno-1-6) # Use CacheMode in CrawlerRunConfig [](#__codelineno-1-7) config = CrawlerRunConfig(cache_mode=CacheMode.BYPASS) [](#__codelineno-1-8) async with AsyncWebCrawler(verbose=True) as crawler: [](#__codelineno-1-9) result = await crawler.arun( [](#__codelineno-1-10) url="https://www.nbcnews.com/business", [](#__codelineno-1-11) config=config # Pass the configuration object [](#__codelineno-1-12) ) [](#__codelineno-1-13) print(len(result.markdown)) [](#__codelineno-1-14) [](#__codelineno-1-15) async def main(): [](#__codelineno-1-16) await use_proxy() [](#__codelineno-1-17) [](#__codelineno-1-18) if __name__ == "__main__": [](#__codelineno-1-19) asyncio.run(main())` Common Migration Patterns ------------------------- | Old Flag | New Mode | | --- | --- | | `bypass_cache=True` | `cache_mode=CacheMode.BYPASS` | | `disable_cache=True` | `cache_mode=CacheMode.DISABLED` | | `no_cache_read=True` | `cache_mode=CacheMode.WRITE_ONLY` | | `no_cache_write=True` | `cache_mode=CacheMode.READ_ONLY` | * * * --- # Installation - Crawl4AI Documentation Installation & Setup (2023 Edition) =================================== 1\. Basic Installation ---------------------- `[](#__codelineno-0-1) pip install crawl4ai` This installs the **core** Crawl4AI library along with essential dependencies. **No** advanced features (like transformers or PyTorch) are included yet. 2\. Initial Setup & Diagnostics ------------------------------- ### 2.1 Run the Setup Command After installing, call: `[](#__codelineno-1-1) crawl4ai-setup` **What does it do?** - Installs or updates required Playwright browsers (Chromium, Firefox, etc.) - Performs OS-level checks (e.g., missing libs on Linux) - Confirms your environment is ready to crawl ### 2.2 Diagnostics Optionally, you can run **diagnostics** to confirm everything is functioning: `[](#__codelineno-2-1) crawl4ai-doctor` This command attempts to: - Check Python version compatibility - Verify Playwright installation - Inspect environment variables or library conflicts If any issues arise, follow its suggestions (e.g., installing additional system packages) and re-run `crawl4ai-setup`. * * * 3\. Verifying Installation: A Simple Crawl (Skip this step if you already run `crawl4ai-doctor`) ------------------------------------------------------------------------------------------------ Below is a minimal Python script demonstrating a **basic** crawl. It uses our new **`BrowserConfig`** and **`CrawlerRunConfig`** for clarity, though no custom settings are passed in this example: `[](#__codelineno-3-1) import asyncio [](#__codelineno-3-2) from crawl4ai import AsyncWebCrawler, BrowserConfig, CrawlerRunConfig [](#__codelineno-3-3) [](#__codelineno-3-4) async def main(): [](#__codelineno-3-5) async with AsyncWebCrawler() as crawler: [](#__codelineno-3-6) result = await crawler.arun( [](#__codelineno-3-7) url="https://www.example.com", [](#__codelineno-3-8) ) [](#__codelineno-3-9) print(result.markdown[:300]) # Show the first 300 characters of extracted text [](#__codelineno-3-10) [](#__codelineno-3-11) if __name__ == "__main__": [](#__codelineno-3-12) asyncio.run(main())` **Expected** outcome: - A headless browser session loads `example.com` - Crawl4AI returns ~300 characters of markdown. If errors occur, rerun `crawl4ai-doctor` or manually ensure Playwright is installed correctly. * * * 4\. Advanced Installation (Optional) ------------------------------------ **Warning**: Only install these **if you truly need them**. They bring in larger dependencies, including big models, which can increase disk usage and memory load significantly. ### 4.1 Torch, Transformers, or All * **Text Clustering (Torch)** `[](#__codelineno-4-1) pip install crawl4ai[torch] [](#__codelineno-4-2) crawl4ai-setup` Installs PyTorch-based features (e.g., cosine similarity or advanced semantic chunking). * **Transformers** `[](#__codelineno-5-1) pip install crawl4ai[transformer] [](#__codelineno-5-2) crawl4ai-setup` Adds Hugging Face-based summarization or generation strategies. * **All Features** `[](#__codelineno-6-1) pip install crawl4ai[all] [](#__codelineno-6-2) crawl4ai-setup` #### (Optional) Pre-Fetching Models `[](#__codelineno-7-1) crawl4ai-download-models` This step caches large models locally (if needed). **Only do this** if your workflow requires them. * * * 5\. Docker (Experimental) ------------------------- We provide a **temporary** Docker approach for testing. **It’s not stable and may break** with future releases. We plan a major Docker revamp in a future stable version, 2025 Q1. If you still want to try: `[](#__codelineno-8-1) docker pull unclecode/crawl4ai:basic [](#__codelineno-8-2) docker run -p 11235:11235 unclecode/crawl4ai:basic` You can then make POST requests to `http://localhost:11235/crawl` to perform crawls. **Production usage** is discouraged until our new Docker approach is ready (planned in Jan or Feb 2025). * * * 6\. Local Server Mode (Legacy) ------------------------------ Some older docs mention running Crawl4AI as a local server. This approach has been **partially replaced** by the new Docker-based prototype and upcoming stable server release. You can experiment, but expect major changes. Official local server instructions will arrive once the new Docker architecture is finalized. * * * Summary ------- 1. **Install** with `pip install crawl4ai` and run `crawl4ai-setup`. 2. **Diagnose** with `crawl4ai-doctor` if you see errors. 3. **Verify** by crawling `example.com` with minimal `BrowserConfig` + `CrawlerRunConfig`. 4. **Advanced** features (Torch, Transformers) are **optional**—avoid them if you don’t need them (they significantly increase resource usage). 5. **Docker** is **experimental**—use at your own risk until the stable version is released. 6. **Local server** references in older docs are largely deprecated; a new solution is in progress. **Got questions?** Check [GitHub issues](https://github.com/unclecode/crawl4ai/issues) for updates or ask the community! * * * --- # arun() - Crawl4AI Documentation `arun()` Parameter Guide (New Approach) ======================================= In Crawl4AI’s **latest** configuration model, nearly all parameters that once went directly to `arun()` are now part of **`CrawlerRunConfig`**. When calling `arun()`, you provide: `[](#__codelineno-0-1) await crawler.arun( [](#__codelineno-0-2) url="https://example.com", [](#__codelineno-0-3) config=my_run_config [](#__codelineno-0-4) )` Below is an organized look at the parameters that can go inside `CrawlerRunConfig`, divided by their functional areas. For **Browser** settings (e.g., `headless`, `browser_type`), see [BrowserConfig](../parameters/) . * * * 1\. Core Usage -------------- `[](#__codelineno-1-1) from crawl4ai import AsyncWebCrawler, CrawlerRunConfig, CacheMode [](#__codelineno-1-2) [](#__codelineno-1-3) async def main(): [](#__codelineno-1-4) run_config = CrawlerRunConfig( [](#__codelineno-1-5) verbose=True, # Detailed logging [](#__codelineno-1-6) cache_mode=CacheMode.ENABLED, # Use normal read/write cache [](#__codelineno-1-7) # ... other parameters [](#__codelineno-1-8) ) [](#__codelineno-1-9) [](#__codelineno-1-10) async with AsyncWebCrawler() as crawler: [](#__codelineno-1-11) result = await crawler.arun( [](#__codelineno-1-12) url="https://example.com", [](#__codelineno-1-13) config=run_config [](#__codelineno-1-14) ) [](#__codelineno-1-15) print(result.cleaned_html[:500])` **Key Fields**: - `verbose=True` logs each crawl step. \- `cache_mode` decides how to read/write the local crawl cache. * * * 2\. Cache Control ----------------- **`cache_mode`** (default: `CacheMode.ENABLED`) Use a built-in enum from `CacheMode`: - `ENABLED`: Normal caching—reads if available, writes if missing. - `DISABLED`: No caching—always refetch pages. - `READ_ONLY`: Reads from cache only; no new writes. - `WRITE_ONLY`: Writes to cache but doesn’t read existing data. - `BYPASS`: Skips reading cache for this crawl (though it might still write if set up that way). `[](#__codelineno-2-1) run_config = CrawlerRunConfig( [](#__codelineno-2-2) cache_mode=CacheMode.BYPASS [](#__codelineno-2-3) )` **Additional flags**: - `bypass_cache=True` acts like `CacheMode.BYPASS`. - `disable_cache=True` acts like `CacheMode.DISABLED`. - `no_cache_read=True` acts like `CacheMode.WRITE_ONLY`. - `no_cache_write=True` acts like `CacheMode.READ_ONLY`. * * * 3\. Content Processing & Selection ---------------------------------- ### 3.1 Text Processing `[](#__codelineno-3-1) run_config = CrawlerRunConfig( [](#__codelineno-3-2) word_count_threshold=10, # Ignore text blocks <10 words [](#__codelineno-3-3) only_text=False, # If True, tries to remove non-text elements [](#__codelineno-3-4) keep_data_attributes=False # Keep or discard data-* attributes [](#__codelineno-3-5) )` ### 3.2 Content Selection `[](#__codelineno-4-1) run_config = CrawlerRunConfig( [](#__codelineno-4-2) css_selector=".main-content", # Focus on .main-content region only [](#__codelineno-4-3) excluded_tags=["form", "nav"], # Remove entire tag blocks [](#__codelineno-4-4) remove_forms=True, # Specifically strip
elements [](#__codelineno-4-5) remove_overlay_elements=True, # Attempt to remove modals/popups [](#__codelineno-4-6) )` ### 3.3 Link Handling `[](#__codelineno-5-1) run_config = CrawlerRunConfig( [](#__codelineno-5-2) exclude_external_links=True, # Remove external links from final content [](#__codelineno-5-3) exclude_social_media_links=True, # Remove links to known social sites [](#__codelineno-5-4) exclude_domains=["ads.example.com"], # Exclude links to these domains [](#__codelineno-5-5) exclude_social_media_domains=["facebook.com","twitter.com"], # Extend the default list [](#__codelineno-5-6) )` ### 3.4 Media Filtering `[](#__codelineno-6-1) run_config = CrawlerRunConfig( [](#__codelineno-6-2) exclude_external_images=True # Strip images from other domains [](#__codelineno-6-3) )` * * * 4\. Page Navigation & Timing ---------------------------- ### 4.1 Basic Browser Flow `[](#__codelineno-7-1) run_config = CrawlerRunConfig( [](#__codelineno-7-2) wait_for="css:.dynamic-content", # Wait for .dynamic-content [](#__codelineno-7-3) delay_before_return_html=2.0, # Wait 2s before capturing final HTML [](#__codelineno-7-4) page_timeout=60000, # Navigation & script timeout (ms) [](#__codelineno-7-5) )` **Key Fields**: - `wait_for`: \- `"css:selector"` or \- `"js:() => boolean"` e.g. `js:() => document.querySelectorAll('.item').length > 10`. * `mean_delay` & `max_range`: define random delays for `arun_many()` calls. * `semaphore_count`: concurrency limit when crawling multiple URLs. ### 4.2 JavaScript Execution `[](#__codelineno-8-1) run_config = CrawlerRunConfig( [](#__codelineno-8-2) js_code=[ [](#__codelineno-8-3) "window.scrollTo(0, document.body.scrollHeight);", [](#__codelineno-8-4) "document.querySelector('.load-more')?.click();" [](#__codelineno-8-5) ], [](#__codelineno-8-6) js_only=False [](#__codelineno-8-7) )` * `js_code` can be a single string or a list of strings. * `js_only=True` means “I’m continuing in the same session with new JS steps, no new full navigation.” ### 4.3 Anti-Bot `[](#__codelineno-9-1) run_config = CrawlerRunConfig( [](#__codelineno-9-2) magic=True, [](#__codelineno-9-3) simulate_user=True, [](#__codelineno-9-4) override_navigator=True [](#__codelineno-9-5) )` \- `magic=True` tries multiple stealth features. \- `simulate_user=True` mimics mouse movements or random delays. \- `override_navigator=True` fakes some navigator properties (like user agent checks). * * * 5\. Session Management ---------------------- **`session_id`**: `[](#__codelineno-10-1) run_config = CrawlerRunConfig( [](#__codelineno-10-2) session_id="my_session123" [](#__codelineno-10-3) )` If re-used in subsequent `arun()` calls, the same tab/page context is continued (helpful for multi-step tasks or stateful browsing). * * * 6\. Screenshot, PDF & Media Options ----------------------------------- `[](#__codelineno-11-1) run_config = CrawlerRunConfig( [](#__codelineno-11-2) screenshot=True, # Grab a screenshot as base64 [](#__codelineno-11-3) screenshot_wait_for=1.0, # Wait 1s before capturing [](#__codelineno-11-4) pdf=True, # Also produce a PDF [](#__codelineno-11-5) image_description_min_word_threshold=5, # If analyzing alt text [](#__codelineno-11-6) image_score_threshold=3, # Filter out low-score images [](#__codelineno-11-7) )` **Where they appear**: - `result.screenshot` → Base64 screenshot string. - `result.pdf` → Byte array with PDF data. * * * 7\. Extraction Strategy ----------------------- **For advanced data extraction** (CSS/LLM-based), set `extraction_strategy`: `[](#__codelineno-12-1) run_config = CrawlerRunConfig( [](#__codelineno-12-2) extraction_strategy=my_css_or_llm_strategy [](#__codelineno-12-3) )` The extracted data will appear in `result.extracted_content`. * * * 8\. Comprehensive Example ------------------------- Below is a snippet combining many parameters: `[](#__codelineno-13-1) import asyncio [](#__codelineno-13-2) from crawl4ai import AsyncWebCrawler, CrawlerRunConfig, CacheMode [](#__codelineno-13-3) from crawl4ai.extraction_strategy import JsonCssExtractionStrategy [](#__codelineno-13-4) [](#__codelineno-13-5) async def main(): [](#__codelineno-13-6) # Example schema [](#__codelineno-13-7) schema = { [](#__codelineno-13-8) "name": "Articles", [](#__codelineno-13-9) "baseSelector": "article.post", [](#__codelineno-13-10) "fields": [ [](#__codelineno-13-11) {"name": "title", "selector": "h2", "type": "text"}, [](#__codelineno-13-12) {"name": "link", "selector": "a", "type": "attribute", "attribute": "href"} [](#__codelineno-13-13) ] [](#__codelineno-13-14) } [](#__codelineno-13-15) [](#__codelineno-13-16) run_config = CrawlerRunConfig( [](#__codelineno-13-17) # Core [](#__codelineno-13-18) verbose=True, [](#__codelineno-13-19) cache_mode=CacheMode.ENABLED, [](#__codelineno-13-20) [](#__codelineno-13-21) # Content [](#__codelineno-13-22) word_count_threshold=10, [](#__codelineno-13-23) css_selector="main.content", [](#__codelineno-13-24) excluded_tags=["nav", "footer"], [](#__codelineno-13-25) exclude_external_links=True, [](#__codelineno-13-26) [](#__codelineno-13-27) # Page & JS [](#__codelineno-13-28) js_code="document.querySelector('.show-more')?.click();", [](#__codelineno-13-29) wait_for="css:.loaded-block", [](#__codelineno-13-30) page_timeout=30000, [](#__codelineno-13-31) [](#__codelineno-13-32) # Extraction [](#__codelineno-13-33) extraction_strategy=JsonCssExtractionStrategy(schema), [](#__codelineno-13-34) [](#__codelineno-13-35) # Session [](#__codelineno-13-36) session_id="persistent_session", [](#__codelineno-13-37) [](#__codelineno-13-38) # Media [](#__codelineno-13-39) screenshot=True, [](#__codelineno-13-40) pdf=True, [](#__codelineno-13-41) [](#__codelineno-13-42) # Anti-bot [](#__codelineno-13-43) simulate_user=True, [](#__codelineno-13-44) magic=True, [](#__codelineno-13-45) ) [](#__codelineno-13-46) [](#__codelineno-13-47) async with AsyncWebCrawler() as crawler: [](#__codelineno-13-48) result = await crawler.arun("https://example.com/posts", config=run_config) [](#__codelineno-13-49) if result.success: [](#__codelineno-13-50) print("HTML length:", len(result.cleaned_html)) [](#__codelineno-13-51) print("Extraction JSON:", result.extracted_content) [](#__codelineno-13-52) if result.screenshot: [](#__codelineno-13-53) print("Screenshot length:", len(result.screenshot)) [](#__codelineno-13-54) if result.pdf: [](#__codelineno-13-55) print("PDF bytes length:", len(result.pdf)) [](#__codelineno-13-56) else: [](#__codelineno-13-57) print("Error:", result.error_message) [](#__codelineno-13-58) [](#__codelineno-13-59) if __name__ == "__main__": [](#__codelineno-13-60) asyncio.run(main())` **What we covered**: 1. **Crawling** the main content region, ignoring external links. 2\. Running **JavaScript** to click “.show-more”. 3. **Waiting** for “.loaded-block” to appear. 4\. Generating a **screenshot** & **PDF** of the final page. 5\. Extracting repeated “article.post” elements with a **CSS-based** extraction strategy. * * * 9\. Best Practices ------------------ 1. **Use `BrowserConfig` for global browser** settings (headless, user agent). 2. **Use `CrawlerRunConfig`** to handle the **specific** crawl needs: content filtering, caching, JS, screenshot, extraction, etc. 3\. Keep your **parameters consistent** in run configs—especially if you’re part of a large codebase with multiple crawls. 4. **Limit** large concurrency (`semaphore_count`) if the site or your system can’t handle it. 5\. For dynamic pages, set `js_code` or `scan_full_page` so you load all content. * * * 10\. Conclusion --------------- All parameters that used to be direct arguments to `arun()` now belong in **`CrawlerRunConfig`**. This approach: * Makes code **clearer** and **more maintainable**. * Minimizes confusion about which arguments affect global vs. per-crawl behavior. * Allows you to create **reusable** config objects for different pages or tasks. For a **full** reference, check out the [CrawlerRunConfig Docs](../parameters/) . Happy crawling with your **structured, flexible** config approach! * * * --- # 0.4.2 - Crawl4AI Documentation 🚀 Crawl4AI 0.4.2 Update: Smarter Crawling Just Got Easier (Dec 12, 2024) ------------------------------------------------------------------------- ### Hey Developers, I’m excited to share Crawl4AI 0.4.2—a major upgrade that makes crawling smarter, faster, and a whole lot more intuitive. I’ve packed in a bunch of new features to simplify your workflows and improve your experience. Let’s cut to the chase! * * * ### 🔧 **Configurable Browser and Crawler Behavior** You’ve asked for better control over how browsers and crawlers are configured, and now you’ve got it. With the new `BrowserConfig` and `CrawlerRunConfig` objects, you can set up your browser and crawling behavior exactly how you want. No more cluttering `arun` with a dozen arguments—just pass in your configs and go. **Example:** `[](#__codelineno-0-1) from crawl4ai import BrowserConfig, CrawlerRunConfig, AsyncWebCrawler [](#__codelineno-0-2) [](#__codelineno-0-3) browser_config = BrowserConfig(headless=True, viewport_width=1920, viewport_height=1080) [](#__codelineno-0-4) crawler_config = CrawlerRunConfig(cache_mode="BYPASS") [](#__codelineno-0-5) [](#__codelineno-0-6) async with AsyncWebCrawler(config=browser_config) as crawler: [](#__codelineno-0-7) result = await crawler.arun(url="https://example.com", config=crawler_config) [](#__codelineno-0-8) print(result.markdown[:500])` This setup is a game-changer for scalability, keeping your code clean and flexible as we add more parameters in the future. Remember: If you like to use the old way, you can still pass arguments directly to `arun` as before, no worries! * * * ### 🔐 **Streamlined Session Management** Here’s the big one: You can now pass local storage and cookies directly. Whether it’s setting values programmatically or importing a saved JSON state, managing sessions has never been easier. This is a must-have for authenticated crawls—just export your storage state once and reuse it effortlessly across runs. **Example:** 1. Open a browser, log in manually, and export the storage state. 2. Import the JSON file for seamless authenticated crawling: `[](#__codelineno-1-1) result = await crawler.arun( [](#__codelineno-1-2) url="https://example.com/protected", [](#__codelineno-1-3) storage_state="my_storage_state.json" [](#__codelineno-1-4) )` * * * ### 🔢 **Handling Large Pages: Supercharged Screenshots and PDF Conversion** Two big upgrades here: * **Blazing-fast long-page screenshots**: Turn extremely long web pages into clean, high-quality screenshots—without breaking a sweat. It’s optimized to handle large content without lag. * **Full-page PDF exports**: Now, you can also convert any page into a PDF with all the details intact. Perfect for archiving or sharing complex layouts. * * * ### 🔧 **Other Cool Stuff** * **Anti-bot enhancements**: Magic mode now handles overlays, user simulation, and anti-detection features like a pro. * **JavaScript execution**: Execute custom JS snippets to handle dynamic content. No more wrestling with endless page interactions. * * * ### 📊 **Performance Boosts and Dev-friendly Updates** * Faster rendering and viewport adjustments for better performance. * Improved cookie and local storage handling for seamless authentication. * Better debugging with detailed logs and actionable error messages. * * * ### 🔠 **Use Cases You’ll Love** 1. **Authenticated Crawls**: Login once, export your storage state, and reuse it across multiple requests without the headache. 2. **Long-page Screenshots**: Perfect for blogs, e-commerce pages, or any endless-scroll website. 3. **PDF Export**: Create professional-looking page PDFs in seconds. * * * ### Let’s Get Crawling Crawl4AI 0.4.2 is ready for you to download and try. I’m always looking for ways to improve, so don’t hold back—share your thoughts and feedback. Happy Crawling! 🚀 * * * --- # Strategies - Crawl4AI Documentation Extraction & Chunking Strategies API ==================================== This documentation covers the API reference for extraction and chunking strategies in Crawl4AI. Extraction Strategies --------------------- All extraction strategies inherit from the base `ExtractionStrategy` class and implement two key methods: - `extract(url: str, html: str) -> List[Dict[str, Any]]` - `run(url: str, sections: List[str]) -> List[Dict[str, Any]]` ### LLMExtractionStrategy Used for extracting structured data using Language Models. `[](#__codelineno-0-1) LLMExtractionStrategy( [](#__codelineno-0-2) # Required Parameters [](#__codelineno-0-3) provider: str = DEFAULT_PROVIDER, # LLM provider (e.g., "ollama/llama2") [](#__codelineno-0-4) api_token: Optional[str] = None, # API token [](#__codelineno-0-5) [](#__codelineno-0-6) # Extraction Configuration [](#__codelineno-0-7) instruction: str = None, # Custom extraction instruction [](#__codelineno-0-8) schema: Dict = None, # Pydantic model schema for structured data [](#__codelineno-0-9) extraction_type: str = "block", # "block" or "schema" [](#__codelineno-0-10) [](#__codelineno-0-11) # Chunking Parameters [](#__codelineno-0-12) chunk_token_threshold: int = 4000, # Maximum tokens per chunk [](#__codelineno-0-13) overlap_rate: float = 0.1, # Overlap between chunks [](#__codelineno-0-14) word_token_rate: float = 0.75, # Word to token conversion rate [](#__codelineno-0-15) apply_chunking: bool = True, # Enable/disable chunking [](#__codelineno-0-16) [](#__codelineno-0-17) # API Configuration [](#__codelineno-0-18) base_url: str = None, # Base URL for API [](#__codelineno-0-19) extra_args: Dict = {}, # Additional provider arguments [](#__codelineno-0-20) verbose: bool = False # Enable verbose logging [](#__codelineno-0-21) )` ### CosineStrategy Used for content similarity-based extraction and clustering. `[](#__codelineno-1-1) CosineStrategy( [](#__codelineno-1-2) # Content Filtering [](#__codelineno-1-3) semantic_filter: str = None, # Topic/keyword filter [](#__codelineno-1-4) word_count_threshold: int = 10, # Minimum words per cluster [](#__codelineno-1-5) sim_threshold: float = 0.3, # Similarity threshold [](#__codelineno-1-6) [](#__codelineno-1-7) # Clustering Parameters [](#__codelineno-1-8) max_dist: float = 0.2, # Maximum cluster distance [](#__codelineno-1-9) linkage_method: str = 'ward', # Clustering method [](#__codelineno-1-10) top_k: int = 3, # Top clusters to return [](#__codelineno-1-11) [](#__codelineno-1-12) # Model Configuration [](#__codelineno-1-13) model_name: str = 'sentence-transformers/all-MiniLM-L6-v2', # Embedding model [](#__codelineno-1-14) [](#__codelineno-1-15) verbose: bool = False # Enable verbose logging [](#__codelineno-1-16) )` ### JsonCssExtractionStrategy Used for CSS selector-based structured data extraction. `[](#__codelineno-2-1) JsonCssExtractionStrategy( [](#__codelineno-2-2) schema: Dict[str, Any], # Extraction schema [](#__codelineno-2-3) verbose: bool = False # Enable verbose logging [](#__codelineno-2-4) ) [](#__codelineno-2-5) [](#__codelineno-2-6) # Schema Structure [](#__codelineno-2-7) schema = { [](#__codelineno-2-8) "name": str, # Schema name [](#__codelineno-2-9) "baseSelector": str, # Base CSS selector [](#__codelineno-2-10) "fields": [ # List of fields to extract [](#__codelineno-2-11) { [](#__codelineno-2-12) "name": str, # Field name [](#__codelineno-2-13) "selector": str, # CSS selector [](#__codelineno-2-14) "type": str, # Field type: "text", "attribute", "html", "regex" [](#__codelineno-2-15) "attribute": str, # For type="attribute" [](#__codelineno-2-16) "pattern": str, # For type="regex" [](#__codelineno-2-17) "transform": str, # Optional: "lowercase", "uppercase", "strip" [](#__codelineno-2-18) "default": Any # Default value if extraction fails [](#__codelineno-2-19) } [](#__codelineno-2-20) ] [](#__codelineno-2-21) }` Chunking Strategies ------------------- All chunking strategies inherit from `ChunkingStrategy` and implement the `chunk(text: str) -> list` method. ### RegexChunking Splits text based on regex patterns. `[](#__codelineno-3-1) RegexChunking( [](#__codelineno-3-2) patterns: List[str] = None # Regex patterns for splitting [](#__codelineno-3-3) # Default: [r'\n\n'] [](#__codelineno-3-4) )` ### SlidingWindowChunking Creates overlapping chunks with a sliding window approach. `[](#__codelineno-4-1) SlidingWindowChunking( [](#__codelineno-4-2) window_size: int = 100, # Window size in words [](#__codelineno-4-3) step: int = 50 # Step size between windows [](#__codelineno-4-4) )` ### OverlappingWindowChunking Creates chunks with specified overlap. `[](#__codelineno-5-1) OverlappingWindowChunking( [](#__codelineno-5-2) window_size: int = 1000, # Chunk size in words [](#__codelineno-5-3) overlap: int = 100 # Overlap size in words [](#__codelineno-5-4) )` Usage Examples -------------- ### LLM Extraction `[](#__codelineno-6-1) from pydantic import BaseModel [](#__codelineno-6-2) from crawl4ai.extraction_strategy import LLMExtractionStrategy [](#__codelineno-6-3) [](#__codelineno-6-4) # Define schema [](#__codelineno-6-5) class Article(BaseModel): [](#__codelineno-6-6) title: str [](#__codelineno-6-7) content: str [](#__codelineno-6-8) author: str [](#__codelineno-6-9) [](#__codelineno-6-10) # Create strategy [](#__codelineno-6-11) strategy = LLMExtractionStrategy( [](#__codelineno-6-12) provider="ollama/llama2", [](#__codelineno-6-13) schema=Article.schema(), [](#__codelineno-6-14) instruction="Extract article details" [](#__codelineno-6-15) ) [](#__codelineno-6-16) [](#__codelineno-6-17) # Use with crawler [](#__codelineno-6-18) result = await crawler.arun( [](#__codelineno-6-19) url="https://example.com/article", [](#__codelineno-6-20) extraction_strategy=strategy [](#__codelineno-6-21) ) [](#__codelineno-6-22) [](#__codelineno-6-23) # Access extracted data [](#__codelineno-6-24) data = json.loads(result.extracted_content)` ### CSS Extraction `[](#__codelineno-7-1) from crawl4ai.extraction_strategy import JsonCssExtractionStrategy [](#__codelineno-7-2) [](#__codelineno-7-3) # Define schema [](#__codelineno-7-4) schema = { [](#__codelineno-7-5) "name": "Product List", [](#__codelineno-7-6) "baseSelector": ".product-card", [](#__codelineno-7-7) "fields": [ [](#__codelineno-7-8) { [](#__codelineno-7-9) "name": "title", [](#__codelineno-7-10) "selector": "h2.title", [](#__codelineno-7-11) "type": "text" [](#__codelineno-7-12) }, [](#__codelineno-7-13) { [](#__codelineno-7-14) "name": "price", [](#__codelineno-7-15) "selector": ".price", [](#__codelineno-7-16) "type": "text", [](#__codelineno-7-17) "transform": "strip" [](#__codelineno-7-18) }, [](#__codelineno-7-19) { [](#__codelineno-7-20) "name": "image", [](#__codelineno-7-21) "selector": "img", [](#__codelineno-7-22) "type": "attribute", [](#__codelineno-7-23) "attribute": "src" [](#__codelineno-7-24) } [](#__codelineno-7-25) ] [](#__codelineno-7-26) } [](#__codelineno-7-27) [](#__codelineno-7-28) # Create and use strategy [](#__codelineno-7-29) strategy = JsonCssExtractionStrategy(schema) [](#__codelineno-7-30) result = await crawler.arun( [](#__codelineno-7-31) url="https://example.com/products", [](#__codelineno-7-32) extraction_strategy=strategy [](#__codelineno-7-33) )` ### Content Chunking `[](#__codelineno-8-1) from crawl4ai.chunking_strategy import OverlappingWindowChunking [](#__codelineno-8-2) [](#__codelineno-8-3) # Create chunking strategy [](#__codelineno-8-4) chunker = OverlappingWindowChunking( [](#__codelineno-8-5) window_size=500, # 500 words per chunk [](#__codelineno-8-6) overlap=50 # 50 words overlap [](#__codelineno-8-7) ) [](#__codelineno-8-8) [](#__codelineno-8-9) # Use with extraction strategy [](#__codelineno-8-10) strategy = LLMExtractionStrategy( [](#__codelineno-8-11) provider="ollama/llama2", [](#__codelineno-8-12) chunking_strategy=chunker [](#__codelineno-8-13) ) [](#__codelineno-8-14) [](#__codelineno-8-15) result = await crawler.arun( [](#__codelineno-8-16) url="https://example.com/long-article", [](#__codelineno-8-17) extraction_strategy=strategy [](#__codelineno-8-18) )` Best Practices -------------- 1. **Choose the Right Strategy** - Use `LLMExtractionStrategy` for complex, unstructured content - Use `JsonCssExtractionStrategy` for well-structured HTML - Use `CosineStrategy` for content similarity and clustering 2. **Optimize Chunking** `[](#__codelineno-9-1) # For long documents [](#__codelineno-9-2) strategy = LLMExtractionStrategy( [](#__codelineno-9-3) chunk_token_threshold=2000, # Smaller chunks [](#__codelineno-9-4) overlap_rate=0.1 # 10% overlap [](#__codelineno-9-5) )` 3. **Handle Errors** `[](#__codelineno-10-1) try: [](#__codelineno-10-2) result = await crawler.arun( [](#__codelineno-10-3) url="https://example.com", [](#__codelineno-10-4) extraction_strategy=strategy [](#__codelineno-10-5) ) [](#__codelineno-10-6) if result.success: [](#__codelineno-10-7) content = json.loads(result.extracted_content) [](#__codelineno-10-8) except Exception as e: [](#__codelineno-10-9) print(f"Extraction failed: {e}")` 4. **Monitor Performance** `[](#__codelineno-11-1) strategy = CosineStrategy( [](#__codelineno-11-2) verbose=True, # Enable logging [](#__codelineno-11-3) word_count_threshold=20, # Filter short content [](#__codelineno-11-4) top_k=5 # Limit results [](#__codelineno-11-5) )` * * * --- # Browser & Crawler Config - Crawl4AI Documentation 1. **BrowserConfig** – Controlling the Browser ============================================== `BrowserConfig` focuses on **how** the browser is launched and behaves. This includes headless mode, proxies, user agents, and other environment tweaks. `[](#__codelineno-0-1) from crawl4ai import AsyncWebCrawler, BrowserConfig [](#__codelineno-0-2) [](#__codelineno-0-3) browser_cfg = BrowserConfig( [](#__codelineno-0-4) browser_type="chromium", [](#__codelineno-0-5) headless=True, [](#__codelineno-0-6) viewport_width=1280, [](#__codelineno-0-7) viewport_height=720, [](#__codelineno-0-8) proxy="http://user:pass@proxy:8080", [](#__codelineno-0-9) user_agent="Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 Chrome/116.0.0.0 Safari/537.36", [](#__codelineno-0-10) )` 1.1 Parameter Highlights ------------------------ | **Parameter** | **Type / Default** | **What It Does** | | --- | --- | --- | | **`browser_type`** | `"chromium"`, `"firefox"`, `"webkit"`
_(default: `"chromium"`)_ | Which browser engine to use. `"chromium"` is typical for many sites, `"firefox"` or `"webkit"` for specialized tests. | | **`headless`** | `bool` (default: `True`) | Headless means no visible UI. `False` is handy for debugging. | | **`viewport_width`** | `int` (default: `1080`) | Initial page width (in px). Useful for testing responsive layouts. | | **`viewport_height`** | `int` (default: `600`) | Initial page height (in px). | | **`proxy`** | `str` (default: `None`) | Single-proxy URL if you want all traffic to go through it, e.g. `"http://user:pass@proxy:8080"`. | | **`proxy_config`** | `dict` (default: `None`) | For advanced or multi-proxy needs, specify details like `{"server": "...", "username": "...", ...}`. | | **`use_persistent_context`** | `bool` (default: `False`) | If `True`, uses a **persistent** browser context (keep cookies, sessions across runs). Also sets `use_managed_browser=True`. | | **`user_data_dir`** | `str or None` (default: `None`) | Directory to store user data (profiles, cookies). Must be set if you want permanent sessions. | | **`ignore_https_errors`** | `bool` (default: `True`) | If `True`, continues despite invalid certificates (common in dev/staging). | | **`java_script_enabled`** | `bool` (default: `True`) | Disable if you want no JS overhead, or if only static content is needed. | | **`cookies`** | `list` (default: `[]`) | Pre-set cookies, each a dict like `{"name": "session", "value": "...", "url": "..."}`. | | **`headers`** | `dict` (default: `{}`) | Extra HTTP headers for every request, e.g. `{"Accept-Language": "en-US"}`. | | **`user_agent`** | `str` (default: Chrome-based UA) | Your custom or random user agent. `user_agent_mode="random"` can shuffle it. | | **`light_mode`** | `bool` (default: `False`) | Disables some background features for performance gains. | | **`text_mode`** | `bool` (default: `False`) | If `True`, tries to disable images/other heavy content for speed. | | **`use_managed_browser`** | `bool` (default: `False`) | For advanced “managed” interactions (debugging, CDP usage). Typically set automatically if persistent context is on. | | **`extra_args`** | `list` (default: `[]`) | Additional flags for the underlying browser process, e.g. `["--disable-extensions"]`. | **Tips**: - Set `headless=False` to visually **debug** how pages load or how interactions proceed. \- If you need **authentication** storage or repeated sessions, consider `use_persistent_context=True` and specify `user_data_dir`. \- For large pages, you might need a bigger `viewport_width` and `viewport_height` to handle dynamic content. * * * 2. **CrawlerRunConfig** – Controlling Each Crawl ================================================ While `BrowserConfig` sets up the **environment**, `CrawlerRunConfig` details **how** each **crawl operation** should behave: caching, content filtering, link or domain blocking, timeouts, JavaScript code, etc. `[](#__codelineno-1-1) from crawl4ai import AsyncWebCrawler, CrawlerRunConfig [](#__codelineno-1-2) [](#__codelineno-1-3) run_cfg = CrawlerRunConfig( [](#__codelineno-1-4) wait_for="css:.main-content", [](#__codelineno-1-5) word_count_threshold=15, [](#__codelineno-1-6) excluded_tags=["nav", "footer"], [](#__codelineno-1-7) exclude_external_links=True, [](#__codelineno-1-8) )` 2.1 Parameter Highlights ------------------------ We group them by category. ### A) **Content Processing** | **Parameter** | **Type / Default** | **What It Does** | | --- | --- | --- | | **`word_count_threshold`** | `int` (default: ~200) | Skips text blocks below X words. Helps ignore trivial sections. | | **`extraction_strategy`** | `ExtractionStrategy` (default: None) | If set, extracts structured data (CSS-based, LLM-based, etc.). | | **`markdown_generator`** | `MarkdownGenerationStrategy` (None) | If you want specialized markdown output (citations, filtering, chunking, etc.). | | **`content_filter`** | `RelevantContentFilter` (None) | Filters out irrelevant text blocks. E.g., `PruningContentFilter` or `BM25ContentFilter`. | | **`css_selector`** | `str` (None) | Retains only the part of the page matching this selector. | | **`excluded_tags`** | `list` (None) | Removes entire tags (e.g. `["script", "style"]`). | | **`excluded_selector`** | `str` (None) | Like `css_selector` but to exclude. E.g. `"#ads, .tracker"`. | | **`only_text`** | `bool` (False) | If `True`, tries to extract text-only content. | | **`prettiify`** | `bool` (False) | If `True`, beautifies final HTML (slower, purely cosmetic). | | **`keep_data_attributes`** | `bool` (False) | If `True`, preserve `data-*` attributes in cleaned HTML. | | **`remove_forms`** | `bool` (False) | If `True`, remove all `` elements. | * * * ### B) **Caching & Session** | **Parameter** | **Type / Default** | **What It Does** | | --- | --- | --- | | **`cache_mode`** | `CacheMode or None` | Controls how caching is handled (`ENABLED`, `BYPASS`, `DISABLED`, etc.). If `None`, typically defaults to `ENABLED`. | | **`session_id`** | `str or None` | Assign a unique ID to reuse a single browser session across multiple `arun()` calls. | | **`bypass_cache`** | `bool` (False) | If `True`, acts like `CacheMode.BYPASS`. | | **`disable_cache`** | `bool` (False) | If `True`, acts like `CacheMode.DISABLED`. | | **`no_cache_read`** | `bool` (False) | If `True`, acts like `CacheMode.WRITE_ONLY` (writes cache but never reads). | | **`no_cache_write`** | `bool` (False) | If `True`, acts like `CacheMode.READ_ONLY` (reads cache but never writes). | Use these for controlling whether you read or write from a local content cache. Handy for large batch crawls or repeated site visits. * * * ### C) **Page Navigation & Timing** | **Parameter** | **Type / Default** | **What It Does** | | --- | --- | --- | | **`wait_until`** | `str` (domcontentloaded) | Condition for navigation to “complete”. Often `"networkidle"` or `"domcontentloaded"`. | | **`page_timeout`** | `int` (60000 ms) | Timeout for page navigation or JS steps. Increase for slow sites. | | **`wait_for`** | `str or None` | Wait for a CSS (`"css:selector"`) or JS (`"js:() => bool"`) condition before content extraction. | | **`wait_for_images`** | `bool` (False) | Wait for images to load before finishing. Slows down if you only want text. | | **`delay_before_return_html`** | `float` (0.1) | Additional pause (seconds) before final HTML is captured. Good for last-second updates. | | **`mean_delay`** and **`max_range`** | `float` (0.1, 0.3) | If you call `arun_many()`, these define random delay intervals between crawls, helping avoid detection or rate limits. | | **`semaphore_count`** | `int` (5) | Max concurrency for `arun_many()`. Increase if you have resources for parallel crawls. | * * * ### D) **Page Interaction** | **Parameter** | **Type / Default** | **What It Does** | | --- | --- | --- | | **`js_code`** | `str or list[str]` (None) | JavaScript to run after load. E.g. `"document.querySelector('button')?.click();"`. | | **`js_only`** | `bool` (False) | If `True`, indicates we’re reusing an existing session and only applying JS. No full reload. | | **`ignore_body_visibility`** | `bool` (True) | Skip checking if `` is visible. Usually best to keep `True`. | | **`scan_full_page`** | `bool` (False) | If `True`, auto-scroll the page to load dynamic content (infinite scroll). | | **`scroll_delay`** | `float` (0.2) | Delay between scroll steps if `scan_full_page=True`. | | **`process_iframes`** | `bool` (False) | Inlines iframe content for single-page extraction. | | **`remove_overlay_elements`** | `bool` (False) | Removes potential modals/popups blocking the main content. | | **`simulate_user`** | `bool` (False) | Simulate user interactions (mouse movements) to avoid bot detection. | | **`override_navigator`** | `bool` (False) | Override `navigator` properties in JS for stealth. | | **`magic`** | `bool` (False) | Automatic handling of popups/consent banners. Experimental. | | **`adjust_viewport_to_content`** | `bool` (False) | Resizes viewport to match page content height. | If your page is a single-page app with repeated JS updates, set `js_only=True` in subsequent calls, plus a `session_id` for reusing the same tab. * * * ### E) **Media Handling** | **Parameter** | **Type / Default** | **What It Does** | | --- | --- | --- | | **`screenshot`** | `bool` (False) | Capture a screenshot (base64) in `result.screenshot`. | | **`screenshot_wait_for`** | `float or None` | Extra wait time before the screenshot. | | **`screenshot_height_threshold`** | `int` (~20000) | If the page is taller than this, alternate screenshot strategies are used. | | **`pdf`** | `bool` (False) | If `True`, returns a PDF in `result.pdf`. | | **`image_description_min_word_threshold`** | `int` (~50) | Minimum words for an image’s alt text or description to be considered valid. | | **`image_score_threshold`** | `int` (~3) | Filter out low-scoring images. The crawler scores images by relevance (size, context, etc.). | | **`exclude_external_images`** | `bool` (False) | Exclude images from other domains. | * * * ### F) **Link/Domain Handling** | **Parameter** | **Type / Default** | **What It Does** | | --- | --- | --- | | **`exclude_social_media_domains`** | `list` (e.g. Facebook/Twitter) | A default list can be extended. Any link to these domains is removed from final output. | | **`exclude_external_links`** | `bool` (False) | Removes all links pointing outside the current domain. | | **`exclude_social_media_links`** | `bool` (False) | Strips links specifically to social sites (like Facebook or Twitter). | | **`exclude_domains`** | `list` (\[\]) | Provide a custom list of domains to exclude (like `["ads.com", "trackers.io"]`). | Use these for link-level content filtering (often to keep crawls “internal” or to remove spammy domains). * * * ### G) **Debug & Logging** | **Parameter** | **Type / Default** | **What It Does** | | --- | --- | --- | | **`verbose`** | `bool` (True) | Prints logs detailing each step of crawling, interactions, or errors. | | **`log_console`** | `bool` (False) | Logs the page’s JavaScript console output if you want deeper JS debugging. | * * * 2.2 Example Usage ----------------- `[](#__codelineno-2-1) import asyncio [](#__codelineno-2-2) from crawl4ai import AsyncWebCrawler, BrowserConfig, CrawlerRunConfig, CacheMode [](#__codelineno-2-3) [](#__codelineno-2-4) async def main(): [](#__codelineno-2-5) # Configure the browser [](#__codelineno-2-6) browser_cfg = BrowserConfig( [](#__codelineno-2-7) headless=False, [](#__codelineno-2-8) viewport_width=1280, [](#__codelineno-2-9) viewport_height=720, [](#__codelineno-2-10) proxy="http://user:pass@myproxy:8080", [](#__codelineno-2-11) text_mode=True [](#__codelineno-2-12) ) [](#__codelineno-2-13) [](#__codelineno-2-14) # Configure the run [](#__codelineno-2-15) run_cfg = CrawlerRunConfig( [](#__codelineno-2-16) cache_mode=CacheMode.BYPASS, [](#__codelineno-2-17) session_id="my_session", [](#__codelineno-2-18) css_selector="main.article", [](#__codelineno-2-19) excluded_tags=["script", "style"], [](#__codelineno-2-20) exclude_external_links=True, [](#__codelineno-2-21) wait_for="css:.article-loaded", [](#__codelineno-2-22) screenshot=True [](#__codelineno-2-23) ) [](#__codelineno-2-24) [](#__codelineno-2-25) async with AsyncWebCrawler(config=browser_cfg) as crawler: [](#__codelineno-2-26) result = await crawler.arun( [](#__codelineno-2-27) url="https://example.com/news", [](#__codelineno-2-28) config=run_cfg [](#__codelineno-2-29) ) [](#__codelineno-2-30) if result.success: [](#__codelineno-2-31) print("Final cleaned_html length:", len(result.cleaned_html)) [](#__codelineno-2-32) if result.screenshot: [](#__codelineno-2-33) print("Screenshot captured (base64, length):", len(result.screenshot)) [](#__codelineno-2-34) else: [](#__codelineno-2-35) print("Crawl failed:", result.error_message) [](#__codelineno-2-36) [](#__codelineno-2-37) if __name__ == "__main__": [](#__codelineno-2-38) asyncio.run(main())` **What’s Happening**: - **`text_mode=True`** avoids loading images and other heavy resources, speeding up the crawl. \- We disable caching (`cache_mode=CacheMode.BYPASS`) to always fetch fresh content. \- We only keep `main.article` content by specifying `css_selector="main.article"`. \- We exclude external links (`exclude_external_links=True`). \- We do a quick screenshot (`screenshot=True`) before finishing. * * * 3\. Putting It All Together --------------------------- * **Use** `BrowserConfig` for **global** browser settings: engine, headless, proxy, user agent. * **Use** `CrawlerRunConfig` for each crawl’s **context**: how to filter content, handle caching, wait for dynamic elements, or run JS. * **Pass** both configs to `AsyncWebCrawler` (the `BrowserConfig`) and then to `arun()` (the `CrawlerRunConfig`). * * * --- # Browser & Crawler Config - Crawl4AI Documentation Browser & Crawler Configuration (Quick Overview) ================================================ Crawl4AI’s flexibility stems from two key classes: 1. **`BrowserConfig`** – Dictates **how** the browser is launched and behaves (e.g., headless or visible, proxy, user agent). 2. **`CrawlerRunConfig`** – Dictates **how** each **crawl** operates (e.g., caching, extraction, timeouts, JavaScript code to run, etc.). In most examples, you create **one** `BrowserConfig` for the entire crawler session, then pass a **fresh** or re-used `CrawlerRunConfig` whenever you call `arun()`. This tutorial shows the most commonly used parameters. If you need advanced or rarely used fields, see the [Configuration Parameters](../../api/parameters/) . * * * 1\. BrowserConfig Essentials ---------------------------- `[](#__codelineno-0-1) class BrowserConfig: [](#__codelineno-0-2) def __init__( [](#__codelineno-0-3) browser_type="chromium", [](#__codelineno-0-4) headless=True, [](#__codelineno-0-5) proxy_config=None, [](#__codelineno-0-6) viewport_width=1080, [](#__codelineno-0-7) viewport_height=600, [](#__codelineno-0-8) verbose=True, [](#__codelineno-0-9) use_persistent_context=False, [](#__codelineno-0-10) user_data_dir=None, [](#__codelineno-0-11) cookies=None, [](#__codelineno-0-12) headers=None, [](#__codelineno-0-13) user_agent=None, [](#__codelineno-0-14) text_mode=False, [](#__codelineno-0-15) light_mode=False, [](#__codelineno-0-16) extra_args=None, [](#__codelineno-0-17) # ... other advanced parameters omitted here [](#__codelineno-0-18) ): [](#__codelineno-0-19) ...` ### Key Fields to Note 1. **`browser_type`** \- Options: `"chromium"`, `"firefox"`, or `"webkit"`. \- Defaults to `"chromium"`. \- If you need a different engine, specify it here. 2. **`headless`** \- `True`: Runs the browser in headless mode (invisible browser). \- `False`: Runs the browser in visible mode, which helps with debugging. 3. **`proxy_config`** \- A dictionary with fields like: `[](#__codelineno-1-1) { [](#__codelineno-1-2) "server": "http://proxy.example.com:8080", [](#__codelineno-1-3) "username": "...", [](#__codelineno-1-4) "password": "..." [](#__codelineno-1-5) }` \- Leave as `None` if a proxy is not required. 4. **`viewport_width` & `viewport_height`**: \- The initial window size. \- Some sites behave differently with smaller or bigger viewports. 5. **`verbose`**: \- If `True`, prints extra logs. \- Handy for debugging. 6. **`use_persistent_context`**: \- If `True`, uses a **persistent** browser profile, storing cookies/local storage across runs. \- Typically also set `user_data_dir` to point to a folder. 7. **`cookies`** & **`headers`**: \- If you want to start with specific cookies or add universal HTTP headers, set them here. \- E.g. `cookies=[{"name": "session", "value": "abc123", "domain": "example.com"}]`. 8. **`user_agent`**: \- Custom User-Agent string. If `None`, a default is used. \- You can also set `user_agent_mode="random"` for randomization (if you want to fight bot detection). 9. **`text_mode`** & **`light_mode`**: \- `text_mode=True` disables images, possibly speeding up text-only crawls. \- `light_mode=True` turns off certain background features for performance. 10. **`extra_args`**: \- Additional flags for the underlying browser. \- E.g. `["--disable-extensions"]`. **Minimal Example**: `[](#__codelineno-2-1) from crawl4ai import AsyncWebCrawler, BrowserConfig [](#__codelineno-2-2) [](#__codelineno-2-3) browser_conf = BrowserConfig( [](#__codelineno-2-4) browser_type="firefox", [](#__codelineno-2-5) headless=False, [](#__codelineno-2-6) text_mode=True [](#__codelineno-2-7) ) [](#__codelineno-2-8) [](#__codelineno-2-9) async with AsyncWebCrawler(config=browser_conf) as crawler: [](#__codelineno-2-10) result = await crawler.arun("https://example.com") [](#__codelineno-2-11) print(result.markdown[:300])` * * * 2\. CrawlerRunConfig Essentials ------------------------------- `[](#__codelineno-3-1) class CrawlerRunConfig: [](#__codelineno-3-2) def __init__( [](#__codelineno-3-3) word_count_threshold=200, [](#__codelineno-3-4) extraction_strategy=None, [](#__codelineno-3-5) markdown_generator=None, [](#__codelineno-3-6) cache_mode=None, [](#__codelineno-3-7) js_code=None, [](#__codelineno-3-8) wait_for=None, [](#__codelineno-3-9) screenshot=False, [](#__codelineno-3-10) pdf=False, [](#__codelineno-3-11) verbose=True, [](#__codelineno-3-12) # ... other advanced parameters omitted [](#__codelineno-3-13) ): [](#__codelineno-3-14) ...` ### Key Fields to Note 1. **`word_count_threshold`**: \- The minimum word count before a block is considered. \- If your site has lots of short paragraphs or items, you can lower it. 2. **`extraction_strategy`**: \- Where you plug in JSON-based extraction (CSS, LLM, etc.). \- If `None`, no structured extraction is done (only raw/cleaned HTML + markdown). 3. **`markdown_generator`**: \- E.g., `DefaultMarkdownGenerator(...)`, controlling how HTML→Markdown conversion is done. \- If `None`, a default approach is used. 4. **`cache_mode`**: \- Controls caching behavior (`ENABLED`, `BYPASS`, `DISABLED`, etc.). \- If `None`, defaults to some level of caching or you can specify `CacheMode.ENABLED`. 5. **`js_code`**: \- A string or list of JS strings to execute. \- Great for “Load More” buttons or user interactions. 6. **`wait_for`**: \- A CSS or JS expression to wait for before extracting content. \- Common usage: `wait_for="css:.main-loaded"` or `wait_for="js:() => window.loaded === true"`. 7. **`screenshot`** & **`pdf`**: \- If `True`, captures a screenshot or PDF after the page is fully loaded. \- The results go to `result.screenshot` (base64) or `result.pdf` (bytes). 8. **`verbose`**: \- Logs additional runtime details. \- Overlaps with the browser’s verbosity if also set to `True` in `BrowserConfig`. **Minimal Example**: `[](#__codelineno-4-1) from crawl4ai import AsyncWebCrawler, CrawlerRunConfig [](#__codelineno-4-2) [](#__codelineno-4-3) crawl_conf = CrawlerRunConfig( [](#__codelineno-4-4) js_code="document.querySelector('button#loadMore')?.click()", [](#__codelineno-4-5) wait_for="css:.loaded-content", [](#__codelineno-4-6) screenshot=True [](#__codelineno-4-7) ) [](#__codelineno-4-8) [](#__codelineno-4-9) async with AsyncWebCrawler() as crawler: [](#__codelineno-4-10) result = await crawler.arun(url="https://example.com", config=crawl_conf) [](#__codelineno-4-11) print(result.screenshot[:100]) # Base64-encoded PNG snippet` * * * 3\. Putting It All Together --------------------------- In a typical scenario, you define **one** `BrowserConfig` for your crawler session, then create **one or more** `CrawlerRunConfig` depending on each call’s needs: `[](#__codelineno-5-1) import asyncio [](#__codelineno-5-2) from crawl4ai import AsyncWebCrawler, BrowserConfig, CrawlerRunConfig, CacheMode [](#__codelineno-5-3) from crawl4ai.extraction_strategy import JsonCssExtractionStrategy [](#__codelineno-5-4) [](#__codelineno-5-5) async def main(): [](#__codelineno-5-6) # 1) Browser config: headless, bigger viewport, no proxy [](#__codelineno-5-7) browser_conf = BrowserConfig( [](#__codelineno-5-8) headless=True, [](#__codelineno-5-9) viewport_width=1280, [](#__codelineno-5-10) viewport_height=720 [](#__codelineno-5-11) ) [](#__codelineno-5-12) [](#__codelineno-5-13) # 2) Example extraction strategy [](#__codelineno-5-14) schema = { [](#__codelineno-5-15) "name": "Articles", [](#__codelineno-5-16) "baseSelector": "div.article", [](#__codelineno-5-17) "fields": [ [](#__codelineno-5-18) {"name": "title", "selector": "h2", "type": "text"}, [](#__codelineno-5-19) {"name": "link", "selector": "a", "type": "attribute", "attribute": "href"} [](#__codelineno-5-20) ] [](#__codelineno-5-21) } [](#__codelineno-5-22) extraction = JsonCssExtractionStrategy(schema) [](#__codelineno-5-23) [](#__codelineno-5-24) # 3) Crawler run config: skip cache, use extraction [](#__codelineno-5-25) run_conf = CrawlerRunConfig( [](#__codelineno-5-26) extraction_strategy=extraction, [](#__codelineno-5-27) cache_mode=CacheMode.BYPASS [](#__codelineno-5-28) ) [](#__codelineno-5-29) [](#__codelineno-5-30) async with AsyncWebCrawler(config=browser_conf) as crawler: [](#__codelineno-5-31) # 4) Execute the crawl [](#__codelineno-5-32) result = await crawler.arun(url="https://example.com/news", config=run_conf) [](#__codelineno-5-33) [](#__codelineno-5-34) if result.success: [](#__codelineno-5-35) print("Extracted content:", result.extracted_content) [](#__codelineno-5-36) else: [](#__codelineno-5-37) print("Error:", result.error_message) [](#__codelineno-5-38) [](#__codelineno-5-39) if __name__ == "__main__": [](#__codelineno-5-40) asyncio.run(main())` * * * 4\. Next Steps -------------- For a **detailed list** of available parameters (including advanced ones), see: * [BrowserConfig and CrawlerRunConfig Reference](../../api/parameters/) You can explore topics like: * **Custom Hooks & Auth** (Inject JavaScript or handle login forms). * **Session Management** (Re-use pages, preserve state across multiple calls). * **Magic Mode** or **Identity-based Crawling** (Fight bot detection by simulating user behavior). * **Advanced Caching** (Fine-tune read/write cache modes). * * * 5\. Conclusion -------------- **BrowserConfig** and **CrawlerRunConfig** give you straightforward ways to define: * **Which** browser to launch, how it should run, and any proxy or user agent needs. * **How** each crawl should behave—caching, timeouts, JavaScript code, extraction strategies, etc. Use them together for **clear, maintainable** code, and when you need more specialized behavior, check out the advanced parameters in the [reference docs](../../api/parameters/) . Happy crawling! * * * --- # Crawler Result - Crawl4AI Documentation Crawl Result and Output ======================= When you call `arun()` on a page, Crawl4AI returns a **`CrawlResult`** object containing everything you might need—raw HTML, a cleaned version, optional screenshots or PDFs, structured extraction results, and more. This document explains those fields and how they map to different output types. * * * 1\. The `CrawlResult` Model --------------------------- Below is the core schema. Each field captures a different aspect of the crawl’s result: `[](#__codelineno-0-1) class MarkdownGenerationResult(BaseModel): [](#__codelineno-0-2) raw_markdown: str [](#__codelineno-0-3) markdown_with_citations: str [](#__codelineno-0-4) references_markdown: str [](#__codelineno-0-5) fit_markdown: Optional[str] = None [](#__codelineno-0-6) fit_html: Optional[str] = None [](#__codelineno-0-7) [](#__codelineno-0-8) class CrawlResult(BaseModel): [](#__codelineno-0-9) url: str [](#__codelineno-0-10) html: str [](#__codelineno-0-11) success: bool [](#__codelineno-0-12) cleaned_html: Optional[str] = None [](#__codelineno-0-13) media: Dict[str, List[Dict]] = {} [](#__codelineno-0-14) links: Dict[str, List[Dict]] = {} [](#__codelineno-0-15) downloaded_files: Optional[List[str]] = None [](#__codelineno-0-16) screenshot: Optional[str] = None [](#__codelineno-0-17) pdf : Optional[bytes] = None [](#__codelineno-0-18) markdown: Optional[Union[str, MarkdownGenerationResult]] = None [](#__codelineno-0-19) markdown_v2: Optional[MarkdownGenerationResult] = None [](#__codelineno-0-20) extracted_content: Optional[str] = None [](#__codelineno-0-21) metadata: Optional[dict] = None [](#__codelineno-0-22) error_message: Optional[str] = None [](#__codelineno-0-23) session_id: Optional[str] = None [](#__codelineno-0-24) response_headers: Optional[dict] = None [](#__codelineno-0-25) status_code: Optional[int] = None [](#__codelineno-0-26) ssl_certificate: Optional[SSLCertificate] = None [](#__codelineno-0-27) class Config: [](#__codelineno-0-28) arbitrary_types_allowed = True` ### Table: Key Fields in `CrawlResult` | Field (Name & Type) | Description | | --- | --- | | **url (`str`)** | The final or actual URL crawled (in case of redirects). | | **html (`str`)** | Original, unmodified page HTML. Good for debugging or custom processing. | | **success (`bool`)** | `True` if the crawl completed without major errors, else `False`. | | **cleaned\_html (`Optional[str]`)** | Sanitized HTML with scripts/styles removed; can exclude tags if configured via `excluded_tags` etc. | | **media (`Dict[str, List[Dict]]`)** | Extracted media info (images, audio, etc.), each with attributes like `src`, `alt`, `score`, etc. | | **links (`Dict[str, List[Dict]]`)** | Extracted link data, split by `internal` and `external`. Each link usually has `href`, `text`, etc. | | **downloaded\_files (`Optional[List[str]]`)** | If `accept_downloads=True` in `BrowserConfig`, this lists the filepaths of saved downloads. | | **screenshot (`Optional[str]`)** | Screenshot of the page (base64-encoded) if `screenshot=True`. | | **pdf (`Optional[bytes]`)** | PDF of the page if `pdf=True`. | | **markdown (`Optional[str or MarkdownGenerationResult]`)** | For now, `markdown_v2` holds a `MarkdownGenerationResult`. Over time, this will be consolidated into `markdown`. The generator can provide raw markdown, citations, references, and optionally `fit_markdown`. | | **markdown\_v2 (`Optional[MarkdownGenerationResult]`)** | Legacy field for detailed markdown output. This will be replaced by `markdown` soon. | | **extracted\_content (`Optional[str]`)** | The output of a structured extraction (CSS/LLM-based) stored as JSON string or other text. | | **metadata (`Optional[dict]`)** | Additional info about the crawl or extracted data. | | **error\_message (`Optional[str]`)** | If `success=False`, contains a short description of what went wrong. | | **session\_id (`Optional[str]`)** | The ID of the session used for multi-page or persistent crawling. | | **response\_headers (`Optional[dict]`)** | HTTP response headers, if captured. | | **status\_code (`Optional[int]`)** | HTTP status code (e.g., 200 for OK). | | **ssl\_certificate (`Optional[SSLCertificate]`)** | SSL certificate info if `fetch_ssl_certificate=True`. | * * * 2\. HTML Variants ----------------- ### `html`: Raw HTML Crawl4AI preserves the exact HTML as `result.html`. Useful for: * Debugging page issues or checking the original content. * Performing your own specialized parse if needed. ### `cleaned_html`: Sanitized If you specify any cleanup or exclusion parameters in `CrawlerRunConfig` (like `excluded_tags`, `remove_forms`, etc.), you’ll see the result here: `[](#__codelineno-1-1) config = CrawlerRunConfig( [](#__codelineno-1-2) excluded_tags=["form", "header", "footer"], [](#__codelineno-1-3) keep_data_attributes=False [](#__codelineno-1-4) ) [](#__codelineno-1-5) result = await crawler.arun("https://example.com", config=config) [](#__codelineno-1-6) print(result.cleaned_html) # Freed of forms, header, footer, data-* attributes` * * * 3\. Markdown Generation ----------------------- ### 3.1 `markdown_v2` (Legacy) vs `markdown` * **`markdown_v2`**: The current location for detailed markdown output, returning a **`MarkdownGenerationResult`** object. * **`markdown`**: Eventually, we’re merging these fields. For now, you might see `result.markdown_v2` used widely in code examples. **`MarkdownGenerationResult`** Fields: | Field | Description | | --- | --- | | **raw\_markdown** | The basic HTML→Markdown conversion. | | **markdown\_with\_citations** | Markdown including inline citations that reference links at the end. | | **references\_markdown** | The references/citations themselves (if `citations=True`). | | **fit\_markdown** | The filtered/“fit” markdown if a content filter was used. | | **fit\_html** | The filtered HTML that generated `fit_markdown`. | ### 3.2 Basic Example with a Markdown Generator `[](#__codelineno-2-1) from crawl4ai import AsyncWebCrawler, CrawlerRunConfig [](#__codelineno-2-2) from crawl4ai.markdown_generation_strategy import DefaultMarkdownGenerator [](#__codelineno-2-3) [](#__codelineno-2-4) config = CrawlerRunConfig( [](#__codelineno-2-5) markdown_generator=DefaultMarkdownGenerator( [](#__codelineno-2-6) options={"citations": True, "body_width": 80} # e.g. pass html2text style options [](#__codelineno-2-7) ) [](#__codelineno-2-8) ) [](#__codelineno-2-9) result = await crawler.arun(url="https://example.com", config=config) [](#__codelineno-2-10) [](#__codelineno-2-11) md_res = result.markdown_v2 # or eventually 'result.markdown' [](#__codelineno-2-12) print(md_res.raw_markdown[:500]) [](#__codelineno-2-13) print(md_res.markdown_with_citations) [](#__codelineno-2-14) print(md_res.references_markdown)` **Note**: If you use a filter like `PruningContentFilter`, you’ll get `fit_markdown` and `fit_html` as well. * * * 4\. Structured Extraction: `extracted_content` ---------------------------------------------- If you run a JSON-based extraction strategy (CSS, XPath, LLM, etc.), the structured data is **not** stored in `markdown`—it’s placed in **`result.extracted_content`** as a JSON string (or sometimes plain text). ### Example: CSS Extraction with `raw://` HTML `[](#__codelineno-3-1) import asyncio [](#__codelineno-3-2) import json [](#__codelineno-3-3) from crawl4ai import AsyncWebCrawler, CrawlerRunConfig, CacheMode [](#__codelineno-3-4) from crawl4ai.extraction_strategy import JsonCssExtractionStrategy [](#__codelineno-3-5) [](#__codelineno-3-6) async def main(): [](#__codelineno-3-7) schema = { [](#__codelineno-3-8) "name": "Example Items", [](#__codelineno-3-9) "baseSelector": "div.item", [](#__codelineno-3-10) "fields": [ [](#__codelineno-3-11) {"name": "title", "selector": "h2", "type": "text"}, [](#__codelineno-3-12) {"name": "link", "selector": "a", "type": "attribute", "attribute": "href"} [](#__codelineno-3-13) ] [](#__codelineno-3-14) } [](#__codelineno-3-15) raw_html = "

Item 1

Link 1
" [](#__codelineno-3-16) [](#__codelineno-3-17) async with AsyncWebCrawler() as crawler: [](#__codelineno-3-18) result = await crawler.arun( [](#__codelineno-3-19) url="raw://" + raw_html, [](#__codelineno-3-20) config=CrawlerRunConfig( [](#__codelineno-3-21) cache_mode=CacheMode.BYPASS, [](#__codelineno-3-22) extraction_strategy=JsonCssExtractionStrategy(schema) [](#__codelineno-3-23) ) [](#__codelineno-3-24) ) [](#__codelineno-3-25) data = json.loads(result.extracted_content) [](#__codelineno-3-26) print(data) [](#__codelineno-3-27) [](#__codelineno-3-28) if __name__ == "__main__": [](#__codelineno-3-29) asyncio.run(main())` Here: - `url="raw://..."` passes the HTML content directly, no network requests. \- The **CSS** extraction strategy populates `result.extracted_content` with the JSON array `[{"title": "...", "link": "..."}]`. * * * 5\. More Fields: Links, Media, and More --------------------------------------- ### 5.1 `links` A dictionary, typically with `"internal"` and `"external"` lists. Each entry might have `href`, `text`, `title`, etc. This is automatically captured if you haven’t disabled link extraction. `[](#__codelineno-4-1) print(result.links["internal"][:3]) # Show first 3 internal links` ### 5.2 `media` Similarly, a dictionary with `"images"`, `"audio"`, `"video"`, etc. Each item could include `src`, `alt`, `score`, and more, if your crawler is set to gather them. `[](#__codelineno-5-1) images = result.media.get("images", []) [](#__codelineno-5-2) for img in images: [](#__codelineno-5-3) print("Image URL:", img["src"], "Alt:", img.get("alt"))` ### 5.3 `screenshot` and `pdf` If you set `screenshot=True` or `pdf=True` in **`CrawlerRunConfig`**, then: * `result.screenshot` contains a base64-encoded PNG string. * `result.pdf` contains raw PDF bytes (you can write them to a file). `[](#__codelineno-6-1) with open("page.pdf", "wb") as f: [](#__codelineno-6-2) f.write(result.pdf)` ### 5.4 `ssl_certificate` If `fetch_ssl_certificate=True`, `result.ssl_certificate` holds details about the site’s SSL cert, such as issuer, validity dates, etc. * * * 6\. Accessing These Fields -------------------------- After you run: `[](#__codelineno-7-1) result = await crawler.arun(url="https://example.com", config=some_config)` Check any field: `[](#__codelineno-8-1) if result.success: [](#__codelineno-8-2) print(result.status_code, result.response_headers) [](#__codelineno-8-3) print("Links found:", len(result.links.get("internal", []))) [](#__codelineno-8-4) if result.markdown_v2: [](#__codelineno-8-5) print("Markdown snippet:", result.markdown_v2.raw_markdown[:200]) [](#__codelineno-8-6) if result.extracted_content: [](#__codelineno-8-7) print("Structured JSON:", result.extracted_content) [](#__codelineno-8-8) else: [](#__codelineno-8-9) print("Error:", result.error_message)` **Remember**: Use `result.markdown_v2` for now. It will eventually become `result.markdown`. * * * 7\. Next Steps -------------- * **Markdown Generation**: Dive deeper into how to configure `DefaultMarkdownGenerator` and various filters. * **Content Filtering**: Learn how to use `BM25ContentFilter` and `PruningContentFilter`. * **Session & Hooks**: If you want to manipulate the page or preserve state across multiple `arun()` calls, see the hooking or session docs. * **LLM Extraction**: For complex or unstructured content requiring AI-driven parsing, check the LLM-based strategies doc. **Enjoy** exploring all that `CrawlResult` offers—whether you need raw HTML, sanitized output, markdown, or fully structured data, Crawl4AI has you covered! * * * --- # Docker Deployment - Crawl4AI Documentation Docker Deployment ================= Crawl4AI provides official Docker images for easy deployment and scalability. This guide covers installation, configuration, and usage of Crawl4AI in Docker environments. Quick Start 🚀 -------------- Pull and run the basic version: `[](#__codelineno-0-1) # Basic run without security [](#__codelineno-0-2) docker pull unclecode/crawl4ai:basic [](#__codelineno-0-3) docker run -p 11235:11235 unclecode/crawl4ai:basic [](#__codelineno-0-4) [](#__codelineno-0-5) # Run with API security enabled [](#__codelineno-0-6) docker run -p 11235:11235 -e CRAWL4AI_API_TOKEN=your_secret_token unclecode/crawl4ai:basic` Running with Docker Compose 🐳 ------------------------------ ### Use Docker Compose (From Local Dockerfile or Docker Hub) Crawl4AI provides flexibility to use Docker Compose for managing your containerized services. You can either build the image locally from the provided `Dockerfile` or use the pre-built image from Docker Hub. ### **Option 1: Using Docker Compose to Build Locally** If you want to build the image locally, use the provided `docker-compose.local.yml` file. `[](#__codelineno-1-1) docker-compose -f docker-compose.local.yml up -d` This will: 1. Build the Docker image from the provided `Dockerfile`. 2. Start the container and expose it on `http://localhost:11235`. * * * ### **Option 2: Using Docker Compose with Pre-Built Image from Hub** If you prefer using the pre-built image on Docker Hub, use the `docker-compose.hub.yml` file. `[](#__codelineno-2-1) docker-compose -f docker-compose.hub.yml up -d` This will: 1. Pull the pre-built image `unclecode/crawl4ai:basic` (or `all`, depending on your configuration). 2. Start the container and expose it on `http://localhost:11235`. * * * ### **Stopping the Running Services** To stop the services started via Docker Compose, you can use: `[](#__codelineno-3-1) docker-compose -f docker-compose.local.yml down [](#__codelineno-3-2) # OR [](#__codelineno-3-3) docker-compose -f docker-compose.hub.yml down` If the containers don’t stop and the application is still running, check the running containers: `[](#__codelineno-4-1) docker ps` Find the `CONTAINER ID` of the running service and stop it forcefully: `[](#__codelineno-5-1) docker stop ` * * * ### **Debugging with Docker Compose** * **Check Logs**: To view the container logs: `[](#__codelineno-6-1) docker-compose -f docker-compose.local.yml logs -f` * **Remove Orphaned Containers**: If the service is still running unexpectedly: `[](#__codelineno-7-1) docker-compose -f docker-compose.local.yml down --remove-orphans` * **Manually Remove Network**: If the network is still in use: `[](#__codelineno-8-1) docker network ls [](#__codelineno-8-2) docker network rm crawl4ai_default` * * * ### Why Use Docker Compose? Docker Compose is the recommended way to deploy Crawl4AI because: 1. It simplifies multi-container setups. 2. Allows you to define environment variables, resources, and ports in a single file. 3. Makes it easier to switch between local development and production-ready images. For example, your `docker-compose.yml` could include API keys, token settings, and memory limits, making deployment quick and consistent. API Security 🔒 --------------- ### Understanding CRAWL4AI\_API\_TOKEN The `CRAWL4AI_API_TOKEN` provides optional security for your Crawl4AI instance: * If `CRAWL4AI_API_TOKEN` is set: All API endpoints (except `/health`) require authentication * If `CRAWL4AI_API_TOKEN` is not set: The API is publicly accessible `[](#__codelineno-9-1) # Secured Instance [](#__codelineno-9-2) docker run -p 11235:11235 -e CRAWL4AI_API_TOKEN=your_secret_token unclecode/crawl4ai:all [](#__codelineno-9-3) [](#__codelineno-9-4) # Unsecured Instance [](#__codelineno-9-5) docker run -p 11235:11235 unclecode/crawl4ai:all` ### Making API Calls For secured instances, include the token in all requests: `[](#__codelineno-10-1) import requests [](#__codelineno-10-2) [](#__codelineno-10-3) # Setup headers if token is being used [](#__codelineno-10-4) api_token = "your_secret_token" # Same token set in CRAWL4AI_API_TOKEN [](#__codelineno-10-5) headers = {"Authorization": f"Bearer {api_token}"} if api_token else {} [](#__codelineno-10-6) [](#__codelineno-10-7) # Making authenticated requests [](#__codelineno-10-8) response = requests.post( [](#__codelineno-10-9) "http://localhost:11235/crawl", [](#__codelineno-10-10) headers=headers, [](#__codelineno-10-11) json={ [](#__codelineno-10-12) "urls": "https://example.com", [](#__codelineno-10-13) "priority": 10 [](#__codelineno-10-14) } [](#__codelineno-10-15) ) [](#__codelineno-10-16) [](#__codelineno-10-17) # Checking task status [](#__codelineno-10-18) task_id = response.json()["task_id"] [](#__codelineno-10-19) status = requests.get( [](#__codelineno-10-20) f"http://localhost:11235/task/{task_id}", [](#__codelineno-10-21) headers=headers [](#__codelineno-10-22) )` ### Using with Docker Compose In your `docker-compose.yml`: `[](#__codelineno-11-1) services: [](#__codelineno-11-2) crawl4ai: [](#__codelineno-11-3) image: unclecode/crawl4ai:all [](#__codelineno-11-4) environment: [](#__codelineno-11-5) - CRAWL4AI_API_TOKEN=${CRAWL4AI_API_TOKEN:-} # Optional [](#__codelineno-11-6) # ... other configuration` Then either: 1. Set in `.env` file: `[](#__codelineno-12-1) CRAWL4AI_API_TOKEN=your_secret_token` 1. Or set via command line: `[](#__codelineno-13-1) CRAWL4AI_API_TOKEN=your_secret_token docker-compose up` > **Security Note**: If you enable the API token, make sure to keep it secure and never commit it to version control. The token will be required for all API endpoints except the health check endpoint (`/health`). Configuration Options 🔧 ------------------------ ### Environment Variables You can configure the service using environment variables: `[](#__codelineno-14-1) # Basic configuration [](#__codelineno-14-2) docker run -p 11235:11235 \ [](#__codelineno-14-3) -e MAX_CONCURRENT_TASKS=5 \ [](#__codelineno-14-4) unclecode/crawl4ai:all [](#__codelineno-14-5) [](#__codelineno-14-6) # With security and LLM support [](#__codelineno-14-7) docker run -p 11235:11235 \ [](#__codelineno-14-8) -e CRAWL4AI_API_TOKEN=your_secret_token \ [](#__codelineno-14-9) -e OPENAI_API_KEY=sk-... \ [](#__codelineno-14-10) -e ANTHROPIC_API_KEY=sk-ant-... \ [](#__codelineno-14-11) unclecode/crawl4ai:all` ### Using Docker Compose (Recommended) 🐳 Create a `docker-compose.yml`: `[](#__codelineno-15-1) version: '3.8' [](#__codelineno-15-2) [](#__codelineno-15-3) services: [](#__codelineno-15-4) crawl4ai: [](#__codelineno-15-5) image: unclecode/crawl4ai:all [](#__codelineno-15-6) ports: [](#__codelineno-15-7) - "11235:11235" [](#__codelineno-15-8) environment: [](#__codelineno-15-9) - CRAWL4AI_API_TOKEN=${CRAWL4AI_API_TOKEN:-} # Optional API security [](#__codelineno-15-10) - MAX_CONCURRENT_TASKS=5 [](#__codelineno-15-11) # LLM Provider Keys [](#__codelineno-15-12) - OPENAI_API_KEY=${OPENAI_API_KEY:-} [](#__codelineno-15-13) - ANTHROPIC_API_KEY=${ANTHROPIC_API_KEY:-} [](#__codelineno-15-14) volumes: [](#__codelineno-15-15) - /dev/shm:/dev/shm [](#__codelineno-15-16) deploy: [](#__codelineno-15-17) resources: [](#__codelineno-15-18) limits: [](#__codelineno-15-19) memory: 4G [](#__codelineno-15-20) reservations: [](#__codelineno-15-21) memory: 1G` You can run it in two ways: 1. Using environment variables directly: `[](#__codelineno-16-1) CRAWL4AI_API_TOKEN=secret123 OPENAI_API_KEY=sk-... docker-compose up` 2. Using a `.env` file (recommended): Create a `.env` file in the same directory: `[](#__codelineno-17-1) # API Security (optional) [](#__codelineno-17-2) CRAWL4AI_API_TOKEN=your_secret_token [](#__codelineno-17-3) [](#__codelineno-17-4) # LLM Provider Keys [](#__codelineno-17-5) OPENAI_API_KEY=sk-... [](#__codelineno-17-6) ANTHROPIC_API_KEY=sk-ant-... [](#__codelineno-17-7) [](#__codelineno-17-8) # Other Configuration [](#__codelineno-17-9) MAX_CONCURRENT_TASKS=5` Then simply run: `[](#__codelineno-18-1) docker-compose up` ### Testing the Deployment 🧪 `[](#__codelineno-19-1) import requests [](#__codelineno-19-2) [](#__codelineno-19-3) # For unsecured instances [](#__codelineno-19-4) def test_unsecured(): [](#__codelineno-19-5) # Health check [](#__codelineno-19-6) health = requests.get("http://localhost:11235/health") [](#__codelineno-19-7) print("Health check:", health.json()) [](#__codelineno-19-8) [](#__codelineno-19-9) # Basic crawl [](#__codelineno-19-10) response = requests.post( [](#__codelineno-19-11) "http://localhost:11235/crawl", [](#__codelineno-19-12) json={ [](#__codelineno-19-13) "urls": "https://www.nbcnews.com/business", [](#__codelineno-19-14) "priority": 10 [](#__codelineno-19-15) } [](#__codelineno-19-16) ) [](#__codelineno-19-17) task_id = response.json()["task_id"] [](#__codelineno-19-18) print("Task ID:", task_id) [](#__codelineno-19-19) [](#__codelineno-19-20) # For secured instances [](#__codelineno-19-21) def test_secured(api_token): [](#__codelineno-19-22) headers = {"Authorization": f"Bearer {api_token}"} [](#__codelineno-19-23) [](#__codelineno-19-24) # Basic crawl with authentication [](#__codelineno-19-25) response = requests.post( [](#__codelineno-19-26) "http://localhost:11235/crawl", [](#__codelineno-19-27) headers=headers, [](#__codelineno-19-28) json={ [](#__codelineno-19-29) "urls": "https://www.nbcnews.com/business", [](#__codelineno-19-30) "priority": 10 [](#__codelineno-19-31) } [](#__codelineno-19-32) ) [](#__codelineno-19-33) task_id = response.json()["task_id"] [](#__codelineno-19-34) print("Task ID:", task_id)` ### LLM Extraction Example 🤖 When you've configured your LLM provider keys (via environment variables or `.env`), you can use LLM extraction: `[](#__codelineno-20-1) request = { [](#__codelineno-20-2) "urls": "https://example.com", [](#__codelineno-20-3) "extraction_config": { [](#__codelineno-20-4) "type": "llm", [](#__codelineno-20-5) "params": { [](#__codelineno-20-6) "provider": "openai/gpt-4", [](#__codelineno-20-7) "instruction": "Extract main topics from the page" [](#__codelineno-20-8) } [](#__codelineno-20-9) } [](#__codelineno-20-10) } [](#__codelineno-20-11) [](#__codelineno-20-12) # Make the request (add headers if using API security) [](#__codelineno-20-13) response = requests.post("http://localhost:11235/crawl", json=request)` > **Note**: Remember to add `.env` to your `.gitignore` to keep your API keys secure! Usage Examples 📝 ----------------- ### Basic Crawling `[](#__codelineno-21-1) request = { [](#__codelineno-21-2) "urls": "https://www.nbcnews.com/business", [](#__codelineno-21-3) "priority": 10 [](#__codelineno-21-4) } [](#__codelineno-21-5) [](#__codelineno-21-6) response = requests.post("http://localhost:11235/crawl", json=request) [](#__codelineno-21-7) task_id = response.json()["task_id"] [](#__codelineno-21-8) [](#__codelineno-21-9) # Get results [](#__codelineno-21-10) result = requests.get(f"http://localhost:11235/task/{task_id}")` ### Structured Data Extraction `[](#__codelineno-22-1) schema = { [](#__codelineno-22-2) "name": "Crypto Prices", [](#__codelineno-22-3) "baseSelector": ".cds-tableRow-t45thuk", [](#__codelineno-22-4) "fields": [ [](#__codelineno-22-5) { [](#__codelineno-22-6) "name": "crypto", [](#__codelineno-22-7) "selector": "td:nth-child(1) h2", [](#__codelineno-22-8) "type": "text", [](#__codelineno-22-9) }, [](#__codelineno-22-10) { [](#__codelineno-22-11) "name": "price", [](#__codelineno-22-12) "selector": "td:nth-child(2)", [](#__codelineno-22-13) "type": "text", [](#__codelineno-22-14) } [](#__codelineno-22-15) ], [](#__codelineno-22-16) } [](#__codelineno-22-17) [](#__codelineno-22-18) request = { [](#__codelineno-22-19) "urls": "https://www.coinbase.com/explore", [](#__codelineno-22-20) "extraction_config": { [](#__codelineno-22-21) "type": "json_css", [](#__codelineno-22-22) "params": {"schema": schema} [](#__codelineno-22-23) } [](#__codelineno-22-24) }` ### Dynamic Content Handling `[](#__codelineno-23-1) request = { [](#__codelineno-23-2) "urls": "https://www.nbcnews.com/business", [](#__codelineno-23-3) "js_code": [ [](#__codelineno-23-4) "const loadMoreButton = Array.from(document.querySelectorAll('button')).find(button => button.textContent.includes('Load More')); loadMoreButton && loadMoreButton.click();" [](#__codelineno-23-5) ], [](#__codelineno-23-6) "wait_for": "article.tease-card:nth-child(10)" [](#__codelineno-23-7) }` ### AI-Powered Extraction (Full Version) `[](#__codelineno-24-1) request = { [](#__codelineno-24-2) "urls": "https://www.nbcnews.com/business", [](#__codelineno-24-3) "extraction_config": { [](#__codelineno-24-4) "type": "cosine", [](#__codelineno-24-5) "params": { [](#__codelineno-24-6) "semantic_filter": "business finance economy", [](#__codelineno-24-7) "word_count_threshold": 10, [](#__codelineno-24-8) "max_dist": 0.2, [](#__codelineno-24-9) "top_k": 3 [](#__codelineno-24-10) } [](#__codelineno-24-11) } [](#__codelineno-24-12) }` Platform-Specific Instructions 💻 --------------------------------- ### macOS `[](#__codelineno-25-1) docker pull unclecode/crawl4ai:basic [](#__codelineno-25-2) docker run -p 11235:11235 unclecode/crawl4ai:basic` ### Ubuntu `[](#__codelineno-26-1) # Basic version [](#__codelineno-26-2) docker pull unclecode/crawl4ai:basic [](#__codelineno-26-3) docker run -p 11235:11235 unclecode/crawl4ai:basic [](#__codelineno-26-4) [](#__codelineno-26-5) # With GPU support [](#__codelineno-26-6) docker pull unclecode/crawl4ai:gpu [](#__codelineno-26-7) docker run --gpus all -p 11235:11235 unclecode/crawl4ai:gpu` ### Windows (PowerShell) `[](#__codelineno-27-1) docker pull unclecode/crawl4ai:basic [](#__codelineno-27-2) docker run -p 11235:11235 unclecode/crawl4ai:basic` Testing 🧪 ---------- Save this as `test_docker.py`: `[](#__codelineno-28-1) import requests [](#__codelineno-28-2) import json [](#__codelineno-28-3) import time [](#__codelineno-28-4) import sys [](#__codelineno-28-5) [](#__codelineno-28-6) class Crawl4AiTester: [](#__codelineno-28-7) def __init__(self, base_url: str = "http://localhost:11235"): [](#__codelineno-28-8) self.base_url = base_url [](#__codelineno-28-9) [](#__codelineno-28-10) def submit_and_wait(self, request_data: dict, timeout: int = 300) -> dict: [](#__codelineno-28-11) # Submit crawl job [](#__codelineno-28-12) response = requests.post(f"{self.base_url}/crawl", json=request_data) [](#__codelineno-28-13) task_id = response.json()["task_id"] [](#__codelineno-28-14) print(f"Task ID: {task_id}") [](#__codelineno-28-15) [](#__codelineno-28-16) # Poll for result [](#__codelineno-28-17) start_time = time.time() [](#__codelineno-28-18) while True: [](#__codelineno-28-19) if time.time() - start_time > timeout: [](#__codelineno-28-20) raise TimeoutError(f"Task {task_id} timeout") [](#__codelineno-28-21) [](#__codelineno-28-22) result = requests.get(f"{self.base_url}/task/{task_id}") [](#__codelineno-28-23) status = result.json() [](#__codelineno-28-24) [](#__codelineno-28-25) if status["status"] == "completed": [](#__codelineno-28-26) return status [](#__codelineno-28-27) [](#__codelineno-28-28) time.sleep(2) [](#__codelineno-28-29) [](#__codelineno-28-30) def test_deployment(): [](#__codelineno-28-31) tester = Crawl4AiTester() [](#__codelineno-28-32) [](#__codelineno-28-33) # Test basic crawl [](#__codelineno-28-34) request = { [](#__codelineno-28-35) "urls": "https://www.nbcnews.com/business", [](#__codelineno-28-36) "priority": 10 [](#__codelineno-28-37) } [](#__codelineno-28-38) [](#__codelineno-28-39) result = tester.submit_and_wait(request) [](#__codelineno-28-40) print("Basic crawl successful!") [](#__codelineno-28-41) print(f"Content length: {len(result['result']['markdown'])}") [](#__codelineno-28-42) [](#__codelineno-28-43) if __name__ == "__main__": [](#__codelineno-28-44) test_deployment()` Advanced Configuration ⚙️ ------------------------- ### Crawler Parameters The `crawler_params` field allows you to configure the browser instance and crawling behavior. Here are key parameters you can use: `[](#__codelineno-29-1) request = { [](#__codelineno-29-2) "urls": "https://example.com", [](#__codelineno-29-3) "crawler_params": { [](#__codelineno-29-4) # Browser Configuration [](#__codelineno-29-5) "headless": True, # Run in headless mode [](#__codelineno-29-6) "browser_type": "chromium", # chromium/firefox/webkit [](#__codelineno-29-7) "user_agent": "custom-agent", # Custom user agent [](#__codelineno-29-8) "proxy": "http://proxy:8080", # Proxy configuration [](#__codelineno-29-9) [](#__codelineno-29-10) # Performance & Behavior [](#__codelineno-29-11) "page_timeout": 30000, # Page load timeout (ms) [](#__codelineno-29-12) "verbose": True, # Enable detailed logging [](#__codelineno-29-13) "semaphore_count": 5, # Concurrent request limit [](#__codelineno-29-14) [](#__codelineno-29-15) # Anti-Detection Features [](#__codelineno-29-16) "simulate_user": True, # Simulate human behavior [](#__codelineno-29-17) "magic": True, # Advanced anti-detection [](#__codelineno-29-18) "override_navigator": True, # Override navigator properties [](#__codelineno-29-19) [](#__codelineno-29-20) # Session Management [](#__codelineno-29-21) "user_data_dir": "./browser-data", # Browser profile location [](#__codelineno-29-22) "use_managed_browser": True, # Use persistent browser [](#__codelineno-29-23) } [](#__codelineno-29-24) }` ### Extra Parameters The `extra` field allows passing additional parameters directly to the crawler's `arun` function: `[](#__codelineno-30-1) request = { [](#__codelineno-30-2) "urls": "https://example.com", [](#__codelineno-30-3) "extra": { [](#__codelineno-30-4) "word_count_threshold": 10, # Min words per block [](#__codelineno-30-5) "only_text": True, # Extract only text [](#__codelineno-30-6) "bypass_cache": True, # Force fresh crawl [](#__codelineno-30-7) "process_iframes": True, # Include iframe content [](#__codelineno-30-8) } [](#__codelineno-30-9) }` ### Complete Examples 1. **Advanced News Crawling** `[](#__codelineno-31-1) request = { [](#__codelineno-31-2) "urls": "https://www.nbcnews.com/business", [](#__codelineno-31-3) "crawler_params": { [](#__codelineno-31-4) "headless": True, [](#__codelineno-31-5) "page_timeout": 30000, [](#__codelineno-31-6) "remove_overlay_elements": True # Remove popups [](#__codelineno-31-7) }, [](#__codelineno-31-8) "extra": { [](#__codelineno-31-9) "word_count_threshold": 50, # Longer content blocks [](#__codelineno-31-10) "bypass_cache": True # Fresh content [](#__codelineno-31-11) }, [](#__codelineno-31-12) "css_selector": ".article-body" [](#__codelineno-31-13) }` 2. **Anti-Detection Configuration** `[](#__codelineno-32-1) request = { [](#__codelineno-32-2) "urls": "https://example.com", [](#__codelineno-32-3) "crawler_params": { [](#__codelineno-32-4) "simulate_user": True, [](#__codelineno-32-5) "magic": True, [](#__codelineno-32-6) "override_navigator": True, [](#__codelineno-32-7) "user_agent": "Mozilla/5.0 ...", [](#__codelineno-32-8) "headers": { [](#__codelineno-32-9) "Accept-Language": "en-US,en;q=0.9" [](#__codelineno-32-10) } [](#__codelineno-32-11) } [](#__codelineno-32-12) }` 3. **LLM Extraction with Custom Parameters** `[](#__codelineno-33-1) request = { [](#__codelineno-33-2) "urls": "https://openai.com/pricing", [](#__codelineno-33-3) "extraction_config": { [](#__codelineno-33-4) "type": "llm", [](#__codelineno-33-5) "params": { [](#__codelineno-33-6) "provider": "openai/gpt-4", [](#__codelineno-33-7) "schema": pricing_schema [](#__codelineno-33-8) } [](#__codelineno-33-9) }, [](#__codelineno-33-10) "crawler_params": { [](#__codelineno-33-11) "verbose": True, [](#__codelineno-33-12) "page_timeout": 60000 [](#__codelineno-33-13) }, [](#__codelineno-33-14) "extra": { [](#__codelineno-33-15) "word_count_threshold": 1, [](#__codelineno-33-16) "only_text": True [](#__codelineno-33-17) } [](#__codelineno-33-18) }` 4. **Session-Based Dynamic Content** `[](#__codelineno-34-1) request = { [](#__codelineno-34-2) "urls": "https://example.com", [](#__codelineno-34-3) "crawler_params": { [](#__codelineno-34-4) "session_id": "dynamic_session", [](#__codelineno-34-5) "headless": False, [](#__codelineno-34-6) "page_timeout": 60000 [](#__codelineno-34-7) }, [](#__codelineno-34-8) "js_code": ["window.scrollTo(0, document.body.scrollHeight);"], [](#__codelineno-34-9) "wait_for": "js:() => document.querySelectorAll('.item').length > 10", [](#__codelineno-34-10) "extra": { [](#__codelineno-34-11) "delay_before_return_html": 2.0 [](#__codelineno-34-12) } [](#__codelineno-34-13) }` 5. **Screenshot with Custom Timing** `[](#__codelineno-35-1) request = { [](#__codelineno-35-2) "urls": "https://example.com", [](#__codelineno-35-3) "screenshot": True, [](#__codelineno-35-4) "crawler_params": { [](#__codelineno-35-5) "headless": True, [](#__codelineno-35-6) "screenshot_wait_for": ".main-content" [](#__codelineno-35-7) }, [](#__codelineno-35-8) "extra": { [](#__codelineno-35-9) "delay_before_return_html": 3.0 [](#__codelineno-35-10) } [](#__codelineno-35-11) }` ### Parameter Reference Table | Category | Parameter | Type | Description | | --- | --- | --- | --- | | Browser | headless | bool | Run browser in headless mode | | Browser | browser\_type | str | Browser engine selection | | Browser | user\_agent | str | Custom user agent string | | Network | proxy | str | Proxy server URL | | Network | headers | dict | Custom HTTP headers | | Timing | page\_timeout | int | Page load timeout (ms) | | Timing | delay\_before\_return\_html | float | Wait before capture | | Anti-Detection | simulate\_user | bool | Human behavior simulation | | Anti-Detection | magic | bool | Advanced protection | | Session | session\_id | str | Browser session ID | | Session | user\_data\_dir | str | Profile directory | | Content | word\_count\_threshold | int | Minimum words per block | | Content | only\_text | bool | Text-only extraction | | Content | process\_iframes | bool | Include iframe content | | Debug | verbose | bool | Detailed logging | | Debug | log\_console | bool | Browser console logs | Troubleshooting 🔍 ------------------ ### Common Issues 1. **Connection Refused** `[](#__codelineno-36-1) Error: Connection refused at localhost:11235` Solution: Ensure the container is running and ports are properly mapped. 2. **Resource Limits** `[](#__codelineno-37-1) Error: No available slots` Solution: Increase MAX\_CONCURRENT\_TASKS or container resources. 3. **GPU Access** `[](#__codelineno-38-1) Error: GPU not found` Solution: Ensure proper NVIDIA drivers and use `--gpus all` flag. ### Debug Mode Access container for debugging: `[](#__codelineno-39-1) docker run -it --entrypoint /bin/bash unclecode/crawl4ai:all` View container logs: `[](#__codelineno-40-1) docker logs [container_id]` Best Practices 🌟 ----------------- 1. **Resource Management** - Set appropriate memory and CPU limits - Monitor resource usage via health endpoint - Use basic version for simple crawling tasks 2. **Scaling** - Use multiple containers for high load - Implement proper load balancing - Monitor performance metrics 3. **Security** - Use environment variables for sensitive data - Implement proper network isolation - Regular security updates API Reference 📚 ---------------- ### Health Check `[](#__codelineno-41-1) GET /health` ### Submit Crawl Task `[](#__codelineno-42-1) POST /crawl [](#__codelineno-42-2) Content-Type: application/json [](#__codelineno-42-3) [](#__codelineno-42-4) { [](#__codelineno-42-5) "urls": "string or array", [](#__codelineno-42-6) "extraction_config": { [](#__codelineno-42-7) "type": "basic|llm|cosine|json_css", [](#__codelineno-42-8) "params": {} [](#__codelineno-42-9) }, [](#__codelineno-42-10) "priority": 1-10, [](#__codelineno-42-11) "ttl": 3600 [](#__codelineno-42-12) }` ### Get Task Status `[](#__codelineno-43-1) GET /task/{task_id}` For more details, visit the [official documentation](https://docs.crawl4ai.com/) . * * * --- # Markdown Generation - Crawl4AI Documentation Markdown Generation Basics ========================== One of Crawl4AI’s core features is generating **clean, structured markdown** from web pages. Originally built to solve the problem of extracting only the “actual” content and discarding boilerplate or noise, Crawl4AI’s markdown system remains one of its biggest draws for AI workflows. In this tutorial, you’ll learn: 1. How to configure the **Default Markdown Generator** 2. How **content filters** (BM25 or Pruning) help you refine markdown and discard junk 3. The difference between raw markdown (`result.markdown`) and filtered markdown (`fit_markdown`) > **Prerequisites** > \- You’ve completed or read [AsyncWebCrawler Basics](../simple-crawling/) > to understand how to run a simple crawl. > \- You know how to configure `CrawlerRunConfig`. * * * 1\. Quick Example ----------------- Here’s a minimal code snippet that uses the **DefaultMarkdownGenerator** with no additional filtering: `[](#__codelineno-0-1) import asyncio [](#__codelineno-0-2) from crawl4ai import AsyncWebCrawler, CrawlerRunConfig [](#__codelineno-0-3) from crawl4ai.markdown_generation_strategy import DefaultMarkdownGenerator [](#__codelineno-0-4) [](#__codelineno-0-5) async def main(): [](#__codelineno-0-6) config = CrawlerRunConfig( [](#__codelineno-0-7) markdown_generator=DefaultMarkdownGenerator() [](#__codelineno-0-8) ) [](#__codelineno-0-9) async with AsyncWebCrawler() as crawler: [](#__codelineno-0-10) result = await crawler.arun("https://example.com", config=config) [](#__codelineno-0-11) [](#__codelineno-0-12) if result.success: [](#__codelineno-0-13) print("Raw Markdown Output:\n") [](#__codelineno-0-14) print(result.markdown) # The unfiltered markdown from the page [](#__codelineno-0-15) else: [](#__codelineno-0-16) print("Crawl failed:", result.error_message) [](#__codelineno-0-17) [](#__codelineno-0-18) if __name__ == "__main__": [](#__codelineno-0-19) asyncio.run(main())` **What’s happening?** \- `CrawlerRunConfig( markdown_generator = DefaultMarkdownGenerator() )` instructs Crawl4AI to convert the final HTML into markdown at the end of each crawl. \- The resulting markdown is accessible via `result.markdown`. * * * 2\. How Markdown Generation Works --------------------------------- ### 2.1 HTML-to-Text Conversion (Forked & Modified) Under the hood, **DefaultMarkdownGenerator** uses a specialized HTML-to-text approach that: * Preserves headings, code blocks, bullet points, etc. * Removes extraneous tags (scripts, styles) that don’t add meaningful content. * Can optionally generate references for links or skip them altogether. A set of **options** (passed as a dict) allows you to customize precisely how HTML converts to markdown. These map to standard html2text-like configuration plus your own enhancements (e.g., ignoring internal links, preserving certain tags verbatim, or adjusting line widths). ### 2.2 Link Citations & References By default, the generator can convert `` elements into `[text][1]` citations, then place the actual links at the bottom of the document. This is handy for research workflows that demand references in a structured manner. ### 2.3 Optional Content Filters Before or after the HTML-to-Markdown step, you can apply a **content filter** (like BM25 or Pruning) to reduce noise and produce a “fit\_markdown”—a heavily pruned version focusing on the page’s main text. We’ll cover these filters shortly. * * * 3\. Configuring the Default Markdown Generator ---------------------------------------------- You can tweak the output by passing an `options` dict to `DefaultMarkdownGenerator`. For example: `[](#__codelineno-1-1) from crawl4ai.markdown_generation_strategy import DefaultMarkdownGenerator [](#__codelineno-1-2) from crawl4ai import AsyncWebCrawler, CrawlerRunConfig [](#__codelineno-1-3) [](#__codelineno-1-4) async def main(): [](#__codelineno-1-5) # Example: ignore all links, don't escape HTML, and wrap text at 80 characters [](#__codelineno-1-6) md_generator = DefaultMarkdownGenerator( [](#__codelineno-1-7) options={ [](#__codelineno-1-8) "ignore_links": True, [](#__codelineno-1-9) "escape_html": False, [](#__codelineno-1-10) "body_width": 80 [](#__codelineno-1-11) } [](#__codelineno-1-12) ) [](#__codelineno-1-13) [](#__codelineno-1-14) config = CrawlerRunConfig( [](#__codelineno-1-15) markdown_generator=md_generator [](#__codelineno-1-16) ) [](#__codelineno-1-17) [](#__codelineno-1-18) async with AsyncWebCrawler() as crawler: [](#__codelineno-1-19) result = await crawler.arun("https://example.com/docs", config=config) [](#__codelineno-1-20) if result.success: [](#__codelineno-1-21) print("Markdown:\n", result.markdown[:500]) # Just a snippet [](#__codelineno-1-22) else: [](#__codelineno-1-23) print("Crawl failed:", result.error_message) [](#__codelineno-1-24) [](#__codelineno-1-25) if __name__ == "__main__": [](#__codelineno-1-26) import asyncio [](#__codelineno-1-27) asyncio.run(main())` Some commonly used `options`: * **`ignore_links`** (bool): Whether to remove all hyperlinks in the final markdown. * **`ignore_images`** (bool): Remove all `![image]()` references. * **`escape_html`** (bool): Turn HTML entities into text (default is often `True`). * **`body_width`** (int): Wrap text at N characters. `0` or `None` means no wrapping. * **`skip_internal_links`** (bool): If `True`, omit `#localAnchors` or internal links referencing the same page. * **`include_sup_sub`** (bool): Attempt to handle `` / `` in a more readable way. * * * 4\. Content Filters ------------------- **Content filters** selectively remove or rank sections of text before turning them into Markdown. This is especially helpful if your page has ads, nav bars, or other clutter you don’t want. ### 4.1 BM25ContentFilter If you have a **search query**, BM25 is a good choice: `[](#__codelineno-2-1) from crawl4ai.markdown_generation_strategy import DefaultMarkdownGenerator [](#__codelineno-2-2) from crawl4ai.content_filter_strategy import BM25ContentFilter [](#__codelineno-2-3) from crawl4ai import CrawlerRunConfig [](#__codelineno-2-4) [](#__codelineno-2-5) bm25_filter = BM25ContentFilter( [](#__codelineno-2-6) user_query="machine learning", [](#__codelineno-2-7) bm25_threshold=1.2, [](#__codelineno-2-8) use_stemming=True [](#__codelineno-2-9) ) [](#__codelineno-2-10) [](#__codelineno-2-11) md_generator = DefaultMarkdownGenerator( [](#__codelineno-2-12) content_filter=bm25_filter, [](#__codelineno-2-13) options={"ignore_links": True} [](#__codelineno-2-14) ) [](#__codelineno-2-15) [](#__codelineno-2-16) config = CrawlerRunConfig(markdown_generator=md_generator)` * **`user_query`**: The term you want to focus on. BM25 tries to keep only content blocks relevant to that query. * **`bm25_threshold`**: Raise it to keep fewer blocks; lower it to keep more. * **`use_stemming`**: If `True`, variations of words match (e.g., “learn,” “learning,” “learnt”). **No query provided?** BM25 tries to glean a context from page metadata, or you can simply treat it as a scorched-earth approach that discards text with low generic score. Realistically, you want to supply a query for best results. ### 4.2 PruningContentFilter If you **don’t** have a specific query, or if you just want a robust “junk remover,” use `PruningContentFilter`. It analyzes text density, link density, HTML structure, and known patterns (like “nav,” “footer”) to systematically prune extraneous or repetitive sections. `[](#__codelineno-3-1) from crawl4ai.content_filter_strategy import PruningContentFilter [](#__codelineno-3-2) [](#__codelineno-3-3) prune_filter = PruningContentFilter( [](#__codelineno-3-4) threshold=0.5, [](#__codelineno-3-5) threshold_type="fixed", # or "dynamic" [](#__codelineno-3-6) min_word_threshold=50 [](#__codelineno-3-7) )` * **`threshold`**: Score boundary. Blocks below this score get removed. * **`threshold_type`**: * `"fixed"`: Straight comparison (`score >= threshold` keeps the block). * `"dynamic"`: The filter adjusts threshold in a data-driven manner. * **`min_word_threshold`**: Discard blocks under N words as likely too short or unhelpful. **When to Use PruningContentFilter** \- You want a broad cleanup without a user query. \- The page has lots of repeated sidebars, footers, or disclaimers that hamper text extraction. * * * 5\. Using Fit Markdown ---------------------- When a content filter is active, the library produces two forms of markdown inside `result.markdown_v2` or (if using the simplified field) `result.markdown`: 1. **`raw_markdown`**: The full unfiltered markdown. 2. **`fit_markdown`**: A “fit” version where the filter has removed or trimmed noisy segments. **Note**: > In earlier examples, you may see references to `result.markdown_v2`. Depending on your library version, you might access `result.markdown`, `result.markdown_v2`, or an object named `MarkdownGenerationResult`. The idea is the same: you’ll have a raw version and a filtered (“fit”) version if a filter is used. `[](#__codelineno-4-1) import asyncio [](#__codelineno-4-2) from crawl4ai import AsyncWebCrawler, CrawlerRunConfig [](#__codelineno-4-3) from crawl4ai.markdown_generation_strategy import DefaultMarkdownGenerator [](#__codelineno-4-4) from crawl4ai.content_filter_strategy import PruningContentFilter [](#__codelineno-4-5) [](#__codelineno-4-6) async def main(): [](#__codelineno-4-7) config = CrawlerRunConfig( [](#__codelineno-4-8) markdown_generator=DefaultMarkdownGenerator( [](#__codelineno-4-9) content_filter=PruningContentFilter(threshold=0.6), [](#__codelineno-4-10) options={"ignore_links": True} [](#__codelineno-4-11) ) [](#__codelineno-4-12) ) [](#__codelineno-4-13) async with AsyncWebCrawler() as crawler: [](#__codelineno-4-14) result = await crawler.arun("https://news.example.com/tech", config=config) [](#__codelineno-4-15) if result.success: [](#__codelineno-4-16) print("Raw markdown:\n", result.markdown) [](#__codelineno-4-17) [](#__codelineno-4-18) # If a filter is used, we also have .fit_markdown: [](#__codelineno-4-19) md_object = result.markdown_v2 # or your equivalent [](#__codelineno-4-20) print("Filtered markdown:\n", md_object.fit_markdown) [](#__codelineno-4-21) else: [](#__codelineno-4-22) print("Crawl failed:", result.error_message) [](#__codelineno-4-23) [](#__codelineno-4-24) if __name__ == "__main__": [](#__codelineno-4-25) asyncio.run(main())` * * * 6\. The `MarkdownGenerationResult` Object ----------------------------------------- If your library stores detailed markdown output in an object like `MarkdownGenerationResult`, you’ll see fields such as: * **`raw_markdown`**: The direct HTML-to-markdown transformation (no filtering). * **`markdown_with_citations`**: A version that moves links to reference-style footnotes. * **`references_markdown`**: A separate string or section containing the gathered references. * **`fit_markdown`**: The filtered markdown if you used a content filter. * **`fit_html`**: The corresponding HTML snippet used to generate `fit_markdown` (helpful for debugging or advanced usage). **Example**: `[](#__codelineno-5-1) md_obj = result.markdown_v2 # your library’s naming may vary [](#__codelineno-5-2) print("RAW:\n", md_obj.raw_markdown) [](#__codelineno-5-3) print("CITED:\n", md_obj.markdown_with_citations) [](#__codelineno-5-4) print("REFERENCES:\n", md_obj.references_markdown) [](#__codelineno-5-5) print("FIT:\n", md_obj.fit_markdown)` **Why Does This Matter?** \- You can supply `raw_markdown` to an LLM if you want the entire text. \- Or feed `fit_markdown` into a vector database to reduce token usage. \- `references_markdown` can help you keep track of link provenance. * * * Below is a **revised section** under “Combining Filters (BM25 + Pruning)” that demonstrates how you can run **two** passes of content filtering without re-crawling, by taking the HTML (or text) from a first pass and feeding it into the second filter. It uses real code patterns from the snippet you provided for **BM25ContentFilter**, which directly accepts **HTML** strings (and can also handle plain text with minimal adaptation). * * * 7\. Combining Filters (BM25 + Pruning) in Two Passes ---------------------------------------------------- You might want to **prune out** noisy boilerplate first (with `PruningContentFilter`), and then **rank what’s left** against a user query (with `BM25ContentFilter`). You don’t have to crawl the page twice. Instead: 1. **First pass**: Apply `PruningContentFilter` directly to the raw HTML from `result.html` (the crawler’s downloaded HTML). 2. **Second pass**: Take the pruned HTML (or text) from step 1, and feed it into `BM25ContentFilter`, focusing on a user query. ### Two-Pass Example `[](#__codelineno-6-1) import asyncio [](#__codelineno-6-2) from crawl4ai import AsyncWebCrawler, CrawlerRunConfig [](#__codelineno-6-3) from crawl4ai.content_filter_strategy import PruningContentFilter, BM25ContentFilter [](#__codelineno-6-4) from bs4 import BeautifulSoup [](#__codelineno-6-5) [](#__codelineno-6-6) async def main(): [](#__codelineno-6-7) # 1. Crawl with minimal or no markdown generator, just get raw HTML [](#__codelineno-6-8) config = CrawlerRunConfig( [](#__codelineno-6-9) # If you only want raw HTML, you can skip passing a markdown_generator [](#__codelineno-6-10) # or provide one but focus on .html in this example [](#__codelineno-6-11) ) [](#__codelineno-6-12) [](#__codelineno-6-13) async with AsyncWebCrawler() as crawler: [](#__codelineno-6-14) result = await crawler.arun("https://example.com/tech-article", config=config) [](#__codelineno-6-15) [](#__codelineno-6-16) if not result.success or not result.html: [](#__codelineno-6-17) print("Crawl failed or no HTML content.") [](#__codelineno-6-18) return [](#__codelineno-6-19) [](#__codelineno-6-20) raw_html = result.html [](#__codelineno-6-21) [](#__codelineno-6-22) # 2. First pass: PruningContentFilter on raw HTML [](#__codelineno-6-23) pruning_filter = PruningContentFilter(threshold=0.5, min_word_threshold=50) [](#__codelineno-6-24) [](#__codelineno-6-25) # filter_content returns a list of "text chunks" or cleaned HTML sections [](#__codelineno-6-26) pruned_chunks = pruning_filter.filter_content(raw_html) [](#__codelineno-6-27) # This list is basically pruned content blocks, presumably in HTML or text form [](#__codelineno-6-28) [](#__codelineno-6-29) # For demonstration, let's combine these chunks back into a single HTML-like string [](#__codelineno-6-30) # or you could do further processing. It's up to your pipeline design. [](#__codelineno-6-31) pruned_html = "\n".join(pruned_chunks) [](#__codelineno-6-32) [](#__codelineno-6-33) # 3. Second pass: BM25ContentFilter with a user query [](#__codelineno-6-34) bm25_filter = BM25ContentFilter( [](#__codelineno-6-35) user_query="machine learning", [](#__codelineno-6-36) bm25_threshold=1.2, [](#__codelineno-6-37) language="english" [](#__codelineno-6-38) ) [](#__codelineno-6-39) [](#__codelineno-6-40) # returns a list of text chunks [](#__codelineno-6-41) bm25_chunks = bm25_filter.filter_content(pruned_html) [](#__codelineno-6-42) [](#__codelineno-6-43) if not bm25_chunks: [](#__codelineno-6-44) print("Nothing matched the BM25 query after pruning.") [](#__codelineno-6-45) return [](#__codelineno-6-46) [](#__codelineno-6-47) # 4. Combine or display final results [](#__codelineno-6-48) final_text = "\n---\n".join(bm25_chunks) [](#__codelineno-6-49) [](#__codelineno-6-50) print("==== PRUNED OUTPUT (first pass) ====") [](#__codelineno-6-51) print(pruned_html[:500], "... (truncated)") # preview [](#__codelineno-6-52) [](#__codelineno-6-53) print("\n==== BM25 OUTPUT (second pass) ====") [](#__codelineno-6-54) print(final_text[:500], "... (truncated)") [](#__codelineno-6-55) [](#__codelineno-6-56) if __name__ == "__main__": [](#__codelineno-6-57) asyncio.run(main())` ### What’s Happening? 1. **Raw HTML**: We crawl once and store the raw HTML in `result.html`. 2. **PruningContentFilter**: Takes HTML + optional parameters. It extracts blocks of text or partial HTML, removing headings/sections deemed “noise.” It returns a **list of text chunks**. 3. **Combine or Transform**: We join these pruned chunks back into a single HTML-like string. (Alternatively, you could store them in a list for further logic—whatever suits your pipeline.) 4. **BM25ContentFilter**: We feed the pruned string into `BM25ContentFilter` with a user query. This second pass further narrows the content to chunks relevant to “machine learning.” **No Re-Crawling**: We used `raw_html` from the first pass, so there’s no need to run `arun()` again—**no second network request**. ### Tips & Variations * **Plain Text vs. HTML**: If your pruned output is mostly text, BM25 can still handle it; just keep in mind it expects a valid string input. If you supply partial HTML (like `"

some text

"`), it will parse it as HTML. * **Chaining in a Single Pipeline**: If your code supports it, you can chain multiple filters automatically. Otherwise, manual two-pass filtering (as shown) is straightforward. * **Adjust Thresholds**: If you see too much or too little text in step one, tweak `threshold=0.5` or `min_word_threshold=50`. Similarly, `bm25_threshold=1.2` can be raised/lowered for more or fewer chunks in step two. ### One-Pass Combination? If your codebase or pipeline design allows applying multiple filters in one pass, you could do so. But often it’s simpler—and more transparent—to run them sequentially, analyzing each step’s result. **Bottom Line**: By **manually chaining** your filtering logic in two passes, you get powerful incremental control over the final content. First, remove “global” clutter with Pruning, then refine further with BM25-based query relevance—without incurring a second network crawl. * * * 8\. Common Pitfalls & Tips -------------------------- 1. **No Markdown Output?** \- Make sure the crawler actually retrieved HTML. If the site is heavily JS-based, you may need to enable dynamic rendering or wait for elements. \- Check if your content filter is too aggressive. Lower thresholds or disable the filter to see if content reappears. 2. **Performance Considerations** \- Very large pages with multiple filters can be slower. Consider `cache_mode` to avoid re-downloading. \- If your final use case is LLM ingestion, consider summarizing further or chunking big texts. 3. **Take Advantage of `fit_markdown`** \- Great for RAG pipelines, semantic search, or any scenario where extraneous boilerplate is unwanted. \- Still verify the textual quality—some sites have crucial data in footers or sidebars. 4. **Adjusting `html2text` Options** \- If you see lots of raw HTML slipping into the text, turn on `escape_html`. \- If code blocks look messy, experiment with `mark_code` or `handle_code_in_pre`. * * * 9\. Summary & Next Steps ------------------------ In this **Markdown Generation Basics** tutorial, you learned to: * Configure the **DefaultMarkdownGenerator** with HTML-to-text options. * Use **BM25ContentFilter** for query-specific extraction or **PruningContentFilter** for general noise removal. * Distinguish between raw and filtered markdown (`fit_markdown`). * Leverage the `MarkdownGenerationResult` object to handle different forms of output (citations, references, etc.). Now you can produce high-quality Markdown from any website, focusing on exactly the content you need—an essential step for powering AI models, summarization pipelines, or knowledge-base queries. **Last Updated**: 2025-01-01 * * * --- # Overview - Crawl4AI Documentation Overview of Some Important Advanced Features ============================================ (Proxy, PDF, Screenshot, SSL, Headers, & Storage State) Crawl4AI offers multiple power-user features that go beyond simple crawling. This tutorial covers: 1. **Proxy Usage** 2. **Capturing PDFs & Screenshots** 3. **Handling SSL Certificates** 4. **Custom Headers** 5. **Session Persistence & Local Storage** > **Prerequisites** > \- You have a basic grasp of [AsyncWebCrawler Basics](../../core/simple-crawling/) > > \- You know how to run or configure your Python environment with Playwright installed * * * 1\. Proxy Usage --------------- If you need to route your crawl traffic through a proxy—whether for IP rotation, geo-testing, or privacy—Crawl4AI supports it via `BrowserConfig.proxy_config`. `[](#__codelineno-0-1) import asyncio [](#__codelineno-0-2) from crawl4ai import AsyncWebCrawler, BrowserConfig, CrawlerRunConfig [](#__codelineno-0-3) [](#__codelineno-0-4) async def main(): [](#__codelineno-0-5) browser_cfg = BrowserConfig( [](#__codelineno-0-6) proxy_config={ [](#__codelineno-0-7) "server": "http://proxy.example.com:8080", [](#__codelineno-0-8) "username": "myuser", [](#__codelineno-0-9) "password": "mypass", [](#__codelineno-0-10) }, [](#__codelineno-0-11) headless=True [](#__codelineno-0-12) ) [](#__codelineno-0-13) crawler_cfg = CrawlerRunConfig( [](#__codelineno-0-14) verbose=True [](#__codelineno-0-15) ) [](#__codelineno-0-16) [](#__codelineno-0-17) async with AsyncWebCrawler(config=browser_cfg) as crawler: [](#__codelineno-0-18) result = await crawler.arun( [](#__codelineno-0-19) url="https://www.whatismyip.com/", [](#__codelineno-0-20) config=crawler_cfg [](#__codelineno-0-21) ) [](#__codelineno-0-22) if result.success: [](#__codelineno-0-23) print("[OK] Page fetched via proxy.") [](#__codelineno-0-24) print("Page HTML snippet:", result.html[:200]) [](#__codelineno-0-25) else: [](#__codelineno-0-26) print("[ERROR]", result.error_message) [](#__codelineno-0-27) [](#__codelineno-0-28) if __name__ == "__main__": [](#__codelineno-0-29) asyncio.run(main())` **Key Points** \- **`proxy_config`** expects a dict with `server` and optional auth credentials. \- Many commercial proxies provide an HTTP/HTTPS “gateway” server that you specify in `server`. \- If your proxy doesn’t need auth, omit `username`/`password`. * * * 2\. Capturing PDFs & Screenshots -------------------------------- Sometimes you need a visual record of a page or a PDF “printout.” Crawl4AI can do both in one pass: `[](#__codelineno-1-1) import os, asyncio [](#__codelineno-1-2) from base64 import b64decode [](#__codelineno-1-3) from crawl4ai import AsyncWebCrawler, CacheMode [](#__codelineno-1-4) [](#__codelineno-1-5) async def main(): [](#__codelineno-1-6) async with AsyncWebCrawler() as crawler: [](#__codelineno-1-7) result = await crawler.arun( [](#__codelineno-1-8) url="https://en.wikipedia.org/wiki/List_of_common_misconceptions", [](#__codelineno-1-9) cache_mode=CacheMode.BYPASS, [](#__codelineno-1-10) pdf=True, [](#__codelineno-1-11) screenshot=True [](#__codelineno-1-12) ) [](#__codelineno-1-13) [](#__codelineno-1-14) if result.success: [](#__codelineno-1-15) # Save screenshot [](#__codelineno-1-16) if result.screenshot: [](#__codelineno-1-17) with open("wikipedia_screenshot.png", "wb") as f: [](#__codelineno-1-18) f.write(b64decode(result.screenshot)) [](#__codelineno-1-19) [](#__codelineno-1-20) # Save PDF [](#__codelineno-1-21) if result.pdf: [](#__codelineno-1-22) with open("wikipedia_page.pdf", "wb") as f: [](#__codelineno-1-23) f.write(result.pdf) [](#__codelineno-1-24) [](#__codelineno-1-25) print("[OK] PDF & screenshot captured.") [](#__codelineno-1-26) else: [](#__codelineno-1-27) print("[ERROR]", result.error_message) [](#__codelineno-1-28) [](#__codelineno-1-29) if __name__ == "__main__": [](#__codelineno-1-30) asyncio.run(main())` **Why PDF + Screenshot?** \- Large or complex pages can be slow or error-prone with “traditional” full-page screenshots. \- Exporting a PDF is more reliable for very long pages. Crawl4AI automatically converts the first PDF page into an image if you request both. **Relevant Parameters** \- **`pdf=True`**: Exports the current page as a PDF (base64-encoded in `result.pdf`). \- **`screenshot=True`**: Creates a screenshot (base64-encoded in `result.screenshot`). \- **`scan_full_page`** or advanced hooking can further refine how the crawler captures content. * * * 3\. Handling SSL Certificates ----------------------------- If you need to verify or export a site’s SSL certificate—for compliance, debugging, or data analysis—Crawl4AI can fetch it during the crawl: `[](#__codelineno-2-1) import asyncio, os [](#__codelineno-2-2) from crawl4ai import AsyncWebCrawler, CrawlerRunConfig, CacheMode [](#__codelineno-2-3) [](#__codelineno-2-4) async def main(): [](#__codelineno-2-5) tmp_dir = os.path.join(os.getcwd(), "tmp") [](#__codelineno-2-6) os.makedirs(tmp_dir, exist_ok=True) [](#__codelineno-2-7) [](#__codelineno-2-8) config = CrawlerRunConfig( [](#__codelineno-2-9) fetch_ssl_certificate=True, [](#__codelineno-2-10) cache_mode=CacheMode.BYPASS [](#__codelineno-2-11) ) [](#__codelineno-2-12) [](#__codelineno-2-13) async with AsyncWebCrawler() as crawler: [](#__codelineno-2-14) result = await crawler.arun(url="https://example.com", config=config) [](#__codelineno-2-15) [](#__codelineno-2-16) if result.success and result.ssl_certificate: [](#__codelineno-2-17) cert = result.ssl_certificate [](#__codelineno-2-18) print("\nCertificate Information:") [](#__codelineno-2-19) print(f"Issuer (CN): {cert.issuer.get('CN', '')}") [](#__codelineno-2-20) print(f"Valid until: {cert.valid_until}") [](#__codelineno-2-21) print(f"Fingerprint: {cert.fingerprint}") [](#__codelineno-2-22) [](#__codelineno-2-23) # Export in multiple formats: [](#__codelineno-2-24) cert.to_json(os.path.join(tmp_dir, "certificate.json")) [](#__codelineno-2-25) cert.to_pem(os.path.join(tmp_dir, "certificate.pem")) [](#__codelineno-2-26) cert.to_der(os.path.join(tmp_dir, "certificate.der")) [](#__codelineno-2-27) [](#__codelineno-2-28) print("\nCertificate exported to JSON/PEM/DER in 'tmp' folder.") [](#__codelineno-2-29) else: [](#__codelineno-2-30) print("[ERROR] No certificate or crawl failed.") [](#__codelineno-2-31) [](#__codelineno-2-32) if __name__ == "__main__": [](#__codelineno-2-33) asyncio.run(main())` **Key Points** \- **`fetch_ssl_certificate=True`** triggers certificate retrieval. \- `result.ssl_certificate` includes methods (`to_json`, `to_pem`, `to_der`) for saving in various formats (handy for server config, Java keystores, etc.). * * * 4\. Custom Headers ------------------ Sometimes you need to set custom headers (e.g., language preferences, authentication tokens, or specialized user-agent strings). You can do this in multiple ways: ``[](#__codelineno-3-1) import asyncio [](#__codelineno-3-2) from crawl4ai import AsyncWebCrawler [](#__codelineno-3-3) [](#__codelineno-3-4) async def main(): [](#__codelineno-3-5) # Option 1: Set headers at the crawler strategy level [](#__codelineno-3-6) crawler1 = AsyncWebCrawler( [](#__codelineno-3-7) # The underlying strategy can accept headers in its constructor [](#__codelineno-3-8) crawler_strategy=None # We'll override below for clarity [](#__codelineno-3-9) ) [](#__codelineno-3-10) crawler1.crawler_strategy.update_user_agent("MyCustomUA/1.0") [](#__codelineno-3-11) crawler1.crawler_strategy.set_custom_headers({ [](#__codelineno-3-12) "Accept-Language": "fr-FR,fr;q=0.9" [](#__codelineno-3-13) }) [](#__codelineno-3-14) result1 = await crawler1.arun("https://www.example.com") [](#__codelineno-3-15) print("Example 1 result success:", result1.success) [](#__codelineno-3-16) [](#__codelineno-3-17) # Option 2: Pass headers directly to `arun()` [](#__codelineno-3-18) crawler2 = AsyncWebCrawler() [](#__codelineno-3-19) result2 = await crawler2.arun( [](#__codelineno-3-20) url="https://www.example.com", [](#__codelineno-3-21) headers={"Accept-Language": "es-ES,es;q=0.9"} [](#__codelineno-3-22) ) [](#__codelineno-3-23) print("Example 2 result success:", result2.success) [](#__codelineno-3-24) [](#__codelineno-3-25) if __name__ == "__main__": [](#__codelineno-3-26) asyncio.run(main())`` **Notes** \- Some sites may react differently to certain headers (e.g., `Accept-Language`). \- If you need advanced user-agent randomization or client hints, see [Identity-Based Crawling (Anti-Bot)](../identity-based-crawling/) or use `UserAgentGenerator`. * * * 5\. Session Persistence & Local Storage --------------------------------------- Crawl4AI can preserve cookies and localStorage so you can continue where you left off—ideal for logging into sites or skipping repeated auth flows. ### 5.1 `storage_state` `[](#__codelineno-4-1) import asyncio [](#__codelineno-4-2) from crawl4ai import AsyncWebCrawler [](#__codelineno-4-3) [](#__codelineno-4-4) async def main(): [](#__codelineno-4-5) storage_dict = { [](#__codelineno-4-6) "cookies": [ [](#__codelineno-4-7) { [](#__codelineno-4-8) "name": "session", [](#__codelineno-4-9) "value": "abcd1234", [](#__codelineno-4-10) "domain": "example.com", [](#__codelineno-4-11) "path": "/", [](#__codelineno-4-12) "expires": 1699999999.0, [](#__codelineno-4-13) "httpOnly": False, [](#__codelineno-4-14) "secure": False, [](#__codelineno-4-15) "sameSite": "None" [](#__codelineno-4-16) } [](#__codelineno-4-17) ], [](#__codelineno-4-18) "origins": [ [](#__codelineno-4-19) { [](#__codelineno-4-20) "origin": "https://example.com", [](#__codelineno-4-21) "localStorage": [ [](#__codelineno-4-22) {"name": "token", "value": "my_auth_token"} [](#__codelineno-4-23) ] [](#__codelineno-4-24) } [](#__codelineno-4-25) ] [](#__codelineno-4-26) } [](#__codelineno-4-27) [](#__codelineno-4-28) # Provide the storage state as a dictionary to start "already logged in" [](#__codelineno-4-29) async with AsyncWebCrawler( [](#__codelineno-4-30) headless=True, [](#__codelineno-4-31) storage_state=storage_dict [](#__codelineno-4-32) ) as crawler: [](#__codelineno-4-33) result = await crawler.arun("https://example.com/protected") [](#__codelineno-4-34) if result.success: [](#__codelineno-4-35) print("Protected page content length:", len(result.html)) [](#__codelineno-4-36) else: [](#__codelineno-4-37) print("Failed to crawl protected page") [](#__codelineno-4-38) [](#__codelineno-4-39) if __name__ == "__main__": [](#__codelineno-4-40) asyncio.run(main())` ### 5.2 Exporting & Reusing State You can sign in once, export the browser context, and reuse it later—without re-entering credentials. * **`await context.storage_state(path="my_storage.json")`**: Exports cookies, localStorage, etc. to a file. * Provide `storage_state="my_storage.json"` on subsequent runs to skip the login step. **See**: [Detailed session management tutorial](../session-management/) or [Explanations → Browser Context & Managed Browser](../identity-based-crawling/) for more advanced scenarios (like multi-step logins, or capturing after interactive pages). * * * Putting It All Together ----------------------- Here’s a snippet that combines multiple “advanced” features (proxy, PDF, screenshot, SSL, custom headers, and session reuse) into one run. Normally, you’d tailor each setting to your project’s needs. `[](#__codelineno-5-1) import os, asyncio [](#__codelineno-5-2) from base64 import b64decode [](#__codelineno-5-3) from crawl4ai import AsyncWebCrawler, BrowserConfig, CrawlerRunConfig, CacheMode [](#__codelineno-5-4) [](#__codelineno-5-5) async def main(): [](#__codelineno-5-6) # 1. Browser config with proxy + headless [](#__codelineno-5-7) browser_cfg = BrowserConfig( [](#__codelineno-5-8) proxy_config={ [](#__codelineno-5-9) "server": "http://proxy.example.com:8080", [](#__codelineno-5-10) "username": "myuser", [](#__codelineno-5-11) "password": "mypass", [](#__codelineno-5-12) }, [](#__codelineno-5-13) headless=True, [](#__codelineno-5-14) ) [](#__codelineno-5-15) [](#__codelineno-5-16) # 2. Crawler config with PDF, screenshot, SSL, custom headers, and ignoring caches [](#__codelineno-5-17) crawler_cfg = CrawlerRunConfig( [](#__codelineno-5-18) pdf=True, [](#__codelineno-5-19) screenshot=True, [](#__codelineno-5-20) fetch_ssl_certificate=True, [](#__codelineno-5-21) cache_mode=CacheMode.BYPASS, [](#__codelineno-5-22) headers={"Accept-Language": "en-US,en;q=0.8"}, [](#__codelineno-5-23) storage_state="my_storage.json", # Reuse session from a previous sign-in [](#__codelineno-5-24) verbose=True, [](#__codelineno-5-25) ) [](#__codelineno-5-26) [](#__codelineno-5-27) # 3. Crawl [](#__codelineno-5-28) async with AsyncWebCrawler(config=browser_cfg) as crawler: [](#__codelineno-5-29) result = await crawler.arun( [](#__codelineno-5-30) url = "https://secure.example.com/protected", [](#__codelineno-5-31) config=crawler_cfg [](#__codelineno-5-32) ) [](#__codelineno-5-33) [](#__codelineno-5-34) if result.success: [](#__codelineno-5-35) print("[OK] Crawled the secure page. Links found:", len(result.links.get("internal", []))) [](#__codelineno-5-36) [](#__codelineno-5-37) # Save PDF & screenshot [](#__codelineno-5-38) if result.pdf: [](#__codelineno-5-39) with open("result.pdf", "wb") as f: [](#__codelineno-5-40) f.write(b64decode(result.pdf)) [](#__codelineno-5-41) if result.screenshot: [](#__codelineno-5-42) with open("result.png", "wb") as f: [](#__codelineno-5-43) f.write(b64decode(result.screenshot)) [](#__codelineno-5-44) [](#__codelineno-5-45) # Check SSL cert [](#__codelineno-5-46) if result.ssl_certificate: [](#__codelineno-5-47) print("SSL Issuer CN:", result.ssl_certificate.issuer.get("CN", "")) [](#__codelineno-5-48) else: [](#__codelineno-5-49) print("[ERROR]", result.error_message) [](#__codelineno-5-50) [](#__codelineno-5-51) if __name__ == "__main__": [](#__codelineno-5-52) asyncio.run(main())` * * * Conclusion & Next Steps ----------------------- You’ve now explored several **advanced** features: * **Proxy Usage** * **PDF & Screenshot** capturing for large or critical pages * **SSL Certificate** retrieval & exporting * **Custom Headers** for language or specialized requests * **Session Persistence** via storage state With these power tools, you can build robust scraping workflows that mimic real user behavior, handle secure sites, capture detailed snapshots, and manage sessions across multiple runs—streamlining your entire data collection pipeline. **Last Updated**: 2025-01-01 * * * --- # Fit Markdown - Crawl4AI Documentation Fit Markdown with Pruning & BM25 ================================ **Fit Markdown** is a specialized **filtered** version of your page’s markdown, focusing on the most relevant content. By default, Crawl4AI converts the entire HTML into a broad **raw\_markdown**. With fit markdown, we apply a **content filter** algorithm (e.g., **Pruning** or **BM25**) to remove or rank low-value sections—such as repetitive sidebars, shallow text blocks, or irrelevancies—leaving a concise textual “core.” * * * 1\. How “Fit Markdown” Works ---------------------------- ### 1.1 The `content_filter` In **`CrawlerRunConfig`**, you can specify a **`content_filter`** to shape how content is pruned or ranked before final markdown generation. A filter’s logic is applied **before** or **during** the HTML→Markdown process, producing: * **`result.markdown_v2.raw_markdown`** (unfiltered) * **`result.markdown_v2.fit_markdown`** (filtered or “fit” version) * **`result.markdown_v2.fit_html`** (the corresponding HTML snippet that produced `fit_markdown`) > **Note**: We’re currently storing the result in `markdown_v2`, but eventually we’ll unify it as `result.markdown`. ### 1.2 Common Filters 1. **PruningContentFilter** – Scores each node by text density, link density, and tag importance, discarding those below a threshold. 2. **BM25ContentFilter** – Focuses on textual relevance using BM25 ranking, especially useful if you have a specific user query (e.g., “machine learning” or “food nutrition”). * * * 2\. PruningContentFilter ------------------------ **Pruning** discards less relevant nodes based on **text density, link density, and tag importance**. It’s a heuristic-based approach—if certain sections appear too “thin” or too “spammy,” they’re pruned. ### 2.1 Usage Example `[](#__codelineno-0-1) import asyncio [](#__codelineno-0-2) from crawl4ai import AsyncWebCrawler, CrawlerRunConfig [](#__codelineno-0-3) from crawl4ai.content_filter_strategy import PruningContentFilter [](#__codelineno-0-4) from crawl4ai.markdown_generation_strategy import DefaultMarkdownGenerator [](#__codelineno-0-5) [](#__codelineno-0-6) async def main(): [](#__codelineno-0-7) # Step 1: Create a pruning filter [](#__codelineno-0-8) prune_filter = PruningContentFilter( [](#__codelineno-0-9) # Lower → more content retained, higher → more content pruned [](#__codelineno-0-10) threshold=0.45, [](#__codelineno-0-11) # "fixed" or "dynamic" [](#__codelineno-0-12) threshold_type="dynamic", [](#__codelineno-0-13) # Ignore nodes with <5 words [](#__codelineno-0-14) min_word_threshold=5 [](#__codelineno-0-15) ) [](#__codelineno-0-16) [](#__codelineno-0-17) # Step 2: Insert it into a Markdown Generator [](#__codelineno-0-18) md_generator = DefaultMarkdownGenerator(content_filter=prune_filter) [](#__codelineno-0-19) [](#__codelineno-0-20) # Step 3: Pass it to CrawlerRunConfig [](#__codelineno-0-21) config = CrawlerRunConfig( [](#__codelineno-0-22) markdown_generator=md_generator [](#__codelineno-0-23) ) [](#__codelineno-0-24) [](#__codelineno-0-25) async with AsyncWebCrawler() as crawler: [](#__codelineno-0-26) result = await crawler.arun( [](#__codelineno-0-27) url="https://news.ycombinator.com", [](#__codelineno-0-28) config=config [](#__codelineno-0-29) ) [](#__codelineno-0-30) [](#__codelineno-0-31) if result.success: [](#__codelineno-0-32) # 'fit_markdown' is your pruned content, focusing on "denser" text [](#__codelineno-0-33) print("Raw Markdown length:", len(result.markdown_v2.raw_markdown)) [](#__codelineno-0-34) print("Fit Markdown length:", len(result.markdown_v2.fit_markdown)) [](#__codelineno-0-35) else: [](#__codelineno-0-36) print("Error:", result.error_message) [](#__codelineno-0-37) [](#__codelineno-0-38) if __name__ == "__main__": [](#__codelineno-0-39) asyncio.run(main())` ### 2.2 Key Parameters * **`min_word_threshold`** (int): If a block has fewer words than this, it’s pruned. * **`threshold_type`** (str): * `"fixed"` → each node must exceed `threshold` (0–1). * `"dynamic"` → node scoring adjusts according to tag type, text/link density, etc. * **`threshold`** (float, default ~0.48): The base or “anchor” cutoff. **Algorithmic Factors**: * **Text density** – Encourages blocks that have a higher ratio of text to overall content. * **Link density** – Penalizes sections that are mostly links. * **Tag importance** – e.g., an `
` or `

` might be more important than a `

`. * **Structural context** – If a node is deeply nested or in a suspected sidebar, it might be deprioritized. * * * 3\. BM25ContentFilter --------------------- **BM25** is a classical text ranking algorithm often used in search engines. If you have a **user query** or rely on page metadata to derive a query, BM25 can identify which text chunks best match that query. ### 3.1 Usage Example `[](#__codelineno-1-1) import asyncio [](#__codelineno-1-2) from crawl4ai import AsyncWebCrawler, CrawlerRunConfig [](#__codelineno-1-3) from crawl4ai.content_filter_strategy import BM25ContentFilter [](#__codelineno-1-4) from crawl4ai.markdown_generation_strategy import DefaultMarkdownGenerator [](#__codelineno-1-5) [](#__codelineno-1-6) async def main(): [](#__codelineno-1-7) # 1) A BM25 filter with a user query [](#__codelineno-1-8) bm25_filter = BM25ContentFilter( [](#__codelineno-1-9) user_query="startup fundraising tips", [](#__codelineno-1-10) # Adjust for stricter or looser results [](#__codelineno-1-11) bm25_threshold=1.2 [](#__codelineno-1-12) ) [](#__codelineno-1-13) [](#__codelineno-1-14) # 2) Insert into a Markdown Generator [](#__codelineno-1-15) md_generator = DefaultMarkdownGenerator(content_filter=bm25_filter) [](#__codelineno-1-16) [](#__codelineno-1-17) # 3) Pass to crawler config [](#__codelineno-1-18) config = CrawlerRunConfig( [](#__codelineno-1-19) markdown_generator=md_generator [](#__codelineno-1-20) ) [](#__codelineno-1-21) [](#__codelineno-1-22) async with AsyncWebCrawler() as crawler: [](#__codelineno-1-23) result = await crawler.arun( [](#__codelineno-1-24) url="https://news.ycombinator.com", [](#__codelineno-1-25) config=config [](#__codelineno-1-26) ) [](#__codelineno-1-27) if result.success: [](#__codelineno-1-28) print("Fit Markdown (BM25 query-based):") [](#__codelineno-1-29) print(result.markdown_v2.fit_markdown) [](#__codelineno-1-30) else: [](#__codelineno-1-31) print("Error:", result.error_message) [](#__codelineno-1-32) [](#__codelineno-1-33) if __name__ == "__main__": [](#__codelineno-1-34) asyncio.run(main())` ### 3.2 Parameters * **`user_query`** (str, optional): E.g. `"machine learning"`. If blank, the filter tries to glean a query from page metadata. * **`bm25_threshold`** (float, default 1.0): * Higher → fewer chunks but more relevant. * Lower → more inclusive. > In more advanced scenarios, you might see parameters like `use_stemming`, `case_sensitive`, or `priority_tags` to refine how text is tokenized or weighted. * * * 4\. Accessing the “Fit” Output ------------------------------ After the crawl, your “fit” content is found in **`result.markdown_v2.fit_markdown`**. In future versions, it will be **`result.markdown.fit_markdown`**. Meanwhile: `[](#__codelineno-2-1) fit_md = result.markdown_v2.fit_markdown [](#__codelineno-2-2) fit_html = result.markdown_v2.fit_html` If the content filter is **BM25**, you might see additional logic or references in `fit_markdown` that highlight relevant segments. If it’s **Pruning**, the text is typically well-cleaned but not necessarily matched to a query. * * * 5\. Code Patterns Recap ----------------------- ### 5.1 Pruning `[](#__codelineno-3-1) prune_filter = PruningContentFilter( [](#__codelineno-3-2) threshold=0.5, [](#__codelineno-3-3) threshold_type="fixed", [](#__codelineno-3-4) min_word_threshold=10 [](#__codelineno-3-5) ) [](#__codelineno-3-6) md_generator = DefaultMarkdownGenerator(content_filter=prune_filter) [](#__codelineno-3-7) config = CrawlerRunConfig(markdown_generator=md_generator) [](#__codelineno-3-8) # => result.markdown_v2.fit_markdown` ### 5.2 BM25 `[](#__codelineno-4-1) bm25_filter = BM25ContentFilter( [](#__codelineno-4-2) user_query="health benefits fruit", [](#__codelineno-4-3) bm25_threshold=1.2 [](#__codelineno-4-4) ) [](#__codelineno-4-5) md_generator = DefaultMarkdownGenerator(content_filter=bm25_filter) [](#__codelineno-4-6) config = CrawlerRunConfig(markdown_generator=md_generator) [](#__codelineno-4-7) # => result.markdown_v2.fit_markdown` * * * 6\. Combining with “word\_count\_threshold” & Exclusions -------------------------------------------------------- Remember you can also specify: `[](#__codelineno-5-1) config = CrawlerRunConfig( [](#__codelineno-5-2) word_count_threshold=10, [](#__codelineno-5-3) excluded_tags=["nav", "footer", "header"], [](#__codelineno-5-4) exclude_external_links=True, [](#__codelineno-5-5) markdown_generator=DefaultMarkdownGenerator( [](#__codelineno-5-6) content_filter=PruningContentFilter(threshold=0.5) [](#__codelineno-5-7) ) [](#__codelineno-5-8) )` Thus, **multi-level** filtering occurs: 1. The crawler’s `excluded_tags` are removed from the HTML first. 2. The content filter (Pruning, BM25, or custom) prunes or ranks the remaining text blocks. 3. The final “fit” content is generated in `result.markdown_v2.fit_markdown`. * * * 7\. Custom Filters ------------------ If you need a different approach (like a specialized ML model or site-specific heuristics), you can create a new class inheriting from `RelevantContentFilter` and implement `filter_content(html)`. Then inject it into your **markdown generator**: `[](#__codelineno-6-1) from crawl4ai.content_filter_strategy import RelevantContentFilter [](#__codelineno-6-2) [](#__codelineno-6-3) class MyCustomFilter(RelevantContentFilter): [](#__codelineno-6-4) def filter_content(self, html, min_word_threshold=None): [](#__codelineno-6-5) # parse HTML, implement custom logic [](#__codelineno-6-6) return [block for block in ... if ... some condition...]` **Steps**: 1. Subclass `RelevantContentFilter`. 2. Implement `filter_content(...)`. 3. Use it in your `DefaultMarkdownGenerator(content_filter=MyCustomFilter(...))`. * * * 8\. Final Thoughts ------------------ **Fit Markdown** is a crucial feature for: * **Summaries**: Quickly get the important text from a cluttered page. * **Search**: Combine with **BM25** to produce content relevant to a query. * **AI Pipelines**: Filter out boilerplate so LLM-based extraction or summarization runs on denser text. **Key Points**: - **PruningContentFilter**: Great if you just want the “meatiest” text without a user query. \- **BM25ContentFilter**: Perfect for query-based extraction or searching. \- Combine with **`excluded_tags`, `exclude_external_links`, `word_count_threshold`** to refine your final “fit” text. \- Fit markdown ends up in **`result.markdown_v2.fit_markdown`**; eventually **`result.markdown.fit_markdown`** in future versions. With these tools, you can **zero in** on the text that truly matters, ignoring spammy or boilerplate content, and produce a concise, relevant “fit markdown” for your AI or data pipelines. Happy pruning and searching! * Last Updated: 2025-01-01 * * * --- # Quick Start - Crawl4AI Documentation Getting Started with Crawl4AI ============================= Welcome to **Crawl4AI**, an open-source LLM-friendly Web Crawler & Scraper. In this tutorial, you’ll: 1. Run your **first crawl** using minimal configuration. 2. Generate **Markdown** output (and learn how it’s influenced by content filters). 3. Experiment with a simple **CSS-based extraction** strategy. 4. See a glimpse of **LLM-based extraction** (including open-source and closed-source model options). 5. Crawl a **dynamic** page that loads content via JavaScript. * * * 1\. Introduction ---------------- Crawl4AI provides: * An asynchronous crawler, **`AsyncWebCrawler`**. * Configurable browser and run settings via **`BrowserConfig`** and **`CrawlerRunConfig`**. * Automatic HTML-to-Markdown conversion via **`DefaultMarkdownGenerator`** (supports optional filters). * Multiple extraction strategies (LLM-based or “traditional” CSS/XPath-based). By the end of this guide, you’ll have performed a basic crawl, generated Markdown, tried out two extraction strategies, and crawled a dynamic page that uses “Load More” buttons or JavaScript updates. * * * 2\. Your First Crawl -------------------- Here’s a minimal Python script that creates an **`AsyncWebCrawler`**, fetches a webpage, and prints the first 300 characters of its Markdown output: `[](#__codelineno-0-1) import asyncio [](#__codelineno-0-2) from crawl4ai import AsyncWebCrawler [](#__codelineno-0-3) [](#__codelineno-0-4) async def main(): [](#__codelineno-0-5) async with AsyncWebCrawler() as crawler: [](#__codelineno-0-6) result = await crawler.arun("https://example.com") [](#__codelineno-0-7) print(result.markdown[:300]) # Print first 300 chars [](#__codelineno-0-8) [](#__codelineno-0-9) if __name__ == "__main__": [](#__codelineno-0-10) asyncio.run(main())` **What’s happening?** - **`AsyncWebCrawler`** launches a headless browser (Chromium by default). - It fetches `https://example.com`. - Crawl4AI automatically converts the HTML into Markdown. You now have a simple, working crawl! * * * 3\. Basic Configuration (Light Introduction) -------------------------------------------- Crawl4AI’s crawler can be heavily customized using two main classes: 1. **`BrowserConfig`**: Controls browser behavior (headless or full UI, user agent, JavaScript toggles, etc.). 2. **`CrawlerRunConfig`**: Controls how each crawl runs (caching, extraction, timeouts, hooking, etc.). Below is an example with minimal usage: `[](#__codelineno-1-1) import asyncio [](#__codelineno-1-2) from crawl4ai import AsyncWebCrawler, BrowserConfig, CrawlerRunConfig, CacheMode [](#__codelineno-1-3) [](#__codelineno-1-4) async def main(): [](#__codelineno-1-5) browser_conf = BrowserConfig(headless=True) # or False to see the browser [](#__codelineno-1-6) run_conf = CrawlerRunConfig( [](#__codelineno-1-7) cache_mode=CacheMode.BYPASS [](#__codelineno-1-8) ) [](#__codelineno-1-9) [](#__codelineno-1-10) async with AsyncWebCrawler(config=browser_conf) as crawler: [](#__codelineno-1-11) result = await crawler.arun( [](#__codelineno-1-12) url="https://example.com", [](#__codelineno-1-13) config=run_conf [](#__codelineno-1-14) ) [](#__codelineno-1-15) print(result.markdown) [](#__codelineno-1-16) [](#__codelineno-1-17) if __name__ == "__main__": [](#__codelineno-1-18) asyncio.run(main())` > IMPORTANT: By default cache mode is set to `CacheMode.ENABLED`. So to have fresh content, you need to set it to `CacheMode.BYPASS` We’ll explore more advanced config in later tutorials (like enabling proxies, PDF output, multi-tab sessions, etc.). For now, just note how you pass these objects to manage crawling. * * * 4\. Generating Markdown Output ------------------------------ By default, Crawl4AI automatically generates Markdown from each crawled page. However, the exact output depends on whether you specify a **markdown generator** or **content filter**. * **`result.markdown`**: The direct HTML-to-Markdown conversion. * **`result.markdown.fit_markdown`**: The same content after applying any configured **content filter** (e.g., `PruningContentFilter`). ### Example: Using a Filter with `DefaultMarkdownGenerator` `[](#__codelineno-2-1) from crawl4ai import AsyncWebCrawler, CrawlerRunConfig [](#__codelineno-2-2) from crawl4ai.content_filter_strategy import PruningContentFilter [](#__codelineno-2-3) from crawl4ai.markdown_generation_strategy import DefaultMarkdownGenerator [](#__codelineno-2-4) [](#__codelineno-2-5) md_generator = DefaultMarkdownGenerator( [](#__codelineno-2-6) content_filter=PruningContentFilter(threshold=0.4, threshold_type="fixed") [](#__codelineno-2-7) ) [](#__codelineno-2-8) [](#__codelineno-2-9) config = CrawlerRunConfig( [](#__codelineno-2-10) cache_mode=CacheMode.BYPASS, [](#__codelineno-2-11) markdown_generator=md_generator [](#__codelineno-2-12) ) [](#__codelineno-2-13) [](#__codelineno-2-14) async with AsyncWebCrawler() as crawler: [](#__codelineno-2-15) result = await crawler.arun("https://news.ycombinator.com", config=config) [](#__codelineno-2-16) print("Raw Markdown length:", len(result.markdown.raw_markdown)) [](#__codelineno-2-17) print("Fit Markdown length:", len(result.markdown.fit_markdown))` **Note**: If you do **not** specify a content filter or markdown generator, you’ll typically see only the raw Markdown. `PruningContentFilter` may adds around `50ms` in processing time. We’ll dive deeper into these strategies in a dedicated **Markdown Generation** tutorial. * * * 5\. Simple Data Extraction (CSS-based) -------------------------------------- Crawl4AI can also extract structured data (JSON) using CSS or XPath selectors. Below is a minimal CSS-based example: `[](#__codelineno-3-1) import asyncio [](#__codelineno-3-2) import json [](#__codelineno-3-3) from crawl4ai import AsyncWebCrawler, CrawlerRunConfig, CacheMode [](#__codelineno-3-4) from crawl4ai.extraction_strategy import JsonCssExtractionStrategy [](#__codelineno-3-5) [](#__codelineno-3-6) async def main(): [](#__codelineno-3-7) schema = { [](#__codelineno-3-8) "name": "Example Items", [](#__codelineno-3-9) "baseSelector": "div.item", [](#__codelineno-3-10) "fields": [ [](#__codelineno-3-11) {"name": "title", "selector": "h2", "type": "text"}, [](#__codelineno-3-12) {"name": "link", "selector": "a", "type": "attribute", "attribute": "href"} [](#__codelineno-3-13) ] [](#__codelineno-3-14) } [](#__codelineno-3-15) [](#__codelineno-3-16) raw_html = "

Item 1

Link 1
" [](#__codelineno-3-17) [](#__codelineno-3-18) async with AsyncWebCrawler() as crawler: [](#__codelineno-3-19) result = await crawler.arun( [](#__codelineno-3-20) url="raw://" + raw_html, [](#__codelineno-3-21) config=CrawlerRunConfig( [](#__codelineno-3-22) cache_mode=CacheMode.BYPASS, [](#__codelineno-3-23) extraction_strategy=JsonCssExtractionStrategy(schema) [](#__codelineno-3-24) ) [](#__codelineno-3-25) ) [](#__codelineno-3-26) # The JSON output is stored in 'extracted_content' [](#__codelineno-3-27) data = json.loads(result.extracted_content) [](#__codelineno-3-28) print(data) [](#__codelineno-3-29) [](#__codelineno-3-30) if __name__ == "__main__": [](#__codelineno-3-31) asyncio.run(main())` **Why is this helpful?** - Great for repetitive page structures (e.g., item listings, articles). - No AI usage or costs. - The crawler returns a JSON string you can parse or store. > Tips: You can pass raw HTML to the crawler instead of a URL. To do so, prefix the HTML with `raw://`. * * * 6\. Simple Data Extraction (LLM-based) -------------------------------------- For more complex or irregular pages, a language model can parse text intelligently into a structure you define. Crawl4AI supports **open-source** or **closed-source** providers: * **Open-Source Models** (e.g., `ollama/llama3.3`, `no_token`) * **OpenAI Models** (e.g., `openai/gpt-4`, requires `api_token`) * Or any provider supported by the underlying library Below is an example using **open-source** style (no token) and closed-source: `[](#__codelineno-4-1) import os [](#__codelineno-4-2) import json [](#__codelineno-4-3) import asyncio [](#__codelineno-4-4) from pydantic import BaseModel, Field [](#__codelineno-4-5) from crawl4ai import AsyncWebCrawler, CrawlerRunConfig [](#__codelineno-4-6) from crawl4ai.extraction_strategy import LLMExtractionStrategy [](#__codelineno-4-7) [](#__codelineno-4-8) class OpenAIModelFee(BaseModel): [](#__codelineno-4-9) model_name: str = Field(..., description="Name of the OpenAI model.") [](#__codelineno-4-10) input_fee: str = Field(..., description="Fee for input token for the OpenAI model.") [](#__codelineno-4-11) output_fee: str = Field( [](#__codelineno-4-12) ..., description="Fee for output token for the OpenAI model." [](#__codelineno-4-13) ) [](#__codelineno-4-14) [](#__codelineno-4-15) async def extract_structured_data_using_llm( [](#__codelineno-4-16) provider: str, api_token: str = None, extra_headers: Dict[str, str] = None [](#__codelineno-4-17) ): [](#__codelineno-4-18) print(f"\n--- Extracting Structured Data with {provider} ---") [](#__codelineno-4-19) [](#__codelineno-4-20) if api_token is None and provider != "ollama": [](#__codelineno-4-21) print(f"API token is required for {provider}. Skipping this example.") [](#__codelineno-4-22) return [](#__codelineno-4-23) [](#__codelineno-4-24) browser_config = BrowserConfig(headless=True) [](#__codelineno-4-25) [](#__codelineno-4-26) extra_args = {"temperature": 0, "top_p": 0.9, "max_tokens": 2000} [](#__codelineno-4-27) if extra_headers: [](#__codelineno-4-28) extra_args["extra_headers"] = extra_headers [](#__codelineno-4-29) [](#__codelineno-4-30) crawler_config = CrawlerRunConfig( [](#__codelineno-4-31) cache_mode=CacheMode.BYPASS, [](#__codelineno-4-32) word_count_threshold=1, [](#__codelineno-4-33) page_timeout=80000, [](#__codelineno-4-34) extraction_strategy=LLMExtractionStrategy( [](#__codelineno-4-35) provider=provider, [](#__codelineno-4-36) api_token=api_token, [](#__codelineno-4-37) schema=OpenAIModelFee.model_json_schema(), [](#__codelineno-4-38) extraction_type="schema", [](#__codelineno-4-39) instruction="""From the crawled content, extract all mentioned model names along with their fees for input and output tokens. [](#__codelineno-4-40) Do not miss any models in the entire content.""", [](#__codelineno-4-41) extra_args=extra_args, [](#__codelineno-4-42) ), [](#__codelineno-4-43) ) [](#__codelineno-4-44) [](#__codelineno-4-45) async with AsyncWebCrawler(config=browser_config) as crawler: [](#__codelineno-4-46) result = await crawler.arun( [](#__codelineno-4-47) url="https://openai.com/api/pricing/", config=crawler_config [](#__codelineno-4-48) ) [](#__codelineno-4-49) print(result.extracted_content) [](#__codelineno-4-50) [](#__codelineno-4-51) if __name__ == "__main__": [](#__codelineno-4-52) # Use ollama with llama3.3 [](#__codelineno-4-53) # asyncio.run( [](#__codelineno-4-54) # extract_structured_data_using_llm( [](#__codelineno-4-55) # provider="ollama/llama3.3", api_token="no-token" [](#__codelineno-4-56) # ) [](#__codelineno-4-57) # ) [](#__codelineno-4-58) [](#__codelineno-4-59) asyncio.run( [](#__codelineno-4-60) extract_structured_data_using_llm( [](#__codelineno-4-61) provider="openai/gpt-4o", api_token=os.getenv("OPENAI_API_KEY") [](#__codelineno-4-62) ) [](#__codelineno-4-63) )` **What’s happening?** - We define a Pydantic schema (`PricingInfo`) describing the fields we want. - The LLM extraction strategy uses that schema and your instructions to transform raw text into structured JSON. - Depending on the **provider** and **api\_token**, you can use local models or a remote API. * * * 7\. Dynamic Content Example --------------------------- Some sites require multiple “page clicks” or dynamic JavaScript updates. Below is an example showing how to **click** a “Next Page” button and wait for new commits to load on GitHub, using **`BrowserConfig`** and **`CrawlerRunConfig`**: `[](#__codelineno-5-1) import asyncio [](#__codelineno-5-2) from crawl4ai import AsyncWebCrawler, BrowserConfig, CrawlerRunConfig, CacheMode [](#__codelineno-5-3) from crawl4ai.extraction_strategy import JsonCssExtractionStrategy [](#__codelineno-5-4) [](#__codelineno-5-5) async def extract_structured_data_using_css_extractor(): [](#__codelineno-5-6) print("\n--- Using JsonCssExtractionStrategy for Fast Structured Output ---") [](#__codelineno-5-7) schema = { [](#__codelineno-5-8) "name": "KidoCode Courses", [](#__codelineno-5-9) "baseSelector": "section.charge-methodology .w-tab-content > div", [](#__codelineno-5-10) "fields": [ [](#__codelineno-5-11) { [](#__codelineno-5-12) "name": "section_title", [](#__codelineno-5-13) "selector": "h3.heading-50", [](#__codelineno-5-14) "type": "text", [](#__codelineno-5-15) }, [](#__codelineno-5-16) { [](#__codelineno-5-17) "name": "section_description", [](#__codelineno-5-18) "selector": ".charge-content", [](#__codelineno-5-19) "type": "text", [](#__codelineno-5-20) }, [](#__codelineno-5-21) { [](#__codelineno-5-22) "name": "course_name", [](#__codelineno-5-23) "selector": ".text-block-93", [](#__codelineno-5-24) "type": "text", [](#__codelineno-5-25) }, [](#__codelineno-5-26) { [](#__codelineno-5-27) "name": "course_description", [](#__codelineno-5-28) "selector": ".course-content-text", [](#__codelineno-5-29) "type": "text", [](#__codelineno-5-30) }, [](#__codelineno-5-31) { [](#__codelineno-5-32) "name": "course_icon", [](#__codelineno-5-33) "selector": ".image-92", [](#__codelineno-5-34) "type": "attribute", [](#__codelineno-5-35) "attribute": "src", [](#__codelineno-5-36) }, [](#__codelineno-5-37) ], [](#__codelineno-5-38) } [](#__codelineno-5-39) [](#__codelineno-5-40) browser_config = BrowserConfig(headless=True, java_script_enabled=True) [](#__codelineno-5-41) [](#__codelineno-5-42) js_click_tabs = """ [](#__codelineno-5-43) (async () => { [](#__codelineno-5-44) const tabs = document.querySelectorAll("section.charge-methodology .tabs-menu-3 > div"); [](#__codelineno-5-45) for(let tab of tabs) { [](#__codelineno-5-46) tab.scrollIntoView(); [](#__codelineno-5-47) tab.click(); [](#__codelineno-5-48) await new Promise(r => setTimeout(r, 500)); [](#__codelineno-5-49) } [](#__codelineno-5-50) })(); [](#__codelineno-5-51) """ [](#__codelineno-5-52) [](#__codelineno-5-53) crawler_config = CrawlerRunConfig( [](#__codelineno-5-54) cache_mode=CacheMode.BYPASS, [](#__codelineno-5-55) extraction_strategy=JsonCssExtractionStrategy(schema), [](#__codelineno-5-56) js_code=[js_click_tabs], [](#__codelineno-5-57) ) [](#__codelineno-5-58) [](#__codelineno-5-59) async with AsyncWebCrawler(config=browser_config) as crawler: [](#__codelineno-5-60) result = await crawler.arun( [](#__codelineno-5-61) url="https://www.kidocode.com/degrees/technology", config=crawler_config [](#__codelineno-5-62) ) [](#__codelineno-5-63) [](#__codelineno-5-64) companies = json.loads(result.extracted_content) [](#__codelineno-5-65) print(f"Successfully extracted {len(companies)} companies") [](#__codelineno-5-66) print(json.dumps(companies[0], indent=2)) [](#__codelineno-5-67) [](#__codelineno-5-68) async def main(): [](#__codelineno-5-69) await extract_structured_data_using_css_extractor() [](#__codelineno-5-70) [](#__codelineno-5-71) if __name__ == "__main__": [](#__codelineno-5-72) asyncio.run(main())` **Key Points**: * **`BrowserConfig(headless=False)`**: We want to watch it click “Next Page.” * **`CrawlerRunConfig(...)`**: We specify the extraction strategy, pass `session_id` to reuse the same page. * **`js_code`** and **`wait_for`** are used for subsequent pages (`page > 0`) to click the “Next” button and wait for new commits to load. * **`js_only=True`** indicates we’re not re-navigating but continuing the existing session. * Finally, we call `kill_session()` to clean up the page and browser session. * * * 8\. Next Steps -------------- Congratulations! You have: 1. Performed a basic crawl and printed Markdown. 2. Used **content filters** with a markdown generator. 3. Extracted JSON via **CSS** or **LLM** strategies. 4. Handled **dynamic** pages with JavaScript triggers. If you’re ready for more, check out: * **Installation**: A deeper dive into advanced installs, Docker usage (experimental), or optional dependencies. * **Hooks & Auth**: Learn how to run custom JavaScript or handle logins with cookies, local storage, etc. * **Deployment**: Explore ephemeral testing in Docker or plan for the upcoming stable Docker release. * **Browser Management**: Delve into user simulation, stealth modes, and concurrency best practices. Crawl4AI is a powerful, flexible tool. Enjoy building out your scrapers, data pipelines, or AI-driven extraction flows. Happy crawling! * * * --- # Content Selection - Crawl4AI Documentation Content Selection ================= Crawl4AI provides multiple ways to **select**, **filter**, and **refine** the content from your crawls. Whether you need to target a specific CSS region, exclude entire tags, filter out external links, or remove certain domains and images, **`CrawlerRunConfig`** offers a wide range of parameters. Below, we show how to configure these parameters and combine them for precise control. * * * 1\. CSS-Based Selection ----------------------- A straightforward way to **limit** your crawl results to a certain region of the page is **`css_selector`** in **`CrawlerRunConfig`**: `[](#__codelineno-0-1) import asyncio [](#__codelineno-0-2) from crawl4ai import AsyncWebCrawler, CrawlerRunConfig [](#__codelineno-0-3) [](#__codelineno-0-4) async def main(): [](#__codelineno-0-5) config = CrawlerRunConfig( [](#__codelineno-0-6) # e.g., first 30 items from Hacker News [](#__codelineno-0-7) css_selector=".athing:nth-child(-n+30)" [](#__codelineno-0-8) ) [](#__codelineno-0-9) async with AsyncWebCrawler() as crawler: [](#__codelineno-0-10) result = await crawler.arun( [](#__codelineno-0-11) url="https://news.ycombinator.com/newest", [](#__codelineno-0-12) config=config [](#__codelineno-0-13) ) [](#__codelineno-0-14) print("Partial HTML length:", len(result.cleaned_html)) [](#__codelineno-0-15) [](#__codelineno-0-16) if __name__ == "__main__": [](#__codelineno-0-17) asyncio.run(main())` **Result**: Only elements matching that selector remain in `result.cleaned_html`. * * * 2\. Content Filtering & Exclusions ---------------------------------- ### 2.1 Basic Overview `[](#__codelineno-1-1) config = CrawlerRunConfig( [](#__codelineno-1-2) # Content thresholds [](#__codelineno-1-3) word_count_threshold=10, # Minimum words per block [](#__codelineno-1-4) [](#__codelineno-1-5) # Tag exclusions [](#__codelineno-1-6) excluded_tags=['form', 'header', 'footer', 'nav'], [](#__codelineno-1-7) [](#__codelineno-1-8) # Link filtering [](#__codelineno-1-9) exclude_external_links=True, [](#__codelineno-1-10) exclude_social_media_links=True, [](#__codelineno-1-11) # Block entire domains [](#__codelineno-1-12) exclude_domains=["adtrackers.com", "spammynews.org"], [](#__codelineno-1-13) exclude_social_media_domains=["facebook.com", "twitter.com"], [](#__codelineno-1-14) [](#__codelineno-1-15) # Media filtering [](#__codelineno-1-16) exclude_external_images=True [](#__codelineno-1-17) )` **Explanation**: * **`word_count_threshold`**: Ignores text blocks under X words. Helps skip trivial blocks like short nav or disclaimers. * **`excluded_tags`**: Removes entire tags (``, `
`, `