# Table of Contents - [Growth - Crawl4AI Documentation (v0.8.x)](#growth-crawl4ai-documentation-v0-8-x-) - [Adaptive Strategies - Crawl4AI Documentation (v0.8.x)](#adaptive-strategies-crawl4ai-documentation-v0-8-x-) - [Home - Crawl4AI Documentation (v0.8.x)](#home-crawl4ai-documentation-v0-8-x-) - [Contributing Guide - Crawl4AI Documentation (v0.8.x)](#contributing-guide-crawl4ai-documentation-v0-8-x-) - [Crawl Dispatcher - Crawl4AI Documentation (v0.8.x)](#crawl-dispatcher-crawl4ai-documentation-v0-8-x-) - [File Downloading - Crawl4AI Documentation (v0.8.x)](#file-downloading-crawl4ai-documentation-v0-8-x-) - [Anti-Bot & Fallback - Crawl4AI Documentation (v0.8.x)](#anti-bot-fallback-crawl4ai-documentation-v0-8-x-) - [Overview - Crawl4AI Documentation (v0.8.x)](#overview-crawl4ai-documentation-v0-8-x-) - [Lazy Loading - Crawl4AI Documentation (v0.8.x)](#lazy-loading-crawl4ai-documentation-v0-8-x-) - [Hooks & Auth - Crawl4AI Documentation (v0.8.x)](#hooks-auth-crawl4ai-documentation-v0-8-x-) - [Identity Based Crawling - Crawl4AI Documentation (v0.8.x)](#identity-based-crawling-crawl4ai-documentation-v0-8-x-) - [Network & Console Capture - Crawl4AI Documentation (v0.8.x)](#network-console-capture-crawl4ai-documentation-v0-8-x-) - [PDF Parsing - Crawl4AI Documentation (v0.8.x)](#pdf-parsing-crawl4ai-documentation-v0-8-x-) - [Multi-URL Crawling - Crawl4AI Documentation (v0.8.x)](#multi-url-crawling-crawl4ai-documentation-v0-8-x-) - [Proxy & Security - Crawl4AI Documentation (v0.8.x)](#proxy-security-crawl4ai-documentation-v0-8-x-) - [SSL Certificate - Crawl4AI Documentation (v0.8.x)](#ssl-certificate-crawl4ai-documentation-v0-8-x-) - [arun_many() - Crawl4AI Documentation (v0.8.x)](#arun-many-crawl4ai-documentation-v0-8-x-) - [Undetected Browser - Crawl4AI Documentation (v0.8.x)](#undetected-browser-crawl4ai-documentation-v0-8-x-) - [Session Management - Crawl4AI Documentation (v0.8.x)](#session-management-crawl4ai-documentation-v0-8-x-) - [AsyncWebCrawler - Crawl4AI Documentation (v0.8.x)](#asyncwebcrawler-crawl4ai-documentation-v0-8-x-) - [arun() - Crawl4AI Documentation (v0.8.x)](#arun-crawl4ai-documentation-v0-8-x-) - [CrawlResult - Crawl4AI Documentation (v0.8.x)](#crawlresult-crawl4ai-documentation-v0-8-x-) - [Build - Crawl4AI Documentation (v0.8.x)](#build-crawl4ai-documentation-v0-8-x-) - [Strategies - Crawl4AI Documentation (v0.8.x)](#strategies-crawl4ai-documentation-v0-8-x-) - [AdaptiveCrawler - Crawl4AI Documentation (v0.8.x)](#adaptivecrawler-crawl4ai-documentation-v0-8-x-) - [Supercharging Your AI Assistant: My Journey to Better LLM Contexts for crawl4ai - Crawl4AI Documentation (v0.8.x)](#supercharging-your-ai-assistant-my-journey-to-better-llm-contexts-for-crawl4ai-crawl4ai-documentation-v0-8-x-) - [Installation 💻 - Crawl4AI Documentation (v0.8.x)](#installation-crawl4ai-documentation-v0-8-x-) - [Adaptive Crawling: Building Dynamic Knowledge That Grows on Demand - Crawl4AI Documentation (v0.8.x)](#adaptive-crawling-building-dynamic-knowledge-that-grows-on-demand-crawl4ai-documentation-v0-8-x-) - [Dockerize hooks - Crawl4AI Documentation (v0.8.x)](#dockerize-hooks-crawl4ai-documentation-v0-8-x-) - [digest() - Crawl4AI Documentation (v0.8.x)](#digest-crawl4ai-documentation-v0-8-x-) - [The LLM Context Protocol: Why Your AI Assistant Needs Memory, Reasoning, and Examples - Crawl4AI Documentation (v0.8.x)](#the-llm-context-protocol-why-your-ai-assistant-needs-memory-reasoning-and-examples-crawl4ai-documentation-v0-8-x-) - [Release Summary for Version 0.4.0 (December 1, 2024) - Crawl4AI Documentation (v0.8.x)](#release-summary-for-version-0-4-0-december-1-2024-crawl4ai-documentation-v0-8-x-) - [0.4.2 - Crawl4AI Documentation (v0.8.x)](#0-4-2-crawl4ai-documentation-v0-8-x-) - [Release Summary for Version 0.4.1 (December 8, 2024): Major Efficiency Boosts with New Features! - Crawl4AI Documentation (v0.8.x)](#release-summary-for-version-0-4-1-december-8-2024-major-efficiency-boosts-with-new-features-crawl4ai-documentation-v0-8-x-) - [Crawl4AI v0.6.0 Release Notes - Crawl4AI Documentation (v0.8.x)](#crawl4ai-v0-6-0-release-notes-crawl4ai-documentation-v0-8-x-) - [🛠️ Crawl4AI v0.7.1: Minor Cleanup Update - Crawl4AI Documentation (v0.8.x)](#-crawl4ai-v0-7-1-minor-cleanup-update-crawl4ai-documentation-v0-8-x-) - [Cache Modes - Crawl4AI Documentation (v0.8.x)](#cache-modes-crawl4ai-documentation-v0-8-x-) - [🚀 Crawl4AI v0.7.2: CI/CD & Dependency Optimization Update - Crawl4AI Documentation (v0.8.x)](#-crawl4ai-v0-7-2-ci-cd-dependency-optimization-update-crawl4ai-documentation-v0-8-x-) - [Crawl4AI 0.4.3: Major Performance Boost & LLM Integration - Crawl4AI Documentation (v0.8.x)](#crawl4ai-0-4-3-major-performance-boost-llm-integration-crawl4ai-documentation-v0-8-x-) - [🚀 Crawl4AI v0.7.3: The Multi-Config Intelligence Update - Crawl4AI Documentation (v0.8.x)](#-crawl4ai-v0-7-3-the-multi-config-intelligence-update-crawl4ai-documentation-v0-8-x-) - [🚀 Crawl4AI v0.7.0: The Adaptive Intelligence Update - Crawl4AI Documentation (v0.8.x)](#-crawl4ai-v0-7-0-the-adaptive-intelligence-update-crawl4ai-documentation-v0-8-x-) - [Crawl4AI v0.7.8: Stability & Bug Fix Release - Crawl4AI Documentation (v0.8.x)](#crawl4ai-v0-7-8-stability-bug-fix-release-crawl4ai-documentation-v0-8-x-) - [Installation - Crawl4AI Documentation (v0.8.x)](#installation-crawl4ai-documentation-v0-8-x-) - [Command Line Interface - Crawl4AI Documentation (v0.8.x)](#command-line-interface-crawl4ai-documentation-v0-8-x-) - [🚀 Crawl4AI v0.7.5: The Docker Hooks & Security Update - Crawl4AI Documentation (v0.8.x)](#-crawl4ai-v0-7-5-the-docker-hooks-security-update-crawl4ai-documentation-v0-8-x-) - [Simple Crawling - Crawl4AI Documentation (v0.8.x)](#simple-crawling-crawl4ai-documentation-v0-8-x-) - [Crawler Result - Crawl4AI Documentation (v0.8.x)](#crawler-result-crawl4ai-documentation-v0-8-x-) - [Fit Markdown - Crawl4AI Documentation (v0.8.x)](#fit-markdown-crawl4ai-documentation-v0-8-x-) - [Code Examples - Crawl4AI Documentation (v0.8.x)](#code-examples-crawl4ai-documentation-v0-8-x-) - [Chunking - Crawl4AI Documentation (v0.8.x)](#chunking-crawl4ai-documentation-v0-8-x-) - [Local Files & Raw HTML - Crawl4AI Documentation (v0.8.x)](#local-files-raw-html-crawl4ai-documentation-v0-8-x-) - [Browser, Crawler & LLM Config - Crawl4AI Documentation (v0.8.x)](#browser-crawler-llm-config-crawl4ai-documentation-v0-8-x-) - [Adaptive Crawling - Crawl4AI Documentation (v0.8.x)](#adaptive-crawling-crawl4ai-documentation-v0-8-x-) - [Clustering Strategies - Crawl4AI Documentation (v0.8.x)](#clustering-strategies-crawl4ai-documentation-v0-8-x-) - [Page Interaction - Crawl4AI Documentation (v0.8.x)](#page-interaction-crawl4ai-documentation-v0-8-x-) - [Content Selection - Crawl4AI Documentation (v0.8.x)](#content-selection-crawl4ai-documentation-v0-8-x-) - [WebScrapingStrategy Migration Guide - Crawl4AI Documentation (v0.8.x)](#webscrapingstrategy-migration-guide-crawl4ai-documentation-v0-8-x-) - [🚀 Crawl4AI v0.7.7: The Self-Hosting & Monitoring Update - Crawl4AI Documentation (v0.8.x)](#-crawl4ai-v0-7-7-the-self-hosting-monitoring-update-crawl4ai-documentation-v0-8-x-) - [Markdown Generation - Crawl4AI Documentation (v0.8.x)](#markdown-generation-crawl4ai-documentation-v0-8-x-) - [Virtual Scroll - Crawl4AI Documentation (v0.8.x)](#virtual-scroll-crawl4ai-documentation-v0-8-x-) - [Demo Apps - Crawl4AI Documentation (v0.8.x)](#demo-apps-crawl4ai-documentation-v0-8-x-) - [LLM Strategies - Crawl4AI Documentation (v0.8.x)](#llm-strategies-crawl4ai-documentation-v0-8-x-) - [Deep Crawling - Crawl4AI Documentation (v0.8.x)](#deep-crawling-crawl4ai-documentation-v0-8-x-) - [Link & Media - Crawl4AI Documentation (v0.8.x)](#link-media-crawl4ai-documentation-v0-8-x-) - [Quick Start - Crawl4AI Documentation (v0.8.x)](#quick-start-crawl4ai-documentation-v0-8-x-) - [C4A-Script Reference - Crawl4AI Documentation (v0.8.x)](#c4a-script-reference-crawl4ai-documentation-v0-8-x-) - [Browser, Crawler & LLM Config - Crawl4AI Documentation (v0.8.x)](#browser-crawler-llm-config-crawl4ai-documentation-v0-8-x-) - [LLM-Free Strategies - Crawl4AI Documentation (v0.8.x)](#llm-free-strategies-crawl4ai-documentation-v0-8-x-) - [Crawl4AI v0.8.0 Release Notes - Crawl4AI Documentation (v0.8.x)](#crawl4ai-v0-8-0-release-notes-crawl4ai-documentation-v0-8-x-) - [URL Seeding - Crawl4AI Documentation (v0.8.x)](#url-seeding-crawl4ai-documentation-v0-8-x-) - [Crawl4AI v0.5.0 Release Notes - Crawl4AI Documentation (v0.8.x)](#crawl4ai-v0-5-0-release-notes-crawl4ai-documentation-v0-8-x-) - [Solving the Virtual Scroll Puzzle: How Crawl4AI Captures What Others Miss - Crawl4AI Documentation (v0.8.x)](#solving-the-virtual-scroll-puzzle-how-crawl4ai-captures-what-others-miss-crawl4ai-documentation-v0-8-x-) - [Migration Guide: Table Extraction v0.7.3 - Crawl4AI Documentation (v0.8.x)](#migration-guide-table-extraction-v0-7-3-crawl4ai-documentation-v0-8-x-) - [Self-Hosting Guide - Crawl4AI Documentation (v0.8.x)](#self-hosting-guide-crawl4ai-documentation-v0-8-x-) - [Brand Book - Crawl4AI Documentation (v0.8.x)](#brand-book-crawl4ai-documentation-v0-8-x-) - [C4A-Script - Crawl4AI Documentation (v0.8.x)](#c4a-script-crawl4ai-documentation-v0-8-x-) - [Crawl4AI v0.7.6 Release Notes - Crawl4AI Documentation (v0.8.x)](#crawl4ai-v0-7-6-release-notes-crawl4ai-documentation-v0-8-x-) - [Table Extraction Strategies - Crawl4AI Documentation (v0.8.x)](#table-extraction-strategies-crawl4ai-documentation-v0-8-x-) - [Blog Home - Crawl4AI Documentation (v0.8.x)](#blog-home-crawl4ai-documentation-v0-8-x-) - [Crawl4AI Complete SDK Documentation - Crawl4AI Documentation (v0.8.x)](#crawl4ai-complete-sdk-documentation-crawl4ai-documentation-v0-8-x-) - [C4A-Script Interactive Tutorial | Crawl4AI](#c4a-script-interactive-tutorial-crawl4ai) - [Crawl4AI LLM Context Builder](#crawl4ai-llm-context-builder) - [Marketplace - Crawl4AI](#marketplace-crawl4ai) - [Admin Dashboard - Crawl4AI Marketplace](#admin-dashboard-crawl4ai-marketplace) - [Crawl4AI Assistant - Chrome Extension for Visual Web Scraping](#crawl4ai-assistant-chrome-extension-for-visual-web-scraping) --- # Growth - Crawl4AI Documentation (v0.8.x) [unclecode/crawl4ai 61.8k 6.3k](https://github.com/unclecode/crawl4ai) ![Crawl4AI](https://docs.crawl4ai.com/assets/images/logo.png) Growth ====== Community growth & adoption metrics for Crawl4AI GitHub Stars 60,904 stargazers Monthly Downloads 914.2K PyPI · latest month Total Downloads 9.72M PyPI cumulative Docker Pulls 1.41M hub.docker.com Forks / Contributors 6,217 / 57 GitHub PyPI Monthly Downloads ---------------------- GitHub Star Growth ------------------ Cumulative PyPI Downloads ------------------------- Daily Download Trend -------------------- GitHub Traffic (14 days) ------------------------ Updated February 24, 2026 · Data from GitHub API, PyPI Stats, Docker Hub * * * \> Feedback [Ask AI](https://docs.crawl4ai.com/core/ask-ai/ "Ask Crawl4AI Assistant") --- # Adaptive Strategies - Crawl4AI Documentation (v0.8.x) [unclecode/crawl4ai 61.8k 6.3k](https://github.com/unclecode/crawl4ai) Advanced Adaptive Strategies ============================ Overview -------- While the default adaptive crawling configuration works well for most use cases, understanding the underlying strategies and scoring mechanisms allows you to fine-tune the crawler for specific domains and requirements. The Three-Layer Scoring System ------------------------------ ### 1\. Coverage Score Coverage measures how comprehensively your knowledge base covers the query terms and related concepts. #### Mathematical Foundation `Coverage(K, Q) = Σ(t ∈ Q) score(t, K) / |Q| where score(t, K) = doc_coverage(t) × (1 + freq_boost(t))` Copy #### Components * **Document Coverage**: Percentage of documents containing the term * **Frequency Boost**: Logarithmic bonus for term frequency * **Query Decomposition**: Handles multi-word queries intelligently #### Tuning Coverage `# For technical documentation with specific terminology config = AdaptiveConfig( confidence_threshold=0.85, # Require high coverage top_k_links=5 # Cast wider net ) # For general topics with synonyms config = AdaptiveConfig( confidence_threshold=0.6, # Lower threshold top_k_links=2 # More focused )` Copy ### 2\. Consistency Score Consistency evaluates whether the information across pages is coherent and non-contradictory. #### How It Works 1. Extracts key statements from each document 2. Compares statements across documents 3. Measures agreement vs. contradiction 4. Returns normalized score (0-1) #### Practical Impact * **High consistency (>0.8)**: Information is reliable and coherent * **Medium consistency (0.5-0.8)**: Some variation, but generally aligned * **Low consistency (<0.5)**: Conflicting information, need more sources ### 3\. Saturation Score Saturation detects when new pages stop providing novel information. #### Detection Algorithm `# Tracks new unique terms per page new_terms_page_1 = 50 new_terms_page_2 = 30 # 60% of first new_terms_page_3 = 15 # 50% of second new_terms_page_4 = 5 # 33% of third # Saturation detected: rapidly diminishing returns` Copy #### Configuration `config = AdaptiveConfig( min_gain_threshold=0.1 # Stop if <10% new information )` Copy Link Ranking Algorithm ---------------------- ### Expected Information Gain Each uncrawled link is scored based on: `ExpectedGain(link) = Relevance × Novelty × Authority` Copy #### 1\. Relevance Scoring Uses BM25 algorithm on link preview text: `relevance = BM25(link.preview_text, query)` Copy Factors: - Term frequency in preview - Inverse document frequency - Preview length normalization #### 2\. Novelty Estimation Measures how different the link appears from already-crawled content: `novelty = 1 - max_similarity(preview, knowledge_base)` Copy Prevents crawling duplicate or highly similar pages. #### 3\. Authority Calculation URL structure and domain analysis: `authority = f(domain_rank, url_depth, url_structure)` Copy Factors: - Domain reputation - URL depth (fewer slashes = higher authority) - Clean URL structure Domain-Specific Configurations ------------------------------ ### Technical Documentation `tech_doc_config = AdaptiveConfig( confidence_threshold=0.85, max_pages=30, top_k_links=3, min_gain_threshold=0.05 # Keep crawling for small gains )` Copy Rationale: - High threshold ensures comprehensive coverage - Lower gain threshold captures edge cases - Moderate link following for depth ### News & Articles `news_config = AdaptiveConfig( confidence_threshold=0.6, max_pages=10, top_k_links=5, min_gain_threshold=0.15 # Stop quickly on repetition )` Copy Rationale: - Lower threshold (articles often repeat information) - Higher gain threshold (avoid duplicate stories) - More links per page (explore different perspectives) ### E-commerce `ecommerce_config = AdaptiveConfig( confidence_threshold=0.7, max_pages=20, top_k_links=2, min_gain_threshold=0.1 )` Copy Rationale: - Balanced threshold for product variations - Focused link following (avoid infinite products) - Standard gain threshold ### Research & Academic `research_config = AdaptiveConfig( confidence_threshold=0.9, max_pages=50, top_k_links=4, min_gain_threshold=0.02 # Very low - capture citations )` Copy Rationale: - Very high threshold for completeness - Many pages allowed for thorough research - Very low gain threshold to capture references Performance Optimization ------------------------ ### Memory Management `# For large crawls, use streaming config = AdaptiveConfig( max_pages=100, save_state=True, state_path="large_crawl.json" ) # Periodically clean state if len(state.knowledge_base) > 1000: # Keep only the top 500 most relevant docs top_content = adaptive.get_relevant_content(top_k=500) keep_indices = {d["index"] for d in top_content} state.knowledge_base = [ doc for i, doc in enumerate(state.knowledge_base) if i in keep_indices ]` Copy ### Parallel Processing `# Use multiple start points start_urls = [ "https://docs.example.com/intro", "https://docs.example.com/api", "https://docs.example.com/guides" ] # Crawl in parallel tasks = [ adaptive.digest(url, query) for url in start_urls ] results = await asyncio.gather(*tasks)` Copy Debugging & Analysis -------------------- ### Enable Verbose Logging `import logging logging.basicConfig(level=logging.DEBUG) adaptive = AdaptiveCrawler(crawler, config, verbose=True)` Copy ### Analyze Crawl Patterns `# After crawling state = await adaptive.digest(start_url, query) # Analyze link selection print("Link selection order:") for i, url in enumerate(state.crawl_order): print(f"{i+1}. {url}") # Analyze term discovery print("\nTerm discovery rate:") for i, new_terms in enumerate(state.new_terms_history): print(f"Page {i+1}: {new_terms} new terms") # Analyze score progression print("\nScore progression:") print(f"Coverage: {state.metrics['coverage_history']}") print(f"Saturation: {state.metrics['saturation_history']}")` Copy ### Export for Analysis `# Export detailed metrics import json metrics = { "query": query, "total_pages": len(state.crawled_urls), "confidence": adaptive.confidence, "coverage_stats": adaptive.coverage_stats, "crawl_order": state.crawl_order, "term_frequencies": dict(state.term_frequencies), "new_terms_history": state.new_terms_history } with open("crawl_analysis.json", "w") as f: json.dump(metrics, f, indent=2)` Copy Custom Strategies ----------------- ### Implementing a Custom Strategy `from crawl4ai.adaptive_crawler import CrawlStrategy class DomainSpecificStrategy(CrawlStrategy): def calculate_coverage(self, state: CrawlState) -> float: # Custom coverage calculation # e.g., weight certain terms more heavily pass def calculate_consistency(self, state: CrawlState) -> float: # Custom consistency logic # e.g., domain-specific validation pass def rank_links(self, links: List[Link], state: CrawlState) -> List[Link]: # Custom link ranking # e.g., prioritize specific URL patterns pass # Use custom strategy adaptive = AdaptiveCrawler( crawler, config=config, strategy=DomainSpecificStrategy() )` Copy ### Combining Strategies `class HybridStrategy(CrawlStrategy): def __init__(self): self.strategies = [ TechnicalDocStrategy(), SemanticSimilarityStrategy(), URLPatternStrategy() ] def calculate_confidence(self, state: CrawlState) -> float: # Weighted combination of strategies scores = [s.calculate_confidence(state) for s in self.strategies] weights = [0.5, 0.3, 0.2] return sum(s * w for s, w in zip(scores, weights))` Copy Best Practices -------------- ### 1\. Start Conservative Begin with default settings and adjust based on results: `# Start with defaults result = await adaptive.digest(url, query) # Analyze and adjust if adaptive.confidence < 0.7: config.max_pages += 10 config.confidence_threshold -= 0.1` Copy ### 2\. Monitor Resource Usage `import psutil # Check memory before large crawls memory_percent = psutil.virtual_memory().percent if memory_percent > 80: config.max_pages = min(config.max_pages, 20)` Copy ### 3\. Use Domain Knowledge `# For API documentation if "api" in start_url: config.top_k_links = 2 # APIs have clear structure # For blogs if "blog" in start_url: config.min_gain_threshold = 0.2 # Avoid similar posts` Copy ### 4\. Validate Results `# Always validate the knowledge base relevant_content = adaptive.get_relevant_content(top_k=10) # Check coverage query_terms = set(query.lower().split()) covered_terms = set() for doc in relevant_content: content_lower = doc['content'].lower() for term in query_terms: if term in content_lower: covered_terms.add(term) coverage_ratio = len(covered_terms) / len(query_terms) print(f"Query term coverage: {coverage_ratio:.0%}")` Copy Next Steps ---------- * Explore [Custom Strategy Implementation](https://docs.crawl4ai.com/advanced/tutorials/custom-adaptive-strategies.md) * Learn about [Knowledge Base Management](https://docs.crawl4ai.com/advanced/tutorials/knowledge-base-management.md) * See [Performance Benchmarks](https://docs.crawl4ai.com/advanced/benchmarks/adaptive-performance.md) Page Copy Page Copy * [Copy as Markdown Copy page for LLMs](https://docs.crawl4ai.com/advanced/adaptive-strategies/#) * [View as Markdown Open raw source](https://docs.crawl4ai.com/advanced/adaptive-strategies/#) * [Open in ChatGPT Ask questions about this page](https://docs.crawl4ai.com/advanced/adaptive-strategies/#) ESC to close * * * \> Feedback [Ask AI](https://docs.crawl4ai.com/core/ask-ai/ "Ask Crawl4AI Assistant") --- # Home - Crawl4AI Documentation (v0.8.x) 🚀🤖 Crawl4AI: Open-Source LLM-Friendly Web Crawler & Scraper ============================================================= [![unclecode%2Fcrawl4ai | Trendshift](https://trendshift.io/api/badge/repositories/11716)](https://trendshift.io/repositories/11716) [![GitHub Stars](https://img.shields.io/github/stars/unclecode/crawl4ai?style=social)](https://github.com/unclecode/crawl4ai/stargazers) [![GitHub Forks](https://img.shields.io/github/forks/unclecode/crawl4ai?style=social)](https://github.com/unclecode/crawl4ai/network/members) [![PyPI version](https://badge.fury.io/py/crawl4ai.svg)](https://badge.fury.io/py/crawl4ai) [![Python Version](https://img.shields.io/pypi/pyversions/crawl4ai)](https://pypi.org/project/crawl4ai/) [![Downloads](https://static.pepy.tech/badge/crawl4ai/month)](https://pepy.tech/project/crawl4ai) [![License](https://img.shields.io/github/license/unclecode/crawl4ai)](https://github.com/unclecode/crawl4ai/blob/main/LICENSE) [![Follow on X](https://img.shields.io/badge/Follow%20on%20X-000000?style=for-the-badge&logo=x&logoColor=white)](https://x.com/crawl4ai) [![Follow on LinkedIn](https://img.shields.io/badge/Follow%20on%20LinkedIn-0077B5?style=for-the-badge&logo=linkedin&logoColor=white)](https://www.linkedin.com/company/crawl4ai) [![Join our Discord](https://img.shields.io/badge/Join%20our%20Discord-5865F2?style=for-the-badge&logo=discord&logoColor=white)](https://discord.gg/jP8KfhDhyN) * * * #### 🚀 Crawl4AI Cloud API — Closed Beta (Launching Soon) Reliable, large-scale web extraction, now built to be _**drastically more cost-effective**_ than any of the existing solutions. 👉 **Apply [here](https://forms.gle/E9MyPaNXACnAMaqG7) for early access** _We’ll be onboarding in phases and working closely with early users. Limited slots._ * * * Crawl4AI is the #1 trending GitHub repository, actively maintained by a vibrant community. It delivers blazing-fast, AI-ready web crawling tailored for large language models, AI agents, and data pipelines. Fully open source, flexible, and built for real-time performance, **Crawl4AI** empowers developers with unmatched speed, precision, and deployment ease. > Enjoy using Crawl4AI? Consider **[becoming a sponsor](https://github.com/sponsors/unclecode) > ** to support ongoing development and community growth! 🆕 AI Assistant Skill Now Available! ------------------------------------ ### 🤖 Crawl4AI Skill for Claude & AI Assistants Supercharge your AI coding assistant with complete Crawl4AI knowledge! Download our comprehensive skill package that includes: * 📚 Complete SDK reference (23K+ words) * 🚀 Ready-to-use extraction scripts * ⚡ Schema generation for efficient scraping * 🔧 Version 0.7.4 compatible [📦 Download Skill Package](https://docs.crawl4ai.com/assets/crawl4ai-skill.zip) Works with Claude, Cursor, Windsurf, and other AI coding assistants. Import the .zip file into your AI assistant's skill/knowledge system. 🎯 New: Adaptive Web Crawling ----------------------------- Crawl4AI now features intelligent adaptive crawling that knows when to stop! Using advanced information foraging algorithms, it determines when sufficient information has been gathered to answer your query. [Learn more about Adaptive Crawling →](https://docs.crawl4ai.com/core/adaptive-crawling/) Quick Start ----------- Here's a quick example to show you how easy it is to use Crawl4AI with its asynchronous capabilities: `[](https://docs.crawl4ai.com/#__codelineno-0-1) import asyncio [](https://docs.crawl4ai.com/#__codelineno-0-2) from crawl4ai import AsyncWebCrawler [](https://docs.crawl4ai.com/#__codelineno-0-3) [](https://docs.crawl4ai.com/#__codelineno-0-4) async def main(): [](https://docs.crawl4ai.com/#__codelineno-0-5) # Create an instance of AsyncWebCrawler [](https://docs.crawl4ai.com/#__codelineno-0-6) async with AsyncWebCrawler() as crawler: [](https://docs.crawl4ai.com/#__codelineno-0-7) # Run the crawler on a URL [](https://docs.crawl4ai.com/#__codelineno-0-8) result = await crawler.arun(url="https://crawl4ai.com") [](https://docs.crawl4ai.com/#__codelineno-0-9) [](https://docs.crawl4ai.com/#__codelineno-0-10) # Print the extracted content [](https://docs.crawl4ai.com/#__codelineno-0-11) print(result.markdown) [](https://docs.crawl4ai.com/#__codelineno-0-12) [](https://docs.crawl4ai.com/#__codelineno-0-13) # Run the async main function [](https://docs.crawl4ai.com/#__codelineno-0-14) asyncio.run(main())` * * * Video Tutorial -------------- * * * What Does Crawl4AI Do? ---------------------- Crawl4AI is a feature-rich crawler and scraper that aims to: 1. **Generate Clean Markdown**: Perfect for RAG pipelines or direct ingestion into LLMs. 2. **Structured Extraction**: Parse repeated patterns with CSS, XPath, or LLM-based extraction. 3. **Advanced Browser Control**: Hooks, proxies, stealth modes, session re-use—fine-grained control. 4. **High Performance**: Parallel crawling, chunk-based extraction, real-time use cases. 5. **Open Source**: No forced API keys, no paywalls—everyone can access their data. **Core Philosophies**: - **Democratize Data**: Free to use, transparent, and highly configurable. \- **LLM Friendly**: Minimally processed, well-structured text, images, and metadata, so AI models can easily consume it. * * * Documentation Structure ----------------------- To help you get started, we’ve organized our docs into clear sections: * **Setup & Installation** Basic instructions to install Crawl4AI via pip or Docker. * **Quick Start** A hands-on introduction showing how to do your first crawl, generate Markdown, and do a simple extraction. * **Core** Deeper guides on single-page crawling, advanced browser/crawler parameters, content filtering, and caching. * **Advanced** Explore link & media handling, lazy loading, hooking & authentication, proxies, session management, and more. * **Extraction** Detailed references for no-LLM (CSS, XPath) vs. LLM-based strategies, chunking, and clustering approaches. * **API Reference** Find the technical specifics of each class and method, including `AsyncWebCrawler`, `arun()`, and `CrawlResult`. Throughout these sections, you’ll find code samples you can **copy-paste** into your environment. If something is missing or unclear, raise an issue or PR. * * * How You Can Support ------------------- * **Star & Fork**: If you find Crawl4AI helpful, star the repo on GitHub or fork it to add your own features. * **File Issues**: Encounter a bug or missing feature? Let us know by filing an issue, so we can improve. * **Pull Requests**: Whether it’s a small fix, a big feature, or better docs—contributions are always welcome. * **Join Discord**: Come chat about web scraping, crawling tips, or AI workflows with the community. * **Spread the Word**: Mention Crawl4AI in your blog posts, talks, or on social media. **Our mission**: to empower everyone—students, researchers, entrepreneurs, data scientists—to access, parse, and shape the world’s data with speed, cost-efficiency, and creative freedom. * * * Quick Links ----------- * **[GitHub Repo](https://github.com/unclecode/crawl4ai) ** * **[Installation Guide](https://docs.crawl4ai.com/core/installation/) ** * **[Quick Start](https://docs.crawl4ai.com/core/quickstart/) ** * **[API Reference](https://docs.crawl4ai.com/api/async-webcrawler/) ** * **[Changelog](https://github.com/unclecode/crawl4ai/blob/main/CHANGELOG.md) ** Thank you for joining me on this journey. Let’s keep building an **open, democratic** approach to data extraction and AI together. Happy Crawling! — _Unclecode, Founder & Maintainer of Crawl4AI_ * * * \> Feedback --- # Contributing Guide - Crawl4AI Documentation (v0.8.x) Contributing to Crawl4AI ======================== Welcome to the Crawl4AI project! As an open-source library for web crawling and AI integration, we value contributions from the community. This guide explains our branching strategy, how to contribute effectively, and the overall release process. Our goal is to maintain a stable, collaborative environment where bug fixes, features, and improvements can be integrated smoothly while allowing for experimental development. We follow a GitFlow-inspired workflow to ensure predictability and quality. Releases occur approximately every two weeks, with a focus on semantic versioning, comprehensive documentation, and user-friendly updates. Core Branches ------------- * **main**: The stable branch containing production-ready code. It's always identical to the latest released version and is tagged for releases. Do not submit PRs directly here. * **develop**: The primary integration branch for ongoing development. This is where all contributions (bug fixes, minor features, documentation updates) are merged. Submit your pull requests targeting this branch. * **next**: Reserved for the lead maintainer (Unclecode) to experiment with major features, refactors, or cutting-edge changes. These are merged into `develop` when ready. * **release/vX.Y.Z**: Temporary branches created from `develop` for final release preparations (e.g., version bumps, demos, release notes). These are short-lived and deleted after the release. Contributor Workflow -------------------- We encourage contributions of all kinds: bug fixes, new features, documentation improvements, tests, or even Docker enhancements. Follow these steps to contribute: 1. **Fork the Repository**: Create your own fork on GitHub. 2. **Create a Branch**: Base your work on the `develop` branch. `[](https://docs.crawl4ai.com/CONTRIBUTING/#__codelineno-0-1) git checkout develop [](https://docs.crawl4ai.com/CONTRIBUTING/#__codelineno-0-2) git checkout -b feature/your-feature-name # Or bugfix/your-bugfix-name` 3. **Make Changes**: * Implement your feature or fix. * If updating documentation (e.g., README.md, mkdocs.yml, or docs/blog/), ensure version references are consistent (e.g., update site\_name in mkdocs.yml to reflect the upcoming version if relevant). * For Docker-related changes (e.g., Dockerfile, docker-compose.yml, or docs/md\_v2/core/docker-deployment.md), test locally and include build instructions in your PR description. * Add tests if applicable (run `pytest` to verify). * Follow code style guidelines (use black for formatting). 4. **Commit and Push**: * Use descriptive commit messages (e.g., "Fix: Resolve issue with async crawling"). * Push to your fork: `git push origin feature/your-feature-name`. 5. **Submit a Pull Request**: * Target the `develop` branch. * Provide a clear description: What does it do? Link to any issues. Include screenshots or code examples if helpful. * If your change affects documentation or Docker, mention how it aligns with the version (e.g., "Updates Docker docs for v0.7.0 compatibility"). * We'll review and merge approved PRs into `develop`. 6. **Discuss Large Changes**: For major features or experimental ideas, open an issue first to align with the project's direction. If your PR involves breaking changes, include a migration guide in the description. Lead Maintainer's Workflow (For Reference) ------------------------------------------ * The lead maintainer (Unclecode) uses the `next` branch for isolated experimental work. * Features from `next` are periodically merged into `develop` (via rebase and merge) to keep everything in sync. * This isolation ensures your contributions aren't disrupted by ongoing major changes. Release Process (High-Level Overview) ------------------------------------- Releases happen bi-weekly to ship improvements regularly. As a contributor, your merged changes in `develop` will be included in the next release unless specified otherwise. Here's a summary of what happens: * **Preparation**: A temporary `release/vX.Y.Z` branch is created from `develop`. Any ready features from `next` are merged here. * **Final Updates**: * Version bump in code (e.g., `__version__.py`). * Creation of a demo script in `examples/` to showcase new features. * Writing release notes in `docs/blog/` (personal "I" voice from Unclecode, with code examples, impacts, and migration guides if needed). * Documentation updates: README.md (highlights, version refs), mkdocs.yml (site\_name with version), docs/blog/index.md (add new release), and copying notes to `docs/md_v2/blog/releases/`. * Docker updates: Dockerfile (version arg), docker-compose.yml, deploy/docker/README.md, and docs/md\_v2/core/docker-deployment.md. A release candidate image (e.g., `X.Y.Z-r1`) is built and tested. * **Testing and Merge**: Full tests run; changes committed and merged to `main` with a tag. * **Publication**: Tagged release on GitHub (with notes), publish to PyPI, and push Docker images (stable and `latest` after testing). * **Sync**: Back-merge to `develop` and reset `next` for the next cycle. Semantic versioning is used: MAJOR for breaking changes, MINOR for features, PATCH for fixes. Pre-releases (e.g., `-rc1`) may be used for testing. If your contribution requires Docker testing or affects docs, it may be part of this step—feel free to suggest updates in your PR. Benefits of This Approach ------------------------- * **Stability**: `main` is always reliable for users. * **Collaboration**: Fixed PR target (`develop`) makes contributing straightforward. * **Isolation**: Experimental work in `next` doesn't block team progress. * **User-Focused**: Releases include demos, detailed notes, and updated docs/Docker for easy adoption. * **Predictability**: Bi-weekly cadence keeps the project active. Checklist for Contributors -------------------------- Before submitting a PR: * \[ \] Based on and targeting `develop`. * \[ \] Tests pass (`pytest`). * \[ \] Docs updated if needed (e.g., version refs in mkdocs.yml, Docker files). * \[ \] No breaking changes without a migration guide. * \[ \] Descriptive title and description. Common Issues ------------- * **Merge Conflicts**: Rebase your branch on latest `develop` before PR. * **Docker Builds**: Test multi-arch (amd64/arm64) locally if changing Dockerfile. * **Version Consistency**: Ensure any version mentions match semantic rules. Communication ------------- * Open issues for discussions or bugs. * Join our Discord (link in README) for real-time help. * After releases, announcements go to GitHub, Discord, and social media. Thanks for contributing to Crawl4AI — we appreciate your help in making it better! _Last Updated: Feb 3, 2026_ * * * \> Feedback --- # Crawl Dispatcher - Crawl4AI Documentation (v0.8.x) Crawl Dispatcher ================ We’re excited to announce a **Crawl Dispatcher** module that can handle **thousands** of crawling tasks simultaneously. By efficiently managing system resources (memory, CPU, network), this dispatcher ensures high-performance data extraction at scale. It also provides **real-time monitoring** of each crawler’s status, memory usage, and overall progress. Stay tuned—this feature is **coming soon** in an upcoming release of Crawl4AI! For the latest news, keep an eye on our changelogs and follow [@unclecode](https://twitter.com/unclecode) on X. Below is a **sample** of how the dispatcher’s performance monitor might look in action: ![Crawl Dispatcher Performance Monitor](https://docs.crawl4ai.com/assets/images/dispatcher.png) We can’t wait to bring you this streamlined, **scalable** approach to multi-URL crawling—**watch this space** for updates! * * * \> Feedback --- # File Downloading - Crawl4AI Documentation (v0.8.x) Download Handling in Crawl4AI ============================= This guide explains how to use Crawl4AI to handle file downloads during crawling. You'll learn how to trigger downloads, specify download locations, and access downloaded files. Enabling Downloads ------------------ To enable downloads, set the `accept_downloads` parameter in the `BrowserConfig` object and pass it to the crawler. `[](https://docs.crawl4ai.com/advanced/file-downloading/#__codelineno-0-1) from crawl4ai.async_configs import BrowserConfig, AsyncWebCrawler [](https://docs.crawl4ai.com/advanced/file-downloading/#__codelineno-0-2) [](https://docs.crawl4ai.com/advanced/file-downloading/#__codelineno-0-3) async def main(): [](https://docs.crawl4ai.com/advanced/file-downloading/#__codelineno-0-4) config = BrowserConfig(accept_downloads=True) # Enable downloads globally [](https://docs.crawl4ai.com/advanced/file-downloading/#__codelineno-0-5) async with AsyncWebCrawler(config=config) as crawler: [](https://docs.crawl4ai.com/advanced/file-downloading/#__codelineno-0-6) # ... your crawling logic ... [](https://docs.crawl4ai.com/advanced/file-downloading/#__codelineno-0-7) [](https://docs.crawl4ai.com/advanced/file-downloading/#__codelineno-0-8) asyncio.run(main())` Specifying Download Location ---------------------------- Specify the download directory using the `downloads_path` attribute in the `BrowserConfig` object. If not provided, Crawl4AI defaults to creating a "downloads" directory inside the `.crawl4ai` folder in your home directory. `[](https://docs.crawl4ai.com/advanced/file-downloading/#__codelineno-1-1) from crawl4ai.async_configs import BrowserConfig [](https://docs.crawl4ai.com/advanced/file-downloading/#__codelineno-1-2) import os [](https://docs.crawl4ai.com/advanced/file-downloading/#__codelineno-1-3) [](https://docs.crawl4ai.com/advanced/file-downloading/#__codelineno-1-4) downloads_path = os.path.join(os.getcwd(), "my_downloads") # Custom download path [](https://docs.crawl4ai.com/advanced/file-downloading/#__codelineno-1-5) os.makedirs(downloads_path, exist_ok=True) [](https://docs.crawl4ai.com/advanced/file-downloading/#__codelineno-1-6) [](https://docs.crawl4ai.com/advanced/file-downloading/#__codelineno-1-7) config = BrowserConfig(accept_downloads=True, downloads_path=downloads_path) [](https://docs.crawl4ai.com/advanced/file-downloading/#__codelineno-1-8) [](https://docs.crawl4ai.com/advanced/file-downloading/#__codelineno-1-9) async def main(): [](https://docs.crawl4ai.com/advanced/file-downloading/#__codelineno-1-10) async with AsyncWebCrawler(config=config) as crawler: [](https://docs.crawl4ai.com/advanced/file-downloading/#__codelineno-1-11) result = await crawler.arun(url="https://example.com") [](https://docs.crawl4ai.com/advanced/file-downloading/#__codelineno-1-12) # ...` Triggering Downloads -------------------- Downloads are typically triggered by user interactions on a web page, such as clicking a download button. Use `js_code` in `CrawlerRunConfig` to simulate these actions and `wait_for` to allow sufficient time for downloads to start. `[](https://docs.crawl4ai.com/advanced/file-downloading/#__codelineno-2-1) from crawl4ai.async_configs import CrawlerRunConfig [](https://docs.crawl4ai.com/advanced/file-downloading/#__codelineno-2-2) [](https://docs.crawl4ai.com/advanced/file-downloading/#__codelineno-2-3) config = CrawlerRunConfig( [](https://docs.crawl4ai.com/advanced/file-downloading/#__codelineno-2-4) js_code=""" [](https://docs.crawl4ai.com/advanced/file-downloading/#__codelineno-2-5) const downloadLink = document.querySelector('a[href$=".exe"]'); [](https://docs.crawl4ai.com/advanced/file-downloading/#__codelineno-2-6) if (downloadLink) { [](https://docs.crawl4ai.com/advanced/file-downloading/#__codelineno-2-7) downloadLink.click(); [](https://docs.crawl4ai.com/advanced/file-downloading/#__codelineno-2-8) } [](https://docs.crawl4ai.com/advanced/file-downloading/#__codelineno-2-9) """, [](https://docs.crawl4ai.com/advanced/file-downloading/#__codelineno-2-10) wait_for=5 # Wait 5 seconds for the download to start [](https://docs.crawl4ai.com/advanced/file-downloading/#__codelineno-2-11) ) [](https://docs.crawl4ai.com/advanced/file-downloading/#__codelineno-2-12) [](https://docs.crawl4ai.com/advanced/file-downloading/#__codelineno-2-13) result = await crawler.arun(url="https://www.python.org/downloads/", config=config)` Accessing Downloaded Files -------------------------- The `downloaded_files` attribute of the `CrawlResult` object contains paths to downloaded files. `[](https://docs.crawl4ai.com/advanced/file-downloading/#__codelineno-3-1) if result.downloaded_files: [](https://docs.crawl4ai.com/advanced/file-downloading/#__codelineno-3-2) print("Downloaded files:") [](https://docs.crawl4ai.com/advanced/file-downloading/#__codelineno-3-3) for file_path in result.downloaded_files: [](https://docs.crawl4ai.com/advanced/file-downloading/#__codelineno-3-4) print(f"- {file_path}") [](https://docs.crawl4ai.com/advanced/file-downloading/#__codelineno-3-5) file_size = os.path.getsize(file_path) [](https://docs.crawl4ai.com/advanced/file-downloading/#__codelineno-3-6) print(f"- File size: {file_size} bytes") [](https://docs.crawl4ai.com/advanced/file-downloading/#__codelineno-3-7) else: [](https://docs.crawl4ai.com/advanced/file-downloading/#__codelineno-3-8) print("No files downloaded.")` Example: Downloading Multiple Files ----------------------------------- `[](https://docs.crawl4ai.com/advanced/file-downloading/#__codelineno-4-1) from crawl4ai.async_configs import BrowserConfig, CrawlerRunConfig [](https://docs.crawl4ai.com/advanced/file-downloading/#__codelineno-4-2) import os [](https://docs.crawl4ai.com/advanced/file-downloading/#__codelineno-4-3) from pathlib import Path [](https://docs.crawl4ai.com/advanced/file-downloading/#__codelineno-4-4) [](https://docs.crawl4ai.com/advanced/file-downloading/#__codelineno-4-5) async def download_multiple_files(url: str, download_path: str): [](https://docs.crawl4ai.com/advanced/file-downloading/#__codelineno-4-6) config = BrowserConfig(accept_downloads=True, downloads_path=download_path) [](https://docs.crawl4ai.com/advanced/file-downloading/#__codelineno-4-7) async with AsyncWebCrawler(config=config) as crawler: [](https://docs.crawl4ai.com/advanced/file-downloading/#__codelineno-4-8) run_config = CrawlerRunConfig( [](https://docs.crawl4ai.com/advanced/file-downloading/#__codelineno-4-9) js_code=""" [](https://docs.crawl4ai.com/advanced/file-downloading/#__codelineno-4-10) const downloadLinks = document.querySelectorAll('a[download]'); [](https://docs.crawl4ai.com/advanced/file-downloading/#__codelineno-4-11) for (const link of downloadLinks) { [](https://docs.crawl4ai.com/advanced/file-downloading/#__codelineno-4-12) link.click(); [](https://docs.crawl4ai.com/advanced/file-downloading/#__codelineno-4-13) // Delay between clicks [](https://docs.crawl4ai.com/advanced/file-downloading/#__codelineno-4-14) await new Promise(r => setTimeout(r, 2000)); [](https://docs.crawl4ai.com/advanced/file-downloading/#__codelineno-4-15) } [](https://docs.crawl4ai.com/advanced/file-downloading/#__codelineno-4-16) """, [](https://docs.crawl4ai.com/advanced/file-downloading/#__codelineno-4-17) wait_for=10 # Wait for all downloads to start [](https://docs.crawl4ai.com/advanced/file-downloading/#__codelineno-4-18) ) [](https://docs.crawl4ai.com/advanced/file-downloading/#__codelineno-4-19) result = await crawler.arun(url=url, config=run_config) [](https://docs.crawl4ai.com/advanced/file-downloading/#__codelineno-4-20) [](https://docs.crawl4ai.com/advanced/file-downloading/#__codelineno-4-21) if result.downloaded_files: [](https://docs.crawl4ai.com/advanced/file-downloading/#__codelineno-4-22) print("Downloaded files:") [](https://docs.crawl4ai.com/advanced/file-downloading/#__codelineno-4-23) for file in result.downloaded_files: [](https://docs.crawl4ai.com/advanced/file-downloading/#__codelineno-4-24) print(f"- {file}") [](https://docs.crawl4ai.com/advanced/file-downloading/#__codelineno-4-25) else: [](https://docs.crawl4ai.com/advanced/file-downloading/#__codelineno-4-26) print("No files downloaded.") [](https://docs.crawl4ai.com/advanced/file-downloading/#__codelineno-4-27) [](https://docs.crawl4ai.com/advanced/file-downloading/#__codelineno-4-28) # Usage [](https://docs.crawl4ai.com/advanced/file-downloading/#__codelineno-4-29) download_path = os.path.join(Path.home(), ".crawl4ai", "downloads") [](https://docs.crawl4ai.com/advanced/file-downloading/#__codelineno-4-30) os.makedirs(download_path, exist_ok=True) [](https://docs.crawl4ai.com/advanced/file-downloading/#__codelineno-4-31) [](https://docs.crawl4ai.com/advanced/file-downloading/#__codelineno-4-32) asyncio.run(download_multiple_files("https://www.python.org/downloads/windows/", download_path))` Important Considerations ------------------------ * **Browser Context:** Downloads are managed within the browser context. Ensure `js_code` correctly targets the download triggers on the webpage. * **Timing:** Use `wait_for` in `CrawlerRunConfig` to manage download timing. * **Error Handling:** Handle errors to manage failed downloads or incorrect paths gracefully. * **Security:** Scan downloaded files for potential security threats before use. This revised guide ensures consistency with the `Crawl4AI` codebase by using `BrowserConfig` and `CrawlerRunConfig` for all download-related configurations. Let me know if further adjustments are needed! * * * \> Feedback --- # Anti-Bot & Fallback - Crawl4AI Documentation (v0.8.x) Anti-Bot Detection & Fallback ============================= When crawling sites protected by anti-bot systems (Akamai, Cloudflare, PerimeterX, DataDome, Imperva, etc.), requests often get blocked with CAPTCHAs, 403 responses, or empty pages. Crawl4AI provides a layered retry and fallback system that automatically detects blocking and escalates through multiple strategies until content is retrieved. How Detection Works ------------------- After each crawl attempt, Crawl4AI inspects the HTTP status code and HTML content for known anti-bot signals: * **HTTP 403/429** with short or empty response bodies * **Challenge pages** — Cloudflare "Just a moment", Akamai "Access Denied", PerimeterX block pages * **CAPTCHA injection** — reCAPTCHA, hCaptcha, or vendor-specific challenges on otherwise empty pages * **Firewall blocks** — Imperva/Incapsula resource iframes, Sucuri firewall pages, Cloudflare error codes Detection uses structural HTML markers (specific element IDs, script sources, form actions) rather than generic keywords to minimize false positives. A normal page that happens to mention "CAPTCHA" or "Cloudflare" in its content will not be flagged. When all attempts fail and blocking is still detected, the result is returned with `success=False` and `error_message` describing the block reason. Configuration Options --------------------- All anti-bot retry options live on `CrawlerRunConfig`: | Parameter | Type | Default | Description | | --- | --- | --- | --- | | `proxy_config` | `ProxyConfig`, `list[ProxyConfig]`, or `None` | `None` | Single proxy or ordered list of proxies to try. Each retry round iterates through the full list. Use `"direct"` or `ProxyConfig.DIRECT` in a list to explicitly try without a proxy. | | `max_retries` | `int` | `0` | Number of retry rounds when blocking is detected. `0` = no retries. | | `fallback_fetch_function` | `async (str) -> str` | `None` | Async function called as last resort. Takes URL, returns raw HTML. | Escalation Chain ---------------- Each retry round tries every proxy in `proxy_config` in order. If all rounds are exhausted and the page is still blocked, the fallback fetch function is called as a last resort. `[](https://docs.crawl4ai.com/advanced/anti-bot-and-fallback/#__codelineno-0-1) For each round (1 + max_retries rounds): [](https://docs.crawl4ai.com/advanced/anti-bot-and-fallback/#__codelineno-0-2) 1. Try proxy_config[0] (or direct if proxy_config is None) [](https://docs.crawl4ai.com/advanced/anti-bot-and-fallback/#__codelineno-0-3) 2. If blocked → try proxy_config[1] [](https://docs.crawl4ai.com/advanced/anti-bot-and-fallback/#__codelineno-0-4) 3. If blocked → try proxy_config[2] [](https://docs.crawl4ai.com/advanced/anti-bot-and-fallback/#__codelineno-0-5) 4. ... continue through all proxies [](https://docs.crawl4ai.com/advanced/anti-bot-and-fallback/#__codelineno-0-6) 5. If any attempt succeeds → done [](https://docs.crawl4ai.com/advanced/anti-bot-and-fallback/#__codelineno-0-7) [](https://docs.crawl4ai.com/advanced/anti-bot-and-fallback/#__codelineno-0-8) If all rounds exhausted and still blocked: [](https://docs.crawl4ai.com/advanced/anti-bot-and-fallback/#__codelineno-0-9) 6. Call fallback_fetch_function(url) → process returned HTML` Worst-case attempts before the fetch function: `(1 + max_retries) x len(proxy_config)` Crawl Stats ----------- Every crawl result includes a `crawl_stats` dict with detailed attempt tracking: `[](https://docs.crawl4ai.com/advanced/anti-bot-and-fallback/#__codelineno-1-1) result.crawl_stats = { [](https://docs.crawl4ai.com/advanced/anti-bot-and-fallback/#__codelineno-1-2) "attempts": 3, # total browser attempts made [](https://docs.crawl4ai.com/advanced/anti-bot-and-fallback/#__codelineno-1-3) "retries": 1, # retry rounds used (0 = succeeded first round) [](https://docs.crawl4ai.com/advanced/anti-bot-and-fallback/#__codelineno-1-4) "proxies_used": [ # ordered list of every attempt [](https://docs.crawl4ai.com/advanced/anti-bot-and-fallback/#__codelineno-1-5) {"proxy": None, "status_code": 403, "blocked": True, "reason": "Akamai block (Reference #)"}, [](https://docs.crawl4ai.com/advanced/anti-bot-and-fallback/#__codelineno-1-6) {"proxy": "proxy.io:8080", "status_code": 403, "blocked": True, "reason": "Akamai block (Reference #)"}, [](https://docs.crawl4ai.com/advanced/anti-bot-and-fallback/#__codelineno-1-7) {"proxy": "premium.io:9090", "status_code": 200, "blocked": False, "reason": ""}, [](https://docs.crawl4ai.com/advanced/anti-bot-and-fallback/#__codelineno-1-8) ], [](https://docs.crawl4ai.com/advanced/anti-bot-and-fallback/#__codelineno-1-9) "fallback_fetch_used": False, # whether fallback_fetch_function was called [](https://docs.crawl4ai.com/advanced/anti-bot-and-fallback/#__codelineno-1-10) "resolved_by": "proxy", # "direct" | "proxy" | "fallback_fetch" | null (all failed) [](https://docs.crawl4ai.com/advanced/anti-bot-and-fallback/#__codelineno-1-11) }` Usage Examples -------------- ### Simple Retry (No Proxy) Retry the crawl up to 3 times when blocking is detected. Useful when blocks are intermittent or IP-based. `[](https://docs.crawl4ai.com/advanced/anti-bot-and-fallback/#__codelineno-2-1) from crawl4ai import AsyncWebCrawler [](https://docs.crawl4ai.com/advanced/anti-bot-and-fallback/#__codelineno-2-2) from crawl4ai.async_configs import BrowserConfig, CrawlerRunConfig [](https://docs.crawl4ai.com/advanced/anti-bot-and-fallback/#__codelineno-2-3) [](https://docs.crawl4ai.com/advanced/anti-bot-and-fallback/#__codelineno-2-4) async with AsyncWebCrawler(config=BrowserConfig(headless=True)) as crawler: [](https://docs.crawl4ai.com/advanced/anti-bot-and-fallback/#__codelineno-2-5) result = await crawler.arun( [](https://docs.crawl4ai.com/advanced/anti-bot-and-fallback/#__codelineno-2-6) url="https://example.com", [](https://docs.crawl4ai.com/advanced/anti-bot-and-fallback/#__codelineno-2-7) config=CrawlerRunConfig(max_retries=3), [](https://docs.crawl4ai.com/advanced/anti-bot-and-fallback/#__codelineno-2-8) )` ### Single Proxy Pass a single `ProxyConfig` — it's used on every attempt. Same behavior as always. `[](https://docs.crawl4ai.com/advanced/anti-bot-and-fallback/#__codelineno-3-1) from crawl4ai.async_configs import ProxyConfig [](https://docs.crawl4ai.com/advanced/anti-bot-and-fallback/#__codelineno-3-2) [](https://docs.crawl4ai.com/advanced/anti-bot-and-fallback/#__codelineno-3-3) config = CrawlerRunConfig( [](https://docs.crawl4ai.com/advanced/anti-bot-and-fallback/#__codelineno-3-4) max_retries=2, [](https://docs.crawl4ai.com/advanced/anti-bot-and-fallback/#__codelineno-3-5) proxy_config=ProxyConfig( [](https://docs.crawl4ai.com/advanced/anti-bot-and-fallback/#__codelineno-3-6) server="http://proxy.example.com:8080", [](https://docs.crawl4ai.com/advanced/anti-bot-and-fallback/#__codelineno-3-7) username="user", [](https://docs.crawl4ai.com/advanced/anti-bot-and-fallback/#__codelineno-3-8) password="pass", [](https://docs.crawl4ai.com/advanced/anti-bot-and-fallback/#__codelineno-3-9) ), [](https://docs.crawl4ai.com/advanced/anti-bot-and-fallback/#__codelineno-3-10) )` ### Direct-First, Then Proxies Try without a proxy first, then escalate to proxies if blocked. Use `ProxyConfig.DIRECT` (or the string `"direct"`) in the list to represent a no-proxy attempt. `[](https://docs.crawl4ai.com/advanced/anti-bot-and-fallback/#__codelineno-4-1) config = CrawlerRunConfig( [](https://docs.crawl4ai.com/advanced/anti-bot-and-fallback/#__codelineno-4-2) max_retries=1, [](https://docs.crawl4ai.com/advanced/anti-bot-and-fallback/#__codelineno-4-3) proxy_config=[ [](https://docs.crawl4ai.com/advanced/anti-bot-and-fallback/#__codelineno-4-4) ProxyConfig.DIRECT, # Try without proxy first [](https://docs.crawl4ai.com/advanced/anti-bot-and-fallback/#__codelineno-4-5) ProxyConfig( [](https://docs.crawl4ai.com/advanced/anti-bot-and-fallback/#__codelineno-4-6) server="http://datacenter-proxy.example.com:8080", [](https://docs.crawl4ai.com/advanced/anti-bot-and-fallback/#__codelineno-4-7) username="user", [](https://docs.crawl4ai.com/advanced/anti-bot-and-fallback/#__codelineno-4-8) password="pass", [](https://docs.crawl4ai.com/advanced/anti-bot-and-fallback/#__codelineno-4-9) ), [](https://docs.crawl4ai.com/advanced/anti-bot-and-fallback/#__codelineno-4-10) ProxyConfig( [](https://docs.crawl4ai.com/advanced/anti-bot-and-fallback/#__codelineno-4-11) server="http://residential-proxy.example.com:9090", [](https://docs.crawl4ai.com/advanced/anti-bot-and-fallback/#__codelineno-4-12) username="user", [](https://docs.crawl4ai.com/advanced/anti-bot-and-fallback/#__codelineno-4-13) password="pass", [](https://docs.crawl4ai.com/advanced/anti-bot-and-fallback/#__codelineno-4-14) ), [](https://docs.crawl4ai.com/advanced/anti-bot-and-fallback/#__codelineno-4-15) ], [](https://docs.crawl4ai.com/advanced/anti-bot-and-fallback/#__codelineno-4-16) )` With this setup, each round tries direct first, then datacenter, then residential. With `max_retries=1`, worst case is 2 rounds x 3 steps = 6 attempts. ### Proxy List (Escalation) Pass a list of proxies. They're tried in order — first one that works wins. Within each retry round, the entire list is tried again. `[](https://docs.crawl4ai.com/advanced/anti-bot-and-fallback/#__codelineno-5-1) config = CrawlerRunConfig( [](https://docs.crawl4ai.com/advanced/anti-bot-and-fallback/#__codelineno-5-2) max_retries=1, [](https://docs.crawl4ai.com/advanced/anti-bot-and-fallback/#__codelineno-5-3) proxy_config=[ [](https://docs.crawl4ai.com/advanced/anti-bot-and-fallback/#__codelineno-5-4) ProxyConfig( [](https://docs.crawl4ai.com/advanced/anti-bot-and-fallback/#__codelineno-5-5) server="http://datacenter-proxy.example.com:8080", [](https://docs.crawl4ai.com/advanced/anti-bot-and-fallback/#__codelineno-5-6) username="user", [](https://docs.crawl4ai.com/advanced/anti-bot-and-fallback/#__codelineno-5-7) password="pass", [](https://docs.crawl4ai.com/advanced/anti-bot-and-fallback/#__codelineno-5-8) ), [](https://docs.crawl4ai.com/advanced/anti-bot-and-fallback/#__codelineno-5-9) ProxyConfig( [](https://docs.crawl4ai.com/advanced/anti-bot-and-fallback/#__codelineno-5-10) server="http://residential-proxy.example.com:9090", [](https://docs.crawl4ai.com/advanced/anti-bot-and-fallback/#__codelineno-5-11) username="user", [](https://docs.crawl4ai.com/advanced/anti-bot-and-fallback/#__codelineno-5-12) password="pass", [](https://docs.crawl4ai.com/advanced/anti-bot-and-fallback/#__codelineno-5-13) ), [](https://docs.crawl4ai.com/advanced/anti-bot-and-fallback/#__codelineno-5-14) ], [](https://docs.crawl4ai.com/advanced/anti-bot-and-fallback/#__codelineno-5-15) )` With this setup, each round tries the datacenter proxy first, then the residential proxy. With `max_retries=1`, worst case is 2 rounds x 2 proxies = 4 attempts. ### Fallback Fetch Function When all browser-based attempts fail, call a custom async function as a last resort. This function receives the URL and must return raw HTML as a string. The returned HTML is processed through the normal pipeline (markdown generation, extraction, etc.). This is useful when you have access to a scraping API, a pre-fetched cache, or any other source of HTML. `[](https://docs.crawl4ai.com/advanced/anti-bot-and-fallback/#__codelineno-6-1) import aiohttp [](https://docs.crawl4ai.com/advanced/anti-bot-and-fallback/#__codelineno-6-2) [](https://docs.crawl4ai.com/advanced/anti-bot-and-fallback/#__codelineno-6-3) async def my_scraping_api(url: str) -> str: [](https://docs.crawl4ai.com/advanced/anti-bot-and-fallback/#__codelineno-6-4) """Fetch HTML via an external scraping API.""" [](https://docs.crawl4ai.com/advanced/anti-bot-and-fallback/#__codelineno-6-5) async with aiohttp.ClientSession() as session: [](https://docs.crawl4ai.com/advanced/anti-bot-and-fallback/#__codelineno-6-6) async with session.get( [](https://docs.crawl4ai.com/advanced/anti-bot-and-fallback/#__codelineno-6-7) "https://api.my-scraping-service.com/fetch", [](https://docs.crawl4ai.com/advanced/anti-bot-and-fallback/#__codelineno-6-8) params={"url": url, "format": "html"}, [](https://docs.crawl4ai.com/advanced/anti-bot-and-fallback/#__codelineno-6-9) headers={"Authorization": "Bearer MY_TOKEN"}, [](https://docs.crawl4ai.com/advanced/anti-bot-and-fallback/#__codelineno-6-10) ) as resp: [](https://docs.crawl4ai.com/advanced/anti-bot-and-fallback/#__codelineno-6-11) if resp.status == 200: [](https://docs.crawl4ai.com/advanced/anti-bot-and-fallback/#__codelineno-6-12) return await resp.text() [](https://docs.crawl4ai.com/advanced/anti-bot-and-fallback/#__codelineno-6-13) raise RuntimeError(f"API error: {resp.status}") [](https://docs.crawl4ai.com/advanced/anti-bot-and-fallback/#__codelineno-6-14) [](https://docs.crawl4ai.com/advanced/anti-bot-and-fallback/#__codelineno-6-15) config = CrawlerRunConfig( [](https://docs.crawl4ai.com/advanced/anti-bot-and-fallback/#__codelineno-6-16) max_retries=1, [](https://docs.crawl4ai.com/advanced/anti-bot-and-fallback/#__codelineno-6-17) fallback_fetch_function=my_scraping_api, [](https://docs.crawl4ai.com/advanced/anti-bot-and-fallback/#__codelineno-6-18) )` The function can do anything — call an API, read from a database, return cached HTML, or make a simple HTTP request with a different library. Crawl4AI does not care how the HTML is obtained. ### Full Escalation (All Features Combined) This example combines every layer: stealth mode, a list of proxies tried in order, retries, and a final fetch function. `[](https://docs.crawl4ai.com/advanced/anti-bot-and-fallback/#__codelineno-7-1) import aiohttp [](https://docs.crawl4ai.com/advanced/anti-bot-and-fallback/#__codelineno-7-2) from crawl4ai import AsyncWebCrawler [](https://docs.crawl4ai.com/advanced/anti-bot-and-fallback/#__codelineno-7-3) from crawl4ai.async_configs import BrowserConfig, CrawlerRunConfig, ProxyConfig [](https://docs.crawl4ai.com/advanced/anti-bot-and-fallback/#__codelineno-7-4) [](https://docs.crawl4ai.com/advanced/anti-bot-and-fallback/#__codelineno-7-5) # Last-resort: fetch HTML via an external service [](https://docs.crawl4ai.com/advanced/anti-bot-and-fallback/#__codelineno-7-6) async def external_fetch(url: str) -> str: [](https://docs.crawl4ai.com/advanced/anti-bot-and-fallback/#__codelineno-7-7) async with aiohttp.ClientSession() as session: [](https://docs.crawl4ai.com/advanced/anti-bot-and-fallback/#__codelineno-7-8) async with session.post( [](https://docs.crawl4ai.com/advanced/anti-bot-and-fallback/#__codelineno-7-9) "https://api.my-service.com/scrape", [](https://docs.crawl4ai.com/advanced/anti-bot-and-fallback/#__codelineno-7-10) json={"url": url, "render_js": True}, [](https://docs.crawl4ai.com/advanced/anti-bot-and-fallback/#__codelineno-7-11) headers={"Authorization": "Bearer MY_TOKEN"}, [](https://docs.crawl4ai.com/advanced/anti-bot-and-fallback/#__codelineno-7-12) ) as resp: [](https://docs.crawl4ai.com/advanced/anti-bot-and-fallback/#__codelineno-7-13) return await resp.text() [](https://docs.crawl4ai.com/advanced/anti-bot-and-fallback/#__codelineno-7-14) [](https://docs.crawl4ai.com/advanced/anti-bot-and-fallback/#__codelineno-7-15) browser_config = BrowserConfig( [](https://docs.crawl4ai.com/advanced/anti-bot-and-fallback/#__codelineno-7-16) headless=True, [](https://docs.crawl4ai.com/advanced/anti-bot-and-fallback/#__codelineno-7-17) enable_stealth=True, [](https://docs.crawl4ai.com/advanced/anti-bot-and-fallback/#__codelineno-7-18) ) [](https://docs.crawl4ai.com/advanced/anti-bot-and-fallback/#__codelineno-7-19) [](https://docs.crawl4ai.com/advanced/anti-bot-and-fallback/#__codelineno-7-20) crawl_config = CrawlerRunConfig( [](https://docs.crawl4ai.com/advanced/anti-bot-and-fallback/#__codelineno-7-21) magic=True, [](https://docs.crawl4ai.com/advanced/anti-bot-and-fallback/#__codelineno-7-22) wait_until="load", [](https://docs.crawl4ai.com/advanced/anti-bot-and-fallback/#__codelineno-7-23) max_retries=2, [](https://docs.crawl4ai.com/advanced/anti-bot-and-fallback/#__codelineno-7-24) [](https://docs.crawl4ai.com/advanced/anti-bot-and-fallback/#__codelineno-7-25) # Proxies tried in order — cheapest first [](https://docs.crawl4ai.com/advanced/anti-bot-and-fallback/#__codelineno-7-26) proxy_config=[ [](https://docs.crawl4ai.com/advanced/anti-bot-and-fallback/#__codelineno-7-27) ProxyConfig( [](https://docs.crawl4ai.com/advanced/anti-bot-and-fallback/#__codelineno-7-28) server="http://datacenter-proxy.example.com:8080", [](https://docs.crawl4ai.com/advanced/anti-bot-and-fallback/#__codelineno-7-29) username="user", [](https://docs.crawl4ai.com/advanced/anti-bot-and-fallback/#__codelineno-7-30) password="pass", [](https://docs.crawl4ai.com/advanced/anti-bot-and-fallback/#__codelineno-7-31) ), [](https://docs.crawl4ai.com/advanced/anti-bot-and-fallback/#__codelineno-7-32) ProxyConfig( [](https://docs.crawl4ai.com/advanced/anti-bot-and-fallback/#__codelineno-7-33) server="http://residential-proxy.example.com:9090", [](https://docs.crawl4ai.com/advanced/anti-bot-and-fallback/#__codelineno-7-34) username="user", [](https://docs.crawl4ai.com/advanced/anti-bot-and-fallback/#__codelineno-7-35) password="pass", [](https://docs.crawl4ai.com/advanced/anti-bot-and-fallback/#__codelineno-7-36) ), [](https://docs.crawl4ai.com/advanced/anti-bot-and-fallback/#__codelineno-7-37) ], [](https://docs.crawl4ai.com/advanced/anti-bot-and-fallback/#__codelineno-7-38) [](https://docs.crawl4ai.com/advanced/anti-bot-and-fallback/#__codelineno-7-39) # Last resort — called after all retries and proxies are exhausted [](https://docs.crawl4ai.com/advanced/anti-bot-and-fallback/#__codelineno-7-40) fallback_fetch_function=external_fetch, [](https://docs.crawl4ai.com/advanced/anti-bot-and-fallback/#__codelineno-7-41) ) [](https://docs.crawl4ai.com/advanced/anti-bot-and-fallback/#__codelineno-7-42) [](https://docs.crawl4ai.com/advanced/anti-bot-and-fallback/#__codelineno-7-43) async with AsyncWebCrawler(config=browser_config) as crawler: [](https://docs.crawl4ai.com/advanced/anti-bot-and-fallback/#__codelineno-7-44) result = await crawler.arun( [](https://docs.crawl4ai.com/advanced/anti-bot-and-fallback/#__codelineno-7-45) url="https://protected-site.com/products", [](https://docs.crawl4ai.com/advanced/anti-bot-and-fallback/#__codelineno-7-46) config=crawl_config, [](https://docs.crawl4ai.com/advanced/anti-bot-and-fallback/#__codelineno-7-47) ) [](https://docs.crawl4ai.com/advanced/anti-bot-and-fallback/#__codelineno-7-48) [](https://docs.crawl4ai.com/advanced/anti-bot-and-fallback/#__codelineno-7-49) if result.success: [](https://docs.crawl4ai.com/advanced/anti-bot-and-fallback/#__codelineno-7-50) print(f"Got {len(result.markdown.raw_markdown)} chars of markdown") [](https://docs.crawl4ai.com/advanced/anti-bot-and-fallback/#__codelineno-7-51) print(f"Resolved by: {result.crawl_stats['resolved_by']}") [](https://docs.crawl4ai.com/advanced/anti-bot-and-fallback/#__codelineno-7-52) print(f"Attempts: {result.crawl_stats['attempts']}") [](https://docs.crawl4ai.com/advanced/anti-bot-and-fallback/#__codelineno-7-53) else: [](https://docs.crawl4ai.com/advanced/anti-bot-and-fallback/#__codelineno-7-54) print(f"All attempts failed: {result.error_message}")` **What happens step by step:** | Round | Attempt | What runs | | --- | --- | --- | | 1 | 1 | Datacenter proxy — blocked | | 1 | 2 | Residential proxy — blocked | | 2 | 1 | Datacenter proxy — blocked | | 2 | 2 | Residential proxy — blocked | | 3 | 1 | Datacenter proxy — blocked | | 3 | 2 | Residential proxy — blocked | | \- | \- | `external_fetch(url)` called — returns HTML | That's up to 6 browser attempts + 1 function call before giving up. Tips ---- * **Start with `max_retries=0`** and a `fallback_fetch_function` if you just want a safety net without burning time on retries. * **Order proxies cheapest-first** — datacenter proxies before residential, residential before premium. * **Combine with stealth mode** — `BrowserConfig(enable_stealth=True)` and `CrawlerRunConfig(magic=True)` reduce the chance of being blocked in the first place. * **`wait_until="load"`** is important for anti-bot sites — the default `domcontentloaded` can return before the anti-bot sensor finishes. * **Check `crawl_stats`** to understand what happened — how many attempts, which proxy worked, whether the fallback function was needed. See Also -------- * [Proxy & Security](https://docs.crawl4ai.com/advanced/proxy-security/) — Proxy setup, authentication, and rotation * [Undetected Browser](https://docs.crawl4ai.com/advanced/undetected-browser/) — Stealth mode and browser fingerprint evasion * [Session Management](https://docs.crawl4ai.com/advanced/session-management/) — Maintaining sessions across requests * * * \> Feedback --- # Overview - Crawl4AI Documentation (v0.8.x) Overview of Some Important Advanced Features ============================================ (Proxy, PDF, Screenshot, SSL, Headers, & Storage State) Crawl4AI offers multiple power-user features that go beyond simple crawling. This tutorial covers: 1. **Proxy Usage** 2. **Capturing PDFs & Screenshots** 3. **Handling SSL Certificates** 4. **Custom Headers** 5. **Session Persistence & Local Storage** 6. **Robots.txt Compliance** > **Prerequisites** > \- You have a basic grasp of [AsyncWebCrawler Basics](https://docs.crawl4ai.com/core/simple-crawling/) > > \- You know how to run or configure your Python environment with Playwright installed * * * 1\. Proxy Usage --------------- If you need to route your crawl traffic through a proxy—whether for IP rotation, geo-testing, or privacy—Crawl4AI supports it via `BrowserConfig.proxy_config`. `[](https://docs.crawl4ai.com/advanced/advanced-features/#__codelineno-0-1) import asyncio [](https://docs.crawl4ai.com/advanced/advanced-features/#__codelineno-0-2) from crawl4ai import AsyncWebCrawler, BrowserConfig, CrawlerRunConfig [](https://docs.crawl4ai.com/advanced/advanced-features/#__codelineno-0-3) [](https://docs.crawl4ai.com/advanced/advanced-features/#__codelineno-0-4) async def main(): [](https://docs.crawl4ai.com/advanced/advanced-features/#__codelineno-0-5) browser_cfg = BrowserConfig( [](https://docs.crawl4ai.com/advanced/advanced-features/#__codelineno-0-6) proxy_config={ [](https://docs.crawl4ai.com/advanced/advanced-features/#__codelineno-0-7) "server": "http://proxy.example.com:8080", [](https://docs.crawl4ai.com/advanced/advanced-features/#__codelineno-0-8) "username": "myuser", [](https://docs.crawl4ai.com/advanced/advanced-features/#__codelineno-0-9) "password": "mypass", [](https://docs.crawl4ai.com/advanced/advanced-features/#__codelineno-0-10) }, [](https://docs.crawl4ai.com/advanced/advanced-features/#__codelineno-0-11) headless=True [](https://docs.crawl4ai.com/advanced/advanced-features/#__codelineno-0-12) ) [](https://docs.crawl4ai.com/advanced/advanced-features/#__codelineno-0-13) crawler_cfg = CrawlerRunConfig( [](https://docs.crawl4ai.com/advanced/advanced-features/#__codelineno-0-14) verbose=True [](https://docs.crawl4ai.com/advanced/advanced-features/#__codelineno-0-15) ) [](https://docs.crawl4ai.com/advanced/advanced-features/#__codelineno-0-16) [](https://docs.crawl4ai.com/advanced/advanced-features/#__codelineno-0-17) async with AsyncWebCrawler(config=browser_cfg) as crawler: [](https://docs.crawl4ai.com/advanced/advanced-features/#__codelineno-0-18) result = await crawler.arun( [](https://docs.crawl4ai.com/advanced/advanced-features/#__codelineno-0-19) url="https://www.whatismyip.com/", [](https://docs.crawl4ai.com/advanced/advanced-features/#__codelineno-0-20) config=crawler_cfg [](https://docs.crawl4ai.com/advanced/advanced-features/#__codelineno-0-21) ) [](https://docs.crawl4ai.com/advanced/advanced-features/#__codelineno-0-22) if result.success: [](https://docs.crawl4ai.com/advanced/advanced-features/#__codelineno-0-23) print("[OK] Page fetched via proxy.") [](https://docs.crawl4ai.com/advanced/advanced-features/#__codelineno-0-24) print("Page HTML snippet:", result.html[:200]) [](https://docs.crawl4ai.com/advanced/advanced-features/#__codelineno-0-25) else: [](https://docs.crawl4ai.com/advanced/advanced-features/#__codelineno-0-26) print("[ERROR]", result.error_message) [](https://docs.crawl4ai.com/advanced/advanced-features/#__codelineno-0-27) [](https://docs.crawl4ai.com/advanced/advanced-features/#__codelineno-0-28) if __name__ == "__main__": [](https://docs.crawl4ai.com/advanced/advanced-features/#__codelineno-0-29) asyncio.run(main())` **Key Points** \- **`proxy_config`** expects a dict with `server` and optional auth credentials. \- Many commercial proxies provide an HTTP/HTTPS “gateway” server that you specify in `server`. \- If your proxy doesn’t need auth, omit `username`/`password`. * * * 2\. Capturing PDFs & Screenshots -------------------------------- Sometimes you need a visual record of a page or a PDF “printout.” Crawl4AI can do both in one pass: `[](https://docs.crawl4ai.com/advanced/advanced-features/#__codelineno-1-1) import os, asyncio [](https://docs.crawl4ai.com/advanced/advanced-features/#__codelineno-1-2) from base64 import b64decode [](https://docs.crawl4ai.com/advanced/advanced-features/#__codelineno-1-3) from crawl4ai import AsyncWebCrawler, CacheMode, CrawlerRunConfig [](https://docs.crawl4ai.com/advanced/advanced-features/#__codelineno-1-4) [](https://docs.crawl4ai.com/advanced/advanced-features/#__codelineno-1-5) async def main(): [](https://docs.crawl4ai.com/advanced/advanced-features/#__codelineno-1-6) run_config = CrawlerRunConfig( [](https://docs.crawl4ai.com/advanced/advanced-features/#__codelineno-1-7) cache_mode=CacheMode.BYPASS, [](https://docs.crawl4ai.com/advanced/advanced-features/#__codelineno-1-8) screenshot=True, [](https://docs.crawl4ai.com/advanced/advanced-features/#__codelineno-1-9) pdf=True [](https://docs.crawl4ai.com/advanced/advanced-features/#__codelineno-1-10) ) [](https://docs.crawl4ai.com/advanced/advanced-features/#__codelineno-1-11) [](https://docs.crawl4ai.com/advanced/advanced-features/#__codelineno-1-12) async with AsyncWebCrawler() as crawler: [](https://docs.crawl4ai.com/advanced/advanced-features/#__codelineno-1-13) result = await crawler.arun( [](https://docs.crawl4ai.com/advanced/advanced-features/#__codelineno-1-14) url="https://en.wikipedia.org/wiki/List_of_common_misconceptions", [](https://docs.crawl4ai.com/advanced/advanced-features/#__codelineno-1-15) config=run_config [](https://docs.crawl4ai.com/advanced/advanced-features/#__codelineno-1-16) ) [](https://docs.crawl4ai.com/advanced/advanced-features/#__codelineno-1-17) if result.success: [](https://docs.crawl4ai.com/advanced/advanced-features/#__codelineno-1-18) print(f"Screenshot data present: {result.screenshot is not None}") [](https://docs.crawl4ai.com/advanced/advanced-features/#__codelineno-1-19) print(f"PDF data present: {result.pdf is not None}") [](https://docs.crawl4ai.com/advanced/advanced-features/#__codelineno-1-20) [](https://docs.crawl4ai.com/advanced/advanced-features/#__codelineno-1-21) if result.screenshot: [](https://docs.crawl4ai.com/advanced/advanced-features/#__codelineno-1-22) print(f"[OK] Screenshot captured, size: {len(result.screenshot)} bytes") [](https://docs.crawl4ai.com/advanced/advanced-features/#__codelineno-1-23) with open("wikipedia_screenshot.png", "wb") as f: [](https://docs.crawl4ai.com/advanced/advanced-features/#__codelineno-1-24) f.write(b64decode(result.screenshot)) [](https://docs.crawl4ai.com/advanced/advanced-features/#__codelineno-1-25) else: [](https://docs.crawl4ai.com/advanced/advanced-features/#__codelineno-1-26) print("[WARN] Screenshot data is None.") [](https://docs.crawl4ai.com/advanced/advanced-features/#__codelineno-1-27) [](https://docs.crawl4ai.com/advanced/advanced-features/#__codelineno-1-28) if result.pdf: [](https://docs.crawl4ai.com/advanced/advanced-features/#__codelineno-1-29) print(f"[OK] PDF captured, size: {len(result.pdf)} bytes") [](https://docs.crawl4ai.com/advanced/advanced-features/#__codelineno-1-30) with open("wikipedia_page.pdf", "wb") as f: [](https://docs.crawl4ai.com/advanced/advanced-features/#__codelineno-1-31) f.write(result.pdf) [](https://docs.crawl4ai.com/advanced/advanced-features/#__codelineno-1-32) else: [](https://docs.crawl4ai.com/advanced/advanced-features/#__codelineno-1-33) print("[WARN] PDF data is None.") [](https://docs.crawl4ai.com/advanced/advanced-features/#__codelineno-1-34) [](https://docs.crawl4ai.com/advanced/advanced-features/#__codelineno-1-35) else: [](https://docs.crawl4ai.com/advanced/advanced-features/#__codelineno-1-36) print("[ERROR]", result.error_message) [](https://docs.crawl4ai.com/advanced/advanced-features/#__codelineno-1-37) [](https://docs.crawl4ai.com/advanced/advanced-features/#__codelineno-1-38) if __name__ == "__main__": [](https://docs.crawl4ai.com/advanced/advanced-features/#__codelineno-1-39) asyncio.run(main())` **Why PDF + Screenshot?** \- Large or complex pages can be slow or error-prone with “traditional” full-page screenshots. \- Exporting a PDF is more reliable for very long pages. Crawl4AI automatically converts the first PDF page into an image if you request both. **Relevant Parameters** \- **`pdf=True`**: Exports the current page as a PDF (base64-encoded in `result.pdf`). \- **`screenshot=True`**: Creates a screenshot (base64-encoded in `result.screenshot`). \- **`scan_full_page`** or advanced hooking can further refine how the crawler captures content. * * * 3\. Handling SSL Certificates ----------------------------- If you need to verify or export a site’s SSL certificate—for compliance, debugging, or data analysis—Crawl4AI can fetch it during the crawl: `[](https://docs.crawl4ai.com/advanced/advanced-features/#__codelineno-2-1) import asyncio, os [](https://docs.crawl4ai.com/advanced/advanced-features/#__codelineno-2-2) from crawl4ai import AsyncWebCrawler, CrawlerRunConfig, CacheMode [](https://docs.crawl4ai.com/advanced/advanced-features/#__codelineno-2-3) [](https://docs.crawl4ai.com/advanced/advanced-features/#__codelineno-2-4) async def main(): [](https://docs.crawl4ai.com/advanced/advanced-features/#__codelineno-2-5) tmp_dir = os.path.join(os.getcwd(), "tmp") [](https://docs.crawl4ai.com/advanced/advanced-features/#__codelineno-2-6) os.makedirs(tmp_dir, exist_ok=True) [](https://docs.crawl4ai.com/advanced/advanced-features/#__codelineno-2-7) [](https://docs.crawl4ai.com/advanced/advanced-features/#__codelineno-2-8) config = CrawlerRunConfig( [](https://docs.crawl4ai.com/advanced/advanced-features/#__codelineno-2-9) fetch_ssl_certificate=True, [](https://docs.crawl4ai.com/advanced/advanced-features/#__codelineno-2-10) cache_mode=CacheMode.BYPASS [](https://docs.crawl4ai.com/advanced/advanced-features/#__codelineno-2-11) ) [](https://docs.crawl4ai.com/advanced/advanced-features/#__codelineno-2-12) [](https://docs.crawl4ai.com/advanced/advanced-features/#__codelineno-2-13) async with AsyncWebCrawler() as crawler: [](https://docs.crawl4ai.com/advanced/advanced-features/#__codelineno-2-14) result = await crawler.arun(url="https://example.com", config=config) [](https://docs.crawl4ai.com/advanced/advanced-features/#__codelineno-2-15) [](https://docs.crawl4ai.com/advanced/advanced-features/#__codelineno-2-16) if result.success and result.ssl_certificate: [](https://docs.crawl4ai.com/advanced/advanced-features/#__codelineno-2-17) cert = result.ssl_certificate [](https://docs.crawl4ai.com/advanced/advanced-features/#__codelineno-2-18) print("\nCertificate Information:") [](https://docs.crawl4ai.com/advanced/advanced-features/#__codelineno-2-19) print(f"Issuer (CN): {cert.issuer.get('CN', '')}") [](https://docs.crawl4ai.com/advanced/advanced-features/#__codelineno-2-20) print(f"Valid until: {cert.valid_until}") [](https://docs.crawl4ai.com/advanced/advanced-features/#__codelineno-2-21) print(f"Fingerprint: {cert.fingerprint}") [](https://docs.crawl4ai.com/advanced/advanced-features/#__codelineno-2-22) [](https://docs.crawl4ai.com/advanced/advanced-features/#__codelineno-2-23) # Export in multiple formats: [](https://docs.crawl4ai.com/advanced/advanced-features/#__codelineno-2-24) cert.to_json(os.path.join(tmp_dir, "certificate.json")) [](https://docs.crawl4ai.com/advanced/advanced-features/#__codelineno-2-25) cert.to_pem(os.path.join(tmp_dir, "certificate.pem")) [](https://docs.crawl4ai.com/advanced/advanced-features/#__codelineno-2-26) cert.to_der(os.path.join(tmp_dir, "certificate.der")) [](https://docs.crawl4ai.com/advanced/advanced-features/#__codelineno-2-27) [](https://docs.crawl4ai.com/advanced/advanced-features/#__codelineno-2-28) print("\nCertificate exported to JSON/PEM/DER in 'tmp' folder.") [](https://docs.crawl4ai.com/advanced/advanced-features/#__codelineno-2-29) else: [](https://docs.crawl4ai.com/advanced/advanced-features/#__codelineno-2-30) print("[ERROR] No certificate or crawl failed.") [](https://docs.crawl4ai.com/advanced/advanced-features/#__codelineno-2-31) [](https://docs.crawl4ai.com/advanced/advanced-features/#__codelineno-2-32) if __name__ == "__main__": [](https://docs.crawl4ai.com/advanced/advanced-features/#__codelineno-2-33) asyncio.run(main())` **Key Points** \- **`fetch_ssl_certificate=True`** triggers certificate retrieval. \- `result.ssl_certificate` includes methods (`to_json`, `to_pem`, `to_der`) for saving in various formats (handy for server config, Java keystores, etc.). * * * 4\. Custom Headers ------------------ Sometimes you need to set custom headers (e.g., language preferences, authentication tokens, or specialized user-agent strings). You can do this in multiple ways: ``[](https://docs.crawl4ai.com/advanced/advanced-features/#__codelineno-3-1) import asyncio [](https://docs.crawl4ai.com/advanced/advanced-features/#__codelineno-3-2) from crawl4ai import AsyncWebCrawler [](https://docs.crawl4ai.com/advanced/advanced-features/#__codelineno-3-3) [](https://docs.crawl4ai.com/advanced/advanced-features/#__codelineno-3-4) async def main(): [](https://docs.crawl4ai.com/advanced/advanced-features/#__codelineno-3-5) # Option 1: Set headers at the crawler strategy level [](https://docs.crawl4ai.com/advanced/advanced-features/#__codelineno-3-6) crawler1 = AsyncWebCrawler( [](https://docs.crawl4ai.com/advanced/advanced-features/#__codelineno-3-7) # The underlying strategy can accept headers in its constructor [](https://docs.crawl4ai.com/advanced/advanced-features/#__codelineno-3-8) crawler_strategy=None # We'll override below for clarity [](https://docs.crawl4ai.com/advanced/advanced-features/#__codelineno-3-9) ) [](https://docs.crawl4ai.com/advanced/advanced-features/#__codelineno-3-10) crawler1.crawler_strategy.update_user_agent("MyCustomUA/1.0") [](https://docs.crawl4ai.com/advanced/advanced-features/#__codelineno-3-11) crawler1.crawler_strategy.set_custom_headers({ [](https://docs.crawl4ai.com/advanced/advanced-features/#__codelineno-3-12) "Accept-Language": "fr-FR,fr;q=0.9" [](https://docs.crawl4ai.com/advanced/advanced-features/#__codelineno-3-13) }) [](https://docs.crawl4ai.com/advanced/advanced-features/#__codelineno-3-14) result1 = await crawler1.arun("https://www.example.com") [](https://docs.crawl4ai.com/advanced/advanced-features/#__codelineno-3-15) print("Example 1 result success:", result1.success) [](https://docs.crawl4ai.com/advanced/advanced-features/#__codelineno-3-16) [](https://docs.crawl4ai.com/advanced/advanced-features/#__codelineno-3-17) # Option 2: Pass headers directly to `arun()` [](https://docs.crawl4ai.com/advanced/advanced-features/#__codelineno-3-18) crawler2 = AsyncWebCrawler() [](https://docs.crawl4ai.com/advanced/advanced-features/#__codelineno-3-19) result2 = await crawler2.arun( [](https://docs.crawl4ai.com/advanced/advanced-features/#__codelineno-3-20) url="https://www.example.com", [](https://docs.crawl4ai.com/advanced/advanced-features/#__codelineno-3-21) headers={"Accept-Language": "es-ES,es;q=0.9"} [](https://docs.crawl4ai.com/advanced/advanced-features/#__codelineno-3-22) ) [](https://docs.crawl4ai.com/advanced/advanced-features/#__codelineno-3-23) print("Example 2 result success:", result2.success) [](https://docs.crawl4ai.com/advanced/advanced-features/#__codelineno-3-24) [](https://docs.crawl4ai.com/advanced/advanced-features/#__codelineno-3-25) if __name__ == "__main__": [](https://docs.crawl4ai.com/advanced/advanced-features/#__codelineno-3-26) asyncio.run(main())`` **Notes** \- Some sites may react differently to certain headers (e.g., `Accept-Language`). \- If you need advanced user-agent randomization or client hints, see [Identity-Based Crawling (Anti-Bot)](https://docs.crawl4ai.com/advanced/identity-based-crawling/) or use `UserAgentGenerator`. * * * 5\. Session Persistence & Local Storage --------------------------------------- Crawl4AI can preserve cookies and localStorage so you can continue where you left off—ideal for logging into sites or skipping repeated auth flows. ### 5.1 `storage_state` `[](https://docs.crawl4ai.com/advanced/advanced-features/#__codelineno-4-1) import asyncio [](https://docs.crawl4ai.com/advanced/advanced-features/#__codelineno-4-2) from crawl4ai import AsyncWebCrawler [](https://docs.crawl4ai.com/advanced/advanced-features/#__codelineno-4-3) [](https://docs.crawl4ai.com/advanced/advanced-features/#__codelineno-4-4) async def main(): [](https://docs.crawl4ai.com/advanced/advanced-features/#__codelineno-4-5) storage_dict = { [](https://docs.crawl4ai.com/advanced/advanced-features/#__codelineno-4-6) "cookies": [ [](https://docs.crawl4ai.com/advanced/advanced-features/#__codelineno-4-7) { [](https://docs.crawl4ai.com/advanced/advanced-features/#__codelineno-4-8) "name": "session", [](https://docs.crawl4ai.com/advanced/advanced-features/#__codelineno-4-9) "value": "abcd1234", [](https://docs.crawl4ai.com/advanced/advanced-features/#__codelineno-4-10) "domain": "example.com", [](https://docs.crawl4ai.com/advanced/advanced-features/#__codelineno-4-11) "path": "/", [](https://docs.crawl4ai.com/advanced/advanced-features/#__codelineno-4-12) "expires": 1699999999.0, [](https://docs.crawl4ai.com/advanced/advanced-features/#__codelineno-4-13) "httpOnly": False, [](https://docs.crawl4ai.com/advanced/advanced-features/#__codelineno-4-14) "secure": False, [](https://docs.crawl4ai.com/advanced/advanced-features/#__codelineno-4-15) "sameSite": "None" [](https://docs.crawl4ai.com/advanced/advanced-features/#__codelineno-4-16) } [](https://docs.crawl4ai.com/advanced/advanced-features/#__codelineno-4-17) ], [](https://docs.crawl4ai.com/advanced/advanced-features/#__codelineno-4-18) "origins": [ [](https://docs.crawl4ai.com/advanced/advanced-features/#__codelineno-4-19) { [](https://docs.crawl4ai.com/advanced/advanced-features/#__codelineno-4-20) "origin": "https://example.com", [](https://docs.crawl4ai.com/advanced/advanced-features/#__codelineno-4-21) "localStorage": [ [](https://docs.crawl4ai.com/advanced/advanced-features/#__codelineno-4-22) {"name": "token", "value": "my_auth_token"} [](https://docs.crawl4ai.com/advanced/advanced-features/#__codelineno-4-23) ] [](https://docs.crawl4ai.com/advanced/advanced-features/#__codelineno-4-24) } [](https://docs.crawl4ai.com/advanced/advanced-features/#__codelineno-4-25) ] [](https://docs.crawl4ai.com/advanced/advanced-features/#__codelineno-4-26) } [](https://docs.crawl4ai.com/advanced/advanced-features/#__codelineno-4-27) [](https://docs.crawl4ai.com/advanced/advanced-features/#__codelineno-4-28) # Provide the storage state as a dictionary to start "already logged in" [](https://docs.crawl4ai.com/advanced/advanced-features/#__codelineno-4-29) async with AsyncWebCrawler( [](https://docs.crawl4ai.com/advanced/advanced-features/#__codelineno-4-30) headless=True, [](https://docs.crawl4ai.com/advanced/advanced-features/#__codelineno-4-31) storage_state=storage_dict [](https://docs.crawl4ai.com/advanced/advanced-features/#__codelineno-4-32) ) as crawler: [](https://docs.crawl4ai.com/advanced/advanced-features/#__codelineno-4-33) result = await crawler.arun("https://example.com/protected") [](https://docs.crawl4ai.com/advanced/advanced-features/#__codelineno-4-34) if result.success: [](https://docs.crawl4ai.com/advanced/advanced-features/#__codelineno-4-35) print("Protected page content length:", len(result.html)) [](https://docs.crawl4ai.com/advanced/advanced-features/#__codelineno-4-36) else: [](https://docs.crawl4ai.com/advanced/advanced-features/#__codelineno-4-37) print("Failed to crawl protected page") [](https://docs.crawl4ai.com/advanced/advanced-features/#__codelineno-4-38) [](https://docs.crawl4ai.com/advanced/advanced-features/#__codelineno-4-39) if __name__ == "__main__": [](https://docs.crawl4ai.com/advanced/advanced-features/#__codelineno-4-40) asyncio.run(main())` ### 5.2 Exporting & Reusing State You can sign in once, export the browser context, and reuse it later—without re-entering credentials. * **`await context.storage_state(path="my_storage.json")`**: Exports cookies, localStorage, etc. to a file. * Provide `storage_state="my_storage.json"` on subsequent runs to skip the login step. **See**: [Detailed session management tutorial](https://docs.crawl4ai.com/advanced/session-management/) or [Explanations → Browser Context & Managed Browser](https://docs.crawl4ai.com/advanced/identity-based-crawling/) for more advanced scenarios (like multi-step logins, or capturing after interactive pages). * * * 6\. Robots.txt Compliance ------------------------- Crawl4AI supports respecting robots.txt rules with efficient caching: `[](https://docs.crawl4ai.com/advanced/advanced-features/#__codelineno-5-1) import asyncio [](https://docs.crawl4ai.com/advanced/advanced-features/#__codelineno-5-2) from crawl4ai import AsyncWebCrawler, CrawlerRunConfig [](https://docs.crawl4ai.com/advanced/advanced-features/#__codelineno-5-3) [](https://docs.crawl4ai.com/advanced/advanced-features/#__codelineno-5-4) async def main(): [](https://docs.crawl4ai.com/advanced/advanced-features/#__codelineno-5-5) # Enable robots.txt checking in config [](https://docs.crawl4ai.com/advanced/advanced-features/#__codelineno-5-6) config = CrawlerRunConfig( [](https://docs.crawl4ai.com/advanced/advanced-features/#__codelineno-5-7) check_robots_txt=True # Will check and respect robots.txt rules [](https://docs.crawl4ai.com/advanced/advanced-features/#__codelineno-5-8) ) [](https://docs.crawl4ai.com/advanced/advanced-features/#__codelineno-5-9) [](https://docs.crawl4ai.com/advanced/advanced-features/#__codelineno-5-10) async with AsyncWebCrawler() as crawler: [](https://docs.crawl4ai.com/advanced/advanced-features/#__codelineno-5-11) result = await crawler.arun( [](https://docs.crawl4ai.com/advanced/advanced-features/#__codelineno-5-12) "https://example.com", [](https://docs.crawl4ai.com/advanced/advanced-features/#__codelineno-5-13) config=config [](https://docs.crawl4ai.com/advanced/advanced-features/#__codelineno-5-14) ) [](https://docs.crawl4ai.com/advanced/advanced-features/#__codelineno-5-15) [](https://docs.crawl4ai.com/advanced/advanced-features/#__codelineno-5-16) if not result.success and result.status_code == 403: [](https://docs.crawl4ai.com/advanced/advanced-features/#__codelineno-5-17) print("Access denied by robots.txt") [](https://docs.crawl4ai.com/advanced/advanced-features/#__codelineno-5-18) [](https://docs.crawl4ai.com/advanced/advanced-features/#__codelineno-5-19) if __name__ == "__main__": [](https://docs.crawl4ai.com/advanced/advanced-features/#__codelineno-5-20) asyncio.run(main())` **Key Points** - Robots.txt files are cached locally for efficiency - Cache is stored in `~/.crawl4ai/robots/robots_cache.db` - Cache has a default TTL of 7 days - If robots.txt can't be fetched, crawling is allowed - Returns 403 status code if URL is disallowed * * * Putting It All Together ----------------------- Here’s a snippet that combines multiple “advanced” features (proxy, PDF, screenshot, SSL, custom headers, and session reuse) into one run. Normally, you’d tailor each setting to your project’s needs. `[](https://docs.crawl4ai.com/advanced/advanced-features/#__codelineno-6-1) import os, asyncio [](https://docs.crawl4ai.com/advanced/advanced-features/#__codelineno-6-2) from base64 import b64decode [](https://docs.crawl4ai.com/advanced/advanced-features/#__codelineno-6-3) from crawl4ai import AsyncWebCrawler, BrowserConfig, CrawlerRunConfig, CacheMode [](https://docs.crawl4ai.com/advanced/advanced-features/#__codelineno-6-4) [](https://docs.crawl4ai.com/advanced/advanced-features/#__codelineno-6-5) async def main(): [](https://docs.crawl4ai.com/advanced/advanced-features/#__codelineno-6-6) # 1. Browser config with proxy + headless [](https://docs.crawl4ai.com/advanced/advanced-features/#__codelineno-6-7) browser_cfg = BrowserConfig( [](https://docs.crawl4ai.com/advanced/advanced-features/#__codelineno-6-8) proxy_config={ [](https://docs.crawl4ai.com/advanced/advanced-features/#__codelineno-6-9) "server": "http://proxy.example.com:8080", [](https://docs.crawl4ai.com/advanced/advanced-features/#__codelineno-6-10) "username": "myuser", [](https://docs.crawl4ai.com/advanced/advanced-features/#__codelineno-6-11) "password": "mypass", [](https://docs.crawl4ai.com/advanced/advanced-features/#__codelineno-6-12) }, [](https://docs.crawl4ai.com/advanced/advanced-features/#__codelineno-6-13) headless=True, [](https://docs.crawl4ai.com/advanced/advanced-features/#__codelineno-6-14) ) [](https://docs.crawl4ai.com/advanced/advanced-features/#__codelineno-6-15) [](https://docs.crawl4ai.com/advanced/advanced-features/#__codelineno-6-16) # 2. Crawler config with PDF, screenshot, SSL, custom headers, and ignoring caches [](https://docs.crawl4ai.com/advanced/advanced-features/#__codelineno-6-17) crawler_cfg = CrawlerRunConfig( [](https://docs.crawl4ai.com/advanced/advanced-features/#__codelineno-6-18) pdf=True, [](https://docs.crawl4ai.com/advanced/advanced-features/#__codelineno-6-19) screenshot=True, [](https://docs.crawl4ai.com/advanced/advanced-features/#__codelineno-6-20) fetch_ssl_certificate=True, [](https://docs.crawl4ai.com/advanced/advanced-features/#__codelineno-6-21) cache_mode=CacheMode.BYPASS, [](https://docs.crawl4ai.com/advanced/advanced-features/#__codelineno-6-22) headers={"Accept-Language": "en-US,en;q=0.8"}, [](https://docs.crawl4ai.com/advanced/advanced-features/#__codelineno-6-23) storage_state="my_storage.json", # Reuse session from a previous sign-in [](https://docs.crawl4ai.com/advanced/advanced-features/#__codelineno-6-24) verbose=True, [](https://docs.crawl4ai.com/advanced/advanced-features/#__codelineno-6-25) ) [](https://docs.crawl4ai.com/advanced/advanced-features/#__codelineno-6-26) [](https://docs.crawl4ai.com/advanced/advanced-features/#__codelineno-6-27) # 3. Crawl [](https://docs.crawl4ai.com/advanced/advanced-features/#__codelineno-6-28) async with AsyncWebCrawler(config=browser_cfg) as crawler: [](https://docs.crawl4ai.com/advanced/advanced-features/#__codelineno-6-29) result = await crawler.arun( [](https://docs.crawl4ai.com/advanced/advanced-features/#__codelineno-6-30) url = "https://secure.example.com/protected", [](https://docs.crawl4ai.com/advanced/advanced-features/#__codelineno-6-31) config=crawler_cfg [](https://docs.crawl4ai.com/advanced/advanced-features/#__codelineno-6-32) ) [](https://docs.crawl4ai.com/advanced/advanced-features/#__codelineno-6-33) [](https://docs.crawl4ai.com/advanced/advanced-features/#__codelineno-6-34) if result.success: [](https://docs.crawl4ai.com/advanced/advanced-features/#__codelineno-6-35) print("[OK] Crawled the secure page. Links found:", len(result.links.get("internal", []))) [](https://docs.crawl4ai.com/advanced/advanced-features/#__codelineno-6-36) [](https://docs.crawl4ai.com/advanced/advanced-features/#__codelineno-6-37) # Save PDF & screenshot [](https://docs.crawl4ai.com/advanced/advanced-features/#__codelineno-6-38) if result.pdf: [](https://docs.crawl4ai.com/advanced/advanced-features/#__codelineno-6-39) with open("result.pdf", "wb") as f: [](https://docs.crawl4ai.com/advanced/advanced-features/#__codelineno-6-40) f.write(b64decode(result.pdf)) [](https://docs.crawl4ai.com/advanced/advanced-features/#__codelineno-6-41) if result.screenshot: [](https://docs.crawl4ai.com/advanced/advanced-features/#__codelineno-6-42) with open("result.png", "wb") as f: [](https://docs.crawl4ai.com/advanced/advanced-features/#__codelineno-6-43) f.write(b64decode(result.screenshot)) [](https://docs.crawl4ai.com/advanced/advanced-features/#__codelineno-6-44) [](https://docs.crawl4ai.com/advanced/advanced-features/#__codelineno-6-45) # Check SSL cert [](https://docs.crawl4ai.com/advanced/advanced-features/#__codelineno-6-46) if result.ssl_certificate: [](https://docs.crawl4ai.com/advanced/advanced-features/#__codelineno-6-47) print("SSL Issuer CN:", result.ssl_certificate.issuer.get("CN", "")) [](https://docs.crawl4ai.com/advanced/advanced-features/#__codelineno-6-48) else: [](https://docs.crawl4ai.com/advanced/advanced-features/#__codelineno-6-49) print("[ERROR]", result.error_message) [](https://docs.crawl4ai.com/advanced/advanced-features/#__codelineno-6-50) [](https://docs.crawl4ai.com/advanced/advanced-features/#__codelineno-6-51) if __name__ == "__main__": [](https://docs.crawl4ai.com/advanced/advanced-features/#__codelineno-6-52) asyncio.run(main())` * * * * * * 7\. Anti-Bot Features (Stealth Mode & Undetected Browser) --------------------------------------------------------- Crawl4AI provides two powerful features to bypass bot detection: ### 7.1 Stealth Mode Stealth mode uses playwright-stealth to modify browser fingerprints and behaviors. Enable it with a simple flag: `[](https://docs.crawl4ai.com/advanced/advanced-features/#__codelineno-7-1) browser_config = BrowserConfig( [](https://docs.crawl4ai.com/advanced/advanced-features/#__codelineno-7-2) enable_stealth=True, # Activates stealth mode [](https://docs.crawl4ai.com/advanced/advanced-features/#__codelineno-7-3) headless=False [](https://docs.crawl4ai.com/advanced/advanced-features/#__codelineno-7-4) )` **When to use**: Sites with basic bot detection (checking navigator.webdriver, plugins, etc.) ### 7.2 Undetected Browser For advanced bot detection, use the undetected browser adapter: `[](https://docs.crawl4ai.com/advanced/advanced-features/#__codelineno-8-1) from crawl4ai import UndetectedAdapter [](https://docs.crawl4ai.com/advanced/advanced-features/#__codelineno-8-2) from crawl4ai.async_crawler_strategy import AsyncPlaywrightCrawlerStrategy [](https://docs.crawl4ai.com/advanced/advanced-features/#__codelineno-8-3) [](https://docs.crawl4ai.com/advanced/advanced-features/#__codelineno-8-4) # Create undetected adapter [](https://docs.crawl4ai.com/advanced/advanced-features/#__codelineno-8-5) adapter = UndetectedAdapter() [](https://docs.crawl4ai.com/advanced/advanced-features/#__codelineno-8-6) strategy = AsyncPlaywrightCrawlerStrategy( [](https://docs.crawl4ai.com/advanced/advanced-features/#__codelineno-8-7) browser_config=browser_config, [](https://docs.crawl4ai.com/advanced/advanced-features/#__codelineno-8-8) browser_adapter=adapter [](https://docs.crawl4ai.com/advanced/advanced-features/#__codelineno-8-9) ) [](https://docs.crawl4ai.com/advanced/advanced-features/#__codelineno-8-10) [](https://docs.crawl4ai.com/advanced/advanced-features/#__codelineno-8-11) async with AsyncWebCrawler(crawler_strategy=strategy, config=browser_config) as crawler: [](https://docs.crawl4ai.com/advanced/advanced-features/#__codelineno-8-12) # Your crawling code` **When to use**: Sites with sophisticated bot detection (Cloudflare, DataDome, etc.) ### 7.3 Combining Both For maximum evasion, combine stealth mode with undetected browser: `[](https://docs.crawl4ai.com/advanced/advanced-features/#__codelineno-9-1) browser_config = BrowserConfig( [](https://docs.crawl4ai.com/advanced/advanced-features/#__codelineno-9-2) enable_stealth=True, # Enable stealth [](https://docs.crawl4ai.com/advanced/advanced-features/#__codelineno-9-3) headless=False [](https://docs.crawl4ai.com/advanced/advanced-features/#__codelineno-9-4) ) [](https://docs.crawl4ai.com/advanced/advanced-features/#__codelineno-9-5) [](https://docs.crawl4ai.com/advanced/advanced-features/#__codelineno-9-6) adapter = UndetectedAdapter() # Use undetected browser` ### Choosing the Right Approach | Detection Level | Recommended Approach | | --- | --- | | No protection | Regular browser | | Basic checks | Regular + Stealth mode | | Advanced protection | Undetected browser | | Maximum evasion | Undetected + Stealth mode | **Best Practice**: Start with regular browser + stealth mode. Only use undetected browser if needed, as it may be slightly slower. See [Undetected Browser Mode](https://docs.crawl4ai.com/advanced/undetected-browser/) for detailed examples. * * * Conclusion & Next Steps ----------------------- You've now explored several **advanced** features: * **Proxy Usage** * **PDF & Screenshot** capturing for large or critical pages * **SSL Certificate** retrieval & exporting * **Custom Headers** for language or specialized requests * **Session Persistence** via storage state * **Robots.txt Compliance** * **Anti-Bot Features** (Stealth Mode & Undetected Browser) With these power tools, you can build robust scraping workflows that mimic real user behavior, handle secure sites, capture detailed snapshots, manage sessions across multiple runs, and bypass bot detection—streamlining your entire data collection pipeline. **Note**: In future versions, we may enable stealth mode and undetected browser by default. For now, users should explicitly enable these features when needed. **Last Updated**: 2025-01-17 * * * \> Feedback --- # Lazy Loading - Crawl4AI Documentation (v0.8.x) Handling Lazy-Loaded Images --------------------------- Many websites now load images **lazily** as you scroll. If you need to ensure they appear in your final crawl (and in `result.media`), consider: 1. **`wait_for_images=True`** – Wait for images to fully load. 2. **`scan_full_page`** – Force the crawler to scroll the entire page, triggering lazy loads. 3. **`scroll_delay`** – Add small delays between scroll steps. **Note**: If the site requires multiple “Load More” triggers or complex interactions, see the [Page Interaction docs](https://docs.crawl4ai.com/core/page-interaction/) . For sites with virtual scrolling (Twitter/Instagram style), see the [Virtual Scroll docs](https://docs.crawl4ai.com/advanced/virtual-scroll/) . ### Example: Ensuring Lazy Images Appear `[](https://docs.crawl4ai.com/advanced/lazy-loading/#__codelineno-0-1) import asyncio [](https://docs.crawl4ai.com/advanced/lazy-loading/#__codelineno-0-2) from crawl4ai import AsyncWebCrawler, CrawlerRunConfig, BrowserConfig [](https://docs.crawl4ai.com/advanced/lazy-loading/#__codelineno-0-3) from crawl4ai.async_configs import CacheMode [](https://docs.crawl4ai.com/advanced/lazy-loading/#__codelineno-0-4) [](https://docs.crawl4ai.com/advanced/lazy-loading/#__codelineno-0-5) async def main(): [](https://docs.crawl4ai.com/advanced/lazy-loading/#__codelineno-0-6) config = CrawlerRunConfig( [](https://docs.crawl4ai.com/advanced/lazy-loading/#__codelineno-0-7) # Force the crawler to wait until images are fully loaded [](https://docs.crawl4ai.com/advanced/lazy-loading/#__codelineno-0-8) wait_for_images=True, [](https://docs.crawl4ai.com/advanced/lazy-loading/#__codelineno-0-9) [](https://docs.crawl4ai.com/advanced/lazy-loading/#__codelineno-0-10) # Option 1: If you want to automatically scroll the page to load images [](https://docs.crawl4ai.com/advanced/lazy-loading/#__codelineno-0-11) scan_full_page=True, # Tells the crawler to try scrolling the entire page [](https://docs.crawl4ai.com/advanced/lazy-loading/#__codelineno-0-12) scroll_delay=0.5, # Delay (seconds) between scroll steps [](https://docs.crawl4ai.com/advanced/lazy-loading/#__codelineno-0-13) [](https://docs.crawl4ai.com/advanced/lazy-loading/#__codelineno-0-14) # Option 2: If the site uses a 'Load More' or JS triggers for images, [](https://docs.crawl4ai.com/advanced/lazy-loading/#__codelineno-0-15) # you can also specify js_code or wait_for logic here. [](https://docs.crawl4ai.com/advanced/lazy-loading/#__codelineno-0-16) [](https://docs.crawl4ai.com/advanced/lazy-loading/#__codelineno-0-17) cache_mode=CacheMode.BYPASS, [](https://docs.crawl4ai.com/advanced/lazy-loading/#__codelineno-0-18) verbose=True [](https://docs.crawl4ai.com/advanced/lazy-loading/#__codelineno-0-19) ) [](https://docs.crawl4ai.com/advanced/lazy-loading/#__codelineno-0-20) [](https://docs.crawl4ai.com/advanced/lazy-loading/#__codelineno-0-21) async with AsyncWebCrawler(config=BrowserConfig(headless=True)) as crawler: [](https://docs.crawl4ai.com/advanced/lazy-loading/#__codelineno-0-22) result = await crawler.arun("https://www.example.com/gallery", config=config) [](https://docs.crawl4ai.com/advanced/lazy-loading/#__codelineno-0-23) [](https://docs.crawl4ai.com/advanced/lazy-loading/#__codelineno-0-24) if result.success: [](https://docs.crawl4ai.com/advanced/lazy-loading/#__codelineno-0-25) images = result.media.get("images", []) [](https://docs.crawl4ai.com/advanced/lazy-loading/#__codelineno-0-26) print("Images found:", len(images)) [](https://docs.crawl4ai.com/advanced/lazy-loading/#__codelineno-0-27) for i, img in enumerate(images[:5]): [](https://docs.crawl4ai.com/advanced/lazy-loading/#__codelineno-0-28) print(f"[Image {i}] URL: {img['src']}, Score: {img.get('score','N/A')}") [](https://docs.crawl4ai.com/advanced/lazy-loading/#__codelineno-0-29) else: [](https://docs.crawl4ai.com/advanced/lazy-loading/#__codelineno-0-30) print("Error:", result.error_message) [](https://docs.crawl4ai.com/advanced/lazy-loading/#__codelineno-0-31) [](https://docs.crawl4ai.com/advanced/lazy-loading/#__codelineno-0-32) if __name__ == "__main__": [](https://docs.crawl4ai.com/advanced/lazy-loading/#__codelineno-0-33) asyncio.run(main())` **Explanation**: * **`wait_for_images=True`** The crawler tries to ensure images have finished loading before finalizing the HTML. * **`scan_full_page=True`** Tells the crawler to attempt scrolling from top to bottom. Each scroll step helps trigger lazy loading. * **`scroll_delay=0.5`** Pause half a second between each scroll step. Helps the site load images before continuing. **When to Use**: * **Lazy-Loading**: If images appear only when the user scrolls into view, `scan_full_page` + `scroll_delay` helps the crawler see them. * **Heavier Pages**: If a page is extremely long, be mindful that scanning the entire page can be slow. Adjust `scroll_delay` or the max scroll steps as needed. * * * Combining with Other Link & Media Filters ----------------------------------------- You can still combine **lazy-load** logic with the usual **exclude\_external\_images**, **exclude\_domains**, or link filtration: `[](https://docs.crawl4ai.com/advanced/lazy-loading/#__codelineno-1-1) config = CrawlerRunConfig( [](https://docs.crawl4ai.com/advanced/lazy-loading/#__codelineno-1-2) wait_for_images=True, [](https://docs.crawl4ai.com/advanced/lazy-loading/#__codelineno-1-3) scan_full_page=True, [](https://docs.crawl4ai.com/advanced/lazy-loading/#__codelineno-1-4) scroll_delay=0.5, [](https://docs.crawl4ai.com/advanced/lazy-loading/#__codelineno-1-5) [](https://docs.crawl4ai.com/advanced/lazy-loading/#__codelineno-1-6) # Filter out external images if you only want local ones [](https://docs.crawl4ai.com/advanced/lazy-loading/#__codelineno-1-7) exclude_external_images=True, [](https://docs.crawl4ai.com/advanced/lazy-loading/#__codelineno-1-8) [](https://docs.crawl4ai.com/advanced/lazy-loading/#__codelineno-1-9) # Exclude certain domains for links [](https://docs.crawl4ai.com/advanced/lazy-loading/#__codelineno-1-10) exclude_domains=["spammycdn.com"], [](https://docs.crawl4ai.com/advanced/lazy-loading/#__codelineno-1-11) )` This approach ensures you see **all** images from the main domain while ignoring external ones, and the crawler physically scrolls the entire page so that lazy-loading triggers. * * * Tips & Troubleshooting ---------------------- 1. **Long Pages** \- Setting `scan_full_page=True` on extremely long or infinite-scroll pages can be resource-intensive. \- Consider using [hooks](https://docs.crawl4ai.com/core/page-interaction/) or specialized logic to load specific sections or “Load More” triggers repeatedly. 2. **Mixed Image Behavior** \- Some sites load images in batches as you scroll. If you’re missing images, increase your `scroll_delay` or call multiple partial scrolls in a loop with JS code or hooks. 3. **Combining with Dynamic Wait** \- If the site has a placeholder that only changes to a real image after a certain event, you might do `wait_for="css:img.loaded"` or a custom JS `wait_for`. 4. **Caching** \- If `cache_mode` is enabled, repeated crawls might skip some network fetches. If you suspect caching is missing new images, set `cache_mode=CacheMode.BYPASS` for fresh fetches. * * * With **lazy-loading** support, **wait\_for\_images**, and **scan\_full\_page** settings, you can capture the entire gallery or feed of images you expect—even if the site only loads them as the user scrolls. Combine these with the standard media filtering and domain exclusion for a complete link & media handling strategy. * * * \> Feedback --- # Hooks & Auth - Crawl4AI Documentation (v0.8.x) Hooks & Auth in AsyncWebCrawler =============================== Crawl4AI’s **hooks** let you customize the crawler at specific points in the pipeline: 1. **`on_browser_created`** – After browser creation. 2. **`on_page_context_created`** – After a new context & page are created. 3. **`before_goto`** – Just before navigating to a page. 4. **`after_goto`** – Right after navigation completes. 5. **`on_user_agent_updated`** – Whenever the user agent changes. 6. **`on_execution_started`** – Once custom JavaScript execution begins. 7. **`before_retrieve_html`** – Just before the crawler retrieves final HTML. 8. **`before_return_html`** – Right before returning the HTML content. **Important**: Avoid heavy tasks in `on_browser_created` since you don’t yet have a page context. If you need to _log in_, do so in **`on_page_context_created`**. > note "Important Hook Usage Warning" **Avoid Misusing Hooks**: Do not manipulate page objects in the wrong hook or at the wrong time, as it can crash the pipeline or produce incorrect results. A common mistake is attempting to handle authentication prematurely—such as creating or closing pages in `on_browser_created`. > > **Use the Right Hook for Auth**: If you need to log in or set tokens, use `on_page_context_created`. This ensures you have a valid page/context to work with, without disrupting the main crawling flow. > > **Identity-Based Crawling**: For robust auth, consider identity-based crawling (or passing a session ID) to preserve state. Run your initial login steps in a separate, well-defined process, then feed that session to your main crawl—rather than shoehorning complex authentication into early hooks. Check out [Identity-Based Crawling](https://docs.crawl4ai.com/advanced/identity-based-crawling/) > for more details. > > **Be Cautious**: Overwriting or removing elements in the wrong hook can compromise the final crawl. Keep hooks focused on smaller tasks (like route filters, custom headers), and let your main logic (crawling, data extraction) proceed normally. Below is an example demonstration. * * * Example: Using Hooks in AsyncWebCrawler --------------------------------------- `[](https://docs.crawl4ai.com/advanced/hooks-auth/#__codelineno-0-1) import asyncio [](https://docs.crawl4ai.com/advanced/hooks-auth/#__codelineno-0-2) import json [](https://docs.crawl4ai.com/advanced/hooks-auth/#__codelineno-0-3) from crawl4ai import AsyncWebCrawler, BrowserConfig, CrawlerRunConfig, CacheMode [](https://docs.crawl4ai.com/advanced/hooks-auth/#__codelineno-0-4) from playwright.async_api import Page, BrowserContext [](https://docs.crawl4ai.com/advanced/hooks-auth/#__codelineno-0-5) [](https://docs.crawl4ai.com/advanced/hooks-auth/#__codelineno-0-6) async def main(): [](https://docs.crawl4ai.com/advanced/hooks-auth/#__codelineno-0-7) print("🔗 Hooks Example: Demonstrating recommended usage") [](https://docs.crawl4ai.com/advanced/hooks-auth/#__codelineno-0-8) [](https://docs.crawl4ai.com/advanced/hooks-auth/#__codelineno-0-9) # 1) Configure the browser [](https://docs.crawl4ai.com/advanced/hooks-auth/#__codelineno-0-10) browser_config = BrowserConfig( [](https://docs.crawl4ai.com/advanced/hooks-auth/#__codelineno-0-11) headless=True, [](https://docs.crawl4ai.com/advanced/hooks-auth/#__codelineno-0-12) verbose=True [](https://docs.crawl4ai.com/advanced/hooks-auth/#__codelineno-0-13) ) [](https://docs.crawl4ai.com/advanced/hooks-auth/#__codelineno-0-14) [](https://docs.crawl4ai.com/advanced/hooks-auth/#__codelineno-0-15) # 2) Configure the crawler run [](https://docs.crawl4ai.com/advanced/hooks-auth/#__codelineno-0-16) crawler_run_config = CrawlerRunConfig( [](https://docs.crawl4ai.com/advanced/hooks-auth/#__codelineno-0-17) js_code="window.scrollTo(0, document.body.scrollHeight);", [](https://docs.crawl4ai.com/advanced/hooks-auth/#__codelineno-0-18) wait_for="body", [](https://docs.crawl4ai.com/advanced/hooks-auth/#__codelineno-0-19) cache_mode=CacheMode.BYPASS [](https://docs.crawl4ai.com/advanced/hooks-auth/#__codelineno-0-20) ) [](https://docs.crawl4ai.com/advanced/hooks-auth/#__codelineno-0-21) [](https://docs.crawl4ai.com/advanced/hooks-auth/#__codelineno-0-22) # 3) Create the crawler instance [](https://docs.crawl4ai.com/advanced/hooks-auth/#__codelineno-0-23) crawler = AsyncWebCrawler(config=browser_config) [](https://docs.crawl4ai.com/advanced/hooks-auth/#__codelineno-0-24) [](https://docs.crawl4ai.com/advanced/hooks-auth/#__codelineno-0-25) # [](https://docs.crawl4ai.com/advanced/hooks-auth/#__codelineno-0-26) # Define Hook Functions [](https://docs.crawl4ai.com/advanced/hooks-auth/#__codelineno-0-27) # [](https://docs.crawl4ai.com/advanced/hooks-auth/#__codelineno-0-28) [](https://docs.crawl4ai.com/advanced/hooks-auth/#__codelineno-0-29) async def on_browser_created(browser, **kwargs): [](https://docs.crawl4ai.com/advanced/hooks-auth/#__codelineno-0-30) # Called once the browser instance is created (but no pages or contexts yet) [](https://docs.crawl4ai.com/advanced/hooks-auth/#__codelineno-0-31) print("[HOOK] on_browser_created - Browser created successfully!") [](https://docs.crawl4ai.com/advanced/hooks-auth/#__codelineno-0-32) # Typically, do minimal setup here if needed [](https://docs.crawl4ai.com/advanced/hooks-auth/#__codelineno-0-33) return browser [](https://docs.crawl4ai.com/advanced/hooks-auth/#__codelineno-0-34) [](https://docs.crawl4ai.com/advanced/hooks-auth/#__codelineno-0-35) async def on_page_context_created(page: Page, context: BrowserContext, **kwargs): [](https://docs.crawl4ai.com/advanced/hooks-auth/#__codelineno-0-36) # Called right after a new page + context are created (ideal for auth or route config). [](https://docs.crawl4ai.com/advanced/hooks-auth/#__codelineno-0-37) print("[HOOK] on_page_context_created - Setting up page & context.") [](https://docs.crawl4ai.com/advanced/hooks-auth/#__codelineno-0-38) [](https://docs.crawl4ai.com/advanced/hooks-auth/#__codelineno-0-39) # Example 1: Route filtering (e.g., block images) [](https://docs.crawl4ai.com/advanced/hooks-auth/#__codelineno-0-40) async def route_filter(route): [](https://docs.crawl4ai.com/advanced/hooks-auth/#__codelineno-0-41) if route.request.resource_type == "image": [](https://docs.crawl4ai.com/advanced/hooks-auth/#__codelineno-0-42) print(f"[HOOK] Blocking image request: {route.request.url}") [](https://docs.crawl4ai.com/advanced/hooks-auth/#__codelineno-0-43) await route.abort() [](https://docs.crawl4ai.com/advanced/hooks-auth/#__codelineno-0-44) else: [](https://docs.crawl4ai.com/advanced/hooks-auth/#__codelineno-0-45) await route.continue_() [](https://docs.crawl4ai.com/advanced/hooks-auth/#__codelineno-0-46) [](https://docs.crawl4ai.com/advanced/hooks-auth/#__codelineno-0-47) await context.route("**", route_filter) [](https://docs.crawl4ai.com/advanced/hooks-auth/#__codelineno-0-48) [](https://docs.crawl4ai.com/advanced/hooks-auth/#__codelineno-0-49) # Example 2: (Optional) Simulate a login scenario [](https://docs.crawl4ai.com/advanced/hooks-auth/#__codelineno-0-50) # (We do NOT create or close pages here, just do quick steps if needed) [](https://docs.crawl4ai.com/advanced/hooks-auth/#__codelineno-0-51) # e.g., await page.goto("https://example.com/login") [](https://docs.crawl4ai.com/advanced/hooks-auth/#__codelineno-0-52) # e.g., await page.fill("input[name='username']", "testuser") [](https://docs.crawl4ai.com/advanced/hooks-auth/#__codelineno-0-53) # e.g., await page.fill("input[name='password']", "password123") [](https://docs.crawl4ai.com/advanced/hooks-auth/#__codelineno-0-54) # e.g., await page.click("button[type='submit']") [](https://docs.crawl4ai.com/advanced/hooks-auth/#__codelineno-0-55) # e.g., await page.wait_for_selector("#welcome") [](https://docs.crawl4ai.com/advanced/hooks-auth/#__codelineno-0-56) # e.g., await context.add_cookies([...]) [](https://docs.crawl4ai.com/advanced/hooks-auth/#__codelineno-0-57) # Then continue [](https://docs.crawl4ai.com/advanced/hooks-auth/#__codelineno-0-58) [](https://docs.crawl4ai.com/advanced/hooks-auth/#__codelineno-0-59) # Example 3: Adjust the viewport [](https://docs.crawl4ai.com/advanced/hooks-auth/#__codelineno-0-60) await page.set_viewport_size({"width": 1080, "height": 600}) [](https://docs.crawl4ai.com/advanced/hooks-auth/#__codelineno-0-61) return page [](https://docs.crawl4ai.com/advanced/hooks-auth/#__codelineno-0-62) [](https://docs.crawl4ai.com/advanced/hooks-auth/#__codelineno-0-63) async def before_goto( [](https://docs.crawl4ai.com/advanced/hooks-auth/#__codelineno-0-64) page: Page, context: BrowserContext, url: str, **kwargs [](https://docs.crawl4ai.com/advanced/hooks-auth/#__codelineno-0-65) ): [](https://docs.crawl4ai.com/advanced/hooks-auth/#__codelineno-0-66) # Called before navigating to each URL. [](https://docs.crawl4ai.com/advanced/hooks-auth/#__codelineno-0-67) print(f"[HOOK] before_goto - About to navigate: {url}") [](https://docs.crawl4ai.com/advanced/hooks-auth/#__codelineno-0-68) # e.g., inject custom headers [](https://docs.crawl4ai.com/advanced/hooks-auth/#__codelineno-0-69) await page.set_extra_http_headers({ [](https://docs.crawl4ai.com/advanced/hooks-auth/#__codelineno-0-70) "Custom-Header": "my-value" [](https://docs.crawl4ai.com/advanced/hooks-auth/#__codelineno-0-71) }) [](https://docs.crawl4ai.com/advanced/hooks-auth/#__codelineno-0-72) return page [](https://docs.crawl4ai.com/advanced/hooks-auth/#__codelineno-0-73) [](https://docs.crawl4ai.com/advanced/hooks-auth/#__codelineno-0-74) async def after_goto( [](https://docs.crawl4ai.com/advanced/hooks-auth/#__codelineno-0-75) page: Page, context: BrowserContext, [](https://docs.crawl4ai.com/advanced/hooks-auth/#__codelineno-0-76) url: str, response, **kwargs [](https://docs.crawl4ai.com/advanced/hooks-auth/#__codelineno-0-77) ): [](https://docs.crawl4ai.com/advanced/hooks-auth/#__codelineno-0-78) # Called after navigation completes. [](https://docs.crawl4ai.com/advanced/hooks-auth/#__codelineno-0-79) print(f"[HOOK] after_goto - Successfully loaded: {url}") [](https://docs.crawl4ai.com/advanced/hooks-auth/#__codelineno-0-80) # e.g., wait for a certain element if we want to verify [](https://docs.crawl4ai.com/advanced/hooks-auth/#__codelineno-0-81) try: [](https://docs.crawl4ai.com/advanced/hooks-auth/#__codelineno-0-82) await page.wait_for_selector('.content', timeout=1000) [](https://docs.crawl4ai.com/advanced/hooks-auth/#__codelineno-0-83) print("[HOOK] Found .content element!") [](https://docs.crawl4ai.com/advanced/hooks-auth/#__codelineno-0-84) except: [](https://docs.crawl4ai.com/advanced/hooks-auth/#__codelineno-0-85) print("[HOOK] .content not found, continuing anyway.") [](https://docs.crawl4ai.com/advanced/hooks-auth/#__codelineno-0-86) return page [](https://docs.crawl4ai.com/advanced/hooks-auth/#__codelineno-0-87) [](https://docs.crawl4ai.com/advanced/hooks-auth/#__codelineno-0-88) async def on_user_agent_updated( [](https://docs.crawl4ai.com/advanced/hooks-auth/#__codelineno-0-89) page: Page, context: BrowserContext, [](https://docs.crawl4ai.com/advanced/hooks-auth/#__codelineno-0-90) user_agent: str, **kwargs [](https://docs.crawl4ai.com/advanced/hooks-auth/#__codelineno-0-91) ): [](https://docs.crawl4ai.com/advanced/hooks-auth/#__codelineno-0-92) # Called whenever the user agent updates. [](https://docs.crawl4ai.com/advanced/hooks-auth/#__codelineno-0-93) print(f"[HOOK] on_user_agent_updated - New user agent: {user_agent}") [](https://docs.crawl4ai.com/advanced/hooks-auth/#__codelineno-0-94) return page [](https://docs.crawl4ai.com/advanced/hooks-auth/#__codelineno-0-95) [](https://docs.crawl4ai.com/advanced/hooks-auth/#__codelineno-0-96) async def on_execution_started(page: Page, context: BrowserContext, **kwargs): [](https://docs.crawl4ai.com/advanced/hooks-auth/#__codelineno-0-97) # Called after custom JavaScript execution begins. [](https://docs.crawl4ai.com/advanced/hooks-auth/#__codelineno-0-98) print("[HOOK] on_execution_started - JS code is running!") [](https://docs.crawl4ai.com/advanced/hooks-auth/#__codelineno-0-99) return page [](https://docs.crawl4ai.com/advanced/hooks-auth/#__codelineno-0-100) [](https://docs.crawl4ai.com/advanced/hooks-auth/#__codelineno-0-101) async def before_retrieve_html(page: Page, context: BrowserContext, **kwargs): [](https://docs.crawl4ai.com/advanced/hooks-auth/#__codelineno-0-102) # Called before final HTML retrieval. [](https://docs.crawl4ai.com/advanced/hooks-auth/#__codelineno-0-103) print("[HOOK] before_retrieve_html - We can do final actions") [](https://docs.crawl4ai.com/advanced/hooks-auth/#__codelineno-0-104) # Example: Scroll again [](https://docs.crawl4ai.com/advanced/hooks-auth/#__codelineno-0-105) await page.evaluate("window.scrollTo(0, document.body.scrollHeight);") [](https://docs.crawl4ai.com/advanced/hooks-auth/#__codelineno-0-106) return page [](https://docs.crawl4ai.com/advanced/hooks-auth/#__codelineno-0-107) [](https://docs.crawl4ai.com/advanced/hooks-auth/#__codelineno-0-108) async def before_return_html( [](https://docs.crawl4ai.com/advanced/hooks-auth/#__codelineno-0-109) page: Page, context: BrowserContext, html: str, **kwargs [](https://docs.crawl4ai.com/advanced/hooks-auth/#__codelineno-0-110) ): [](https://docs.crawl4ai.com/advanced/hooks-auth/#__codelineno-0-111) # Called just before returning the HTML in the result. [](https://docs.crawl4ai.com/advanced/hooks-auth/#__codelineno-0-112) print(f"[HOOK] before_return_html - HTML length: {len(html)}") [](https://docs.crawl4ai.com/advanced/hooks-auth/#__codelineno-0-113) return page [](https://docs.crawl4ai.com/advanced/hooks-auth/#__codelineno-0-114) [](https://docs.crawl4ai.com/advanced/hooks-auth/#__codelineno-0-115) # [](https://docs.crawl4ai.com/advanced/hooks-auth/#__codelineno-0-116) # Attach Hooks [](https://docs.crawl4ai.com/advanced/hooks-auth/#__codelineno-0-117) # [](https://docs.crawl4ai.com/advanced/hooks-auth/#__codelineno-0-118) [](https://docs.crawl4ai.com/advanced/hooks-auth/#__codelineno-0-119) crawler.crawler_strategy.set_hook("on_browser_created", on_browser_created) [](https://docs.crawl4ai.com/advanced/hooks-auth/#__codelineno-0-120) crawler.crawler_strategy.set_hook( [](https://docs.crawl4ai.com/advanced/hooks-auth/#__codelineno-0-121) "on_page_context_created", on_page_context_created [](https://docs.crawl4ai.com/advanced/hooks-auth/#__codelineno-0-122) ) [](https://docs.crawl4ai.com/advanced/hooks-auth/#__codelineno-0-123) crawler.crawler_strategy.set_hook("before_goto", before_goto) [](https://docs.crawl4ai.com/advanced/hooks-auth/#__codelineno-0-124) crawler.crawler_strategy.set_hook("after_goto", after_goto) [](https://docs.crawl4ai.com/advanced/hooks-auth/#__codelineno-0-125) crawler.crawler_strategy.set_hook( [](https://docs.crawl4ai.com/advanced/hooks-auth/#__codelineno-0-126) "on_user_agent_updated", on_user_agent_updated [](https://docs.crawl4ai.com/advanced/hooks-auth/#__codelineno-0-127) ) [](https://docs.crawl4ai.com/advanced/hooks-auth/#__codelineno-0-128) crawler.crawler_strategy.set_hook( [](https://docs.crawl4ai.com/advanced/hooks-auth/#__codelineno-0-129) "on_execution_started", on_execution_started [](https://docs.crawl4ai.com/advanced/hooks-auth/#__codelineno-0-130) ) [](https://docs.crawl4ai.com/advanced/hooks-auth/#__codelineno-0-131) crawler.crawler_strategy.set_hook( [](https://docs.crawl4ai.com/advanced/hooks-auth/#__codelineno-0-132) "before_retrieve_html", before_retrieve_html [](https://docs.crawl4ai.com/advanced/hooks-auth/#__codelineno-0-133) ) [](https://docs.crawl4ai.com/advanced/hooks-auth/#__codelineno-0-134) crawler.crawler_strategy.set_hook( [](https://docs.crawl4ai.com/advanced/hooks-auth/#__codelineno-0-135) "before_return_html", before_return_html [](https://docs.crawl4ai.com/advanced/hooks-auth/#__codelineno-0-136) ) [](https://docs.crawl4ai.com/advanced/hooks-auth/#__codelineno-0-137) [](https://docs.crawl4ai.com/advanced/hooks-auth/#__codelineno-0-138) await crawler.start() [](https://docs.crawl4ai.com/advanced/hooks-auth/#__codelineno-0-139) [](https://docs.crawl4ai.com/advanced/hooks-auth/#__codelineno-0-140) # 4) Run the crawler on an example page [](https://docs.crawl4ai.com/advanced/hooks-auth/#__codelineno-0-141) url = "https://example.com" [](https://docs.crawl4ai.com/advanced/hooks-auth/#__codelineno-0-142) result = await crawler.arun(url, config=crawler_run_config) [](https://docs.crawl4ai.com/advanced/hooks-auth/#__codelineno-0-143) [](https://docs.crawl4ai.com/advanced/hooks-auth/#__codelineno-0-144) if result.success: [](https://docs.crawl4ai.com/advanced/hooks-auth/#__codelineno-0-145) print("\nCrawled URL:", result.url) [](https://docs.crawl4ai.com/advanced/hooks-auth/#__codelineno-0-146) print("HTML length:", len(result.html)) [](https://docs.crawl4ai.com/advanced/hooks-auth/#__codelineno-0-147) else: [](https://docs.crawl4ai.com/advanced/hooks-auth/#__codelineno-0-148) print("Error:", result.error_message) [](https://docs.crawl4ai.com/advanced/hooks-auth/#__codelineno-0-149) [](https://docs.crawl4ai.com/advanced/hooks-auth/#__codelineno-0-150) await crawler.close() [](https://docs.crawl4ai.com/advanced/hooks-auth/#__codelineno-0-151) [](https://docs.crawl4ai.com/advanced/hooks-auth/#__codelineno-0-152) if __name__ == "__main__": [](https://docs.crawl4ai.com/advanced/hooks-auth/#__codelineno-0-153) asyncio.run(main())` * * * Hook Lifecycle Summary ---------------------- 1. **`on_browser_created`**: \- Browser is up, but **no** pages or contexts yet. \- Light setup only—don’t try to open or close pages here (that belongs in `on_page_context_created`). 2. **`on_page_context_created`**: \- Perfect for advanced **auth** or route blocking. \- You have a **page** + **context** ready but haven’t navigated to the target URL yet. 3. **`before_goto`**: \- Right before navigation. Typically used for setting **custom headers** or logging the target URL. 4. **`after_goto`**: \- After page navigation is done. Good place for verifying content or waiting on essential elements. 5. **`on_user_agent_updated`**: \- Whenever the user agent changes (for stealth or different UA modes). 6. **`on_execution_started`**: \- If you set `js_code` or run custom scripts, this runs once your JS is about to start. 7. **`before_retrieve_html`**: \- Just before the final HTML snapshot is taken. Often you do a final scroll or lazy-load triggers here. 8. **`before_return_html`**: \- The last hook before returning HTML to the `CrawlResult`. Good for logging HTML length or minor modifications. * * * When to Handle Authentication ----------------------------- **Recommended**: Use **`on_page_context_created`** if you need to: * Navigate to a login page or fill forms * Set cookies or localStorage tokens * Block resource routes to avoid ads This ensures the newly created context is under your control **before** `arun()` navigates to the main URL. * * * Additional Considerations ------------------------- * **Session Management**: If you want multiple `arun()` calls to reuse a single session, pass `session_id=` in your `CrawlerRunConfig`. Hooks remain the same. * **Performance**: Hooks can slow down crawling if they do heavy tasks. Keep them concise. * **Error Handling**: If a hook fails, the overall crawl might fail. Catch exceptions or handle them gracefully. * **Concurrency**: If you run `arun_many()`, each URL triggers these hooks in parallel. Ensure your hooks are thread/async-safe. * * * Conclusion ---------- Hooks provide **fine-grained** control over: * **Browser** creation (light tasks only) * **Page** and **context** creation (auth, route blocking) * **Navigation** phases * **Final HTML** retrieval Follow the recommended usage: - **Login** or advanced tasks in `on_page_context_created` \- **Custom headers** or logs in `before_goto` / `after_goto` \- **Scrolling** or final checks in `before_retrieve_html` / `before_return_html` * * * \> Feedback --- # Identity Based Crawling - Crawl4AI Documentation (v0.8.x) Preserve Your Identity with Crawl4AI ==================================== Crawl4AI empowers you to navigate and interact with the web using your **authentic digital identity**, ensuring you’re recognized as a human and not mistaken for a bot. This tutorial covers: 1. **Managed Browsers** – The recommended approach for persistent profiles and identity-based crawling. 2. **Magic Mode** – A simplified fallback solution for quick automation without persistent identity. * * * 1\. Managed Browsers: Your Digital Identity Solution ---------------------------------------------------- **Managed Browsers** let developers create and use **persistent browser profiles**. These profiles store local storage, cookies, and other session data, letting you browse as your **real self**—complete with logins, preferences, and cookies. ### Key Benefits * **Authentic Browsing Experience**: Retain session data and browser fingerprints as though you’re a normal user. * **Effortless Configuration**: Once you log in or solve CAPTCHAs in your chosen data directory, you can re-run crawls without repeating those steps. * **Empowered Data Access**: If you can see the data in your own browser, you can automate its retrieval with your genuine identity. * * * Below is a **partial update** to your **Managed Browsers** tutorial, specifically the section about **creating a user-data directory** using **Playwright’s Chromium** binary rather than a system-wide Chrome/Edge. We’ll show how to **locate** that binary and launch it with a `--user-data-dir` argument to set up your profile. You can then point `BrowserConfig.user_data_dir` to that folder for subsequent crawls. * * * ### Creating a User Data Directory (Command-Line Approach via Playwright) If you installed Crawl4AI (which installs Playwright under the hood), you already have a Playwright-managed Chromium on your system. Follow these steps to launch that **Chromium** from your command line, specifying a **custom** data directory: 1. **Find** the Playwright Chromium binary: - On most systems, installed browsers go under a `~/.cache/ms-playwright/` folder or similar path. \- To see an overview of installed browsers, run: `[](https://docs.crawl4ai.com/advanced/identity-based-crawling/#__codelineno-0-1) python -m playwright install --dry-run` or `[](https://docs.crawl4ai.com/advanced/identity-based-crawling/#__codelineno-1-1) playwright install --dry-run` (depending on your environment). This shows where Playwright keeps Chromium. * For instance, you might see a path like: `[](https://docs.crawl4ai.com/advanced/identity-based-crawling/#__codelineno-2-1) ~/.cache/ms-playwright/chromium-1234/chrome-linux/chrome` on Linux, or a corresponding folder on macOS/Windows. 2. **Launch** the Playwright Chromium binary with a **custom** user-data directory: `[](https://docs.crawl4ai.com/advanced/identity-based-crawling/#__codelineno-3-1) # Linux example [](https://docs.crawl4ai.com/advanced/identity-based-crawling/#__codelineno-3-2) ~/.cache/ms-playwright/chromium-1234/chrome-linux/chrome \ [](https://docs.crawl4ai.com/advanced/identity-based-crawling/#__codelineno-3-3) --user-data-dir=/home//my_chrome_profile` `[](https://docs.crawl4ai.com/advanced/identity-based-crawling/#__codelineno-4-1) # macOS example (Playwright’s internal binary) [](https://docs.crawl4ai.com/advanced/identity-based-crawling/#__codelineno-4-2) ~/Library/Caches/ms-playwright/chromium-1234/chrome-mac/Chromium.app/Contents/MacOS/Chromium \ [](https://docs.crawl4ai.com/advanced/identity-based-crawling/#__codelineno-4-3) --user-data-dir=/Users//my_chrome_profile` `[](https://docs.crawl4ai.com/advanced/identity-based-crawling/#__codelineno-5-1) # Windows example (PowerShell/cmd) [](https://docs.crawl4ai.com/advanced/identity-based-crawling/#__codelineno-5-2) "C:\Users\\AppData\Local\ms-playwright\chromium-1234\chrome-win\chrome.exe" ^ [](https://docs.crawl4ai.com/advanced/identity-based-crawling/#__codelineno-5-3) --user-data-dir="C:\Users\\my_chrome_profile"` **Replace** the path with the actual subfolder indicated in your `ms-playwright` cache structure. \- This **opens** a fresh Chromium with your new or existing data folder. \- **Log into** any sites or configure your browser the way you want. \- **Close** when done—your profile data is saved in that folder. 3. **Use** that folder in **`BrowserConfig.user_data_dir`**: `[](https://docs.crawl4ai.com/advanced/identity-based-crawling/#__codelineno-6-1) from crawl4ai import AsyncWebCrawler, BrowserConfig, CrawlerRunConfig [](https://docs.crawl4ai.com/advanced/identity-based-crawling/#__codelineno-6-2) [](https://docs.crawl4ai.com/advanced/identity-based-crawling/#__codelineno-6-3) browser_config = BrowserConfig( [](https://docs.crawl4ai.com/advanced/identity-based-crawling/#__codelineno-6-4) headless=True, [](https://docs.crawl4ai.com/advanced/identity-based-crawling/#__codelineno-6-5) use_managed_browser=True, [](https://docs.crawl4ai.com/advanced/identity-based-crawling/#__codelineno-6-6) user_data_dir="/home//my_chrome_profile", [](https://docs.crawl4ai.com/advanced/identity-based-crawling/#__codelineno-6-7) browser_type="chromium" [](https://docs.crawl4ai.com/advanced/identity-based-crawling/#__codelineno-6-8) )` \- Next time you run your code, it reuses that folder—**preserving** your session data, cookies, local storage, etc. * * * ### Creating a Profile Using the Crawl4AI CLI (Easiest) If you prefer a guided, interactive setup, use the built-in CLI to create and manage persistent browser profiles. 1.⠀Launch the profile manager: `[](https://docs.crawl4ai.com/advanced/identity-based-crawling/#__codelineno-7-1) crwl profiles` 2.⠀Choose "Create new profile" and enter a profile name. A Chromium window opens so you can log in to sites and configure settings. When finished, return to the terminal and press `q` to save the profile. 3.⠀Profiles are saved under `~/.crawl4ai/profiles/` (for example: `/home//.crawl4ai/profiles/test_profile_1`) along with a `storage_state.json` for cookies and session data. 4.⠀Optionally, choose "List profiles" in the CLI to view available profiles and their paths. 5.⠀Use the saved path with `BrowserConfig.user_data_dir`: `[](https://docs.crawl4ai.com/advanced/identity-based-crawling/#__codelineno-8-1) from crawl4ai import AsyncWebCrawler, BrowserConfig [](https://docs.crawl4ai.com/advanced/identity-based-crawling/#__codelineno-8-2) [](https://docs.crawl4ai.com/advanced/identity-based-crawling/#__codelineno-8-3) profile_path = "/home//.crawl4ai/profiles/test_profile_1" [](https://docs.crawl4ai.com/advanced/identity-based-crawling/#__codelineno-8-4) [](https://docs.crawl4ai.com/advanced/identity-based-crawling/#__codelineno-8-5) browser_config = BrowserConfig( [](https://docs.crawl4ai.com/advanced/identity-based-crawling/#__codelineno-8-6) headless=True, [](https://docs.crawl4ai.com/advanced/identity-based-crawling/#__codelineno-8-7) use_managed_browser=True, [](https://docs.crawl4ai.com/advanced/identity-based-crawling/#__codelineno-8-8) user_data_dir=profile_path, [](https://docs.crawl4ai.com/advanced/identity-based-crawling/#__codelineno-8-9) browser_type="chromium", [](https://docs.crawl4ai.com/advanced/identity-based-crawling/#__codelineno-8-10) ) [](https://docs.crawl4ai.com/advanced/identity-based-crawling/#__codelineno-8-11) [](https://docs.crawl4ai.com/advanced/identity-based-crawling/#__codelineno-8-12) async with AsyncWebCrawler(config=browser_config) as crawler: [](https://docs.crawl4ai.com/advanced/identity-based-crawling/#__codelineno-8-13) result = await crawler.arun(url="https://example.com/private")` The CLI also supports listing and deleting profiles, and even testing a crawl directly from the menu. * * * 3\. Using Managed Browsers in Crawl4AI -------------------------------------- Once you have a data directory with your session data, pass it to **`BrowserConfig`**: `[](https://docs.crawl4ai.com/advanced/identity-based-crawling/#__codelineno-9-1) import asyncio [](https://docs.crawl4ai.com/advanced/identity-based-crawling/#__codelineno-9-2) from crawl4ai import AsyncWebCrawler, BrowserConfig, CrawlerRunConfig [](https://docs.crawl4ai.com/advanced/identity-based-crawling/#__codelineno-9-3) [](https://docs.crawl4ai.com/advanced/identity-based-crawling/#__codelineno-9-4) async def main(): [](https://docs.crawl4ai.com/advanced/identity-based-crawling/#__codelineno-9-5) # 1) Reference your persistent data directory [](https://docs.crawl4ai.com/advanced/identity-based-crawling/#__codelineno-9-6) browser_config = BrowserConfig( [](https://docs.crawl4ai.com/advanced/identity-based-crawling/#__codelineno-9-7) headless=True, # 'True' for automated runs [](https://docs.crawl4ai.com/advanced/identity-based-crawling/#__codelineno-9-8) verbose=True, [](https://docs.crawl4ai.com/advanced/identity-based-crawling/#__codelineno-9-9) use_managed_browser=True, # Enables persistent browser strategy [](https://docs.crawl4ai.com/advanced/identity-based-crawling/#__codelineno-9-10) browser_type="chromium", [](https://docs.crawl4ai.com/advanced/identity-based-crawling/#__codelineno-9-11) user_data_dir="/path/to/my-chrome-profile" [](https://docs.crawl4ai.com/advanced/identity-based-crawling/#__codelineno-9-12) ) [](https://docs.crawl4ai.com/advanced/identity-based-crawling/#__codelineno-9-13) [](https://docs.crawl4ai.com/advanced/identity-based-crawling/#__codelineno-9-14) # 2) Standard crawl config [](https://docs.crawl4ai.com/advanced/identity-based-crawling/#__codelineno-9-15) crawl_config = CrawlerRunConfig( [](https://docs.crawl4ai.com/advanced/identity-based-crawling/#__codelineno-9-16) wait_for="css:.logged-in-content" [](https://docs.crawl4ai.com/advanced/identity-based-crawling/#__codelineno-9-17) ) [](https://docs.crawl4ai.com/advanced/identity-based-crawling/#__codelineno-9-18) [](https://docs.crawl4ai.com/advanced/identity-based-crawling/#__codelineno-9-19) async with AsyncWebCrawler(config=browser_config) as crawler: [](https://docs.crawl4ai.com/advanced/identity-based-crawling/#__codelineno-9-20) result = await crawler.arun(url="https://example.com/private", config=crawl_config) [](https://docs.crawl4ai.com/advanced/identity-based-crawling/#__codelineno-9-21) if result.success: [](https://docs.crawl4ai.com/advanced/identity-based-crawling/#__codelineno-9-22) print("Successfully accessed private data with your identity!") [](https://docs.crawl4ai.com/advanced/identity-based-crawling/#__codelineno-9-23) else: [](https://docs.crawl4ai.com/advanced/identity-based-crawling/#__codelineno-9-24) print("Error:", result.error_message) [](https://docs.crawl4ai.com/advanced/identity-based-crawling/#__codelineno-9-25) [](https://docs.crawl4ai.com/advanced/identity-based-crawling/#__codelineno-9-26) if __name__ == "__main__": [](https://docs.crawl4ai.com/advanced/identity-based-crawling/#__codelineno-9-27) asyncio.run(main())` ### Workflow 1. **Login** externally (via CLI or your normal Chrome with `--user-data-dir=...`). 2. **Close** that browser. 3. **Use** the same folder in `user_data_dir=` in Crawl4AI. 4. **Crawl** – The site sees your identity as if you’re the same user who just logged in. * * * 4\. Magic Mode: Simplified Automation ------------------------------------- If you **don’t** need a persistent profile or identity-based approach, **Magic Mode** offers a quick way to simulate human-like browsing without storing long-term data. `[](https://docs.crawl4ai.com/advanced/identity-based-crawling/#__codelineno-10-1) from crawl4ai import AsyncWebCrawler, CrawlerRunConfig [](https://docs.crawl4ai.com/advanced/identity-based-crawling/#__codelineno-10-2) [](https://docs.crawl4ai.com/advanced/identity-based-crawling/#__codelineno-10-3) async with AsyncWebCrawler() as crawler: [](https://docs.crawl4ai.com/advanced/identity-based-crawling/#__codelineno-10-4) result = await crawler.arun( [](https://docs.crawl4ai.com/advanced/identity-based-crawling/#__codelineno-10-5) url="https://example.com", [](https://docs.crawl4ai.com/advanced/identity-based-crawling/#__codelineno-10-6) config=CrawlerRunConfig( [](https://docs.crawl4ai.com/advanced/identity-based-crawling/#__codelineno-10-7) magic=True, # Simplifies a lot of interaction [](https://docs.crawl4ai.com/advanced/identity-based-crawling/#__codelineno-10-8) remove_overlay_elements=True, [](https://docs.crawl4ai.com/advanced/identity-based-crawling/#__codelineno-10-9) page_timeout=60000 [](https://docs.crawl4ai.com/advanced/identity-based-crawling/#__codelineno-10-10) ) [](https://docs.crawl4ai.com/advanced/identity-based-crawling/#__codelineno-10-11) )` **Magic Mode**: * Simulates a user-like experience * Randomizes user agent & navigator * Randomizes interactions & timings * Masks automation signals * Attempts pop-up handling **But** it’s no substitute for **true** user-based sessions if you want a fully legitimate identity-based solution. * * * 5\. Comparing Managed Browsers vs. Magic Mode --------------------------------------------- | Feature | **Managed Browsers** | **Magic Mode** | | --- | --- | --- | | **Session Persistence** | Full localStorage/cookies retained in user\_data\_dir | No persistent data (fresh each run) | | **Genuine Identity** | Real user profile with full rights & preferences | Emulated user-like patterns, but no actual identity | | **Complex Sites** | Best for login-gated sites or heavy config | Simple tasks, minimal login or config needed | | **Setup** | External creation of user\_data\_dir, then use in Crawl4AI | Single-line approach (`magic=True`) | | **Reliability** | Extremely consistent (same data across runs) | Good for smaller tasks, can be less stable | * * * 6\. Using the BrowserProfiler Class ----------------------------------- Crawl4AI provides a dedicated `BrowserProfiler` class for managing browser profiles, making it easy to create, list, and delete profiles for identity-based browsing. ### Creating and Managing Profiles with BrowserProfiler The `BrowserProfiler` class offers a comprehensive API for browser profile management: `[](https://docs.crawl4ai.com/advanced/identity-based-crawling/#__codelineno-11-1) import asyncio [](https://docs.crawl4ai.com/advanced/identity-based-crawling/#__codelineno-11-2) from crawl4ai import BrowserProfiler [](https://docs.crawl4ai.com/advanced/identity-based-crawling/#__codelineno-11-3) [](https://docs.crawl4ai.com/advanced/identity-based-crawling/#__codelineno-11-4) async def manage_profiles(): [](https://docs.crawl4ai.com/advanced/identity-based-crawling/#__codelineno-11-5) # Create a profiler instance [](https://docs.crawl4ai.com/advanced/identity-based-crawling/#__codelineno-11-6) profiler = BrowserProfiler() [](https://docs.crawl4ai.com/advanced/identity-based-crawling/#__codelineno-11-7) [](https://docs.crawl4ai.com/advanced/identity-based-crawling/#__codelineno-11-8) # Create a profile interactively - opens a browser window [](https://docs.crawl4ai.com/advanced/identity-based-crawling/#__codelineno-11-9) profile_path = await profiler.create_profile( [](https://docs.crawl4ai.com/advanced/identity-based-crawling/#__codelineno-11-10) profile_name="my-login-profile" # Optional: name your profile [](https://docs.crawl4ai.com/advanced/identity-based-crawling/#__codelineno-11-11) ) [](https://docs.crawl4ai.com/advanced/identity-based-crawling/#__codelineno-11-12) [](https://docs.crawl4ai.com/advanced/identity-based-crawling/#__codelineno-11-13) print(f"Profile saved at: {profile_path}") [](https://docs.crawl4ai.com/advanced/identity-based-crawling/#__codelineno-11-14) [](https://docs.crawl4ai.com/advanced/identity-based-crawling/#__codelineno-11-15) # List all available profiles [](https://docs.crawl4ai.com/advanced/identity-based-crawling/#__codelineno-11-16) profiles = profiler.list_profiles() [](https://docs.crawl4ai.com/advanced/identity-based-crawling/#__codelineno-11-17) [](https://docs.crawl4ai.com/advanced/identity-based-crawling/#__codelineno-11-18) for profile in profiles: [](https://docs.crawl4ai.com/advanced/identity-based-crawling/#__codelineno-11-19) print(f"Profile: {profile['name']}") [](https://docs.crawl4ai.com/advanced/identity-based-crawling/#__codelineno-11-20) print(f" Path: {profile['path']}") [](https://docs.crawl4ai.com/advanced/identity-based-crawling/#__codelineno-11-21) print(f" Created: {profile['created']}") [](https://docs.crawl4ai.com/advanced/identity-based-crawling/#__codelineno-11-22) print(f" Browser type: {profile['type']}") [](https://docs.crawl4ai.com/advanced/identity-based-crawling/#__codelineno-11-23) [](https://docs.crawl4ai.com/advanced/identity-based-crawling/#__codelineno-11-24) # Get a specific profile path by name [](https://docs.crawl4ai.com/advanced/identity-based-crawling/#__codelineno-11-25) specific_profile = profiler.get_profile_path("my-login-profile") [](https://docs.crawl4ai.com/advanced/identity-based-crawling/#__codelineno-11-26) [](https://docs.crawl4ai.com/advanced/identity-based-crawling/#__codelineno-11-27) # Delete a profile when no longer needed [](https://docs.crawl4ai.com/advanced/identity-based-crawling/#__codelineno-11-28) success = profiler.delete_profile("old-profile-name") [](https://docs.crawl4ai.com/advanced/identity-based-crawling/#__codelineno-11-29) [](https://docs.crawl4ai.com/advanced/identity-based-crawling/#__codelineno-11-30) asyncio.run(manage_profiles())` **How profile creation works:** 1. A browser window opens for you to interact with 2. You log in to websites, set preferences, etc. 3. When you're done, press 'q' in the terminal to close the browser 4. The profile is saved in the Crawl4AI profiles directory 5. You can use the returned path with `BrowserConfig.user_data_dir` ### Interactive Profile Management The `BrowserProfiler` also offers an interactive management console that guides you through profile creation, listing, and deletion: `[](https://docs.crawl4ai.com/advanced/identity-based-crawling/#__codelineno-12-1) import asyncio [](https://docs.crawl4ai.com/advanced/identity-based-crawling/#__codelineno-12-2) from crawl4ai import BrowserProfiler, AsyncWebCrawler, BrowserConfig [](https://docs.crawl4ai.com/advanced/identity-based-crawling/#__codelineno-12-3) [](https://docs.crawl4ai.com/advanced/identity-based-crawling/#__codelineno-12-4) # Define a function to use a profile for crawling [](https://docs.crawl4ai.com/advanced/identity-based-crawling/#__codelineno-12-5) async def crawl_with_profile(profile_path, url): [](https://docs.crawl4ai.com/advanced/identity-based-crawling/#__codelineno-12-6) browser_config = BrowserConfig( [](https://docs.crawl4ai.com/advanced/identity-based-crawling/#__codelineno-12-7) headless=True, [](https://docs.crawl4ai.com/advanced/identity-based-crawling/#__codelineno-12-8) use_managed_browser=True, [](https://docs.crawl4ai.com/advanced/identity-based-crawling/#__codelineno-12-9) user_data_dir=profile_path [](https://docs.crawl4ai.com/advanced/identity-based-crawling/#__codelineno-12-10) ) [](https://docs.crawl4ai.com/advanced/identity-based-crawling/#__codelineno-12-11) [](https://docs.crawl4ai.com/advanced/identity-based-crawling/#__codelineno-12-12) async with AsyncWebCrawler(config=browser_config) as crawler: [](https://docs.crawl4ai.com/advanced/identity-based-crawling/#__codelineno-12-13) result = await crawler.arun(url) [](https://docs.crawl4ai.com/advanced/identity-based-crawling/#__codelineno-12-14) return result [](https://docs.crawl4ai.com/advanced/identity-based-crawling/#__codelineno-12-15) [](https://docs.crawl4ai.com/advanced/identity-based-crawling/#__codelineno-12-16) async def main(): [](https://docs.crawl4ai.com/advanced/identity-based-crawling/#__codelineno-12-17) # Create a profiler instance [](https://docs.crawl4ai.com/advanced/identity-based-crawling/#__codelineno-12-18) profiler = BrowserProfiler() [](https://docs.crawl4ai.com/advanced/identity-based-crawling/#__codelineno-12-19) [](https://docs.crawl4ai.com/advanced/identity-based-crawling/#__codelineno-12-20) # Launch the interactive profile manager [](https://docs.crawl4ai.com/advanced/identity-based-crawling/#__codelineno-12-21) # Passing the crawl function as a callback adds a "crawl with profile" option [](https://docs.crawl4ai.com/advanced/identity-based-crawling/#__codelineno-12-22) await profiler.interactive_manager(crawl_callback=crawl_with_profile) [](https://docs.crawl4ai.com/advanced/identity-based-crawling/#__codelineno-12-23) [](https://docs.crawl4ai.com/advanced/identity-based-crawling/#__codelineno-12-24) asyncio.run(main())` ### Legacy Methods For backward compatibility, the previous methods on `ManagedBrowser` are still available, but they delegate to the new `BrowserProfiler` class: `[](https://docs.crawl4ai.com/advanced/identity-based-crawling/#__codelineno-13-1) from crawl4ai.browser_manager import ManagedBrowser [](https://docs.crawl4ai.com/advanced/identity-based-crawling/#__codelineno-13-2) [](https://docs.crawl4ai.com/advanced/identity-based-crawling/#__codelineno-13-3) # These methods still work but use BrowserProfiler internally [](https://docs.crawl4ai.com/advanced/identity-based-crawling/#__codelineno-13-4) profiles = ManagedBrowser.list_profiles()` ### Complete Example See the full example in `docs/examples/identity_based_browsing.py` for a complete demonstration of creating and using profiles for authenticated browsing using the new `BrowserProfiler` class. * * * 7\. Locale, Timezone, and Geolocation Control --------------------------------------------- In addition to using persistent profiles, Crawl4AI supports customizing your browser's locale, timezone, and geolocation settings. These features enhance your identity-based browsing experience by allowing you to control how websites perceive your location and regional settings. ### Setting Locale and Timezone You can set the browser's locale and timezone through `CrawlerRunConfig`: `[](https://docs.crawl4ai.com/advanced/identity-based-crawling/#__codelineno-14-1) from crawl4ai import AsyncWebCrawler, CrawlerRunConfig [](https://docs.crawl4ai.com/advanced/identity-based-crawling/#__codelineno-14-2) [](https://docs.crawl4ai.com/advanced/identity-based-crawling/#__codelineno-14-3) async with AsyncWebCrawler() as crawler: [](https://docs.crawl4ai.com/advanced/identity-based-crawling/#__codelineno-14-4) result = await crawler.arun( [](https://docs.crawl4ai.com/advanced/identity-based-crawling/#__codelineno-14-5) url="https://example.com", [](https://docs.crawl4ai.com/advanced/identity-based-crawling/#__codelineno-14-6) config=CrawlerRunConfig( [](https://docs.crawl4ai.com/advanced/identity-based-crawling/#__codelineno-14-7) # Set browser locale (language and region formatting) [](https://docs.crawl4ai.com/advanced/identity-based-crawling/#__codelineno-14-8) locale="fr-FR", # French (France) [](https://docs.crawl4ai.com/advanced/identity-based-crawling/#__codelineno-14-9) [](https://docs.crawl4ai.com/advanced/identity-based-crawling/#__codelineno-14-10) # Set browser timezone [](https://docs.crawl4ai.com/advanced/identity-based-crawling/#__codelineno-14-11) timezone_id="Europe/Paris", [](https://docs.crawl4ai.com/advanced/identity-based-crawling/#__codelineno-14-12) [](https://docs.crawl4ai.com/advanced/identity-based-crawling/#__codelineno-14-13) # Other normal options... [](https://docs.crawl4ai.com/advanced/identity-based-crawling/#__codelineno-14-14) magic=True, [](https://docs.crawl4ai.com/advanced/identity-based-crawling/#__codelineno-14-15) page_timeout=60000 [](https://docs.crawl4ai.com/advanced/identity-based-crawling/#__codelineno-14-16) ) [](https://docs.crawl4ai.com/advanced/identity-based-crawling/#__codelineno-14-17) )` **How it works:** - `locale` affects language preferences, date formats, number formats, etc. - `timezone_id` affects JavaScript's Date object and time-related functionality - These settings are applied when creating the browser context and maintained throughout the session ### Configuring Geolocation Control the GPS coordinates reported by the browser's geolocation API: `[](https://docs.crawl4ai.com/advanced/identity-based-crawling/#__codelineno-15-1) from crawl4ai import AsyncWebCrawler, CrawlerRunConfig, GeolocationConfig [](https://docs.crawl4ai.com/advanced/identity-based-crawling/#__codelineno-15-2) [](https://docs.crawl4ai.com/advanced/identity-based-crawling/#__codelineno-15-3) async with AsyncWebCrawler() as crawler: [](https://docs.crawl4ai.com/advanced/identity-based-crawling/#__codelineno-15-4) result = await crawler.arun( [](https://docs.crawl4ai.com/advanced/identity-based-crawling/#__codelineno-15-5) url="https://maps.google.com", # Or any location-aware site [](https://docs.crawl4ai.com/advanced/identity-based-crawling/#__codelineno-15-6) config=CrawlerRunConfig( [](https://docs.crawl4ai.com/advanced/identity-based-crawling/#__codelineno-15-7) # Configure precise GPS coordinates [](https://docs.crawl4ai.com/advanced/identity-based-crawling/#__codelineno-15-8) geolocation=GeolocationConfig( [](https://docs.crawl4ai.com/advanced/identity-based-crawling/#__codelineno-15-9) latitude=48.8566, # Paris coordinates [](https://docs.crawl4ai.com/advanced/identity-based-crawling/#__codelineno-15-10) longitude=2.3522, [](https://docs.crawl4ai.com/advanced/identity-based-crawling/#__codelineno-15-11) accuracy=100 # Accuracy in meters (optional) [](https://docs.crawl4ai.com/advanced/identity-based-crawling/#__codelineno-15-12) ), [](https://docs.crawl4ai.com/advanced/identity-based-crawling/#__codelineno-15-13) [](https://docs.crawl4ai.com/advanced/identity-based-crawling/#__codelineno-15-14) # This site will see you as being in Paris [](https://docs.crawl4ai.com/advanced/identity-based-crawling/#__codelineno-15-15) page_timeout=60000 [](https://docs.crawl4ai.com/advanced/identity-based-crawling/#__codelineno-15-16) ) [](https://docs.crawl4ai.com/advanced/identity-based-crawling/#__codelineno-15-17) )` **Important notes:** - When `geolocation` is specified, the browser is automatically granted permission to access location - Websites using the Geolocation API will receive the exact coordinates you specify - This affects map services, store locators, delivery services, etc. - Combined with the appropriate `locale` and `timezone_id`, you can create a fully consistent location profile ### Combining with Managed Browsers These settings work perfectly with managed browsers for a complete identity solution: `[](https://docs.crawl4ai.com/advanced/identity-based-crawling/#__codelineno-16-1) from crawl4ai import ( [](https://docs.crawl4ai.com/advanced/identity-based-crawling/#__codelineno-16-2) AsyncWebCrawler, BrowserConfig, CrawlerRunConfig, [](https://docs.crawl4ai.com/advanced/identity-based-crawling/#__codelineno-16-3) GeolocationConfig [](https://docs.crawl4ai.com/advanced/identity-based-crawling/#__codelineno-16-4) ) [](https://docs.crawl4ai.com/advanced/identity-based-crawling/#__codelineno-16-5) [](https://docs.crawl4ai.com/advanced/identity-based-crawling/#__codelineno-16-6) browser_config = BrowserConfig( [](https://docs.crawl4ai.com/advanced/identity-based-crawling/#__codelineno-16-7) use_managed_browser=True, [](https://docs.crawl4ai.com/advanced/identity-based-crawling/#__codelineno-16-8) user_data_dir="/path/to/my-profile", [](https://docs.crawl4ai.com/advanced/identity-based-crawling/#__codelineno-16-9) browser_type="chromium" [](https://docs.crawl4ai.com/advanced/identity-based-crawling/#__codelineno-16-10) ) [](https://docs.crawl4ai.com/advanced/identity-based-crawling/#__codelineno-16-11) [](https://docs.crawl4ai.com/advanced/identity-based-crawling/#__codelineno-16-12) crawl_config = CrawlerRunConfig( [](https://docs.crawl4ai.com/advanced/identity-based-crawling/#__codelineno-16-13) # Location settings [](https://docs.crawl4ai.com/advanced/identity-based-crawling/#__codelineno-16-14) locale="es-MX", # Spanish (Mexico) [](https://docs.crawl4ai.com/advanced/identity-based-crawling/#__codelineno-16-15) timezone_id="America/Mexico_City", [](https://docs.crawl4ai.com/advanced/identity-based-crawling/#__codelineno-16-16) geolocation=GeolocationConfig( [](https://docs.crawl4ai.com/advanced/identity-based-crawling/#__codelineno-16-17) latitude=19.4326, # Mexico City [](https://docs.crawl4ai.com/advanced/identity-based-crawling/#__codelineno-16-18) longitude=-99.1332 [](https://docs.crawl4ai.com/advanced/identity-based-crawling/#__codelineno-16-19) ) [](https://docs.crawl4ai.com/advanced/identity-based-crawling/#__codelineno-16-20) ) [](https://docs.crawl4ai.com/advanced/identity-based-crawling/#__codelineno-16-21) [](https://docs.crawl4ai.com/advanced/identity-based-crawling/#__codelineno-16-22) async with AsyncWebCrawler(config=browser_config) as crawler: [](https://docs.crawl4ai.com/advanced/identity-based-crawling/#__codelineno-16-23) result = await crawler.arun(url="https://example.com", config=crawl_config)` Combining persistent profiles with precise geolocation and region settings gives you complete control over your digital identity. 8\. Summary ----------- * **Create** your user-data directory either: * By launching Chrome/Chromium externally with `--user-data-dir=/some/path` * Or by using the built-in `BrowserProfiler.create_profile()` method * Or through the interactive interface with `profiler.interactive_manager()` * **Log in** or configure sites as needed, then close the browser * **Reference** that folder in `BrowserConfig(user_data_dir="...")` + `use_managed_browser=True` * **Customize** identity aspects with `locale`, `timezone_id`, and `geolocation` * **List and reuse** profiles with `BrowserProfiler.list_profiles()` * **Manage** your profiles with the dedicated `BrowserProfiler` class * Enjoy **persistent** sessions that reflect your real identity * If you only need quick, ephemeral automation, **Magic Mode** might suffice **Recommended**: Always prefer a **Managed Browser** for robust, identity-based crawling and simpler interactions with complex sites. Use **Magic Mode** for quick tasks or prototypes where persistent data is unnecessary. With these approaches, you preserve your **authentic** browsing environment, ensuring the site sees you exactly as a normal user—no repeated logins or wasted time. * * * \> Feedback --- # Network & Console Capture - Crawl4AI Documentation (v0.8.x) Network Requests & Console Message Capturing ============================================ Crawl4AI can capture all network requests and browser console messages during a crawl, which is invaluable for debugging, security analysis, or understanding page behavior. Configuration ------------- To enable network and console capturing, use these configuration options: `[](https://docs.crawl4ai.com/advanced/network-console-capture/#__codelineno-0-1) from crawl4ai import AsyncWebCrawler, CrawlerRunConfig [](https://docs.crawl4ai.com/advanced/network-console-capture/#__codelineno-0-2) [](https://docs.crawl4ai.com/advanced/network-console-capture/#__codelineno-0-3) # Enable both network request capture and console message capture [](https://docs.crawl4ai.com/advanced/network-console-capture/#__codelineno-0-4) config = CrawlerRunConfig( [](https://docs.crawl4ai.com/advanced/network-console-capture/#__codelineno-0-5) capture_network_requests=True, # Capture all network requests and responses [](https://docs.crawl4ai.com/advanced/network-console-capture/#__codelineno-0-6) capture_console_messages=True # Capture all browser console output [](https://docs.crawl4ai.com/advanced/network-console-capture/#__codelineno-0-7) )` Example Usage ------------- `[](https://docs.crawl4ai.com/advanced/network-console-capture/#__codelineno-1-1) import asyncio [](https://docs.crawl4ai.com/advanced/network-console-capture/#__codelineno-1-2) import json [](https://docs.crawl4ai.com/advanced/network-console-capture/#__codelineno-1-3) from crawl4ai import AsyncWebCrawler, CrawlerRunConfig [](https://docs.crawl4ai.com/advanced/network-console-capture/#__codelineno-1-4) [](https://docs.crawl4ai.com/advanced/network-console-capture/#__codelineno-1-5) async def main(): [](https://docs.crawl4ai.com/advanced/network-console-capture/#__codelineno-1-6) # Enable both network request capture and console message capture [](https://docs.crawl4ai.com/advanced/network-console-capture/#__codelineno-1-7) config = CrawlerRunConfig( [](https://docs.crawl4ai.com/advanced/network-console-capture/#__codelineno-1-8) capture_network_requests=True, [](https://docs.crawl4ai.com/advanced/network-console-capture/#__codelineno-1-9) capture_console_messages=True [](https://docs.crawl4ai.com/advanced/network-console-capture/#__codelineno-1-10) ) [](https://docs.crawl4ai.com/advanced/network-console-capture/#__codelineno-1-11) [](https://docs.crawl4ai.com/advanced/network-console-capture/#__codelineno-1-12) async with AsyncWebCrawler() as crawler: [](https://docs.crawl4ai.com/advanced/network-console-capture/#__codelineno-1-13) result = await crawler.arun( [](https://docs.crawl4ai.com/advanced/network-console-capture/#__codelineno-1-14) url="https://example.com", [](https://docs.crawl4ai.com/advanced/network-console-capture/#__codelineno-1-15) config=config [](https://docs.crawl4ai.com/advanced/network-console-capture/#__codelineno-1-16) ) [](https://docs.crawl4ai.com/advanced/network-console-capture/#__codelineno-1-17) [](https://docs.crawl4ai.com/advanced/network-console-capture/#__codelineno-1-18) if result.success: [](https://docs.crawl4ai.com/advanced/network-console-capture/#__codelineno-1-19) # Analyze network requests [](https://docs.crawl4ai.com/advanced/network-console-capture/#__codelineno-1-20) if result.network_requests: [](https://docs.crawl4ai.com/advanced/network-console-capture/#__codelineno-1-21) print(f"Captured {len(result.network_requests)} network events") [](https://docs.crawl4ai.com/advanced/network-console-capture/#__codelineno-1-22) [](https://docs.crawl4ai.com/advanced/network-console-capture/#__codelineno-1-23) # Count request types [](https://docs.crawl4ai.com/advanced/network-console-capture/#__codelineno-1-24) request_count = len([r for r in result.network_requests if r.get("event_type") == "request"]) [](https://docs.crawl4ai.com/advanced/network-console-capture/#__codelineno-1-25) response_count = len([r for r in result.network_requests if r.get("event_type") == "response"]) [](https://docs.crawl4ai.com/advanced/network-console-capture/#__codelineno-1-26) failed_count = len([r for r in result.network_requests if r.get("event_type") == "request_failed"]) [](https://docs.crawl4ai.com/advanced/network-console-capture/#__codelineno-1-27) [](https://docs.crawl4ai.com/advanced/network-console-capture/#__codelineno-1-28) print(f"Requests: {request_count}, Responses: {response_count}, Failed: {failed_count}") [](https://docs.crawl4ai.com/advanced/network-console-capture/#__codelineno-1-29) [](https://docs.crawl4ai.com/advanced/network-console-capture/#__codelineno-1-30) # Find API calls [](https://docs.crawl4ai.com/advanced/network-console-capture/#__codelineno-1-31) api_calls = [r for r in result.network_requests [](https://docs.crawl4ai.com/advanced/network-console-capture/#__codelineno-1-32) if r.get("event_type") == "request" and "api" in r.get("url", "")] [](https://docs.crawl4ai.com/advanced/network-console-capture/#__codelineno-1-33) if api_calls: [](https://docs.crawl4ai.com/advanced/network-console-capture/#__codelineno-1-34) print(f"Detected {len(api_calls)} API calls:") [](https://docs.crawl4ai.com/advanced/network-console-capture/#__codelineno-1-35) for call in api_calls[:3]: # Show first 3 [](https://docs.crawl4ai.com/advanced/network-console-capture/#__codelineno-1-36) print(f" - {call.get('method')} {call.get('url')}") [](https://docs.crawl4ai.com/advanced/network-console-capture/#__codelineno-1-37) [](https://docs.crawl4ai.com/advanced/network-console-capture/#__codelineno-1-38) # Analyze console messages [](https://docs.crawl4ai.com/advanced/network-console-capture/#__codelineno-1-39) if result.console_messages: [](https://docs.crawl4ai.com/advanced/network-console-capture/#__codelineno-1-40) print(f"Captured {len(result.console_messages)} console messages") [](https://docs.crawl4ai.com/advanced/network-console-capture/#__codelineno-1-41) [](https://docs.crawl4ai.com/advanced/network-console-capture/#__codelineno-1-42) # Group by type [](https://docs.crawl4ai.com/advanced/network-console-capture/#__codelineno-1-43) message_types = {} [](https://docs.crawl4ai.com/advanced/network-console-capture/#__codelineno-1-44) for msg in result.console_messages: [](https://docs.crawl4ai.com/advanced/network-console-capture/#__codelineno-1-45) msg_type = msg.get("type", "unknown") [](https://docs.crawl4ai.com/advanced/network-console-capture/#__codelineno-1-46) message_types[msg_type] = message_types.get(msg_type, 0) + 1 [](https://docs.crawl4ai.com/advanced/network-console-capture/#__codelineno-1-47) [](https://docs.crawl4ai.com/advanced/network-console-capture/#__codelineno-1-48) print("Message types:", message_types) [](https://docs.crawl4ai.com/advanced/network-console-capture/#__codelineno-1-49) [](https://docs.crawl4ai.com/advanced/network-console-capture/#__codelineno-1-50) # Show errors (often the most important) [](https://docs.crawl4ai.com/advanced/network-console-capture/#__codelineno-1-51) errors = [msg for msg in result.console_messages if msg.get("type") == "error"] [](https://docs.crawl4ai.com/advanced/network-console-capture/#__codelineno-1-52) if errors: [](https://docs.crawl4ai.com/advanced/network-console-capture/#__codelineno-1-53) print(f"Found {len(errors)} console errors:") [](https://docs.crawl4ai.com/advanced/network-console-capture/#__codelineno-1-54) for err in errors[:2]: # Show first 2 [](https://docs.crawl4ai.com/advanced/network-console-capture/#__codelineno-1-55) print(f" - {err.get('text', '')[:100]}") [](https://docs.crawl4ai.com/advanced/network-console-capture/#__codelineno-1-56) [](https://docs.crawl4ai.com/advanced/network-console-capture/#__codelineno-1-57) # Export all captured data to a file for detailed analysis [](https://docs.crawl4ai.com/advanced/network-console-capture/#__codelineno-1-58) with open("network_capture.json", "w") as f: [](https://docs.crawl4ai.com/advanced/network-console-capture/#__codelineno-1-59) json.dump({ [](https://docs.crawl4ai.com/advanced/network-console-capture/#__codelineno-1-60) "url": result.url, [](https://docs.crawl4ai.com/advanced/network-console-capture/#__codelineno-1-61) "network_requests": result.network_requests or [], [](https://docs.crawl4ai.com/advanced/network-console-capture/#__codelineno-1-62) "console_messages": result.console_messages or [] [](https://docs.crawl4ai.com/advanced/network-console-capture/#__codelineno-1-63) }, f, indent=2) [](https://docs.crawl4ai.com/advanced/network-console-capture/#__codelineno-1-64) [](https://docs.crawl4ai.com/advanced/network-console-capture/#__codelineno-1-65) print("Exported detailed capture data to network_capture.json") [](https://docs.crawl4ai.com/advanced/network-console-capture/#__codelineno-1-66) [](https://docs.crawl4ai.com/advanced/network-console-capture/#__codelineno-1-67) if __name__ == "__main__": [](https://docs.crawl4ai.com/advanced/network-console-capture/#__codelineno-1-68) asyncio.run(main())` Captured Data Structure ----------------------- ### Network Requests The `result.network_requests` contains a list of dictionaries, each representing a network event with these common fields: | Field | Description | | --- | --- | | `event_type` | Type of event: `"request"`, `"response"`, or `"request_failed"` | | `url` | The URL of the request | | `timestamp` | Unix timestamp when the event was captured | #### Request Event Fields `[](https://docs.crawl4ai.com/advanced/network-console-capture/#__codelineno-2-1) { [](https://docs.crawl4ai.com/advanced/network-console-capture/#__codelineno-2-2) "event_type": "request", [](https://docs.crawl4ai.com/advanced/network-console-capture/#__codelineno-2-3) "url": "https://example.com/api/data.json", [](https://docs.crawl4ai.com/advanced/network-console-capture/#__codelineno-2-4) "method": "GET", [](https://docs.crawl4ai.com/advanced/network-console-capture/#__codelineno-2-5) "headers": {"User-Agent": "...", "Accept": "..."}, [](https://docs.crawl4ai.com/advanced/network-console-capture/#__codelineno-2-6) "post_data": "key=value&otherkey=value", [](https://docs.crawl4ai.com/advanced/network-console-capture/#__codelineno-2-7) "resource_type": "fetch", [](https://docs.crawl4ai.com/advanced/network-console-capture/#__codelineno-2-8) "is_navigation_request": false, [](https://docs.crawl4ai.com/advanced/network-console-capture/#__codelineno-2-9) "timestamp": 1633456789.123 [](https://docs.crawl4ai.com/advanced/network-console-capture/#__codelineno-2-10) }` #### Response Event Fields `[](https://docs.crawl4ai.com/advanced/network-console-capture/#__codelineno-3-1) { [](https://docs.crawl4ai.com/advanced/network-console-capture/#__codelineno-3-2) "event_type": "response", [](https://docs.crawl4ai.com/advanced/network-console-capture/#__codelineno-3-3) "url": "https://example.com/api/data.json", [](https://docs.crawl4ai.com/advanced/network-console-capture/#__codelineno-3-4) "status": 200, [](https://docs.crawl4ai.com/advanced/network-console-capture/#__codelineno-3-5) "status_text": "OK", [](https://docs.crawl4ai.com/advanced/network-console-capture/#__codelineno-3-6) "headers": {"Content-Type": "application/json", "Cache-Control": "..."}, [](https://docs.crawl4ai.com/advanced/network-console-capture/#__codelineno-3-7) "from_service_worker": false, [](https://docs.crawl4ai.com/advanced/network-console-capture/#__codelineno-3-8) "request_timing": {"requestTime": 1234.56, "receiveHeadersEnd": 1234.78}, [](https://docs.crawl4ai.com/advanced/network-console-capture/#__codelineno-3-9) "timestamp": 1633456789.456 [](https://docs.crawl4ai.com/advanced/network-console-capture/#__codelineno-3-10) }` #### Failed Request Event Fields `[](https://docs.crawl4ai.com/advanced/network-console-capture/#__codelineno-4-1) { [](https://docs.crawl4ai.com/advanced/network-console-capture/#__codelineno-4-2) "event_type": "request_failed", [](https://docs.crawl4ai.com/advanced/network-console-capture/#__codelineno-4-3) "url": "https://example.com/missing.png", [](https://docs.crawl4ai.com/advanced/network-console-capture/#__codelineno-4-4) "method": "GET", [](https://docs.crawl4ai.com/advanced/network-console-capture/#__codelineno-4-5) "resource_type": "image", [](https://docs.crawl4ai.com/advanced/network-console-capture/#__codelineno-4-6) "failure_text": "net::ERR_ABORTED 404", [](https://docs.crawl4ai.com/advanced/network-console-capture/#__codelineno-4-7) "timestamp": 1633456789.789 [](https://docs.crawl4ai.com/advanced/network-console-capture/#__codelineno-4-8) }` ### Console Messages The `result.console_messages` contains a list of dictionaries, each representing a console message with these common fields: | Field | Description | | --- | --- | | `type` | Message type: `"log"`, `"error"`, `"warning"`, `"info"`, etc. | | `text` | The message text | | `timestamp` | Unix timestamp when the message was captured | #### Console Message Example `[](https://docs.crawl4ai.com/advanced/network-console-capture/#__codelineno-5-1) { [](https://docs.crawl4ai.com/advanced/network-console-capture/#__codelineno-5-2) "type": "error", [](https://docs.crawl4ai.com/advanced/network-console-capture/#__codelineno-5-3) "text": "Uncaught TypeError: Cannot read property 'length' of undefined", [](https://docs.crawl4ai.com/advanced/network-console-capture/#__codelineno-5-4) "location": "https://example.com/script.js:123:45", [](https://docs.crawl4ai.com/advanced/network-console-capture/#__codelineno-5-5) "timestamp": 1633456790.123 [](https://docs.crawl4ai.com/advanced/network-console-capture/#__codelineno-5-6) }` Key Benefits ------------ * **Full Request Visibility**: Capture all network activity including: * Requests (URLs, methods, headers, post data) * Responses (status codes, headers, timing) * Failed requests (with error messages) * **Console Message Access**: View all JavaScript console output: * Log messages * Warnings * Errors with stack traces * Developer debugging information * **Debugging Power**: Identify issues such as: * Failed API calls or resource loading * JavaScript errors affecting page functionality * CORS or other security issues * Hidden API endpoints and data flows * **Security Analysis**: Detect: * Unexpected third-party requests * Data leakage in request payloads * Suspicious script behavior * **Performance Insights**: Analyze: * Request timing data * Resource loading patterns * Potential bottlenecks Use Cases --------- 1. **API Discovery**: Identify hidden endpoints and data flows in single-page applications 2. **Debugging**: Track down JavaScript errors affecting page functionality 3. **Security Auditing**: Detect unwanted third-party requests or data leakage 4. **Performance Analysis**: Identify slow-loading resources 5. **Ad/Tracker Analysis**: Detect and catalog advertising or tracking calls This capability is especially valuable for complex sites with heavy JavaScript, single-page applications, or when you need to understand the exact communication happening between a browser and servers. * * * \> Feedback --- # PDF Parsing - Crawl4AI Documentation (v0.8.x) PDF Processing Strategies ========================= Crawl4AI provides specialized strategies for handling and extracting content from PDF files. These strategies allow you to seamlessly integrate PDF processing into your crawling workflows, whether the PDFs are hosted online or stored locally. `PDFCrawlerStrategy` -------------------- ### Overview `PDFCrawlerStrategy` is an implementation of `AsyncCrawlerStrategy` designed specifically for PDF documents. Instead of interpreting the input URL as an HTML webpage, this strategy treats it as a pointer to a PDF file. It doesn't perform deep crawling or HTML parsing itself but rather prepares the PDF source for a dedicated PDF scraping strategy. Its primary role is to identify the PDF source (web URL or local file) and pass it along the processing pipeline in a way that `AsyncWebCrawler` can handle. ### When to Use Use `PDFCrawlerStrategy` when you need to: - Process PDF files using the `AsyncWebCrawler`. - Handle PDFs from both web URLs (e.g., `https://example.com/document.pdf`) and local file paths (e.g., `file:///path/to/your/document.pdf`). - Integrate PDF content extraction into a unified `CrawlResult` object, allowing consistent handling of PDF data alongside web page data. ### Key Methods and Their Behavior * **`__init__(self, logger: AsyncLogger = None)`**: * Initializes the strategy. * `logger`: An optional `AsyncLogger` instance (from `crawl4ai.async_logger`) for logging purposes. * **`async crawl(self, url: str, **kwargs) -> AsyncCrawlResponse`**: * This method is called by the `AsyncWebCrawler` during the `arun` process. * It takes the `url` (which should point to a PDF) and creates a minimal `AsyncCrawlResponse`. * The `html` attribute of this response is typically empty or a placeholder, as the actual PDF content processing is deferred to the `PDFContentScrapingStrategy` (or a similar PDF-aware scraping strategy). * It sets `response_headers` to indicate "application/pdf" and `status_code` to 200. * **`async close(self)`**: * A method for cleaning up any resources used by the strategy. For `PDFCrawlerStrategy`, this is usually minimal. * **`async __aenter__(self)` / `async __aexit__(self, exc_type, exc_val, exc_tb)`**: * Enables asynchronous context management for the strategy, allowing it to be used with `async with`. ### Example Usage `[](https://docs.crawl4ai.com/advanced/pdf-parsing/#__codelineno-0-1) import asyncio [](https://docs.crawl4ai.com/advanced/pdf-parsing/#__codelineno-0-2) from crawl4ai import AsyncWebCrawler, CrawlerRunConfig [](https://docs.crawl4ai.com/advanced/pdf-parsing/#__codelineno-0-3) from crawl4ai.processors.pdf import PDFCrawlerStrategy, PDFContentScrapingStrategy [](https://docs.crawl4ai.com/advanced/pdf-parsing/#__codelineno-0-4) [](https://docs.crawl4ai.com/advanced/pdf-parsing/#__codelineno-0-5) async def main(): [](https://docs.crawl4ai.com/advanced/pdf-parsing/#__codelineno-0-6) # Initialize the PDF crawler strategy [](https://docs.crawl4ai.com/advanced/pdf-parsing/#__codelineno-0-7) pdf_crawler_strategy = PDFCrawlerStrategy() [](https://docs.crawl4ai.com/advanced/pdf-parsing/#__codelineno-0-8) [](https://docs.crawl4ai.com/advanced/pdf-parsing/#__codelineno-0-9) # PDFCrawlerStrategy is typically used in conjunction with PDFContentScrapingStrategy [](https://docs.crawl4ai.com/advanced/pdf-parsing/#__codelineno-0-10) # The scraping strategy handles the actual PDF content extraction [](https://docs.crawl4ai.com/advanced/pdf-parsing/#__codelineno-0-11) pdf_scraping_strategy = PDFContentScrapingStrategy() [](https://docs.crawl4ai.com/advanced/pdf-parsing/#__codelineno-0-12) run_config = CrawlerRunConfig(scraping_strategy=pdf_scraping_strategy) [](https://docs.crawl4ai.com/advanced/pdf-parsing/#__codelineno-0-13) [](https://docs.crawl4ai.com/advanced/pdf-parsing/#__codelineno-0-14) async with AsyncWebCrawler(crawler_strategy=pdf_crawler_strategy) as crawler: [](https://docs.crawl4ai.com/advanced/pdf-parsing/#__codelineno-0-15) # Example with a remote PDF URL [](https://docs.crawl4ai.com/advanced/pdf-parsing/#__codelineno-0-16) pdf_url = "https://arxiv.org/pdf/2310.06825.pdf" # A public PDF from arXiv [](https://docs.crawl4ai.com/advanced/pdf-parsing/#__codelineno-0-17) [](https://docs.crawl4ai.com/advanced/pdf-parsing/#__codelineno-0-18) print(f"Attempting to process PDF: {pdf_url}") [](https://docs.crawl4ai.com/advanced/pdf-parsing/#__codelineno-0-19) result = await crawler.arun(url=pdf_url, config=run_config) [](https://docs.crawl4ai.com/advanced/pdf-parsing/#__codelineno-0-20) [](https://docs.crawl4ai.com/advanced/pdf-parsing/#__codelineno-0-21) if result.success: [](https://docs.crawl4ai.com/advanced/pdf-parsing/#__codelineno-0-22) print(f"Successfully processed PDF: {result.url}") [](https://docs.crawl4ai.com/advanced/pdf-parsing/#__codelineno-0-23) print(f"Metadata Title: {result.metadata.get('title', 'N/A')}") [](https://docs.crawl4ai.com/advanced/pdf-parsing/#__codelineno-0-24) # Further processing of result.markdown, result.media, etc. [](https://docs.crawl4ai.com/advanced/pdf-parsing/#__codelineno-0-25) # would be done here, based on what PDFContentScrapingStrategy extracts. [](https://docs.crawl4ai.com/advanced/pdf-parsing/#__codelineno-0-26) if result.markdown and hasattr(result.markdown, 'raw_markdown'): [](https://docs.crawl4ai.com/advanced/pdf-parsing/#__codelineno-0-27) print(f"Extracted text (first 200 chars): {result.markdown.raw_markdown[:200]}...") [](https://docs.crawl4ai.com/advanced/pdf-parsing/#__codelineno-0-28) else: [](https://docs.crawl4ai.com/advanced/pdf-parsing/#__codelineno-0-29) print("No markdown (text) content extracted.") [](https://docs.crawl4ai.com/advanced/pdf-parsing/#__codelineno-0-30) else: [](https://docs.crawl4ai.com/advanced/pdf-parsing/#__codelineno-0-31) print(f"Failed to process PDF: {result.error_message}") [](https://docs.crawl4ai.com/advanced/pdf-parsing/#__codelineno-0-32) [](https://docs.crawl4ai.com/advanced/pdf-parsing/#__codelineno-0-33) if __name__ == "__main__": [](https://docs.crawl4ai.com/advanced/pdf-parsing/#__codelineno-0-34) asyncio.run(main())` ### Pros and Cons **Pros:** - Enables `AsyncWebCrawler` to handle PDF sources directly using familiar `arun` calls. - Provides a consistent interface for specifying PDF sources (URLs or local paths). - Abstracts the source handling, allowing a separate scraping strategy to focus on PDF content parsing. **Cons:** - Does not perform any PDF data extraction itself; it strictly relies on a compatible scraping strategy (like `PDFContentScrapingStrategy`) to process the PDF. - Has limited utility on its own; most of its value comes from being paired with a PDF-specific content scraping strategy. * * * `PDFContentScrapingStrategy` ---------------------------- ### Overview `PDFContentScrapingStrategy` is an implementation of `ContentScrapingStrategy` designed to extract text, metadata, and optionally images from PDF documents. It is intended to be used in conjunction with a crawler strategy that can provide it with a PDF source, such as `PDFCrawlerStrategy`. This strategy uses the `NaivePDFProcessorStrategy` internally to perform the low-level PDF parsing. ### When to Use Use `PDFContentScrapingStrategy` when your `AsyncWebCrawler` (often configured with `PDFCrawlerStrategy`) needs to: - Extract textual content page by page from a PDF document. - Retrieve standard metadata embedded within the PDF (e.g., title, author, subject, creation date, page count). - Optionally, extract images contained within the PDF pages. These images can be saved to a local directory or made available for further processing. - Produce a `ScrapingResult` that can be converted into a `CrawlResult`, making PDF content accessible in a manner similar to HTML web content (e.g., text in `result.markdown`, metadata in `result.metadata`). ### Key Configuration Attributes When initializing `PDFContentScrapingStrategy`, you can configure its behavior using the following attributes: - **`extract_images: bool = False`**: If `True`, the strategy will attempt to extract images from the PDF. - **`save_images_locally: bool = False`**: If `True` (and `extract_images` is also `True`), extracted images will be saved to disk in the `image_save_dir`. If `False`, image data might be available in another form (e.g., base64, depending on the underlying processor) but not saved as separate files by this strategy. - **`image_save_dir: str = None`**: Specifies the directory where extracted images should be saved if `save_images_locally` is `True`. If `None`, a default or temporary directory might be used. - **`batch_size: int = 4`**: Defines how many PDF pages are processed in a single batch. This can be useful for managing memory when dealing with very large PDF documents. - **`logger: AsyncLogger = None`**: An optional `AsyncLogger` instance for logging. ### Key Methods and Their Behavior * **`__init__(self, save_images_locally: bool = False, extract_images: bool = False, image_save_dir: str = None, batch_size: int = 4, logger: AsyncLogger = None)`**: * Initializes the strategy with configurations for image handling, batch processing, and logging. It sets up an internal `NaivePDFProcessorStrategy` instance which performs the actual PDF parsing. * **`scrap(self, url: str, html: str, **params) -> ScrapingResult`**: * This is the primary synchronous method called by the crawler (via `ascrap`) to process the PDF. * `url`: The path or URL to the PDF file (provided by `PDFCrawlerStrategy` or similar). * `html`: Typically an empty string when used with `PDFCrawlerStrategy`, as the content is a PDF, not HTML. * It first ensures the PDF is accessible locally (downloads it to a temporary file if `url` is remote). * It then uses its internal PDF processor to extract text, metadata, and images (if configured). * The extracted information is compiled into a `ScrapingResult` object: * `cleaned_html`: Contains an HTML-like representation of the PDF, where each page's content is often wrapped in a `
` with page number information. * `media`: A dictionary where `media["images"]` will contain information about extracted images if `extract_images` was `True`. * `links`: A dictionary where `links["urls"]` can contain URLs found within the PDF content. * `metadata`: A dictionary holding PDF metadata (e.g., title, author, num\_pages). * **`async ascrap(self, url: str, html: str, **kwargs) -> ScrapingResult`**: * The asynchronous version of `scrap`. Under the hood, it typically runs the synchronous `scrap` method in a separate thread using `asyncio.to_thread` to avoid blocking the event loop. * **`_get_pdf_path(self, url: str) -> str`**: * A private helper method to manage PDF file access. If the `url` is remote (http/https), it downloads the PDF to a temporary local file and returns its path. If `url` indicates a local file (`file://` or a direct path), it resolves and returns the local path. ### Example Usage `[](https://docs.crawl4ai.com/advanced/pdf-parsing/#__codelineno-1-1) import asyncio [](https://docs.crawl4ai.com/advanced/pdf-parsing/#__codelineno-1-2) from crawl4ai import AsyncWebCrawler, CrawlerRunConfig [](https://docs.crawl4ai.com/advanced/pdf-parsing/#__codelineno-1-3) from crawl4ai.processors.pdf import PDFCrawlerStrategy, PDFContentScrapingStrategy [](https://docs.crawl4ai.com/advanced/pdf-parsing/#__codelineno-1-4) import os # For creating image directory [](https://docs.crawl4ai.com/advanced/pdf-parsing/#__codelineno-1-5) [](https://docs.crawl4ai.com/advanced/pdf-parsing/#__codelineno-1-6) async def main(): [](https://docs.crawl4ai.com/advanced/pdf-parsing/#__codelineno-1-7) # Define the directory for saving extracted images [](https://docs.crawl4ai.com/advanced/pdf-parsing/#__codelineno-1-8) image_output_dir = "./my_pdf_images" [](https://docs.crawl4ai.com/advanced/pdf-parsing/#__codelineno-1-9) os.makedirs(image_output_dir, exist_ok=True) [](https://docs.crawl4ai.com/advanced/pdf-parsing/#__codelineno-1-10) [](https://docs.crawl4ai.com/advanced/pdf-parsing/#__codelineno-1-11) # Configure the PDF content scraping strategy [](https://docs.crawl4ai.com/advanced/pdf-parsing/#__codelineno-1-12) # Enable image extraction and specify where to save them [](https://docs.crawl4ai.com/advanced/pdf-parsing/#__codelineno-1-13) pdf_scraping_cfg = PDFContentScrapingStrategy( [](https://docs.crawl4ai.com/advanced/pdf-parsing/#__codelineno-1-14) extract_images=True, [](https://docs.crawl4ai.com/advanced/pdf-parsing/#__codelineno-1-15) save_images_locally=True, [](https://docs.crawl4ai.com/advanced/pdf-parsing/#__codelineno-1-16) image_save_dir=image_output_dir, [](https://docs.crawl4ai.com/advanced/pdf-parsing/#__codelineno-1-17) batch_size=2 # Process 2 pages at a time for demonstration [](https://docs.crawl4ai.com/advanced/pdf-parsing/#__codelineno-1-18) ) [](https://docs.crawl4ai.com/advanced/pdf-parsing/#__codelineno-1-19) [](https://docs.crawl4ai.com/advanced/pdf-parsing/#__codelineno-1-20) # The PDFCrawlerStrategy is needed to tell AsyncWebCrawler how to "crawl" a PDF [](https://docs.crawl4ai.com/advanced/pdf-parsing/#__codelineno-1-21) pdf_crawler_cfg = PDFCrawlerStrategy() [](https://docs.crawl4ai.com/advanced/pdf-parsing/#__codelineno-1-22) [](https://docs.crawl4ai.com/advanced/pdf-parsing/#__codelineno-1-23) # Configure the overall crawl run [](https://docs.crawl4ai.com/advanced/pdf-parsing/#__codelineno-1-24) run_cfg = CrawlerRunConfig( [](https://docs.crawl4ai.com/advanced/pdf-parsing/#__codelineno-1-25) scraping_strategy=pdf_scraping_cfg # Use our PDF scraping strategy [](https://docs.crawl4ai.com/advanced/pdf-parsing/#__codelineno-1-26) ) [](https://docs.crawl4ai.com/advanced/pdf-parsing/#__codelineno-1-27) [](https://docs.crawl4ai.com/advanced/pdf-parsing/#__codelineno-1-28) # Initialize the crawler with the PDF-specific crawler strategy [](https://docs.crawl4ai.com/advanced/pdf-parsing/#__codelineno-1-29) async with AsyncWebCrawler(crawler_strategy=pdf_crawler_cfg) as crawler: [](https://docs.crawl4ai.com/advanced/pdf-parsing/#__codelineno-1-30) pdf_url = "https://arxiv.org/pdf/2310.06825.pdf" # Example PDF [](https://docs.crawl4ai.com/advanced/pdf-parsing/#__codelineno-1-31) [](https://docs.crawl4ai.com/advanced/pdf-parsing/#__codelineno-1-32) print(f"Starting PDF processing for: {pdf_url}") [](https://docs.crawl4ai.com/advanced/pdf-parsing/#__codelineno-1-33) result = await crawler.arun(url=pdf_url, config=run_cfg) [](https://docs.crawl4ai.com/advanced/pdf-parsing/#__codelineno-1-34) [](https://docs.crawl4ai.com/advanced/pdf-parsing/#__codelineno-1-35) if result.success: [](https://docs.crawl4ai.com/advanced/pdf-parsing/#__codelineno-1-36) print("\n--- PDF Processing Successful ---") [](https://docs.crawl4ai.com/advanced/pdf-parsing/#__codelineno-1-37) print(f"Processed URL: {result.url}") [](https://docs.crawl4ai.com/advanced/pdf-parsing/#__codelineno-1-38) [](https://docs.crawl4ai.com/advanced/pdf-parsing/#__codelineno-1-39) print("\n--- Metadata ---") [](https://docs.crawl4ai.com/advanced/pdf-parsing/#__codelineno-1-40) for key, value in result.metadata.items(): [](https://docs.crawl4ai.com/advanced/pdf-parsing/#__codelineno-1-41) print(f" {key.replace('_', ' ').title()}: {value}") [](https://docs.crawl4ai.com/advanced/pdf-parsing/#__codelineno-1-42) [](https://docs.crawl4ai.com/advanced/pdf-parsing/#__codelineno-1-43) if result.markdown and hasattr(result.markdown, 'raw_markdown'): [](https://docs.crawl4ai.com/advanced/pdf-parsing/#__codelineno-1-44) print(f"\n--- Extracted Text (Markdown Snippet) ---") [](https://docs.crawl4ai.com/advanced/pdf-parsing/#__codelineno-1-45) print(result.markdown.raw_markdown[:500].strip() + "...") [](https://docs.crawl4ai.com/advanced/pdf-parsing/#__codelineno-1-46) else: [](https://docs.crawl4ai.com/advanced/pdf-parsing/#__codelineno-1-47) print("\nNo text (markdown) content extracted.") [](https://docs.crawl4ai.com/advanced/pdf-parsing/#__codelineno-1-48) [](https://docs.crawl4ai.com/advanced/pdf-parsing/#__codelineno-1-49) if result.media and result.media.get("images"): [](https://docs.crawl4ai.com/advanced/pdf-parsing/#__codelineno-1-50) print(f"\n--- Image Extraction ---") [](https://docs.crawl4ai.com/advanced/pdf-parsing/#__codelineno-1-51) print(f"Extracted {len(result.media['images'])} image(s).") [](https://docs.crawl4ai.com/advanced/pdf-parsing/#__codelineno-1-52) for i, img_info in enumerate(result.media["images"][:2]): # Show info for first 2 images [](https://docs.crawl4ai.com/advanced/pdf-parsing/#__codelineno-1-53) print(f" Image {i+1}:") [](https://docs.crawl4ai.com/advanced/pdf-parsing/#__codelineno-1-54) print(f" Page: {img_info.get('page')}") [](https://docs.crawl4ai.com/advanced/pdf-parsing/#__codelineno-1-55) print(f" Format: {img_info.get('format', 'N/A')}") [](https://docs.crawl4ai.com/advanced/pdf-parsing/#__codelineno-1-56) if img_info.get('path'): [](https://docs.crawl4ai.com/advanced/pdf-parsing/#__codelineno-1-57) print(f" Saved at: {img_info.get('path')}") [](https://docs.crawl4ai.com/advanced/pdf-parsing/#__codelineno-1-58) else: [](https://docs.crawl4ai.com/advanced/pdf-parsing/#__codelineno-1-59) print("\nNo images were extracted (or extract_images was False).") [](https://docs.crawl4ai.com/advanced/pdf-parsing/#__codelineno-1-60) else: [](https://docs.crawl4ai.com/advanced/pdf-parsing/#__codelineno-1-61) print(f"\n--- PDF Processing Failed ---") [](https://docs.crawl4ai.com/advanced/pdf-parsing/#__codelineno-1-62) print(f"Error: {result.error_message}") [](https://docs.crawl4ai.com/advanced/pdf-parsing/#__codelineno-1-63) [](https://docs.crawl4ai.com/advanced/pdf-parsing/#__codelineno-1-64) if __name__ == "__main__": [](https://docs.crawl4ai.com/advanced/pdf-parsing/#__codelineno-1-65) asyncio.run(main())` ### Pros and Cons **Pros:** - Provides a comprehensive way to extract text, metadata, and (optionally) images from PDF documents. - Handles both remote PDFs (via URL) and local PDF files. - Configurable image extraction allows saving images to disk or accessing their data. - Integrates smoothly with the `CrawlResult` object structure, making PDF-derived data accessible in a way consistent with web-scraped data. - The `batch_size` parameter can help in managing memory consumption when processing large or numerous PDF pages. **Cons:** - Extraction quality and performance can vary significantly depending on the PDF's complexity, encoding, and whether it's image-based (scanned) or text-based. - Image extraction can be resource-intensive (both CPU and disk space if `save_images_locally` is true). - Relies on `NaivePDFProcessorStrategy` internally, which might have limitations with very complex layouts, encrypted PDFs, or forms compared to more sophisticated PDF parsing libraries. Scanned PDFs will not yield text unless an OCR step is performed (which is not part of this strategy by default). - Link extraction from PDFs can be basic and depends on how hyperlinks are embedded in the document. * * * \> Feedback --- # Multi-URL Crawling - Crawl4AI Documentation (v0.8.x) Advanced Multi-URL Crawling with Dispatchers ============================================ > **Heads Up**: Crawl4AI supports advanced dispatchers for **parallel** or **throttled** crawling, providing dynamic rate limiting and memory usage checks. The built-in `arun_many()` function uses these dispatchers to handle concurrency efficiently. 1\. Introduction ---------------- When crawling many URLs: * **Basic**: Use `arun()` in a loop (simple but less efficient) * **Better**: Use `arun_many()`, which efficiently handles multiple URLs with proper concurrency control * **Best**: Customize dispatcher behavior for your specific needs (memory management, rate limits, etc.) **Why Dispatchers?** * **Adaptive**: Memory-based dispatchers can pause or slow down based on system resources * **Rate-limiting**: Built-in rate limiting with exponential backoff for 429/503 responses * **Real-time Monitoring**: Live dashboard of ongoing tasks, memory usage, and performance * **Flexibility**: Choose between memory-adaptive or semaphore-based concurrency * * * 2\. Core Components ------------------- ### 2.1 Rate Limiter `[](https://docs.crawl4ai.com/advanced/multi-url-crawling/#__codelineno-0-1) class RateLimiter: [](https://docs.crawl4ai.com/advanced/multi-url-crawling/#__codelineno-0-2) def __init__( [](https://docs.crawl4ai.com/advanced/multi-url-crawling/#__codelineno-0-3) # Random delay range between requests [](https://docs.crawl4ai.com/advanced/multi-url-crawling/#__codelineno-0-4) base_delay: Tuple[float, float] = (1.0, 3.0), [](https://docs.crawl4ai.com/advanced/multi-url-crawling/#__codelineno-0-5) [](https://docs.crawl4ai.com/advanced/multi-url-crawling/#__codelineno-0-6) # Maximum backoff delay [](https://docs.crawl4ai.com/advanced/multi-url-crawling/#__codelineno-0-7) max_delay: float = 60.0, [](https://docs.crawl4ai.com/advanced/multi-url-crawling/#__codelineno-0-8) [](https://docs.crawl4ai.com/advanced/multi-url-crawling/#__codelineno-0-9) # Retries before giving up [](https://docs.crawl4ai.com/advanced/multi-url-crawling/#__codelineno-0-10) max_retries: int = 3, [](https://docs.crawl4ai.com/advanced/multi-url-crawling/#__codelineno-0-11) [](https://docs.crawl4ai.com/advanced/multi-url-crawling/#__codelineno-0-12) # Status codes triggering backoff [](https://docs.crawl4ai.com/advanced/multi-url-crawling/#__codelineno-0-13) rate_limit_codes: List[int] = [429, 503] [](https://docs.crawl4ai.com/advanced/multi-url-crawling/#__codelineno-0-14) )` Here’s the revised and simplified explanation of the **RateLimiter**, focusing on constructor parameters and adhering to your markdown style and mkDocs guidelines. #### RateLimiter Constructor Parameters The **RateLimiter** is a utility that helps manage the pace of requests to avoid overloading servers or getting blocked due to rate limits. It operates internally to delay requests and handle retries but can be configured using its constructor parameters. **Parameters of the `RateLimiter` constructor:** 1. **`base_delay`** (`Tuple[float, float]`, default: `(1.0, 3.0)`)   The range for a random delay (in seconds) between consecutive requests to the same domain. * A random delay is chosen between `base_delay[0]` and `base_delay[1]` for each request. * This prevents sending requests at a predictable frequency, reducing the chances of triggering rate limits. **Example:** If `base_delay = (2.0, 5.0)`, delays could be randomly chosen as `2.3s`, `4.1s`, etc. * * * 2. **`max_delay`** (`float`, default: `60.0`)   The maximum allowable delay when rate-limiting errors occur. * When servers return rate-limit responses (e.g., 429 or 503), the delay increases exponentially with jitter. * The `max_delay` ensures the delay doesn’t grow unreasonably high, capping it at this value. **Example:** For a `max_delay = 30.0`, even if backoff calculations suggest a delay of `45s`, it will cap at `30s`. * * * 3. **`max_retries`** (`int`, default: `3`)   The maximum number of retries for a request if rate-limiting errors occur. * After encountering a rate-limit response, the `RateLimiter` retries the request up to this number of times. * If all retries fail, the request is marked as failed, and the process continues. **Example:** If `max_retries = 3`, the system retries a failed request three times before giving up. * * * 4. **`rate_limit_codes`** (`List[int]`, default: `[429, 503]`)   A list of HTTP status codes that trigger the rate-limiting logic. * These status codes indicate the server is overwhelmed or actively limiting requests. * You can customize this list to include other codes based on specific server behavior. **Example:** If `rate_limit_codes = [429, 503, 504]`, the crawler will back off on these three error codes. * * * **How to Use the `RateLimiter`:** Here’s an example of initializing and using a `RateLimiter` in your project: `[](https://docs.crawl4ai.com/advanced/multi-url-crawling/#__codelineno-1-1) from crawl4ai import RateLimiter [](https://docs.crawl4ai.com/advanced/multi-url-crawling/#__codelineno-1-2) [](https://docs.crawl4ai.com/advanced/multi-url-crawling/#__codelineno-1-3) # Create a RateLimiter with custom settings [](https://docs.crawl4ai.com/advanced/multi-url-crawling/#__codelineno-1-4) rate_limiter = RateLimiter( [](https://docs.crawl4ai.com/advanced/multi-url-crawling/#__codelineno-1-5) base_delay=(2.0, 4.0), # Random delay between 2-4 seconds [](https://docs.crawl4ai.com/advanced/multi-url-crawling/#__codelineno-1-6) max_delay=30.0, # Cap delay at 30 seconds [](https://docs.crawl4ai.com/advanced/multi-url-crawling/#__codelineno-1-7) max_retries=5, # Retry up to 5 times on rate-limiting errors [](https://docs.crawl4ai.com/advanced/multi-url-crawling/#__codelineno-1-8) rate_limit_codes=[429, 503] # Handle these HTTP status codes [](https://docs.crawl4ai.com/advanced/multi-url-crawling/#__codelineno-1-9) ) [](https://docs.crawl4ai.com/advanced/multi-url-crawling/#__codelineno-1-10) [](https://docs.crawl4ai.com/advanced/multi-url-crawling/#__codelineno-1-11) # RateLimiter will handle delays and retries internally [](https://docs.crawl4ai.com/advanced/multi-url-crawling/#__codelineno-1-12) # No additional setup is required for its operation` The `RateLimiter` integrates seamlessly with dispatchers like `MemoryAdaptiveDispatcher` and `SemaphoreDispatcher`, ensuring requests are paced correctly without user intervention. Its internal mechanisms manage delays and retries to avoid overwhelming servers while maximizing efficiency. ### 2.2 Crawler Monitor The CrawlerMonitor provides real-time visibility into crawling operations: `[](https://docs.crawl4ai.com/advanced/multi-url-crawling/#__codelineno-2-1) from crawl4ai import CrawlerMonitor, DisplayMode [](https://docs.crawl4ai.com/advanced/multi-url-crawling/#__codelineno-2-2) monitor = CrawlerMonitor( [](https://docs.crawl4ai.com/advanced/multi-url-crawling/#__codelineno-2-3) # Maximum rows in live display [](https://docs.crawl4ai.com/advanced/multi-url-crawling/#__codelineno-2-4) max_visible_rows=15, [](https://docs.crawl4ai.com/advanced/multi-url-crawling/#__codelineno-2-5) [](https://docs.crawl4ai.com/advanced/multi-url-crawling/#__codelineno-2-6) # DETAILED or AGGREGATED view [](https://docs.crawl4ai.com/advanced/multi-url-crawling/#__codelineno-2-7) display_mode=DisplayMode.DETAILED [](https://docs.crawl4ai.com/advanced/multi-url-crawling/#__codelineno-2-8) )` **Display Modes**: 1. **DETAILED**: Shows individual task status, memory usage, and timing 2. **AGGREGATED**: Displays summary statistics and overall progress * * * 3\. Available Dispatchers ------------------------- ### 3.1 MemoryAdaptiveDispatcher (Default) Automatically manages concurrency based on system memory usage: `[](https://docs.crawl4ai.com/advanced/multi-url-crawling/#__codelineno-3-1) from crawl4ai.async_dispatcher import MemoryAdaptiveDispatcher [](https://docs.crawl4ai.com/advanced/multi-url-crawling/#__codelineno-3-2) [](https://docs.crawl4ai.com/advanced/multi-url-crawling/#__codelineno-3-3) dispatcher = MemoryAdaptiveDispatcher( [](https://docs.crawl4ai.com/advanced/multi-url-crawling/#__codelineno-3-4) memory_threshold_percent=90.0, # Pause if memory exceeds this [](https://docs.crawl4ai.com/advanced/multi-url-crawling/#__codelineno-3-5) check_interval=1.0, # How often to check memory [](https://docs.crawl4ai.com/advanced/multi-url-crawling/#__codelineno-3-6) max_session_permit=10, # Maximum concurrent tasks [](https://docs.crawl4ai.com/advanced/multi-url-crawling/#__codelineno-3-7) rate_limiter=RateLimiter( # Optional rate limiting [](https://docs.crawl4ai.com/advanced/multi-url-crawling/#__codelineno-3-8) base_delay=(1.0, 2.0), [](https://docs.crawl4ai.com/advanced/multi-url-crawling/#__codelineno-3-9) max_delay=30.0, [](https://docs.crawl4ai.com/advanced/multi-url-crawling/#__codelineno-3-10) max_retries=2 [](https://docs.crawl4ai.com/advanced/multi-url-crawling/#__codelineno-3-11) ), [](https://docs.crawl4ai.com/advanced/multi-url-crawling/#__codelineno-3-12) monitor=CrawlerMonitor( # Optional monitoring [](https://docs.crawl4ai.com/advanced/multi-url-crawling/#__codelineno-3-13) max_visible_rows=15, [](https://docs.crawl4ai.com/advanced/multi-url-crawling/#__codelineno-3-14) display_mode=DisplayMode.DETAILED [](https://docs.crawl4ai.com/advanced/multi-url-crawling/#__codelineno-3-15) ) [](https://docs.crawl4ai.com/advanced/multi-url-crawling/#__codelineno-3-16) )` **Constructor Parameters:** 1. **`memory_threshold_percent`** (`float`, default: `90.0`)   Specifies the memory usage threshold (as a percentage). If system memory usage exceeds this value, the dispatcher pauses crawling to prevent system overload. 2. **`check_interval`** (`float`, default: `1.0`)   The interval (in seconds) at which the dispatcher checks system memory usage. 3. **`max_session_permit`** (`int`, default: `10`)   The maximum number of concurrent crawling tasks allowed. This ensures resource limits are respected while maintaining concurrency. 4. **`memory_wait_timeout`** (`float`, default: `600.0`)   Optional timeout (in seconds). If memory usage exceeds `memory_threshold_percent` for longer than this duration, a `MemoryError` is raised. 5. **`rate_limiter`** (`RateLimiter`, default: `None`)   Optional rate-limiting logic to avoid server-side blocking (e.g., for handling 429 or 503 errors). See **RateLimiter** for details. 6. **`monitor`** (`CrawlerMonitor`, default: `None`)   Optional monitoring for real-time task tracking and performance insights. See **CrawlerMonitor** for details. * * * ### 3.2 SemaphoreDispatcher Provides simple concurrency control with a fixed limit: `[](https://docs.crawl4ai.com/advanced/multi-url-crawling/#__codelineno-4-1) from crawl4ai.async_dispatcher import SemaphoreDispatcher [](https://docs.crawl4ai.com/advanced/multi-url-crawling/#__codelineno-4-2) [](https://docs.crawl4ai.com/advanced/multi-url-crawling/#__codelineno-4-3) dispatcher = SemaphoreDispatcher( [](https://docs.crawl4ai.com/advanced/multi-url-crawling/#__codelineno-4-4) max_session_permit=20, # Maximum concurrent tasks [](https://docs.crawl4ai.com/advanced/multi-url-crawling/#__codelineno-4-5) rate_limiter=RateLimiter( # Optional rate limiting [](https://docs.crawl4ai.com/advanced/multi-url-crawling/#__codelineno-4-6) base_delay=(0.5, 1.0), [](https://docs.crawl4ai.com/advanced/multi-url-crawling/#__codelineno-4-7) max_delay=10.0 [](https://docs.crawl4ai.com/advanced/multi-url-crawling/#__codelineno-4-8) ), [](https://docs.crawl4ai.com/advanced/multi-url-crawling/#__codelineno-4-9) monitor=CrawlerMonitor( # Optional monitoring [](https://docs.crawl4ai.com/advanced/multi-url-crawling/#__codelineno-4-10) max_visible_rows=15, [](https://docs.crawl4ai.com/advanced/multi-url-crawling/#__codelineno-4-11) display_mode=DisplayMode.DETAILED [](https://docs.crawl4ai.com/advanced/multi-url-crawling/#__codelineno-4-12) ) [](https://docs.crawl4ai.com/advanced/multi-url-crawling/#__codelineno-4-13) )` **Constructor Parameters:** 1. **`max_session_permit`** (`int`, default: `20`)   The maximum number of concurrent crawling tasks allowed, irrespective of semaphore slots. 2. **`rate_limiter`** (`RateLimiter`, default: `None`)   Optional rate-limiting logic to avoid overwhelming servers. See **RateLimiter** for details. 3. **`monitor`** (`CrawlerMonitor`, default: `None`)   Optional monitoring for tracking task progress and resource usage. See **CrawlerMonitor** for details. * * * 4\. Usage Examples ------------------ ### 4.1 Batch Processing (Default) `[](https://docs.crawl4ai.com/advanced/multi-url-crawling/#__codelineno-5-1) async def crawl_batch(): [](https://docs.crawl4ai.com/advanced/multi-url-crawling/#__codelineno-5-2) browser_config = BrowserConfig(headless=True, verbose=False) [](https://docs.crawl4ai.com/advanced/multi-url-crawling/#__codelineno-5-3) run_config = CrawlerRunConfig( [](https://docs.crawl4ai.com/advanced/multi-url-crawling/#__codelineno-5-4) cache_mode=CacheMode.BYPASS, [](https://docs.crawl4ai.com/advanced/multi-url-crawling/#__codelineno-5-5) stream=False # Default: get all results at once [](https://docs.crawl4ai.com/advanced/multi-url-crawling/#__codelineno-5-6) ) [](https://docs.crawl4ai.com/advanced/multi-url-crawling/#__codelineno-5-7) [](https://docs.crawl4ai.com/advanced/multi-url-crawling/#__codelineno-5-8) dispatcher = MemoryAdaptiveDispatcher( [](https://docs.crawl4ai.com/advanced/multi-url-crawling/#__codelineno-5-9) memory_threshold_percent=70.0, [](https://docs.crawl4ai.com/advanced/multi-url-crawling/#__codelineno-5-10) check_interval=1.0, [](https://docs.crawl4ai.com/advanced/multi-url-crawling/#__codelineno-5-11) max_session_permit=10, [](https://docs.crawl4ai.com/advanced/multi-url-crawling/#__codelineno-5-12) monitor=CrawlerMonitor( [](https://docs.crawl4ai.com/advanced/multi-url-crawling/#__codelineno-5-13) display_mode=DisplayMode.DETAILED [](https://docs.crawl4ai.com/advanced/multi-url-crawling/#__codelineno-5-14) ) [](https://docs.crawl4ai.com/advanced/multi-url-crawling/#__codelineno-5-15) ) [](https://docs.crawl4ai.com/advanced/multi-url-crawling/#__codelineno-5-16) [](https://docs.crawl4ai.com/advanced/multi-url-crawling/#__codelineno-5-17) async with AsyncWebCrawler(config=browser_config) as crawler: [](https://docs.crawl4ai.com/advanced/multi-url-crawling/#__codelineno-5-18) # Get all results at once [](https://docs.crawl4ai.com/advanced/multi-url-crawling/#__codelineno-5-19) results = await crawler.arun_many( [](https://docs.crawl4ai.com/advanced/multi-url-crawling/#__codelineno-5-20) urls=urls, [](https://docs.crawl4ai.com/advanced/multi-url-crawling/#__codelineno-5-21) config=run_config, [](https://docs.crawl4ai.com/advanced/multi-url-crawling/#__codelineno-5-22) dispatcher=dispatcher [](https://docs.crawl4ai.com/advanced/multi-url-crawling/#__codelineno-5-23) ) [](https://docs.crawl4ai.com/advanced/multi-url-crawling/#__codelineno-5-24) [](https://docs.crawl4ai.com/advanced/multi-url-crawling/#__codelineno-5-25) # Process all results after completion [](https://docs.crawl4ai.com/advanced/multi-url-crawling/#__codelineno-5-26) for result in results: [](https://docs.crawl4ai.com/advanced/multi-url-crawling/#__codelineno-5-27) if result.success: [](https://docs.crawl4ai.com/advanced/multi-url-crawling/#__codelineno-5-28) await process_result(result) [](https://docs.crawl4ai.com/advanced/multi-url-crawling/#__codelineno-5-29) else: [](https://docs.crawl4ai.com/advanced/multi-url-crawling/#__codelineno-5-30) print(f"Failed to crawl {result.url}: {result.error_message}")` **Review:** \- **Purpose:** Executes a batch crawl with all URLs processed together after crawling is complete. \- **Dispatcher:** Uses `MemoryAdaptiveDispatcher` to manage concurrency and system memory. \- **Stream:** Disabled (`stream=False`), so all results are collected at once for post-processing. \- **Best Use Case:** When you need to analyze results in bulk rather than individually during the crawl. * * * ### 4.2 Streaming Mode `[](https://docs.crawl4ai.com/advanced/multi-url-crawling/#__codelineno-6-1) async def crawl_streaming(): [](https://docs.crawl4ai.com/advanced/multi-url-crawling/#__codelineno-6-2) browser_config = BrowserConfig(headless=True, verbose=False) [](https://docs.crawl4ai.com/advanced/multi-url-crawling/#__codelineno-6-3) run_config = CrawlerRunConfig( [](https://docs.crawl4ai.com/advanced/multi-url-crawling/#__codelineno-6-4) cache_mode=CacheMode.BYPASS, [](https://docs.crawl4ai.com/advanced/multi-url-crawling/#__codelineno-6-5) stream=True # Enable streaming mode [](https://docs.crawl4ai.com/advanced/multi-url-crawling/#__codelineno-6-6) ) [](https://docs.crawl4ai.com/advanced/multi-url-crawling/#__codelineno-6-7) [](https://docs.crawl4ai.com/advanced/multi-url-crawling/#__codelineno-6-8) dispatcher = MemoryAdaptiveDispatcher( [](https://docs.crawl4ai.com/advanced/multi-url-crawling/#__codelineno-6-9) memory_threshold_percent=70.0, [](https://docs.crawl4ai.com/advanced/multi-url-crawling/#__codelineno-6-10) check_interval=1.0, [](https://docs.crawl4ai.com/advanced/multi-url-crawling/#__codelineno-6-11) max_session_permit=10, [](https://docs.crawl4ai.com/advanced/multi-url-crawling/#__codelineno-6-12) monitor=CrawlerMonitor( [](https://docs.crawl4ai.com/advanced/multi-url-crawling/#__codelineno-6-13) display_mode=DisplayMode.DETAILED [](https://docs.crawl4ai.com/advanced/multi-url-crawling/#__codelineno-6-14) ) [](https://docs.crawl4ai.com/advanced/multi-url-crawling/#__codelineno-6-15) ) [](https://docs.crawl4ai.com/advanced/multi-url-crawling/#__codelineno-6-16) [](https://docs.crawl4ai.com/advanced/multi-url-crawling/#__codelineno-6-17) async with AsyncWebCrawler(config=browser_config) as crawler: [](https://docs.crawl4ai.com/advanced/multi-url-crawling/#__codelineno-6-18) # Process results as they become available [](https://docs.crawl4ai.com/advanced/multi-url-crawling/#__codelineno-6-19) async for result in await crawler.arun_many( [](https://docs.crawl4ai.com/advanced/multi-url-crawling/#__codelineno-6-20) urls=urls, [](https://docs.crawl4ai.com/advanced/multi-url-crawling/#__codelineno-6-21) config=run_config, [](https://docs.crawl4ai.com/advanced/multi-url-crawling/#__codelineno-6-22) dispatcher=dispatcher [](https://docs.crawl4ai.com/advanced/multi-url-crawling/#__codelineno-6-23) ): [](https://docs.crawl4ai.com/advanced/multi-url-crawling/#__codelineno-6-24) if result.success: [](https://docs.crawl4ai.com/advanced/multi-url-crawling/#__codelineno-6-25) # Process each result immediately [](https://docs.crawl4ai.com/advanced/multi-url-crawling/#__codelineno-6-26) await process_result(result) [](https://docs.crawl4ai.com/advanced/multi-url-crawling/#__codelineno-6-27) else: [](https://docs.crawl4ai.com/advanced/multi-url-crawling/#__codelineno-6-28) print(f"Failed to crawl {result.url}: {result.error_message}")` **Review:** \- **Purpose:** Enables streaming to process results as soon as they’re available. \- **Dispatcher:** Uses `MemoryAdaptiveDispatcher` for concurrency and memory management. \- **Stream:** Enabled (`stream=True`), allowing real-time processing during crawling. \- **Best Use Case:** When you need to act on results immediately, such as for real-time analytics or progressive data storage. * * * ### 4.3 Semaphore-based Crawling `[](https://docs.crawl4ai.com/advanced/multi-url-crawling/#__codelineno-7-1) async def crawl_with_semaphore(urls): [](https://docs.crawl4ai.com/advanced/multi-url-crawling/#__codelineno-7-2) browser_config = BrowserConfig(headless=True, verbose=False) [](https://docs.crawl4ai.com/advanced/multi-url-crawling/#__codelineno-7-3) run_config = CrawlerRunConfig(cache_mode=CacheMode.BYPASS) [](https://docs.crawl4ai.com/advanced/multi-url-crawling/#__codelineno-7-4) [](https://docs.crawl4ai.com/advanced/multi-url-crawling/#__codelineno-7-5) dispatcher = SemaphoreDispatcher( [](https://docs.crawl4ai.com/advanced/multi-url-crawling/#__codelineno-7-6) semaphore_count=5, [](https://docs.crawl4ai.com/advanced/multi-url-crawling/#__codelineno-7-7) rate_limiter=RateLimiter( [](https://docs.crawl4ai.com/advanced/multi-url-crawling/#__codelineno-7-8) base_delay=(0.5, 1.0), [](https://docs.crawl4ai.com/advanced/multi-url-crawling/#__codelineno-7-9) max_delay=10.0 [](https://docs.crawl4ai.com/advanced/multi-url-crawling/#__codelineno-7-10) ), [](https://docs.crawl4ai.com/advanced/multi-url-crawling/#__codelineno-7-11) monitor=CrawlerMonitor( [](https://docs.crawl4ai.com/advanced/multi-url-crawling/#__codelineno-7-12) max_visible_rows=15, [](https://docs.crawl4ai.com/advanced/multi-url-crawling/#__codelineno-7-13) display_mode=DisplayMode.DETAILED [](https://docs.crawl4ai.com/advanced/multi-url-crawling/#__codelineno-7-14) ) [](https://docs.crawl4ai.com/advanced/multi-url-crawling/#__codelineno-7-15) ) [](https://docs.crawl4ai.com/advanced/multi-url-crawling/#__codelineno-7-16) [](https://docs.crawl4ai.com/advanced/multi-url-crawling/#__codelineno-7-17) async with AsyncWebCrawler(config=browser_config) as crawler: [](https://docs.crawl4ai.com/advanced/multi-url-crawling/#__codelineno-7-18) results = await crawler.arun_many( [](https://docs.crawl4ai.com/advanced/multi-url-crawling/#__codelineno-7-19) urls, [](https://docs.crawl4ai.com/advanced/multi-url-crawling/#__codelineno-7-20) config=run_config, [](https://docs.crawl4ai.com/advanced/multi-url-crawling/#__codelineno-7-21) dispatcher=dispatcher [](https://docs.crawl4ai.com/advanced/multi-url-crawling/#__codelineno-7-22) ) [](https://docs.crawl4ai.com/advanced/multi-url-crawling/#__codelineno-7-23) return results` **Review:** \- **Purpose:** Uses `SemaphoreDispatcher` to limit concurrency with a fixed number of slots. \- **Dispatcher:** Configured with a semaphore to control parallel crawling tasks. \- **Rate Limiter:** Prevents servers from being overwhelmed by pacing requests. \- **Best Use Case:** When you want precise control over the number of concurrent requests, independent of system memory. * * * ### 4.4 Robots.txt Consideration `[](https://docs.crawl4ai.com/advanced/multi-url-crawling/#__codelineno-8-1) import asyncio [](https://docs.crawl4ai.com/advanced/multi-url-crawling/#__codelineno-8-2) from crawl4ai import AsyncWebCrawler, CrawlerRunConfig, CacheMode [](https://docs.crawl4ai.com/advanced/multi-url-crawling/#__codelineno-8-3) [](https://docs.crawl4ai.com/advanced/multi-url-crawling/#__codelineno-8-4) async def main(): [](https://docs.crawl4ai.com/advanced/multi-url-crawling/#__codelineno-8-5) urls = [ [](https://docs.crawl4ai.com/advanced/multi-url-crawling/#__codelineno-8-6) "https://example1.com", [](https://docs.crawl4ai.com/advanced/multi-url-crawling/#__codelineno-8-7) "https://example2.com", [](https://docs.crawl4ai.com/advanced/multi-url-crawling/#__codelineno-8-8) "https://example3.com" [](https://docs.crawl4ai.com/advanced/multi-url-crawling/#__codelineno-8-9) ] [](https://docs.crawl4ai.com/advanced/multi-url-crawling/#__codelineno-8-10) [](https://docs.crawl4ai.com/advanced/multi-url-crawling/#__codelineno-8-11) config = CrawlerRunConfig( [](https://docs.crawl4ai.com/advanced/multi-url-crawling/#__codelineno-8-12) cache_mode=CacheMode.ENABLED, [](https://docs.crawl4ai.com/advanced/multi-url-crawling/#__codelineno-8-13) check_robots_txt=True, # Will respect robots.txt for each URL [](https://docs.crawl4ai.com/advanced/multi-url-crawling/#__codelineno-8-14) semaphore_count=3 # Max concurrent requests [](https://docs.crawl4ai.com/advanced/multi-url-crawling/#__codelineno-8-15) ) [](https://docs.crawl4ai.com/advanced/multi-url-crawling/#__codelineno-8-16) [](https://docs.crawl4ai.com/advanced/multi-url-crawling/#__codelineno-8-17) async with AsyncWebCrawler() as crawler: [](https://docs.crawl4ai.com/advanced/multi-url-crawling/#__codelineno-8-18) async for result in crawler.arun_many(urls, config=config): [](https://docs.crawl4ai.com/advanced/multi-url-crawling/#__codelineno-8-19) if result.success: [](https://docs.crawl4ai.com/advanced/multi-url-crawling/#__codelineno-8-20) print(f"Successfully crawled {result.url}") [](https://docs.crawl4ai.com/advanced/multi-url-crawling/#__codelineno-8-21) elif result.status_code == 403 and "robots.txt" in result.error_message: [](https://docs.crawl4ai.com/advanced/multi-url-crawling/#__codelineno-8-22) print(f"Skipped {result.url} - blocked by robots.txt") [](https://docs.crawl4ai.com/advanced/multi-url-crawling/#__codelineno-8-23) else: [](https://docs.crawl4ai.com/advanced/multi-url-crawling/#__codelineno-8-24) print(f"Failed to crawl {result.url}: {result.error_message}") [](https://docs.crawl4ai.com/advanced/multi-url-crawling/#__codelineno-8-25) [](https://docs.crawl4ai.com/advanced/multi-url-crawling/#__codelineno-8-26) if __name__ == "__main__": [](https://docs.crawl4ai.com/advanced/multi-url-crawling/#__codelineno-8-27) asyncio.run(main())` **Review:** \- **Purpose:** Ensures compliance with `robots.txt` rules for ethical and legal web crawling. \- **Configuration:** Set `check_robots_txt=True` to validate each URL against `robots.txt` before crawling. \- **Dispatcher:** Handles requests with concurrency limits (`semaphore_count=3`). \- **Best Use Case:** When crawling websites that strictly enforce robots.txt policies or for responsible crawling practices. * * * 5\. Dispatch Results -------------------- Each crawl result includes dispatch information: `[](https://docs.crawl4ai.com/advanced/multi-url-crawling/#__codelineno-9-1) @dataclass [](https://docs.crawl4ai.com/advanced/multi-url-crawling/#__codelineno-9-2) class DispatchResult: [](https://docs.crawl4ai.com/advanced/multi-url-crawling/#__codelineno-9-3) task_id: str [](https://docs.crawl4ai.com/advanced/multi-url-crawling/#__codelineno-9-4) memory_usage: float [](https://docs.crawl4ai.com/advanced/multi-url-crawling/#__codelineno-9-5) peak_memory: float [](https://docs.crawl4ai.com/advanced/multi-url-crawling/#__codelineno-9-6) start_time: datetime [](https://docs.crawl4ai.com/advanced/multi-url-crawling/#__codelineno-9-7) end_time: datetime [](https://docs.crawl4ai.com/advanced/multi-url-crawling/#__codelineno-9-8) error_message: str = ""` Access via `result.dispatch_result`: `[](https://docs.crawl4ai.com/advanced/multi-url-crawling/#__codelineno-10-1) for result in results: [](https://docs.crawl4ai.com/advanced/multi-url-crawling/#__codelineno-10-2) if result.success: [](https://docs.crawl4ai.com/advanced/multi-url-crawling/#__codelineno-10-3) dr = result.dispatch_result [](https://docs.crawl4ai.com/advanced/multi-url-crawling/#__codelineno-10-4) print(f"URL: {result.url}") [](https://docs.crawl4ai.com/advanced/multi-url-crawling/#__codelineno-10-5) print(f"Memory: {dr.memory_usage:.1f}MB") [](https://docs.crawl4ai.com/advanced/multi-url-crawling/#__codelineno-10-6) print(f"Duration: {dr.end_time - dr.start_time}")` 6\. URL-Specific Configurations ------------------------------- When crawling diverse content types, you often need different configurations for different URLs. For example: - PDFs need specialized extraction - Blog pages benefit from content filtering - Dynamic sites need JavaScript execution - API endpoints need JSON parsing ### 6.1 Basic URL Pattern Matching `[](https://docs.crawl4ai.com/advanced/multi-url-crawling/#__codelineno-11-1) from crawl4ai import AsyncWebCrawler, CrawlerRunConfig, MatchMode [](https://docs.crawl4ai.com/advanced/multi-url-crawling/#__codelineno-11-2) from crawl4ai.processors.pdf import PDFContentScrapingStrategy [](https://docs.crawl4ai.com/advanced/multi-url-crawling/#__codelineno-11-3) from crawl4ai.extraction_strategy import JsonCssExtractionStrategy [](https://docs.crawl4ai.com/advanced/multi-url-crawling/#__codelineno-11-4) from crawl4ai.content_filter_strategy import PruningContentFilter [](https://docs.crawl4ai.com/advanced/multi-url-crawling/#__codelineno-11-5) from crawl4ai.markdown_generation_strategy import DefaultMarkdownGenerator [](https://docs.crawl4ai.com/advanced/multi-url-crawling/#__codelineno-11-6) [](https://docs.crawl4ai.com/advanced/multi-url-crawling/#__codelineno-11-7) async def crawl_mixed_content(): [](https://docs.crawl4ai.com/advanced/multi-url-crawling/#__codelineno-11-8) # Configure different strategies for different content [](https://docs.crawl4ai.com/advanced/multi-url-crawling/#__codelineno-11-9) configs = [ [](https://docs.crawl4ai.com/advanced/multi-url-crawling/#__codelineno-11-10) # PDF files - specialized extraction [](https://docs.crawl4ai.com/advanced/multi-url-crawling/#__codelineno-11-11) CrawlerRunConfig( [](https://docs.crawl4ai.com/advanced/multi-url-crawling/#__codelineno-11-12) url_matcher="*.pdf", [](https://docs.crawl4ai.com/advanced/multi-url-crawling/#__codelineno-11-13) scraping_strategy=PDFContentScrapingStrategy() [](https://docs.crawl4ai.com/advanced/multi-url-crawling/#__codelineno-11-14) ), [](https://docs.crawl4ai.com/advanced/multi-url-crawling/#__codelineno-11-15) [](https://docs.crawl4ai.com/advanced/multi-url-crawling/#__codelineno-11-16) # Blog/article pages - content filtering [](https://docs.crawl4ai.com/advanced/multi-url-crawling/#__codelineno-11-17) CrawlerRunConfig( [](https://docs.crawl4ai.com/advanced/multi-url-crawling/#__codelineno-11-18) url_matcher=["*/blog/*", "*/article/*"], [](https://docs.crawl4ai.com/advanced/multi-url-crawling/#__codelineno-11-19) markdown_generator=DefaultMarkdownGenerator( [](https://docs.crawl4ai.com/advanced/multi-url-crawling/#__codelineno-11-20) content_filter=PruningContentFilter(threshold=0.48) [](https://docs.crawl4ai.com/advanced/multi-url-crawling/#__codelineno-11-21) ) [](https://docs.crawl4ai.com/advanced/multi-url-crawling/#__codelineno-11-22) ), [](https://docs.crawl4ai.com/advanced/multi-url-crawling/#__codelineno-11-23) [](https://docs.crawl4ai.com/advanced/multi-url-crawling/#__codelineno-11-24) # Dynamic pages - JavaScript execution [](https://docs.crawl4ai.com/advanced/multi-url-crawling/#__codelineno-11-25) CrawlerRunConfig( [](https://docs.crawl4ai.com/advanced/multi-url-crawling/#__codelineno-11-26) url_matcher=lambda url: 'github.com' in url, [](https://docs.crawl4ai.com/advanced/multi-url-crawling/#__codelineno-11-27) js_code="window.scrollTo(0, 500);" [](https://docs.crawl4ai.com/advanced/multi-url-crawling/#__codelineno-11-28) ), [](https://docs.crawl4ai.com/advanced/multi-url-crawling/#__codelineno-11-29) [](https://docs.crawl4ai.com/advanced/multi-url-crawling/#__codelineno-11-30) # API endpoints - JSON extraction [](https://docs.crawl4ai.com/advanced/multi-url-crawling/#__codelineno-11-31) CrawlerRunConfig( [](https://docs.crawl4ai.com/advanced/multi-url-crawling/#__codelineno-11-32) url_matcher=lambda url: 'api' in url or url.endswith('.json'), [](https://docs.crawl4ai.com/advanced/multi-url-crawling/#__codelineno-11-33) # Custome settings for JSON extraction [](https://docs.crawl4ai.com/advanced/multi-url-crawling/#__codelineno-11-34) ), [](https://docs.crawl4ai.com/advanced/multi-url-crawling/#__codelineno-11-35) [](https://docs.crawl4ai.com/advanced/multi-url-crawling/#__codelineno-11-36) # Default config for everything else [](https://docs.crawl4ai.com/advanced/multi-url-crawling/#__codelineno-11-37) CrawlerRunConfig() # No url_matcher means it matches ALL URLs (fallback) [](https://docs.crawl4ai.com/advanced/multi-url-crawling/#__codelineno-11-38) ] [](https://docs.crawl4ai.com/advanced/multi-url-crawling/#__codelineno-11-39) [](https://docs.crawl4ai.com/advanced/multi-url-crawling/#__codelineno-11-40) # Mixed URLs [](https://docs.crawl4ai.com/advanced/multi-url-crawling/#__codelineno-11-41) urls = [ [](https://docs.crawl4ai.com/advanced/multi-url-crawling/#__codelineno-11-42) "https://www.w3.org/WAI/ER/tests/xhtml/testfiles/resources/pdf/dummy.pdf", [](https://docs.crawl4ai.com/advanced/multi-url-crawling/#__codelineno-11-43) "https://blog.python.org/", [](https://docs.crawl4ai.com/advanced/multi-url-crawling/#__codelineno-11-44) "https://github.com/microsoft/playwright", [](https://docs.crawl4ai.com/advanced/multi-url-crawling/#__codelineno-11-45) "https://httpbin.org/json", [](https://docs.crawl4ai.com/advanced/multi-url-crawling/#__codelineno-11-46) "https://example.com/" [](https://docs.crawl4ai.com/advanced/multi-url-crawling/#__codelineno-11-47) ] [](https://docs.crawl4ai.com/advanced/multi-url-crawling/#__codelineno-11-48) [](https://docs.crawl4ai.com/advanced/multi-url-crawling/#__codelineno-11-49) async with AsyncWebCrawler() as crawler: [](https://docs.crawl4ai.com/advanced/multi-url-crawling/#__codelineno-11-50) results = await crawler.arun_many( [](https://docs.crawl4ai.com/advanced/multi-url-crawling/#__codelineno-11-51) urls=urls, [](https://docs.crawl4ai.com/advanced/multi-url-crawling/#__codelineno-11-52) config=configs # Pass list of configs [](https://docs.crawl4ai.com/advanced/multi-url-crawling/#__codelineno-11-53) ) [](https://docs.crawl4ai.com/advanced/multi-url-crawling/#__codelineno-11-54) [](https://docs.crawl4ai.com/advanced/multi-url-crawling/#__codelineno-11-55) for result in results: [](https://docs.crawl4ai.com/advanced/multi-url-crawling/#__codelineno-11-56) print(f"{result.url}: {len(result.markdown)} chars")` ### 6.2 Advanced Pattern Matching **Important**: A `CrawlerRunConfig` without `url_matcher` (or with `url_matcher=None`) matches ALL URLs. This makes it perfect as a default/fallback configuration. The `url_matcher` parameter supports three types of patterns: #### Glob Patterns (Strings) `[](https://docs.crawl4ai.com/advanced/multi-url-crawling/#__codelineno-12-1) # Simple patterns [](https://docs.crawl4ai.com/advanced/multi-url-crawling/#__codelineno-12-2) "*.pdf" # Any PDF file [](https://docs.crawl4ai.com/advanced/multi-url-crawling/#__codelineno-12-3) "*/api/*" # Any URL with /api/ in path [](https://docs.crawl4ai.com/advanced/multi-url-crawling/#__codelineno-12-4) "https://*.example.com/*" # Subdomain matching [](https://docs.crawl4ai.com/advanced/multi-url-crawling/#__codelineno-12-5) "*://example.com/blog/*" # Any protocol` #### Custom Functions `[](https://docs.crawl4ai.com/advanced/multi-url-crawling/#__codelineno-13-1) # Complex logic with lambdas [](https://docs.crawl4ai.com/advanced/multi-url-crawling/#__codelineno-13-2) lambda url: url.startswith('https://') and 'secure' in url [](https://docs.crawl4ai.com/advanced/multi-url-crawling/#__codelineno-13-3) lambda url: len(url) > 50 and url.count('/') > 5 [](https://docs.crawl4ai.com/advanced/multi-url-crawling/#__codelineno-13-4) lambda url: any(domain in url for domain in ['api.', 'data.', 'feed.'])` #### Mixed Lists with AND/OR Logic `[](https://docs.crawl4ai.com/advanced/multi-url-crawling/#__codelineno-14-1) # Combine multiple conditions [](https://docs.crawl4ai.com/advanced/multi-url-crawling/#__codelineno-14-2) CrawlerRunConfig( [](https://docs.crawl4ai.com/advanced/multi-url-crawling/#__codelineno-14-3) url_matcher=[ [](https://docs.crawl4ai.com/advanced/multi-url-crawling/#__codelineno-14-4) "https://*", # Must be HTTPS [](https://docs.crawl4ai.com/advanced/multi-url-crawling/#__codelineno-14-5) lambda url: 'internal' in url, # Must contain 'internal' [](https://docs.crawl4ai.com/advanced/multi-url-crawling/#__codelineno-14-6) lambda url: not url.endswith('.pdf') # Must not be PDF [](https://docs.crawl4ai.com/advanced/multi-url-crawling/#__codelineno-14-7) ], [](https://docs.crawl4ai.com/advanced/multi-url-crawling/#__codelineno-14-8) match_mode=MatchMode.AND # ALL conditions must match [](https://docs.crawl4ai.com/advanced/multi-url-crawling/#__codelineno-14-9) )` ### 6.3 Practical Example: News Site Crawler `[](https://docs.crawl4ai.com/advanced/multi-url-crawling/#__codelineno-15-1) async def crawl_news_site(): [](https://docs.crawl4ai.com/advanced/multi-url-crawling/#__codelineno-15-2) dispatcher = MemoryAdaptiveDispatcher( [](https://docs.crawl4ai.com/advanced/multi-url-crawling/#__codelineno-15-3) memory_threshold_percent=70.0, [](https://docs.crawl4ai.com/advanced/multi-url-crawling/#__codelineno-15-4) rate_limiter=RateLimiter(base_delay=(1.0, 2.0)) [](https://docs.crawl4ai.com/advanced/multi-url-crawling/#__codelineno-15-5) ) [](https://docs.crawl4ai.com/advanced/multi-url-crawling/#__codelineno-15-6) [](https://docs.crawl4ai.com/advanced/multi-url-crawling/#__codelineno-15-7) configs = [ [](https://docs.crawl4ai.com/advanced/multi-url-crawling/#__codelineno-15-8) # Homepage - light extraction [](https://docs.crawl4ai.com/advanced/multi-url-crawling/#__codelineno-15-9) CrawlerRunConfig( [](https://docs.crawl4ai.com/advanced/multi-url-crawling/#__codelineno-15-10) url_matcher=lambda url: url.rstrip('/') == 'https://news.ycombinator.com', [](https://docs.crawl4ai.com/advanced/multi-url-crawling/#__codelineno-15-11) css_selector="nav, .headline", [](https://docs.crawl4ai.com/advanced/multi-url-crawling/#__codelineno-15-12) extraction_strategy=None [](https://docs.crawl4ai.com/advanced/multi-url-crawling/#__codelineno-15-13) ), [](https://docs.crawl4ai.com/advanced/multi-url-crawling/#__codelineno-15-14) [](https://docs.crawl4ai.com/advanced/multi-url-crawling/#__codelineno-15-15) # Article pages - full extraction [](https://docs.crawl4ai.com/advanced/multi-url-crawling/#__codelineno-15-16) CrawlerRunConfig( [](https://docs.crawl4ai.com/advanced/multi-url-crawling/#__codelineno-15-17) url_matcher="*/article/*", [](https://docs.crawl4ai.com/advanced/multi-url-crawling/#__codelineno-15-18) extraction_strategy=CosineStrategy( [](https://docs.crawl4ai.com/advanced/multi-url-crawling/#__codelineno-15-19) semantic_filter="article content", [](https://docs.crawl4ai.com/advanced/multi-url-crawling/#__codelineno-15-20) word_count_threshold=100 [](https://docs.crawl4ai.com/advanced/multi-url-crawling/#__codelineno-15-21) ), [](https://docs.crawl4ai.com/advanced/multi-url-crawling/#__codelineno-15-22) screenshot=True, [](https://docs.crawl4ai.com/advanced/multi-url-crawling/#__codelineno-15-23) excluded_tags=["nav", "aside", "footer"] [](https://docs.crawl4ai.com/advanced/multi-url-crawling/#__codelineno-15-24) ), [](https://docs.crawl4ai.com/advanced/multi-url-crawling/#__codelineno-15-25) [](https://docs.crawl4ai.com/advanced/multi-url-crawling/#__codelineno-15-26) # Author pages - metadata focus [](https://docs.crawl4ai.com/advanced/multi-url-crawling/#__codelineno-15-27) CrawlerRunConfig( [](https://docs.crawl4ai.com/advanced/multi-url-crawling/#__codelineno-15-28) url_matcher="*/author/*", [](https://docs.crawl4ai.com/advanced/multi-url-crawling/#__codelineno-15-29) extraction_strategy=JsonCssExtractionStrategy({ [](https://docs.crawl4ai.com/advanced/multi-url-crawling/#__codelineno-15-30) "name": "h1.author-name", [](https://docs.crawl4ai.com/advanced/multi-url-crawling/#__codelineno-15-31) "bio": ".author-bio", [](https://docs.crawl4ai.com/advanced/multi-url-crawling/#__codelineno-15-32) "articles": "article.post-card h2" [](https://docs.crawl4ai.com/advanced/multi-url-crawling/#__codelineno-15-33) }) [](https://docs.crawl4ai.com/advanced/multi-url-crawling/#__codelineno-15-34) ), [](https://docs.crawl4ai.com/advanced/multi-url-crawling/#__codelineno-15-35) [](https://docs.crawl4ai.com/advanced/multi-url-crawling/#__codelineno-15-36) # Everything else [](https://docs.crawl4ai.com/advanced/multi-url-crawling/#__codelineno-15-37) CrawlerRunConfig() [](https://docs.crawl4ai.com/advanced/multi-url-crawling/#__codelineno-15-38) ] [](https://docs.crawl4ai.com/advanced/multi-url-crawling/#__codelineno-15-39) [](https://docs.crawl4ai.com/advanced/multi-url-crawling/#__codelineno-15-40) async with AsyncWebCrawler() as crawler: [](https://docs.crawl4ai.com/advanced/multi-url-crawling/#__codelineno-15-41) results = await crawler.arun_many( [](https://docs.crawl4ai.com/advanced/multi-url-crawling/#__codelineno-15-42) urls=news_urls, [](https://docs.crawl4ai.com/advanced/multi-url-crawling/#__codelineno-15-43) config=configs, [](https://docs.crawl4ai.com/advanced/multi-url-crawling/#__codelineno-15-44) dispatcher=dispatcher [](https://docs.crawl4ai.com/advanced/multi-url-crawling/#__codelineno-15-45) )` ### 6.4 Best Practices 1. **Order Matters**: Configs are evaluated in order - put specific patterns before general ones 2. **Default Config Behavior**: 3. A config without `url_matcher` matches ALL URLs 4. Always include a default config as the last item if you want to handle all URLs 5. Without a default config, unmatched URLs will fail with "No matching configuration found" 6. **Test Your Patterns**: Use the config's `is_match()` method to test patterns: `[](https://docs.crawl4ai.com/advanced/multi-url-crawling/#__codelineno-16-1) config = CrawlerRunConfig(url_matcher="*.pdf") [](https://docs.crawl4ai.com/advanced/multi-url-crawling/#__codelineno-16-2) print(config.is_match("https://example.com/doc.pdf")) # True [](https://docs.crawl4ai.com/advanced/multi-url-crawling/#__codelineno-16-3) [](https://docs.crawl4ai.com/advanced/multi-url-crawling/#__codelineno-16-4) default_config = CrawlerRunConfig() # No url_matcher [](https://docs.crawl4ai.com/advanced/multi-url-crawling/#__codelineno-16-5) print(default_config.is_match("https://any-url.com")) # True - matches everything!` 7. **Optimize for Performance**: 8. Disable JS for static content 9. Skip screenshots for data APIs 10. Use appropriate extraction strategies 7\. Summary ----------- 1. **Two Dispatcher Types**: * MemoryAdaptiveDispatcher (default): Dynamic concurrency based on memory * SemaphoreDispatcher: Fixed concurrency limit 2. **Optional Components**: * RateLimiter: Smart request pacing and backoff * CrawlerMonitor: Real-time progress visualization 3. **Key Benefits**: * Automatic memory management * Built-in rate limiting * Live progress monitoring * Flexible concurrency control Choose the dispatcher that best fits your needs: * **MemoryAdaptiveDispatcher**: For large crawls or limited resources * **SemaphoreDispatcher**: For simple, fixed-concurrency scenarios * * * \> Feedback --- # Proxy & Security - Crawl4AI Documentation (v0.8.x) Proxy & Security ================ This guide covers proxy configuration and security features in Crawl4AI, including SSL certificate analysis and proxy rotation strategies. Understanding Proxy Configuration --------------------------------- Crawl4AI recommends configuring proxies per request through `CrawlerRunConfig.proxy_config`. This gives you precise control, enables rotation strategies, and keeps examples simple enough to copy, paste, and run. Basic Proxy Setup ----------------- Configure proxies that apply to each crawl operation: `[](https://docs.crawl4ai.com/advanced/proxy-security/#__codelineno-0-1) import asyncio [](https://docs.crawl4ai.com/advanced/proxy-security/#__codelineno-0-2) from crawl4ai import AsyncWebCrawler, BrowserConfig, CrawlerRunConfig, ProxyConfig [](https://docs.crawl4ai.com/advanced/proxy-security/#__codelineno-0-3) [](https://docs.crawl4ai.com/advanced/proxy-security/#__codelineno-0-4) run_config = CrawlerRunConfig(proxy_config=ProxyConfig(server="http://proxy.example.com:8080")) [](https://docs.crawl4ai.com/advanced/proxy-security/#__codelineno-0-5) # run_config = CrawlerRunConfig(proxy_config={"server": "http://proxy.example.com:8080"}) [](https://docs.crawl4ai.com/advanced/proxy-security/#__codelineno-0-6) # run_config = CrawlerRunConfig(proxy_config="http://proxy.example.com:8080") [](https://docs.crawl4ai.com/advanced/proxy-security/#__codelineno-0-7) [](https://docs.crawl4ai.com/advanced/proxy-security/#__codelineno-0-8) [](https://docs.crawl4ai.com/advanced/proxy-security/#__codelineno-0-9) async def main(): [](https://docs.crawl4ai.com/advanced/proxy-security/#__codelineno-0-10) browser_config = BrowserConfig() [](https://docs.crawl4ai.com/advanced/proxy-security/#__codelineno-0-11) async with AsyncWebCrawler(config=browser_config) as crawler: [](https://docs.crawl4ai.com/advanced/proxy-security/#__codelineno-0-12) result = await crawler.arun(url="https://example.com", config=run_config) [](https://docs.crawl4ai.com/advanced/proxy-security/#__codelineno-0-13) print(f"Success: {result.success} -> {result.url}") [](https://docs.crawl4ai.com/advanced/proxy-security/#__codelineno-0-14) [](https://docs.crawl4ai.com/advanced/proxy-security/#__codelineno-0-15) [](https://docs.crawl4ai.com/advanced/proxy-security/#__codelineno-0-16) if __name__ == "__main__": [](https://docs.crawl4ai.com/advanced/proxy-security/#__codelineno-0-17) asyncio.run(main())` Why request-level? `CrawlerRunConfig.proxy_config` keeps each request self-contained, so swapping proxies or rotation strategies is just a matter of building a new run configuration. Supported Proxy Formats ----------------------- The `ProxyConfig.from_string()` method supports multiple formats: `[](https://docs.crawl4ai.com/advanced/proxy-security/#__codelineno-1-1) from crawl4ai import ProxyConfig [](https://docs.crawl4ai.com/advanced/proxy-security/#__codelineno-1-2) [](https://docs.crawl4ai.com/advanced/proxy-security/#__codelineno-1-3) # HTTP proxy with authentication [](https://docs.crawl4ai.com/advanced/proxy-security/#__codelineno-1-4) proxy1 = ProxyConfig.from_string("http://user:pass@192.168.1.1:8080") [](https://docs.crawl4ai.com/advanced/proxy-security/#__codelineno-1-5) [](https://docs.crawl4ai.com/advanced/proxy-security/#__codelineno-1-6) # HTTPS proxy [](https://docs.crawl4ai.com/advanced/proxy-security/#__codelineno-1-7) proxy2 = ProxyConfig.from_string("https://proxy.example.com:8080") [](https://docs.crawl4ai.com/advanced/proxy-security/#__codelineno-1-8) [](https://docs.crawl4ai.com/advanced/proxy-security/#__codelineno-1-9) # SOCKS5 proxy [](https://docs.crawl4ai.com/advanced/proxy-security/#__codelineno-1-10) proxy3 = ProxyConfig.from_string("socks5://proxy.example.com:1080") [](https://docs.crawl4ai.com/advanced/proxy-security/#__codelineno-1-11) [](https://docs.crawl4ai.com/advanced/proxy-security/#__codelineno-1-12) # Simple IP:port format [](https://docs.crawl4ai.com/advanced/proxy-security/#__codelineno-1-13) proxy4 = ProxyConfig.from_string("192.168.1.1:8080") [](https://docs.crawl4ai.com/advanced/proxy-security/#__codelineno-1-14) [](https://docs.crawl4ai.com/advanced/proxy-security/#__codelineno-1-15) # IP:port:user:pass format [](https://docs.crawl4ai.com/advanced/proxy-security/#__codelineno-1-16) proxy5 = ProxyConfig.from_string("192.168.1.1:8080:user:pass")` Authenticated Proxies --------------------- For proxies requiring authentication: `[](https://docs.crawl4ai.com/advanced/proxy-security/#__codelineno-2-1) import asyncio [](https://docs.crawl4ai.com/advanced/proxy-security/#__codelineno-2-2) from crawl4ai import AsyncWebCrawler,BrowserConfig, CrawlerRunConfig, ProxyConfig [](https://docs.crawl4ai.com/advanced/proxy-security/#__codelineno-2-3) [](https://docs.crawl4ai.com/advanced/proxy-security/#__codelineno-2-4) run_config = CrawlerRunConfig( [](https://docs.crawl4ai.com/advanced/proxy-security/#__codelineno-2-5) proxy_config=ProxyConfig( [](https://docs.crawl4ai.com/advanced/proxy-security/#__codelineno-2-6) server="http://proxy.example.com:8080", [](https://docs.crawl4ai.com/advanced/proxy-security/#__codelineno-2-7) username="your_username", [](https://docs.crawl4ai.com/advanced/proxy-security/#__codelineno-2-8) password="your_password", [](https://docs.crawl4ai.com/advanced/proxy-security/#__codelineno-2-9) ) [](https://docs.crawl4ai.com/advanced/proxy-security/#__codelineno-2-10) ) [](https://docs.crawl4ai.com/advanced/proxy-security/#__codelineno-2-11) # Or dictionary style: [](https://docs.crawl4ai.com/advanced/proxy-security/#__codelineno-2-12) # run_config = CrawlerRunConfig(proxy_config={ [](https://docs.crawl4ai.com/advanced/proxy-security/#__codelineno-2-13) # "server": "http://proxy.example.com:8080", [](https://docs.crawl4ai.com/advanced/proxy-security/#__codelineno-2-14) # "username": "your_username", [](https://docs.crawl4ai.com/advanced/proxy-security/#__codelineno-2-15) # "password": "your_password", [](https://docs.crawl4ai.com/advanced/proxy-security/#__codelineno-2-16) # }) [](https://docs.crawl4ai.com/advanced/proxy-security/#__codelineno-2-17) [](https://docs.crawl4ai.com/advanced/proxy-security/#__codelineno-2-18) [](https://docs.crawl4ai.com/advanced/proxy-security/#__codelineno-2-19) async def main(): [](https://docs.crawl4ai.com/advanced/proxy-security/#__codelineno-2-20) browser_config = BrowserConfig() [](https://docs.crawl4ai.com/advanced/proxy-security/#__codelineno-2-21) async with AsyncWebCrawler(config=browser_config) as crawler: [](https://docs.crawl4ai.com/advanced/proxy-security/#__codelineno-2-22) result = await crawler.arun(url="https://example.com", config=run_config) [](https://docs.crawl4ai.com/advanced/proxy-security/#__codelineno-2-23) print(f"Success: {result.success} -> {result.url}") [](https://docs.crawl4ai.com/advanced/proxy-security/#__codelineno-2-24) [](https://docs.crawl4ai.com/advanced/proxy-security/#__codelineno-2-25) [](https://docs.crawl4ai.com/advanced/proxy-security/#__codelineno-2-26) if __name__ == "__main__": [](https://docs.crawl4ai.com/advanced/proxy-security/#__codelineno-2-27) asyncio.run(main())` Environment Variable Configuration ---------------------------------- Load proxies from environment variables for easy configuration: `[](https://docs.crawl4ai.com/advanced/proxy-security/#__codelineno-3-1) import os [](https://docs.crawl4ai.com/advanced/proxy-security/#__codelineno-3-2) from crawl4ai import ProxyConfig, CrawlerRunConfig [](https://docs.crawl4ai.com/advanced/proxy-security/#__codelineno-3-3) [](https://docs.crawl4ai.com/advanced/proxy-security/#__codelineno-3-4) # Set environment variable [](https://docs.crawl4ai.com/advanced/proxy-security/#__codelineno-3-5) os.environ["PROXIES"] = "ip1:port1:user1:pass1,ip2:port2:user2:pass2,ip3:port3" [](https://docs.crawl4ai.com/advanced/proxy-security/#__codelineno-3-6) [](https://docs.crawl4ai.com/advanced/proxy-security/#__codelineno-3-7) # Load all proxies [](https://docs.crawl4ai.com/advanced/proxy-security/#__codelineno-3-8) proxies = ProxyConfig.from_env() [](https://docs.crawl4ai.com/advanced/proxy-security/#__codelineno-3-9) print(f"Loaded {len(proxies)} proxies") [](https://docs.crawl4ai.com/advanced/proxy-security/#__codelineno-3-10) [](https://docs.crawl4ai.com/advanced/proxy-security/#__codelineno-3-11) # Use first proxy [](https://docs.crawl4ai.com/advanced/proxy-security/#__codelineno-3-12) if proxies: [](https://docs.crawl4ai.com/advanced/proxy-security/#__codelineno-3-13) run_config = CrawlerRunConfig(proxy_config=proxies[0])` Rotating Proxies ---------------- Crawl4AI supports automatic proxy rotation to distribute requests across multiple proxy servers. Rotation is applied per request using a rotation strategy on `CrawlerRunConfig`. ### Proxy Rotation (recommended) `[](https://docs.crawl4ai.com/advanced/proxy-security/#__codelineno-4-1) import asyncio [](https://docs.crawl4ai.com/advanced/proxy-security/#__codelineno-4-2) import re [](https://docs.crawl4ai.com/advanced/proxy-security/#__codelineno-4-3) from crawl4ai import AsyncWebCrawler, BrowserConfig, CrawlerRunConfig, CacheMode, ProxyConfig [](https://docs.crawl4ai.com/advanced/proxy-security/#__codelineno-4-4) from crawl4ai.proxy_strategy import RoundRobinProxyStrategy [](https://docs.crawl4ai.com/advanced/proxy-security/#__codelineno-4-5) [](https://docs.crawl4ai.com/advanced/proxy-security/#__codelineno-4-6) async def main(): [](https://docs.crawl4ai.com/advanced/proxy-security/#__codelineno-4-7) # Load proxies from environment [](https://docs.crawl4ai.com/advanced/proxy-security/#__codelineno-4-8) proxies = ProxyConfig.from_env() [](https://docs.crawl4ai.com/advanced/proxy-security/#__codelineno-4-9) if not proxies: [](https://docs.crawl4ai.com/advanced/proxy-security/#__codelineno-4-10) print("No proxies found! Set PROXIES environment variable.") [](https://docs.crawl4ai.com/advanced/proxy-security/#__codelineno-4-11) return [](https://docs.crawl4ai.com/advanced/proxy-security/#__codelineno-4-12) [](https://docs.crawl4ai.com/advanced/proxy-security/#__codelineno-4-13) # Create rotation strategy [](https://docs.crawl4ai.com/advanced/proxy-security/#__codelineno-4-14) proxy_strategy = RoundRobinProxyStrategy(proxies) [](https://docs.crawl4ai.com/advanced/proxy-security/#__codelineno-4-15) [](https://docs.crawl4ai.com/advanced/proxy-security/#__codelineno-4-16) # Configure per-request with proxy rotation [](https://docs.crawl4ai.com/advanced/proxy-security/#__codelineno-4-17) browser_config = BrowserConfig(headless=True, verbose=False) [](https://docs.crawl4ai.com/advanced/proxy-security/#__codelineno-4-18) run_config = CrawlerRunConfig( [](https://docs.crawl4ai.com/advanced/proxy-security/#__codelineno-4-19) cache_mode=CacheMode.BYPASS, [](https://docs.crawl4ai.com/advanced/proxy-security/#__codelineno-4-20) proxy_rotation_strategy=proxy_strategy, [](https://docs.crawl4ai.com/advanced/proxy-security/#__codelineno-4-21) ) [](https://docs.crawl4ai.com/advanced/proxy-security/#__codelineno-4-22) [](https://docs.crawl4ai.com/advanced/proxy-security/#__codelineno-4-23) async with AsyncWebCrawler(config=browser_config) as crawler: [](https://docs.crawl4ai.com/advanced/proxy-security/#__codelineno-4-24) urls = ["https://httpbin.org/ip"] * (len(proxies) * 2) # Test each proxy twice [](https://docs.crawl4ai.com/advanced/proxy-security/#__codelineno-4-25) [](https://docs.crawl4ai.com/advanced/proxy-security/#__codelineno-4-26) print(f"🚀 Testing {len(proxies)} proxies with rotation...") [](https://docs.crawl4ai.com/advanced/proxy-security/#__codelineno-4-27) results = await crawler.arun_many(urls=urls, config=run_config) [](https://docs.crawl4ai.com/advanced/proxy-security/#__codelineno-4-28) [](https://docs.crawl4ai.com/advanced/proxy-security/#__codelineno-4-29) for i, result in enumerate(results): [](https://docs.crawl4ai.com/advanced/proxy-security/#__codelineno-4-30) if result.success: [](https://docs.crawl4ai.com/advanced/proxy-security/#__codelineno-4-31) # Extract IP from response [](https://docs.crawl4ai.com/advanced/proxy-security/#__codelineno-4-32) ip_match = re.search(r'(?:[0-9]{1,3}\.){3}[0-9]{1,3}', result.html) [](https://docs.crawl4ai.com/advanced/proxy-security/#__codelineno-4-33) if ip_match: [](https://docs.crawl4ai.com/advanced/proxy-security/#__codelineno-4-34) detected_ip = ip_match.group(0) [](https://docs.crawl4ai.com/advanced/proxy-security/#__codelineno-4-35) proxy_index = i % len(proxies) [](https://docs.crawl4ai.com/advanced/proxy-security/#__codelineno-4-36) expected_ip = proxies[proxy_index].ip [](https://docs.crawl4ai.com/advanced/proxy-security/#__codelineno-4-37) [](https://docs.crawl4ai.com/advanced/proxy-security/#__codelineno-4-38) print(f"✅ Request {i+1}: Proxy {proxy_index+1} -> IP {detected_ip}") [](https://docs.crawl4ai.com/advanced/proxy-security/#__codelineno-4-39) if detected_ip == expected_ip: [](https://docs.crawl4ai.com/advanced/proxy-security/#__codelineno-4-40) print(" 🎯 IP matches proxy configuration") [](https://docs.crawl4ai.com/advanced/proxy-security/#__codelineno-4-41) else: [](https://docs.crawl4ai.com/advanced/proxy-security/#__codelineno-4-42) print(f" ⚠️ IP mismatch (expected {expected_ip})") [](https://docs.crawl4ai.com/advanced/proxy-security/#__codelineno-4-43) else: [](https://docs.crawl4ai.com/advanced/proxy-security/#__codelineno-4-44) print(f"❌ Request {i+1}: Could not extract IP from response") [](https://docs.crawl4ai.com/advanced/proxy-security/#__codelineno-4-45) else: [](https://docs.crawl4ai.com/advanced/proxy-security/#__codelineno-4-46) print(f"❌ Request {i+1}: Failed - {result.error_message}") [](https://docs.crawl4ai.com/advanced/proxy-security/#__codelineno-4-47) [](https://docs.crawl4ai.com/advanced/proxy-security/#__codelineno-4-48) if __name__ == "__main__": [](https://docs.crawl4ai.com/advanced/proxy-security/#__codelineno-4-49) asyncio.run(main())` SSL Certificate Analysis ------------------------ Combine proxy usage with SSL certificate inspection for enhanced security analysis. SSL certificate fetching is configured per request via `CrawlerRunConfig`. ### Per-Request SSL Certificate Analysis `[](https://docs.crawl4ai.com/advanced/proxy-security/#__codelineno-5-1) import asyncio [](https://docs.crawl4ai.com/advanced/proxy-security/#__codelineno-5-2) from crawl4ai import AsyncWebCrawler, BrowserConfig, CrawlerRunConfig [](https://docs.crawl4ai.com/advanced/proxy-security/#__codelineno-5-3) [](https://docs.crawl4ai.com/advanced/proxy-security/#__codelineno-5-4) run_config = CrawlerRunConfig( [](https://docs.crawl4ai.com/advanced/proxy-security/#__codelineno-5-5) proxy_config={ [](https://docs.crawl4ai.com/advanced/proxy-security/#__codelineno-5-6) "server": "http://proxy.example.com:8080", [](https://docs.crawl4ai.com/advanced/proxy-security/#__codelineno-5-7) "username": "user", [](https://docs.crawl4ai.com/advanced/proxy-security/#__codelineno-5-8) "password": "pass", [](https://docs.crawl4ai.com/advanced/proxy-security/#__codelineno-5-9) }, [](https://docs.crawl4ai.com/advanced/proxy-security/#__codelineno-5-10) fetch_ssl_certificate=True, # Enable SSL certificate analysis for this request [](https://docs.crawl4ai.com/advanced/proxy-security/#__codelineno-5-11) ) [](https://docs.crawl4ai.com/advanced/proxy-security/#__codelineno-5-12) [](https://docs.crawl4ai.com/advanced/proxy-security/#__codelineno-5-13) [](https://docs.crawl4ai.com/advanced/proxy-security/#__codelineno-5-14) async def main(): [](https://docs.crawl4ai.com/advanced/proxy-security/#__codelineno-5-15) browser_config = BrowserConfig() [](https://docs.crawl4ai.com/advanced/proxy-security/#__codelineno-5-16) async with AsyncWebCrawler(config=browser_config) as crawler: [](https://docs.crawl4ai.com/advanced/proxy-security/#__codelineno-5-17) result = await crawler.arun(url="https://example.com", config=run_config) [](https://docs.crawl4ai.com/advanced/proxy-security/#__codelineno-5-18) [](https://docs.crawl4ai.com/advanced/proxy-security/#__codelineno-5-19) if result.success: [](https://docs.crawl4ai.com/advanced/proxy-security/#__codelineno-5-20) print(f"✅ Crawled via proxy: {result.url}") [](https://docs.crawl4ai.com/advanced/proxy-security/#__codelineno-5-21) [](https://docs.crawl4ai.com/advanced/proxy-security/#__codelineno-5-22) # Analyze SSL certificate [](https://docs.crawl4ai.com/advanced/proxy-security/#__codelineno-5-23) if result.ssl_certificate: [](https://docs.crawl4ai.com/advanced/proxy-security/#__codelineno-5-24) cert = result.ssl_certificate [](https://docs.crawl4ai.com/advanced/proxy-security/#__codelineno-5-25) print("🔒 SSL Certificate Info:") [](https://docs.crawl4ai.com/advanced/proxy-security/#__codelineno-5-26) print(f" Issuer: {cert.issuer}") [](https://docs.crawl4ai.com/advanced/proxy-security/#__codelineno-5-27) print(f" Subject: {cert.subject}") [](https://docs.crawl4ai.com/advanced/proxy-security/#__codelineno-5-28) print(f" Valid until: {cert.valid_until}") [](https://docs.crawl4ai.com/advanced/proxy-security/#__codelineno-5-29) print(f" Fingerprint: {cert.fingerprint}") [](https://docs.crawl4ai.com/advanced/proxy-security/#__codelineno-5-30) [](https://docs.crawl4ai.com/advanced/proxy-security/#__codelineno-5-31) # Export certificate [](https://docs.crawl4ai.com/advanced/proxy-security/#__codelineno-5-32) cert.to_json("certificate.json") [](https://docs.crawl4ai.com/advanced/proxy-security/#__codelineno-5-33) print("💾 Certificate exported to certificate.json") [](https://docs.crawl4ai.com/advanced/proxy-security/#__codelineno-5-34) else: [](https://docs.crawl4ai.com/advanced/proxy-security/#__codelineno-5-35) print("⚠️ No SSL certificate information available") [](https://docs.crawl4ai.com/advanced/proxy-security/#__codelineno-5-36) [](https://docs.crawl4ai.com/advanced/proxy-security/#__codelineno-5-37) [](https://docs.crawl4ai.com/advanced/proxy-security/#__codelineno-5-38) if __name__ == "__main__": [](https://docs.crawl4ai.com/advanced/proxy-security/#__codelineno-5-39) asyncio.run(main())` Security Best Practices ----------------------- ### 1\. Proxy Rotation for Anonymity `[](https://docs.crawl4ai.com/advanced/proxy-security/#__codelineno-6-1) from crawl4ai import CrawlerRunConfig, ProxyConfig [](https://docs.crawl4ai.com/advanced/proxy-security/#__codelineno-6-2) from crawl4ai.proxy_strategy import RoundRobinProxyStrategy [](https://docs.crawl4ai.com/advanced/proxy-security/#__codelineno-6-3) [](https://docs.crawl4ai.com/advanced/proxy-security/#__codelineno-6-4) # Use multiple proxies to avoid IP blocking [](https://docs.crawl4ai.com/advanced/proxy-security/#__codelineno-6-5) proxies = ProxyConfig.from_env("PROXIES") [](https://docs.crawl4ai.com/advanced/proxy-security/#__codelineno-6-6) strategy = RoundRobinProxyStrategy(proxies) [](https://docs.crawl4ai.com/advanced/proxy-security/#__codelineno-6-7) [](https://docs.crawl4ai.com/advanced/proxy-security/#__codelineno-6-8) # Configure rotation per request (recommended) [](https://docs.crawl4ai.com/advanced/proxy-security/#__codelineno-6-9) run_config = CrawlerRunConfig(proxy_rotation_strategy=strategy) [](https://docs.crawl4ai.com/advanced/proxy-security/#__codelineno-6-10) [](https://docs.crawl4ai.com/advanced/proxy-security/#__codelineno-6-11) # For a fixed proxy across all requests, just reuse the same run_config instance [](https://docs.crawl4ai.com/advanced/proxy-security/#__codelineno-6-12) static_run_config = run_config` ### 2\. SSL Certificate Verification `[](https://docs.crawl4ai.com/advanced/proxy-security/#__codelineno-7-1) from crawl4ai import CrawlerRunConfig [](https://docs.crawl4ai.com/advanced/proxy-security/#__codelineno-7-2) [](https://docs.crawl4ai.com/advanced/proxy-security/#__codelineno-7-3) # Always verify SSL certificates when possible [](https://docs.crawl4ai.com/advanced/proxy-security/#__codelineno-7-4) # Per-request (affects specific requests) [](https://docs.crawl4ai.com/advanced/proxy-security/#__codelineno-7-5) run_config = CrawlerRunConfig(fetch_ssl_certificate=True)` ### 3\. Environment Variable Security `[](https://docs.crawl4ai.com/advanced/proxy-security/#__codelineno-8-1) # Use environment variables for sensitive proxy credentials [](https://docs.crawl4ai.com/advanced/proxy-security/#__codelineno-8-2) # Avoid hardcoding usernames/passwords in code [](https://docs.crawl4ai.com/advanced/proxy-security/#__codelineno-8-3) export PROXIES="ip1:port1:user1:pass1,ip2:port2:user2:pass2"` ### 4\. SOCKS5 for Enhanced Security `[](https://docs.crawl4ai.com/advanced/proxy-security/#__codelineno-9-1) from crawl4ai import CrawlerRunConfig [](https://docs.crawl4ai.com/advanced/proxy-security/#__codelineno-9-2) [](https://docs.crawl4ai.com/advanced/proxy-security/#__codelineno-9-3) # Prefer SOCKS5 proxies for better protocol support [](https://docs.crawl4ai.com/advanced/proxy-security/#__codelineno-9-4) run_config = CrawlerRunConfig(proxy_config="socks5://proxy.example.com:1080")` Migration from Deprecated `proxy` Parameter ------------------------------------------- * "Deprecation Notice" The legacy `proxy` argument on `BrowserConfig` is deprecated. Configure proxies through `CrawlerRunConfig.proxy_config` so each request fully describes its network settings. `[](https://docs.crawl4ai.com/advanced/proxy-security/#__codelineno-10-1) # Old (deprecated) approach [](https://docs.crawl4ai.com/advanced/proxy-security/#__codelineno-10-2) # from crawl4ai import BrowserConfig [](https://docs.crawl4ai.com/advanced/proxy-security/#__codelineno-10-3) # browser_config = BrowserConfig(proxy="http://proxy.example.com:8080") [](https://docs.crawl4ai.com/advanced/proxy-security/#__codelineno-10-4) [](https://docs.crawl4ai.com/advanced/proxy-security/#__codelineno-10-5) # New (preferred) approach [](https://docs.crawl4ai.com/advanced/proxy-security/#__codelineno-10-6) from crawl4ai import CrawlerRunConfig [](https://docs.crawl4ai.com/advanced/proxy-security/#__codelineno-10-7) run_config = CrawlerRunConfig(proxy_config="http://proxy.example.com:8080")` ### Safe Logging of Proxies `[](https://docs.crawl4ai.com/advanced/proxy-security/#__codelineno-11-1) from crawl4ai import ProxyConfig [](https://docs.crawl4ai.com/advanced/proxy-security/#__codelineno-11-2) [](https://docs.crawl4ai.com/advanced/proxy-security/#__codelineno-11-3) def safe_proxy_repr(proxy: ProxyConfig): [](https://docs.crawl4ai.com/advanced/proxy-security/#__codelineno-11-4) if getattr(proxy, "username", None): [](https://docs.crawl4ai.com/advanced/proxy-security/#__codelineno-11-5) return f"{proxy.server} (auth: ****)" [](https://docs.crawl4ai.com/advanced/proxy-security/#__codelineno-11-6) return proxy.server` Troubleshooting --------------- ### Common Issues * "Proxy connection failed" * Verify the proxy server is reachable from your network. * Double-check authentication credentials. * Ensure the protocol matches (`http`, `https`, or `socks5`). * "SSL certificate errors" * Some proxies break SSL inspection; switch proxies if you see repeated failures. * Consider temporarily disabling certificate fetching to isolate the issue. * "Environment variables not loading" * Confirm `PROXIES` (or your custom env var) is set before running the script. * Check formatting: `ip:port:user:pass,ip:port:user:pass`. * "Proxy rotation not working" * Ensure `ProxyConfig.from_env()` actually loaded entries (`len(proxies) > 0`). * Attach `proxy_rotation_strategy` to `CrawlerRunConfig`. * Validate the proxy definitions you pass into the strategy. See Also -------- * [Anti-Bot Detection & Fallback](https://docs.crawl4ai.com/advanced/anti-bot-and-fallback/) — Automatic retry with proxy escalation and fallback functions when anti-bot blocking is detected * * * \> Feedback --- # SSL Certificate - Crawl4AI Documentation (v0.8.x) `SSLCertificate` Reference ========================== The **`SSLCertificate`** class encapsulates an SSL certificate’s data and allows exporting it in various formats (PEM, DER, JSON, or text). It’s used within **Crawl4AI** whenever you set **`fetch_ssl_certificate=True`** in your **`CrawlerRunConfig`**. 1\. Overview ------------ **Location**: `crawl4ai/ssl_certificate.py` `[](https://docs.crawl4ai.com/advanced/ssl-certificate/#__codelineno-0-1) class SSLCertificate: [](https://docs.crawl4ai.com/advanced/ssl-certificate/#__codelineno-0-2) """ [](https://docs.crawl4ai.com/advanced/ssl-certificate/#__codelineno-0-3) Represents an SSL certificate with methods to export in various formats. [](https://docs.crawl4ai.com/advanced/ssl-certificate/#__codelineno-0-4) [](https://docs.crawl4ai.com/advanced/ssl-certificate/#__codelineno-0-5) Main Methods: [](https://docs.crawl4ai.com/advanced/ssl-certificate/#__codelineno-0-6) - from_url(url, timeout=10) [](https://docs.crawl4ai.com/advanced/ssl-certificate/#__codelineno-0-7) - from_file(file_path) [](https://docs.crawl4ai.com/advanced/ssl-certificate/#__codelineno-0-8) - from_binary(binary_data) [](https://docs.crawl4ai.com/advanced/ssl-certificate/#__codelineno-0-9) - to_json(filepath=None) [](https://docs.crawl4ai.com/advanced/ssl-certificate/#__codelineno-0-10) - to_pem(filepath=None) [](https://docs.crawl4ai.com/advanced/ssl-certificate/#__codelineno-0-11) - to_der(filepath=None) [](https://docs.crawl4ai.com/advanced/ssl-certificate/#__codelineno-0-12) ... [](https://docs.crawl4ai.com/advanced/ssl-certificate/#__codelineno-0-13) [](https://docs.crawl4ai.com/advanced/ssl-certificate/#__codelineno-0-14) Common Properties: [](https://docs.crawl4ai.com/advanced/ssl-certificate/#__codelineno-0-15) - issuer [](https://docs.crawl4ai.com/advanced/ssl-certificate/#__codelineno-0-16) - subject [](https://docs.crawl4ai.com/advanced/ssl-certificate/#__codelineno-0-17) - valid_from [](https://docs.crawl4ai.com/advanced/ssl-certificate/#__codelineno-0-18) - valid_until [](https://docs.crawl4ai.com/advanced/ssl-certificate/#__codelineno-0-19) - fingerprint [](https://docs.crawl4ai.com/advanced/ssl-certificate/#__codelineno-0-20) """` ### Typical Use Case 1. You **enable** certificate fetching in your crawl by: `[](https://docs.crawl4ai.com/advanced/ssl-certificate/#__codelineno-1-1) CrawlerRunConfig(fetch_ssl_certificate=True, ...)` 2. After `arun()`, if `result.ssl_certificate` is present, it’s an instance of **`SSLCertificate`**. 3. You can **read** basic properties (issuer, subject, validity) or **export** them in multiple formats. * * * 2\. Construction & Fetching --------------------------- ### 2.1 **`from_url(url, timeout=10)`** Manually load an SSL certificate from a given URL (port 443). Typically used internally, but you can call it directly if you want: `[](https://docs.crawl4ai.com/advanced/ssl-certificate/#__codelineno-2-1) cert = SSLCertificate.from_url("https://example.com") [](https://docs.crawl4ai.com/advanced/ssl-certificate/#__codelineno-2-2) if cert: [](https://docs.crawl4ai.com/advanced/ssl-certificate/#__codelineno-2-3) print("Fingerprint:", cert.fingerprint)` ### 2.2 **`from_file(file_path)`** Load from a file containing certificate data in ASN.1 or DER. Rarely needed unless you have local cert files: `[](https://docs.crawl4ai.com/advanced/ssl-certificate/#__codelineno-3-1) cert = SSLCertificate.from_file("/path/to/cert.der")` ### 2.3 **`from_binary(binary_data)`** Initialize from raw binary. E.g., if you captured it from a socket or another source: `[](https://docs.crawl4ai.com/advanced/ssl-certificate/#__codelineno-4-1) cert = SSLCertificate.from_binary(raw_bytes)` * * * 3\. Common Properties --------------------- After obtaining a **`SSLCertificate`** instance (e.g. `result.ssl_certificate` from a crawl), you can read: 1. **`issuer`** _(dict)_ \- E.g. `{"CN": "My Root CA", "O": "..."}` 2. **`subject`** _(dict)_ \- E.g. `{"CN": "example.com", "O": "ExampleOrg"}` 3. **`valid_from`** _(str)_ \- NotBefore date/time. Often in ASN.1/UTC format. 4. **`valid_until`** _(str)_ \- NotAfter date/time. 5. **`fingerprint`** _(str)_ \- The SHA-256 digest (lowercase hex). \- E.g. `"d14d2e..."` * * * 4\. Export Methods ------------------ Once you have a **`SSLCertificate`** object, you can **export** or **inspect** it: ### 4.1 **`to_json(filepath=None)` → `Optional[str]`** * Returns a JSON string containing the parsed certificate fields. * If `filepath` is provided, saves it to disk instead, returning `None`. **Usage**: `[](https://docs.crawl4ai.com/advanced/ssl-certificate/#__codelineno-5-1) json_data = cert.to_json() # returns JSON string [](https://docs.crawl4ai.com/advanced/ssl-certificate/#__codelineno-5-2) cert.to_json("certificate.json") # writes file, returns None` ### 4.2 **`to_pem(filepath=None)` → `Optional[str]`** * Returns a PEM-encoded string (common for web servers). * If `filepath` is provided, saves it to disk instead. `[](https://docs.crawl4ai.com/advanced/ssl-certificate/#__codelineno-6-1) pem_str = cert.to_pem() # in-memory PEM string [](https://docs.crawl4ai.com/advanced/ssl-certificate/#__codelineno-6-2) cert.to_pem("/path/to/cert.pem") # saved to file` ### 4.3 **`to_der(filepath=None)` → `Optional[bytes]`** * Returns the original DER (binary ASN.1) bytes. * If `filepath` is specified, writes the bytes there instead. `[](https://docs.crawl4ai.com/advanced/ssl-certificate/#__codelineno-7-1) der_bytes = cert.to_der() [](https://docs.crawl4ai.com/advanced/ssl-certificate/#__codelineno-7-2) cert.to_der("certificate.der")` ### 4.4 (Optional) **`export_as_text()`** * If you see a method like `export_as_text()`, it typically returns an OpenSSL-style textual representation. * Not always needed, but can help for debugging or manual inspection. * * * 5\. Example Usage in Crawl4AI ----------------------------- Below is a minimal sample showing how the crawler obtains an SSL cert from a site, then reads or exports it. The code snippet: `[](https://docs.crawl4ai.com/advanced/ssl-certificate/#__codelineno-8-1) import asyncio [](https://docs.crawl4ai.com/advanced/ssl-certificate/#__codelineno-8-2) import os [](https://docs.crawl4ai.com/advanced/ssl-certificate/#__codelineno-8-3) from crawl4ai import AsyncWebCrawler, CrawlerRunConfig, CacheMode [](https://docs.crawl4ai.com/advanced/ssl-certificate/#__codelineno-8-4) [](https://docs.crawl4ai.com/advanced/ssl-certificate/#__codelineno-8-5) async def main(): [](https://docs.crawl4ai.com/advanced/ssl-certificate/#__codelineno-8-6) tmp_dir = "tmp" [](https://docs.crawl4ai.com/advanced/ssl-certificate/#__codelineno-8-7) os.makedirs(tmp_dir, exist_ok=True) [](https://docs.crawl4ai.com/advanced/ssl-certificate/#__codelineno-8-8) [](https://docs.crawl4ai.com/advanced/ssl-certificate/#__codelineno-8-9) config = CrawlerRunConfig( [](https://docs.crawl4ai.com/advanced/ssl-certificate/#__codelineno-8-10) fetch_ssl_certificate=True, [](https://docs.crawl4ai.com/advanced/ssl-certificate/#__codelineno-8-11) cache_mode=CacheMode.BYPASS [](https://docs.crawl4ai.com/advanced/ssl-certificate/#__codelineno-8-12) ) [](https://docs.crawl4ai.com/advanced/ssl-certificate/#__codelineno-8-13) [](https://docs.crawl4ai.com/advanced/ssl-certificate/#__codelineno-8-14) async with AsyncWebCrawler() as crawler: [](https://docs.crawl4ai.com/advanced/ssl-certificate/#__codelineno-8-15) result = await crawler.arun("https://example.com", config=config) [](https://docs.crawl4ai.com/advanced/ssl-certificate/#__codelineno-8-16) if result.success and result.ssl_certificate: [](https://docs.crawl4ai.com/advanced/ssl-certificate/#__codelineno-8-17) cert = result.ssl_certificate [](https://docs.crawl4ai.com/advanced/ssl-certificate/#__codelineno-8-18) # 1. Basic Info [](https://docs.crawl4ai.com/advanced/ssl-certificate/#__codelineno-8-19) print("Issuer CN:", cert.issuer.get("CN", "")) [](https://docs.crawl4ai.com/advanced/ssl-certificate/#__codelineno-8-20) print("Valid until:", cert.valid_until) [](https://docs.crawl4ai.com/advanced/ssl-certificate/#__codelineno-8-21) print("Fingerprint:", cert.fingerprint) [](https://docs.crawl4ai.com/advanced/ssl-certificate/#__codelineno-8-22) [](https://docs.crawl4ai.com/advanced/ssl-certificate/#__codelineno-8-23) # 2. Export [](https://docs.crawl4ai.com/advanced/ssl-certificate/#__codelineno-8-24) cert.to_json(os.path.join(tmp_dir, "certificate.json")) [](https://docs.crawl4ai.com/advanced/ssl-certificate/#__codelineno-8-25) cert.to_pem(os.path.join(tmp_dir, "certificate.pem")) [](https://docs.crawl4ai.com/advanced/ssl-certificate/#__codelineno-8-26) cert.to_der(os.path.join(tmp_dir, "certificate.der")) [](https://docs.crawl4ai.com/advanced/ssl-certificate/#__codelineno-8-27) [](https://docs.crawl4ai.com/advanced/ssl-certificate/#__codelineno-8-28) if __name__ == "__main__": [](https://docs.crawl4ai.com/advanced/ssl-certificate/#__codelineno-8-29) asyncio.run(main())` * * * 6\. Notes & Best Practices -------------------------- 1. **Timeout**: `SSLCertificate.from_url` internally uses a default **10s** socket connect and wraps SSL. 2. **Binary Form**: The certificate is loaded in ASN.1 (DER) form, then re-parsed by `OpenSSL.crypto`. 3. **Validation**: This does **not** validate the certificate chain or trust store. It only fetches and parses. 4. **Integration**: Within Crawl4AI, you typically just set `fetch_ssl_certificate=True` in `CrawlerRunConfig`; the final result’s `ssl_certificate` is automatically built. 5. **Export**: If you need to store or analyze a cert, the `to_json` and `to_pem` are quite universal. * * * ### Summary * **`SSLCertificate`** is a convenience class for capturing and exporting the **TLS certificate** from your crawled site(s). * Common usage is in the **`CrawlResult.ssl_certificate`** field, accessible after setting `fetch_ssl_certificate=True`. * Offers quick access to essential certificate details (`issuer`, `subject`, `fingerprint`) and is easy to export (PEM, DER, JSON) for further analysis or server usage. Use it whenever you need **insight** into a site’s certificate or require some form of cryptographic or compliance check. * * * \> Feedback --- # arun_many() - Crawl4AI Documentation (v0.8.x) `arun_many(...)` Reference ========================== > **Note**: This function is very similar to [`arun()`](https://docs.crawl4ai.com/api/arun/) > but focused on **concurrent** or **batch** crawling. If you’re unfamiliar with `arun()` usage, please read that doc first, then review this for differences. Function Signature ------------------ ``[](https://docs.crawl4ai.com/api/arun_many/#__codelineno-0-1) async def arun_many( [](https://docs.crawl4ai.com/api/arun_many/#__codelineno-0-2) urls: Union[List[str], List[Any]], [](https://docs.crawl4ai.com/api/arun_many/#__codelineno-0-3) config: Optional[Union[CrawlerRunConfig, List[CrawlerRunConfig]]] = None, [](https://docs.crawl4ai.com/api/arun_many/#__codelineno-0-4) dispatcher: Optional[BaseDispatcher] = None, [](https://docs.crawl4ai.com/api/arun_many/#__codelineno-0-5) ... [](https://docs.crawl4ai.com/api/arun_many/#__codelineno-0-6) ) -> Union[List[CrawlResult], AsyncGenerator[CrawlResult, None]]: [](https://docs.crawl4ai.com/api/arun_many/#__codelineno-0-7) """ [](https://docs.crawl4ai.com/api/arun_many/#__codelineno-0-8) Crawl multiple URLs concurrently or in batches. [](https://docs.crawl4ai.com/api/arun_many/#__codelineno-0-9) [](https://docs.crawl4ai.com/api/arun_many/#__codelineno-0-10) :param urls: A list of URLs (or tasks) to crawl. [](https://docs.crawl4ai.com/api/arun_many/#__codelineno-0-11) :param config: (Optional) Either: [](https://docs.crawl4ai.com/api/arun_many/#__codelineno-0-12) - A single `CrawlerRunConfig` applying to all URLs [](https://docs.crawl4ai.com/api/arun_many/#__codelineno-0-13) - A list of `CrawlerRunConfig` objects with url_matcher patterns [](https://docs.crawl4ai.com/api/arun_many/#__codelineno-0-14) :param dispatcher: (Optional) A concurrency controller (e.g. MemoryAdaptiveDispatcher). [](https://docs.crawl4ai.com/api/arun_many/#__codelineno-0-15) ... [](https://docs.crawl4ai.com/api/arun_many/#__codelineno-0-16) :return: Either a list of `CrawlResult` objects, or an async generator if streaming is enabled. [](https://docs.crawl4ai.com/api/arun_many/#__codelineno-0-17) """`` Differences from `arun()` ------------------------- 1. **Multiple URLs**: * Instead of crawling a single URL, you pass a list of them (strings or tasks).  * The function returns either a **list** of `CrawlResult` or an **async generator** if streaming is enabled. 2. **Concurrency & Dispatchers**: * **`dispatcher`** param allows advanced concurrency control.  * If omitted, a default dispatcher (like `MemoryAdaptiveDispatcher`) is used internally.  * Dispatchers handle concurrency, rate limiting, and memory-based adaptive throttling (see [Multi-URL Crawling](https://docs.crawl4ai.com/advanced/multi-url-crawling/) ). 3. **Streaming Support**: * Enable streaming by setting `stream=True` in your `CrawlerRunConfig`. * When streaming, use `async for` to process results as they become available. * Ideal for processing large numbers of URLs without waiting for all to complete. 4. **Parallel** Execution\*\*: * `arun_many()` can run multiple requests concurrently under the hood.  * Each `CrawlResult` might also include a **`dispatch_result`** with concurrency details (like memory usage, start/end times). ### Basic Example (Batch Mode) `[](https://docs.crawl4ai.com/api/arun_many/#__codelineno-1-1) # Minimal usage: The default dispatcher will be used [](https://docs.crawl4ai.com/api/arun_many/#__codelineno-1-2) results = await crawler.arun_many( [](https://docs.crawl4ai.com/api/arun_many/#__codelineno-1-3) urls=["https://site1.com", "https://site2.com"], [](https://docs.crawl4ai.com/api/arun_many/#__codelineno-1-4) config=CrawlerRunConfig(stream=False) # Default behavior [](https://docs.crawl4ai.com/api/arun_many/#__codelineno-1-5) ) [](https://docs.crawl4ai.com/api/arun_many/#__codelineno-1-6) [](https://docs.crawl4ai.com/api/arun_many/#__codelineno-1-7) for res in results: [](https://docs.crawl4ai.com/api/arun_many/#__codelineno-1-8) if res.success: [](https://docs.crawl4ai.com/api/arun_many/#__codelineno-1-9) print(res.url, "crawled OK!") [](https://docs.crawl4ai.com/api/arun_many/#__codelineno-1-10) else: [](https://docs.crawl4ai.com/api/arun_many/#__codelineno-1-11) print("Failed:", res.url, "-", res.error_message)` ### Streaming Example `[](https://docs.crawl4ai.com/api/arun_many/#__codelineno-2-1) config = CrawlerRunConfig( [](https://docs.crawl4ai.com/api/arun_many/#__codelineno-2-2) stream=True, # Enable streaming mode [](https://docs.crawl4ai.com/api/arun_many/#__codelineno-2-3) cache_mode=CacheMode.BYPASS [](https://docs.crawl4ai.com/api/arun_many/#__codelineno-2-4) ) [](https://docs.crawl4ai.com/api/arun_many/#__codelineno-2-5) [](https://docs.crawl4ai.com/api/arun_many/#__codelineno-2-6) # Process results as they complete [](https://docs.crawl4ai.com/api/arun_many/#__codelineno-2-7) async for result in await crawler.arun_many( [](https://docs.crawl4ai.com/api/arun_many/#__codelineno-2-8) urls=["https://site1.com", "https://site2.com", "https://site3.com"], [](https://docs.crawl4ai.com/api/arun_many/#__codelineno-2-9) config=config [](https://docs.crawl4ai.com/api/arun_many/#__codelineno-2-10) ): [](https://docs.crawl4ai.com/api/arun_many/#__codelineno-2-11) if result.success: [](https://docs.crawl4ai.com/api/arun_many/#__codelineno-2-12) print(f"Just completed: {result.url}") [](https://docs.crawl4ai.com/api/arun_many/#__codelineno-2-13) # Process each result immediately [](https://docs.crawl4ai.com/api/arun_many/#__codelineno-2-14) process_result(result)` ### With a Custom Dispatcher `[](https://docs.crawl4ai.com/api/arun_many/#__codelineno-3-1) dispatcher = MemoryAdaptiveDispatcher( [](https://docs.crawl4ai.com/api/arun_many/#__codelineno-3-2) memory_threshold_percent=70.0, [](https://docs.crawl4ai.com/api/arun_many/#__codelineno-3-3) max_session_permit=10 [](https://docs.crawl4ai.com/api/arun_many/#__codelineno-3-4) ) [](https://docs.crawl4ai.com/api/arun_many/#__codelineno-3-5) results = await crawler.arun_many( [](https://docs.crawl4ai.com/api/arun_many/#__codelineno-3-6) urls=["https://site1.com", "https://site2.com", "https://site3.com"], [](https://docs.crawl4ai.com/api/arun_many/#__codelineno-3-7) config=my_run_config, [](https://docs.crawl4ai.com/api/arun_many/#__codelineno-3-8) dispatcher=dispatcher [](https://docs.crawl4ai.com/api/arun_many/#__codelineno-3-9) )` ### URL-Specific Configurations Instead of using one config for all URLs, provide a list of configs with `url_matcher` patterns: `[](https://docs.crawl4ai.com/api/arun_many/#__codelineno-4-1) from crawl4ai import CrawlerRunConfig, MatchMode [](https://docs.crawl4ai.com/api/arun_many/#__codelineno-4-2) from crawl4ai.processors.pdf import PDFContentScrapingStrategy [](https://docs.crawl4ai.com/api/arun_many/#__codelineno-4-3) from crawl4ai.extraction_strategy import JsonCssExtractionStrategy [](https://docs.crawl4ai.com/api/arun_many/#__codelineno-4-4) from crawl4ai.content_filter_strategy import PruningContentFilter [](https://docs.crawl4ai.com/api/arun_many/#__codelineno-4-5) from crawl4ai.markdown_generation_strategy import DefaultMarkdownGenerator [](https://docs.crawl4ai.com/api/arun_many/#__codelineno-4-6) [](https://docs.crawl4ai.com/api/arun_many/#__codelineno-4-7) # PDF files - specialized extraction [](https://docs.crawl4ai.com/api/arun_many/#__codelineno-4-8) pdf_config = CrawlerRunConfig( [](https://docs.crawl4ai.com/api/arun_many/#__codelineno-4-9) url_matcher="*.pdf", [](https://docs.crawl4ai.com/api/arun_many/#__codelineno-4-10) scraping_strategy=PDFContentScrapingStrategy() [](https://docs.crawl4ai.com/api/arun_many/#__codelineno-4-11) ) [](https://docs.crawl4ai.com/api/arun_many/#__codelineno-4-12) [](https://docs.crawl4ai.com/api/arun_many/#__codelineno-4-13) # Blog/article pages - content filtering [](https://docs.crawl4ai.com/api/arun_many/#__codelineno-4-14) blog_config = CrawlerRunConfig( [](https://docs.crawl4ai.com/api/arun_many/#__codelineno-4-15) url_matcher=["*/blog/*", "*/article/*", "*python.org*"], [](https://docs.crawl4ai.com/api/arun_many/#__codelineno-4-16) markdown_generator=DefaultMarkdownGenerator( [](https://docs.crawl4ai.com/api/arun_many/#__codelineno-4-17) content_filter=PruningContentFilter(threshold=0.48) [](https://docs.crawl4ai.com/api/arun_many/#__codelineno-4-18) ) [](https://docs.crawl4ai.com/api/arun_many/#__codelineno-4-19) ) [](https://docs.crawl4ai.com/api/arun_many/#__codelineno-4-20) [](https://docs.crawl4ai.com/api/arun_many/#__codelineno-4-21) # Dynamic pages - JavaScript execution [](https://docs.crawl4ai.com/api/arun_many/#__codelineno-4-22) github_config = CrawlerRunConfig( [](https://docs.crawl4ai.com/api/arun_many/#__codelineno-4-23) url_matcher=lambda url: 'github.com' in url, [](https://docs.crawl4ai.com/api/arun_many/#__codelineno-4-24) js_code="window.scrollTo(0, 500);" [](https://docs.crawl4ai.com/api/arun_many/#__codelineno-4-25) ) [](https://docs.crawl4ai.com/api/arun_many/#__codelineno-4-26) [](https://docs.crawl4ai.com/api/arun_many/#__codelineno-4-27) # API endpoints - JSON extraction [](https://docs.crawl4ai.com/api/arun_many/#__codelineno-4-28) api_config = CrawlerRunConfig( [](https://docs.crawl4ai.com/api/arun_many/#__codelineno-4-29) url_matcher=lambda url: 'api' in url or url.endswith('.json'), [](https://docs.crawl4ai.com/api/arun_many/#__codelineno-4-30) # Custome settings for JSON extraction [](https://docs.crawl4ai.com/api/arun_many/#__codelineno-4-31) ) [](https://docs.crawl4ai.com/api/arun_many/#__codelineno-4-32) [](https://docs.crawl4ai.com/api/arun_many/#__codelineno-4-33) # Default fallback config [](https://docs.crawl4ai.com/api/arun_many/#__codelineno-4-34) default_config = CrawlerRunConfig() # No url_matcher means it never matches except as fallback [](https://docs.crawl4ai.com/api/arun_many/#__codelineno-4-35) [](https://docs.crawl4ai.com/api/arun_many/#__codelineno-4-36) # Pass the list of configs - first match wins! [](https://docs.crawl4ai.com/api/arun_many/#__codelineno-4-37) results = await crawler.arun_many( [](https://docs.crawl4ai.com/api/arun_many/#__codelineno-4-38) urls=[ [](https://docs.crawl4ai.com/api/arun_many/#__codelineno-4-39) "https://www.w3.org/WAI/ER/tests/xhtml/testfiles/resources/pdf/dummy.pdf", # → pdf_config [](https://docs.crawl4ai.com/api/arun_many/#__codelineno-4-40) "https://blog.python.org/", # → blog_config [](https://docs.crawl4ai.com/api/arun_many/#__codelineno-4-41) "https://github.com/microsoft/playwright", # → github_config [](https://docs.crawl4ai.com/api/arun_many/#__codelineno-4-42) "https://httpbin.org/json", # → api_config [](https://docs.crawl4ai.com/api/arun_many/#__codelineno-4-43) "https://example.com/" # → default_config [](https://docs.crawl4ai.com/api/arun_many/#__codelineno-4-44) ], [](https://docs.crawl4ai.com/api/arun_many/#__codelineno-4-45) config=[pdf_config, blog_config, github_config, api_config, default_config] [](https://docs.crawl4ai.com/api/arun_many/#__codelineno-4-46) )` **URL Matching Features**: - **String patterns**: `"*.pdf"`, `"*/blog/*"`, `"*python.org*"` - **Function matchers**: `lambda url: 'api' in url` - **Mixed patterns**: Combine strings and functions with `MatchMode.OR` or `MatchMode.AND` - **First match wins**: Configs are evaluated in order **Key Points**: - Each URL is processed by the same or separate sessions, depending on the dispatcher’s strategy. - `dispatch_result` in each `CrawlResult` (if using concurrency) can hold memory and timing info.  - If you need to handle authentication or session IDs, pass them in each individual task or within your run config. - **Important**: Always include a default config (without `url_matcher`) as the last item if you want to handle all URLs. Otherwise, unmatched URLs will fail. ### Return Value Either a **list** of [`CrawlResult`](https://docs.crawl4ai.com/api/crawl-result/) objects, or an **async generator** if streaming is enabled. You can iterate to check `result.success` or read each item’s `extracted_content`, `markdown`, or `dispatch_result`. * * * Dispatcher Reference -------------------- * **`MemoryAdaptiveDispatcher`**: Dynamically manages concurrency based on system memory usage.  * **`SemaphoreDispatcher`**: Fixed concurrency limit, simpler but less adaptive.  For advanced usage or custom settings, see [Multi-URL Crawling with Dispatchers](https://docs.crawl4ai.com/advanced/multi-url-crawling/) . * * * Common Pitfalls --------------- 1. **Large Lists**: If you pass thousands of URLs, be mindful of memory or rate-limits. A dispatcher can help.  2. **Session Reuse**: If you need specialized logins or persistent contexts, ensure your dispatcher or tasks handle sessions accordingly.  3. **Error Handling**: Each `CrawlResult` might fail for different reasons—always check `result.success` or the `error_message` before proceeding. * * * Conclusion ---------- Use `arun_many()` when you want to **crawl multiple URLs** simultaneously or in controlled parallel tasks. If you need advanced concurrency features (like memory-based adaptive throttling or complex rate-limiting), provide a **dispatcher**. Each result is a standard `CrawlResult`, possibly augmented with concurrency stats (`dispatch_result`) for deeper inspection. For more details on concurrency logic and dispatchers, see the [Advanced Multi-URL Crawling](https://docs.crawl4ai.com/advanced/multi-url-crawling/) docs. * * * \> Feedback --- # Undetected Browser - Crawl4AI Documentation (v0.8.x) Undetected Browser Mode ======================= Overview -------- Crawl4AI offers two powerful anti-bot features to help you access websites with bot detection: 1. **Stealth Mode** - Uses playwright-stealth to modify browser fingerprints and behaviors 2. **Undetected Browser Mode** - Advanced browser adapter with deep-level patches for sophisticated bot detection This guide covers both features and helps you choose the right approach for your needs. Anti-Bot Features Comparison ---------------------------- | Feature | Regular Browser | Stealth Mode | Undetected Browser | | --- | --- | --- | --- | | WebDriver Detection | ❌ | ✅ | ✅ | | Navigator Properties | ❌ | ✅ | ✅ | | Plugin Emulation | ❌ | ✅ | ✅ | | CDP Detection | ❌ | Partial | ✅ | | Deep Browser Patches | ❌ | ❌ | ✅ | | Performance Impact | None | Minimal | Moderate | | Setup Complexity | None | None | Minimal | When to Use Each Approach ------------------------- ### Use Regular Browser + Stealth Mode When: * Sites have basic bot detection (checking navigator.webdriver, plugins, etc.) * You need good performance with basic protection * Sites check for common automation indicators ### Use Undetected Browser When: * Sites employ sophisticated bot detection services (Cloudflare, DataDome, etc.) * Stealth mode alone isn't sufficient * You're willing to trade some performance for better evasion ### Best Practice: Progressive Enhancement 1. **Start with**: Regular browser + Stealth mode 2. **If blocked**: Switch to Undetected browser 3. **If still blocked**: Combine Undetected browser + Stealth mode Stealth Mode ------------ Stealth mode is the simpler anti-bot solution that works with both regular and undetected browsers: `[](https://docs.crawl4ai.com/advanced/undetected-browser/#__codelineno-0-1) from crawl4ai import AsyncWebCrawler, BrowserConfig [](https://docs.crawl4ai.com/advanced/undetected-browser/#__codelineno-0-2) [](https://docs.crawl4ai.com/advanced/undetected-browser/#__codelineno-0-3) # Enable stealth mode with regular browser [](https://docs.crawl4ai.com/advanced/undetected-browser/#__codelineno-0-4) browser_config = BrowserConfig( [](https://docs.crawl4ai.com/advanced/undetected-browser/#__codelineno-0-5) enable_stealth=True, # Simple flag to enable [](https://docs.crawl4ai.com/advanced/undetected-browser/#__codelineno-0-6) headless=False # Better for avoiding detection [](https://docs.crawl4ai.com/advanced/undetected-browser/#__codelineno-0-7) ) [](https://docs.crawl4ai.com/advanced/undetected-browser/#__codelineno-0-8) [](https://docs.crawl4ai.com/advanced/undetected-browser/#__codelineno-0-9) async with AsyncWebCrawler(config=browser_config) as crawler: [](https://docs.crawl4ai.com/advanced/undetected-browser/#__codelineno-0-10) result = await crawler.arun("https://example.com")` ### What Stealth Mode Does: * Removes `navigator.webdriver` flag * Modifies browser fingerprints * Emulates realistic plugin behavior * Adjusts navigator properties * Fixes common automation leaks Undetected Browser Mode ----------------------- For sites with sophisticated bot detection that stealth mode can't bypass, use the undetected browser adapter: ### Key Features * **Drop-in Replacement**: Uses the same API as regular browser mode * **Enhanced Stealth**: Built-in patches to evade common detection methods * **Browser Adapter Pattern**: Seamlessly switch between regular and undetected modes * **Automatic Installation**: `crawl4ai-setup` installs all necessary browser dependencies ### Quick Start `[](https://docs.crawl4ai.com/advanced/undetected-browser/#__codelineno-1-1) import asyncio [](https://docs.crawl4ai.com/advanced/undetected-browser/#__codelineno-1-2) from crawl4ai import ( [](https://docs.crawl4ai.com/advanced/undetected-browser/#__codelineno-1-3) AsyncWebCrawler, [](https://docs.crawl4ai.com/advanced/undetected-browser/#__codelineno-1-4) BrowserConfig, [](https://docs.crawl4ai.com/advanced/undetected-browser/#__codelineno-1-5) CrawlerRunConfig, [](https://docs.crawl4ai.com/advanced/undetected-browser/#__codelineno-1-6) UndetectedAdapter [](https://docs.crawl4ai.com/advanced/undetected-browser/#__codelineno-1-7) ) [](https://docs.crawl4ai.com/advanced/undetected-browser/#__codelineno-1-8) from crawl4ai.async_crawler_strategy import AsyncPlaywrightCrawlerStrategy [](https://docs.crawl4ai.com/advanced/undetected-browser/#__codelineno-1-9) [](https://docs.crawl4ai.com/advanced/undetected-browser/#__codelineno-1-10) async def main(): [](https://docs.crawl4ai.com/advanced/undetected-browser/#__codelineno-1-11) # Create the undetected adapter [](https://docs.crawl4ai.com/advanced/undetected-browser/#__codelineno-1-12) undetected_adapter = UndetectedAdapter() [](https://docs.crawl4ai.com/advanced/undetected-browser/#__codelineno-1-13) [](https://docs.crawl4ai.com/advanced/undetected-browser/#__codelineno-1-14) # Create browser config [](https://docs.crawl4ai.com/advanced/undetected-browser/#__codelineno-1-15) browser_config = BrowserConfig( [](https://docs.crawl4ai.com/advanced/undetected-browser/#__codelineno-1-16) headless=False, # Headless mode can be detected easier [](https://docs.crawl4ai.com/advanced/undetected-browser/#__codelineno-1-17) verbose=True, [](https://docs.crawl4ai.com/advanced/undetected-browser/#__codelineno-1-18) ) [](https://docs.crawl4ai.com/advanced/undetected-browser/#__codelineno-1-19) [](https://docs.crawl4ai.com/advanced/undetected-browser/#__codelineno-1-20) # Create the crawler strategy with undetected adapter [](https://docs.crawl4ai.com/advanced/undetected-browser/#__codelineno-1-21) crawler_strategy = AsyncPlaywrightCrawlerStrategy( [](https://docs.crawl4ai.com/advanced/undetected-browser/#__codelineno-1-22) browser_config=browser_config, [](https://docs.crawl4ai.com/advanced/undetected-browser/#__codelineno-1-23) browser_adapter=undetected_adapter [](https://docs.crawl4ai.com/advanced/undetected-browser/#__codelineno-1-24) ) [](https://docs.crawl4ai.com/advanced/undetected-browser/#__codelineno-1-25) [](https://docs.crawl4ai.com/advanced/undetected-browser/#__codelineno-1-26) # Create the crawler with our custom strategy [](https://docs.crawl4ai.com/advanced/undetected-browser/#__codelineno-1-27) async with AsyncWebCrawler( [](https://docs.crawl4ai.com/advanced/undetected-browser/#__codelineno-1-28) crawler_strategy=crawler_strategy, [](https://docs.crawl4ai.com/advanced/undetected-browser/#__codelineno-1-29) config=browser_config [](https://docs.crawl4ai.com/advanced/undetected-browser/#__codelineno-1-30) ) as crawler: [](https://docs.crawl4ai.com/advanced/undetected-browser/#__codelineno-1-31) # Your crawling code here [](https://docs.crawl4ai.com/advanced/undetected-browser/#__codelineno-1-32) result = await crawler.arun( [](https://docs.crawl4ai.com/advanced/undetected-browser/#__codelineno-1-33) url="https://example.com", [](https://docs.crawl4ai.com/advanced/undetected-browser/#__codelineno-1-34) config=CrawlerRunConfig() [](https://docs.crawl4ai.com/advanced/undetected-browser/#__codelineno-1-35) ) [](https://docs.crawl4ai.com/advanced/undetected-browser/#__codelineno-1-36) print(result.markdown[:500]) [](https://docs.crawl4ai.com/advanced/undetected-browser/#__codelineno-1-37) [](https://docs.crawl4ai.com/advanced/undetected-browser/#__codelineno-1-38) asyncio.run(main())` Combining Both Features ----------------------- For maximum evasion, combine stealth mode with undetected browser: `[](https://docs.crawl4ai.com/advanced/undetected-browser/#__codelineno-2-1) from crawl4ai import AsyncWebCrawler, BrowserConfig, UndetectedAdapter [](https://docs.crawl4ai.com/advanced/undetected-browser/#__codelineno-2-2) from crawl4ai.async_crawler_strategy import AsyncPlaywrightCrawlerStrategy [](https://docs.crawl4ai.com/advanced/undetected-browser/#__codelineno-2-3) [](https://docs.crawl4ai.com/advanced/undetected-browser/#__codelineno-2-4) # Create browser config with stealth enabled [](https://docs.crawl4ai.com/advanced/undetected-browser/#__codelineno-2-5) browser_config = BrowserConfig( [](https://docs.crawl4ai.com/advanced/undetected-browser/#__codelineno-2-6) enable_stealth=True, # Enable stealth mode [](https://docs.crawl4ai.com/advanced/undetected-browser/#__codelineno-2-7) headless=False [](https://docs.crawl4ai.com/advanced/undetected-browser/#__codelineno-2-8) ) [](https://docs.crawl4ai.com/advanced/undetected-browser/#__codelineno-2-9) [](https://docs.crawl4ai.com/advanced/undetected-browser/#__codelineno-2-10) # Create undetected adapter [](https://docs.crawl4ai.com/advanced/undetected-browser/#__codelineno-2-11) adapter = UndetectedAdapter() [](https://docs.crawl4ai.com/advanced/undetected-browser/#__codelineno-2-12) [](https://docs.crawl4ai.com/advanced/undetected-browser/#__codelineno-2-13) # Create strategy with both features [](https://docs.crawl4ai.com/advanced/undetected-browser/#__codelineno-2-14) strategy = AsyncPlaywrightCrawlerStrategy( [](https://docs.crawl4ai.com/advanced/undetected-browser/#__codelineno-2-15) browser_config=browser_config, [](https://docs.crawl4ai.com/advanced/undetected-browser/#__codelineno-2-16) browser_adapter=adapter [](https://docs.crawl4ai.com/advanced/undetected-browser/#__codelineno-2-17) ) [](https://docs.crawl4ai.com/advanced/undetected-browser/#__codelineno-2-18) [](https://docs.crawl4ai.com/advanced/undetected-browser/#__codelineno-2-19) async with AsyncWebCrawler( [](https://docs.crawl4ai.com/advanced/undetected-browser/#__codelineno-2-20) crawler_strategy=strategy, [](https://docs.crawl4ai.com/advanced/undetected-browser/#__codelineno-2-21) config=browser_config [](https://docs.crawl4ai.com/advanced/undetected-browser/#__codelineno-2-22) ) as crawler: [](https://docs.crawl4ai.com/advanced/undetected-browser/#__codelineno-2-23) result = await crawler.arun("https://protected-site.com")` Examples -------- ### Example 1: Basic Stealth Mode `[](https://docs.crawl4ai.com/advanced/undetected-browser/#__codelineno-3-1) import asyncio [](https://docs.crawl4ai.com/advanced/undetected-browser/#__codelineno-3-2) from crawl4ai import AsyncWebCrawler, BrowserConfig, CrawlerRunConfig [](https://docs.crawl4ai.com/advanced/undetected-browser/#__codelineno-3-3) [](https://docs.crawl4ai.com/advanced/undetected-browser/#__codelineno-3-4) async def test_stealth_mode(): [](https://docs.crawl4ai.com/advanced/undetected-browser/#__codelineno-3-5) # Simple stealth mode configuration [](https://docs.crawl4ai.com/advanced/undetected-browser/#__codelineno-3-6) browser_config = BrowserConfig( [](https://docs.crawl4ai.com/advanced/undetected-browser/#__codelineno-3-7) enable_stealth=True, [](https://docs.crawl4ai.com/advanced/undetected-browser/#__codelineno-3-8) headless=False [](https://docs.crawl4ai.com/advanced/undetected-browser/#__codelineno-3-9) ) [](https://docs.crawl4ai.com/advanced/undetected-browser/#__codelineno-3-10) [](https://docs.crawl4ai.com/advanced/undetected-browser/#__codelineno-3-11) async with AsyncWebCrawler(config=browser_config) as crawler: [](https://docs.crawl4ai.com/advanced/undetected-browser/#__codelineno-3-12) result = await crawler.arun( [](https://docs.crawl4ai.com/advanced/undetected-browser/#__codelineno-3-13) url="https://bot.sannysoft.com", [](https://docs.crawl4ai.com/advanced/undetected-browser/#__codelineno-3-14) config=CrawlerRunConfig(screenshot=True) [](https://docs.crawl4ai.com/advanced/undetected-browser/#__codelineno-3-15) ) [](https://docs.crawl4ai.com/advanced/undetected-browser/#__codelineno-3-16) [](https://docs.crawl4ai.com/advanced/undetected-browser/#__codelineno-3-17) if result.success: [](https://docs.crawl4ai.com/advanced/undetected-browser/#__codelineno-3-18) print("✓ Successfully accessed bot detection test site") [](https://docs.crawl4ai.com/advanced/undetected-browser/#__codelineno-3-19) # Save screenshot to verify detection results [](https://docs.crawl4ai.com/advanced/undetected-browser/#__codelineno-3-20) if result.screenshot: [](https://docs.crawl4ai.com/advanced/undetected-browser/#__codelineno-3-21) import base64 [](https://docs.crawl4ai.com/advanced/undetected-browser/#__codelineno-3-22) with open("stealth_test.png", "wb") as f: [](https://docs.crawl4ai.com/advanced/undetected-browser/#__codelineno-3-23) f.write(base64.b64decode(result.screenshot)) [](https://docs.crawl4ai.com/advanced/undetected-browser/#__codelineno-3-24) print("✓ Screenshot saved - check for green (passed) tests") [](https://docs.crawl4ai.com/advanced/undetected-browser/#__codelineno-3-25) [](https://docs.crawl4ai.com/advanced/undetected-browser/#__codelineno-3-26) asyncio.run(test_stealth_mode())` ### Example 2: Undetected Browser Mode `[](https://docs.crawl4ai.com/advanced/undetected-browser/#__codelineno-4-1) import asyncio [](https://docs.crawl4ai.com/advanced/undetected-browser/#__codelineno-4-2) from crawl4ai import ( [](https://docs.crawl4ai.com/advanced/undetected-browser/#__codelineno-4-3) AsyncWebCrawler, [](https://docs.crawl4ai.com/advanced/undetected-browser/#__codelineno-4-4) BrowserConfig, [](https://docs.crawl4ai.com/advanced/undetected-browser/#__codelineno-4-5) CrawlerRunConfig, [](https://docs.crawl4ai.com/advanced/undetected-browser/#__codelineno-4-6) UndetectedAdapter [](https://docs.crawl4ai.com/advanced/undetected-browser/#__codelineno-4-7) ) [](https://docs.crawl4ai.com/advanced/undetected-browser/#__codelineno-4-8) from crawl4ai.async_crawler_strategy import AsyncPlaywrightCrawlerStrategy [](https://docs.crawl4ai.com/advanced/undetected-browser/#__codelineno-4-9) [](https://docs.crawl4ai.com/advanced/undetected-browser/#__codelineno-4-10) [](https://docs.crawl4ai.com/advanced/undetected-browser/#__codelineno-4-11) async def main(): [](https://docs.crawl4ai.com/advanced/undetected-browser/#__codelineno-4-12) # Create browser config [](https://docs.crawl4ai.com/advanced/undetected-browser/#__codelineno-4-13) browser_config = BrowserConfig( [](https://docs.crawl4ai.com/advanced/undetected-browser/#__codelineno-4-14) headless=False, [](https://docs.crawl4ai.com/advanced/undetected-browser/#__codelineno-4-15) verbose=True, [](https://docs.crawl4ai.com/advanced/undetected-browser/#__codelineno-4-16) ) [](https://docs.crawl4ai.com/advanced/undetected-browser/#__codelineno-4-17) [](https://docs.crawl4ai.com/advanced/undetected-browser/#__codelineno-4-18) # Create the undetected adapter [](https://docs.crawl4ai.com/advanced/undetected-browser/#__codelineno-4-19) undetected_adapter = UndetectedAdapter() [](https://docs.crawl4ai.com/advanced/undetected-browser/#__codelineno-4-20) [](https://docs.crawl4ai.com/advanced/undetected-browser/#__codelineno-4-21) # Create the crawler strategy with the undetected adapter [](https://docs.crawl4ai.com/advanced/undetected-browser/#__codelineno-4-22) crawler_strategy = AsyncPlaywrightCrawlerStrategy( [](https://docs.crawl4ai.com/advanced/undetected-browser/#__codelineno-4-23) browser_config=browser_config, [](https://docs.crawl4ai.com/advanced/undetected-browser/#__codelineno-4-24) browser_adapter=undetected_adapter [](https://docs.crawl4ai.com/advanced/undetected-browser/#__codelineno-4-25) ) [](https://docs.crawl4ai.com/advanced/undetected-browser/#__codelineno-4-26) [](https://docs.crawl4ai.com/advanced/undetected-browser/#__codelineno-4-27) # Create the crawler with our custom strategy [](https://docs.crawl4ai.com/advanced/undetected-browser/#__codelineno-4-28) async with AsyncWebCrawler( [](https://docs.crawl4ai.com/advanced/undetected-browser/#__codelineno-4-29) crawler_strategy=crawler_strategy, [](https://docs.crawl4ai.com/advanced/undetected-browser/#__codelineno-4-30) config=browser_config [](https://docs.crawl4ai.com/advanced/undetected-browser/#__codelineno-4-31) ) as crawler: [](https://docs.crawl4ai.com/advanced/undetected-browser/#__codelineno-4-32) # Configure the crawl [](https://docs.crawl4ai.com/advanced/undetected-browser/#__codelineno-4-33) crawler_config = CrawlerRunConfig( [](https://docs.crawl4ai.com/advanced/undetected-browser/#__codelineno-4-34) markdown_generator=DefaultMarkdownGenerator( [](https://docs.crawl4ai.com/advanced/undetected-browser/#__codelineno-4-35) content_filter=PruningContentFilter() [](https://docs.crawl4ai.com/advanced/undetected-browser/#__codelineno-4-36) ), [](https://docs.crawl4ai.com/advanced/undetected-browser/#__codelineno-4-37) capture_console_messages=True, # Test adapter console capture [](https://docs.crawl4ai.com/advanced/undetected-browser/#__codelineno-4-38) ) [](https://docs.crawl4ai.com/advanced/undetected-browser/#__codelineno-4-39) [](https://docs.crawl4ai.com/advanced/undetected-browser/#__codelineno-4-40) # Test on a site that typically detects bots [](https://docs.crawl4ai.com/advanced/undetected-browser/#__codelineno-4-41) print("Testing undetected adapter...") [](https://docs.crawl4ai.com/advanced/undetected-browser/#__codelineno-4-42) result: CrawlResult = await crawler.arun( [](https://docs.crawl4ai.com/advanced/undetected-browser/#__codelineno-4-43) url="https://www.helloworld.org", [](https://docs.crawl4ai.com/advanced/undetected-browser/#__codelineno-4-44) config=crawler_config [](https://docs.crawl4ai.com/advanced/undetected-browser/#__codelineno-4-45) ) [](https://docs.crawl4ai.com/advanced/undetected-browser/#__codelineno-4-46) [](https://docs.crawl4ai.com/advanced/undetected-browser/#__codelineno-4-47) print(f"Status: {result.status_code}") [](https://docs.crawl4ai.com/advanced/undetected-browser/#__codelineno-4-48) print(f"Success: {result.success}") [](https://docs.crawl4ai.com/advanced/undetected-browser/#__codelineno-4-49) print(f"Console messages captured: {len(result.console_messages or [])}") [](https://docs.crawl4ai.com/advanced/undetected-browser/#__codelineno-4-50) print(f"Markdown content (first 500 chars):\n{result.markdown.raw_markdown[:500]}") [](https://docs.crawl4ai.com/advanced/undetected-browser/#__codelineno-4-51) [](https://docs.crawl4ai.com/advanced/undetected-browser/#__codelineno-4-52) [](https://docs.crawl4ai.com/advanced/undetected-browser/#__codelineno-4-53) if __name__ == "__main__": [](https://docs.crawl4ai.com/advanced/undetected-browser/#__codelineno-4-54) asyncio.run(main())` Browser Adapter Pattern ----------------------- The undetected browser support is implemented using an adapter pattern, allowing seamless switching between different browser implementations: `[](https://docs.crawl4ai.com/advanced/undetected-browser/#__codelineno-5-1) # Regular browser adapter (default) [](https://docs.crawl4ai.com/advanced/undetected-browser/#__codelineno-5-2) from crawl4ai import PlaywrightAdapter [](https://docs.crawl4ai.com/advanced/undetected-browser/#__codelineno-5-3) regular_adapter = PlaywrightAdapter() [](https://docs.crawl4ai.com/advanced/undetected-browser/#__codelineno-5-4) [](https://docs.crawl4ai.com/advanced/undetected-browser/#__codelineno-5-5) # Undetected browser adapter [](https://docs.crawl4ai.com/advanced/undetected-browser/#__codelineno-5-6) from crawl4ai import UndetectedAdapter [](https://docs.crawl4ai.com/advanced/undetected-browser/#__codelineno-5-7) undetected_adapter = UndetectedAdapter()` The adapter handles: - JavaScript execution - Console message capture - Error handling - Browser-specific optimizations Best Practices -------------- 1. **Avoid Headless Mode**: Detection is easier in headless mode `[](https://docs.crawl4ai.com/advanced/undetected-browser/#__codelineno-6-1) browser_config = BrowserConfig(headless=False)` 2. **Use Reasonable Delays**: Don't rush through pages `[](https://docs.crawl4ai.com/advanced/undetected-browser/#__codelineno-7-1) crawler_config = CrawlerRunConfig( [](https://docs.crawl4ai.com/advanced/undetected-browser/#__codelineno-7-2) wait_time=3.0, # Wait 3 seconds after page load [](https://docs.crawl4ai.com/advanced/undetected-browser/#__codelineno-7-3) delay_before_return_html=2.0 # Additional delay [](https://docs.crawl4ai.com/advanced/undetected-browser/#__codelineno-7-4) )` 3. **Rotate User Agents**: You can customize user agents `[](https://docs.crawl4ai.com/advanced/undetected-browser/#__codelineno-8-1) browser_config = BrowserConfig( [](https://docs.crawl4ai.com/advanced/undetected-browser/#__codelineno-8-2) headers={"User-Agent": "your-user-agent"} [](https://docs.crawl4ai.com/advanced/undetected-browser/#__codelineno-8-3) )` 4. **Handle Failures Gracefully**: Some sites may still detect and block `[](https://docs.crawl4ai.com/advanced/undetected-browser/#__codelineno-9-1) if not result.success: [](https://docs.crawl4ai.com/advanced/undetected-browser/#__codelineno-9-2) print(f"Crawl failed: {result.error_message}")` Advanced Usage Tips ------------------- ### Progressive Detection Handling `[](https://docs.crawl4ai.com/advanced/undetected-browser/#__codelineno-10-1) async def crawl_with_progressive_evasion(url): [](https://docs.crawl4ai.com/advanced/undetected-browser/#__codelineno-10-2) # Step 1: Try regular browser with stealth [](https://docs.crawl4ai.com/advanced/undetected-browser/#__codelineno-10-3) browser_config = BrowserConfig( [](https://docs.crawl4ai.com/advanced/undetected-browser/#__codelineno-10-4) enable_stealth=True, [](https://docs.crawl4ai.com/advanced/undetected-browser/#__codelineno-10-5) headless=False [](https://docs.crawl4ai.com/advanced/undetected-browser/#__codelineno-10-6) ) [](https://docs.crawl4ai.com/advanced/undetected-browser/#__codelineno-10-7) [](https://docs.crawl4ai.com/advanced/undetected-browser/#__codelineno-10-8) async with AsyncWebCrawler(config=browser_config) as crawler: [](https://docs.crawl4ai.com/advanced/undetected-browser/#__codelineno-10-9) result = await crawler.arun(url) [](https://docs.crawl4ai.com/advanced/undetected-browser/#__codelineno-10-10) if result.success and "Access Denied" not in result.html: [](https://docs.crawl4ai.com/advanced/undetected-browser/#__codelineno-10-11) return result [](https://docs.crawl4ai.com/advanced/undetected-browser/#__codelineno-10-12) [](https://docs.crawl4ai.com/advanced/undetected-browser/#__codelineno-10-13) # Step 2: If blocked, try undetected browser [](https://docs.crawl4ai.com/advanced/undetected-browser/#__codelineno-10-14) print("Regular + stealth blocked, trying undetected browser...") [](https://docs.crawl4ai.com/advanced/undetected-browser/#__codelineno-10-15) [](https://docs.crawl4ai.com/advanced/undetected-browser/#__codelineno-10-16) adapter = UndetectedAdapter() [](https://docs.crawl4ai.com/advanced/undetected-browser/#__codelineno-10-17) strategy = AsyncPlaywrightCrawlerStrategy( [](https://docs.crawl4ai.com/advanced/undetected-browser/#__codelineno-10-18) browser_config=browser_config, [](https://docs.crawl4ai.com/advanced/undetected-browser/#__codelineno-10-19) browser_adapter=adapter [](https://docs.crawl4ai.com/advanced/undetected-browser/#__codelineno-10-20) ) [](https://docs.crawl4ai.com/advanced/undetected-browser/#__codelineno-10-21) [](https://docs.crawl4ai.com/advanced/undetected-browser/#__codelineno-10-22) async with AsyncWebCrawler( [](https://docs.crawl4ai.com/advanced/undetected-browser/#__codelineno-10-23) crawler_strategy=strategy, [](https://docs.crawl4ai.com/advanced/undetected-browser/#__codelineno-10-24) config=browser_config [](https://docs.crawl4ai.com/advanced/undetected-browser/#__codelineno-10-25) ) as crawler: [](https://docs.crawl4ai.com/advanced/undetected-browser/#__codelineno-10-26) result = await crawler.arun(url) [](https://docs.crawl4ai.com/advanced/undetected-browser/#__codelineno-10-27) return result` Installation ------------ The undetected browser dependencies are automatically installed when you run: `[](https://docs.crawl4ai.com/advanced/undetected-browser/#__codelineno-11-1) crawl4ai-setup` This command installs all necessary browser dependencies for both regular and undetected modes. Limitations ----------- * **Performance**: Slightly slower than regular mode due to additional patches * **Headless Detection**: Some sites can still detect headless mode * **Resource Usage**: May use more resources than regular mode * **Not 100% Guaranteed**: Advanced anti-bot services are constantly evolving Troubleshooting --------------- ### Browser Not Found Run the setup command: `[](https://docs.crawl4ai.com/advanced/undetected-browser/#__codelineno-12-1) crawl4ai-setup` ### Detection Still Occurring Try combining with other features: `[](https://docs.crawl4ai.com/advanced/undetected-browser/#__codelineno-13-1) crawler_config = CrawlerRunConfig( [](https://docs.crawl4ai.com/advanced/undetected-browser/#__codelineno-13-2) simulate_user=True, # Add user simulation [](https://docs.crawl4ai.com/advanced/undetected-browser/#__codelineno-13-3) magic=True, # Enable magic mode [](https://docs.crawl4ai.com/advanced/undetected-browser/#__codelineno-13-4) wait_time=5.0, # Longer waits [](https://docs.crawl4ai.com/advanced/undetected-browser/#__codelineno-13-5) )` ### Performance Issues If experiencing slow performance: `[](https://docs.crawl4ai.com/advanced/undetected-browser/#__codelineno-14-1) # Use selective undetected mode only for protected sites [](https://docs.crawl4ai.com/advanced/undetected-browser/#__codelineno-14-2) if is_protected_site(url): [](https://docs.crawl4ai.com/advanced/undetected-browser/#__codelineno-14-3) adapter = UndetectedAdapter() [](https://docs.crawl4ai.com/advanced/undetected-browser/#__codelineno-14-4) else: [](https://docs.crawl4ai.com/advanced/undetected-browser/#__codelineno-14-5) adapter = PlaywrightAdapter() # Default adapter` Future Plans ------------ **Note**: In future versions of Crawl4AI, we may enable stealth mode and undetected browser by default to provide better out-of-the-box success rates. For now, users should explicitly enable these features when needed. Conclusion ---------- Crawl4AI provides flexible anti-bot solutions: 1. **Start Simple**: Use regular browser + stealth mode for most sites 2. **Escalate if Needed**: Switch to undetected browser for sophisticated protection 3. **Combine for Maximum Effect**: Use both features together when facing the toughest challenges Remember: - Always respect robots.txt and website terms of service - Use appropriate delays to avoid overwhelming servers - Consider the performance trade-offs of each approach - Test progressively to find the minimum necessary evasion level See Also -------- * [Advanced Features](https://docs.crawl4ai.com/advanced/advanced-features/) - Overview of all advanced features * [Proxy & Security](https://docs.crawl4ai.com/advanced/proxy-security/) - Using proxies with anti-bot features * [Session Management](https://docs.crawl4ai.com/advanced/session-management/) - Maintaining sessions across requests * [Identity Based Crawling](https://docs.crawl4ai.com/advanced/identity-based-crawling/) - Additional anti-detection strategies * [Anti-Bot Detection & Fallback](https://docs.crawl4ai.com/advanced/anti-bot-and-fallback/) - Automatic retry and proxy escalation when blocking is detected * * * \> Feedback --- # Session Management - Crawl4AI Documentation (v0.8.x) Session Management ================== Session management in Crawl4AI is a powerful feature that allows you to maintain state across multiple requests, making it particularly suitable for handling complex multi-step crawling tasks. It enables you to reuse the same browser tab (or page object) across sequential actions and crawls, which is beneficial for: * **Performing JavaScript actions before and after crawling.** * **Executing multiple sequential crawls faster** without needing to reopen tabs or allocate memory repeatedly. **Note:** This feature is designed for sequential workflows and is not suitable for parallel operations. * * * #### Basic Session Usage Use `BrowserConfig` and `CrawlerRunConfig` to maintain state with a `session_id`: `[](https://docs.crawl4ai.com/advanced/session-management/#__codelineno-0-1) from crawl4ai.async_configs import BrowserConfig, CrawlerRunConfig [](https://docs.crawl4ai.com/advanced/session-management/#__codelineno-0-2) [](https://docs.crawl4ai.com/advanced/session-management/#__codelineno-0-3) async with AsyncWebCrawler() as crawler: [](https://docs.crawl4ai.com/advanced/session-management/#__codelineno-0-4) session_id = "my_session" [](https://docs.crawl4ai.com/advanced/session-management/#__codelineno-0-5) [](https://docs.crawl4ai.com/advanced/session-management/#__codelineno-0-6) # Define configurations [](https://docs.crawl4ai.com/advanced/session-management/#__codelineno-0-7) config1 = CrawlerRunConfig( [](https://docs.crawl4ai.com/advanced/session-management/#__codelineno-0-8) url="https://example.com/page1", session_id=session_id [](https://docs.crawl4ai.com/advanced/session-management/#__codelineno-0-9) ) [](https://docs.crawl4ai.com/advanced/session-management/#__codelineno-0-10) config2 = CrawlerRunConfig( [](https://docs.crawl4ai.com/advanced/session-management/#__codelineno-0-11) url="https://example.com/page2", session_id=session_id [](https://docs.crawl4ai.com/advanced/session-management/#__codelineno-0-12) ) [](https://docs.crawl4ai.com/advanced/session-management/#__codelineno-0-13) [](https://docs.crawl4ai.com/advanced/session-management/#__codelineno-0-14) # First request [](https://docs.crawl4ai.com/advanced/session-management/#__codelineno-0-15) result1 = await crawler.arun(config=config1) [](https://docs.crawl4ai.com/advanced/session-management/#__codelineno-0-16) [](https://docs.crawl4ai.com/advanced/session-management/#__codelineno-0-17) # Subsequent request using the same session [](https://docs.crawl4ai.com/advanced/session-management/#__codelineno-0-18) result2 = await crawler.arun(config=config2) [](https://docs.crawl4ai.com/advanced/session-management/#__codelineno-0-19) [](https://docs.crawl4ai.com/advanced/session-management/#__codelineno-0-20) # Clean up when done [](https://docs.crawl4ai.com/advanced/session-management/#__codelineno-0-21) await crawler.crawler_strategy.kill_session(session_id)` * * * #### Dynamic Content with Sessions Here's an example of crawling GitHub commits across multiple pages while preserving session state: `[](https://docs.crawl4ai.com/advanced/session-management/#__codelineno-1-1) from crawl4ai.async_configs import CrawlerRunConfig [](https://docs.crawl4ai.com/advanced/session-management/#__codelineno-1-2) from crawl4ai import JsonCssExtractionStrategy [](https://docs.crawl4ai.com/advanced/session-management/#__codelineno-1-3) from crawl4ai.cache_context import CacheMode [](https://docs.crawl4ai.com/advanced/session-management/#__codelineno-1-4) [](https://docs.crawl4ai.com/advanced/session-management/#__codelineno-1-5) async def crawl_dynamic_content(): [](https://docs.crawl4ai.com/advanced/session-management/#__codelineno-1-6) url = "https://github.com/microsoft/TypeScript/commits/main" [](https://docs.crawl4ai.com/advanced/session-management/#__codelineno-1-7) session_id = "wait_for_session" [](https://docs.crawl4ai.com/advanced/session-management/#__codelineno-1-8) all_commits = [] [](https://docs.crawl4ai.com/advanced/session-management/#__codelineno-1-9) [](https://docs.crawl4ai.com/advanced/session-management/#__codelineno-1-10) js_next_page = """ [](https://docs.crawl4ai.com/advanced/session-management/#__codelineno-1-11) const commits = document.querySelectorAll('li[data-testid="commit-row-item"] h4'); [](https://docs.crawl4ai.com/advanced/session-management/#__codelineno-1-12) if (commits.length > 0) { [](https://docs.crawl4ai.com/advanced/session-management/#__codelineno-1-13) window.lastCommit = commits[0].textContent.trim(); [](https://docs.crawl4ai.com/advanced/session-management/#__codelineno-1-14) } [](https://docs.crawl4ai.com/advanced/session-management/#__codelineno-1-15) const button = document.querySelector('a[data-testid="pagination-next-button"]'); [](https://docs.crawl4ai.com/advanced/session-management/#__codelineno-1-16) if (button) {button.click(); console.log('button clicked') } [](https://docs.crawl4ai.com/advanced/session-management/#__codelineno-1-17) """ [](https://docs.crawl4ai.com/advanced/session-management/#__codelineno-1-18) [](https://docs.crawl4ai.com/advanced/session-management/#__codelineno-1-19) wait_for = """() => { [](https://docs.crawl4ai.com/advanced/session-management/#__codelineno-1-20) const commits = document.querySelectorAll('li[data-testid="commit-row-item"] h4'); [](https://docs.crawl4ai.com/advanced/session-management/#__codelineno-1-21) if (commits.length === 0) return false; [](https://docs.crawl4ai.com/advanced/session-management/#__codelineno-1-22) const firstCommit = commits[0].textContent.trim(); [](https://docs.crawl4ai.com/advanced/session-management/#__codelineno-1-23) return firstCommit !== window.lastCommit; [](https://docs.crawl4ai.com/advanced/session-management/#__codelineno-1-24) }""" [](https://docs.crawl4ai.com/advanced/session-management/#__codelineno-1-25) [](https://docs.crawl4ai.com/advanced/session-management/#__codelineno-1-26) schema = { [](https://docs.crawl4ai.com/advanced/session-management/#__codelineno-1-27) "name": "Commit Extractor", [](https://docs.crawl4ai.com/advanced/session-management/#__codelineno-1-28) "baseSelector": "li[data-testid='commit-row-item']", [](https://docs.crawl4ai.com/advanced/session-management/#__codelineno-1-29) "fields": [ [](https://docs.crawl4ai.com/advanced/session-management/#__codelineno-1-30) { [](https://docs.crawl4ai.com/advanced/session-management/#__codelineno-1-31) "name": "title", [](https://docs.crawl4ai.com/advanced/session-management/#__codelineno-1-32) "selector": "h4 a", [](https://docs.crawl4ai.com/advanced/session-management/#__codelineno-1-33) "type": "text", [](https://docs.crawl4ai.com/advanced/session-management/#__codelineno-1-34) "transform": "strip", [](https://docs.crawl4ai.com/advanced/session-management/#__codelineno-1-35) }, [](https://docs.crawl4ai.com/advanced/session-management/#__codelineno-1-36) ], [](https://docs.crawl4ai.com/advanced/session-management/#__codelineno-1-37) } [](https://docs.crawl4ai.com/advanced/session-management/#__codelineno-1-38) extraction_strategy = JsonCssExtractionStrategy(schema, verbose=True) [](https://docs.crawl4ai.com/advanced/session-management/#__codelineno-1-39) [](https://docs.crawl4ai.com/advanced/session-management/#__codelineno-1-40) [](https://docs.crawl4ai.com/advanced/session-management/#__codelineno-1-41) browser_config = BrowserConfig( [](https://docs.crawl4ai.com/advanced/session-management/#__codelineno-1-42) verbose=True, [](https://docs.crawl4ai.com/advanced/session-management/#__codelineno-1-43) headless=False, [](https://docs.crawl4ai.com/advanced/session-management/#__codelineno-1-44) ) [](https://docs.crawl4ai.com/advanced/session-management/#__codelineno-1-45) [](https://docs.crawl4ai.com/advanced/session-management/#__codelineno-1-46) async with AsyncWebCrawler(config=browser_config) as crawler: [](https://docs.crawl4ai.com/advanced/session-management/#__codelineno-1-47) for page in range(3): [](https://docs.crawl4ai.com/advanced/session-management/#__codelineno-1-48) crawler_config = CrawlerRunConfig( [](https://docs.crawl4ai.com/advanced/session-management/#__codelineno-1-49) session_id=session_id, [](https://docs.crawl4ai.com/advanced/session-management/#__codelineno-1-50) css_selector="li[data-testid='commit-row-item']", [](https://docs.crawl4ai.com/advanced/session-management/#__codelineno-1-51) extraction_strategy=extraction_strategy, [](https://docs.crawl4ai.com/advanced/session-management/#__codelineno-1-52) js_code=js_next_page if page > 0 else None, [](https://docs.crawl4ai.com/advanced/session-management/#__codelineno-1-53) wait_for=wait_for if page > 0 else None, [](https://docs.crawl4ai.com/advanced/session-management/#__codelineno-1-54) js_only=page > 0, [](https://docs.crawl4ai.com/advanced/session-management/#__codelineno-1-55) cache_mode=CacheMode.BYPASS, [](https://docs.crawl4ai.com/advanced/session-management/#__codelineno-1-56) capture_console_messages=True, [](https://docs.crawl4ai.com/advanced/session-management/#__codelineno-1-57) ) [](https://docs.crawl4ai.com/advanced/session-management/#__codelineno-1-58) [](https://docs.crawl4ai.com/advanced/session-management/#__codelineno-1-59) result = await crawler.arun(url=url, config=crawler_config) [](https://docs.crawl4ai.com/advanced/session-management/#__codelineno-1-60) [](https://docs.crawl4ai.com/advanced/session-management/#__codelineno-1-61) if result.console_messages: [](https://docs.crawl4ai.com/advanced/session-management/#__codelineno-1-62) print(f"Page {page + 1} console messages:", result.console_messages) [](https://docs.crawl4ai.com/advanced/session-management/#__codelineno-1-63) [](https://docs.crawl4ai.com/advanced/session-management/#__codelineno-1-64) if result.extracted_content: [](https://docs.crawl4ai.com/advanced/session-management/#__codelineno-1-65) # print(f"Page {page + 1} result:", result.extracted_content) [](https://docs.crawl4ai.com/advanced/session-management/#__codelineno-1-66) commits = json.loads(result.extracted_content) [](https://docs.crawl4ai.com/advanced/session-management/#__codelineno-1-67) all_commits.extend(commits) [](https://docs.crawl4ai.com/advanced/session-management/#__codelineno-1-68) print(f"Page {page + 1}: Found {len(commits)} commits") [](https://docs.crawl4ai.com/advanced/session-management/#__codelineno-1-69) else: [](https://docs.crawl4ai.com/advanced/session-management/#__codelineno-1-70) print(f"Page {page + 1}: No content extracted") [](https://docs.crawl4ai.com/advanced/session-management/#__codelineno-1-71) [](https://docs.crawl4ai.com/advanced/session-management/#__codelineno-1-72) print(f"Successfully crawled {len(all_commits)} commits across 3 pages") [](https://docs.crawl4ai.com/advanced/session-management/#__codelineno-1-73) # Clean up session [](https://docs.crawl4ai.com/advanced/session-management/#__codelineno-1-74) await crawler.crawler_strategy.kill_session(session_id)` * * * Example 1: Basic Session-Based Crawling --------------------------------------- A simple example using session-based crawling: `[](https://docs.crawl4ai.com/advanced/session-management/#__codelineno-2-1) import asyncio [](https://docs.crawl4ai.com/advanced/session-management/#__codelineno-2-2) from crawl4ai.async_configs import BrowserConfig, CrawlerRunConfig [](https://docs.crawl4ai.com/advanced/session-management/#__codelineno-2-3) from crawl4ai.cache_context import CacheMode [](https://docs.crawl4ai.com/advanced/session-management/#__codelineno-2-4) [](https://docs.crawl4ai.com/advanced/session-management/#__codelineno-2-5) async def basic_session_crawl(): [](https://docs.crawl4ai.com/advanced/session-management/#__codelineno-2-6) async with AsyncWebCrawler() as crawler: [](https://docs.crawl4ai.com/advanced/session-management/#__codelineno-2-7) session_id = "dynamic_content_session" [](https://docs.crawl4ai.com/advanced/session-management/#__codelineno-2-8) url = "https://example.com/dynamic-content" [](https://docs.crawl4ai.com/advanced/session-management/#__codelineno-2-9) [](https://docs.crawl4ai.com/advanced/session-management/#__codelineno-2-10) for page in range(3): [](https://docs.crawl4ai.com/advanced/session-management/#__codelineno-2-11) config = CrawlerRunConfig( [](https://docs.crawl4ai.com/advanced/session-management/#__codelineno-2-12) url=url, [](https://docs.crawl4ai.com/advanced/session-management/#__codelineno-2-13) session_id=session_id, [](https://docs.crawl4ai.com/advanced/session-management/#__codelineno-2-14) js_code="document.querySelector('.load-more-button').click();" if page > 0 else None, [](https://docs.crawl4ai.com/advanced/session-management/#__codelineno-2-15) css_selector=".content-item", [](https://docs.crawl4ai.com/advanced/session-management/#__codelineno-2-16) cache_mode=CacheMode.BYPASS [](https://docs.crawl4ai.com/advanced/session-management/#__codelineno-2-17) ) [](https://docs.crawl4ai.com/advanced/session-management/#__codelineno-2-18) [](https://docs.crawl4ai.com/advanced/session-management/#__codelineno-2-19) result = await crawler.arun(config=config) [](https://docs.crawl4ai.com/advanced/session-management/#__codelineno-2-20) print(f"Page {page + 1}: Found {result.extracted_content.count('.content-item')} items") [](https://docs.crawl4ai.com/advanced/session-management/#__codelineno-2-21) [](https://docs.crawl4ai.com/advanced/session-management/#__codelineno-2-22) await crawler.crawler_strategy.kill_session(session_id) [](https://docs.crawl4ai.com/advanced/session-management/#__codelineno-2-23) [](https://docs.crawl4ai.com/advanced/session-management/#__codelineno-2-24) asyncio.run(basic_session_crawl())` This example shows: 1. Reusing the same `session_id` across multiple requests. 2. Executing JavaScript to load more content dynamically. 3. Properly closing the session to free resources. * * * Advanced Technique 1: Custom Execution Hooks -------------------------------------------- > Warning: You might feel confused by the end of the next few examples 😅, so make sure you are comfortable with the order of the parts before you start this. Use custom hooks to handle complex scenarios, such as waiting for content to load dynamically: `[](https://docs.crawl4ai.com/advanced/session-management/#__codelineno-3-1) async def advanced_session_crawl_with_hooks(): [](https://docs.crawl4ai.com/advanced/session-management/#__codelineno-3-2) first_commit = "" [](https://docs.crawl4ai.com/advanced/session-management/#__codelineno-3-3) [](https://docs.crawl4ai.com/advanced/session-management/#__codelineno-3-4) async def on_execution_started(page): [](https://docs.crawl4ai.com/advanced/session-management/#__codelineno-3-5) nonlocal first_commit [](https://docs.crawl4ai.com/advanced/session-management/#__codelineno-3-6) try: [](https://docs.crawl4ai.com/advanced/session-management/#__codelineno-3-7) while True: [](https://docs.crawl4ai.com/advanced/session-management/#__codelineno-3-8) await page.wait_for_selector("li.commit-item h4") [](https://docs.crawl4ai.com/advanced/session-management/#__codelineno-3-9) commit = await page.query_selector("li.commit-item h4") [](https://docs.crawl4ai.com/advanced/session-management/#__codelineno-3-10) commit = await commit.evaluate("(element) => element.textContent").strip() [](https://docs.crawl4ai.com/advanced/session-management/#__codelineno-3-11) if commit and commit != first_commit: [](https://docs.crawl4ai.com/advanced/session-management/#__codelineno-3-12) first_commit = commit [](https://docs.crawl4ai.com/advanced/session-management/#__codelineno-3-13) break [](https://docs.crawl4ai.com/advanced/session-management/#__codelineno-3-14) await asyncio.sleep(0.5) [](https://docs.crawl4ai.com/advanced/session-management/#__codelineno-3-15) except Exception as e: [](https://docs.crawl4ai.com/advanced/session-management/#__codelineno-3-16) print(f"Warning: New content didn't appear: {e}") [](https://docs.crawl4ai.com/advanced/session-management/#__codelineno-3-17) [](https://docs.crawl4ai.com/advanced/session-management/#__codelineno-3-18) async with AsyncWebCrawler() as crawler: [](https://docs.crawl4ai.com/advanced/session-management/#__codelineno-3-19) session_id = "commit_session" [](https://docs.crawl4ai.com/advanced/session-management/#__codelineno-3-20) url = "https://github.com/example/repo/commits/main" [](https://docs.crawl4ai.com/advanced/session-management/#__codelineno-3-21) crawler.crawler_strategy.set_hook("on_execution_started", on_execution_started) [](https://docs.crawl4ai.com/advanced/session-management/#__codelineno-3-22) [](https://docs.crawl4ai.com/advanced/session-management/#__codelineno-3-23) js_next_page = """document.querySelector('a.pagination-next').click();""" [](https://docs.crawl4ai.com/advanced/session-management/#__codelineno-3-24) [](https://docs.crawl4ai.com/advanced/session-management/#__codelineno-3-25) for page in range(3): [](https://docs.crawl4ai.com/advanced/session-management/#__codelineno-3-26) config = CrawlerRunConfig( [](https://docs.crawl4ai.com/advanced/session-management/#__codelineno-3-27) url=url, [](https://docs.crawl4ai.com/advanced/session-management/#__codelineno-3-28) session_id=session_id, [](https://docs.crawl4ai.com/advanced/session-management/#__codelineno-3-29) js_code=js_next_page if page > 0 else None, [](https://docs.crawl4ai.com/advanced/session-management/#__codelineno-3-30) css_selector="li.commit-item", [](https://docs.crawl4ai.com/advanced/session-management/#__codelineno-3-31) js_only=page > 0, [](https://docs.crawl4ai.com/advanced/session-management/#__codelineno-3-32) cache_mode=CacheMode.BYPASS [](https://docs.crawl4ai.com/advanced/session-management/#__codelineno-3-33) ) [](https://docs.crawl4ai.com/advanced/session-management/#__codelineno-3-34) [](https://docs.crawl4ai.com/advanced/session-management/#__codelineno-3-35) result = await crawler.arun(config=config) [](https://docs.crawl4ai.com/advanced/session-management/#__codelineno-3-36) print(f"Page {page + 1}: Found {len(result.extracted_content)} commits") [](https://docs.crawl4ai.com/advanced/session-management/#__codelineno-3-37) [](https://docs.crawl4ai.com/advanced/session-management/#__codelineno-3-38) await crawler.crawler_strategy.kill_session(session_id) [](https://docs.crawl4ai.com/advanced/session-management/#__codelineno-3-39) [](https://docs.crawl4ai.com/advanced/session-management/#__codelineno-3-40) asyncio.run(advanced_session_crawl_with_hooks())` This technique ensures new content loads before the next action. * * * Advanced Technique 2: Integrated JavaScript Execution and Waiting ----------------------------------------------------------------- Combine JavaScript execution and waiting logic for concise handling of dynamic content: `[](https://docs.crawl4ai.com/advanced/session-management/#__codelineno-4-1) async def integrated_js_and_wait_crawl(): [](https://docs.crawl4ai.com/advanced/session-management/#__codelineno-4-2) async with AsyncWebCrawler() as crawler: [](https://docs.crawl4ai.com/advanced/session-management/#__codelineno-4-3) session_id = "integrated_session" [](https://docs.crawl4ai.com/advanced/session-management/#__codelineno-4-4) url = "https://github.com/example/repo/commits/main" [](https://docs.crawl4ai.com/advanced/session-management/#__codelineno-4-5) [](https://docs.crawl4ai.com/advanced/session-management/#__codelineno-4-6) js_next_page_and_wait = """ [](https://docs.crawl4ai.com/advanced/session-management/#__codelineno-4-7) (async () => { [](https://docs.crawl4ai.com/advanced/session-management/#__codelineno-4-8) const getCurrentCommit = () => document.querySelector('li.commit-item h4').textContent.trim(); [](https://docs.crawl4ai.com/advanced/session-management/#__codelineno-4-9) const initialCommit = getCurrentCommit(); [](https://docs.crawl4ai.com/advanced/session-management/#__codelineno-4-10) document.querySelector('a.pagination-next').click(); [](https://docs.crawl4ai.com/advanced/session-management/#__codelineno-4-11) while (getCurrentCommit() === initialCommit) { [](https://docs.crawl4ai.com/advanced/session-management/#__codelineno-4-12) await new Promise(resolve => setTimeout(resolve, 100)); [](https://docs.crawl4ai.com/advanced/session-management/#__codelineno-4-13) } [](https://docs.crawl4ai.com/advanced/session-management/#__codelineno-4-14) })(); [](https://docs.crawl4ai.com/advanced/session-management/#__codelineno-4-15) """ [](https://docs.crawl4ai.com/advanced/session-management/#__codelineno-4-16) [](https://docs.crawl4ai.com/advanced/session-management/#__codelineno-4-17) for page in range(3): [](https://docs.crawl4ai.com/advanced/session-management/#__codelineno-4-18) config = CrawlerRunConfig( [](https://docs.crawl4ai.com/advanced/session-management/#__codelineno-4-19) url=url, [](https://docs.crawl4ai.com/advanced/session-management/#__codelineno-4-20) session_id=session_id, [](https://docs.crawl4ai.com/advanced/session-management/#__codelineno-4-21) js_code=js_next_page_and_wait if page > 0 else None, [](https://docs.crawl4ai.com/advanced/session-management/#__codelineno-4-22) css_selector="li.commit-item", [](https://docs.crawl4ai.com/advanced/session-management/#__codelineno-4-23) js_only=page > 0, [](https://docs.crawl4ai.com/advanced/session-management/#__codelineno-4-24) cache_mode=CacheMode.BYPASS [](https://docs.crawl4ai.com/advanced/session-management/#__codelineno-4-25) ) [](https://docs.crawl4ai.com/advanced/session-management/#__codelineno-4-26) [](https://docs.crawl4ai.com/advanced/session-management/#__codelineno-4-27) result = await crawler.arun(config=config) [](https://docs.crawl4ai.com/advanced/session-management/#__codelineno-4-28) print(f"Page {page + 1}: Found {len(result.extracted_content)} commits") [](https://docs.crawl4ai.com/advanced/session-management/#__codelineno-4-29) [](https://docs.crawl4ai.com/advanced/session-management/#__codelineno-4-30) await crawler.crawler_strategy.kill_session(session_id) [](https://docs.crawl4ai.com/advanced/session-management/#__codelineno-4-31) [](https://docs.crawl4ai.com/advanced/session-management/#__codelineno-4-32) asyncio.run(integrated_js_and_wait_crawl())` * * * #### Common Use Cases for Sessions 1. **Authentication Flows**: Login and interact with secured pages. 2. **Pagination Handling**: Navigate through multiple pages. 3. **Form Submissions**: Fill forms, submit, and process results. 4. **Multi-step Processes**: Complete workflows that span multiple actions. 5. **Dynamic Content Navigation**: Handle JavaScript-rendered or event-triggered content. * * * \> Feedback --- # AsyncWebCrawler - Crawl4AI Documentation (v0.8.x) AsyncWebCrawler =============== The **`AsyncWebCrawler`** is the core class for asynchronous web crawling in Crawl4AI. You typically create it **once**, optionally customize it with a **`BrowserConfig`** (e.g., headless, user agent), then **run** multiple **`arun()`** calls with different **`CrawlerRunConfig`** objects. **Recommended usage**: 1. **Create** a `BrowserConfig` for global browser settings.  2. **Instantiate** `AsyncWebCrawler(config=browser_config)`.  3. **Use** the crawler in an async context manager (`async with`) or manage start/close manually.  4. **Call** `arun(url, config=crawler_run_config)` for each page you want. * * * 1. Constructor Overview ----------------------- `[](https://docs.crawl4ai.com/api/async-webcrawler/#__codelineno-0-1) class AsyncWebCrawler: [](https://docs.crawl4ai.com/api/async-webcrawler/#__codelineno-0-2) def __init__( [](https://docs.crawl4ai.com/api/async-webcrawler/#__codelineno-0-3) self, [](https://docs.crawl4ai.com/api/async-webcrawler/#__codelineno-0-4) crawler_strategy: Optional[AsyncCrawlerStrategy] = None, [](https://docs.crawl4ai.com/api/async-webcrawler/#__codelineno-0-5) config: Optional[BrowserConfig] = None, [](https://docs.crawl4ai.com/api/async-webcrawler/#__codelineno-0-6) always_bypass_cache: bool = False, # deprecated [](https://docs.crawl4ai.com/api/async-webcrawler/#__codelineno-0-7) always_by_pass_cache: Optional[bool] = None, # also deprecated [](https://docs.crawl4ai.com/api/async-webcrawler/#__codelineno-0-8) base_directory: str = ..., [](https://docs.crawl4ai.com/api/async-webcrawler/#__codelineno-0-9) thread_safe: bool = False, [](https://docs.crawl4ai.com/api/async-webcrawler/#__codelineno-0-10) **kwargs, [](https://docs.crawl4ai.com/api/async-webcrawler/#__codelineno-0-11) ): [](https://docs.crawl4ai.com/api/async-webcrawler/#__codelineno-0-12) """ [](https://docs.crawl4ai.com/api/async-webcrawler/#__codelineno-0-13) Create an AsyncWebCrawler instance. [](https://docs.crawl4ai.com/api/async-webcrawler/#__codelineno-0-14) [](https://docs.crawl4ai.com/api/async-webcrawler/#__codelineno-0-15) Args: [](https://docs.crawl4ai.com/api/async-webcrawler/#__codelineno-0-16) crawler_strategy: [](https://docs.crawl4ai.com/api/async-webcrawler/#__codelineno-0-17) (Advanced) Provide a custom crawler strategy if needed. [](https://docs.crawl4ai.com/api/async-webcrawler/#__codelineno-0-18) config: [](https://docs.crawl4ai.com/api/async-webcrawler/#__codelineno-0-19) A BrowserConfig object specifying how the browser is set up. [](https://docs.crawl4ai.com/api/async-webcrawler/#__codelineno-0-20) always_bypass_cache: [](https://docs.crawl4ai.com/api/async-webcrawler/#__codelineno-0-21) (Deprecated) Use CrawlerRunConfig.cache_mode instead. [](https://docs.crawl4ai.com/api/async-webcrawler/#__codelineno-0-22) base_directory: [](https://docs.crawl4ai.com/api/async-webcrawler/#__codelineno-0-23) Folder for storing caches/logs (if relevant). [](https://docs.crawl4ai.com/api/async-webcrawler/#__codelineno-0-24) thread_safe: [](https://docs.crawl4ai.com/api/async-webcrawler/#__codelineno-0-25) If True, attempts some concurrency safeguards. Usually False. [](https://docs.crawl4ai.com/api/async-webcrawler/#__codelineno-0-26) **kwargs: [](https://docs.crawl4ai.com/api/async-webcrawler/#__codelineno-0-27) Additional legacy or debugging parameters. [](https://docs.crawl4ai.com/api/async-webcrawler/#__codelineno-0-28) """ [](https://docs.crawl4ai.com/api/async-webcrawler/#__codelineno-0-29) ) [](https://docs.crawl4ai.com/api/async-webcrawler/#__codelineno-0-30) [](https://docs.crawl4ai.com/api/async-webcrawler/#__codelineno-0-31) ### Typical Initialization [](https://docs.crawl4ai.com/api/async-webcrawler/#__codelineno-0-32) [](https://docs.crawl4ai.com/api/async-webcrawler/#__codelineno-0-33) ```python [](https://docs.crawl4ai.com/api/async-webcrawler/#__codelineno-0-34) from crawl4ai import AsyncWebCrawler, BrowserConfig [](https://docs.crawl4ai.com/api/async-webcrawler/#__codelineno-0-35) [](https://docs.crawl4ai.com/api/async-webcrawler/#__codelineno-0-36) browser_cfg = BrowserConfig( [](https://docs.crawl4ai.com/api/async-webcrawler/#__codelineno-0-37) browser_type="chromium", [](https://docs.crawl4ai.com/api/async-webcrawler/#__codelineno-0-38) headless=True, [](https://docs.crawl4ai.com/api/async-webcrawler/#__codelineno-0-39) verbose=True [](https://docs.crawl4ai.com/api/async-webcrawler/#__codelineno-0-40) ) [](https://docs.crawl4ai.com/api/async-webcrawler/#__codelineno-0-41) [](https://docs.crawl4ai.com/api/async-webcrawler/#__codelineno-0-42) crawler = AsyncWebCrawler(config=browser_cfg)` **Notes**: * **Legacy** parameters like `always_bypass_cache` remain for backward compatibility, but prefer to set **caching** in `CrawlerRunConfig`. * * * 2. Lifecycle: Start/Close or Context Manager -------------------------------------------- ### 2.1 Context Manager (Recommended) `[](https://docs.crawl4ai.com/api/async-webcrawler/#__codelineno-1-1) async with AsyncWebCrawler(config=browser_cfg) as crawler: [](https://docs.crawl4ai.com/api/async-webcrawler/#__codelineno-1-2) result = await crawler.arun("https://example.com") [](https://docs.crawl4ai.com/api/async-webcrawler/#__codelineno-1-3) # The crawler automatically starts/closes resources` When the `async with` block ends, the crawler cleans up (closes the browser, etc.). ### 2.2 Manual Start & Close `[](https://docs.crawl4ai.com/api/async-webcrawler/#__codelineno-2-1) crawler = AsyncWebCrawler(config=browser_cfg) [](https://docs.crawl4ai.com/api/async-webcrawler/#__codelineno-2-2) await crawler.start() [](https://docs.crawl4ai.com/api/async-webcrawler/#__codelineno-2-3) [](https://docs.crawl4ai.com/api/async-webcrawler/#__codelineno-2-4) result1 = await crawler.arun("https://example.com") [](https://docs.crawl4ai.com/api/async-webcrawler/#__codelineno-2-5) result2 = await crawler.arun("https://another.com") [](https://docs.crawl4ai.com/api/async-webcrawler/#__codelineno-2-6) [](https://docs.crawl4ai.com/api/async-webcrawler/#__codelineno-2-7) await crawler.close()` Use this style if you have a **long-running** application or need full control of the crawler’s lifecycle. * * * 3. Primary Method: `arun()` --------------------------- `[](https://docs.crawl4ai.com/api/async-webcrawler/#__codelineno-3-1) async def arun( [](https://docs.crawl4ai.com/api/async-webcrawler/#__codelineno-3-2) self, [](https://docs.crawl4ai.com/api/async-webcrawler/#__codelineno-3-3) url: str, [](https://docs.crawl4ai.com/api/async-webcrawler/#__codelineno-3-4) config: Optional[CrawlerRunConfig] = None, [](https://docs.crawl4ai.com/api/async-webcrawler/#__codelineno-3-5) # Legacy parameters for backward compatibility... [](https://docs.crawl4ai.com/api/async-webcrawler/#__codelineno-3-6) ) -> CrawlResult: [](https://docs.crawl4ai.com/api/async-webcrawler/#__codelineno-3-7) ...` ### 3.1 New Approach You pass a `CrawlerRunConfig` object that sets up everything about a crawl—content filtering, caching, session reuse, JS code, screenshots, etc. `[](https://docs.crawl4ai.com/api/async-webcrawler/#__codelineno-4-1) import asyncio [](https://docs.crawl4ai.com/api/async-webcrawler/#__codelineno-4-2) from crawl4ai import CrawlerRunConfig, CacheMode [](https://docs.crawl4ai.com/api/async-webcrawler/#__codelineno-4-3) [](https://docs.crawl4ai.com/api/async-webcrawler/#__codelineno-4-4) run_cfg = CrawlerRunConfig( [](https://docs.crawl4ai.com/api/async-webcrawler/#__codelineno-4-5) cache_mode=CacheMode.BYPASS, [](https://docs.crawl4ai.com/api/async-webcrawler/#__codelineno-4-6) css_selector="main.article", [](https://docs.crawl4ai.com/api/async-webcrawler/#__codelineno-4-7) word_count_threshold=10, [](https://docs.crawl4ai.com/api/async-webcrawler/#__codelineno-4-8) screenshot=True [](https://docs.crawl4ai.com/api/async-webcrawler/#__codelineno-4-9) ) [](https://docs.crawl4ai.com/api/async-webcrawler/#__codelineno-4-10) [](https://docs.crawl4ai.com/api/async-webcrawler/#__codelineno-4-11) async with AsyncWebCrawler(config=browser_cfg) as crawler: [](https://docs.crawl4ai.com/api/async-webcrawler/#__codelineno-4-12) result = await crawler.arun("https://example.com/news", config=run_cfg) [](https://docs.crawl4ai.com/api/async-webcrawler/#__codelineno-4-13) print("Crawled HTML length:", len(result.cleaned_html)) [](https://docs.crawl4ai.com/api/async-webcrawler/#__codelineno-4-14) if result.screenshot: [](https://docs.crawl4ai.com/api/async-webcrawler/#__codelineno-4-15) print("Screenshot base64 length:", len(result.screenshot))` ### 3.2 Legacy Parameters Still Accepted For **backward** compatibility, `arun()` can still accept direct arguments like `css_selector=...`, `word_count_threshold=...`, etc., but we strongly advise migrating them into a **`CrawlerRunConfig`**. * * * 4. Batch Processing: `arun_many()` ---------------------------------- `[](https://docs.crawl4ai.com/api/async-webcrawler/#__codelineno-5-1) async def arun_many( [](https://docs.crawl4ai.com/api/async-webcrawler/#__codelineno-5-2) self, [](https://docs.crawl4ai.com/api/async-webcrawler/#__codelineno-5-3) urls: List[str], [](https://docs.crawl4ai.com/api/async-webcrawler/#__codelineno-5-4) config: Optional[CrawlerRunConfig] = None, [](https://docs.crawl4ai.com/api/async-webcrawler/#__codelineno-5-5) # Legacy parameters maintained for backwards compatibility... [](https://docs.crawl4ai.com/api/async-webcrawler/#__codelineno-5-6) ) -> List[CrawlResult]: [](https://docs.crawl4ai.com/api/async-webcrawler/#__codelineno-5-7) """ [](https://docs.crawl4ai.com/api/async-webcrawler/#__codelineno-5-8) Process multiple URLs with intelligent rate limiting and resource monitoring. [](https://docs.crawl4ai.com/api/async-webcrawler/#__codelineno-5-9) """` ### 4.1 Resource-Aware Crawling The `arun_many()` method now uses an intelligent dispatcher that: * Monitors system memory usage * Implements adaptive rate limiting * Provides detailed progress monitoring * Manages concurrent crawls efficiently ### 4.2 Example Usage Check page [Multi-url Crawling](https://docs.crawl4ai.com/advanced/multi-url-crawling/) for a detailed example of how to use `arun_many()`. ``[](https://docs.crawl4ai.com/api/async-webcrawler/#__codelineno-6-1) ### 4.3 Key Features [](https://docs.crawl4ai.com/api/async-webcrawler/#__codelineno-6-2) [](https://docs.crawl4ai.com/api/async-webcrawler/#__codelineno-6-3) 1. **Rate Limiting** [](https://docs.crawl4ai.com/api/async-webcrawler/#__codelineno-6-4) [](https://docs.crawl4ai.com/api/async-webcrawler/#__codelineno-6-5) - Automatic delay between requests [](https://docs.crawl4ai.com/api/async-webcrawler/#__codelineno-6-6) - Exponential backoff on rate limit detection [](https://docs.crawl4ai.com/api/async-webcrawler/#__codelineno-6-7) - Domain-specific rate limiting [](https://docs.crawl4ai.com/api/async-webcrawler/#__codelineno-6-8) - Configurable retry strategy [](https://docs.crawl4ai.com/api/async-webcrawler/#__codelineno-6-9) [](https://docs.crawl4ai.com/api/async-webcrawler/#__codelineno-6-10) 2. **Resource Monitoring** [](https://docs.crawl4ai.com/api/async-webcrawler/#__codelineno-6-11) [](https://docs.crawl4ai.com/api/async-webcrawler/#__codelineno-6-12) - Memory usage tracking [](https://docs.crawl4ai.com/api/async-webcrawler/#__codelineno-6-13) - Adaptive concurrency based on system load [](https://docs.crawl4ai.com/api/async-webcrawler/#__codelineno-6-14) - Automatic pausing when resources are constrained [](https://docs.crawl4ai.com/api/async-webcrawler/#__codelineno-6-15) [](https://docs.crawl4ai.com/api/async-webcrawler/#__codelineno-6-16) 3. **Progress Monitoring** [](https://docs.crawl4ai.com/api/async-webcrawler/#__codelineno-6-17) [](https://docs.crawl4ai.com/api/async-webcrawler/#__codelineno-6-18) - Detailed or aggregated progress display [](https://docs.crawl4ai.com/api/async-webcrawler/#__codelineno-6-19) - Real-time status updates [](https://docs.crawl4ai.com/api/async-webcrawler/#__codelineno-6-20) - Memory usage statistics [](https://docs.crawl4ai.com/api/async-webcrawler/#__codelineno-6-21) [](https://docs.crawl4ai.com/api/async-webcrawler/#__codelineno-6-22) 4. **Error Handling** [](https://docs.crawl4ai.com/api/async-webcrawler/#__codelineno-6-23) [](https://docs.crawl4ai.com/api/async-webcrawler/#__codelineno-6-24) - Graceful handling of rate limits [](https://docs.crawl4ai.com/api/async-webcrawler/#__codelineno-6-25) - Automatic retries with backoff [](https://docs.crawl4ai.com/api/async-webcrawler/#__codelineno-6-26) - Detailed error reporting [](https://docs.crawl4ai.com/api/async-webcrawler/#__codelineno-6-27) [](https://docs.crawl4ai.com/api/async-webcrawler/#__codelineno-6-28) --- [](https://docs.crawl4ai.com/api/async-webcrawler/#__codelineno-6-29) [](https://docs.crawl4ai.com/api/async-webcrawler/#__codelineno-6-30) ## 5. `CrawlResult` Output [](https://docs.crawl4ai.com/api/async-webcrawler/#__codelineno-6-31) [](https://docs.crawl4ai.com/api/async-webcrawler/#__codelineno-6-32) Each `arun()` returns a **`CrawlResult`** containing: [](https://docs.crawl4ai.com/api/async-webcrawler/#__codelineno-6-33) [](https://docs.crawl4ai.com/api/async-webcrawler/#__codelineno-6-34) - `url`: Final URL (if redirected). [](https://docs.crawl4ai.com/api/async-webcrawler/#__codelineno-6-35) - `html`: Original HTML. [](https://docs.crawl4ai.com/api/async-webcrawler/#__codelineno-6-36) - `cleaned_html`: Sanitized HTML. [](https://docs.crawl4ai.com/api/async-webcrawler/#__codelineno-6-37) - `markdown_v2`: Deprecated. Instead just use regular `markdown` [](https://docs.crawl4ai.com/api/async-webcrawler/#__codelineno-6-38) - `extracted_content`: If an extraction strategy was used (JSON for CSS/LLM strategies). [](https://docs.crawl4ai.com/api/async-webcrawler/#__codelineno-6-39) - `screenshot`, `pdf`: If screenshots/PDF requested. [](https://docs.crawl4ai.com/api/async-webcrawler/#__codelineno-6-40) - `media`, `links`: Information about discovered images/links. [](https://docs.crawl4ai.com/api/async-webcrawler/#__codelineno-6-41) - `success`, `error_message`: Status info. [](https://docs.crawl4ai.com/api/async-webcrawler/#__codelineno-6-42) [](https://docs.crawl4ai.com/api/async-webcrawler/#__codelineno-6-43) For details, see [CrawlResult doc](./crawl-result.md). [](https://docs.crawl4ai.com/api/async-webcrawler/#__codelineno-6-44) [](https://docs.crawl4ai.com/api/async-webcrawler/#__codelineno-6-45) --- [](https://docs.crawl4ai.com/api/async-webcrawler/#__codelineno-6-46) [](https://docs.crawl4ai.com/api/async-webcrawler/#__codelineno-6-47) ## 6. Quick Example [](https://docs.crawl4ai.com/api/async-webcrawler/#__codelineno-6-48) [](https://docs.crawl4ai.com/api/async-webcrawler/#__codelineno-6-49) Below is an example hooking it all together: [](https://docs.crawl4ai.com/api/async-webcrawler/#__codelineno-6-50) [](https://docs.crawl4ai.com/api/async-webcrawler/#__codelineno-6-51) ```python [](https://docs.crawl4ai.com/api/async-webcrawler/#__codelineno-6-52) import asyncio [](https://docs.crawl4ai.com/api/async-webcrawler/#__codelineno-6-53) from crawl4ai import AsyncWebCrawler, BrowserConfig, CrawlerRunConfig, CacheMode [](https://docs.crawl4ai.com/api/async-webcrawler/#__codelineno-6-54) from crawl4ai import JsonCssExtractionStrategy [](https://docs.crawl4ai.com/api/async-webcrawler/#__codelineno-6-55) import json [](https://docs.crawl4ai.com/api/async-webcrawler/#__codelineno-6-56) [](https://docs.crawl4ai.com/api/async-webcrawler/#__codelineno-6-57) async def main(): [](https://docs.crawl4ai.com/api/async-webcrawler/#__codelineno-6-58) # 1. Browser config [](https://docs.crawl4ai.com/api/async-webcrawler/#__codelineno-6-59) browser_cfg = BrowserConfig( [](https://docs.crawl4ai.com/api/async-webcrawler/#__codelineno-6-60) browser_type="firefox", [](https://docs.crawl4ai.com/api/async-webcrawler/#__codelineno-6-61) headless=False, [](https://docs.crawl4ai.com/api/async-webcrawler/#__codelineno-6-62) verbose=True [](https://docs.crawl4ai.com/api/async-webcrawler/#__codelineno-6-63) ) [](https://docs.crawl4ai.com/api/async-webcrawler/#__codelineno-6-64) [](https://docs.crawl4ai.com/api/async-webcrawler/#__codelineno-6-65) # 2. Run config [](https://docs.crawl4ai.com/api/async-webcrawler/#__codelineno-6-66) schema = { [](https://docs.crawl4ai.com/api/async-webcrawler/#__codelineno-6-67) "name": "Articles", [](https://docs.crawl4ai.com/api/async-webcrawler/#__codelineno-6-68) "baseSelector": "article.post", [](https://docs.crawl4ai.com/api/async-webcrawler/#__codelineno-6-69) "fields": [ [](https://docs.crawl4ai.com/api/async-webcrawler/#__codelineno-6-70) { [](https://docs.crawl4ai.com/api/async-webcrawler/#__codelineno-6-71) "name": "title", [](https://docs.crawl4ai.com/api/async-webcrawler/#__codelineno-6-72) "selector": "h2", [](https://docs.crawl4ai.com/api/async-webcrawler/#__codelineno-6-73) "type": "text" [](https://docs.crawl4ai.com/api/async-webcrawler/#__codelineno-6-74) }, [](https://docs.crawl4ai.com/api/async-webcrawler/#__codelineno-6-75) { [](https://docs.crawl4ai.com/api/async-webcrawler/#__codelineno-6-76) "name": "url", [](https://docs.crawl4ai.com/api/async-webcrawler/#__codelineno-6-77) "selector": "a", [](https://docs.crawl4ai.com/api/async-webcrawler/#__codelineno-6-78) "type": "attribute", [](https://docs.crawl4ai.com/api/async-webcrawler/#__codelineno-6-79) "attribute": "href" [](https://docs.crawl4ai.com/api/async-webcrawler/#__codelineno-6-80) } [](https://docs.crawl4ai.com/api/async-webcrawler/#__codelineno-6-81) ] [](https://docs.crawl4ai.com/api/async-webcrawler/#__codelineno-6-82) } [](https://docs.crawl4ai.com/api/async-webcrawler/#__codelineno-6-83) [](https://docs.crawl4ai.com/api/async-webcrawler/#__codelineno-6-84) run_cfg = CrawlerRunConfig( [](https://docs.crawl4ai.com/api/async-webcrawler/#__codelineno-6-85) cache_mode=CacheMode.BYPASS, [](https://docs.crawl4ai.com/api/async-webcrawler/#__codelineno-6-86) extraction_strategy=JsonCssExtractionStrategy(schema), [](https://docs.crawl4ai.com/api/async-webcrawler/#__codelineno-6-87) word_count_threshold=15, [](https://docs.crawl4ai.com/api/async-webcrawler/#__codelineno-6-88) remove_overlay_elements=True, [](https://docs.crawl4ai.com/api/async-webcrawler/#__codelineno-6-89) wait_for="css:.post" # Wait for posts to appear [](https://docs.crawl4ai.com/api/async-webcrawler/#__codelineno-6-90) ) [](https://docs.crawl4ai.com/api/async-webcrawler/#__codelineno-6-91) [](https://docs.crawl4ai.com/api/async-webcrawler/#__codelineno-6-92) async with AsyncWebCrawler(config=browser_cfg) as crawler: [](https://docs.crawl4ai.com/api/async-webcrawler/#__codelineno-6-93) result = await crawler.arun( [](https://docs.crawl4ai.com/api/async-webcrawler/#__codelineno-6-94) url="https://example.com/blog", [](https://docs.crawl4ai.com/api/async-webcrawler/#__codelineno-6-95) config=run_cfg [](https://docs.crawl4ai.com/api/async-webcrawler/#__codelineno-6-96) ) [](https://docs.crawl4ai.com/api/async-webcrawler/#__codelineno-6-97) [](https://docs.crawl4ai.com/api/async-webcrawler/#__codelineno-6-98) if result.success: [](https://docs.crawl4ai.com/api/async-webcrawler/#__codelineno-6-99) print("Cleaned HTML length:", len(result.cleaned_html)) [](https://docs.crawl4ai.com/api/async-webcrawler/#__codelineno-6-100) if result.extracted_content: [](https://docs.crawl4ai.com/api/async-webcrawler/#__codelineno-6-101) articles = json.loads(result.extracted_content) [](https://docs.crawl4ai.com/api/async-webcrawler/#__codelineno-6-102) print("Extracted articles:", articles[:2]) [](https://docs.crawl4ai.com/api/async-webcrawler/#__codelineno-6-103) else: [](https://docs.crawl4ai.com/api/async-webcrawler/#__codelineno-6-104) print("Error:", result.error_message) [](https://docs.crawl4ai.com/api/async-webcrawler/#__codelineno-6-105) [](https://docs.crawl4ai.com/api/async-webcrawler/#__codelineno-6-106) asyncio.run(main())`` **Explanation**: * We define a **`BrowserConfig`** with Firefox, no headless, and `verbose=True`.  * We define a **`CrawlerRunConfig`** that **bypasses cache**, uses a **CSS** extraction schema, has a `word_count_threshold=15`, etc.  * We pass them to `AsyncWebCrawler(config=...)` and `arun(url=..., config=...)`. * * * 7. Best Practices & Migration Notes ----------------------------------- 1. **Use** `BrowserConfig` for **global** settings about the browser’s environment.  2. **Use** `CrawlerRunConfig` for **per-crawl** logic (caching, content filtering, extraction strategies, wait conditions).  3. **Avoid** legacy parameters like `css_selector` or `word_count_threshold` directly in `arun()`. Instead: `[](https://docs.crawl4ai.com/api/async-webcrawler/#__codelineno-7-1) run_cfg = CrawlerRunConfig(css_selector=".main-content", word_count_threshold=20) [](https://docs.crawl4ai.com/api/async-webcrawler/#__codelineno-7-2) result = await crawler.arun(url="...", config=run_cfg)` 4. **Context Manager** usage is simplest unless you want a persistent crawler across many calls. * * * 8. Summary ---------- **AsyncWebCrawler** is your entry point to asynchronous crawling: * **Constructor** accepts **`BrowserConfig`** (or defaults).  * **`arun(url, config=CrawlerRunConfig)`** is the main method for single-page crawls.  * **`arun_many(urls, config=CrawlerRunConfig)`** handles concurrency across multiple URLs.  * For advanced lifecycle control, use `start()` and `close()` explicitly.  **Migration**: * If you used `AsyncWebCrawler(browser_type="chromium", css_selector="...")`, move browser settings to `BrowserConfig(...)` and content/crawl logic to `CrawlerRunConfig(...)`. This modular approach ensures your code is **clean**, **scalable**, and **easy to maintain**. For any advanced or rarely used parameters, see the [BrowserConfig docs](https://docs.crawl4ai.com/api/parameters/) . * * * \> Feedback --- # arun() - Crawl4AI Documentation (v0.8.x) `arun()` Parameter Guide (New Approach) ======================================= In Crawl4AI’s **latest** configuration model, nearly all parameters that once went directly to `arun()` are now part of **`CrawlerRunConfig`**. When calling `arun()`, you provide: `[](https://docs.crawl4ai.com/api/arun/#__codelineno-0-1) await crawler.arun( [](https://docs.crawl4ai.com/api/arun/#__codelineno-0-2) url="https://example.com", [](https://docs.crawl4ai.com/api/arun/#__codelineno-0-3) config=my_run_config [](https://docs.crawl4ai.com/api/arun/#__codelineno-0-4) )` Below is an organized look at the parameters that can go inside `CrawlerRunConfig`, divided by their functional areas. For **Browser** settings (e.g., `headless`, `browser_type`), see [BrowserConfig](https://docs.crawl4ai.com/api/parameters/) . * * * 1. Core Usage ------------- `[](https://docs.crawl4ai.com/api/arun/#__codelineno-1-1) from crawl4ai import AsyncWebCrawler, CrawlerRunConfig, CacheMode [](https://docs.crawl4ai.com/api/arun/#__codelineno-1-2) [](https://docs.crawl4ai.com/api/arun/#__codelineno-1-3) async def main(): [](https://docs.crawl4ai.com/api/arun/#__codelineno-1-4) run_config = CrawlerRunConfig( [](https://docs.crawl4ai.com/api/arun/#__codelineno-1-5) verbose=True, # Detailed logging [](https://docs.crawl4ai.com/api/arun/#__codelineno-1-6) cache_mode=CacheMode.ENABLED, # Use normal read/write cache [](https://docs.crawl4ai.com/api/arun/#__codelineno-1-7) check_robots_txt=True, # Respect robots.txt rules [](https://docs.crawl4ai.com/api/arun/#__codelineno-1-8) # ... other parameters [](https://docs.crawl4ai.com/api/arun/#__codelineno-1-9) ) [](https://docs.crawl4ai.com/api/arun/#__codelineno-1-10) [](https://docs.crawl4ai.com/api/arun/#__codelineno-1-11) async with AsyncWebCrawler() as crawler: [](https://docs.crawl4ai.com/api/arun/#__codelineno-1-12) result = await crawler.arun( [](https://docs.crawl4ai.com/api/arun/#__codelineno-1-13) url="https://example.com", [](https://docs.crawl4ai.com/api/arun/#__codelineno-1-14) config=run_config [](https://docs.crawl4ai.com/api/arun/#__codelineno-1-15) ) [](https://docs.crawl4ai.com/api/arun/#__codelineno-1-16) [](https://docs.crawl4ai.com/api/arun/#__codelineno-1-17) # Check if blocked by robots.txt [](https://docs.crawl4ai.com/api/arun/#__codelineno-1-18) if not result.success and result.status_code == 403: [](https://docs.crawl4ai.com/api/arun/#__codelineno-1-19) print(f"Error: {result.error_message}")` **Key Fields**: - `verbose=True` logs each crawl step.  - `cache_mode` decides how to read/write the local crawl cache. * * * 2. Cache Control ---------------- **`cache_mode`** (default: `CacheMode.ENABLED`) Use a built-in enum from `CacheMode`: * `ENABLED`: Normal caching—reads if available, writes if missing. * `DISABLED`: No caching—always refetch pages. * `READ_ONLY`: Reads from cache only; no new writes. * `WRITE_ONLY`: Writes to cache but doesn’t read existing data. * `BYPASS`: Skips reading cache for this crawl (though it might still write if set up that way). `[](https://docs.crawl4ai.com/api/arun/#__codelineno-2-1) run_config = CrawlerRunConfig( [](https://docs.crawl4ai.com/api/arun/#__codelineno-2-2) cache_mode=CacheMode.BYPASS [](https://docs.crawl4ai.com/api/arun/#__codelineno-2-3) )` **Additional flags**: * `bypass_cache=True` acts like `CacheMode.BYPASS`. * `disable_cache=True` acts like `CacheMode.DISABLED`. * `no_cache_read=True` acts like `CacheMode.WRITE_ONLY`. * `no_cache_write=True` acts like `CacheMode.READ_ONLY`. * * * 3. Content Processing & Selection --------------------------------- ### 3.1 Text Processing `[](https://docs.crawl4ai.com/api/arun/#__codelineno-3-1) run_config = CrawlerRunConfig( [](https://docs.crawl4ai.com/api/arun/#__codelineno-3-2) word_count_threshold=10, # Ignore text blocks <10 words [](https://docs.crawl4ai.com/api/arun/#__codelineno-3-3) only_text=False, # If True, tries to remove non-text elements [](https://docs.crawl4ai.com/api/arun/#__codelineno-3-4) keep_data_attributes=False # Keep or discard data-* attributes [](https://docs.crawl4ai.com/api/arun/#__codelineno-3-5) )` ### 3.2 Content Selection `[](https://docs.crawl4ai.com/api/arun/#__codelineno-4-1) run_config = CrawlerRunConfig( [](https://docs.crawl4ai.com/api/arun/#__codelineno-4-2) css_selector=".main-content", # Focus on .main-content region only [](https://docs.crawl4ai.com/api/arun/#__codelineno-4-3) excluded_tags=["form", "nav"], # Remove entire tag blocks [](https://docs.crawl4ai.com/api/arun/#__codelineno-4-4) remove_forms=True, # Specifically strip
elements [](https://docs.crawl4ai.com/api/arun/#__codelineno-4-5) remove_overlay_elements=True, # Attempt to remove modals/popups [](https://docs.crawl4ai.com/api/arun/#__codelineno-4-6) )` ### 3.3 Link Handling `[](https://docs.crawl4ai.com/api/arun/#__codelineno-5-1) run_config = CrawlerRunConfig( [](https://docs.crawl4ai.com/api/arun/#__codelineno-5-2) exclude_external_links=True, # Remove external links from final content [](https://docs.crawl4ai.com/api/arun/#__codelineno-5-3) exclude_social_media_links=True, # Remove links to known social sites [](https://docs.crawl4ai.com/api/arun/#__codelineno-5-4) exclude_domains=["ads.example.com"], # Exclude links to these domains [](https://docs.crawl4ai.com/api/arun/#__codelineno-5-5) exclude_social_media_domains=["facebook.com","twitter.com"], # Extend the default list [](https://docs.crawl4ai.com/api/arun/#__codelineno-5-6) )` ### 3.4 Media Filtering `[](https://docs.crawl4ai.com/api/arun/#__codelineno-6-1) run_config = CrawlerRunConfig( [](https://docs.crawl4ai.com/api/arun/#__codelineno-6-2) exclude_external_images=True # Strip images from other domains [](https://docs.crawl4ai.com/api/arun/#__codelineno-6-3) )` * * * 4. Page Navigation & Timing --------------------------- ### 4.1 Basic Browser Flow `[](https://docs.crawl4ai.com/api/arun/#__codelineno-7-1) run_config = CrawlerRunConfig( [](https://docs.crawl4ai.com/api/arun/#__codelineno-7-2) wait_for="css:.dynamic-content", # Wait for .dynamic-content [](https://docs.crawl4ai.com/api/arun/#__codelineno-7-3) delay_before_return_html=2.0, # Wait 2s before capturing final HTML [](https://docs.crawl4ai.com/api/arun/#__codelineno-7-4) page_timeout=60000, # Navigation & script timeout (ms) [](https://docs.crawl4ai.com/api/arun/#__codelineno-7-5) )` **Key Fields**: * `wait_for`: * `"css:selector"` or * `"js:() => boolean"` e.g. `js:() => document.querySelectorAll('.item').length > 10`. * `mean_delay` & `max_range`: define random delays for `arun_many()` calls.  * `semaphore_count`: concurrency limit when crawling multiple URLs. ### 4.2 JavaScript Execution `[](https://docs.crawl4ai.com/api/arun/#__codelineno-8-1) run_config = CrawlerRunConfig( [](https://docs.crawl4ai.com/api/arun/#__codelineno-8-2) js_code=[ [](https://docs.crawl4ai.com/api/arun/#__codelineno-8-3) "window.scrollTo(0, document.body.scrollHeight);", [](https://docs.crawl4ai.com/api/arun/#__codelineno-8-4) "document.querySelector('.load-more')?.click();" [](https://docs.crawl4ai.com/api/arun/#__codelineno-8-5) ], [](https://docs.crawl4ai.com/api/arun/#__codelineno-8-6) js_only=False [](https://docs.crawl4ai.com/api/arun/#__codelineno-8-7) )` * `js_code` can be a single string or a list of strings.  * `js_only=True` means “I’m continuing in the same session with new JS steps, no new full navigation.” ### 4.3 Anti-Bot `[](https://docs.crawl4ai.com/api/arun/#__codelineno-9-1) run_config = CrawlerRunConfig( [](https://docs.crawl4ai.com/api/arun/#__codelineno-9-2) magic=True, [](https://docs.crawl4ai.com/api/arun/#__codelineno-9-3) simulate_user=True, [](https://docs.crawl4ai.com/api/arun/#__codelineno-9-4) override_navigator=True [](https://docs.crawl4ai.com/api/arun/#__codelineno-9-5) )` \- `magic=True` tries multiple stealth features.  - `simulate_user=True` mimics mouse movements or random delays.  - `override_navigator=True` fakes some navigator properties (like user agent checks). * * * 5. Session Management --------------------- **`session_id`**: `[](https://docs.crawl4ai.com/api/arun/#__codelineno-10-1) run_config = CrawlerRunConfig( [](https://docs.crawl4ai.com/api/arun/#__codelineno-10-2) session_id="my_session123" [](https://docs.crawl4ai.com/api/arun/#__codelineno-10-3) )` If re-used in subsequent `arun()` calls, the same tab/page context is continued (helpful for multi-step tasks or stateful browsing). * * * 6. Screenshot, PDF & Media Options ---------------------------------- `[](https://docs.crawl4ai.com/api/arun/#__codelineno-11-1) run_config = CrawlerRunConfig( [](https://docs.crawl4ai.com/api/arun/#__codelineno-11-2) screenshot=True, # Grab a screenshot as base64 [](https://docs.crawl4ai.com/api/arun/#__codelineno-11-3) screenshot_wait_for=1.0, # Wait 1s before capturing [](https://docs.crawl4ai.com/api/arun/#__codelineno-11-4) pdf=True, # Also produce a PDF [](https://docs.crawl4ai.com/api/arun/#__codelineno-11-5) image_description_min_word_threshold=5, # If analyzing alt text [](https://docs.crawl4ai.com/api/arun/#__codelineno-11-6) image_score_threshold=3, # Filter out low-score images [](https://docs.crawl4ai.com/api/arun/#__codelineno-11-7) )` **Where they appear**: - `result.screenshot` → Base64 screenshot string. - `result.pdf` → Byte array with PDF data. * * * 7. Extraction Strategy ---------------------- **For advanced data extraction** (CSS/LLM-based), set `extraction_strategy`: `[](https://docs.crawl4ai.com/api/arun/#__codelineno-12-1) run_config = CrawlerRunConfig( [](https://docs.crawl4ai.com/api/arun/#__codelineno-12-2) extraction_strategy=my_css_or_llm_strategy [](https://docs.crawl4ai.com/api/arun/#__codelineno-12-3) )` The extracted data will appear in `result.extracted_content`. * * * 8. Comprehensive Example ------------------------ Below is a snippet combining many parameters: `[](https://docs.crawl4ai.com/api/arun/#__codelineno-13-1) import asyncio [](https://docs.crawl4ai.com/api/arun/#__codelineno-13-2) from crawl4ai import AsyncWebCrawler, CrawlerRunConfig, CacheMode [](https://docs.crawl4ai.com/api/arun/#__codelineno-13-3) from crawl4ai import JsonCssExtractionStrategy [](https://docs.crawl4ai.com/api/arun/#__codelineno-13-4) [](https://docs.crawl4ai.com/api/arun/#__codelineno-13-5) async def main(): [](https://docs.crawl4ai.com/api/arun/#__codelineno-13-6) # Example schema [](https://docs.crawl4ai.com/api/arun/#__codelineno-13-7) schema = { [](https://docs.crawl4ai.com/api/arun/#__codelineno-13-8) "name": "Articles", [](https://docs.crawl4ai.com/api/arun/#__codelineno-13-9) "baseSelector": "article.post", [](https://docs.crawl4ai.com/api/arun/#__codelineno-13-10) "fields": [ [](https://docs.crawl4ai.com/api/arun/#__codelineno-13-11) {"name": "title", "selector": "h2", "type": "text"}, [](https://docs.crawl4ai.com/api/arun/#__codelineno-13-12) {"name": "link", "selector": "a", "type": "attribute", "attribute": "href"} [](https://docs.crawl4ai.com/api/arun/#__codelineno-13-13) ] [](https://docs.crawl4ai.com/api/arun/#__codelineno-13-14) } [](https://docs.crawl4ai.com/api/arun/#__codelineno-13-15) [](https://docs.crawl4ai.com/api/arun/#__codelineno-13-16) run_config = CrawlerRunConfig( [](https://docs.crawl4ai.com/api/arun/#__codelineno-13-17) # Core [](https://docs.crawl4ai.com/api/arun/#__codelineno-13-18) verbose=True, [](https://docs.crawl4ai.com/api/arun/#__codelineno-13-19) cache_mode=CacheMode.ENABLED, [](https://docs.crawl4ai.com/api/arun/#__codelineno-13-20) check_robots_txt=True, # Respect robots.txt rules [](https://docs.crawl4ai.com/api/arun/#__codelineno-13-21) [](https://docs.crawl4ai.com/api/arun/#__codelineno-13-22) # Content [](https://docs.crawl4ai.com/api/arun/#__codelineno-13-23) word_count_threshold=10, [](https://docs.crawl4ai.com/api/arun/#__codelineno-13-24) css_selector="main.content", [](https://docs.crawl4ai.com/api/arun/#__codelineno-13-25) excluded_tags=["nav", "footer"], [](https://docs.crawl4ai.com/api/arun/#__codelineno-13-26) exclude_external_links=True, [](https://docs.crawl4ai.com/api/arun/#__codelineno-13-27) [](https://docs.crawl4ai.com/api/arun/#__codelineno-13-28) # Page & JS [](https://docs.crawl4ai.com/api/arun/#__codelineno-13-29) js_code="document.querySelector('.show-more')?.click();", [](https://docs.crawl4ai.com/api/arun/#__codelineno-13-30) wait_for="css:.loaded-block", [](https://docs.crawl4ai.com/api/arun/#__codelineno-13-31) page_timeout=30000, [](https://docs.crawl4ai.com/api/arun/#__codelineno-13-32) [](https://docs.crawl4ai.com/api/arun/#__codelineno-13-33) # Extraction [](https://docs.crawl4ai.com/api/arun/#__codelineno-13-34) extraction_strategy=JsonCssExtractionStrategy(schema), [](https://docs.crawl4ai.com/api/arun/#__codelineno-13-35) [](https://docs.crawl4ai.com/api/arun/#__codelineno-13-36) # Session [](https://docs.crawl4ai.com/api/arun/#__codelineno-13-37) session_id="persistent_session", [](https://docs.crawl4ai.com/api/arun/#__codelineno-13-38) [](https://docs.crawl4ai.com/api/arun/#__codelineno-13-39) # Media [](https://docs.crawl4ai.com/api/arun/#__codelineno-13-40) screenshot=True, [](https://docs.crawl4ai.com/api/arun/#__codelineno-13-41) pdf=True, [](https://docs.crawl4ai.com/api/arun/#__codelineno-13-42) [](https://docs.crawl4ai.com/api/arun/#__codelineno-13-43) # Anti-bot [](https://docs.crawl4ai.com/api/arun/#__codelineno-13-44) simulate_user=True, [](https://docs.crawl4ai.com/api/arun/#__codelineno-13-45) magic=True, [](https://docs.crawl4ai.com/api/arun/#__codelineno-13-46) ) [](https://docs.crawl4ai.com/api/arun/#__codelineno-13-47) [](https://docs.crawl4ai.com/api/arun/#__codelineno-13-48) async with AsyncWebCrawler() as crawler: [](https://docs.crawl4ai.com/api/arun/#__codelineno-13-49) result = await crawler.arun("https://example.com/posts", config=run_config) [](https://docs.crawl4ai.com/api/arun/#__codelineno-13-50) if result.success: [](https://docs.crawl4ai.com/api/arun/#__codelineno-13-51) print("HTML length:", len(result.cleaned_html)) [](https://docs.crawl4ai.com/api/arun/#__codelineno-13-52) print("Extraction JSON:", result.extracted_content) [](https://docs.crawl4ai.com/api/arun/#__codelineno-13-53) if result.screenshot: [](https://docs.crawl4ai.com/api/arun/#__codelineno-13-54) print("Screenshot length:", len(result.screenshot)) [](https://docs.crawl4ai.com/api/arun/#__codelineno-13-55) if result.pdf: [](https://docs.crawl4ai.com/api/arun/#__codelineno-13-56) print("PDF bytes length:", len(result.pdf)) [](https://docs.crawl4ai.com/api/arun/#__codelineno-13-57) else: [](https://docs.crawl4ai.com/api/arun/#__codelineno-13-58) print("Error:", result.error_message) [](https://docs.crawl4ai.com/api/arun/#__codelineno-13-59) [](https://docs.crawl4ai.com/api/arun/#__codelineno-13-60) if __name__ == "__main__": [](https://docs.crawl4ai.com/api/arun/#__codelineno-13-61) asyncio.run(main())` **What we covered**: 1. **Crawling** the main content region, ignoring external links.  2. Running **JavaScript** to click “.show-more”.  3. **Waiting** for “.loaded-block” to appear.  4. Generating a **screenshot** & **PDF** of the final page.  5. Extracting repeated “article.post” elements with a **CSS-based** extraction strategy. * * * 9. Best Practices ----------------- 1. **Use `BrowserConfig` for global browser** settings (headless, user agent).  2. **Use `CrawlerRunConfig`** to handle the **specific** crawl needs: content filtering, caching, JS, screenshot, extraction, etc.  3. Keep your **parameters consistent** in run configs—especially if you’re part of a large codebase with multiple crawls.  4. **Limit** large concurrency (`semaphore_count`) if the site or your system can’t handle it.  5. For dynamic pages, set `js_code` or `scan_full_page` so you load all content. * * * 10. Conclusion -------------- All parameters that used to be direct arguments to `arun()` now belong in **`CrawlerRunConfig`**. This approach: * Makes code **clearer** and **more maintainable**.  * Minimizes confusion about which arguments affect global vs. per-crawl behavior.  * Allows you to create **reusable** config objects for different pages or tasks. For a **full** reference, check out the [CrawlerRunConfig Docs](https://docs.crawl4ai.com/api/parameters/) .  Happy crawling with your **structured, flexible** config approach! * * * \> Feedback --- # CrawlResult - Crawl4AI Documentation (v0.8.x) `CrawlResult` Reference ======================= The **`CrawlResult`** class encapsulates everything returned after a single crawl operation. It provides the **raw or processed content**, details on links and media, plus optional metadata (like screenshots, PDFs, or extracted JSON). **Location**: `crawl4ai/crawler/models.py` (for reference) `[](https://docs.crawl4ai.com/api/crawl-result/#__codelineno-0-1) class CrawlResult(BaseModel): [](https://docs.crawl4ai.com/api/crawl-result/#__codelineno-0-2) url: str [](https://docs.crawl4ai.com/api/crawl-result/#__codelineno-0-3) html: str [](https://docs.crawl4ai.com/api/crawl-result/#__codelineno-0-4) success: bool [](https://docs.crawl4ai.com/api/crawl-result/#__codelineno-0-5) cleaned_html: Optional[str] = None [](https://docs.crawl4ai.com/api/crawl-result/#__codelineno-0-6) fit_html: Optional[str] = None # Preprocessed HTML optimized for extraction [](https://docs.crawl4ai.com/api/crawl-result/#__codelineno-0-7) media: Dict[str, List[Dict]] = {} [](https://docs.crawl4ai.com/api/crawl-result/#__codelineno-0-8) links: Dict[str, List[Dict]] = {} [](https://docs.crawl4ai.com/api/crawl-result/#__codelineno-0-9) downloaded_files: Optional[List[str]] = None [](https://docs.crawl4ai.com/api/crawl-result/#__codelineno-0-10) screenshot: Optional[str] = None [](https://docs.crawl4ai.com/api/crawl-result/#__codelineno-0-11) pdf : Optional[bytes] = None [](https://docs.crawl4ai.com/api/crawl-result/#__codelineno-0-12) mhtml: Optional[str] = None [](https://docs.crawl4ai.com/api/crawl-result/#__codelineno-0-13) markdown: Optional[Union[str, MarkdownGenerationResult]] = None [](https://docs.crawl4ai.com/api/crawl-result/#__codelineno-0-14) extracted_content: Optional[str] = None [](https://docs.crawl4ai.com/api/crawl-result/#__codelineno-0-15) metadata: Optional[dict] = None [](https://docs.crawl4ai.com/api/crawl-result/#__codelineno-0-16) error_message: Optional[str] = None [](https://docs.crawl4ai.com/api/crawl-result/#__codelineno-0-17) session_id: Optional[str] = None [](https://docs.crawl4ai.com/api/crawl-result/#__codelineno-0-18) response_headers: Optional[dict] = None [](https://docs.crawl4ai.com/api/crawl-result/#__codelineno-0-19) status_code: Optional[int] = None [](https://docs.crawl4ai.com/api/crawl-result/#__codelineno-0-20) redirected_status_code: Optional[int] = None [](https://docs.crawl4ai.com/api/crawl-result/#__codelineno-0-21) ssl_certificate: Optional[SSLCertificate] = None [](https://docs.crawl4ai.com/api/crawl-result/#__codelineno-0-22) dispatch_result: Optional[DispatchResult] = None [](https://docs.crawl4ai.com/api/crawl-result/#__codelineno-0-23) ...` Below is a **field-by-field** explanation and possible usage patterns. * * * 1\. Basic Crawl Info -------------------- ### 1.1 **`url`** _(str)_ **What**: The final crawled URL (after any redirects). **Usage**: `[](https://docs.crawl4ai.com/api/crawl-result/#__codelineno-1-1) print(result.url) # e.g., "https://example.com/"` ### 1.2 **`success`** _(bool)_ **What**: `True` if the crawl pipeline ended without major errors; `False` otherwise. **Usage**: `[](https://docs.crawl4ai.com/api/crawl-result/#__codelineno-2-1) if not result.success: [](https://docs.crawl4ai.com/api/crawl-result/#__codelineno-2-2) print(f"Crawl failed: {result.error_message}")` ### 1.3 **`status_code`** _(Optional\[int\])_ **What**: The page's HTTP status code (e.g., 200, 404). When the page was reached via redirect, this is the status code of the **first** response in the redirect chain (e.g., 301 or 302). **Usage**: `[](https://docs.crawl4ai.com/api/crawl-result/#__codelineno-3-1) if result.status_code == 404: [](https://docs.crawl4ai.com/api/crawl-result/#__codelineno-3-2) print("Page not found!")` ### 1.4 **`redirected_status_code`** _(Optional\[int\])_ **What**: The HTTP status code of the **final** redirect destination. For a 302→200 redirect, `status_code` is 302 and `redirected_status_code` is 200. `None` for non-HTTP requests (raw HTML, local files). **Usage**: `[](https://docs.crawl4ai.com/api/crawl-result/#__codelineno-4-1) if result.status_code in (301, 302) and result.redirected_status_code == 200: [](https://docs.crawl4ai.com/api/crawl-result/#__codelineno-4-2) print(f"Redirected to {result.redirected_url} (OK)")` ### 1.5 **`error_message`** _(Optional\[str\])_ **What**: If `success=False`, a textual description of the failure. **Usage**: `[](https://docs.crawl4ai.com/api/crawl-result/#__codelineno-5-1) if not result.success: [](https://docs.crawl4ai.com/api/crawl-result/#__codelineno-5-2) print("Error:", result.error_message)` ### 1.5 **`session_id`** _(Optional\[str\])_ **What**: The ID used for reusing a browser context across multiple calls. **Usage**: `[](https://docs.crawl4ai.com/api/crawl-result/#__codelineno-6-1) # If you used session_id="login_session" in CrawlerRunConfig, see it here: [](https://docs.crawl4ai.com/api/crawl-result/#__codelineno-6-2) print("Session:", result.session_id)` ### 1.6 **`response_headers`** _(Optional\[dict\])_ **What**: Final HTTP response headers. **Usage**: `[](https://docs.crawl4ai.com/api/crawl-result/#__codelineno-7-1) if result.response_headers: [](https://docs.crawl4ai.com/api/crawl-result/#__codelineno-7-2) print("Server:", result.response_headers.get("Server", "Unknown"))` ### 1.7 **`ssl_certificate`** _(Optional\[SSLCertificate\])_ **What**: If `fetch_ssl_certificate=True` in your CrawlerRunConfig, **`result.ssl_certificate`** contains a [**`SSLCertificate`**](https://docs.crawl4ai.com/advanced/ssl-certificate/) object describing the site's certificate. You can export the cert in multiple formats (PEM/DER/JSON) or access its properties like `issuer`, `subject`, `valid_from`, `valid_until`, etc. **Usage**: `[](https://docs.crawl4ai.com/api/crawl-result/#__codelineno-8-1) if result.ssl_certificate: [](https://docs.crawl4ai.com/api/crawl-result/#__codelineno-8-2) print("Issuer:", result.ssl_certificate.issuer)` * * * 2\. Raw / Cleaned Content ------------------------- ### 2.1 **`html`** _(str)_ **What**: The **original** unmodified HTML from the final page load. **Usage**: `[](https://docs.crawl4ai.com/api/crawl-result/#__codelineno-9-1) # Possibly large [](https://docs.crawl4ai.com/api/crawl-result/#__codelineno-9-2) print(len(result.html))` ### 2.2 **`cleaned_html`** _(Optional\[str\])_ **What**: A sanitized HTML version—scripts, styles, or excluded tags are removed based on your `CrawlerRunConfig`. **Usage**: `[](https://docs.crawl4ai.com/api/crawl-result/#__codelineno-10-1) print(result.cleaned_html[:500]) # Show a snippet` * * * 3\. Markdown Fields ------------------- ### 3.1 The Markdown Generation Approach Crawl4AI can convert HTML→Markdown, optionally including: * **Raw** markdown * **Links as citations** (with a references section) * **Fit** markdown if a **content filter** is used (like Pruning or BM25) **`MarkdownGenerationResult`** includes: - **`raw_markdown`** _(str)_: The full HTML→Markdown conversion. \- **`markdown_with_citations`** _(str)_: Same markdown, but with link references as academic-style citations. \- **`references_markdown`** _(str)_: The reference list or footnotes at the end. \- **`fit_markdown`** _(Optional\[str\])_: If content filtering (Pruning/BM25) was applied, the filtered "fit" text. \- **`fit_html`** _(Optional\[str\])_: The HTML that led to `fit_markdown`. **Usage**: `[](https://docs.crawl4ai.com/api/crawl-result/#__codelineno-11-1) if result.markdown: [](https://docs.crawl4ai.com/api/crawl-result/#__codelineno-11-2) md_res = result.markdown [](https://docs.crawl4ai.com/api/crawl-result/#__codelineno-11-3) print("Raw MD:", md_res.raw_markdown[:300]) [](https://docs.crawl4ai.com/api/crawl-result/#__codelineno-11-4) print("Citations MD:", md_res.markdown_with_citations[:300]) [](https://docs.crawl4ai.com/api/crawl-result/#__codelineno-11-5) print("References:", md_res.references_markdown) [](https://docs.crawl4ai.com/api/crawl-result/#__codelineno-11-6) if md_res.fit_markdown: [](https://docs.crawl4ai.com/api/crawl-result/#__codelineno-11-7) print("Pruned text:", md_res.fit_markdown[:300])` ### 3.2 **`markdown`** _(Optional\[Union\[str, MarkdownGenerationResult\]\])_ **What**: Holds the `MarkdownGenerationResult`. **Usage**: `[](https://docs.crawl4ai.com/api/crawl-result/#__codelineno-12-1) print(result.markdown.raw_markdown[:200]) [](https://docs.crawl4ai.com/api/crawl-result/#__codelineno-12-2) print(result.markdown.fit_markdown) [](https://docs.crawl4ai.com/api/crawl-result/#__codelineno-12-3) print(result.markdown.fit_html)` **Important**: "Fit" content (in `fit_markdown`/`fit_html`) exists in result.markdown, only if you used a **filter** (like **PruningContentFilter** or **BM25ContentFilter**) within a `MarkdownGenerationStrategy`. * * * 4\. Media & Links ----------------- ### 4.1 **`media`** _(Dict\[str, List\[Dict\]\])_ **What**: Contains info about discovered images, videos, or audio. Typically keys: `"images"`, `"videos"`, `"audios"`. **Common Fields** in each item: * `src` _(str)_: Media URL * `alt` or `title` _(str)_: Descriptive text * `score` _(float)_: Relevance score if the crawler's heuristic found it "important" * `desc` or `description` _(Optional\[str\])_: Additional context extracted from surrounding text **Usage**: `[](https://docs.crawl4ai.com/api/crawl-result/#__codelineno-13-1) images = result.media.get("images", []) [](https://docs.crawl4ai.com/api/crawl-result/#__codelineno-13-2) for img in images: [](https://docs.crawl4ai.com/api/crawl-result/#__codelineno-13-3) if img.get("score", 0) > 5: [](https://docs.crawl4ai.com/api/crawl-result/#__codelineno-13-4) print("High-value image:", img["src"])` ### 4.2 **`links`** _(Dict\[str, List\[Dict\]\])_ **What**: Holds internal and external link data. Usually two keys: `"internal"` and `"external"`. **Common Fields**: * `href` _(str)_: The link target * `text` _(str)_: Link text * `title` _(str)_: Title attribute * `context` _(str)_: Surrounding text snippet * `domain` _(str)_: If external, the domain **Usage**: `[](https://docs.crawl4ai.com/api/crawl-result/#__codelineno-14-1) for link in result.links["internal"]: [](https://docs.crawl4ai.com/api/crawl-result/#__codelineno-14-2) print(f"Internal link to {link['href']} with text {link['text']}")` * * * 5\. Additional Fields --------------------- ### 5.1 **`extracted_content`** _(Optional\[str\])_ **What**: If you used **`extraction_strategy`** (CSS, LLM, etc.), the structured output (JSON). **Usage**: `[](https://docs.crawl4ai.com/api/crawl-result/#__codelineno-15-1) if result.extracted_content: [](https://docs.crawl4ai.com/api/crawl-result/#__codelineno-15-2) data = json.loads(result.extracted_content) [](https://docs.crawl4ai.com/api/crawl-result/#__codelineno-15-3) print(data)` ### 5.2 **`downloaded_files`** _(Optional\[List\[str\]\])_ **What**: If `accept_downloads=True` in your `BrowserConfig` + `downloads_path`, lists local file paths for downloaded items. **Usage**: `[](https://docs.crawl4ai.com/api/crawl-result/#__codelineno-16-1) if result.downloaded_files: [](https://docs.crawl4ai.com/api/crawl-result/#__codelineno-16-2) for file_path in result.downloaded_files: [](https://docs.crawl4ai.com/api/crawl-result/#__codelineno-16-3) print("Downloaded:", file_path)` ### 5.3 **`screenshot`** _(Optional\[str\])_ **What**: Base64-encoded screenshot if `screenshot=True` in `CrawlerRunConfig`. **Usage**: `[](https://docs.crawl4ai.com/api/crawl-result/#__codelineno-17-1) import base64 [](https://docs.crawl4ai.com/api/crawl-result/#__codelineno-17-2) if result.screenshot: [](https://docs.crawl4ai.com/api/crawl-result/#__codelineno-17-3) with open("page.png", "wb") as f: [](https://docs.crawl4ai.com/api/crawl-result/#__codelineno-17-4) f.write(base64.b64decode(result.screenshot))` ### 5.4 **`pdf`** _(Optional\[bytes\])_ **What**: Raw PDF bytes if `pdf=True` in `CrawlerRunConfig`. **Usage**: `[](https://docs.crawl4ai.com/api/crawl-result/#__codelineno-18-1) if result.pdf: [](https://docs.crawl4ai.com/api/crawl-result/#__codelineno-18-2) with open("page.pdf", "wb") as f: [](https://docs.crawl4ai.com/api/crawl-result/#__codelineno-18-3) f.write(result.pdf)` ### 5.5 **`mhtml`** _(Optional\[str\])_ **What**: MHTML snapshot of the page if `capture_mhtml=True` in `CrawlerRunConfig`. MHTML (MIME HTML) format preserves the entire web page with all its resources (CSS, images, scripts, etc.) in a single file. **Usage**: `[](https://docs.crawl4ai.com/api/crawl-result/#__codelineno-19-1) if result.mhtml: [](https://docs.crawl4ai.com/api/crawl-result/#__codelineno-19-2) with open("page.mhtml", "w", encoding="utf-8") as f: [](https://docs.crawl4ai.com/api/crawl-result/#__codelineno-19-3) f.write(result.mhtml)` ### 5.6 **`metadata`** _(Optional\[dict\])_ **What**: Page-level metadata if discovered (title, description, OG data, etc.). **Usage**: `[](https://docs.crawl4ai.com/api/crawl-result/#__codelineno-20-1) if result.metadata: [](https://docs.crawl4ai.com/api/crawl-result/#__codelineno-20-2) print("Title:", result.metadata.get("title")) [](https://docs.crawl4ai.com/api/crawl-result/#__codelineno-20-3) print("Author:", result.metadata.get("author"))` * * * 6\. `dispatch_result` (optional) -------------------------------- A `DispatchResult` object providing additional concurrency and resource usage information when crawling URLs in parallel (e.g., via `arun_many()` with custom dispatchers). It contains: * **`task_id`**: A unique identifier for the parallel task. * **`memory_usage`** (float): The memory (in MB) used at the time of completion. * **`peak_memory`** (float): The peak memory usage (in MB) recorded during the task's execution. * **`start_time`** / **`end_time`** (datetime): Time range for this crawling task. * **`error_message`** (str): Any dispatcher- or concurrency-related error encountered. `[](https://docs.crawl4ai.com/api/crawl-result/#__codelineno-21-1) # Example usage: [](https://docs.crawl4ai.com/api/crawl-result/#__codelineno-21-2) for result in results: [](https://docs.crawl4ai.com/api/crawl-result/#__codelineno-21-3) if result.success and result.dispatch_result: [](https://docs.crawl4ai.com/api/crawl-result/#__codelineno-21-4) dr = result.dispatch_result [](https://docs.crawl4ai.com/api/crawl-result/#__codelineno-21-5) print(f"URL: {result.url}, Task ID: {dr.task_id}") [](https://docs.crawl4ai.com/api/crawl-result/#__codelineno-21-6) print(f"Memory: {dr.memory_usage:.1f} MB (Peak: {dr.peak_memory:.1f} MB)") [](https://docs.crawl4ai.com/api/crawl-result/#__codelineno-21-7) print(f"Duration: {dr.end_time - dr.start_time}")` > **Note**: This field is typically populated when using `arun_many(...)` alongside a **dispatcher** (e.g., `MemoryAdaptiveDispatcher` or `SemaphoreDispatcher`). If no concurrency or dispatcher is used, `dispatch_result` may remain `None`. * * * 7\. Network Requests & Console Messages --------------------------------------- When you enable network and console message capturing in `CrawlerRunConfig` using `capture_network_requests=True` and `capture_console_messages=True`, the `CrawlResult` will include these fields: ### 7.1 **`network_requests`** _(Optional\[List\[Dict\[str, Any\]\]\])_ **What**: A list of dictionaries containing information about all network requests, responses, and failures captured during the crawl. **Structure**: - Each item has an `event_type` field that can be `"request"`, `"response"`, or `"request_failed"`. - Request events include `url`, `method`, `headers`, `post_data`, `resource_type`, and `is_navigation_request`. - Response events include `url`, `status`, `status_text`, `headers`, and `request_timing`. - Failed request events include `url`, `method`, `resource_type`, and `failure_text`. - All events include a `timestamp` field. **Usage**: `[](https://docs.crawl4ai.com/api/crawl-result/#__codelineno-22-1) if result.network_requests: [](https://docs.crawl4ai.com/api/crawl-result/#__codelineno-22-2) # Count different types of events [](https://docs.crawl4ai.com/api/crawl-result/#__codelineno-22-3) requests = [r for r in result.network_requests if r.get("event_type") == "request"] [](https://docs.crawl4ai.com/api/crawl-result/#__codelineno-22-4) responses = [r for r in result.network_requests if r.get("event_type") == "response"] [](https://docs.crawl4ai.com/api/crawl-result/#__codelineno-22-5) failures = [r for r in result.network_requests if r.get("event_type") == "request_failed"] [](https://docs.crawl4ai.com/api/crawl-result/#__codelineno-22-6) [](https://docs.crawl4ai.com/api/crawl-result/#__codelineno-22-7) print(f"Captured {len(requests)} requests, {len(responses)} responses, and {len(failures)} failures") [](https://docs.crawl4ai.com/api/crawl-result/#__codelineno-22-8) [](https://docs.crawl4ai.com/api/crawl-result/#__codelineno-22-9) # Analyze API calls [](https://docs.crawl4ai.com/api/crawl-result/#__codelineno-22-10) api_calls = [r for r in requests if "api" in r.get("url", "")] [](https://docs.crawl4ai.com/api/crawl-result/#__codelineno-22-11) [](https://docs.crawl4ai.com/api/crawl-result/#__codelineno-22-12) # Identify failed resources [](https://docs.crawl4ai.com/api/crawl-result/#__codelineno-22-13) for failure in failures: [](https://docs.crawl4ai.com/api/crawl-result/#__codelineno-22-14) print(f"Failed to load: {failure.get('url')} - {failure.get('failure_text')}")` ### 7.2 **`console_messages`** _(Optional\[List\[Dict\[str, Any\]\]\])_ **What**: A list of dictionaries containing all browser console messages captured during the crawl. **Structure**: - Each item has a `type` field indicating the message type (e.g., `"log"`, `"error"`, `"warning"`, etc.). - The `text` field contains the actual message text. - Some messages include `location` information (URL, line, column). - All messages include a `timestamp` field. **Usage**: `[](https://docs.crawl4ai.com/api/crawl-result/#__codelineno-23-1) if result.console_messages: [](https://docs.crawl4ai.com/api/crawl-result/#__codelineno-23-2) # Count messages by type [](https://docs.crawl4ai.com/api/crawl-result/#__codelineno-23-3) message_types = {} [](https://docs.crawl4ai.com/api/crawl-result/#__codelineno-23-4) for msg in result.console_messages: [](https://docs.crawl4ai.com/api/crawl-result/#__codelineno-23-5) msg_type = msg.get("type", "unknown") [](https://docs.crawl4ai.com/api/crawl-result/#__codelineno-23-6) message_types[msg_type] = message_types.get(msg_type, 0) + 1 [](https://docs.crawl4ai.com/api/crawl-result/#__codelineno-23-7) [](https://docs.crawl4ai.com/api/crawl-result/#__codelineno-23-8) print(f"Message type counts: {message_types}") [](https://docs.crawl4ai.com/api/crawl-result/#__codelineno-23-9) [](https://docs.crawl4ai.com/api/crawl-result/#__codelineno-23-10) # Display errors (which are usually most important) [](https://docs.crawl4ai.com/api/crawl-result/#__codelineno-23-11) for msg in result.console_messages: [](https://docs.crawl4ai.com/api/crawl-result/#__codelineno-23-12) if msg.get("type") == "error": [](https://docs.crawl4ai.com/api/crawl-result/#__codelineno-23-13) print(f"Error: {msg.get('text')}")` These fields provide deep visibility into the page's network activity and browser console, which is invaluable for debugging, security analysis, and understanding complex web applications. For more details on network and console capturing, see the [Network & Console Capture documentation](https://docs.crawl4ai.com/advanced/network-console-capture/) . * * * 8\. Example: Accessing Everything --------------------------------- `[](https://docs.crawl4ai.com/api/crawl-result/#__codelineno-24-1) async def handle_result(result: CrawlResult): [](https://docs.crawl4ai.com/api/crawl-result/#__codelineno-24-2) if not result.success: [](https://docs.crawl4ai.com/api/crawl-result/#__codelineno-24-3) print("Crawl error:", result.error_message) [](https://docs.crawl4ai.com/api/crawl-result/#__codelineno-24-4) return [](https://docs.crawl4ai.com/api/crawl-result/#__codelineno-24-5) [](https://docs.crawl4ai.com/api/crawl-result/#__codelineno-24-6) # Basic info [](https://docs.crawl4ai.com/api/crawl-result/#__codelineno-24-7) print("Crawled URL:", result.url) [](https://docs.crawl4ai.com/api/crawl-result/#__codelineno-24-8) print("Status code:", result.status_code) [](https://docs.crawl4ai.com/api/crawl-result/#__codelineno-24-9) [](https://docs.crawl4ai.com/api/crawl-result/#__codelineno-24-10) # HTML [](https://docs.crawl4ai.com/api/crawl-result/#__codelineno-24-11) print("Original HTML size:", len(result.html)) [](https://docs.crawl4ai.com/api/crawl-result/#__codelineno-24-12) print("Cleaned HTML size:", len(result.cleaned_html or "")) [](https://docs.crawl4ai.com/api/crawl-result/#__codelineno-24-13) [](https://docs.crawl4ai.com/api/crawl-result/#__codelineno-24-14) # Markdown output [](https://docs.crawl4ai.com/api/crawl-result/#__codelineno-24-15) if result.markdown: [](https://docs.crawl4ai.com/api/crawl-result/#__codelineno-24-16) print("Raw Markdown:", result.markdown.raw_markdown[:300]) [](https://docs.crawl4ai.com/api/crawl-result/#__codelineno-24-17) print("Citations Markdown:", result.markdown.markdown_with_citations[:300]) [](https://docs.crawl4ai.com/api/crawl-result/#__codelineno-24-18) if result.markdown.fit_markdown: [](https://docs.crawl4ai.com/api/crawl-result/#__codelineno-24-19) print("Fit Markdown:", result.markdown.fit_markdown[:200]) [](https://docs.crawl4ai.com/api/crawl-result/#__codelineno-24-20) [](https://docs.crawl4ai.com/api/crawl-result/#__codelineno-24-21) # Media & Links [](https://docs.crawl4ai.com/api/crawl-result/#__codelineno-24-22) if "images" in result.media: [](https://docs.crawl4ai.com/api/crawl-result/#__codelineno-24-23) print("Image count:", len(result.media["images"])) [](https://docs.crawl4ai.com/api/crawl-result/#__codelineno-24-24) if "internal" in result.links: [](https://docs.crawl4ai.com/api/crawl-result/#__codelineno-24-25) print("Internal link count:", len(result.links["internal"])) [](https://docs.crawl4ai.com/api/crawl-result/#__codelineno-24-26) [](https://docs.crawl4ai.com/api/crawl-result/#__codelineno-24-27) # Extraction strategy result [](https://docs.crawl4ai.com/api/crawl-result/#__codelineno-24-28) if result.extracted_content: [](https://docs.crawl4ai.com/api/crawl-result/#__codelineno-24-29) print("Structured data:", result.extracted_content) [](https://docs.crawl4ai.com/api/crawl-result/#__codelineno-24-30) [](https://docs.crawl4ai.com/api/crawl-result/#__codelineno-24-31) # Screenshot/PDF/MHTML [](https://docs.crawl4ai.com/api/crawl-result/#__codelineno-24-32) if result.screenshot: [](https://docs.crawl4ai.com/api/crawl-result/#__codelineno-24-33) print("Screenshot length:", len(result.screenshot)) [](https://docs.crawl4ai.com/api/crawl-result/#__codelineno-24-34) if result.pdf: [](https://docs.crawl4ai.com/api/crawl-result/#__codelineno-24-35) print("PDF bytes length:", len(result.pdf)) [](https://docs.crawl4ai.com/api/crawl-result/#__codelineno-24-36) if result.mhtml: [](https://docs.crawl4ai.com/api/crawl-result/#__codelineno-24-37) print("MHTML length:", len(result.mhtml)) [](https://docs.crawl4ai.com/api/crawl-result/#__codelineno-24-38) [](https://docs.crawl4ai.com/api/crawl-result/#__codelineno-24-39) # Network and console capturing [](https://docs.crawl4ai.com/api/crawl-result/#__codelineno-24-40) if result.network_requests: [](https://docs.crawl4ai.com/api/crawl-result/#__codelineno-24-41) print(f"Network requests captured: {len(result.network_requests)}") [](https://docs.crawl4ai.com/api/crawl-result/#__codelineno-24-42) # Analyze request types [](https://docs.crawl4ai.com/api/crawl-result/#__codelineno-24-43) req_types = {} [](https://docs.crawl4ai.com/api/crawl-result/#__codelineno-24-44) for req in result.network_requests: [](https://docs.crawl4ai.com/api/crawl-result/#__codelineno-24-45) if "resource_type" in req: [](https://docs.crawl4ai.com/api/crawl-result/#__codelineno-24-46) req_types[req["resource_type"]] = req_types.get(req["resource_type"], 0) + 1 [](https://docs.crawl4ai.com/api/crawl-result/#__codelineno-24-47) print(f"Resource types: {req_types}") [](https://docs.crawl4ai.com/api/crawl-result/#__codelineno-24-48) [](https://docs.crawl4ai.com/api/crawl-result/#__codelineno-24-49) if result.console_messages: [](https://docs.crawl4ai.com/api/crawl-result/#__codelineno-24-50) print(f"Console messages captured: {len(result.console_messages)}") [](https://docs.crawl4ai.com/api/crawl-result/#__codelineno-24-51) # Count by message type [](https://docs.crawl4ai.com/api/crawl-result/#__codelineno-24-52) msg_types = {} [](https://docs.crawl4ai.com/api/crawl-result/#__codelineno-24-53) for msg in result.console_messages: [](https://docs.crawl4ai.com/api/crawl-result/#__codelineno-24-54) msg_types[msg.get("type", "unknown")] = msg_types.get(msg.get("type", "unknown"), 0) + 1 [](https://docs.crawl4ai.com/api/crawl-result/#__codelineno-24-55) print(f"Message types: {msg_types}")` * * * 9\. Key Points & Future ----------------------- 1. **Deprecated legacy properties of CrawlResult** \- `markdown_v2` - Deprecated in v0.5. Just use `markdown`. It holds the `MarkdownGenerationResult` now! - `fit_markdown` and `fit_html` - Deprecated in v0.5. They can now be accessed via `MarkdownGenerationResult` in `result.markdown`. eg: `result.markdown.fit_markdown` and `result.markdown.fit_html` 2. **Fit Content** \- **`fit_markdown`** and **`fit_html`** appear in MarkdownGenerationResult, only if you used a content filter (like **PruningContentFilter** or **BM25ContentFilter**) inside your **MarkdownGenerationStrategy** or set them directly. \- If no filter is used, they remain `None`. 3. **References & Citations** \- If you enable link citations in your `DefaultMarkdownGenerator` (`options={"citations": True}`), you’ll see `markdown_with_citations` plus a **`references_markdown`** block. This helps large language models or academic-like referencing. 4. **Links & Media** \- `links["internal"]` and `links["external"]` group discovered anchors by domain. \- `media["images"]` / `["videos"]` / `["audios"]` store extracted media elements with optional scoring or context. 5. **Error Cases** \- If `success=False`, check `error_message` (e.g., timeouts, invalid URLs). \- `status_code` might be `None` if we failed before an HTTP response. Use **`CrawlResult`** to glean all final outputs and feed them into your data pipelines, AI models, or archives. With the synergy of a properly configured **BrowserConfig** and **CrawlerRunConfig**, the crawler can produce robust, structured results here in **`CrawlResult`**. * * * \> Feedback --- # Build - Crawl4AI Documentation (v0.8.x) O**Prompt for AI Coding Assistant: Create an Interactive LLM Context Builder Page** **Objective:** Your task is to create an interactive HTML webpage with JavaScript functionality that allows users to select and combine different `crawl4ai` LLM context files into a single downloadable Markdown (`.md`) file. This tool will empower users to craft tailored context for their AI assistants based on their specific needs. **Core Functionality:** 1. **Display `crawl4ai` Components:** The page will list all available `crawl4ai` documentation components. 2. **Select Context Types:** For each component, users can select which types of context they want to include: * Memory (API facts) * Reasoning (How-to/why) * Examples (Code snippets) (All should be selected by default for each initially selected component). 3. **Special "Aggregate" Contexts:** Include options for special, pre-combined contexts: * "Vibe Coding" (a curated mix for general AI prompting) * "All Library Context" (a comprehensive aggregation of all memory, reasoning, and examples for the entire library). 4. **Fetch and Concatenate:** When the user clicks a "Download Combined Context" button: * The JavaScript will fetch the content of all selected Markdown files from the server (from a predefined folder, e.g., `/llmtxt/`). * It will concatenate the content of these files into a single string. 5. **Client-Side Download:** The concatenated content will be offered to the user as a download (e.g., `custom_crawl4ai_context.md`). **Input/Assumptions:** * **Context Files Location:** All individual context Markdown files are located on the server in a publicly accessible folder named `llmtxt/`. * **File Naming Convention:** Files follow the pattern: `crawl4ai_{{component_name}}_[memory|reasoning|examples]_content.llm.md`. * `{{component_name}}` can contain underscores (e.g., `deep_crawling`, `config_objects`). * The special contexts will have names like `crawl4ai_vibe_content.llm.md` and `crawl4ai_all_content.llm.md`. * **Component List:** You will be provided with a list of `crawl4ai` components. For this implementation, use the following list: * `core` * `config_objects` * `deep_crawling` * `deployment` (covers Installation & Docker Deployment) * `extraction` (covers Structured Data Extraction) * `markdown` (covers Markdown Generation Algorithm) * `pdf_processing` * _(No separate "Vibe Coding" or "All Library Context" in this list, as they are special top-level selections)_ **Detailed UI/UX Requirements:** 1. **Main Page Structure:** * **Header:** "Crawl4AI Interactive LLM Context Builder" * **Introduction:** Briefly explain the purpose of the tool (from the `USING_LLM_CONTEXTS.md` content you helped draft: "Supercharging Your AI Assistant..."). * **Selection Area:** * **Special Aggregate Contexts (Radio Buttons or Prominent Checkboxes):** * \[ \] "Vibe Coding Context" (`crawl4ai_vibe_content.llm.md`) * \[ \] "All Library Context (Comprehensive)" (`crawl4ai_all_content.llm.md`) * _Behavior:_ Selecting one of these might disable individual component selections (or vice-versa) to avoid redundancy, or simply add them to the list. Consider user experience here. A simple approach is that if an aggregate is selected, it's the _only_ thing downloaded. * **Individual Component Selection (Table or List of Checkboxes):** * A section titled "Select Individual Components & Context Types:" * For each component in the provided list: * A master checkbox for the component itself (e.g., `[ ] Core Functionality`). Selected by default. * Nested checkboxes (indented or grouped) for context types, enabled only if the parent component is checked: * `[x] Memory (API Facts)` * `[x] Reasoning (How-to/Why)` * `[x] Examples (Code Snippets)` (These three sub-checkboxes should be selected by default if the parent component is selected). * **Action Button:** * A button: "Generate & Download Combined Context" * **Status/Feedback Area:** (Optional, but good UX) * Display messages like "Fetching files...", "Combining context...", "Download starting..." or error messages. **Final Output:** * A single HTML file (e.g., `interactive_context_builder.html`). * Associated JavaScript code (can be inline within `