# Table of Contents - [PlaywrightCrawler | API | Crawlee for JavaScript · Build reliable crawlers. Fast.](#playwrightcrawler-api-crawlee-for-javascript-build-reliable-crawlers-fast-) - [createAdaptivePlaywrightRouter | API | Crawlee for JavaScript · Build reliable crawlers. Fast.](#createadaptiveplaywrightrouter-api-crawlee-for-javascript-build-reliable-crawlers-fast-) - [AdaptivePlaywrightCrawler | API | Crawlee for JavaScript · Build reliable crawlers. Fast.](#adaptiveplaywrightcrawler-api-crawlee-for-javascript-build-reliable-crawlers-fast-) - [AdaptivePlaywrightCrawlerContext | API | Crawlee for JavaScript · Build reliable crawlers. Fast.](#adaptiveplaywrightcrawlercontext-api-crawlee-for-javascript-build-reliable-crawlers-fast-) - [@crawlee/playwright | API | Crawlee for JavaScript · Build reliable crawlers. Fast.](#-crawlee-playwright-api-crawlee-for-javascript-build-reliable-crawlers-fast-) - [playwrightClickElements | API | Crawlee for JavaScript · Build reliable crawlers. Fast.](#playwrightclickelements-api-crawlee-for-javascript-build-reliable-crawlers-fast-) - [createPlaywrightRouter | API | Crawlee for JavaScript · Build reliable crawlers. Fast.](#createplaywrightrouter-api-crawlee-for-javascript-build-reliable-crawlers-fast-) - [launchPlaywright | API | Crawlee for JavaScript · Build reliable crawlers. Fast.](#launchplaywright-api-crawlee-for-javascript-build-reliable-crawlers-fast-) - [AdaptivePlaywrightCrawlerOptions | API | Crawlee for JavaScript · Build reliable crawlers. Fast.](#adaptiveplaywrightcrawleroptions-api-crawlee-for-javascript-build-reliable-crawlers-fast-) - [PlaywrightCrawlingContext | API | Crawlee for JavaScript · Build reliable crawlers. Fast.](#playwrightcrawlingcontext-api-crawlee-for-javascript-build-reliable-crawlers-fast-) - [PlaywrightHook | API | Crawlee for JavaScript · Build reliable crawlers. Fast.](#playwrighthook-api-crawlee-for-javascript-build-reliable-crawlers-fast-) - [PlaywrightCrawlerOptions | API | Crawlee for JavaScript · Build reliable crawlers. Fast.](#playwrightcrawleroptions-api-crawlee-for-javascript-build-reliable-crawlers-fast-) - [PlaywrightRequestHandler | API | Crawlee for JavaScript · Build reliable crawlers. Fast.](#playwrightrequesthandler-api-crawlee-for-javascript-build-reliable-crawlers-fast-) - [PlaywrightLaunchContext | API | Crawlee for JavaScript · Build reliable crawlers. Fast.](#playwrightlaunchcontext-api-crawlee-for-javascript-build-reliable-crawlers-fast-) - [Changelog | API | Crawlee for JavaScript · Build reliable crawlers. Fast.](#changelog-api-crawlee-for-javascript-build-reliable-crawlers-fast-) - [playwrightUtils | API | Crawlee for JavaScript · Build reliable crawlers. Fast.](#playwrightutils-api-crawlee-for-javascript-build-reliable-crawlers-fast-) --- # PlaywrightCrawler | API | Crawlee for JavaScript · Build reliable crawlers. Fast. [Skip to main content](#__docusaurus_skipToContent_fallback) Version: 3.13 On this page Provides a simple framework for parallel crawling of web pages using headless Chromium, Firefox and Webkit browsers with [Playwright](https://github.com/microsoft/playwright) . The URLs to crawl are fed either from a static list of URLs or from a dynamic queue of URLs enabling recursive crawling of websites. Since `Playwright` uses headless browser to download web pages and extract data, it is useful for crawling of websites that require to execute JavaScript. If the target website doesn't need JavaScript, consider using [CheerioCrawler](/js/api/cheerio-crawler/class/CheerioCrawler) , which downloads the pages using raw HTTP requests and is about 10x faster. The source URLs are represented using [Request](/js/api/core/class/Request) objects that are fed from [RequestList](/js/api/core/class/RequestList) or [RequestQueue](/js/api/core/class/RequestQueue) instances provided by the [PlaywrightCrawlerOptions.requestList](/js/api/playwright-crawler/interface/PlaywrightCrawlerOptions#requestList) or [PlaywrightCrawlerOptions.requestQueue](/js/api/playwright-crawler/interface/PlaywrightCrawlerOptions#requestQueue) constructor options, respectively. If both [PlaywrightCrawlerOptions.requestList](/js/api/playwright-crawler/interface/PlaywrightCrawlerOptions#requestList) and [PlaywrightCrawlerOptions.requestQueue](/js/api/playwright-crawler/interface/PlaywrightCrawlerOptions#requestQueue) are used, the instance first processes URLs from the [RequestList](/js/api/core/class/RequestList) and automatically enqueues all of them to [RequestQueue](/js/api/core/class/RequestQueue) before it starts their processing. This ensures that a single URL is not crawled multiple times. The crawler finishes when there are no more [Request](/js/api/core/class/Request) objects to crawl. `PlaywrightCrawler` opens a new Chrome page (i.e. tab) for each [Request](/js/api/core/class/Request) object to crawl and then calls the function provided by user as the [PlaywrightCrawlerOptions.requestHandler](/js/api/playwright-crawler/interface/PlaywrightCrawlerOptions#requestHandler) option. New pages are only opened when there is enough free CPU and memory available, using the functionality provided by the [AutoscaledPool](/js/api/core/class/AutoscaledPool) class. All [AutoscaledPool](/js/api/core/class/AutoscaledPool) configuration options can be passed to the [PlaywrightCrawlerOptions.autoscaledPoolOptions](/js/api/playwright-crawler/interface/PlaywrightCrawlerOptions#autoscaledPoolOptions) parameter of the `PlaywrightCrawler` constructor. For user convenience, the `minConcurrency` and `maxConcurrency` [AutoscaledPoolOptions](/js/api/core/interface/AutoscaledPoolOptions) are available directly in the `PlaywrightCrawler` constructor. Note that the pool of Playwright instances is internally managed by the [BrowserPool](https://github.com/apify/browser-pool) class. **Example usage:** const crawler = new PlaywrightCrawler({ async requestHandler({ page, request }) { // This function is called to extract data from a single web page // 'page' is an instance of Playwright.Page with page.goto(request.url) already called // 'request' is an instance of Request class with information about the page to load await Dataset.pushData({ title: await page.title(), url: request.url, succeeded: true, }) }, async failedRequestHandler({ request }) { // This function is called when the crawling of a request failed too many times await Dataset.pushData({ url: request.url, succeeded: false, errors: request.errorMessages, }) },});await crawler.run([ 'http://www.example.com/page-1', 'http://www.example.com/page-2',]); ### Hierarchy * [BrowserCrawler](/js/api/browser-crawler/class/BrowserCrawler) <{ browserPlugins: \[[PlaywrightPlugin](/js/api/browser-pool/class/PlaywrightPlugin)\ \] }, LaunchOptions, [PlaywrightCrawlingContext](/js/api/playwright-crawler/interface/PlaywrightCrawlingContext) \> * _PlaywrightCrawler_ * [AdaptivePlaywrightCrawler](/js/api/playwright-crawler/class/AdaptivePlaywrightCrawler) Index[](#Index) ---------------- ### Constructors * [constructor](#constructor) ### Properties * [autoscaledPool](#autoscaledPool) * [browserPool](#browserPool) * [config](#config) * [hasFinishedBefore](#hasFinishedBefore) * [launchContext](#launchContext) * [log](#log) * [proxyConfiguration](#proxyConfiguration) * [requestList](#requestList) * [requestQueue](#requestQueue) * [router](#router) * [running](#running) * [sessionPool](#sessionPool) * [stats](#stats) ### Methods * [addRequests](#addRequests) * [exportData](#exportData) * [getData](#getData) * [getDataset](#getDataset) * [getRequestQueue](#getRequestQueue) * [pushData](#pushData) * [run](#run) * [setStatusMessage](#setStatusMessage) * [stop](#stop) * [useState](#useState) Constructors[](#Constructors) ------------------------------ ### [](#constructor) [](https://github.com/apify/crawlee/blob/master/packages/playwright-crawler/src/internals/playwright-crawler.ts#L204) constructor * **new PlaywrightCrawler**(options, config): [PlaywrightCrawler](/js/api/playwright-crawler/class/PlaywrightCrawler) * All `PlaywrightCrawler` parameters are passed via an options object. * * * #### Parameters * ##### options: [PlaywrightCrawlerOptions](/js/api/playwright-crawler/interface/PlaywrightCrawlerOptions) = {} * ##### config: [Configuration](/js/api/core/class/Configuration) = ... #### Returns [PlaywrightCrawler](/js/api/playwright-crawler/class/PlaywrightCrawler) Properties[](#Properties) -------------------------- ### [](#autoscaledPool) [](https://github.com/apify/crawlee/blob/master/packages/basic-crawler/src/internals/basic-crawler.ts#L480) optionalinheritedautoscaledPool autoscaledPool?: [AutoscaledPool](/js/api/core/class/AutoscaledPool) A reference to the underlying [AutoscaledPool](/js/api/core/class/AutoscaledPool) class that manages the concurrency of the crawler. > _NOTE:_ This property is only initialized after calling the [`crawler.run()`](/js/api/basic-crawler/class/BasicCrawler#run) > function. We can use it to change the concurrency settings on the fly, to pause the crawler by calling [`autoscaledPool.pause()`](/js/api/core/class/AutoscaledPool#pause) > or to abort it by calling [`autoscaledPool.abort()`](/js/api/core/class/AutoscaledPool#abort) > . ### [](#browserPool) [](https://github.com/apify/crawlee/blob/master/packages/browser-crawler/src/internals/browser-crawler.ts#L325) inheritedbrowserPool browserPool: [BrowserPool](/js/api/browser-pool/class/BrowserPool) <{ browserPlugins: \[[PlaywrightPlugin](/js/api/browser-pool/class/PlaywrightPlugin)\ \] }, \[[PlaywrightPlugin](/js/api/browser-pool/class/PlaywrightPlugin)\ \], [BrowserController](/js/api/browser-pool/class/BrowserController) , undefined | LaunchOptions, Browser, undefined | { acceptDownloads?: boolean; baseURL?: string; bypassCSP?: boolean; clientCertificates?: { cert?: Buffer; certPath?: string; key?: Buffer; keyPath?: string; origin: string; passphrase?: string; pfx?: Buffer; pfxPath?: string }\[\]; colorScheme?: null | light | dark | no-preference; deviceScaleFactor?: number; extraHTTPHeaders?: {}; forcedColors?: null | active | none; geolocation?: { accuracy?: number; latitude: number; longitude: number }; hasTouch?: boolean; httpCredentials?: { origin?: string; password: string; send?: unauthorized | always; username: string }; ignoreHTTPSErrors?: boolean; isMobile?: boolean; javaScriptEnabled?: boolean; locale?: string; logger?: Logger; offline?: boolean; permissions?: string\[\]; proxy?: { bypass?: string; password?: string; server: string; username?: string }; recordHar?: { content?: omit | embed | attach; mode?: full | minimal; omitContent?: boolean; path: string; urlFilter?: string | RegExp }; recordVideo?: { dir: string; size?: { height: number; width: number } }; reducedMotion?: null | reduce | no-preference; screen?: { height: number; width: number }; serviceWorkers?: allow | block; storageState?: string | { cookies: { domain: string; expires: number; httpOnly: boolean; name: string; path: string; sameSite: Strict | Lax | None; secure: boolean; value: string }\[\]; origins: { localStorage: { name: string; value: string }\[\]; origin: string }\[\] }; strictSelectors?: boolean; timezoneId?: string; userAgent?: string; videoSize?: { height: number; width: number }; videosPath?: string; viewport?: null | { height: number; width: number } }, Page\>, [LaunchContext](/js/api/browser-pool/class/LaunchContext) , undefined | LaunchOptions, Browser, undefined | { acceptDownloads?: boolean; baseURL?: string; bypassCSP?: boolean; clientCertificates?: { cert?: Buffer; certPath?: string; key?: Buffer; keyPath?: string; origin: string; passphrase?: string; pfx?: Buffer; pfxPath?: string }\[\]; colorScheme?: null | light | dark | no-preference; deviceScaleFactor?: number; extraHTTPHeaders?: {}; forcedColors?: null | active | none; geolocation?: { accuracy?: number; latitude: number; longitude: number }; hasTouch?: boolean; httpCredentials?: { origin?: string; password: string; send?: unauthorized | always; username: string }; ignoreHTTPSErrors?: boolean; isMobile?: boolean; javaScriptEnabled?: boolean; locale?: string; logger?: Logger; offline?: boolean; permissions?: string\[\]; proxy?: { bypass?: string; password?: string; server: string; username?: string }; recordHar?: { content?: omit | embed | attach; mode?: full | minimal; omitContent?: boolean; path: string; urlFilter?: string | RegExp }; recordVideo?: { dir: string; size?: { height: number; width: number } }; reducedMotion?: null | reduce | no-preference; screen?: { height: number; width: number }; serviceWorkers?: allow | block; storageState?: string | { cookies: { domain: string; expires: number; httpOnly: boolean; name: string; path: string; sameSite: Strict | Lax | None; secure: boolean; value: string }\[\]; origins: { localStorage: { name: string; value: string }\[\]; origin: string }\[\] }; strictSelectors?: boolean; timezoneId?: string; userAgent?: string; videoSize?: { height: number; width: number }; videosPath?: string; viewport?: null | { height: number; width: number } }, Page\>, undefined | { acceptDownloads?: boolean; baseURL?: string; bypassCSP?: boolean; clientCertificates?: { cert?: Buffer; certPath?: string; key?: Buffer; keyPath?: string; origin: string; passphrase?: string; pfx?: Buffer; pfxPath?: string }\[\]; colorScheme?: null | light | dark | no-preference; deviceScaleFactor?: number; extraHTTPHeaders?: {}; forcedColors?: null | active | none; geolocation?: { accuracy?: number; latitude: number; longitude: number }; hasTouch?: boolean; httpCredentials?: { origin?: string; password: string; send?: unauthorized | always; username: string }; ignoreHTTPSErrors?: boolean; isMobile?: boolean; javaScriptEnabled?: boolean; locale?: string; logger?: Logger; offline?: boolean; permissions?: string\[\]; proxy?: { bypass?: string; password?: string; server: string; username?: string }; recordHar?: { content?: omit | embed | attach; mode?: full | minimal; omitContent?: boolean; path: string; urlFilter?: string | RegExp }; recordVideo?: { dir: string; size?: { height: number; width: number } }; reducedMotion?: null | reduce | no-preference; screen?: { height: number; width: number }; serviceWorkers?: allow | block; storageState?: string | { cookies: { domain: string; expires: number; httpOnly: boolean; name: string; path: string; sameSite: Strict | Lax | None; secure: boolean; value: string }\[\]; origins: { localStorage: { name: string; value: string }\[\]; origin: string }\[\] }; strictSelectors?: boolean; timezoneId?: string; userAgent?: string; videoSize?: { height: number; width: number }; videosPath?: string; viewport?: null | { height: number; width: number } }, Page\> A reference to the underlying [BrowserPool](/js/api/browser-pool/class/BrowserPool) class that manages the crawler's browsers. ### [](#config) [](https://github.com/apify/crawlee/blob/master/packages/playwright-crawler/src/internals/playwright-crawler.ts#L206) readonlyinheritedconfig config: [Configuration](/js/api/core/class/Configuration) = ... ### [](#hasFinishedBefore) [](https://github.com/apify/crawlee/blob/master/packages/basic-crawler/src/internals/basic-crawler.ts#L489) inheritedhasFinishedBefore hasFinishedBefore: boolean = false ### [](#launchContext) [](https://github.com/apify/crawlee/blob/master/packages/browser-crawler/src/internals/browser-crawler.ts#L327) inheritedlaunchContext launchContext: [BrowserLaunchContext](/js/api/browser-crawler/interface/BrowserLaunchContext) ### [](#log) [](https://github.com/apify/crawlee/blob/master/packages/basic-crawler/src/internals/basic-crawler.ts#L491) readonlyinheritedlog log: [Log](/js/api/core/class/Log) ### [](#proxyConfiguration) [](https://github.com/apify/crawlee/blob/master/packages/browser-crawler/src/internals/browser-crawler.ts#L320) optionalinheritedproxyConfiguration proxyConfiguration?: [ProxyConfiguration](/js/api/core/class/ProxyConfiguration) A reference to the underlying [ProxyConfiguration](/js/api/core/class/ProxyConfiguration) class that manages the crawler's proxies. Only available if used by the crawler. ### [](#requestList) [](https://github.com/apify/crawlee/blob/master/packages/basic-crawler/src/internals/basic-crawler.ts#L458) optionalinheritedrequestList requestList?: [IRequestList](/js/api/core/interface/IRequestList) A reference to the underlying [RequestList](/js/api/core/class/RequestList) class that manages the crawler's [requests](/js/api/core/class/Request) . Only available if used by the crawler. ### [](#requestQueue) [](https://github.com/apify/crawlee/blob/master/packages/basic-crawler/src/internals/basic-crawler.ts#L465) optionalinheritedrequestQueue requestQueue?: [RequestProvider](/js/api/core/class/RequestProvider) Dynamic queue of URLs to be processed. This is useful for recursive crawling of websites. A reference to the underlying [RequestQueue](/js/api/core/class/RequestQueue) class that manages the crawler's [requests](/js/api/core/class/Request) . Only available if used by the crawler. ### [](#router) [](https://github.com/apify/crawlee/blob/master/packages/basic-crawler/src/internals/basic-crawler.ts#L486) readonlyinheritedrouter router: [RouterHandler](/js/api/core/interface/RouterHandler) <{ request: [LoadedRequest](/js/api/core#LoadedRequest) <[Request](/js/api/core/class/Request) \> } & Omit<[PlaywrightCrawlingContext](/js/api/playwright-crawler/interface/PlaywrightCrawlingContext) , request\>\> = ... Default [Router](/js/api/core/class/Router) instance that will be used if we don't specify any [`requestHandler`](/js/api/basic-crawler/interface/BasicCrawlerOptions#requestHandler) . See [`router.addHandler()`](/js/api/core/class/Router#addHandler) and [`router.addDefaultHandler()`](/js/api/core/class/Router#addDefaultHandler) . ### [](#running) [](https://github.com/apify/crawlee/blob/master/packages/basic-crawler/src/internals/basic-crawler.ts#L488) inheritedrunning running: boolean = false ### [](#sessionPool) [](https://github.com/apify/crawlee/blob/master/packages/basic-crawler/src/internals/basic-crawler.ts#L471) optionalinheritedsessionPool sessionPool?: [SessionPool](/js/api/core/class/SessionPool) A reference to the underlying [SessionPool](/js/api/core/class/SessionPool) class that manages the crawler's [sessions](/js/api/core/class/Session) . Only available if used by the crawler. ### [](#stats) [](https://github.com/apify/crawlee/blob/master/packages/basic-crawler/src/internals/basic-crawler.ts#L452) readonlyinheritedstats stats: [Statistics](/js/api/core/class/Statistics) A reference to the underlying [Statistics](/js/api/core/class/Statistics) class that collects and logs run statistics for requests. Methods[](#Methods) -------------------- ### [](#addRequests) [](https://github.com/apify/crawlee/blob/master/packages/basic-crawler/src/internals/basic-crawler.ts#L1027) inheritedaddRequests * **addRequests**(requests, options): Promise<[CrawlerAddRequestsResult](/js/api/basic-crawler/interface/CrawlerAddRequestsResult) \> * Adds requests to the queue in batches. By default, it will resolve after the initial batch is added, and continue adding the rest in background. You can configure the batch size via `batchSize` option and the sleep time in between the batches via `waitBetweenBatchesMillis`. If you want to wait for all batches to be added to the queue, you can use the `waitForAllRequestsToBeAdded` promise you get in the response object. This is an alias for calling `addRequestsBatched()` on the implicit `RequestQueue` for this crawler instance. * * * #### Parameters * ##### requests: (string | [Source](/js/api/core#Source) )\[\] The requests to add * ##### options: [CrawlerAddRequestsOptions](/js/api/basic-crawler/interface/CrawlerAddRequestsOptions) = {} Options for the request queue #### Returns Promise<[CrawlerAddRequestsResult](/js/api/basic-crawler/interface/CrawlerAddRequestsResult) \> ### [](#exportData) [](https://github.com/apify/crawlee/blob/master/packages/basic-crawler/src/internals/basic-crawler.ts#L1062) inheritedexportData * **exportData**(path, format, options): Promise * Retrieves all the data from the default crawler [Dataset](/js/api/core/class/Dataset) and exports them to the specified format. Supported formats are currently 'json' and 'csv', and will be inferred from the `path` automatically. * * * #### Parameters * ##### path: string * ##### optionalformat: json | csv * ##### optionaloptions: [DatasetExportOptions](/js/api/core/interface/DatasetExportOptions) #### Returns Promise ### [](#getData) [](https://github.com/apify/crawlee/blob/master/packages/basic-crawler/src/internals/basic-crawler.ts#L1053) inheritedgetData * **getData**(...args): Promise<[DatasetContent](/js/api/core/interface/DatasetContent) \> * Retrieves data from the default crawler [Dataset](/js/api/core/class/Dataset) by calling [Dataset.getData](/js/api/core/class/Dataset#getData) . * * * #### Parameters * ##### rest...args: \[options: [DatasetDataOptions](/js/api/core/interface/DatasetDataOptions)\ \] #### Returns Promise<[DatasetContent](/js/api/core/interface/DatasetContent) \> ### [](#getDataset) [](https://github.com/apify/crawlee/blob/master/packages/basic-crawler/src/internals/basic-crawler.ts#L1046) inheritedgetDataset * **getDataset**(idOrName): Promise<[Dataset](/js/api/core/class/Dataset) \> * Retrieves the specified [Dataset](/js/api/core/class/Dataset) , or the default crawler [Dataset](/js/api/core/class/Dataset) . * * * #### Parameters * ##### optionalidOrName: string #### Returns Promise<[Dataset](/js/api/core/class/Dataset) \> ### [](#getRequestQueue) [](https://github.com/apify/crawlee/blob/master/packages/basic-crawler/src/internals/basic-crawler.ts#L999) inheritedgetRequestQueue * **getRequestQueue**(): Promise<[RequestProvider](/js/api/core/class/RequestProvider) \> * #### Returns Promise<[RequestProvider](/js/api/core/class/RequestProvider) \> ### [](#pushData) [](https://github.com/apify/crawlee/blob/master/packages/basic-crawler/src/internals/basic-crawler.ts#L1038) inheritedpushData * **pushData**(data, datasetIdOrName): Promise * Pushes data to the specified [Dataset](/js/api/core/class/Dataset) , or the default crawler [Dataset](/js/api/core/class/Dataset) by calling [Dataset.pushData](/js/api/core/class/Dataset#pushData) . * * * #### Parameters * ##### data: Dictionary | Dictionary\[\] * ##### optionaldatasetIdOrName: string #### Returns Promise ### [](#run) [](https://github.com/apify/crawlee/blob/master/packages/basic-crawler/src/internals/basic-crawler.ts#L872) inheritedrun * **run**(requests, options): Promise<[FinalStatistics](/js/api/core/interface/FinalStatistics) \> * Runs the crawler. Returns a promise that resolves once all the requests are processed and `autoscaledPool.isFinished` returns `true`. We can use the `requests` parameter to enqueue the initial requests — it is a shortcut for running [`crawler.addRequests()`](/js/api/basic-crawler/class/BasicCrawler#addRequests) before [`crawler.run()`](/js/api/basic-crawler/class/BasicCrawler#run) . * * * #### Parameters * ##### optionalrequests: (string | [Request](/js/api/core/class/Request) | [RequestOptions](/js/api/core/interface/RequestOptions) )\[\] The requests to add. * ##### optionaloptions: [CrawlerRunOptions](/js/api/basic-crawler/interface/CrawlerRunOptions) Options for the request queue. #### Returns Promise<[FinalStatistics](/js/api/core/interface/FinalStatistics) \> ### [](#setStatusMessage) [](https://github.com/apify/crawlee/blob/master/packages/basic-crawler/src/internals/basic-crawler.ts#L796) inheritedsetStatusMessage * **setStatusMessage**(message, options): Promise * This method is periodically called by the crawler, every `statusMessageLoggingInterval` seconds. * * * #### Parameters * ##### message: string * ##### options: [SetStatusMessageOptions](/js/api/types/interface/SetStatusMessageOptions) = {} #### Returns Promise ### [](#stop) [](https://github.com/apify/crawlee/blob/master/packages/basic-crawler/src/internals/basic-crawler.ts#L987) inheritedstop * **stop**(message): void * Gracefully stops the current run of the crawler. All the tasks active at the time of calling this method will be allowed to finish. * * * #### Parameters * ##### message: string = 'The crawler has been gracefully stopped.' #### Returns void ### [](#useState) [](https://github.com/apify/crawlee/blob/master/packages/basic-crawler/src/internals/basic-crawler.ts#L1011) inheriteduseState * **useState**(defaultValue): Promise * #### Parameters * ##### defaultValue: State = ... #### Returns Promise **Page Options** Hide Inherited * [constructor](#constructor) * [autoscaledPool](#autoscaledPool) * [browserPool](#browserPool) * [config](#config) * [hasFinishedBefore](#hasFinishedBefore) * [launchContext](#launchContext) * [log](#log) * [proxyConfiguration](#proxyConfiguration) * [requestList](#requestList) * [requestQueue](#requestQueue) * [router](#router) * [running](#running) * [sessionPool](#sessionPool) * [stats](#stats) * [addRequests](#addRequests) * [exportData](#exportData) * [getData](#getData) * [getDataset](#getDataset) * [getRequestQueue](#getRequestQueue) * [pushData](#pushData) * [run](#run) * [setStatusMessage](#setStatusMessage) * [stop](#stop) * [useState](#useState) --- # createAdaptivePlaywrightRouter | API | Crawlee for JavaScript · Build reliable crawlers. Fast. [Skip to main content](#__docusaurus_skipToContent_fallback) Version: 3.13 ### Callable * **createAdaptivePlaywrightRouter**(routes): [RouterHandler](/js/api/core/interface/RouterHandler) * * * * #### Parameters * ##### optionalroutes: [RouterRoutes](/js/api/core#RouterRoutes) #### Returns [RouterHandler](/js/api/core/interface/RouterHandler) --- # AdaptivePlaywrightCrawler | API | Crawlee for JavaScript · Build reliable crawlers. Fast. [Skip to main content](#__docusaurus_skipToContent_fallback) Version: 3.13 On this page experimental An extension of [PlaywrightCrawler](/js/api/playwright-crawler/class/PlaywrightCrawler) that uses a more limited request handler interface so that it is able to switch to HTTP-only crawling when it detects it may be possible. **Example usage:** const crawler = new AdaptivePlaywrightCrawler({ renderingTypeDetectionRatio: 0.1, async requestHandler({ querySelector, pushData, enqueueLinks, request, log }) { // This function is called to extract data from a single web page const $prices = await querySelector('span.price') await pushData({ url: request.url, price: $prices.filter(':contains("$")').first().text(), }) await enqueueLinks({ selector: '.pagination a' }) },});await crawler.run([ 'http://www.example.com/page-1', 'http://www.example.com/page-2',]); ### Hierarchy * [PlaywrightCrawler](/js/api/playwright-crawler/class/PlaywrightCrawler) * _AdaptivePlaywrightCrawler_ Index[](#Index) ---------------- ### Constructors * [constructor](#constructor) ### Properties * [autoscaledPool](#autoscaledPool) * [browserPool](#browserPool) * [config](#config) * [hasFinishedBefore](#hasFinishedBefore) * [launchContext](#launchContext) * [log](#log) * [proxyConfiguration](#proxyConfiguration) * [requestList](#requestList) * [requestQueue](#requestQueue) * [router](#router) * [running](#running) * [sessionPool](#sessionPool) * [stats](#stats) ### Methods * [addRequests](#addRequests) * [exportData](#exportData) * [getData](#getData) * [getDataset](#getDataset) * [getRequestQueue](#getRequestQueue) * [pushData](#pushData) * [run](#run) * [setStatusMessage](#setStatusMessage) * [stop](#stop) * [useState](#useState) Constructors[](#Constructors) ------------------------------ ### [](#constructor) [](https://github.com/apify/crawlee/blob/master/packages/playwright-crawler/src/internals/adaptive-playwright-crawler.ts#L226) constructor * **new AdaptivePlaywrightCrawler**(options, config): [AdaptivePlaywrightCrawler](/js/api/playwright-crawler/class/AdaptivePlaywrightCrawler) * experimental #### Parameters * ##### options: [AdaptivePlaywrightCrawlerOptions](/js/api/playwright-crawler/interface/AdaptivePlaywrightCrawlerOptions) = {} * ##### config: [Configuration](/js/api/core/class/Configuration) = ... #### Returns [AdaptivePlaywrightCrawler](/js/api/playwright-crawler/class/AdaptivePlaywrightCrawler) Properties[](#Properties) -------------------------- ### [](#autoscaledPool) [](https://github.com/apify/crawlee/blob/master/packages/basic-crawler/src/internals/basic-crawler.ts#L480) optionalinheritedautoscaledPool experimental autoscaledPool?: [AutoscaledPool](/js/api/core/class/AutoscaledPool) A reference to the underlying [AutoscaledPool](/js/api/core/class/AutoscaledPool) class that manages the concurrency of the crawler. > _NOTE:_ This property is only initialized after calling the [`crawler.run()`](/js/api/basic-crawler/class/BasicCrawler#run) > function. We can use it to change the concurrency settings on the fly, to pause the crawler by calling [`autoscaledPool.pause()`](/js/api/core/class/AutoscaledPool#pause) > or to abort it by calling [`autoscaledPool.abort()`](/js/api/core/class/AutoscaledPool#abort) > . ### [](#browserPool) [](https://github.com/apify/crawlee/blob/master/packages/browser-crawler/src/internals/browser-crawler.ts#L325) inheritedbrowserPool experimental browserPool: [BrowserPool](/js/api/browser-pool/class/BrowserPool) <{ browserPlugins: \[[PlaywrightPlugin](/js/api/browser-pool/class/PlaywrightPlugin)\ \] }, \[[PlaywrightPlugin](/js/api/browser-pool/class/PlaywrightPlugin)\ \], [BrowserController](/js/api/browser-pool/class/BrowserController) , undefined | LaunchOptions, Browser, undefined | { acceptDownloads?: boolean; baseURL?: string; bypassCSP?: boolean; clientCertificates?: { cert?: Buffer; certPath?: string; key?: Buffer; keyPath?: string; origin: string; passphrase?: string; pfx?: Buffer; pfxPath?: string }\[\]; colorScheme?: null | light | dark | no-preference; deviceScaleFactor?: number; extraHTTPHeaders?: {}; forcedColors?: null | active | none; geolocation?: { accuracy?: number; latitude: number; longitude: number }; hasTouch?: boolean; httpCredentials?: { origin?: string; password: string; send?: unauthorized | always; username: string }; ignoreHTTPSErrors?: boolean; isMobile?: boolean; javaScriptEnabled?: boolean; locale?: string; logger?: Logger; offline?: boolean; permissions?: string\[\]; proxy?: { bypass?: string; password?: string; server: string; username?: string }; recordHar?: { content?: omit | embed | attach; mode?: full | minimal; omitContent?: boolean; path: string; urlFilter?: string | RegExp }; recordVideo?: { dir: string; size?: { height: number; width: number } }; reducedMotion?: null | reduce | no-preference; screen?: { height: number; width: number }; serviceWorkers?: allow | block; storageState?: string | { cookies: { domain: string; expires: number; httpOnly: boolean; name: string; path: string; sameSite: Strict | Lax | None; secure: boolean; value: string }\[\]; origins: { localStorage: { name: string; value: string }\[\]; origin: string }\[\] }; strictSelectors?: boolean; timezoneId?: string; userAgent?: string; videoSize?: { height: number; width: number }; videosPath?: string; viewport?: null | { height: number; width: number } }, Page\>, [LaunchContext](/js/api/browser-pool/class/LaunchContext) , undefined | LaunchOptions, Browser, undefined | { acceptDownloads?: boolean; baseURL?: string; bypassCSP?: boolean; clientCertificates?: { cert?: Buffer; certPath?: string; key?: Buffer; keyPath?: string; origin: string; passphrase?: string; pfx?: Buffer; pfxPath?: string }\[\]; colorScheme?: null | light | dark | no-preference; deviceScaleFactor?: number; extraHTTPHeaders?: {}; forcedColors?: null | active | none; geolocation?: { accuracy?: number; latitude: number; longitude: number }; hasTouch?: boolean; httpCredentials?: { origin?: string; password: string; send?: unauthorized | always; username: string }; ignoreHTTPSErrors?: boolean; isMobile?: boolean; javaScriptEnabled?: boolean; locale?: string; logger?: Logger; offline?: boolean; permissions?: string\[\]; proxy?: { bypass?: string; password?: string; server: string; username?: string }; recordHar?: { content?: omit | embed | attach; mode?: full | minimal; omitContent?: boolean; path: string; urlFilter?: string | RegExp }; recordVideo?: { dir: string; size?: { height: number; width: number } }; reducedMotion?: null | reduce | no-preference; screen?: { height: number; width: number }; serviceWorkers?: allow | block; storageState?: string | { cookies: { domain: string; expires: number; httpOnly: boolean; name: string; path: string; sameSite: Strict | Lax | None; secure: boolean; value: string }\[\]; origins: { localStorage: { name: string; value: string }\[\]; origin: string }\[\] }; strictSelectors?: boolean; timezoneId?: string; userAgent?: string; videoSize?: { height: number; width: number }; videosPath?: string; viewport?: null | { height: number; width: number } }, Page\>, undefined | { acceptDownloads?: boolean; baseURL?: string; bypassCSP?: boolean; clientCertificates?: { cert?: Buffer; certPath?: string; key?: Buffer; keyPath?: string; origin: string; passphrase?: string; pfx?: Buffer; pfxPath?: string }\[\]; colorScheme?: null | light | dark | no-preference; deviceScaleFactor?: number; extraHTTPHeaders?: {}; forcedColors?: null | active | none; geolocation?: { accuracy?: number; latitude: number; longitude: number }; hasTouch?: boolean; httpCredentials?: { origin?: string; password: string; send?: unauthorized | always; username: string }; ignoreHTTPSErrors?: boolean; isMobile?: boolean; javaScriptEnabled?: boolean; locale?: string; logger?: Logger; offline?: boolean; permissions?: string\[\]; proxy?: { bypass?: string; password?: string; server: string; username?: string }; recordHar?: { content?: omit | embed | attach; mode?: full | minimal; omitContent?: boolean; path: string; urlFilter?: string | RegExp }; recordVideo?: { dir: string; size?: { height: number; width: number } }; reducedMotion?: null | reduce | no-preference; screen?: { height: number; width: number }; serviceWorkers?: allow | block; storageState?: string | { cookies: { domain: string; expires: number; httpOnly: boolean; name: string; path: string; sameSite: Strict | Lax | None; secure: boolean; value: string }\[\]; origins: { localStorage: { name: string; value: string }\[\]; origin: string }\[\] }; strictSelectors?: boolean; timezoneId?: string; userAgent?: string; videoSize?: { height: number; width: number }; videosPath?: string; viewport?: null | { height: number; width: number } }, Page\> A reference to the underlying [BrowserPool](/js/api/browser-pool/class/BrowserPool) class that manages the crawler's browsers. ### [](#config) [](https://github.com/apify/crawlee/blob/master/packages/playwright-crawler/src/internals/adaptive-playwright-crawler.ts#L228) readonlyinheritedconfig experimental config: [Configuration](/js/api/core/class/Configuration) = ... ### [](#hasFinishedBefore) [](https://github.com/apify/crawlee/blob/master/packages/basic-crawler/src/internals/basic-crawler.ts#L489) inheritedhasFinishedBefore experimental hasFinishedBefore: boolean = false ### [](#launchContext) [](https://github.com/apify/crawlee/blob/master/packages/browser-crawler/src/internals/browser-crawler.ts#L327) inheritedlaunchContext experimental launchContext: [BrowserLaunchContext](/js/api/browser-crawler/interface/BrowserLaunchContext) ### [](#log) [](https://github.com/apify/crawlee/blob/master/packages/basic-crawler/src/internals/basic-crawler.ts#L491) readonlyinheritedlog experimental log: [Log](/js/api/core/class/Log) ### [](#proxyConfiguration) [](https://github.com/apify/crawlee/blob/master/packages/browser-crawler/src/internals/browser-crawler.ts#L320) optionalinheritedproxyConfiguration experimental proxyConfiguration?: [ProxyConfiguration](/js/api/core/class/ProxyConfiguration) A reference to the underlying [ProxyConfiguration](/js/api/core/class/ProxyConfiguration) class that manages the crawler's proxies. Only available if used by the crawler. ### [](#requestList) [](https://github.com/apify/crawlee/blob/master/packages/basic-crawler/src/internals/basic-crawler.ts#L458) optionalinheritedrequestList experimental requestList?: [IRequestList](/js/api/core/interface/IRequestList) A reference to the underlying [RequestList](/js/api/core/class/RequestList) class that manages the crawler's [requests](/js/api/core/class/Request) . Only available if used by the crawler. ### [](#requestQueue) [](https://github.com/apify/crawlee/blob/master/packages/basic-crawler/src/internals/basic-crawler.ts#L465) optionalinheritedrequestQueue experimental requestQueue?: [RequestProvider](/js/api/core/class/RequestProvider) Dynamic queue of URLs to be processed. This is useful for recursive crawling of websites. A reference to the underlying [RequestQueue](/js/api/core/class/RequestQueue) class that manages the crawler's [requests](/js/api/core/class/Request) . Only available if used by the crawler. ### [](#router) [](https://github.com/apify/crawlee/blob/master/packages/playwright-crawler/src/internals/adaptive-playwright-crawler.ts#L223) readonlyrouter experimental router: [RouterHandler](/js/api/core/interface/RouterHandler) <[AdaptivePlaywrightCrawlerContext](/js/api/playwright-crawler/interface/AdaptivePlaywrightCrawlerContext) \> = ... Default [Router](/js/api/core/class/Router) instance that will be used if we don't specify any [`requestHandler`](/js/api/playwright-crawler/interface/AdaptivePlaywrightCrawlerOptions#requestHandler) . See [`router.addHandler()`](/js/api/core/class/Router#addHandler) and [`router.addDefaultHandler()`](/js/api/core/class/Router#addDefaultHandler) . ### [](#running) [](https://github.com/apify/crawlee/blob/master/packages/basic-crawler/src/internals/basic-crawler.ts#L488) inheritedrunning experimental running: boolean = false ### [](#sessionPool) [](https://github.com/apify/crawlee/blob/master/packages/basic-crawler/src/internals/basic-crawler.ts#L471) optionalinheritedsessionPool experimental sessionPool?: [SessionPool](/js/api/core/class/SessionPool) A reference to the underlying [SessionPool](/js/api/core/class/SessionPool) class that manages the crawler's [sessions](/js/api/core/class/Session) . Only available if used by the crawler. ### [](#stats) [](https://github.com/apify/crawlee/blob/master/packages/playwright-crawler/src/internals/adaptive-playwright-crawler.ts#L216) readonlystats experimental stats: AdaptivePlaywrightCrawlerStatistics A reference to the underlying [Statistics](/js/api/core/class/Statistics) class that collects and logs run statistics for requests. Methods[](#Methods) -------------------- ### [](#addRequests) [](https://github.com/apify/crawlee/blob/master/packages/basic-crawler/src/internals/basic-crawler.ts#L1027) inheritedaddRequests * **addRequests**(requests, options): Promise<[CrawlerAddRequestsResult](/js/api/basic-crawler/interface/CrawlerAddRequestsResult) \> * experimental Adds requests to the queue in batches. By default, it will resolve after the initial batch is added, and continue adding the rest in background. You can configure the batch size via `batchSize` option and the sleep time in between the batches via `waitBetweenBatchesMillis`. If you want to wait for all batches to be added to the queue, you can use the `waitForAllRequestsToBeAdded` promise you get in the response object. This is an alias for calling `addRequestsBatched()` on the implicit `RequestQueue` for this crawler instance. * * * #### Parameters * ##### requests: (string | [Source](/js/api/core#Source) )\[\] The requests to add * ##### options: [CrawlerAddRequestsOptions](/js/api/basic-crawler/interface/CrawlerAddRequestsOptions) = {} Options for the request queue #### Returns Promise<[CrawlerAddRequestsResult](/js/api/basic-crawler/interface/CrawlerAddRequestsResult) \> ### [](#exportData) [](https://github.com/apify/crawlee/blob/master/packages/basic-crawler/src/internals/basic-crawler.ts#L1062) inheritedexportData * **exportData**(path, format, options): Promise * experimental Retrieves all the data from the default crawler [Dataset](/js/api/core/class/Dataset) and exports them to the specified format. Supported formats are currently 'json' and 'csv', and will be inferred from the `path` automatically. * * * #### Parameters * ##### path: string * ##### optionalformat: json | csv * ##### optionaloptions: [DatasetExportOptions](/js/api/core/interface/DatasetExportOptions) #### Returns Promise ### [](#getData) [](https://github.com/apify/crawlee/blob/master/packages/basic-crawler/src/internals/basic-crawler.ts#L1053) inheritedgetData * **getData**(...args): Promise<[DatasetContent](/js/api/core/interface/DatasetContent) \> * experimental Retrieves data from the default crawler [Dataset](/js/api/core/class/Dataset) by calling [Dataset.getData](/js/api/core/class/Dataset#getData) . * * * #### Parameters * ##### rest...args: \[options: [DatasetDataOptions](/js/api/core/interface/DatasetDataOptions)\ \] #### Returns Promise<[DatasetContent](/js/api/core/interface/DatasetContent) \> ### [](#getDataset) [](https://github.com/apify/crawlee/blob/master/packages/basic-crawler/src/internals/basic-crawler.ts#L1046) inheritedgetDataset * **getDataset**(idOrName): Promise<[Dataset](/js/api/core/class/Dataset) \> * experimental Retrieves the specified [Dataset](/js/api/core/class/Dataset) , or the default crawler [Dataset](/js/api/core/class/Dataset) . * * * #### Parameters * ##### optionalidOrName: string #### Returns Promise<[Dataset](/js/api/core/class/Dataset) \> ### [](#getRequestQueue) [](https://github.com/apify/crawlee/blob/master/packages/basic-crawler/src/internals/basic-crawler.ts#L999) inheritedgetRequestQueue * **getRequestQueue**(): Promise<[RequestProvider](/js/api/core/class/RequestProvider) \> * experimental #### Returns Promise<[RequestProvider](/js/api/core/class/RequestProvider) \> ### [](#pushData) [](https://github.com/apify/crawlee/blob/master/packages/basic-crawler/src/internals/basic-crawler.ts#L1038) inheritedpushData * **pushData**(data, datasetIdOrName): Promise * experimental Pushes data to the specified [Dataset](/js/api/core/class/Dataset) , or the default crawler [Dataset](/js/api/core/class/Dataset) by calling [Dataset.pushData](/js/api/core/class/Dataset#pushData) . * * * #### Parameters * ##### data: Dictionary | Dictionary\[\] * ##### optionaldatasetIdOrName: string #### Returns Promise ### [](#run) [](https://github.com/apify/crawlee/blob/master/packages/basic-crawler/src/internals/basic-crawler.ts#L872) inheritedrun * **run**(requests, options): Promise<[FinalStatistics](/js/api/core/interface/FinalStatistics) \> * experimental Runs the crawler. Returns a promise that resolves once all the requests are processed and `autoscaledPool.isFinished` returns `true`. We can use the `requests` parameter to enqueue the initial requests — it is a shortcut for running [`crawler.addRequests()`](/js/api/basic-crawler/class/BasicCrawler#addRequests) before [`crawler.run()`](/js/api/basic-crawler/class/BasicCrawler#run) . * * * #### Parameters * ##### optionalrequests: (string | [Request](/js/api/core/class/Request) | [RequestOptions](/js/api/core/interface/RequestOptions) )\[\] The requests to add. * ##### optionaloptions: [CrawlerRunOptions](/js/api/basic-crawler/interface/CrawlerRunOptions) Options for the request queue. #### Returns Promise<[FinalStatistics](/js/api/core/interface/FinalStatistics) \> ### [](#setStatusMessage) [](https://github.com/apify/crawlee/blob/master/packages/basic-crawler/src/internals/basic-crawler.ts#L796) inheritedsetStatusMessage * **setStatusMessage**(message, options): Promise * experimental This method is periodically called by the crawler, every `statusMessageLoggingInterval` seconds. * * * #### Parameters * ##### message: string * ##### options: [SetStatusMessageOptions](/js/api/types/interface/SetStatusMessageOptions) = {} #### Returns Promise ### [](#stop) [](https://github.com/apify/crawlee/blob/master/packages/basic-crawler/src/internals/basic-crawler.ts#L987) inheritedstop * **stop**(message): void * experimental Gracefully stops the current run of the crawler. All the tasks active at the time of calling this method will be allowed to finish. * * * #### Parameters * ##### message: string = 'The crawler has been gracefully stopped.' #### Returns void ### [](#useState) [](https://github.com/apify/crawlee/blob/master/packages/basic-crawler/src/internals/basic-crawler.ts#L1011) inheriteduseState * **useState**(defaultValue): Promise * experimental #### Parameters * ##### defaultValue: State = ... #### Returns Promise **Page Options** Hide Inherited * [constructor](#constructor) * [autoscaledPool](#autoscaledPool) * [browserPool](#browserPool) * [config](#config) * [hasFinishedBefore](#hasFinishedBefore) * [launchContext](#launchContext) * [log](#log) * [proxyConfiguration](#proxyConfiguration) * [requestList](#requestList) * [requestQueue](#requestQueue) * [router](#router) * [running](#running) * [sessionPool](#sessionPool) * [stats](#stats) * [addRequests](#addRequests) * [exportData](#exportData) * [getData](#getData) * [getDataset](#getDataset) * [getRequestQueue](#getRequestQueue) * [pushData](#pushData) * [run](#run) * [setStatusMessage](#setStatusMessage) * [stop](#stop) * [useState](#useState) --- # AdaptivePlaywrightCrawlerContext | API | Crawlee for JavaScript · Build reliable crawlers. Fast. [Skip to main content](#__docusaurus_skipToContent_fallback) Version: 3.13 On this page ### Hierarchy * [RestrictedCrawlingContext](/js/api/core/interface/RestrictedCrawlingContext) * _AdaptivePlaywrightCrawlerContext_ Index[](#Index) ---------------- ### Properties * [addRequests](#addRequests) * [enqueueLinks](#enqueueLinks) * [getKeyValueStore](#getKeyValueStore) * [id](#id) * [log](#log) * [proxyInfo](#proxyInfo) * [request](#request) * [session](#session) * [useState](#useState) ### Methods * [parseWithCheerio](#parseWithCheerio) * [pushData](#pushData) * [querySelector](#querySelector) * [waitForSelector](#waitForSelector) Properties[](#Properties) -------------------------- ### [](#addRequests) [](https://github.com/apify/crawlee/blob/master/packages/core/src/crawlers/crawler_commons.ts#L88) inheritedaddRequests addRequests: (requestsLike, options) => Promise Add requests directly to the request queue. * * * #### Type declaration * * (requestsLike, options): Promise * #### Parameters * ##### requestsLike: readonly (string | ReadonlyObjectDeep\> & { regex?: RegExp; requestsFromUrl?: string }\> | ReadonlyObjectDeep<[Request](/js/api/core/class/Request) \>)\[\] * ##### optionaloptions: ReadonlyObjectDeep<[RequestQueueOperationOptions](/js/api/core/interface/RequestQueueOperationOptions) \> Options for the request queue #### Returns Promise ### [](#enqueueLinks) [](https://github.com/apify/crawlee/blob/master/packages/core/src/crawlers/crawler_commons.ts#L80) inheritedenqueueLinks enqueueLinks: (options) => Promise This function automatically finds and enqueues links from the current page, adding them to the [RequestQueue](/js/api/core/class/RequestQueue) currently used by the crawler. Optionally, the function allows you to filter the target links' URLs using an array of globs or regular expressions and override settings of the enqueued [Request](/js/api/core/class/Request) objects. Check out the [Crawl a website with relative links](/js/docs/examples/crawl-relative-links) example for more details regarding its usage. **Example usage** async requestHandler({ enqueueLinks }) { await enqueueLinks({ globs: [ 'https://www.example.com/handbags/*', ], });}, * * * #### Type declaration * * (options): Promise * #### Parameters * ##### optionaloptions: ReadonlyObjectDeep\> All `enqueueLinks()` parameters are passed via an options object. #### Returns Promise ### [](#getKeyValueStore) [](https://github.com/apify/crawlee/blob/master/packages/core/src/crawlers/crawler_commons.ts#L101) inheritedgetKeyValueStore getKeyValueStore: (idOrName) => Promise\> Get a key-value store with given name or id, or the default one for the crawler. * * * #### Type declaration * * (idOrName): Promise\> * #### Parameters * ##### optionalidOrName: string #### Returns Promise\> ### [](#id) [](https://github.com/apify/crawlee/blob/master/packages/core/src/crawlers/crawler_commons.ts#L33) inheritedid id: string ### [](#log) [](https://github.com/apify/crawlee/blob/master/packages/core/src/crawlers/crawler_commons.ts#L108) inheritedlog log: [Log](/js/api/core/class/Log) A preconfigured logger for the request handler. ### [](#proxyInfo) [](https://github.com/apify/crawlee/blob/master/packages/core/src/crawlers/crawler_commons.ts#L40) optionalinheritedproxyInfo proxyInfo?: [ProxyInfo](/js/api/core/interface/ProxyInfo) An object with information about currently used proxy by the crawler and configured by the [ProxyConfiguration](/js/api/core/class/ProxyConfiguration) class. ### [](#request) [](https://github.com/apify/crawlee/blob/master/packages/core/src/crawlers/crawler_commons.ts#L45) inheritedrequest request: [Request](/js/api/core/class/Request) The original [Request](/js/api/core/class/Request) object. ### [](#session) [](https://github.com/apify/crawlee/blob/master/packages/core/src/crawlers/crawler_commons.ts#L34) optionalinheritedsession session?: [Session](/js/api/core/class/Session) ### [](#useState) [](https://github.com/apify/crawlee/blob/master/packages/core/src/crawlers/crawler_commons.ts#L96) inheriteduseState useState: (defaultValue) => Promise Returns the state - a piece of mutable persistent data shared across all the request handler runs. * * * #### Type declaration * * (defaultValue): Promise * #### Parameters * ##### optionaldefaultValue: State #### Returns Promise Methods[](#Methods) -------------------- ### [](#parseWithCheerio) [](https://github.com/apify/crawlee/blob/master/packages/playwright-crawler/src/internals/adaptive-playwright-crawler.ts#L124) parseWithCheerio * **parseWithCheerio**(selector, timeoutMs): Promise * Returns Cheerio handle for `page.content()`, allowing to work with the data same way as with [CheerioCrawler](/js/api/cheerio-crawler/class/CheerioCrawler) . When provided with the `selector` argument, it will first look for the selector with a 5s timeout. **Example usage:** async requestHandler({ parseWithCheerio }) { const $ = await parseWithCheerio(); const title = $('title').text();}); * * * #### Parameters * ##### optionalselector: string * ##### optionaltimeoutMs: number #### Returns Promise ### [](#pushData) [](https://github.com/apify/crawlee/blob/master/packages/core/src/crawlers/crawler_commons.ts#L54) inheritedpushData * **pushData**(data, datasetIdOrName): Promise * This function allows you to push data to a [Dataset](/js/api/core/class/Dataset) specified by name, or the one currently used by the crawler. Shortcut for `crawler.pushData()`. * * * #### Parameters * ##### optionaldata: ReadonlyDeep Data to be pushed to the default dataset. * ##### optionaldatasetIdOrName: string #### Returns Promise ### [](#querySelector) [](https://github.com/apify/crawlee/blob/master/packages/playwright-crawler/src/internals/adaptive-playwright-crawler.ts#L95) querySelector * **querySelector**(selector, timeoutMs): Promise\> * Wait for an element matching the selector to appear and return a Cheerio object of matched elements. Timeout defaults to 5s. * * * #### Parameters * ##### selector: string * ##### optionaltimeoutMs: number #### Returns Promise\> ### [](#waitForSelector) [](https://github.com/apify/crawlee/blob/master/packages/playwright-crawler/src/internals/adaptive-playwright-crawler.ts#L110) waitForSelector * **waitForSelector**(selector, timeoutMs): Promise * Wait for an element matching the selector to appear. Timeout defaults to 5s. **Example usage:** async requestHandler({ waitForSelector, parseWithCheerio }) { await waitForSelector('article h1'); const $ = await parseWithCheerio(); const title = $('title').text();}); * * * #### Parameters * ##### selector: string * ##### optionaltimeoutMs: number #### Returns Promise **Page Options** Hide Inherited * [addRequests](#addRequests) * [enqueueLinks](#enqueueLinks) * [getKeyValueStore](#getKeyValueStore) * [id](#id) * [log](#log) * [proxyInfo](#proxyInfo) * [request](#request) * [session](#session) * [useState](#useState) * [parseWithCheerio](#parseWithCheerio) * [pushData](#pushData) * [querySelector](#querySelector) * [waitForSelector](#waitForSelector) --- # @crawlee/playwright | API | Crawlee for JavaScript · Build reliable crawlers. Fast. [Skip to main content](#__docusaurus_skipToContent_fallback) Version: 3.13 On this page Provides a simple framework for parallel crawling of web pages using headless Chromium, Firefox and Webkit browsers with [Playwright](https://github.com/microsoft/playwright) . The URLs to crawl are fed either from a static list of URLs or from a dynamic queue of URLs enabling recursive crawling of websites. Since `Playwright` uses headless browser to download web pages and extract data, it is useful for crawling of websites that require to execute JavaScript. If the target website doesn't need JavaScript, consider using [CheerioCrawler](/js/api/cheerio-crawler/class/CheerioCrawler) , which downloads the pages using raw HTTP requests and is about 10x faster. The source URLs are represented using [Request](/js/api/core/class/Request) objects that are fed from [RequestList](/js/api/core/class/RequestList) or [RequestQueue](/js/api/core/class/RequestQueue) instances provided by the [PlaywrightCrawlerOptions.requestList](/js/api/playwright-crawler/interface/PlaywrightCrawlerOptions#requestList) or [PlaywrightCrawlerOptions.requestQueue](/js/api/playwright-crawler/interface/PlaywrightCrawlerOptions#requestQueue) constructor options, respectively. If both [PlaywrightCrawlerOptions.requestList](/js/api/playwright-crawler/interface/PlaywrightCrawlerOptions#requestList) and [PlaywrightCrawlerOptions.requestQueue](/js/api/playwright-crawler/interface/PlaywrightCrawlerOptions#requestQueue) are used, the instance first processes URLs from the [RequestList](/js/api/core/class/RequestList) and automatically enqueues all of them to [RequestQueue](/js/api/core/class/RequestQueue) before it starts their processing. This ensures that a single URL is not crawled multiple times. The crawler finishes when there are no more [Request](/js/api/core/class/Request) objects to crawl. `PlaywrightCrawler` opens a new Chrome page (i.e. tab) for each [Request](/js/api/core/class/Request) object to crawl and then calls the function provided by user as the [PlaywrightCrawlerOptions.requestHandler](/js/api/playwright-crawler/interface/PlaywrightCrawlerOptions#requestHandler) option. New pages are only opened when there is enough free CPU and memory available, using the functionality provided by the [AutoscaledPool](/js/api/core/class/AutoscaledPool) class. All [AutoscaledPool](/js/api/core/class/AutoscaledPool) configuration options can be passed to the [PlaywrightCrawlerOptions.autoscaledPoolOptions](/js/api/playwright-crawler/interface/PlaywrightCrawlerOptions#autoscaledPoolOptions) parameter of the `PlaywrightCrawler` constructor. For user convenience, the `minConcurrency` and `maxConcurrency` [AutoscaledPoolOptions](/js/api/core/interface/AutoscaledPoolOptions) are available directly in the `PlaywrightCrawler` constructor. Note that the pool of Playwright instances is internally managed by the [BrowserPool](https://github.com/apify/browser-pool) class. Example usage[​](#example-usage "Direct link to Example usage") ---------------------------------------------------------------- const crawler = new PlaywrightCrawler({ async requestHandler({ page, request }) { // This function is called to extract data from a single web page // 'page' is an instance of Playwright.Page with page.goto(request.url) already called // 'request' is an instance of Request class with information about the page to load await Dataset.pushData({ title: await page.title(), url: request.url, succeeded: true, }) }, async failedRequestHandler({ request }) { // This function is called when the crawling of a request failed too many times await Dataset.pushData({ url: request.url, succeeded: false, errors: request.errorMessages, }) },});await crawler.run([ 'http://www.example.com/page-1', 'http://www.example.com/page-2',]); Index[](#Index) ---------------- ### Crawlers * [PlaywrightCrawler](/js/api/playwright-crawler/class/PlaywrightCrawler) ### Other * [AddRequestsBatchedOptions](/js/api/playwright-crawler#AddRequestsBatchedOptions) * [AddRequestsBatchedResult](/js/api/playwright-crawler#AddRequestsBatchedResult) * [AutoscaledPool](/js/api/playwright-crawler#AutoscaledPool) * [AutoscaledPoolOptions](/js/api/playwright-crawler#AutoscaledPoolOptions) * [BaseHttpClient](/js/api/playwright-crawler#BaseHttpClient) * [BaseHttpResponseData](/js/api/playwright-crawler#BaseHttpResponseData) * [BASIC\_CRAWLER\_TIMEOUT\_BUFFER\_SECS](/js/api/playwright-crawler#BASIC_CRAWLER_TIMEOUT_BUFFER_SECS) * [BasicCrawler](/js/api/playwright-crawler#BasicCrawler) * [BasicCrawlerOptions](/js/api/playwright-crawler#BasicCrawlerOptions) * [BasicCrawlingContext](/js/api/playwright-crawler#BasicCrawlingContext) * [BLOCKED\_STATUS\_CODES](/js/api/playwright-crawler#BLOCKED_STATUS_CODES) * [BrowserCrawler](/js/api/playwright-crawler#BrowserCrawler) * [BrowserCrawlerOptions](/js/api/playwright-crawler#BrowserCrawlerOptions) * [BrowserCrawlingContext](/js/api/playwright-crawler#BrowserCrawlingContext) * [BrowserErrorHandler](/js/api/playwright-crawler#BrowserErrorHandler) * [BrowserHook](/js/api/playwright-crawler#BrowserHook) * [BrowserLaunchContext](/js/api/playwright-crawler#BrowserLaunchContext) * [BrowserRequestHandler](/js/api/playwright-crawler#BrowserRequestHandler) * [checkStorageAccess](/js/api/playwright-crawler#checkStorageAccess) * [ClientInfo](/js/api/playwright-crawler#ClientInfo) * [Configuration](/js/api/playwright-crawler#Configuration) * [ConfigurationOptions](/js/api/playwright-crawler#ConfigurationOptions) * [Cookie](/js/api/playwright-crawler#Cookie) * [CrawlerAddRequestsOptions](/js/api/playwright-crawler#CrawlerAddRequestsOptions) * [CrawlerAddRequestsResult](/js/api/playwright-crawler#CrawlerAddRequestsResult) * [CrawlerExperiments](/js/api/playwright-crawler#CrawlerExperiments) * [CrawlerRunOptions](/js/api/playwright-crawler#CrawlerRunOptions) * [CrawlingContext](/js/api/playwright-crawler#CrawlingContext) * [createBasicRouter](/js/api/playwright-crawler#createBasicRouter) * [CreateContextOptions](/js/api/playwright-crawler#CreateContextOptions) * [CreateSession](/js/api/playwright-crawler#CreateSession) * [CriticalError](/js/api/playwright-crawler#CriticalError) * [Dataset](/js/api/playwright-crawler#Dataset) * [DatasetConsumer](/js/api/playwright-crawler#DatasetConsumer) * [DatasetContent](/js/api/playwright-crawler#DatasetContent) * [DatasetDataOptions](/js/api/playwright-crawler#DatasetDataOptions) * [DatasetExportOptions](/js/api/playwright-crawler#DatasetExportOptions) * [DatasetExportToOptions](/js/api/playwright-crawler#DatasetExportToOptions) * [DatasetIteratorOptions](/js/api/playwright-crawler#DatasetIteratorOptions) * [DatasetMapper](/js/api/playwright-crawler#DatasetMapper) * [DatasetOptions](/js/api/playwright-crawler#DatasetOptions) * [DatasetReducer](/js/api/playwright-crawler#DatasetReducer) * [enqueueLinks](/js/api/playwright-crawler#enqueueLinks) * [EnqueueLinksOptions](/js/api/playwright-crawler#EnqueueLinksOptions) * [EnqueueStrategy](/js/api/playwright-crawler#EnqueueStrategy) * [ErrnoException](/js/api/playwright-crawler#ErrnoException) * [ErrorHandler](/js/api/playwright-crawler#ErrorHandler) * [ErrorSnapshotter](/js/api/playwright-crawler#ErrorSnapshotter) * [ErrorTracker](/js/api/playwright-crawler#ErrorTracker) * [ErrorTrackerOptions](/js/api/playwright-crawler#ErrorTrackerOptions) * [EventManager](/js/api/playwright-crawler#EventManager) * [EventType](/js/api/playwright-crawler#EventType) * [EventTypeName](/js/api/playwright-crawler#EventTypeName) * [filterRequestsByPatterns](/js/api/playwright-crawler#filterRequestsByPatterns) * [FinalStatistics](/js/api/playwright-crawler#FinalStatistics) * [GetUserDataFromRequest](/js/api/playwright-crawler#GetUserDataFromRequest) * [GlobInput](/js/api/playwright-crawler#GlobInput) * [GlobObject](/js/api/playwright-crawler#GlobObject) * [GotScrapingHttpClient](/js/api/playwright-crawler#GotScrapingHttpClient) * [HttpRequest](/js/api/playwright-crawler#HttpRequest) * [HttpRequestOptions](/js/api/playwright-crawler#HttpRequestOptions) * [HttpResponse](/js/api/playwright-crawler#HttpResponse) * [IRequestList](/js/api/playwright-crawler#IRequestList) * [IStorage](/js/api/playwright-crawler#IStorage) * [KeyConsumer](/js/api/playwright-crawler#KeyConsumer) * [KeyValueStore](/js/api/playwright-crawler#KeyValueStore) * [KeyValueStoreIteratorOptions](/js/api/playwright-crawler#KeyValueStoreIteratorOptions) * [KeyValueStoreOptions](/js/api/playwright-crawler#KeyValueStoreOptions) * [LoadedRequest](/js/api/playwright-crawler#LoadedRequest) * [LocalEventManager](/js/api/playwright-crawler#LocalEventManager) * [log](/js/api/playwright-crawler#log) * [Log](/js/api/playwright-crawler#Log) * [Logger](/js/api/playwright-crawler#Logger) * [LoggerJson](/js/api/playwright-crawler#LoggerJson) * [LoggerOptions](/js/api/playwright-crawler#LoggerOptions) * [LoggerText](/js/api/playwright-crawler#LoggerText) * [LogLevel](/js/api/playwright-crawler#LogLevel) * [MAX\_POOL\_SIZE](/js/api/playwright-crawler#MAX_POOL_SIZE) * [NonRetryableError](/js/api/playwright-crawler#NonRetryableError) * [PERSIST\_STATE\_KEY](/js/api/playwright-crawler#PERSIST_STATE_KEY) * [PersistenceOptions](/js/api/playwright-crawler#PersistenceOptions) * [PlaywrightDirectNavigationOptions](/js/api/playwright-crawler#PlaywrightDirectNavigationOptions) * [processHttpRequestOptions](/js/api/playwright-crawler#processHttpRequestOptions) * [ProxyConfiguration](/js/api/playwright-crawler#ProxyConfiguration) * [ProxyConfigurationFunction](/js/api/playwright-crawler#ProxyConfigurationFunction) * [ProxyConfigurationOptions](/js/api/playwright-crawler#ProxyConfigurationOptions) * [ProxyInfo](/js/api/playwright-crawler#ProxyInfo) * [PseudoUrl](/js/api/playwright-crawler#PseudoUrl) * [PseudoUrlInput](/js/api/playwright-crawler#PseudoUrlInput) * [PseudoUrlObject](/js/api/playwright-crawler#PseudoUrlObject) * [purgeDefaultStorages](/js/api/playwright-crawler#purgeDefaultStorages) * [PushErrorMessageOptions](/js/api/playwright-crawler#PushErrorMessageOptions) * [QueueOperationInfo](/js/api/playwright-crawler#QueueOperationInfo) * [RecordOptions](/js/api/playwright-crawler#RecordOptions) * [RedirectHandler](/js/api/playwright-crawler#RedirectHandler) * [RegExpInput](/js/api/playwright-crawler#RegExpInput) * [RegExpObject](/js/api/playwright-crawler#RegExpObject) * [Request](/js/api/playwright-crawler#Request) * [RequestHandler](/js/api/playwright-crawler#RequestHandler) * [RequestHandlerResult](/js/api/playwright-crawler#RequestHandlerResult) * [RequestList](/js/api/playwright-crawler#RequestList) * [RequestListOptions](/js/api/playwright-crawler#RequestListOptions) * [RequestListSourcesFunction](/js/api/playwright-crawler#RequestListSourcesFunction) * [RequestListState](/js/api/playwright-crawler#RequestListState) * [RequestOptions](/js/api/playwright-crawler#RequestOptions) * [RequestProvider](/js/api/playwright-crawler#RequestProvider) * [RequestProviderOptions](/js/api/playwright-crawler#RequestProviderOptions) * [RequestQueue](/js/api/playwright-crawler#RequestQueue) * [RequestQueueOperationOptions](/js/api/playwright-crawler#RequestQueueOperationOptions) * [RequestQueueOptions](/js/api/playwright-crawler#RequestQueueOptions) * [RequestQueueV1](/js/api/playwright-crawler#RequestQueueV1) * [RequestQueueV2](/js/api/playwright-crawler#RequestQueueV2) * [RequestState](/js/api/playwright-crawler#RequestState) * [RequestTransform](/js/api/playwright-crawler#RequestTransform) * [ResponseLike](/js/api/playwright-crawler#ResponseLike) * [ResponseTypes](/js/api/playwright-crawler#ResponseTypes) * [RestrictedCrawlingContext](/js/api/playwright-crawler#RestrictedCrawlingContext) * [RetryRequestError](/js/api/playwright-crawler#RetryRequestError) * [Router](/js/api/playwright-crawler#Router) * [RouterHandler](/js/api/playwright-crawler#RouterHandler) * [RouterRoutes](/js/api/playwright-crawler#RouterRoutes) * [Session](/js/api/playwright-crawler#Session) * [SessionError](/js/api/playwright-crawler#SessionError) * [SessionOptions](/js/api/playwright-crawler#SessionOptions) * [SessionPool](/js/api/playwright-crawler#SessionPool) * [SessionPoolOptions](/js/api/playwright-crawler#SessionPoolOptions) * [SessionState](/js/api/playwright-crawler#SessionState) * [SitemapRequestList](/js/api/playwright-crawler#SitemapRequestList) * [SitemapRequestListOptions](/js/api/playwright-crawler#SitemapRequestListOptions) * [SnapshotResult](/js/api/playwright-crawler#SnapshotResult) * [Snapshotter](/js/api/playwright-crawler#Snapshotter) * [SnapshotterOptions](/js/api/playwright-crawler#SnapshotterOptions) * [Source](/js/api/playwright-crawler#Source) * [StatisticPersistedState](/js/api/playwright-crawler#StatisticPersistedState) * [Statistics](/js/api/playwright-crawler#Statistics) * [StatisticsOptions](/js/api/playwright-crawler#StatisticsOptions) * [StatisticState](/js/api/playwright-crawler#StatisticState) * [StatusMessageCallback](/js/api/playwright-crawler#StatusMessageCallback) * [StatusMessageCallbackParams](/js/api/playwright-crawler#StatusMessageCallbackParams) * [StorageClient](/js/api/playwright-crawler#StorageClient) * [StorageManagerOptions](/js/api/playwright-crawler#StorageManagerOptions) * [StreamingHttpResponse](/js/api/playwright-crawler#StreamingHttpResponse) * [SystemInfo](/js/api/playwright-crawler#SystemInfo) * [SystemStatus](/js/api/playwright-crawler#SystemStatus) * [SystemStatusOptions](/js/api/playwright-crawler#SystemStatusOptions) * [TieredProxy](/js/api/playwright-crawler#TieredProxy) * [tryAbsoluteURL](/js/api/playwright-crawler#tryAbsoluteURL) * [UrlPatternObject](/js/api/playwright-crawler#UrlPatternObject) * [useState](/js/api/playwright-crawler#useState) * [UseStateOptions](/js/api/playwright-crawler#UseStateOptions) * [withCheckedStorageAccess](/js/api/playwright-crawler#withCheckedStorageAccess) * [playwrightClickElements](/js/api/playwright-crawler/namespace/playwrightClickElements) * [playwrightUtils](/js/api/playwright-crawler/namespace/playwrightUtils) * [AdaptivePlaywrightCrawler](/js/api/playwright-crawler/class/AdaptivePlaywrightCrawler) * [AdaptivePlaywrightCrawlerContext](/js/api/playwright-crawler/interface/AdaptivePlaywrightCrawlerContext) * [AdaptivePlaywrightCrawlerOptions](/js/api/playwright-crawler/interface/AdaptivePlaywrightCrawlerOptions) * [PlaywrightCrawlerOptions](/js/api/playwright-crawler/interface/PlaywrightCrawlerOptions) * [PlaywrightCrawlingContext](/js/api/playwright-crawler/interface/PlaywrightCrawlingContext) * [PlaywrightHook](/js/api/playwright-crawler/interface/PlaywrightHook) * [PlaywrightLaunchContext](/js/api/playwright-crawler/interface/PlaywrightLaunchContext) * [PlaywrightRequestHandler](/js/api/playwright-crawler/interface/PlaywrightRequestHandler) * [PlaywrightGotoOptions](/js/api/playwright-crawler#PlaywrightGotoOptions) * [RenderingType](/js/api/playwright-crawler#RenderingType) * [createAdaptivePlaywrightRouter](/js/api/playwright-crawler/function/createAdaptivePlaywrightRouter) * [createPlaywrightRouter](/js/api/playwright-crawler/function/createPlaywrightRouter) * [launchPlaywright](/js/api/playwright-crawler/function/launchPlaywright) Other[](#__CATEGORY__) ----------------------- ### [](#AddRequestsBatchedOptions) [](https://github.com/apify/crawlee/blob/master/packages/core/src/storages/request_provider.ts#L852) AddRequestsBatchedOptions Re-exports [AddRequestsBatchedOptions](/js/api/core/interface/AddRequestsBatchedOptions) ### [](#AddRequestsBatchedResult) [](https://github.com/apify/crawlee/blob/master/packages/core/src/storages/request_provider.ts#L870) AddRequestsBatchedResult Re-exports [AddRequestsBatchedResult](/js/api/core/interface/AddRequestsBatchedResult) ### [](#AutoscaledPool) [](https://github.com/apify/crawlee/blob/master/packages/core/src/autoscaling/autoscaled_pool.ts#L179) AutoscaledPool Re-exports [AutoscaledPool](/js/api/core/class/AutoscaledPool) ### [](#AutoscaledPoolOptions) [](https://github.com/apify/crawlee/blob/master/packages/core/src/autoscaling/autoscaled_pool.ts#L15) AutoscaledPoolOptions Re-exports [AutoscaledPoolOptions](/js/api/core/interface/AutoscaledPoolOptions) ### [](#BaseHttpClient) [](https://github.com/apify/crawlee/blob/master/packages/core/src/http_clients/base-http-client.ts#L179) BaseHttpClient Re-exports [BaseHttpClient](/js/api/core/interface/BaseHttpClient) ### [](#BaseHttpResponseData) [](https://github.com/apify/crawlee/blob/master/packages/core/src/http_clients/base-http-client.ts#L130) BaseHttpResponseData Re-exports [BaseHttpResponseData](/js/api/core/interface/BaseHttpResponseData) ### [](#BASIC_CRAWLER_TIMEOUT_BUFFER_SECS) [](https://github.com/apify/crawlee/blob/master/packages/basic-crawler/src/internals/constants.ts#L6) BASIC\_CRAWLER\_TIMEOUT\_BUFFER\_SECS Re-exports [BASIC\_CRAWLER\_TIMEOUT\_BUFFER\_SECS](/js/api/basic-crawler#BASIC_CRAWLER_TIMEOUT_BUFFER_SECS) ### [](#BasicCrawler) [](https://github.com/apify/crawlee/blob/master/packages/basic-crawler/src/internals/basic-crawler.ts#L446) BasicCrawler Re-exports [BasicCrawler](/js/api/basic-crawler/class/BasicCrawler) ### [](#BasicCrawlerOptions) [](https://github.com/apify/crawlee/blob/master/packages/basic-crawler/src/internals/basic-crawler.ts#L130) BasicCrawlerOptions Re-exports [BasicCrawlerOptions](/js/api/basic-crawler/interface/BasicCrawlerOptions) ### [](#BasicCrawlingContext) [](https://github.com/apify/crawlee/blob/master/packages/basic-crawler/src/internals/basic-crawler.ts#L68) BasicCrawlingContext Re-exports [BasicCrawlingContext](/js/api/basic-crawler/interface/BasicCrawlingContext) ### [](#BLOCKED_STATUS_CODES) [](https://github.com/apify/crawlee/blob/master/packages/core/src/session_pool/consts.ts#L1) BLOCKED\_STATUS\_CODES Re-exports [BLOCKED\_STATUS\_CODES](/js/api/core#BLOCKED_STATUS_CODES) ### [](#BrowserCrawler) [](https://github.com/apify/crawlee/blob/master/packages/browser-crawler/src/internals/browser-crawler.ts#L310) BrowserCrawler Re-exports [BrowserCrawler](/js/api/browser-crawler/class/BrowserCrawler) ### [](#BrowserCrawlerOptions) [](https://github.com/apify/crawlee/blob/master/packages/browser-crawler/src/internals/browser-crawler.ts#L71) BrowserCrawlerOptions Re-exports [BrowserCrawlerOptions](/js/api/browser-crawler/interface/BrowserCrawlerOptions) ### [](#BrowserCrawlingContext) [](https://github.com/apify/crawlee/blob/master/packages/browser-crawler/src/internals/browser-crawler.ts#L48) BrowserCrawlingContext Re-exports [BrowserCrawlingContext](/js/api/browser-crawler/interface/BrowserCrawlingContext) ### [](#BrowserErrorHandler) [](https://github.com/apify/crawlee/blob/master/packages/browser-crawler/src/internals/browser-crawler.ts#L63) BrowserErrorHandler Re-exports [BrowserErrorHandler](/js/api/browser-crawler#BrowserErrorHandler) ### [](#BrowserHook) [](https://github.com/apify/crawlee/blob/master/packages/browser-crawler/src/internals/browser-crawler.ts#L66) BrowserHook Re-exports [BrowserHook](/js/api/browser-crawler#BrowserHook) ### [](#BrowserLaunchContext) [](https://github.com/apify/crawlee/blob/master/packages/browser-crawler/src/internals/browser-launcher.ts#L14) BrowserLaunchContext Re-exports [BrowserLaunchContext](/js/api/browser-crawler/interface/BrowserLaunchContext) ### [](#BrowserRequestHandler) [](https://github.com/apify/crawlee/blob/master/packages/browser-crawler/src/internals/browser-crawler.ts#L60) BrowserRequestHandler Re-exports [BrowserRequestHandler](/js/api/browser-crawler#BrowserRequestHandler) ### [](#checkStorageAccess) [](https://github.com/apify/crawlee/blob/master/packages/core/src/storages/access_checking.ts#L10) checkStorageAccess Re-exports [checkStorageAccess](/js/api/core/function/checkStorageAccess) ### [](#ClientInfo) [](https://github.com/apify/crawlee/blob/master/packages/core/src/autoscaling/system_status.ts#L79) ClientInfo Re-exports [ClientInfo](/js/api/core/interface/ClientInfo) ### [](#Configuration) [](https://github.com/apify/crawlee/blob/master/packages/core/src/configuration.ts#L246) Configuration Re-exports [Configuration](/js/api/core/class/Configuration) ### [](#ConfigurationOptions) [](https://github.com/apify/crawlee/blob/master/packages/core/src/configuration.ts#L15) ConfigurationOptions Re-exports [ConfigurationOptions](/js/api/core/interface/ConfigurationOptions) ### [](#Cookie) [](https://github.com/apify/crawlee/blob/master/packages/core/src/index.ts#L18) Cookie Re-exports [Cookie](/js/api/core/interface/Cookie) ### [](#CrawlerAddRequestsOptions) [](https://github.com/apify/crawlee/blob/master/packages/basic-crawler/src/internals/basic-crawler.ts#L1751) CrawlerAddRequestsOptions Re-exports [CrawlerAddRequestsOptions](/js/api/basic-crawler/interface/CrawlerAddRequestsOptions) ### [](#CrawlerAddRequestsResult) [](https://github.com/apify/crawlee/blob/master/packages/basic-crawler/src/internals/basic-crawler.ts#L1753) CrawlerAddRequestsResult Re-exports [CrawlerAddRequestsResult](/js/api/basic-crawler/interface/CrawlerAddRequestsResult) ### [](#CrawlerExperiments) [](https://github.com/apify/crawlee/blob/master/packages/basic-crawler/src/internals/basic-crawler.ts#L372) CrawlerExperiments Re-exports [CrawlerExperiments](/js/api/basic-crawler/interface/CrawlerExperiments) ### [](#CrawlerRunOptions) [](https://github.com/apify/crawlee/blob/master/packages/basic-crawler/src/internals/basic-crawler.ts#L1755) CrawlerRunOptions Re-exports [CrawlerRunOptions](/js/api/basic-crawler/interface/CrawlerRunOptions) ### [](#CrawlingContext) [](https://github.com/apify/crawlee/blob/master/packages/core/src/crawlers/crawler_commons.ts#L111) CrawlingContext Re-exports [CrawlingContext](/js/api/core/interface/CrawlingContext) ### [](#createBasicRouter) [](https://github.com/apify/crawlee/blob/master/packages/basic-crawler/src/internals/basic-crawler.ts#L1797) createBasicRouter Re-exports [createBasicRouter](/js/api/basic-crawler/function/createBasicRouter) ### [](#CreateContextOptions) [](https://github.com/apify/crawlee/blob/master/packages/basic-crawler/src/internals/basic-crawler.ts#L1745) CreateContextOptions Re-exports [CreateContextOptions](/js/api/basic-crawler/interface/CreateContextOptions) ### [](#CreateSession) [](https://github.com/apify/crawlee/blob/master/packages/core/src/session_pool/session_pool.ts#L21) CreateSession Re-exports [CreateSession](/js/api/core/interface/CreateSession) ### [](#CriticalError) [](https://github.com/apify/crawlee/blob/master/packages/core/src/errors.ts#L10) CriticalError Re-exports [CriticalError](/js/api/core/class/CriticalError) ### [](#Dataset) [](https://github.com/apify/crawlee/blob/master/packages/core/src/storages/dataset.ts#L225) Dataset Re-exports [Dataset](/js/api/core/class/Dataset) ### [](#DatasetConsumer) [](https://github.com/apify/crawlee/blob/master/packages/core/src/storages/dataset.ts#L684) DatasetConsumer Re-exports [DatasetConsumer](/js/api/core/interface/DatasetConsumer) ### [](#DatasetContent) [](https://github.com/apify/crawlee/blob/master/packages/core/src/storages/dataset.ts#L722) DatasetContent Re-exports [DatasetContent](/js/api/core/interface/DatasetContent) ### [](#DatasetDataOptions) [](https://github.com/apify/crawlee/blob/master/packages/core/src/storages/dataset.ts#L91) DatasetDataOptions Re-exports [DatasetDataOptions](/js/api/core/interface/DatasetDataOptions) ### [](#DatasetExportOptions) [](https://github.com/apify/crawlee/blob/master/packages/core/src/storages/dataset.ts#L143) DatasetExportOptions Re-exports [DatasetExportOptions](/js/api/core/interface/DatasetExportOptions) ### [](#DatasetExportToOptions) [](https://github.com/apify/crawlee/blob/master/packages/core/src/storages/dataset.ts#L169) DatasetExportToOptions Re-exports [DatasetExportToOptions](/js/api/core/interface/DatasetExportToOptions) ### [](#DatasetIteratorOptions) [](https://github.com/apify/crawlee/blob/master/packages/core/src/storages/dataset.ts#L145) DatasetIteratorOptions Re-exports [DatasetIteratorOptions](/js/api/core/interface/DatasetIteratorOptions) ### [](#DatasetMapper) [](https://github.com/apify/crawlee/blob/master/packages/core/src/storages/dataset.ts#L695) DatasetMapper Re-exports [DatasetMapper](/js/api/core/interface/DatasetMapper) ### [](#DatasetOptions) [](https://github.com/apify/crawlee/blob/master/packages/core/src/storages/dataset.ts#L716) DatasetOptions Re-exports [DatasetOptions](/js/api/core/interface/DatasetOptions) ### [](#DatasetReducer) [](https://github.com/apify/crawlee/blob/master/packages/core/src/storages/dataset.ts#L707) DatasetReducer Re-exports [DatasetReducer](/js/api/core/interface/DatasetReducer) ### [](#enqueueLinks) [](https://github.com/apify/crawlee/blob/master/packages/core/src/enqueue_links/enqueue_links.ts#L241) enqueueLinks Re-exports [enqueueLinks](/js/api/core/function/enqueueLinks) ### [](#EnqueueLinksOptions) [](https://github.com/apify/crawlee/blob/master/packages/core/src/enqueue_links/enqueue_links.ts#L19) EnqueueLinksOptions Re-exports [EnqueueLinksOptions](/js/api/core/interface/EnqueueLinksOptions) ### [](#EnqueueStrategy) [](https://github.com/apify/crawlee/blob/master/packages/core/src/enqueue_links/enqueue_links.ts#L183) EnqueueStrategy Re-exports [EnqueueStrategy](/js/api/core/enum/EnqueueStrategy) ### [](#ErrnoException) [](https://github.com/apify/crawlee/blob/master/packages/core/src/crawlers/error_tracker.ts#L9) ErrnoException Re-exports [ErrnoException](/js/api/core/interface/ErrnoException) ### [](#ErrorHandler) [](https://github.com/apify/crawlee/blob/master/packages/basic-crawler/src/internals/basic-crawler.ts#L111) ErrorHandler Re-exports [ErrorHandler](/js/api/basic-crawler#ErrorHandler) ### [](#ErrorSnapshotter) [](https://github.com/apify/crawlee/blob/master/packages/core/src/crawlers/error_snapshotter.ts#L42) ErrorSnapshotter Re-exports [ErrorSnapshotter](/js/api/core/class/ErrorSnapshotter) ### [](#ErrorTracker) [](https://github.com/apify/crawlee/blob/master/packages/core/src/crawlers/error_tracker.ts#L286) ErrorTracker Re-exports [ErrorTracker](/js/api/core/class/ErrorTracker) ### [](#ErrorTrackerOptions) [](https://github.com/apify/crawlee/blob/master/packages/core/src/crawlers/error_tracker.ts#L17) ErrorTrackerOptions Re-exports [ErrorTrackerOptions](/js/api/core/interface/ErrorTrackerOptions) ### [](#EventManager) [](https://github.com/apify/crawlee/blob/master/packages/core/src/events/event_manager.ts#L23) EventManager Re-exports [EventManager](/js/api/core/class/EventManager) ### [](#EventType) [](https://github.com/apify/crawlee/blob/master/packages/core/src/events/event_manager.ts#L8) EventType Re-exports [EventType](/js/api/core/enum/EventType) ### [](#EventTypeName) [](https://github.com/apify/crawlee/blob/master/packages/core/src/events/event_manager.ts#L16) EventTypeName Re-exports [EventTypeName](/js/api/core#EventTypeName) ### [](#filterRequestsByPatterns) [](https://github.com/apify/crawlee/blob/master/packages/core/src/enqueue_links/shared.ts#L200) filterRequestsByPatterns Re-exports [filterRequestsByPatterns](/js/api/core/function/filterRequestsByPatterns) ### [](#FinalStatistics) [](https://github.com/apify/crawlee/blob/master/packages/core/src/autoscaling/system_status.ts#L85) FinalStatistics Re-exports [FinalStatistics](/js/api/core/interface/FinalStatistics) ### [](#GetUserDataFromRequest) [](https://github.com/apify/crawlee/blob/master/packages/core/src/router.ts#L15) GetUserDataFromRequest Re-exports [GetUserDataFromRequest](/js/api/core#GetUserDataFromRequest) ### [](#GlobInput) [](https://github.com/apify/crawlee/blob/master/packages/core/src/enqueue_links/shared.ts#L39) GlobInput Re-exports [GlobInput](/js/api/core#GlobInput) ### [](#GlobObject) [](https://github.com/apify/crawlee/blob/master/packages/core/src/enqueue_links/shared.ts#L34) GlobObject Re-exports [GlobObject](/js/api/core#GlobObject) ### [](#GotScrapingHttpClient) [](https://github.com/apify/crawlee/blob/master/packages/core/src/http_clients/got-scraping-http-client.ts#L17) GotScrapingHttpClient Re-exports [GotScrapingHttpClient](/js/api/core/class/GotScrapingHttpClient) ### [](#HttpRequest) [](https://github.com/apify/crawlee/blob/master/packages/core/src/http_clients/base-http-client.ts#L78) HttpRequest Re-exports [HttpRequest](/js/api/core/interface/HttpRequest) ### [](#HttpRequestOptions) [](https://github.com/apify/crawlee/blob/master/packages/core/src/http_clients/base-http-client.ts#L111) HttpRequestOptions Re-exports [HttpRequestOptions](/js/api/core/interface/HttpRequestOptions) ### [](#HttpResponse) [](https://github.com/apify/crawlee/blob/master/packages/core/src/http_clients/base-http-client.ts#L152) HttpResponse Re-exports [HttpResponse](/js/api/core/interface/HttpResponse) ### [](#IRequestList) [](https://github.com/apify/crawlee/blob/master/packages/core/src/storages/request_list.ts#L26) IRequestList Re-exports [IRequestList](/js/api/core/interface/IRequestList) ### [](#IStorage) [](https://github.com/apify/crawlee/blob/master/packages/core/src/storages/storage_manager.ts#L14) IStorage Re-exports [IStorage](/js/api/core/interface/IStorage) ### [](#KeyConsumer) [](https://github.com/apify/crawlee/blob/master/packages/core/src/storages/key_value_store.ts#L699) KeyConsumer Re-exports [KeyConsumer](/js/api/core/interface/KeyConsumer) ### [](#KeyValueStore) [](https://github.com/apify/crawlee/blob/master/packages/core/src/storages/key_value_store.ts#L106) KeyValueStore Re-exports [KeyValueStore](/js/api/core/class/KeyValueStore) ### [](#KeyValueStoreIteratorOptions) [](https://github.com/apify/crawlee/blob/master/packages/core/src/storages/key_value_store.ts#L723) KeyValueStoreIteratorOptions Re-exports [KeyValueStoreIteratorOptions](/js/api/core/interface/KeyValueStoreIteratorOptions) ### [](#KeyValueStoreOptions) [](https://github.com/apify/crawlee/blob/master/packages/core/src/storages/key_value_store.ts#L709) KeyValueStoreOptions Re-exports [KeyValueStoreOptions](/js/api/core/interface/KeyValueStoreOptions) ### [](#LoadedRequest) [](https://github.com/apify/crawlee/blob/master/packages/core/src/crawlers/crawler_commons.ts#L21) LoadedRequest Re-exports [LoadedRequest](/js/api/core#LoadedRequest) ### [](#LocalEventManager) [](https://github.com/apify/crawlee/blob/master/packages/core/src/events/local_event_manager.ts#L10) LocalEventManager Re-exports [LocalEventManager](/js/api/core/class/LocalEventManager) ### [](#log) [](https://github.com/apify/crawlee/blob/master/packages/core/src/log.ts#L3) log Re-exports [log](/js/api/core#log) ### [](#Log) [](https://github.com/apify/crawlee/blob/master/packages/core/src/log.ts#L3) Log Re-exports [Log](/js/api/core/class/Log) ### [](#Logger) [](https://github.com/apify/crawlee/blob/master/packages/core/src/log.ts#L3) Logger Re-exports [Logger](/js/api/core/class/Logger) ### [](#LoggerJson) [](https://github.com/apify/crawlee/blob/master/packages/core/src/log.ts#L3) LoggerJson Re-exports [LoggerJson](/js/api/core/class/LoggerJson) ### [](#LoggerOptions) [](https://github.com/apify/crawlee/blob/master/packages/core/src/log.ts#L3) LoggerOptions Re-exports [LoggerOptions](/js/api/core/interface/LoggerOptions) ### [](#LoggerText) [](https://github.com/apify/crawlee/blob/master/packages/core/src/log.ts#L3) LoggerText Re-exports [LoggerText](/js/api/core/class/LoggerText) ### [](#LogLevel) [](https://github.com/apify/crawlee/blob/master/packages/core/src/log.ts#L3) LogLevel Re-exports [LogLevel](/js/api/core/enum/LogLevel) ### [](#MAX_POOL_SIZE) [](https://github.com/apify/crawlee/blob/master/packages/core/src/session_pool/consts.ts#L3) MAX\_POOL\_SIZE Re-exports [MAX\_POOL\_SIZE](/js/api/core#MAX_POOL_SIZE) ### [](#NonRetryableError) [](https://github.com/apify/crawlee/blob/master/packages/core/src/errors.ts#L4) NonRetryableError Re-exports [NonRetryableError](/js/api/core/class/NonRetryableError) ### [](#PERSIST_STATE_KEY) [](https://github.com/apify/crawlee/blob/master/packages/core/src/session_pool/consts.ts#L2) PERSIST\_STATE\_KEY Re-exports [PERSIST\_STATE\_KEY](/js/api/core#PERSIST_STATE_KEY) ### [](#PersistenceOptions) [](https://github.com/apify/crawlee/blob/master/packages/core/src/crawlers/statistics.ts#L46) PersistenceOptions Re-exports [PersistenceOptions](/js/api/core/interface/PersistenceOptions) ### [](#PlaywrightDirectNavigationOptions) [](https://github.com/apify/crawlee/blob/master/packages/playwright-crawler/src/index.ts#L8) PlaywrightDirectNavigationOptions Renames and re-exports [DirectNavigationOptions](/js/api/playwright-crawler/namespace/playwrightUtils#DirectNavigationOptions) ### [](#processHttpRequestOptions) [](https://github.com/apify/crawlee/blob/master/packages/core/src/http_clients/base-http-client.ts#L196) processHttpRequestOptions Re-exports [processHttpRequestOptions](/js/api/core/function/processHttpRequestOptions) ### [](#ProxyConfiguration) [](https://github.com/apify/crawlee/blob/master/packages/core/src/proxy_configuration.ts#L200) ProxyConfiguration Re-exports [ProxyConfiguration](/js/api/core/class/ProxyConfiguration) ### [](#ProxyConfigurationFunction) [](https://github.com/apify/crawlee/blob/master/packages/core/src/proxy_configuration.ts#L8) ProxyConfigurationFunction Re-exports [ProxyConfigurationFunction](/js/api/core/interface/ProxyConfigurationFunction) ### [](#ProxyConfigurationOptions) [](https://github.com/apify/crawlee/blob/master/packages/core/src/proxy_configuration.ts#L12) ProxyConfigurationOptions Re-exports [ProxyConfigurationOptions](/js/api/core/interface/ProxyConfigurationOptions) ### [](#ProxyInfo) [](https://github.com/apify/crawlee/blob/master/packages/core/src/proxy_configuration.ts#L77) ProxyInfo Re-exports [ProxyInfo](/js/api/core/interface/ProxyInfo) ### [](#PseudoUrl) [](https://github.com/apify/crawlee/blob/master/packages/core/src/index.ts#L17) PseudoUrl Re-exports [PseudoUrl](/js/api/core/class/PseudoUrl) ### [](#PseudoUrlInput) [](https://github.com/apify/crawlee/blob/master/packages/core/src/enqueue_links/shared.ts#L32) PseudoUrlInput Re-exports [PseudoUrlInput](/js/api/core#PseudoUrlInput) ### [](#PseudoUrlObject) [](https://github.com/apify/crawlee/blob/master/packages/core/src/enqueue_links/shared.ts#L27) PseudoUrlObject Re-exports [PseudoUrlObject](/js/api/core#PseudoUrlObject) ### [](#purgeDefaultStorages) [](https://github.com/apify/crawlee/blob/master/packages/core/src/storages/utils.ts#L33) [](https://github.com/apify/crawlee/blob/master/packages/core/src/storages/utils.ts#L45) [](https://github.com/apify/crawlee/blob/master/packages/core/src/storages/utils.ts#L46) purgeDefaultStorages Re-exports [purgeDefaultStorages](/js/api/core/function/purgeDefaultStorages) ### [](#PushErrorMessageOptions) [](https://github.com/apify/crawlee/blob/master/packages/core/src/request.ts#L523) PushErrorMessageOptions Re-exports [PushErrorMessageOptions](/js/api/core/interface/PushErrorMessageOptions) ### [](#QueueOperationInfo) [](https://github.com/apify/crawlee/blob/master/packages/core/src/index.ts#L18) QueueOperationInfo Re-exports [QueueOperationInfo](/js/api/core/interface/QueueOperationInfo) ### [](#RecordOptions) [](https://github.com/apify/crawlee/blob/master/packages/core/src/storages/key_value_store.ts#L716) RecordOptions Re-exports [RecordOptions](/js/api/core/interface/RecordOptions) ### [](#RedirectHandler) [](https://github.com/apify/crawlee/blob/master/packages/core/src/http_clients/base-http-client.ts#L171) RedirectHandler Re-exports [RedirectHandler](/js/api/core#RedirectHandler) ### [](#RegExpInput) [](https://github.com/apify/crawlee/blob/master/packages/core/src/enqueue_links/shared.ts#L46) RegExpInput Re-exports [RegExpInput](/js/api/core#RegExpInput) ### [](#RegExpObject) [](https://github.com/apify/crawlee/blob/master/packages/core/src/enqueue_links/shared.ts#L41) RegExpObject Re-exports [RegExpObject](/js/api/core#RegExpObject) ### [](#Request) [](https://github.com/apify/crawlee/blob/master/packages/core/src/request.ts#L81) Request Re-exports [Request](/js/api/core/class/Request) ### [](#RequestHandler) [](https://github.com/apify/crawlee/blob/master/packages/basic-crawler/src/internals/basic-crawler.ts#L107) RequestHandler Re-exports [RequestHandler](/js/api/basic-crawler#RequestHandler) ### [](#RequestHandlerResult) [](https://github.com/apify/crawlee/blob/master/packages/core/src/crawlers/crawler_commons.ts#L174) RequestHandlerResult Re-exports [RequestHandlerResult](/js/api/core/class/RequestHandlerResult) ### [](#RequestList) [](https://github.com/apify/crawlee/blob/master/packages/core/src/storages/request_list.ts#L306) RequestList Re-exports [RequestList](/js/api/core/class/RequestList) ### [](#RequestListOptions) [](https://github.com/apify/crawlee/blob/master/packages/core/src/storages/request_list.ts#L97) RequestListOptions Re-exports [RequestListOptions](/js/api/core/interface/RequestListOptions) ### [](#RequestListSourcesFunction) [](https://github.com/apify/crawlee/blob/master/packages/core/src/storages/request_list.ts#L1006) RequestListSourcesFunction Re-exports [RequestListSourcesFunction](/js/api/core#RequestListSourcesFunction) ### [](#RequestListState) [](https://github.com/apify/crawlee/blob/master/packages/core/src/storages/request_list.ts#L994) RequestListState Re-exports [RequestListState](/js/api/core/interface/RequestListState) ### [](#RequestOptions) [](https://github.com/apify/crawlee/blob/master/packages/core/src/request.ts#L424) RequestOptions Re-exports [RequestOptions](/js/api/core/interface/RequestOptions) ### [](#RequestProvider) [](https://github.com/apify/crawlee/blob/master/packages/core/src/storages/request_provider.ts#L30) RequestProvider Re-exports [RequestProvider](/js/api/core/class/RequestProvider) ### [](#RequestProviderOptions) [](https://github.com/apify/crawlee/blob/master/packages/core/src/storages/request_provider.ts#L794) RequestProviderOptions Re-exports [RequestProviderOptions](/js/api/core/interface/RequestProviderOptions) ### [](#RequestQueue) [](https://github.com/apify/crawlee/blob/master/packages/core/src/storages/index.ts#L6) RequestQueue Re-exports [RequestQueue](/js/api/core/class/RequestQueue) ### [](#RequestQueueOperationOptions) [](https://github.com/apify/crawlee/blob/master/packages/core/src/storages/request_provider.ts#L821) RequestQueueOperationOptions Re-exports [RequestQueueOperationOptions](/js/api/core/interface/RequestQueueOperationOptions) ### [](#RequestQueueOptions) [](https://github.com/apify/crawlee/blob/master/packages/core/src/storages/request_provider.ts#L810) RequestQueueOptions Re-exports [RequestQueueOptions](/js/api/core/interface/RequestQueueOptions) ### [](#RequestQueueV1) [](https://github.com/apify/crawlee/blob/master/packages/core/src/storages/index.ts#L5) RequestQueueV1 Re-exports [RequestQueueV1](/js/api/core/class/RequestQueueV1) ### [](#RequestQueueV2) [](https://github.com/apify/crawlee/blob/master/packages/core/src/storages/index.ts#L7) RequestQueueV2 Re-exports [RequestQueueV2](/js/api/core#RequestQueueV2) ### [](#RequestState) [](https://github.com/apify/crawlee/blob/master/packages/core/src/request.ts#L39) RequestState Re-exports [RequestState](/js/api/core/enum/RequestState) ### [](#RequestTransform) [](https://github.com/apify/crawlee/blob/master/packages/core/src/enqueue_links/shared.ts#L265) RequestTransform Re-exports [RequestTransform](/js/api/core/interface/RequestTransform) ### [](#ResponseLike) [](https://github.com/apify/crawlee/blob/master/packages/core/src/cookie_utils.ts#L7) ResponseLike Re-exports [ResponseLike](/js/api/core/interface/ResponseLike) ### [](#ResponseTypes) [](https://github.com/apify/crawlee/blob/master/packages/core/src/http_clients/base-http-client.ts#L39) ResponseTypes Re-exports [ResponseTypes](/js/api/core/interface/ResponseTypes) ### [](#RestrictedCrawlingContext) [](https://github.com/apify/crawlee/blob/master/packages/core/src/crawlers/crawler_commons.ts#L30) RestrictedCrawlingContext Re-exports [RestrictedCrawlingContext](/js/api/core/interface/RestrictedCrawlingContext) ### [](#RetryRequestError) [](https://github.com/apify/crawlee/blob/master/packages/core/src/errors.ts#L22) RetryRequestError Re-exports [RetryRequestError](/js/api/core/class/RetryRequestError) ### [](#Router) [](https://github.com/apify/crawlee/blob/master/packages/core/src/router.ts#L86) Router Re-exports [Router](/js/api/core/class/Router) ### [](#RouterHandler) [](https://github.com/apify/crawlee/blob/master/packages/core/src/router.ts#L10) RouterHandler Re-exports [RouterHandler](/js/api/core/interface/RouterHandler) ### [](#RouterRoutes) [](https://github.com/apify/crawlee/blob/master/packages/core/src/router.ts#L17) RouterRoutes Re-exports [RouterRoutes](/js/api/core#RouterRoutes) ### [](#Session) [](https://github.com/apify/crawlee/blob/master/packages/core/src/session_pool/session.ts#L99) Session Re-exports [Session](/js/api/core/class/Session) ### [](#SessionError) [](https://github.com/apify/crawlee/blob/master/packages/core/src/errors.ts#L33) SessionError Re-exports [SessionError](/js/api/core/class/SessionError) ### [](#SessionOptions) [](https://github.com/apify/crawlee/blob/master/packages/core/src/session_pool/session.ts#L36) SessionOptions Re-exports [SessionOptions](/js/api/core/interface/SessionOptions) ### [](#SessionPool) [](https://github.com/apify/crawlee/blob/master/packages/core/src/session_pool/session_pool.ts#L136) SessionPool Re-exports [SessionPool](/js/api/core/class/SessionPool) ### [](#SessionPoolOptions) [](https://github.com/apify/crawlee/blob/master/packages/core/src/session_pool/session_pool.ts#L29) SessionPoolOptions Re-exports [SessionPoolOptions](/js/api/core/interface/SessionPoolOptions) ### [](#SessionState) [](https://github.com/apify/crawlee/blob/master/packages/core/src/session_pool/session.ts#L23) SessionState Re-exports [SessionState](/js/api/core/interface/SessionState) ### [](#SitemapRequestList) [](https://github.com/apify/crawlee/blob/master/packages/core/src/storages/sitemap_request_list.ts#L110) SitemapRequestList Re-exports [SitemapRequestList](/js/api/core/class/SitemapRequestList) ### [](#SitemapRequestListOptions) [](https://github.com/apify/crawlee/blob/master/packages/core/src/storages/sitemap_request_list.ts#L56) SitemapRequestListOptions Re-exports [SitemapRequestListOptions](/js/api/core/interface/SitemapRequestListOptions) ### [](#SnapshotResult) [](https://github.com/apify/crawlee/blob/master/packages/core/src/crawlers/error_snapshotter.ts#L16) SnapshotResult Re-exports [SnapshotResult](/js/api/core/interface/SnapshotResult) ### [](#Snapshotter) [](https://github.com/apify/crawlee/blob/master/packages/core/src/autoscaling/snapshotter.ts#L117) Snapshotter Re-exports [Snapshotter](/js/api/core/class/Snapshotter) ### [](#SnapshotterOptions) [](https://github.com/apify/crawlee/blob/master/packages/core/src/autoscaling/snapshotter.ts#L18) SnapshotterOptions Re-exports [SnapshotterOptions](/js/api/core/interface/SnapshotterOptions) ### [](#Source) [](https://github.com/apify/crawlee/blob/master/packages/core/src/request.ts#L539) Source Re-exports [Source](/js/api/core#Source) ### [](#StatisticPersistedState) [](https://github.com/apify/crawlee/blob/master/packages/core/src/crawlers/statistics.ts#L479) StatisticPersistedState Re-exports [StatisticPersistedState](/js/api/core/interface/StatisticPersistedState) ### [](#Statistics) [](https://github.com/apify/crawlee/blob/master/packages/core/src/crawlers/statistics.ts#L64) Statistics Re-exports [Statistics](/js/api/core/class/Statistics) ### [](#StatisticsOptions) [](https://github.com/apify/crawlee/blob/master/packages/core/src/crawlers/statistics.ts#L433) StatisticsOptions Re-exports [StatisticsOptions](/js/api/core/interface/StatisticsOptions) ### [](#StatisticState) [](https://github.com/apify/crawlee/blob/master/packages/core/src/crawlers/statistics.ts#L493) StatisticState Re-exports [StatisticState](/js/api/core/interface/StatisticState) ### [](#StatusMessageCallback) [](https://github.com/apify/crawlee/blob/master/packages/basic-crawler/src/internals/basic-crawler.ts#L125) StatusMessageCallback Re-exports [StatusMessageCallback](/js/api/basic-crawler#StatusMessageCallback) ### [](#StatusMessageCallbackParams) [](https://github.com/apify/crawlee/blob/master/packages/basic-crawler/src/internals/basic-crawler.ts#L115) StatusMessageCallbackParams Re-exports [StatusMessageCallbackParams](/js/api/basic-crawler/interface/StatusMessageCallbackParams) ### [](#StorageClient) [](https://github.com/apify/crawlee/blob/master/packages/core/src/index.ts#L18) StorageClient Re-exports [StorageClient](/js/api/core/interface/StorageClient) ### [](#StorageManagerOptions) [](https://github.com/apify/crawlee/blob/master/packages/core/src/storages/storage_manager.ts#L156) StorageManagerOptions Re-exports [StorageManagerOptions](/js/api/core/interface/StorageManagerOptions) ### [](#StreamingHttpResponse) [](https://github.com/apify/crawlee/blob/master/packages/core/src/http_clients/base-http-client.ts#L162) StreamingHttpResponse Re-exports [StreamingHttpResponse](/js/api/core/interface/StreamingHttpResponse) ### [](#SystemInfo) [](https://github.com/apify/crawlee/blob/master/packages/core/src/autoscaling/system_status.ts#L10) SystemInfo Re-exports [SystemInfo](/js/api/core/interface/SystemInfo) ### [](#SystemStatus) [](https://github.com/apify/crawlee/blob/master/packages/core/src/autoscaling/system_status.ts#L120) SystemStatus Re-exports [SystemStatus](/js/api/core/class/SystemStatus) ### [](#SystemStatusOptions) [](https://github.com/apify/crawlee/blob/master/packages/core/src/autoscaling/system_status.ts#L35) SystemStatusOptions Re-exports [SystemStatusOptions](/js/api/core/interface/SystemStatusOptions) ### [](#TieredProxy) [](https://github.com/apify/crawlee/blob/master/packages/core/src/proxy_configuration.ts#L42) TieredProxy Re-exports [TieredProxy](/js/api/core/interface/TieredProxy) ### [](#tryAbsoluteURL) [](https://github.com/apify/crawlee/blob/master/packages/core/src/enqueue_links/shared.ts#L10) tryAbsoluteURL Re-exports [tryAbsoluteURL](/js/api/core/function/tryAbsoluteURL) ### [](#UrlPatternObject) [](https://github.com/apify/crawlee/blob/master/packages/core/src/enqueue_links/shared.ts#L22) UrlPatternObject Re-exports [UrlPatternObject](/js/api/core#UrlPatternObject) ### [](#useState) [](https://github.com/apify/crawlee/blob/master/packages/core/src/storages/utils.ts#L87) useState Re-exports [useState](/js/api/core/function/useState) ### [](#UseStateOptions) [](https://github.com/apify/crawlee/blob/master/packages/core/src/storages/utils.ts#L69) UseStateOptions Re-exports [UseStateOptions](/js/api/core/interface/UseStateOptions) ### [](#withCheckedStorageAccess) [](https://github.com/apify/crawlee/blob/master/packages/core/src/storages/access_checking.ts#L18) withCheckedStorageAccess Re-exports [withCheckedStorageAccess](/js/api/core/function/withCheckedStorageAccess) ### [](#PlaywrightGotoOptions) [](https://github.com/apify/crawlee/blob/master/packages/playwright-crawler/src/internals/playwright-crawler.ts#L26) PlaywrightGotoOptions PlaywrightGotoOptions: Parameters\[1\] ### [](#RenderingType) [](https://github.com/apify/crawlee/blob/master/packages/playwright-crawler/src/internals/utils/rendering-type-prediction.ts#L5) RenderingType RenderingType: clientOnly | static **Page Options** Hide Inherited * [AddRequestsBatchedOptions](#AddRequestsBatchedOptions) * [AddRequestsBatchedResult](#AddRequestsBatchedResult) * [AutoscaledPool](#AutoscaledPool) * [AutoscaledPoolOptions](#AutoscaledPoolOptions) * [BaseHttpClient](#BaseHttpClient) * [BaseHttpResponseData](#BaseHttpResponseData) * [BASIC\_CRAWLER\_TIMEOUT\_BUFFER\_SECS](#BASIC_CRAWLER_TIMEOUT_BUFFER_SECS) * [BasicCrawler](#BasicCrawler) * [BasicCrawlerOptions](#BasicCrawlerOptions) * [BasicCrawlingContext](#BasicCrawlingContext) * [BLOCKED\_STATUS\_CODES](#BLOCKED_STATUS_CODES) * [BrowserCrawler](#BrowserCrawler) * [BrowserCrawlerOptions](#BrowserCrawlerOptions) * [BrowserCrawlingContext](#BrowserCrawlingContext) * [BrowserErrorHandler](#BrowserErrorHandler) * [BrowserHook](#BrowserHook) * [BrowserLaunchContext](#BrowserLaunchContext) * [BrowserRequestHandler](#BrowserRequestHandler) * [checkStorageAccess](#checkStorageAccess) * [ClientInfo](#ClientInfo) * [Configuration](#Configuration) * [ConfigurationOptions](#ConfigurationOptions) * [Cookie](#Cookie) * [CrawlerAddRequestsOptions](#CrawlerAddRequestsOptions) * [CrawlerAddRequestsResult](#CrawlerAddRequestsResult) * [CrawlerExperiments](#CrawlerExperiments) * [CrawlerRunOptions](#CrawlerRunOptions) * [CrawlingContext](#CrawlingContext) * [createBasicRouter](#createBasicRouter) * [CreateContextOptions](#CreateContextOptions) * [CreateSession](#CreateSession) * [CriticalError](#CriticalError) * [Dataset](#Dataset) * [DatasetConsumer](#DatasetConsumer) * [DatasetContent](#DatasetContent) * [DatasetDataOptions](#DatasetDataOptions) * [DatasetExportOptions](#DatasetExportOptions) * [DatasetExportToOptions](#DatasetExportToOptions) * [DatasetIteratorOptions](#DatasetIteratorOptions) * [DatasetMapper](#DatasetMapper) * [DatasetOptions](#DatasetOptions) * [DatasetReducer](#DatasetReducer) * [enqueueLinks](#enqueueLinks) * [EnqueueLinksOptions](#EnqueueLinksOptions) * [EnqueueStrategy](#EnqueueStrategy) * [ErrnoException](#ErrnoException) * [ErrorHandler](#ErrorHandler) * [ErrorSnapshotter](#ErrorSnapshotter) * [ErrorTracker](#ErrorTracker) * [ErrorTrackerOptions](#ErrorTrackerOptions) * [EventManager](#EventManager) * [EventType](#EventType) * [EventTypeName](#EventTypeName) * [filterRequestsByPatterns](#filterRequestsByPatterns) * [FinalStatistics](#FinalStatistics) * [GetUserDataFromRequest](#GetUserDataFromRequest) * [GlobInput](#GlobInput) * [GlobObject](#GlobObject) * [GotScrapingHttpClient](#GotScrapingHttpClient) * [HttpRequest](#HttpRequest) * [HttpRequestOptions](#HttpRequestOptions) * [HttpResponse](#HttpResponse) * [IRequestList](#IRequestList) * [IStorage](#IStorage) * [KeyConsumer](#KeyConsumer) * [KeyValueStore](#KeyValueStore) * [KeyValueStoreIteratorOptions](#KeyValueStoreIteratorOptions) * [KeyValueStoreOptions](#KeyValueStoreOptions) * [LoadedRequest](#LoadedRequest) * [LocalEventManager](#LocalEventManager) * [log](#log) * [Log](#Log) * [Logger](#Logger) * [LoggerJson](#LoggerJson) * [LoggerOptions](#LoggerOptions) * [LoggerText](#LoggerText) * [LogLevel](#LogLevel) * [MAX\_POOL\_SIZE](#MAX_POOL_SIZE) * [NonRetryableError](#NonRetryableError) * [PERSIST\_STATE\_KEY](#PERSIST_STATE_KEY) * [PersistenceOptions](#PersistenceOptions) * [PlaywrightDirectNavigationOptions](#PlaywrightDirectNavigationOptions) * [processHttpRequestOptions](#processHttpRequestOptions) * [ProxyConfiguration](#ProxyConfiguration) * [ProxyConfigurationFunction](#ProxyConfigurationFunction) * [ProxyConfigurationOptions](#ProxyConfigurationOptions) * [ProxyInfo](#ProxyInfo) * [PseudoUrl](#PseudoUrl) * [PseudoUrlInput](#PseudoUrlInput) * [PseudoUrlObject](#PseudoUrlObject) * [purgeDefaultStorages](#purgeDefaultStorages) * [PushErrorMessageOptions](#PushErrorMessageOptions) * [QueueOperationInfo](#QueueOperationInfo) * [RecordOptions](#RecordOptions) * [RedirectHandler](#RedirectHandler) * [RegExpInput](#RegExpInput) * [RegExpObject](#RegExpObject) * [Request](#Request) * [RequestHandler](#RequestHandler) * [RequestHandlerResult](#RequestHandlerResult) * [RequestList](#RequestList) * [RequestListOptions](#RequestListOptions) * [RequestListSourcesFunction](#RequestListSourcesFunction) * [RequestListState](#RequestListState) * [RequestOptions](#RequestOptions) * [RequestProvider](#RequestProvider) * [RequestProviderOptions](#RequestProviderOptions) * [RequestQueue](#RequestQueue) * [RequestQueueOperationOptions](#RequestQueueOperationOptions) * [RequestQueueOptions](#RequestQueueOptions) * [RequestQueueV1](#RequestQueueV1) * [RequestQueueV2](#RequestQueueV2) * [RequestState](#RequestState) * [RequestTransform](#RequestTransform) * [ResponseLike](#ResponseLike) * [ResponseTypes](#ResponseTypes) * [RestrictedCrawlingContext](#RestrictedCrawlingContext) * [RetryRequestError](#RetryRequestError) * [Router](#Router) * [RouterHandler](#RouterHandler) * [RouterRoutes](#RouterRoutes) * [Session](#Session) * [SessionError](#SessionError) * [SessionOptions](#SessionOptions) * [SessionPool](#SessionPool) * [SessionPoolOptions](#SessionPoolOptions) * [SessionState](#SessionState) * [SitemapRequestList](#SitemapRequestList) * [SitemapRequestListOptions](#SitemapRequestListOptions) * [SnapshotResult](#SnapshotResult) * [Snapshotter](#Snapshotter) * [SnapshotterOptions](#SnapshotterOptions) * [Source](#Source) * [StatisticPersistedState](#StatisticPersistedState) * [Statistics](#Statistics) * [StatisticsOptions](#StatisticsOptions) * [StatisticState](#StatisticState) * [StatusMessageCallback](#StatusMessageCallback) * [StatusMessageCallbackParams](#StatusMessageCallbackParams) * [StorageClient](#StorageClient) * [StorageManagerOptions](#StorageManagerOptions) * [StreamingHttpResponse](#StreamingHttpResponse) * [SystemInfo](#SystemInfo) * [SystemStatus](#SystemStatus) * [SystemStatusOptions](#SystemStatusOptions) * [TieredProxy](#TieredProxy) * [tryAbsoluteURL](#tryAbsoluteURL) * [UrlPatternObject](#UrlPatternObject) * [useState](#useState) * [UseStateOptions](#UseStateOptions) * [withCheckedStorageAccess](#withCheckedStorageAccess) * [PlaywrightGotoOptions](#PlaywrightGotoOptions) * [RenderingType](#RenderingType) --- # playwrightClickElements | API | Crawlee for JavaScript · Build reliable crawlers. Fast. [Skip to main content](#__docusaurus_skipToContent_fallback) Version: 3.13 On this page Index[](#Index) ---------------- ### References * [enqueueLinksByClickingElements](/js/api/playwright-crawler/namespace/playwrightClickElements#enqueueLinksByClickingElements) ### Interfaces * [EnqueueLinksByClickingElementsOptions](/js/api/playwright-crawler/namespace/playwrightClickElements#EnqueueLinksByClickingElementsOptions) References[](#References) -------------------------- ### [](#enqueueLinksByClickingElements) [](https://github.com/apify/crawlee/blob/master/packages/playwright-crawler/src/internals/enqueue-links/click-elements.ts#L213) enqueueLinksByClickingElements Re-exports [enqueueLinksByClickingElements](/js/api/playwright-crawler/namespace/playwrightUtils#enqueueLinksByClickingElements) Interfaces[](#Interfaces) -------------------------- ### [](#EnqueueLinksByClickingElementsOptions) [](https://github.com/apify/crawlee/blob/master/packages/playwright-crawler/src/internals/enqueue-links/click-elements.ts#L29) EnqueueLinksByClickingElementsOptions EnqueueLinksByClickingElementsOptions: ### [](#clickOptions) [](https://github.com/apify/crawlee/blob/master/packages/playwright-crawler/src/internals/enqueue-links/click-elements.ts#L55) optionalclickOptions clickOptions?: { button?: left | right | middle; clickCount?: number; delay?: number; force?: boolean; modifiers?: (Alt | Control | ControlOrMeta | Meta | Shift)\[\]; noWaitAfter?: boolean; position?: { x: number; y: number }; strict?: boolean; timeout?: number; trial?: boolean } Click options for use in Playwright click handler. * * * #### Type declaration * ##### externaloptionalbutton?: left | right | middle Defaults to `left`. * ##### externaloptionalclickCount?: number defaults to 1. See \[UIEvent.detail\]. * ##### externaloptionaldelay?: number Time to wait between `mousedown` and `mouseup` in milliseconds. Defaults to 0. * ##### externaloptionalforce?: boolean Whether to bypass the [actionability](https://playwright.dev/docs/actionability) checks. Defaults to `false`. * ##### externaloptionalmodifiers?: (Alt | Control | ControlOrMeta | Meta | Shift)\[\] Modifier keys to press. Ensures that only these modifiers are pressed during the operation, and then restores current modifiers back. If not specified, currently pressed modifiers are used. "ControlOrMeta" resolves to "Control" on Windows and Linux and to "Meta" on macOS. * ##### externaloptionalnoWaitAfter?: boolean Actions that initiate navigations are waiting for these navigations to happen and for pages to start loading. You can opt out of waiting via setting this flag. You would only need this option in the exceptional cases such as navigating to inaccessible pages. Defaults to `false`. **@deprecated** This option will default to `true` in the future. * ##### externaloptionalposition?: { x: number; y: number } A point to use relative to the top-left corner of element padding box. If not specified, uses some visible point of the element. * ##### externalx: number * ##### externaly: number * ##### externaloptionalstrict?: boolean When true, the call requires selector to resolve to a single element. If given selector resolves to more than one element, the call throws an exception. * ##### externaloptionaltimeout?: number Maximum time in milliseconds. Defaults to `0` - no timeout. The default value can be changed via `actionTimeout` option in the config, or by using the [browserContext.setDefaultTimeout(timeout)](https://playwright.dev/docs/api/class-browsercontext#browser-context-set-default-timeout) or [page.setDefaultTimeout(timeout)](https://playwright.dev/docs/api/class-page#page-set-default-timeout) methods. * ##### externaloptionaltrial?: boolean When set, this method only performs the [actionability](https://playwright.dev/docs/actionability) checks and skips the action. Defaults to `false`. Useful to wait until the element is ready for the action without performing it. Note that keyboard `modifiers` will be pressed regardless of `trial` to allow testing elements which are only visible when those keys are pressed. ### [](#forefront) [](https://github.com/apify/crawlee/blob/master/packages/playwright-crawler/src/internals/enqueue-links/click-elements.ts#L163) optionalforefront forefront?: boolean = false If set to `true`: * while adding the request to the queue: the request will be added to the foremost position in the queue. * while reclaiming the request: the request will be placed to the beginning of the queue, so that it's returned in the next call to [RequestQueue.fetchNextRequest](/js/api/core/class/RequestQueue#fetchNextRequest) . By default, it's put to the end of the queue. ### [](#globs) [](https://github.com/apify/crawlee/blob/master/packages/playwright-crawler/src/internals/enqueue-links/click-elements.ts#L71) optionalglobs globs?: [GlobInput](/js/api/core#GlobInput) \[\] An array of glob pattern strings or plain objects containing glob pattern strings matching the URLs to be enqueued. The plain objects must include at least the `glob` property, which holds the glob pattern string. All remaining keys will be used as request options for the corresponding enqueued [Request](/js/api/core/class/Request) objects. The matching is always case-insensitive. If you need case-sensitive matching, use `regexps` property directly. If `globs` is an empty array or `undefined`, then the function enqueues all the intercepted navigation requests produced by the page after clicking on elements matching the provided CSS selector. ### [](#label) [](https://github.com/apify/crawlee/blob/master/packages/playwright-crawler/src/internals/enqueue-links/click-elements.ts#L50) optionallabel label?: string Sets [Request.label](/js/api/core/class/Request#label) for newly enqueued requests. ### [](#maxWaitForPageIdleSecs) [](https://github.com/apify/crawlee/blob/master/packages/playwright-crawler/src/internals/enqueue-links/click-elements.ts#L153) optionalmaxWaitForPageIdleSecs maxWaitForPageIdleSecs?: number = 5 This is the maximum period for which the function will keep tracking events, even if more events keep coming. Its purpose is to prevent a deadlock in the page by periodic events, often unrelated to the clicking itself. See `waitForPageIdleSecs` above for an explanation. ### [](#page) [](https://github.com/apify/crawlee/blob/master/packages/playwright-crawler/src/internals/enqueue-links/click-elements.ts#L33) page page: Page Playwright [`Page`](https://playwright.dev/docs/api/class-page) object. ### [](#pseudoUrls) [](https://github.com/apify/crawlee/blob/master/packages/playwright-crawler/src/internals/enqueue-links/click-elements.ts#L105) optionalpseudoUrls pseudoUrls?: [PseudoUrlInput](/js/api/core#PseudoUrlInput) \[\] _NOTE:_ In future versions of SDK the options will be removed. Please use `globs` or `regexps` instead. An array of [PseudoUrl](/js/api/core/class/PseudoUrl) strings or plain objects containing [PseudoUrl](/js/api/core/class/PseudoUrl) strings matching the URLs to be enqueued. The plain objects must include at least the `purl` property, which holds the pseudo-URL pattern string. All remaining keys will be used as request options for the corresponding enqueued [Request](/js/api/core/class/Request) objects. With a pseudo-URL string, the matching is always case-insensitive. If you need case-sensitive matching, use `regexps` property directly. If `pseudoUrls` is an empty array or `undefined`, then the function enqueues all the intercepted navigation requests produced by the page after clicking on elements matching the provided CSS selector. **@deprecated** prefer using `globs` or `regexps` instead ### [](#regexps) [](https://github.com/apify/crawlee/blob/master/packages/playwright-crawler/src/internals/enqueue-links/click-elements.ts#L84) optionalregexps regexps?: [RegExpInput](/js/api/core#RegExpInput) \[\] An array of regular expressions or plain objects containing regular expressions matching the URLs to be enqueued. The plain objects must include at least the `regexp` property, which holds the regular expression. All remaining keys will be used as request options for the corresponding enqueued [Request](/js/api/core/class/Request) objects. If `regexps` is an empty array or `undefined`, then the function enqueues all the intercepted navigation requests produced by the page after clicking on elements matching the provided CSS selector. ### [](#requestQueue) [](https://github.com/apify/crawlee/blob/master/packages/playwright-crawler/src/internals/enqueue-links/click-elements.ts#L38) requestQueue requestQueue: [RequestProvider](/js/api/core/class/RequestProvider) A request queue to which the URLs will be enqueued. ### [](#selector) [](https://github.com/apify/crawlee/blob/master/packages/playwright-crawler/src/internals/enqueue-links/click-elements.ts#L44) selector selector: string A CSS selector matching elements to be clicked on. Unlike in enqueueLinks, there is no default value. This is to prevent suboptimal use of this function by using it too broadly. ### [](#skipNavigation) [](https://github.com/apify/crawlee/blob/master/packages/playwright-crawler/src/internals/enqueue-links/click-elements.ts#L169) optionalskipNavigation skipNavigation?: boolean = false If set to `true`, tells the crawler to skip navigation and process the request directly. ### [](#transformRequestFunction) [](https://github.com/apify/crawlee/blob/master/packages/playwright-crawler/src/internals/enqueue-links/click-elements.ts#L128) optionaltransformRequestFunction transformRequestFunction?: [RequestTransform](/js/api/core/interface/RequestTransform) Just before a new [Request](/js/api/core/class/Request) is constructed and enqueued to the [RequestQueue](/js/api/core/class/RequestQueue) , this function can be used to remove it or modify its contents such as `userData`, `payload` or, most importantly `uniqueKey`. This is useful when you need to enqueue multiple `Requests` to the queue that share the same URL, but differ in methods or payloads, or to dynamically update or create `userData`. For example: by adding `useExtendedUniqueKey: true` to the `request` object, `uniqueKey` will be computed from a combination of `url`, `method` and `payload` which enables crawling of websites that navigate using form submits (POST requests). **Example:** { transformRequestFunction: (request) => { request.userData.foo = 'bar'; request.useExtendedUniqueKey = true; return request; }} ### [](#userData) [](https://github.com/apify/crawlee/blob/master/packages/playwright-crawler/src/internals/enqueue-links/click-elements.ts#L47) optionaluserData userData?: Dictionary Sets [Request.userData](/js/api/core/class/Request#userData) for newly enqueued requests. ### [](#waitForPageIdleSecs) [](https://github.com/apify/crawlee/blob/master/packages/playwright-crawler/src/internals/enqueue-links/click-elements.ts#L145) optionalwaitForPageIdleSecs waitForPageIdleSecs?: number = 1 Clicking in the page triggers various asynchronous operations that lead to new URLs being shown by the browser. It could be a simple JavaScript redirect or opening of a new tab in the browser. These events often happen only some time after the actual click. Requests typically take milliseconds while new tabs open in hundreds of milliseconds. To be able to capture all those events, the `enqueueLinksByClickingElements()` function repeatedly waits for the `waitForPageIdleSecs`. By repeatedly we mean that whenever a relevant event is triggered, the timer is restarted. As long as new events keep coming, the function will not return, unless the below `maxWaitForPageIdleSecs` timeout is reached. You may want to reduce this for example when you're sure that your clicks do not open new tabs, or increase when you're not getting all the expected URLs. **Page Options** Hide Inherited * [enqueueLinksByClickingElements](#enqueueLinksByClickingElements) * [EnqueueLinksByClickingElementsOptions](#EnqueueLinksByClickingElementsOptions) --- # createPlaywrightRouter | API | Crawlee for JavaScript · Build reliable crawlers. Fast. [Skip to main content](#__docusaurus_skipToContent_fallback) Version: 3.13 ### Callable * **createPlaywrightRouter**(routes): [RouterHandler](/js/api/core/interface/RouterHandler) * * * * Creates new [Router](/js/api/core/class/Router) instance that works based on request labels. This instance can then serve as a `requestHandler` of your [PlaywrightCrawler](/js/api/playwright-crawler/class/PlaywrightCrawler) . Defaults to the [PlaywrightCrawlingContext](/js/api/playwright-crawler/interface/PlaywrightCrawlingContext) . > Serves as a shortcut for using `Router.create()`. import { PlaywrightCrawler, createPlaywrightRouter } from 'crawlee';const router = createPlaywrightRouter();router.addHandler('label-a', async (ctx) => { ctx.log.info('...');});router.addDefaultHandler(async (ctx) => { ctx.log.info('...');});const crawler = new PlaywrightCrawler({ requestHandler: router,});await crawler.run(); * * * #### Parameters * ##### optionalroutes: [RouterRoutes](/js/api/core#RouterRoutes) #### Returns [RouterHandler](/js/api/core/interface/RouterHandler) --- # launchPlaywright | API | Crawlee for JavaScript · Build reliable crawlers. Fast. [Skip to main content](#__docusaurus_skipToContent_fallback) Version: 3.13 ### Callable * **launchPlaywright**(launchContext, config): Promise * * * * Launches headless browsers using Playwright pre-configured to work within the Apify platform. The function has the same return value as `browserType.launch()`. See [Playwright documentation](https://playwright.dev/docs/api/class-browsertype) for more details. The `launchPlaywright()` function alters the following Playwright options: * Passes the setting from the `CRAWLEE_HEADLESS` environment variable to the `headless` option, unless it was already defined by the caller or `CRAWLEE_XVFB` environment variable is set to `1`. Note that Apify Actor cloud platform automatically sets `CRAWLEE_HEADLESS=1` to all running actors. * Takes the `proxyUrl` option, validates it and adds it to `launchOptions` in a proper format. The proxy URL must define a port number and have one of the following schemes: `http://`, `https://`, `socks4://` or `socks5://`. If the proxy is HTTP (i.e. has the `http://` scheme) and contains username or password, the `launchPlaywright` functions sets up an anonymous proxy HTTP to make the proxy work with headless Chrome. For more information, read the [blog post about proxy-chain library](https://blog.apify.com/how-to-make-headless-chrome-and-puppeteer-use-a-proxy-server-with-authentication-249a21a79212) . To use this function, you need to have the [Playwright](https://www.npmjs.com/package/playwright) NPM package installed in your project. When running on the Apify Platform, you can achieve that simply by using the `apify/actor-node-playwright-*` base Docker image for your actor - see [Apify Actor documentation](https://docs.apify.com/actor/build#base-images) for details. * * * #### Parameters * ##### optionallaunchContext: [PlaywrightLaunchContext](/js/api/playwright-crawler/interface/PlaywrightLaunchContext) Optional settings passed to `browserType.launch()`. In addition to [Playwright's options](https://playwright.dev/docs/api/class-browsertype?_highlight=launch#browsertypelaunchoptions) the object may contain our own [PlaywrightLaunchContext](/js/api/playwright-crawler/interface/PlaywrightLaunchContext) that enable additional features. * ##### optionalconfig: [Configuration](/js/api/core/class/Configuration) = ... #### Returns Promise Promise that resolves to Playwright's `Browser` instance. --- # AdaptivePlaywrightCrawlerOptions | API | Crawlee for JavaScript · Build reliable crawlers. Fast. [Skip to main content](#__docusaurus_skipToContent_fallback) Version: 3.13 On this page ### Hierarchy * Omit<[PlaywrightCrawlerOptions](/js/api/playwright-crawler/interface/PlaywrightCrawlerOptions) , requestHandler | handlePageFunction\> * _AdaptivePlaywrightCrawlerOptions_ Index[](#Index) ---------------- ### Properties * [autoscaledPoolOptions](#autoscaledPoolOptions) * [browserPoolOptions](#browserPoolOptions) * [errorHandler](#errorHandler) * [experiments](#experiments) * [failedRequestHandler](#failedRequestHandler) * [headless](#headless) * [httpClient](#httpClient) * [ignoreIframes](#ignoreIframes) * [ignoreShadowRoots](#ignoreShadowRoots) * [keepAlive](#keepAlive) * [launchContext](#launchContext) * [maxConcurrency](#maxConcurrency) * [maxRequestRetries](#maxRequestRetries) * [maxRequestsPerCrawl](#maxRequestsPerCrawl) * [maxRequestsPerMinute](#maxRequestsPerMinute) * [maxSessionRotations](#maxSessionRotations) * [minConcurrency](#minConcurrency) * [navigationTimeoutSecs](#navigationTimeoutSecs) * [persistCookiesPerSession](#persistCookiesPerSession) * [postNavigationHooks](#postNavigationHooks) * [preNavigationHooks](#preNavigationHooks) * [proxyConfiguration](#proxyConfiguration) * [renderingTypeDetectionRatio](#renderingTypeDetectionRatio) * [renderingTypePredictor](#renderingTypePredictor) * [requestHandler](#requestHandler) * [requestHandlerTimeoutSecs](#requestHandlerTimeoutSecs) * [requestList](#requestList) * [requestQueue](#requestQueue) * [resultChecker](#resultChecker) * [resultComparator](#resultComparator) * [retryOnBlocked](#retryOnBlocked) * [sameDomainDelaySecs](#sameDomainDelaySecs) * [sessionPoolOptions](#sessionPoolOptions) * [statisticsOptions](#statisticsOptions) * [statusMessageCallback](#statusMessageCallback) * [statusMessageLoggingInterval](#statusMessageLoggingInterval) * [useSessionPool](#useSessionPool) Properties[](#Properties) -------------------------- ### [](#autoscaledPoolOptions) [](https://github.com/apify/crawlee/blob/master/packages/basic-crawler/src/internals/basic-crawler.ts#L271) optionalinheritedautoscaledPoolOptions autoscaledPoolOptions?: [AutoscaledPoolOptions](/js/api/core/interface/AutoscaledPoolOptions) Custom options passed to the underlying [AutoscaledPool](/js/api/core/class/AutoscaledPool) constructor. > _NOTE:_ The [`runTaskFunction`](/js/api/core/interface/AutoscaledPoolOptions#runTaskFunction) > and [`isTaskReadyFunction`](/js/api/core/interface/AutoscaledPoolOptions#isTaskReadyFunction) > options are provided by the crawler and cannot be overridden. However, we can provide a custom implementation of [`isFinishedFunction`](/js/api/core/interface/AutoscaledPoolOptions#isFinishedFunction) > . ### [](#browserPoolOptions) [](https://github.com/apify/crawlee/blob/master/packages/browser-crawler/src/internals/browser-crawler.ts#L190) optionalinheritedbrowserPoolOptions browserPoolOptions?: Partial<[BrowserPoolOptions](/js/api/browser-pool/interface/BrowserPoolOptions) <[BrowserPlugin](/js/api/browser-pool/class/BrowserPlugin) <[CommonLibrary](/js/api/browser-pool/interface/CommonLibrary) , undefined | Dictionary, CommonBrowser, unknown, CommonPage\>\>\> & Partial<[BrowserPoolHooks](/js/api/browser-pool/interface/BrowserPoolHooks) <[BrowserController](/js/api/browser-pool/class/BrowserController) , undefined | LaunchOptions, Browser, undefined | { acceptDownloads?: boolean; baseURL?: string; bypassCSP?: boolean; clientCertificates?: { cert?: ... | ...; certPath?: ... | ...; key?: ... | ...; keyPath?: ... | ...; origin: string; passphrase?: ... | ...; pfx?: ... | ...; pfxPath?: ... | ... }\[\]; colorScheme?: null | light | dark | no-preference; deviceScaleFactor?: number; extraHTTPHeaders?: {}; forcedColors?: null | active | none; geolocation?: { accuracy?: number; latitude: number; longitude: number }; hasTouch?: boolean; httpCredentials?: { origin?: string; password: string; send?: unauthorized | always; username: string }; ignoreHTTPSErrors?: boolean; isMobile?: boolean; javaScriptEnabled?: boolean; locale?: string; logger?: Logger; offline?: boolean; permissions?: string\[\]; proxy?: { bypass?: string; password?: string; server: string; username?: string }; recordHar?: { content?: omit | embed | attach; mode?: full | minimal; omitContent?: boolean; path: string; urlFilter?: string | RegExp }; recordVideo?: { dir: string; size?: { height: ...; width: ... } }; reducedMotion?: null | reduce | no-preference; screen?: { height: number; width: number }; serviceWorkers?: allow | block; storageState?: string | { cookies: { domain: ...; expires: ...; httpOnly: ...; name: ...; path: ...; sameSite: ...; secure: ...; value: ... }\[\]; origins: { localStorage: ...; origin: ... }\[\] }; strictSelectors?: boolean; timezoneId?: string; userAgent?: string; videoSize?: { height: number; width: number }; videosPath?: string; viewport?: null | { height: number; width: number } }, Page\>, [LaunchContext](/js/api/browser-pool/class/LaunchContext) , undefined | LaunchOptions, Browser, undefined | { acceptDownloads?: boolean; baseURL?: string; bypassCSP?: boolean; clientCertificates?: { cert?: ... | ...; certPath?: ... | ...; key?: ... | ...; keyPath?: ... | ...; origin: string; passphrase?: ... | ...; pfx?: ... | ...; pfxPath?: ... | ... }\[\]; colorScheme?: null | light | dark | no-preference; deviceScaleFactor?: number; extraHTTPHeaders?: {}; forcedColors?: null | active | none; geolocation?: { accuracy?: number; latitude: number; longitude: number }; hasTouch?: boolean; httpCredentials?: { origin?: string; password: string; send?: unauthorized | always; username: string }; ignoreHTTPSErrors?: boolean; isMobile?: boolean; javaScriptEnabled?: boolean; locale?: string; logger?: Logger; offline?: boolean; permissions?: string\[\]; proxy?: { bypass?: string; password?: string; server: string; username?: string }; recordHar?: { content?: omit | embed | attach; mode?: full | minimal; omitContent?: boolean; path: string; urlFilter?: string | RegExp }; recordVideo?: { dir: string; size?: { height: ...; width: ... } }; reducedMotion?: null | reduce | no-preference; screen?: { height: number; width: number }; serviceWorkers?: allow | block; storageState?: string | { cookies: { domain: ...; expires: ...; httpOnly: ...; name: ...; path: ...; sameSite: ...; secure: ...; value: ... }\[\]; origins: { localStorage: ...; origin: ... }\[\] }; strictSelectors?: boolean; timezoneId?: string; userAgent?: string; videoSize?: { height: number; width: number }; videosPath?: string; viewport?: null | { height: number; width: number } }, Page\>, Page\>\> Custom options passed to the underlying [BrowserPool](/js/api/browser-pool/class/BrowserPool) constructor. We can tweak those to fine-tune browser management. ### [](#errorHandler) [](https://github.com/apify/crawlee/blob/master/packages/browser-crawler/src/internals/browser-crawler.ts#L159) optionalinheritederrorHandler errorHandler?: [BrowserErrorHandler](/js/api/browser-crawler#BrowserErrorHandler) <[PlaywrightCrawlingContext](/js/api/playwright-crawler/interface/PlaywrightCrawlingContext) \> User-provided function that allows modifying the request object before it gets retried by the crawler. It's executed before each retry for the requests that failed less than [`maxRequestRetries`](/js/api/browser-crawler/interface/BrowserCrawlerOptions#maxRequestRetries) times. The function receives the [BrowserCrawlingContext](/js/api/browser-crawler/interface/BrowserCrawlingContext) (actual context will be enhanced with the crawler specific properties) as the first argument, where the [`request`](/js/api/browser-crawler/interface/BrowserCrawlingContext#request) corresponds to the request to be retried. Second argument is the `Error` instance that represents the last error thrown during processing of the request. ### [](#experiments) [](https://github.com/apify/crawlee/blob/master/packages/basic-crawler/src/internals/basic-crawler.ts#L351) optionalinheritedexperiments experiments?: [CrawlerExperiments](/js/api/basic-crawler/interface/CrawlerExperiments) Enables experimental features of Crawlee, which can alter the behavior of the crawler. WARNING: these options are not guaranteed to be stable and may change or be removed at any time. ### [](#failedRequestHandler) [](https://github.com/apify/crawlee/blob/master/packages/browser-crawler/src/internals/browser-crawler.ts#L170) optionalinheritedfailedRequestHandler failedRequestHandler?: [BrowserErrorHandler](/js/api/browser-crawler#BrowserErrorHandler) <[PlaywrightCrawlingContext](/js/api/playwright-crawler/interface/PlaywrightCrawlingContext) \> A function to handle requests that failed more than `option.maxRequestRetries` times. The function receives the [BrowserCrawlingContext](/js/api/browser-crawler/interface/BrowserCrawlingContext) (actual context will be enhanced with the crawler specific properties) as the first argument, where the [`request`](/js/api/browser-crawler/interface/BrowserCrawlingContext#request) corresponds to the failed request. Second argument is the `Error` instance that represents the last error thrown during processing of the request. ### [](#headless) [](https://github.com/apify/crawlee/blob/master/packages/browser-crawler/src/internals/browser-crawler.ts#L256) optionalinheritedheadless headless?: boolean | new | old Whether to run browser in headless mode. Defaults to `true`. Can be also set via [Configuration](/js/api/core/class/Configuration) . ### [](#httpClient) [](https://github.com/apify/crawlee/blob/master/packages/basic-crawler/src/internals/basic-crawler.ts#L363) optionalinheritedhttpClient httpClient?: [BaseHttpClient](/js/api/core/interface/BaseHttpClient) HTTP client implementation for the `sendRequest` context helper and for plain HTTP crawling. Defaults to a new instance of [GotScrapingHttpClient](/js/api/core/class/GotScrapingHttpClient) ### [](#ignoreIframes) [](https://github.com/apify/crawlee/blob/master/packages/browser-crawler/src/internals/browser-crawler.ts#L268) optionalinheritedignoreIframes ignoreIframes?: boolean Whether to ignore `iframes` when processing the page content via `parseWithCheerio` helper. By default, `iframes` are expanded automatically. Use this option to disable this behavior. ### [](#ignoreShadowRoots) [](https://github.com/apify/crawlee/blob/master/packages/browser-crawler/src/internals/browser-crawler.ts#L262) optionalinheritedignoreShadowRoots ignoreShadowRoots?: boolean Whether to ignore custom elements (and their #shadow-roots) when processing the page content via `parseWithCheerio` helper. By default, they are expanded automatically. Use this option to disable this behavior. ### [](#keepAlive) [](https://github.com/apify/crawlee/blob/master/packages/basic-crawler/src/internals/basic-crawler.ts#L299) optionalinheritedkeepAlive keepAlive?: boolean Allows to keep the crawler alive even if the [RequestQueue](/js/api/core/class/RequestQueue) gets empty. By default, the `crawler.run()` will resolve once the queue is empty. With `keepAlive: true` it will keep running, waiting for more requests to come. Use `crawler.stop()` to exit the crawler gracefully, or `crawler.teardown()` to stop it immediately. ### [](#launchContext) [](https://github.com/apify/crawlee/blob/master/packages/playwright-crawler/src/internals/playwright-crawler.ts#L33) optionalinheritedlaunchContext launchContext?: [PlaywrightLaunchContext](/js/api/playwright-crawler/interface/PlaywrightLaunchContext) The same options as used by [launchPlaywright](/js/api/playwright-crawler/function/launchPlaywright) . ### [](#maxConcurrency) [](https://github.com/apify/crawlee/blob/master/packages/basic-crawler/src/internals/basic-crawler.ts#L285) optionalinheritedmaxConcurrency maxConcurrency?: number Sets the maximum concurrency (parallelism) for the crawl. Shortcut for the AutoscaledPool [`maxConcurrency`](/js/api/core/interface/AutoscaledPoolOptions#maxConcurrency) option. ### [](#maxRequestRetries) [](https://github.com/apify/crawlee/blob/master/packages/basic-crawler/src/internals/basic-crawler.ts#L240) optionalinheritedmaxRequestRetries maxRequestRetries?: number = 3 Indicates how many times the request is retried if [`requestHandler`](/js/api/basic-crawler/interface/BasicCrawlerOptions#requestHandler) fails. ### [](#maxRequestsPerCrawl) [](https://github.com/apify/crawlee/blob/master/packages/basic-crawler/src/internals/basic-crawler.ts#L262) optionalinheritedmaxRequestsPerCrawl maxRequestsPerCrawl?: number Maximum number of pages that the crawler will open. The crawl will stop when this limit is reached. This value should always be set in order to prevent infinite loops in misconfigured crawlers. > _NOTE:_ In cases of parallel crawling, the actual number of pages visited might be slightly higher than this value. ### [](#maxRequestsPerMinute) [](https://github.com/apify/crawlee/blob/master/packages/basic-crawler/src/internals/basic-crawler.ts#L292) optionalinheritedmaxRequestsPerMinute maxRequestsPerMinute?: number The maximum number of requests per minute the crawler should run. By default, this is set to `Infinity`, but we can pass any positive, non-zero integer. Shortcut for the AutoscaledPool [`maxTasksPerMinute`](/js/api/core/interface/AutoscaledPoolOptions#maxTasksPerMinute) option. ### [](#maxSessionRotations) [](https://github.com/apify/crawlee/blob/master/packages/basic-crawler/src/internals/basic-crawler.ts#L255) optionalinheritedmaxSessionRotations maxSessionRotations?: number = 10 Maximum number of session rotations per request. The crawler will automatically rotate the session in case of a proxy error or if it gets blocked by the website. The session rotations are not counted towards the [`maxRequestRetries`](/js/api/basic-crawler/interface/BasicCrawlerOptions#maxRequestRetries) limit. ### [](#minConcurrency) [](https://github.com/apify/crawlee/blob/master/packages/basic-crawler/src/internals/basic-crawler.ts#L279) optionalinheritedminConcurrency minConcurrency?: number Sets the minimum concurrency (parallelism) for the crawl. Shortcut for the AutoscaledPool [`minConcurrency`](/js/api/core/interface/AutoscaledPoolOptions#minConcurrency) option. > _WARNING:_ If we set this value too high with respect to the available system memory and CPU, our crawler will run extremely slow or crash. If not sure, it's better to keep the default value and the concurrency will scale up automatically. ### [](#navigationTimeoutSecs) [](https://github.com/apify/crawlee/blob/master/packages/browser-crawler/src/internals/browser-crawler.ts#L244) optionalinheritednavigationTimeoutSecs navigationTimeoutSecs?: number Timeout in which page navigation needs to finish, in seconds. ### [](#persistCookiesPerSession) [](https://github.com/apify/crawlee/blob/master/packages/browser-crawler/src/internals/browser-crawler.ts#L250) optionalinheritedpersistCookiesPerSession persistCookiesPerSession?: boolean Defines whether the cookies should be persisted for sessions. This can only be used when `useSessionPool` is set to `true`. ### [](#postNavigationHooks) [](https://github.com/apify/crawlee/blob/master/packages/playwright-crawler/src/internals/playwright-crawler.ts#L124) optionalinheritedpostNavigationHooks postNavigationHooks?: [PlaywrightHook](/js/api/playwright-crawler/interface/PlaywrightHook) \[\] Async functions that are sequentially evaluated after the navigation. Good for checking if the navigation was successful. The function accepts `crawlingContext` as the only parameter. Example: postNavigationHooks: [ async (crawlingContext) => { const { page } = crawlingContext; if (hasCaptcha(page)) { await solveCaptcha (page); } },] ### [](#preNavigationHooks) [](https://github.com/apify/crawlee/blob/master/packages/playwright-crawler/src/internals/playwright-crawler.ts#L107) optionalinheritedpreNavigationHooks preNavigationHooks?: [PlaywrightHook](/js/api/playwright-crawler/interface/PlaywrightHook) \[\] Async functions that are sequentially evaluated before the navigation. Good for setting additional cookies or browser properties before navigation. The function accepts two parameters, `crawlingContext` and `gotoOptions`, which are passed to the `page.goto()` function the crawler calls to navigate. Example: preNavigationHooks: [ async (crawlingContext, gotoOptions) => { const { page } = crawlingContext; await page.evaluate((attr) => { window.foo = attr; }, 'bar'); },] Modyfing `pageOptions` is supported only in Playwright incognito. See [PrePageCreateHook](/js/api/browser-pool#PrePageCreateHook) ### [](#proxyConfiguration) [](https://github.com/apify/crawlee/blob/master/packages/browser-crawler/src/internals/browser-crawler.ts#L197) optionalinheritedproxyConfiguration proxyConfiguration?: [ProxyConfiguration](/js/api/core/class/ProxyConfiguration) If set, the crawler will be configured for all connections to use the Proxy URLs provided and rotated according to the configuration. ### [](#renderingTypeDetectionRatio) [](https://github.com/apify/crawlee/blob/master/packages/playwright-crawler/src/internals/adaptive-playwright-crawler.ts#L146) optionalrenderingTypeDetectionRatio renderingTypeDetectionRatio?: number Specifies the frequency of rendering type detection checks - 0.1 means roughly 10% of requests. Defaults to 0.1 (so 10%). ### [](#renderingTypePredictor) [](https://github.com/apify/crawlee/blob/master/packages/playwright-crawler/src/internals/adaptive-playwright-crawler.ts#L166) optionalrenderingTypePredictor renderingTypePredictor?: Pick A custom rendering type predictor ### [](#requestHandler) [](https://github.com/apify/crawlee/blob/master/packages/playwright-crawler/src/internals/adaptive-playwright-crawler.ts#L140) optionalrequestHandler requestHandler?: (crawlingContext) => Awaitable Function that is called to process each request. The function receives the AdaptivePlaywrightCrawlingContext as an argument, and it must refrain from calling code with side effects, other than the methods of the crawling context. Any other side effects may be invoked repeatedly by the crawler, which can lead to inconsistent results. The function must return a promise, which is then awaited by the crawler. If the function throws an exception, the crawler will try to re-crawl the request later, up to `option.maxRequestRetries` times. * * * #### Type declaration * * (crawlingContext): Awaitable * #### Parameters * ##### crawlingContext: { request: [LoadedRequest](/js/api/core#LoadedRequest) <[Request](/js/api/core/class/Request) \> } & Omit<[AdaptivePlaywrightCrawlerContext](/js/api/playwright-crawler/interface/AdaptivePlaywrightCrawlerContext) , request\> #### Returns Awaitable ### [](#requestHandlerTimeoutSecs) [](https://github.com/apify/crawlee/blob/master/packages/basic-crawler/src/internals/basic-crawler.ts#L192) optionalinheritedrequestHandlerTimeoutSecs requestHandlerTimeoutSecs?: number = 60 Timeout in which the function passed as [`requestHandler`](/js/api/basic-crawler/interface/BasicCrawlerOptions#requestHandler) needs to finish, in seconds. ### [](#requestList) [](https://github.com/apify/crawlee/blob/master/packages/basic-crawler/src/internals/basic-crawler.ts#L178) optionalinheritedrequestList requestList?: [IRequestList](/js/api/core/interface/IRequestList) Static list of URLs to be processed. If not provided, the crawler will open the default request queue when the [`crawler.addRequests()`](/js/api/basic-crawler/class/BasicCrawler#addRequests) function is called. > Alternatively, `requests` parameter of [`crawler.run()`](/js/api/basic-crawler/class/BasicCrawler#run) > could be used to enqueue the initial requests - it is a shortcut for running `crawler.addRequests()` before the `crawler.run()`. ### [](#requestQueue) [](https://github.com/apify/crawlee/blob/master/packages/basic-crawler/src/internals/basic-crawler.ts#L186) optionalinheritedrequestQueue requestQueue?: [RequestProvider](/js/api/core/class/RequestProvider) Dynamic queue of URLs to be processed. This is useful for recursive crawling of websites. If not provided, the crawler will open the default request queue when the [`crawler.addRequests()`](/js/api/basic-crawler/class/BasicCrawler#addRequests) function is called. > Alternatively, `requests` parameter of [`crawler.run()`](/js/api/basic-crawler/class/BasicCrawler#run) > could be used to enqueue the initial requests - it is a shortcut for running `crawler.addRequests()` before the `crawler.run()`. ### [](#resultChecker) [](https://github.com/apify/crawlee/blob/master/packages/playwright-crawler/src/internals/adaptive-playwright-crawler.ts#L153) optionalresultChecker resultChecker?: (result) => boolean An optional callback that is called on dataset items found by the request handler in plain HTTP mode. If it returns false, the request is retried in a browser. If no callback is specified, every dataset item is considered valid. * * * #### Type declaration * * (result): boolean * #### Parameters * ##### result: [RequestHandlerResult](/js/api/core/class/RequestHandlerResult) #### Returns boolean ### [](#resultComparator) [](https://github.com/apify/crawlee/blob/master/packages/playwright-crawler/src/internals/adaptive-playwright-crawler.ts#L161) optionalresultComparator resultComparator?: (resultA, resultB) => boolean An optional callback used in rendering type detection. On each detection, the result of the plain HTTP run is compared to that of the browser one. If the callback returns true, the results are considered equal and the target site is considered static. If no result comparator is specified, but there is a `resultChecker`, any site where the `resultChecker` returns true is considered static. If neither `resultComparator` nor `resultChecker` are specified, a deep comparison of returned dataset items is used as a default. * * * #### Type declaration * * (resultA, resultB): boolean * #### Parameters * ##### resultA: [RequestHandlerResult](/js/api/core/class/RequestHandlerResult) * ##### resultB: [RequestHandlerResult](/js/api/core/class/RequestHandlerResult) #### Returns boolean ### [](#retryOnBlocked) [](https://github.com/apify/crawlee/blob/master/packages/basic-crawler/src/internals/basic-crawler.ts#L342) optionalinheritedretryOnBlocked retryOnBlocked?: boolean If set to `true`, the crawler will automatically try to bypass any detected bot protection. Currently supports: * [**Cloudflare** Bot Management](https://www.cloudflare.com/products/bot-management/) * [**Google Search** Rate Limiting](https://www.google.com/sorry/) ### [](#sameDomainDelaySecs) [](https://github.com/apify/crawlee/blob/master/packages/basic-crawler/src/internals/basic-crawler.ts#L246) optionalinheritedsameDomainDelaySecs sameDomainDelaySecs?: number = 0 Indicates how much time (in seconds) to wait before crawling another same domain request. ### [](#sessionPoolOptions) [](https://github.com/apify/crawlee/blob/master/packages/basic-crawler/src/internals/basic-crawler.ts#L310) optionalinheritedsessionPoolOptions sessionPoolOptions?: [SessionPoolOptions](/js/api/core/interface/SessionPoolOptions) The configuration options for [SessionPool](/js/api/core/class/SessionPool) to use. ### [](#statisticsOptions) [](https://github.com/apify/crawlee/blob/master/packages/basic-crawler/src/internals/basic-crawler.ts#L357) optionalinheritedstatisticsOptions statisticsOptions?: [StatisticsOptions](/js/api/core/interface/StatisticsOptions) Customize the way statistics collecting works, such as logging interval or whether to output them to the Key-Value store. ### [](#statusMessageCallback) [](https://github.com/apify/crawlee/blob/master/packages/basic-crawler/src/internals/basic-crawler.ts#L333) optionalinheritedstatusMessageCallback statusMessageCallback?: [StatusMessageCallback](/js/api/basic-crawler#StatusMessageCallback) <[BasicCrawlingContext](/js/api/basic-crawler/interface/BasicCrawlingContext) , [BasicCrawler](/js/api/basic-crawler/class/BasicCrawler) <[BasicCrawlingContext](/js/api/basic-crawler/interface/BasicCrawlingContext) \>\> Allows overriding the default status message. The callback needs to call `crawler.setStatusMessage()` explicitly. The default status message is provided in the parameters. const crawler = new CheerioCrawler({ statusMessageCallback: async (ctx) => { return ctx.crawler.setStatusMessage(`this is status message from ${new Date().toISOString()}`, { level: 'INFO' }); // log level defaults to 'DEBUG' }, statusMessageLoggingInterval: 1, // defaults to 10s async requestHandler({ $, enqueueLinks, request, log }) { // ... },}); ### [](#statusMessageLoggingInterval) [](https://github.com/apify/crawlee/blob/master/packages/basic-crawler/src/internals/basic-crawler.ts#L315) optionalinheritedstatusMessageLoggingInterval statusMessageLoggingInterval?: number Defines the length of the interval for calling the `setStatusMessage` in seconds. ### [](#useSessionPool) [](https://github.com/apify/crawlee/blob/master/packages/basic-crawler/src/internals/basic-crawler.ts#L305) optionalinheriteduseSessionPool useSessionPool?: boolean Basic crawler will initialize the [SessionPool](/js/api/core/class/SessionPool) with the corresponding [`sessionPoolOptions`](/js/api/core/interface/SessionPoolOptions) . The session instance will be than available in the [`requestHandler`](/js/api/basic-crawler/interface/BasicCrawlerOptions#requestHandler) . **Page Options** Hide Inherited * [autoscaledPoolOptions](#autoscaledPoolOptions) * [browserPoolOptions](#browserPoolOptions) * [errorHandler](#errorHandler) * [experiments](#experiments) * [failedRequestHandler](#failedRequestHandler) * [headless](#headless) * [httpClient](#httpClient) * [ignoreIframes](#ignoreIframes) * [ignoreShadowRoots](#ignoreShadowRoots) * [keepAlive](#keepAlive) * [launchContext](#launchContext) * [maxConcurrency](#maxConcurrency) * [maxRequestRetries](#maxRequestRetries) * [maxRequestsPerCrawl](#maxRequestsPerCrawl) * [maxRequestsPerMinute](#maxRequestsPerMinute) * [maxSessionRotations](#maxSessionRotations) * [minConcurrency](#minConcurrency) * [navigationTimeoutSecs](#navigationTimeoutSecs) * [persistCookiesPerSession](#persistCookiesPerSession) * [postNavigationHooks](#postNavigationHooks) * [preNavigationHooks](#preNavigationHooks) * [proxyConfiguration](#proxyConfiguration) * [renderingTypeDetectionRatio](#renderingTypeDetectionRatio) * [renderingTypePredictor](#renderingTypePredictor) * [requestHandler](#requestHandler) * [requestHandlerTimeoutSecs](#requestHandlerTimeoutSecs) * [requestList](#requestList) * [requestQueue](#requestQueue) * [resultChecker](#resultChecker) * [resultComparator](#resultComparator) * [retryOnBlocked](#retryOnBlocked) * [sameDomainDelaySecs](#sameDomainDelaySecs) * [sessionPoolOptions](#sessionPoolOptions) * [statisticsOptions](#statisticsOptions) * [statusMessageCallback](#statusMessageCallback) * [statusMessageLoggingInterval](#statusMessageLoggingInterval) * [useSessionPool](#useSessionPool) --- # PlaywrightCrawlingContext | API | Crawlee for JavaScript · Build reliable crawlers. Fast. [Skip to main content](#__docusaurus_skipToContent_fallback) Version: 3.13 On this page ### Hierarchy * [BrowserCrawlingContext](/js/api/browser-crawler/interface/BrowserCrawlingContext) <[PlaywrightCrawler](/js/api/playwright-crawler/class/PlaywrightCrawler) , Page, Response, [PlaywrightController](/js/api/browser-pool/class/PlaywrightController) , UserData\> * PlaywrightContextUtils * _PlaywrightCrawlingContext_ Index[](#Index) ---------------- ### Properties * [addRequests](#addRequests) * [browserController](#browserController) * [crawler](#crawler) * [getKeyValueStore](#getKeyValueStore) * [id](#id) * [log](#log) * [page](#page) * [proxyInfo](#proxyInfo) * [request](#request) * [response](#response) * [session](#session) * [useState](#useState) ### Methods * [blockRequests](#blockRequests) * [closeCookieModals](#closeCookieModals) * [compileScript](#compileScript) * [enqueueLinks](#enqueueLinks) * [enqueueLinksByClickingElements](#enqueueLinksByClickingElements) * [handleCloudflareChallenge](#handleCloudflareChallenge) * [infiniteScroll](#infiniteScroll) * [injectFile](#injectFile) * [injectJQuery](#injectJQuery) * [parseWithCheerio](#parseWithCheerio) * [pushData](#pushData) * [saveSnapshot](#saveSnapshot) * [sendRequest](#sendRequest) * [waitForSelector](#waitForSelector) Properties[](#Properties) -------------------------- ### [](#addRequests) [](https://github.com/apify/crawlee/blob/master/packages/core/src/crawlers/crawler_commons.ts#L88) inheritedaddRequests addRequests: (requestsLike, options) => Promise Add requests directly to the request queue. * * * #### Type declaration * * (requestsLike, options): Promise * #### Parameters * ##### requestsLike: readonly (string | ReadonlyObjectDeep\> & { regex?: RegExp; requestsFromUrl?: string }\> | ReadonlyObjectDeep<[Request](/js/api/core/class/Request) \>)\[\] * ##### optionaloptions: ReadonlyObjectDeep<[RequestQueueOperationOptions](/js/api/core/interface/RequestQueueOperationOptions) \> Options for the request queue #### Returns Promise ### [](#browserController) [](https://github.com/apify/crawlee/blob/master/packages/browser-crawler/src/internals/browser-crawler.ts#L55) inheritedbrowserController browserController: [PlaywrightController](/js/api/browser-pool/class/PlaywrightController) ### [](#crawler) [](https://github.com/apify/crawlee/blob/master/packages/core/src/crawlers/crawler_commons.ts#L113) inheritedcrawler crawler: [PlaywrightCrawler](/js/api/playwright-crawler/class/PlaywrightCrawler) ### [](#getKeyValueStore) [](https://github.com/apify/crawlee/blob/master/packages/core/src/crawlers/crawler_commons.ts#L147) inheritedgetKeyValueStore getKeyValueStore: (idOrName) => Promise<[KeyValueStore](/js/api/core/class/KeyValueStore) \> Get a key-value store with given name or id, or the default one for the crawler. * * * #### Type declaration * * (idOrName): Promise<[KeyValueStore](/js/api/core/class/KeyValueStore) \> * #### Parameters * ##### optionalidOrName: string #### Returns Promise<[KeyValueStore](/js/api/core/class/KeyValueStore) \> ### [](#id) [](https://github.com/apify/crawlee/blob/master/packages/core/src/crawlers/crawler_commons.ts#L33) inheritedid id: string ### [](#log) [](https://github.com/apify/crawlee/blob/master/packages/core/src/crawlers/crawler_commons.ts#L108) inheritedlog log: [Log](/js/api/core/class/Log) A preconfigured logger for the request handler. ### [](#page) [](https://github.com/apify/crawlee/blob/master/packages/browser-crawler/src/internals/browser-crawler.ts#L56) inheritedpage page: Page ### [](#proxyInfo) [](https://github.com/apify/crawlee/blob/master/packages/core/src/crawlers/crawler_commons.ts#L40) optionalinheritedproxyInfo proxyInfo?: [ProxyInfo](/js/api/core/interface/ProxyInfo) An object with information about currently used proxy by the crawler and configured by the [ProxyConfiguration](/js/api/core/class/ProxyConfiguration) class. ### [](#request) [](https://github.com/apify/crawlee/blob/master/packages/core/src/crawlers/crawler_commons.ts#L45) inheritedrequest request: [Request](/js/api/core/class/Request) The original [Request](/js/api/core/class/Request) object. ### [](#response) [](https://github.com/apify/crawlee/blob/master/packages/browser-crawler/src/internals/browser-crawler.ts#L57) optionalinheritedresponse response?: Response ### [](#session) [](https://github.com/apify/crawlee/blob/master/packages/core/src/crawlers/crawler_commons.ts#L34) optionalinheritedsession session?: [Session](/js/api/core/class/Session) ### [](#useState) [](https://github.com/apify/crawlee/blob/master/packages/core/src/crawlers/crawler_commons.ts#L96) inheriteduseState useState: (defaultValue) => Promise Returns the state - a piece of mutable persistent data shared across all the request handler runs. * * * #### Type declaration * * (defaultValue): Promise * #### Parameters * ##### optionaldefaultValue: State #### Returns Promise Methods[](#Methods) -------------------- ### [](#blockRequests) [](https://github.com/apify/crawlee/blob/master/packages/playwright-crawler/src/internals/utils/playwright-utils.ts#L860) inheritedblockRequests * **blockRequests**(options): Promise * Forces the Playwright browser tab to block loading URLs that match a provided pattern. This is useful to speed up crawling of websites, since it reduces the amount of data that needs to be downloaded from the web, but it may break some websites or unexpectedly prevent loading of resources. By default, the function will block all URLs including the following patterns: [".css", ".jpg", ".jpeg", ".png", ".svg", ".gif", ".woff", ".pdf", ".zip"] If you want to extend this list further, use the `extraUrlPatterns` option, which will keep blocking the default patterns, as well as add your custom ones. If you would like to block only specific patterns, use the `urlPatterns` option, which will override the defaults and block only URLs with your custom patterns. This function does not use Playwright's request interception and therefore does not interfere with browser cache. It's also faster than blocking requests using interception, because the blocking happens directly in the browser without the round-trip to Node.js, but it does not provide the extra benefits of request interception. The function will never block main document loads and their respective redirects. **Example usage** preNavigationHooks: [ async ({ blockRequests }) => { // Block all requests to URLs that include `adsbygoogle.js` and also all defaults. await blockRequests({ extraUrlPatterns: ['adsbygoogle.js'], }); },], * * * #### Parameters * ##### optionaloptions: [BlockRequestsOptions](/js/api/playwright-crawler/namespace/playwrightUtils#BlockRequestsOptions) #### Returns Promise ### [](#closeCookieModals) [](https://github.com/apify/crawlee/blob/master/packages/playwright-crawler/src/internals/utils/playwright-utils.ts#L979) inheritedcloseCookieModals * **closeCookieModals**(): Promise * Tries to close cookie consent modals on the page. Based on the I Don't Care About Cookies browser extension. * * * #### Returns Promise ### [](#compileScript) [](https://github.com/apify/crawlee/blob/master/packages/playwright-crawler/src/internals/utils/playwright-utils.ts#L974) inheritedcompileScript * **compileScript**(scriptString, ctx): [CompiledScriptFunction](/js/api/playwright-crawler/namespace/playwrightUtils#CompiledScriptFunction) * Compiles a Playwright script into an async function that may be executed at any time by providing it with the following object: { page: Page, request: Request,} Where `page` is a Playwright [`Page`](https://playwright.dev/docs/api/class-page) and `request` is a [Request](/js/api/core/class/Request) . The function is compiled by using the `scriptString` parameter as the function's body, so any limitations to function bodies apply. Return value of the compiled function is the return value of the function body = the `scriptString` parameter. As a security measure, no globals such as `process` or `require` are accessible from within the function body. Note that the function does not provide a safe sandbox and even though globals are not easily accessible, malicious code may still execute in the main process via prototype manipulation. Therefore you should only use this function to execute sanitized or safe code. Custom context may also be provided using the `context` parameter. To improve security, make sure to only pass the really necessary objects to the context. Preferably making secured copies beforehand. * * * #### Parameters * ##### scriptString: string * ##### optionalctx: Dictionary #### Returns [CompiledScriptFunction](/js/api/playwright-crawler/namespace/playwrightUtils#CompiledScriptFunction) ### [](#enqueueLinks) [](https://github.com/apify/crawlee/blob/master/packages/core/src/crawlers/crawler_commons.ts#L140) inheritedenqueueLinks * **enqueueLinks**(options): Promise<[BatchAddRequestsResult](/js/api/types/interface/BatchAddRequestsResult) \> * This function automatically finds and enqueues links from the current page, adding them to the [RequestQueue](/js/api/core/class/RequestQueue) currently used by the crawler. Optionally, the function allows you to filter the target links' URLs using an array of globs or regular expressions and override settings of the enqueued [Request](/js/api/core/class/Request) objects. Check out the [Crawl a website with relative links](/js/docs/examples/crawl-relative-links) example for more details regarding its usage. **Example usage** async requestHandler({ enqueueLinks }) { await enqueueLinks({ globs: [ 'https://www.example.com/handbags/*', ], });}, * * * #### Parameters * ##### optionaloptions: ReadonlyObjectDeep\> & Pick<[EnqueueLinksOptions](/js/api/core/interface/EnqueueLinksOptions) , requestQueue\> All `enqueueLinks()` parameters are passed via an options object. #### Returns Promise<[BatchAddRequestsResult](/js/api/types/interface/BatchAddRequestsResult) \> Promise that resolves to [BatchAddRequestsResult](/js/api/types/interface/BatchAddRequestsResult) object. ### [](#enqueueLinksByClickingElements) [](https://github.com/apify/crawlee/blob/master/packages/playwright-crawler/src/internals/utils/playwright-utils.ts#L944) inheritedenqueueLinksByClickingElements * **enqueueLinksByClickingElements**(options): Promise<[BatchAddRequestsResult](/js/api/types/interface/BatchAddRequestsResult) \> * The function finds elements matching a specific CSS selector in a Playwright page, clicks all those elements using a mouse move and a left mouse button click and intercepts all the navigation requests that are subsequently produced by the page. The intercepted requests, including their methods, headers and payloads are then enqueued to a provided [RequestQueue](/js/api/core/class/RequestQueue) . This is useful to crawl JavaScript heavy pages where links are not available in `href` elements, but rather navigations are triggered in click handlers. If you're looking to find URLs in `href` attributes of the page, see enqueueLinks. Optionally, the function allows you to filter the target links' URLs using an array of [PseudoUrl](/js/api/core/class/PseudoUrl) objects and override settings of the enqueued [Request](/js/api/core/class/Request) objects. **IMPORTANT**: To be able to do this, this function uses various mutations on the page, such as changing the Z-index of elements being clicked and their visibility. Therefore, it is recommended to only use this function as the last operation in the page. **USING HEADFUL BROWSER**: When using a headful browser, this function will only be able to click elements in the focused tab, effectively limiting concurrency to 1. In headless mode, full concurrency can be achieved. **PERFORMANCE**: Clicking elements with a mouse and intercepting requests is not a low level operation that takes nanoseconds. It's not very CPU intensive, but it takes time. We strongly recommend limiting the scope of the clicking as much as possible by using a specific selector that targets only the elements that you assume or know will produce a navigation. You can certainly click everything by using the `*` selector, but be prepared to wait minutes to get results on a large and complex page. **Example usage** async requestHandler({ enqueueLinksByClickingElements }) { await enqueueLinksByClickingElements({ selector: 'a.product-detail', globs: [ 'https://www.example.com/handbags/**' 'https://www.example.com/purses/**' ], });}); * * * #### Parameters * ##### options: Omit<[EnqueueLinksByClickingElementsOptions](/js/api/playwright-crawler/namespace/playwrightClickElements#EnqueueLinksByClickingElementsOptions) , requestQueue | page\> #### Returns Promise<[BatchAddRequestsResult](/js/api/types/interface/BatchAddRequestsResult) \> Promise that resolves to [BatchAddRequestsResult](/js/api/types/interface/BatchAddRequestsResult) object. ### [](#handleCloudflareChallenge) [](https://github.com/apify/crawlee/blob/master/packages/playwright-crawler/src/internals/utils/playwright-utils.ts#L1001) inheritedhandleCloudflareChallenge * **handleCloudflareChallenge**(options): Promise * This helper tries to solve the Cloudflare challenge automatically by clicking on the checkbox. It will try to detect the Cloudflare page, click on the checkbox, and wait for 10 seconds (configurable via `sleepSecs` option) for the page to load. Use this in the `postNavigationHooks`, a failures will result in a SessionError which will be automatically retried, so only successful requests will get into the `requestHandler`. Works best with camoufox. **Example usage** postNavigationHooks: [ async ({ handleCloudflareChallenge }) => { await handleCloudflareChallenge(); },], * * * #### Parameters * ##### optionaloptions: HandleCloudflareChallengeOptions #### Returns Promise ### [](#infiniteScroll) [](https://github.com/apify/crawlee/blob/master/packages/playwright-crawler/src/internals/utils/playwright-utils.ts#L895) inheritedinfiniteScroll * **infiniteScroll**(options): Promise * Scrolls to the bottom of a page, or until it times out. Loads dynamic content when it hits the bottom of a page, and then continues scrolling. * * * #### Parameters * ##### optionaloptions: [InfiniteScrollOptions](/js/api/playwright-crawler/namespace/playwrightUtils#InfiniteScrollOptions) #### Returns Promise ### [](#injectFile) [](https://github.com/apify/crawlee/blob/master/packages/playwright-crawler/src/internals/utils/playwright-utils.ts#L795) inheritedinjectFile * **injectFile**(filePath, options): Promise * Injects a JavaScript file into current `page`. Unlike Playwright's `addScriptTag` function, this function works on pages with arbitrary Cross-Origin Resource Sharing (CORS) policies. File contents are cached for up to 10 files to limit file system access. * * * #### Parameters * ##### filePath: string * ##### optionaloptions: [InjectFileOptions](/js/api/playwright-crawler/namespace/playwrightUtils#InjectFileOptions) #### Returns Promise ### [](#injectJQuery) [](https://github.com/apify/crawlee/blob/master/packages/playwright-crawler/src/internals/utils/playwright-utils.ts#L822) inheritedinjectJQuery * **injectJQuery**(): Promise * Injects the [jQuery](https://jquery.com/) library into current `page`. jQuery is often useful for various web scraping and crawling tasks. For example, it can help extract text from HTML elements using CSS selectors. Beware that the injected jQuery object will be set to the `window.$` variable and thus it might cause conflicts with other libraries included by the page that use the same variable name (e.g. another version of jQuery). This can affect functionality of page's scripts. The injected jQuery will survive page navigations and reloads. **Example usage:** async requestHandler({ page, injectJQuery }) { await injectJQuery(); const title = await page.evaluate(() => { return $('head title').text(); });}); Note that `injectJQuery()` does not affect the Playwright [`page.$()`](https://playwright.dev/docs/api/class-page#page-query-selector) function in any way. * * * #### Returns Promise ### [](#parseWithCheerio) [](https://github.com/apify/crawlee/blob/master/packages/playwright-crawler/src/internals/utils/playwright-utils.ts#L889) inheritedparseWithCheerio * **parseWithCheerio**(selector, timeoutMs): Promise * Returns Cheerio handle for `page.content()`, allowing to work with the data same way as with [CheerioCrawler](/js/api/cheerio-crawler/class/CheerioCrawler) . When provided with the `selector` argument, it waits for it to be available first. **Example usage:** async requestHandler({ parseWithCheerio }) { const $ = await parseWithCheerio(); const title = $('title').text();}); * * * #### Parameters * ##### optionalselector: string * ##### optionaltimeoutMs: number #### Returns Promise ### [](#pushData) [](https://github.com/apify/crawlee/blob/master/packages/core/src/crawlers/crawler_commons.ts#L54) inheritedpushData * **pushData**(data, datasetIdOrName): Promise * This function allows you to push data to a [Dataset](/js/api/core/class/Dataset) specified by name, or the one currently used by the crawler. Shortcut for `crawler.pushData()`. * * * #### Parameters * ##### optionaldata: ReadonlyDeep Data to be pushed to the default dataset. * ##### optionaldatasetIdOrName: string #### Returns Promise ### [](#saveSnapshot) [](https://github.com/apify/crawlee/blob/master/packages/playwright-crawler/src/internals/utils/playwright-utils.ts#L901) inheritedsaveSnapshot * **saveSnapshot**(options): Promise * Saves a full screenshot and HTML of the current page into a Key-Value store. * * * #### Parameters * ##### optionaloptions: [SaveSnapshotOptions](/js/api/playwright-crawler/namespace/playwrightUtils#SaveSnapshotOptions) #### Returns Promise ### [](#sendRequest) [](https://github.com/apify/crawlee/blob/master/packages/core/src/crawlers/crawler_commons.ts#L166) inheritedsendRequest * **sendRequest**(overrideOptions): Promise\> * Fires HTTP request via [`got-scraping`](/js/docs/guides/got-scraping) , allowing to override the request options on the fly. This is handy when you work with a browser crawler but want to execute some requests outside it (e.g. API requests). Check the [Skipping navigations for certain requests](/js/docs/examples/skip-navigation) example for more detailed explanation of how to do that. async requestHandler({ sendRequest }) { const { body } = await sendRequest({ // override headers only headers: { ... }, });}, * * * #### Parameters * ##### optionaloverrideOptions: Partial #### Returns Promise\> ### [](#waitForSelector) [](https://github.com/apify/crawlee/blob/master/packages/playwright-crawler/src/internals/utils/playwright-utils.ts#L875) inheritedwaitForSelector * **waitForSelector**(selector, timeoutMs): Promise * Wait for an element matching the selector to appear. Timeout defaults to 5s. **Example usage:** async requestHandler({ waitForSelector, parseWithCheerio }) { await waitForSelector('article h1'); const $ = await parseWithCheerio(); const title = $('title').text();}); * * * #### Parameters * ##### selector: string * ##### optionaltimeoutMs: number #### Returns Promise **Page Options** Hide Inherited * [addRequests](#addRequests) * [browserController](#browserController) * [crawler](#crawler) * [getKeyValueStore](#getKeyValueStore) * [id](#id) * [log](#log) * [page](#page) * [proxyInfo](#proxyInfo) * [request](#request) * [response](#response) * [session](#session) * [useState](#useState) * [blockRequests](#blockRequests) * [closeCookieModals](#closeCookieModals) * [compileScript](#compileScript) * [enqueueLinks](#enqueueLinks) * [enqueueLinksByClickingElements](#enqueueLinksByClickingElements) * [handleCloudflareChallenge](#handleCloudflareChallenge) * [infiniteScroll](#infiniteScroll) * [injectFile](#injectFile) * [injectJQuery](#injectJQuery) * [parseWithCheerio](#parseWithCheerio) * [pushData](#pushData) * [saveSnapshot](#saveSnapshot) * [sendRequest](#sendRequest) * [waitForSelector](#waitForSelector) --- # PlaywrightHook | API | Crawlee for JavaScript · Build reliable crawlers. Fast. [Skip to main content](#__docusaurus_skipToContent_fallback) Version: 3.13 ### Hierarchy * [BrowserHook](/js/api/browser-crawler#BrowserHook) <[PlaywrightCrawlingContext](/js/api/playwright-crawler/interface/PlaywrightCrawlingContext) , [PlaywrightGotoOptions](/js/api/playwright-crawler#PlaywrightGotoOptions) \> * _PlaywrightHook_ ### Callable * **PlaywrightHook**(crawlingContext, gotoOptions): Awaitable * * * * #### Parameters * ##### crawlingContext: [PlaywrightCrawlingContext](/js/api/playwright-crawler/interface/PlaywrightCrawlingContext) * ##### gotoOptions: undefined | { referer?: string; timeout?: number; waitUntil?: domcontentloaded | load | networkidle | commit } * ##### externaloptionalreferer: string Referer header value. If provided it will take preference over the referer header value set by [page.setExtraHTTPHeaders(headers)](https://playwright.dev/docs/api/class-page#page-set-extra-http-headers) . * ##### externaloptionaltimeout: number Maximum operation time in milliseconds. Defaults to `0` - no timeout. The default value can be changed via `navigationTimeout` option in the config, or by using the [browserContext.setDefaultNavigationTimeout(timeout)](https://playwright.dev/docs/api/class-browsercontext#browser-context-set-default-navigation-timeout) , [browserContext.setDefaultTimeout(timeout)](https://playwright.dev/docs/api/class-browsercontext#browser-context-set-default-timeout) , [page.setDefaultNavigationTimeout(timeout)](https://playwright.dev/docs/api/class-page#page-set-default-navigation-timeout) or [page.setDefaultTimeout(timeout)](https://playwright.dev/docs/api/class-page#page-set-default-timeout) methods. * ##### externaloptionalwaitUntil: domcontentloaded | load | networkidle | commit When to consider operation succeeded, defaults to `load`. Events can be either: * `'domcontentloaded'` - consider operation to be finished when the `DOMContentLoaded` event is fired. * `'load'` - consider operation to be finished when the `load` event is fired. * `'networkidle'` - **DISCOURAGED** consider operation to be finished when there are no network connections for at least `500` ms. Don't use this method for testing, rely on web assertions to assess readiness instead. * `'commit'` - consider operation to be finished when network response is received and the document started loading. #### Returns Awaitable --- # PlaywrightCrawlerOptions | API | Crawlee for JavaScript · Build reliable crawlers. Fast. [Skip to main content](#__docusaurus_skipToContent_fallback) Version: 3.13 On this page ### Hierarchy * [BrowserCrawlerOptions](/js/api/browser-crawler/interface/BrowserCrawlerOptions) <[PlaywrightCrawlingContext](/js/api/playwright-crawler/interface/PlaywrightCrawlingContext) , { browserPlugins: \[[PlaywrightPlugin](/js/api/browser-pool/class/PlaywrightPlugin)\ \] }\> * _PlaywrightCrawlerOptions_ Index[](#Index) ---------------- ### Properties * [autoscaledPoolOptions](#autoscaledPoolOptions) * [browserPoolOptions](#browserPoolOptions) * [errorHandler](#errorHandler) * [experiments](#experiments) * [failedRequestHandler](#failedRequestHandler) * [headless](#headless) * [httpClient](#httpClient) * [ignoreIframes](#ignoreIframes) * [ignoreShadowRoots](#ignoreShadowRoots) * [keepAlive](#keepAlive) * [launchContext](#launchContext) * [maxConcurrency](#maxConcurrency) * [maxRequestRetries](#maxRequestRetries) * [maxRequestsPerCrawl](#maxRequestsPerCrawl) * [maxRequestsPerMinute](#maxRequestsPerMinute) * [maxSessionRotations](#maxSessionRotations) * [minConcurrency](#minConcurrency) * [navigationTimeoutSecs](#navigationTimeoutSecs) * [persistCookiesPerSession](#persistCookiesPerSession) * [postNavigationHooks](#postNavigationHooks) * [preNavigationHooks](#preNavigationHooks) * [proxyConfiguration](#proxyConfiguration) * [requestHandler](#requestHandler) * [requestHandlerTimeoutSecs](#requestHandlerTimeoutSecs) * [requestList](#requestList) * [requestQueue](#requestQueue) * [retryOnBlocked](#retryOnBlocked) * [sameDomainDelaySecs](#sameDomainDelaySecs) * [sessionPoolOptions](#sessionPoolOptions) * [statisticsOptions](#statisticsOptions) * [statusMessageCallback](#statusMessageCallback) * [statusMessageLoggingInterval](#statusMessageLoggingInterval) * [useSessionPool](#useSessionPool) Properties[](#Properties) -------------------------- ### [](#autoscaledPoolOptions) [](https://github.com/apify/crawlee/blob/master/packages/basic-crawler/src/internals/basic-crawler.ts#L271) optionalinheritedautoscaledPoolOptions autoscaledPoolOptions?: [AutoscaledPoolOptions](/js/api/core/interface/AutoscaledPoolOptions) Custom options passed to the underlying [AutoscaledPool](/js/api/core/class/AutoscaledPool) constructor. > _NOTE:_ The [`runTaskFunction`](/js/api/core/interface/AutoscaledPoolOptions#runTaskFunction) > and [`isTaskReadyFunction`](/js/api/core/interface/AutoscaledPoolOptions#isTaskReadyFunction) > options are provided by the crawler and cannot be overridden. However, we can provide a custom implementation of [`isFinishedFunction`](/js/api/core/interface/AutoscaledPoolOptions#isFinishedFunction) > . ### [](#browserPoolOptions) [](https://github.com/apify/crawlee/blob/master/packages/browser-crawler/src/internals/browser-crawler.ts#L190) optionalinheritedbrowserPoolOptions browserPoolOptions?: Partial<[BrowserPoolOptions](/js/api/browser-pool/interface/BrowserPoolOptions) <[BrowserPlugin](/js/api/browser-pool/class/BrowserPlugin) <[CommonLibrary](/js/api/browser-pool/interface/CommonLibrary) , undefined | Dictionary, CommonBrowser, unknown, CommonPage\>\>\> & Partial<[BrowserPoolHooks](/js/api/browser-pool/interface/BrowserPoolHooks) <[BrowserController](/js/api/browser-pool/class/BrowserController) , undefined | LaunchOptions, Browser, undefined | { acceptDownloads?: boolean; baseURL?: string; bypassCSP?: boolean; clientCertificates?: { cert?: ... | ...; certPath?: ... | ...; key?: ... | ...; keyPath?: ... | ...; origin: string; passphrase?: ... | ...; pfx?: ... | ...; pfxPath?: ... | ... }\[\]; colorScheme?: null | light | dark | no-preference; deviceScaleFactor?: number; extraHTTPHeaders?: {}; forcedColors?: null | active | none; geolocation?: { accuracy?: number; latitude: number; longitude: number }; hasTouch?: boolean; httpCredentials?: { origin?: string; password: string; send?: unauthorized | always; username: string }; ignoreHTTPSErrors?: boolean; isMobile?: boolean; javaScriptEnabled?: boolean; locale?: string; logger?: Logger; offline?: boolean; permissions?: string\[\]; proxy?: { bypass?: string; password?: string; server: string; username?: string }; recordHar?: { content?: omit | embed | attach; mode?: full | minimal; omitContent?: boolean; path: string; urlFilter?: string | RegExp }; recordVideo?: { dir: string; size?: { height: ...; width: ... } }; reducedMotion?: null | reduce | no-preference; screen?: { height: number; width: number }; serviceWorkers?: allow | block; storageState?: string | { cookies: { domain: ...; expires: ...; httpOnly: ...; name: ...; path: ...; sameSite: ...; secure: ...; value: ... }\[\]; origins: { localStorage: ...; origin: ... }\[\] }; strictSelectors?: boolean; timezoneId?: string; userAgent?: string; videoSize?: { height: number; width: number }; videosPath?: string; viewport?: null | { height: number; width: number } }, Page\>, [LaunchContext](/js/api/browser-pool/class/LaunchContext) , undefined | LaunchOptions, Browser, undefined | { acceptDownloads?: boolean; baseURL?: string; bypassCSP?: boolean; clientCertificates?: { cert?: ... | ...; certPath?: ... | ...; key?: ... | ...; keyPath?: ... | ...; origin: string; passphrase?: ... | ...; pfx?: ... | ...; pfxPath?: ... | ... }\[\]; colorScheme?: null | light | dark | no-preference; deviceScaleFactor?: number; extraHTTPHeaders?: {}; forcedColors?: null | active | none; geolocation?: { accuracy?: number; latitude: number; longitude: number }; hasTouch?: boolean; httpCredentials?: { origin?: string; password: string; send?: unauthorized | always; username: string }; ignoreHTTPSErrors?: boolean; isMobile?: boolean; javaScriptEnabled?: boolean; locale?: string; logger?: Logger; offline?: boolean; permissions?: string\[\]; proxy?: { bypass?: string; password?: string; server: string; username?: string }; recordHar?: { content?: omit | embed | attach; mode?: full | minimal; omitContent?: boolean; path: string; urlFilter?: string | RegExp }; recordVideo?: { dir: string; size?: { height: ...; width: ... } }; reducedMotion?: null | reduce | no-preference; screen?: { height: number; width: number }; serviceWorkers?: allow | block; storageState?: string | { cookies: { domain: ...; expires: ...; httpOnly: ...; name: ...; path: ...; sameSite: ...; secure: ...; value: ... }\[\]; origins: { localStorage: ...; origin: ... }\[\] }; strictSelectors?: boolean; timezoneId?: string; userAgent?: string; videoSize?: { height: number; width: number }; videosPath?: string; viewport?: null | { height: number; width: number } }, Page\>, Page\>\> Custom options passed to the underlying [BrowserPool](/js/api/browser-pool/class/BrowserPool) constructor. We can tweak those to fine-tune browser management. ### [](#errorHandler) [](https://github.com/apify/crawlee/blob/master/packages/browser-crawler/src/internals/browser-crawler.ts#L159) optionalinheritederrorHandler errorHandler?: [BrowserErrorHandler](/js/api/browser-crawler#BrowserErrorHandler) <[PlaywrightCrawlingContext](/js/api/playwright-crawler/interface/PlaywrightCrawlingContext) \> User-provided function that allows modifying the request object before it gets retried by the crawler. It's executed before each retry for the requests that failed less than [`maxRequestRetries`](/js/api/browser-crawler/interface/BrowserCrawlerOptions#maxRequestRetries) times. The function receives the [BrowserCrawlingContext](/js/api/browser-crawler/interface/BrowserCrawlingContext) (actual context will be enhanced with the crawler specific properties) as the first argument, where the [`request`](/js/api/browser-crawler/interface/BrowserCrawlingContext#request) corresponds to the request to be retried. Second argument is the `Error` instance that represents the last error thrown during processing of the request. ### [](#experiments) [](https://github.com/apify/crawlee/blob/master/packages/basic-crawler/src/internals/basic-crawler.ts#L351) optionalinheritedexperiments experiments?: [CrawlerExperiments](/js/api/basic-crawler/interface/CrawlerExperiments) Enables experimental features of Crawlee, which can alter the behavior of the crawler. WARNING: these options are not guaranteed to be stable and may change or be removed at any time. ### [](#failedRequestHandler) [](https://github.com/apify/crawlee/blob/master/packages/browser-crawler/src/internals/browser-crawler.ts#L170) optionalinheritedfailedRequestHandler failedRequestHandler?: [BrowserErrorHandler](/js/api/browser-crawler#BrowserErrorHandler) <[PlaywrightCrawlingContext](/js/api/playwright-crawler/interface/PlaywrightCrawlingContext) \> A function to handle requests that failed more than `option.maxRequestRetries` times. The function receives the [BrowserCrawlingContext](/js/api/browser-crawler/interface/BrowserCrawlingContext) (actual context will be enhanced with the crawler specific properties) as the first argument, where the [`request`](/js/api/browser-crawler/interface/BrowserCrawlingContext#request) corresponds to the failed request. Second argument is the `Error` instance that represents the last error thrown during processing of the request. ### [](#headless) [](https://github.com/apify/crawlee/blob/master/packages/browser-crawler/src/internals/browser-crawler.ts#L256) optionalinheritedheadless headless?: boolean | new | old Whether to run browser in headless mode. Defaults to `true`. Can be also set via [Configuration](/js/api/core/class/Configuration) . ### [](#httpClient) [](https://github.com/apify/crawlee/blob/master/packages/basic-crawler/src/internals/basic-crawler.ts#L363) optionalinheritedhttpClient httpClient?: [BaseHttpClient](/js/api/core/interface/BaseHttpClient) HTTP client implementation for the `sendRequest` context helper and for plain HTTP crawling. Defaults to a new instance of [GotScrapingHttpClient](/js/api/core/class/GotScrapingHttpClient) ### [](#ignoreIframes) [](https://github.com/apify/crawlee/blob/master/packages/browser-crawler/src/internals/browser-crawler.ts#L268) optionalinheritedignoreIframes ignoreIframes?: boolean Whether to ignore `iframes` when processing the page content via `parseWithCheerio` helper. By default, `iframes` are expanded automatically. Use this option to disable this behavior. ### [](#ignoreShadowRoots) [](https://github.com/apify/crawlee/blob/master/packages/browser-crawler/src/internals/browser-crawler.ts#L262) optionalinheritedignoreShadowRoots ignoreShadowRoots?: boolean Whether to ignore custom elements (and their #shadow-roots) when processing the page content via `parseWithCheerio` helper. By default, they are expanded automatically. Use this option to disable this behavior. ### [](#keepAlive) [](https://github.com/apify/crawlee/blob/master/packages/basic-crawler/src/internals/basic-crawler.ts#L299) optionalinheritedkeepAlive keepAlive?: boolean Allows to keep the crawler alive even if the [RequestQueue](/js/api/core/class/RequestQueue) gets empty. By default, the `crawler.run()` will resolve once the queue is empty. With `keepAlive: true` it will keep running, waiting for more requests to come. Use `crawler.stop()` to exit the crawler gracefully, or `crawler.teardown()` to stop it immediately. ### [](#launchContext) [](https://github.com/apify/crawlee/blob/master/packages/playwright-crawler/src/internals/playwright-crawler.ts#L33) optionallaunchContext launchContext?: [PlaywrightLaunchContext](/js/api/playwright-crawler/interface/PlaywrightLaunchContext) The same options as used by [launchPlaywright](/js/api/playwright-crawler/function/launchPlaywright) . ### [](#maxConcurrency) [](https://github.com/apify/crawlee/blob/master/packages/basic-crawler/src/internals/basic-crawler.ts#L285) optionalinheritedmaxConcurrency maxConcurrency?: number Sets the maximum concurrency (parallelism) for the crawl. Shortcut for the AutoscaledPool [`maxConcurrency`](/js/api/core/interface/AutoscaledPoolOptions#maxConcurrency) option. ### [](#maxRequestRetries) [](https://github.com/apify/crawlee/blob/master/packages/basic-crawler/src/internals/basic-crawler.ts#L240) optionalinheritedmaxRequestRetries maxRequestRetries?: number = 3 Indicates how many times the request is retried if [`requestHandler`](/js/api/basic-crawler/interface/BasicCrawlerOptions#requestHandler) fails. ### [](#maxRequestsPerCrawl) [](https://github.com/apify/crawlee/blob/master/packages/basic-crawler/src/internals/basic-crawler.ts#L262) optionalinheritedmaxRequestsPerCrawl maxRequestsPerCrawl?: number Maximum number of pages that the crawler will open. The crawl will stop when this limit is reached. This value should always be set in order to prevent infinite loops in misconfigured crawlers. > _NOTE:_ In cases of parallel crawling, the actual number of pages visited might be slightly higher than this value. ### [](#maxRequestsPerMinute) [](https://github.com/apify/crawlee/blob/master/packages/basic-crawler/src/internals/basic-crawler.ts#L292) optionalinheritedmaxRequestsPerMinute maxRequestsPerMinute?: number The maximum number of requests per minute the crawler should run. By default, this is set to `Infinity`, but we can pass any positive, non-zero integer. Shortcut for the AutoscaledPool [`maxTasksPerMinute`](/js/api/core/interface/AutoscaledPoolOptions#maxTasksPerMinute) option. ### [](#maxSessionRotations) [](https://github.com/apify/crawlee/blob/master/packages/basic-crawler/src/internals/basic-crawler.ts#L255) optionalinheritedmaxSessionRotations maxSessionRotations?: number = 10 Maximum number of session rotations per request. The crawler will automatically rotate the session in case of a proxy error or if it gets blocked by the website. The session rotations are not counted towards the [`maxRequestRetries`](/js/api/basic-crawler/interface/BasicCrawlerOptions#maxRequestRetries) limit. ### [](#minConcurrency) [](https://github.com/apify/crawlee/blob/master/packages/basic-crawler/src/internals/basic-crawler.ts#L279) optionalinheritedminConcurrency minConcurrency?: number Sets the minimum concurrency (parallelism) for the crawl. Shortcut for the AutoscaledPool [`minConcurrency`](/js/api/core/interface/AutoscaledPoolOptions#minConcurrency) option. > _WARNING:_ If we set this value too high with respect to the available system memory and CPU, our crawler will run extremely slow or crash. If not sure, it's better to keep the default value and the concurrency will scale up automatically. ### [](#navigationTimeoutSecs) [](https://github.com/apify/crawlee/blob/master/packages/browser-crawler/src/internals/browser-crawler.ts#L244) optionalinheritednavigationTimeoutSecs navigationTimeoutSecs?: number Timeout in which page navigation needs to finish, in seconds. ### [](#persistCookiesPerSession) [](https://github.com/apify/crawlee/blob/master/packages/browser-crawler/src/internals/browser-crawler.ts#L250) optionalinheritedpersistCookiesPerSession persistCookiesPerSession?: boolean Defines whether the cookies should be persisted for sessions. This can only be used when `useSessionPool` is set to `true`. ### [](#postNavigationHooks) [](https://github.com/apify/crawlee/blob/master/packages/playwright-crawler/src/internals/playwright-crawler.ts#L124) optionalpostNavigationHooks postNavigationHooks?: [PlaywrightHook](/js/api/playwright-crawler/interface/PlaywrightHook) \[\] Async functions that are sequentially evaluated after the navigation. Good for checking if the navigation was successful. The function accepts `crawlingContext` as the only parameter. Example: postNavigationHooks: [ async (crawlingContext) => { const { page } = crawlingContext; if (hasCaptcha(page)) { await solveCaptcha (page); } },] ### [](#preNavigationHooks) [](https://github.com/apify/crawlee/blob/master/packages/playwright-crawler/src/internals/playwright-crawler.ts#L107) optionalpreNavigationHooks preNavigationHooks?: [PlaywrightHook](/js/api/playwright-crawler/interface/PlaywrightHook) \[\] Async functions that are sequentially evaluated before the navigation. Good for setting additional cookies or browser properties before navigation. The function accepts two parameters, `crawlingContext` and `gotoOptions`, which are passed to the `page.goto()` function the crawler calls to navigate. Example: preNavigationHooks: [ async (crawlingContext, gotoOptions) => { const { page } = crawlingContext; await page.evaluate((attr) => { window.foo = attr; }, 'bar'); },] Modyfing `pageOptions` is supported only in Playwright incognito. See [PrePageCreateHook](/js/api/browser-pool#PrePageCreateHook) ### [](#proxyConfiguration) [](https://github.com/apify/crawlee/blob/master/packages/browser-crawler/src/internals/browser-crawler.ts#L197) optionalinheritedproxyConfiguration proxyConfiguration?: [ProxyConfiguration](/js/api/core/class/ProxyConfiguration) If set, the crawler will be configured for all connections to use the Proxy URLs provided and rotated according to the configuration. ### [](#requestHandler) [](https://github.com/apify/crawlee/blob/master/packages/playwright-crawler/src/internals/playwright-crawler.ts#L59) optionalrequestHandler requestHandler?: [PlaywrightRequestHandler](/js/api/playwright-crawler/interface/PlaywrightRequestHandler) Function that is called to process each request. The function receives the [PlaywrightCrawlingContext](/js/api/playwright-crawler/interface/PlaywrightCrawlingContext) as an argument, where: * `request` is an instance of the [Request](/js/api/core/class/Request) object with details about the URL to open, HTTP method etc. * `page` is an instance of the `Playwright` [`Page`](https://playwright.dev/docs/api/class-page) * `browserController` is an instance of the [`BrowserController`](https://github.com/apify/browser-pool#browsercontroller) , * `response` is an instance of the `Playwright` [`Response`](https://playwright.dev/docs/api/class-response) , which is the main resource response as returned by `page.goto(request.url)`. The function must return a promise, which is then awaited by the crawler. If the function throws an exception, the crawler will try to re-crawl the request later, up to `option.maxRequestRetries` times. If all the retries fail, the crawler calls the function provided to the `failedRequestHandler` parameter. To make this work, you should **always** let your function throw exceptions rather than catch them. The exceptions are logged to the request using the [Request.pushErrorMessage](/js/api/core/class/Request#pushErrorMessage) function. ### [](#requestHandlerTimeoutSecs) [](https://github.com/apify/crawlee/blob/master/packages/basic-crawler/src/internals/basic-crawler.ts#L192) optionalinheritedrequestHandlerTimeoutSecs requestHandlerTimeoutSecs?: number = 60 Timeout in which the function passed as [`requestHandler`](/js/api/basic-crawler/interface/BasicCrawlerOptions#requestHandler) needs to finish, in seconds. ### [](#requestList) [](https://github.com/apify/crawlee/blob/master/packages/basic-crawler/src/internals/basic-crawler.ts#L178) optionalinheritedrequestList requestList?: [IRequestList](/js/api/core/interface/IRequestList) Static list of URLs to be processed. If not provided, the crawler will open the default request queue when the [`crawler.addRequests()`](/js/api/basic-crawler/class/BasicCrawler#addRequests) function is called. > Alternatively, `requests` parameter of [`crawler.run()`](/js/api/basic-crawler/class/BasicCrawler#run) > could be used to enqueue the initial requests - it is a shortcut for running `crawler.addRequests()` before the `crawler.run()`. ### [](#requestQueue) [](https://github.com/apify/crawlee/blob/master/packages/basic-crawler/src/internals/basic-crawler.ts#L186) optionalinheritedrequestQueue requestQueue?: [RequestProvider](/js/api/core/class/RequestProvider) Dynamic queue of URLs to be processed. This is useful for recursive crawling of websites. If not provided, the crawler will open the default request queue when the [`crawler.addRequests()`](/js/api/basic-crawler/class/BasicCrawler#addRequests) function is called. > Alternatively, `requests` parameter of [`crawler.run()`](/js/api/basic-crawler/class/BasicCrawler#run) > could be used to enqueue the initial requests - it is a shortcut for running `crawler.addRequests()` before the `crawler.run()`. ### [](#retryOnBlocked) [](https://github.com/apify/crawlee/blob/master/packages/basic-crawler/src/internals/basic-crawler.ts#L342) optionalinheritedretryOnBlocked retryOnBlocked?: boolean If set to `true`, the crawler will automatically try to bypass any detected bot protection. Currently supports: * [**Cloudflare** Bot Management](https://www.cloudflare.com/products/bot-management/) * [**Google Search** Rate Limiting](https://www.google.com/sorry/) ### [](#sameDomainDelaySecs) [](https://github.com/apify/crawlee/blob/master/packages/basic-crawler/src/internals/basic-crawler.ts#L246) optionalinheritedsameDomainDelaySecs sameDomainDelaySecs?: number = 0 Indicates how much time (in seconds) to wait before crawling another same domain request. ### [](#sessionPoolOptions) [](https://github.com/apify/crawlee/blob/master/packages/basic-crawler/src/internals/basic-crawler.ts#L310) optionalinheritedsessionPoolOptions sessionPoolOptions?: [SessionPoolOptions](/js/api/core/interface/SessionPoolOptions) The configuration options for [SessionPool](/js/api/core/class/SessionPool) to use. ### [](#statisticsOptions) [](https://github.com/apify/crawlee/blob/master/packages/basic-crawler/src/internals/basic-crawler.ts#L357) optionalinheritedstatisticsOptions statisticsOptions?: [StatisticsOptions](/js/api/core/interface/StatisticsOptions) Customize the way statistics collecting works, such as logging interval or whether to output them to the Key-Value store. ### [](#statusMessageCallback) [](https://github.com/apify/crawlee/blob/master/packages/basic-crawler/src/internals/basic-crawler.ts#L333) optionalinheritedstatusMessageCallback statusMessageCallback?: [StatusMessageCallback](/js/api/basic-crawler#StatusMessageCallback) <[BasicCrawlingContext](/js/api/basic-crawler/interface/BasicCrawlingContext) , [BasicCrawler](/js/api/basic-crawler/class/BasicCrawler) <[BasicCrawlingContext](/js/api/basic-crawler/interface/BasicCrawlingContext) \>\> Allows overriding the default status message. The callback needs to call `crawler.setStatusMessage()` explicitly. The default status message is provided in the parameters. const crawler = new CheerioCrawler({ statusMessageCallback: async (ctx) => { return ctx.crawler.setStatusMessage(`this is status message from ${new Date().toISOString()}`, { level: 'INFO' }); // log level defaults to 'DEBUG' }, statusMessageLoggingInterval: 1, // defaults to 10s async requestHandler({ $, enqueueLinks, request, log }) { // ... },}); ### [](#statusMessageLoggingInterval) [](https://github.com/apify/crawlee/blob/master/packages/basic-crawler/src/internals/basic-crawler.ts#L315) optionalinheritedstatusMessageLoggingInterval statusMessageLoggingInterval?: number Defines the length of the interval for calling the `setStatusMessage` in seconds. ### [](#useSessionPool) [](https://github.com/apify/crawlee/blob/master/packages/basic-crawler/src/internals/basic-crawler.ts#L305) optionalinheriteduseSessionPool useSessionPool?: boolean Basic crawler will initialize the [SessionPool](/js/api/core/class/SessionPool) with the corresponding [`sessionPoolOptions`](/js/api/core/interface/SessionPoolOptions) . The session instance will be than available in the [`requestHandler`](/js/api/basic-crawler/interface/BasicCrawlerOptions#requestHandler) . **Page Options** Hide Inherited * [autoscaledPoolOptions](#autoscaledPoolOptions) * [browserPoolOptions](#browserPoolOptions) * [errorHandler](#errorHandler) * [experiments](#experiments) * [failedRequestHandler](#failedRequestHandler) * [headless](#headless) * [httpClient](#httpClient) * [ignoreIframes](#ignoreIframes) * [ignoreShadowRoots](#ignoreShadowRoots) * [keepAlive](#keepAlive) * [launchContext](#launchContext) * [maxConcurrency](#maxConcurrency) * [maxRequestRetries](#maxRequestRetries) * [maxRequestsPerCrawl](#maxRequestsPerCrawl) * [maxRequestsPerMinute](#maxRequestsPerMinute) * [maxSessionRotations](#maxSessionRotations) * [minConcurrency](#minConcurrency) * [navigationTimeoutSecs](#navigationTimeoutSecs) * [persistCookiesPerSession](#persistCookiesPerSession) * [postNavigationHooks](#postNavigationHooks) * [preNavigationHooks](#preNavigationHooks) * [proxyConfiguration](#proxyConfiguration) * [requestHandler](#requestHandler) * [requestHandlerTimeoutSecs](#requestHandlerTimeoutSecs) * [requestList](#requestList) * [requestQueue](#requestQueue) * [retryOnBlocked](#retryOnBlocked) * [sameDomainDelaySecs](#sameDomainDelaySecs) * [sessionPoolOptions](#sessionPoolOptions) * [statisticsOptions](#statisticsOptions) * [statusMessageCallback](#statusMessageCallback) * [statusMessageLoggingInterval](#statusMessageLoggingInterval) * [useSessionPool](#useSessionPool) --- # PlaywrightRequestHandler | API | Crawlee for JavaScript · Build reliable crawlers. Fast. [Skip to main content](#__docusaurus_skipToContent_fallback) Version: 3.13 ### Hierarchy * [BrowserRequestHandler](/js/api/browser-crawler#BrowserRequestHandler) \> * _PlaywrightRequestHandler_ ### Callable * **PlaywrightRequestHandler**(inputs): Awaitable * * * * #### Parameters * ##### inputs: { request: [LoadedRequest](/js/api/core#LoadedRequest) <[LoadedRequest](/js/api/core#LoadedRequest) <[Request](/js/api/core/class/Request) \>\> } & Omit<{ request: [LoadedRequest](/js/api/core#LoadedRequest) <[Request](/js/api/core/class/Request) \> } & Omit<[PlaywrightCrawlingContext](/js/api/playwright-crawler/interface/PlaywrightCrawlingContext) , request\>, request\> #### Returns Awaitable --- # PlaywrightLaunchContext | API | Crawlee for JavaScript · Build reliable crawlers. Fast. [Skip to main content](#__docusaurus_skipToContent_fallback) Version: 3.13 On this page Apify extends the launch options of Playwright. You can use any of the Playwright compatible [`LaunchOptions`](https://playwright.dev/docs/api/class-browsertype#browsertypelaunchoptions) options by providing the `launchOptions` property. **Example:** // launch a headless Chrome (not Chromium)const launchContext = { // Apify helpers useChrome: true, proxyUrl: 'http://user:password@some.proxy.com' // Native Playwright options launchOptions: { headless: true, args: ['--some-flag'], }} ### Hierarchy * [BrowserLaunchContext](/js/api/browser-crawler/interface/BrowserLaunchContext) * _PlaywrightLaunchContext_ Index[](#Index) ---------------- ### Properties * [browserPerProxy](#browserPerProxy) * [experimentalContainers](#experimentalContainers) * [launcher](#launcher) * [launchOptions](#launchOptions) * [proxyUrl](#proxyUrl) * [useChrome](#useChrome) * [useIncognitoPages](#useIncognitoPages) * [userAgent](#userAgent) * [userDataDir](#userDataDir) Properties[](#Properties) -------------------------- ### [](#browserPerProxy) [](https://github.com/apify/crawlee/blob/master/packages/browser-crawler/src/internals/browser-launcher.ts#L40) optionalinheritedbrowserPerProxy browserPerProxy?: boolean If set to `true`, the crawler respects the proxy url generated for the given request. This aligns the browser-based crawlers with the `HttpCrawler`. Might cause performance issues, as Crawlee might launch too many browser instances. ### [](#experimentalContainers) [](https://github.com/apify/crawlee/blob/master/packages/playwright-crawler/src/internals/playwright-launcher.ts#L66) optionalexperimentalContainers experimental experimentalContainers?: boolean Like `useIncognitoPages`, but for persistent contexts, so cache is used for faster loading. Works best with Firefox. Unstable on Chromium. ### [](#launcher) [](https://github.com/apify/crawlee/blob/master/packages/playwright-crawler/src/internals/playwright-launcher.ts#L79) optionallauncher launcher?: BrowserType<{}\> By default this function uses `require("playwright").chromium`. If you want to use a different browser you can pass it by this property as e.g. `require("playwright").firefox` ### [](#launchOptions) [](https://github.com/apify/crawlee/blob/master/packages/playwright-crawler/src/internals/playwright-launcher.ts#L33) optionallaunchOptions launchOptions?: LaunchOptions & { acceptDownloads?: boolean; args?: string\[\]; baseURL?: string; bypassCSP?: boolean; channel?: string; chromiumSandbox?: boolean; clientCertificates?: { cert?: Buffer; certPath?: string; key?: Buffer; keyPath?: string; origin: string; passphrase?: string; pfx?: Buffer; pfxPath?: string }\[\]; colorScheme?: null | light | dark | no-preference; deviceScaleFactor?: number; devtools?: boolean; downloadsPath?: string; env?: {}; executablePath?: string; extraHTTPHeaders?: {}; firefoxUserPrefs?: {}; forcedColors?: null | active | none; geolocation?: { accuracy?: number; latitude: number; longitude: number }; handleSIGHUP?: boolean; handleSIGINT?: boolean; handleSIGTERM?: boolean; hasTouch?: boolean; headless?: boolean; httpCredentials?: { origin?: string; password: string; send?: unauthorized | always; username: string }; ignoreDefaultArgs?: boolean | string\[\]; ignoreHTTPSErrors?: boolean; isMobile?: boolean; javaScriptEnabled?: boolean; locale?: string; logger?: Logger; offline?: boolean; permissions?: string\[\]; proxy?: { bypass?: string; password?: string; server: string; username?: string }; recordHar?: { content?: omit | embed | attach; mode?: full | minimal; omitContent?: boolean; path: string; urlFilter?: string | RegExp }; recordVideo?: { dir: string; size?: { height: number; width: number } }; reducedMotion?: null | reduce | no-preference; screen?: { height: number; width: number }; serviceWorkers?: allow | block; slowMo?: number; strictSelectors?: boolean; timeout?: number; timezoneId?: string; tracesDir?: string; userAgent?: string; videoSize?: { height: number; width: number }; videosPath?: string; viewport?: null | { height: number; width: number } } `browserType.launch` [options](https://playwright.dev/docs/api/class-browsertype#browser-type-launch) or `browserType.launchContextOptions` [options](https://playwright.dev/docs/api/class-browsertype#browser-type-launch-persistent-context) ### [](#proxyUrl) [](https://github.com/apify/crawlee/blob/master/packages/playwright-crawler/src/internals/playwright-launcher.ts#L41) optionalproxyUrl proxyUrl?: string URL to a HTTP proxy server. It must define the port number, and it may also contain proxy username and password. Example: `http://bob:pass123@proxy.example.com:1234`. ### [](#useChrome) [](https://github.com/apify/crawlee/blob/master/packages/playwright-crawler/src/internals/playwright-launcher.ts#L52) optionaluseChrome useChrome?: boolean = false If `true` and `executablePath` is not set, Playwright will launch full Google Chrome browser available on the machine rather than the bundled Chromium. The path to Chrome executable is taken from the `CRAWLEE_CHROME_EXECUTABLE_PATH` environment variable if provided, or defaults to the typical Google Chrome executable location specific for the operating system. By default, this option is `false`. ### [](#useIncognitoPages) [](https://github.com/apify/crawlee/blob/master/packages/playwright-crawler/src/internals/playwright-launcher.ts#L59) optionaluseIncognitoPages useIncognitoPages?: boolean = false With this option selected, all pages will be opened in a new incognito browser context. This means they will not share cookies nor cache and their resources will not be throttled by one another. ### [](#userAgent) [](https://github.com/apify/crawlee/blob/master/packages/browser-crawler/src/internals/browser-launcher.ts#L68) optionalinheriteduserAgent userAgent?: string The `User-Agent` HTTP header used by the browser. If not provided, the function sets `User-Agent` to a reasonable default to reduce the chance of detection of the crawler. ### [](#userDataDir) [](https://github.com/apify/crawlee/blob/master/packages/playwright-crawler/src/internals/playwright-launcher.ts#L73) optionaluserDataDir userDataDir?: string Sets the [User Data Directory](https://chromium.googlesource.com/chromium/src/+/master/docs/user_data_dir.md) path. The user data directory contains profile data such as history, bookmarks, and cookies, as well as other per-installation local state. If not specified, a temporary directory is used instead. **Page Options** Hide Inherited * [browserPerProxy](#browserPerProxy) * [experimentalContainers](#experimentalContainers) * [launcher](#launcher) * [launchOptions](#launchOptions) * [proxyUrl](#proxyUrl) * [useChrome](#useChrome) * [useIncognitoPages](#useIncognitoPages) * [userAgent](#userAgent) * [userDataDir](#userDataDir) --- # Changelog | API | Crawlee for JavaScript · Build reliable crawlers. Fast. [Skip to main content](#__docusaurus_skipToContent_fallback) Version: 3.13 On this page All notable changes to this project will be documented in this file. See [Conventional Commits](https://conventionalcommits.org) for commit guidelines. ### Features[​](#features "Direct link to Features") * **playwright:** add `handleCloudflareChallenge` helper ([#2865](https://github.com/apify/crawlee/issues/2865) ) ([9a1725f](https://github.com/apify/crawlee/commit/9a1725f7b87fb70194fc31858500cb35639fb964) ) [3.12.2](https://github.com/apify/crawlee/compare/v3.12.1...v3.12.2) (2025-01-27)[​](#3122-2025-01-27 "Direct link to 3122-2025-01-27") ----------------------------------------------------------------------------------------------------------------------------------------- **Note:** Version bump only for package @crawlee/playwright [3.12.1](https://github.com/apify/crawlee/compare/v3.12.0...v3.12.1) (2024-12-04)[​](#3121-2024-12-04 "Direct link to 3121-2024-12-04") ----------------------------------------------------------------------------------------------------------------------------------------- **Note:** Version bump only for package @crawlee/playwright [3.12.0](https://github.com/apify/crawlee/compare/v3.11.5...v3.12.0) (2024-11-04) ================================================================================== ### Bug Fixes[​](#bug-fixes "Direct link to Bug Fixes") * ignore errors from iframe content extraction ([#2714](https://github.com/apify/crawlee/issues/2714) ) ([627e5c2](https://github.com/apify/crawlee/commit/627e5c2fbadce63c7e631217cd0e735597c0ce08) ), closes [#2708](https://github.com/apify/crawlee/issues/2708) [3.11.5](https://github.com/apify/crawlee/compare/v3.11.4...v3.11.5) (2024-10-04)[​](#3115-2024-10-04 "Direct link to 3115-2024-10-04") ----------------------------------------------------------------------------------------------------------------------------------------- ### Bug Fixes[​](#bug-fixes-1 "Direct link to Bug Fixes") * **core:** accept `UInt8Array` in `KVS.setValue()` ([#2682](https://github.com/apify/crawlee/issues/2682) ) ([8ef0e60](https://github.com/apify/crawlee/commit/8ef0e60ca6fb2f4ec1b0d1aec6dcd53fcfb398b3) ) [3.11.4](https://github.com/apify/crawlee/compare/v3.11.3...v3.11.4) (2024-09-23)[​](#3114-2024-09-23 "Direct link to 3114-2024-09-23") ----------------------------------------------------------------------------------------------------------------------------------------- **Note:** Version bump only for package @crawlee/playwright [3.11.3](https://github.com/apify/crawlee/compare/v3.11.2...v3.11.3) (2024-09-03)[​](#3113-2024-09-03 "Direct link to 3113-2024-09-03") ----------------------------------------------------------------------------------------------------------------------------------------- **Note:** Version bump only for package @crawlee/playwright [3.11.2](https://github.com/apify/crawlee/compare/v3.11.1...v3.11.2) (2024-08-28)[​](#3112-2024-08-28 "Direct link to 3112-2024-08-28") ----------------------------------------------------------------------------------------------------------------------------------------- **Note:** Version bump only for package @crawlee/playwright [3.11.1](https://github.com/apify/crawlee/compare/v3.11.0...v3.11.1) (2024-07-24)[​](#3111-2024-07-24 "Direct link to 3111-2024-07-24") ----------------------------------------------------------------------------------------------------------------------------------------- **Note:** Version bump only for package @crawlee/playwright [3.11.0](https://github.com/apify/crawlee/compare/v3.10.5...v3.11.0) (2024-07-09) ================================================================================== ### Features[​](#features-1 "Direct link to Features") * add `iframe` expansion to `parseWithCheerio` in browsers ([#2542](https://github.com/apify/crawlee/issues/2542) ) ([328d085](https://github.com/apify/crawlee/commit/328d08598807782b3712bd543e394fe9a000a85d) ), closes [#2507](https://github.com/apify/crawlee/issues/2507) * add `ignoreIframes` opt-out from the Cheerio iframe expansion ([#2562](https://github.com/apify/crawlee/issues/2562) ) ([474a8dc](https://github.com/apify/crawlee/commit/474a8dc06a567cde0651d385fdac9c350ddf4508) ) [3.10.5](https://github.com/apify/crawlee/compare/v3.10.4...v3.10.5) (2024-06-12)[​](#3105-2024-06-12 "Direct link to 3105-2024-06-12") ----------------------------------------------------------------------------------------------------------------------------------------- ### Bug Fixes[​](#bug-fixes-2 "Direct link to Bug Fixes") * allow creating new adaptive crawler instance without any parameters ([9b7f595](https://github.com/apify/crawlee/commit/9b7f595a2d70cab5c50e188581b21b0ef7e51780) ) * fix detection of HTTP site when using the `useState` in adaptive crawler ([#2530](https://github.com/apify/crawlee/issues/2530) ) ([7e195c1](https://github.com/apify/crawlee/commit/7e195c17cf1d9beae7f6f068fe505f1334a3a5b3) ) * mark `context.request.loadedUrl` and `id` as required inside the request handler ([#2531](https://github.com/apify/crawlee/issues/2531) ) ([2b54660](https://github.com/apify/crawlee/commit/2b546600691d84852a2f9ef42f273cecf818d66d) ) [3.10.4](https://github.com/apify/crawlee/compare/v3.10.3...v3.10.4) (2024-06-11)[​](#3104-2024-06-11 "Direct link to 3104-2024-06-11") ----------------------------------------------------------------------------------------------------------------------------------------- ### Bug Fixes[​](#bug-fixes-3 "Direct link to Bug Fixes") * **playwright:** allow passing new context options in `launchOptions` on type level ([0519d40](https://github.com/apify/crawlee/commit/0519d4099d257bbc40ed091c131a674ea5f8d731) ), closes [#1849](https://github.com/apify/crawlee/issues/1849) [3.10.3](https://github.com/apify/crawlee/compare/v3.10.2...v3.10.3) (2024-06-07)[​](#3103-2024-06-07 "Direct link to 3103-2024-06-07") ----------------------------------------------------------------------------------------------------------------------------------------- ### Bug Fixes[​](#bug-fixes-4 "Direct link to Bug Fixes") * **adaptive-crawler:** log only once for the committed request handler execution ([#2524](https://github.com/apify/crawlee/issues/2524) ) ([533bd3f](https://github.com/apify/crawlee/commit/533bd3f04671d54273f0861664d316269d08fbfb) ) * respect implicit router when no `requestHandler` is provided in `AdaptiveCrawler` ([#2518](https://github.com/apify/crawlee/issues/2518) ) ([31083aa](https://github.com/apify/crawlee/commit/31083aa27ddd51827f73c7ac4290379ec7a81283) ) ### Features[​](#features-2 "Direct link to Features") * add `waitForSelector` context helper + `parseWithCheerio` in adaptive crawler ([#2522](https://github.com/apify/crawlee/issues/2522) ) ([6f88e73](https://github.com/apify/crawlee/commit/6f88e738d43ab4774dc4ef3f78775a5d88728e0d) ) [3.10.2](https://github.com/apify/crawlee/compare/v3.10.1...v3.10.2) (2024-06-03)[​](#3102-2024-06-03 "Direct link to 3102-2024-06-03") ----------------------------------------------------------------------------------------------------------------------------------------- **Note:** Version bump only for package @crawlee/playwright [3.10.1](https://github.com/apify/crawlee/compare/v3.10.0...v3.10.1) (2024-05-23)[​](#3101-2024-05-23 "Direct link to 3101-2024-05-23") ----------------------------------------------------------------------------------------------------------------------------------------- **Note:** Version bump only for package @crawlee/playwright [3.10.0](https://github.com/apify/crawlee/compare/v3.9.2...v3.10.0) (2024-05-16) ================================================================================= ### Bug Fixes[​](#bug-fixes-5 "Direct link to Bug Fixes") * do not drop statistics on migration/resurrection/resume ([#2462](https://github.com/apify/crawlee/issues/2462) ) ([8ce7dd4](https://github.com/apify/crawlee/commit/8ce7dd4ae6a3718dac95e784a53bd5661c827edc) ) [3.9.2](https://github.com/apify/crawlee/compare/v3.9.1...v3.9.2) (2024-04-17)[​](#392-2024-04-17 "Direct link to 392-2024-04-17") ------------------------------------------------------------------------------------------------------------------------------------ **Note:** Version bump only for package @crawlee/playwright [3.9.1](https://github.com/apify/crawlee/compare/v3.9.0...v3.9.1) (2024-04-11)[​](#391-2024-04-11 "Direct link to 391-2024-04-11") ------------------------------------------------------------------------------------------------------------------------------------ **Note:** Version bump only for package @crawlee/playwright [3.9.0](https://github.com/apify/crawlee/compare/v3.8.2...v3.9.0) (2024-04-10) =============================================================================== ### Features[​](#features-3 "Direct link to Features") * `createAdaptivePlaywrightRouter` utility ([#2415](https://github.com/apify/crawlee/issues/2415) ) ([cee4778](https://github.com/apify/crawlee/commit/cee477814e4901d025c5376205ad884c2fe08e0e) ), closes [#2407](https://github.com/apify/crawlee/issues/2407) * expand #shadow-root elements automatically in `parseWithCheerio` helper ([#2396](https://github.com/apify/crawlee/issues/2396) ) ([a05b3a9](https://github.com/apify/crawlee/commit/a05b3a93a9b57926b353df0e79d846b5024c42ac) ) [3.8.2](https://github.com/apify/crawlee/compare/v3.8.1...v3.8.2) (2024-03-21)[​](#382-2024-03-21 "Direct link to 382-2024-03-21") ------------------------------------------------------------------------------------------------------------------------------------ ### Features[​](#features-4 "Direct link to Features") * implement global storage access checking and use it to prevent unwanted side effects in adaptive crawler ([#2371](https://github.com/apify/crawlee/issues/2371) ) ([fb3b7da](https://github.com/apify/crawlee/commit/fb3b7da402522ddff8c7394ac1253ba8aeac984c) ), closes [#2364](https://github.com/apify/crawlee/issues/2364) [3.8.1](https://github.com/apify/crawlee/compare/v3.8.0...v3.8.1) (2024-02-22)[​](#381-2024-02-22 "Direct link to 381-2024-02-22") ------------------------------------------------------------------------------------------------------------------------------------ **Note:** Version bump only for package @crawlee/playwright [3.8.0](https://github.com/apify/crawlee/compare/v3.7.3...v3.8.0) (2024-02-21) =============================================================================== ### Features[​](#features-5 "Direct link to Features") * adaptive playwright crawler ([#2316](https://github.com/apify/crawlee/issues/2316) ) ([8e4218a](https://github.com/apify/crawlee/commit/8e4218ada03cf485751def46f8c465b2d2a825c7) ) [3.7.3](https://github.com/apify/crawlee/compare/v3.7.2...v3.7.3) (2024-01-30)[​](#373-2024-01-30 "Direct link to 373-2024-01-30") ------------------------------------------------------------------------------------------------------------------------------------ **Note:** Version bump only for package @crawlee/playwright [3.7.2](https://github.com/apify/crawlee/compare/v3.7.1...v3.7.2) (2024-01-09)[​](#372-2024-01-09 "Direct link to 372-2024-01-09") ------------------------------------------------------------------------------------------------------------------------------------ **Note:** Version bump only for package @crawlee/playwright [3.7.1](https://github.com/apify/crawlee/compare/v3.7.0...v3.7.1) (2024-01-02)[​](#371-2024-01-02 "Direct link to 371-2024-01-02") ------------------------------------------------------------------------------------------------------------------------------------ **Note:** Version bump only for package @crawlee/playwright [3.7.0](https://github.com/apify/crawlee/compare/v3.6.2...v3.7.0) (2023-12-21) =============================================================================== **Note:** Version bump only for package @crawlee/playwright [3.6.2](https://github.com/apify/crawlee/compare/v3.6.1...v3.6.2) (2023-11-26)[​](#362-2023-11-26 "Direct link to 362-2023-11-26") ------------------------------------------------------------------------------------------------------------------------------------ **Note:** Version bump only for package @crawlee/playwright [3.6.1](https://github.com/apify/crawlee/compare/v3.6.0...v3.6.1) (2023-11-15)[​](#361-2023-11-15 "Direct link to 361-2023-11-15") ------------------------------------------------------------------------------------------------------------------------------------ ### Features[​](#features-6 "Direct link to Features") * **puppeteer:** enable `new` headless mode ([#1910](https://github.com/apify/crawlee/issues/1910) ) ([7fc999c](https://github.com/apify/crawlee/commit/7fc999cf4658ca69b97f16d434444081998470f4) ) [3.6.0](https://github.com/apify/crawlee/compare/v3.5.8...v3.6.0) (2023-11-15) =============================================================================== ### Bug Fixes[​](#bug-fixes-6 "Direct link to Bug Fixes") * add `skipNavigation` option to `enqueueLinks` ([#2153](https://github.com/apify/crawlee/issues/2153) ) ([118515d](https://github.com/apify/crawlee/commit/118515d2ba534b99be2f23436f6abe41d66a8e07) ) [3.5.8](https://github.com/apify/crawlee/compare/v3.5.7...v3.5.8) (2023-10-17)[​](#358-2023-10-17 "Direct link to 358-2023-10-17") ------------------------------------------------------------------------------------------------------------------------------------ **Note:** Version bump only for package @crawlee/playwright [3.5.7](https://github.com/apify/crawlee/compare/v3.5.6...v3.5.7) (2023-10-05)[​](#357-2023-10-05 "Direct link to 357-2023-10-05") ------------------------------------------------------------------------------------------------------------------------------------ **Note:** Version bump only for package @crawlee/playwright [3.5.6](https://github.com/apify/crawlee/compare/v3.5.5...v3.5.6) (2023-10-04)[​](#356-2023-10-04 "Direct link to 356-2023-10-04") ------------------------------------------------------------------------------------------------------------------------------------ **Note:** Version bump only for package @crawlee/playwright [3.5.5](https://github.com/apify/crawlee/compare/v3.5.4...v3.5.5) (2023-10-02)[​](#355-2023-10-02 "Direct link to 355-2023-10-02") ------------------------------------------------------------------------------------------------------------------------------------ ### Bug Fixes[​](#bug-fixes-7 "Direct link to Bug Fixes") * allow to use any version of puppeteer or playwright ([#2102](https://github.com/apify/crawlee/issues/2102) ) ([0cafceb](https://github.com/apify/crawlee/commit/0cafceb2966d430dd1b2a1b619fe66da1c951f4c) ), closes [#2101](https://github.com/apify/crawlee/issues/2101) ### Features[​](#features-7 "Direct link to Features") * Request Queue v2 ([#1975](https://github.com/apify/crawlee/issues/1975) ) ([70a77ee](https://github.com/apify/crawlee/commit/70a77ee15f984e9ae67cd584fc58ace7e55346db) ), closes [#1365](https://github.com/apify/crawlee/issues/1365) [3.5.4](https://github.com/apify/crawlee/compare/v3.5.3...v3.5.4) (2023-09-11)[​](#354-2023-09-11 "Direct link to 354-2023-09-11") ------------------------------------------------------------------------------------------------------------------------------------ ### Bug Fixes[​](#bug-fixes-8 "Direct link to Bug Fixes") * various helpers opening KVS now respect Configuration ([#2071](https://github.com/apify/crawlee/issues/2071) ) ([59dbb16](https://github.com/apify/crawlee/commit/59dbb164699774e5a6718e98d0a4e8f630f35323) ) [3.5.3](https://github.com/apify/crawlee/compare/v3.5.2...v3.5.3) (2023-08-31)[​](#353-2023-08-31 "Direct link to 353-2023-08-31") ------------------------------------------------------------------------------------------------------------------------------------ ### Bug Fixes[​](#bug-fixes-9 "Direct link to Bug Fixes") * pin all internal dependencies ([#2041](https://github.com/apify/crawlee/issues/2041) ) ([d6f2b17](https://github.com/apify/crawlee/commit/d6f2b172d4a6776137c7893ca798d5b4a9408e79) ), closes [#2040](https://github.com/apify/crawlee/issues/2040) [3.5.2](https://github.com/apify/crawlee/compare/v3.5.1...v3.5.2) (2023-08-21)[​](#352-2023-08-21 "Direct link to 352-2023-08-21") ------------------------------------------------------------------------------------------------------------------------------------ **Note:** Version bump only for package @crawlee/playwright [3.5.1](https://github.com/apify/crawlee/compare/v3.5.0...v3.5.1) (2023-08-16)[​](#351-2023-08-16 "Direct link to 351-2023-08-16") ------------------------------------------------------------------------------------------------------------------------------------ **Note:** Version bump only for package @crawlee/playwright [3.5.0](https://github.com/apify/crawlee/compare/v3.4.2...v3.5.0) (2023-07-31) =============================================================================== ### Features[​](#features-8 "Direct link to Features") * add `closeCookieModals` context helper for Playwright and Puppeteer ([#1927](https://github.com/apify/crawlee/issues/1927) ) ([98d93bb](https://github.com/apify/crawlee/commit/98d93bb6713ec219baa83db2ad2cd1d7621a3339) ) * **core:** use `RequestQueue.addBatchedRequests()` in `enqueueLinks` helper ([4d61ca9](https://github.com/apify/crawlee/commit/4d61ca934072f8bbb680c842d8b1c9a4452ee73a) ), closes [#1995](https://github.com/apify/crawlee/issues/1995) [3.4.2](https://github.com/apify/crawlee/compare/v3.4.1...v3.4.2) (2023-07-19)[​](#342-2023-07-19 "Direct link to 342-2023-07-19") ------------------------------------------------------------------------------------------------------------------------------------ **Note:** Version bump only for package @crawlee/playwright [3.4.1](https://github.com/apify/crawlee/compare/v3.4.0...v3.4.1) (2023-07-13)[​](#341-2023-07-13 "Direct link to 341-2023-07-13") ------------------------------------------------------------------------------------------------------------------------------------ **Note:** Version bump only for package @crawlee/playwright [3.4.0](https://github.com/apify/crawlee/compare/v3.3.3...v3.4.0) (2023-06-12) =============================================================================== ### Features[​](#features-9 "Direct link to Features") * infiniteScroll has maxScrollHeight limit ([#1945](https://github.com/apify/crawlee/issues/1945) ) ([44997bb](https://github.com/apify/crawlee/commit/44997bba5bbf33ddb7dbac2f3e26d4bee60d4f47) ) [3.3.3](https://github.com/apify/crawlee/compare/v3.3.2...v3.3.3) (2023-05-31)[​](#333-2023-05-31 "Direct link to 333-2023-05-31") ------------------------------------------------------------------------------------------------------------------------------------ **Note:** Version bump only for package @crawlee/playwright [3.3.2](https://github.com/apify/crawlee/compare/v3.3.1...v3.3.2) (2023-05-11)[​](#332-2023-05-11 "Direct link to 332-2023-05-11") ------------------------------------------------------------------------------------------------------------------------------------ ### Features[​](#features-10 "Direct link to Features") * **router:** allow inline router definition ([#1877](https://github.com/apify/crawlee/issues/1877) ) ([2d241c9](https://github.com/apify/crawlee/commit/2d241c9f88964ebd41a181069c378b6b7b5bf262) ) [3.3.1](https://github.com/apify/crawlee/compare/v3.3.0...v3.3.1) (2023-04-11)[​](#331-2023-04-11 "Direct link to 331-2023-04-11") ------------------------------------------------------------------------------------------------------------------------------------ ### Bug Fixes[​](#bug-fixes-10 "Direct link to Bug Fixes") * infiniteScroll() not working in Firefox ([#1826](https://github.com/apify/crawlee/issues/1826) ) ([4286c5d](https://github.com/apify/crawlee/commit/4286c5d29b94aec3f4d3835bbf36b7fafcaec8f0) ), closes [#1821](https://github.com/apify/crawlee/issues/1821) * **jsdom:** delay closing of the window and add some polyfills ([2e81618](https://github.com/apify/crawlee/commit/2e81618afb5f3890495e3e5fcfa037eb3319edc9) ) [3.3.0](https://github.com/apify/crawlee/compare/v3.2.2...v3.3.0) (2023-03-09) =============================================================================== **Note:** Version bump only for package @crawlee/playwright [3.2.2](https://github.com/apify/crawlee/compare/v3.2.1...v3.2.2) (2023-02-08)[​](#322-2023-02-08 "Direct link to 322-2023-02-08") ------------------------------------------------------------------------------------------------------------------------------------ **Note:** Version bump only for package @crawlee/playwright [3.2.1](https://github.com/apify/crawlee/compare/v3.2.0...v3.2.1) (2023-02-07)[​](#321-2023-02-07 "Direct link to 321-2023-02-07") ------------------------------------------------------------------------------------------------------------------------------------ **Note:** Version bump only for package @crawlee/playwright [3.2.0](https://github.com/apify/crawlee/compare/v3.1.4...v3.2.0) (2023-02-07) =============================================================================== ### Bug Fixes[​](#bug-fixes-11 "Direct link to Bug Fixes") * allow `userData` option in `enqueueLinksByClickingElements` ([#1749](https://github.com/apify/crawlee/issues/1749) ) ([736f85d](https://github.com/apify/crawlee/commit/736f85d4a3b99a06d0f99f91e33e71976a9458a3) ), closes [#1617](https://github.com/apify/crawlee/issues/1617) * declare missing dependency on `tslib` ([27e96c8](https://github.com/apify/crawlee/commit/27e96c80c26e7fc31809a4b518d699573cb8c662) ), closes [#1747](https://github.com/apify/crawlee/issues/1747) * update playwright to 1.29.2 and make peer dep. less strict ([#1735](https://github.com/apify/crawlee/issues/1735) ) ([c654fcd](https://github.com/apify/crawlee/commit/c654fcdea06fb203b7952ed97650190cc0e74394) ), closes [#1723](https://github.com/apify/crawlee/issues/1723) ### Features[​](#features-11 "Direct link to Features") * add `forefront` option to all `enqueueLinks` variants ([#1760](https://github.com/apify/crawlee/issues/1760) ) ([a01459d](https://github.com/apify/crawlee/commit/a01459dffb51162e676354f0aa4811a1d36affa9) ), closes [#1483](https://github.com/apify/crawlee/issues/1483) [3.1.4](https://github.com/apify/crawlee/compare/v3.1.3...v3.1.4) (2022-12-14)[​](#314-2022-12-14 "Direct link to 314-2022-12-14") ------------------------------------------------------------------------------------------------------------------------------------ **Note:** Version bump only for package @crawlee/playwright [3.1.3](https://github.com/apify/crawlee/compare/v3.1.2...v3.1.3) (2022-12-07)[​](#313-2022-12-07 "Direct link to 313-2022-12-07") ------------------------------------------------------------------------------------------------------------------------------------ **Note:** Version bump only for package @crawlee/playwright 3.1.2 (2022-11-15)[​](#312-2022-11-15 "Direct link to 3.1.2 (2022-11-15)") --------------------------------------------------------------------------- **Note:** Version bump only for package @crawlee/playwright 3.1.1 (2022-11-07)[​](#311-2022-11-07 "Direct link to 3.1.1 (2022-11-07)") --------------------------------------------------------------------------- **Note:** Version bump only for package @crawlee/playwright 3.1.0 (2022-10-13) ================== **Note:** Version bump only for package @crawlee/playwright [3.0.4](https://github.com/apify/crawlee/compare/v3.0.3...v3.0.4) (2022-08-22)[​](#304-2022-08-22 "Direct link to 304-2022-08-22") ------------------------------------------------------------------------------------------------------------------------------------ ### Features[​](#features-12 "Direct link to Features") * enable tab-as-a-container for Firefox ([#1456](https://github.com/apify/crawlee/issues/1456) ) ([ae5ba4f](https://github.com/apify/crawlee/commit/ae5ba4f15fd6d14f444486234753ce1781c74cc8) ) **Page Options** Hide Inherited * [Features](#features) * [3.12.2 (2025-01-27)](#3122-2025-01-27) * [3.12.1 (2024-12-04)](#3121-2024-12-04) * [Bug Fixes](#bug-fixes) * [3.11.5 (2024-10-04)](#3115-2024-10-04) * [Bug Fixes](#bug-fixes-1) * [3.11.4 (2024-09-23)](#3114-2024-09-23) * [3.11.3 (2024-09-03)](#3113-2024-09-03) * [3.11.2 (2024-08-28)](#3112-2024-08-28) * [3.11.1 (2024-07-24)](#3111-2024-07-24) * [Features](#features-1) * [3.10.5 (2024-06-12)](#3105-2024-06-12) * [Bug Fixes](#bug-fixes-2) * [3.10.4 (2024-06-11)](#3104-2024-06-11) * [Bug Fixes](#bug-fixes-3) * [3.10.3 (2024-06-07)](#3103-2024-06-07) * [Bug Fixes](#bug-fixes-4) * [Features](#features-2) * [3.10.2 (2024-06-03)](#3102-2024-06-03) * [3.10.1 (2024-05-23)](#3101-2024-05-23) * [Bug Fixes](#bug-fixes-5) * [3.9.2 (2024-04-17)](#392-2024-04-17) * [3.9.1 (2024-04-11)](#391-2024-04-11) * [Features](#features-3) * [3.8.2 (2024-03-21)](#382-2024-03-21) * [Features](#features-4) * [3.8.1 (2024-02-22)](#381-2024-02-22) * [Features](#features-5) * [3.7.3 (2024-01-30)](#373-2024-01-30) * [3.7.2 (2024-01-09)](#372-2024-01-09) * [3.7.1 (2024-01-02)](#371-2024-01-02) * [3.6.2 (2023-11-26)](#362-2023-11-26) * [3.6.1 (2023-11-15)](#361-2023-11-15) * [Features](#features-6) * [Bug Fixes](#bug-fixes-6) * [3.5.8 (2023-10-17)](#358-2023-10-17) * [3.5.7 (2023-10-05)](#357-2023-10-05) * [3.5.6 (2023-10-04)](#356-2023-10-04) * [3.5.5 (2023-10-02)](#355-2023-10-02) * [Bug Fixes](#bug-fixes-7) * [Features](#features-7) * [3.5.4 (2023-09-11)](#354-2023-09-11) * [Bug Fixes](#bug-fixes-8) * [3.5.3 (2023-08-31)](#353-2023-08-31) * [Bug Fixes](#bug-fixes-9) * [3.5.2 (2023-08-21)](#352-2023-08-21) * [3.5.1 (2023-08-16)](#351-2023-08-16) * [Features](#features-8) * [3.4.2 (2023-07-19)](#342-2023-07-19) * [3.4.1 (2023-07-13)](#341-2023-07-13) * [Features](#features-9) * [3.3.3 (2023-05-31)](#333-2023-05-31) * [3.3.2 (2023-05-11)](#332-2023-05-11) * [Features](#features-10) * [3.3.1 (2023-04-11)](#331-2023-04-11) * [Bug Fixes](#bug-fixes-10) * [3.2.2 (2023-02-08)](#322-2023-02-08) * [3.2.1 (2023-02-07)](#321-2023-02-07) * [Bug Fixes](#bug-fixes-11) * [Features](#features-11) * [3.1.4 (2022-12-14)](#314-2022-12-14) * [3.1.3 (2022-12-07)](#313-2022-12-07) * [3.1.2 (2022-11-15)](#312-2022-11-15) * [3.1.1 (2022-11-07)](#311-2022-11-07) * [3.0.4 (2022-08-22)](#304-2022-08-22) * [Features](#features-12) --- # playwrightUtils | API | Crawlee for JavaScript · Build reliable crawlers. Fast. [Skip to main content](#__docusaurus_skipToContent_fallback) Version: 3.13 On this page A namespace that contains various utilities for [Playwright](https://github.com/microsoft/playwright) - the headless Chrome Node API. **Example usage:** import { launchPlaywright, playwrightUtils } from 'crawlee';// Navigate to https://www.example.com in Playwright with a POST requestconst browser = await launchPlaywright();const page = await browser.newPage();await playwrightUtils.gotoExtended(page, { url: 'https://example.com, method: 'POST',}); Index[](#Index) ---------------- ### Interfaces * [BlockRequestsOptions](/js/api/playwright-crawler/namespace/playwrightUtils#BlockRequestsOptions) * [CompiledScriptParams](/js/api/playwright-crawler/namespace/playwrightUtils#CompiledScriptParams) * [DirectNavigationOptions](/js/api/playwright-crawler/namespace/playwrightUtils#DirectNavigationOptions) * [InfiniteScrollOptions](/js/api/playwright-crawler/namespace/playwrightUtils#InfiniteScrollOptions) * [InjectFileOptions](/js/api/playwright-crawler/namespace/playwrightUtils#InjectFileOptions) * [SaveSnapshotOptions](/js/api/playwright-crawler/namespace/playwrightUtils#SaveSnapshotOptions) ### Type Aliases * [CompiledScriptFunction](/js/api/playwright-crawler/namespace/playwrightUtils#CompiledScriptFunction) ### Functions * [blockRequests](/js/api/playwright-crawler/namespace/playwrightUtils#blockRequests) * [closeCookieModals](/js/api/playwright-crawler/namespace/playwrightUtils#closeCookieModals) * [compileScript](/js/api/playwright-crawler/namespace/playwrightUtils#compileScript) * [enqueueLinksByClickingElements](/js/api/playwright-crawler/namespace/playwrightUtils#enqueueLinksByClickingElements) * [gotoExtended](/js/api/playwright-crawler/namespace/playwrightUtils#gotoExtended) * [infiniteScroll](/js/api/playwright-crawler/namespace/playwrightUtils#infiniteScroll) * [injectFile](/js/api/playwright-crawler/namespace/playwrightUtils#injectFile) * [injectJQuery](/js/api/playwright-crawler/namespace/playwrightUtils#injectJQuery) * [parseWithCheerio](/js/api/playwright-crawler/namespace/playwrightUtils#parseWithCheerio) * [registerUtilsToContext](/js/api/playwright-crawler/namespace/playwrightUtils#registerUtilsToContext) * [saveSnapshot](/js/api/playwright-crawler/namespace/playwrightUtils#saveSnapshot) Interfaces[](#Interfaces) -------------------------- ### [](#BlockRequestsOptions) [](https://github.com/apify/crawlee/blob/master/packages/playwright-crawler/src/internals/utils/playwright-utils.ts#L56) BlockRequestsOptions BlockRequestsOptions: ### [](#extraUrlPatterns) [](https://github.com/apify/crawlee/blob/master/packages/playwright-crawler/src/internals/utils/playwright-utils.ts#L68) optionalextraUrlPatterns extraUrlPatterns?: string\[\] If you just want to append to the default blocked patterns, use this property. ### [](#urlPatterns) [](https://github.com/apify/crawlee/blob/master/packages/playwright-crawler/src/internals/utils/playwright-utils.ts#L63) optionalurlPatterns urlPatterns?: string\[\] The patterns of URLs to block from being loaded by the browser. Only `*` can be used as a wildcard. It is also automatically added to the beginning and end of the pattern. This limitation is enforced by the DevTools protocol. `.png` is the same as `*.png*`. ### [](#CompiledScriptParams) [](https://github.com/apify/crawlee/blob/master/packages/playwright-crawler/src/internals/utils/playwright-utils.ts#L305) CompiledScriptParams CompiledScriptParams: ### [](#page) [](https://github.com/apify/crawlee/blob/master/packages/playwright-crawler/src/internals/utils/playwright-utils.ts#L306) page page: Page ### [](#request) [](https://github.com/apify/crawlee/blob/master/packages/playwright-crawler/src/internals/utils/playwright-utils.ts#L307) request request: [Request](/js/api/core/class/Request) ### [](#DirectNavigationOptions) [](https://github.com/apify/crawlee/blob/master/packages/playwright-crawler/src/internals/utils/playwright-utils.ts#L146) DirectNavigationOptions DirectNavigationOptions: ### [](#referer) [](https://github.com/apify/crawlee/blob/master/packages/playwright-crawler/src/internals/utils/playwright-utils.ts#L166) optionalreferer referer?: string Referer header value. If provided it will take preference over the referer header value set by page.setExtraHTTPHeaders(headers). ### [](#timeout) [](https://github.com/apify/crawlee/blob/master/packages/playwright-crawler/src/internals/utils/playwright-utils.ts#L153) optionaltimeout timeout?: number Maximum operation time in milliseconds, defaults to 30 seconds, pass `0` to disable timeout. The default value can be changed by using the browserContext.setDefaultNavigationTimeout(timeout), browserContext.setDefaultTimeout(timeout), page.setDefaultNavigationTimeout(timeout) or page.setDefaultTimeout(timeout) methods. ### [](#waitUntil) [](https://github.com/apify/crawlee/blob/master/packages/playwright-crawler/src/internals/utils/playwright-utils.ts#L161) optionalwaitUntil waitUntil?: domcontentloaded | load | networkidle When to consider operation succeeded, defaults to `load`. Events can be either: * `'domcontentloaded'` - consider operation to be finished when the `DOMContentLoaded` event is fired. * `'load'` - consider operation to be finished when the `load` event is fired. * `'networkidle'` - consider operation to be finished when there are no network connections for at least `500` ms. ### [](#InfiniteScrollOptions) [](https://github.com/apify/crawlee/blob/master/packages/playwright-crawler/src/internals/utils/playwright-utils.ts#L354) InfiniteScrollOptions InfiniteScrollOptions: ### [](#buttonSelector) [](https://github.com/apify/crawlee/blob/master/packages/playwright-crawler/src/internals/utils/playwright-utils.ts#L382) optionalbuttonSelector buttonSelector?: string Optionally checks and clicks a button if it appears while scrolling. This is required on some websites for the scroll to work. ### [](#maxScrollHeight) [](https://github.com/apify/crawlee/blob/master/packages/playwright-crawler/src/internals/utils/playwright-utils.ts#L365) optionalmaxScrollHeight maxScrollHeight?: number = 0 How many pixels to scroll down. If 0, will scroll until bottom of page. ### [](#scrollDownAndUp) [](https://github.com/apify/crawlee/blob/master/packages/playwright-crawler/src/internals/utils/playwright-utils.ts#L377) optionalscrollDownAndUp scrollDownAndUp?: boolean = false If true, it will scroll up a bit after each scroll down. This is required on some websites for the scroll to work. ### [](#stopScrollCallback) [](https://github.com/apify/crawlee/blob/master/packages/playwright-crawler/src/internals/utils/playwright-utils.ts#L387) optionalstopScrollCallback stopScrollCallback?: () => unknown This function is called after every scroll and stops the scrolling process if it returns `true`. The function can be `async`. * * * #### Type declaration * * (): unknown * #### Returns unknown ### [](#timeoutSecs) [](https://github.com/apify/crawlee/blob/master/packages/playwright-crawler/src/internals/utils/playwright-utils.ts#L359) optionaltimeoutSecs timeoutSecs?: number = 0 How many seconds to scroll for. If 0, will scroll until bottom of page. ### [](#waitForSecs) [](https://github.com/apify/crawlee/blob/master/packages/playwright-crawler/src/internals/utils/playwright-utils.ts#L371) optionalwaitForSecs waitForSecs?: number = 4 How many seconds to wait for no new content to load before exit. ### [](#InjectFileOptions) [](https://github.com/apify/crawlee/blob/master/packages/playwright-crawler/src/internals/utils/playwright-utils.ts#L47) InjectFileOptions InjectFileOptions: ### [](#surviveNavigations) [](https://github.com/apify/crawlee/blob/master/packages/playwright-crawler/src/internals/utils/playwright-utils.ts#L53) optionalsurviveNavigations surviveNavigations?: boolean Enables the injected script to survive page navigations and reloads without need to be re-injected manually. This does not mean, however, that internal state will be preserved. Just that it will be automatically re-injected on each navigation before any other scripts get the chance to execute. ### [](#SaveSnapshotOptions) [](https://github.com/apify/crawlee/blob/master/packages/playwright-crawler/src/internals/utils/playwright-utils.ts#L496) SaveSnapshotOptions SaveSnapshotOptions: ### [](#config) [](https://github.com/apify/crawlee/blob/master/packages/playwright-crawler/src/internals/utils/playwright-utils.ts#L531) optionalconfig config?: [Configuration](/js/api/core/class/Configuration) = [Configuration](/js/api/core/class/Configuration) Configuration of the crawler that will be used to save the snapshot. ### [](#key) [](https://github.com/apify/crawlee/blob/master/packages/playwright-crawler/src/internals/utils/playwright-utils.ts#L501) optionalkey key?: string = ‘SNAPSHOT’ Key under which the screenshot and HTML will be saved. `.jpg` will be appended for screenshot and `.html` for HTML. ### [](#keyValueStoreName) [](https://github.com/apify/crawlee/blob/master/packages/playwright-crawler/src/internals/utils/playwright-utils.ts#L525) optionalkeyValueStoreName keyValueStoreName?: null | string = null | string Name or id of the Key-Value store where snapshot is saved. By default it is saved to default Key-Value store. ### [](#saveHtml) [](https://github.com/apify/crawlee/blob/master/packages/playwright-crawler/src/internals/utils/playwright-utils.ts#L519) optionalsaveHtml saveHtml?: boolean = true If true, it will save a full HTML of the current page as a record with `key` appended by `.html`. ### [](#saveScreenshot) [](https://github.com/apify/crawlee/blob/master/packages/playwright-crawler/src/internals/utils/playwright-utils.ts#L513) optionalsaveScreenshot saveScreenshot?: boolean = true If true, it will save a full screenshot of the current page as a record with `key` appended by `.jpg`. ### [](#screenshotQuality) [](https://github.com/apify/crawlee/blob/master/packages/playwright-crawler/src/internals/utils/playwright-utils.ts#L507) optionalscreenshotQuality screenshotQuality?: number = 50 The quality of the image, between 0-100. Higher quality images have bigger size and require more storage. Type Aliases[](#Type Aliases) ------------------------------ ### [](#CompiledScriptFunction) [](https://github.com/apify/crawlee/blob/master/packages/playwright-crawler/src/internals/utils/playwright-utils.ts#L310) CompiledScriptFunction CompiledScriptFunction: (params) => Promise #### Type declaration * * (params): Promise * #### Parameters * ##### params: [CompiledScriptParams](/js/api/playwright-crawler/namespace/playwrightUtils#CompiledScriptParams) #### Returns Promise Functions[](#Functions) ------------------------ ### [](#blockRequests) [](https://github.com/apify/crawlee/blob/master/packages/playwright-crawler/src/internals/utils/playwright-utils.ts#L281) blockRequests * **blockRequests**(page, options): Promise * > This is a **Chromium-only feature.** > > Using this option with Firefox and WebKit browsers doesn't have any effect. To set up request blocking for these browsers, use `page.route()` instead. Forces the Playwright browser tab to block loading URLs that match a provided pattern. This is useful to speed up crawling of websites, since it reduces the amount of data that needs to be downloaded from the web, but it may break some websites or unexpectedly prevent loading of resources. By default, the function will block all URLs including the following patterns: [".css", ".jpg", ".jpeg", ".png", ".svg", ".gif", ".woff", ".pdf", ".zip"] If you want to extend this list further, use the `extraUrlPatterns` option, which will keep blocking the default patterns, as well as add your custom ones. If you would like to block only specific patterns, use the `urlPatterns` option, which will override the defaults and block only URLs with your custom patterns. This function does not use Playwright's request interception and therefore does not interfere with browser cache. It's also faster than blocking requests using interception, because the blocking happens directly in the browser without the round-trip to Node.js, but it does not provide the extra benefits of request interception. The function will never block main document loads and their respective redirects. **Example usage** import { launchPlaywright, playwrightUtils } from 'crawlee';const browser = await launchPlaywright();const page = await browser.newPage();// Block all requests to URLs that include `adsbygoogle.js` and also all defaults.await playwrightUtils.blockRequests(page, { extraUrlPatterns: ['adsbygoogle.js'],});await page.goto('https://cnn.com'); * * * #### Parameters * ##### page: Page Playwright [`Page`](https://playwright.dev/docs/api/class-page) object. * ##### optionaloptions: [BlockRequestsOptions](/js/api/playwright-crawler/namespace/playwrightUtils#BlockRequestsOptions) = {} #### Returns Promise ### [](#closeCookieModals) [](https://github.com/apify/crawlee/blob/master/packages/playwright-crawler/src/internals/utils/playwright-utils.ts#L641) closeCookieModals * **closeCookieModals**(page): Promise * #### Parameters * ##### page: Page #### Returns Promise ### [](#compileScript) [](https://github.com/apify/crawlee/blob/master/packages/playwright-crawler/src/internals/utils/playwright-utils.ts#L338) compileScript * **compileScript**(scriptString, context): [CompiledScriptFunction](/js/api/playwright-crawler/namespace/playwrightUtils#CompiledScriptFunction) * Compiles a Playwright script into an async function that may be executed at any time by providing it with the following object: { page: Page, request: Request,} Where `page` is a Playwright [`Page`](https://playwright.dev/docs/api/class-page) and `request` is a [Request](/js/api/core/class/Request) . The function is compiled by using the `scriptString` parameter as the function's body, so any limitations to function bodies apply. Return value of the compiled function is the return value of the function body = the `scriptString` parameter. As a security measure, no globals such as `process` or `require` are accessible from within the function body. Note that the function does not provide a safe sandbox and even though globals are not easily accessible, malicious code may still execute in the main process via prototype manipulation. Therefore you should only use this function to execute sanitized or safe code. Custom context may also be provided using the `context` parameter. To improve security, make sure to only pass the really necessary objects to the context. Preferably making secured copies beforehand. * * * #### Parameters * ##### scriptString: string * ##### context: Dictionary = ... #### Returns [CompiledScriptFunction](/js/api/playwright-crawler/namespace/playwrightUtils#CompiledScriptFunction) ### [](#enqueueLinksByClickingElements) [](https://github.com/apify/crawlee/blob/master/packages/playwright-crawler/src/internals/enqueue-links/click-elements.ts#L213) enqueueLinksByClickingElements * **enqueueLinksByClickingElements**(options): Promise<[BatchAddRequestsResult](/js/api/types/interface/BatchAddRequestsResult) \> * The function finds elements matching a specific CSS selector in a Playwright page, clicks all those elements using a mouse move and a left mouse button click and intercepts all the navigation requests that are subsequently produced by the page. The intercepted requests, including their methods, headers and payloads are then enqueued to a provided [RequestQueue](/js/api/core/class/RequestQueue) . This is useful to crawl JavaScript heavy pages where links are not available in `href` elements, but rather navigations are triggered in click handlers. If you're looking to find URLs in `href` attributes of the page, see enqueueLinks. Optionally, the function allows you to filter the target links' URLs using an array of [PseudoUrl](/js/api/core/class/PseudoUrl) objects and override settings of the enqueued [Request](/js/api/core/class/Request) objects. **IMPORTANT**: To be able to do this, this function uses various mutations on the page, such as changing the Z-index of elements being clicked and their visibility. Therefore, it is recommended to only use this function as the last operation in the page. **USING HEADFUL BROWSER**: When using a headful browser, this function will only be able to click elements in the focused tab, effectively limiting concurrency to 1. In headless mode, full concurrency can be achieved. **PERFORMANCE**: Clicking elements with a mouse and intercepting requests is not a low level operation that takes nanoseconds. It's not very CPU intensive, but it takes time. We strongly recommend limiting the scope of the clicking as much as possible by using a specific selector that targets only the elements that you assume or know will produce a navigation. You can certainly click everything by using the `*` selector, but be prepared to wait minutes to get results on a large and complex page. **Example usage** await playwrightUtils.enqueueLinksByClickingElements({ page, requestQueue, selector: 'a.product-detail', pseudoUrls: [ 'https://www.example.com/handbags/[.*]' 'https://www.example.com/purses/[.*]' ],}); * * * #### Parameters * ##### options: [EnqueueLinksByClickingElementsOptions](/js/api/playwright-crawler/namespace/playwrightClickElements#EnqueueLinksByClickingElementsOptions) #### Returns Promise<[BatchAddRequestsResult](/js/api/types/interface/BatchAddRequestsResult) \> Promise that resolves to [BatchAddRequestsResult](/js/api/types/interface/BatchAddRequestsResult) object. ### [](#gotoExtended) [](https://github.com/apify/crawlee/blob/master/packages/playwright-crawler/src/internals/utils/playwright-utils.ts#L181) gotoExtended * **gotoExtended**(page, request, gotoOptions): Promise * Extended version of Playwright's `page.goto()` allowing to perform requests with HTTP method other than GET, with custom headers and POST payload. URL, method, headers and payload are taken from request parameter that must be an instance of Request class. _NOTE:_ In recent versions of Playwright using requests other than GET, overriding headers and adding payloads disables browser cache which degrades performance. * * * #### Parameters * ##### page: Page Playwright [`Page`](https://playwright.dev/docs/api/class-page) object. * ##### request: [Request](/js/api/core/class/Request) * ##### optionalgotoOptions: [DirectNavigationOptions](/js/api/playwright-crawler/namespace/playwrightUtils#DirectNavigationOptions) = {} Custom options for `page.goto()`. #### Returns Promise ### [](#infiniteScroll) [](https://github.com/apify/crawlee/blob/master/packages/playwright-crawler/src/internals/utils/playwright-utils.ts#L396) infiniteScroll * **infiniteScroll**(page, options): Promise * Scrolls to the bottom of a page, or until it times out. Loads dynamic content when it hits the bottom of a page, and then continues scrolling. * * * #### Parameters * ##### page: Page Playwright [`Page`](https://playwright.dev/docs/api/class-page) object. * ##### optionaloptions: [InfiniteScrollOptions](/js/api/playwright-crawler/namespace/playwrightUtils#InfiniteScrollOptions) = {} #### Returns Promise ### [](#injectFile) [](https://github.com/apify/crawlee/blob/master/packages/playwright-crawler/src/internals/utils/playwright-utils.ts#L87) injectFile * **injectFile**(page, filePath, options): Promise * Injects a JavaScript file into a Playwright page. Unlike Playwright's `addScriptTag` function, this function works on pages with arbitrary Cross-Origin Resource Sharing (CORS) policies. File contents are cached for up to 10 files to limit file system access. * * * #### Parameters * ##### page: Page Playwright [`Page`](https://playwright.dev/docs/api/class-page) object. * ##### filePath: string File path * ##### optionaloptions: [InjectFileOptions](/js/api/playwright-crawler/namespace/playwrightUtils#InjectFileOptions) = {} #### Returns Promise ### [](#injectJQuery) [](https://github.com/apify/crawlee/blob/master/packages/playwright-crawler/src/internals/utils/playwright-utils.ts#L141) injectJQuery * **injectJQuery**(page, options): Promise * Injects the [jQuery](https://jquery.com/) library into a Playwright page. jQuery is often useful for various web scraping and crawling tasks. For example, it can help extract text from HTML elements using CSS selectors. Beware that the injected jQuery object will be set to the `window.$` variable and thus it might cause conflicts with other libraries included by the page that use the same variable name (e.g. another version of jQuery). This can affect functionality of page's scripts. The injected jQuery will survive page navigations and reloads by default. **Example usage:** await playwrightUtils.injectJQuery(page);const title = await page.evaluate(() => { return $('head title').text();}); Note that `injectJQuery()` does not affect the Playwright [`page.$()`](https://playwright.dev/docs/api/class-page#page-query-selector) function in any way. * * * #### Parameters * ##### page: Page Playwright [`Page`](https://playwright.dev/docs/api/class-page) object. * ##### optionaloptions: { surviveNavigations?: boolean } * ##### optionalsurviveNavigations: boolean Opt-out option to disable the JQuery reinjection after navigation. #### Returns Promise ### [](#parseWithCheerio) [](https://github.com/apify/crawlee/blob/master/packages/playwright-crawler/src/internals/utils/playwright-utils.ts#L600) parseWithCheerio * **parseWithCheerio**(page, ignoreShadowRoots, ignoreIframes): Promise<[CheerioRoot](/js/api/utils#CheerioRoot) \> * Returns Cheerio handle for `page.content()`, allowing to work with the data same way as with [CheerioCrawler](/js/api/cheerio-crawler/class/CheerioCrawler) . **Example usage:** const $ = await playwrightUtils.parseWithCheerio(page);const title = $('title').text(); * * * #### Parameters * ##### page: Page Playwright [`Page`](https://playwright.dev/docs/api/class-page) object. * ##### ignoreShadowRoots: boolean = false * ##### ignoreIframes: boolean = false #### Returns Promise<[CheerioRoot](/js/api/utils#CheerioRoot) \> ### [](#registerUtilsToContext) [](https://github.com/apify/crawlee/blob/master/packages/playwright-crawler/src/internals/utils/playwright-utils.ts#L1004) registerUtilsToContext * **registerUtilsToContext**(context, crawlerOptions): void * #### Parameters * ##### context: [PlaywrightCrawlingContext](/js/api/playwright-crawler/interface/PlaywrightCrawlingContext) * ##### crawlerOptions: [PlaywrightCrawlerOptions](/js/api/playwright-crawler/interface/PlaywrightCrawlerOptions) #### Returns void ### [](#saveSnapshot) [](https://github.com/apify/crawlee/blob/master/packages/playwright-crawler/src/internals/utils/playwright-utils.ts#L539) saveSnapshot * **saveSnapshot**(page, options): Promise * Saves a full screenshot and HTML of the current page into a Key-Value store. * * * #### Parameters * ##### page: Page Playwright [`Page`](https://playwright.dev/docs/api/class-page) object. * ##### optionaloptions: [SaveSnapshotOptions](/js/api/playwright-crawler/namespace/playwrightUtils#SaveSnapshotOptions) = {} #### Returns Promise **Page Options** Hide Inherited * [BlockRequestsOptions](#BlockRequestsOptions) * [CompiledScriptParams](#CompiledScriptParams) * [DirectNavigationOptions](#DirectNavigationOptions) * [InfiniteScrollOptions](#InfiniteScrollOptions) * [InjectFileOptions](#InjectFileOptions) * [SaveSnapshotOptions](#SaveSnapshotOptions) * [CompiledScriptFunction](#CompiledScriptFunction) * [blockRequests](#blockRequests) * [closeCookieModals](#closeCookieModals) * [compileScript](#compileScript) * [enqueueLinksByClickingElements](#enqueueLinksByClickingElements) * [gotoExtended](#gotoExtended) * [infiniteScroll](#infiniteScroll) * [injectFile](#injectFile) * [injectJQuery](#injectJQuery) * [parseWithCheerio](#parseWithCheerio) * [registerUtilsToContext](#registerUtilsToContext) * [saveSnapshot](#saveSnapshot) ---