# Table of Contents - [Welcome! | Cyberdrop-DL](#welcome-cyberdrop-dl) - [Migration to Cyberdrop-DL-Patched | Cyberdrop-DL](#migration-to-cyberdrop-dl-patched-cyberdrop-dl) - [Getting Started | Cyberdrop-DL](#getting-started-cyberdrop-dl) - [Running CDL on Android | Cyberdrop-DL](#running-cdl-on-android-cyberdrop-dl) - [Frequently Asked Questions | Cyberdrop-DL](#frequently-asked-questions-cyberdrop-dl) - [Cyberdrop-DL Install | Cyberdrop-DL](#cyberdrop-dl-install-cyberdrop-dl) - [SimpCity Support Dropped | Cyberdrop-DL](#simpcity-support-dropped-cyberdrop-dl) - [Transition to V8 | Cyberdrop-DL](#transition-to-v8-cyberdrop-dl) - [Special Type Options | Cyberdrop-DL](#special-type-options-cyberdrop-dl) - [Authentication | Cyberdrop-DL](#authentication-cyberdrop-dl) - [Configuration Options | Cyberdrop-DL](#configuration-options-cyberdrop-dl) - [Settings | Cyberdrop-DL](#settings-cyberdrop-dl) - [Notifications | Cyberdrop-DL](#notifications-cyberdrop-dl) - [Welcome! | Cyberdrop-DL Development Docs | Cyberdrop-DL](#welcome-cyberdrop-dl-development-docs-cyberdrop-dl) - [Global Settings | Cyberdrop-DL](#global-settings-cyberdrop-dl) - [Migration to Cyberdrop-DL-Patched | Cyberdrop-DL Development Docs | Cyberdrop-DL](#migration-to-cyberdrop-dl-patched-cyberdrop-dl-development-docs-cyberdrop-dl) - [Files | Cyberdrop-DL](#files-cyberdrop-dl) - [UI Options | Cyberdrop-DL](#ui-options-cyberdrop-dl) - [Running CDL on Android | Cyberdrop-DL Development Docs | Cyberdrop-DL](#running-cdl-on-android-cyberdrop-dl-development-docs-cyberdrop-dl) - [SimpCity Support Dropped | Cyberdrop-DL Development Docs | Cyberdrop-DL](#simpcity-support-dropped-cyberdrop-dl-development-docs-cyberdrop-dl) - [Browser Cookies | Cyberdrop-DL](#browser-cookies-cyberdrop-dl) - [File Size Limits | Cyberdrop-DL](#file-size-limits-cyberdrop-dl) - [Generic Crawlers | Cyberdrop-DL](#generic-crawlers-cyberdrop-dl) - [Getting Started | Cyberdrop-DL Development Docs | Cyberdrop-DL](#getting-started-cyberdrop-dl-development-docs-cyberdrop-dl) - [Transition to V8 | Cyberdrop-DL Development Docs | Cyberdrop-DL](#transition-to-v8-cyberdrop-dl-development-docs-cyberdrop-dl) - [Media Duration Limits | Cyberdrop-DL](#media-duration-limits-cyberdrop-dl) - [Frequently Asked Questions | Cyberdrop-DL Development Docs | Cyberdrop-DL](#frequently-asked-questions-cyberdrop-dl-development-docs-cyberdrop-dl) - [General | Cyberdrop-DL](#general-cyberdrop-dl) - [Cyberdrop-DL Install | Cyberdrop-DL Development Docs | Cyberdrop-DL](#cyberdrop-dl-install-cyberdrop-dl-development-docs-cyberdrop-dl) - [Dupe Cleanup Options | Cyberdrop-DL](#dupe-cleanup-options-cyberdrop-dl) - [Settings | Cyberdrop-DL Development Docs | Cyberdrop-DL](#settings-cyberdrop-dl-development-docs-cyberdrop-dl) - [Rate Limiting Options | Cyberdrop-DL](#rate-limiting-options-cyberdrop-dl) - [CLI Arguments | Cyberdrop-DL](#cli-arguments-cyberdrop-dl) - [Ignore Options | Cyberdrop-DL](#ignore-options-cyberdrop-dl) - [Logs | Cyberdrop-DL](#logs-cyberdrop-dl) - [Authentication | Cyberdrop-DL Development Docs | Cyberdrop-DL](#authentication-cyberdrop-dl-development-docs-cyberdrop-dl) - [Special Type Options | Cyberdrop-DL Development Docs | Cyberdrop-DL](#special-type-options-cyberdrop-dl-development-docs-cyberdrop-dl) - [Dupe Cleanup Options | Cyberdrop-DL Development Docs | Cyberdrop-DL](#dupe-cleanup-options-cyberdrop-dl-development-docs-cyberdrop-dl) - [Media Duration Limits | Cyberdrop-DL Development Docs | Cyberdrop-DL](#media-duration-limits-cyberdrop-dl-development-docs-cyberdrop-dl) - [Notifications | Cyberdrop-DL Development Docs | Cyberdrop-DL](#notifications-cyberdrop-dl-development-docs-cyberdrop-dl) - [Browser Cookies | Cyberdrop-DL Development Docs | Cyberdrop-DL](#browser-cookies-cyberdrop-dl-development-docs-cyberdrop-dl) - [Configuration Options | Cyberdrop-DL Development Docs | Cyberdrop-DL](#configuration-options-cyberdrop-dl-development-docs-cyberdrop-dl) - [UI Options | Cyberdrop-DL Development Docs | Cyberdrop-DL](#ui-options-cyberdrop-dl-development-docs-cyberdrop-dl) - [Files | Cyberdrop-DL Development Docs | Cyberdrop-DL](#files-cyberdrop-dl-development-docs-cyberdrop-dl) - [Sorting | Cyberdrop-DL](#sorting-cyberdrop-dl) - [Global Settings | Cyberdrop-DL Development Docs | Cyberdrop-DL](#global-settings-cyberdrop-dl-development-docs-cyberdrop-dl) - [Runtime Options | Cyberdrop-DL](#runtime-options-cyberdrop-dl) - [Generic Crawlers | Cyberdrop-DL Development Docs | Cyberdrop-DL](#generic-crawlers-cyberdrop-dl-development-docs-cyberdrop-dl) - [Ignore Options | Cyberdrop-DL Development Docs | Cyberdrop-DL](#ignore-options-cyberdrop-dl-development-docs-cyberdrop-dl) - [Supported Websites | Cyberdrop-DL](#supported-websites-cyberdrop-dl) - [Logs | Cyberdrop-DL Development Docs | Cyberdrop-DL](#logs-cyberdrop-dl-development-docs-cyberdrop-dl) - [General | Cyberdrop-DL Development Docs | Cyberdrop-DL](#general-cyberdrop-dl-development-docs-cyberdrop-dl) - [Rate Limiting Options | Cyberdrop-DL Development Docs | Cyberdrop-DL](#rate-limiting-options-cyberdrop-dl-development-docs-cyberdrop-dl) - [Sorting | Cyberdrop-DL Development Docs | Cyberdrop-DL](#sorting-cyberdrop-dl-development-docs-cyberdrop-dl) - [Runtime Options | Cyberdrop-DL Development Docs | Cyberdrop-DL](#runtime-options-cyberdrop-dl-development-docs-cyberdrop-dl) - [Download Options | Cyberdrop-DL](#download-options-cyberdrop-dl) - [CLI Arguments | Cyberdrop-DL Development Docs | Cyberdrop-DL](#cli-arguments-cyberdrop-dl-development-docs-cyberdrop-dl) - [Download Options | Cyberdrop-DL Development Docs | Cyberdrop-DL](#download-options-cyberdrop-dl-development-docs-cyberdrop-dl) - [File Size Limits | Cyberdrop-DL Development Docs | Cyberdrop-DL](#file-size-limits-cyberdrop-dl-development-docs-cyberdrop-dl) - [Supported Websites | Cyberdrop-DL Development Docs | Cyberdrop-DL](#supported-websites-cyberdrop-dl-development-docs-cyberdrop-dl) --- # Welcome! | Cyberdrop-DL Cyberdrop-DL is an open source asynchronous downloader maintained by [jbsparrowarrow-up-right](https://github.com/jbsparrow) . It's goal is to act as a bulk downloader for various sites you will find around the web. [globe-pointerSupported Websiteschevron-right](https://script-ware.gitbook.io/cyberdrop-dl/reference/supported-websites) [hashtag](https://script-ware.gitbook.io/cyberdrop-dl#want-to-jump-right-in) Want to jump right in? -------------------------------------------------------------------------------------------------------- Explore the Getting Started guide to quickly familiarize yourself with the setup process and get started with the software. [bullseye-arrowGetting Startedchevron-right](https://script-ware.gitbook.io/cyberdrop-dl/getting-started) [hashtag](https://script-ware.gitbook.io/cyberdrop-dl#want-to-customize-your-experience) Want to customize your experience? -------------------------------------------------------------------------------------------------------------------------------- Take a closer look and begin exploring the available configuration options and CLI arguments: [slidersConfiguration Optionschevron-right](https://script-ware.gitbook.io/cyberdrop-dl/reference/configuration-options) [rectangle-terminalCLI Argumentschevron-right](https://script-ware.gitbook.io/cyberdrop-dl/reference/cli-arguments) [envelope-open-textNotificationschevron-right](https://script-ware.gitbook.io/cyberdrop-dl/reference/notifications) [NextGetting Startedchevron-right](https://script-ware.gitbook.io/cyberdrop-dl/getting-started) Last updated 1 year ago sun-brightdesktopmoon sun-brightdesktopmoon --- # Migration to Cyberdrop-DL-Patched | Cyberdrop-DL If you used the original version of the package (`cyberdrop-dl` instead of `cyberdrop-dl-patched`), you can follow the steps below to migrate. circle-exclamation You may need to adjust your config to make sure it's compatible with the newer versions. See: [Transition to V8arrow-up-right](https://script-ware.gitbook.io/cyberdrop-dl/upgrade) [hashtag](https://script-ware.gitbook.io/cyberdrop-dl/migration-to-cyberdrop-dl-patched#if-you-installed-on-your-own-using-pip) If you installed on your own using pip --------------------------------------------------------------------------------------------------------------------------------------------------------------------------- You can simply uninstall the old version and install the new one using the following commands: Copy pip uninstall -y cyberdrop-dl pip install cyberdrop-dl-patched The command to use the new version will be `cyberdrop-dl-patched`. [hashtag](https://script-ware.gitbook.io/cyberdrop-dl/migration-to-cyberdrop-dl-patched#if-you-installed-using-the-start-files) If you installed using the start files --------------------------------------------------------------------------------------------------------------------------------------------------------------------------- You can find the new ones here: [new start scriptsarrow-up-right](https://github.com/jbsparrow/CyberDropDownloader/releases/latest) Put the new start scripts on the same folder as the old ones, them delete the old ones. You also have to delete any `venv` or `.venv` folder (if you have any) [PreviousFrequently Asked Questionschevron-left](https://script-ware.gitbook.io/cyberdrop-dl/frequently-asked-questions) [NextSimpCity Support Droppedchevron-right](https://script-ware.gitbook.io/cyberdrop-dl/simpcity-support-dropped) Last updated 5 months ago sun-brightdesktopmoon sun-brightdesktopmoon --- # Getting Started | Cyberdrop-DL [hashtag](https://script-ware.gitbook.io/cyberdrop-dl/getting-started#installing-cyberdrop-dl) Installing Cyberdrop-DL --------------------------------------------------------------------------------------------------------------------------- There are two ways to install Cyberdrop-DL. The first is the easy method, where you simply download the start scripts. The start scripts will automatically install a compatible python version and then install Cyberdrop-DL. The second method is installing cyberdrop-dl directly from pypi using `uv`, `pipx` or `pip` For instructions, see: [downloadCyberdrop-DL Installchevron-right](https://script-ware.gitbook.io/cyberdrop-dl/getting-started/cyberdrop-dl-install) [androidRunning CDL on Androidchevron-right](https://script-ware.gitbook.io/cyberdrop-dl/getting-started/cyberdrop-dl-on-android) [hashtag](https://script-ware.gitbook.io/cyberdrop-dl/getting-started#what-now) What now? ---------------------------------------------------------------------------------------------- If you downloaded the start scripts, just open the start script with the name of the OS you are using. For a manual install, execute the program with this command: Copy cyberdrop-dl-patched On the main screen, you can use the 'Edit URLs' option to add the URLs for the files you wish to download, them select the `download` option. That's it! However, Cyberdrop-DL has a ton of configuration options if you want more control over the downloads. You may want to review the following: [slidersConfiguration Optionschevron-right](https://script-ware.gitbook.io/cyberdrop-dl/reference/configuration-options) [rectangle-terminalCLI Argumentschevron-right](https://script-ware.gitbook.io/cyberdrop-dl/reference/cli-arguments) [envelope-open-textNotificationschevron-right](https://script-ware.gitbook.io/cyberdrop-dl/reference/notifications) By default, all config files are under the `AppData` folder, which is created on the same directory that CDL it run from. You may also want to peek at what websites the program actually supports: [globe-pointerSupported Websiteschevron-right](https://script-ware.gitbook.io/cyberdrop-dl/reference/supported-websites) If you have any issues, perhaps the [FAQ](https://script-ware.gitbook.io/cyberdrop-dl/frequently-asked-questions) might help you! [PreviousWelcome!chevron-left](https://script-ware.gitbook.io/cyberdrop-dl) [NextCyberdrop-DL Installchevron-right](https://script-ware.gitbook.io/cyberdrop-dl/getting-started/cyberdrop-dl-install) Last updated 5 months ago sun-brightdesktopmoon sun-brightdesktopmoon --- # Running CDL on Android | Cyberdrop-DL Cyberdrop-dl is a terminal app. That means you need a terminal emulator to run it. The defacto choice in Android is [termuxarrow-up-right](https://termux.dev/en/) . circle-info Most of the dependencies need to be compiled from source. A rust compiler is required. This means the installation could take several minutes, especially on low end phones circle-exclamation Compiling from source also requires a lot of extra storage. You will need at least 3.4GB just for the installation of CDL [hashtag](https://script-ware.gitbook.io/cyberdrop-dl/getting-started/cyberdrop-dl-on-android#id-1.-install-termux) 1\. Install `termux` --------------------------------------------------------------------------------------------------------------------------------------------- Termux wiki: [https://wiki.termux.com/wiki/Installationarrow-up-right](https://wiki.termux.com/wiki/Installation) Install termux from [F-droid (recommended)arrow-up-right](https://f-droid.org/packages/com.termux/) or from the [Google Playstore (restricted version)arrow-up-right](https://play.google.com/store/apps/details?id=com.termux) : [hashtag](https://script-ware.gitbook.io/cyberdrop-dl/getting-started/cyberdrop-dl-on-android#id-2.-install-cyberdrop-dl-patched) 2\. Install `cyberdrop-dl-patched` ------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Run the following commands inside termux Copy # Get storage access permission termux-setup-storage # Install dependencies pkg upgrade -y pkg install rust which micro libjpeg-turbo python uv -y # Install CDL and setup appdata folder uv tool install cyberdrop-dl-patched uv tool update-shell mkdir /sdcard/cyberdrop-dl echo 'alias cyberdrop-dl="cyberdrop-dl --portrait --appdata-folder /sdcard/cyberdrop-dl"' >> ~/.bashrc source ~/.bashrc [hashtag](https://script-ware.gitbook.io/cyberdrop-dl/getting-started/cyberdrop-dl-on-android#how-to-update-cyberdrop-dl-patched) How to update `cyberdrop-dl-patched`? ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Run this command inside termux: [PreviousCyberdrop-DL Installchevron-left](https://script-ware.gitbook.io/cyberdrop-dl/getting-started/cyberdrop-dl-install) [NextTransition to V8chevron-right](https://script-ware.gitbook.io/cyberdrop-dl/upgrade) Last updated 5 months ago * [1\. Install termux](https://script-ware.gitbook.io/cyberdrop-dl/getting-started/cyberdrop-dl-on-android#id-1.-install-termux) * [2\. Install cyberdrop-dl-patched](https://script-ware.gitbook.io/cyberdrop-dl/getting-started/cyberdrop-dl-on-android#id-2.-install-cyberdrop-dl-patched) * [How to update cyberdrop-dl-patched?](https://script-ware.gitbook.io/cyberdrop-dl/getting-started/cyberdrop-dl-on-android#how-to-update-cyberdrop-dl-patched) sun-brightdesktopmoon Copy uv tool upgrade cyberdrop-dl-patched sun-brightdesktopmoon --- # Frequently Asked Questions | Cyberdrop-DL [hashtag](https://script-ware.gitbook.io/cyberdrop-dl/frequently-asked-questions#what-does-this-do) What does this do? --------------------------------------------------------------------------------------------------------------------------- This is a bulk downloader for the supported sites. It supports resumable downloading (you can close and reopen the program at any time and it will pick up where it left off), and keeps track of your download history to avoid downloading files you've already downloaded in the past. [hashtag](https://script-ware.gitbook.io/cyberdrop-dl/frequently-asked-questions#how-do-i-update-this) How do I update this? --------------------------------------------------------------------------------------------------------------------------------- If you are using one of the provided start files, it should do so automatically. Keep in mind that they will only update to the newest version within the same major version. ex: if you are using v8 start scripts, they will update to the latest v8 release. When v9 is out, you will need to download the new start scripts [hashtag](https://script-ware.gitbook.io/cyberdrop-dl/frequently-asked-questions#why-do-i-get-ddos-guard-error-downloading-from-x-site) Why do i get `DDoS-Guard` error downloading from `x` site? ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- You may need to import cookies. Follow the instructions here: [How to extract cookies (DDoSGuard or login errors) #839arrow-up-right](https://github.com/jbsparrow/CyberDropDownloader/discussions/839) [hashtag](https://script-ware.gitbook.io/cyberdrop-dl/frequently-asked-questions#im-trying-to-report-a-bug-and-they-ask-me-for-a-logs-file.-where-is-this-file) I'm trying to report a bug and they ask me for a logs file. Where is this file? ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- By default, it'll be in `./AppData/configs//logs/` The `AppData` folder is created inside the folder where you run cyberdrop-dl from [hashtag](https://script-ware.gitbook.io/cyberdrop-dl/frequently-asked-questions#what-does-scrape_failures-and-download_failures-mean) What does `SCRAPE_FAILURES` and `DOWNLOAD_FAILURES` mean? ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Quite simply, almost all of them you see will be HTTP Status codes. Such as: 404 - Not Found (dead link) You check [this page to learn about what each error code meansarrow-up-right](https://developer.mozilla.org/en-US/docs/Web/HTTP/Status) . circle-info Any "Unknown" error, is usually coding related, or it'll be something like the program not being able to find a file extension for a file. You can report unknown error on the issue tracker: [https://github.com/jbsparrow/CyberDropDownloader/issuesarrow-up-right](https://github.com/jbsparrow/CyberDropDownloader/issues) [hashtag](https://script-ware.gitbook.io/cyberdrop-dl/frequently-asked-questions#why-are-all-the-files-skipped) Why are all the files skipped? --------------------------------------------------------------------------------------------------------------------------------------------------- The program tracks your download history and will skip any files you've previously downloaded to avoid duplicates. You can disable this behavior by using the `--ignore-history` CLI argument or setting `ignore_history` to `true` in the config [hashtag](https://script-ware.gitbook.io/cyberdrop-dl/frequently-asked-questions#the-screen-is-flickering) The screen is flickering? ----------------------------------------------------------------------------------------------------------------------------------------- This issue is likely related to the limitations of the traditional command prompt, which has remained largely unchanged over time. For Windows 10 users, it's recommended to install and use [Windows Terminalarrow-up-right](https://aka.ms/terminal) to run Cyberdrop-DL. Windows Terminal is the default on Windows 11. [hashtag](https://script-ware.gitbook.io/cyberdrop-dl/frequently-asked-questions#cyberdrop-dl-is-not-a-recognized-internal-command) `cyberdrop-dl` is not a recognized internal command -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- This issue is caused by an improper installation of Python, specifically Python not being added to the system PATH. It is recommended to revisit the [Getting Started](https://script-ware.gitbook.io/cyberdrop-dl/getting-started) guide and follow the steps provided to reinstall or use one of the latest start scripts [hashtag](https://script-ware.gitbook.io/cyberdrop-dl/frequently-asked-questions#how-do-i-scrape-forum-threads) How do I scrape forum threads? --------------------------------------------------------------------------------------------------------------------------------------------------- You need to import cookies to use as authentication for those sites. Follow the instructions here: [How to extract cookies (DDoSGuard or login errors) #839arrow-up-right](https://github.com/jbsparrow/CyberDropDownloader/discussions/839) [hashtag](https://script-ware.gitbook.io/cyberdrop-dl/frequently-asked-questions#why-are-the-filenames-the-way-they-are) Why are the filenames the way they are? --------------------------------------------------------------------------------------------------------------------------------------------------------------------- Filenames are taken directly from the source website. Blame whoever uploaded it. [hashtag](https://script-ware.gitbook.io/cyberdrop-dl/frequently-asked-questions#how-do-i-fix-ssl-certificate_verify_failed) How do I fix `[SSL: CERTIFICATE_VERIFY_FAILED]`? ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- This should only appear on macOS, so these instructions are for mac users. Go to your applications folder, find the python folder inside of it. Run the `Install Certificates` file in that folder. Go back to where you are running Cyberdrop-DL and delete the `venv` folder if one exists (if not, don't worry). Then try running the program again. [hashtag](https://script-ware.gitbook.io/cyberdrop-dl/frequently-asked-questions#a-thread-site-i-follow-has-new-posts-but-cyberdrop-dl-is-not-detecting-downloading-them-why) A thread/site i follow has new posts but `cyberdrop-dl` is not detecting / downloading them, why? ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ `cyberdrop-dl` caches requests to made to sites to speed up re-runs and minimize load on those sites. By default, forums are cached for 30 days and any other site is cached for 7 days. You can run with `--disable-cache` to temporarily disable the cache (CLI only) or change the default values to 0. See: [\--forum-cache-expire-afterarrow-up-right](https://script-ware.gitbook.io/cyberdrop-dl/reference/configuration-options/global-settings/rate-limiting-options#forum_cache_expire_after) [\--file-host-cache-expire-afterarrow-up-right](https://script-ware.gitbook.io/cyberdrop-dl/reference/configuration-options/global-settings/rate-limiting-options#file_host_cache_expire_after) [PreviousTransition to V8chevron-left](https://script-ware.gitbook.io/cyberdrop-dl/upgrade) [NextMigration to Cyberdrop-DL-Patchedchevron-right](https://script-ware.gitbook.io/cyberdrop-dl/migration-to-cyberdrop-dl-patched) Last updated 5 months ago * [What does this do?](https://script-ware.gitbook.io/cyberdrop-dl/frequently-asked-questions#what-does-this-do) * [How do I update this?](https://script-ware.gitbook.io/cyberdrop-dl/frequently-asked-questions#how-do-i-update-this) * [Why do i get DDoS-Guard error downloading from x site?](https://script-ware.gitbook.io/cyberdrop-dl/frequently-asked-questions#why-do-i-get-ddos-guard-error-downloading-from-x-site) * [I'm trying to report a bug and they ask me for a logs file. Where is this file?](https://script-ware.gitbook.io/cyberdrop-dl/frequently-asked-questions#im-trying-to-report-a-bug-and-they-ask-me-for-a-logs-file.-where-is-this-file) * [What does SCRAPE\_FAILURES and DOWNLOAD\_FAILURES mean?](https://script-ware.gitbook.io/cyberdrop-dl/frequently-asked-questions#what-does-scrape_failures-and-download_failures-mean) * [Why are all the files skipped?](https://script-ware.gitbook.io/cyberdrop-dl/frequently-asked-questions#why-are-all-the-files-skipped) * [The screen is flickering?](https://script-ware.gitbook.io/cyberdrop-dl/frequently-asked-questions#the-screen-is-flickering) * [cyberdrop-dl is not a recognized internal command](https://script-ware.gitbook.io/cyberdrop-dl/frequently-asked-questions#cyberdrop-dl-is-not-a-recognized-internal-command) * [How do I scrape forum threads?](https://script-ware.gitbook.io/cyberdrop-dl/frequently-asked-questions#how-do-i-scrape-forum-threads) * [Why are the filenames the way they are?](https://script-ware.gitbook.io/cyberdrop-dl/frequently-asked-questions#why-are-the-filenames-the-way-they-are) * [How do I fix \[SSL: CERTIFICATE\_VERIFY\_FAILED\]?](https://script-ware.gitbook.io/cyberdrop-dl/frequently-asked-questions#how-do-i-fix-ssl-certificate_verify_failed) * [A thread/site i follow has new posts but cyberdrop-dl is not detecting / downloading them, why?](https://script-ware.gitbook.io/cyberdrop-dl/frequently-asked-questions#a-thread-site-i-follow-has-new-posts-but-cyberdrop-dl-is-not-detecting-downloading-them-why) sun-brightdesktopmoon sun-brightdesktopmoon --- # Cyberdrop-DL Install | Cyberdrop-DL [hashtag](https://script-ware.gitbook.io/cyberdrop-dl/getting-started/cyberdrop-dl-install#using-start-scripts-from-release-page) Using Start Scripts (From Release Page) ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ This is the simplest method to get the program up and running. Pre-configured start files are provided that will automatically install python, install cyberdrop-dl, update, and launch the program for you. circle-info The start scripts only work on 64bits operating systems. If you are running a 32bit OS, you need to install directly from pypi and may need to compile some dependencies You can download them here: [https://github.com/jbsparrow/CyberDropDownloader/releases/latestarrow-up-right](https://github.com/jbsparrow/CyberDropDownloader/releases/latest) You only need to download the `Cyberdrop-DL_.zip` file, you don't need to worry about the other files. Extract the contents of the zip file to any location where you'd like the program to run and store downloaded files. The extracted files will include a start file for Windows, macOS, and Linux If you're using Windows or Linux, simply open the start file, and it will handle the rest for you circle-info If you are using Windows, **DO NOT** run the script as admin On macOS, you should be able to open the start file and have everything set up automatically. However, in some rare cases, macOS users may need to run the following command first:: Copy xcode-select --install chevron-rightOptional: Running Cyberdrop-DL Script with Custom Parameters[hashtag](https://script-ware.gitbook.io/cyberdrop-dl/getting-started/cyberdrop-dl-install#optional-running-cyberdrop-dl-script-with-custom-parameters) You can open the start script from the zip in a text editor like notepad. At the top of the file, you will this line: Copy set "COMMANDLINE_ARGS=" `COMMANDLINE_ARGS`: Provide any arguments to pass to Cyberdrop-Dl. For more information, refer to the [CLI Arguments section](https://script-ware.gitbook.io/cyberdrop-dl/reference/cli-arguments) circle-info You **MUST** put the values _inside_ the double quotes. Ex: `set "COMMANDLINE_ARGS=--disable-cache"` [hashtag](https://script-ware.gitbook.io/cyberdrop-dl/getting-started/cyberdrop-dl-install#manual-install) Manual Install ------------------------------------------------------------------------------------------------------------------------------ ### [hashtag](https://script-ware.gitbook.io/cyberdrop-dl/getting-started/cyberdrop-dl-install#id-1.-using-uv) 1\. Using `uv` The recommended way to install Cyberdrop-DL is using [`uv`arrow-up-right](https://docs.astral.sh/uv) . macOS / Linux Windows Once you have `uv`, you can install Cyberdrop-DL using: ### [hashtag](https://script-ware.gitbook.io/cyberdrop-dl/getting-started/cyberdrop-dl-install#id-2.-manual-python-install) 2\. Manual Python Install If you do not want to or can't use `uv`, you will need to install a compatible python version manually. If you don't have python, you can find and download it from their official website: [https://www.python.org/downloads/arrow-up-right](https://www.python.org/downloads/) circle-info Cyberdrop-DL requires python 3.11, 3.12 or python 3.13. The **RECOMMENDED** version is 3.12+ circle-exclamation If you are using a version of Cyberdrop-DL from the previous repository (`cyberdrop-dl` instead of `cyberdrop-dl-patched`), you **MUST** uninstall it before installing the patched version. Once you have python, you can install Cyberdrop-DL directly from pypi using `pipx` or `pip`. In a command prompt/terminal window: pipx pip * Install `pipx` (if you don't have it already) * Install cdl circle-exclamation Using bare `pip` to install `cyberdrop-dl-patched` is discouraged as it may lead to dependency conflicts with global installs and an inconsistent environment. Consider using `uv` or `pipx` [PreviousGetting Startedchevron-left](https://script-ware.gitbook.io/cyberdrop-dl/getting-started) [NextRunning CDL on Androidchevron-right](https://script-ware.gitbook.io/cyberdrop-dl/getting-started/cyberdrop-dl-on-android) Last updated 5 months ago * [Using Start Scripts (From Release Page)](https://script-ware.gitbook.io/cyberdrop-dl/getting-started/cyberdrop-dl-install#using-start-scripts-from-release-page) * [Manual Install](https://script-ware.gitbook.io/cyberdrop-dl/getting-started/cyberdrop-dl-install#manual-install) * [1\. Using uv](https://script-ware.gitbook.io/cyberdrop-dl/getting-started/cyberdrop-dl-install#id-1.-using-uv) * [2\. Manual Python Install](https://script-ware.gitbook.io/cyberdrop-dl/getting-started/cyberdrop-dl-install#id-2.-manual-python-install) sun-brightdesktopmoon Copy curl -LsSf https://astral.sh/uv/install.sh | sh Copy powershell -ExecutionPolicy ByPass -c "irm https://astral.sh/uv/install.ps1 | iex" Copy uv tool install cyberdrop-dl-patched Copy pip uninstall cyberdrop-dl Copy pip install pipx Copy pipx install cyberdrop-dl-patched Copy pip install cyberdrop-dl-patched sun-brightdesktopmoon --- # SimpCity Support Dropped | Cyberdrop-DL [hashtag](https://script-ware.gitbook.io/cyberdrop-dl/simpcity-support-dropped#removal-of-simpcity-scraping-support) Removal of SimpCity Scraping Support -------------------------------------------------------------------------------------------------------------------------------------------------------------- Support for scraping SimpCity has been removed temporarily. This may be disappointing to some users, but the explanation provided should clarify the reasoning behind this decision. Following the recent DDOS attacks, SimpCity has implemented the following security measures to protect their website: * SimpCity has implemented a DDOS-Guard browser check. * Access is now restricted to [whitelisted email domainsarrow-up-right](https://simpcity.su/threads/emails-august-2024.365869/) . * [New rate limitsarrow-up-right](https://simpcity.su/threads/rate-limit-429-error.397746/) have been introduced to protect the site. [hashtag](https://script-ware.gitbook.io/cyberdrop-dl/simpcity-support-dropped#why-scraping-simpcity-has-been-disabled) Why Scraping SimpCity Has Been Disabled -------------------------------------------------------------------------------------------------------------------------------------------------------------------- Cyberdrop-DL allows users to scrape a model’s entire thread quickly, downloading large amounts of content. While convenient, this has created several issues: * Large-scale scraping can result in downloading content you may never view. * Such scraping puts additional load on their servers, particularly during DDOS attacks. * Scraping provides no direct benefit to SimpCity, and in light of these attacks, they have chosen not to support automated scraping. For these reasons, SimpCity has removed the Cyberdrop-DL thread, and scraping for their site has been disabled in order to reduce the strain on their servers. [hashtag](https://script-ware.gitbook.io/cyberdrop-dl/simpcity-support-dropped#how-to-reduce-the-impact-of-cyberdrop-dl-usage) How to Reduce the Impact of Cyberdrop-DL Usage ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- To help reduce the load on other websites, the `update_last_forum_post` setting has been enabled by default in all users' configs. This setting speeds up scrapes by picking up where the last session left off, rather than re-scraping entire threads. You can disable this setting if you prefer, but it is strongly discouraged. In addition, the default rate-limiting settings have been adjusted to further reduce the impact on websites. [hashtag](https://script-ware.gitbook.io/cyberdrop-dl/simpcity-support-dropped#usage-best-practices) Usage Best Practices ------------------------------------------------------------------------------------------------------------------------------ I encourage you to use the program responsibly. Here are some tips to help minimize the impact on websites: * Avoid running multiple scraping sessions repeatedly in a short span of time. * Regularly clean up your URLs file by removing links you no longer need, and reducing how often you run the program. * Be selective with what you download. It’s helpful to quickly scan content to avoid downloading files you'll delete later. [hashtag](https://script-ware.gitbook.io/cyberdrop-dl/simpcity-support-dropped#alternatives-to-scraping-simpcity) Alternatives to Scraping SimpCity -------------------------------------------------------------------------------------------------------------------------------------------------------- If you wish to continue downloading from SimpCity, you can use Tampermonkey scripts like these ones: * [SimpCity Tampermonkey Forum Downloaderarrow-up-right](https://simpcity.su/threads/forum-post-downloader-tampermonkey-script.96714/) . * [Forums Link Grabberarrow-up-right](https://github.com/Garcarius/forumslinkgraber) [PreviousMigration to Cyberdrop-DL-Patchedchevron-left](https://script-ware.gitbook.io/cyberdrop-dl/migration-to-cyberdrop-dl-patched) [NextSupported Websiteschevron-right](https://script-ware.gitbook.io/cyberdrop-dl/reference/supported-websites) Last updated 5 months ago * [Removal of SimpCity Scraping Support](https://script-ware.gitbook.io/cyberdrop-dl/simpcity-support-dropped#removal-of-simpcity-scraping-support) * [Why Scraping SimpCity Has Been Disabled](https://script-ware.gitbook.io/cyberdrop-dl/simpcity-support-dropped#why-scraping-simpcity-has-been-disabled) * [How to Reduce the Impact of Cyberdrop-DL Usage](https://script-ware.gitbook.io/cyberdrop-dl/simpcity-support-dropped#how-to-reduce-the-impact-of-cyberdrop-dl-usage) * [Usage Best Practices](https://script-ware.gitbook.io/cyberdrop-dl/simpcity-support-dropped#usage-best-practices) * [Alternatives to Scraping SimpCity](https://script-ware.gitbook.io/cyberdrop-dl/simpcity-support-dropped#alternatives-to-scraping-simpcity) sun-brightdesktopmoon sun-brightdesktopmoon --- # Transition to V8 | Cyberdrop-DL triangle-exclamation V6 introduced some breaking changes: Using a more strict config validation logic, replacing `md5` with `xxh128` as the default hashing algorithm, using a new database schema, among others. It's recommended to do a manual backup of your current `AppData` folder. You won't be able to rollback to a previous version after the transfer is completed. You can learn more about these changes on the [release announcementarrow-up-right](https://github.com/jbsparrow/CyberDropDownloader/blob/master/CHANGELOG.md#600---2024-12-23) circle-exclamation Even after a successful configuration migration, the program may not start if some values from the previous version’s config are no longer valid. Please follow the on-screen instructions to resolve the issue. You can use the [Config Options page](https://script-ware.gitbook.io/cyberdrop-dl/reference/configuration-options) as reference for valid config values [hashtag](https://script-ware.gitbook.io/cyberdrop-dl/upgrade#import-v5-v6-or-v7-config-into-v8) Import V5, V6 or v7 config into v8 ---------------------------------------------------------------------------------------------------------------------------------------- Good news! If you have a v5-v7 config, CDL can import it. The program will automatically detect the config in the current folder and handle the migration process for you. All you have to do is replace your old start script (for Windows, that's the `.bat` file) with the v8 start script. Please note that after the migration is complete, you may still need to manually adjust some values in the config. If you are migrating from v5 and the migration fails, the database will be rolled back to its previous state. An automatic backup is also made before the migration begins. However, keep in mind that the backup only includes the database, not the config files. circle-info Support for v5 and v6 (config, database and auto-migration) is deprecated and will be removed in v9 [hashtag](https://script-ware.gitbook.io/cyberdrop-dl/upgrade#import-v4-config) Import V4 config ----------------------------------------------------------------------------------------------------- circle-exclamation V4 import is no longer supported. Last version with v4 support was `6.5.0`. To upgrade to the latest version, you need to perform a sequential upgrade. First, update CDL to `6.5.0`, import your v4 files, and then update to the latest version. You can install `6.5.0` with pip by running: Copy pip install cyberdrop-dl-patched==6.5.0 Built into Cyberdrop-DL is a tool that allows you to import both your configs and your history DB from a v4 version of Cyberdrop-DL. You can use it by selecting "Import V4 Items". ### [hashtag](https://script-ware.gitbook.io/cyberdrop-dl/upgrade#import-your-v4-configs) Import Your V4 Configs The config files will be located in the folder that you were previously running Cyberdrop-DL in. Start CDL, choose the option `Import v4 config` and follow the on screen instructions circle-info If you weren't using the config previously, you don't need to import it. However, if you were primarily using CLI Arguments with V4, some of the arguments has been replaced. If you don't end up using the import feature, make sure you also change the default config in the program if that's something you want to do. ### [hashtag](https://script-ware.gitbook.io/cyberdrop-dl/upgrade#import-your-v4-database) Import Your V4 Database For a lot of people, the `download_history.sqlite` file will be in the same folder as your start file (or wherever you are running Cyberdrop-DL). If it's not there, you can find it here: Windows: macOS: Linux: To import it, start CDL, choose the option `Import v4 database` and follow the on screen instructions circle-info The old `download_history.sqlite` file is no longer used by Cyberdrop-DL. After you import it, you can delete the old one. If you don't want to import previous download history, you can just delete it. [PreviousRunning CDL on Androidchevron-left](https://script-ware.gitbook.io/cyberdrop-dl/getting-started/cyberdrop-dl-on-android) [NextFrequently Asked Questionschevron-right](https://script-ware.gitbook.io/cyberdrop-dl/frequently-asked-questions) Last updated 5 months ago sun-brightdesktopmoon Copy C:\Users\\AppData\Local\Cyberdrop-DL\Cyberdrop-DL\download_history.sqlite Copy /Library/Application Support/Cyberdrop-DL/Cyberdrop-DL/download_history.sqlite Copy /home//.local/share/Cyberdrop-DL/Cyberdrop-DL/download_history.sqlite sun-brightdesktopmoon --- # Special Type Options | Cyberdrop-DL [hashtag](https://script-ware.gitbook.io/cyberdrop-dl/reference/configuration-options/special_setting_types#additiveargs) `AdditiveArgs` --------------------------------------------------------------------------------------------------------------------------------------------- Special type for some settings that accept multiple values as input (`lists` or `sets`). When passing values as CLI arguments, if the first value is `+`, any value after it will be added to the values specified in the config file instead of overriding it. Similarly, if the first value is `-`, any value after it will be removed from the values specified in the config ### [hashtag](https://script-ware.gitbook.io/cyberdrop-dl/reference/configuration-options/special_setting_types#examples) Examples If you have the following `skip_hosts` setting in you config file: Copy skip_hosts: - drive.google.com - youtube.com - facebook.com You will get the following results: Value you used Value CDL interpreted Details `--skip-hosts instagram.com` `[instagram.com]` config values were overridden by CLI values `--skip-hosts "+" instagram.com` `[drive.google.com, youtube.com, facebook.com, instagram.com]` config values and CLI values were merged `--skip-hosts "-" drive.google.com youtube.com` `[facebook.com]` CLI values were removed from the config values circle-info Always use quotes for "+" and "-" to make sure your shell does not try to parse them as additional flags [hashtag](https://script-ware.gitbook.io/cyberdrop-dl/reference/configuration-options/special_setting_types#boolflag) `BoolFlag` ------------------------------------------------------------------------------------------------------------------------------------- A special kind of `bool`. Within a config file, it can have a `true` or `false` value. However, when used via CLI, the value is assumed from the flag name. The normal name is `true`, and prefixing the name with `--no` means `false`. ### [hashtag](https://script-ware.gitbook.io/cyberdrop-dl/reference/configuration-options/special_setting_types#examples-1) Examples Value you used Value CDL interpreted `--auto-dedupe` auto-dedupe: `true` `--no-auto-dedupe` auto-dedupe: `false` [hashtag](https://script-ware.gitbook.io/cyberdrop-dl/reference/configuration-options/special_setting_types#bytesize) `ByteSize` ------------------------------------------------------------------------------------------------------------------------------------- A special kind of `int` that also accepts suffixes like `GB`, `MiB` and `KB` to specify valid values In conformance with IEC 80000-13 Standard, `1KB` means `1000 bytes`, and `1KiB` means `1024 bytes`. In general, including a middle 'i' will cause the unit to be interpreted as a power of 2, rather than a power of 10. ### [hashtag](https://script-ware.gitbook.io/cyberdrop-dl/reference/configuration-options/special_setting_types#examples-2) Examples Value you used Value CDL interpreted `1GB` `1.000.000.000 bytes` `512` `512 bytes` `1MiB` `1.048.576 bytes` circle-info `ByteSize` is also used for some settings to specify speed and it's interpreted as ` / second`. ex: `8MB` means `8MB/s` [PreviousConfiguration Optionschevron-left](https://script-ware.gitbook.io/cyberdrop-dl/reference/configuration-options) [NextAuthenticationchevron-right](https://script-ware.gitbook.io/cyberdrop-dl/reference/configuration-options/authentication) Last updated 5 months ago * [AdditiveArgs](https://script-ware.gitbook.io/cyberdrop-dl/reference/configuration-options/special_setting_types#additiveargs) * [Examples](https://script-ware.gitbook.io/cyberdrop-dl/reference/configuration-options/special_setting_types#examples) * [BoolFlag](https://script-ware.gitbook.io/cyberdrop-dl/reference/configuration-options/special_setting_types#boolflag) * [Examples](https://script-ware.gitbook.io/cyberdrop-dl/reference/configuration-options/special_setting_types#examples-1) * [ByteSize](https://script-ware.gitbook.io/cyberdrop-dl/reference/configuration-options/special_setting_types#bytesize) * [Examples](https://script-ware.gitbook.io/cyberdrop-dl/reference/configuration-options/special_setting_types#examples-2) sun-brightdesktopmoon sun-brightdesktopmoon --- # Authentication | Cyberdrop-DL All the options in these list are optional. The default value for all of them is an empty `str` Type Default `str` `""` chevron-rightCoomer[hashtag](https://script-ware.gitbook.io/cyberdrop-dl/reference/configuration-options/authentication#coomer) * `session` Value of the `session` cookie from the website. This will allow CDL to download from your favoruites by using `https://coomer.st/favorites` as an input URL. chevron-rightKemono[hashtag](https://script-ware.gitbook.io/cyberdrop-dl/reference/configuration-options/authentication#kemono) * `session` Value of the `session` cookie from the website. This will allow CDL to download from your favoruites by using `https://kemono.cr/favorites` as an input URL. chevron-rightForums[hashtag](https://script-ware.gitbook.io/cyberdrop-dl/reference/configuration-options/authentication#forums) circle-exclamation Logging to forums with `Authentication` was deprecated in v6.7.0 You need to use cookie files. See: [How to extract cookies (DDoSGuard or login errors) #839arrow-up-right](https://github.com/jbsparrow/CyberDropDownloader/discussions/839) chevron-rightGoFile[hashtag](https://script-ware.gitbook.io/cyberdrop-dl/reference/configuration-options/authentication#gofile) If you decide to pay for GoFile Premium (faster downloads, etc.) you can provide your API key to Cyberdrop-DL in order for the program to use it. * `api_key` You can get your API key here: [https://gofile.io/myProfilearrow-up-right](https://gofile.io/myProfile) chevron-rightJDownloader[hashtag](https://script-ware.gitbook.io/cyberdrop-dl/reference/configuration-options/authentication#jdownloader) Under JDownloader 2 settings -> MyJDownloader: * `username` * `password` * `device` chevron-rightPixelDrain[hashtag](https://script-ware.gitbook.io/cyberdrop-dl/reference/configuration-options/authentication#pixeldrain) If you decide to pay for PixelDrain premium (faster downloads, etc.) you can provide your API key to Cyberdrop-DL in order for the program to use it. * `api_key` You can get your API key here: [https://pixeldrain.com/user/api\_keysarrow-up-right](https://pixeldrain.com/user/api_keys) chevron-rightReal-Debrid[hashtag](https://script-ware.gitbook.io/cyberdrop-dl/reference/configuration-options/authentication#real-debrid) In order to download files from sites supported by real-debrid, you'll need to get the API token from your account. * `api_key` You can get your API key here (you must be logged in): [https://real-debrid.com/apitokenarrow-up-right](https://real-debrid.com/apitoken) chevron-rightReddit[hashtag](https://script-ware.gitbook.io/cyberdrop-dl/reference/configuration-options/authentication#reddit) In order to scrape files from Reddit, you'll need to create an app on reddit's website (it's free): [https://www.reddit.com/prefs/appsarrow-up-right](https://www.reddit.com/prefs/apps) Select `script` as the app type. Any name can be used. For the redirect URI you can use any website, even a fake one like `http://your_username.cyberdrop-dl`. Click `create app` to get your credentials. ![](https://script-ware.gitbook.io/cyberdrop-dl/~gitbook/image?url=https%3A%2F%2F2790613906-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252FkSSbsrZtznOSkleG0nUZ%252Fuploads%252Fgit-blob-f93398ada05724268bd07398f61ccc6e4a112ad7%252Freddit_personal_script_setup_1.png%3Falt%3Dmedia&width=768&dpr=3&quality=100&sign=59c4f5fb&sv=2) reddit\_personal\_script\_setup\_1 ![](https://script-ware.gitbook.io/cyberdrop-dl/~gitbook/image?url=https%3A%2F%2F2790613906-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252FkSSbsrZtznOSkleG0nUZ%252Fuploads%252Fgit-blob-d4567c1b4960f9278f014e1d6e677a33bf3c9ea1%252Freddit_personal_script_setup_2.png%3Falt%3Dmedia&width=768&dpr=3&quality=100&sign=d8eb11e3&sv=2) reddit\_personal\_script\_setup\_2 After generating the app, you need to provide Cyberdrop-DL these values: * `personal_use_script` * `secret` chevron-rightMega.nz[hashtag](https://script-ware.gitbook.io/cyberdrop-dl/reference/configuration-options/authentication#mega.nz) If you have premium or want to download files/folders only shared with you, provide CDL these values: * `email` * `password` circle-exclamation Accounts with 2 factor authentication (2FA) are NOT supported [PreviousSpecial Type Optionschevron-left](https://script-ware.gitbook.io/cyberdrop-dl/reference/configuration-options/special_setting_types) [NextSettingschevron-right](https://script-ware.gitbook.io/cyberdrop-dl/reference/configuration-options/settings) Last updated 3 months ago sun-brightdesktopmoon sun-brightdesktopmoon --- # Configuration Options | Cyberdrop-DL There are three type of config files: config settings, global settings and authentication settings. [user-lockAuthenticationchevron-right](https://script-ware.gitbook.io/cyberdrop-dl/reference/configuration-options/authentication) [earth-americasGlobal Settingschevron-right](https://script-ware.gitbook.io/cyberdrop-dl/reference/configuration-options/global-settings) [screwdriver-wrenchSettingschevron-right](https://script-ware.gitbook.io/cyberdrop-dl/reference/configuration-options/settings) circle-info **Global Settings** and **Authentication Settings** are "global". They apply to _all_ config settings. You can set them once and never touch them again. Each config setting will be setup by default to have a separate `URLs.txt` files and separate log files. [PreviousSupported Websiteschevron-left](https://script-ware.gitbook.io/cyberdrop-dl/reference/supported-websites) [NextSpecial Type Optionschevron-right](https://script-ware.gitbook.io/cyberdrop-dl/reference/configuration-options/special_setting_types) Last updated 3 months ago sun-brightdesktopmoon sun-brightdesktopmoon --- # Settings | Cyberdrop-DL [Browser Cookieschevron-right](https://script-ware.gitbook.io/cyberdrop-dl/reference/configuration-options/settings/browser_cookies) [Download Optionschevron-right](https://script-ware.gitbook.io/cyberdrop-dl/reference/configuration-options/settings/download_options) [Dupe Cleanup Optionschevron-right](https://script-ware.gitbook.io/cyberdrop-dl/reference/configuration-options/settings/dupe_cleanup_options) [File Size Limitschevron-right](https://script-ware.gitbook.io/cyberdrop-dl/reference/configuration-options/settings/file_size_limits) [Fileschevron-right](https://script-ware.gitbook.io/cyberdrop-dl/reference/configuration-options/settings/files) [Ignore Optionschevron-right](https://script-ware.gitbook.io/cyberdrop-dl/reference/configuration-options/settings/ignore_options) [Logschevron-right](https://script-ware.gitbook.io/cyberdrop-dl/reference/configuration-options/settings/logs) [Media Duration Limitschevron-right](https://script-ware.gitbook.io/cyberdrop-dl/reference/configuration-options/settings/media_duration_limits) [Runtime Optionschevron-right](https://script-ware.gitbook.io/cyberdrop-dl/reference/configuration-options/settings/runtime_options) [Sortingchevron-right](https://script-ware.gitbook.io/cyberdrop-dl/reference/configuration-options/settings/sorting) [PreviousAuthenticationchevron-left](https://script-ware.gitbook.io/cyberdrop-dl/reference/configuration-options/authentication) [NextBrowser Cookieschevron-right](https://script-ware.gitbook.io/cyberdrop-dl/reference/configuration-options/settings/browser_cookies) Last updated 1 year ago sun-brightdesktopmoon sun-brightdesktopmoon --- # Notifications | Cyberdrop-DL Cyberdrop-DL generates a report at the end of a run with stats about all the downloads, total runtime, errors, deduplication report, etc. By default, this report is only shown in the console and at the end of the main log file. You can set up CDL to sent you the report via discord, email, a native notification of your OS, telegram and many other services. [hashtag](https://script-ware.gitbook.io/cyberdrop-dl/reference/notifications#notifications-via-discord) Notifications via Discord --------------------------------------------------------------------------------------------------------------------------------------- To get notifications via discord, you need to provide a discord `webhook_url` inside the `setting.yaml` of the config you are running. You can learn how to setup a webhook following the [official discord guidearrow-up-right](https://support.discord.com/hc/en-us/articles/228383668-Intro-to-Webhooks) . Optionally, you can add the tag `attach_logs=` as a prefix to your webhook url. This will tell CDL to include a copy of the main log as an attachment to Discord. [hashtag](https://script-ware.gitbook.io/cyberdrop-dl/reference/notifications#notifications-to-other-services-via-apprise) Notifications to other services (via Apprise) ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Cyberdrop-DL uses [Apprisearrow-up-right](https://github.com/caronc/apprise) to send notifications to any of the services than they support. ### [hashtag](https://script-ware.gitbook.io/cyberdrop-dl/reference/notifications#how-to-setup-apprise) How to setup Apprise To send notifications via Apprise, you need to create an `apprise.txt` file inside `AppData/Configs/`, where `` if the config you want to use. The file must contain a list of URLs and they must be in the format of one of the supported apprise services. You can check the full list of supported services [herearrow-up-right](https://github.com/caronc/apprise/wiki) and the URL format than each one uses [herearrow-up-right](https://github.com/caronc/apprise?tab=readme-ov-file#supported-notifications) . Apprise services also support the `attach_logs=` tag to send the main log as an attachment. ### [hashtag](https://script-ware.gitbook.io/cyberdrop-dl/reference/notifications#troubleshooting-apprise-notifications) Troubleshooting Apprise notifications Cyberdrop-DL will show you a message at the end of a run telling you if the apprise notifications were successfully sent or not. If you are having trouble getting notifications via Apprise, follow their [troubleshooting guidearrow-up-right](https://github.com/caronc/apprise/wiki/Troubleshooting) . circle-info When running on Windows, Cyberdrop-DL will setup OS notifications by default. You can disable them by deleting the `windows://` line from the default `apprise.txt` file. You can also completely delete the file if you don't have any other notification setup. [hashtag](https://script-ware.gitbook.io/cyberdrop-dl/reference/notifications#examples) Examples ----------------------------------------------------------------------------------------------------- Email Email + Logs Native OS notifications Discord + Logs To get notifications via email, use this URL format in your `apprise.txt` file: Add `attach_logs` to your email URL in your `apprise.txt` file: Some operating systems require additional dependencies for notifications to work. Cyberdrop-DL includes the required dependencies for Windows. Follow the url on the OS name to get additional information on how to set them up. OS Syntax [Linux (DBus Notifications)arrow-up-right](https://github.com/caronc/apprise/wiki/Notify_dbus) `dbus://` `qt://` `glib://` `kde://` [Linux (Gnome Notifications)arrow-up-right](https://github.com/caronc/apprise/wiki/Notify_gnome) `gnome://` [macOSarrow-up-right](https://github.com/caronc/apprise/wiki/Notify_macosx) `macosx://` [Windowsarrow-up-right](https://github.com/caronc/apprise/wiki/Notify_windows) `windows://` Add `attach_logs` to the `webhook_url` config option: [PreviousCLI Argumentschevron-left](https://script-ware.gitbook.io/cyberdrop-dl/reference/cli-arguments) Last updated 5 months ago * [Notifications via Discord](https://script-ware.gitbook.io/cyberdrop-dl/reference/notifications#notifications-via-discord) * [Notifications to other services (via Apprise)](https://script-ware.gitbook.io/cyberdrop-dl/reference/notifications#notifications-to-other-services-via-apprise) * [How to setup Apprise](https://script-ware.gitbook.io/cyberdrop-dl/reference/notifications#how-to-setup-apprise) * [Troubleshooting Apprise notifications](https://script-ware.gitbook.io/cyberdrop-dl/reference/notifications#troubleshooting-apprise-notifications) * [Examples](https://script-ware.gitbook.io/cyberdrop-dl/reference/notifications#examples) sun-brightdesktopmoon Copy mailto://user:password@domain.com Copy attach_logs=mailto://user:password@domain.com Copy attach_logs=https://discord.com/api/webhooks/webhook_id/webhook_token sun-brightdesktopmoon --- # Welcome! | Cyberdrop-DL Development Docs | Cyberdrop-DL Cyberdrop-DL is an open source asynchronous downloader maintained by [jbsparrowarrow-up-right](https://github.com/jbsparrow) . It's goal is to act as a bulk downloader for various sites you will find around the web. [globe-pointerSupported Websiteschevron-right](https://script-ware.gitbook.io/cyberdrop-dl/dev/reference/supported-websites) [hashtag](https://script-ware.gitbook.io/cyberdrop-dl/dev#want-to-jump-right-in) Want to jump right in? ------------------------------------------------------------------------------------------------------------ Explore the Getting Started guide to quickly familiarize yourself with the setup process and get started with the software. [bullseye-arrowGetting Startedchevron-right](https://script-ware.gitbook.io/cyberdrop-dl/dev/getting-started) [hashtag](https://script-ware.gitbook.io/cyberdrop-dl/dev#want-to-customize-your-experience) Want to customize your experience? ------------------------------------------------------------------------------------------------------------------------------------ Take a closer look and begin exploring the available configuration options and CLI arguments: [slidersConfiguration Optionschevron-right](https://script-ware.gitbook.io/cyberdrop-dl/dev/reference/configuration-options) [rectangle-terminalCLI Argumentschevron-right](https://script-ware.gitbook.io/cyberdrop-dl/dev/reference/cli-arguments) [envelope-open-textNotificationschevron-right](https://script-ware.gitbook.io/cyberdrop-dl/dev/reference/notifications) [NextGetting Startedchevron-right](https://script-ware.gitbook.io/cyberdrop-dl/dev/getting-started) Last updated 1 year ago sun-brightdesktopmoon sun-brightdesktopmoon --- # Global Settings | Cyberdrop-DL [Generalchevron-right](https://script-ware.gitbook.io/cyberdrop-dl/reference/configuration-options/global-settings/general) [Rate Limiting Optionschevron-right](https://script-ware.gitbook.io/cyberdrop-dl/reference/configuration-options/global-settings/rate-limiting-options) [UI Optionschevron-right](https://script-ware.gitbook.io/cyberdrop-dl/reference/configuration-options/global-settings/ui-options) [Generic Crawlerschevron-right](https://script-ware.gitbook.io/cyberdrop-dl/reference/configuration-options/global-settings/generic_crawlers_instances) [PreviousSortingchevron-left](https://script-ware.gitbook.io/cyberdrop-dl/reference/configuration-options/settings/sorting) [NextGeneralchevron-right](https://script-ware.gitbook.io/cyberdrop-dl/reference/configuration-options/global-settings/general) Last updated 1 year ago sun-brightdesktopmoon sun-brightdesktopmoon --- # Migration to Cyberdrop-DL-Patched | Cyberdrop-DL Development Docs | Cyberdrop-DL If you used the original version of the package (`cyberdrop-dl` instead of `cyberdrop-dl-patched`), you can follow the steps below to migrate. circle-exclamation You may need to adjust your config to make sure it's compatible with the newer versions. See: [Transition to V8arrow-up-right](https://script-ware.gitbook.io/cyberdrop-dl/upgrade) [hashtag](https://script-ware.gitbook.io/cyberdrop-dl/dev/migration-to-cyberdrop-dl-patched#if-you-installed-on-your-own-using-pip) If you installed on your own using pip ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- You can simply uninstall the old version and install the new one using the following commands: Copy pip uninstall -y cyberdrop-dl pip install cyberdrop-dl-patched The command to use the new version will be `cyberdrop-dl-patched`. [hashtag](https://script-ware.gitbook.io/cyberdrop-dl/dev/migration-to-cyberdrop-dl-patched#if-you-installed-using-the-start-files) If you installed using the start files ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- You can find the new ones here: [new start scriptsarrow-up-right](https://github.com/jbsparrow/CyberDropDownloader/releases/latest) Put the new start scripts on the same folder as the old ones, them delete the old ones. You also have to delete any `venv` or `.venv` folder (if you have any) [PreviousFrequently Asked Questionschevron-left](https://script-ware.gitbook.io/cyberdrop-dl/dev/frequently-asked-questions) [NextSimpCity Support Droppedchevron-right](https://script-ware.gitbook.io/cyberdrop-dl/dev/simpcity-support-dropped) Last updated 5 months ago sun-brightdesktopmoon sun-brightdesktopmoon --- # Files | Cyberdrop-DL [hashtag](https://script-ware.gitbook.io/cyberdrop-dl/reference/configuration-options/settings/files#download_folder) `download_folder` -------------------------------------------------------------------------------------------------------------------------------------------- Type Default `Path` `Downloads` The path to the folder you want Cyberdrop-DL to download files to. [hashtag](https://script-ware.gitbook.io/cyberdrop-dl/reference/configuration-options/settings/files#dump_json) `dump_json` -------------------------------------------------------------------------------------------------------------------------------- Type Default `bool` `False` If enabled, CDL will created a [json linesarrow-up-right](https://jsonlines.org/) files with the information about every file downloaded in the current run. The path to this file will be the same as `--main-log` but with the extension `.results.jsonl` Each line in the file will contain the following details (this may change on future versions): Copy { "url": "https://store9.gofile.io/download/web/7c88c147-ABCD-4e4d-9a6c-12345678/a_video.mp4", "referer": "https://gofile.io/d/ABC123", "download_folder": "Downloads/Cyberdrop-DL Downloads/test_album (GoFile)", "filename": "0hxte0li0o931lwgcrzbz_source.mp4", "original_filename": "a_video.mp4", "download_filename": "0hxte0li0o931lwgcrzbz_source.mp4", "filesize": 12054723, "ext": ".mp4", "debrid_link": null, "duration": null, "album_id": "ABC123", "datetime": "2025-01-22T11:00:07", "parents": ["https://a_forum.com/threads/.54321/post-123123"], "parent_threads": ["https://a_forum.com/threads/.54321"], "partial_file": "Downloads/Cyberdrop-DL Downloads/test_album (GoFile)/a_video.mp4.part", "complete_file": "Downloads/Cyberdrop-DL Downloads/test_album (GoFile)/a_video.mp4", "hash": "xxh128:53ee56b7bfafa31b8780a572e9783df3", "downloaded": true, "attempts": 1 } [hashtag](https://script-ware.gitbook.io/cyberdrop-dl/reference/configuration-options/settings/files#input_file) `input_file` ---------------------------------------------------------------------------------------------------------------------------------- Type Default `Path` `AppData/Configs/{config}/URLs.txt` The path to the text file containing the URLs you want to download. Each line should be a single URL. You can also use `html` code. Cyberdrop-DL will parse all the links on the HTML [hashtag](https://script-ware.gitbook.io/cyberdrop-dl/reference/configuration-options/settings/files#save_pages_html) `save_pages_html` -------------------------------------------------------------------------------------------------------------------------------------------- Type Default `bool` `False` CDL will save to disk a copy of every requests as an html file. The files will be saved to a folder named `cdl_responses`, located in the same folder as the main log file. circle-info Not every request made by CDL returns an HTML page (ex: API requests generally return JSON data). Only HTML responses will be saved [PreviousFile Size Limitschevron-left](https://script-ware.gitbook.io/cyberdrop-dl/reference/configuration-options/settings/file_size_limits) [NextIgnore Optionschevron-right](https://script-ware.gitbook.io/cyberdrop-dl/reference/configuration-options/settings/ignore_options) Last updated 5 months ago * [download\_folder](https://script-ware.gitbook.io/cyberdrop-dl/reference/configuration-options/settings/files#download_folder) * [dump\_json](https://script-ware.gitbook.io/cyberdrop-dl/reference/configuration-options/settings/files#dump_json) * [input\_file](https://script-ware.gitbook.io/cyberdrop-dl/reference/configuration-options/settings/files#input_file) * [save\_pages\_html](https://script-ware.gitbook.io/cyberdrop-dl/reference/configuration-options/settings/files#save_pages_html) sun-brightdesktopmoon sun-brightdesktopmoon --- # UI Options | Cyberdrop-DL [hashtag](https://script-ware.gitbook.io/cyberdrop-dl/reference/configuration-options/global-settings/ui-options#downloading_item_limit) `downloading_item_limit` ---------------------------------------------------------------------------------------------------------------------------------------------------------------------- Type Default `PositiveInt` `5` This is the limit on the number of items shown in the UI (while downloading) before they are simply added to the overflow number (`and other files`) [hashtag](https://script-ware.gitbook.io/cyberdrop-dl/reference/configuration-options/global-settings/ui-options#refresh_rate) `refresh_rate` -------------------------------------------------------------------------------------------------------------------------------------------------- Type Default `PositiveInt` `10` This is the refresh rate per second for the UI. [hashtag](https://script-ware.gitbook.io/cyberdrop-dl/reference/configuration-options/global-settings/ui-options#scraping_item_limit) `scraping_item_limit` ---------------------------------------------------------------------------------------------------------------------------------------------------------------- Type Default `PositiveInt` `5` This is the limit on the number of items shown in the UI (while scraping) before they are simply added to the overflow number (`and other links`) [hashtag](https://script-ware.gitbook.io/cyberdrop-dl/reference/configuration-options/global-settings/ui-options#vi_mode) `vi_mode` ---------------------------------------------------------------------------------------------------------------------------------------- Type Default `bool` `false` This enables vi/vim key binds while editing/entering text in CDL. [PreviousRate Limiting Optionschevron-left](https://script-ware.gitbook.io/cyberdrop-dl/reference/configuration-options/global-settings/rate-limiting-options) [NextGeneric Crawlerschevron-right](https://script-ware.gitbook.io/cyberdrop-dl/reference/configuration-options/global-settings/generic_crawlers_instances) Last updated 1 year ago * [downloading\_item\_limit](https://script-ware.gitbook.io/cyberdrop-dl/reference/configuration-options/global-settings/ui-options#downloading_item_limit) * [refresh\_rate](https://script-ware.gitbook.io/cyberdrop-dl/reference/configuration-options/global-settings/ui-options#refresh_rate) * [scraping\_item\_limit](https://script-ware.gitbook.io/cyberdrop-dl/reference/configuration-options/global-settings/ui-options#scraping_item_limit) * [vi\_mode](https://script-ware.gitbook.io/cyberdrop-dl/reference/configuration-options/global-settings/ui-options#vi_mode) sun-brightdesktopmoon sun-brightdesktopmoon --- # Running CDL on Android | Cyberdrop-DL Development Docs | Cyberdrop-DL Cyberdrop-dl is a terminal app. That means you need a terminal emulator to run it. The defacto choice in Android is [termuxarrow-up-right](https://termux.dev/en/) . circle-info Most of the dependencies need to be compiled from source. A rust compiler is required. This means the installation could take several minutes, especially on low end phones circle-exclamation Compiling from source also requires a lot of extra storage. You will need at least 3.4GB just for the installation of CDL [hashtag](https://script-ware.gitbook.io/cyberdrop-dl/dev/getting-started/cyberdrop-dl-on-android#id-1.-install-termux) 1\. Install `termux` ------------------------------------------------------------------------------------------------------------------------------------------------- Termux wiki: [https://wiki.termux.com/wiki/Installationarrow-up-right](https://wiki.termux.com/wiki/Installation) Install termux from [F-droid (recommended)arrow-up-right](https://f-droid.org/packages/com.termux/) or from the [Google Playstore (restricted version)arrow-up-right](https://play.google.com/store/apps/details?id=com.termux) : [hashtag](https://script-ware.gitbook.io/cyberdrop-dl/dev/getting-started/cyberdrop-dl-on-android#id-2.-install-cyberdrop-dl-patched) 2\. Install `cyberdrop-dl-patched` ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Run the following commands inside termux Copy # Get storage access permission termux-setup-storage # Install dependencies pkg upgrade -y pkg install rust which micro libjpeg-turbo python uv -y # Install CDL and setup appdata folder uv tool install cyberdrop-dl-patched uv tool update-shell mkdir /sdcard/cyberdrop-dl echo 'alias cyberdrop-dl="cyberdrop-dl --portrait --appdata-folder /sdcard/cyberdrop-dl"' >> ~/.bashrc source ~/.bashrc [hashtag](https://script-ware.gitbook.io/cyberdrop-dl/dev/getting-started/cyberdrop-dl-on-android#how-to-update-cyberdrop-dl-patched) How to update `cyberdrop-dl-patched`? -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Run this command inside termux: [PreviousCyberdrop-DL Installchevron-left](https://script-ware.gitbook.io/cyberdrop-dl/dev/getting-started/cyberdrop-dl-install) [NextTransition to V8chevron-right](https://script-ware.gitbook.io/cyberdrop-dl/dev/upgrade) Last updated 5 months ago * [1\. Install termux](https://script-ware.gitbook.io/cyberdrop-dl/dev/getting-started/cyberdrop-dl-on-android#id-1.-install-termux) * [2\. Install cyberdrop-dl-patched](https://script-ware.gitbook.io/cyberdrop-dl/dev/getting-started/cyberdrop-dl-on-android#id-2.-install-cyberdrop-dl-patched) * [How to update cyberdrop-dl-patched?](https://script-ware.gitbook.io/cyberdrop-dl/dev/getting-started/cyberdrop-dl-on-android#how-to-update-cyberdrop-dl-patched) sun-brightdesktopmoon Copy uv tool upgrade cyberdrop-dl-patched sun-brightdesktopmoon --- # SimpCity Support Dropped | Cyberdrop-DL Development Docs | Cyberdrop-DL [hashtag](https://script-ware.gitbook.io/cyberdrop-dl/dev/simpcity-support-dropped#removal-of-simpcity-scraping-support) Removal of SimpCity Scraping Support ------------------------------------------------------------------------------------------------------------------------------------------------------------------ Support for scraping SimpCity has been removed temporarily. This may be disappointing to some users, but the explanation provided should clarify the reasoning behind this decision. Following the recent DDOS attacks, SimpCity has implemented the following security measures to protect their website: * SimpCity has implemented a DDOS-Guard browser check. * Access is now restricted to [whitelisted email domainsarrow-up-right](https://simpcity.su/threads/emails-august-2024.365869/) . * [New rate limitsarrow-up-right](https://simpcity.su/threads/rate-limit-429-error.397746/) have been introduced to protect the site. [hashtag](https://script-ware.gitbook.io/cyberdrop-dl/dev/simpcity-support-dropped#why-scraping-simpcity-has-been-disabled) Why Scraping SimpCity Has Been Disabled ------------------------------------------------------------------------------------------------------------------------------------------------------------------------ Cyberdrop-DL allows users to scrape a model’s entire thread quickly, downloading large amounts of content. While convenient, this has created several issues: * Large-scale scraping can result in downloading content you may never view. * Such scraping puts additional load on their servers, particularly during DDOS attacks. * Scraping provides no direct benefit to SimpCity, and in light of these attacks, they have chosen not to support automated scraping. For these reasons, SimpCity has removed the Cyberdrop-DL thread, and scraping for their site has been disabled in order to reduce the strain on their servers. [hashtag](https://script-ware.gitbook.io/cyberdrop-dl/dev/simpcity-support-dropped#how-to-reduce-the-impact-of-cyberdrop-dl-usage) How to Reduce the Impact of Cyberdrop-DL Usage -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- To help reduce the load on other websites, the `update_last_forum_post` setting has been enabled by default in all users' configs. This setting speeds up scrapes by picking up where the last session left off, rather than re-scraping entire threads. You can disable this setting if you prefer, but it is strongly discouraged. In addition, the default rate-limiting settings have been adjusted to further reduce the impact on websites. [hashtag](https://script-ware.gitbook.io/cyberdrop-dl/dev/simpcity-support-dropped#usage-best-practices) Usage Best Practices ---------------------------------------------------------------------------------------------------------------------------------- I encourage you to use the program responsibly. Here are some tips to help minimize the impact on websites: * Avoid running multiple scraping sessions repeatedly in a short span of time. * Regularly clean up your URLs file by removing links you no longer need, and reducing how often you run the program. * Be selective with what you download. It’s helpful to quickly scan content to avoid downloading files you'll delete later. [hashtag](https://script-ware.gitbook.io/cyberdrop-dl/dev/simpcity-support-dropped#alternatives-to-scraping-simpcity) Alternatives to Scraping SimpCity ------------------------------------------------------------------------------------------------------------------------------------------------------------ If you wish to continue downloading from SimpCity, you can use Tampermonkey scripts like these ones: * [SimpCity Tampermonkey Forum Downloaderarrow-up-right](https://simpcity.su/threads/forum-post-downloader-tampermonkey-script.96714/) . * [Forums Link Grabberarrow-up-right](https://github.com/Garcarius/forumslinkgraber) [PreviousMigration to Cyberdrop-DL-Patchedchevron-left](https://script-ware.gitbook.io/cyberdrop-dl/dev/migration-to-cyberdrop-dl-patched) [NextSupported Websiteschevron-right](https://script-ware.gitbook.io/cyberdrop-dl/dev/reference/supported-websites) Last updated 5 months ago * [Removal of SimpCity Scraping Support](https://script-ware.gitbook.io/cyberdrop-dl/dev/simpcity-support-dropped#removal-of-simpcity-scraping-support) * [Why Scraping SimpCity Has Been Disabled](https://script-ware.gitbook.io/cyberdrop-dl/dev/simpcity-support-dropped#why-scraping-simpcity-has-been-disabled) * [How to Reduce the Impact of Cyberdrop-DL Usage](https://script-ware.gitbook.io/cyberdrop-dl/dev/simpcity-support-dropped#how-to-reduce-the-impact-of-cyberdrop-dl-usage) * [Usage Best Practices](https://script-ware.gitbook.io/cyberdrop-dl/dev/simpcity-support-dropped#usage-best-practices) * [Alternatives to Scraping SimpCity](https://script-ware.gitbook.io/cyberdrop-dl/dev/simpcity-support-dropped#alternatives-to-scraping-simpcity) sun-brightdesktopmoon sun-brightdesktopmoon --- # Browser Cookies | Cyberdrop-DL Cyberdrop-DL can extract cookies from your browser. These can be used for websites that require login or to pass DDoS-Guard challenges. Only cookies from supported websites are extracted circle-exclamation The `user-agent` config value **MUST** match the `user-agent` of the browser from which you imported the cookies. If they do not match, the cookies will not work [hashtag](https://script-ware.gitbook.io/cyberdrop-dl/reference/configuration-options/settings/browser_cookies#auto_import) `auto_import` ---------------------------------------------------------------------------------------------------------------------------------------------- Type Default `bool` `false` Toggles automatic import of cookies at the start of each run [hashtag](https://script-ware.gitbook.io/cyberdrop-dl/reference/configuration-options/settings/browser_cookies#browser) `browser` -------------------------------------------------------------------------------------------------------------------------------------- Type Default `BROWSER` `firefox` A browser to use for extraction. Browser must be a supported browser's name. ### [hashtag](https://script-ware.gitbook.io/cyberdrop-dl/reference/configuration-options/settings/browser_cookies#supported-browsers) Supported Browsers Browser Windows Linux MacOS Arc ❌ ❌ ✅ Brave ❌ ✅ ✅ Chrome ❌ ✅ ✅ Chromium ❌ ✅ ✅ Edge ❌ ✅ ✅ Firefox ✅ ✅ ✅ LibreWolf ✅ ✅ ✅ Lynx ❌ ✅ ❌ Opera ❌ ✅ ✅ Opera\_GX ❌ ❌ ✅ Safari ❌ ❌ ✅ Vivaldi ❌ ✅ ✅ W3M ❌ ✅ ❌ [hashtag](https://script-ware.gitbook.io/cyberdrop-dl/reference/configuration-options/settings/browser_cookies#sites) `sites` ---------------------------------------------------------------------------------------------------------------------------------- Type Default `list[DOMAINS]` `[<>]` List of domains to extract cookies from. You can put any domain on the list, even if they are not officially supported. [hashtag](https://script-ware.gitbook.io/cyberdrop-dl/reference/configuration-options/settings/browser_cookies#manual-cookie-extraction) Manual Cookie Extraction ---------------------------------------------------------------------------------------------------------------------------------------------------------------------- If cookie extraction fails, you can manually extract the cookies from your browser using tools like [cookie-editorarrow-up-right](https://cookie-editor.com/) and save them at `AppData/Cookies/.txt`. The file must be a Netscape formatted cookie file. You can use any name for the file as long as it has a `.txt` extension. See: [How to extract cookies (DDoSGuard or login errors) #839arrow-up-right](https://github.com/jbsparrow/CyberDropDownloader/discussions/839) for detailed instructions circle-info Multiple cookie files are supported. You could have a `SocialMediaGirls.txt` file and a `cyberdrop.txt` file, for example [PreviousSettingschevron-left](https://script-ware.gitbook.io/cyberdrop-dl/reference/configuration-options/settings) [NextDownload Optionschevron-right](https://script-ware.gitbook.io/cyberdrop-dl/reference/configuration-options/settings/download_options) Last updated 5 months ago * [auto\_import](https://script-ware.gitbook.io/cyberdrop-dl/reference/configuration-options/settings/browser_cookies#auto_import) * [browser](https://script-ware.gitbook.io/cyberdrop-dl/reference/configuration-options/settings/browser_cookies#browser) * [Supported Browsers](https://script-ware.gitbook.io/cyberdrop-dl/reference/configuration-options/settings/browser_cookies#supported-browsers) * [sites](https://script-ware.gitbook.io/cyberdrop-dl/reference/configuration-options/settings/browser_cookies#sites) * [Manual Cookie Extraction](https://script-ware.gitbook.io/cyberdrop-dl/reference/configuration-options/settings/browser_cookies#manual-cookie-extraction) sun-brightdesktopmoon sun-brightdesktopmoon --- # File Size Limits | Cyberdrop-DL You can provide the maximum and minimum file size for each file "type". All options on this category take a `ByteSize` input ([more info here](https://script-ware.gitbook.io/cyberdrop-dl/reference/configuration-options/special_setting_types#bytesize) ). Type Default `ByteSize` `0` Setting any of these options to `0` means that limit is `disabled` * `maximum_image_filesize` * `minimum_image_filesize` * `maximum_video_filesize` * `minimum_video_filesize` * `maximum_other_filesize` * `minimum_other_filesize` [PreviousDupe Cleanup Optionschevron-left](https://script-ware.gitbook.io/cyberdrop-dl/reference/configuration-options/settings/dupe_cleanup_options) [NextFileschevron-right](https://script-ware.gitbook.io/cyberdrop-dl/reference/configuration-options/settings/files) Last updated 8 months ago sun-brightdesktopmoon sun-brightdesktopmoon --- # Generic Crawlers | Cyberdrop-DL CDL has some "generic" crawlers disabled by default. Generic crawlers are designed to work on any site that uses a **specific** framework. Users can supply a list of sites to map to these crawlers, and CDL will then be able to download from them. The URL in the list should be the primary URL of the site. ex: `https://forums.docker.com/` Supported generic crawlers: * `chevereto`: This works on any site that uses [Cheveretoarrow-up-right](https://chevereto.com//) . * `discourse`: This works on any forum that uses [Discoursearrow-up-right](https://www.discourse.org/) . * `wordpress_media`: This crawler should work on any [WordPressarrow-up-right](https://wordpress.com/) site where content primarily consists of images or galleries. The images need to be hosted on the site itself. It requires sites to have a public WordPress REST API. * `wordpress_html`: This works on any WordPress site. It scrapes the actual HTML of the site, which means it works on sites that have embedded third-party media like videos or links to hosting sites. It is always slower than `wordpress_media`. [hashtag](https://script-ware.gitbook.io/cyberdrop-dl/reference/configuration-options/global-settings/generic_crawlers_instances#generic_crawlers_instances) generic\_crawlers\_instances ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- [hashtag](https://script-ware.gitbook.io/cyberdrop-dl/reference/configuration-options/global-settings/generic_crawlers_instances#chevereto) `chevereto` ------------------------------------------------------------------------------------------------------------------------------------------------------------ Type Default `list[HttpURL]` `[]` [hashtag](https://script-ware.gitbook.io/cyberdrop-dl/reference/configuration-options/global-settings/generic_crawlers_instances#discourse) `discourse` ------------------------------------------------------------------------------------------------------------------------------------------------------------ Type Default `list[HttpURL]` `[]` [hashtag](https://script-ware.gitbook.io/cyberdrop-dl/reference/configuration-options/global-settings/generic_crawlers_instances#wordpress_media) `wordpress_media` ------------------------------------------------------------------------------------------------------------------------------------------------------------------------ Type Default `list[HttpURL]` `[]` [hashtag](https://script-ware.gitbook.io/cyberdrop-dl/reference/configuration-options/global-settings/generic_crawlers_instances#wordpress_html) `wordpress_html` ---------------------------------------------------------------------------------------------------------------------------------------------------------------------- Type Default `list[HttpURL]` `[]` [PreviousUI Optionschevron-left](https://script-ware.gitbook.io/cyberdrop-dl/reference/configuration-options/global-settings/ui-options) [NextCLI Argumentschevron-right](https://script-ware.gitbook.io/cyberdrop-dl/reference/cli-arguments) Last updated 5 months ago * [generic\_crawlers\_instances](https://script-ware.gitbook.io/cyberdrop-dl/reference/configuration-options/global-settings/generic_crawlers_instances#generic_crawlers_instances) * [chevereto](https://script-ware.gitbook.io/cyberdrop-dl/reference/configuration-options/global-settings/generic_crawlers_instances#chevereto) * [discourse](https://script-ware.gitbook.io/cyberdrop-dl/reference/configuration-options/global-settings/generic_crawlers_instances#discourse) * [wordpress\_media](https://script-ware.gitbook.io/cyberdrop-dl/reference/configuration-options/global-settings/generic_crawlers_instances#wordpress_media) * [wordpress\_html](https://script-ware.gitbook.io/cyberdrop-dl/reference/configuration-options/global-settings/generic_crawlers_instances#wordpress_html) sun-brightdesktopmoon sun-brightdesktopmoon --- # Getting Started | Cyberdrop-DL Development Docs | Cyberdrop-DL [hashtag](https://script-ware.gitbook.io/cyberdrop-dl/dev/getting-started#installing-cyberdrop-dl) Installing Cyberdrop-DL ------------------------------------------------------------------------------------------------------------------------------- There are two ways to install Cyberdrop-DL. The first is the easy method, where you simply download the start scripts. The start scripts will automatically install a compatible python version and then install Cyberdrop-DL. The second method is installing cyberdrop-dl directly from pypi using `uv`, `pipx` or `pip` For instructions, see: [downloadCyberdrop-DL Installchevron-right](https://script-ware.gitbook.io/cyberdrop-dl/dev/getting-started/cyberdrop-dl-install) [androidRunning CDL on Androidchevron-right](https://script-ware.gitbook.io/cyberdrop-dl/dev/getting-started/cyberdrop-dl-on-android) [hashtag](https://script-ware.gitbook.io/cyberdrop-dl/dev/getting-started#what-now) What now? -------------------------------------------------------------------------------------------------- If you downloaded the start scripts, just open the start script with the name of the OS you are using. For a manual install, execute the program with this command: Copy cyberdrop-dl-patched On the main screen, you can use the 'Edit URLs' option to add the URLs for the files you wish to download, them select the `download` option. That's it! However, Cyberdrop-DL has a ton of configuration options if you want more control over the downloads. You may want to review the following: [slidersConfiguration Optionschevron-right](https://script-ware.gitbook.io/cyberdrop-dl/dev/reference/configuration-options) [rectangle-terminalCLI Argumentschevron-right](https://script-ware.gitbook.io/cyberdrop-dl/dev/reference/cli-arguments) [envelope-open-textNotificationschevron-right](https://script-ware.gitbook.io/cyberdrop-dl/dev/reference/notifications) By default, all config files are under the `AppData` folder, which is created on the same directory that CDL it run from. You may also want to peek at what websites the program actually supports: [globe-pointerSupported Websiteschevron-right](https://script-ware.gitbook.io/cyberdrop-dl/dev/reference/supported-websites) If you have any issues, perhaps the [FAQ](https://script-ware.gitbook.io/cyberdrop-dl/dev/frequently-asked-questions) might help you! [PreviousWelcome!chevron-left](https://script-ware.gitbook.io/cyberdrop-dl/dev) [NextCyberdrop-DL Installchevron-right](https://script-ware.gitbook.io/cyberdrop-dl/dev/getting-started/cyberdrop-dl-install) Last updated 5 months ago sun-brightdesktopmoon sun-brightdesktopmoon --- # Transition to V8 | Cyberdrop-DL Development Docs | Cyberdrop-DL triangle-exclamation V6 introduced some breaking changes: Using a more strict config validation logic, replacing `md5` with `xxh128` as the default hashing algorithm, using a new database schema, among others. It's recommended to do a manual backup of your current `AppData` folder. You won't be able to rollback to a previous version after the transfer is completed. You can learn more about these changes on the [release announcementarrow-up-right](https://github.com/jbsparrow/CyberDropDownloader/blob/master/CHANGELOG.md#600---2024-12-23) circle-exclamation Even after a successful configuration migration, the program may not start if some values from the previous version’s config are no longer valid. Please follow the on-screen instructions to resolve the issue. You can use the [Config Options page](https://script-ware.gitbook.io/cyberdrop-dl/dev/reference/configuration-options) as reference for valid config values [hashtag](https://script-ware.gitbook.io/cyberdrop-dl/dev/upgrade#import-v5-v6-or-v7-config-into-v8) Import V5, V6 or v7 config into v8 -------------------------------------------------------------------------------------------------------------------------------------------- Good news! If you have a v5-v7 config, CDL can import it. The program will automatically detect the config in the current folder and handle the migration process for you. All you have to do is replace your old start script (for Windows, that's the `.bat` file) with the v8 start script. Please note that after the migration is complete, you may still need to manually adjust some values in the config. If you are migrating from v5 and the migration fails, the database will be rolled back to its previous state. An automatic backup is also made before the migration begins. However, keep in mind that the backup only includes the database, not the config files. circle-info Support for v5 and v6 (config, database and auto-migration) is deprecated and will be removed in v9 [hashtag](https://script-ware.gitbook.io/cyberdrop-dl/dev/upgrade#import-v4-config) Import V4 config --------------------------------------------------------------------------------------------------------- circle-exclamation V4 import is no longer supported. Last version with v4 support was `6.5.0`. To upgrade to the latest version, you need to perform a sequential upgrade. First, update CDL to `6.5.0`, import your v4 files, and then update to the latest version. You can install `6.5.0` with pip by running: Copy pip install cyberdrop-dl-patched==6.5.0 Built into Cyberdrop-DL is a tool that allows you to import both your configs and your history DB from a v4 version of Cyberdrop-DL. You can use it by selecting "Import V4 Items". ### [hashtag](https://script-ware.gitbook.io/cyberdrop-dl/dev/upgrade#import-your-v4-configs) Import Your V4 Configs The config files will be located in the folder that you were previously running Cyberdrop-DL in. Start CDL, choose the option `Import v4 config` and follow the on screen instructions circle-info If you weren't using the config previously, you don't need to import it. However, if you were primarily using CLI Arguments with V4, some of the arguments has been replaced. If you don't end up using the import feature, make sure you also change the default config in the program if that's something you want to do. ### [hashtag](https://script-ware.gitbook.io/cyberdrop-dl/dev/upgrade#import-your-v4-database) Import Your V4 Database For a lot of people, the `download_history.sqlite` file will be in the same folder as your start file (or wherever you are running Cyberdrop-DL). If it's not there, you can find it here: Windows: macOS: Linux: To import it, start CDL, choose the option `Import v4 database` and follow the on screen instructions circle-info The old `download_history.sqlite` file is no longer used by Cyberdrop-DL. After you import it, you can delete the old one. If you don't want to import previous download history, you can just delete it. [PreviousRunning CDL on Androidchevron-left](https://script-ware.gitbook.io/cyberdrop-dl/dev/getting-started/cyberdrop-dl-on-android) [NextFrequently Asked Questionschevron-right](https://script-ware.gitbook.io/cyberdrop-dl/dev/frequently-asked-questions) Last updated 5 months ago sun-brightdesktopmoon Copy C:\Users\\AppData\Local\Cyberdrop-DL\Cyberdrop-DL\download_history.sqlite Copy /Library/Application Support/Cyberdrop-DL/Cyberdrop-DL/download_history.sqlite Copy /home//.local/share/Cyberdrop-DL/Cyberdrop-DL/download_history.sqlite sun-brightdesktopmoon --- # Media Duration Limits | Cyberdrop-DL You can provide the maximum and minimum duration for audio and video files. Type Default `timedelta` `0s` * A `timedelta` input is expected to be a valid ISO 8601 timespan, ex: `P10DT2H30M10S` * An `int` input is assumed to be the number of days * A `str` input is expected to be in the format; ` `, ex: `10 days`. Setting any of these options to `0` means that limit is `disabled` * `minimum_audio_duration` * `maximum_audio_duration` * `minimum_video_duration` * `maximum_video_duration` [PreviousLogschevron-left](https://script-ware.gitbook.io/cyberdrop-dl/reference/configuration-options/settings/logs) [NextRuntime Optionschevron-right](https://script-ware.gitbook.io/cyberdrop-dl/reference/configuration-options/settings/runtime_options) Last updated 1 year ago sun-brightdesktopmoon sun-brightdesktopmoon --- # Frequently Asked Questions | Cyberdrop-DL Development Docs | Cyberdrop-DL [hashtag](https://script-ware.gitbook.io/cyberdrop-dl/dev/frequently-asked-questions#what-does-this-do) What does this do? ------------------------------------------------------------------------------------------------------------------------------- This is a bulk downloader for the supported sites. It supports resumable downloading (you can close and reopen the program at any time and it will pick up where it left off), and keeps track of your download history to avoid downloading files you've already downloaded in the past. [hashtag](https://script-ware.gitbook.io/cyberdrop-dl/dev/frequently-asked-questions#how-do-i-update-this) How do I update this? ------------------------------------------------------------------------------------------------------------------------------------- If you are using one of the provided start files, it should do so automatically. Keep in mind that they will only update to the newest version within the same major version. ex: if you are using v8 start scripts, they will update to the latest v8 release. When v9 is out, you will need to download the new start scripts [hashtag](https://script-ware.gitbook.io/cyberdrop-dl/dev/frequently-asked-questions#why-do-i-get-ddos-guard-error-downloading-from-x-site) Why do i get `DDoS-Guard` error downloading from `x` site? ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- You may need to import cookies. Follow the instructions here: [How to extract cookies (DDoSGuard or login errors) #839arrow-up-right](https://github.com/jbsparrow/CyberDropDownloader/discussions/839) [hashtag](https://script-ware.gitbook.io/cyberdrop-dl/dev/frequently-asked-questions#im-trying-to-report-a-bug-and-they-ask-me-for-a-logs-file.-where-is-this-file) I'm trying to report a bug and they ask me for a logs file. Where is this file? -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- By default, it'll be in `./AppData/configs//logs/` The `AppData` folder is created inside the folder where you run cyberdrop-dl from [hashtag](https://script-ware.gitbook.io/cyberdrop-dl/dev/frequently-asked-questions#what-does-scrape_failures-and-download_failures-mean) What does `SCRAPE_FAILURES` and `DOWNLOAD_FAILURES` mean? --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Quite simply, almost all of them you see will be HTTP Status codes. Such as: 404 - Not Found (dead link) You check [this page to learn about what each error code meansarrow-up-right](https://developer.mozilla.org/en-US/docs/Web/HTTP/Status) . circle-info Any "Unknown" error, is usually coding related, or it'll be something like the program not being able to find a file extension for a file. You can report unknown error on the issue tracker: [https://github.com/jbsparrow/CyberDropDownloader/issuesarrow-up-right](https://github.com/jbsparrow/CyberDropDownloader/issues) [hashtag](https://script-ware.gitbook.io/cyberdrop-dl/dev/frequently-asked-questions#why-are-all-the-files-skipped) Why are all the files skipped? ------------------------------------------------------------------------------------------------------------------------------------------------------- The program tracks your download history and will skip any files you've previously downloaded to avoid duplicates. You can disable this behavior by using the `--ignore-history` CLI argument or setting `ignore_history` to `true` in the config [hashtag](https://script-ware.gitbook.io/cyberdrop-dl/dev/frequently-asked-questions#the-screen-is-flickering) The screen is flickering? --------------------------------------------------------------------------------------------------------------------------------------------- This issue is likely related to the limitations of the traditional command prompt, which has remained largely unchanged over time. For Windows 10 users, it's recommended to install and use [Windows Terminalarrow-up-right](https://aka.ms/terminal) to run Cyberdrop-DL. Windows Terminal is the default on Windows 11. [hashtag](https://script-ware.gitbook.io/cyberdrop-dl/dev/frequently-asked-questions#cyberdrop-dl-is-not-a-recognized-internal-command) `cyberdrop-dl` is not a recognized internal command ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ This issue is caused by an improper installation of Python, specifically Python not being added to the system PATH. It is recommended to revisit the [Getting Started](https://script-ware.gitbook.io/cyberdrop-dl/dev/getting-started) guide and follow the steps provided to reinstall or use one of the latest start scripts [hashtag](https://script-ware.gitbook.io/cyberdrop-dl/dev/frequently-asked-questions#how-do-i-scrape-forum-threads) How do I scrape forum threads? ------------------------------------------------------------------------------------------------------------------------------------------------------- You need to import cookies to use as authentication for those sites. Follow the instructions here: [How to extract cookies (DDoSGuard or login errors) #839arrow-up-right](https://github.com/jbsparrow/CyberDropDownloader/discussions/839) [hashtag](https://script-ware.gitbook.io/cyberdrop-dl/dev/frequently-asked-questions#why-are-the-filenames-the-way-they-are) Why are the filenames the way they are? ------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Filenames are taken directly from the source website. Blame whoever uploaded it. [hashtag](https://script-ware.gitbook.io/cyberdrop-dl/dev/frequently-asked-questions#how-do-i-fix-ssl-certificate_verify_failed) How do I fix `[SSL: CERTIFICATE_VERIFY_FAILED]`? -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- This should only appear on macOS, so these instructions are for mac users. Go to your applications folder, find the python folder inside of it. Run the `Install Certificates` file in that folder. Go back to where you are running Cyberdrop-DL and delete the `venv` folder if one exists (if not, don't worry). Then try running the program again. [hashtag](https://script-ware.gitbook.io/cyberdrop-dl/dev/frequently-asked-questions#a-thread-site-i-follow-has-new-posts-but-cyberdrop-dl-is-not-detecting-downloading-them-why) A thread/site i follow has new posts but `cyberdrop-dl` is not detecting / downloading them, why? ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- `cyberdrop-dl` caches requests to made to sites to speed up re-runs and minimize load on those sites. By default, forums are cached for 30 days and any other site is cached for 7 days. You can run with `--disable-cache` to temporarily disable the cache (CLI only) or change the default values to 0. See: [\--forum-cache-expire-afterarrow-up-right](https://script-ware.gitbook.io/cyberdrop-dl/reference/configuration-options/global-settings/rate-limiting-options#forum_cache_expire_after) [\--file-host-cache-expire-afterarrow-up-right](https://script-ware.gitbook.io/cyberdrop-dl/reference/configuration-options/global-settings/rate-limiting-options#file_host_cache_expire_after) [PreviousTransition to V8chevron-left](https://script-ware.gitbook.io/cyberdrop-dl/dev/upgrade) [NextMigration to Cyberdrop-DL-Patchedchevron-right](https://script-ware.gitbook.io/cyberdrop-dl/dev/migration-to-cyberdrop-dl-patched) Last updated 5 months ago * [What does this do?](https://script-ware.gitbook.io/cyberdrop-dl/dev/frequently-asked-questions#what-does-this-do) * [How do I update this?](https://script-ware.gitbook.io/cyberdrop-dl/dev/frequently-asked-questions#how-do-i-update-this) * [Why do i get DDoS-Guard error downloading from x site?](https://script-ware.gitbook.io/cyberdrop-dl/dev/frequently-asked-questions#why-do-i-get-ddos-guard-error-downloading-from-x-site) * [I'm trying to report a bug and they ask me for a logs file. Where is this file?](https://script-ware.gitbook.io/cyberdrop-dl/dev/frequently-asked-questions#im-trying-to-report-a-bug-and-they-ask-me-for-a-logs-file.-where-is-this-file) * [What does SCRAPE\_FAILURES and DOWNLOAD\_FAILURES mean?](https://script-ware.gitbook.io/cyberdrop-dl/dev/frequently-asked-questions#what-does-scrape_failures-and-download_failures-mean) * [Why are all the files skipped?](https://script-ware.gitbook.io/cyberdrop-dl/dev/frequently-asked-questions#why-are-all-the-files-skipped) * [The screen is flickering?](https://script-ware.gitbook.io/cyberdrop-dl/dev/frequently-asked-questions#the-screen-is-flickering) * [cyberdrop-dl is not a recognized internal command](https://script-ware.gitbook.io/cyberdrop-dl/dev/frequently-asked-questions#cyberdrop-dl-is-not-a-recognized-internal-command) * [How do I scrape forum threads?](https://script-ware.gitbook.io/cyberdrop-dl/dev/frequently-asked-questions#how-do-i-scrape-forum-threads) * [Why are the filenames the way they are?](https://script-ware.gitbook.io/cyberdrop-dl/dev/frequently-asked-questions#why-are-the-filenames-the-way-they-are) * [How do I fix \[SSL: CERTIFICATE\_VERIFY\_FAILED\]?](https://script-ware.gitbook.io/cyberdrop-dl/dev/frequently-asked-questions#how-do-i-fix-ssl-certificate_verify_failed) * [A thread/site i follow has new posts but cyberdrop-dl is not detecting / downloading them, why?](https://script-ware.gitbook.io/cyberdrop-dl/dev/frequently-asked-questions#a-thread-site-i-follow-has-new-posts-but-cyberdrop-dl-is-not-detecting-downloading-them-why) sun-brightdesktopmoon sun-brightdesktopmoon --- # General | Cyberdrop-DL [hashtag](https://script-ware.gitbook.io/cyberdrop-dl/reference/configuration-options/global-settings/general#disable_crawlers) `disable_crawlers` ------------------------------------------------------------------------------------------------------------------------------------------------------- Type Default Additional Info `list[NonEmptyStr]` `[]` This is an [`AdditiveArg`](https://script-ware.gitbook.io/cyberdrop-dl/reference/configuration-options/special_setting_types#additiveargs) You can supply a list of crawlers to disable for the current run. This will make CDL completely ignore the crawler, as if the site was not supported. However, links from the site will still be processed by Real-Debrid (if enabled), Jdownloader (If enabled) and the Generic crawler (If enabled), in that order. The list should be valid crawlers names. The name of the crawler is the name of the primary site they support. ex: `4Chan`, `Bunkrr`, `Dropbox` Crawlers names correspond to the column `site` in the [supported sites pagearrow-up-right](https://script-ware.gitbook.io/cyberdrop-dl/reference/supported-websites#supported-sites) . [hashtag](https://script-ware.gitbook.io/cyberdrop-dl/reference/configuration-options/global-settings/general#enable_generic_crawler) `enable_generic_crawler` ------------------------------------------------------------------------------------------------------------------------------------------------------------------- Type Default `bool` `false` CDL has a generic crawler that will try to download from unsupported sites. Setting this to `true` will enable it. circle-info CDL will always try to download from unsupported URLs if the last part of the URL has a known file extension. ex: `.jpg` [hashtag](https://script-ware.gitbook.io/cyberdrop-dl/reference/configuration-options/global-settings/general#flaresolverr) `flaresolverr` ----------------------------------------------------------------------------------------------------------------------------------------------- Type Default `HttpURL` or `null` `null` [FlareSolverrarrow-up-right](https://github.com/FlareSolverr/FlareSolverr) is a proxy server to bypass Cloudflare and `DDoS-Guard` protection. The provided value must be a valid `http` URL of an existing flaresolverr instance. Ex: `http://192.168.1.44:4000` circle-info `0.0.0.0` is NOT a valid IP address. To set up a flaresolverr instance running on the same machine as CDL, use `127.0.0.1` as the IP circle-exclamation This wiki does not cover flaresolverr setup process. If you need help, refer to their documentation. Please do not open issues related to flaresolverr or `DDoS-Guard`. See: [How to extract cookies (DDoSGuard or login errors) #839arrow-up-right](https://github.com/jbsparrow/CyberDropDownloader/discussions/839) for alternatives using cookies [hashtag](https://script-ware.gitbook.io/cyberdrop-dl/reference/configuration-options/global-settings/general#max_file_name_length) `max_file_name_length` --------------------------------------------------------------------------------------------------------------------------------------------------------------- Type Default `PositiveInt` `95` This is the maximum number of characters a filename should have. CDL will truncate filenames longer that this. [hashtag](https://script-ware.gitbook.io/cyberdrop-dl/reference/configuration-options/global-settings/general#max_folder_name_length) `max_folder_name_length` ------------------------------------------------------------------------------------------------------------------------------------------------------------------- Type Default `PositiveInt` `95` This is the maximum number of characters a folder should have. CDL will truncate folders longer that this. [hashtag](https://script-ware.gitbook.io/cyberdrop-dl/reference/configuration-options/global-settings/general#proxy) `proxy` --------------------------------------------------------------------------------------------------------------------------------- Type Default `HttpURL` or `null` `null` The proxy you want CDL to use. Only `http` proxies are supported. Ex: `https://user:password@ip:port` [hashtag](https://script-ware.gitbook.io/cyberdrop-dl/reference/configuration-options/global-settings/general#required_free_space) `required_free_space` ------------------------------------------------------------------------------------------------------------------------------------------------------------- Type Default Restrictions `ByteSize` `5GB` `>=512MB` This is the minimum amount of free space require to start new downloads. circle-info If you set a value lower than `512MB`, CDL will override it with `512MB` [hashtag](https://script-ware.gitbook.io/cyberdrop-dl/reference/configuration-options/global-settings/general#ssl_context) `ssl_context` --------------------------------------------------------------------------------------------------------------------------------------------- Type Default `NonEmptyStr` or None `truststore+certifi` Context that will used to verify SSL connections. Valid values are: * `truststore`: Will use certificates already included with the OS * `certifi`: Will use certificates bundled with the `certifi` version available at the release of the current CDL version * `truststore+certifi`: Will use certificates already included with the OS, with a fallback to `certifi` for missing certificates * `None`: Will completely disable SSL verification, allowing insecure connections. Setting this to `None` will allow the program to connect to websites without SSL encryption (insecurely). triangle-exclamation Sensitive data may be exposed using an insecure connection. For your safety, is recommended to always use a secure HTTPS connection. [hashtag](https://script-ware.gitbook.io/cyberdrop-dl/reference/configuration-options/global-settings/general#user_agent) `user_agent` ------------------------------------------------------------------------------------------------------------------------------------------- Type Default `NonEmptyStr` `Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:142.0) Gecko/20100101 Firefox/142.0` The user agent is the signature of your browser. Some sites use it to identify if the requests come from a human or a robot. You can google "what is my user agent" to get yours. circle-info If you use flaresolverr, this value MUST match with flaresolverr's user agent. Otherwise, flaresolverr cookies won't work [PreviousGlobal Settingschevron-left](https://script-ware.gitbook.io/cyberdrop-dl/reference/configuration-options/global-settings) [NextRate Limiting Optionschevron-right](https://script-ware.gitbook.io/cyberdrop-dl/reference/configuration-options/global-settings/rate-limiting-options) Last updated 5 months ago * [disable\_crawlers](https://script-ware.gitbook.io/cyberdrop-dl/reference/configuration-options/global-settings/general#disable_crawlers) * [enable\_generic\_crawler](https://script-ware.gitbook.io/cyberdrop-dl/reference/configuration-options/global-settings/general#enable_generic_crawler) * [flaresolverr](https://script-ware.gitbook.io/cyberdrop-dl/reference/configuration-options/global-settings/general#flaresolverr) * [max\_file\_name\_length](https://script-ware.gitbook.io/cyberdrop-dl/reference/configuration-options/global-settings/general#max_file_name_length) * [max\_folder\_name\_length](https://script-ware.gitbook.io/cyberdrop-dl/reference/configuration-options/global-settings/general#max_folder_name_length) * [proxy](https://script-ware.gitbook.io/cyberdrop-dl/reference/configuration-options/global-settings/general#proxy) * [required\_free\_space](https://script-ware.gitbook.io/cyberdrop-dl/reference/configuration-options/global-settings/general#required_free_space) * [ssl\_context](https://script-ware.gitbook.io/cyberdrop-dl/reference/configuration-options/global-settings/general#ssl_context) * [user\_agent](https://script-ware.gitbook.io/cyberdrop-dl/reference/configuration-options/global-settings/general#user_agent) sun-brightdesktopmoon sun-brightdesktopmoon --- # Cyberdrop-DL Install | Cyberdrop-DL Development Docs | Cyberdrop-DL [hashtag](https://script-ware.gitbook.io/cyberdrop-dl/dev/getting-started/cyberdrop-dl-install#using-start-scripts-from-release-page) Using Start Scripts (From Release Page) ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- This is the simplest method to get the program up and running. Pre-configured start files are provided that will automatically install python, install cyberdrop-dl, update, and launch the program for you. circle-info The start scripts only work on 64bits operating systems. If you are running a 32bit OS, you need to install directly from pypi and may need to compile some dependencies You can download them here: [https://github.com/jbsparrow/CyberDropDownloader/releases/latestarrow-up-right](https://github.com/jbsparrow/CyberDropDownloader/releases/latest) You only need to download the `Cyberdrop-DL_.zip` file, you don't need to worry about the other files. Extract the contents of the zip file to any location where you'd like the program to run and store downloaded files. The extracted files will include a start file for Windows, macOS, and Linux If you're using Windows or Linux, simply open the start file, and it will handle the rest for you circle-info If you are using Windows, **DO NOT** run the script as admin On macOS, you should be able to open the start file and have everything set up automatically. However, in some rare cases, macOS users may need to run the following command first:: Copy xcode-select --install chevron-rightOptional: Running Cyberdrop-DL Script with Custom Parameters[hashtag](https://script-ware.gitbook.io/cyberdrop-dl/dev/getting-started/cyberdrop-dl-install#optional-running-cyberdrop-dl-script-with-custom-parameters) You can open the start script from the zip in a text editor like notepad. At the top of the file, you will this line: Copy set "COMMANDLINE_ARGS=" `COMMANDLINE_ARGS`: Provide any arguments to pass to Cyberdrop-Dl. For more information, refer to the [CLI Arguments section](https://script-ware.gitbook.io/cyberdrop-dl/dev/reference/cli-arguments) circle-info You **MUST** put the values _inside_ the double quotes. Ex: `set "COMMANDLINE_ARGS=--disable-cache"` [hashtag](https://script-ware.gitbook.io/cyberdrop-dl/dev/getting-started/cyberdrop-dl-install#manual-install) Manual Install ---------------------------------------------------------------------------------------------------------------------------------- ### [hashtag](https://script-ware.gitbook.io/cyberdrop-dl/dev/getting-started/cyberdrop-dl-install#id-1.-using-uv) 1\. Using `uv` The recommended way to install Cyberdrop-DL is using [`uv`arrow-up-right](https://docs.astral.sh/uv) . macOS / Linux Windows Once you have `uv`, you can install Cyberdrop-DL using: ### [hashtag](https://script-ware.gitbook.io/cyberdrop-dl/dev/getting-started/cyberdrop-dl-install#id-2.-manual-python-install) 2\. Manual Python Install If you do not want to or can't use `uv`, you will need to install a compatible python version manually. If you don't have python, you can find and download it from their official website: [https://www.python.org/downloads/arrow-up-right](https://www.python.org/downloads/) circle-info Cyberdrop-DL requires python 3.11, 3.12 or python 3.13. The **RECOMMENDED** version is 3.12+ circle-exclamation If you are using a version of Cyberdrop-DL from the previous repository (`cyberdrop-dl` instead of `cyberdrop-dl-patched`), you **MUST** uninstall it before installing the patched version. Once you have python, you can install Cyberdrop-DL directly from pypi using `pipx` or `pip`. In a command prompt/terminal window: pipx pip * Install `pipx` (if you don't have it already) * Install cdl circle-exclamation Using bare `pip` to install `cyberdrop-dl-patched` is discouraged as it may lead to dependency conflicts with global installs and an inconsistent environment. Consider using `uv` or `pipx` [PreviousGetting Startedchevron-left](https://script-ware.gitbook.io/cyberdrop-dl/dev/getting-started) [NextRunning CDL on Androidchevron-right](https://script-ware.gitbook.io/cyberdrop-dl/dev/getting-started/cyberdrop-dl-on-android) Last updated 5 months ago * [Using Start Scripts (From Release Page)](https://script-ware.gitbook.io/cyberdrop-dl/dev/getting-started/cyberdrop-dl-install#using-start-scripts-from-release-page) * [Manual Install](https://script-ware.gitbook.io/cyberdrop-dl/dev/getting-started/cyberdrop-dl-install#manual-install) * [1\. Using uv](https://script-ware.gitbook.io/cyberdrop-dl/dev/getting-started/cyberdrop-dl-install#id-1.-using-uv) * [2\. Manual Python Install](https://script-ware.gitbook.io/cyberdrop-dl/dev/getting-started/cyberdrop-dl-install#id-2.-manual-python-install) sun-brightdesktopmoon Copy curl -LsSf https://astral.sh/uv/install.sh | sh Copy powershell -ExecutionPolicy ByPass -c "irm https://astral.sh/uv/install.ps1 | iex" Copy uv tool install cyberdrop-dl-patched Copy pip uninstall cyberdrop-dl Copy pip install pipx Copy pipx install cyberdrop-dl-patched Copy pip install cyberdrop-dl-patched sun-brightdesktopmoon --- # Dupe Cleanup Options | Cyberdrop-DL Cyberdrop-DL maintains an internal database of all downloaded files, indexed by their hashes. This can be used to automatically delete newly downloaded files if they were downloaded before. To enable auto dupe cleanup: 1. Set `hashing` to `IN_PLACE` or `POST_DOWNLOAD` 2. Set `auto_dedupe` to `true` [hashtag](https://script-ware.gitbook.io/cyberdrop-dl/reference/configuration-options/settings/dupe_cleanup_options#add_md5_hash) `add_md5_hash` ----------------------------------------------------------------------------------------------------------------------------------------------------- Type Default `bool` `false` If enabled, calculates the `md5` hash for every downloaded file and store it in the internal database. This allows the program to match files with sites that provide hash information before downloading them, skipping the download altogether if a match is found circle-info `md5` was the default hashing algorithm of Cyberdrop-DL V5. Since v6, the default algorithm is `xxhash128`. [hashtag](https://script-ware.gitbook.io/cyberdrop-dl/reference/configuration-options/settings/dupe_cleanup_options#add_sha256_hash) `add_sha256_hash` ----------------------------------------------------------------------------------------------------------------------------------------------------------- Type Default `bool` `false` If enabled, calculates the `sha256` hash for every downloaded file and store it in the internal database. This allows the program to match files with sites that provide hash information before downloading them, skipping the download altogether if a match is found [hashtag](https://script-ware.gitbook.io/cyberdrop-dl/reference/configuration-options/settings/dupe_cleanup_options#auto_dedupe) `auto_dedupe` --------------------------------------------------------------------------------------------------------------------------------------------------- Type Default `bool` `false` Enables deduping files functionality. Needs `hashing` to be enabled The `auto_dedupe` feature scans the database for files with matching hashes and sizes, automatically deleting any duplicates and retaining only the oldest copy. Deletion will only occur if two or more matching files are found during the database search. circle-exclamation Auto dedupe will delete files if you have _ever_ downloaded them before, even if you no longer have the original download on disk [hashtag](https://script-ware.gitbook.io/cyberdrop-dl/reference/configuration-options/settings/dupe_cleanup_options#hashing) `hashing` ------------------------------------------------------------------------------------------------------------------------------------------- Type Default Restrictions `HASHING` `OFF` Must be `OFF`, `IN_PLACE` or `POST_DOWNLOAD` There are three possible options for hashing: 1. `OFF`: disables hashing 2. `IN_PLACE`: performs hashing after each download 3. `POST_DOWNLOAD`: performs hashing after all downloads have completed The default hashing algorithm is `xxh128`. You can enable additional hashing algorithms, but you can not replace the default. [hashtag](https://script-ware.gitbook.io/cyberdrop-dl/reference/configuration-options/settings/dupe_cleanup_options#send_deleted_to_trash) `send_deleted_to_trash` ----------------------------------------------------------------------------------------------------------------------------------------------------------------------- Type Default `bool` `false` Deduped files are sent to the trash bin instead of being deleted [PreviousDownload Optionschevron-left](https://script-ware.gitbook.io/cyberdrop-dl/reference/configuration-options/settings/download_options) [NextFile Size Limitschevron-right](https://script-ware.gitbook.io/cyberdrop-dl/reference/configuration-options/settings/file_size_limits) Last updated 5 months ago * [add\_md5\_hash](https://script-ware.gitbook.io/cyberdrop-dl/reference/configuration-options/settings/dupe_cleanup_options#add_md5_hash) * [add\_sha256\_hash](https://script-ware.gitbook.io/cyberdrop-dl/reference/configuration-options/settings/dupe_cleanup_options#add_sha256_hash) * [auto\_dedupe](https://script-ware.gitbook.io/cyberdrop-dl/reference/configuration-options/settings/dupe_cleanup_options#auto_dedupe) * [hashing](https://script-ware.gitbook.io/cyberdrop-dl/reference/configuration-options/settings/dupe_cleanup_options#hashing) * [send\_deleted\_to\_trash](https://script-ware.gitbook.io/cyberdrop-dl/reference/configuration-options/settings/dupe_cleanup_options#send_deleted_to_trash) sun-brightdesktopmoon sun-brightdesktopmoon --- # Settings | Cyberdrop-DL Development Docs | Cyberdrop-DL [Browser Cookieschevron-right](https://script-ware.gitbook.io/cyberdrop-dl/dev/reference/configuration-options/settings/browser_cookies) [Download Optionschevron-right](https://script-ware.gitbook.io/cyberdrop-dl/dev/reference/configuration-options/settings/download_options) [Dupe Cleanup Optionschevron-right](https://script-ware.gitbook.io/cyberdrop-dl/dev/reference/configuration-options/settings/dupe_cleanup_options) [File Size Limitschevron-right](https://script-ware.gitbook.io/cyberdrop-dl/dev/reference/configuration-options/settings/file_size_limits) [Fileschevron-right](https://script-ware.gitbook.io/cyberdrop-dl/dev/reference/configuration-options/settings/files) [Ignore Optionschevron-right](https://script-ware.gitbook.io/cyberdrop-dl/dev/reference/configuration-options/settings/ignore_options) [Logschevron-right](https://script-ware.gitbook.io/cyberdrop-dl/dev/reference/configuration-options/settings/logs) [Media Duration Limitschevron-right](https://script-ware.gitbook.io/cyberdrop-dl/dev/reference/configuration-options/settings/media_duration_limits) [Runtime Optionschevron-right](https://script-ware.gitbook.io/cyberdrop-dl/dev/reference/configuration-options/settings/runtime_options) [Sortingchevron-right](https://script-ware.gitbook.io/cyberdrop-dl/dev/reference/configuration-options/settings/sorting) [PreviousAuthenticationchevron-left](https://script-ware.gitbook.io/cyberdrop-dl/dev/reference/configuration-options/authentication) [NextBrowser Cookieschevron-right](https://script-ware.gitbook.io/cyberdrop-dl/dev/reference/configuration-options/settings/browser_cookies) Last updated 1 year ago sun-brightdesktopmoon sun-brightdesktopmoon --- # Rate Limiting Options | Cyberdrop-DL [hashtag](https://script-ware.gitbook.io/cyberdrop-dl/reference/configuration-options/global-settings/rate-limiting-options#connection_timeout) `connection_timeout` ------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Type Default `PositiveInt` `15` The number of seconds to wait while connecting to a website before timing out circle-info This value will also be used for Flaresolverr (if enabled) as the max number of seconds to solve a CAPTCHA challenge [hashtag](https://script-ware.gitbook.io/cyberdrop-dl/reference/configuration-options/global-settings/rate-limiting-options#download_attempts) `download_attempts` ----------------------------------------------------------------------------------------------------------------------------------------------------------------------- Type Default `PositiveInt` `2` The number of download attempts per file. Some conditions are never retried (such as a `404` HTTP status) [hashtag](https://script-ware.gitbook.io/cyberdrop-dl/reference/configuration-options/global-settings/rate-limiting-options#download_delay) `download_delay` ----------------------------------------------------------------------------------------------------------------------------------------------------------------- Type Default `NonNegativeFloat` `0.0` This is the number of seconds to wait between downloads to the same domain. Some domains have internal limits set by the program, which can not be modified: * `bunkr`: 0.5 * `coomer`: 0.5 * `kemono`: 0.5 * `nhentai.net`: 1 [hashtag](https://script-ware.gitbook.io/cyberdrop-dl/reference/configuration-options/global-settings/rate-limiting-options#download_speed_limit) `download_speed_limit` ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Type Default `ByteSize` `0` circle-info This setting specifies speed and it's interpreted as ` / second`. ex: `8MB` means `8MB/s` This is the max rate of downloading in bytes (per second) for all downloads combined. Set to `0` to disable [hashtag](https://script-ware.gitbook.io/cyberdrop-dl/reference/configuration-options/global-settings/rate-limiting-options#file_host_cache_expire_after) `file_host_cache_expire_after` --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Type Default `timedelta`, `str` or `int` `7 days` Cyberdrop-DL caches the requests made to any website. This setting controls how long responses to file host websites are stored before expiring. * A `timedelta` input is expected to be a valid ISO 8601 timespan, ex: `P10DT2H30M10S` * An `int` input is assumed to be the number of days * A `str` input is expected to be in the format; ` `, ex: `10 days`. ### [hashtag](https://script-ware.gitbook.io/cyberdrop-dl/reference/configuration-options/global-settings/rate-limiting-options#valid-str-units) Valid `str` units * `year(s)` * `month(s)` * `week(s)` * `day(s)` * `hour(s)` * `minute(s)` * `second(s)` * `millisecond(s)` * `microsecond(s)` circle-info You can set the value to `0` to disable caching [hashtag](https://script-ware.gitbook.io/cyberdrop-dl/reference/configuration-options/global-settings/rate-limiting-options#forum_cache_expire_after) `forum_cache_expire_after` ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Type Default `timedelta`, `str` or `int` `4 weeks` Same as `file_host_cache_expire_after` but applied to forums requests. [hashtag](https://script-ware.gitbook.io/cyberdrop-dl/reference/configuration-options/global-settings/rate-limiting-options#jitter) `jitter` ------------------------------------------------------------------------------------------------------------------------------------------------- Type Default `NonNegativeFloat` `0.0` Additional number of seconds to wait in between downloads. CDL will wait an additional random number of seconds in between 0 and the `jitter` value. [hashtag](https://script-ware.gitbook.io/cyberdrop-dl/reference/configuration-options/global-settings/rate-limiting-options#max_simultaneous_downloads) `max_simultaneous_downloads` ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Type Default `PositiveInt` `15` This is the maximum number of files that can be downloaded simultaneously. [hashtag](https://script-ware.gitbook.io/cyberdrop-dl/reference/configuration-options/global-settings/rate-limiting-options#max_simultaneous_downloads_per_domain) `max_simultaneous_downloads_per_domain` --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Type Default `PositiveInt` `5` This is the maximum number of files that can be downloaded from a single domain simultaneously. Some domains have internal limits set by the program, which can not be modified: * `bunkr`: 1 per unique server * `cyberdrop`: 1 * `cyberfile`: 1 * `noodlemagazine`: 2 * `4chan`: 1 * `pixeldrain`: 2 * `xxxbunker`: 2 [hashtag](https://script-ware.gitbook.io/cyberdrop-dl/reference/configuration-options/global-settings/rate-limiting-options#rate_limit) `rate_limit` --------------------------------------------------------------------------------------------------------------------------------------------------------- Type Default `PositiveFloat` `50.0` circle-info This setting specifies speed and it's interpreted as ` / second`. ex: `50` means `50 requests / second` This is the maximum number of requests that can be made by the program per second. [hashtag](https://script-ware.gitbook.io/cyberdrop-dl/reference/configuration-options/global-settings/rate-limiting-options#read_timeout) `read_timeout` ------------------------------------------------------------------------------------------------------------------------------------------------------------- Type Default `PositiveFloat` or `null` `300.0` The number of seconds to wait while reading data from a website before timing out. A `null` value will make CDL keep the socket connection open indefinitely, even if the server is not sending data anymore [PreviousGeneralchevron-left](https://script-ware.gitbook.io/cyberdrop-dl/reference/configuration-options/global-settings/general) [NextUI Optionschevron-right](https://script-ware.gitbook.io/cyberdrop-dl/reference/configuration-options/global-settings/ui-options) Last updated 3 months ago * [connection\_timeout](https://script-ware.gitbook.io/cyberdrop-dl/reference/configuration-options/global-settings/rate-limiting-options#connection_timeout) * [download\_attempts](https://script-ware.gitbook.io/cyberdrop-dl/reference/configuration-options/global-settings/rate-limiting-options#download_attempts) * [download\_delay](https://script-ware.gitbook.io/cyberdrop-dl/reference/configuration-options/global-settings/rate-limiting-options#download_delay) * [download\_speed\_limit](https://script-ware.gitbook.io/cyberdrop-dl/reference/configuration-options/global-settings/rate-limiting-options#download_speed_limit) * [file\_host\_cache\_expire\_after](https://script-ware.gitbook.io/cyberdrop-dl/reference/configuration-options/global-settings/rate-limiting-options#file_host_cache_expire_after) * [Valid str units](https://script-ware.gitbook.io/cyberdrop-dl/reference/configuration-options/global-settings/rate-limiting-options#valid-str-units) * [forum\_cache\_expire\_after](https://script-ware.gitbook.io/cyberdrop-dl/reference/configuration-options/global-settings/rate-limiting-options#forum_cache_expire_after) * [jitter](https://script-ware.gitbook.io/cyberdrop-dl/reference/configuration-options/global-settings/rate-limiting-options#jitter) * [max\_simultaneous\_downloads](https://script-ware.gitbook.io/cyberdrop-dl/reference/configuration-options/global-settings/rate-limiting-options#max_simultaneous_downloads) * [max\_simultaneous\_downloads\_per\_domain](https://script-ware.gitbook.io/cyberdrop-dl/reference/configuration-options/global-settings/rate-limiting-options#max_simultaneous_downloads_per_domain) * [rate\_limit](https://script-ware.gitbook.io/cyberdrop-dl/reference/configuration-options/global-settings/rate-limiting-options#rate_limit) * [read\_timeout](https://script-ware.gitbook.io/cyberdrop-dl/reference/configuration-options/global-settings/rate-limiting-options#read_timeout) sun-brightdesktopmoon sun-brightdesktopmoon --- # CLI Arguments | Cyberdrop-DL circle-info CLI inputs always take priority over config values. circle-info Use `-` instead of `_` to separate words in an config option name when using it as a CLI argument: Ex: `auto_dedupe` needs to be `auto-dedupe` when using it via the CLI You can pass any of the **Config Settings** and **Global Settings** options as a cli argument for the program. For items not explained below, you can find their counterparts in the configuration options to see what they do. [hashtag](https://script-ware.gitbook.io/cyberdrop-dl/reference/cli-arguments#cli-only-arguments) CLI only arguments ------------------------------------------------------------------------------------------------------------------------- ### [hashtag](https://script-ware.gitbook.io/cyberdrop-dl/reference/cli-arguments#appdata-folder) `appdata-folder` Type Default `Path` `` Folder where Cyberdrop-DL will store it's database, cache and config files. ### [hashtag](https://script-ware.gitbook.io/cyberdrop-dl/reference/cli-arguments#completed-after) `completed-after` Type Default `date` `None` Only retry downloads that were completed on or after this date. The date should be in ISO 8601 format, for example, `2021-12-23` circle-info This option has no effect unless you run CDL with `--retry-all` ### [hashtag](https://script-ware.gitbook.io/cyberdrop-dl/reference/cli-arguments#completed-before) `completed-before` Type Default `date` `None` Only retry downloads that were completed on or before this date. The date should be in ISO 8601 format, for example, `2021-12-23` circle-info This option has no effect unless you run CDL with `--retry-all` ### [hashtag](https://script-ware.gitbook.io/cyberdrop-dl/reference/cli-arguments#config) `config` Type Default `str` `None` Name of config to load. Use `ALL` to run all configs sequentially ### [hashtag](https://script-ware.gitbook.io/cyberdrop-dl/reference/cli-arguments#config-file) `config-file` Type Default `Path` `None` Path to the CDL `settings.yaml` file to load circle-info If both `config` and `config-file` are supplied, `config-file` takes priority ### [hashtag](https://script-ware.gitbook.io/cyberdrop-dl/reference/cli-arguments#disable-cache) `disable-cache` Type Default Action `BoolFlag` `False` `store_true` Disables read/writes to requests cache for the current run only. All config settings or arguments related to the cache (ex: `file_host_cache_expire_after`) will be ignored. ### [hashtag](https://script-ware.gitbook.io/cyberdrop-dl/reference/cli-arguments#download) `download` Type Default Action `BoolFlag` `False` `store_true` Skips UI, start download immediately ### [hashtag](https://script-ware.gitbook.io/cyberdrop-dl/reference/cli-arguments#download-tiktok-audios) `download-tiktok-audios` Type Default Action `BoolFlag` `False` `store_true` Download TikTok audios from posts and save them as separate files ### [hashtag](https://script-ware.gitbook.io/cyberdrop-dl/reference/cli-arguments#download-tiktok-src-quality-videos) `download-tiktok-src-quality-videos` Type Default Action `BoolFlag` `False` `store_true` By default, CDL will download the "optimized for streaming" version of tiktok videos. Setting this option to `True` will download videos in original (source) quality. `_original` will be added as a suffix to their filename. circle-exclamation This will make video downloads several times slower When it is set to `False` (the default) CDL can download 50 videos with a single request. When it is set to `True` , CDL needs to make at least 3 requests _per_ video to download them. There's also a daily limit of the API CDL uses: 5000 requests per day per IP Setting this option to `True` will consume the daily limit faster ### [hashtag](https://script-ware.gitbook.io/cyberdrop-dl/reference/cli-arguments#impersonate) `impersonate` Type Default Action `chrome", "edge", "safari", "safari_ios", "chrome_android", "firefox"`, `BoolFlag` or `null` `null` `store_true` or `store_const` Impersonation allows CDL to make requests and appear to be a legitimate web browser. This helps bypass bot-protection on some sites and it's required for any site that only accepts HTTP2 connections. * The default value (`null`) means CDL will automatically use impersonation for crawlers that were programmed to use it. * Passing the flag without any value (`--impersonate`) is the same as `--impersonate True`: CDL will use impersonation for ALL requests, using the default impersonation target * Passing an specific target (ex: `--impersonate chrome_android`) will make CDL use impersonation for all requests, using that tarjet circle-info The current default target is `chrome`. The default target can change on any new release without notice ### [hashtag](https://script-ware.gitbook.io/cyberdrop-dl/reference/cli-arguments#max-items-retry) `max-items-retry` Type Default `NonNegativeInt` `0` Max number of links to retry. Using `0` means no limit circle-info This option has no effect unless you run CDL with one of the retry options: `--retry-all`, `--retry-failed` or `--retry-maintenance` ### [hashtag](https://script-ware.gitbook.io/cyberdrop-dl/reference/cli-arguments#portrait) `portrait` Type Default Action `BoolFlag` `False` `store_true` Force CDL to run with a vertical layout ### [hashtag](https://script-ware.gitbook.io/cyberdrop-dl/reference/cli-arguments#print-stats) `print-stats` Type Default Action `BoolFlag` `True` `store_false` Show stats report at the end of a run ### [hashtag](https://script-ware.gitbook.io/cyberdrop-dl/reference/cli-arguments#retry-all) `retry-all` Type Default Action `BoolFlag` `False` `store_true` Retry all downloads ### [hashtag](https://script-ware.gitbook.io/cyberdrop-dl/reference/cli-arguments#retry-failed) `retry-failed` Type Default Action `BoolFlag` `False` `store_true` Retry failed downloads ### [hashtag](https://script-ware.gitbook.io/cyberdrop-dl/reference/cli-arguments#retry-maintenance) `retry-maintenance` Type Default Action `BoolFlag` `False` `store_true` Retry download of maintenance files (bunkr). Requires files to be hashed ### [hashtag](https://script-ware.gitbook.io/cyberdrop-dl/reference/cli-arguments#show-supported-sites) `show-supported-sites` Type Default Action `BoolFlag` `False` `store_true` Shows a list of all supported sites and exits ### [hashtag](https://script-ware.gitbook.io/cyberdrop-dl/reference/cli-arguments#ui) `ui` Type Default `CaseInsensitiveStrEnum` `FULLSCREEN` UI can have 1 of these values: * `DISABLED` : no output at all * `ACTIVITY` : only shows a spinner with the text `running CDL...` * `SIMPLE`: shows spinner + simplified progress bar * `FULLSCREEN`: shows the normal UI/progress view circle-info Values are case insensitive, ex: both `disabled` and `DISABLED` are valid [hashtag](https://script-ware.gitbook.io/cyberdrop-dl/reference/cli-arguments#overview) Overview ----------------------------------------------------------------------------------------------------- Bool arguments like options within `Download Options`, `Ignore Options`, `Runtime Options`, etc. can be prefixed with `--no-` to negate them. Ex: `--no-auto-dedupe` will disable auto dedupe, overriding whatever the config option was set to. [PreviousGeneric Crawlerschevron-left](https://script-ware.gitbook.io/cyberdrop-dl/reference/configuration-options/global-settings/generic_crawlers_instances) [NextNotificationschevron-right](https://script-ware.gitbook.io/cyberdrop-dl/reference/notifications) Last updated 3 months ago sun-brightdesktopmoon Copy usage: cyberdrop-dl [OPTIONS] URL [URL...] Bulk asynchronous downloader for multiple file hosts options: -h, --help show this help message and exit -V, --version show CDL version number and exit CLI-only options: LINK(S) link(s) to content to download (passing multiple links is supported) --appdata-folder APPDATA_FOLDER AppData folder path --completed-after COMPLETED_AFTER only retry downloads that were completed on or after this date --completed-before COMPLETED_BEFORE only retry downloads that were completed on or before this date --config CONFIG name of config to load --config-file CONFIG_FILE path to the CDL settings.yaml file to load --disable-cache temporarily disable the requests cache --download skips UI, start download immediately --download-tiktok-audios download TikTok audios from posts and save them as separate files --download-tiktok-src-quality-videos download TikTok videos in source quality --impersonate [IMPERSONATE] Use this target as impersonation for all scrape requests --max-items-retry MAX_ITEMS_RETRY max number of links to retry --portrait force CDL to run with a vertical layout --print-stats show stats report at the end of a run --retry-all retry all downloads --retry-failed retry failed downloads --retry-maintenance retry download of maintenance files (bunkr). Requires files to be hashed --show-supported-sites shows a list of supported sites and exits --ui UI DISABLED, ACTIVITY, SIMPLE or FULLSCREEN browser_cookies: --auto-import, --no-auto-import --browser BROWSER --sites [SITES ...] download_options: --block-download-sub-folders, --no-block-download-sub-folders --disable-download-attempt-limit, --no-disable-download-attempt-limit --disable-file-timestamps, --no-disable-file-timestamps --include-album-id-in-folder-name, --no-include-album-id-in-folder-name --include-thread-id-in-folder-name, --no-include-thread-id-in-folder-name --maximum-number-of-children [MAXIMUM_NUMBER_OF_CHILDREN ...] --remove-domains-from-folder-names, --no-remove-domains-from-folder-names --remove-generated-id-from-filenames, --no-remove-generated-id-from-filenames --scrape-single-forum-post, --no-scrape-single-forum-post --separate-posts-format SEPARATE_POSTS_FORMAT --separate-posts, --no-separate-posts --skip-download-mark-completed, --no-skip-download-mark-completed --skip-referer-seen-before, --no-skip-referer-seen-before --maximum-thread-depth MAXIMUM_THREAD_DEPTH --maximum-thread-folder-depth MAXIMUM_THREAD_FOLDER_DEPTH dupe_cleanup_options: --add-md5-hash, --no-add-md5-hash --add-sha256-hash, --no-add-sha256-hash --auto-dedupe, --no-auto-dedupe --hashing HASHING --send-deleted-to-trash, --no-send-deleted-to-trash file_size_limits: --maximum-image-size MAXIMUM_IMAGE_SIZE --maximum-other-size MAXIMUM_OTHER_SIZE --maximum-video-size MAXIMUM_VIDEO_SIZE --minimum-image-size MINIMUM_IMAGE_SIZE --minimum-other-size MINIMUM_OTHER_SIZE --minimum-video-size MINIMUM_VIDEO_SIZE media_duration_limits: --maximum-video-duration MAXIMUM_VIDEO_DURATION --maximum-audio-duration MAXIMUM_AUDIO_DURATION --minimum-video-duration MINIMUM_VIDEO_DURATION --minimum-audio-duration MINIMUM_AUDIO_DURATION files: -d, --download-folder DOWNLOAD_FOLDER -j, --dump-json, --no-dump-json -i, --input-file INPUT_FILE --save-pages-html, --no-save-pages-html ignore_options: --exclude-audio, --no-exclude-audio --exclude-images, --no-exclude-images --exclude-other, --no-exclude-other --exclude-videos, --no-exclude-videos --filename-regex-filter FILENAME_REGEX_FILTER --ignore-coomer-ads, --no-ignore-coomer-ads --ignore-coomer-post-content, --no-ignore-coomer-post-content --only-hosts [ONLY_HOSTS ...] --skip-hosts [SKIP_HOSTS ...] --exclude-files-with-no-extension, --no-exclude-files-with-no-extension --exclude-before EXCLUDE_BEFORE --exclude-after EXCLUDE_AFTER logs: --download-error-urls DOWNLOAD_ERROR_URLS --last-forum-post LAST_FORUM_POST --log-folder LOG_FOLDER --log-line-width LOG_LINE_WIDTH --logs-expire-after LOGS_EXPIRE_AFTER --main-log MAIN_LOG --rotate-logs, --no-rotate-logs --scrape-error-urls SCRAPE_ERROR_URLS --unsupported-urls UNSUPPORTED_URLS --webhook WEBHOOK runtime_options: --console-log-level CONSOLE_LOG_LEVEL --deep-scrape, --no-deep-scrape --delete-partial-files, --no-delete-partial-files --ignore-history, --no-ignore-history --jdownloader-autostart, --no-jdownloader-autostart --jdownloader-download-dir JDOWNLOADER_DOWNLOAD_DIR --jdownloader-whitelist [JDOWNLOADER_WHITELIST ...] --log-level LOG_LEVEL --send-unsupported-to-jdownloader, --no-send-unsupported-to-jdownloader --skip-check-for-empty-folders, --no-skip-check-for-empty-folders --skip-check-for-partial-files, --no-skip-check-for-partial-files --slow-download-speed SLOW_DOWNLOAD_SPEED --update-last-forum-post, --no-update-last-forum-post sorting: --scan-folder SCAN_FOLDER --sort-downloads, --no-sort-downloads --sort-folder SORT_FOLDER --sort-incrementer-format SORT_INCREMENTER_FORMAT --sorted-audio SORTED_AUDIO --sorted-image SORTED_IMAGE --sorted-other SORTED_OTHER --sorted-video SORTED_VIDEO general: --ssl-context SSL_CONTEXT --disable-crawlers [DISABLE_CRAWLERS ...] --enable-generic-crawler, --no-enable-generic-crawler --flaresolverr FLARESOLVERR --max-file-name-length MAX_FILE_NAME_LENGTH --max-folder-name-length MAX_FOLDER_NAME_LENGTH --proxy PROXY --required-free-space REQUIRED_FREE_SPACE --user-agent USER_AGENT rate_limiting_options: --download-attempts DOWNLOAD_ATTEMPTS --download-delay DOWNLOAD_DELAY --download-speed-limit DOWNLOAD_SPEED_LIMIT --file-host-cache-expire-after FILE_HOST_CACHE_EXPIRE_AFTER --forum-cache-expire-after FORUM_CACHE_EXPIRE_AFTER --jitter JITTER --max-simultaneous-downloads-per-domain MAX_SIMULTANEOUS_DOWNLOADS_PER_DOMAIN --max-simultaneous-downloads MAX_SIMULTANEOUS_DOWNLOADS --rate-limit RATE_LIMIT --connection-timeout CONNECTION_TIMEOUT --read-timeout READ_TIMEOUT ui_options: --downloading-item-limit DOWNLOADING_ITEM_LIMIT --refresh-rate REFRESH_RATE --scraping-item-limit SCRAPING_ITEM_LIMIT --vi-mode, --no-vi-mode generic_crawlers_instances: --wordpress-media [WORDPRESS_MEDIA ...] --wordpress-html [WORDPRESS_HTML ...] --discourse [DISCOURSE ...] --chevereto [CHEVERETO ...] sun-brightdesktopmoon --- # Ignore Options | Cyberdrop-DL [hashtag](https://script-ware.gitbook.io/cyberdrop-dl/reference/configuration-options/settings/ignore_options#exclude_audio) `exclude_audio` ------------------------------------------------------------------------------------------------------------------------------------------------- Type Default `bool` `false` When this is set to `true`, the program will skip downloading audio files. [hashtag](https://script-ware.gitbook.io/cyberdrop-dl/reference/configuration-options/settings/ignore_options#exclude_images) `exclude_images` --------------------------------------------------------------------------------------------------------------------------------------------------- Type Default `bool` `false` When this is set to `true`, the program will skip downloading image files. [hashtag](https://script-ware.gitbook.io/cyberdrop-dl/reference/configuration-options/settings/ignore_options#exclude_videos) `exclude_videos` --------------------------------------------------------------------------------------------------------------------------------------------------- Type Default `bool` `false` When this is set to `true`, the program will skip downloading video files. [hashtag](https://script-ware.gitbook.io/cyberdrop-dl/reference/configuration-options/settings/ignore_options#exclude_other) `exclude_other` ------------------------------------------------------------------------------------------------------------------------------------------------- Type Default `bool` `false` When this is set to `true`, the program will skip downloading non media files files. [hashtag](https://script-ware.gitbook.io/cyberdrop-dl/reference/configuration-options/settings/ignore_options#exclude_files_with_no_extension) `exclude_files_with_no_extension` ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Type Default `bool` `true` When this is set to `true`, the program will skip downloading files without an extension. circle-info CDL internally assumes any file without an extension is an `.mp4` file. That means any option that applies to videos like `--exclude-videos` and `--minimum-video-size` will apply to them. The actual file will still be downloaded without an extension [hashtag](https://script-ware.gitbook.io/cyberdrop-dl/reference/configuration-options/settings/ignore_options#filename_regex_filter) `filename_regex_filter` ----------------------------------------------------------------------------------------------------------------------------------------------------------------- Type Default `NonEmptyStr` or `null` `null` Any download with a filename that matches this regex expression will be skipped [hashtag](https://script-ware.gitbook.io/cyberdrop-dl/reference/configuration-options/settings/ignore_options#ignore_coomer_ads) `ignore_coomer_ads` --------------------------------------------------------------------------------------------------------------------------------------------------------- Type Default `bool` `false` When this is set to `true`, CDL will skip posts marked as `#ad` by models. Despite the name, this option affects all kemono based sites (Nekohouse, Kemono and Coomer) circle-info This requires fetching all posts with the tag `#ad` first and them filter them out from the normal ones. This means at least 1 additional request per profile is needed, depending on how many ads the profile has. [hashtag](https://script-ware.gitbook.io/cyberdrop-dl/reference/configuration-options/settings/ignore_options#ignore_coomer_post_content) `ignore_coomer_post_content` --------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Type Default `bool` `true` When this is set to `false`, CDL will scan the text inside each post for URLs and process them. Despite the name, this option affects all kemono based sites (Nekohouse, Kemono and Coomer) circle-exclamation This requires making 50x more requests when downloading an entire profile. Only enable this option if you actually need it [hashtag](https://script-ware.gitbook.io/cyberdrop-dl/reference/configuration-options/settings/ignore_options#only_hosts) `only_hosts` ------------------------------------------------------------------------------------------------------------------------------------------- Type Default Additional Info `list[NonEmptyStr]` `[]` This is an [`AdditiveArg`](https://script-ware.gitbook.io/cyberdrop-dl/reference/configuration-options/special_setting_types#additiveargs) You can supply hosts that you'd like the program to exclusively scrape/download from. This setting accepts any domain, even if they are no supported. [hashtag](https://script-ware.gitbook.io/cyberdrop-dl/reference/configuration-options/settings/ignore_options#skip_hosts) `skip_hosts` ------------------------------------------------------------------------------------------------------------------------------------------- Type Default Additional Info `list[NonEmptyStr]` `[]` This is an [`AdditiveArg`](https://script-ware.gitbook.io/cyberdrop-dl/reference/configuration-options/special_setting_types#additiveargs) You can supply hosts that you'd like the program to skip, to not scrape/download from them. This setting accepts any domain, even if they are no supported. [hashtag](https://script-ware.gitbook.io/cyberdrop-dl/reference/configuration-options/settings/ignore_options#exclude_before) `exclude_before` --------------------------------------------------------------------------------------------------------------------------------------------------- Type Default Additional Info `date` or `null` `null` The date should a valid ISO 8601 format, for example, `2021-12-23` Do not download files uploaded before this date. [hashtag](https://script-ware.gitbook.io/cyberdrop-dl/reference/configuration-options/settings/ignore_options#exclude_after) `exclude_after` ------------------------------------------------------------------------------------------------------------------------------------------------- Type Default Additional Info `date` or `null` `null` The date should a valid ISO 8601 format, for example, `2021-12-23` Do not download files uploaded after this date. [PreviousFileschevron-left](https://script-ware.gitbook.io/cyberdrop-dl/reference/configuration-options/settings/files) [NextLogschevron-right](https://script-ware.gitbook.io/cyberdrop-dl/reference/configuration-options/settings/logs) Last updated 3 months ago * [exclude\_audio](https://script-ware.gitbook.io/cyberdrop-dl/reference/configuration-options/settings/ignore_options#exclude_audio) * [exclude\_images](https://script-ware.gitbook.io/cyberdrop-dl/reference/configuration-options/settings/ignore_options#exclude_images) * [exclude\_videos](https://script-ware.gitbook.io/cyberdrop-dl/reference/configuration-options/settings/ignore_options#exclude_videos) * [exclude\_other](https://script-ware.gitbook.io/cyberdrop-dl/reference/configuration-options/settings/ignore_options#exclude_other) * [exclude\_files\_with\_no\_extension](https://script-ware.gitbook.io/cyberdrop-dl/reference/configuration-options/settings/ignore_options#exclude_files_with_no_extension) * [filename\_regex\_filter](https://script-ware.gitbook.io/cyberdrop-dl/reference/configuration-options/settings/ignore_options#filename_regex_filter) * [ignore\_coomer\_ads](https://script-ware.gitbook.io/cyberdrop-dl/reference/configuration-options/settings/ignore_options#ignore_coomer_ads) * [ignore\_coomer\_post\_content](https://script-ware.gitbook.io/cyberdrop-dl/reference/configuration-options/settings/ignore_options#ignore_coomer_post_content) * [only\_hosts](https://script-ware.gitbook.io/cyberdrop-dl/reference/configuration-options/settings/ignore_options#only_hosts) * [skip\_hosts](https://script-ware.gitbook.io/cyberdrop-dl/reference/configuration-options/settings/ignore_options#skip_hosts) * [exclude\_before](https://script-ware.gitbook.io/cyberdrop-dl/reference/configuration-options/settings/ignore_options#exclude_before) * [exclude\_after](https://script-ware.gitbook.io/cyberdrop-dl/reference/configuration-options/settings/ignore_options#exclude_after) sun-brightdesktopmoon sun-brightdesktopmoon --- # Logs | Cyberdrop-DL [hashtag](https://script-ware.gitbook.io/cyberdrop-dl/reference/configuration-options/settings/logs#download_error_urls) `download_error_urls` --------------------------------------------------------------------------------------------------------------------------------------------------- Type Default Restrictions `Path` `Download_Error_URLs.csv` extension will always be overridden to `.csv` Path of the download error log. For relative paths, the final path will be `log_folder` / `download_error_urls` Cyberdrop-DL will output the links it fails to download, the reason and their origin in CSV format. [hashtag](https://script-ware.gitbook.io/cyberdrop-dl/reference/configuration-options/settings/logs#last_forum_post) `last_forum_post` ------------------------------------------------------------------------------------------------------------------------------------------- Type Default Restrictions `Path` `Last_Scraped_Forum_Posts.csv` extension will always be overridden to `.csv` Path of the forum-post log file. For relative paths, the final path will be `log_folder` / `last_forum_post` Cyberdrop-DL will store the link to the last forum posts it scraped from a given forum thread in this file. [hashtag](https://script-ware.gitbook.io/cyberdrop-dl/reference/configuration-options/settings/logs#log_folder) `log_folder` --------------------------------------------------------------------------------------------------------------------------------- Type Default `Path` `AppData/Configs/{config}/Logs` The path to the location you want Cyberdrop-DL to store logs in. [hashtag](https://script-ware.gitbook.io/cyberdrop-dl/reference/configuration-options/settings/logs#log_line_width) `log_line_width` ----------------------------------------------------------------------------------------------------------------------------------------- Type Default Restrictions `PositiveInt` `240` `>=50` Line width to use when writing to the main log file [hashtag](https://script-ware.gitbook.io/cyberdrop-dl/reference/configuration-options/settings/logs#logs_expire_after) `logs_expire_after` ----------------------------------------------------------------------------------------------------------------------------------------------- Type Default `timedelta` or `null` `null` With `rotate_logs` enabled, this setting specifies the retention period for log files before they are deleted. * A `timedelta` input is expected to be a valid ISO 8601 timespan, ex: `P10DT2H30M10S` * An `int` input is assumed to be the number of days * A `str` input is expected to be in the format; ` `, ex: `10 days`. * A `null` value means disabled (never delete any logs) circle-exclamation Any `.log` or `.csv` file within `log_folder` will be deleted, even if CDL did not create them circle-info Log files with an absolute path not relative to `log_folder` will never be deleted [hashtag](https://script-ware.gitbook.io/cyberdrop-dl/reference/configuration-options/settings/logs#main_log) `main_log` ----------------------------------------------------------------------------------------------------------------------------- Type Default Restrictions `Path` `downloader.log` extension will always be overridden to `.log` Path of main log file. For relative paths, the final path will be `log_folder` / `main_log` [hashtag](https://script-ware.gitbook.io/cyberdrop-dl/reference/configuration-options/settings/logs#rotate_logs) `rotate_logs` ----------------------------------------------------------------------------------------------------------------------------------- Type Default `bool` `false` If enabled, Cyberdrop-DL will add the current date and time as a suffix to each log file, in the format `YYMMDD_HHMMSS` Every log file will be created inside a sub folder with the current date This will prevent overriding old log files [hashtag](https://script-ware.gitbook.io/cyberdrop-dl/reference/configuration-options/settings/logs#scrape_error_urls) `scrape_error_urls` ----------------------------------------------------------------------------------------------------------------------------------------------- Type Default Restrictions `Path` `Scrape_Error_URLs.csv` extension will always be overridden to `.csv` What you want Cyberdrop-DL to call the scrape error log. For relative paths, the final path will be `log_folder` / `scrape_error_urls` Cyberdrop-DL will output the links it fails to scrape, the reason and their origin in CSV format. [hashtag](https://script-ware.gitbook.io/cyberdrop-dl/reference/configuration-options/settings/logs#unsupported_urls) `unsupported_urls` --------------------------------------------------------------------------------------------------------------------------------------------- Type Default Restrictions `Path` `Unsupported_URLs.csv` extension will always be overridden to `.csv` Path of the unsupported log file. For relative paths, the final path will be `log_folder` / `unsupported_urls` Cyberdrop-DL will output links it can't download to this file. [hashtag](https://script-ware.gitbook.io/cyberdrop-dl/reference/configuration-options/settings/logs#webhook) `webhook` --------------------------------------------------------------------------------------------------------------------------- Type Default Restrictions `AppriseURL` or `null` `null` The scheme of the URL must be `http` or `https` The URL of a webhook that you want to send download stats to (Ex: Discord). You can add the optional tag `attach_logs=` as a prefix to include a copy of the main log as an attachment. Example: > `attach_logs=https://discord.com/api/webhooks/webhook_id/webhook_token` [PreviousIgnore Optionschevron-left](https://script-ware.gitbook.io/cyberdrop-dl/reference/configuration-options/settings/ignore_options) [NextMedia Duration Limitschevron-right](https://script-ware.gitbook.io/cyberdrop-dl/reference/configuration-options/settings/media_duration_limits) Last updated 8 months ago * [download\_error\_urls](https://script-ware.gitbook.io/cyberdrop-dl/reference/configuration-options/settings/logs#download_error_urls) * [last\_forum\_post](https://script-ware.gitbook.io/cyberdrop-dl/reference/configuration-options/settings/logs#last_forum_post) * [log\_folder](https://script-ware.gitbook.io/cyberdrop-dl/reference/configuration-options/settings/logs#log_folder) * [log\_line\_width](https://script-ware.gitbook.io/cyberdrop-dl/reference/configuration-options/settings/logs#log_line_width) * [logs\_expire\_after](https://script-ware.gitbook.io/cyberdrop-dl/reference/configuration-options/settings/logs#logs_expire_after) * [main\_log](https://script-ware.gitbook.io/cyberdrop-dl/reference/configuration-options/settings/logs#main_log) * [rotate\_logs](https://script-ware.gitbook.io/cyberdrop-dl/reference/configuration-options/settings/logs#rotate_logs) * [scrape\_error\_urls](https://script-ware.gitbook.io/cyberdrop-dl/reference/configuration-options/settings/logs#scrape_error_urls) * [unsupported\_urls](https://script-ware.gitbook.io/cyberdrop-dl/reference/configuration-options/settings/logs#unsupported_urls) * [webhook](https://script-ware.gitbook.io/cyberdrop-dl/reference/configuration-options/settings/logs#webhook) sun-brightdesktopmoon sun-brightdesktopmoon --- # Authentication | Cyberdrop-DL Development Docs | Cyberdrop-DL All the options in these list are optional. The default value for all of them is an empty `str` Type Default `str` `""` chevron-rightCoomer[hashtag](https://script-ware.gitbook.io/cyberdrop-dl/dev/reference/configuration-options/authentication#coomer) * `session` Value of the `session` cookie from the website. This will allow CDL to download from your favoruites by using `https://coomer.st/favorites` as an input URL. chevron-rightKemono[hashtag](https://script-ware.gitbook.io/cyberdrop-dl/dev/reference/configuration-options/authentication#kemono) * `session` Value of the `session` cookie from the website. This will allow CDL to download from your favoruites by using `https://kemono.cr/favorites` as an input URL. chevron-rightForums[hashtag](https://script-ware.gitbook.io/cyberdrop-dl/dev/reference/configuration-options/authentication#forums) circle-exclamation Logging to forums with `Authentication` was deprecated in v6.7.0 You need to use cookie files. See: [How to extract cookies (DDoSGuard or login errors) #839arrow-up-right](https://github.com/jbsparrow/CyberDropDownloader/discussions/839) chevron-rightGoFile[hashtag](https://script-ware.gitbook.io/cyberdrop-dl/dev/reference/configuration-options/authentication#gofile) If you decide to pay for GoFile Premium (faster downloads, etc.) you can provide your API key to Cyberdrop-DL in order for the program to use it. * `api_key` You can get your API key here: [https://gofile.io/myProfilearrow-up-right](https://gofile.io/myProfile) chevron-rightJDownloader[hashtag](https://script-ware.gitbook.io/cyberdrop-dl/dev/reference/configuration-options/authentication#jdownloader) Under JDownloader 2 settings -> MyJDownloader: * `username` * `password` * `device` chevron-rightPixelDrain[hashtag](https://script-ware.gitbook.io/cyberdrop-dl/dev/reference/configuration-options/authentication#pixeldrain) If you decide to pay for PixelDrain premium (faster downloads, etc.) you can provide your API key to Cyberdrop-DL in order for the program to use it. * `api_key` You can get your API key here: [https://pixeldrain.com/user/api\_keysarrow-up-right](https://pixeldrain.com/user/api_keys) chevron-rightReal-Debrid[hashtag](https://script-ware.gitbook.io/cyberdrop-dl/dev/reference/configuration-options/authentication#real-debrid) In order to download files from sites supported by real-debrid, you'll need to get the API token from your account. * `api_key` You can get your API key here (you must be logged in): [https://real-debrid.com/apitokenarrow-up-right](https://real-debrid.com/apitoken) chevron-rightMega.nz[hashtag](https://script-ware.gitbook.io/cyberdrop-dl/dev/reference/configuration-options/authentication#mega.nz) If you have premium or want to download files/folders only shared with you, provide CDL these values: * `email` * `password` circle-exclamation Accounts with 2 factor authentication (2FA) are NOT supported [PreviousSpecial Type Optionschevron-left](https://script-ware.gitbook.io/cyberdrop-dl/dev/reference/configuration-options/special_setting_types) [NextSettingschevron-right](https://script-ware.gitbook.io/cyberdrop-dl/dev/reference/configuration-options/settings) Last updated 1 month ago sun-brightdesktopmoon sun-brightdesktopmoon --- # Special Type Options | Cyberdrop-DL Development Docs | Cyberdrop-DL [hashtag](https://script-ware.gitbook.io/cyberdrop-dl/dev/reference/configuration-options/special_setting_types#additiveargs) `AdditiveArgs` ------------------------------------------------------------------------------------------------------------------------------------------------- Special type for some settings that accept multiple values as input (`lists` or `sets`). When passing values as CLI arguments, if the first value is `+`, any value after it will be added to the values specified in the config file instead of overriding it. Similarly, if the first value is `-`, any value after it will be removed from the values specified in the config ### [hashtag](https://script-ware.gitbook.io/cyberdrop-dl/dev/reference/configuration-options/special_setting_types#examples) Examples If you have the following `skip_hosts` setting in you config file: Copy skip_hosts: - drive.google.com - youtube.com - facebook.com You will get the following results: Value you used Value CDL interpreted Details `--skip-hosts instagram.com` `[instagram.com]` config values were overridden by CLI values `--skip-hosts "+" instagram.com` `[drive.google.com, youtube.com, facebook.com, instagram.com]` config values and CLI values were merged `--skip-hosts "-" drive.google.com youtube.com` `[facebook.com]` CLI values were removed from the config values circle-info Always use quotes for "+" and "-" to make sure your shell does not try to parse them as additional flags [hashtag](https://script-ware.gitbook.io/cyberdrop-dl/dev/reference/configuration-options/special_setting_types#boolflag) `BoolFlag` ----------------------------------------------------------------------------------------------------------------------------------------- A special kind of `bool`. Within a config file, it can have a `true` or `false` value. However, when used via CLI, the value is assumed from the flag name. The normal name is `true`, and prefixing the name with `--no` means `false`. ### [hashtag](https://script-ware.gitbook.io/cyberdrop-dl/dev/reference/configuration-options/special_setting_types#examples-1) Examples Value you used Value CDL interpreted `--auto-dedupe` auto-dedupe: `true` `--no-auto-dedupe` auto-dedupe: `false` [hashtag](https://script-ware.gitbook.io/cyberdrop-dl/dev/reference/configuration-options/special_setting_types#bytesize) `ByteSize` ----------------------------------------------------------------------------------------------------------------------------------------- A special kind of `int` that also accepts suffixes like `GB`, `MiB` and `KB` to specify valid values In conformance with IEC 80000-13 Standard, `1KB` means `1000 bytes`, and `1KiB` means `1024 bytes`. In general, including a middle 'i' will cause the unit to be interpreted as a power of 2, rather than a power of 10. ### [hashtag](https://script-ware.gitbook.io/cyberdrop-dl/dev/reference/configuration-options/special_setting_types#examples-2) Examples Value you used Value CDL interpreted `1GB` `1.000.000.000 bytes` `512` `512 bytes` `1MiB` `1.048.576 bytes` circle-info `ByteSize` is also used for some settings to specify speed and it's interpreted as ` / second`. ex: `8MB` means `8MB/s` [PreviousConfiguration Optionschevron-left](https://script-ware.gitbook.io/cyberdrop-dl/dev/reference/configuration-options) [NextAuthenticationchevron-right](https://script-ware.gitbook.io/cyberdrop-dl/dev/reference/configuration-options/authentication) Last updated 5 months ago * [AdditiveArgs](https://script-ware.gitbook.io/cyberdrop-dl/dev/reference/configuration-options/special_setting_types#additiveargs) * [Examples](https://script-ware.gitbook.io/cyberdrop-dl/dev/reference/configuration-options/special_setting_types#examples) * [BoolFlag](https://script-ware.gitbook.io/cyberdrop-dl/dev/reference/configuration-options/special_setting_types#boolflag) * [Examples](https://script-ware.gitbook.io/cyberdrop-dl/dev/reference/configuration-options/special_setting_types#examples-1) * [ByteSize](https://script-ware.gitbook.io/cyberdrop-dl/dev/reference/configuration-options/special_setting_types#bytesize) * [Examples](https://script-ware.gitbook.io/cyberdrop-dl/dev/reference/configuration-options/special_setting_types#examples-2) sun-brightdesktopmoon sun-brightdesktopmoon --- # Dupe Cleanup Options | Cyberdrop-DL Development Docs | Cyberdrop-DL Cyberdrop-DL maintains an internal database of all downloaded files, indexed by their hashes. This can be used to automatically delete newly downloaded files if they were downloaded before. To enable auto dupe cleanup: 1. Set `hashing` to `IN_PLACE` or `POST_DOWNLOAD` 2. Set `auto_dedupe` to `true` [hashtag](https://script-ware.gitbook.io/cyberdrop-dl/dev/reference/configuration-options/settings/dupe_cleanup_options#add_md5_hash) `add_md5_hash` --------------------------------------------------------------------------------------------------------------------------------------------------------- Type Default `bool` `false` If enabled, calculates the `md5` hash for every downloaded file and store it in the internal database. This allows the program to match files with sites that provide hash information before downloading them, skipping the download altogether if a match is found circle-info `md5` was the default hashing algorithm of Cyberdrop-DL V5. Since v6, the default algorithm is `xxhash128`. [hashtag](https://script-ware.gitbook.io/cyberdrop-dl/dev/reference/configuration-options/settings/dupe_cleanup_options#add_sha256_hash) `add_sha256_hash` --------------------------------------------------------------------------------------------------------------------------------------------------------------- Type Default `bool` `false` If enabled, calculates the `sha256` hash for every downloaded file and store it in the internal database. This allows the program to match files with sites that provide hash information before downloading them, skipping the download altogether if a match is found [hashtag](https://script-ware.gitbook.io/cyberdrop-dl/dev/reference/configuration-options/settings/dupe_cleanup_options#auto_dedupe) `auto_dedupe` ------------------------------------------------------------------------------------------------------------------------------------------------------- Type Default `bool` `false` Enables deduping files functionality. Needs `hashing` to be enabled The `auto_dedupe` feature scans the database for files with matching hashes and sizes, automatically deleting any duplicates and retaining only the oldest copy. Deletion will only occur if two or more matching files are found during the database search. circle-exclamation Auto dedupe will delete files if you have _ever_ downloaded them before, even if you no longer have the original download on disk [hashtag](https://script-ware.gitbook.io/cyberdrop-dl/dev/reference/configuration-options/settings/dupe_cleanup_options#hashing) `hashing` ----------------------------------------------------------------------------------------------------------------------------------------------- Type Default Restrictions `HASHING` `OFF` Must be `OFF`, `IN_PLACE` or `POST_DOWNLOAD` There are three possible options for hashing: 1. `OFF`: disables hashing 2. `IN_PLACE`: performs hashing after each download 3. `POST_DOWNLOAD`: performs hashing after all downloads have completed The default hashing algorithm is `xxh128`. You can enable additional hashing algorithms, but you can not replace the default. [hashtag](https://script-ware.gitbook.io/cyberdrop-dl/dev/reference/configuration-options/settings/dupe_cleanup_options#send_deleted_to_trash) `send_deleted_to_trash` --------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Type Default `bool` `false` Deduped files are sent to the trash bin instead of being deleted [PreviousDownload Optionschevron-left](https://script-ware.gitbook.io/cyberdrop-dl/dev/reference/configuration-options/settings/download_options) [NextFile Size Limitschevron-right](https://script-ware.gitbook.io/cyberdrop-dl/dev/reference/configuration-options/settings/file_size_limits) Last updated 5 months ago * [add\_md5\_hash](https://script-ware.gitbook.io/cyberdrop-dl/dev/reference/configuration-options/settings/dupe_cleanup_options#add_md5_hash) * [add\_sha256\_hash](https://script-ware.gitbook.io/cyberdrop-dl/dev/reference/configuration-options/settings/dupe_cleanup_options#add_sha256_hash) * [auto\_dedupe](https://script-ware.gitbook.io/cyberdrop-dl/dev/reference/configuration-options/settings/dupe_cleanup_options#auto_dedupe) * [hashing](https://script-ware.gitbook.io/cyberdrop-dl/dev/reference/configuration-options/settings/dupe_cleanup_options#hashing) * [send\_deleted\_to\_trash](https://script-ware.gitbook.io/cyberdrop-dl/dev/reference/configuration-options/settings/dupe_cleanup_options#send_deleted_to_trash) sun-brightdesktopmoon sun-brightdesktopmoon --- # Media Duration Limits | Cyberdrop-DL Development Docs | Cyberdrop-DL You can provide the maximum and minimum duration for audio and video files. Type Default `timedelta` `0s` * A `timedelta` input is expected to be a valid ISO 8601 timespan, ex: `P10DT2H30M10S` * An `int` input is assumed to be the number of days * A `str` input is expected to be in the format; ` `, ex: `10 days`. Setting any of these options to `0` means that limit is `disabled` * `minimum_audio_duration` * `maximum_audio_duration` * `minimum_video_duration` * `maximum_video_duration` [PreviousLogschevron-left](https://script-ware.gitbook.io/cyberdrop-dl/dev/reference/configuration-options/settings/logs) [NextRuntime Optionschevron-right](https://script-ware.gitbook.io/cyberdrop-dl/dev/reference/configuration-options/settings/runtime_options) Last updated 1 year ago sun-brightdesktopmoon sun-brightdesktopmoon --- # Notifications | Cyberdrop-DL Development Docs | Cyberdrop-DL Cyberdrop-DL generates a report at the end of a run with stats about all the downloads, total runtime, errors, deduplication report, etc. By default, this report is only shown in the console and at the end of the main log file. You can set up CDL to sent you the report via discord, email, a native notification of your OS, telegram and many other services. [hashtag](https://script-ware.gitbook.io/cyberdrop-dl/dev/reference/notifications#notifications-via-discord) Notifications via Discord ------------------------------------------------------------------------------------------------------------------------------------------- To get notifications via discord, you need to provide a discord `webhook_url` inside the `setting.yaml` of the config you are running. You can learn how to setup a webhook following the [official discord guidearrow-up-right](https://support.discord.com/hc/en-us/articles/228383668-Intro-to-Webhooks) . Optionally, you can add the tag `attach_logs=` as a prefix to your webhook url. This will tell CDL to include a copy of the main log as an attachment to Discord. [hashtag](https://script-ware.gitbook.io/cyberdrop-dl/dev/reference/notifications#notifications-to-other-services-via-apprise) Notifications to other services (via Apprise) --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Cyberdrop-DL uses [Apprisearrow-up-right](https://github.com/caronc/apprise) to send notifications to any of the services than they support. circle-info `apprise` is an an optional dependency; It's not installed by default with CDL. To install `cyberdrop-dl` with `apprise`, especify it as an extra while installing cyberdrop: ex: Copy uv tool install cyberdrop-dl-patched[apprise] ### [hashtag](https://script-ware.gitbook.io/cyberdrop-dl/dev/reference/notifications#how-to-setup-apprise) How to setup Apprise To send notifications via Apprise, you need to create an `apprise.txt` file inside `AppData/Configs/`, where `` if the config you want to use. The file must contain a list of URLs and they must be in the format of one of the supported apprise services. You can check the full list of supported services [herearrow-up-right](https://github.com/caronc/apprise/wiki) and the URL format than each one uses [herearrow-up-right](https://github.com/caronc/apprise?tab=readme-ov-file#supported-notifications) . Apprise services also support the `attach_logs=` tag to send the main log as an attachment. ### [hashtag](https://script-ware.gitbook.io/cyberdrop-dl/dev/reference/notifications#troubleshooting-apprise-notifications) Troubleshooting Apprise notifications Cyberdrop-DL will show you a message at the end of a run telling you if the apprise notifications were successfully sent or not. If you are having trouble getting notifications via Apprise, follow their [troubleshooting guidearrow-up-right](https://github.com/caronc/apprise/wiki/Troubleshooting) . circle-info When running on Windows, Cyberdrop-DL will setup OS notifications by default. You can disable them by deleting the `windows://` line from the default `apprise.txt` file. You can also completely delete the file if you don't have any other notification setup. [hashtag](https://script-ware.gitbook.io/cyberdrop-dl/dev/reference/notifications#examples) Examples --------------------------------------------------------------------------------------------------------- Email Email + Logs Native OS notifications Discord + Logs To get notifications via email, use this URL format in your `apprise.txt` file: Add `attach_logs` to your email URL in your `apprise.txt` file: Some operating systems require additional dependencies for notifications to work. Cyberdrop-DL includes the required dependencies for Windows. Follow the url on the OS name to get additional information on how to set them up. OS Syntax [Linux (DBus Notifications)arrow-up-right](https://github.com/caronc/apprise/wiki/Notify_dbus) `dbus://` `qt://` `glib://` `kde://` [Linux (Gnome Notifications)arrow-up-right](https://github.com/caronc/apprise/wiki/Notify_gnome) `gnome://` [macOSarrow-up-right](https://github.com/caronc/apprise/wiki/Notify_macosx) `macosx://` [Windowsarrow-up-right](https://github.com/caronc/apprise/wiki/Notify_windows) `windows://` Add `attach_logs` to the `webhook_url` config option: [PreviousCLI Argumentschevron-left](https://script-ware.gitbook.io/cyberdrop-dl/dev/reference/cli-arguments) Last updated 1 month ago * [Notifications via Discord](https://script-ware.gitbook.io/cyberdrop-dl/dev/reference/notifications#notifications-via-discord) * [Notifications to other services (via Apprise)](https://script-ware.gitbook.io/cyberdrop-dl/dev/reference/notifications#notifications-to-other-services-via-apprise) * [How to setup Apprise](https://script-ware.gitbook.io/cyberdrop-dl/dev/reference/notifications#how-to-setup-apprise) * [Troubleshooting Apprise notifications](https://script-ware.gitbook.io/cyberdrop-dl/dev/reference/notifications#troubleshooting-apprise-notifications) * [Examples](https://script-ware.gitbook.io/cyberdrop-dl/dev/reference/notifications#examples) sun-brightdesktopmoon Copy mailto://user:password@domain.com Copy attach_logs=mailto://user:password@domain.com Copy attach_logs=https://discord.com/api/webhooks/webhook_id/webhook_token sun-brightdesktopmoon --- # Browser Cookies | Cyberdrop-DL Development Docs | Cyberdrop-DL Cyberdrop-DL can extract cookies from your browser. These can be used for websites that require login or to pass DDoS-Guard challenges. Only cookies from supported websites are extracted circle-exclamation The `user-agent` config value **MUST** match the `user-agent` of the browser from which you imported the cookies. If they do not match, the cookies will not work [hashtag](https://script-ware.gitbook.io/cyberdrop-dl/dev/reference/configuration-options/settings/browser_cookies#auto_import) `auto_import` -------------------------------------------------------------------------------------------------------------------------------------------------- Type Default `bool` `false` Toggles automatic import of cookies at the start of each run [hashtag](https://script-ware.gitbook.io/cyberdrop-dl/dev/reference/configuration-options/settings/browser_cookies#browser) `browser` ------------------------------------------------------------------------------------------------------------------------------------------ Type Default `BROWSER` `firefox` A browser to use for extraction. Browser must be a supported browser's name. ### [hashtag](https://script-ware.gitbook.io/cyberdrop-dl/dev/reference/configuration-options/settings/browser_cookies#supported-browsers) Supported Browsers Browser Windows Linux MacOS Arc ❌ ❌ ✅ Brave ❌ ✅ ✅ Chrome ❌ ✅ ✅ Chromium ❌ ✅ ✅ Edge ❌ ✅ ✅ Firefox ✅ ✅ ✅ LibreWolf ✅ ✅ ✅ Lynx ❌ ✅ ❌ Opera ❌ ✅ ✅ Opera\_GX ❌ ❌ ✅ Safari ❌ ❌ ✅ Vivaldi ❌ ✅ ✅ W3M ❌ ✅ ❌ [hashtag](https://script-ware.gitbook.io/cyberdrop-dl/dev/reference/configuration-options/settings/browser_cookies#sites) `sites` -------------------------------------------------------------------------------------------------------------------------------------- Type Default `list[DOMAINS]` `[<>]` List of domains to extract cookies from. You can put any domain on the list, even if they are not officially supported. [hashtag](https://script-ware.gitbook.io/cyberdrop-dl/dev/reference/configuration-options/settings/browser_cookies#manual-cookie-extraction) Manual Cookie Extraction -------------------------------------------------------------------------------------------------------------------------------------------------------------------------- If cookie extraction fails, you can manually extract the cookies from your browser using tools like [cookie-editorarrow-up-right](https://cookie-editor.com/) and save them at `AppData/Cookies/.txt`. The file must be a Netscape formatted cookie file. You can use any name for the file as long as it has a `.txt` extension. See: [How to extract cookies (DDoSGuard or login errors) #839arrow-up-right](https://github.com/jbsparrow/CyberDropDownloader/discussions/839) for detailed instructions circle-info Multiple cookie files are supported. You could have a `SocialMediaGirls.txt` file and a `cyberdrop.txt` file, for example [PreviousSettingschevron-left](https://script-ware.gitbook.io/cyberdrop-dl/dev/reference/configuration-options/settings) [NextDownload Optionschevron-right](https://script-ware.gitbook.io/cyberdrop-dl/dev/reference/configuration-options/settings/download_options) Last updated 5 months ago * [auto\_import](https://script-ware.gitbook.io/cyberdrop-dl/dev/reference/configuration-options/settings/browser_cookies#auto_import) * [browser](https://script-ware.gitbook.io/cyberdrop-dl/dev/reference/configuration-options/settings/browser_cookies#browser) * [Supported Browsers](https://script-ware.gitbook.io/cyberdrop-dl/dev/reference/configuration-options/settings/browser_cookies#supported-browsers) * [sites](https://script-ware.gitbook.io/cyberdrop-dl/dev/reference/configuration-options/settings/browser_cookies#sites) * [Manual Cookie Extraction](https://script-ware.gitbook.io/cyberdrop-dl/dev/reference/configuration-options/settings/browser_cookies#manual-cookie-extraction) sun-brightdesktopmoon sun-brightdesktopmoon --- # Configuration Options | Cyberdrop-DL Development Docs | Cyberdrop-DL There are three type of config files: config settings, global settings and authentication settings. [user-lockAuthenticationchevron-right](https://script-ware.gitbook.io/cyberdrop-dl/dev/reference/configuration-options/authentication) [earth-americasGlobal Settingschevron-right](https://script-ware.gitbook.io/cyberdrop-dl/dev/reference/configuration-options/global-settings) [screwdriver-wrenchSettingschevron-right](https://script-ware.gitbook.io/cyberdrop-dl/dev/reference/configuration-options/settings) circle-info **Global Settings** and **Authentication Settings** are "global". They apply to _all_ config settings. You can set them once and never touch them again. Each config setting will be setup by default to have a separate `URLs.txt` files and separate log files. [PreviousSupported Websiteschevron-left](https://script-ware.gitbook.io/cyberdrop-dl/dev/reference/supported-websites) [NextSpecial Type Optionschevron-right](https://script-ware.gitbook.io/cyberdrop-dl/dev/reference/configuration-options/special_setting_types) Last updated 3 months ago sun-brightdesktopmoon sun-brightdesktopmoon --- # UI Options | Cyberdrop-DL Development Docs | Cyberdrop-DL [hashtag](https://script-ware.gitbook.io/cyberdrop-dl/dev/reference/configuration-options/global-settings/ui-options#downloading_item_limit) `downloading_item_limit` -------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Type Default `PositiveInt` `5` This is the limit on the number of items shown in the UI (while downloading) before they are simply added to the overflow number (`and other files`) [hashtag](https://script-ware.gitbook.io/cyberdrop-dl/dev/reference/configuration-options/global-settings/ui-options#refresh_rate) `refresh_rate` ------------------------------------------------------------------------------------------------------------------------------------------------------ Type Default `PositiveInt` `10` This is the refresh rate per second for the UI. [hashtag](https://script-ware.gitbook.io/cyberdrop-dl/dev/reference/configuration-options/global-settings/ui-options#scraping_item_limit) `scraping_item_limit` -------------------------------------------------------------------------------------------------------------------------------------------------------------------- Type Default `PositiveInt` `5` This is the limit on the number of items shown in the UI (while scraping) before they are simply added to the overflow number (`and other links`) [hashtag](https://script-ware.gitbook.io/cyberdrop-dl/dev/reference/configuration-options/global-settings/ui-options#vi_mode) `vi_mode` -------------------------------------------------------------------------------------------------------------------------------------------- Type Default `bool` `false` This enables vi/vim key binds while editing/entering text in CDL. [PreviousRate Limiting Optionschevron-left](https://script-ware.gitbook.io/cyberdrop-dl/dev/reference/configuration-options/global-settings/rate-limiting-options) [NextGeneric Crawlerschevron-right](https://script-ware.gitbook.io/cyberdrop-dl/dev/reference/configuration-options/global-settings/generic_crawlers_instances) Last updated 1 year ago * [downloading\_item\_limit](https://script-ware.gitbook.io/cyberdrop-dl/dev/reference/configuration-options/global-settings/ui-options#downloading_item_limit) * [refresh\_rate](https://script-ware.gitbook.io/cyberdrop-dl/dev/reference/configuration-options/global-settings/ui-options#refresh_rate) * [scraping\_item\_limit](https://script-ware.gitbook.io/cyberdrop-dl/dev/reference/configuration-options/global-settings/ui-options#scraping_item_limit) * [vi\_mode](https://script-ware.gitbook.io/cyberdrop-dl/dev/reference/configuration-options/global-settings/ui-options#vi_mode) sun-brightdesktopmoon sun-brightdesktopmoon --- # Files | Cyberdrop-DL Development Docs | Cyberdrop-DL [hashtag](https://script-ware.gitbook.io/cyberdrop-dl/dev/reference/configuration-options/settings/files#download_folder) `download_folder` ------------------------------------------------------------------------------------------------------------------------------------------------ Type Default `Path` `Downloads` The path to the folder you want Cyberdrop-DL to download files to. [hashtag](https://script-ware.gitbook.io/cyberdrop-dl/dev/reference/configuration-options/settings/files#dump_json) `dump_json` ------------------------------------------------------------------------------------------------------------------------------------ Type Default `bool` `False` If enabled, CDL will created a [json linesarrow-up-right](https://jsonlines.org/) files with the information about every file downloaded in the current run. The path to this file will be the same as `--main-log` but with the extension `.results.jsonl` Each line in the file will contain the following details (this may change on future versions): Copy { "url": "https://store9.gofile.io/download/web/7c88c147-ABCD-4e4d-9a6c-12345678/a_video.mp4", "referer": "https://gofile.io/d/ABC123", "download_folder": "Downloads/Cyberdrop-DL Downloads/test_album (GoFile)", "filename": "0hxte0li0o931lwgcrzbz_source.mp4", "original_filename": "a_video.mp4", "download_filename": "0hxte0li0o931lwgcrzbz_source.mp4", "filesize": 12054723, "ext": ".mp4", "debrid_link": null, "duration": null, "album_id": "ABC123", "datetime": "2025-01-22T11:00:07", "parents": ["https://a_forum.com/threads/.54321/post-123123"], "parent_threads": ["https://a_forum.com/threads/.54321"], "partial_file": "Downloads/Cyberdrop-DL Downloads/test_album (GoFile)/a_video.mp4.part", "complete_file": "Downloads/Cyberdrop-DL Downloads/test_album (GoFile)/a_video.mp4", "hash": "xxh128:53ee56b7bfafa31b8780a572e9783df3", "downloaded": true, "attempts": 1 } [hashtag](https://script-ware.gitbook.io/cyberdrop-dl/dev/reference/configuration-options/settings/files#input_file) `input_file` -------------------------------------------------------------------------------------------------------------------------------------- Type Default `Path` `AppData/Configs/{config}/URLs.txt` The path to the text file containing the URLs you want to download. Each line should be a single URL. You can also use `html` code. Cyberdrop-DL will parse all the links on the HTML [hashtag](https://script-ware.gitbook.io/cyberdrop-dl/dev/reference/configuration-options/settings/files#save_pages_html) `save_pages_html` ------------------------------------------------------------------------------------------------------------------------------------------------ Type Default `bool` `False` CDL will save to disk a copy of every requests as an html file. The files will be saved to a folder named `cdl_responses`, located in the same folder as the main log file. circle-info Not every request made by CDL returns an HTML page (ex: API requests generally return JSON data). Only HTML responses will be saved [PreviousFile Size Limitschevron-left](https://script-ware.gitbook.io/cyberdrop-dl/dev/reference/configuration-options/settings/file_size_limits) [NextIgnore Optionschevron-right](https://script-ware.gitbook.io/cyberdrop-dl/dev/reference/configuration-options/settings/ignore_options) Last updated 5 months ago * [download\_folder](https://script-ware.gitbook.io/cyberdrop-dl/dev/reference/configuration-options/settings/files#download_folder) * [dump\_json](https://script-ware.gitbook.io/cyberdrop-dl/dev/reference/configuration-options/settings/files#dump_json) * [input\_file](https://script-ware.gitbook.io/cyberdrop-dl/dev/reference/configuration-options/settings/files#input_file) * [save\_pages\_html](https://script-ware.gitbook.io/cyberdrop-dl/dev/reference/configuration-options/settings/files#save_pages_html) sun-brightdesktopmoon sun-brightdesktopmoon --- # Sorting | Cyberdrop-DL Cyberdrop-DL has a file sorter built in, but it's disabled by default You can use the field names below to create a custom path format. You can also use essentially none of them and have a hard coded path. However, `filename` and `ext` must always be used. Common fields for sorting format options (supported for `audio`, `videos`, `images` and `other`): > `base_dir`: the name of highest level folder inside `scan_folder`. This normally is the model name or the thread name > > `ext`: the file extension (suffix) > > `file_date`: the file date. This is a datetime object, which means it accepts a custom format spec ex: `{file_date:%Y-%m}` > > `file_date_iso`: the file date as an iso 8601 string (`%Y-%m-%d`) > > `file_date_us`: the file date in the US format (`%Y-%d-%m`) > > `filename`: the file's name (stem) > > `parent_dir`: the name of the folder where the file is located at (its parent folder). This is normally the album name for photos or the post name for forums/reddit if `separate_post` is enabled > > `sort_dir`: the same path as `sort_folder` from the download options [hashtag](https://script-ware.gitbook.io/cyberdrop-dl/reference/configuration-options/settings/sorting#scan_folder) `scan_folder` -------------------------------------------------------------------------------------------------------------------------------------- Type Default `Path` or `null` `null` Sets the starting point for the file scan Each direct child of the `scan_folder` is recursively scanned, and files are moved based on your settings. If this is set to `null` (the default), the value of `download_dir` from the download options is used. [hashtag](https://script-ware.gitbook.io/cyberdrop-dl/reference/configuration-options/settings/sorting#sort_downloads) `sort_downloads` -------------------------------------------------------------------------------------------------------------------------------------------- Type Default `bool` `false` Setting this to `true` will allow Cyberdrop-DL to sort downloads after a run is complete. [hashtag](https://script-ware.gitbook.io/cyberdrop-dl/reference/configuration-options/settings/sorting#sort_folder) `sort_folder` -------------------------------------------------------------------------------------------------------------------------------------- Type Default `Path` `Downloads/Cyberdrop-DL Sorted Downloads` This is the path to the folder you'd like sorted downloads to be stored in. circle-exclamation Setting `sort_folder` to the same value as `scan_folder` is not supported and will lead to expected results [hashtag](https://script-ware.gitbook.io/cyberdrop-dl/reference/configuration-options/settings/sorting#sort_incrementer_format) `sort_incrementer_format` -------------------------------------------------------------------------------------------------------------------------------------------------------------- Type Default `NonEmptyStr` `({i})` When naming collisions happen, Cyberdrop-DL will rename files automatically > `image.jpg` -> `image (1).jpg`. You can modify the format as needed, but it must include `{i}` to specify where the auto-increment value should be placed [hashtag](https://script-ware.gitbook.io/cyberdrop-dl/reference/configuration-options/settings/sorting#sorted_audio) `sorted_audio` ---------------------------------------------------------------------------------------------------------------------------------------- Type Default `NonEmptyStr` or `null` `{sort_dir}/{base_dir}/Audio/{filename}{ext}` This is the format for the directory structure and naming scheme for audio files. Set to `null` to skip sorting audio files In addition to the common sorting format fields, this option supports: > `bitrate`: file bit rate. This is an `int` > > `duration`: audio total runtime in seconds. This is an `int` > > `length`: same as `duration` > > `sample_rate`: audio sample rate. This is an `int` [hashtag](https://script-ware.gitbook.io/cyberdrop-dl/reference/configuration-options/settings/sorting#sorted_image) `sorted_image` ---------------------------------------------------------------------------------------------------------------------------------------- Type Default `NonEmptyStr` or `null` `{sort_dir}/{base_dir}/Images/{filename}{ext}` This is the format for the directory structure and naming scheme for image files. Set to `null` to skip sorting image files In addition to the common sorting format fields, this option supports: > `height`: vertical pixel count. This is an `int` > > `width`: horizontal pixel count. This is an `int` > > `resolution`: `width`x`height` ex. 1080x1920. This is a `str` [hashtag](https://script-ware.gitbook.io/cyberdrop-dl/reference/configuration-options/settings/sorting#sorted_video) `sorted_video` ---------------------------------------------------------------------------------------------------------------------------------------- Type Default `NonEmptyStr` or `null` `{sort_dir}/{base_dir}/Video/{filename}{ext}` This is the format for the directory structure and naming scheme for video files. Set to `null` to skip sorting video files In addition to the common sorting format fields, this option supports: > `codec`: ex. h264. This is a `str`. It could potentially be `None` for some files > > `duration`: video total runtime in seconds. This is an `int` > > `fps`: ex. `24`. This represents a number but is a `str`. It could potentially be `None` for some files > > `length`: same as `duration` > > `height`: vertical pixel count. This is an `int` > > `width`: horizontal pixel count.This is an `int` > > `resolution`: `width`x`height` ex. 1080x1920. This is a `str` [hashtag](https://script-ware.gitbook.io/cyberdrop-dl/reference/configuration-options/settings/sorting#sorted_other) `sorted_other` ---------------------------------------------------------------------------------------------------------------------------------------- Type Default `NonEmptyStr` or `null` `{sort_dir}/{base_dir}/Other/{filename}{ext}` This is the format for the directory structure and naming scheme for other files. Set to `null` to skip sorting other files [hashtag](https://script-ware.gitbook.io/cyberdrop-dl/reference/configuration-options/settings/sorting#group-urls) Group URLs ---------------------------------------------------------------------------------------------------------------------------------- It is possible to treat a list of URLs as a group, allowing them to be downloaded to a single folder. To define a group, put a title above the URLs you want to be in the group, using the format: `--- ` or `=== `. To define the end of a group, add a new group with no name. (`---` or `===`) Here is an example URL file with two groups: Those downloads would be sorted as follows: Loose Files Group 1 Group 2 file1.jpg file2.jpg file5.jpg file4.jpg file3.jpg file6.jpg file7.jpg [PreviousRuntime Optionschevron-left](https://script-ware.gitbook.io/cyberdrop-dl/reference/configuration-options/settings/runtime_options) [NextGlobal Settingschevron-right](https://script-ware.gitbook.io/cyberdrop-dl/reference/configuration-options/global-settings) Last updated 5 months ago * [scan\_folder](https://script-ware.gitbook.io/cyberdrop-dl/reference/configuration-options/settings/sorting#scan_folder) * [sort\_downloads](https://script-ware.gitbook.io/cyberdrop-dl/reference/configuration-options/settings/sorting#sort_downloads) * [sort\_folder](https://script-ware.gitbook.io/cyberdrop-dl/reference/configuration-options/settings/sorting#sort_folder) * [sort\_incrementer\_format](https://script-ware.gitbook.io/cyberdrop-dl/reference/configuration-options/settings/sorting#sort_incrementer_format) * [sorted\_audio](https://script-ware.gitbook.io/cyberdrop-dl/reference/configuration-options/settings/sorting#sorted_audio) * [sorted\_image](https://script-ware.gitbook.io/cyberdrop-dl/reference/configuration-options/settings/sorting#sorted_image) * [sorted\_video](https://script-ware.gitbook.io/cyberdrop-dl/reference/configuration-options/settings/sorting#sorted_video) * [sorted\_other](https://script-ware.gitbook.io/cyberdrop-dl/reference/configuration-options/settings/sorting#sorted_other) * [Group URLs](https://script-ware.gitbook.io/cyberdrop-dl/reference/configuration-options/settings/sorting#group-urls) sun-brightdesktopmoon Copy https://example.com/file1.jpg === Group 1 https://example.com/file2.jpg https://example.com/file3.jpg === https://example.com/file4.jpg --- Group 2 https://example.com/file5.jpg https://example.com/file6.jpg === https://example.com/file7.jpg sun-brightdesktopmoon --- # Global Settings | Cyberdrop-DL Development Docs | Cyberdrop-DL [Generalchevron-right](https://script-ware.gitbook.io/cyberdrop-dl/dev/reference/configuration-options/global-settings/general) [Rate Limiting Optionschevron-right](https://script-ware.gitbook.io/cyberdrop-dl/dev/reference/configuration-options/global-settings/rate-limiting-options) [UI Optionschevron-right](https://script-ware.gitbook.io/cyberdrop-dl/dev/reference/configuration-options/global-settings/ui-options) [Generic Crawlerschevron-right](https://script-ware.gitbook.io/cyberdrop-dl/dev/reference/configuration-options/global-settings/generic_crawlers_instances) [PreviousSortingchevron-left](https://script-ware.gitbook.io/cyberdrop-dl/dev/reference/configuration-options/settings/sorting) [NextGeneralchevron-right](https://script-ware.gitbook.io/cyberdrop-dl/dev/reference/configuration-options/global-settings/general) Last updated 1 year ago sun-brightdesktopmoon sun-brightdesktopmoon --- # Runtime Options | Cyberdrop-DL These are higher level options that effect the overarching functions of the program. [hashtag](https://script-ware.gitbook.io/cyberdrop-dl/reference/configuration-options/settings/runtime_options#console_log_level) `console_log_level` ---------------------------------------------------------------------------------------------------------------------------------------------------------- Type Default `NonNegativeInt` `100` Same as `log_level` but it controls which messages are shown on the console. circle-info This option is ignored unless CDL is running with `--ui DISABLED` [hashtag](https://script-ware.gitbook.io/cyberdrop-dl/reference/configuration-options/settings/runtime_options#deep_scrape) `deep_scrape` ---------------------------------------------------------------------------------------------------------------------------------------------- Type Default `bool` `false` Cyberdrop-DL uses a some tricks to try to reduce the number of requests it needs to make while scraping a site. However, this may cause a few links to be skipped. Use `--deep-scrape` to disable this functionality and always make a new request. This setting is temporary and will always reset to `false` after each run circle-exclamation Use this option only when absolutely necessary, as it will significantly increase the number of requests being made. For example, scraping an album normally takes one single request. With `--deep-scrape`, CDL will make `n` requests per album, where `n` is the total number of items in the album [hashtag](https://script-ware.gitbook.io/cyberdrop-dl/reference/configuration-options/settings/runtime_options#delete_partial_files) `delete_partial_files` ---------------------------------------------------------------------------------------------------------------------------------------------------------------- Type Default `bool` `false` Files downloaded by CDL have a `.part` extension (`.cdl_hls` for HLS segments). CDL only changes the extension to the original one after a successful download. This allows CDL to resume downloads on subsequent runs. Setting this to `true` will delete any `.part` and `.cdl_hls` files in the download folder. [hashtag](https://script-ware.gitbook.io/cyberdrop-dl/reference/configuration-options/settings/runtime_options#ignore_history) `ignore_history` ---------------------------------------------------------------------------------------------------------------------------------------------------- Type Default `bool` `false` By default, the program tracks your downloads in a database to prevent downloading the same files multiple times, to save time and reduce strain on the servers you're downloading from. Setting this to `true` will cause the program to ignore the database, and will allow you to re-download files. [hashtag](https://script-ware.gitbook.io/cyberdrop-dl/reference/configuration-options/settings/runtime_options#jdownloader_autostart) `jdownloader_autostart` ------------------------------------------------------------------------------------------------------------------------------------------------------------------ Type Default `bool` `false` Setting this to `true` will make jdownloader start downloads as soon as they are sent. circle-info This option has no effect unless `send_unsupported_to_jdownloader` is `true` [hashtag](https://script-ware.gitbook.io/cyberdrop-dl/reference/configuration-options/settings/runtime_options#jdownloader_download_dir) `jdownloader_download_dir` ------------------------------------------------------------------------------------------------------------------------------------------------------------------------ Type Default `Path` or `null` `null` The `download_dir` jdownloader will use. A `null` value (the default) will make jdownloader use the same `download_dir` as Cyberdrop-DL. Use this option as path mapping when jdownloader is running on a different host / docker. circle-info This option has no effect unless `send_unsupported_to_jdownloader` is `true` [hashtag](https://script-ware.gitbook.io/cyberdrop-dl/reference/configuration-options/settings/runtime_options#jdownloader_whitelist) `jdownloader_whitelist` ------------------------------------------------------------------------------------------------------------------------------------------------------------------ Type Default `list[NonEmptyStr]` `[]` List of domain names. An unsupported URL will only be sent to jdownloader if its host is found on the list. An empty whitelist (the default) will disable this functionality, sending any unsupported URL to jdownloader. circle-info This option has no effect unless `send_unsupported_to_jdownloader` is `true` [hashtag](https://script-ware.gitbook.io/cyberdrop-dl/reference/configuration-options/settings/runtime_options#log_level) `log_level` ------------------------------------------------------------------------------------------------------------------------------------------ Type Default `NonNegativeInt` `10` Defines the logging level for messages, according to [Python logging levelsarrow-up-right](https://docs.python.org/3/library/logging.html#levels) . Only messages with a level equal or greater than this setting will be logged, with the exception of `NOTSET` which will disable all logging messages. Setting a value above 50 will also disable logging entirely. Level Value Description `NOTSET` 0 Disables all log messages `DEBUG` 10 Offers detailed information, typically useful to troubleshoot problems `INFO` 20 Provides general information about scrape and download progress `WARNING` 30 Potential issues or something that might need attention (e.g. `Login wasn't provided for `) `ERROR` 40 Serious problem preventing Cyberdrop-DL to execute some function `CRITICAL` 50 Fatal error that causes Cyberdrop-DL to exit immediately circle-info Using anything other that `DEBUG` makes troubleshooting issues harder. Practically speaking, you should _only_ change this if you expect CDL to run for an extended period (with a large number of input URLs) to minimize the log files sizes circle-info There is one more verbose level of logs: `DEVELOPER` It includes pager logs, posts logs, individual requests logs (their URLs and their response code), requests cache hits and misses, file system locks logs and local variable values from within a traceback. It's not exposed as a config option because it is too verbose. You can set the environment variable `CDL_DEBUG_LOG_FOLDER` to any valid folder and CDL will create an additional log file with the level set to `DEVELOPER` inside it. circle-exclamation As the name suggests, the `DEVELOPER` log level is only meant for developers or, more generally, people that want and have the know how to troubleshoot CDL issues on their own. You should never share those logs with _anyone_ as it could potentially have raw information in the tracebacks like cookies, auth credentials, IPs, etc. [hashtag](https://script-ware.gitbook.io/cyberdrop-dl/reference/configuration-options/settings/runtime_options#send_unsupported_to_jdownloader) `send_unsupported_to_jdownloader` -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Type Default `bool` `false` Cyberdrop-DL has integration with jdownloader. This will allow you to download URLs that Cyberdrop-DL finds but do not support. However, this setting is disabled by default. Setting this to `true`, will send unsupported links over. [hashtag](https://script-ware.gitbook.io/cyberdrop-dl/reference/configuration-options/settings/runtime_options#skip_check_for_empty_folders) `skip_check_for_empty_folders` -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Type Default `bool` `false` After a run is complete, the program will do a check (and remove) any empty files and folders in the download and scan folder. Setting this to `true` will disable this functionality. [hashtag](https://script-ware.gitbook.io/cyberdrop-dl/reference/configuration-options/settings/runtime_options#skip_check_for_partial_files) `skip_check_for_partial_files` -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Type Default `bool` `false` After a run is complete, the program will do a check to see if any partially downloaded files remain in the downloads folder and will notify you of them. Setting this to `true` will skip this check. [hashtag](https://script-ware.gitbook.io/cyberdrop-dl/reference/configuration-options/settings/runtime_options#slow_download_speed) `slow_download_speed` -------------------------------------------------------------------------------------------------------------------------------------------------------------- Type Default `ByteSize` `0` circle-info This setting specifies speed and it's interpreted as ` / second`. ex: `8MB` means `8MB/s` Downloads with a speed lower than this value for more than 10 seconds will be skipped. Set to `0` to disable [hashtag](https://script-ware.gitbook.io/cyberdrop-dl/reference/configuration-options/settings/runtime_options#update_last_forum_post) `update_last_forum_post` -------------------------------------------------------------------------------------------------------------------------------------------------------------------- Type Default `bool` `false` Updates `input_file` content, adding the last scraped post id to every forum thread URL [PreviousMedia Duration Limitschevron-left](https://script-ware.gitbook.io/cyberdrop-dl/reference/configuration-options/settings/media_duration_limits) [NextSortingchevron-right](https://script-ware.gitbook.io/cyberdrop-dl/reference/configuration-options/settings/sorting) Last updated 3 months ago * [console\_log\_level](https://script-ware.gitbook.io/cyberdrop-dl/reference/configuration-options/settings/runtime_options#console_log_level) * [deep\_scrape](https://script-ware.gitbook.io/cyberdrop-dl/reference/configuration-options/settings/runtime_options#deep_scrape) * [delete\_partial\_files](https://script-ware.gitbook.io/cyberdrop-dl/reference/configuration-options/settings/runtime_options#delete_partial_files) * [ignore\_history](https://script-ware.gitbook.io/cyberdrop-dl/reference/configuration-options/settings/runtime_options#ignore_history) * [jdownloader\_autostart](https://script-ware.gitbook.io/cyberdrop-dl/reference/configuration-options/settings/runtime_options#jdownloader_autostart) * [jdownloader\_download\_dir](https://script-ware.gitbook.io/cyberdrop-dl/reference/configuration-options/settings/runtime_options#jdownloader_download_dir) * [jdownloader\_whitelist](https://script-ware.gitbook.io/cyberdrop-dl/reference/configuration-options/settings/runtime_options#jdownloader_whitelist) * [log\_level](https://script-ware.gitbook.io/cyberdrop-dl/reference/configuration-options/settings/runtime_options#log_level) * [send\_unsupported\_to\_jdownloader](https://script-ware.gitbook.io/cyberdrop-dl/reference/configuration-options/settings/runtime_options#send_unsupported_to_jdownloader) * [skip\_check\_for\_empty\_folders](https://script-ware.gitbook.io/cyberdrop-dl/reference/configuration-options/settings/runtime_options#skip_check_for_empty_folders) * [skip\_check\_for\_partial\_files](https://script-ware.gitbook.io/cyberdrop-dl/reference/configuration-options/settings/runtime_options#skip_check_for_partial_files) * [slow\_download\_speed](https://script-ware.gitbook.io/cyberdrop-dl/reference/configuration-options/settings/runtime_options#slow_download_speed) * [update\_last\_forum\_post](https://script-ware.gitbook.io/cyberdrop-dl/reference/configuration-options/settings/runtime_options#update_last_forum_post) sun-brightdesktopmoon sun-brightdesktopmoon --- # Generic Crawlers | Cyberdrop-DL Development Docs | Cyberdrop-DL CDL has some "generic" crawlers disabled by default. Generic crawlers are designed to work on any site that uses a **specific** framework. Users can supply a list of sites to map to these crawlers, and CDL will then be able to download from them. The URL in the list should be the primary URL of the site. ex: `https://forums.docker.com/` Supported generic crawlers: * `chevereto`: This works on any site that uses [Cheveretoarrow-up-right](https://chevereto.com//) . * `discourse`: This works on any forum that uses [Discoursearrow-up-right](https://www.discourse.org/) . * `wordpress_media`: This crawler should work on any [WordPressarrow-up-right](https://wordpress.com/) site where content primarily consists of images or galleries. The images need to be hosted on the site itself. It requires sites to have a public WordPress REST API. * `wordpress_html`: This works on any WordPress site. It scrapes the actual HTML of the site, which means it works on sites that have embedded third-party media like videos or links to hosting sites. It is always slower than `wordpress_media`. [hashtag](https://script-ware.gitbook.io/cyberdrop-dl/dev/reference/configuration-options/global-settings/generic_crawlers_instances#generic_crawlers_instances) generic\_crawlers\_instances -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- [hashtag](https://script-ware.gitbook.io/cyberdrop-dl/dev/reference/configuration-options/global-settings/generic_crawlers_instances#chevereto) `chevereto` ---------------------------------------------------------------------------------------------------------------------------------------------------------------- Type Default `list[HttpURL]` `[]` [hashtag](https://script-ware.gitbook.io/cyberdrop-dl/dev/reference/configuration-options/global-settings/generic_crawlers_instances#discourse) `discourse` ---------------------------------------------------------------------------------------------------------------------------------------------------------------- Type Default `list[HttpURL]` `[]` [hashtag](https://script-ware.gitbook.io/cyberdrop-dl/dev/reference/configuration-options/global-settings/generic_crawlers_instances#wordpress_media) `wordpress_media` ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Type Default `list[HttpURL]` `[]` [hashtag](https://script-ware.gitbook.io/cyberdrop-dl/dev/reference/configuration-options/global-settings/generic_crawlers_instances#wordpress_html) `wordpress_html` -------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Type Default `list[HttpURL]` `[]` [PreviousUI Optionschevron-left](https://script-ware.gitbook.io/cyberdrop-dl/dev/reference/configuration-options/global-settings/ui-options) [NextCLI Argumentschevron-right](https://script-ware.gitbook.io/cyberdrop-dl/dev/reference/cli-arguments) Last updated 5 months ago * [generic\_crawlers\_instances](https://script-ware.gitbook.io/cyberdrop-dl/dev/reference/configuration-options/global-settings/generic_crawlers_instances#generic_crawlers_instances) * [chevereto](https://script-ware.gitbook.io/cyberdrop-dl/dev/reference/configuration-options/global-settings/generic_crawlers_instances#chevereto) * [discourse](https://script-ware.gitbook.io/cyberdrop-dl/dev/reference/configuration-options/global-settings/generic_crawlers_instances#discourse) * [wordpress\_media](https://script-ware.gitbook.io/cyberdrop-dl/dev/reference/configuration-options/global-settings/generic_crawlers_instances#wordpress_media) * [wordpress\_html](https://script-ware.gitbook.io/cyberdrop-dl/dev/reference/configuration-options/global-settings/generic_crawlers_instances#wordpress_html) sun-brightdesktopmoon sun-brightdesktopmoon --- # Ignore Options | Cyberdrop-DL Development Docs | Cyberdrop-DL [hashtag](https://script-ware.gitbook.io/cyberdrop-dl/dev/reference/configuration-options/settings/ignore_options#exclude_audio) `exclude_audio` ----------------------------------------------------------------------------------------------------------------------------------------------------- Type Default `bool` `false` When this is set to `true`, the program will skip downloading audio files. [hashtag](https://script-ware.gitbook.io/cyberdrop-dl/dev/reference/configuration-options/settings/ignore_options#exclude_images) `exclude_images` ------------------------------------------------------------------------------------------------------------------------------------------------------- Type Default `bool` `false` When this is set to `true`, the program will skip downloading image files. [hashtag](https://script-ware.gitbook.io/cyberdrop-dl/dev/reference/configuration-options/settings/ignore_options#exclude_videos) `exclude_videos` ------------------------------------------------------------------------------------------------------------------------------------------------------- Type Default `bool` `false` When this is set to `true`, the program will skip downloading video files. [hashtag](https://script-ware.gitbook.io/cyberdrop-dl/dev/reference/configuration-options/settings/ignore_options#exclude_other) `exclude_other` ----------------------------------------------------------------------------------------------------------------------------------------------------- Type Default `bool` `false` When this is set to `true`, the program will skip downloading non media files files. [hashtag](https://script-ware.gitbook.io/cyberdrop-dl/dev/reference/configuration-options/settings/ignore_options#exclude_files_with_no_extension) `exclude_files_with_no_extension` ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Type Default `bool` `true` When this is set to `true`, the program will skip downloading files without an extension. circle-info CDL internally assumes any file without an extension is an `.mp4` file. That means any option that applies to videos like `--exclude-videos` and `--minimum-video-size` will apply to them. The actual file will still be downloaded without an extension [hashtag](https://script-ware.gitbook.io/cyberdrop-dl/dev/reference/configuration-options/settings/ignore_options#filename_regex_filter) `filename_regex_filter` --------------------------------------------------------------------------------------------------------------------------------------------------------------------- Type Default `NonEmptyStr` or `null` `null` Any download with a filename that matches this regex expression will be skipped [hashtag](https://script-ware.gitbook.io/cyberdrop-dl/dev/reference/configuration-options/settings/ignore_options#ignore_coomer_ads) `ignore_coomer_ads` ------------------------------------------------------------------------------------------------------------------------------------------------------------- Type Default `bool` `false` When this is set to `true`, CDL will skip posts marked as `#ad` by models. Despite the name, this option affects all kemono based sites (Nekohouse, Kemono and Coomer) circle-info This requires fetching all posts with the tag `#ad` first and them filter them out from the normal ones. This means at least 1 additional request per profile is needed, depending on how many ads the profile has. [hashtag](https://script-ware.gitbook.io/cyberdrop-dl/dev/reference/configuration-options/settings/ignore_options#ignore_coomer_post_content) `ignore_coomer_post_content` ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Type Default `bool` `true` When this is set to `false`, CDL will scan the text inside each post for URLs and process them. Despite the name, this option affects all kemono based sites (Nekohouse, Kemono and Coomer) circle-exclamation This requires making 50x more requests when downloading an entire profile. Only enable this option if you actually need it [hashtag](https://script-ware.gitbook.io/cyberdrop-dl/dev/reference/configuration-options/settings/ignore_options#only_hosts) `only_hosts` ----------------------------------------------------------------------------------------------------------------------------------------------- Type Default Additional Info `list[NonEmptyStr]` `[]` This is an [`AdditiveArg`](https://script-ware.gitbook.io/cyberdrop-dl/dev/reference/configuration-options/special_setting_types#additiveargs) You can supply hosts that you'd like the program to exclusively scrape/download from. This setting accepts any domain, even if they are no supported. [hashtag](https://script-ware.gitbook.io/cyberdrop-dl/dev/reference/configuration-options/settings/ignore_options#skip_hosts) `skip_hosts` ----------------------------------------------------------------------------------------------------------------------------------------------- Type Default Additional Info `list[NonEmptyStr]` `[]` This is an [`AdditiveArg`](https://script-ware.gitbook.io/cyberdrop-dl/dev/reference/configuration-options/special_setting_types#additiveargs) You can supply hosts that you'd like the program to skip, to not scrape/download from them. This setting accepts any domain, even if they are no supported. [hashtag](https://script-ware.gitbook.io/cyberdrop-dl/dev/reference/configuration-options/settings/ignore_options#exclude_before) `exclude_before` ------------------------------------------------------------------------------------------------------------------------------------------------------- Type Default Additional Info `date` or `null` `null` The date should a valid ISO 8601 format, for example, `2021-12-23` Do not download files uploaded before this date. [hashtag](https://script-ware.gitbook.io/cyberdrop-dl/dev/reference/configuration-options/settings/ignore_options#exclude_after) `exclude_after` ----------------------------------------------------------------------------------------------------------------------------------------------------- Type Default Additional Info `date` or `null` `null` The date should a valid ISO 8601 format, for example, `2021-12-23` Do not download files uploaded after this date. [PreviousFileschevron-left](https://script-ware.gitbook.io/cyberdrop-dl/dev/reference/configuration-options/settings/files) [NextLogschevron-right](https://script-ware.gitbook.io/cyberdrop-dl/dev/reference/configuration-options/settings/logs) Last updated 3 months ago * [exclude\_audio](https://script-ware.gitbook.io/cyberdrop-dl/dev/reference/configuration-options/settings/ignore_options#exclude_audio) * [exclude\_images](https://script-ware.gitbook.io/cyberdrop-dl/dev/reference/configuration-options/settings/ignore_options#exclude_images) * [exclude\_videos](https://script-ware.gitbook.io/cyberdrop-dl/dev/reference/configuration-options/settings/ignore_options#exclude_videos) * [exclude\_other](https://script-ware.gitbook.io/cyberdrop-dl/dev/reference/configuration-options/settings/ignore_options#exclude_other) * [exclude\_files\_with\_no\_extension](https://script-ware.gitbook.io/cyberdrop-dl/dev/reference/configuration-options/settings/ignore_options#exclude_files_with_no_extension) * [filename\_regex\_filter](https://script-ware.gitbook.io/cyberdrop-dl/dev/reference/configuration-options/settings/ignore_options#filename_regex_filter) * [ignore\_coomer\_ads](https://script-ware.gitbook.io/cyberdrop-dl/dev/reference/configuration-options/settings/ignore_options#ignore_coomer_ads) * [ignore\_coomer\_post\_content](https://script-ware.gitbook.io/cyberdrop-dl/dev/reference/configuration-options/settings/ignore_options#ignore_coomer_post_content) * [only\_hosts](https://script-ware.gitbook.io/cyberdrop-dl/dev/reference/configuration-options/settings/ignore_options#only_hosts) * [skip\_hosts](https://script-ware.gitbook.io/cyberdrop-dl/dev/reference/configuration-options/settings/ignore_options#skip_hosts) * [exclude\_before](https://script-ware.gitbook.io/cyberdrop-dl/dev/reference/configuration-options/settings/ignore_options#exclude_before) * [exclude\_after](https://script-ware.gitbook.io/cyberdrop-dl/dev/reference/configuration-options/settings/ignore_options#exclude_after) sun-brightdesktopmoon sun-brightdesktopmoon --- # Supported Websites | Cyberdrop-DL For a full list of all supported sites, see [supported sites](https://script-ware.gitbook.io/cyberdrop-dl/reference/supported-websites#supported-sites) [hashtag](https://script-ware.gitbook.io/cyberdrop-dl/reference/supported-websites#password-protected-content-hosts) Password Protected Content Hosts ---------------------------------------------------------------------------------------------------------------------------------------------------------- Cyberdrop-DL can download password protected files and folders from these hosts. User must include the password as a query parameter in the input URL, adding `?password=` to it. Example: `https://cyberfile.me/folder/xUGg?password=1234` Domain GoFile Cyberfile Chevereto Sites (`JPG5`, `ImagePond.net`,`ImgLike`) Iceyfile.com Transfer.it Koofr.eu Sites supported by Real-Debrid [hashtag](https://script-ware.gitbook.io/cyberdrop-dl/reference/supported-websites#additional-content-hosts-with-real-debrid) Additional Content Hosts with Real-Debrid ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Cyberdrop-DL has integration with Real-Debrid as download service to support additional hosts. In order to enable Real-Debrid, user must provide their API token inside the `authentication.yaml` file. You can get your API token from this URL (you must be logged in): [https://real-debrid.com/apitokenarrow-up-right](https://real-debrid.com/apitoken) Supported domains via Real-Debrid include `rapidgator`, `4shared.com`, `fikper.com`, `k2s`, `etc`. List of all supported domains can be found here (250+): [https://api.real-debrid.com/rest/1.0/hosts/domainsarrow-up-right](https://api.real-debrid.com/rest/1.0/hosts/domains) circle-info CDL will only use Real-Debrid for unsupported sites. To use it for a site that CDL supports, ex: `mega.nz`, you have to disable the `mega.nz` crawler. See: https://script-ware.gitbook.io/cyberdrop-dl/reference/configuration-options/global-settings/general#disable\_crawlers [hashtag](https://script-ware.gitbook.io/cyberdrop-dl/reference/supported-websites#supported-sites) Supported sites ------------------------------------------------------------------------------------------------------------------------ List of sites supported by cyberdrop-dl-patched as of version 8.10.0 Site Primary URL Supported Domains Supported Paths 4chan https://boards.4chan.org 4chan.\* Board: - `/` Thread: - `/thread` 8Muses https://comics.8muses.com 8muses.\* Album: - `/comics/album/...` AllPornComix https://forum.allporncomix.com allporncomix.\* Attachments: - `/(attachments|data|uploads)/...` Threads: - `/(thread|topic|tema|threads|topics|temas)/` - `/goto/` - `/posts/` **NOTES** - base crawler: Xenforo Anontransfer.com https://anontransfer.com anontransfer.com Direct Link: - `/download-direct.php?dir=&file=` - `/uploads//` File: - `/d/` Folder: - `/f/` AnySex https://anysex.com anysex.\* Album: - `/photos//...` Search: - `/search/...` Video: - `/video//...` ArchiveBate https://www.archivebate.store archivebate.\* Video: - `/watch/` aShemaleTube https://www.ashemaletube.com ashemaletube.\* Model: - `/creators/...` - `/model/...` - `/pornstars/...` Playlist: - `/playlists/...` User: - `/profiles/...` Video: - `/videos/...` Bandcamp https://bandcamp.com bandcamp.\* Album: - `/album/` Song: - `/track/` **NOTES** - You can set 'CDL\_BANDCAMP\_FORMATS' env var to a comma separated list of formats to download (Ordered by preference) \[Default = 'mp3-320,mp3,aac-hi,wav,flac,vorbis,aiff,alas'\] Beeg.com https://beeg.com beeg.com Video: - `/` - `/video/` Bellazon https://www.bellazon.com/main bellazon.\* Attachments: - `/(attachments|data|uploads)/...` Threads: - `/(thread|topic|tema|threads|topics|temas)/` - `/goto/` - `/posts/` **NOTES** - base crawler: Invision BestPrettyGirl https://bestprettygirl.com bestprettygirl.com All Posts: - `/posts/` Category: - `/category/` Date Range: - `...?after=` - `...?before=` - `...?before=` Post: - `//` Tag: - `/tag/` **NOTES** - For `Date Range`, must be a valid iso 8601 date, ex: `2022-12-06`. `Date Range` can be combined with `Category`, `Tag` and `All Posts`. ex: To only download categories from a date range: , `/category/?before=` Box https://www.box.com app.box.com Embedded File or Folder: - `app.box.com/embed/s?sh=` - `app.box.com/embed_widget/s?sh=` File or Folder: - `app.box.com/s?sh=` Bunkr https://bunkr.site bunkr.\* Album: - `/a/` Direct Links File: - `/` - `/f/` Video: - `/v/` Bunkr-Albums.io https://bunkr-albums.io bunkr-albums.io Search: - `/s?search=` BuzzHeavier https://buzzheavier.com buzzheavier.com Direct Links Camwhores.tv https://www.camwhores.tv camwhores.tv Categories: - `/categories/...` Members: - `/members/` Search: - `/search/...` Tags: - `/tags/...` Videos: - `/videos/...` Catbox https://catbox.moe files.catbox.moe litter.catbox.moe Direct Links CelebForum https://celebforum.to celebforum.\* Attachments: - `/(attachments|data|uploads)/...` Threads: - `/(thread|topic|tema|threads|topics|temas)/` - `/goto/` - `/posts/` **NOTES** - base crawler: Xenforo Chevereto ::GENERIC CRAWLER:: Album: - `/a/` - `/a/.` - `/album/` - `/album/.` Category: - `/category/` Direct Links Image: - `/image/` - `/image/.` - `/img/` - `/img/.` Profile: - `/` Video: - `/video/` - `/video/.` - `/videos/` - `/videos/.` CloudflareStream https://cloudflarestream.com cloudflarestream.com videodelivery.net Public Video: - `/` - `//iframe` - `//watch` - `/embed/___.js?video=` Restricted Access Video: - `/` - `//iframe` - `//watch` - `/embed/___.js?video=` Coomer https://coomer.st coomer.party coomer.st coomer.su Direct links: - `/data/...` - `/thumbnail/...` Favorites: - `/account/favorites/posts|artists` - `/favorites?type=post|artist` Individual Post: - `//user//post/` Model: - `//user/` Search: - `/search?q=...` Cyberdrop https://cyberdrop.cr cyberdrop.\* cyberdrop.cr cyberdrop.me cyberdrop.to k1-cd.cdn.gigachad-cdn.ru Album: - `/a/` Direct links: - `/api/file/d/` File: - `/e/` - `/f/` Cyberfile https://cyberfile.me cyberfile.\* Files: - `/` - `//` Public Folders: - `/folder/` - `/folder//` Shared folders: - `/shared/` DesiVideo https://desivideo.net desivideo.net Search: - `/search?s=` Video: - `/videos//...` DirtyShip https://dirtyship.com dirtyship.\* Category: - `/category/...` Gallery: - `/gallery/...` Photo: - `/gallery/.../...` Tag: - `/tag/...` Video: - `/` Discourse ::GENERIC CRAWLER:: Attachments: - `/uploads/...` Topic: - `/t//` - `/t///` **NOTES** - If the URL includes , posts with a number lower that it won't be scraped DoodStream https://doodstream.com all3do.com do7go.com dood.re dood.yt doodcdn. _doodstream._ doodstream.co vidply.com Video: - `/e/` Dropbox https://www.dropbox.com dropbox.\* File: - `/s/...` - `/scl/fi/?rlkey=...` - `/scl/fo//?preview=&rlkey=...` Folder: - `/scl/fo//?rlkey=...` - `/sh/...` E-Hentai https://e-hentai.org e-hentai.\* Album: - `/g/...` File: - `/s/...` E621 https://e621.net e621.net Pools: - `/pools/...` Post: - `/posts/...` Tags: - `/posts?tags=...` eFukt https://efukt.com efukt.com Gif: - `/view.gif.php?id=` Homepage: - `/` Photo: - `/pics/....` Series: - `/series/` Video: - `/...` ePorner https://www.eporner.com eporner.\* Categories: - `/cat/...` Channels: - `/channel/...` Gallery: - `/gallery/...` Photo: - `/photo/...` Pornstar: - `/pornstar/...` Profile: - `/profile/...` Search: - `/search/...` Video: - `/-` - `/embed/` - `/hd-porn/` Erome https://www.erome.com erome.\* Album: - `/a/` Profile: - `/` Search: - `/search?q=` Erome.fan https://erome.fan erome.fan Album: - `/a/` Profile: - `/a/category/` Search: - `/search/` EveriaClub https://everia.club everia.club All Posts: - `/posts/` Category: - `/category/` Date Range: - `...?after=` - `...?before=` - `...?before=` Post: - `//` Tag: - `/tag/` **NOTES** - For `Date Range`, must be a valid iso 8601 date, ex: `2022-12-06`. `Date Range` can be combined with `Category`, `Tag` and `All Posts`. ex: To only download categories from a date range: , `/category/?before=` F95Zone https://f95zone.to f95zone.\* Attachments: - `/(attachments|data|uploads)/...` Threads: - `/(thread|topic|tema|threads|topics|temas)/` - `/goto/` - `/posts/` **NOTES** - base crawler: Xenforo Fapello https://fapello.su fapello.\* Individual Post: - `/.../...` Model: - `/...` Fileditch https://fileditchfiles.me fileditch.\* Direct Links FilesVC https://files.vc files.vc Direct Links Flugel-Anime https://flugelanime.com flugel-anime.\* Any path: - `/...` Forums.plex.tv https://forums.plex.tv forums.plex.tv Attachments: - `/uploads/...` Topic: - `/t//` - `/t///` **NOTES** - If the URL includes , posts with a number lower that it won't be scraped FSIBlog https://fsiblog5.com fsiblog.club fsiblog.com fsiblog1.club fsiblog1.com fsiblog2.club fsiblog2.com fsiblog3.club fsiblog3.com fsiblog4.club fsiblog4.com fsiblog5.club fsiblog5.com Posts: - `//` Search: - `?s=<query>` FuckingFast https://fuckingfast.co fuckingfast.co Direct links: - `/<file_id>` FuXXX https://fuxxx.com fuxxx.com fuxxx.tube Video: - `/embed/<video_id>/...` - `/videos/<video_id>/...` GirlsReleased https://www.girlsreleased.com girlsreleased.\* Model: - `/model/<model_id>/<model_name>` Set: - `/set/<set_id>` Site: - `/site/<site>` GoFile https://gofile.io gofile.\* Direct link: - `/download/<content_id>/<filename>` - `/download/web/<content_id>/<filename>` Folder / File: - `/d/<content_id>` **NOTES** - Use `password` as a query param to download password protected folders - ex: https://gofile.io/d/ABC654?password=1234 GoogleDrive https://drive.google.com docs.google drive.google drive.usercontent.google.com Docs: - `/document/d/<file_id>` Files: - `/file/d/<file_id>` Folders: - `/drive/folders/<folder_id>` - `/embeddedfolderview/<folder_id>` Sheets: - `/spreadsheets/d/<file_id>` Slides: - `/presentation/d/<file_id>` **NOTES** - You can download sheets, slides and docs in a custom format by using it as a query param. ex: https://docs.google.com/document/d/1ZzEzJbemBMPm46O2q5VcGNoPbqDu9AhhUc2djQbvbTY?format=ods Valid Formats: document: - docx (default) - epub - md - odt - pdf - rtf - txt - zip presentation: - odp - pptx (default) spreadsheets: - csv - html - ods - tsv - xslx (default) GooglePhotos https://photos.google.com photos.app.goo.gl photos.google.com Album: - `/share/<album_id>` Photo: - `/album/<album_id>/photo/<photo_id>` **NOTES** - Only downloads 'optimized' images, NOT original quality - Can NOT download videos HClips https://hclips.com hclips.com hclips.tube privatehomeclips.com privatehomeclips.tube Video: - `/embed/<video_id>/...` - `/videos/<video_id>/...` HDZog https://hdzog.com hdzog.com hdzog.tube Video: - `/embed/<video_id>/...` - `/videos/<video_id>/...` Hianime https://hianime.to aniwatch.to aniwatchtv.to hianime.to zoro.to Anime: - `/<name>-<anime_id>` Episode: - `/<name>-<anime_id>?ep=<episode_id>` - `/watch/<name>-<anime_id>?ep=<episode_id>` **NOTES** - You can select the language to be downloaded by using a 'lang' query param. Valid options: 'sub' or 'dub'. Default: 'sub'If the chosen language is not available, CDL will use the first one available Hitomi.la https://hitomi.la hitomi.la Collection: - `/artist/...` - `/character/...` - `/group/...` - `/series/...` - `/tag/...` - `/type/...` Gallery: - `/anime/...` - `/cg/...` - `/doujinshi/...` - `/galleries/...` - `/gamecg/...` - `/imageset/...` - `/manga/...` - `/reader/...` Search: - `/search.html?<query>` HotLeaksTV https://hotleaks.tv hotleaks.tv Model: - `/<model_id>` Video: - `/<model_id>/video/<video_id>` HotLeakVip https://hotleak.vip hotleak.vip Model: - `/<model_id>` Video: - `/<model_id>/video/<video_id>` HotMovs https://hotmovs.com hotmovs.com hotmovs.tube Video: - `/embed/<video_id>/...` - `/videos/<video_id>/...` HotPic https://hotpic.cc 2385290.xyz hotpic.\* Album: - `/album/...` Image: - `/i/...` Iceyfile https://iceyfile.com iceyfile.\* Files: - `/<file_id>` - `/<file_id>/<file_name>` Public Folders: - `/folder/<folder_id>` - `/folder/<folder_id>/<folder_name>` Shared folders: - `/shared/<share_key>` ImageBam https://www.imagebam.com imagebam.\* Gallery: - `/gallery/<id>` Gallery or Image: - `/view/<id>` Image: - `/image/<id>` - `images<x>.imagebam.com/<id>` Thumbnails: - `thumbs<x>.imagebam.com/<id>` ImagePond https://imagepond.net imagepond.net Album: - `/a/<id>` - `/a/<name>.<id>` - `/album/<id>` - `/album/<name>.<id>` Category: - `/category/<name>` Direct Links Image: - `/image/<id>` - `/image/<name>.<id>` - `/img/<id>` - `/img/<name>.<id>` Profile: - `/<user_name>` Video: - `/video/<id>` - `/video/<name>.<id>` - `/videos/<id>` - `/videos/<name>.<id>` ImgBB https://ibb.co ibb.co imgbb.co Album: - `/album/<album_id>` Image: - `/<image_id>` Profile: - `<user_name>.imgbb.co/` ImgBox https://imgbox.com imgbox.\* Album: - `/g/...` Direct Links Image: - `/...` ImgLike https://imglike.com imglike.com Album: - `/a/<id>` - `/a/<name>.<id>` - `/album/<id>` - `/album/<name>.<id>` Category: - `/category/<name>` Direct Links Image: - `/image/<id>` - `/image/<name>.<id>` - `/img/<id>` - `/img/<name>.<id>` Profile: - `/<user_name>` Video: - `/video/<id>` - `/video/<name>.<id>` - `/videos/<id>` - `/videos/<name>.<id>` Imgur https://imgur.com imgur.\* Album: - `/a/<album_id>` Direct links: - `https://i.imgur.com/<image_id>.<ext>` Gallery: - `/gallery/<slug>-<album_id>` Image: - `/<image_id>` Imx.to https://imx.to imx.to Gallery: - `/g/<gallery_id>` Image: - `/i/...` - `/u/i/...` Thumbnail: - `/t/...` - `/u/t/` IncestFlix https://www.incestflix.com incestflix.\* Tag: - `/tag/...` Video: - `/watch/...` InfluencerBitches https://influencerbitches.com influencerbitches.\* Model: - `/model/...` InPorn https://inporn.com inporn.com inporn.tube Video: - `/embed/<video_id>/...` - `/videos/<video_id>/...` JPG5 https://jpg6.su host.church jpeg.pet jpg.church jpg.fish jpg.fishing jpg.homes jpg.pet jpg1.su jpg2.su jpg3.su jpg4.su jpg5.su jpg6.su jpg7.cr selti-delivery.ru Album: - `/a/<id>` - `/a/<name>.<id>` - `/album/<id>` - `/album/<name>.<id>` Category: - `/category/<name>` Direct Links Image: - `/image/<id>` - `/image/<name>.<id>` - `/img/<id>` - `/img/<name>.<id>` Profile: - `/<user_name>` Kemono https://kemono.cr kemono.cr kemono.party kemono.su Direct links: - `/data/...` - `/thumbnail/...` Discord Server: - `/discord/<server_id>` Discord Server Channel: - `/discord/server/<server_id>/<channel_id>#...` Favorites: - `/account/favorites/posts|artists` - `/favorites?type=post|artist` Individual Post: - `/<service>/user/<user_id>/post/<post_id>` Model: - `/<service>/user/<user_id>` Search: - `/search?q=...` Koofr https://koofr.eu k00.fr koofr.eu koofr.net Public Share: - `/links/<content_id>` - `https://k00.fr/<short_id>` LeakedModels https://leakedmodels.com/forum leakedmodels.\* Attachments: - `/(attachments|data|uploads)/...` Threads: - `/(thread|topic|tema|threads|topics|temas)/<thread_name_and_id>` - `/goto/<post_id>` - `/posts/<post_id>` **NOTES** - base crawler: Xenforo LeakedZone https://leakedzone.com leakedzone.\* Model: - `/<model_id>` Video: - `/<model_id>/video/<video_id>` Luscious https://members.luscious.net luscious.\* Album: - `/albums/...` LuxureTV https://luxuretv.com luxuretv.\* Search: - `/searchgate/videos/<search>/...` Video: - `/videos/<name>-<id>.html` Masahub https://masahub.com lol49.com masa49.com masafun.net masahub.com masahub2.com vido99.com Search: - `?s=<query>` Videos: - `/title` Mediafire https://www.mediafire.com mediafire.\* File: - `/file/<quick_key>` - `?<quick_key>` Folder: - `/folder/<folder_key>` Megacloud https://megacloud.blog megacloud.\* Embed v3: - `/embed-2/v3` MegaNz https://mega.nz mega.co.nz mega.io mega.nz File: - `/!#<file_id>!<share_key>` - `/file/<file_id>#<share_key>` - `/folder/<folder_id>#<share_key>/file/<file_id>` Folder: - `/F!#<folder_id>!<share_key>` - `/folder/<folder_id>#<share_key>` Subfolder: - `/folder/<folder_id>#<share_key>/folder/<subfolder_id>` **NOTES** - Downloads can not be resumed. Partial downloads will always be deleted and new downloads will start over MissAV https://missav.ws missav.\* Genres: - `/genres/<genre>` Labels: - `/labels/<label>` Makers: - `/makers/<maker>` Search: - `/search/<search>` Tags: - `/tags/<tag>` Video: - `/...` MixDrop https://mixdrop.sb mixdrop. _mxdrop._ File: - `/e/<file_id>` - `/f/<file_id>` Motherless https://motherless.com motherless.\* Group: - `/g/<group_name>` - `/gi/<image>` - `/gv/<video>` Image: - `/...` User: - `/f/...` - `/u/...` Video: - `pending` **NOTES** - Galleries are NOT supported MyDesi https://lolpol.com fry99.com lolpol.com mydesi.net Search: - `/search/<query>` Videos: - `/title` Nekohouse https://nekohouse.su nekohouse.\* Direct links: - \`/(data nHentai https://nhentai.net nhentai.net Collections: - `artist` - `character` - `favorites` - `group` - `parody` - `search` - `tag` Gallery: - `/g/<gallery_id>` NoodleMagazine https://noodlemagazine.com noodlemagazine.\* Search: - `/video/<search_query` Video: - `/watch/<video_id>` nsfw.xxx https://nsfw.xxx nsfw.xxx Category: - `/category/<name>` Post: - `/post/<id>` Search: - `/search?q=<query>` Subreddit: - `/r/<subreddit>` User: - `/user/<username>` NudoStar https://nudostar.com/forum nudostar.\* Attachments: - `/(attachments|data|uploads)/...` Threads: - `/(thread|topic|tema|threads|topics|temas)/<thread_name_and_id>` - `/goto/<post_id>` - `/posts/<post_id>` **NOTES** - base crawler: Xenforo NudoStarTV https://nudostar.tv nudostar.tv Model: - `/models/...` ok.ru https://ok.ru odnoklassniki.ru ok.ru Channel: - `/profile/<username>/c<channel_id>` - `/video/c<channel_id>` Video: - `/video/<video_id>` OmegaScans https://omegascans.org omegascans.\* Chapter: - `/series/.../...` Direct Links Series: - `/series/...` OneDrive https://onedrive.com 1drv.ms onedrive.live.com Access Link: - `https://onedrive.live.com/?authkey=<KEY>&id=<ID>&cid=<CID>` Share Link (anyone can access): - `https://1drv.ms/b/<KEY>` - `https://1drv.ms/f/<KEY>` - `https://1drv.ms/t/<KEY>` - `https://1drv.ms/u/<KEY>` pCloud https://www.pcloud.com e.pc.cd pc.cd pcloud.\* Public File or folder: - `?code=<share_code>` - `e.pc.cd/<short_code>` - `u.pc.cd/<short_code>` PimpAndHost https://pimpandhost.com pimpandhost.\* Album: - `/album/...` Image: - `/image/...` PimpBunny https://pimpbunny.com pimpbunny.com Album: - `/albums/<album_name>` Category: - `/categories/<category>` Model Albums: - `/albums/models/<model_name>` Models: - `/onlyfans-models/<model_name>` Tag: - `/tags/<tag>` Videos: - `/videos/...` PixelDrain https://pixeldrain.com pd.1drv.eu.org pd.cybar.xyz pixeldra.in pixeldrain.com pixeldrain.net File: - `/api/file/<file_id>` - `/l/<list_id>#item=<file_index>` - `/u/<file_id>` Filesystem: - `/api/filesystem/<path>...` - `/d/<id>` Folder: - `/api/list/<list_id>` - `/l/<list_id>` **NOTES** - text files will not be downloaded but their content will be parsed for URLs PixHost https://pixhost.to pixhost.org pixhost.to Gallery: - `/gallery/<gallery_id>` Image: - `/show/<image_id>` Thumbnail: - `/thumbs/..` Pkmncards https://pkmncards.com pkmncards.\* Card: - `/card/...` Series: - `/series/...` Set: - `/set/...` PMVHaven https://pmvhaven.com pmvhaven.\* Playlist: - `/playlists/...` Search results: - `/search/...` Users: - `/profile/...` - `/users/...` Video: - `/video/...` PornHub https://www.pornhub.com pornhub.\* Album: - `/album/...` Channel: - `/channel/...` Gif: - `/gif/...` Photo: - `/photo/...` Playlist: - `/playlist/...` Profile: - `/model/...` - `/pornstar/...` - `/user/...` Video: - `/embed/<video_id>` - `/view_video.php?viewkey=<video_id>` PornPics https://pornpics.com pornpics.\* Categories: - `/categories/....` Channels: - `/channels/...` Direct Links Gallery: - `/galleries/...` Pornstars: - `/pornstars/...` Search: - `/?q=<query>` Tags: - `/tags/...` Porntrex https://www.porntrex.com porntrex.\* Album: - `/albums/...` Category: - `/categories/...` Model: - `/models/...` Playlist: - `/playlists/...` Search: - `/search/...` Tag: - `/tags/...` User: - `/members/...` Video: - `/video/...` PornZog https://pornzog.com pornzog.\* Video: - `/video/...` PostImg https://postimg.cc postimages.org postimg.cc postimg.org Album: - `/gallery/<album_id>/...` Direct links: - `i.postimg.cc/<image_id>/...` Image: - `/<image_id>/...` Ranoz.gg https://ranoz.gg qiwi.gg ranoz.gg File: - `/d/<file_id>` - `/file/<file_id>` RealBooru https://realbooru.com realbooru.\* File: - `?id=...` Tags: - `?tags=...` Reddit https://www.reddit.com redd.it reddit.\* Direct Links Subreddit:: - `/r/<subreddit>` User: - `/u/<user>` - `/user/<user>` - `/user/<user>/...` RedGifs https://www.redgifs.com redgifs.\* Embeds: - `/ifr/<gif_id>` Gif: - `/watch/<gif_id>` Image: - `/i/<image_id>` User: - `/users/<user>` Rootz.so https://www.rootz.so rootz.so File: - `/d/<file_id>` - `/file/<file_id>` Rule34Vault https://rule34vault.com rule34vault.\* Playlist: - `/playlists/view/...` Post: - `/post/...` Tag: - `/...` Rule34Video https://rule34video.com rule34video.\* Members: - `/members/...` Models: - `/models/...` Search: - `/search/...` Tags: - `/tags/...` Video: - `/video/<id>/<name>` - `/videos/<id>/<name>` Rule34XXX https://rule34.xxx rule34.xxx File: - `?id=...` Tag: - `?tags=...` Rule34XYZ https://rule34.xyz rule34.xyz Playlist: - `/playlists/view/...` Post: - `/post/...` Tag: - `/...` Rumble https://rumble.com rumble.\* Channel: - `/c/<name>` Embed: - `/embed/<video_id>` User: - `/user/<name>` Video: - `<video_id>-<video-title>.html` Safe.Soul https://safe.soul.lol safe.soul.lol Album: - `/a/<album_id>` File: - `/<file_id>` Scrolller https://scrolller.com scrolller.\* Subreddit: - `/r/...` SendNow https://send.now send.now Direct Links SendVid https://sendvid.com sendvid.\* Direct Links Embeds: - `/embed/...` Videos: - `/...` Sex.com https://sex.com sex.\* Shorts Profiles: - `/shorts/<profile>` SocialMediaGirls https://forums.socialmediagirls.com socialmediagirls.\* Attachments: - `/(attachments|data|uploads)/...` Threads: - `/(thread|topic|tema|threads|topics|temas)/<thread_name_and_id>` - `/goto/<post_id>` - `/posts/<post_id>` **NOTES** - base crawler: Xenforo SpankBang https://spankbang.com spankbang.\* Playlist: - `/playlist/<playlist-id>` Video: - `/embed/<video_id>` - `/play/<video_id>` - `/playlist/<playlist-id>-<video_id>` - `/video/<video_id>` Streamable https://streamable.com streamable.\* Video: - `/...` Streamtape https://streamtape.com streamtape.com Player: - `/e/<video_id>` Videos: - `/v/<video_id>` ThisVid https://thisvid.com thisvid.\* Albums: - `/albums/<album_name>` Categories: - `/categories/...` Image: - `/albums/<album_name>/<image_name>` Members: - `/members/<member_id>` Search: - `/search/?q=...` Tags: - `/tags/...` Videos: - `/videos/...` ThotHub https://thothub.to thothub.\* Albums: - `/albums/<album_id>/<album_name>` Videos: - `/videos/...` TikTok https://www.tiktok.com tiktok.\* Photo: - `/@<user>/photo/<photo_id>` User: - `/@<user>` Video: - `/@<user>/video/<video_id>` TitsInTops https://titsintops.com/phpBB2 titsintops.\* Attachments: - `/(attachments|data|uploads)/...` Threads: - `/(thread|topic|tema|threads|topics|temas)/<thread_name_and_id>` - `/goto/<post_id>` - `/posts/<post_id>` **NOTES** - base crawler: Xenforo TNAFlix https://www.tnaflix.com tnaflix.\* Channel: - `/channel/...` Profile: - `/profile/...` Search: - `/search?what=<query>` Video: - `/<category>/<title>/video<video_id>` Tokyomotion https://www.tokyomotion.net tokyomotion.\* Albums: - `/album/<album_id>` - `/user/<user>/albums/` Photo: - `/photo/<photo_id>` - `/user/<user>/favorite/photos` Playlist: - `/user/<user>/favorite/videos` Profiles: - `/user/<user>` Search Results: - `/search?...` Video: - `/video/<video_id>` Toonily https://toonily.com toonily.\* Chapter: - `/serie/<name>/chapter-<chapter-id>` Serie: - `/serie/<name>` Tranny.One https://www.tranny.one tranny.one Album: - `/pics/album/<album_id>` Pornstars: - `/pornstar/<model_id>/<model_name>` Search: - `/search/<search_query>` Video: - `/view/<video_id>` Transfer.it https://transfer.it transfer.it Transfer: - `/t/<transfer_id>` TransFlix https://transflix.net transflix.\* Search: - `/search/?q=...` Video: - `/video/<name>-<video_id>` TubePornClassic https://tubepornclassic.com tubepornclassic.com tubepornclassic.tube Video: - `/embed/<video_id>/...` - `/videos/<video_id>/...` TurboVid https://turbovid.cr saint.to saint2.cr saint2.su turbo.cr turbovid.cr Album: - `/a/<album_id>` Direct links: - `/data/...` Search: - `library?q=<query>` Video: - `/d/<file_id>` - `/embed/<file_id>` - `/v/<file_id>` Twitch https://www.twitch.tv twitch.\* Clip: - `/<user>/clip/<slug>` - `/embed?clip=<slug>` - `https://clips.twitch.tv/<slug>` Collection: - `/collections/<collection_id>` VOD: - `/<user>/v/<vod_id>` - `/video/<vod_id>` - `/videos/<vod_id>` - `?video=<vod_id>` TwitterImages https://twimg.com twimg.\* Photo: - `/...` TWPornStars https://www.twpornstars.com indiantw.com twanal.com twgaymuscle.com twgays.com twlesbian.com twmilf.com twonfans.com twpornstars.com twteens.com twtiktoks.com Photo: - `/...` TXXX https://txxx.com txxx.com txxx.tube videotxxx.com videotxxx.tube Video: - `/embed/<video_id>/...` - `/videos/<video_id>/...` UPornia https://upornia.com upornia.com upornia.tube Video: - `/embed/<video_id>/...` - `/videos/<video_id>/...` ViperGirls https://vipergirls.to viper.click vipergirls.to Threads: - `/goto/<post_id>` - `/posts/<post_id>` - `/threads/<thread_name>` Vipr.im https://vipr.im vipr.im Image: - `/...` Thumbnail: - `/th/...` VJav https://vjav.com vjav.com vjav.tube Video: - `/embed/<video_id>/...` - `/videos/<video_id>/...` Voe.sx https://voe.sx alejandrocenturyoil.com diananatureforeign.com heatherwholeinvolve.com jennifercertaindevelopment.com jilliandescribecompany.com jonathansociallike.com mariatheserepublican.com maxfinishseveral.com nathanfromsubject.com richardsignfish.com robertordercharacter.com sarahnewspaperbeat.com voe.sx Embed: - `/e/video_id` VoyeurHit https://voyeurhit.com voyeurhit.com voyeurhit.tube Video: - `/embed/<video_id>/...` - `/videos/<video_id>/...` VXXX https://vxxx.com vxxx.com vxxx.tube Video: - `/video-<video-id>` WeTransfer https://wetransfer.com we.tl wetransfer.com Direct links: - `download.wetransfer.com/...` Public link: - `wetransfer.com/downloads/<file_id>/<security_hash>` Share Link: - `wetransfer.com/downloads/<file_id>/<recipient_id>/<security_hash>` Short Link: - `we.tl/<short_file_id>` WordPressHTML ::GENERIC CRAWLER:: All Posts: - `/posts/` Category: - `/category/<category_slug>` Date Range: - `...?after=<date>` - `...?before=<date&after=<date>` - `...?before=<date>` Post: - `/<post_slug>/` Tag: - `/tag/<tag_slug>` **NOTES** - For `Date Range`, must be a valid iso 8601 date, ex: `2022-12-06`. `Date Range` can be combined with `Category`, `Tag` and `All Posts`. ex: To only download categories from a date range: , `/category/<category_slug>?before=<date>` WordPressMedia ::GENERIC CRAWLER:: All Posts: - `/posts/` Category: - `/category/<category_slug>` Date Range: - `...?after=<date>` - `...?before=<date&after=<date>` - `...?before=<date>` Post: - `/<post_slug>/` Tag: - `/tag/<tag_slug>` **NOTES** - For `Date Range`, must be a valid iso 8601 date, ex: `2022-12-06`. `Date Range` can be combined with `Category`, `Tag` and `All Posts`. ex: To only download categories from a date range: , `/category/<category_slug>?before=<date>` Xasiat https://www.xasiat.com xasiat.\* Album: - `/albums/<id>/<name>` Images: - `/get_image/...` Videos: - `/videos/<id>/<name>` XBunker https://xbunker.nu xbunker.\* Attachments: - `/(attachments|data|uploads)/...` Threads: - `/(thread|topic|tema|threads|topics|temas)/<thread_name_and_id>` - `/goto/<post_id>` - `/posts/<post_id>` **NOTES** - base crawler: Xenforo XBunkr https://xbunkr.com xbunkr.\* Albums: - `/a/...` Direct Links XGroovy https://xgroovy.com xgroovy.\* Channel: - `/<category>/channels/...` - `/channels/...` Gif: - `/<category>/gifs/<gif_id>/...` - `/gifs/<gif_id>/...` Images: - `/<category>/photos/<photo_id>/...` - `/photos/<photo_id>/...` Pornstar: - `/<category>/pornstars/<pornstar_id>/...` - `/pornstars/<pornstar_id>/...` Search: - `/<category>/search/...` - `/search/...` Tag: - `/<category>/tags/...` - `/tags/...` Video: - `/<category>/videos/<video_id>/...` - `/videos/<video_id>/...` xHamster https://xhamster.com xhamster.\* Creator: - `/creators/<creator_name>` Creator Galleries: - `/creators/<creator_name>/photos` Creator Videos: - `/creators/<creator_name>/exclusive` Gallery: - `/photos/gallery/<gallery_name_or_id>` User: - `/users/<user_name>` - `/users/profiles/<user_name>` User Galleries: - `/users/<user_name>/photos` User Videos: - `/users/<user_name>/videos` Video: - `/videos/<title>` XMilf https://xmilf.com xmilf.com xmilf.tube Video: - `/embed/<video_id>/...` - `/videos/<video_id>/...` xVideos https://www.xvideos.com xv-ru.com xvideos-ar.com xvideos-india.com xvideos.com xvideos.es Account: - `/<channel_name>` - `/amateur|amateur-channels|amateurs|channel|channel-channels|channels|pornstar|pornstar-channels|pornstars|profile|profile-channels|profiles/<name>` Account Photos: - `/<channel_name>#_tabPhotos` - `/<channel_name>/photos/...` - `/amateur|amateur-channels|amateurs|channel|channel-channels|channels|pornstar|pornstar-channels|pornstars|profile|profile-channels|profiles/<name>#_tabPhotos` - `/amateur|amateur-channels|amateurs|channel|channel-channels|channels|pornstar|pornstar-channels|pornstars|profile|profile-channels|profiles/<name>/photos/...` Account Quickies: - `/<channel_name>#quickies` - `/amateur|amateur-channels|amateurs|channel|channel-channels|channels|pornstar|pornstar-channels|pornstars|profile|profile-channels|profiles/<name>#quickies` Account Videos: - `/<channel_name>#_tabVideos` - `/amateur|amateur-channels|amateurs|channel|channel-channels|channels|pornstar|pornstar-channels|pornstars|profile|profile-channels|profiles/<name>#_tabVideos` Video: - `/amateur|amateur-channels|amateurs|channel|channel-channels|channels|pornstar|pornstar-channels|pornstars|profile|profile-channels|profiles#quickies/(a|h|v)/<video_id>` - `/video.<encoded_id>/<title>` - `/video<id>/<title>` XXXBunker https://xxxbunker.com xxxbunker.\* Category: - `/categories/<category>` Search: - `/search/<video_id>` User Favorites: - `/<username>/favoritevideos` Video: - `/<video_id>` YandexDisk https://disk.yandex.com.tr disk.yandex yadi.sk File: - `/d/<folder_id>/<file_name>` - `/i/<file_id>` Folder: - `/d/<folder_id>` **NOTES** - Does NOT support nested folders YouJizz https://www.youjizz.com youjizz.\* Video: - `/videos/<video_name>` - `/videos/embed/<video_id>` [PreviousSimpCity Support Droppedchevron-left](https://script-ware.gitbook.io/cyberdrop-dl/simpcity-support-dropped) [NextConfiguration Optionschevron-right](https://script-ware.gitbook.io/cyberdrop-dl/reference/configuration-options) Last updated 1 month ago * [Password Protected Content Hosts](https://script-ware.gitbook.io/cyberdrop-dl/reference/supported-websites#password-protected-content-hosts) * [Additional Content Hosts with Real-Debrid](https://script-ware.gitbook.io/cyberdrop-dl/reference/supported-websites#additional-content-hosts-with-real-debrid) * [Supported sites](https://script-ware.gitbook.io/cyberdrop-dl/reference/supported-websites#supported-sites) sun-brightdesktopmoon sun-brightdesktopmoon --- # Logs | Cyberdrop-DL Development Docs | Cyberdrop-DL [hashtag](https://script-ware.gitbook.io/cyberdrop-dl/dev/reference/configuration-options/settings/logs#download_error_urls) `download_error_urls` ------------------------------------------------------------------------------------------------------------------------------------------------------- Type Default Restrictions `Path` `Download_Error_URLs.csv` extension will always be overridden to `.csv` Path of the download error log. For relative paths, the final path will be `log_folder` / `download_error_urls` Cyberdrop-DL will output the links it fails to download, the reason and their origin in CSV format. [hashtag](https://script-ware.gitbook.io/cyberdrop-dl/dev/reference/configuration-options/settings/logs#last_forum_post) `last_forum_post` ----------------------------------------------------------------------------------------------------------------------------------------------- Type Default Restrictions `Path` `Last_Scraped_Forum_Posts.csv` extension will always be overridden to `.csv` Path of the forum-post log file. For relative paths, the final path will be `log_folder` / `last_forum_post` Cyberdrop-DL will store the link to the last forum posts it scraped from a given forum thread in this file. [hashtag](https://script-ware.gitbook.io/cyberdrop-dl/dev/reference/configuration-options/settings/logs#log_folder) `log_folder` ------------------------------------------------------------------------------------------------------------------------------------- Type Default `Path` `AppData/Configs/{config}/Logs` The path to the location you want Cyberdrop-DL to store logs in. [hashtag](https://script-ware.gitbook.io/cyberdrop-dl/dev/reference/configuration-options/settings/logs#log_line_width) `log_line_width` --------------------------------------------------------------------------------------------------------------------------------------------- Type Default Restrictions `PositiveInt` `240` `>=50` Line width to use when writing to the main log file [hashtag](https://script-ware.gitbook.io/cyberdrop-dl/dev/reference/configuration-options/settings/logs#logs_expire_after) `logs_expire_after` --------------------------------------------------------------------------------------------------------------------------------------------------- Type Default `timedelta` or `null` `null` With `rotate_logs` enabled, this setting specifies the retention period for log files before they are deleted. * A `timedelta` input is expected to be a valid ISO 8601 timespan, ex: `P10DT2H30M10S` * An `int` input is assumed to be the number of days * A `str` input is expected to be in the format; `<value> <unit>`, ex: `10 days`. * A `null` value means disabled (never delete any logs) circle-exclamation Any `.log` or `.csv` file within `log_folder` will be deleted, even if CDL did not create them circle-info Log files with an absolute path not relative to `log_folder` will never be deleted [hashtag](https://script-ware.gitbook.io/cyberdrop-dl/dev/reference/configuration-options/settings/logs#main_log) `main_log` --------------------------------------------------------------------------------------------------------------------------------- Type Default Restrictions `Path` `downloader.log` extension will always be overridden to `.log` Path of main log file. For relative paths, the final path will be `log_folder` / `main_log` [hashtag](https://script-ware.gitbook.io/cyberdrop-dl/dev/reference/configuration-options/settings/logs#rotate_logs) `rotate_logs` --------------------------------------------------------------------------------------------------------------------------------------- Type Default `bool` `false` If enabled, Cyberdrop-DL will add the current date and time as a suffix to each log file, in the format `YYMMDD_HHMMSS` Every log file will be created inside a sub folder with the current date This will prevent overriding old log files [hashtag](https://script-ware.gitbook.io/cyberdrop-dl/dev/reference/configuration-options/settings/logs#scrape_error_urls) `scrape_error_urls` --------------------------------------------------------------------------------------------------------------------------------------------------- Type Default Restrictions `Path` `Scrape_Error_URLs.csv` extension will always be overridden to `.csv` What you want Cyberdrop-DL to call the scrape error log. For relative paths, the final path will be `log_folder` / `scrape_error_urls` Cyberdrop-DL will output the links it fails to scrape, the reason and their origin in CSV format. [hashtag](https://script-ware.gitbook.io/cyberdrop-dl/dev/reference/configuration-options/settings/logs#unsupported_urls) `unsupported_urls` ------------------------------------------------------------------------------------------------------------------------------------------------- Type Default Restrictions `Path` `Unsupported_URLs.csv` extension will always be overridden to `.csv` Path of the unsupported log file. For relative paths, the final path will be `log_folder` / `unsupported_urls` Cyberdrop-DL will output links it can't download to this file. [hashtag](https://script-ware.gitbook.io/cyberdrop-dl/dev/reference/configuration-options/settings/logs#webhook) `webhook` ------------------------------------------------------------------------------------------------------------------------------- Type Default Restrictions `AppriseURL` or `null` `null` The scheme of the URL must be `http` or `https` The URL of a webhook that you want to send download stats to (Ex: Discord). You can add the optional tag `attach_logs=` as a prefix to include a copy of the main log as an attachment. Example: > `attach_logs=https://discord.com/api/webhooks/webhook_id/webhook_token` [PreviousIgnore Optionschevron-left](https://script-ware.gitbook.io/cyberdrop-dl/dev/reference/configuration-options/settings/ignore_options) [NextMedia Duration Limitschevron-right](https://script-ware.gitbook.io/cyberdrop-dl/dev/reference/configuration-options/settings/media_duration_limits) Last updated 8 months ago * [download\_error\_urls](https://script-ware.gitbook.io/cyberdrop-dl/dev/reference/configuration-options/settings/logs#download_error_urls) * [last\_forum\_post](https://script-ware.gitbook.io/cyberdrop-dl/dev/reference/configuration-options/settings/logs#last_forum_post) * [log\_folder](https://script-ware.gitbook.io/cyberdrop-dl/dev/reference/configuration-options/settings/logs#log_folder) * [log\_line\_width](https://script-ware.gitbook.io/cyberdrop-dl/dev/reference/configuration-options/settings/logs#log_line_width) * [logs\_expire\_after](https://script-ware.gitbook.io/cyberdrop-dl/dev/reference/configuration-options/settings/logs#logs_expire_after) * [main\_log](https://script-ware.gitbook.io/cyberdrop-dl/dev/reference/configuration-options/settings/logs#main_log) * [rotate\_logs](https://script-ware.gitbook.io/cyberdrop-dl/dev/reference/configuration-options/settings/logs#rotate_logs) * [scrape\_error\_urls](https://script-ware.gitbook.io/cyberdrop-dl/dev/reference/configuration-options/settings/logs#scrape_error_urls) * [unsupported\_urls](https://script-ware.gitbook.io/cyberdrop-dl/dev/reference/configuration-options/settings/logs#unsupported_urls) * [webhook](https://script-ware.gitbook.io/cyberdrop-dl/dev/reference/configuration-options/settings/logs#webhook) sun-brightdesktopmoon sun-brightdesktopmoon --- # General | Cyberdrop-DL Development Docs | Cyberdrop-DL [hashtag](https://script-ware.gitbook.io/cyberdrop-dl/dev/reference/configuration-options/global-settings/general#disable_crawlers) `disable_crawlers` ----------------------------------------------------------------------------------------------------------------------------------------------------------- Type Default Additional Info `list[NonEmptyStr]` `[]` This is an [`AdditiveArg`](https://script-ware.gitbook.io/cyberdrop-dl/dev/reference/configuration-options/special_setting_types#additiveargs) You can supply a list of crawlers to disable for the current run. This will make CDL completely ignore the crawler, as if the site was not supported. However, links from the site will still be processed by Real-Debrid (if enabled), Jdownloader (If enabled) and the Generic crawler (If enabled), in that order. The list should be valid crawlers names. The name of the crawler is the name of the primary site they support. ex: `4Chan`, `Bunkrr`, `Dropbox` Crawlers names correspond to the column `site` in the [supported sites pagearrow-up-right](https://script-ware.gitbook.io/cyberdrop-dl/reference/supported-websites#supported-sites) . [hashtag](https://script-ware.gitbook.io/cyberdrop-dl/dev/reference/configuration-options/global-settings/general#enable_generic_crawler) `enable_generic_crawler` ----------------------------------------------------------------------------------------------------------------------------------------------------------------------- Type Default `bool` `false` CDL has a generic crawler that will try to download from unsupported sites. Setting this to `true` will enable it. circle-info CDL will always try to download from unsupported URLs if the last part of the URL has a known file extension. ex: `.jpg` [hashtag](https://script-ware.gitbook.io/cyberdrop-dl/dev/reference/configuration-options/global-settings/general#flaresolverr) `flaresolverr` --------------------------------------------------------------------------------------------------------------------------------------------------- Type Default `HttpURL` or `null` `null` [FlareSolverrarrow-up-right](https://github.com/FlareSolverr/FlareSolverr) is a proxy server to bypass Cloudflare and `DDoS-Guard` protection. The provided value must be a valid `http` URL of an existing flaresolverr instance. Ex: `http://192.168.1.44:4000` circle-info `0.0.0.0` is NOT a valid IP address. To set up a flaresolverr instance running on the same machine as CDL, use `127.0.0.1` as the IP circle-exclamation This wiki does not cover flaresolverr setup process. If you need help, refer to their documentation. Please do not open issues related to flaresolverr or `DDoS-Guard`. See: [How to extract cookies (DDoSGuard or login errors) #839arrow-up-right](https://github.com/jbsparrow/CyberDropDownloader/discussions/839) for alternatives using cookies [hashtag](https://script-ware.gitbook.io/cyberdrop-dl/dev/reference/configuration-options/global-settings/general#max_file_name_length) `max_file_name_length` ------------------------------------------------------------------------------------------------------------------------------------------------------------------- Type Default `PositiveInt` `95` This is the maximum number of characters a filename should have. CDL will truncate filenames longer that this. [hashtag](https://script-ware.gitbook.io/cyberdrop-dl/dev/reference/configuration-options/global-settings/general#max_folder_name_length) `max_folder_name_length` ----------------------------------------------------------------------------------------------------------------------------------------------------------------------- Type Default `PositiveInt` `95` This is the maximum number of characters a folder should have. CDL will truncate folders longer that this. [hashtag](https://script-ware.gitbook.io/cyberdrop-dl/dev/reference/configuration-options/global-settings/general#proxy) `proxy` ------------------------------------------------------------------------------------------------------------------------------------- Type Default `HttpURL` or `null` `null` The proxy you want CDL to use. Only `http` proxies are supported. Ex: `https://user:password@ip:port` [hashtag](https://script-ware.gitbook.io/cyberdrop-dl/dev/reference/configuration-options/global-settings/general#required_free_space) `required_free_space` ----------------------------------------------------------------------------------------------------------------------------------------------------------------- Type Default Restrictions `ByteSize` `5GB` `>=512MB` This is the minimum amount of free space require to start new downloads. circle-info If you set a value lower than `512MB`, CDL will override it with `512MB` [hashtag](https://script-ware.gitbook.io/cyberdrop-dl/dev/reference/configuration-options/global-settings/general#ssl_context) `ssl_context` ------------------------------------------------------------------------------------------------------------------------------------------------- Type Default `NonEmptyStr` or None `truststore+certifi` Context that will used to verify SSL connections. Valid values are: * `truststore`: Will use certificates already included with the OS * `certifi`: Will use certificates bundled with the `certifi` version available at the release of the current CDL version * `truststore+certifi`: Will use certificates already included with the OS, with a fallback to `certifi` for missing certificates * `None`: Will completely disable SSL verification, allowing insecure connections. Setting this to `None` will allow the program to connect to websites without SSL encryption (insecurely). triangle-exclamation Sensitive data may be exposed using an insecure connection. For your safety, is recommended to always use a secure HTTPS connection. [hashtag](https://script-ware.gitbook.io/cyberdrop-dl/dev/reference/configuration-options/global-settings/general#user_agent) `user_agent` ----------------------------------------------------------------------------------------------------------------------------------------------- Type Default `NonEmptyStr` `Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:142.0) Gecko/20100101 Firefox/142.0` The user agent is the signature of your browser. Some sites use it to identify if the requests come from a human or a robot. You can google "what is my user agent" to get yours. circle-info If you use flaresolverr, this value MUST match with flaresolverr's user agent. Otherwise, flaresolverr cookies won't work [PreviousGlobal Settingschevron-left](https://script-ware.gitbook.io/cyberdrop-dl/dev/reference/configuration-options/global-settings) [NextRate Limiting Optionschevron-right](https://script-ware.gitbook.io/cyberdrop-dl/dev/reference/configuration-options/global-settings/rate-limiting-options) Last updated 5 months ago * [disable\_crawlers](https://script-ware.gitbook.io/cyberdrop-dl/dev/reference/configuration-options/global-settings/general#disable_crawlers) * [enable\_generic\_crawler](https://script-ware.gitbook.io/cyberdrop-dl/dev/reference/configuration-options/global-settings/general#enable_generic_crawler) * [flaresolverr](https://script-ware.gitbook.io/cyberdrop-dl/dev/reference/configuration-options/global-settings/general#flaresolverr) * [max\_file\_name\_length](https://script-ware.gitbook.io/cyberdrop-dl/dev/reference/configuration-options/global-settings/general#max_file_name_length) * [max\_folder\_name\_length](https://script-ware.gitbook.io/cyberdrop-dl/dev/reference/configuration-options/global-settings/general#max_folder_name_length) * [proxy](https://script-ware.gitbook.io/cyberdrop-dl/dev/reference/configuration-options/global-settings/general#proxy) * [required\_free\_space](https://script-ware.gitbook.io/cyberdrop-dl/dev/reference/configuration-options/global-settings/general#required_free_space) * [ssl\_context](https://script-ware.gitbook.io/cyberdrop-dl/dev/reference/configuration-options/global-settings/general#ssl_context) * [user\_agent](https://script-ware.gitbook.io/cyberdrop-dl/dev/reference/configuration-options/global-settings/general#user_agent) sun-brightdesktopmoon sun-brightdesktopmoon --- # Rate Limiting Options | Cyberdrop-DL Development Docs | Cyberdrop-DL [hashtag](https://script-ware.gitbook.io/cyberdrop-dl/dev/reference/configuration-options/global-settings/rate-limiting-options#connection_timeout) `connection_timeout` ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Type Default `PositiveInt` `15` The number of seconds to wait while connecting to a website before timing out circle-info This value will also be used for Flaresolverr (if enabled) as the max number of seconds to solve a CAPTCHA challenge [hashtag](https://script-ware.gitbook.io/cyberdrop-dl/dev/reference/configuration-options/global-settings/rate-limiting-options#download_attempts) `download_attempts` --------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Type Default `PositiveInt` `2` The number of download attempts per file. Some conditions are never retried (such as a `404` HTTP status) [hashtag](https://script-ware.gitbook.io/cyberdrop-dl/dev/reference/configuration-options/global-settings/rate-limiting-options#download_delay) `download_delay` --------------------------------------------------------------------------------------------------------------------------------------------------------------------- Type Default `NonNegativeFloat` `0.0` This is the number of seconds to wait between downloads to the same domain. Some domains have internal limits set by the program, which can not be modified: * `bunkr`: 0.5 * `coomer`: 0.5 * `kemono`: 0.5 * `nhentai.net`: 1 [hashtag](https://script-ware.gitbook.io/cyberdrop-dl/dev/reference/configuration-options/global-settings/rate-limiting-options#download_speed_limit) `download_speed_limit` --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Type Default `ByteSize` `0` circle-info This setting specifies speed and it's interpreted as `<value> / second`. ex: `8MB` means `8MB/s` This is the max rate of downloading in bytes (per second) for all downloads combined. Set to `0` to disable [hashtag](https://script-ware.gitbook.io/cyberdrop-dl/dev/reference/configuration-options/global-settings/rate-limiting-options#file_host_cache_expire_after) `file_host_cache_expire_after` ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Type Default `timedelta`, `str` or `int` `7 days` Cyberdrop-DL caches the requests made to any website. This setting controls how long responses to file host websites are stored before expiring. * A `timedelta` input is expected to be a valid ISO 8601 timespan, ex: `P10DT2H30M10S` * An `int` input is assumed to be the number of days * A `str` input is expected to be in the format; `<value> <unit>`, ex: `10 days`. ### [hashtag](https://script-ware.gitbook.io/cyberdrop-dl/dev/reference/configuration-options/global-settings/rate-limiting-options#valid-str-units) Valid `str` units * `year(s)` * `month(s)` * `week(s)` * `day(s)` * `hour(s)` * `minute(s)` * `second(s)` * `millisecond(s)` * `microsecond(s)` circle-info You can set the value to `0` to disable caching [hashtag](https://script-ware.gitbook.io/cyberdrop-dl/dev/reference/configuration-options/global-settings/rate-limiting-options#forum_cache_expire_after) `forum_cache_expire_after` ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Type Default `timedelta`, `str` or `int` `4 weeks` Same as `file_host_cache_expire_after` but applied to forums requests. [hashtag](https://script-ware.gitbook.io/cyberdrop-dl/dev/reference/configuration-options/global-settings/rate-limiting-options#jitter) `jitter` ----------------------------------------------------------------------------------------------------------------------------------------------------- Type Default `NonNegativeFloat` `0.0` Additional number of seconds to wait in between downloads. CDL will wait an additional random number of seconds in between 0 and the `jitter` value. [hashtag](https://script-ware.gitbook.io/cyberdrop-dl/dev/reference/configuration-options/global-settings/rate-limiting-options#max_simultaneous_downloads) `max_simultaneous_downloads` --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Type Default `PositiveInt` `15` This is the maximum number of files that can be downloaded simultaneously. [hashtag](https://script-ware.gitbook.io/cyberdrop-dl/dev/reference/configuration-options/global-settings/rate-limiting-options#max_simultaneous_downloads_per_domain) `max_simultaneous_downloads_per_domain` ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Type Default `PositiveInt` `5` This is the maximum number of files that can be downloaded from a single domain simultaneously. Some domains have internal limits set by the program, which can not be modified: * `bunkr`: 1 per unique server * `cyberfile`: 1 * `noodlemagazine`: 2 * `4chan`: 1 * `pixeldrain`: 2 * `xxxbunker`: 2 [hashtag](https://script-ware.gitbook.io/cyberdrop-dl/dev/reference/configuration-options/global-settings/rate-limiting-options#rate_limit) `rate_limit` ------------------------------------------------------------------------------------------------------------------------------------------------------------- Type Default `PositiveFloat` `50.0` circle-info This setting specifies speed and it's interpreted as `<value> / second`. ex: `50` means `50 requests / second` This is the maximum number of requests that can be made by the program per second. [hashtag](https://script-ware.gitbook.io/cyberdrop-dl/dev/reference/configuration-options/global-settings/rate-limiting-options#read_timeout) `read_timeout` ----------------------------------------------------------------------------------------------------------------------------------------------------------------- Type Default `PositiveFloat` or `null` `300.0` The number of seconds to wait while reading data from a website before timing out. A `null` value will make CDL keep the socket connection open indefinitely, even if the server is not sending data anymore [PreviousGeneralchevron-left](https://script-ware.gitbook.io/cyberdrop-dl/dev/reference/configuration-options/global-settings/general) [NextUI Optionschevron-right](https://script-ware.gitbook.io/cyberdrop-dl/dev/reference/configuration-options/global-settings/ui-options) Last updated 1 month ago * [connection\_timeout](https://script-ware.gitbook.io/cyberdrop-dl/dev/reference/configuration-options/global-settings/rate-limiting-options#connection_timeout) * [download\_attempts](https://script-ware.gitbook.io/cyberdrop-dl/dev/reference/configuration-options/global-settings/rate-limiting-options#download_attempts) * [download\_delay](https://script-ware.gitbook.io/cyberdrop-dl/dev/reference/configuration-options/global-settings/rate-limiting-options#download_delay) * [download\_speed\_limit](https://script-ware.gitbook.io/cyberdrop-dl/dev/reference/configuration-options/global-settings/rate-limiting-options#download_speed_limit) * [file\_host\_cache\_expire\_after](https://script-ware.gitbook.io/cyberdrop-dl/dev/reference/configuration-options/global-settings/rate-limiting-options#file_host_cache_expire_after) * [Valid str units](https://script-ware.gitbook.io/cyberdrop-dl/dev/reference/configuration-options/global-settings/rate-limiting-options#valid-str-units) * [forum\_cache\_expire\_after](https://script-ware.gitbook.io/cyberdrop-dl/dev/reference/configuration-options/global-settings/rate-limiting-options#forum_cache_expire_after) * [jitter](https://script-ware.gitbook.io/cyberdrop-dl/dev/reference/configuration-options/global-settings/rate-limiting-options#jitter) * [max\_simultaneous\_downloads](https://script-ware.gitbook.io/cyberdrop-dl/dev/reference/configuration-options/global-settings/rate-limiting-options#max_simultaneous_downloads) * [max\_simultaneous\_downloads\_per\_domain](https://script-ware.gitbook.io/cyberdrop-dl/dev/reference/configuration-options/global-settings/rate-limiting-options#max_simultaneous_downloads_per_domain) * [rate\_limit](https://script-ware.gitbook.io/cyberdrop-dl/dev/reference/configuration-options/global-settings/rate-limiting-options#rate_limit) * [read\_timeout](https://script-ware.gitbook.io/cyberdrop-dl/dev/reference/configuration-options/global-settings/rate-limiting-options#read_timeout) sun-brightdesktopmoon sun-brightdesktopmoon --- # Sorting | Cyberdrop-DL Development Docs | Cyberdrop-DL Cyberdrop-DL has a file sorter built in, but it's disabled by default You can use the field names below to create a custom path format. You can also use essentially none of them and have a hard coded path. However, `filename` and `ext` must always be used. Common fields for sorting format options (supported for `audio`, `videos`, `images` and `other`): > `base_dir`: the name of highest level folder inside `scan_folder`. This normally is the model name or the thread name > > `ext`: the file extension (suffix) > > `file_date`: the file date. This is a datetime object, which means it accepts a custom format spec ex: `{file_date:%Y-%m}` > > `file_date_iso`: the file date as an iso 8601 string (`%Y-%m-%d`) > > `file_date_us`: the file date in the US format (`%Y-%d-%m`) > > `filename`: the file's name (stem) > > `parent_dir`: the name of the folder where the file is located at (its parent folder). This is normally the album name for photos or the post name for forums/reddit if `separate_post` is enabled > > `sort_dir`: the same path as `sort_folder` from the download options [hashtag](https://script-ware.gitbook.io/cyberdrop-dl/dev/reference/configuration-options/settings/sorting#scan_folder) `scan_folder` ------------------------------------------------------------------------------------------------------------------------------------------ Type Default `Path` or `null` `null` Sets the starting point for the file scan Each direct child of the `scan_folder` is recursively scanned, and files are moved based on your settings. If this is set to `null` (the default), the value of `download_dir` from the download options is used. [hashtag](https://script-ware.gitbook.io/cyberdrop-dl/dev/reference/configuration-options/settings/sorting#sort_downloads) `sort_downloads` ------------------------------------------------------------------------------------------------------------------------------------------------ Type Default `bool` `false` Setting this to `true` will allow Cyberdrop-DL to sort downloads after a run is complete. [hashtag](https://script-ware.gitbook.io/cyberdrop-dl/dev/reference/configuration-options/settings/sorting#sort_folder) `sort_folder` ------------------------------------------------------------------------------------------------------------------------------------------ Type Default `Path` `Downloads/Cyberdrop-DL Sorted Downloads` This is the path to the folder you'd like sorted downloads to be stored in. circle-exclamation Setting `sort_folder` to the same value as `scan_folder` is not supported and will lead to expected results [hashtag](https://script-ware.gitbook.io/cyberdrop-dl/dev/reference/configuration-options/settings/sorting#sort_incrementer_format) `sort_incrementer_format` ------------------------------------------------------------------------------------------------------------------------------------------------------------------ Type Default `NonEmptyStr` `({i})` When naming collisions happen, Cyberdrop-DL will rename files automatically > `image.jpg` -> `image (1).jpg`. You can modify the format as needed, but it must include `{i}` to specify where the auto-increment value should be placed [hashtag](https://script-ware.gitbook.io/cyberdrop-dl/dev/reference/configuration-options/settings/sorting#sorted_audio) `sorted_audio` -------------------------------------------------------------------------------------------------------------------------------------------- Type Default `NonEmptyStr` or `null` `{sort_dir}/{base_dir}/Audio/{filename}{ext}` This is the format for the directory structure and naming scheme for audio files. Set to `null` to skip sorting audio files In addition to the common sorting format fields, this option supports: > `bitrate`: file bit rate. This is an `int` > > `duration`: audio total runtime in seconds. This is an `int` > > `length`: same as `duration` > > `sample_rate`: audio sample rate. This is an `int` [hashtag](https://script-ware.gitbook.io/cyberdrop-dl/dev/reference/configuration-options/settings/sorting#sorted_image) `sorted_image` -------------------------------------------------------------------------------------------------------------------------------------------- Type Default `NonEmptyStr` or `null` `{sort_dir}/{base_dir}/Images/{filename}{ext}` This is the format for the directory structure and naming scheme for image files. Set to `null` to skip sorting image files In addition to the common sorting format fields, this option supports: > `height`: vertical pixel count. This is an `int` > > `width`: horizontal pixel count. This is an `int` > > `resolution`: `width`x`height` ex. 1080x1920. This is a `str` [hashtag](https://script-ware.gitbook.io/cyberdrop-dl/dev/reference/configuration-options/settings/sorting#sorted_video) `sorted_video` -------------------------------------------------------------------------------------------------------------------------------------------- Type Default `NonEmptyStr` or `null` `{sort_dir}/{base_dir}/Video/{filename}{ext}` This is the format for the directory structure and naming scheme for video files. Set to `null` to skip sorting video files In addition to the common sorting format fields, this option supports: > `codec`: ex. h264. This is a `str`. It could potentially be `None` for some files > > `duration`: video total runtime in seconds. This is an `int` > > `fps`: ex. `24`. This represents a number but is a `str`. It could potentially be `None` for some files > > `length`: same as `duration` > > `height`: vertical pixel count. This is an `int` > > `width`: horizontal pixel count.This is an `int` > > `resolution`: `width`x`height` ex. 1080x1920. This is a `str` [hashtag](https://script-ware.gitbook.io/cyberdrop-dl/dev/reference/configuration-options/settings/sorting#sorted_other) `sorted_other` -------------------------------------------------------------------------------------------------------------------------------------------- Type Default `NonEmptyStr` or `null` `{sort_dir}/{base_dir}/Other/{filename}{ext}` This is the format for the directory structure and naming scheme for other files. Set to `null` to skip sorting other files [hashtag](https://script-ware.gitbook.io/cyberdrop-dl/dev/reference/configuration-options/settings/sorting#group-urls) Group URLs -------------------------------------------------------------------------------------------------------------------------------------- It is possible to treat a list of URLs as a group, allowing them to be downloaded to a single folder. To define a group, put a title above the URLs you want to be in the group, using the format: `--- <group_name>` or `=== <group_name>`. To define the end of a group, add a new group with no name. (`---` or `===`) Here is an example URL file with two groups: Those downloads would be sorted as follows: Loose Files Group 1 Group 2 file1.jpg file2.jpg file5.jpg file4.jpg file3.jpg file6.jpg file7.jpg [PreviousRuntime Optionschevron-left](https://script-ware.gitbook.io/cyberdrop-dl/dev/reference/configuration-options/settings/runtime_options) [NextGlobal Settingschevron-right](https://script-ware.gitbook.io/cyberdrop-dl/dev/reference/configuration-options/global-settings) Last updated 5 months ago * [scan\_folder](https://script-ware.gitbook.io/cyberdrop-dl/dev/reference/configuration-options/settings/sorting#scan_folder) * [sort\_downloads](https://script-ware.gitbook.io/cyberdrop-dl/dev/reference/configuration-options/settings/sorting#sort_downloads) * [sort\_folder](https://script-ware.gitbook.io/cyberdrop-dl/dev/reference/configuration-options/settings/sorting#sort_folder) * [sort\_incrementer\_format](https://script-ware.gitbook.io/cyberdrop-dl/dev/reference/configuration-options/settings/sorting#sort_incrementer_format) * [sorted\_audio](https://script-ware.gitbook.io/cyberdrop-dl/dev/reference/configuration-options/settings/sorting#sorted_audio) * [sorted\_image](https://script-ware.gitbook.io/cyberdrop-dl/dev/reference/configuration-options/settings/sorting#sorted_image) * [sorted\_video](https://script-ware.gitbook.io/cyberdrop-dl/dev/reference/configuration-options/settings/sorting#sorted_video) * [sorted\_other](https://script-ware.gitbook.io/cyberdrop-dl/dev/reference/configuration-options/settings/sorting#sorted_other) * [Group URLs](https://script-ware.gitbook.io/cyberdrop-dl/dev/reference/configuration-options/settings/sorting#group-urls) sun-brightdesktopmoon Copy https://example.com/file1.jpg === Group 1 https://example.com/file2.jpg https://example.com/file3.jpg === https://example.com/file4.jpg --- Group 2 https://example.com/file5.jpg https://example.com/file6.jpg === https://example.com/file7.jpg sun-brightdesktopmoon --- # Runtime Options | Cyberdrop-DL Development Docs | Cyberdrop-DL These are higher level options that effect the overarching functions of the program. [hashtag](https://script-ware.gitbook.io/cyberdrop-dl/dev/reference/configuration-options/settings/runtime_options#console_log_level) `console_log_level` -------------------------------------------------------------------------------------------------------------------------------------------------------------- Type Default `NonNegativeInt` `100` Same as `log_level` but it controls which messages are shown on the console. circle-info This option is ignored unless CDL is running with `--ui DISABLED` [hashtag](https://script-ware.gitbook.io/cyberdrop-dl/dev/reference/configuration-options/settings/runtime_options#deep_scrape) `deep_scrape` -------------------------------------------------------------------------------------------------------------------------------------------------- Type Default `bool` `false` Cyberdrop-DL uses a some tricks to try to reduce the number of requests it needs to make while scraping a site. However, this may cause a few links to be skipped. Use `--deep-scrape` to disable this functionality and always make a new request. This setting is temporary and will always reset to `false` after each run circle-exclamation Use this option only when absolutely necessary, as it will significantly increase the number of requests being made. For example, scraping an album normally takes one single request. With `--deep-scrape`, CDL will make `n` requests per album, where `n` is the total number of items in the album [hashtag](https://script-ware.gitbook.io/cyberdrop-dl/dev/reference/configuration-options/settings/runtime_options#delete_partial_files) `delete_partial_files` -------------------------------------------------------------------------------------------------------------------------------------------------------------------- Type Default `bool` `false` Files downloaded by CDL have a `.part` extension (`.cdl_hls` for HLS segments). CDL only changes the extension to the original one after a successful download. This allows CDL to resume downloads on subsequent runs. Setting this to `true` will delete any `.part` and `.cdl_hls` files in the download folder. [hashtag](https://script-ware.gitbook.io/cyberdrop-dl/dev/reference/configuration-options/settings/runtime_options#ignore_history) `ignore_history` -------------------------------------------------------------------------------------------------------------------------------------------------------- Type Default `bool` `false` By default, the program tracks your downloads in a database to prevent downloading the same files multiple times, to save time and reduce strain on the servers you're downloading from. Setting this to `true` will cause the program to ignore the database, and will allow you to re-download files. [hashtag](https://script-ware.gitbook.io/cyberdrop-dl/dev/reference/configuration-options/settings/runtime_options#jdownloader_autostart) `jdownloader_autostart` ---------------------------------------------------------------------------------------------------------------------------------------------------------------------- Type Default `bool` `false` Setting this to `true` will make jdownloader start downloads as soon as they are sent. circle-info This option has no effect unless `send_unsupported_to_jdownloader` is `true` [hashtag](https://script-ware.gitbook.io/cyberdrop-dl/dev/reference/configuration-options/settings/runtime_options#jdownloader_download_dir) `jdownloader_download_dir` ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Type Default `Path` or `null` `null` The `download_dir` jdownloader will use. A `null` value (the default) will make jdownloader use the same `download_dir` as Cyberdrop-DL. Use this option as path mapping when jdownloader is running on a different host / docker. circle-info This option has no effect unless `send_unsupported_to_jdownloader` is `true` [hashtag](https://script-ware.gitbook.io/cyberdrop-dl/dev/reference/configuration-options/settings/runtime_options#jdownloader_whitelist) `jdownloader_whitelist` ---------------------------------------------------------------------------------------------------------------------------------------------------------------------- Type Default `list[NonEmptyStr]` `[]` List of domain names. An unsupported URL will only be sent to jdownloader if its host is found on the list. An empty whitelist (the default) will disable this functionality, sending any unsupported URL to jdownloader. circle-info This option has no effect unless `send_unsupported_to_jdownloader` is `true` [hashtag](https://script-ware.gitbook.io/cyberdrop-dl/dev/reference/configuration-options/settings/runtime_options#log_level) `log_level` ---------------------------------------------------------------------------------------------------------------------------------------------- Type Default `NonNegativeInt` `10` Defines the logging level for messages, according to [Python logging levelsarrow-up-right](https://docs.python.org/3/library/logging.html#levels) . Only messages with a level equal or greater than this setting will be logged, with the exception of `NOTSET` which will disable all logging messages. Setting a value above 50 will also disable logging entirely. Level Value Description `NOTSET` 0 Disables all log messages `DEBUG` 10 Offers detailed information, typically useful to troubleshoot problems `INFO` 20 Provides general information about scrape and download progress `WARNING` 30 Potential issues or something that might need attention (e.g. `Login wasn't provided for <FORUM>`) `ERROR` 40 Serious problem preventing Cyberdrop-DL to execute some function `CRITICAL` 50 Fatal error that causes Cyberdrop-DL to exit immediately circle-info Using anything other that `DEBUG` makes troubleshooting issues harder. Practically speaking, you should _only_ change this if you expect CDL to run for an extended period (with a large number of input URLs) to minimize the log files sizes circle-info There is one more verbose level of logs: `DEVELOPER` It includes pager logs, posts logs, individual requests logs (their URLs and their response code), requests cache hits and misses, file system locks logs and local variable values from within a traceback. It's not exposed as a config option because it is too verbose. You can set the environment variable `CDL_DEBUG_LOG_FOLDER` to any valid folder and CDL will create an additional log file with the level set to `DEVELOPER` inside it. circle-exclamation As the name suggests, the `DEVELOPER` log level is only meant for developers or, more generally, people that want and have the know how to troubleshoot CDL issues on their own. You should never share those logs with _anyone_ as it could potentially have raw information in the tracebacks like cookies, auth credentials, IPs, etc. [hashtag](https://script-ware.gitbook.io/cyberdrop-dl/dev/reference/configuration-options/settings/runtime_options#send_unsupported_to_jdownloader) `send_unsupported_to_jdownloader` ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ Type Default `bool` `false` Cyberdrop-DL has integration with jdownloader. This will allow you to download URLs that Cyberdrop-DL finds but do not support. However, this setting is disabled by default. Setting this to `true`, will send unsupported links over. [hashtag](https://script-ware.gitbook.io/cyberdrop-dl/dev/reference/configuration-options/settings/runtime_options#skip_check_for_empty_folders) `skip_check_for_empty_folders` ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ Type Default `bool` `false` After a run is complete, the program will do a check (and remove) any empty files and folders in the download and scan folder. Setting this to `true` will disable this functionality. [hashtag](https://script-ware.gitbook.io/cyberdrop-dl/dev/reference/configuration-options/settings/runtime_options#skip_check_for_partial_files) `skip_check_for_partial_files` ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ Type Default `bool` `false` After a run is complete, the program will do a check to see if any partially downloaded files remain in the downloads folder and will notify you of them. Setting this to `true` will skip this check. [hashtag](https://script-ware.gitbook.io/cyberdrop-dl/dev/reference/configuration-options/settings/runtime_options#slow_download_speed) `slow_download_speed` ------------------------------------------------------------------------------------------------------------------------------------------------------------------ Type Default `ByteSize` `0` circle-info This setting specifies speed and it's interpreted as `<value> / second`. ex: `8MB` means `8MB/s` Downloads with a speed lower than this value for more than 10 seconds will be skipped. Set to `0` to disable [hashtag](https://script-ware.gitbook.io/cyberdrop-dl/dev/reference/configuration-options/settings/runtime_options#update_last_forum_post) `update_last_forum_post` ------------------------------------------------------------------------------------------------------------------------------------------------------------------------ Type Default `bool` `false` Updates `input_file` content, adding the last scraped post id to every forum thread URL [PreviousMedia Duration Limitschevron-left](https://script-ware.gitbook.io/cyberdrop-dl/dev/reference/configuration-options/settings/media_duration_limits) [NextSortingchevron-right](https://script-ware.gitbook.io/cyberdrop-dl/dev/reference/configuration-options/settings/sorting) Last updated 3 months ago * [console\_log\_level](https://script-ware.gitbook.io/cyberdrop-dl/dev/reference/configuration-options/settings/runtime_options#console_log_level) * [deep\_scrape](https://script-ware.gitbook.io/cyberdrop-dl/dev/reference/configuration-options/settings/runtime_options#deep_scrape) * [delete\_partial\_files](https://script-ware.gitbook.io/cyberdrop-dl/dev/reference/configuration-options/settings/runtime_options#delete_partial_files) * [ignore\_history](https://script-ware.gitbook.io/cyberdrop-dl/dev/reference/configuration-options/settings/runtime_options#ignore_history) * [jdownloader\_autostart](https://script-ware.gitbook.io/cyberdrop-dl/dev/reference/configuration-options/settings/runtime_options#jdownloader_autostart) * [jdownloader\_download\_dir](https://script-ware.gitbook.io/cyberdrop-dl/dev/reference/configuration-options/settings/runtime_options#jdownloader_download_dir) * [jdownloader\_whitelist](https://script-ware.gitbook.io/cyberdrop-dl/dev/reference/configuration-options/settings/runtime_options#jdownloader_whitelist) * [log\_level](https://script-ware.gitbook.io/cyberdrop-dl/dev/reference/configuration-options/settings/runtime_options#log_level) * [send\_unsupported\_to\_jdownloader](https://script-ware.gitbook.io/cyberdrop-dl/dev/reference/configuration-options/settings/runtime_options#send_unsupported_to_jdownloader) * [skip\_check\_for\_empty\_folders](https://script-ware.gitbook.io/cyberdrop-dl/dev/reference/configuration-options/settings/runtime_options#skip_check_for_empty_folders) * [skip\_check\_for\_partial\_files](https://script-ware.gitbook.io/cyberdrop-dl/dev/reference/configuration-options/settings/runtime_options#skip_check_for_partial_files) * [slow\_download\_speed](https://script-ware.gitbook.io/cyberdrop-dl/dev/reference/configuration-options/settings/runtime_options#slow_download_speed) * [update\_last\_forum\_post](https://script-ware.gitbook.io/cyberdrop-dl/dev/reference/configuration-options/settings/runtime_options#update_last_forum_post) sun-brightdesktopmoon sun-brightdesktopmoon --- # Download Options | Cyberdrop-DL [hashtag](https://script-ware.gitbook.io/cyberdrop-dl/reference/configuration-options/settings/download_options#block_download_sub_folders) `block_download_sub_folders` ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Type Default `bool` `false` When this is set to `true`, downloads that would be in a folder structure like: > `Downloads/folderA/folderB/folderC/image.jpg` will be changed to: > `Downloads/folderA/image.jpg` [hashtag](https://script-ware.gitbook.io/cyberdrop-dl/reference/configuration-options/settings/download_options#disable_download_attempts) `disable_download_attempts` --------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Type Default `bool` `false` By default the program will retry a download twice. You can set this to `true` to disable it and always retry until the download completes. However, to make sure the program will not run endlessly, there are certain situations where a file will never be retried, like if the program receives a `404` HTTP status, meaning the link is dead. [hashtag](https://script-ware.gitbook.io/cyberdrop-dl/reference/configuration-options/settings/download_options#disable_file_timestamps) `disable_file_timestamps` ----------------------------------------------------------------------------------------------------------------------------------------------------------------------- Type Default `bool` `false` By default the program will do it's absolute best to try and find the upload date of a file. It'll then set the `last modified` and `last accessed` dates on the file to match. On Windows and macOS, it will also try to set the `created` date. Setting this to `true` will disable this function, and the dates for those metadata entries will be the date the file was downloaded. [hashtag](https://script-ware.gitbook.io/cyberdrop-dl/reference/configuration-options/settings/download_options#include_album_id_in_folder_name) `include_album_id_in_folder_name` --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Type Default `bool` `false` Setting this to `true` will include the album ID (random alphanumeric string) of the album in the download folder name. [hashtag](https://script-ware.gitbook.io/cyberdrop-dl/reference/configuration-options/settings/download_options#include_thread_id_in_folder_name) `include_thread_id_in_folder_name` ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Type Default `bool` `false` Setting this to `true` will include the thread ID (random alphanumeric string) of the forum thread in the download folder name. [hashtag](https://script-ware.gitbook.io/cyberdrop-dl/reference/configuration-options/settings/download_options#maximum_number_of_children) `maximum_number_of_children` ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Type Default `list[NonNegativeInt]` `[]` Limit the number of items to scrape using a tuple of up to 4 positions. Each position defines the maximum number of sub-items (`children_limit`) a specific type of `scrape_item` will have: 1. Max number of children from a **FORUM URL** 2. Max number of children from a **FORUM POST** 3. Max number of children from a **FILE HOST PROFILE** 4. Max number of children from a **FILE HOST ALBUM** Using `0` on any position means no limit on the number of children for that type of `scrape_item`. Any tailing value not supplied is assumed as `0` ### [hashtag](https://script-ware.gitbook.io/cyberdrop-dl/reference/configuration-options/settings/download_options#examples) Examples example 1 example 2 example 3 example 4 Limit **FORUM** scrape to 15 posts max, grab all links and media within those posts, but only scrape a maximum of 10 items from each link in a post: Only grab the first link from each post in a forum, but that link will have no `children_limit`: Only grab the first **POST** / **ALBUM** from a **FILE\_HOST\_PROFILE** No **FORUM** limit, no **FORUM\_POST** limit, no **FILE\_HOST\_PROFILE** limit, maximum of 20 items from any **FILE\_HOST\_ALBUM**: [hashtag](https://script-ware.gitbook.io/cyberdrop-dl/reference/configuration-options/settings/download_options#remove_domains_from_folder_names) `remove_domains_from_folder_names` ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Type Default `bool` `false` Setting this to `true` will remove the "(DOMAIN)" portion of folder names on new downloads. [hashtag](https://script-ware.gitbook.io/cyberdrop-dl/reference/configuration-options/settings/download_options#remove_generated_id_from_filenames) `remove_generated_id_from_filenames` --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Type Default `bool` `false` Setting this to `true` will remove the alphanumeric ID added to the end of filenames by some websites. This option only works for URLs from `cyberdrop.me` at the moment. Multipart archive filenames will be corrected to follow the proper naming pattern for their format. Supported formats: `.rar` `.7z` `.tar` `.gz` `.bz2` `.zip` [hashtag](https://script-ware.gitbook.io/cyberdrop-dl/reference/configuration-options/settings/download_options#scrape_single_forum_post) `scrape_single_forum_post` ------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Type Default `bool` `false` Setting this to `true` will prevent Cyberdrop-DL from scraping an entire thread if the input URL had an specific post in it. CDL will only download files within that post. For most forum sites, the post id is part of the fragment in the URL. ex: `/thread/iphone-16-16e-16-plus-16-pro-16-promax.256047/page-64#post-7512404` has a post id of `7512404` If `scrape_single_forum_post` is `false`, CDL will download all post in the thread, from post `7512404` until the last post If `scrape_single_forum_post` is `true`, CDL will only download files within post `7512404` itself and stop. [hashtag](https://script-ware.gitbook.io/cyberdrop-dl/reference/configuration-options/settings/download_options#separate_posts) `separate_posts` ----------------------------------------------------------------------------------------------------------------------------------------------------- Type Default `bool` `false` Setting this to `true` will separate content from forum and site posts into separate folders. This option only works with sites that have 'posts': * `Forums` * `Discourse` * `reddit` * `Tiktok` * `Coomer`, `Kemono` and `Nekohouse`. For some sites, this value is hardcorded to `true` because each post is always an individual page: * `Wordpress` * `eFukt` [hashtag](https://script-ware.gitbook.io/cyberdrop-dl/reference/configuration-options/settings/download_options#separate_posts_format) `separate_posts_format` ------------------------------------------------------------------------------------------------------------------------------------------------------------------- Type Default `NonEmptyStr` `{default}` This is the format for the directory created when using `--separate-posts`. Unique Path Flags: > `date`: date of the post. This is a python `datetime` object > > `id`: The post id. This is always a `string`, even if some sites use numbers > > `number`: This no longer means anything. Currently, it always has the same value as `id` > > `title`: post title. This is a `string` circle-exclamation Not all sites support all possible flags. Ex: Posts from reddit only support the `title` flag If you use a format with a field that the site does not support, CDL will replace it with `UNKNOWN_<FIELD_NAME>` ex: using the format `reddit post #{id}` will result in `reddit post #UNKNOWN_ID` Setting it to `{default}` will use the default format, which is different for each crawler: Site Default Format `Coomer`, `Kemono` and `Nekohouse` `{date} - {title}` `Forums (Xenforo/vBulletin/Invision)` `{date} - {id} - {title}` `Discourse` `{date} - {id} - {title}` `Reddit` `{title}` `WordPress` `{date:%Y-%m-%d} - {id} - {title}` `eFukt` `{date:%Y-%m-%d} {title}` `Tiktok` `{date:%Y-%m-%d} - {id}` A date without a `format_spec` defaults to ISO 8601 format You can use any valid format string supported by python, with the following restrictions: * You can not have positional arguments in the format string. ex: `post {0} from date {1}` * You can not have unnamed fields in the format string. ex: `post {} from date {}` * You can not perform operations within the format string. ex: `post {id + 1} from date {date}` * All the fields named in the format string must be valid fields for that format option. CDL will validate this at startup [hashtag](https://script-ware.gitbook.io/cyberdrop-dl/reference/configuration-options/settings/download_options#skip_download_mark_completed) `skip_download_mark_completed` --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Type Default `bool` `false` Setting this to `true` will skip the download process for every file and mark them as downloaded in the database. [hashtag](https://script-ware.gitbook.io/cyberdrop-dl/reference/configuration-options/settings/download_options#skip_referer_seen_before) `skip_referer_seen_before` ------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Type Default `bool` `false` Setting this to `true` will skip downloading files from any referer that have been scraped before. The file (s) will always be skipped, regardless of whether the referer was successfully scraped or not [hashtag](https://script-ware.gitbook.io/cyberdrop-dl/reference/configuration-options/settings/download_options#maximum_thread_depth) `maximum_thread_depth` ----------------------------------------------------------------------------------------------------------------------------------------------------------------- Type Default `NonNegativeInt` 0 circle-exclamation It is not recommended to set this above the default value of `0`, as there is a high chance of infinite nesting in certain cases. For example, when dealing with Megathreads, if a Megathread is linked to another Megathread, you could end up scraping an undesirable amount of data. Restricts how many levels deep the scraper is allowed to go while scraping a thread A value of `0` means only the top level thread will be scraped circle-info This setting is hardcoded to `0` for Discourse sites ### [hashtag](https://script-ware.gitbook.io/cyberdrop-dl/reference/configuration-options/settings/download_options#example) Example Consider CDL finds the following sub-threads while scraping an input URL: * With `maximum_thread_depth` = 0, CDL will only download files in `thread_01`, all the other threads will be ignored * With `maximum_thread_depth` = 1, CDL will only download files in `thread_01` to `thread_08`. All threads from `thread_09` to `thread_12` will be ignored * With `maximum_thread_depth` >= 2, CDL will download files from all the threads in this case [hashtag](https://script-ware.gitbook.io/cyberdrop-dl/reference/configuration-options/settings/download_options#maximum_thread_folder_depth) `maximum_thread_folder_depth` ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Type Default `NonNegativeInt` or `None` `None` Restricts the max number of nested folders CDL will create when `maximum_thread_depth` is greater that 0 Values: * `None`: Create as many nested folders as required (AKA, the same number as `maximum_thread_depth` allows) * `0`: Do not create subfolders, use a flat structure for any nested thread. * `1+`: Create a max of `n` folders ### [hashtag](https://script-ware.gitbook.io/cyberdrop-dl/reference/configuration-options/settings/download_options#example-5) Example * With `maximum_thread_folder_depth` = None: * With `maximum_thread_folder_depth` = 0: * With `maximum_thread_folder_depth` = 1: [PreviousBrowser Cookieschevron-left](https://script-ware.gitbook.io/cyberdrop-dl/reference/configuration-options/settings/browser_cookies) [NextDupe Cleanup Optionschevron-right](https://script-ware.gitbook.io/cyberdrop-dl/reference/configuration-options/settings/dupe_cleanup_options) Last updated 5 months ago * [block\_download\_sub\_folders](https://script-ware.gitbook.io/cyberdrop-dl/reference/configuration-options/settings/download_options#block_download_sub_folders) * [disable\_download\_attempts](https://script-ware.gitbook.io/cyberdrop-dl/reference/configuration-options/settings/download_options#disable_download_attempts) * [disable\_file\_timestamps](https://script-ware.gitbook.io/cyberdrop-dl/reference/configuration-options/settings/download_options#disable_file_timestamps) * [include\_album\_id\_in\_folder\_name](https://script-ware.gitbook.io/cyberdrop-dl/reference/configuration-options/settings/download_options#include_album_id_in_folder_name) * [include\_thread\_id\_in\_folder\_name](https://script-ware.gitbook.io/cyberdrop-dl/reference/configuration-options/settings/download_options#include_thread_id_in_folder_name) * [maximum\_number\_of\_children](https://script-ware.gitbook.io/cyberdrop-dl/reference/configuration-options/settings/download_options#maximum_number_of_children) * [Examples](https://script-ware.gitbook.io/cyberdrop-dl/reference/configuration-options/settings/download_options#examples) * [remove\_domains\_from\_folder\_names](https://script-ware.gitbook.io/cyberdrop-dl/reference/configuration-options/settings/download_options#remove_domains_from_folder_names) * [remove\_generated\_id\_from\_filenames](https://script-ware.gitbook.io/cyberdrop-dl/reference/configuration-options/settings/download_options#remove_generated_id_from_filenames) * [scrape\_single\_forum\_post](https://script-ware.gitbook.io/cyberdrop-dl/reference/configuration-options/settings/download_options#scrape_single_forum_post) * [separate\_posts](https://script-ware.gitbook.io/cyberdrop-dl/reference/configuration-options/settings/download_options#separate_posts) * [separate\_posts\_format](https://script-ware.gitbook.io/cyberdrop-dl/reference/configuration-options/settings/download_options#separate_posts_format) * [skip\_download\_mark\_completed](https://script-ware.gitbook.io/cyberdrop-dl/reference/configuration-options/settings/download_options#skip_download_mark_completed) * [skip\_referer\_seen\_before](https://script-ware.gitbook.io/cyberdrop-dl/reference/configuration-options/settings/download_options#skip_referer_seen_before) * [maximum\_thread\_depth](https://script-ware.gitbook.io/cyberdrop-dl/reference/configuration-options/settings/download_options#maximum_thread_depth) * [Example](https://script-ware.gitbook.io/cyberdrop-dl/reference/configuration-options/settings/download_options#example) * [maximum\_thread\_folder\_depth](https://script-ware.gitbook.io/cyberdrop-dl/reference/configuration-options/settings/download_options#maximum_thread_folder_depth) * [Example](https://script-ware.gitbook.io/cyberdrop-dl/reference/configuration-options/settings/download_options#example-5) sun-brightdesktopmoon Copy --maximum-number-of-children 15 0 10 Copy --maximum-number-of-children 0 1 Copy --maximum-number-of-children 0 0 1 Copy --maximum-number-of-children 0 0 0 20 Copy └── thread_01 ├── thread_02 ├── thread_03 │ ├── thread_09 │ ├── thread_10 │ └── thread_11 ├── thread_04 ├── thread_05 ├── thread_06 ├── thread_07 │ └── thread_12 └── thread_08 Copy └── thread_01 ├── thread_02 ├── thread_03 │ ├── thread_09 │ ├── thread_10 │ └── thread_11 ├── thread_04 ├── thread_05 ├── thread_06 ├── thread_07 │ └── thread_12 └── thread_08 Copy ├── thread_01 ├── thread_02 ├── thread_03 ├── thread_09 ├── thread_10 ├── thread_11 ├── thread_04 ├── thread_05 ├── thread_06 ├── thread_07 ├── thread_12 └── thread_08 Copy └── thread_01 ├── thread_02 ├── thread_03 ├── thread_09 ├── thread_10 ├── thread_11 ├── thread_04 ├── thread_05 ├── thread_06 ├── thread_07 ├── thread_12 └── thread_08 sun-brightdesktopmoon --- # CLI Arguments | Cyberdrop-DL Development Docs | Cyberdrop-DL circle-info CLI inputs always take priority over config values. circle-info Use `-` instead of `_` to separate words in an config option name when using it as a CLI argument: Ex: `auto_dedupe` needs to be `auto-dedupe` when using it via the CLI You can pass any of the **Config Settings** and **Global Settings** options as a cli argument for the program. For items not explained below, you can find their counterparts in the configuration options to see what they do. [hashtag](https://script-ware.gitbook.io/cyberdrop-dl/dev/reference/cli-arguments#cli-only-arguments) CLI only arguments ----------------------------------------------------------------------------------------------------------------------------- ### [hashtag](https://script-ware.gitbook.io/cyberdrop-dl/dev/reference/cli-arguments#appdata-folder) `appdata-folder` Type Default `Path` `<Current Working Directory>` Folder where Cyberdrop-DL will store it's database, cache and config files. ### [hashtag](https://script-ware.gitbook.io/cyberdrop-dl/dev/reference/cli-arguments#completed-after) `completed-after` Type Default `date` `None` Only retry downloads that were completed on or after this date. The date should be in ISO 8601 format, for example, `2021-12-23` circle-info This option has no effect unless you run CDL with `--retry-all` ### [hashtag](https://script-ware.gitbook.io/cyberdrop-dl/dev/reference/cli-arguments#completed-before) `completed-before` Type Default `date` `None` Only retry downloads that were completed on or before this date. The date should be in ISO 8601 format, for example, `2021-12-23` circle-info This option has no effect unless you run CDL with `--retry-all` ### [hashtag](https://script-ware.gitbook.io/cyberdrop-dl/dev/reference/cli-arguments#config) `config` Type Default `str` `None` Name of config to load. ### [hashtag](https://script-ware.gitbook.io/cyberdrop-dl/dev/reference/cli-arguments#config-file) `config-file` Type Default `Path` `None` Path to the CDL `settings.yaml` file to load circle-info If both `config` and `config-file` are supplied, `config-file` takes priority ### [hashtag](https://script-ware.gitbook.io/cyberdrop-dl/dev/reference/cli-arguments#disable-cache) `disable-cache` Type Default Action `BoolFlag` `False` `store_true` Disables read/writes to requests cache for the current run only. All config settings or arguments related to the cache (ex: `file_host_cache_expire_after`) will be ignored. ### [hashtag](https://script-ware.gitbook.io/cyberdrop-dl/dev/reference/cli-arguments#download) `download` Type Default Action `BoolFlag` `False` `store_true` Skips UI, start download immediately ### [hashtag](https://script-ware.gitbook.io/cyberdrop-dl/dev/reference/cli-arguments#download-tiktok-audios) `download-tiktok-audios` Type Default Action `BoolFlag` `False` `store_true` Download TikTok audios from posts and save them as separate files ### [hashtag](https://script-ware.gitbook.io/cyberdrop-dl/dev/reference/cli-arguments#download-tiktok-src-quality-videos) `download-tiktok-src-quality-videos` Type Default Action `BoolFlag` `False` `store_true` By default, CDL will download the "optimized for streaming" version of tiktok videos. Setting this option to `True` will download videos in original (source) quality. `_original` will be added as a suffix to their filename. circle-exclamation This will make video downloads several times slower When it is set to `False` (the default) CDL can download 50 videos with a single request. When it is set to `True` , CDL needs to make at least 3 requests _per_ video to download them. There's also a daily limit of the API CDL uses: 5000 requests per day per IP Setting this option to `True` will consume the daily limit faster ### [hashtag](https://script-ware.gitbook.io/cyberdrop-dl/dev/reference/cli-arguments#impersonate) `impersonate` Type Default Action `chrome", "edge", "safari", "safari_ios", "chrome_android", "firefox"`, `BoolFlag` or `null` `null` `store_true` or `store_const` Impersonation allows CDL to make requests and appear to be a legitimate web browser. This helps bypass bot-protection on some sites and it's required for any site that only accepts HTTP2 connections. * The default value (`null`) means CDL will automatically use impersonation for crawlers that were programmed to use it. * Passing the flag without any value (`--impersonate`) is the same as `--impersonate True`: CDL will use impersonation for ALL requests, using the default impersonation target * Passing an specific target (ex: `--impersonate chrome_android`) will make CDL use impersonation for all requests, using that tarjet circle-info The current default target is `chrome`. The default target can change on any new release without notice ### [hashtag](https://script-ware.gitbook.io/cyberdrop-dl/dev/reference/cli-arguments#max-items-retry) `max-items-retry` Type Default `NonNegativeInt` `0` Max number of links to retry. Using `0` means no limit circle-info This option has no effect unless you run CDL with one of the retry options: `--retry-all`, `--retry-failed` or `--retry-maintenance` ### [hashtag](https://script-ware.gitbook.io/cyberdrop-dl/dev/reference/cli-arguments#portrait) `portrait` Type Default Action `BoolFlag` `False` `store_true` Force CDL to run with a vertical layout ### [hashtag](https://script-ware.gitbook.io/cyberdrop-dl/dev/reference/cli-arguments#print-stats) `print-stats` Type Default Action `BoolFlag` `True` `store_false` Show stats report at the end of a run ### [hashtag](https://script-ware.gitbook.io/cyberdrop-dl/dev/reference/cli-arguments#retry-all) `retry-all` Type Default Action `BoolFlag` `False` `store_true` Retry all downloads ### [hashtag](https://script-ware.gitbook.io/cyberdrop-dl/dev/reference/cli-arguments#retry-failed) `retry-failed` Type Default Action `BoolFlag` `False` `store_true` Retry failed downloads ### [hashtag](https://script-ware.gitbook.io/cyberdrop-dl/dev/reference/cli-arguments#retry-maintenance) `retry-maintenance` Type Default Action `BoolFlag` `False` `store_true` Retry download of maintenance files (bunkr). Requires files to be hashed ### [hashtag](https://script-ware.gitbook.io/cyberdrop-dl/dev/reference/cli-arguments#show-supported-sites) `show-supported-sites` Type Default Action `BoolFlag` `False` `store_true` Shows a list of all supported sites and exits ### [hashtag](https://script-ware.gitbook.io/cyberdrop-dl/dev/reference/cli-arguments#ui) `ui` Type Default `CaseInsensitiveStrEnum` `FULLSCREEN` UI can have 1 of these values: * `DISABLED` : no output at all * `ACTIVITY` : only shows a spinner with the text `running CDL...` * `SIMPLE`: shows spinner + simplified progress bar * `FULLSCREEN`: shows the normal UI/progress view circle-info Values are case insensitive, ex: both `disabled` and `DISABLED` are valid [hashtag](https://script-ware.gitbook.io/cyberdrop-dl/dev/reference/cli-arguments#overview) Overview --------------------------------------------------------------------------------------------------------- Bool arguments like options within `Download Options`, `Ignore Options`, `Runtime Options`, etc. can be prefixed with `--no-` to negate them. Ex: `--no-auto-dedupe` will disable auto dedupe, overriding whatever the config option was set to. [PreviousGeneric Crawlerschevron-left](https://script-ware.gitbook.io/cyberdrop-dl/dev/reference/configuration-options/global-settings/generic_crawlers_instances) [NextNotificationschevron-right](https://script-ware.gitbook.io/cyberdrop-dl/dev/reference/notifications) Last updated 1 month ago sun-brightdesktopmoon Copy usage: cyberdrop-dl [OPTIONS] URL [URL...] Bulk asynchronous downloader for multiple file hosts options: -h, --help show this help message and exit -V, --version show CDL version number and exit CLI-only options: LINK(S) link(s) to content to download (passing multiple links is supported) --appdata-folder APPDATA_FOLDER AppData folder path --completed-after COMPLETED_AFTER only retry downloads that were completed on or after this date --completed-before COMPLETED_BEFORE only retry downloads that were completed on or before this date --config CONFIG name of config to load --config-file CONFIG_FILE path to the CDL settings.yaml file to load --disable-cache temporarily disable the requests cache --download skips UI, start download immediately --download-tiktok-audios download TikTok audios from posts and save them as separate files --download-tiktok-src-quality-videos download TikTok videos in source quality --impersonate [IMPERSONATE] Use this target as impersonation for all scrape requests --max-items-retry MAX_ITEMS_RETRY max number of links to retry --portrait force CDL to run with a vertical layout --print-stats show stats report at the end of a run --retry-all retry all downloads --retry-failed retry failed downloads --retry-maintenance retry download of maintenance files (bunkr). Requires files to be hashed --show-supported-sites shows a list of supported sites and exits --ui UI DISABLED, ACTIVITY, SIMPLE or FULLSCREEN browser_cookies: --auto-import, --no-auto-import --browser BROWSER --sites [SITES ...] download_options: --block-download-sub-folders, --no-block-download-sub-folders --disable-download-attempt-limit, --no-disable-download-attempt-limit --disable-file-timestamps, --no-disable-file-timestamps --include-album-id-in-folder-name, --no-include-album-id-in-folder-name --include-thread-id-in-folder-name, --no-include-thread-id-in-folder-name --maximum-number-of-children [MAXIMUM_NUMBER_OF_CHILDREN ...] --remove-domains-from-folder-names, --no-remove-domains-from-folder-names --remove-generated-id-from-filenames, --no-remove-generated-id-from-filenames --scrape-single-forum-post, --no-scrape-single-forum-post --separate-posts-format SEPARATE_POSTS_FORMAT --separate-posts, --no-separate-posts --skip-download-mark-completed, --no-skip-download-mark-completed --skip-referer-seen-before, --no-skip-referer-seen-before --maximum-thread-depth MAXIMUM_THREAD_DEPTH --maximum-thread-folder-depth MAXIMUM_THREAD_FOLDER_DEPTH dupe_cleanup_options: --add-md5-hash, --no-add-md5-hash --add-sha256-hash, --no-add-sha256-hash --auto-dedupe, --no-auto-dedupe --hashing HASHING --send-deleted-to-trash, --no-send-deleted-to-trash file_size_limits: --maximum-image-size MAXIMUM_IMAGE_SIZE --maximum-other-size MAXIMUM_OTHER_SIZE --maximum-video-size MAXIMUM_VIDEO_SIZE --minimum-image-size MINIMUM_IMAGE_SIZE --minimum-other-size MINIMUM_OTHER_SIZE --minimum-video-size MINIMUM_VIDEO_SIZE media_duration_limits: --maximum-video-duration MAXIMUM_VIDEO_DURATION --maximum-audio-duration MAXIMUM_AUDIO_DURATION --minimum-video-duration MINIMUM_VIDEO_DURATION --minimum-audio-duration MINIMUM_AUDIO_DURATION files: -d, --download-folder DOWNLOAD_FOLDER -j, --dump-json, --no-dump-json -i, --input-file INPUT_FILE --save-pages-html, --no-save-pages-html ignore_options: --exclude-audio, --no-exclude-audio --exclude-images, --no-exclude-images --exclude-other, --no-exclude-other --exclude-videos, --no-exclude-videos --filename-regex-filter FILENAME_REGEX_FILTER --ignore-coomer-ads, --no-ignore-coomer-ads --ignore-coomer-post-content, --no-ignore-coomer-post-content --only-hosts [ONLY_HOSTS ...] --skip-hosts [SKIP_HOSTS ...] --exclude-files-with-no-extension, --no-exclude-files-with-no-extension --exclude-before EXCLUDE_BEFORE --exclude-after EXCLUDE_AFTER logs: --download-error-urls DOWNLOAD_ERROR_URLS --last-forum-post LAST_FORUM_POST --log-folder LOG_FOLDER --log-line-width LOG_LINE_WIDTH --logs-expire-after LOGS_EXPIRE_AFTER --main-log MAIN_LOG --rotate-logs, --no-rotate-logs --scrape-error-urls SCRAPE_ERROR_URLS --unsupported-urls UNSUPPORTED_URLS --webhook WEBHOOK runtime_options: --console-log-level CONSOLE_LOG_LEVEL --deep-scrape, --no-deep-scrape --delete-partial-files, --no-delete-partial-files --ignore-history, --no-ignore-history --jdownloader-autostart, --no-jdownloader-autostart --jdownloader-download-dir JDOWNLOADER_DOWNLOAD_DIR --jdownloader-whitelist [JDOWNLOADER_WHITELIST ...] --log-level LOG_LEVEL --send-unsupported-to-jdownloader, --no-send-unsupported-to-jdownloader --skip-check-for-empty-folders, --no-skip-check-for-empty-folders --skip-check-for-partial-files, --no-skip-check-for-partial-files --slow-download-speed SLOW_DOWNLOAD_SPEED --update-last-forum-post, --no-update-last-forum-post sorting: --scan-folder SCAN_FOLDER --sort-downloads, --no-sort-downloads --sort-folder SORT_FOLDER --sort-incrementer-format SORT_INCREMENTER_FORMAT --sorted-audio SORTED_AUDIO --sorted-image SORTED_IMAGE --sorted-other SORTED_OTHER --sorted-video SORTED_VIDEO general: --ssl-context SSL_CONTEXT --disable-crawlers [DISABLE_CRAWLERS ...] --enable-generic-crawler, --no-enable-generic-crawler --flaresolverr FLARESOLVERR --max-file-name-length MAX_FILE_NAME_LENGTH --max-folder-name-length MAX_FOLDER_NAME_LENGTH --proxy PROXY --required-free-space REQUIRED_FREE_SPACE --user-agent USER_AGENT rate_limiting_options: --download-attempts DOWNLOAD_ATTEMPTS --download-delay DOWNLOAD_DELAY --download-speed-limit DOWNLOAD_SPEED_LIMIT --file-host-cache-expire-after FILE_HOST_CACHE_EXPIRE_AFTER --forum-cache-expire-after FORUM_CACHE_EXPIRE_AFTER --jitter JITTER --max-simultaneous-downloads-per-domain MAX_SIMULTANEOUS_DOWNLOADS_PER_DOMAIN --max-simultaneous-downloads MAX_SIMULTANEOUS_DOWNLOADS --rate-limit RATE_LIMIT --connection-timeout CONNECTION_TIMEOUT --read-timeout READ_TIMEOUT ui_options: --downloading-item-limit DOWNLOADING_ITEM_LIMIT --refresh-rate REFRESH_RATE --scraping-item-limit SCRAPING_ITEM_LIMIT --vi-mode, --no-vi-mode generic_crawlers_instances: --wordpress-media [WORDPRESS_MEDIA ...] --wordpress-html [WORDPRESS_HTML ...] --discourse [DISCOURSE ...] --chevereto [CHEVERETO ...] sun-brightdesktopmoon --- # Download Options | Cyberdrop-DL Development Docs | Cyberdrop-DL [hashtag](https://script-ware.gitbook.io/cyberdrop-dl/dev/reference/configuration-options/settings/download_options#block_download_sub_folders) `block_download_sub_folders` --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Type Default `bool` `false` When this is set to `true`, downloads that would be in a folder structure like: > `Downloads/folderA/folderB/folderC/image.jpg` will be changed to: > `Downloads/folderA/image.jpg` [hashtag](https://script-ware.gitbook.io/cyberdrop-dl/dev/reference/configuration-options/settings/download_options#disable_download_attempts) `disable_download_attempts` ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Type Default `bool` `false` By default the program will retry a download twice. You can set this to `true` to disable it and always retry until the download completes. However, to make sure the program will not run endlessly, there are certain situations where a file will never be retried, like if the program receives a `404` HTTP status, meaning the link is dead. [hashtag](https://script-ware.gitbook.io/cyberdrop-dl/dev/reference/configuration-options/settings/download_options#disable_file_timestamps) `disable_file_timestamps` --------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Type Default `bool` `false` By default the program will do it's absolute best to try and find the upload date of a file. It'll then set the `last modified` and `last accessed` dates on the file to match. On Windows and macOS, it will also try to set the `created` date. Setting this to `true` will disable this function, and the dates for those metadata entries will be the date the file was downloaded. [hashtag](https://script-ware.gitbook.io/cyberdrop-dl/dev/reference/configuration-options/settings/download_options#include_album_id_in_folder_name) `include_album_id_in_folder_name` ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Type Default `bool` `false` Setting this to `true` will include the album ID (random alphanumeric string) of the album in the download folder name. [hashtag](https://script-ware.gitbook.io/cyberdrop-dl/dev/reference/configuration-options/settings/download_options#include_thread_id_in_folder_name) `include_thread_id_in_folder_name` --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Type Default `bool` `false` Setting this to `true` will include the thread ID (random alphanumeric string) of the forum thread in the download folder name. [hashtag](https://script-ware.gitbook.io/cyberdrop-dl/dev/reference/configuration-options/settings/download_options#maximum_number_of_children) `maximum_number_of_children` --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Type Default `list[NonNegativeInt]` `[]` Limit the number of items to scrape using a tuple of up to 4 positions. Each position defines the maximum number of sub-items (`children_limit`) a specific type of `scrape_item` will have: 1. Max number of children from a **FORUM URL** 2. Max number of children from a **FORUM POST** 3. Max number of children from a **FILE HOST PROFILE** 4. Max number of children from a **FILE HOST ALBUM** Using `0` on any position means no limit on the number of children for that type of `scrape_item`. Any tailing value not supplied is assumed as `0` ### [hashtag](https://script-ware.gitbook.io/cyberdrop-dl/dev/reference/configuration-options/settings/download_options#examples) Examples example 1 example 2 example 3 example 4 Limit **FORUM** scrape to 15 posts max, grab all links and media within those posts, but only scrape a maximum of 10 items from each link in a post: Only grab the first link from each post in a forum, but that link will have no `children_limit`: Only grab the first **POST** / **ALBUM** from a **FILE\_HOST\_PROFILE** No **FORUM** limit, no **FORUM\_POST** limit, no **FILE\_HOST\_PROFILE** limit, maximum of 20 items from any **FILE\_HOST\_ALBUM**: [hashtag](https://script-ware.gitbook.io/cyberdrop-dl/dev/reference/configuration-options/settings/download_options#remove_domains_from_folder_names) `remove_domains_from_folder_names` --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Type Default `bool` `false` Setting this to `true` will remove the "(DOMAIN)" portion of folder names on new downloads. [hashtag](https://script-ware.gitbook.io/cyberdrop-dl/dev/reference/configuration-options/settings/download_options#remove_generated_id_from_filenames) `remove_generated_id_from_filenames` ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Type Default `bool` `false` Setting this to `true` will remove the alphanumeric ID added to the end of filenames by some websites. This option only works for URLs from `cyberdrop.me` at the moment. Multipart archive filenames will be corrected to follow the proper naming pattern for their format. Supported formats: `.rar` `.7z` `.tar` `.gz` `.bz2` `.zip` [hashtag](https://script-ware.gitbook.io/cyberdrop-dl/dev/reference/configuration-options/settings/download_options#scrape_single_forum_post) `scrape_single_forum_post` ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Type Default `bool` `false` Setting this to `true` will prevent Cyberdrop-DL from scraping an entire thread if the input URL had an specific post in it. CDL will only download files within that post. For most forum sites, the post id is part of the fragment in the URL. ex: `/thread/iphone-16-16e-16-plus-16-pro-16-promax.256047/page-64#post-7512404` has a post id of `7512404` If `scrape_single_forum_post` is `false`, CDL will download all post in the thread, from post `7512404` until the last post If `scrape_single_forum_post` is `true`, CDL will only download files within post `7512404` itself and stop. [hashtag](https://script-ware.gitbook.io/cyberdrop-dl/dev/reference/configuration-options/settings/download_options#separate_posts) `separate_posts` --------------------------------------------------------------------------------------------------------------------------------------------------------- Type Default `bool` `false` Setting this to `true` will separate content from forum and site posts into separate folders. This option only works with sites that have 'posts': * `Forums` * `Discourse` * `Tiktok` * `Coomer`, `Kemono` and `Nekohouse`. For some sites, this value is hardcorded to `true` because each post is always an individual page: * `Wordpress` * `eFukt` [hashtag](https://script-ware.gitbook.io/cyberdrop-dl/dev/reference/configuration-options/settings/download_options#separate_posts_format) `separate_posts_format` ----------------------------------------------------------------------------------------------------------------------------------------------------------------------- Type Default `NonEmptyStr` `{default}` This is the format for the directory created when using `--separate-posts`. Unique Path Flags: > `date`: date of the post. This is a python `datetime` object > > `id`: The post id. This is always a `string`, even if some sites use numbers > > `number`: This no longer means anything. Currently, it always has the same value as `id` > > `title`: post title. This is a `string` circle-exclamation Not all sites support all possible flags. If you use a format with a field that the site does not support, CDL will replace it with `UNKNOWN_<FIELD_NAME>` Setting it to `{default}` will use the default format, which is different for each crawler: Site Default Format `Coomer`, `Kemono` and `Nekohouse` `{date} - {title}` `Forums (Xenforo/vBulletin/Invision)` `{date} - {id} - {title}` `Discourse` `{date} - {id} - {title}` `Reddit` `{title}` `WordPress` `{date:%Y-%m-%d} - {id} - {title}` `eFukt` `{date:%Y-%m-%d} {title}` `Tiktok` `{date:%Y-%m-%d} - {id}` A date without a `format_spec` defaults to ISO 8601 format You can use any valid format string supported by python, with the following restrictions: * You can not have positional arguments in the format string. ex: `post {0} from date {1}` * You can not have unnamed fields in the format string. ex: `post {} from date {}` * You can not perform operations within the format string. ex: `post {id + 1} from date {date}` * All the fields named in the format string must be valid fields for that format option. CDL will validate this at startup [hashtag](https://script-ware.gitbook.io/cyberdrop-dl/dev/reference/configuration-options/settings/download_options#skip_download_mark_completed) `skip_download_mark_completed` ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Type Default `bool` `false` Setting this to `true` will skip the download process for every file and mark them as downloaded in the database. [hashtag](https://script-ware.gitbook.io/cyberdrop-dl/dev/reference/configuration-options/settings/download_options#maximum_thread_depth) `maximum_thread_depth` --------------------------------------------------------------------------------------------------------------------------------------------------------------------- Type Default `NonNegativeInt` 0 circle-exclamation It is not recommended to set this above the default value of `0`, as there is a high chance of infinite nesting in certain cases. For example, when dealing with Megathreads, if a Megathread is linked to another Megathread, you could end up scraping an undesirable amount of data. Restricts how many levels deep the scraper is allowed to go while scraping a thread A value of `0` means only the top level thread will be scraped circle-info This setting is hardcoded to `0` for Discourse sites ### [hashtag](https://script-ware.gitbook.io/cyberdrop-dl/dev/reference/configuration-options/settings/download_options#example) Example Consider CDL finds the following sub-threads while scraping an input URL: * With `maximum_thread_depth` = 0, CDL will only download files in `thread_01`, all the other threads will be ignored * With `maximum_thread_depth` = 1, CDL will only download files in `thread_01` to `thread_08`. All threads from `thread_09` to `thread_12` will be ignored * With `maximum_thread_depth` >= 2, CDL will download files from all the threads in this case [hashtag](https://script-ware.gitbook.io/cyberdrop-dl/dev/reference/configuration-options/settings/download_options#maximum_thread_folder_depth) `maximum_thread_folder_depth` ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Type Default `NonNegativeInt` or `None` `None` Restricts the max number of nested folders CDL will create when `maximum_thread_depth` is greater that 0 Values: * `None`: Create as many nested folders as required (AKA, the same number as `maximum_thread_depth` allows) * `0`: Do not create subfolders, use a flat structure for any nested thread. * `1+`: Create a max of `n` folders ### [hashtag](https://script-ware.gitbook.io/cyberdrop-dl/dev/reference/configuration-options/settings/download_options#example-5) Example * With `maximum_thread_folder_depth` = None: * With `maximum_thread_folder_depth` = 0: * With `maximum_thread_folder_depth` = 1: [PreviousBrowser Cookieschevron-left](https://script-ware.gitbook.io/cyberdrop-dl/dev/reference/configuration-options/settings/browser_cookies) [NextDupe Cleanup Optionschevron-right](https://script-ware.gitbook.io/cyberdrop-dl/dev/reference/configuration-options/settings/dupe_cleanup_options) Last updated 1 month ago * [block\_download\_sub\_folders](https://script-ware.gitbook.io/cyberdrop-dl/dev/reference/configuration-options/settings/download_options#block_download_sub_folders) * [disable\_download\_attempts](https://script-ware.gitbook.io/cyberdrop-dl/dev/reference/configuration-options/settings/download_options#disable_download_attempts) * [disable\_file\_timestamps](https://script-ware.gitbook.io/cyberdrop-dl/dev/reference/configuration-options/settings/download_options#disable_file_timestamps) * [include\_album\_id\_in\_folder\_name](https://script-ware.gitbook.io/cyberdrop-dl/dev/reference/configuration-options/settings/download_options#include_album_id_in_folder_name) * [include\_thread\_id\_in\_folder\_name](https://script-ware.gitbook.io/cyberdrop-dl/dev/reference/configuration-options/settings/download_options#include_thread_id_in_folder_name) * [maximum\_number\_of\_children](https://script-ware.gitbook.io/cyberdrop-dl/dev/reference/configuration-options/settings/download_options#maximum_number_of_children) * [Examples](https://script-ware.gitbook.io/cyberdrop-dl/dev/reference/configuration-options/settings/download_options#examples) * [remove\_domains\_from\_folder\_names](https://script-ware.gitbook.io/cyberdrop-dl/dev/reference/configuration-options/settings/download_options#remove_domains_from_folder_names) * [remove\_generated\_id\_from\_filenames](https://script-ware.gitbook.io/cyberdrop-dl/dev/reference/configuration-options/settings/download_options#remove_generated_id_from_filenames) * [scrape\_single\_forum\_post](https://script-ware.gitbook.io/cyberdrop-dl/dev/reference/configuration-options/settings/download_options#scrape_single_forum_post) * [separate\_posts](https://script-ware.gitbook.io/cyberdrop-dl/dev/reference/configuration-options/settings/download_options#separate_posts) * [separate\_posts\_format](https://script-ware.gitbook.io/cyberdrop-dl/dev/reference/configuration-options/settings/download_options#separate_posts_format) * [skip\_download\_mark\_completed](https://script-ware.gitbook.io/cyberdrop-dl/dev/reference/configuration-options/settings/download_options#skip_download_mark_completed) * [maximum\_thread\_depth](https://script-ware.gitbook.io/cyberdrop-dl/dev/reference/configuration-options/settings/download_options#maximum_thread_depth) * [Example](https://script-ware.gitbook.io/cyberdrop-dl/dev/reference/configuration-options/settings/download_options#example) * [maximum\_thread\_folder\_depth](https://script-ware.gitbook.io/cyberdrop-dl/dev/reference/configuration-options/settings/download_options#maximum_thread_folder_depth) * [Example](https://script-ware.gitbook.io/cyberdrop-dl/dev/reference/configuration-options/settings/download_options#example-5) sun-brightdesktopmoon Copy --maximum-number-of-children 15 0 10 Copy --maximum-number-of-children 0 1 Copy --maximum-number-of-children 0 0 1 Copy --maximum-number-of-children 0 0 0 20 Copy └── thread_01 ├── thread_02 ├── thread_03 │ ├── thread_09 │ ├── thread_10 │ └── thread_11 ├── thread_04 ├── thread_05 ├── thread_06 ├── thread_07 │ └── thread_12 └── thread_08 Copy └── thread_01 ├── thread_02 ├── thread_03 │ ├── thread_09 │ ├── thread_10 │ └── thread_11 ├── thread_04 ├── thread_05 ├── thread_06 ├── thread_07 │ └── thread_12 └── thread_08 Copy ├── thread_01 ├── thread_02 ├── thread_03 ├── thread_09 ├── thread_10 ├── thread_11 ├── thread_04 ├── thread_05 ├── thread_06 ├── thread_07 ├── thread_12 └── thread_08 Copy └── thread_01 ├── thread_02 ├── thread_03 ├── thread_09 ├── thread_10 ├── thread_11 ├── thread_04 ├── thread_05 ├── thread_06 ├── thread_07 ├── thread_12 └── thread_08 sun-brightdesktopmoon --- # File Size Limits | Cyberdrop-DL Development Docs | Cyberdrop-DL You can provide the maximum and minimum file size for each file "type". All options on this category take a `ByteSize` input ([more info here](https://script-ware.gitbook.io/cyberdrop-dl/dev/reference/configuration-options/special_setting_types#bytesize) ). Type Default `ByteSize` `0` Setting any of these options to `0` means that limit is `disabled` * `maximum_image_filesize` * `minimum_image_filesize` * `maximum_video_filesize` * `minimum_video_filesize` * `maximum_other_filesize` * `minimum_other_filesize` [PreviousDupe Cleanup Optionschevron-left](https://script-ware.gitbook.io/cyberdrop-dl/dev/reference/configuration-options/settings/dupe_cleanup_options) [NextFileschevron-right](https://script-ware.gitbook.io/cyberdrop-dl/dev/reference/configuration-options/settings/files) Last updated 8 months ago sun-brightdesktopmoon sun-brightdesktopmoon --- # Supported Websites | Cyberdrop-DL Development Docs | Cyberdrop-DL For a full list of all supported sites, see [supported sites](https://script-ware.gitbook.io/cyberdrop-dl/dev/reference/supported-websites#supported-sites) [hashtag](https://script-ware.gitbook.io/cyberdrop-dl/dev/reference/supported-websites#password-protected-content-hosts) Password Protected Content Hosts -------------------------------------------------------------------------------------------------------------------------------------------------------------- Cyberdrop-DL can download password protected files and folders from these hosts. User must include the password as a query parameter in the input URL, adding `?password=<URL_PASSWORD>` to it. Example: `https://cyberfile.me/folder/xUGg?password=1234` Domain GoFile Cyberfile Chevereto Sites (`JPG5`, `ImagePond.net`,`ImgLike`) Iceyfile.com Transfer.it Koofr.eu Sites supported by Real-Debrid [hashtag](https://script-ware.gitbook.io/cyberdrop-dl/dev/reference/supported-websites#additional-content-hosts-with-real-debrid) Additional Content Hosts with Real-Debrid -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Cyberdrop-DL has integration with Real-Debrid as download service to support additional hosts. In order to enable Real-Debrid, user must provide their API token inside the `authentication.yaml` file. You can get your API token from this URL (you must be logged in): [https://real-debrid.com/apitokenarrow-up-right](https://real-debrid.com/apitoken) Supported domains via Real-Debrid include `rapidgator`, `4shared.com`, `fikper.com`, `k2s`, `etc`. List of all supported domains can be found here (250+): [https://api.real-debrid.com/rest/1.0/hosts/domainsarrow-up-right](https://api.real-debrid.com/rest/1.0/hosts/domains) circle-info CDL will only use Real-Debrid for unsupported sites. To use it for a site that CDL supports, ex: `mega.nz`, you have to disable the `mega.nz` crawler. See: https://script-ware.gitbook.io/cyberdrop-dl/reference/configuration-options/global-settings/general#disable\_crawlers [hashtag](https://script-ware.gitbook.io/cyberdrop-dl/dev/reference/supported-websites#supported-sites) Supported sites ---------------------------------------------------------------------------------------------------------------------------- List of sites supported by cyberdrop-dl-patched as of version 8.10.0 Site Primary URL Supported Domains Supported Paths 4chan https://boards.4chan.org 4chan.\* Board: - `/<board>` Thread: - `/thread` 8Muses https://comics.8muses.com 8muses.\* Album: - `/comics/album/...` AllPornComix https://forum.allporncomix.com allporncomix.\* Attachments: - `/(attachments|data|uploads)/...` Threads: - `/(thread|topic|tema|threads|topics|temas)/<thread_name_and_id>` - `/goto/<post_id>` - `/posts/<post_id>` **NOTES** - base crawler: Xenforo Anontransfer.com https://anontransfer.com anontransfer.com Direct Link: - `/download-direct.php?dir=<file_id>&file=<filename>` - `/uploads/<file_id>/<filename>` File: - `/d/<file_id>` Folder: - `/f/<folder_uuid>` AnySex https://anysex.com anysex.\* Album: - `/photos/<album_id>/...` Search: - `/search/...` Video: - `/video/<video_id>/...` ArchiveBate https://www.archivebate.store archivebate.\* Video: - `/watch/<video_id>` aShemaleTube https://www.ashemaletube.com ashemaletube.\* Model: - `/creators/...` - `/model/...` - `/pornstars/...` Playlist: - `/playlists/...` User: - `/profiles/...` Video: - `/videos/...` Bandcamp https://bandcamp.com bandcamp.\* Album: - `/album/<slug>` Song: - `/track/<slug>` **NOTES** - You can set 'CDL\_BANDCAMP\_FORMATS' env var to a comma separated list of formats to download (Ordered by preference) \[Default = 'mp3-320,mp3,aac-hi,wav,flac,vorbis,aiff,alas'\] Beeg.com https://beeg.com beeg.com Video: - `/<video_id>` - `/video/<video_id>` Bellazon https://www.bellazon.com/main bellazon.\* Attachments: - `/(attachments|data|uploads)/...` Threads: - `/(thread|topic|tema|threads|topics|temas)/<thread_name_and_id>` - `/goto/<post_id>` - `/posts/<post_id>` **NOTES** - base crawler: Invision BestPrettyGirl https://bestprettygirl.com bestprettygirl.com All Posts: - `/posts/` Category: - `/category/<category_slug>` Date Range: - `...?after=<date>` - `...?before=<date&after=<date>` - `...?before=<date>` Post: - `/<post_slug>/` Tag: - `/tag/<tag_slug>` **NOTES** - For `Date Range`, must be a valid iso 8601 date, ex: `2022-12-06`. `Date Range` can be combined with `Category`, `Tag` and `All Posts`. ex: To only download categories from a date range: , `/category/<category_slug>?before=<date>` Box https://www.box.com app.box.com Embedded File or Folder: - `app.box.com/embed/s?sh=<share_code>` - `app.box.com/embed_widget/s?sh=<share_code>` File or Folder: - `app.box.com/s?sh=<share_code>` Bunkr https://bunkr.site bunkr.\* Album: - `/a/<album_id>` Direct Links File: - `/<slug>` - `/f/<slug>` Video: - `/v/<slug>` Bunkr-Albums.io https://bunkr-albums.io bunkr-albums.io Search: - `/s?search=<query>` BuzzHeavier https://buzzheavier.com buzzheavier.com Direct Links Camwhores.tv https://www.camwhores.tv camwhores.tv Categories: - `/categories/...` Members: - `/members/<member_id>` Search: - `/search/...` Tags: - `/tags/...` Videos: - `/videos/...` Catbox https://catbox.moe files.catbox.moe litter.catbox.moe Direct Links CelebForum https://celebforum.to celebforum.\* Attachments: - `/(attachments|data|uploads)/...` Threads: - `/(thread|topic|tema|threads|topics|temas)/<thread_name_and_id>` - `/goto/<post_id>` - `/posts/<post_id>` **NOTES** - base crawler: Xenforo Chevereto ::GENERIC CRAWLER:: Album: - `/a/<id>` - `/a/<name>.<id>` - `/album/<id>` - `/album/<name>.<id>` Category: - `/category/<name>` Direct Links Image: - `/image/<id>` - `/image/<name>.<id>` - `/img/<id>` - `/img/<name>.<id>` Profile: - `/<user_name>` Video: - `/video/<id>` - `/video/<name>.<id>` - `/videos/<id>` - `/videos/<name>.<id>` CloudflareStream https://cloudflarestream.com cloudflarestream.com videodelivery.net Public Video: - `/<video_uid>` - `/<video_uid>/iframe` - `/<video_uid>/watch` - `/embed/___.js?video=<video_uid>` Restricted Access Video: - `/<jwt_access_token>` - `/<jwt_access_token>/iframe` - `/<jwt_access_token>/watch` - `/embed/___.js?video=<jwt_access_token>` Coomer https://coomer.st coomer.party coomer.st coomer.su Direct links: - `/data/...` - `/thumbnail/...` Favorites: - `/account/favorites/posts|artists` - `/favorites?type=post|artist` Individual Post: - `/<service>/user/<user_id>/post/<post_id>` Model: - `/<service>/user/<user_id>` Search: - `/search?q=...` Cyberdrop https://cyberdrop.cr cyberdrop.\* cyberdrop.cr cyberdrop.me cyberdrop.to k1-cd.cdn.gigachad-cdn.ru Album: - `/a/<album_id>` Direct links: - `/api/file/d/<file_id>` File: - `/e/<file_id>` - `/f/<file_id>` Cyberfile https://cyberfile.me cyberfile.\* Files: - `/<file_id>` - `/<file_id>/<file_name>` Public Folders: - `/folder/<folder_id>` - `/folder/<folder_id>/<folder_name>` Shared folders: - `/shared/<share_key>` DesiVideo https://desivideo.net desivideo.net Search: - `/search?s=<query>` Video: - `/videos/<video_id>/...` DirtyShip https://dirtyship.com dirtyship.\* Category: - `/category/...` Gallery: - `/gallery/...` Photo: - `/gallery/.../...` Tag: - `/tag/...` Video: - `/<video_name>` Discourse ::GENERIC CRAWLER:: Attachments: - `/uploads/...` Topic: - `/t/<topic_name>/<topic_id>` - `/t/<topic_name>/<topic_id>/<post_number>` **NOTES** - If the URL includes <post\_number>, posts with a number lower that it won't be scraped DoodStream https://doodstream.com all3do.com do7go.com dood.re dood.yt doodcdn. _doodstream._ doodstream.co vidply.com Video: - `/e/<video_id>` Dropbox https://www.dropbox.com dropbox.\* File: - `/s/...` - `/scl/fi/<link_key>?rlkey=...` - `/scl/fo/<link_key>/<secure_hash>?preview=<filename>&rlkey=...` Folder: - `/scl/fo/<link_key>/<secure_hash>?rlkey=...` - `/sh/...` E-Hentai https://e-hentai.org e-hentai.\* Album: - `/g/...` File: - `/s/...` E621 https://e621.net e621.net Pools: - `/pools/...` Post: - `/posts/...` Tags: - `/posts?tags=...` eFukt https://efukt.com efukt.com Gif: - `/view.gif.php?id=<id>` Homepage: - `/` Photo: - `/pics/....` Series: - `/series/<series_name>` Video: - `/...` ePorner https://www.eporner.com eporner.\* Categories: - `/cat/...` Channels: - `/channel/...` Gallery: - `/gallery/...` Photo: - `/photo/...` Pornstar: - `/pornstar/...` Profile: - `/profile/...` Search: - `/search/...` Video: - `/<video_name>-<video-id>` - `/embed/<video_id>` - `/hd-porn/<video_id>` Erome https://www.erome.com erome.\* Album: - `/a/<album_id>` Profile: - `/<name>` Search: - `/search?q=<query>` Erome.fan https://erome.fan erome.fan Album: - `/a/<album_id>` Profile: - `/a/category/<name>` Search: - `/search/<query>` EveriaClub https://everia.club everia.club All Posts: - `/posts/` Category: - `/category/<category_slug>` Date Range: - `...?after=<date>` - `...?before=<date&after=<date>` - `...?before=<date>` Post: - `/<post_slug>/` Tag: - `/tag/<tag_slug>` **NOTES** - For `Date Range`, must be a valid iso 8601 date, ex: `2022-12-06`. `Date Range` can be combined with `Category`, `Tag` and `All Posts`. ex: To only download categories from a date range: , `/category/<category_slug>?before=<date>` F95Zone https://f95zone.to f95zone.\* Attachments: - `/(attachments|data|uploads)/...` Threads: - `/(thread|topic|tema|threads|topics|temas)/<thread_name_and_id>` - `/goto/<post_id>` - `/posts/<post_id>` **NOTES** - base crawler: Xenforo Fapello https://fapello.su fapello.\* Individual Post: - `/.../...` Model: - `/...` Fileditch https://fileditchfiles.me fileditch.\* Direct Links FilesVC https://files.vc files.vc Direct Links Flugel-Anime https://flugelanime.com flugel-anime.\* Any path: - `/...` Forums.plex.tv https://forums.plex.tv forums.plex.tv Attachments: - `/uploads/...` Topic: - `/t/<topic_name>/<topic_id>` - `/t/<topic_name>/<topic_id>/<post_number>` **NOTES** - If the URL includes <post\_number>, posts with a number lower that it won't be scraped FSIBlog https://fsiblog5.com fsiblog.club fsiblog.com fsiblog1.club fsiblog1.com fsiblog2.club fsiblog2.com fsiblog3.club fsiblog3.com fsiblog4.club fsiblog4.com fsiblog5.club fsiblog5.com Posts: - `/<category>/<title>` Search: - `?s=<query>` FuckingFast https://fuckingfast.co fuckingfast.co Direct links: - `/<file_id>` FuXXX https://fuxxx.com fuxxx.com fuxxx.tube Video: - `/embed/<video_id>/...` - `/videos/<video_id>/...` GirlsReleased https://www.girlsreleased.com girlsreleased.\* Model: - `/model/<model_id>/<model_name>` Set: - `/set/<set_id>` Site: - `/site/<site>` GoFile https://gofile.io gofile.\* Direct link: - `/download/<content_id>/<filename>` - `/download/web/<content_id>/<filename>` Folder / File: - `/d/<content_id>` **NOTES** - Use `password` as a query param to download password protected folders - ex: https://gofile.io/d/ABC654?password=1234 GoogleDrive https://drive.google.com docs.google drive.google drive.usercontent.google.com Docs: - `/document/d/<file_id>` Files: - `/file/d/<file_id>` Folders: - `/drive/folders/<folder_id>` - `/embeddedfolderview/<folder_id>` Sheets: - `/spreadsheets/d/<file_id>` Slides: - `/presentation/d/<file_id>` **NOTES** - You can download sheets, slides and docs in a custom format by using it as a query param. ex: https://docs.google.com/document/d/1ZzEzJbemBMPm46O2q5VcGNoPbqDu9AhhUc2djQbvbTY?format=ods Valid Formats: document: - docx (default) - epub - md - odt - pdf - rtf - txt - zip presentation: - odp - pptx (default) spreadsheets: - csv - html - ods - tsv - xslx (default) GooglePhotos https://photos.google.com photos.app.goo.gl photos.google.com Album: - `/share/<album_id>` Photo: - `/album/<album_id>/photo/<photo_id>` **NOTES** - Only downloads 'optimized' images, NOT original quality - Can NOT download videos HClips https://hclips.com hclips.com hclips.tube privatehomeclips.com privatehomeclips.tube Video: - `/embed/<video_id>/...` - `/videos/<video_id>/...` HDZog https://hdzog.com hdzog.com hdzog.tube Video: - `/embed/<video_id>/...` - `/videos/<video_id>/...` Hianime https://hianime.to aniwatch.to aniwatchtv.to hianime.to zoro.to Anime: - `/<name>-<anime_id>` Episode: - `/<name>-<anime_id>?ep=<episode_id>` - `/watch/<name>-<anime_id>?ep=<episode_id>` **NOTES** - You can select the language to be downloaded by using a 'lang' query param. Valid options: 'sub' or 'dub'. Default: 'sub'If the chosen language is not available, CDL will use the first one available Hitomi.la https://hitomi.la hitomi.la Collection: - `/artist/...` - `/character/...` - `/group/...` - `/series/...` - `/tag/...` - `/type/...` Gallery: - `/anime/...` - `/cg/...` - `/doujinshi/...` - `/galleries/...` - `/gamecg/...` - `/imageset/...` - `/manga/...` - `/reader/...` Search: - `/search.html?<query>` HotLeaksTV https://hotleaks.tv hotleaks.tv Model: - `/<model_id>` Video: - `/<model_id>/video/<video_id>` HotLeakVip https://hotleak.vip hotleak.vip Model: - `/<model_id>` Video: - `/<model_id>/video/<video_id>` HotMovs https://hotmovs.com hotmovs.com hotmovs.tube Video: - `/embed/<video_id>/...` - `/videos/<video_id>/...` HotPic https://hotpic.cc 2385290.xyz hotpic.\* Album: - `/album/...` Image: - `/i/...` Iceyfile https://iceyfile.com iceyfile.\* Files: - `/<file_id>` - `/<file_id>/<file_name>` Public Folders: - `/folder/<folder_id>` - `/folder/<folder_id>/<folder_name>` Shared folders: - `/shared/<share_key>` ImageBam https://www.imagebam.com imagebam.\* Gallery: - `/gallery/<id>` Gallery or Image: - `/view/<id>` Image: - `/image/<id>` - `images<x>.imagebam.com/<id>` Thumbnails: - `thumbs<x>.imagebam.com/<id>` ImagePond https://imagepond.net imagepond.net Album: - `/a/<id>` - `/a/<name>.<id>` - `/album/<id>` - `/album/<name>.<id>` Category: - `/category/<name>` Direct Links Image: - `/image/<id>` - `/image/<name>.<id>` - `/img/<id>` - `/img/<name>.<id>` Profile: - `/<user_name>` Video: - `/video/<id>` - `/video/<name>.<id>` - `/videos/<id>` - `/videos/<name>.<id>` ImgBB https://ibb.co ibb.co imgbb.co Album: - `/album/<album_id>` Image: - `/<image_id>` Profile: - `<user_name>.imgbb.co/` ImgBox https://imgbox.com imgbox.\* Album: - `/g/...` Direct Links Image: - `/...` ImgLike https://imglike.com imglike.com Album: - `/a/<id>` - `/a/<name>.<id>` - `/album/<id>` - `/album/<name>.<id>` Category: - `/category/<name>` Direct Links Image: - `/image/<id>` - `/image/<name>.<id>` - `/img/<id>` - `/img/<name>.<id>` Profile: - `/<user_name>` Video: - `/video/<id>` - `/video/<name>.<id>` - `/videos/<id>` - `/videos/<name>.<id>` Imgur https://imgur.com imgur.\* Album: - `/a/<album_id>` Direct links: - `https://i.imgur.com/<image_id>.<ext>` Gallery: - `/gallery/<slug>-<album_id>` Image: - `/<image_id>` Imx.to https://imx.to imx.to Gallery: - `/g/<gallery_id>` Image: - `/i/...` - `/u/i/...` Thumbnail: - `/t/...` - `/u/t/` IncestFlix https://www.incestflix.com incestflix.\* Tag: - `/tag/...` Video: - `/watch/...` InfluencerBitches https://influencerbitches.com influencerbitches.\* Model: - `/model/...` InPorn https://inporn.com inporn.com inporn.tube Video: - `/embed/<video_id>/...` - `/videos/<video_id>/...` JPG5 https://jpg6.su host.church jpeg.pet jpg.church jpg.fish jpg.fishing jpg.homes jpg.pet jpg1.su jpg2.su jpg3.su jpg4.su jpg5.su jpg6.su jpg7.cr selti-delivery.ru Album: - `/a/<id>` - `/a/<name>.<id>` - `/album/<id>` - `/album/<name>.<id>` Category: - `/category/<name>` Direct Links Image: - `/image/<id>` - `/image/<name>.<id>` - `/img/<id>` - `/img/<name>.<id>` Profile: - `/<user_name>` Kemono https://kemono.cr kemono.cr kemono.party kemono.su Direct links: - `/data/...` - `/thumbnail/...` Discord Server: - `/discord/<server_id>` Discord Server Channel: - `/discord/server/<server_id>/<channel_id>#...` Favorites: - `/account/favorites/posts|artists` - `/favorites?type=post|artist` Individual Post: - `/<service>/user/<user_id>/post/<post_id>` Model: - `/<service>/user/<user_id>` Search: - `/search?q=...` Koofr https://koofr.eu k00.fr koofr.eu koofr.net Public Share: - `/links/<content_id>` - `https://k00.fr/<short_id>` LeakedModels https://leakedmodels.com/forum leakedmodels.\* Attachments: - `/(attachments|data|uploads)/...` Threads: - `/(thread|topic|tema|threads|topics|temas)/<thread_name_and_id>` - `/goto/<post_id>` - `/posts/<post_id>` **NOTES** - base crawler: Xenforo LeakedZone https://leakedzone.com leakedzone.\* Model: - `/<model_id>` Video: - `/<model_id>/video/<video_id>` Luscious https://members.luscious.net luscious.\* Album: - `/albums/...` LuxureTV https://luxuretv.com luxuretv.\* Search: - `/searchgate/videos/<search>/...` Video: - `/videos/<name>-<id>.html` Masahub https://masahub.com lol49.com masa49.com masafun.net masahub.com masahub2.com vido99.com Search: - `?s=<query>` Videos: - `/title` Mediafire https://www.mediafire.com mediafire.\* File: - `/file/<quick_key>` - `?<quick_key>` Folder: - `/folder/<folder_key>` Megacloud https://megacloud.blog megacloud.\* Embed v3: - `/embed-2/v3` MegaNz https://mega.nz mega.co.nz mega.io mega.nz File: - `/!#<file_id>!<share_key>` - `/file/<file_id>#<share_key>` - `/folder/<folder_id>#<share_key>/file/<file_id>` Folder: - `/F!#<folder_id>!<share_key>` - `/folder/<folder_id>#<share_key>` Subfolder: - `/folder/<folder_id>#<share_key>/folder/<subfolder_id>` **NOTES** - Downloads can not be resumed. Partial downloads will always be deleted and new downloads will start over MissAV https://missav.ws missav.\* Genres: - `/genres/<genre>` Labels: - `/labels/<label>` Makers: - `/makers/<maker>` Search: - `/search/<search>` Tags: - `/tags/<tag>` Video: - `/...` MixDrop https://mixdrop.sb mixdrop. _mxdrop._ File: - `/e/<file_id>` - `/f/<file_id>` Motherless https://motherless.com motherless.\* Group: - `/g/<group_name>` - `/gi/<image>` - `/gv/<video>` Image: - `/...` User: - `/f/...` - `/u/...` Video: - `pending` **NOTES** - Galleries are NOT supported MyDesi https://lolpol.com fry99.com lolpol.com mydesi.net Search: - `/search/<query>` Videos: - `/title` Nekohouse https://nekohouse.su nekohouse.\* Direct links: - \`/(data nHentai https://nhentai.net nhentai.net Collections: - `artist` - `character` - `favorites` - `group` - `parody` - `search` - `tag` Gallery: - `/g/<gallery_id>` NoodleMagazine https://noodlemagazine.com noodlemagazine.\* Search: - `/video/<search_query` Video: - `/watch/<video_id>` nsfw.xxx https://nsfw.xxx nsfw.xxx Category: - `/category/<name>` Post: - `/post/<id>` Search: - `/search?q=<query>` Subreddit: - `/r/<subreddit>` User: - `/user/<username>` NudoStar https://nudostar.com/forum nudostar.\* Attachments: - `/(attachments|data|uploads)/...` Threads: - `/(thread|topic|tema|threads|topics|temas)/<thread_name_and_id>` - `/goto/<post_id>` - `/posts/<post_id>` **NOTES** - base crawler: Xenforo NudoStarTV https://nudostar.tv nudostar.tv Model: - `/models/...` ok.ru https://ok.ru odnoklassniki.ru ok.ru Channel: - `/profile/<username>/c<channel_id>` - `/video/c<channel_id>` Video: - `/video/<video_id>` OmegaScans https://omegascans.org omegascans.\* Chapter: - `/series/.../...` Direct Links Series: - `/series/...` OneDrive https://onedrive.com 1drv.ms onedrive.live.com Access Link: - `https://onedrive.live.com/?authkey=<KEY>&id=<ID>&cid=<CID>` Share Link (anyone can access): - `https://1drv.ms/b/<KEY>` - `https://1drv.ms/f/<KEY>` - `https://1drv.ms/t/<KEY>` - `https://1drv.ms/u/<KEY>` pCloud https://www.pcloud.com e.pc.cd pc.cd pcloud.\* Public File or folder: - `?code=<share_code>` - `e.pc.cd/<short_code>` - `u.pc.cd/<short_code>` PimpAndHost https://pimpandhost.com pimpandhost.\* Album: - `/album/...` Image: - `/image/...` PimpBunny https://pimpbunny.com pimpbunny.com Album: - `/albums/<album_name>` Category: - `/categories/<category>` Model Albums: - `/albums/models/<model_name>` Models: - `/onlyfans-models/<model_name>` Tag: - `/tags/<tag>` Videos: - `/videos/...` PixelDrain https://pixeldrain.com pd.1drv.eu.org pd.cybar.xyz pixeldra.in pixeldrain.com pixeldrain.net File: - `/api/file/<file_id>` - `/l/<list_id>#item=<file_index>` - `/u/<file_id>` Filesystem: - `/api/filesystem/<path>...` - `/d/<id>` Folder: - `/api/list/<list_id>` - `/l/<list_id>` **NOTES** - text files will not be downloaded but their content will be parsed for URLs PixHost https://pixhost.to pixhost.org pixhost.to Gallery: - `/gallery/<gallery_id>` Image: - `/show/<image_id>` Thumbnail: - `/thumbs/..` Pkmncards https://pkmncards.com pkmncards.\* Card: - `/card/...` Series: - `/series/...` Set: - `/set/...` PMVHaven https://pmvhaven.com pmvhaven.\* Playlist: - `/playlists/...` Search results: - `/search/...` Users: - `/profile/...` - `/users/...` Video: - `/video/...` PornHub https://www.pornhub.com pornhub.\* Album: - `/album/...` Channel: - `/channel/...` Gif: - `/gif/...` Photo: - `/photo/...` Playlist: - `/playlist/...` Profile: - `/model/...` - `/pornstar/...` - `/user/...` Video: - `/embed/<video_id>` - `/view_video.php?viewkey=<video_id>` PornPics https://pornpics.com pornpics.\* Categories: - `/categories/....` Channels: - `/channels/...` Direct Links Gallery: - `/galleries/...` Pornstars: - `/pornstars/...` Search: - `/?q=<query>` Tags: - `/tags/...` Porntrex https://www.porntrex.com porntrex.\* Album: - `/albums/...` Category: - `/categories/...` Model: - `/models/...` Playlist: - `/playlists/...` Search: - `/search/...` Tag: - `/tags/...` User: - `/members/...` Video: - `/video/...` PornZog https://pornzog.com pornzog.\* Video: - `/video/...` PostImg https://postimg.cc postimages.org postimg.cc postimg.org Album: - `/gallery/<album_id>/...` Direct links: - `i.postimg.cc/<image_id>/...` Image: - `/<image_id>/...` Ranoz.gg https://ranoz.gg qiwi.gg ranoz.gg File: - `/d/<file_id>` - `/file/<file_id>` RealBooru https://realbooru.com realbooru.\* File: - `?id=...` Tags: - `?tags=...` Reddit https://www.reddit.com redd.it reddit.\* Direct Links Subreddit:: - `/r/<subreddit>` User: - `/u/<user>` - `/user/<user>` - `/user/<user>/...` RedGifs https://www.redgifs.com redgifs.\* Embeds: - `/ifr/<gif_id>` Gif: - `/watch/<gif_id>` Image: - `/i/<image_id>` User: - `/users/<user>` Rootz.so https://www.rootz.so rootz.so File: - `/d/<file_id>` - `/file/<file_id>` Rule34Vault https://rule34vault.com rule34vault.\* Playlist: - `/playlists/view/...` Post: - `/post/...` Tag: - `/...` Rule34Video https://rule34video.com rule34video.\* Members: - `/members/...` Models: - `/models/...` Search: - `/search/...` Tags: - `/tags/...` Video: - `/video/<id>/<name>` - `/videos/<id>/<name>` Rule34XXX https://rule34.xxx rule34.xxx File: - `?id=...` Tag: - `?tags=...` Rule34XYZ https://rule34.xyz rule34.xyz Playlist: - `/playlists/view/...` Post: - `/post/...` Tag: - `/...` Rumble https://rumble.com rumble.\* Channel: - `/c/<name>` Embed: - `/embed/<video_id>` User: - `/user/<name>` Video: - `<video_id>-<video-title>.html` Safe.Soul https://safe.soul.lol safe.soul.lol Album: - `/a/<album_id>` File: - `/<file_id>` Scrolller https://scrolller.com scrolller.\* Subreddit: - `/r/...` SendNow https://send.now send.now Direct Links SendVid https://sendvid.com sendvid.\* Direct Links Embeds: - `/embed/...` Videos: - `/...` Sex.com https://sex.com sex.\* Shorts Profiles: - `/shorts/<profile>` SocialMediaGirls https://forums.socialmediagirls.com socialmediagirls.\* Attachments: - `/(attachments|data|uploads)/...` Threads: - `/(thread|topic|tema|threads|topics|temas)/<thread_name_and_id>` - `/goto/<post_id>` - `/posts/<post_id>` **NOTES** - base crawler: Xenforo SpankBang https://spankbang.com spankbang.\* Playlist: - `/playlist/<playlist-id>` Video: - `/embed/<video_id>` - `/play/<video_id>` - `/playlist/<playlist-id>-<video_id>` - `/video/<video_id>` Streamable https://streamable.com streamable.\* Video: - `/...` Streamtape https://streamtape.com streamtape.com Player: - `/e/<video_id>` Videos: - `/v/<video_id>` ThisVid https://thisvid.com thisvid.\* Albums: - `/albums/<album_name>` Categories: - `/categories/...` Image: - `/albums/<album_name>/<image_name>` Members: - `/members/<member_id>` Search: - `/search/?q=...` Tags: - `/tags/...` Videos: - `/videos/...` ThotHub https://thothub.to thothub.\* Albums: - `/albums/<album_id>/<album_name>` Videos: - `/videos/...` TikTok https://www.tiktok.com tiktok.\* Photo: - `/@<user>/photo/<photo_id>` User: - `/@<user>` Video: - `/@<user>/video/<video_id>` TitsInTops https://titsintops.com/phpBB2 titsintops.\* Attachments: - `/(attachments|data|uploads)/...` Threads: - `/(thread|topic|tema|threads|topics|temas)/<thread_name_and_id>` - `/goto/<post_id>` - `/posts/<post_id>` **NOTES** - base crawler: Xenforo TNAFlix https://www.tnaflix.com tnaflix.\* Channel: - `/channel/...` Profile: - `/profile/...` Search: - `/search?what=<query>` Video: - `/<category>/<title>/video<video_id>` Tokyomotion https://www.tokyomotion.net tokyomotion.\* Albums: - `/album/<album_id>` - `/user/<user>/albums/` Photo: - `/photo/<photo_id>` - `/user/<user>/favorite/photos` Playlist: - `/user/<user>/favorite/videos` Profiles: - `/user/<user>` Search Results: - `/search?...` Video: - `/video/<video_id>` Toonily https://toonily.com toonily.\* Chapter: - `/serie/<name>/chapter-<chapter-id>` Serie: - `/serie/<name>` Tranny.One https://www.tranny.one tranny.one Album: - `/pics/album/<album_id>` Pornstars: - `/pornstar/<model_id>/<model_name>` Search: - `/search/<search_query>` Video: - `/view/<video_id>` Transfer.it https://transfer.it transfer.it Transfer: - `/t/<transfer_id>` TransFlix https://transflix.net transflix.\* Search: - `/search/?q=...` Video: - `/video/<name>-<video_id>` TubePornClassic https://tubepornclassic.com tubepornclassic.com tubepornclassic.tube Video: - `/embed/<video_id>/...` - `/videos/<video_id>/...` TurboVid https://turbovid.cr saint.to saint2.cr saint2.su turbo.cr turbovid.cr Album: - `/a/<album_id>` Direct links: - `/data/...` Search: - `library?q=<query>` Video: - `/d/<file_id>` - `/embed/<file_id>` - `/v/<file_id>` Twitch https://www.twitch.tv twitch.\* Clip: - `/<user>/clip/<slug>` - `/embed?clip=<slug>` - `https://clips.twitch.tv/<slug>` Collection: - `/collections/<collection_id>` VOD: - `/<user>/v/<vod_id>` - `/video/<vod_id>` - `/videos/<vod_id>` - `?video=<vod_id>` TwitterImages https://twimg.com twimg.\* Photo: - `/...` TWPornStars https://www.twpornstars.com indiantw.com twanal.com twgaymuscle.com twgays.com twlesbian.com twmilf.com twonfans.com twpornstars.com twteens.com twtiktoks.com Photo: - `/...` TXXX https://txxx.com txxx.com txxx.tube videotxxx.com videotxxx.tube Video: - `/embed/<video_id>/...` - `/videos/<video_id>/...` UPornia https://upornia.com upornia.com upornia.tube Video: - `/embed/<video_id>/...` - `/videos/<video_id>/...` ViperGirls https://vipergirls.to viper.click vipergirls.to Threads: - `/goto/<post_id>` - `/posts/<post_id>` - `/threads/<thread_name>` Vipr.im https://vipr.im vipr.im Image: - `/...` Thumbnail: - `/th/...` VJav https://vjav.com vjav.com vjav.tube Video: - `/embed/<video_id>/...` - `/videos/<video_id>/...` Voe.sx https://voe.sx alejandrocenturyoil.com diananatureforeign.com heatherwholeinvolve.com jennifercertaindevelopment.com jilliandescribecompany.com jonathansociallike.com mariatheserepublican.com maxfinishseveral.com nathanfromsubject.com richardsignfish.com robertordercharacter.com sarahnewspaperbeat.com voe.sx Embed: - `/e/video_id` VoyeurHit https://voyeurhit.com voyeurhit.com voyeurhit.tube Video: - `/embed/<video_id>/...` - `/videos/<video_id>/...` VXXX https://vxxx.com vxxx.com vxxx.tube Video: - `/video-<video-id>` WeTransfer https://wetransfer.com we.tl wetransfer.com Direct links: - `download.wetransfer.com/...` Public link: - `wetransfer.com/downloads/<file_id>/<security_hash>` Share Link: - `wetransfer.com/downloads/<file_id>/<recipient_id>/<security_hash>` Short Link: - `we.tl/<short_file_id>` WordPressHTML ::GENERIC CRAWLER:: All Posts: - `/posts/` Category: - `/category/<category_slug>` Date Range: - `...?after=<date>` - `...?before=<date&after=<date>` - `...?before=<date>` Post: - `/<post_slug>/` Tag: - `/tag/<tag_slug>` **NOTES** - For `Date Range`, must be a valid iso 8601 date, ex: `2022-12-06`. `Date Range` can be combined with `Category`, `Tag` and `All Posts`. ex: To only download categories from a date range: , `/category/<category_slug>?before=<date>` WordPressMedia ::GENERIC CRAWLER:: All Posts: - `/posts/` Category: - `/category/<category_slug>` Date Range: - `...?after=<date>` - `...?before=<date&after=<date>` - `...?before=<date>` Post: - `/<post_slug>/` Tag: - `/tag/<tag_slug>` **NOTES** - For `Date Range`, must be a valid iso 8601 date, ex: `2022-12-06`. `Date Range` can be combined with `Category`, `Tag` and `All Posts`. ex: To only download categories from a date range: , `/category/<category_slug>?before=<date>` Xasiat https://www.xasiat.com xasiat.\* Album: - `/albums/<id>/<name>` Images: - `/get_image/...` Videos: - `/videos/<id>/<name>` XBunker https://xbunker.nu xbunker.\* Attachments: - `/(attachments|data|uploads)/...` Threads: - `/(thread|topic|tema|threads|topics|temas)/<thread_name_and_id>` - `/goto/<post_id>` - `/posts/<post_id>` **NOTES** - base crawler: Xenforo XBunkr https://xbunkr.com xbunkr.\* Albums: - `/a/...` Direct Links XGroovy https://xgroovy.com xgroovy.\* Channel: - `/<category>/channels/...` - `/channels/...` Gif: - `/<category>/gifs/<gif_id>/...` - `/gifs/<gif_id>/...` Images: - `/<category>/photos/<photo_id>/...` - `/photos/<photo_id>/...` Pornstar: - `/<category>/pornstars/<pornstar_id>/...` - `/pornstars/<pornstar_id>/...` Search: - `/<category>/search/...` - `/search/...` Tag: - `/<category>/tags/...` - `/tags/...` Video: - `/<category>/videos/<video_id>/...` - `/videos/<video_id>/...` xHamster https://xhamster.com xhamster.\* Creator: - `/creators/<creator_name>` Creator Galleries: - `/creators/<creator_name>/photos` Creator Videos: - `/creators/<creator_name>/exclusive` Gallery: - `/photos/gallery/<gallery_name_or_id>` User: - `/users/<user_name>` - `/users/profiles/<user_name>` User Galleries: - `/users/<user_name>/photos` User Videos: - `/users/<user_name>/videos` Video: - `/videos/<title>` XMilf https://xmilf.com xmilf.com xmilf.tube Video: - `/embed/<video_id>/...` - `/videos/<video_id>/...` xVideos https://www.xvideos.com xv-ru.com xvideos-ar.com xvideos-india.com xvideos.com xvideos.es Account: - `/<channel_name>` - `/amateur|amateur-channels|amateurs|channel|channel-channels|channels|pornstar|pornstar-channels|pornstars|profile|profile-channels|profiles/<name>` Account Photos: - `/<channel_name>#_tabPhotos` - `/<channel_name>/photos/...` - `/amateur|amateur-channels|amateurs|channel|channel-channels|channels|pornstar|pornstar-channels|pornstars|profile|profile-channels|profiles/<name>#_tabPhotos` - `/amateur|amateur-channels|amateurs|channel|channel-channels|channels|pornstar|pornstar-channels|pornstars|profile|profile-channels|profiles/<name>/photos/...` Account Quickies: - `/<channel_name>#quickies` - `/amateur|amateur-channels|amateurs|channel|channel-channels|channels|pornstar|pornstar-channels|pornstars|profile|profile-channels|profiles/<name>#quickies` Account Videos: - `/<channel_name>#_tabVideos` - `/amateur|amateur-channels|amateurs|channel|channel-channels|channels|pornstar|pornstar-channels|pornstars|profile|profile-channels|profiles/<name>#_tabVideos` Video: - `/amateur|amateur-channels|amateurs|channel|channel-channels|channels|pornstar|pornstar-channels|pornstars|profile|profile-channels|profiles#quickies/(a|h|v)/<video_id>` - `/video.<encoded_id>/<title>` - `/video<id>/<title>` XXXBunker https://xxxbunker.com xxxbunker.\* Category: - `/categories/<category>` Search: - `/search/<video_id>` User Favorites: - `/<username>/favoritevideos` Video: - `/<video_id>` YandexDisk https://disk.yandex.com.tr disk.yandex yadi.sk File: - `/d/<folder_id>/<file_name>` - `/i/<file_id>` Folder: - `/d/<folder_id>` **NOTES** - Does NOT support nested folders YouJizz https://www.youjizz.com youjizz.\* Video: - `/videos/<video_name>` - `/videos/embed/<video_id>` [PreviousSimpCity Support Droppedchevron-left](https://script-ware.gitbook.io/cyberdrop-dl/dev/simpcity-support-dropped) [NextConfiguration Optionschevron-right](https://script-ware.gitbook.io/cyberdrop-dl/dev/reference/configuration-options) Last updated 1 month ago * [Password Protected Content Hosts](https://script-ware.gitbook.io/cyberdrop-dl/dev/reference/supported-websites#password-protected-content-hosts) * [Additional Content Hosts with Real-Debrid](https://script-ware.gitbook.io/cyberdrop-dl/dev/reference/supported-websites#additional-content-hosts-with-real-debrid) * [Supported sites](https://script-ware.gitbook.io/cyberdrop-dl/dev/reference/supported-websites#supported-sites) sun-brightdesktopmoon sun-brightdesktopmoon ---