# Table of Contents - [About DolphinDB](#about-dolphindb) - [Database](#database) - [Feature Settings](#feature-settings) - [Programming Guide](#programming-guide) - [Getting Started](#getting-started) - [Migration](#migration) - [Maintenance](#maintenance) - [Documentation](#documentation) - [Deployment](#deployment) - [Troubleshooting](#troubleshooting) - [Distributed Architecture](#distributed-architecture) - [Function References](#function-references) - [Streaming](#streaming) - [GUI Clients](#gui-clients) - [Tutorials](#tutorials) - [Web Interface](#web-interface) - [Data Types and Data Forms](#data-types-and-data-forms) - [Data Partitioning](#data-partitioning) - [API & Connector](#api-connector) - [Plugins](#plugins) - [Multi-Model Database](#multi-model-database) - [Catalog](#catalog) - [Distributed Database Overview](#distributed-database-overview) - [S12009](#s12009) - [Release Notes](#release-notes) - [Data Acquisition Platform](#data-acquisition-platform) - [Database Operations](#database-operations) - [Inter-node Communication Protocols](#inter-node-communication-protocols) - [Cluster Deployment and Upgrade](#cluster-deployment-and-upgrade) - [DolphinDB Python Parser](#dolphindb-python-parser) - [Standalone Deployment and Upgrade](#standalone-deployment-and-upgrade) - [Configuration](#configuration) - [Prepare a Linux Machine for Deployment](#prepare-a-linux-machine-for-deployment) - [Visual Studio Code Extension](#visual-studio-code-extension) - [Tiered Storage](#tiered-storage) - [DolphinDB GUI](#dolphindb-gui) - [TSDB Storage Engine](#tsdb-storage-engine) - [Primary Key Storage Engine](#primary-key-storage-engine) - [pipeline](#pipeline) - [SQL Reference](#sql-reference) - [VectorDB](#vectordb) - [OLAP Storage Engine](#olap-storage-engine) - [Standalone Deployment on Docker](#standalone-deployment-on-docker) - [compress](#compress) - [mr](#mr) - [Programming Guide](#programming-guide) - [Shell](#shell) - [Getting Started](#getting-started) - [Configuration](#configuration) - [streamEngineParser](#streamengineparser) - [Querying Tables](#querying-tables) - [Cluster](#cluster) - [Job](#job) - [Quick Start](#quick-start) - [Quick Start](#quick-start) - [Complex Event Processing](#complex-event-processing) - [Stream](#stream) - [Stream for DolphinDB](#stream-for-dolphindb) - [R API](#r-api) - [JavaScript API](#javascript-api) - [Application in Streaming](#application-in-streaming) - [ops](#ops) - [create](#create) - [Log](#log) - [Historical Data Replay](#historical-data-replay) - [Technical Analysis Indicator Library](#technical-analysis-indicator-library) - [CEP Streaming Engine](#cep-streaming-engine) - [Unknown](#unknown) - [Import Data](#import-data) - [Grafana Datasource](#grafana-datasource) - [Trading Calendar](#trading-calendar) - [Overview](#overview) - [context by](#context-by) - [From kdb+ to DolphinDB](#from-kdb-to-dolphindb) - [Modules](#modules) - [WorldQuant 101 Alphas](#worldquant-101-alphas) - [Fundamentals](#fundamentals) - [group by](#group-by) - [pivot by](#pivot-by) - [Unified Batch and Stream Processing](#unified-batch-and-stream-processing) - [Performance Monitoring](#performance-monitoring) - [Single Sign-On](#single-sign-on) - [Graceful Shutdown](#graceful-shutdown) - [Backup and Restore](#backup-and-restore) - [Node Crashes](#node-crashes) - [Node Startup Exception](#node-startup-exception) - [C# API](#c-api) - [Join Engines](#join-engines) - [Batch Job Management](#batch-job-management) - [Error Code Reference](#error-code-reference) - [accumulate](#accumulate) - [Historical Data Replay](#historical-data-replay) - [Streaming Engines](#streaming-engines) - [Web API](#web-api) - [Server Connection Issues](#server-connection-issues) - [System Freeze](#system-freeze) - [Scale out a DolphinDB Cluster](#scale-out-a-dolphindb-cluster) - [JDBC Connector](#jdbc-connector) - [Functions](#functions) --- # About DolphinDB [Jump to main content](#wh_topic_body) About DolphinDB =============== Welcome to DolphinDB documentation! DolphinDB is a real-time platform for analytics and stream processing, powered by a high-performance time series database. It offers capabilities related to efficient writes, fast queries, complex analysis, distributed parallel computing, and low-latency stream computing. It also supports high availability. The following figure illustrates the architecture of the DolphinDB system: ![](images/getting_started.png) Figure 1. DolphinDB Architecture Distributed Architecture ------------------------ * [**Distributed File System**](Tutorials/database.html) : Data is logically distributed across data nodes, with a controller managing all chunk metadata, including chunk and replica information, chunk version numbers, etc. This distributed system simplifies data management and enhances fault tolerance and scalability to ensure high data availability and continuous execution of computing tasks. * **Various Scaling Options**: A DolphinDB cluster can be scaled online and offline by adding/removing more nodes or increasing/decreasing resources for a single node. DolphinDB also provides seamless and efficient data migration and rebalancing to ensure the even distribution of chunk replicas across a cluster after scaling. * **Distributed Transaction**: DolphinDB guarantees [ACID](Tutorials/database.html#52-transactions) (atomicity, consistency, isolation, and durability) and supports snapshot isolation. * **High Availability**: To maintain stable service and business continuity against single-node failure, DolphinDB offers high availability solutions for controllers, data nodes, and clients. * **Comprehensive Backup and Recovery**: The backup and recovery mechanism is flexible and can be tailored to specific needs, ensuring data security. * **Cluster Replication**: With low latency, high throughput, and high fault tolerance, this solution enables asynchronous data replication across clusters, enhancing datasecurity. Storage ------- * DolphinDB supports various storage engines, including TSDB, OLAP, PKEY, IMOLTP, and VectorDB, each tailored for different application scenarios: * [**TSDB**](Database/Multi-Model%20Database/tsdb_storage_engine.html) : Utilizes the PAX model for row-column hybrid storage, delivering significant performance for point queries. * [**OLAP**](Database/Multi-Model%20Database/olap_storage_engine.html) : Utilizes columnar storage, which is more suitable for aggregate computations over long time spans. * [**PKEY**](Database/Multi-Model%20Database/primary_key_storage_eng.html) : Ensures uniqueness of primary key and supports real-time updates and efficient queries. The PKEY engine is specifically engineered to seamlessly integrate with OLTP (Online Transaction Processing) systems based on CDC (change data capture). * **IMOLTP**: Organizes the in-memory data in rows, supports transactions, and handles high-frequency, high-concurrency updates and queries using B+ tree indexing. * [**VECTORDB**](Database/Multi-Model%20Database/vectordb.html) : Supports vector indexing and enables fast approximate nearest neighbor searches (ANNS), catering to large-scale vector data retrieval and response needs. * Supports various lossless data [compression](Functions/c/compress.html) algorithms, including LZ4, Delta-of-delta, Zstandard, Chimp, and dictionary compression, with a compression ratio of 4:1 to 10:1. * Supports [tiered storage](Tutorials/tiered_storage.html) , which stores hot and cold data in different ways to reduce storage costs. Batch Processing ---------------- * **Rich Function Library**: DolphinDB boasts over 2,000 built-in functions across various domains, covering a wide range of data processing needs. It also supports user-defined functions, helping users tackle complex application scenarios. * **Diverse Computing Models**: DolphinDB’s distributed framework integrates a variety of computing models, including [pipeline](Functions/p/pipeline.html) , [Map-Reduce](Functions/m/mr.html) , and iterative computing. * **Multi-Diagram Programming**: DolphinDB seamlessly combines SQL with functions and expressions to support complex data analysis directly within the database, significantly improving data processing speed and efficiency. * **[Vectorized Programming](Tutorials/DolphinDB_Programming_Guide.html) **: Vectorization converts loops into parallel operations, processing multiple data elements simultaneously through single CPU instructions. This is particularly suitable for time-series data represented with a vector, reducing the cost of script interpreting and optimizing numerous algorithms. * **Multi-Machine and Multi-Core CPU Resources**: DolphinDB fully utilizes the CPU resources and achieves efficient parallel processing of massive data. Stream Processing ----------------- * DolphinDB supports publishing and subscribing to [stream tables](Tutorials/streaming_tutorial.html) . * **Streaming Engines**: DolphinDB integrates various streaming engines and built-in functions/operators for complex business needs. Powerful streaming pipelines can be built manually by calling these engines or automatically by using the [streamEngineParser](Functions/s/streamEngineParser.html) . * [**Historical Data Replay**](Tutorials/historical_data_replay.html) : DolphinDB’s replay functionality allows simulating real-time data flows by replaying multiple data streams into a single synchronized timeline, which is essentially a way to backtest how trading strategies would have performed under past conditions. * **Unified Stream and Batch Processing**: Factors or functions written in batch processing of historical data can be used for real-time calculation in production, ensuring that the results of stream calculations are completely consistent with those of batch calculations. This feature not only greatly facilitates testing, validation, and retrospective analysis for users, but also significantly reduces the development costs of factor implementation. * **Real-Time Data Ingestion**: DolphinDB supports the ingestion of real-time streaming data from various sources, with automatic format conversion of the data. * [**Complex Event Processing (CEP) Engine**](Streaming/ComplexEventProcessing.html) : DolphinDB’s CEP engine enables users to capture specific events from the received real-time data streams and perform predefined actions on the events. * **Sub-Millisecond Latency**: DolphinDB boasts sub-millisecond data latency, ensuring optimal performance for real-time data processing. Multi-Paradigm Programming -------------------------- * **Turing-Complete Programming Language**: DolphinDB supports [multi-paradigm programming](Tutorials/DolphinDB_Programming_Guide.html) including imperative programming, functional programming, vector programming, and SQL programming. With concise syntax, flexible usage, and strong expressiveness, it contributes to a leap in development efficiency. * [**Support for SQL-92 Standard**](Programming/SQLStatements/SQLStatement.html#common-features-with-standard-sql) : DolphinDB further extends the SQL-92 syntax with additional features, such as `cgroup by` (for cumulative grouping calculations) and `pivot by` (for column/table rearrangement). It is also compatible with mainstream SQL databases such as Oracle and MySQL. * [**Python Parser**](Programming/py_par_intro.html) : As an interpreter for Python language, DolphinDB Python Parser supports some native Python objects (e.g., dict, list, tuple, set), syntax, and functionalities of the pandas library. Users can directly use Python syntax in DolphinDB clients to access and manipulate DolphinDB data. Comprehensive Ecosystem ----------------------- * **Rich [SDK](API/chap_api.html "API in different languages for users to call DolphinDB from their programs") and Client Ecosystem**: DolphinDB offers a wide range of APIs in different programming languages (e.g., [Python](https://docs.dolphindb.com/en/pydoc/chap1_quickstart_landingpage.html) , [C++](https://docs.dolphindb.com/en/cppdoc/chap1_quickstart_landingpage.html) , [Java](https://docs.dolphindb.com/en/javadoc/overview.html) , [C#](API/Csharp.html) , [R](API/R.html) , [JavaScript](API/JavaScript.html) , etc.) and clients (e.g., [Web Interface](Database/Clients/Web/web.html) , [VS Code Extension](Tutorials/vscode_extension.html) , Java GUI, DolphinDB terminal, etc.). * [**Various Plugins**](Plugins/chap_plugin.html "Guidance on the use of plugins and plugin development for multiple application scenarios") : DolphinDB supports plugins across different fields, covering data access, financial scenarios, message queues, numerical computing, machine learning, networking, cloud storage, format conversion, etc. * [**Modular Design**](Tutorials/modules.html) : DolphinDB provides various modules, including [technical analysis indicator library](Tutorials/ta.html) , [ops](Tutorials/ops.html) , [trading calendar](Tutorials/trading_calendar.html) , and [WorldQuant 101 alphas](Tutorials/wq101alpha.html) . It also enables users to save user-defined functions as reusable modules to simplify the scripts for function calling and optimize system maintenance efficiency. * **Third-Party Tools**: DolphinDB integrates third-party tools such as [Grafana](API/grafana_datasource.html) to connect various solutions for data transfer and visualization. --- # Database [Jump to main content](#wh_topic_body) Database ======== Core features of DolphinDB's distributed architecture, basic database operations, and configuration setup. --- # Feature Settings [Jump to main content](#wh_topic_body) Feature Settings ================ This page displays configurable features that users can choose to enable or disable based on their business needs. Currently, the following two features can be configured through this page: * **Guide for Financial Scenario**: Designed for financial clients. When this feature is enabled, an entrance for **Guide for Financial Scenario** will be displayed in the left panel. Through this entrance, users can access the page to create databases and tables that are commonly used in financial analyses. * **Guide for IoT Scenario**: Designed for IoT clients. Refer to **Guide for Financial Scenario**. The users' requirements for databases and tables are collected through the guides. Then, the corresponding scripts to create databases and tables will be generated, leading to the convenient and efficient creation of databases and tables. Guide for Financial Scenario ---------------------------- In financial scenarios, many factors (such as daily increments, etc.) can affect the parameter settings when creating databases and tables. This page includes common factors to help users configure parameters based on their needs, ensuring the creation of suitable databases and tables. The creation of databases and tables includes the following four steps: **Create Database**, **Create Table**, **Script Preview** and **Execution Result**. ### Create Database **Create Database** page is for collecting user requirements for databases and tables. On this page, users can choose to create tables under an existing database or add a new database to create tables. * **Using Existing Database**: Select the database name from the dropdown list and click **Next** to proceed to the **Create Table** page. Alternatively, you can click **Import Database Configuration** in the top right corner to fill in the page by importing a database configuration file. ![](../../../images/Web%20Interface/feature_1.png) * **Using Existing Database**: Select the database name from the dropdown list and click **Next** to proceed to the **Create Table** page. Alternatively, you can click **Import Database Configuration** in the top right corner to fill in the page by importing a database configuration file. ![](../../../images/Web%20Interface/feature_2.png) ### Create Table On this page, users can fill in the relevant information to generate the table creation script. Currently, there are two methods to fill in the information: * Enter text or select information directly on the interface to generate the database table creation script. * **Generate via CSV/TXT File**: Click the **Import Table** button to import a table file and generate the script. The table file can be stored on the server or locally. After completing the information, click the **Generate Script** button to proceed to the **Script Preview** page. Alternatively, click **Back** to return to the **Create Database** page. The following two images show the **Create Table** pages for **Using Existing Database** and **Creating New Database**. Most of the information on both pages is the same. The difference is that when creating a table under an existing database, the **Filter Columns** will be displayed at the bottom of the page. When creating a new database table, users need to select the time column and the symbol column (this column is displayed only if the daily increment exceeds 1 million rows) as partitioning columns. ![](../../../images/Web%20Interface/feature_3.png) ![](../../../images/Web%20Interface/feature_4.png) **Note:** The **Filter Columns** shown at the bottom of the page in the images above is displayed only when the storage engine is set to **TSDB**. Here, you can specify the columns that are frequently used as filter conditions in queries. The filter columns must meet the following conditions: * No more than 2 columns. * The data type must be enumerable, such as INT, STRING, SYMBOL, etc. ### Script Review This page is used to display the scripts generated through the aforementioned operations. Users can browse the scripts on this page but cannot modify the script content. If the script is correct, click **Execute Now** to create the table. You can also click **Export Table Configuration** to export the database table configuration information to a JSON file. If you need to modify the script, click **Back** to return to the **Create Table** page to make changes, or copy the script and modify the content. ![](../../../images/Web%20Interface/feature_5.png) Guide for IoT Scenario ---------------------- IoT data includes characteristics such as the number of measurements, acquisition frequency, and daily incrementX-axis. These data are usually stored in TSDB databases for efficient storage and query. Most of the features on the **Guide for IoT Scenario** are the same as those on the **Guide for Financial Scenario**. However, it offers two types of table creation pages based on the user's familiarity with the database: a basic version and an advanced version. ![](../../../images/Web%20Interface/feature_6.png) ### Basic version On the basic version page, the system provides default configuration information for storage, indexing, and other related details, reducing the difficulty for users to create databases and tables. Users provide the necessary information through the interface forms to generate the script, which can be executed to create the tables. ![](../../../images/Web%20Interface/feature_7.png) This section explains how to set **Filter Columns** in **Guide for IoT Scenario**. For other fields, refer to **Guide for Financial Scenario** instructions above. **Filter Columns** are columns frequently used as filter conditions in queries. Filter columns should be prioritized based on the importance of the filter conditions, with the most important filters listed first. Specify the filter columns according to the following criteria: * If the data is time-series data, or if the data is non-time-series but exceeds 2 million rows: the first filter column should be the time column, and the second should be the device ID column. * Other cases: select the device ID column. ### Advanced version Compared to the basic version, the advanced version requires users to customize more attributes. Users can define the storage engine type, partitioning columns, keeping duplicates, and other information on this page. In the advanced version, database and table information is filled out in two steps: Basic Information and Advanced Information. The method for filling out Basic Information is the same as described in the basic version and will not be repeated here. This section will provide detailed instructions on how to fill out the Advanced Information. ![](../../../images/Web%20Interface/feature_8.png) As shown in the image above, you need to select the storage engine, choose whether to allow concurrent writes to the same partition, set the partitioning columns, and define whether to keep duplicates. Refer to the following guidelines for each field: **Storage Engine:** Select the storage engine based on business characteristics. **Enable Concurrent Writes to the Same Partition:** If this feature is enabled, concurrent operations on the same partition are allowed. This can improve write speed but may result in a very small probability of data inconsistency. **Partitioning columns:** This configuration will recommend partitioning columns and data types based on the basic information. Users are advised to follow the recommended information when making selections. **Time Column:** When the TSDB engine is selected and the data is non-time-series with a total volume exceeding 2 million rows, this column must be chosen. **Filter Columns:** This configuration item will recommend common filter columns and data types based on the basic information. Users are advised to select no more than the recommended number of filter columns. **Keep Duplicates:** This configuration offers three strategies for handling data with the same sortColumn values in the same partition: ALL(retaining all data); LAST(retaining only the most recent data); FIRST(retaining only the first record). After completing the above configurations, click **Generate Script** to create the script. Alternatively, click **Back** to return to the information page to make modifications, or copy the script for further editing. --- # Programming Guide [Jump to main content](#wh_topic_body) Programming Guide ================= DolphinDB scripting language and SQL reference --- # Getting Started [Jump to main content](#wh_topic_body) Getting Started =============== This quick start guide offers a brief walkthrough on deploying the DolphinDB server, creating databases and tables, importing data, and performing queries. Preparations ------------ ### Download and Install 1. Download the DolphinDB server package on the [official website](https://dolphindb.com/product#downloads-top) . 2. Extract the package to a local directory. No further installation is required after extraction. ### Run DolphinDB Server The extracted _server_ directory contains files related to DolphinDB Web Interface, DolphinDB server, plugins, modules, and a license file. DolohinDB offers various deployment options. For details, refer to [Deployment](Deployment/chap_ddb_deployment.html "Deploy and upgrade DolphinDB on premise or in the cloud") . The following example introduces how to launch DolphinDB server in the standalone mode. Execute the startup command corresponding to the operating system. * Linux: Open a terminal and execute the following command line: cd path_to_DolphinDB_server // Actual path where DolphinDB/server is located chmod +x dolphindb // Modify file permissions on the first startup ./dolphindb ![](images/getting_started/1-1.png) * Windows: Open the _dolphindb.exe_ file. ![](images/getting_started/1-2.png) If the DolphinDB server is launched, you will see the corresponding interface as shown in the images above, displaying the current server’s version number, license type (trial/test/commercial), expiration date, andcompilation date. **Note**: * To use a different license, rename the new license file to _dolphindb.lic_ and replace the old file. * The default port number is 8848. To customize the port, modify the _localSite_ parameter in the configuration file _dolphindb.cfg_ in the same directory. For example, you can specify _localSite_ as “localhost:8890“ and restart the server. ### Connect Clients to DolphinDB Server DolphinDB integrates various [clients](Database/Clients/clients.html "Use DolphindB client tools, including VS Code Extension (recommended), GUI, and Web Interface") , including VS Code Extension, Web Interface, and GUI. This section takes the Web Interface as an example to introduce how to connect clients to DolphinDB server. **Prerequisite**: Ensure that the firewall allows access to the corresponding port. 1. Enter `:` in the browser's address bar. `:` refers to the IP address and port number of the DolphinDB server. For example, to access the locally deployed standalone server through the Web Interface, enter `http://localhost:8848`. 2. Login Click the **To log in** button in the upper left corner, enter the username and password ("admin" and "123456" for initial login), and click **Log in**. **Note**: Without logging in, you can execute scripts but cannot view the schema of distributed databases and may lack access to certain features. The figure below illustrates the layout of the Web Interface. For more information, refer to [Shell](Database/Clients/Web/Shell.html) . ![](images/getting_started/1-3.png) Create Databases and Tables --------------------------- In DolphinDB, databases and tables can be created using functions `database`/`table` or the CREATE statement. The following example uses SQL statements to create a database with composite partitioning on date and symbols as well as a table in that database based on the sample stock data (including “TradeTime”, “SecurityID”, “TotalVolumeTrade”, and “TotalValueTrade”). First, use the [CREATE statement](Programming/SQLStatements/create.html) to create an OLAP database named db. Enter the following script into the editor of the Web Interface and click **Execute** (shortcut: **Ctrl+E**). CREATE DATABASE "dfs://db" PARTITIONED BY VALUE(2020.01.01..2021.01.01), HASH([SYMBOL, 4]) **Note**: DolphinDB supports various partition types. A suitable partition type not only helps users evenly divide data based on business characteristics but also improves performance and increases data throughput. For more information, refer to [Distributed Database Overview](Tutorials/database.html) . Next, use the CREATE statement to create a partitioned table named tb in the database db, with “TradeTime” and “SecurityID” as the partitioning columns. Enter and execute the following script into the editor. CREATE TABLE "dfs://db"."tb"( TradeTime TIMESTAMP SecurityID SYMBOL TotalVolumeTrade LONG TotalValueTrade DOUBLE ) partitioned by TradeTime, SecurityID Click the 🗘 icon in the upper right corner of the database browser to view the created database and table. ![](images/getting_started/2-1.png) You can execute the following script to view the schema of the partitioned table tb. loadTable("dfs://db", "tb").schema().colDefs Output: ![](images/getting_started/2-2.png) Import Data ----------- Download the sample data [testdata.csv](images/getting_started/testdata.csv) . Execute the following script to import the sample data from disk into the distributed table. Note that `"path_to_testdata.csv"` should be replaced with the actual path to the sample data. tmp = loadTable("dfs://db", "tb").schema().colDefs schemaTB = table(tmp.name as name, tmp.typeString as type) loadTextEx(dbHandle=database("dfs://db"), tableName="tb", partitionColumns=`TradeTime`SecurityID, filename="path_to_testdata.csv", schema=schemaTB) After import, use `loadTable` to load the data into memory, then check with a SELECT query. tb = loadTable("dfs://db", "tb") select * from tb Output: ![](images/getting_started/3-1.png) **Note**: This example displays the result with two decimal places. You can click the ⚙ icon to customize the decimal places. Perform Queries --------------- DolphinDB seamlessly combines SQL with functions and expressions. The following are some simple queries performed on the partitioned table tb. **Example 1.**Count the number of rows. Query the total number of records in tb. SELECT count(*) FROM tb //Output: 16 **Example 2.** Filter records with specific values. Query all records satisfying SecurityID = \`A00001 in tb. SELECT * FROM tb WHERE SecurityID = `A00001 Output: ![](images/getting_started/4-1.png) **Example 3.**Group records with group by. Use [group by](Programming/SQLStatements/groupby.html) to group the data by SecurityID and query the average of TotalVolumeTrade and the maximum of TotalValueTrade for each security. SELECT avg(TotalVolumeTrade), max(TotalValueTrade) FROM tb group by SecurityID Output: ![](images/getting_started/4-2.png) **Example 4.** Group records with context by. [context by](Programming/SQLStatements/contextBy.html) is a unique feature in DolphinDB for group calculation. Unlike `group by`, it returns a vector of the same size as the group's records, which can be used with aggregate functions, moving window functions, cumulative functions, etc., facilitating time series analysis.Use `context by` to group the data by SecurityID and query the last two records for each security. SELECT TradeTime, SecurityID, deltas(TotalVolumeTrade) FROM tb CONTEXT BY SecurityID Output: ![](images/getting_started/4-3.png) **Example 5.**Rearrange columns on two dimensions. `pivot by` is a unique feature in DolphinDB used to rearrange a column (or multiple columns) of a table on two dimensions. Use [pivot by](Programming/SQLStatements/pivotBy.html) to specify TradeTime as the row index and SecurityID as the column index to view the values of TotalValueTrade. select TotalValueTrade from tb pivot by TradeTime, SecurityID Output: ![](images/getting_started/4-4.png) Next Steps ---------- To continue exploring DolphinDB, here are some things you might like to try: * **For Data Analysts**: Focus on key [SQL](Programming/SQLStatements/SQLStatement.html) operations such as aggregation, time series data processing, and table joining. Pay special attention to streaming subscriptions and engines to tackle high-throughput, low-latency, and complex real-time analysis scenarios. * **For Database Developers**: Focus on [core concepts of databases](Database/chap_database.html "Core features of DolphinDB's distributed architecture, basic database operations, and configuration setup.") and [DolphinDB scripting language](Programming/DataTypesandStructures/DataTypesandStructures.html) to leverage DolphinDB's high performance to the largest extent and develop more efficient applications. * **For Database Administrators**: Focus on [database maintenance](Maintenance/chap_maintenance.html "Database maintenance and system management") and [troubleshooting](Troubleshooting/ErrorCode.html) to ensure the stable performance and effective management of the DolphinDB system. Technical Support ----------------- For any issues with DolphinDB, you can seek technical support through the following channels: | Channel | Description | | --- | --- | | StackOverflow | Search for related information or post your questions on [StackOverflow](https://stackoverflow.com/questions/tagged/dolphindb)
.

**Note**: Please provide as much detail as possible in your questions, including the product version, operating system, OS kernel version, etc. Clearly describe the issue and attach relevant files or screenshots so our team can assist you more efficiently. | | Tutorials | DolphinDB offers [tutorials](Tutorials/chap_tutorials.html "Tutorials on database concepts and application scenarios")
covering various scenarios and needs. | | Twitter/LinkedIn | Find us on [Twitter](https://x.com/DolphinDB_Inc?ref_src=twsrc%5Egoogle%7Ctwcamp%5Eserp%7Ctwgr%5Eauthor)
and [LinkedIn](https://www.linkedin.com/company/dolphindb/)
and follow our account for the latest announcements and updates. | | E-mail | Email your inquiries to support@dolphindb.com and our team will respond in time. For more complex issues, additional time may be needed for analysis and reply. Please:

* Describe your issue in detail, including the hardware and software environment and the specific business scenario.
* Provide your contact information (e.g., phone number) for further communication. | | Slack | Contact our team on [Slack](https://dolphindb.slack.com/)
. | Video Courses ------------- DolphinDB offers high-quality videos on [YouTube](https://www.youtube.com/@dolphindb8736/videos) , covering technical discussions, product introductions, and recorded public lectures. Feel free to explore these video resources! --- # Migration [Jump to main content](#wh_topic_body) Migration ========= Import and migrate data from various data sources to DolphinDB databases --- # Maintenance [Jump to main content](#wh_topic_body) Maintenance =========== Database maintenance and system management --- # Documentation [Jump to main content](#content) [About DolphinDB](about_dolphindb.html) [Getting Started](getting_started.html) [Deployment](Deployment/chap_ddb_deployment.html) Deploy and upgrade DolphinDB on premise or in the cloud [Standalone Deployment and Upgrade](Tutorials/standalone_deployment.html) [Cluster Deployment and Upgrade](Deployment/cluster_deployment.html) [Cloud Deployment and Upgrade](Tutorials/docker_single_deployment.html) [Prepare a Linux Machine for Deployment](Tutorials/prep_linux_for_deploy.html) [GUI Clients](Database/Clients/clients.html) Use DolphindB client tools, including VS Code Extension (recommended), GUI, and Web Interface [Database](Database/chap_database.html) Core features of DolphinDB's distributed architecture, basic database operations, and configuration setup. [Distributed Architecture](Database/DatabaseandDistributedComputing/DistributedDatabase.html) [Multi-Model Database](Database/Multi-Model%20Database/multi_model_database.html) [Data Partitioning](Database/data_partitioning.html) [Database Operations](Database/DatabaseOperations/database_operation.html) Basic database operations include creating and dropping databases and tables, appending data, updating and deleting records, querying data, and performing table joins. [Catalog](Database/Catalog.html) [Inter-node Communication Protocols](Database/inter_node_communication_protocols.html) [Configuration](Database/Configuration/configuration.html) [Streaming](Streaming/chap_streaming.html) Publish, subscribe to, and process streaming data [Fundamentals](Streaming/str_funcs.html) [Streaming Engines](Streaming/streaming_engines.html) [Join Engines](Streaming/join_engines.html) [Historical Data Replay](Streaming/str_replay.html) [Unified Batch and Stream Processing](Streaming/unified_batch_and_stream_processing.html) [Complex Event Processing](Streaming/ComplexEventProcessing.html) [Migration](Migration/chap_data_migration.html) Import and migrate data from various data sources to DolphinDB databases [Import Data](Migration/import_data.html) [Migrate Data](Tutorials/kdb_to_ddb.html) [Maintenance](Maintenance/chap_maintenance.html) Database maintenance and system management [Cluster Management](Tutorials/cluster_scaleout.html) [Job Management](Maintenance/BatchJobManagement.html) [Performance Monitoring](Maintenance/PerformanceMonitoring.html) [Login Management](Maintenance/single_sign_on.html) [Graceful Shutdown](Maintenance/shutdown.html) [Security](Maintenance/BackupandRestore.html) [Troubleshooting](Troubleshooting/chap_troubleshooting.html) Common errors and corresponding solutions [Server Connection Issues](Troubleshooting/server_connection_issues.html) [Node Crashes](Troubleshooting/node_crashes.html) [Node Startup Exception](Troubleshooting/node_startup_exception.html) [System Freeze](Troubleshooting/system_freeze.html) [Slow Queries or Writes](Troubleshooting/slow_queries_or_writes.html) [Error Code Reference](Troubleshooting/ErrorCode.html) [Programming Guide](Programming/programming_basics.html) DolphinDB scripting language and SQL reference [DolphinDB Scripting Language](Programming/DataTypesandStructures/DataTypesandStructures.html) [SQL Reference](Programming/SQLStatements/SQLStatement.html) [DolphinDB Python Parser](Programming/py_par_intro.html) [Function References](Functions/chap_function_references.html) Reference for all built-in functions [Categories](Functions/funcs_by_topics.html) [Functions](Functions/category.html) [Themes](Functions/Themes/mFunctions.html) [Templates](Functions/Templates/accumulate.html) [API & Connector](API/chap_api.html) API in different languages for users to call DolphinDB from their programs [C++ API](https://docs.dolphindb.com/en/cppdoc/chap1_quickstart_landingpage.html) [C# API](API/Csharp.html) [Java API](https://docs.dolphindb.com/en/javadoc/overview.html) [JavaScript API](API/JavaScript.html) [JDBC Connector](API/JDBC.html) [Web API](API/Json.html) [Python API](https://docs.dolphindb.com/en/pydoc/chap1_quickstart_landingpage.html) [R API](API/R.html) [REST API](API/rest_api.html) [Grafana Datasource](API/grafana_datasource.html) [Plugins](Plugins/chap_plugin.html) Guidance on the use of plugins and plugin development for multiple application scenarios [Introduction](Plugins/ddb_plugin.html) [Plugin Development](Plugins/plugin_development_tutorial.html) [Arrow](Plugins/formatArrow.html) [AWSS3](Plugins/aws.html) [Feather](Plugins/feather.html) [gp](Plugins/gp.html) [gurobi](Plugins/gurobi.html) [HBase](Plugins/hbase.html) [HDF5](Plugins/hdf5.html) [HDFS](Plugins/hdfs.html) [HttpClient](Plugins/httpClient.html) [Kafka](Plugins/kafka.html) [Kdb+](Plugins/kdb.html) [LDAP](Plugins/ldap.html) [lgbm](Plugins/lgbm.html) [LibTorch](Plugins/libtorch.html) [mat](Plugins/mat.html) [MongoDB](Plugins/mongodb.html) [MQTT](Plugins/mqttClient.html) [mseed](Plugins/mseed.html) [MySQL](Plugins/mysql.html) [ODBC](Plugins/odbc.html) [opc](Plugins/opc.html) [opcua](Plugins/opcua.html) [orc](Plugins/orc.html) [parquet](Plugins/parquet.html) [Py](Plugins/py.html) [redis](Plugins/redis.html) [RabbitMQ](Plugins/rabbitmq.html) [RocketMQ](Plugins/rocketmq.html) [SevenZip](Plugins/SevenZip.html) [signal](Plugins/signal.html) [SVM](Plugins/svm.html) [TCPSocket](Plugins/tcpsocket.html) [WebSocket](Plugins/websocket.html) [xgboost](Plugins/xgboost.html) [zmq](Plugins/zmq.html) [zip](Plugins/zip.html) [zlib](Plugins/zlib.html) [Tutorials](Tutorials/chap_tutorials.html) Tutorials on database concepts and application scenarios [Distributed Database](Tutorials/async_replication.html) [Data Import](Tutorials/import_data.html) [Programming](Tutorials/add_column.html) [Streaming](Tutorials/streamingHA.html) [System Management](Tutorials/backup_and_restore.html) [Modules](Tutorials/modules.html) [Application Scenarios](Tutorials/best_practices_for_market_data_replay.html) [API](Tutorials/python_api_install_offline.html) [Release Notes](ReleaseNotes/chap_release_notes.html) Release notes and compatibility changes for DolphinDB server, plugins and APIs [Server](ReleaseNotes/server/server_rn.html) [Plugins](ReleaseNotes/plugin_rn.html) [API & Connector](ReleaseNotes/api_rn.html) [Clients](ReleaseNotes/clients_rn.html) --- # Deployment [Jump to main content](#wh_topic_body) Deployment ========== Deploy and upgrade DolphinDB on premise or in the cloud DolphinDB offers flexible deployment options to accommodate diverse business needs. | Feature | Standalone | Single-Machine Cluster | Multi-Machine Cluster | | --- | --- | --- | --- | | Multiple storage engines | √ | √ | √ | | Transaction | √ | √ | √ | | Distributed computing | √ | √ | √ | | Real-time streaming | √ | √ | √ | | System management and APIs | √ | √ | √ | | Cloud deployment | √ | √ | √ | | Scalability | × | √ | √ | | High availability through redundancy | × | √ | √ | | Fault tolerance for machine failures | × | × | √ | --- # Troubleshooting [Jump to main content](#wh_topic_body) Troubleshooting =============== Common errors and corresponding solutions --- # Distributed Architecture [Jump to main content](#wh_topic_body) Distributed Architecture ======================== DolphinDB's distributed database architecture combines a shared-nothing model with a logical file system. In the shared-nothing model, each node operates independently with its own computing and storage resources. This design allows for features like high availability, online recovery, distributed transactions, and asynchronous replication. The logical file system stores metadata on controllers and distributes partitioned data across data nodes. It provides unified data access through a global directory, abstracting physical storage locations from users. This design delivers efficient distributed storage while simplifying data access for users. --- # Function References [Jump to main content](#wh_topic_body) Function References =================== Reference for all built-in functions Notations --------- For unary functions, we use X as the argument. For binary functions, we use X and Y as the first and second argument, respectively. Without clarification, X/Y indicates an object including scalar, vector, matrix, dictionary and table. We have functions such as [take](t/take.html) , [rand](r/rand.html) , [norm](n/norm.html) that require a scalar to be one of the arguments. For these cases, we use lower case letters such as "a", "b", etc. Contents within \[\] are optional. **Unary function syntax:** (1) `(X)` // standard function calling (2) ` X` // parenthesis around X are ignored when putting a function name in front of parameter X; not good for nested function calling (3) `X.()` // parenthesis around X may be ignored when X is an object; it is more suited for nested function calling **Binary function syntax:** (1) `(X, Y)` // standard function calling (2) `X Y` // put the function name between two parameters; not good for nested function calling (3) `X. (Y)` // parenthesis around X may be ignored when X is an object; it is more suited for nested function calling Calling a function by the object-oriented style can be applied to both constants and variables. For brevity, we will only introduce the first form `(X)` or `(X, Y)` in the syntax section for all unary and binary functions of this chapter. Examples -------- // 3 ways to call function log log(10); // output: 2.302585 log 10; // output: 2.302585 (10).log(); // output: 2.302585 // a pair of parenthesis is mandatory here in order to treat the constant 10 as an object x = 5 3 1 10; log(x); // output: [1.609438,1.098612,0,2.302585] log x; // output: [1.609438,1.098612,0,2.302585] x.log(); // output: [1.609438,1.098612,0,2.302585] // a pair of parenthesis are ignored here because x is an vector object (5 3 1 10).log() // output: [1.609438,1.098612,0,2.302585] // a pair of parenthesis are mandatory here in order to treat constant vector 5 2 1 10 as an vector object sin(add(log(5),7)); // output: 0.727959 Performance Tip --------------- The two arguments of most binary operators/functions don't need to be the same data type. The system tries its best to do the conversion. However, it is not recommended to use different data types because it causes performance loss when a significant amount of data type conversion is involved. --- # Streaming [Jump to main content](#wh_topic_body) Streaming ========= Publish, subscribe to, and process streaming data --- # GUI Clients [Jump to main content](#wh_topic_body) GUI Clients =========== Use DolphindB client tools, including VS Code Extension (recommended), GUI, and Web Interface --- # Tutorials [Jump to main content](#wh_topic_body) Tutorials ========= Tutorials on database concepts and application scenarios --- # Web Interface [Jump to main content](#wh_topic_body) Web Interface ============= Users can manage a DolphinDB cluster from the controller in the web-based interface. ![](../images/nb_index_0.png) Please note that if there is no interaction between DolphinDB Web Interface and DolphinDB server for longer than 10 minutes, the interface is automatically disconnected from DolphinDB server. The web interface is only suitable for ad-hoc executions of a few lines of script. We strongly recommend using DolphinDB GUI or DolphinDB VS Code extension as they provide a more convenient and more stable environment. --- # Data Types and Data Forms [Jump to main content](#wh_topic_body) Data Types and Data Forms ========================= The DolphinDB system allocates memory and decides what to be stored in the reserved memory based on the data type of a variable. This chapter introduces data types, categories, and forms are first summarized in a table and then explained with examples. The codes and results in the paragraphs with light gray background are the same as seen from a console. Users can copy and paste these codes to a console, execute and then check the results. --- # Data Partitioning [Jump to main content](#wh_topic_body) Data Partitioning ================= In distributed systems, data partitioning is a technique that splits large datasets into smaller chunks based on certain rules. These chunks are then distributed across different nodes or servers for query optimization and resource management. Benefits of Data Partitioning ----------------------------- A well-designed data partitioning strategy offers several key advantages: * **Improved Query Performance:** Querying with partitioning columns allows the system to scan only relevant partitions, avoiding costly full table scans. * **Easier Maintenance:** Partitioning makes large tables more manageable with small chunks. Routine database tasks like backup, restore, and data migration can be coordinated more effectively by retrieving smaller units. * **Optimized Resource Utilization:** By distributing data across multiple nodes, a single task can be broken down into multiple subtasks, each executed on different nodes. Such parallelism harnesses the power of distributed computing, improving processing speed and overall system performance. Partition Type and Scheme ------------------------- When creating a database in DolphinDB, you can define how the data within the database is partitioned by specifying two key elements: the partition type and the partition scheme. DolphinDB databases support the following basic **partition types**: RANGE, HASH, VALUE, LIST. To achieve more flexible data partitioning, DolphinDB also supports composite partitioning (COMPO), which combines two or three dimensions of partition types. The **partition scheme** defines the range, set of values, or number of buckets that establishes the scope or boundaries for dividing data based on one or more specific columns, i.e., **partitioning columns**. In following sections, we'll explore various partitioning strategies and demonstrate how to create databases (with `database` function) with different partition types. **Note:** * Before proceeding, make sure that you have administrator access or the DB\_OWNER privilege (which can be checked with the `getUserAccess` function). If you do not have the necessary permissions, please contact the administrator for authorization. Alternatively, login with the default admin account using ``login(`admin,`123456)``. ### Range Partitioning Range partitioning is suitable for storing time series data, or continuous numerical data. Partitions of the RANGE type are defined by ranges where the boundaries are two consecutive elements from the partition scheme vector. The starting value is inclusive, while the ending value is exclusive. For example, to create a database db with 2 partitions: \[0,5) and \[5,10), specify `partitionScheme = 0 5 10`. Then, create a partitioned table with the partitioning column ID in db.\ \ n=1000000\ ID=rand(10, n)\ x=rand(1.0, n)\ t=table(ID, x)\ db=database("dfs://rangedb", RANGE, 0 5 10)\ \ pt = db.createPartitionedTable(t, `pt, `ID)\ pt.append!(t)\ \ pt=loadTable(db,`pt)\ select count(x) from pt;\ \ Once data is appended to pt, the system will organize it based on the ID values: records with ID values ranging from 0 to 4 will be stored in partition _0\_5_ and 5 to 9 in _5\_10_.\ \ \ ![](images/data_partitioning/1.png) \ \ **Note**: The partition scheme of a range-partitioned database can be appended after it is created. See function [addRangePartitions](../Functions/a/addRangePartitions.html)\ for details.\ \ ### Hash Partitioning\ \ Hash partitioning creates a specified number of partitions using a hash function on the partitioning column. This method helps balance data distribution, reducing the risk of overloaded partitions. However, it may still result in uneven partition sizes if the partitioning column values are skewed. For efficient retrieval of data within a range, range or value partitions are preferable to hash partitions.\ \ For example, to create a database db with 2 partitions based on an INT column, specify `partitionScheme = [INT, 2]`. Then, create a partitioned table with the partitioning column ID in db.\ \ n=1000000\ ID=rand(10, n)\ x=rand(1.0, n)\ t=table(ID, x)\ db=database("dfs://hashdb", HASH, [INT, 2])\ \ pt = db.createPartitionedTable(t, `pt, `ID)\ pt.append!(t)\ \ pt=loadTable(db,`pt)\ select count(x) from pt;\ \ Once data is appended to pt, the system will organize it based on hash values. Records sharing the same hash value are grouped into the same partition (e.g., Key0, Key1).\ \ \ ![](images/data_partitioning/2.png) \ \ ### Value Partitioning\ \ Value partitioning creates partitions based on specific values in the partitioning column, typically applied to date and time columns (see section Partitions on Temporal Values).\ \ For example, to create a database db with 204 monthly partitions from January 2000 to December 2016, specify _partitionScheme_ as a vector, `2000.01M..2016.12M`, with each element corresponding to a partition. Then, create a partitioned table with the partitioning column month in db.\ \ n=1000000\ month=take(2000.01M..2016.12M, n)\ x=rand(1.0, n)\ t=table(month, x)\ \ db=database("dfs://valuedb", VALUE, 2000.01M..2016.12M)\ \ pt = db.createPartitionedTable(t, `pt, `month)\ pt.append!(t)\ \ pt=loadTable(db,`pt)\ select count(x) from pt;\ \ Once data is appended to pt, the system will organize it based on month. For example, data in 2000.01 will be stored in _200001M_.\ \ \ ![](images/data_partitioning/3.png) \ \ **Note**: The partition scheme of a value-partitioned database can be appended after it is created. See function [addValuePartitions](../Functions/a/addValuePartitions.html)\ and configuration parameter [_newValuePartitionPolicy_](Configuration/reference.html#topic_syl_jvr_ncc)\ for details.\ \ ### List Partitioning\ \ List partitioning partitions data based on specified sets of enum values. It's particularly useful when you need to include multiple partitioning column values into a single partition.\ \ **Note**: The difference between the list type and value type is that all the elements in a value partition scheme are scalars, whereas each element in a list partition scheme may be a vector.\ \ For example, to create a database db with 2 partitions based on symbol, specify _partitionScheme_ as a vector, ``[`IBM`ORCL`MSFT, `GOOG`FB]``, with each element grouped into a partition. Then, create a partitioned table with the partitioning column “ticker” in db.\ \ n=1000000\ symbol = rand(`MSFT`GOOG`FB`ORCL`IBM,n)\ x=rand(1.0, n)\ t=table(symbol, x)\ \ db=database("dfs://listdb", LIST, [`IBM`ORCL`MSFT, `GOOG`FB])\ pt = db.createPartitionedTable(t, `pt, `symbol)\ pt.append!(t)\ \ pt=loadTable(db,`pt)\ select count(x) from pt;\ \ Once data is appended to pt, the system will automatically organize it based on symbol: records with symbol=\`IBM\`ORCL\`MSFT are stored in _List0_ and symbol=\`GOOG\`FB in _List1_.\ \ \ ![](images/data_partitioning/4.png) \ \ ### Composite Partitioning\ \ Composite partitioning allows for flexible, multi-dimensional data organization.\ \ For example, to create a database db based on 2-dimension partitioning, specify _partitionScheme_ as a vector, `[dbDate, dbID]`, with each element corresponding to a database handle. dbDate is value-partitioned and dbID is range-partitioned. Then, create a partitioned table with partitioning columns “date” and “ID“ in db.\ \ n=1000000\ ID=rand(100, n)\ dates=2017.08.07..2017.08.11\ date=rand(dates, n)\ x=rand(10.0, n)\ t=table(ID, date, x)\ \ dbDate = database(, VALUE, 2017.08.07..2017.08.11)\ dbID=database(, RANGE, 0 50 100)\ db = database("dfs://compoDB", COMPO, [dbDate, dbID])\ \ pt = db.createPartitionedTable(t, `pt, `date`ID)\ pt.append!(t)\ \ pt=loadTable(db,`pt)\ select count(x) from pt;\ \ Once data is appended to pt, the system will organize it along both dimensions: VALUE+RANGE. The value domain has 5 partitions. Click on a date partition, we can see the range domain has 2 partitions:\ \ \ ![](images/data_partitioning/5.png) \ \ **Note**: If one of the partitioning columns is of value domain, it can be appended with new values after it is created. See function [addValuePartitions](../Functions/a/addValuePartitions.html)\ and configuration parameter [_newValuePartitionPolicy_](Configuration/reference.html#topic_syl_jvr_ncc)\ for details.\ \ ### Adaptable Data Partitioning: User-Defined Rules\ \ DolphinDB provides a custom solution for data partitioning based on user-defined rules to flexibly accommodate diverse partitioning requirements. For example:\ \ * Applying prefix functions to futures and options contract data from exchanges;\ * Partitioning based on IoT metric IDs;\ * Extracting date and time information from encoded data.\ \ The parameter _partitionColumns_ of `createPartitionedTable` function and `create` statement supports specifying a user-defined function to a column and using the transformed result as a partitioning column.\ \ For instance, for data with a column in the format `id_date_id` (e.g., ax1ve\_20240101\_e37f6), users can partition by date using a user-defined function:\ \ // Define a function to extract the date information\ def myPartitionFunc(str,a,b) {\ return temporalParse(substr(str, a, b),"yyyyMMdd")\ }\ \ // Create a database\ data = ["ax1ve_20240101_e37f6", "91f86_20240103_b781d", "475b4_20240101_6d9b2", "239xj_20240102_x983n","2940x_20240102_d9237"]\ tb = table(data as id_date, 1..5 as value, `a`b`c`d`e as sym)\ db = database("dfs://testdb", VALUE, 2024.02.01..2024.02.02)\ \ // Use myPartitionFunc to process the data column\ pt = db.createPartitionedTable(table=tb, tableName=`pt, \ partitionColumns=["myPartitionFunc(id_date, 6, 8)"])\ pt.append!(tb)\ \ select * from pt\ \ Records queried by the `select` statement are read and returned by partition. The query result shows that table pt is partitioned by the date information extracted from the id\_date column.\ \ \ ![](images/data_partitioning/6.png) \ \ Key Considerations for Partition Design\ ---------------------------------------\ \ A good partition strategy can reduce latency and improve query performance and throughput. This section explores key factors to consider when designing an optimal partitioning scheme.\ \ ### Select Appropriate Partitioning Columns\ \ When implementing partitions in a database, the choice of partitioning column(s) is a crucial decision made during table creation. This choice significantly affects subsequent data operations, especially queries involving or related to the partitioning columns.\ \ **Supported Data Types**\ \ In DolphinDB, the data types that can be used for partitioning columns include integers (CHAR, SHORT, INT), temporal (DATE, MONTH, TIME, SECOND, MINUTE, DATETIME, DATEHOUR), STRING and SYMBOL. The HASH domain also supports LONG, UUID, IPADDR and INT128 types. Although a partitioning column can be of STRING type, for optimal performance, we recommend converting a STRING type column into SYMBOL type to be used as a partitioning column.\ \ FLOAT and DOUBLE types cannot be used as a partitioning column.\ \ db=database("dfs://rangedb1", RANGE, 0.0 5.0 10.0);\ // report an error: The data type DOUBLE can't be used for a partition column.\ \ **Select Columns Frequently Used**\ \ Select partitioning columns that align with frequent database updates in your business context. For example, many tasks on financial databases involve stock symbols and dates. Therefore stock symbols and dates are natural candidates as partitioning columns.\ \ **Note**: DolphinDB doesn't allow multiple threads or processes to write to the same partition simultaneously when updating the database. Considering we may need to update data of a date or of a stock, if we use other partitioning columns such as trading hour, multiple writers may write to the same partition simultaneously and cause problems.\ \ Additionally, a partitioning column is equivalent to a physical index on a table. If a query uses a partitioning column in **where** conditions, the system can quickly load the target data instead of scanning the entire table. Therefore, partitioning columns should be the columns that are frequently used in **where** conditions for optimal performance.\ \ ### Partition Size Should Not Be Too Large\ \ The columns in a partition are stored as separate files on disk after compression. When a query reads the partition, the system loads the necessary columns into memory after decompression. Too large partitions may slow down the system, as they may cause insufficient memory with multiple threads running in parallel, or they may make the system swap data between disk and memory too frequently. As a rule of thumb, assume the available memory of a data node is S and the number of workers is W, then it is recommended that a partition is less than S/8W in memory after decompression，For example, with available memory of 32GB and 8 workers, a single partition should be smaller than 32GB/8/8=512MB.\ \ The number of subtasks of a query is the same as the number of partitions involved. Therefore if partitions are too large, the system cannot fully take advantage of distributed computing on multiple partitions.\ \ DolphinDB is designed as an OLAP database system. It supports appending data to tables, but does not support deleting or modifying individual rows. To modify certain rows, we need to overwrite entire partitions containing these rows. If the partitions are too large, it would be too slow to modify data. When we copy or move selected data between nodes, entire partitions with the selected data are copied or moved. If the partitions are too large, it would be too slow to copy or move data.\ \ Considering all the factors mentioned above, we recommend the size of a partition does not exceed 1GB before compression. This restriction can be adjusted based on the specific situations. For example, when we work with a table with hundreds of columns and only a small subset of columns are actually used, we can relax the upper limit of the partition size range.\ \ To reduce partitions sizes, we can (1) use composite partitions (COMPO); (2) increase the number of partitions; (3) use value partitions instead of range partitions.\ \ ### Partition Size Should Not Be Too Small\ \ If partition size is too small, a query or computing job may generate a large number of subtasks. This increases the time and resources in communicating between the controller node and data nodes, and between different controller nodes. Too small partitions also result in innefficient reading/writing of small files on disk and therefore. Lastly, the metadata of all partitions are stored in the memory of the controller node. Too many small partitions may make the controller node run out of memory. We recommend that the partition size is larger than 100 MB on average before compression. Based on the above-mentioned analysis, we recommend the size of a partition be between 100MB and 1GB.\ \ The distribution of trading activities of stocks is highly skewed. Some stocks are extremely active but a majority of stocks are much less active. For a composite partition with partitioning columns of trading days and stock tickers, if we use value partitions on both partitioning columns, we will have many extremely small partitions for illiquid stocks. In this case we recommend range partitions on stock tickers where many illiquid stocks can be grouped into one partition, resulting a more evenly partitioned dataset for better performance.\ \ ### How to Partition Data Evenly\ \ Significant differences among partition sizes may cause load imbalance. Some nodes may have heavy workloads while other nodes idle. If a task is divided into multiple subtasks, it returns the final result only after the last subtask is finished. As each subtask works on a different partition, if data is not distributed evenly among partitions, it may increase the execution time.\ \ A useful tool for partitioning data evenly is function `cutPoints(X, N, [freq])`, where X is a vector; N means the number of buckets the elements of X will be grouped into; the optional argument freq is a vector of the same length as X indicating the frequency of each element in X. It returns a vector with (N+1) elements such that the elements of X are evenly distributed within each of the N buckets indicated by the vector. It can be used to generate the partition scheme of a range domain in a distributed database.\ \ In the following example, we construct a composite partition on date and symbols for high frequency stock quotes data. We use the data of 2007.08.01 to determine 128 stock symbol ranges with equal number of rows on 2007.08.01, and apply these ranges on the entire dataset.\ \ t = ploadText(WORK_DIR+"/TAQ20070801.csv")\ t = select count(*) as ct from t where date=2007.08.01 group by symbol\ buckets = cutPoints(t.symbol, 128, t.ct)\ \ dateDomain = database("", VALUE, 2017.07.01..2018.06.30)\ symDomain = database("", RANGE, buckets)\ stockDB = database("dfs://stockDBTest", COMPO, [dateDomain, symDomain]);\ \ ### Partitions on Temporal Values\ \ In the following example, we create a distributed database with a value partition on date. The partitioning scheme extends to the year 2030 to accommodate updates in future time periods.\ \ dateDB = database("dfs://testDate", VALUE, 2000.01.01 .. 2030.01.01)\ \ When using a time column as the partitioning column, the data type of the partition scheme does not need to be the same as the data type of a partitioning column. For example, if we use month as the partition scheme in a value partition, the data type of the partitioning column can be month, date, datetime, timestamp, or nanotimestamp.\ \ **Note**: While DolphinDB supports TIME, SECOND, and DATETIME for partitioning columns, avoid using them for value partitions. These time types create millions of small partitions, making both creation and querying inefficient.\ \ ### Partition Colocation\ \ It may be time consuming to join multiple tables in a distributed database, as the partitions that need to be joined with may be located on different nodes and therefore data need to be copied and moved across nodes. DolphinDB ensures that the same partitions of all the tables in the same distributed database are stored at the same node. This makes it highly efficient to join these tables. DolphinDB does not support joining tables from different partitioned databases.\ \ dateDomain = database("", VALUE, 2018.05.01..2018.07.01)\ symDomain = database("", RANGE, string('A'..'Z') join `ZZZZZ)\ stockDB = database("dfs://stockDB", COMPO, [dateDomain, symDomain])\ \ quoteSchema = table(10:0, `sym`date`time`bid`bidSize`ask`askSize, [SYMBOL,DATE,TIME,DOUBLE,INT,DOUBLE,INT])\ stockDB.createPartitionedTable(quoteSchema, "quotes", `date`sym)\ \ tradeSchema = table(10:0, `sym`date`time`price`vol, [SYMBOL,DATE,TIME,DOUBLE,INT])\ stockDB.createPartitionedTable(tradeSchema, "trades", `date`sym)\ \ In the examples above, the distributed tables quotes and trades are located in the same distributed database. --- # API & Connector [Jump to main content](#wh_topic_body) API & Connector =============== API in different languages for users to call DolphinDB from their programs --- # Plugins [Jump to main content](#wh_topic_body) Plugins ======= Guidance on the use of plugins and plugin development for multiple application scenarios | Usage | Plugin | | --- | --- | | Data Import and Export | * Hbase
* HDFS
* Kdb+
* MongoDB
* MySQL
* ODBC
* OPC
* OPCUA | | Message Queue | * Kafka
* MQTT
* ZeroMQ | | Network | * HttpClient | | Cloud Storage | * AWS | | Machine Learning | * XGBoost | | Extension | * Py | | Format | * Arrow
* Feather
* HDF5
* mat
* Parquet
* Zip
* Zlib | --- # Multi-Model Database [Jump to main content](#wh_topic_body) Multi-Model Database ==================== DolphinDB is a multi-model database where multiple storage engines are integrated within a unified database architecture. This design enables storage and querying of data using various data models, providing an efficient and integrated solution to complex data governance scenarios in fields like finance and IoT. Multi-Model Architecture ------------------------ DolphinDB employs an architecture (see image below) consisting of four layers, from top to bottom: * Query layer: Compiles and parses programming languages, supporting DolphinScript (including SQL-92 compliant statements) and Python. * Computation layer: Abstracts common functionalities in distributed computation like generic operators, query optimization and vectorization, ensuring efficient resource utilization across all computational tasks. * Storage engine layer: Consists of multiple storage engines that manage in-partition data storage. While the storage engines are optimized for different scenarios with custom storage and indexing designs, they all implement standardized read and write interfaces. The other layers interact with these interfaces without knowing the exact storage engine in use. * Storage layer: Manages distributed storage, handling data persistence, partitioning, replication, transactions, load balancing, recovery, and metadata management. ![](images/1.png) Storage Engines --------------- The database's layered, unified architecture allows new storage engines to be integrated with minimal changes. A new storage engine only needs to provide a specific data model (including indexing implementation and data file structure), along with interfaces for other layers to read and write data. These interfaces work with abstract representations of tables and vectors with storage details hidden, thus reducing inter-layer coupling. For example, the SQL engine can process queries using abstract representations of tables and columns (treated as vectors in the underlying computation layer) without concern for storage details. The storage of data in each partition is entirely managed by the storage engines and can vary in implementation. For example, in the TSDB engine, a column in a partition consists of multiple blocks, forming a logically contiguous but physically scattered vector. Conversely, the OLAP engine stores a column within a partition as a physically contiguous vector. While the distributed file system (DFS) in the underlying storage layer manages entire database data storage across nodes, the storage engines handle in-partition operations, including reads, writes, indexing, and decompression. During writes, data is initially written to in-memory tables per partition via the storage engine's interface, then persisted to disk based on DFS-specified locations. For reads, the storage engine retrieves data from disk by partition and passes it to the computation layer for processing. Storage engine is specified during database creation through the _engine_ parameter of the [database](../../Functions/d/database.html) function. While you can create databases under different storage engines, each database is tied to a single storage engine. DolphinDB provides the following storage engines: | Storage Engine | Use Cases | | --- | --- | | [OLAP](olap_storage_engine.html) | Large-scale data analysis (e.g., querying trading volumes for all stocks in a specific time period) | | [TSDB](tsdb_storage_engine.html) | Most time-series processing scenarios; balances analytical capabilities and query performance | | [VectorDB](vectordb.html) | Fast retrieval with massive datasets (e.g., search engines, AI generative models) | | [PKEY](primary_key_storage_eng.html) | Real-time updates and efficient queries (e.g., real-time data analysis through CDC integration with OLTP systems) | | [IMOLTP](../../Functions/c/createIMOLTPTable.html) | High-concurrency, low-latency OLTP (e.g., financial trading systems); dataset size must be within memory capacity | | [TextDB](textdb.html) | Market sentiment analysis, information filtering and processing in finance; large-scale log data management, real-time search and analysis in IoT | Except for the in-memory IMOLTP, all other storage engines support horizontal partitioning of large data sets based on a distributed file system, ensuring data is evenly dispersed across data nodes. All storage engines support horizontal partitioning of large data sets based on a distributed file system, ensuring data is evenly dispersed across data nodes. --- # Catalog [Jump to main content](#wh_topic_body) Catalog ======= DolphinDB 3.0 has introduced the catalog feature exclusively for DFS databases to provide users with a more convenient and standardized database access experience, as well as facilitate integration with third-party software. Catalogs allow users to uniformly manage databases and tables of varying partitioning schemes, and access them using standard SQL syntax. Concepts -------- **Syntax** The syntax for referencing tables within a catalog is as follows: ..[@] where * catalog is a STRING scalar indicating the catalog name. * schema is a STRING scalar indicating the schema name. * table is a STRING scalar indicating the table name. * cluster\_identifier (optional) is a STRING scalar indicating the cluster name (configured by _clusterName_). **Note**: In contrast to tables within a catalog that can be referenced directly with `..
`, tables within a database must first be loaded with `loadTable` before performing any subsequent operations. DolphinDB's catalog consists of the following three hierarchies: * **Catalog**: The highest level of organization within the database management system (DBMS). A catalog holds one or more schemas, representing the complete set of schemas that a user can access. * **Schema**: The logical container for table objects. In DolphinDB, a schema corresponds to a database in the Database > Table structure. Note that this concept of schema should be distinguished from the table schema which refers to the structure of the table. * **Table**: The primary component within a schema. Tables organize information into rows and columns. ![](../images/catalog.png) Note: When using SQL statements for table operations, tables within a catalog can be referenced by specifying `catalog.schema.table`. In contrast, tables within a database must first be loaded with `loadTable` before performing any subsequent operations. Operations on Catalogs ---------------------- ### Working with Catalogs The name of a catalog or schema can contain letters, digits and underscores and must begin with a letter. The name is case-insensitive when referenced in statements or functions. **(1) Creating catalogs** To create a catalog, use function `createCatalog`. For example, create a catalog "trading": createCatalog("trading") To switch to a specific catalog, execute `use` statement and specify the `catalog` keyword. For example, switch to catalog "trading": use catalog trading; Once a catalog is used, subsequent operations would use the catalog by default unless a different catalog is explicitly specified. **(2) Creating schemas** To create a schema, use `create database` statement and specify _catalog.schema_ instead of _directory_. Syntax create database catalog.schema partitioned by partitionType(partitionScheme),[partitionType(partitionScheme),partitionType(partitionScheme)], [engine='OLAP'], [atomic='TRANS'], [chunkGranularity='TABLE'] For example, create a schema "stock" using VALUE partitions and OLAP storage engine. create database stock partitioned by VALUE(`IBM`MSFT`GM`C`YHOO`GOOG), engine='OLAP' Since section (1) already specifies a default catalog "trading", `stock` in the above script is equivalent to `trading.stock`. If no catalog is used, the creation would fail and an error of "The catalog doesn't exist" will be thrown. To add an existing database to a catalog, use `createSchema`. For example, add schema "stock2" referencing database "dfs://db1" to the catalog "trading". database(directory="dfs://db1", partitionType=RANGE, partitionScheme=0 5 10) //通过 database 函数创建的 db1 createSchema("trading", "dfs://db1", "stock2") **(3) Creating tables** To create a table in a schema, use `create table` statement and specify _catalog.schema.tableName_ instead of _dbPath_. Syntax create table catalog.schema.tableName( schema[columnDescription] ) [partitioned by partitionColumns], [sortColumns], [keepDuplicates=ALL], [sortKeyMappingFunction] For example: create table stock.quote ( id INT, date DATE[comment="time_col", compress="delta"], value DOUBLE, ) partitioned by id Since section (1) already specifies a default catalog "trading", `stock.quote` in the above script is equivalent to `trading.stock.quote`. **(4) Dropping catalogs** To drop a catalog, call function `dropCatalog`. For example: dropCatalog("catalog1") **(5) Dropping schemas from catalogs** To drop a schema from a catalog, use `drop` statement. Syntax drop table [if exists] catalog.schema where _catalog.schema_ specifies the schema to be dropped. If a default catalog is already used, the _catalog_ can be omitted. **(6) Dropping tables from schemas** To drop a table from a schema, use `drop` statement. Syntax drop table [if exists] catalog.schema.tableName where _catalog.schema.tableName_ specifies the table to be dropped. If a default catalog is already used, the _catalog_ can be omitted. ### SQL Examples The following SQL statements can be used when working with catalogs. | Statement | Type | Description | | --- | --- | --- | | create | DDL | Create schemas/tables | | alter | DDL | Add columns to tables | | drop | DDL | Drop schemas/tables | | update | DML | Update tables | | delete | DML | Delete records from tables | | select | DQL | Access tables | Note: When using these SQL statements on catalogs/schemas, a default catalog must be used in the current session or a catalog must be specified in the statement, otherwise an error would be raised. Use the following script to generate sample data using the above created catalog "trading": create database stock partitioned by VALUE(1..6), engine='OLAP' database(directory="dfs://db1", partitionType=RANGE, partitionScheme=0 5 10) createSchema("trading", "dfs://db1", "stock2") create table stock.quote ( id INT, date DATE[comment="time_col", compress="delta"], value DOUBLE, ) partitioned by id dbUrl = exec dbUrl from getSchemaByCatalog("trading") where schema = "stock" id = 1 2 3 4 5 6 1 2 3 4 5 6 date = 2023.01.01..2023.01.12 value = 1..12 data = table( id, date, value) loadTable(dbUrl[0], "quote").append!(data) select * from stock.quote Table trading.stock.quote: | id | date | value | | --- | --- | --- | | 1 | 2023.01.01 | 1 | | 1 | 2023.01.07 | 7 | | 2 | 2023.01.02 | 2 | | 2 | 2023.01.08 | 8 | | 3 | 2023.01.03 | 3 | | 3 | 2023.01.09 | 9 | | 4 | 2023.01.04 | 4 | | 4 | 2023.01.10 | 10 | | 5 | 2023.01.05 | 5 | | 5 | 2023.01.11 | 11 | Example 1. Select data from "stock.quote" with filtering conditions: select * from stock.quote where id = 1 | id | date | value | | --- | --- | --- | | 1 | 2023.01.01 | 1 | | 1 | 2023.01.07 | 7 | Example 2. Update data of "stock.quote": update stock.quote set value = -1.0 where id = 2 select * from stock.quote where id = 2 | id | date | value | | --- | --- | --- | | 2 | 2023.01.02 | \-1 | | 2 | 2023.01.08 | \-1 | Example 3. Delete data from "stock.quote": delete from stock.quote where id = 3 select * from stock.quote where id = 3 // returns an empty table Note: If a script contains a variable with the same name as schema, the query may fail to access the schema. For example, if a variable stock is defined and shares the same name as the schema, stock in the script will be parsed as the variable name and an error will be raised. stock=1 select * from stock.quote where id = 1; //getMember method not supported Such variables can be dropped with `undef`. undef(`stock) Management of Catalogs ---------------------- **(1) Checking default catalog** To view the default catalog used in the current session, call `getCurrentCatalog`. use catalog trading; getCurrentCatalog() // output: "trading" use catalog trading2; getCurrentCatalog() // output: "trading2" **(2) Renaming catalogs/schemas** To rename a catalog, call function `renameCatalog`. renameCatalog("trading", "trading2") getAllCatalogs() // output: ["trading2"] To rename a schema, call function `renameSchema`. renameSchema("trading", "stock", "stock1") exec schema from getSchemaByCatalog("trading") // output: ["stock1"] **(3) Checking schema info** To get info on schemas within a catalog, call function `getSchemaByCatalog`. A schema created with `create database` has a unique identifier of _dbUrl_ in the format "dfs://" which is unaltered even after renamed. getSchemaByCatalog("trading") | schema | dbUrl | | --- | --- | | stock | dfs://trading\_stock\_1712077295373 | **(4) Checking logs on catalog operations** To view info of operations on catalogs and schemas, you can check the ACL Audit log on the controller. The log message is displayed in the following format: ACL Audit: function createSchema [catalog=trading,dbUrl=dfs://db1,schema=stock2], called by user [xxx] User Access Control ------------------- ### Catalog-Level Access Control The following access privileges with prefix "CATALOG\_" can be set for databases/tables within a specific catalog. | Privilege | Description | | --- | --- | | CATALOG\_MANAGE | MANAGE permission for operations including adding schemas, drop and rename catalogs | | CATALOG\_READ | READ permission for all tables within a catalog | | CATALOG\_WRITE | WRITE permission for write, update, and insert operations on all tables within a catalog | | CATALOG\_INSERT | INSERT permission for all tables within a catalog | | CATALOG\_UPDATE | UPDATE permission for all tables within a catalog | | CATALOG\_DELETE | DELETE permission for all tables within a catalog | The parameter _objs_ of function `grant`/`deny`/`revoke` should be the catalog name. For example, grant user Adam CATALOG\_READ access on catalog "trading". grant("Adam", CATALOG_READ, "trading") ### Schema-Level Access Control The following access privileges with prefix "SCHEMA\_" can be set for schemas. | Privilege | Description | | --- | --- | | SCHEMA\_MANAGE | MANAGE permission for schemas | | SCHEMAOBJ\_CREATE | CREATE permission for all add DDL operations on tables within a schema | | SCHEMAOBJ\_DELETE | DELETE permission for all delete DDL operations on tables within a schema | | SCHEMA\_READ | READ permission for all tables within a schema | | SCHEMA\_WRITE | WRITE permission for write, update, and insert operations on all tables within a schema | | SCHEMA\_INSERT | INSERT permission for all tables within a schema | | SCHEMA\_UPDATE | UPDATE permission for all tables within a schema | | SCHEMA\_DELETE | DELETE permission for all tables within a schema | The parameter _objs_ of function `grant`/`deny`/`revoke` should be in the format of `catalog.schema`. For example, grant user Adam SCHEMA\_READ access on schema "trading.stock". grant("Adam", SCHEMA_READ, "trading.stock") Note: When managing permissions across multiple clusters, remember two key requirements: * Append `@` to usernames to specify their cluster membership; * Append `@` to _objs_ to indicate which cluster's resources are being accessed. For example, to grant user 1 from cluster 2 access to the trading.stock.quote table in cluster 1, you can execute the following script on the MoM node. grant("user1@cluster2", TABLE_READ, "trading.stock.quote@cluster1") Appendix: Operation Reference ----------------------------- This section provides a comprehensive summary of SQL statements and functions related to catalog management in DolphinDB, serving as a quick reference for users. | Function/ Statement | Description | | --- | --- | | `use` | Switch the current catalog. | | `create` | Create a schema or a table. | | `alter` | Add a column to an existing table. | | `drop` | Delete a schema or a table. | | `select` | Access data in a table. | | `update` | Update records in a table. | | `delete` | Delete records in a table. | | `setDefaultCatalog` | Set the default catalog for the current session. | | `existsCatalog` | Check if a specified catalog exists. | | `createCatalog` | Create a new catalog. | | `createSchema` | Add a specified database as a schema to a specified catalog. | | `dropDatabase` | Delete a database and referenced schema. | | `dropCatalog` | Delete a specified catalog. | | `dropSchema` | Delete a specified schema within a catalog. | | `getCurrentCatalog` | View the current catalog for the session. | | `getAllCatalogs` | Retrieve all catalogs. | | `getSchemaByCatalog` | Retrieve all schemas within a specified catalog. | | `renameCatalog` | Rename a catalog. | | `renameSchema` | Rename a schema. | | `getUserAccess` | Query the permissions assigned to a specific user or the combined effective permissions. | | `getGroupAccess` | Query the permissions of a group. | --- # Distributed Database Overview [Jump to main content](#wh_topic_body) Distributed Database Overview ============================= 1\. Benefits of Distributed Databases ------------------------------------- Partitioned databases can significantly reduce latency and improve throughput. * Partitioning makes large tables more manageable by enabling users to access subsets of data quickly and efficiently, while maintaining the integrity of a data collection. * Partitioning helps the system utilize all cluster resources. Data can be partitioned to multiple nodes and a task can be divided into multiple subtasks on these nodes. With distributed computing, these subtasks can be executed simultaneously to improve performance. 2\. The Differences Between DolphinDB and MPP Databases Regarding Partitions ---------------------------------------------------------------------------- MPP (Massive Parallel Processing) databases are widely adopted in popular systems such as Greenplum, Amazon Reshift, etc. In MPP a leader node connects to all client applications. DolphinDB adolpts a peer-to-peer structure in database and doesn't have a leader node. Each client application can connect to any compute node. Therefore DolphinDB doesn't have performance bottlenecks at the leader node. MPP databases usually split data across nodes with hashed sharding first (horizontal partitioning), then within each node conduct further partitioning (vertical partitioning). In DolphinDB, the partitioning logic and partition storage are independent of each other. The distributed file system (DFS) determines the storage locations of the partitions. Partitioning storage is optimized across the entire cluster. Compared with MPP databases, DolphinDB provides more fine-grained partitions. Data is more evenly distributed across partitions to better utilize cluster resources. As the distributed file system provides excellent partition management, fault tolerance, replica management and transaction management, a single table in DolphinDB can support millions of partitions. DolphinDB supports data storage and fast queries on datasets up to PBs. With DFS, database storage is independent of data nodes, making it extremely convenient to scale out the clusters. ![](images/DolphinDBvMPP.PNG) 3\. Partition Domains --------------------- DolphinDB supports range, hash, value, list and composite partitions. * Range partitions use ranges to form partitions. Each range is defined by 2 adjacent elements of a partition scheme vector. It is the most commonly used partition type. * Hash partitions use a hash function on a partitioning column. Hash partition is a convenient way to generate a given number of partitions. * In a value domain, each element of the partition scheme vector corresponds to a partition. * A list domain partitions data according to a list. It is more flexible than a range domain. * The composite domain is suitable for situations where multiple columns are frequently used in SQL **where** or **group by** clauses on large tables. A composite domain can have 2 or 3 partitioning columns. Each column can be of range, hash, value or list domain. For example, we can use a value domain on trading days, and a range domain on stock symbols. The order of the partitioning columns is irrelevant. When we create a new distributed database, we need to specify parameters "partitionType" and "partitionScheme". When we reopen an existing distributed database, we only need to specify "directory". We cannot overwrite an existing distributed database with a different "partitionType" or "partitionScheme". When we use aggregate functions on a partitioned table, we can achieve optimal performance if **group by** columns are also the partitioning columns. To call the function database, you must log in as an administrator, or a user with granted privilege **DB\_OWNER**. For example, you can log in using the default admin account: login(userId=`admin, password=`123456) The following examples are executed on local drives with an admin already logged in on Windows. To run in Linux servers or DFS clusters, just change the directory of the databases accordingly. ### 3.1. RANGE Domain In a range domain (RANGE), partitions are determined by ranges whose boundaries are two adjacent elements of the partition scheme vector. The starting value is inclusive and the ending value is exclusive. In the example below, database db has 2 partitions: \[0,5) and \[5,10). Table t is saved as a partitioned table pt with the partitioning column of ID in database db.\ \ n=1000000\ ID=rand(10, n)\ x=rand(1.0, n)\ t=table(ID, x)\ db=database("dfs://rangedb", RANGE, 0 5 10)\ pt = db.createPartitionedTable(t, `pt, `ID)\ pt.append!(t);\ \ pt=loadTable(db,`pt)\ select count(x) from pt;\ \ \ ![](images/database/range.png) \ \ The partition scheme of a range domain can be appended after it is created. Please check function `addRangePartitions` in user manual for details.\ \ ### 3.2. HASH Domain\ \ In a hash domain (HASH), partitions are determined by a hash function on the partitioning column. Hash partition is a convenient way to generate a given number of partitions. However, there might be significant differences between the partition sizes if the distribution of the partitioning column values is skewed. To locate observations on a continuous range in the partitioning column, it is more efficient to use range partitions or value partitions than hash partitions.\ \ In the example below, database db has 2 partitions. Table t is saved as a partitioned table pt with the partitioning column of ID in database db.\ \ n=1000000\ ID=rand(10, n)\ x=rand(1.0, n)\ t=table(ID, x)\ db=database("dfs://hashdb", HASH, [INT, 2])\ \ pt = db.createPartitionedTable(t, `pt, `ID)\ pt.append!(t)\ \ pt=loadTable(db,`pt)\ select count(x) from pt\ \ ### 3.3. VALUE Domain\ \ In a value domain (VALUE), each element of the partition scheme vector corresponds to a partition.\ \ n=1000000\ month=take(2000.01M..2016.12M, n)\ x=rand(1.0, n)\ t=table(month, x)\ \ db=database("dfs://valuedb", VALUE, 2000.01M..2016.12M)\ pt = db.createPartitionedTable(t, `pt, `month)\ pt.append!(t)\ \ pt=loadTable(db,`pt)\ select count(x) from pt;\ \ The example above defines a database db with 204 partitions. Each of these partitions is a month between January 2000 and December 2016. In database db, table t is saved as a partitioned table pt with the partitioning column of month.\ \ \ ![](images/database/value.png) \ \ The partition scheme of a value domain can be appended with new values after it is created. Please check function `addValuePartitions` in user manual for details.\ \ ### 3.4. LIST Domain\ \ In a list domain, each element of a vector represents a partition. The difference between a list domain and a value domain is that all the elements in a value domain partition scheme are scalars, whereas each element in a list domain partition scheme may be a vector.\ \ n=1000000\ ticker = rand(`MSFT`GOOG`FB`ORCL`IBM,n);\ x=rand(1.0, n)\ t=table(ticker, x)\ \ db=database("dfs://listdb", LIST, [`IBM`ORCL`MSFT, `GOOG`FB])\ pt = db.createPartitionedTable(t, `pt, `ticker)\ pt.append!(t)\ \ pt=loadTable(db,`pt)\ select count(x) from pt;\ \ The database above has 2 partitions. The first partition contains 3 tickers and the second contains 2 tickers.\ \ \ ![](images/database/list.png) \ \ ### 3.5. COMPO Domain\ \ A composite domain (COMPO) can have 2 or 3 partitioning columns. Each partitioning column can be of range, value, or list domain. The order of the partitioning columns is irrelevant.\ \ n=1000000\ ID=rand(100, n)\ dates=2017.08.07..2017.08.11\ date=rand(dates, n)\ x=rand(10.0, n)\ t=table(ID, date, x)\ \ dbDate = database(, VALUE, 2017.08.07..2017.08.11)\ dbID=database(, RANGE, 0 50 100)\ db = database("dfs://compoDB", COMPO, [dbDate, dbID])\ pt = db.createPartitionedTable(t, `pt, `date`ID)\ pt.append!(t)\ \ pt=loadTable(db,`pt)\ select count(x) from pt;\ \ The value domain has 5 partitions:\ \ \ ![](images/database/hier1.png) \ \ Click on a date partition, we can see the range domain has 2 partitions:\ \ \ ![](images/database/hier2.png) \ \ If one of the partitioning columns of a composite domain is of value domain, it can be appended with new values after it is created. Please check function `addValuePartitions` in user manual for details.\ \ 4\. Partition Guidelines\ ------------------------\ \ A good partition scheme can reduce latency and improve query performance and throughput. In this section we discuss the issues to consider in determining the optimal partitioning scheme.\ \ ### 4.1. Select Appropriate Partitioning Columns\ \ In DolphinDB, the data types that can be used for partitioning columns include integers (CHAR, SHORT, INT), temporal (DATE, MONTH, TIME, SECOND, MINUTE, DATETIME, DATEHOUR), STRING and SYMBOL. The HASH domain also supports LONG, UUID, IPADDR and INT128 types. Although a partitioning column can be of STRING type, for optimal performance, we recommend converting a STRING type column into SYMBOL type to be used as a partitioning column.\ \ FLOAT and DOUBLE types cannot be used as a partitioning column.\ \ db=database("dfs://rangedb1", RANGE, 0.0 5.0 10.0);\ The data type DOUBLE can't be used for a partition column.\ \ Although DolphinDB supports TIME, SECOND and DATETIME for partitioning columns, please avoid using them for value partitions. Otherwise each partition may be too small. It is very time consuming to create and query millions of small partitions.\ \ The partitioning columns should be relevant in most database transactions, especially database updates. For example, many tasks on financial databases involve stock symbols and dates. Therefore stock symbols and dates are natural candidates as partitioning columns. As will be discussed in section Transactions, DolphinDB doesn't allow multiple threads or processes to write to the same partition simultaneously when updating the database. Considering we may need to update data of a date or of a stock, if we use other partitioning columns such as trading hour, multiple writers may write to the same partition simultaneously and cause problems.\ \ A partitioning column is equivalent to a physical index on a table. If a query uses a partition column in **where** conditions, the system can quickly load the target data instead of scanning the entire table. Therefore, partitioning columns should be the columns that are frequently used in **where** conditions for optimal performance.\ \ ### 4.2. Partition Size Should Not Be Too Large\ \ The columns in a partition are stored as separate files on disk after compression. When a query reads the partition, the system loads the necessary columns into memory after decompression. Too large partitions may slow down the system, as they may cause insufficient memory with multiple threads running in parallel, or they may make the system swap data between disk and memory too frequently. As a rule of thumb, assume the available memory of a data node is S and the number of workers is W, then it is recommended that a partition is less than S/8W in memory after decompression,For example, with available memory of 32GB and 8 workers, a single partition should be smaller than 32GB/8/8=512MB.\ \ The number of subtasks of a query is the same as the number of partitions involved. Therefore if partitions are too large, the system cannot fully take advantage of distributed computing on multiple partitions.\ \ DolphinDB is designed as an OLAP database system. It supports appending data to tables, but does not support deleting or modifying individual rows. To modify certain rows, we need to overwrite entire partitions containing these rows. If the partitions are too large, it would be too slow to modify data. When we copy or move selected data between nodes, entire partitions with the selected data are copied or moved. If the partitions are too large, it would be too slow to copy or move data.\ \ Considering all the factors mentioned above, we recommend the size of a partition does not exceed 1GB before compression. This restriction can be adjusted based on the specific situations. For example, when we work with a table with hundreds of columns and only a small subset of columns are actually used, we can relax the upper limit of the partition size range.\ \ To reduce partitions sizes, we can (1) use composite partitions (COMPO); (2) increase the number of partitions; (3) use value partitions instead of range partitions.\ \ ### 4.3. Partition Size Should Not Be Too Small\ \ If partition size is too small, a query or computing job may generate a large number of subtasks. This increases the time and resources in communicating between the controller node and data nodes, and between different controller nodes. Too small partitions also result in innefficient reading/writing of small files on disk and therefore. Lastly, the metadata of all partitions are stored in the memory of the controller node. Too many small partitions may make the controller node run out of memory. We recommend that the partition size is larger than 100 MB on average before compression.\ \ Based on the analysis in the above section, we recommend the size of a partition be between 100MB and 1GB.\ \ The distribution of trading activities of stocks is highly skewed. Some stocks are extremely active but a majority of stocks are much less active. For a composite partition with partitioning columns of trading days and stock tickers, if we use value partitions on both partitioning columns, we will have many extremely small partitions for illiquid stocks. In this case we recommend range partitions on stock tickers where many illiquid stocks can be grouped into one partition, resulting a more evenly distributed dataset for better performance.\ \ ### 4.4. How to Partition Data Evenly\ \ Significant differences among partition sizes may cause load imbalance. Some nodes may have heavy workloads while other nodes idle. If a task is divided into multiple subtasks, it returns the final result only after the last subtask is finished. As each subtask works on a different partition, if data is not distributed evenly among partitions, it may increase the execution time.\ \ A useful tool for partitioning data evenly is function `cutPoints(X, N, [freq])`, where X is a vector; N means the number of buckets the elements of X will be grouped into; the optional argument freq is a vector of the same length as X indicating the frequency of each element in X. It returns a vector with (N+1) elements such that the elements of X are evenly distributed within each of the N buckets indicated by the vector. It can be used to generate the partition scheme of a range domain in a distributed database.\ \ In the following example, we construct a composite partition on date and symbols for high frequency stock quotes data. We use the data of 2007.08.01 to determine 128 stock symbol ranges with equal number of rows on 2007.08.01, and apply these ranges on the entire dataset.\ \ t = ploadText(WORK_DIR+"/TAQ20070801.csv")\ t = select count(*) as ct from t where date=2007.08.01 group by symbol\ buckets = cutPoints(t.symbol, 128, t.ct)\ \ dateDomain = database("", VALUE, 2017.07.01..2018.06.30)\ symDomain = database("", RANGE, buckets)\ stockDB = database("dfs://stockDBTest", COMPO, [dateDomain, symDomain]);\ \ ### 4.5. Partitions on Temporal Variables\ \ In the following example, we create a distributed database with a value partition on date. The partitioning scheme extends to the year 2030 to accommodate updates in future time periods.\ \ dateDB = database("dfs://testDate", VALUE, 2000.01.01 .. 2030.01.01)\ \ To partition the database by year, you can use RANGE partitions. The following example partitions the databases by year from 2020 to 2029.\ \ yearDB = database("dfs://testYear", RANGE, date(2020.01M + 12 * 0 .. 10)) \ \ When using temporal variables as a partition column, the data type of the partition scheme does not need to be the same as the data type of a partitioning column. For example, if we use month as the partition scheme in a value partition, the data type of the partitioning column can be month, date, datetime, timestamp, or nanotimestamp.\ \ ### 4.6. Partition Colocation\ \ It may be time consuming to join multiple tables in a distributed database, as the partitions that need to be joined with may be located on different nodes and therefore data need to be copied and moved across nodes. DolphinDB ensures that the same partitions of all the tables in the same distributed database are stored at the same node. This makes it highly efficient to join these tables. DolphinDB does not support joining tables from different distributed databases.\ \ dateDomain = database("", VALUE, 2018.05.01..2018.07.01)\ symDomain = database("", RANGE, string('A'..'Z') join `ZZZZZ)\ stockDB = database("dfs://stockDB", COMPO, [dateDomain, symDomain])\ \ quoteSchema = table(10:0, `sym`date`time`bid`bidSize`ask`askSize, [SYMBOL,DATE,TIME,DOUBLE,INT,DOUBLE,INT])\ stockDB.createPartitionedTable(quoteSchema, "quotes", `date`sym)\ \ tradeSchema = table(10:0, `sym`date`time`price`vol, [SYMBOL,DATE,TIME,DOUBLE,INT])\ stockDB.createPartitionedTable(tradeSchema, "trades", `date`sym)\ \ In the examples above, the distributed tables quotes and trades are located in the same distributed database.\ \ 5\. Import Data into Distributed Databases\ ------------------------------------------\ \ DolphinDB is an OLAP database system. It is designed for fast storage and query/computing of massive structured data and for high performance data processing with the in-memory database and streaming functionalities. It is not an OLTP system for frequent updates. When appending new data to a database on disk in DolphinDB, compressed data is inserted at the end of partitions or files in batches, similar to Hadoop HDFS. To update or delete existing rows, entire partitions that contain these rows need to be deleted first and then rewritten.\ \ ### 5.1. Replicas\ \ We can make multiple replicas for each partition in DolphinDB. The number of replicas is 2 by default and can be changed by setting the configuration parameter "dfsReplicationFactor".\ \ There are 2 purposes of replicas: (1) Fault tolerance: If a node is down, the system can use a replica for ongoing projects. (2) Load balance: With a large number of concurrent users, replicas can improve system throughput and decrease latency.\ \ DolphinDB adopts two-phase commit protocol to ensure strong consistency of all replicas of the same partition on different nodes when writing data into the database.\ \ The configuration parameter "dfsReplicaReliabilityLevel" in the controller configuration file (controller.cfg) determines whether multiple replicas are allowed to reside on nodes of the same physical server. In development stage, we can set it to 0 allowing multiple replicas on the same machine. In production stage, however, we should set it to 1 to ensure fault tolerance.\ \ ### 5.2. Transactions\ \ The DFS table engine in DolphinDB supports transactions, i.e., it guarantees ACID (atomicity, consistency, isolation and durability). The DFS table engine uses MVCC for transactions and supports snapshot isolation. With snapshot isolation, reading and writing do not block each other, therefore read operations in a data warehouse is optimized.\ \ To optimize the performance of queries and computing tasks in the data warehouse, DolphinDB has the following restrictions on transactions:\ \ * A transaction cannot involve both read operations and write operations.\ * A write transaction can write to multiple partitions, but a partition cannot be written to by multiple transactions simultaneously. If transaction A attempts to lock a partition while the partition is locked by another transaction, the system will immediately throw an exception and transaction A will be rolled back.\ \ ### 5.3. Parallel Data Writing\ \ In DolphinDB, a single table can have millions of partitions. This facilitates fast parallel data loading. Parallel data loading is especially important when huge amount of data is imported into DolphinDB, or when real-time data is persisted to the data warehouse with low latency.\ \ The following example loads stock quotes data to database stockDB in parallel. The data is stored in csv files, with each file representing a different day. The database stockDB uses a composite partition on date and stock symbol. For each file, create a jobID prefix with the file name, and use function `submitJob` to execute function `loadTextEx` to load data into database stockDB. Use command `pnodeRun` to execute the loading task at each data node of the cluster.\ \ dateDomain = database("", VALUE, 2018.05.01..2018.07.01)\ symDomain = database("", RANGE, string('A'..'Z') join `ZZZZZ)\ stockDB = database("dfs://stockDB", COMPO, [dateDomain, symDomain])\ quoteSchema = table(10:0, `sym`date`time`bid`bidSize`ask`askSize, [SYMBOL,DATE,TIME,DOUBLE,INT,DOUBLE,INT])\ stockDB.createPartitionedTable(quoteSchema, "quotes", `date`sym)\ \ def loadJob(){\ fileDir='/stockData'\ filenames = exec filename from files(fileDir)\ db = database("dfs://stockDB")\ \ for(fname in filenames){\ jobId = fname.strReplace(".csv", "")\ submitJob(jobId,, loadTextEx{db, "quotes", `date`sym, fileDir+'/'+fname})\ }\ }\ \ pnodeRun(loadJob);\ \ When multiple writers load data in parallel, we must make sure they wouldn't write data to the same partition simultaneously. Otherwise, at leat one transaction will fail. In the aforementioned example, different data files represent different partitions. Therefore, none of the writing jobs write to the same partition as another job.\ \ ### 5.4. Import Data\ \ We can also use function `append!` to import data into DolphinDB databases.\ \ n = 1000000\ syms = `IBM`MSFT`GM`C`FB`GOOG`V`F`XOM`AMZN`TSLA`PG`S\ time = 09:30:00 + rand(21600000, n)\ bid = rand(10.0, n)\ bidSize = 1 + rand(100, n)\ ask = rand(10.0, n)\ askSize = 1 + rand(100, n)\ quotes = table(rand(syms, n) as sym, 2018.05.04 as date, time, bid, bidSize, ask, askSize)\ \ loadTable("dfs://stockDB", "quotes").append!(quotes);\ \ #### 5.4.1. Import Data from Text Files\ \ DolphinDB provides 3 function to load text files: `loadText`, `ploadText`, and `loadTextEx`.\ \ workDir = "C:/DolphinDB/Data"\ if(!exists(workDir)) mkdir(workDir)\ quotes.saveText(workDir + "/quotes.csv")\ \ `loadText` and `ploadText` load text files smaller than available memory. `ploadText` loads data in parallel as a partitioned table in memory and is faster than `loadText`.\ \ t=loadText(workDir + "/trades.csv")\ ploadTable("dfs://stockDB", "quotes").append!(t)\ \ `loadTextEx` can load files much larger than available memory and can load data directly to databases. `loadTextEx` implicitly executes function `append!`.\ \ db = database("dfs://stockDB")\ loadTextEx(db, "quotes", `date`sym, workDir + "/quotes.csv")\ \ #### 5.4.2. Subscibe to a Stream Table and Import Data in Batches\ \ In the following example we subscribe to a stream table quotes\_stream. The streaming data from quotes\_stream will be inserted to table quotes if the incoming data reaches 10,000 rows or if 6 seconds have elapsed since the last time streaming data were inserted into table quotes, whichever occurs first.\ \ dfsQuotes = loadTable("dfs://stockDB", "quotes")\ saveQuotesToDFS=def(mutable t, msg): t.append!(select today() as date,* from msg)\ subscribeTable(, "quotes_stream", "quotes", -1, saveQuotesToDFS{dfsQuotes}, true, 10000, 6)\ \ #### 5.4.3. Import from Databases of Other Vendors via ODBC\ \ The following example imports table TAQquotes from MySQL.\ \ loadPlugin("/DOLPHINDB_DIR/server/plugins/odbc/odbc.cfg")\ conn=odbc::connect("Driver=MySQL;Data Source = mysql-stock;server=127.0.0.1;uid=[xxx];pwd=[xxx];database=stockDB")\ t=odbc::query(conn,"select * from quotes")\ loadTable("dfs://stockDB", "quotes").append!(t)\ \ #### 5.4.4. Import Data with Programming APIs\ \ DolphinDB provides APIs for Python, Java, C++, C#, R and JavaScript. After getting data with these languages, we can call function `append!` to import data into distributed databases in DolphinDB. The following script is an example for Java API.\ \ DBConnection conn = new DBConnection();\ \ Connect to a DolphinDB server:\ \ conn.connect("localhost", 8848, "admin", "123456");\ \ Define function saveQuotes:\ \ conn.run("def saveQuotes(t){ loadTable('dfs://stockDB','quotes').append!(t)}");\ \ Prepare a table:\ \ BasicTable quotes = ...\ \ Call function saveQuotes:\ \ List args = new ArrayList(1);\ args.add(quotes);\ conn.run("saveQuotes", args);\ \ 6\. Queries on Partitioned Tables\ ---------------------------------\ \ Most distributed queries do not involve all partitions of a distributed table. It could save a significant amount of time if the system can narrow down relevant partitions before loading and processing data.\ \ DolphinDB conducts partition pruning based on relational operators (<, <=, =, ==, >, >=, in, between) and logical operators (or, and) with the partitioning column(s).\ \ n=10000000\ id=take(1..1000, n).sort()\ date=1989.12.31+take(1..365, n)\ announcementDate=date+rand(1..10, n)\ x=rand(1.0, n)\ y=rand(10, n)\ t=table(id, date, announcementDate, x, y)\ db=database("dfs://rangedb1", RANGE, [1990.01.01, 1990.03.01, 1990.05.01, 1990.07.01, 1990.09.01, 1990.11.01, 1991.01.01])\ pt = db.createPartitionedTable(t, `pt, `date)\ pt.append!(t);\ \ pt=db.loadTable(`pt);\ \ The following queries can narrow down relevant partitions:\ \ select * from pt where date>1990.04.01 and date<1990.06.01;\ \ The system determines that only 2 partitions (\[1990.03.01, 1990.05.01) and \[1990.05.01, 1990.07.01)) are relevant to this query.\ \ select * from pt where date>1990.12.01-10;\ \ The system determines that only 1 partition \[1990.11.01, 1991.01.01) is relevant to this query.\ \ select count(*) from pt where date between 1990.08.01:1990.12.01 group by date;\ \ The system determines that only 3 partitions (\[1990.07.01, 1990.09.01), \[1990.09.01, 1990.11.01) and \[1990.11.01, 1991.01.01)) are relevant to this query.\ \ select * from pt where y<5 and date between 1990.08.01:1990.08.31;\ \ The system narrows down the relevant partitions to \[1990.07.01, 1990.09.01). Please note that in this step, the system ignores the condition of y<5. After loading the relevant partition, the system will use the condition of y<5 to further filter the data.\ \ The following queries cannot narrow down the relevant partitions. If used on a huge partitioned table, they will take a long time to finish. For this reason these queries should be avoided.\ \ select * from pt where date+10>1990.08.01;\ \ select * from pt where 1990.08.01 defined by the InfiniBand specification: RC (Reliable Connected)/RD (Reliable Datagram)/UC (Unreliable Connected)/UD (Unreliable Datagram) | | Data Transfer Method | Data copying, full-duplex communication | Zero-copy, direct memory access | | Memory Access | Through operating system kernel | Bypasses operating system kernel | | Programming Model | Socket API | Verbs API | | Advantages | Widely compatible, flexible and robust | High throughput, low latency data transmission, with large-scale parallel processing support | RDMA Performance Testing ------------------------ In scenarios with high network communication volume, enabling RDMA can lead to significant performance improvements. The test case below illustrates RDMA's memory writing performance. **Test Scenario:** Create an in-memory table on one node. Then initiate concurrent queries (sent through `submitJob`) from another node using the `rpc` function. The goal is to determine how many concurrent operations are needed to achieve 100 Gbit/s bandwidth (The fewer concurrent operations required, the better the performance). **Test Conditions:** * The in-memory table consists of 10 INT columns, 10 LONG columns, 10 FLOAT columns, and 10 DOUBLE columns, with a total size of approximately 3.8 GB. **Test Subjects:** * DolphinDB RPC using IPoIB * DolphinDB RPC using native RDMA **Test Results (Measured in Gbit/s):** ![](../images/rdma.png) **Conclusion:** As shown in the above plot, the NIC utilization with RDMA is twice that of IPoIB. Related Functions and Configurations ------------------------------------ Functions: [getConnections](../Functions/g/getConnections.html) Configuration parameters: [Inter-node Communication](Configuration/reference.html#topic_hzh_c2r_ncc__table_ql3_p2r_ncc) --- # Cluster Deployment and Upgrade [Jump to main content](#wh_topic_body) Cluster Deployment and Upgrade ============================== A DolphinDB cluster consists of 4 types of nodes: controller, agent, data node, and compute node. * **Controller**: The controllers are the core of a DolphinDB cluster. They collect heartbeats from agents and data nodes, monitor the node status, and manage metadata and transactions of the distributed file system. A single-machine or multi-machine cluster has only one controller. For a high-availability cluster with multiple controllers forming a Raft group, DolphinDB uses the Raft protocol to ensure strong consistency of metadata across controllers, * **Agent**: An agent executes the commands issued by a controller to start/stop local data nodes. Each physical server can have only one agent configured. * **Data node**: Data nodes store data and execute queries (or more complex computations). Each physical server can have multiple data nodes configured. * **Compute node**: Compute nodes handle queries and computations. They respond to client requests and return results. Compute nodes work with data nodes to effectively isolate storage and computing resources. Each physical server can have zero or multiple compute nodes configured. --- # DolphinDB Python Parser [Jump to main content](#wh_topic_body) DolphinDB Python Parser ======================= Introduction ------------ DolphinDB Python Parser interprets Python language and provides an execution environment for Python scripts. Unlike conventional Python, it is unrestricted by Global Interpreter Lock (GIL), unleashing the full potential of multi-core and distributed computing. By sharing the object system and runtime environment with DolphinDB scripts, Python Parser can directly access DolphinDB's storage engines and apply its built-in computing engine. Additionally, pandas, a third-party library for Python Parser, has been implemented. The following diagram briefly shows the structure of the script parsing system of DolphinDB. ![](images/0.1.png) Based on Python 3.10, Python Parser currently supports the most commonly used syntax in Python. For details, please refer to Syntax. In addition to Python syntax, Python Parser is also compatible with DolphinDB's distinctive syntax and introduces extended syntax including SQL, enhancing its expressiveness, as illustrated in the following examples: 1. In Python Parser, SQL query statements can be executed directly without using an API. def query(t): return select count(*) from t 2. As a built-in data type of Python Parser, temporal objects can be created directly with a literal constant (e.g. `2012.06M`), without entering `import datetime`. Currently, Python Parser is fully compatible with [constant data types in DolphinDB](DataTypesandStructures/DataTypesandStructures.dita) . month = 2012.06M For details about extended Python Parser syntax, please refer to Extended Syntax. **Note**: The current version of DolphinDB Python Parser is still in the Alpha trial phase. Differences from CPython ------------------------ The fundamental difference between Python Parser and CPython lies in their object models. Python Parser shares the same object models with DolphinDB, enabling seamless integration between the two. For example, Python Parser implements integers in the same way as C++ implements `int`. In contrast, CPython implements integer objects based on `struct _longobject`, which contains both data and type information. In addition, CPython initiates a new independent process for each execution. In comparison, Python Parser runs on the DolphinDB server. Therefore, before each execution, it is necessary to connect Python Parser to the server and create a session using client tools, such as GUI and Visual Studio Code (VS Code) Extension for DolphinDB. Differences from DolphinDB Python API ------------------------------------- Python API needs to be connected to the DolphinDB server in the Python environment before interacting with DolphinDB by executing DolphinDB scripts. In comparison, Python Parser runs directly on the DolphinDB server. --- # Standalone Deployment and Upgrade [Jump to main content](#wh_topic_body) Standalone Deployment and Upgrade ================================= This tutorial is a quick start guide describing how to deploy the DolphinDB server standalone and update the server and license file. You can also find solutions to common issues in the FAQ section. Standalone Deployment on Linux OS --------------------------------- ### Step 1: Download * Official website: [DolphinDB](https://dolphindb.com/product#downloads-top) * Or you can download DolphinDB with a shell command: wget https://www.dolphindb.com/downloads/DolphinDB_Linux64_V${release}.zip -O dolphindb.zip `${release}` refers to the version of DolphinDB server. For example, you can download Linux64 server 2.00.11.3 with the following command: wget https://www.dolphindb.com/downloads/DolphinDB_Linux64_V2.00.11.3.zip -O dolphindb.zip To download the ABI or JIT version of DolphinDB server, add "ABI" or "JIT" after the version number (linked with an underscore). For example, you can download Linux64 ABI server 2.00.11.3 with the following command: wget https://www.dolphindb.com/downloads/DolphinDB_Linux64_V2.00.11.3_ABI.zip -O dolphindb.zip Download Linux64 JIT server 2.00.11.3 with the following command: wget https://www.dolphindb.com/downloads/DolphinDB_Linux64_V2.00.11.3_JIT.zip -O dolphindb.zip * Then extract the installation package to the specified directory (`/path/to/directory`): unzip dolphindb.zip -d **Note**: The directory name cannot contain any space characters, otherwise the startup of the data node will fail. ### Step 2: Update License File If you have obtained the Enterprise Edition license, use it to replace the following file: /DolphinDB/server/dolphindb.lic Otherwise, you can continue to use the community version of DolphinDB, which allows up to 2 nodes, 2 CPU cores and 8 GB RAM per node. ### Step 3: Run DolphinDB Server Navigate to the folder _/DolphinDB/server/_. The file permissions need to be modified for the first startup. Execute the following shell command: chmod +x dolphindb * Linux console mode: ./dolphindb The default port number of the system is 8848. To change it (e.g., to 8900), use the following command line: ./dolphindb -localSite localhost:8900:local8900 * Linux background mode: sh startSingle.sh To check whether the node was successfully started, execute the following shell command: ps aux|grep dolphindb The following information indicates a successful startup: ![SingleNodeValidation](images/deploy_standalone/start_singlenode_linux_backend_success.png) ### Step 4: Check the Node Status on DolphinDB Web Enter the deployment server IP address and port number (8848 by default) in the browser to navigate to the DolphinDB Web. The server address (_ip_:_port_) used in this tutorial is 10.0.0.82:8848. Below is the web interface. ![SingleNodeStatusWeb](images/deploy_standalone/checknode_status_singlenode_web.png) **Note**: If the browser and DolphinDB are not deployed on the same server, you should turn off the firewall or open the corresponding port beforehand. Standalone Deployment on Windows OS ----------------------------------- ### Step 1: Download * Official website: [DolphinDB](https://www.dolphindb.com/alone/alone.php?id=75) * Extract the installation package to the specified directory: C:\DolphinDB **Note**: The directory name cannot contain any space characters, otherwise the startup of the data node will fail. For example, do not extract it to the Program Files folder on Windows. ### Step 2: Update License File If you have obtained the Enterprise Edition license, use it to replace the following file: C:\DolphinDB\server\dolphindb.lic Otherwise, you can continue to use the community version of DolphinDB, which allows up to 8GB of memory use for 20 years. ### Step 3: Run DolphinDB Server Navigate to the folder _C:\\DolphinDB\\server_: ![start_singlenode_win_folder](images/deploy_standalone/start_singlenode_win_folder.png) * Windows console mode: Double click to execute _dolphindb.exe_: ![singlenode_win_dbclickddbexe](images/deploy_standalone/singlenode_win_dbclickddbexe.png) The default port number of the system is 8848. You can change it by modifying the _localSite_ parameter in config file _dolphindb.cfg_. * Windows background mode: Double click to execute _backgroundSingle.vbs_, and check DolphinDB process in your Task Manager. ![singlenode_win_backend_vbs](images/deploy_standalone/singlenode_win_backend_vbs.png) Or you can check the process in Command Prompt with the following command. tasklist|findstr "dolphindb" ![singlenode_win_findstr](images/deploy_standalone/singlenode_win_findstr.png) ### Step 4: Check the Node Status on DolphinDB Web Enter the deployment server IP address and port number (8848 by default) in the browser to navigate into the DolphinDB Web. The server address (_ip_:_port_) used in this tutorial is 10.0.0.82:8848. Below is the web interface. ![singlenode_win_web_checknode](images/deploy_standalone/singlenode_win_web_checknode.png) **Note**: If the browser and DolphinDB are not deployed on the same server, you should turn off the firewall or open the corresponding port beforehand. Upgrade DolphinDB Server ------------------------ ### Upgrade on Linux **Step 1: Close the Server** Navigate to the folder _/DolphinDB/server/clusterDemo_ to execute the following command: ./stopAllNode.sh **Step 2: Backup the Metadata** The default directory to save the metadata for a standalone mode is: /DolphinDB/server/local8848/dfsMeta/ /DolphinDB/server/local8848/storage/CHUNK_METADATA/ You can execute the following command to back up the metadata: mkdir backup cp -r local8848/dfsMeta/ backup/dfsMeta cp -r local8848/storage/CHUNK_METADATA/ backup/CHUNK_METADATA **Note**: If the backup files are not in the above default directories, check the directories specified by the configuration parameters _dfsMetaDir_ and _chunkMetaDir_. If the configuration parameters are not modified but the configuration parameter _volumes_ is specified, then you can find the _CHUNK\_METADATA_ under the _volumes_ directory. **Step 3: Upgrade** **Note**: When the server is upgraded to a certain version, the plugin should also be upgraded to the corresponding version. * Online Upgrade Navigate to the folder _/DolphinDB/server/clusterDemo_ to execute the following command: ./upgrade.sh The following prompt is returned: ![singlenode_linux_upgrade_online_tip1](images/deploy_standalone/singlenode_linux_upgrade_online_tip1.png) Type "y" and press Enter: ![singlenode_linux_upgrade_online_tip2](images/deploy_standalone/singlenode_linux_upgrade_online_tip2.png) Type "1" and press Enter: ![singlenode_linux_upgrade_online_tip3](images/deploy_standalone/singlenode_linux_upgrade_online_tip3.png) Type a version number and press Enter. To upgrade to version 2.00.9.1, for example, enter 2.00.9.1 and press Enter. The following prompt indicates a successful upgrade. ![singlenode_linux_upgrade_online_success](images/deploy_standalone/singlenode_linux_upgrade_online_success.png) * Offline Upgrade Download a new version of server package from [DolphinDB website](https://www.dolphindb.com/alone/alone.php?id=75) Upload the installation package to _/DolphinDB/server/clusterDemo_. Take version 2.00.9.1 as an example. ![singlenode_linux_upgrade_offline_1](images/deploy_standalone/singlenode_linux_upgrade_offline_1.png) Navigate to the folder _/DolphinDB/server/clusterDemo_ to execute the following command: ./upgrade.sh The following prompt is returned: ![singlenode_linux_upgrade_offline_2](images/deploy_standalone/singlenode_linux_upgrade_offline_2.png) Type "y" and press Enter: ![singlenode_linux_upgrade_offline_3](images/deploy_standalone/singlenode_linux_upgrade_offline_3.png) Type "2" and press Enter: ![singlenode_linux_upgrade_offline_4](images/deploy_standalone/singlenode_linux_upgrade_offline_4.png) Type a version number and press Enter. To upgrade to version 2.00.9.1, for example, enter 2.00.9.1 and press Enter. The following prompt indicates a successful upgrade. ![singlenode_linux_upgrade_offline_5](images/deploy_standalone/singlenode_linux_upgrade_offline_5.png) **Step 4: Restart the Server** Navigate to the folder _/DolphinDB/server_ to start the server with the following command: sh startSingle.sh Open the web interface and execute the following script to check the current version of DolphinDB. version() ### Upgrade on Windows **Step 1: Close the Server** * In console mode, close the foreground process. * In background mode, close DolphinDB process from Task Manager. **Step 2: Back up the Metadata** The default directory to save the metadata for a standalone mode is: C:\DolphinDB\server\local8848\dfsMeta\ C:\DolphinDB\DolphinDB\server\local8848\storage\CHUNK_METADATA\ Create a new folder _backup_ under _C:\\DolphinDB_, and copy the following files to it: * file _dfsMeta_ under _C:\\DolphinDB\\server\\local8848_ * file _CHUNK\_METADATA_ under _C:\\DolphinDB\\server\\local8848\\storage_ ![singlenode_win_upgade_1](images/deploy_standalone/singlenode_win_upgade_1.png) **Note**: If the backup files are not in the above default directories, check the directories specified by the configuration parameters _dfsMetaDir_ and _chunkMetaDir_. If the configuration parameters are not modified but the configuration parameter _volumes_ is specified, then you can find the _CHUNK\_METADATA_ under the _volumes_ directory. **Step 3: Upgrade** * Download a new version of server package from [DolphinDB website](https://www.dolphindb.com/alone/alone.php?id=75) * Replace the existing server with all files (except _dolphindb.cfg_ and _dolphindb.lic_) in the current _\\DolphinDB\\server_ folder. **Note**: When the server is upgraded to a certain version, the plugin should also be upgraded to the corresponding version. **Step 4: Restart the Server** Double click to execute _dolphindb.exe_. Open the web interface and execute the following script to check the current version of DolphinDB. version() Update License File ------------------- ### Step 1: Replace the License File Replace an existing license file with a new one. License file path on Linux: /DolphinDB/server/dolphindb.lic License file path on Windows: C:\DolphinDB\server\dolphindb.lic ### Step 2: Update License File * Online Update Execute the following script in web interface: updateLicense() **Note:** * The client name of the license cannot be changed. * The number of nodes, memory size, and the number of CPU cores cannot be smaller than the original license. * The update takes effect only on the node where the function is executed. Therefore, in a cluster mode, the function needs to be run on all controllers, agents, data nodes, and compute nodes. * The license type must be either commercial (paid) or free. * Offline Update Restart DolphinDB server to complete the updates. FAQ --- **Q1: Failed to start the server for the port is occupied by other programs** The default port number of the system is 8848. If you cannot start the server, you can first check the log file _dolphindb.log_ under _/DolphinDB/server_. If the following error occurs, it indicates that the specified port is occupied by other programs. :Failed to bind the socket on port 8848 with error code 98 In such case, you can change to another free port in the config file. **Q2: Failed to access the web interface** Despite the server running and the server address (_ip_:_port_) being correct, the web interface remains inaccessible. ![singlenode_faq_1](images/deploy_standalone/singlenode_faq_1.jpg) A common reason for the above problem is that the browser and DolphinDB are not deployed on the same server, and a firewall is enabled on the server where DolphinDB is deployed. You can solve this issue by turning off the firewall or by opening the corresponding port. **Q3: Roll back a failed upgrade on Linux** If you cannot start DolphinDB server after upgrade, you can follow steps below to roll back to the previous version. * Step 1: Restore Metadata Files Navigate to the folder _/DolphinDB/server_ to restore metadata files from backup with the following commands: cp -r backup/dfsMeta/ local8848/dfsMeta cp -r backup/CHUNK_METADATA/ local8848/storage/CHUNK_METADATA * Step 2: Restore Program Files Download the previous version of server package from the official website. Replace the server that failed to update with all files (except _dolphindb.cfg_ and _dolphindb.lic_) just downloaded. **Q4: Roll back a failed upgrade on Windows** If you cannot start DolphinDB server after upgrade, you can follow steps below to roll back to the previous version. * Step 1: Restore Metadata Files Use metadata files from folder backup to replace the following files: file _dfsMeta_ under _local8848_ file _CHUNK\_METADATA_ under _local8848/storage_ * Step 2: Restore Program Files Download the previous version of server package from the official website. Replace the server that failed to update with all files (except _dolphindb.cfg_ and _dolphindb.lic_) just downloaded. **Q5: Failed to update the license file** Updating the license file online has to meet the requirements listed in [Step 2: Update License File](#step-2-update-license-file-2) . If not, you can choose to update offline or apply for an [Enterprise Edition License](https://www.dolphindb.com) . **Q6: Change configuration** For more details on configuration parameters, refer to [Configuration](../Database/Configuration/configuration.html) . If you encounter performance problems, you can contact our team on [Slack](https://dolphindb.slack.com/) for technical support. --- # Configuration [Jump to main content](#wh_topic_body) Configuration ============= DolphinDB offers configuration parameters that allow users to tailor configurations based on their specific situations. This helps optimize utilization of the machine's hardware resources. Use function [getConfig](../../Functions/g/getConfig.html) to check the configuration information. Note: Most parameters can be configured on both the data node and compute node. Since the compute node does not store data, there is no need to configure disk-related parameters. --- # Prepare a Linux Machine for Deployment [Jump to main content](#wh_topic_body) Prepare a Linux Machine for Deployment ====================================== This tutorial introduces how to set up the environment for DolphinDB deployment and helps you choose an appropriate deployment option suitable for your needs. If you already have a Linux machine configured, you can skip this tutorial. Environment Setup ----------------- ### Prerequisites #### Supported Platforms DolphinDB supports Linux x86 and ARM architecture. We use CentOS 7.9.2009 as an example to introduce system configurations. * If you use Linux system, it requires Linux kernel version 2.6.19 or higher. Stable release of CentOS 7 is recommended. * If you use Ubuntu system: * Select the _/home_ partition and choose the XFS file system as the format type in the Storage configuration during setup, thus formatting the _/home_ folder with the XFS file system; * Use command `sudo ufw` disable to turn off the firewall. See `man ufw` for more examples. * If you need to customize the core dumps, use command `sudo systemctl disable apport.service` to disable apport. #### Software Dependency DolphinDB relies on GCC 4.8.5 or higher. Use the following command to install GCC on CentOS 7 stable release: # yum install -y gcc #### Recommended Hardware Configuration To optimize the system performance, the following hardware configurations are recommended: * **Metadata & redo log**: It is recommended to use one SSD with small capacity; for higher reliability requirements, RAID1 with two SSDs are recommended; * **Data entity**: Use SSDs for better performance; or implement concurrent operations with multiple mechanical hard drives as a more economic choice. **Note**: The disk capacity and read/write performance using 1 SSD or multiple HDDs depend on actual business needs. ### File System and Inode #### Initialize the File System **Recommended**: For Linux system, XFS file system is recommended. It has the capability to not only accommodate hard links but also allows for the dynamic adjustment of the number of inode. If there are not enough inodes on the file system, you may encounter issues when DolphinDB attempts to write new files. In such case, additional inodes need to be added dynamically. **Not Recommended**: File system that does not support hard links, such as beegfs, is not recommended, due to its slow update performance. The root user or a user with root privileges (who has to add "sudo" to the beginning of the command) connects to the CentOS server via SSH. Use the `df -T` command to check the file system type: # df -T Filesystem Type 1K-blocks Used Avail Use% Mounted on devtmpfs devtmpfs 3992420 0 3992420 0% /dev tmpfs tmpfs 4004356 0 4004356 0% /dev/shm tmpfs tmpfs 4004356 8748 3995608 1% /run tmpfs tmpfs 4004356 0 4004356 0% /sys/fs/cgroup /dev/mapper/centos-root xfs 52403200 1598912 50804288 4% / /dev/sda1 xfs 1038336 153388 884948 15% /boot tmpfs tmpfs 800872 0 800872 0% /run/user/0 /dev/mapper/centos-home xfs 42970624 33004 42937620 1% /home The _/home_ folder is formatted in xfs, and the DolphinDB data file can be put under this directory. **Note**: The XFS file system is recommended only for the folder string data files. The installation, metadata, and log folder can be ext4 or other file systems. If the _/home_ folder is formatted in ext4 or other file systems that cannot change the number of inodes, follow steps below to format it with the XFS file system: (1) Back up data in _/home_ # cd / # cp -R /home /tmp (2) Unmount _/home_ and remove the corresponding logical volume # umount /home # lvremove /dev/mapper/centos-home Do you really want to remove active logical volume centos/home? [y/n]: y Logical volume "home" successfully removed (3) Check the available free space on your hard drive # vgdisplay | grep Free Free PE / Size 10527 / 41.12 GiB # left 41.12 GB (4) Create a new logical volume _/home_ and format it in xfs # lvcreate -L 41G -n home centos # the size of new logical volume is determined by available free space WARNING: xfs signature detected on /dev/centos/home at offset 0. Wipe it? [y/n]: y Wiping xfs signature on /dev/centos/home. Logical volume "home" created. # mkfs.xfs /dev/mapper/centos-home (5) Mount the _/home_ directory, and restore the data # mount /dev/mapper/centos-home /home # mv /tmp/home/* /home/ # chown owner /home/owner # Reassign privileges to the owner. It has to be modified with the user name. #### Set the Number of inodes for XFS File System If there is enough disk space for DolphinDB, but you cannot write data because there is no inode available, you can follow the steps below to increase the number of inodes: (1) Check the inode information: # xfs_info /dev/mapper/centos-home | grep imaxpct data = bsize=4096 blocks=10747904, imaxpct=25 # df -i | grep /dev/mapper/centos-home file system Inode used(I) avail(I) use(I)% Mounted on /dev/mapper/centos-home 21495808 7 21495801 1% /home The results show that 25% of the space under the _/dev/mapper/centos-home_ volume is allocated for inodes, and the number of available inodes is 21495801. (2) Increase the number of inodes: # xfs_growfs -m 30 /dev/mapper/centos-home meta-data=/dev/mapper/centos-home isize=512 agcount=4, agsize=2686976 blks = sectsz=512 attr=2, projid32bit=1 = crc=1 finobt=0 spinodes=0 data = bsize=4096 blocks=10747904, imaxpct=25 = sunit=0 swidth=0 blks naming =version 2 bsize=4096 ascii-ci=0 ftype=1 log =internal bsize=4096 blocks=5248, version=2 = sectsz=512 sunit=0 blks, lazy-count=1 realtime =none extsz=4096 blocks=0, rtextents=0 inode max percent changed from 25 to 30 (3) Check the inode information again: # df -i | grep /dev/mapper/centos-home /dev/mapper/centos-home 25794944 7 25794937 1% /home The number of available inodes has increased to 25794937. #### Set the Number of inodes for Ext File System When formatting a drive with an ext file system like ext4, it is important to consider the file size and growth to ensure the appropriate inode number. For example, you can specify the number of inodes with the `-N` option in the `mkfs.ext4` command. The default number of inode for the ext4 file system is relatively small, especially when the drive is particularly large, so it is recommended to set a larger value. ### Mount on a New Disk Performance improvement of disk I/O is crucial for big data processing. Recommended hardware configurations can be found in section 1.1.3. After physically installing a new hard drive or adding one via a cloud service provider, you can connect to the CentOS server via SSH. To add a new hard drive with a capacity of 50G via a virtual machine, follow these steps: (1) Check the disk information by `fdisk -l`: # fdisk -l ... Disk /dev/sdb: 53.7 GB, 53687091200 bytes, 104857600 sectors Units = sectors of 1 * 512 = 512 bytes Sector size(logical/physical): 512 bytes/ 512 bytes I/O size(minimum/optimal): 512 bytes / 512 bytes ... It can be seen that _/dev/sdb_ is the newly added 50G disk. (2) Use `fdisk` to create and modify partitions: # fdisk /dev/sdb Welcome to fdisk (util-linux 2.23.2). Changes will remain in memory only, until you decide to write them. Be careful before using the write command. Device does not contain a recognized partition table Created a new DOS disklabel with disk identifier 0x7becd49e Command(m for help): n # create a new partition Partition type: p primary (0 primary, 0 extended, 4 free) e extended Select (default p): p Partition number (1-4, default 1): First sector (2048-104857599, default 2048): the default value 2048 is used Last sector, +sectors or +size{K,M,G} (2048-104857599, default 104857599): the default value 104857599 is used Created a new partition 1 of type `Linux` and of size 50GiB Command(m for help): q (3) Use command `mkfs.xfs` to create an XFS file system: # mkfs.xfs /dev/sdb meta-data=/dev/sdb isize=512 agcount=4, agsize=3276800 blks = sectsz=512 attr=2, projid32bit=1 = crc=1 finobt=0, sparse=0 data = bsize=4096 blocks=13107200, imaxpct=25 = sunit=0 swidth=0 blks naming =version 2 bsize=4096 ascii-ci=0 ftype=1 log =internal log bsize=4096 blocks=6400, version=2 = sectsz=512 sunit=0 blks, lazy-count=1 realtime =none extsz=4096 blocks=0, rtextents=0 (4) Create a mount point and mount the drive: # mkdir -p /mnt/dev1 # mount /dev/sdb /mnt/dev1 (5) Check the file system type of the new disk: # df -Th Filesystem Type Size Used Avail Use% Mounted on devtmpfs devtmpfs 3.9G 0 3.9G 0% /dev tmpfs tmpfs 3.9G 0 3.9G 0% /dev/shm tmpfs tmpfs 3.9G 8.6M 3.9G 1% /run tmpfs tmpfs 3.9G 0 3.9G 0% /sys/fs/cgroup /dev/mapper/centos-root xfs 50G 1.6G 49G 4% / /dev/sda1 xfs 1014M 150M 865M 15% /boot /dev/mapper/centos-home xfs 41G 33M 41G 1% /home tmpfs tmpfs 783M 0 783M 0% /run/user/0 /dev/sdb xfs 50G 33M 50G 1% /mnt/dev1 (6) Set auto-mount at startup: # blkid /dev/sdb /dev/sdb: UUID="29ecb452-6454-4288-bda9-23cebcf9c755" TYPE="xfs" # vi /etc/fstab UUID=29ecb452-6454-4288-bda9-23cebcf9c755 /mnt/dev1 xfs defaults 0 0 Note that the UUID is used to identify the new disk, which avoids identification errors when the disk is replaced or its location is changed. ### Enable core dump [core dump](https://en.wikipedia.org/wiki/Core_dump) consists of the recorded state of the working memory of a computer program when the program has crashed or otherwise terminated abnormally. Therefore, if there is enough space, it is recommended to enable core dump. Below are steps to enable core dump: (1) Create a path to store core files: # mkdir /var/crash # The folder can be modified. **Note**: * Set aside enough space for the folder to hold core files, where the maximum value is about _maxMemSize_; * Ensure that DolphinDB has write access to the folder. (2) Enable core dump. # vi /etc/security/limits.conf * soft core unlimited It takes effect for all processes. You can also specify the process name to enable core dump only for DolphinDB. Unlimited indicates unlimited size for core files. (3) Set the output path and format for core files. # vi /etc/sysctl.conf kernel.core_pattern = /var/crash/core-%e-%s-%u-%g-%p-%t Format: * %e - the executable filename * %s - number of signal causing dump * %u - numeric real UID of dumped process * %p - PID of dumped process * %g - numeric real GID of dumped process * %t - time of dump * %h - hostname (4) Reboot and check if the changes take effect. # ulimit -c unlimited We recommend starting DolphinDB in standalone mode to verify whether the core dump configuration takes effect. $ cd /path_to_dolphindb $ chmod +x ./startSingle.sh $ ./startSingle.sh $ kill -11 dolphindb_pid $ ls /var/crash/ | grep dolphindb_pid # Check if there is a core file in the folder. ### Increase the Maximum Number of Open Files The number of files open simultaneously when running DolphinDB may exceed the default maximum number 1024 on CentOS 7. It is recommended to configure the maximum number of open files to 102400. You can follow steps below: (1) Check the maximum number of open files: # ulimit -n # on a process level 1024 # cat /proc/sys/fs/file-max # on a kernel level 763964 It can be seen that the maximum number of file descriptors open in a process is 1024, and 763964 in a Linux host. (2) Set the maximum limit to 102400 for the processes. # vi /etc/security/limits.conf * soft nofile 102400 * hard nofile 102400 It takes effect for all processes. You can also specify the process name to enable core dump only for DolphinDB. (3) If the configured value for the maximum number of open files in a host is less than 102400, you need to change it to a value no less than 102400: # vi /etc/sysctl.conf fs.file-max = 102400 The maximum limit of open files in the host here is 763964, which is greater than 102400, so it is not modified. (4) Reboot and check if the changes take effect. # ulimit -n # on a process level 102400 # cat /proc/sys/fs/file-max # on a kernel level 763964 ### NTPD If a multi-machine cluster is deployed, it is necessary to configure NTPD (Network Time Protocol Daemon) for time synchronization to ensure the time order of transactions. We configure NTPD server and client on two virtual machines. | Virtual Machine | IP | Subnet Mask | NTPD | | --- | --- | --- | --- | | VM 1 | 192.168.0.30 | 255.255.254.0 | server; time synchronization with cn.pool.ntp.org | | VM 2 | 192.168.0.31 | 255.255.254.0 | client; time synchronization with 192.168.0.30 | Configure with the following steps: (1) Install the NTP package. # yum install ntp (2) Set the time of VM 1 in synchronization with [cn.pool.ntp.org](http://cn.pool.ntp.org) # vi /etc/ntp.conf ... # Hosts on local network are less restricted. restrict 192.168.0.0 mask 255.255.254.0 nomodify notrap # Use public servers from the pool.ntp.org project. # Please consider joining the pool (http://www.pool.ntp.org/join.html). server 0.cn.pool.ntp.org iburst server 1.cn.pool.ntp.org iburst server 2.cn.pool.ntp.org iburst server 3.cn.pool.ntp.org iburst ... The 4th line configures the IP subnet to request the NTP service on the local network with the address 192.168.0.0 and subnet mask 255.255.254.0. The nomodify option prevents client to modify the configuration of the server. The notrap option uses the trap facility for remote event logging. (3) Enable the firewall on VM 1 and add the NTP service. # firewall-cmd --add-service=ntp --permanent # firewall-cmd --reload (4) Set the time of VM 2 in synchronization with VM 1. # vi /etc/ntp.conf ... # Use public servers from the pool.ntp.org project. # Please consider joining the pool (http://www.pool.ntp.org/join.html). server 192.168.0.30 iburst # server 0.centos.pool.ntp.org iburst # server 1.centos.pool.ntp.org iburst # server 2.centos.pool.ntp.org iburst # server 3.centos.pool.ntp.org iburst ... (5) Start NTPD on VM 1 and VM 2. # systemctl enable ntpd # systemctl restart ntpd (6) Wait for a few seconds, and then check the status of NTPD. VM 1: # ntpstat synchronised to NTP server (202.112.31.197) at stratum 2 time correct to within 41 ms polling server every 64 s VM 2: # ntpstat synchronised to NTP server (192.168.0.30) at stratum 3 time correct to within 243 ms polling server every 64 s ### Configure a Firewall Before running DolphinDB, you need to configure the firewall to open the ports of each node, or turn off the firewall. Take 8900 ~ 8903 TCP ports for example: # firewall-cmd --add-port=8900-8903/tcp --permanent # firewall-cmd --reload # systemctl restart firewalld Turn off the firewall using the following commands if it is not required: # systemctl stop firewalld # systemctl disable firewalld.service ### Swap Swap space, also known as virtual memory, is used as an extension of RAM. If the system needs more memory resources and the RAM is full, inactive pages in memory are moved to the swap space. Swap space allows a computer to effectively utilize more memory than is physically available, but it can also cause performance degradation if it is used excessively. If system performance prioritize to memory, it is recommended to disable Swap with the following steps: (1) Check available swap space. # free -h total used free shared buff/cache available Mem: 7.6G 378M 7.1G 8.5M 136M 7.1G Swap: 7.9G 0B 7.9G It can be seen that the total swap memory available is 7.9G. (2) Disable swap space temporarily. # swapoff -a (3) Check available swap space again. # free -h total used free shared buff/cache available Mem: 7.6G 378M 7.1G 8.5M 136M 7.1G Swap: 0B 0B 0B The total swap memory available is 0B. The swap space has been disabled. (4) To permanently disable swap space, comment out the line with the value of `swap` in the second column: # vi /etc/fstab ... # /dev/mapper/centos-swap swap swap defaults 0 0 ... --- # Visual Studio Code Extension [Jump to main content](#wh_topic_body) Visual Studio Code Extension ============================ Microsoft Visual Studio Code (VS Code) is a powerful and lightweight code editor with a rich extensibility model. VS Code extensions let you add languages, debuggers, and tools to your installation to support your development workflow. Install the DolphinDB Extension for VS Code to add the DolphinDB scripting language in VS Code, which enables you to write and execute scripts in VS Code to operate the DolphinDB database and access its data. Features -------- * Code highlighting * Code completion for keywords, constants, built-in functions * Documentation and parameter hints for built-in functions * Displays code execution results and `print()` output in the integrated terminal * Displays running script status in bottom status bar with option to click to cancel * Displays data structures like tables, vectors, matrices in browser pop-up windows * Displays connections, databases and session variables in the sidebar * Displays tables, vectors, and matrices in browser pop-up windows * Exports DolphinDB tables to disk (.csv file) ![](../images/vscode_1.png) Getting Started --------------- **Installation** 1. Install or upgrade VS Code to the latest version (above v1.68.0) at [https://code.visualstudio.com/](https://code.visualstudio.com/) 2. Install extension: Search for "dolphindb" in the VS Code **Extensions** view and click **Install**. Note: If the installation fails due to network reasons, open [Visual Studio Markstplace](https://marketplace.visualstudio.com/items?itemName=dolphindb.dolphindb-vscode) in your browser, click the **Version History** tab, download the latest version of the extension (with the suffix `.vsix`) to your local computer and drag and drop it to the Extensions view in VS Code. 3. After the installation is complete, quit all windows in the VS Code and restart the program for the installation to take effect. Otherwise you may not be able to view the variables in the browser (see below). **View and configure server connections** 1. After installing the DolphinDB extension, an icon will be added to the Activity Bar in the VS Code. Click the icon, the DolphinDB sidebar will be displayed with three views including CONNECTIONS, DATABASES, and VARIABLES.![](../images/vscode_2.png) 2. Click `File > Preferences > Settings` in the menu bar to open the VS Code settings.Enter "dolphindb" in the search box, click `edit in settings.json` to open the `dolphindb.connections` configuration item in the `settings.json` configuration file. 3. The `dolphindb.connections` configuration item is an array of objects where each object (wrapped by curly brackets "{ }") represents a connection. You can add or modify the connection objects to create or edit session connections as appropriate. `name` and `url` are required atrributes, and different connection objects must have different `name`. When connecting to a DolphinDB server, by default you'll be logged in as the DolphinDB admin ("autologin": true). Hover over an attribute to view its description. **Open or create a DolphinDB script file** * If the script file name is suffixed with `.dos` (short for DolphinDB Script), the extension will automatically recognize the DolphinDB language and enable syntax highlighting, code completion and prompts. * If the script file is not a `.dos` file and has a different suffix, such as `.txt`, please manually associate the DolphinDB language by following the steps below: 1. Click the **Select Language Mode** button in the lower right corner of the VS Code editor (see screenshot below): ![](../images/vscode_lang.png) 2. Enter "dolphindb" in the language selection pop-up box and press `Enter` to associate the current file with the DolphinDB language:![](../images/vscode_3.png) **Execute code** 1. In the opened DolphinDB script file, use the shortcut command `Ctrl` + `E` to send the code to the DolphinDB server for execution. When the code is executed for the first time, it will automatically connect to the selected connection listed under the DOLPHINDB tab. * Only selected code in the script will be sent to the server for execution. * If no code is selected, the line where the current cursor is located will be sent to the server for execution. 2. After the code is executed, there will be text-based output in the terminal below the VS Code editor. If the last statement of the executed code returns a table, array, or matrix, it will automatically switch to the DolphinDB area of the panel below the VS Code editor. Forms to display data structures such as tables, vectors, and matrices. It is recommended to drag the contents of the DolphinDB tab to the right side of the terminal, as shown in the figure below:![](../images/vscode_4.png)![](../images/vscode_5.png) **Switch connections and inspect session variables** After executing the code, you can perform the following actions (see screenshot below): * Switch the connection used to execute the code (the original connection will not be disconnected) * Click the "Disconnect" icon to the right of the connection to manually disconnect from server * View the value of a session variable * Use the 2 icons dispalyed to the right of each variable (excluding scalars or pairs): * Click the icon on the left to view the variables in the DolphinDB area of the lower panel of the editor * Click the icon on the right to directly open a browser pop-up window and view the variables in the pop-up window (you need to configure the browser to allow the pop-up window, see later). The popup function requires an open `DolphinDB Data Browser` tab in the browser (the URL may be [http://localhost:8321/](http://localhost:8321/) ). If this tab is missing, the plugin will automatically open this page first.![](../images/vscode_8.png) Note: To use the "Inspect Variable" icon, configure your browser to allow pop-ups. **Expand function documentation** * When entering a DolphinDB built-in function in the VS Code editor, suggestions and the documentation for the function will pop up as you type. Click the arrow icon ("Read More") next to the function prompt to dispaly/hide the documentation.![](../images/vscode_10.png) * You can also hover the mouse over a the name of a built-in function in your code to view its documentation. The accompanying documentation for the function will expand to the side. Debugging --------- The DolphinDB extension for VS Code provides debugging support for user scripts, which allows real-time tracing of script execution, displaying of intermediate variable values and showing function call stack information. For details, see [Debugging](vscode_debug.html) . File Importing and Exporting ---------------------------- DolphinDB's VS Code extension supports users to upload files. Users are supported to upload files in the following two ways: 1. Select the file to be uploaded in the explorer of VS Code and right-click, and select "DolphinDB: Upload to Server" in the right-click menu 2. After opening the file to be uploaded, click the upload button in the upper right corner of the VS Code interface. 3. After that, the user needs to input the path of the file uploaded to the server (cannot be empty), press Enter, and wait for the prompt "The file was uploaded successfully". DolphinDB's VS Code extension also supports users to export files. It requires the server version to be 2.00.11 or higher. Users can export tables with the following steps: 1. Execute scripts and display the table in the panel. 2. Click on the export icon in the top-left corner. 3. A file explorer window will open, prompting you to choose the desired directory and enter a filename for the exported file. 4. Click “Save” to export the file to the specified location. Upon successful export, a notification message will appear in the bottom-right corner. In addition, users can customize the mapping relationship between the local path and the server path by configuring the mappings property of dolphindb.connections, so that the plug-in can map the server path according to the mappings in the subsequent file upload process. 1. In the VS Code settings, select DolphinDB under extension, open the setting.json file, add or modify mappings in the connection that needs to be configured, the "key" on the left is the local address, and the "value" on the right is the server address.![](../images/vscode_11.png) The plugin will map the path according to the mappings configured by the user in the current connection. For example, the mappings configured in the user's current connection are: { "/path/to/local/": "/path/at/remote/", "/path/to/local/dir1/": "/data/server/dir1/", "D:/path/to/local/": "/data/server/", "default": "/data/server/" } 2. When users upload files, the path mapping rules are as follows: * Mapping is performed in an automatic manner. The key represents the local path, and the value represents the server path. After the configuration is complete, the longest matching item will be selected as the upload path. For example, the file path uploaded by the user is `/path/to/local/dir1/file.dos`, and `/path/to/local/` and `/path/to/local/dir1/` exist at the same time to match the user path, but the longest matching item `/path/to/local/dir1/` will be matched first. * The defalut field can be configured as a default match. If the current path does not match other items in dolphindb.mappings, the server path corresponding to the default will be used as the upload path. For example, the file path uploaded by the user is `/user/dosuments/file.dos`. At this time, if the rest of the mappings cannot be matched, the server path mapped by the `default` field will be used as the upload path, that is, `/data/server/file.dos` * If there is no match in dolphindb.mappings, use `getHomeDir() + /uploads/ + filename` as the upload path. Customization ------------- **Bolding function names** Add the following textmate rules to the VS Code configuration file `settings.json` "editor.tokenColorCustomizations": { "textMateRules": [\ // function: bold\ { "scope": "entity.name.function", "settings": { "fontStyle": "bold" }},\ { "scope": "support.function", "settings": { "fontStyle": "bold" }} ,\ ] }, FAQ --- **Code execution** * If there is no response to the `Ctrl + E` shortcut key, it may be that the DolphinDB language is not associated (syntax highlighting does not take effect at this time), or the shortcut key conflicts with other plugins, and you need to customize the shortcut key: Go to `File > Preferences > Keyboard Shortcuts` (`File > Preferences > Keyboard Shortcuts`) of VSCode to modify, enter `ctrl+e` in the search box, and delete other plug-ins that conflict with the `DolphinDB: Execute Code` hot key. * If you have been stuck in execution after executing the code, open `Help` > `Switch Developer Tools (DevTools)` at the top of vscode and switch to the console tab in the pop-up window to see if there is `Webview fatal error: Error: Could not register service workers: InvalidStateError: Failed to register a ServiceWorker: The document is in an invalid state.`. If there is such an error, please restart VS Code. If it still can't be solved, try the following methods to end all `Code.exe` processes and delete the service worker cache: * linux: `pkill code && rm -rf .config/Code/Service\ Worker/{CacheStorage,ScriptCache}` * windows: * 1. After exiting vscode, open the task manager and end all remaining vscode zombie processes `taskkill /F /IM Code.exe` 2. Open `C:/Users/your username/AppData/Roaming/Code/Service Worker/` in the file manager 3. Delete `CacheStorage` and `ScriptCache` two folders Reference [https://github.com/microsoft/vscode/issues/125993](https://github.com/microsoft/vscode/issues/125993) * If there is no automatic switch to the DolphinDB view at the bottom after executing the code and returning to the form, you need to reset the position of the DolphinDB view, as shown in the figure below.![](../images/vscode_7.png) * If the dataview (data view) panel is right-clicked and hidden, it cannot be displayed again, and the execution script has been stuck in execution. You need to execute the `defs()` function, then press `ctrl + shift + p` to call out the command panel, search for open view, click to open the view, and then search for the data view (English name is dataview), click to open. * VS Code has a memory limit of about `1 GB`. It is recommended to use `limit` to limit the number of returned records; or assign the result to a variable, such as `a = select * from`, and then click the button next to the variable in the sidebar to perform paging lazy loading and retrieve single page data on demand. * In order to display tables and other data in the browser, each VSCode window will start a local HTTP server, and its available port range can be configured through `dolphindb.ports`, the default is `8321-8420`, hover the mouse over ports to view Detailed explanation. The function displayed in the pop-up window in the browser requires a version of the past two years, such as Chrome 100+ or Edge 100+ or Firefox 100+. Development ----------- # Install the latest version of nodejs # https://nodejs.org/en/download/current/ # Install the pnpm package manager corepack enable corepack prepare pnpm@latest --activate git clone https://github.com/dolphindb/vscode-extension.git cd vscode-extension # Install project dependencies pnpm install # copy .vscode/settings.template.json to .vscode/settings.json cp .vscode/settings.template.json .vscode/settings.json # Refer to scripts in package.json # Build the development version pnpm run dev # Switch to the debug panel in VSCode and start the ddb.ext debugging task (you need to disable or uninstall the installed dolphindb plugin first) --- # Tiered Storage [Jump to main content](#wh_topic_body) Tiered Storage ============== 1\. Introduction ---------------- Tiered storage is a storage architecture that uses multiple tiers of storage with different speeds and capacities. Old data that is infrequently accessed ("cold data") can consume a lot of resources on high-performance storage media like SSDs. With tiered storage, cold data can be automatically moved from fast disks (e.g., SSDs) to slower, less expensive disks (e.g., HDDs) or object storage services, allowing frequently accessed ("hot") data to reside on high-speed storage tiers. This tutorial describes how to configure and use tiered storage in DolphinDB. 2\. Configuring Tiered Storage ------------------------------ ### 2.1. Specifying Configuration Parameters Before implementing tiered storage, the required configuration parameters must be specified. **1\. Cold data storage** coldVolumes=file:/{your_local_path},s3://{your_bucket_name}/{s3_path_prefix} _coldVolumes_ specifies the volumes to store the cold data. Multiple (local or S3) directories can be specified with a comma delimiter. A local path starts with the identifier "file://"; An S3 path is in the format of "s3:///" where "s3path" must be specified. For example: coldVolumes=file://home/mypath/hdd/,s3://bucket1/data/ **Note:** To prevent data from being overwritten by other nodes, specify a unique storage path for each data node. The storage path can be defined using a macro in the format: `/home/mypath/hdd/`, where`` is the alias of each data node. This action ensures that the system saves data for each node to a directory named after the node's alias. **2\. Amazon S3** If an S3 path is specified for _coldVolumes_, load the DolphinDB AWS plugin and configure the related parameters (`s3AccessKeyId`, `s3SecretAccessKey`, `s3Region`, `Endpoint`) accordingly. pluginDir=plugins //specify the directory for plugins. The AWS S3 plugin must be saved under the plugins directory. preloadModules=plugins::awss3 //the AWS S3 plugin will be automatically loaded on server startup s3AccessKeyId={your_access_key_id} s3SecretAccessKey={your_access_screet_key} s3Region={your_s3_region} s3Endpoint={your_s3_Endpoint} For more information on the DolphinDB AWS S3 plugin, see [README](../Plugins/aws.html) . ### 2.2. Setting Data Retention Period Use the _hoursToColdVolume_ parameter of the [setRetentionPolicy](../Functions/s/setRetentionPolicy.html) function to specify the data retention period. For example, cold data storage volumes have been specified as: coldVolumes=file://home/dolphindb/tiered_store/ The range of data that will be migrated to cold storage depends on the time values in the data. This requires the database to adopt time-based partitioning as one of its partitioning schemes. In this example, we create a VALUE partitioned DFS database using the time column, then append the data for the past 15 days to the database: db = database("dfs://db1", VALUE, (date(now()) - 14)..date(now())) //create a database VALUE partitioned on dates data = table((date(now()) - 14)..date(now()) as cdate, 1..15 as val) //create a table with a column containing dates of the last 15 days tbl = db.createPartitionedTable(data, "table1", `cdate) tbl.append!(data) Call [setRetentionPolicy](../Functions/s/setRetentionPolicy.html) to specify: Migrate data which has been kept for over 5 days (120 hours) to the cold volumes, and delete data which has been kept for over 30 days (720 hours). As the database has only one partitioning scheme on date values, the parameter _retentionDimension_ (indicating the level of the time-based partitions) is 0. setRetentionPolicy(dbHandle=db, retentionHours=720, retentionDimension=0, hoursToColdVolume=120) ### 2.3. Triggering Data Migration Once the retention policies are set, DolphinDB will perform migration checks every 1 hour in the background. In this example, we use the [moveHotDataToColdVolume](../Functions/m/moveHotDataToColdVolume.html) function to manually trigger a migration. pnodeRun(moveHotDataToColdVolume) //initiate data migration on each data node As a result, DolphinDB starts the tasks to migrate data in the range of \[current day - 15 days, current day - 5 days). Call [getRecoveryTaskStatus](../Functions/g/getRecoveryTaskStatus.html)\ to view the status of the migration tasks.\ \ rpc(getControllerAlias(), getRecoveryTaskStatus) //the result shows the tasks have been created\ \ Sample output (some columns are omitted):\ \ | **TaskId** | **TaskType** | **ChunkPath** | **Source** | **Dest** | **Status** |\ | --- | --- | --- | --- | --- | --- |\ | 2059a13f-00d7-1c9e-a644-7a23ca7bbdc2 | LoadRebalance | /db1/20230209/4 | NODE0 | NODE0 | Finish |\ | … | … | … | … | … | … |\ \ **Note:**\ \ 1. If multiple paths are specified for `coldVolumes`, the data will be transferred randomly to one of the specified directories.\ 2. If `coldVolumes` is a local path, the migration is conducted locally by copying files directly.\ 3. If `coldVolumes` is an AWS S3 path, data will be uploaded using the AWS S3 plugin in a multi-threaded manner. Migration to S3 may be slower than to a local path.\ 4. During migration, partitions involved will temporarily become unavailable for reads/writes.\ \ After migration completes, migrated partitions become read-only. You can query data in these partitions using `select` statements, but cannot `update`, `delete`, or `append!` the data.\ \ select * from tbl where cdate = date(now()) - 10 //query migrated data with the select statement\ update tbl set val = 0 where cdate = date(now()) - 10 //an error will be reported after executing the update statement \ \ Note that executing drop operations (e.g. `dropTable`, `dropPartition`, `dropDatabase`) will remove the corresponding data in the object storage service.\ \ To check permissions for partitions, call `getClusterChunksStatus`:\ \ rpc(getControllerAlias(), getClusterChunksStatus)\ \ Sample output (some columns are omitted):\ \ | **chunkId** | **file** | **permission** |\ | --- | --- | --- |\ | ef23ce84-f455-06b7-6842-c85e46acdaac | /db1/20230216/4 | READ\_ONLY _(indicating that this partition has been migrated)_ |\ | 260ab856-f796-4a87-3d4b-993632fb09d9 | /db1/20230223/4 | READ\_WRITE _(indicating that this partition has not been migrated)_ |\ \ 3\. Tiered Storage Architecture\ -------------------------------\ \ ### 3.1. Auto-Migration Trigger\ \ Once retention policies ([setRetentionPolicy](../Functions/s/setRetentionPolicy.html)\ ) have been defined, a background worker checks for data pending migration every hour based on each time-based partition. The worker searches in the time range \[current time - hoursToColdVolume - 10 days, current time minus hoursToColdVolume).\ \ If there is data that needs to be migrated, migration tasks are created. Once the migration starts, the system does not usually migrate all partitions of all databases simultaneously. Instead, migration occurs database by database, with only a portion of a single database's data migrated each hour. This strategy reduces system pressure and maximizes data availability.\ \ For example, databases "dfs://db1" and "dfs://db2" both partition data by time. The _hoursToColdVolume_ parameter is set to 120h, meaning data will be retained for 5 days:\ \ * At 17:00 on February 20, 2023, the system may migrate all db1 partitions in the range \[2023.02.05, 2023.02.15).\ * At 18:00 on February 20, 2023, the system may migrate all db2 partitions in \[2023.02.05, 2023.02.15).\ * If the system has been performing checks and handling migration tasks throughout the day, it will complete migrating partitions in \[2023.02.05 to 2023.02.15) in both databases by the end of February 20, 2023.\ \ ### 3.2. Data Migration Process\ \ DolphinDB's tiered storage leverages its data recovery mechanism to migrate partition replicas on each node to slower storage media - either local disks or AWS S3. The data migration process is as follows:\ \ 1. A user defines the _hoursToColdVolume_ parameter (data retention time) via the [setRetentionPolicy](../Functions/s/setRetentionPolicy.html)\ function.\ 2. A background worker identifies data eligible for migration based on partitions and creates corresponding recovery tasks.\ 3. The system executes tasks by uploading or copying data files to AWS S3 or local directories.\ 4. The partition metadata and paths are updated, and migrated partition permissions are changed to `READ_ONLY`.\ \ ### 3.3. Access to Migrated Data\ \ When you query data stored in S3 using a SQL "select" statement, the system uses methods of the DolphinDB S3 plugin to perform the necessary operations, such as reading data and file lengths, and listing all files in a directory. Aside from these S3-specific operations, the data access process works the same as usual.\ \ However, since accessing S3 over a network is much slower than reading from a local disk, and because data saved to S3 remains static, DolphinDB caches some S3 file metadata (such as file length) as well as the data itself. As a result, if the cached data matches a query, it can be returned directly from the cache instead of retrieving it from S3 over the network.\ \ 4\. Conclusion\ --------------\ \ DolphinDB's tiered storage feature allows cold data to be migrated regularly to slower disks or cloud storage. Users can access migrated data by using SQL "select" statements. --- # DolphinDB GUI [Jump to main content](#wh_topic_body) DolphinDB GUI ============= The DolphinDB GUI is a platform where users can edit and execute programs, manage database contents, and draw graphs. ![](../images/ide.png) Specifically, the DolphinDB GUI allows users to: 1. Establish or modify DolphinDB server connections. 2. Manage projects by editing and managing scripts. 3. Execute scripts and monitor execution progress. 4. Review or export DolphinDB script results. 5. Manage databases. Users can browse or delete datasets and generated objects. 6. Generate charts and graphs. DolphinDB supports commonly used chart types, such as line charts, bar charts, pie charts, scatter charts, histograms, etc. --- # TSDB Storage Engine [Jump to main content](#wh_topic_body) TSDB Storage Engine =================== TSDB is a storage engine based on the [LSM-Tree](https://en.wikipedia.org/wiki/Log-structured_merge-tree) (Log Structured Merge Tree) model. TSDB is optimized for handling time-series data, providing improved performance and efficiency in data storage, retrieval, and analysis. Design ------ ### LSM-Tree The LSM-tree is an ordered data storage structure primarily used in write-intensive applications. It optimizes both read and write operations using a combination of in-memory and disk-based structures. LSM-tree provides high write throughput and efficient space utilization through compaction. ### Sort Columns Sort columns (specified by the _sortColumns_ parameter of function `createPartitionedTable`) are a unique structure specific to the TSDB engine, playing a crucial role in the storage and retrieval processes. Sort columns are used to sort data. Data for each write is sorted based onsort columns before being written to the disk, ensuring data within transactions and level files is ordered. Note that there is no guaranteed order between different level files or within a single partition. This storage structure facilitates efficient query indexing and data deduplication. **Query Indexing** - **Sort Key** In DolphinDB, a sort key functions as an index to facilitate querying. When multiple sort columns are specified, the last must be a time column and all other columns form the sort key. If there is only one sort column, that column becomes the sort key. Records sharing the same sort key are sorted by timestamp and stored together in blocks. These blocks are the basic units for querying and compression. Maintaining sort key metadata (e.g., block offset) allows queries to quickly locate relevant blocks, enabling efficient access. This enhances retrieval speed, especially for range queries on sort key or exact matches on the first sort column. **Data Deduplication** TSDB implements data deduplication primarily to handle cases where records with identical _sortColumns_ values are encountered. Deduplication takes place during data writing or level file compaction. The deduplication policy is controlled by the _keepDuplicates_ parameter (set during table creation) with three options: ALL, LAST, and FIRST. Note: The TSDB deduplication policy prevents duplicate results, but doesn't eliminate redundant data on disk. When a query is executed, the system reads the relevant data blocks from level files based on the sort key. It then deduplicates this data in memory before returning the final result. Storage Structure ----------------- The following figure illustrates the storage structure of the TSDB engine. It adopts an LSM-Tree structure, comprising two parts: memory and disk. ![](images/TSDB/1.png) ### Cache Engine * **Write Buffer** (unsorted): Receives newly written data. * **Sorted Buffer**: After the unsorted write buffer reaches a threshold, it is converted into a read-only sorted buffer with its data sorted according to the specified _sortColumns_. ![](images/TSDB/2.png) ### Redo Log The redo log is a type of log file used to record database transactions, capturing all changes made to the database. It primarily serves to ensure the atomicity and durability of transactions. In the TSDB engine, data is first written to the redo log as separate redo log files. Metadata is data that provides information about other data, including information such as database structure, table definitions, constraints on data, indexes, and other database objects. Although metadata does not directly log transactions, it is typically stored in _meta.log_ to ensure that the database can be restructured during recovery. ### Level File **Level File Structure** After multiple sorted write buffers are created and their total size reaches a threshold, data in all the sorted write buffers is written to level files on disk. The Partition Attributes Across (PAX) layout is used: records are first sorted by the sort key entry and records with the same sort key are stored with column files. Each column's data is divided into multiple blocks, and the addresses of these blocks along with their corresponding sort key information are recorded at the end of the level file. A block is the smallest unit for query and compression, with its internal data sorted in the order of the time column. The structure of a level file is as follows: ![](images/TSDB/3.png) * **Header**: Contains reserved fields, table structure details, and transaction-related information. * **Sorted Col Data**: Data is sorted by the sort key entry, where each sort key column sequentially stores the block data. * **Zonemap**: Stores pre-aggregated information for the data, including the minimum, maximum, sum, and not-null count for each column. * **Indexes**: Contains index information for the sorted col data, such as the number of sort key entries, the record count for each sort key entry, the file offset for data in blocks, checksums, etc. * **Footer**: Stores the starting position of the zonemap, enabling the system to locate the pre-aggregation and indexing regions. Zonemap and indexes are loaded into the memory during query operations for indexing purposes. **Level File Levels** The organization of level files is as follows: ![](images/TSDB/4.png) Level files are divided into 4 levels, from Level 0 to Level 3. The higher the level, the bigger the size of each level file. Data from the cache engine is divided into 32 MB chunks, compressed, and stored by partition in the level file at Level 0. Lower level files are merged to generate higher-level files. Consequently, files at lower levels are relatively newer, while files at higher levels are relatively older. Read and Write Operations ------------------------- ### Writing Data When writing to a TSDB database, the data will be written through the following process: 1. **Recorded in the****redo log.** 2. **Cached in the****memory (cache engine)**: Simultaneously with the redo log write, data is written to and sorted in the TSDB cache engine. When data is initially written, it is appended to the end of the write buffer. After the unsorted write buffer reaches a threshold, it is sorted according to the specified _sortColumns_ and converted into a read-only sorted buffer. 3. **Flushed to disk**: When the cache engine's data volume reaches the predefined _TSDBCacheEngineSize_, the TSDB engine initiates a flush operation. The system writes the data by partition to level 0 files (each approximately 32MB in size). During the flush operation, data is sorted globally. Data undergoes two sorting operations: being sorted by sort columns first and then being sorted globally. This approach leverages the divide-and-conquer algorithm to enhance sorting efficiency. 4. **Background compaction**: The system automatically triggers a compaction when (1) the number of level files at a lower level exceeds 10, or the total file size at a specific level surpasses the size of a single-level file at the next higher level. The system merges these level files into a level file at the higher level; (2) TSDB level files meet the compaction criteria. In cases where there may be too many files without compaction, users can use `triggerTSDBCompaction` to manually trigger compaction. ### Reading Data The TSDB engine introduces an indexing mechanism based on the sort key, which is more suitable for point queries. Additionally, the TSDB engine does not use user-space caching but relies solely on the operating system's cache. The specific query process is as follows: 1. **Partition Pruning**: The system narrows down the relevant partitions based on the filtering condition. 2. **Loading Indexes**: The system traverses all level files in the involved partitions and loads the index information into memory. A lazy caching strategy is used here, which means that the index information will not be immediately loaded into memory until the first query hits the partition. Once loaded, the index information for that partition remains cached in memory unless evicted due to memory constraints. Subsequent queries involving this partition can skip this step and directly read index information from memory. Note: The size of the memory area for storing index data is determined by _TSDBLevelFileIndexCacheSize_. Users can monitor the memory usage of these indexes in real time using the [getLevelFileIndexCacheStatus](../../Functions/g/getLevelFileIndexCacheStatus.html) function. If the loaded index data exceeds the configured size, the system will employ LRU (Least Recently Used) to manage it. The threshold for this process can be adjusted by _TSDBLevelFileIndexCacheInvalidPercent_. 3. **Searching Data in Cache**: The system searches the data in the cache engine: performing sequential scan in the write buffer or performing a binary search in the sorted buffer. 4. **Searching Data on Disk**: Based on the index, the system locates the data blocks of the queried fields in the level files and decompresses them into memory. If the query's filtering conditions include sort columns, the index can be used to accelerate the query. 5. **Returning Query Results**: Merge the results from steps 3 and 4 and return them. **Indexing Mechanism** ![](images/TSDB/5.png) The in-memory index consists of two parts: * Block offset: indexes of the level file. * Zonemap: zonemap of the level file. The specific indexing process is as follows: 1. **Locating the sort key entry by indexes**: The indexes record the block address offsets for each field under every sort key entry in the level file. If the query condition includes the sort key field, the system can narrow down search range by partition pruning. Once a sort key index is hit, the system retrieves the address offsets for all blocks corresponding to the sort key entry. For example, if the sort columns are deviceId, location, and time, and the query condition is “deviceId = 0”, the system quickly locates all sort key entries where “deviceId = 0” and retrieves all block data. 2. **Locating blocks by zonemap**: After locating the sort key entry in step 1, the system checks the corresponding minimum and maximum values in the zonemap. If the query condition specifies additional fields, the system can further filter out blocks unrequired. For example, if the sort columns are deviceId, location, and time, and the query condition is the “time between 13:30 and 15:00”, the system uses the zonemap information of the time column to quickly identify that the relevant block is “block2”. The system then uses the indexes to locate the address offset of block2 across all level files, reads the block data from disk, and applies any necessary deduplication before returning the results to the user. ### Updating Data The data update process in the TSDB engine varies depending on the deduplication policy settings. 1. **Partition Pruning**: The system narrows down the relevant partitions based on the filtering condition. 2. **Updating Data in Memory**: The system loads all data from the corresponding partition into the memory and updates it. 3. **Writing Updated Data to a New Version Directory** (When _keepDuplicates_\=ALL/FIRST): The system writes the updated data back to the database and uses a new version directory (default is _\__) to store it. Old files are removed periodically (default is 30min). **Appending Updated Data** (When _keepDuplicates_\=LAST): The system directly appends the updated data to the level 0 files. Updates and changes are appended to new files rather than modifying existing level files. DolphinDB offers three methods for updating tables: * [update](../../Programming/SQLStatements/update.html) : Standard SQL syntax. * [sqlUpdate](../../Functions/s/sqlUpdate.html) : Dynamically generates the metacode of the SQL update statement. * [upsert!](../../Functions/u/upsert!.html) : Inserts rows into a keyed table, indexed table or DFS table if the values of the primary key do not already exist, or update them if the primary key do. Note: * Update operations with _keepDuplicates_\=LAST are advisable for scenarios requiring frequent updates. * Update operations with _keepDuplicates_\=ALL/FIRST modify the entire partition. It is important to ensure that the total size of partitions in each update does not exceed available system memory to avoid overflow. ### Deleting Data The data deletion process in the TSDB engine varies depending on the deduplication policy settings. When keeping all records or only the first record, the data deletion process is similar to the update process. The specific process is: 1. **Partition Pruning:** The system narrows down the relevant partitions based on the filtering condition. 2. **Deleting Data in Memory:** The system loads all data from the corresponding partition into memory and deletes data based on conditions. 3. **Writing Back of Retained Data:** The system writes the retained data back to the database and uses a new version (default is "_\__") to save it. Old files are removed periodically (default is 30min). When keeping only the latest record, soft deletion can be enabled through the _softDelete_ parameter. Soft deletion is recommended for scenarios requiring frequent data deletions, significantly improving deletion performance. Features of the TSDB Engine --------------------------- * Queries with partitioning columns and sort columns in filtering conditions have high performance. * Data sorting and deduplication are supported. * Suitable for storing in wide formats with hundreds or thousands of columns. It also supports complex data types and forms such as BLOB type and array vectors. * If only the last record is kept for duplicate records (set _keepDuplicates_ = LAST for function createPartitionedTable), to update a record, only the level file that this record belongs to needs to be rewritten instead of an entire partition. Note: * Relatively lower write throughput as data needs to be sorted in the cache engine and the level files might be compacted. * Relatively lower performance when reading data from an entire partition or columns in an entire partition. Usage Example ------------- ### Creating a Database Use the database function (with _engine_ specified as “TSDB”) to create a COMPO-partitioned TSDB database. (1) create with SQL statement: create database "dfs://test_tsdb" partitioned by VALUE(2020.01.01..2021.01.01),HASH([SYMBOL, 100]), engine='TSDB' (2) create by `database`: dbName="dfs://test_tsdb" db1 = database(, VALUE, 2020.01.01..2021.01.01) db2 = database(, HASH, [SYMBOL, 100]) db = database(directory=dbName, partitionType=COMPO, partitionScheme=[db1, db2], engine="TSDB") ### Creating a Table There are two ways to create a table within a TSDB database. (1) create with SQL statement (only partitioned table is supported): create table dbPath.tableName ( schema[columnDescription] ) [partitioned by partitionColumns], [sortColumns], [keepDuplicates=ALL], [sortKeyMappingFunction] For example, create a partitioned table within the TSDB database. tbName="pt1" colName = "SecurityID""TradeDate""TradeTime""TradePrice""TradeQty""TradeAmount""BuyNo""SellNo" colType = "SYMBOL""DATE""TIME""DOUBLE""INT""DOUBLE""INT""INT" tbSchema = table(1:0, colName, colType) db.createPartitionedTable(table=tbSchema, tableName=tbName, partitionColumns="TradeDate""SecurityID", compressMeth (2) create by `createPartitionedTable`: To create a partitioned table: createPartitionedTable(dbHandle, table, tableName, [partitionColumns], [compressMethods], [sortColumns], [keepDuplicates=ALL], [sortKeyMappingFunction]) To create a dimension table: createDimensionTable(dbHandle, table, tableName, [compressMethods], [sortColumns], [keepDuplicates=ALL], [sortKeyMappingFunction]) Since DolphinDB 3.00.1, users can design partitioning schemes by custom rules. A function call can be passed in when specifying partitioning columns and data is partitioned based on the function result. Below is an example: If the id\_date column contains data like ax1ve\_20240101\_e37f6 or 91f86\_20240102\_b781d, and you want to partition based on the date (e.g., 20240101), you can create the table as follows: // define a function to process the partitioning column data def myPartitionFunc(str) { return temporalParse(substr(str, 6, 8), "yyyyMMdd") } (1) create with SQL statement: create database "dfs://partitionFunc" partitioned by VALUE(2024.02.01..2024.02.02) create table "dfs://partitionFunc"."pt"( id_date STRING, ts TIMESTAMP, value DOUBLE ) partitioned by myPartitionFunc(id_date) (2) create by `createPartitionedTable`: db = database("dfs://partitionFunc", VALUE, 2024.02.01..2024.02.02) tb=table(100:0,["id_date", "ts", "value"],[STRING,TIMESTAMP, DOUBLE]) db.createPartitionedTable(table=tb, tableName=`pt, partitionColumns=["myPartitionFunc(id_date)"]) The example above provides a simple demonstration of how to create databases and tables in the TSDB engine. In applications, it's important to configure the database and table parameters based on the specific requirements. Below are some key parameter setting recommendations for creating databases and tables. Considerations for Databases and Tables Design ---------------------------------------------- ### Database **Partitioning Design** Recommended partition size for a TSDB database is 400MB - 1GB (before compression). Distributed queries load data by partition for parallel processing. Too large partitions may lead to insufficient memory, reduced query parallelism, and decreased update/delete efficiency. Conversely, too small partitions can result in excessive subtasks, increased node load, and metadata explosion on the control node. For detailed information, refer to [Data Partitioning](../data_partitioning.html) . **Concurrent Writes to the Same Partition - _atomic_** * ‘TRANS' (default): Concurrent writes to the same partition are not allowed. * ‘CHUNK': Multi-threaded concurrent writes to the same partition are allowed. **Note**: _atomic='CHUNK'_ may compromise transaction atomicity due to potential data loss from failed retries. ### Table **compressMethods** _compressMethods_ is a dictionary indicating which compression methods are used for specified columns. * Use SYMBOL for highly repetitive strings. There are less than 2^21 (2,097,152) unique values in a SYMBOL column of the table (refer to [S00003](../../Maintenance/ErrorCodeReference/S00003.html) for more information). * The delta (delta-of-delta encoding) compression method can be used for time-series data and sequential data (integers). * In other cases, use Lz4 compression method. **sortColumns** _sortColumns_ is a STRING scalar/vector that specifies the column(s) used to sort the ingested data within each partition. * The number of columns specified by _sortColumns_ should not exceed 4 and the number of sort key entries within each partition should not exceed 2000 for optimal performance. * Typical combinations: SecurityID+timestamp (finance), deviceID+timestamp (IoT). * Avoid using primary keys or unique constraints as _sortColumns_. * It is recommended to specify frequently-queried columns for _sortColumns_ and sort them in the descending order of query frequency, which ensures that frequently-used data is readily available during query processing. * If only one column is specified for _sortColumns_, the column is used as the sort key. Block data is unordered, so queries must scan every block. If multiple columns are specified for _sortColumns_, the last column must be a time column and the preceding columns are used as the sort keys. Data within blocks is sorted by time, so if the query includes time-based conditions, it can potentially skip full block scans. * _sortColumns_ can only be of INTEGER, TEMPORAL, STRING, or SYMBOL type. _sortKey_ can only be of TIME, TIMESTAMP, NANOTIME, or NANOTIMESTAMP type. Note: DolphinDB does not support constraints. It is not recommended to set primary keys or unique constraints as sort columns. Doing so would cause each row of data to become a sort key entry, severely degrading both write and query performance, and significantly increasing memory consumption. **sortKeyMappingFunction** _sortKeyMappingFunction_ is a vector of unary functions used for dimensionality reduction when sort key entries exceed recommended limits. Example: For 5,000 stocks partitioned by date with _SecurityID + timestamp_ as _sortColumns_, resulting in about 5,000 sort keys per partition, use _sortKeyMappingFunction=\[hashBucket{, 500}\]_ to reduce this to 500 sort key entries. **keepDuplicates** _keepDuplicates_ specifies how to deal with records with duplicate _sortColumns_ values with three options: ALL (keep all records), LAST (only keep the last record) and FIRST (only keep the first record). * If there are no specific requirements for deduplication, consider the following factors to determine the optimal _keepDuplicates_ setting: * High-Frequency Updates: Use _keepDuplicates=_LAST for more efficient append updates. * Query Performance: Use _keepDuplicates=_ALL to avoid extra deduplication overhead during queries. * When _atomic_\=CHUNK, use _keepDuplicates=_FIRST/ LAST. In case of concurrent write failures, data can be rewritten directly without the need for deletion and reinsertion. Performance Tuning ------------------ Factors such as partitioning and indexing can significantly impact the performance of data writing and querying in the TSDB engine. Therefore, it is crucial to understand the relevant parameter configurations when creating databases and tables. This section will focus on how to optimize TSDB's writing and querying performance through careful parameter settings. * **TSDBCacheEngineSize**: The capacity (in GB) of the TSDB cache engine. The default value is 1. If the parameter is set too small, data in the cache engine may be flushed to disk too frequently, thus affecting the write performance; If set too high, a large volume of cached data is not flushed to disk until it reaches _TSDBCacheEngineSize_ (or after 10 minutes). If power failure or shutdown occurs in such cases, numerous redo logs are to be replayed when the system is restarting, causing a slow startup. * **TSDBLevelFileIndexCacheSize**: The upper limit of the size (in GB) of index data (level file indexes and zone maps). The default value is 5% of _maxMemSize_. If set too small, frequent index swapping may occur. The zonemap, a memory-intensive component, can be estimated using the formula: (number of partitions) × (number of sort keys) × (4 × sum of each field's byte size). Here, 4 represents the four pre-aggregation metrics: min, max, sum, and notnullcount. * **TSDBAsyncSortingWorkerNum**: A non-negative integer indicating the number of threads for asynchronous sorting in the TSDB cache engine. The default value is 1. Increasing this value when sufficient CPU resources are available to improve write performance. * **TSDBCacheFlushWorkNum**: The number of worker threads for flushing the TSDB cache engine to disk. The default value is the number of disk volumes specified by _volumes_. If the configured value is less than the number of disk volumes, the default value is used. Typically, this configuration does not need to be modified. --- # Primary Key Storage Engine [Jump to main content](#wh_topic_body) Primary Key Storage Engine ========================== DolphinDB version 3.00.1 introduces a brand new storage engine - Primary Key Engine (PKEY) - designed to store data with unique identifier to each record within a table. This enhancement results in faster sorting, searching, and querying operations. Design Considerations --------------------- The PKEY engine is specifically engineered to seamlessly integrate with OLTP (Online Transaction Processing) systems based on CDC (change data capture). In normal cases, transaction processing systems involve a large number of update and delete operations in addition to insert operations. When synchronizing data from a transaction processing system, a downstream database or data warehouse addresses the following key requirements: * Data integrity and consistency: Preserves the primary key constraint of the source OLTP system. * Near real-time data manipulation: Enables swift handling of updates, insertions, and deletions on a per row basis to managing write-heavy workloads typically associated with transactional databases. * Flexible ad hoc queries: Supports on-the-fly queries. While both TSDB and OLAP engines can, to some extent, address the requirement of preserving primary keys, they each face certain challenges in fully meeting this need. * The OLAP storage plan faces challenges with real-time writes, as rewriting data files is necessary for each write operation. * The TSDB storage plan presents challenges for data query, particularly with dynamic ad hoc queries that may require multiple index types, due to: (1) expensive deduplication costs; (2) inability to utilize indexes for non-primary key column. ### Popular Strategies The following approaches are commonly considered for handling data updates and queries. #### Copy-on-Write This strategy duplicates data files when changes occur, writing updates to a new copy. * **Pros**: This method excels in query performance and simplifies primary key uniqueness management during queries. * **Cons**: It's less suitable for frequent update scenarios due to the costly process of rewriting the data file with each write operation. * **Examples** of systems using this approach include Hudi, Delta Lake, and DolphinDB's OLAP engine. #### Merge-on-Read It employs a strategy where updates and changes are appended to new files rather than modifying existing data files. Then during data queries, multiple versions of data files need to be read online to deduplicate data and find the latest data. * **Pros**: This approach offers superior write performance and is well-suited for frequent write scenarios, as it doesn't strictly enforce uniqueness during writes. * **Cons**: It has limitations in query performance optimization, as queries must read and merge new files for deduplication. Additionally, it restricts the ability to filter data using non-primary key column indexes. * **Examples** of systems using this approach include ClickHouse, StarRocks Unique Key table, and DolphinDB's TSDB engine (_keepDuplicates_\=LAST). #### Merge-on-Write This strategy is realized by using the primary key index and the **delete bitmap**. When data is written, the delete bitmap is updated to mark rows for deletion or overwriting. Then during data reading, rows that have been marked are directly filtered out, and only the latest data would be read. * **Pros:** This strategy ensures that only the latest record among the records with the same primary key value needs to be read during queries, which eliminates the need to merge multiple versions of data files. Moreover, predicates and indexes can be pushed down to the underlying data, which greatly improves query performance. * **Cons:** The need to update the delete bitmap for each file introduces some overhead, potentially impacting write performance. * **Examples** of systems using this approach include Hologres, ByteHouse, Doris Unique Model, StarRocks Primary Key table. ### Solutions for the PKEY Engine Considering the scenario for streaming data in real time from transaction processing systems into DolphinDB, the constraints of the TSDB and OLAP solutions highlight the need for a solution that can create a trade-off between excellent query performance and the ability to handle high-frequency, real-time write operations effectively. To address these challenges, the PKEY storage engine implements two key design principles: **LSM Tree Architecture + Merge-on-Write**. This design enables real-time data inserts and updates through sequential writing, significantly enhancing write performance. It completes all data deduplication work during the data write phase, thus providing excellent query performance. Storage Structure ----------------- The PKEY engine shares similarities with the TSDB engine, which is developed based on LSM Tree, utilizing a column-oriented storage structure. In PKEY databases, both the Immutable MemTable and level files sort data based on the primary key. A key distinction lies in how data updates and deletes are performed: The PKEY engine writes a delete bitmap file (see [File Structure](#file_structure) for details) that tracks the position in that file of records to be deleted. The following figure illustrates the storage structure of the PKEY database. ![](images/primary_key_01.png) The storage structure of PKEY database consists of the following components: **In Memory,** * **MemTable (Cache Engine)**: Two types of memory tables are maintained: * Mutable MemTable: Receives newly written data. * Immutable MemTable: When the Mutable MemTable reaches its capacity, its data is sorted by the primary key and converted into an Immutable MemTable. * **Stashed Primary Key Buffer**: A double-buffer design is adopted during the disk flushing process. It temporarily stores two key pieces of information: (1) the primary key and (2) the CID (commit ID) column of the data. It enables background updates of delete bitmap and facilitates deduplication during query. **On Disk,** * **REDO**: Logs transactions to ensure atomicity and durability. Write operations first record data in the redo log before inserting it into the MemTable. * **META**: Contains storage engine metadata, including meta.log for meta information and multi-version delete bitmap files. * **STORAGE**: Houses the data of the storage engine, organized into four levels. Each level contains multiple level files. Records within a level file are sorted by their primary key. File Structure -------------- ### Level File In the PKEY engine, the Level File concept is analogous to SSTable (Sorted Strings Table). The PKEY storage engine implements a lazy leveling strategy for compaction: * Level 0-2: Multiple level files may have overlapping primary key ranges. * Level 3: Ranges of primary keys of level files never overlap. This approach is designed to minimize read amplification (i.e., the number of disk _reads_ per query) and space amplification (i.e., the ratio of the size of data files on disk to data size in the database) at level 3, optimizing query performance and storage efficiency. **Level File Name** Level files in the PKEY engine are named using a three-part structure to facilitate efficient management and data ordering: * **level ID**: Indicates the level of the file (starting from 0). * **flush ID**: Represents the most recent flushing operation that contributed data to this file. * **sequence number**: Denotes the file's position (starting from 0) within its **sorted run**. A sorted run consists of one or multiple level files and each level file belongs to exactly one sorted run. Within a sorted run, ranges of primary keys of level files never overlap. For example, a file named "1-000000014-001" indicates that this file is on the Level 1 (the second level), containing data from flush operation 14, and it is the second file in its sorted run. **Level File Structure** The structure of level files in a PKEY database is composed of six main components. See the following figure. ![](images/primary_key_02.png) where, * **Header**: Contains file metadata, including maximum and minimum CIDs, column type info, and primary key column details. * **Data Block**: Stores data, which is sorted by primary key both within and between blocks. Each block can hold up to 8192 rows. * **Index Block**: Supports two index types: Bloomfilter and ZoneMap. * **Data Block Index**: Stores information of data blocks, including column number, file offsets, and row counts. * **Index Block Index**: Stores information of index blocks, including index type, column number, and file offsets. * **Footer**: Stores the file offsets for the Data Block Index and Index Block Index. For optimal file access efficiency, all six components are aligned to a 4 KB boundary. ### Delete Bitmap File The PKEY storage engine employs the delete bitmap to track data deletions and overwrites across its level files. Each delete bitmap is approximately 120KB in size, which enables storing deletion markers for up to 1 million rows of data. Therefore, one delete bitmap file is sufficient to store the deletion status for all level files in a partition. The following figure illustrates a delete bitmap file structure. ![](images/primary_key_03.png) where, * **Bitmap**: Stores the delete bitmap for each level file within the partition, indicating whether the row has been deleted or overwritten. If a row is deleted, mark the row as 1. Each bitmap consists of a binary sequence. For example, the Bitmap 1 in the above figure indicates that the 2nd, 6th, 9th, 11th, 12th and 15th rows of the corresponding level file have been marked as deleted or overwritten. * **Footer**: Stores the information of corresponding level files, including file offset, level ID, flush ID, and sequence number. **Multi-Version Delete Bitmap Files** The delete bitmap file is stored in multiple versions to avoid conflicts between query and write processes. Delete bitmap files are named with the flush ID. For example, a file named "delmap-000000014" is the delete bitmap for level files containing the updated data with flush ID 14. Each delete bitmap file will take the version with the latest flush ID as the snapshot. The _meta.log_ file tracks the latest flush ID for delete bitmaps in each partition, which enables queries to retrieve the most recent delete bitmap version from the metadata for accurate data deduplication. Read and Write Operations ------------------------- The following figure demonstrates the workflow of read and write operations in a PKEY database. ![](images/primary_key_04.png) ### Writing Data When writing to a PKEY database, the data will be written through the following phases: 1. **Recorded in the** **redo log**: To ensure atomicity and durability. 2. **Cached in the** **memory (cache engine)**: After the redo log is successfully written, the data moves to the cache engine, which involves two steps: 1. Data is first cached to the **Mutable MemTable**. 2. Once the Mutable MemTable reaches 8MB, the records are sorted by primary key and it is converted to an **Immutable MemTable**. 3. **Flushed to disk**: When the cache engine's data volume reaches the predefined _PKEYCacheEngineSize_, the PKEY engine initiates a flush operation. This process writes all in-memory data to disk, creating the first level of storage. 4. **Background delete bitmap update**: During data flushing (step 3), the system extracts and temporarily stores the primary key and CID of the data in the **Stashed Primary Key buffer**, based on which the delete bitmap will be updated asynchronously in batches as a background operation. Depending on the number and size of files, updating one batch of the delete bitmap usually takes longer than a single flushing of data. In order to mark as many deleted or overwritten rows as possible in a delete bitmap update, the system will check whether the size of Stashed Primary Key reaches _PKEYDeleteBitmapUpdateThreshold_ to trigger the update of delete bitmap. The delete bitmap update follows these steps: 1. Uses each file's ZoneMap to determine which data blocks require merging. 2. Uses binary search to check and update the delete bitmap. For rows with identical primary keys, the row with the lower CID on disk is marked as deleted. 3. Writes updated delete bitmaps for all files into a single delete bitmap file, naming it with the most recent flush ID. 5. **Background Compaction**: After each flushing and compaction operation, a background thread evaluates the need for level file compaction based on specific trigger conditions. If compaction is required, it proceeds according to a predetermined file selection strategy. 1. Trigger condition: Checks if (1) the amount of data, or (2) the percentage of deleted or overwritten rows in each level exceeds the threshold. When multiple levels meet the condition, a scoring system is applied to determine which level has priority for compaction. 2. File selection policy: Once a level is chosen for compaction, the system selects files within that level: Starting from the oldest file, the system identifies overlapping sorted runs and calculates a score for each potential set of files to compact. If the trigger condition is the amount of data, files with the most sorted runs are favored. If the trigger condition is the ratio of deleted rows, files with the most deleted rows are prioritized. ### Reading Data The PKEY engine's query process shares similarities with the TSDB engine when the TSDB database only keeps the last record for duplicate values (_keepDuplicates_ = LAST). However, the PKEY engine distinguishes itself by pushing down predicate conditions to the process before deduplication and utilizing Bloomfilter and other indexes for query acceleration. The data retrieval process unfolds as follows: 1. **In-Memory Data Access**: Initially retrieves required columns from the cache engine. For primary key-based point queries, it employs a binary search across each MemTable. 2. **Metadata Snapshot Retrieval**: The latest versions of level file info, and delete bitmaps are fetched from the metadata. 3. **Disk Data Extraction**: This phase involves multi-level filtering using various indexes to create a bitmap for each file, identifying relevant data blocks: 1. Bloomfilter: For point queries or exact-match queries on Bloomfilter-indexed columns, a file-level bloom filter screens each level file to identify potential result-containing files. 2. ZoneMap: For exact-match or range queries, block-level filtering is conducted on each level file to pinpoint qualifying blocks. 3. Delete Bitmap: Following initial filtering, the file's delete bitmap is cross-referenced with the row number bitmap to exclude deleted and overwritten rows. 4. **Predicate Evaluation**: The predicate pushdown is performed. 5. **Result Deduplication**: Query results undergo a deduplication process, considering both the cache engine data and the stashed primary keys in memory. If both the cache engine and stashed primary key are empty, this step is skipped. 6. **Result Delivery**: After filtering and deduplication, the system returns the final query result to the user. ### Updating Data The PKEY engine treats an update operation as a combination of a targeted query followed by a data write. This process unfolds in the following steps: 1. **Query Phase**: The system executes a query based on the given predicate conditions to locate the specific data that needs updating. 2. **Modification Phase**: Once the relevant data is retrieved, the system updates the records and corresponding delete bitmaps. 3. **Write Phase**: The updated data is then inserted into the database following the standard write process of the PKEY engine. ### Deleting Data Similar to the update process, an deletion operation unfolds in the following steps: 1. **Query Phase**: The system executes a query based on the given predicate conditions to locate the specific data that needs deleting. 2. **Write Phase**: Then marks the data as deleted in delete bitmap. The corresponding CID will be updated as a negative number as a delete marker. Features of the PKEY Engine --------------------------- The PKEY storage engine is designed to create a trade-off between read and write operations, offering a set of distinctive features: * **Primary key integrity**: Ensures uniqueness of primary keys while supporting near real-time updates. * **Efficient point queries**: Excels in queries using primary key columns with predicate conditions, delivering high-speed retrievals. * **Multiple indexing for range queries**: * Bloomfilter: Facilitates file-level filtering, ideal for point queries on high-cardinality columns. * ZoneMap: Enables block-level data filtering, particularly effective for time-related column queries. * **Competitive Write Performance**: At moderate data throughput levels, the write performance of PKEY engine is comparable to TSDB engine. Note: As throughput increases, write time grows more rapidly compared to TSDB engine. Configuration Options --------------------- The following lists the potential configurations that a PKEY database may require. Before using a PKEY database, configure with appropriate values. * **PKEYRedoLogDir** and **PKEYMetaLogDir**: Paths where the PKEY meta log files and redo log files will be stored. To improve write performance, it is recommended to configure the path to SSDs. * **PKEYCacheEngineSize**: The capacity (in GB) of the PKEY cache engine. The default value is 1. For scenarios with large write volumes, setting it higher can improve write performance. Conversely, in situations where query performance is critical, a lower value might be more appropriate. * **PKEYBlockCacheSize**: The capacity (in GB) of the PKEY block cache. The default value is 1. Setting it higher for optimal query performance. * **PKEYDeleteBitmapUpdateThreshold**: The buffer size threshold (in MB) for updating the PKEY delete bitmap. The default value is 100. * If set too small, the delete bitmap will be updated too frequently, consuming excessive disk bandwidth. This can negatively impact queries, cache engine flushing, and level file compaction. * If set too large, the deduplication process during queries becomes more resource-intensive, potentially increasing query time. The system recovery process after a reboot may be significantly slower. * **PKEYStashedPrimaryKeyBufferSize**: The buffer size (in MB) for the stashed primary key. The default value is 1024. Under excessive write load, frequent disk flushing and delayed buffer clearing due to slow delete bitmap updates will result in the buffer to grow continuously. This parameter serves as a safeguard against potential OOM issues that could arise from an indefinitely growing staging buffer during periods of high write activity. * **PKEYBackgroundWorkerPerVolume**: The size of worker pool for the PKEY level file compaction and delete bitmap updates on each volume. The default and minimum value is 1. * **PKEYCacheFlushWorkerNumPerVolume**: The size of worker pool for the PKEY cache engine flushing on each volume. The default and minimum value is 1. Usage Example ------------- ### Creating a Database Use the database function (with _engine_ specified as "PKEY") to create a COMPO-partitioned PKEY database. dbName="dfs://test_pkey" db1 = database(, VALUE, 2020.01.01..2021.01.01) db2 = database(, HASH, [SYMBOL, 100]) db = database(directory=dbName, partitionType=COMPO, partitionScheme=[db1, db2], engine="PKEY") ### Creating a Table There are two ways to create a table within a PKEY database. 1. create with built-in functions To create a partitioned table createPartitionedTable(dbHandle, table, tableName, [partitionColumns], [compressMethods], [primaryKey], [indexes]) To create a dimension table createDimensionTable(dbHandle, table, tableName, [compressMethods], [primaryKey], [indexes]) 2. create with SQL statement (only partitioned table is supported) create table dbPath.tableName ( schema[columnDescription] ) [partitioned by partitionColumns], [primaryKey] For example, create a partitioned table within the PKEY database. tbName = "pt1" colName = `SecurityID`TradeDate`TradeTime`TradePrice`TradeQty`TradeAmount`BuyNo`SellNo colType = `SYMBOL`DATE`TIME`DOUBLE`INT`DOUBLE`INT`INT tbSchema = table(1:0, colName, colType) // create with built-in function db.createPartitionedTable(table=tbSchema, tableName=tbName, partitionColumns=`TradeDate`SecurityID, primaryKey=`SecurityID`TradeDate`TradeTime, indexes={"BuyNo": "bloomfilter"}) // create with SQL statement create table "dfs://test_pkey"."pt1"( SecurityID SYMBOL, TradeDate DATE, TradeTime TIME, TradePrice DOUBLE, TradeQty INT, TradeAmount DOUBLE, BuyNo INT [indexes="bloomfilter"], SellNo INT ) partitioned by TradeDate,SecurityID primaryKey=`SecurityID`TradeDate`TradeTime Performance Tuning ------------------ This section introduces how to improve your read and write performance by properly setting the following parameters. **partitionColumns** _partitionColumns_ defines the database's partitioning strategy, which directly impacts data distribution and access patterns. For maximum write efficiency, structure your data and queries so that each batch write or update operation affects as few partitions as possible. * Avoid using continuously increasing IDs or unique IDs as partitioning columns, as this can lead to poor write performance due to constant creation of new partitions. * Instead, choose partitioning columns that naturally group related data together, allowing for more localized write operations. In this regard, you can adopt a custom solution for data partitioning based on user-defined rules to flexibly accommodate diverse partitioning requirements. For instance, for data with a column in the format `id_date_id` (e.g., ax1ve\_20240101\_e37f6), you can partition by date using a user-defined function: // Define a function to extract the date information def myPartitionFunc(str,a,b) { return temporalParse(substr(str, a, b),"yyyyMMdd") } // Use myPartitionFunc to process the data column pt = db.createPartitionedTable(table=tb, tableName=`pt, partitionColumns=["myPartitionFunc(DateId, 6, 8)","SecurityID"], primaryKey=`SecurityID`DateId`TradeTime, indexes={"BuyNo": "bloomfilter"}) **primaryKey** The _primaryKey_ parameter defines the database's primary key columns and should contain all partitioning columns. For optimal performance, it's recommended to prioritize ordered columns such as timestamps or incrementing IDs. This enhances storage efficiency by grouping related data together and thus improves query performance. **indexes** The _indexes_ parameter sets the index type for non-primary key columns. Currently, only "bloomfilter" index type is available. Bloomfilter indexing excels in point queries on high-cardinality columns (e.g., ID card numbers, order numbers, foreign keys from upstreams). Columns not specified in _indexes_ default to ZoneMap indexing, which is particularly effective for range queries, especially on time-based columns. ZoneMap performance improves with more concentrated batches of written data, making it ideal for columns such as order time or event occurrence time. --- # pipeline [Jump to main content](#wh_topic_body) pipeline ======== Syntax ------ `pipeline(initTasks, followers, [queueDepth=2])` Arguments --------- **initTasks** is the collection of the initial steps of all tasks, which is represented by a zero-argument function. For example, if we have 10 tasks then _initialTasks_ is a tuple of 10 zero-argument functions. **followers** is a set of unary functions, each of which represents a step of the task after the initial step. If a task consists of N steps, followers should have N-1 unary functions. The output of a follower is the input of the next follower. The last follower may or may not return an object. The initial step of tasks is executed in the main thread (the thread that accepted the tasks) and the remaining steps will be executed in separate threads. To execute tasks of N steps, the system creates N-1 threads and these threads will be destroyed upon completion of the job. **queueDepth** is the maximum depth of the queue for the next step. The intermediate result of each step is stored in the queue for the next follower. When the queue is full, the execution will stop when the next step comsumes data from the queue. The deeper the queue, the less waiting time for the next step. However, a deeper queue consumes more memory. _queueDepth_ is optional and the default value is 2. Details ------- Optimize tasks that meet the following conditions through multithreading: 1. Can be decomposed into multiple sub-tasks. 2. Each subtask contains multiple steps. 3. The kth step of the ith subtask can only be executed after the (k-1)th step of the ith subtask and the kth step of the (i-1)th subtask are completed. If the last step (follower) returns an object, the `pipeline` function returns a tuple. Otherwise, it returns nothing. Examples -------- In the following example, we need to convert the partitioned table stockData into a CSV file. This table contains data from 2008 to 2018 and exceeds the available memory of the system, so we cannot load the entire table into memory and then converted it into a CSV file. The task can be divided into multiple sub-tasks, each of which consists of two steps: load one month of data into memory, and then store the data in the CSV file. To store the data of a month in the CSV file, it must be ensured that the data of the month has been loaded into the memory, and the that data of the previous month has been stored in the CSV file. v = 2000.01M..2018.12M def loadData(m){ return select * from loadTable("dfs://stockDB", "stockData") where TradingTime between datetime(date(m)) : datetime(date(m+1)) } def saveData(tb){ tb.saveText("/hdd/hdd0/data/stockData.csv",',', true) } pipeline(each(partial{loadData}, v),saveData); --- # SQL Reference [Jump to main content](#wh_topic_body) SQL Reference ============= This chapter covers how to use SQL to access, retrieve and manipulate data in DolphinDB. DolphinDB's SQL syntax is very similar to the ANSI SQL language in a Relational Database Management System (RDBMS) such as MySQL, Oracle, SQL Server, etc. Syntax ------ select [top_clause] column_expressions from table_name | table_expression [where filtering_conditions] [grouping_clause [having_clause] | order_clause] Common features with ANSI SQL ----------------------------- * Support `select`, `insert`, `update` and `delete` statement for retrieving, inserting, updating and deleting records in a table, respectively. Since 1.30.17/2.00.5, DolphinDB supports the create statement to create a database (table), and the alter statement to add columns to a table. Starting in version 1.30.22/2.00.10, all DolphinDB SQL keywords can be written in all-capital letters (e.g., SELECT, FROM, WHERE). * Support the `where` clause. * Support `group by` and `order by` clauses. * Support table join, including `inner join`, `left join`, `left semijoin`, `full join`. * Starting from version 2.00.10, line breaks are supported for SQL statements. Note that: * Keywords with multiple words (such as ORDER BY, GROUP BY, UNION ALL, INNER JOIN) cannot be split into two lines. * If an alias for a column or table is not specified with keyword _as_, it must follow the original name without a line break. What are the differences? ------------------------- * Most functions can be directly called in SQL queries. * Other differences as listed below. | ANSI SQL syntax | DolphinDB syntax | Explanation | | --- | --- | --- | | N/A | context by | context by is an innovation of DolphinDB. It simplifies processing time-series data withineach group. With group by, each group returns a scalar; with context by, each group returnsa vector of the same size as the group's records. | | N/A | pivot by | pivot by transforms a vector into a matrix or table. | | N/A | cgroup by | Perform cumulative grouping calculations | | N/A | map | Execute the SQL statement on each partition separately, then merge the results. | | N/A | aj | Asof join. It takes each record in the left table as a reference and checks if there isa match in the right table. If there is no match, the most recent observation will be chosen.If there are more than one match, the last one will be chosen. | | N/A | wj, pwj | Window join and prevailing window join. They are a generalization of asof join. For each rowin the left table, window join applies aggregate functions on a window of rows in the right table.If the right table doesn't have a matching value for the window, prevailing window join willfill it with the last value before the window and then apply the aggregate functions. | Compatibility for SQL Dialects ------------------------------ Since version 1.30.22/2.00.10, DolphinDB has enhanced the compatibility for Oracle and MySQL dialects. In addition to supporting ANSI SQL syntax, DolphinDB deals with the inconsistent behaviors of functions with the same name due to dialect-specific features: You can select a dialect mode in a session and run scripts written in that dialect. Currently, three dialect modes are available: DolphinDB, Oracle and MySQL. **Note:** * Scripts written in DolphinDB language can be correctly parsed in Oracle or MySQL dialect mode. * Only part of the functions or features of Oracle/MySQL are supported: | SQL Dialect | Features | Functions (case insensitive) | | --- | --- | --- | | Oracle | Comment symbols: --, /\*\*/

Concatenation operator:\| | asciistr, concat, decode, instr, nvl, nvl2, rank, regexp\_like, to\_char, to\_date, to\_number, and trunc

Note: to\_char only accepts numeric, DATE, DATEHOUR, and DATETIME types.

For syntax references of Oracle SQL functions, see [SQL Language Reference](https://docs.oracle.com/en/database/oracle/oracle-database/19/sqlrf/Functions.html#GUID-D079EFD3-C683-441F-977E-2C9503089982)
. | | MySQL | | sysdate

For syntax references of MySQL functions, see [MySQL :: MySQL 8.0 Reference Manual :: 12 Functions and Operators](https://dev.mysql.com/doc/refman/8.0/en/functions.html)
. | --- # VectorDB [Jump to main content](#wh_topic_body) VectorDB ======== Since version 3.00.1, DolphinDB introduces VectorDB, a vector retrieval database system built on the TSDB storage engine. It applies rapid approximate nearest neighbor search (ANNS) to array vector columns. Key features of DolphinDB VectorDB include: * Persistent storage of vector indexes, allowing pre-built indexes to be read from disk without rebuilding for each query. * Efficient vector similarity search for massive vector data using rapid approximate nearest neighbor searches. * Hybrid searches combining keyword-based search algorithms with vector search techniques for comprehensive and precise results. Implementation -------------- This section introduces the implementation of vector storage, indexing and search in VectorDB. ### Vector Storage In DolphinDB, vector data can be stored as [array vectors](../../Programming/DataTypesandStructures/DataForms/Vector/arrayVector.html) , a data form natively supported by the TSDB storage engine. This allows for creating a DFS table with vector indexing using TSDB as its underlying architecture for efficient vector data management. ### Vector Indexing Vector indexing is constructed based on Facebook's open-source library, [Faiss](https://github.com/facebookresearch/faiss) . When creating a DFS partitioned table, if an index is specified for an array vector column, a vector index is constructed based on the specified index type and stored in each level file. Index types supported in VectorDB and their trade-offs will be explained in the following section. ### Vector Search VectorDB uses [Euclidean distance (L2 distance)](https://en.wikipedia.org/wiki/Euclidean_distance) to measure similarity between high-dimensional vector data. L2 distance is widely used and performs well in low-dimensional spaces, though its effectiveness can deteriorate in high-dimensional spaces due to the curse of dimensionality. For a vector search query written in the following format, VectorDB selects an appropriate search method to retrieve the _K_ nearest vectors to the query vector. SELECT {col1,...,colx} FROM table [where] ORDER BY rowEuclidean(, queryVector) LIMIT **Basic Vector Search** If no `where` clause is specified, the basic search process is as follows: 1. Iterate through all level files (_N_ in total) on disk and read out vector indexes, obtaining _N \* K_ preliminary query results. 2. Perform brute-force search on data (if any) in the cache engine, yielding an additional _K_ results. 3. Merge sort the _(N + 1) \* K_ query results based on their L2 distance from the query vector in ascending order, and return the _K_ closest results. **Hybrid Search** If `where` conditions are specified, VectorDB uses hybrid searches combining keyword-based search algorithms with vector search techniques to improve the accuracy and relevance of search results. The process for hybrid search is as follows: 1. Filter data based on the `where` conditions. 2. Divide the selected data by level file. 3. For each level file, search and obtain query results. 4. Merge sort these query results and return the _K_ closest results. Supported Index Types --------------------- Vector index is a data structure for efficient vectors retrieval, featuring more time and space-efficient vector similarity query capabilities. Through vector indexes, users can quickly query several vectors most similar to a target vector. Currently, VectorDB supports the following index types: 1. **Flat**: Exact nearest neighbor retrieval through brute-force search. Ideal for small datasets or when highest accuracy is required. 2. **PQ (Product Quantization)**: A lossy algorithm that splits vectors into sub-vectors and applies quantization for memory efficiency. Suitable for retrieving large datasets such as large databases and video libraries. 3. **IVF (Inverted File)**: Uses k-means clustering to divide vector data into clusters and builds an inverted index for each cluster. Common in image retrieval and text retrieval applications. 4. **IVFPQ (Inverted File Product Quantization)**: Combines IVF and PQ techniques. Balanced approach for large-scale datasets requiring speed, accuracy, and memory efficiency, such as large-scale recommendation systems or user matching in social networks. 5. **HNSW (Hierarchical Navigable Small World)**: Builds a multi-level navigable graph structure for efficient search. Ideal for very large datasets with high requirements for speed, accuracy, and dynamic updates, such as real-time recommendation systems, online search, and Retrieval-Augmented Generation (RAG) applications. | **Index** | **Index Size** | **Retrieval Speed** | **Accuracy** | | --- | --- | --- | --- | | Flat | 100 - 10,000 | Slow | High | | PQ | 100,000 - 10M+ | Medium | Low | | IVF | 10,000 - 1M+ | Fast | Medium | | IVFPQ | 1M - 10M+ | Very Fast | Low-Medium | | HNSW | 100M - 1B+ | Fastest | Medium | Considerations When Using VectorDB ---------------------------------- To create a DFS partitioned table in VectorDB, you can use either the createPartitionedTable function or the SQL CREATE statement, and specify the vector column, index type, and vector dimension for the _indexes_ parameter. For detailed function usage, see the function page. Note the following considerations when using VectorDB: (1) Creating Databases VectorDB databases can only be created with the TSDB engine, i.e., must set _engine_\="TSDB" when using the `database` function to create a database. (2) Creating Tables * Vector indexing is supported only for DFS partitioned tables with _keepDuplicates_ = ALL (i.e., without record deduplication). * The table can contain only one vector column of FLOAT\[\] type. Each row in this column represents a vector, and all these vectors must have the same dimension. The dimension (i.e., the number of elements in each vector) is specified by _dim_ in the _indexes_ parameter. * The dimension of vectors inserted into the vector column must be consistent with _dim_. (3) Searching Vectors VectorDB will only use the vector indexing to accelerate the retrieval if the following conditions are met, otherwise it will only load data from all level files and perform an exhaustive search. * No table joins are used in queries. * The `order by` clause must sort in ascending order and can only use `rowEuclidean` to compute distances. * The first parameter passed to `rowEuclidean` must be the vector column, i.e., `rowEuclidean(, queryVector)`. * A `limit` clause must be specified. * If a `where` clause is specified, it must not include any sort columns. * The query cannot include clauses such as `group by` and `having`. Note: Vectors in the cache engine do not have indexes and cannot be retrieved using vector indexing. To accelerate retrieving these vectors, you can use `flushTSDBCache` to write the data from the cache to disk before querying. Usage Example ------------- // Create a database db = database(directory="dfs://indexesTest", partitionType=VALUE, partitionScheme=1..10, engine="TSDB") // Create a table schema with col3 of FLOAT[] type schematb = table(1:0, `col0`col1`col2`col3, [INT, INT, TIMESTAMP, FLOAT[]]) // Specify a vector index for 'col3' pt = createPartitionedTable(dbHandle=db, table=schematb, tableName=`pt, partitionColumns=`col0, sortColumns=`col1`col2, indexes={"col3":"vectorindex(type=flat, dim=5)"}) tmp = cj(table(1..10 as col0), cj(table(1..10 as col1), table(now()+1..10 as col2))) join table(arrayVector(1..1000*5,1..5000) as col3) // Insert data into the table and force flush the data to disk pt.tableInsert(tmp) flushTSDBCache() select * from pt where col20 and _dataSync_\=1 (redo log enabled). The OLAP engine supports ACID transactions and implements MVCC (Multiversion Concurrency Control ) for snapshot isolation. When writing to the OLAP database, the data will be written through the following phases: 1. **Recorded in theredo log.** 2. **Cached in thememory (cache engine)**: Simultaneously with the redo log write, data is written to and sorted in the OLAP cache engine. 3. **Flushed to disk**: When the cache engine's data volume reaches the predefined 30% of the _OLAPCacheEngineSize_, the OLAP engine initiates a flush operation. The cached data is appended to the column files within the partition. Reading Data ------------ The OLAP engine stores each column of data as a columnar file. Therefore, when reading data, only the necessary columnar files are read from the disk, decompressed, and loaded into memory. The specific process is as follows: 1. The system narrows down the relevant partitions based on the filtering condition. 2. The system loads the data files into memory and applies the filters. 3. The system reads the corresponding data of the other columns based on the selected column data. This approach allows the OLAP engine to maintain high performance for queries with high throughput. However, for updates or deletions, the entire partition must be loaded into memory to make the changes, which incurs significant performance overhead. Updating Data ------------- The OLAP engine supports ACID transactions and implements MVCC for snapshot isolation. For each update, the system reads the columnar files from the corresponding partition, performs the update in memory, and creates a new version to store the new data. Unchanged columns are managed using hard links to improve performance and reduce unnecessary data reads and writes. Until the transaction is committed, the old version of data can still be accessed. If an update involves multiple partitions and one fails, the system rolls back the changes in all partitions. To meet different requirements, DolphinDB provides three methods for updating tables: * [update](../../Programming/SQLStatements/update.html) : Standard SQL syntax. * [sqlUpdate](../../Functions/s/sqlUpdate.html) : Dynamically generates the metacode of the SQL update statement. * [upsert!](../../Functions/u/upsert!.html) : Inserts rows into a keyed table, indexed table or DFS table if the values of the primary key do not already exist, or update them if the primary key do. Deleting Data ------------- The data deletion process in the OLAP engine follows the same steps as the update process. Data is read by partition, the deletion is performed, and a new version directory is written. Transactions ensure data consistency, and MVCC guarantees read consistency. If a deletion involves multiple partitions and one fails, the system rolls back the changes in all partitions. To meet different requirements, DolphinDB provides the following methods for deleting data: * [dropPartition](../../Functions/d/dropPartition.html) : Removes data from the partition. Optionally removes partition schema. * [delete](../../Programming/SQLStatements/delete.html) : Deletes datafrom the table. * [sqlDelete](../../Functions/s/sqlDelete.html) : Dynamically generates the metacode of the SQL update statement. * [dropTable](../../Functions/d/dropTable.html) : Drops tables. * [truncate](../../Functions/t/truncate.html) : Deletes all data in a table but keep its schema. Features of the OLAP Engine --------------------------- * Data is stored in the table in the same order as it is written, ensuring highly efficient data writing. * It is suitable for scenarios where entire partition or specific columns within a partition need to be read. * It is suitable for large-scale data scanning and full table scans. Note: * Deduplication is not supported. * It is not suitable for tables with more than a few hundred columns. --- # Standalone Deployment on Docker [Jump to main content](#wh_topic_body) Standalone Deployment on Docker =============================== Docker is a lightweight tool for resource isolation, and there is no significant performance difference between running DolphinDB in a Docker and a non-Docker environment. This tutorial is a quick start guide describing how to deploy the DolphinDB server standalone on Docker. Environment Setup ----------------- * **Install Docker** You can download and install Docker Desktop from its [offical website](https://www.docker.com/products/docker) . * **Pull the Docker image of DolphinDB** The latest Docker images are available on [Docker hub repository](https://hub.docker.com/u/dolphindb) . Take the tag v2.00.5 for example: docker pull dolphindb/dolphindb:v2.00.5 * **Host** | Host | IP | Docker Service | Mount Point | | --- | --- | --- | --- | | host1 | xxx.xxx.xx.xx | dolphindb | /ddbdocker | Quick Starts ------------ ### Docker (X86 Architecture) (1) Run the following script on your machine to create a Docker container: docker run -itd --name dolphindb \ --hostname host1 \ -p 8848:8848 \ -v /etc:/dolphindb/etc \ dolphindb/dolphindb:v2.00.5 \ sh Parameters: * `--hostname`: The host name (which is used to collect machine fingerprint for Enterprise Edition License). You can check the host name with the hostname shell command. * `-p`: The first port 8848 is the host port, and the latter one is the port mapped to the DolphinDB container (the default port). After startup, you can access DolphinDB through the local port 8848. * `-v`: Map the /etc directory of the host to the container (/dolphindb/etc). It is used to collect machine fingerprint for Enterprise Edition License. * `--name`: The name of the container. You can perform operations on the container, such as checking container-related information, by specifying a container name. If not specified, a random name will be assigned. Note that the container name must be unique. * `dolphindb/dolphindb:v2.00.5`: It is a required parameter, which is used to specify the image. The image name must be complete (including dolphindb repository and version), for example, dolphindb/dolphindb:v2.00.5. (2) Run the following script to check whether the container is successfully created: docker ps|grep dolphindb The following information indicates a successful operation: 347bfa54df86 dolphindb/dolphindb:v2.00.5 "sh -c 'cd /data/ddb…" 20 seconds ago Up 19 seconds 0.0.0.0:8848->8848/tcp, :::8848->8848/tcp dolphindb (3) Connect to DolphinDB with the client and conduct a test. ### Docker (ARM Architecture) (1) Run the following script on your machine to create a Docker container: docker run -itd --name dolphindb \ --hostname cnserver10 \ -p 8848:8848 \ -v /etc:/dolphindb/etc \ dolphindb/dolphindb-arm64:v2.00.7 \ sh Parameters: * `--hostname`: The host name (which is used to collect machine fingerprint for Enterprise Edition License). You can check the host name with the hostname shell command. * `-p`: The first port 8848 is the host port, and the latter one is the port mapped to the DolphinDB container (the default port). After startup, you can access DolphinDB through the local port 8848. * `-v`: Map the /etc directory of the host to the container (/dolphindb/etc). It is used to ccollect machine fingerprint for Enterprise Edition License. * `--name`: The name of the container. You can perform operations on the container, such as checking container-related information, by specifying a container name. If not specified, a random name will be assigned. Note that the container name must be unique. * `dolphindb/dolphindb-arm64:v2.00.7`: It is a required parameter, which is used to specify the image. The image name must be complete (including dolphindb repository and version), for example, dolphindb/dolphindb-arm64:v2.00.7. (2) Run the following script to check whether the container is successfully created: docker ps|grep dolphindb The following information indicates a successful operation: 347bfa54df86 dolphindb/dolphindb-arm64:v2.00.7 "sh -c 'cd /data/ddb…" 20 seconds ago Up 19 seconds 0.0.0.0:8848->8848/tcp, :::8848->8848/tcp dolphindb (3) Connect to DolphinDB with the client and conduct a test. Production Environment for Deployment (X86 Architecture) -------------------------------------------------------- ### Online Deployment (1) Run the following script to download the installation package and configure the mapping file (take v2.00.5 for example): git clone --depth=1 https://github.com/dolphindb/dolphindb-k8s && \ cd dolphindb_k8s/docker-single && \ sh map_dir.sh (2) Run the following command: tree /ddbdocker Expected output: ├── ddbdocker │ ├── data │ ├── ddb_related │ │ ├── dolphindb.cfg │ │ └── dolphindb.lic │ └── plugins │ ├── hdf5 │ │ ├── libPluginHdf5.so │ │ └── PluginHdf5.txt │ ├── mysql │ │ ├── libPluginMySQL.so │ │ └── PluginMySQL.txt │ ├── odbc │ │ ├── libPluginODBC.so │ │ └── PluginODBC.txt │ └── parquet │ ├── libPluginParquet.so │ └── PluginParquet.tx Description for the above files/folders: | Name | Function | Directory of Host | Directory of Container | | --- | --- | --- | --- | | dolphindb.cfg | the configuration file for a standalone mode | /ddbdocker/ddb\_related/dolphindb.cfg | /data/ddb/server/dolphindb.cfg | | dolphindb.lic | license file of DolphinDB (contact the IT for an Enterprise Edition License) | /ddbdocker/ ddb\_related/dolphindb.lic | /data/ddb/server/dolphindb.lic | | plugins | stores DolphinDB plugins | /ddbdocker/plugins | /data/ddb/server/plugins | | data | stores DolphinDB files on data nodes (e.g., metadata, stream data, tables, etc.) | /ddbdocker/data | /data/ddb/server/data | (3) Start the container with the following script: docker run -itd --name dolphindb \ -p 8848:8848 \ --ulimit nofile=1000000:1000000 \ -v /etc:/dolphindb/etc \ -v /ddbdocker/ddb_related/dolphindb.cfg:/data/ddb/server/dolphindb.cfg \ -v /ddbdocker/ddb_related/dolphindb.lic:/data/ddb/server/dolphindb.lic \ -v /ddbdocker/plugins:/data/ddb/server/plugins \ -v /ddbdocker/data:/data/ddb/server/data \ dolphindb/dolphindb:v2.00.5 \ sh \ -stdoutLog 1 The expected container ID: `3cdfbab788d0054a80c450e67d5273fb155e30b26a6ec6ef8821b832522474f5`. (4) Connect to DolphinDB with the client and conduct a test. ### Offline Deployment (1) Download the corresponding DolphinDB server image from the [official website](https://dolphindb.com) . (2) Add `map_dir.sh` to the same directory as the downloaded ZIP archives as follows: #!/bin/bash set -e clear # Create a new folder to store the corresponding files dir1=/ddbdocker dir2=/ddbdocker/ddb_related if [ ! -e "$dir1" ]; then mkdir $dir1 fi if [ ! -e "$dir2" ]; then mkdir $dir2 fi # Obtain the installation package ddb_zip=$(ls ./DolphinDB_Linux64_V*.zip) ddb_name=$(basename $ddb_zip .zip) if [ -e $ddb_zip ]; then echo -e "Upload installation package successfully" "\033[32m UpLoadSuccess\033[0m"\ else\ echo -e "The installation package cannot be found, please upload to this directory" "\033[31m UpLoadFailure\033[0m"\ echo ""\ sleep 1\ exit\ fi\ unzip "${ddb_zip}" -d "${ddb_name}" \ if [ -d ./$ddb_name ];then\ echo -e "Unzip installation package successfully" "\033[32m UnzipSuccess\033[0m"\ else\ echo -e "Unzip installation package failed, check if the installation package is downloaded completely" "\033[31m UnzipFailure\033[0m"\ echo ""\ sleep 1\ exit\ fi\ \ # Get the source and target files (folders) under the corresponding paths\ source_f1=./${ddb_name}/server/dolphindb.cfg\ source_f2=./${ddb_name}/server/dolphindb.lic\ source_dir4=./${ddb_name}/server/plugins\ \ f1=/ddbdocker/ddb_related/dolphindb.cfg\ f2=/ddbdocker/ddb_related/dolphindb.lic\ dir3=/ddbdocker/data\ dir4=/ddbdocker/plugins\ \ \ # Iterate over the target paths and set conditional statements to determine if existing files are to be overwritten\ function isCovered() { \ \ for i in $*;\ do\ if [ -e $i ];\ then\ #echo $i\ read -p "The $i has already existed, would you want to recover or clean it and other similar ones?(y/n)" answer\ if [ $answer=="y" ];\ then\ break\ else\ echo ""\ sleep 1\ exit\ fi\ fi\ done\ }\ \ isCovered $f1 $f2 $dir3 $dir4\ \ # Make a copy of the corresponding file (folder)\ cp -rpf $source_f1 $f1\ cp -rpf $source_f2 $f2\ \ if [ -e "$dir3" ]; \ then\ rm -rf $dir3\ fi\ \ mkdir $dir3\ cp -rpf $source_dir4 $dir4\ \ # Delete the installation package\ rm -rf ./${ddb_name}\ \ \ (3) Run the following command:\ \ sh map_dir.sh && tree /ddbdocker\ \ Expected output:\ \ ├── ddbdocker\ │ ├── data\ │ ├── ddb_related\ │ │ ├── dolphindb.cfg\ │ │ └── dolphindb.lic\ │ └── plugins\ │ ├── hdf5\ │ │ ├── libPluginHdf5.so\ │ │ └── PluginHdf5.txt\ │ ├── mysql\ │ │ ├── libPluginMySQL.so\ │ │ └── PluginMySQL.txt\ │ ├── odbc\ │ │ ├── libPluginODBC.so\ │ │ └── PluginODBC.txt\ │ └── parquet\ │ ├── libPluginParquet.so\ │ └── PluginParquet.tx\ \ Description for the above files/folders:\ \ | Name | Function | Directory of Host | Directory of Container |\ | --- | --- | --- | --- |\ | dolphindb.cfg | the configuration file for a standalone mode | /ddbdocker/ddb\_related/dolphindb.cfg | /data/ddb/server/dolphindb.cfg |\ | dolphindb.lic | license file of DolphinDB (contact the IT for an Enterprise Edition License) | /ddbdocker/ ddb\_related/dolphindb.lic | /data/ddb/server/dolphindb.lic |\ | plugins | stores DolphinDB plugins | /ddbdocker/plugins | /data/ddb/server/plugins |\ | data | stores DolphinDB files on data nodes (e.g., metadata, stream data, tables, etc.) | /ddbdocker/data | /data/ddb/server/data |\ \ (4) Start the container with the following script:\ \ docker run -itd --name dolphindb \\ -p 8848:8848 \\ --ulimit nofile=1000000:1000000 \\ -v /etc:/dolphindb/etc \\ -v /ddbdocker/ddb_related/dolphindb.cfg:/data/ddb/server/dolphindb.cfg \\ -v /ddbdocker/ddb_related/dolphindb.lic:/data/ddb/server/dolphindb.lic \\ -v /ddbdocker/plugins:/data/ddb/server/plugins \\ -v /ddbdocker/data:/data/ddb/server/data \\ dolphindb/dolphindb:v2.00.5 \\ sh \\ -stdoutLog 1\ \ The expected container ID: `3cdfbab788d0054a80c450e67d5273fb155e30b26a6ec6ef8821b832522474f5`.\ \ (5) Connect to DolphinDB with the client and conduct a test.\ \ FAQ\ ---\ \ **How to modify a configuration file?**\ \ Find the mapped DolphinDB configuration file _dolphindb.cfg_ on the host, e.g., _/ddbdocker/ddb\_related/dolphindb.cfg_.\ \ localSite=localhost:8848:local8848\ mode=single\ maxMemSize=32\ maxConnections=512\ workerNum=4\ localExecutors=3\ dataSync=1\ chunkCacheEngineMemSize=2\ newValuePartitionPolicy=add\ maxPubConnections=64\ subExecutors=4\ subPort=8849\ lanCluster=0\ \ * `localSite`: IP address, port number and alias of the controller node. The default localSite for a standalone mode is _localhost:8848:local8848_, where 8848 is the the port number (mandatory) on which DolphinDB is running.\ \ * `workerNum`: The size of worker pool for regular interactive jobs. The default value is the number of CPU cores.\ \ * `localExecutors`: The number of local executors. The default value is the number of CPU cores - 1.\ \ * `maxMemSize`: The maximum memory (in units of GB) allocated to DolphinDB. If set to 0, it means no limits on memory usage. It is recommended to set it to a value lower than the machine's memory capacity.\ \ For more configuration parameters, refer to [Reference](../Database/Configuration/reference.html)\ .\ \ \ **How to upgrade?**\ \ (1) Pull the new version of DolphinDB image (take version 2.00.6 for example):\ \ docker pull dolphindb/dolphindb:v2.00.6\ \ (2) Upgrade with the following commands:\ \ docker run -itd --name dolphindb \\ -p 8848:8848 \\ --ulimit nofile=1000000:1000000 \\ -v /etc:/dolphindb/etc \\ -v /ddbdocker/ddb_related/dolphindb.cfg:/data/ddb/server/dolphindb.cfg \\ -v /ddbdocker/ddb_related/dolphindb.lic:/data/ddb/server/dolphindb.lic \\ -v /ddbdocker/plugins:/data/ddb/server/plugins \\ -v /ddbdocker/data:/data/ddb/server/data \\ dolphindb/dolphindb:v2.00.6 \\ sh \\ -stdoutLog 1\ \ The upgrade operation of DolphinDB server only applies to the installation package. Therefore, to avoid unnecessary replacement of configuration files, license files, plugins, and built-in data files that are beyond the scope of upgrade operation, make sure to back up the following files and directories before upgrading:\ \ * Configuration file: dolphindb.cfg;\ * License file: dolphindb.lic;\ * Directory where plugins are located: /plugins/;\ * Data file: /data/.\ \ **Note**:\ \ * Since the name of Docker container must be unique, if the previous container is kept, the new container name must be different from the previous one;\ * The port number is not occupied.\ \ (3) Connect to the node via GUI and execute the following command to check the version number to see whether the upgrade is successful.\ \ version()\ \ Expected output:\ \ 2.00.6\ \ Troubleshooting\ ---------------\ \ (1) **Error messages**:\ \ Can't find time zone database. Please use parameter tzdb to set the root directory of time zone database.\ \ **Solution**: If the Docker image does not contain the `tzdata` package (e.g., `ubuntu:latest`), the above error will be reported at startup. Execute the following command to install the `tzdata` package:\ \ ```bash\ apt-get install tzdata\ ```\ \ (2) **Error messages**:\ \ ```bash\ : Failed to retrieve machine fingerprint\ ```\ \ **Solution**: Since the official license requires machine fingerprint which Docker does not have access to, the above exception will be thrown in the logs at startup. You need to add the following parameters to `docker run`:\ \ ```bash\ -v /etc:/dolphindb/etc \ ```\ \ (3) **Error messages**:\ \ ```bash\ docker: Error response from daemon: driver failed programming external connectivity on endpoint dolphindb (178a842284d64fbe128ff3f1188ead76ef4072c9149226f8bf62dc7795a58603): Error starting userland proxy: listen tcp4 0.0.0.0:8848: bind: address already in use.\ ```\ \ **Solution**: Change the port number of the host, or use the following command to check whether the port is available or not:\ \ ```shell\ lsof -i:8848\ COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME\ dolphindb 17238 root 6u IPv4 231927135 0t0 TCP *:8848 (LISTEN)\ ``` --- # compress [Jump to main content](#wh_topic_body) compress ======== Syntax ------ `compress(X, [method='lz4'])` Arguments --------- **X** is a vector or a table. **method** (optional) is a string indicating the compression algorithm. The available options are: * "lz4" (by default) is suitable for almost all data types. Although the "lz4" method may not achieve the highest compression ratio, it provides fast compression and decompression speeds. * "delta" option applies delta-of-delta algorithm, which is particularly suitable for data types like SHORT, INT, LONG, and date/time data. * "zstd" is also suitable for almost all data types. It provides a higher compression ratio compared to "lz4", but the compression and decompression speed is about half as fast as "lz4". * "chimp" is suitable for DOUBLE type data with decimal parts not exceeding three digits in length. Details ------- Compress a vector or a table with the specified compression algorithm. The compressed variable needs to be decompressed with function [decompress](../d/decompress.html) before it can be used in a calculation. Examples -------- x=1..100000000 y=compress(x, "delta"); y.typestr(); // output: HUGE COMPRESSED VECTOR z=compress(x, "zstd"); z.typestr(); // output: HUGE COMPRESSED VECTOR select name, bytes from objs() where name in `x`y; | name | bytes | | --- | --- | | x | 402653952 | | y | 13634544 | Please note that if function `size` is applied on the compressed vector y, the result is the length of the compressed vector y instead of the original vector x. To extract information about x from y, we need to decompress y first. y.size(); // output: 12670932 z=decompress(y); z.size(); // output: 100000000 Related functions: [decompress](../d/decompress.html) --- # mr [Jump to main content](#wh_topic_body) mr == Syntax ------ `mr(ds, mapFunc, [reduceFunc], [finalFunc], [parallel=true])` Arguments --------- **ds** is the list of data sources. This required parameter must be a tuple and each element of the tuple is a data source object. Even if there is only one data source, we still need a tuple to wrap the data source. **mapFunc** is the map function. It accepts one and only one argument, which is the materialized data entity from a data source. If we would like the map function to accept more parameters in addition to the materialized data source, we can use a [PartialApplication](../../Programming/FunctionalProgramming/PartialApplication.html) to convert a multiple-parameter function to a unary function. The number of map function calls is the same as the number of data sources. The map function returns a regular object (scalar, pair, array, matrix, table, set, or dictionary) or a tuple (containing multiple regular objects). **reduceFunc** (optional) is the binary reduce function that combines two map function call results. The reduce function in most cases is trivial. An example is the addition function. The reduce function is optional. If the reduce function is not specified, the system returns all individual map call results to the final function. **finalFunc** (optional) is the final function which accepts one and only one parameter. The output of the last reduce function call is the input of the final function. If it is not specified, the system returns the individual map function call results. **parallel** (optional) is a boolean flag indicating whether to execute the map function in parallel locally. The default value is true, i.e., enabling parallel computing. When there is very limited available memory and each map call needs a large amount of memory, we can disable parallel computing to prevent the out-of-memory problem. We may also want to disable the parallel option in other scenarios. For example, we may need to disable the parallel option to prevent multiple threads from writing to the same partition simultaneously. Details ------- The Map-Reduce function is the core function of DolphinDB's generic distributed computing framework. Examples -------- The following is an example of distributed linear regression. Suppose _X_ is the matrix of independent variables and _y_ is the dependent variable. _X_ and _y_ are stored in multiple data sources. To estimate the ordinary least square parameters, we need to calculate X TX and X Ty. We can calculate the tuple of (X TX, X Ty) from each data source, then aggregate the results from all data sources to get X TX and XT y for the entire dataset. def myOLSMap(table, yColName, xColNames, intercept){ if(intercept) x = matrix(take(1.0, table.rows()), table[xColNames]) else x = matrix(table[xColNames]) xt = x.transpose(); return xt.dot(x), xt.dot(table[yColName]) } def myOLSFinal(result){ xtx = result[0] xty = result[1] return xtx.inv().dot(xty)[0] } def myOLSEx(ds, yColName, xColNames, intercept){ return mr(ds, myOLSMap{, yColName, xColNames, intercept}, +, myOLSFinal) } In the example above, we define the map function and final function. In practice, we may define transformation functions for data sources as well. These functions only need to be defined in the local instance. Users don't need to compile them or deploy them to the remote instances. The distributed computing framework in DolphinDB handles these complicated issues for end users on the fly. As a frequently used analytics tool, the distributed ordinary least square linear regression is implemented in the core library of DolphinDB already. The built-in version ([olsEx](../o/olsEx.html) ) provides more features. --- # Programming Guide [Jump to main content](#wh_topic_body) Programming Guide ================= To develop big data applications, in addition to a distributed database that can store huge volumes of data and a distributed computing framework capable of making efficient use of multi-cores and multi-nodes, we also need a programming language that is organically integrated with the distributed database and the distributed computing framework. DolphinDB drew inspiration from popular programming languages such as SQL and Python. It provides a highly expressive programming language that offers high performance and enables rapid development. 1\. Vector Programming ---------------------- Vector programming is the most basic programming paradigm in DolphinDB. Most of the functions in DolphinDB can use vectors as input parameters. There are 2 types of functions depending on the data form of the result: an aggregate function returns a scalar; a vector function returns a vector of the same length as input vectors. Vector programming has 3 main advantages: (1) concise script; (2) significant reduction in the cost of interpreting the scripting language; (3) can optimize numerous algorithms. As time-series data can be represented with a vector, time-series analysis is particularly suitable for vector programming. The following example sums two huge vectors. Imperative programming with 'for' statements takes more than 100 times longer to execute than vector programming. n = 10000000 a = rand(1.0, n) b = rand(1.0, n) //'for' statement in imperative programming: c = array(DOUBLE, n) for(i in 0 : n) c[i] = a[i] + b[i] //vector programming: c = a + b Vector programming is essentially batch processing of homogeneous data. With it, the instructions can be optimized with vectorization in compiling, and numerous algorithms can be optimized. For instance, to calculate moving average with n rows of data and a window size of k, the runtime complexity is O(nk) without batch computing. When we move to a new window, however, only one data point changes. To get the new average, we only need to adjust for this data point. Therefore, the runtime complexity of batch computing is O(n). DolphinDB has optimized most of the moving window fuctions with runtime in the order of O(n). These functions include: `mmax`, `mmin`, `mimax`, `mimin`, `mavg`, `msum`, `mcount`, `mstd`, `mvar`, `mrank`, `mcorr`, `mcovar`, `mbeta` and `mmed`. The following example shows that optimized `mavg` function significantly improves the performance of moving average calculation. n = 10000000 a = rand(1.0, n) window = 60 //compute each window respectively with `avg` function: timer moving(avg, a, window) // Time elapsed: 4039.23 ms //batch computing with `mavg` function: timer mavg(a, window) // Time elapsed: 12.968 ms Vector programming also has its limitations. First, not all operations can be conducted with vectorized computing. In machine learning and statistical analysis, there are numerious occasions where we can only process data through iteration sentence by sentence instead of vectorization. To improve system performance in these situations, you can use DolphinDB just-in-time (JIT) compilation version, which dynamically compiles script to machine code. Second, languages like MATLAB and R usually load an entire vector into a contiguous memory block. Sometimes large segments of contiguous memory may not be found due to the memory fragmentation problem. DolphinDB introduces big array to handle this problem. Big array can represent a vector with segmented memory blocks. Whether the system uses big array is dynamically determined and transparent to users. Usually, compared with contiguous memory, the performance loss is 1%~5% for scanning a big array and 20%~30% for random access. DolphinDB sacrifices acceptable performance for improved system availability in return. 2\. SQL Programming ------------------- DolphinDB SQL supports most ANSI SQL features and also offers many new features. ### 2.1. Integration of SQL and Programming Languages In DolphinDB, the scripting language is fully integrated with SQL statements. * SQL query can be assigned directly to a variable or as a function parameter. * SQL query statements can quote variables or functions. If a SQL query involves a distributed table, the variables and functions in the SQL query can be automatically serialized to the corresponding nodes. * SQL statements can be dynamically generated script. * SQL statements can operate not only on a table, but also on other data forms (including scalar, vector, matrix, set and dictionary). Tables can be converted into other data forms. //generate a table of employee wages emp_wage = table(take(1..10, 100) as id, take(2017.10M + 1..10, 100).sort() as month, take(5000 5500 6000 6500, 100) as wage) //compute the average wage for each employee. The employee list is stored in the array 'empIds'. empIds = 3 4 6 7 9 select avg(wage) from emp_wage where id in empIds group by id id avg_wage -- -------- 3 5500 4 6000 6 6000 7 5500 9 5500 //display average wage and corresponding employee's name. Get employee name from the dictionary 'empName'. empName = dict(1..10, `Alice`Bob`Jerry`Jessica`Mike`Tim`Henry`Anna`Kevin`Jones) select empName[first(id)] as name, avg(wage) from emp_wage where id in empIds group by id id name avg_wage -- ------- -------- 3 Jerry 5500 4 Jessica 6000 6 Tim 6000 7 Henry 5500 9 Kevin 5500 In the examples above, SQL queries quote a vector and a dictionary directly. By integrating the programming language and SQL queries, the system uses hash tables to solve a problem that would have required subqueries and table joins. It makes the script more concise and easier to understand. More importantly, it improves the performance. DolphinDB introduces a SQL keyword `exec`. Compared with `select`, the result returned by `exec` can be a scalar, vector or matrix. The following example uses `exec` and `pivot by` to return a matrix. exec first(wage) from emp_wage pivot by month, id 1 2 3 4 5 6 7 8 9 10 ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- 2017.11M|5000 5500 6000 6500 5000 5500 6000 6500 5000 5500 2017.12M|6000 6500 5000 5500 6000 6500 5000 5500 6000 6500 2018.01M|5000 5500 6000 6500 5000 5500 6000 6500 5000 5500 2018.02M|6000 6500 5000 5500 6000 6500 5000 5500 6000 6500 2018.03M|5000 5500 6000 6500 5000 5500 6000 6500 5000 5500 2018.04M|6000 6500 5000 5500 6000 6500 5000 5500 6000 6500 2018.05M|5000 5500 6000 6500 5000 5500 6000 6500 5000 5500 2018.06M|6000 6500 5000 5500 6000 6500 5000 5500 6000 6500 2018.07M|5000 5500 6000 6500 5000 5500 6000 6500 5000 5500 2018.08M|6000 6500 5000 5500 6000 6500 5000 5500 6000 6500 ### 2.2. Support for Panel Data Some calculations with panel data return the same number of rows as the input data, such as converting stock prices into stock returns and calculating moving window statistics. For these tasks SQL server and PostgreSQL use window functions. DolphinDB offers `context by` clause in SQL statements. Compared with window functions, `context by` clause has the following advantages: * Can be used with not only `select` statements but also `update` statements. * Window functions can only group existing fields, while `context by` can also group calculated fields. * Window functions can only use a limited number of functions, whereas `context by` clause has no limits on what functions can be used. Besides, `context by` can use arbitrary expressions, such as a combination of multiple functions. * `context by` can be used with `having` clause to filter rows within each group. In the following example, we use `context by` to calculate daily stock returns and ranks. //calculate daily stock returns assuming data for each stock are already ordered. update trades set ret = ratios(price) - 1.0 context by sym //calculate the daily rank(in descending order of stock return)of each stock within each day. select date, symbol, ret, rank(ret, false) + 1 as rank from trades where isValid(ret) context by date //select top 10 stocks each day select date, symbol, ret from trades where isValid(ret) context by date having rank(ret, false) < 10 A paper [101 Formulaic Alphas](http://www.followingthetrend.com/?mdocs-file=3424) introduced 101 quant factors used by WorldQuant, one of Wall Street's top quantitative hedge funds. A quant hedge fund calculated these factors with C#. One of the most complicated factor (No.98) needs hundreds of lines of code in C# and takes about 30 minutes to calculate with data for 3,000 stocks in 10 years. In contrast, the same calcuation in DolphinDB only needs 4 lines of code and takes only 2 seconds, an almost 3 orders of magnitude improvement in performance. //Implementation of No.98 alpha factor. 'stock' is the input table. def alpha98(stock){ t = select code, valueDate, adv5, adv15, open, vwap from stock order by valueDate update t set rank_open = rank(open), rank_adv15 = rank(adv15) context by valueDate update t set decay7 = mavg(mcorr(vwap, msum(adv5, 26), 5), 1..7), decay8 = mavg(mrank(9 - mimin(mcorr(rank_open, rank_adv15, 21), 9), true, 7), 1..8) context by code return select code, valueDate, rank(decay7)-rank(decay8) as A98 from t context by valueDate } ### 2.3. Support for Time Series Data DolphinDB uses columnar data storage and vector programming, which are very friendly for time-series data analysis. * DolphinDB supports temporal data types with various precisions up to nanosecond. High-frequency data can be conveniently converted into low-frequency data such as second-level, minute-level and hour-level with SQL statements. It can also be converted into data for intervals with specified length with function `bar` and SQL clause `group by`. * DolphinDB offers convenient time-series functions, such as lead, lag, moving window, cumulative window and so on. More importantly, these functions are all optimized with performance 1 to 2 orders of magnitude better than other systems. * DolphinDB designs 2 ways of time series joins: asof join and window join. The following example calculates the average salary of a group of people for 3 months before a certain point in time with function `wj`. For more details about function `wj`, please refer to [window join](../Programming/SQLStatements/TableJoiners/windowjoin.html) . p = table(1 2 3 as id, 2018.06M 2018.07M 2018.07M as month) s = table(1 2 1 2 1 2 as id, 2018.04M + 0 0 1 1 2 2 as month, 4500 5000 6000 5000 6000 4500 as wage) select * from wj(p, s, -3:-1,,`id`month) id month avg_wage -- -------- ----------- 1 2018.06M 5250 2 2018.07M 4833.333333 3 2018.07M A typical application of window join in quant finance is to calculate transaction costs with a 'trades' table and a 'quotes' table. //table 'trades' sym date time price qty ---- ---------- ------------ ------ --- IBM 2018.06.01 10:01:01.005 143.19 100 MSFT 2018.06.01 10:01:04.006 107.94 200 ... //table 'quotes' sym date time bid ask bidSize askSize ---- ---------- ------------ ------ ------ ------- ------- IBM 2018.06.01 10:01:01.006 143.18 143.21 400 200 MSFT 2018.06.01 10:01:04.010 107.92 107.97 800 100 ... dateRange = 2018.05.01 : 2018.08.01 //select the last quote before each transaction with asof join, and use the average of bid and ask as the benchmark for transaction costs. select sum(abs(price - (bid+ask)/2.0)*qty)/sum(price*qty) as cost from aj(trades, quotes, `date`sym`time) where date between dataRange group by sym //select all the quotes in the 10 ms before each transaction with window join, and calculate the average mid price as the benchmark for transaction costs. select sum(abs(price - mid)*qty)/sum(price*qty) as cost from pwj(trades, quotes, -10:0, ,`date`sym`time) where date between dataRange group by sym ### 2.4. Other new features DolphinDB SQL offers various other new features. For examples: * User-defined functions can be used in SQL on the local node or distributed environment without compiling, packaging or deploying. * SQL is fully integrated with the distributed computing framework in DolphinDB, making in-database analytics more convenient and efficient. * DolphinDB supports composite column, which can output multiple return values of complex analysis functions into one row of a table. When using composite columns in SQL statements, the function must output a key-value pair or an array. Otherwise you can convert the result into one of these two types with a user-defined function. For details please refer to [User Manual](../Programming/SQLStatements/groupby.html) . factor1=3.2 1.2 5.9 6.9 11.1 9.6 1.4 7.3 2.0 0.1 6.1 2.9 6.3 8.4 5.6 factor2=1.7 1.3 4.2 6.8 9.2 1.3 1.4 7.8 7.9 9.9 9.3 4.6 7.8 2.4 8.7 t=table(take(1 2 3, 15).sort() as id, 1..15 as y, factor1, factor2) //run `ols` for each id: y = alpha + beta1 * factor1 + beta2 * factor2, and output parameters alpha, beta1 and beta2. select ols(y, [factor1,factor2], true, 0) as `alpha`beta1`beta2 from t group by id id alpha beta1 beta2 -- --------- --------- --------- 1 1.063991 -0.258685 0.732795 2 6.886877 -0.148325 0.303584 3 11.833867 0.272352 -0.065526 //output R2 with a user-defined function. def myols(y,x){ r=ols(y,x,true,2) return r.Coefficient.beta join r.RegressionStat.statistics[0] } select myols(y,[factor1,factor2]) as `alpha`beta1`beta2`R2 from t group by id id alpha beta1 beta2 R2 -- --------- --------- --------- -------- 1 1.063991 -0.258685 0.732795 0.946056 2 6.886877 -0.148325 0.303584 0.992413 3 11.833867 0.272352 -0.065526 0.144837 3\. Imperative Programming -------------------------- Like most scripting languages (e.g., Python, JavaScript) and various strongly-typed, compiled languages (e.g., C++, C, Java), DolphinDB supports imperative programming, which means script can be executed sentence by sentence. DolphinDB currently supports 18 types of statements, including the most commonly used assignment statements, branch statements (e.g., 'if..else') and loop statements (e.g., 'for', 'do..while'). For more details, please refer to [Programming Statements](../Programming/ProgrammingStatements/programming_statements.html) . DolphinDB supports single and multiple assignment: x = 1 2 3 y = 4 5 y += 2 x, y = y, x //swap the value of x and y x, y =1 2 3, 4 5 DolphinDB supports loop statements, including `for` loop and `do..while` loop. The iterator of a `for` loop can be a range (not including the upper bound), an array, a matrix or a table. //loop from 1 to 100 to compute the cumulative sum: s = 0 for(x in 1:101) s += x print s //loop over an array to compute the sum of all the elements in the array: s = 0; for(x in 1 3 5 9 15) s += x print s //loop over a matrix to compute the average of each column and then print them: m = matrix(1 2 3, 4 5 6, 7 8 9) for(c in m) print c.max() //loop over a table "product" to compute the sales of each product: t= table(["TV set", "Phone", "PC"] as productId, 1200 600 800 as price, 10 20 7 as qty) for(row in t) print row.productId + ": " + row.price * row.qty DolphinDB supports branch statement `if..else` as in other programming languages. if(condition){ } else{ } With large amounts of data, control statements (e.g., `for`, `if..else`) are very inefficient. We recommend using vector programming, functional programming and SQL programmming to process large amounts of data. 4\. Functional Programming -------------------------- DolphinDB supports functional programming including: (1) pure function; (2) user defined function (udf); (3) lambda function; (4) higher order function; (5) partial application; (6) closure. For more details, please refer to [Functional Programming](../Programming/FunctionalProgramming/FunctionalProgramming.html) . ### 4.1. User Defined Function & Lambda Function A user-defined function can be created with or without function name (usually lamdba function). DolphinDB is different from Python in that only function parameters and local variables can be referred inside a function. This greatly improves the software quality with slight reduction in the flexibility of syntactic sugar. //define a function to return weekdays def getWorkDays(dates){ return dates[def(x):weekday(x) between 1:5] } getWorkDays(2018.07.01 2018.08.01 2018.09.01 2018.10.01) [2018.08.01, 2018.10.01] In the example above, we defined a lambda function getWorkDays(). The function accepts an array of dates and returns weekdays. ### 4.2. Higher Order Function In this section we focus on higher-order functions and partial applications. A higher-order function accepts another function as a parameter. In DolphinDB, higher-order functions are mainly used as template functions. Usually the first parameter is another function. For example, object A has m elements and object B has n elements. To calculate pairwise correlation of the elements in A and B, we need to use function `corr` on all possible pairs of elements in A and elements in B, and arrange the results in an m\*n matrix. In DolphinDB, the aforementioned steps can be simplified with a high-order function `cross`: cross(corr, A, B). DolphinDB provides many such template functions including `all`, `any`, `each`, `loop`, `eachLeft`, `eachRight`, `eachPre`, `eachPost`, `accumulate`, `reduce`, `groupby`, `contextby`, `pivot`, `cross`, `moving`, `rolling`, etc. In the following example, we use 3 higher-order functions to calculate the pairwise correlation with tick-level stock data. //simulate 100 million data points n=10000000 syms = rand(`FB`GOOG`MSFT`AMZN`IBM, n) time = 09:30:00.000 + rand(21600000, n) price = 500.0 + rand(500.0, n) //generate a pivot table with `pivot` function priceMatrix = pivot(avg, price, time.minute(), syms) //generate stock return per minute for each stock (each column of the matrix) with functions `each` and `ratios` retMatrix = each(ratios, priceMatrix) - 1 //compute pairwise correlation with funtions `cross` and `corr` corrMatrix = cross(corr, retMatrix, retMatrix) AMZN FB GOOG IBM MSFT --------- --------- --------- --------- --------- AMZN|1 0.015181 -0.056245 0.005822 0.084104 FB |0.015181 1 -0.028113 0.034159 -0.117279 GOOG|-0.056245 -0.028113 1 -0.039278 -0.025165 IBM |0.005822 0.034159 -0.039278 1 -0.049922 MSFT|0.084104 -0.117279 -0.025165 -0.049922 1 ### 4.3. Partial Application Partial application fixes part or all of the parameters of a function to produce a new function with fewer parameters. In DolphinDB, function calls use () and partial applications use {}. For example, the `ratios` function is a partial application with the higher order function `eachPre`: `eachPre{ratio}`. The following 2 lines of script are equivalent: `retMatrix = each(ratios, priceMatrix) - 1` retMatrix = each(eachPre{ratio}, priceMatrix) - 1 Partial applications are often used with higher order functions. Higher order functions usually have certain requirements for parameters, and partial applications can ensure that the parameters meet these requirements. For example, to calculate the correlation between vector a and each column in a matrix m, we can use the higher order function `each`. However, if we use function `corr`, vector a and matrix m as the parameters of `each`, the system would try to calculate the correlation between each element of vector a and each column of matrix m, which will throw an exception. We can use partial application to form a new function corr, and then use it as a parameter for the higher order function `each` together with matrix m. We can also use 'for' statement for this problem, but the script is lengthy and the execution takes longer time. a = 12 14 18 m = matrix(5 6 7, 1 3 2, 8 7 11) //calculate correlation with `each` and partial applications each(corr{a}, m) //calculate with 'for' statement cols = m.columns() c = array(DOUBLE, cols) for(i in 0:cols) c[i] = corr(a, m[i]) Partial application can also be used to keep a function in a state. Usually the result of a function is completely determined by the input parameters. Occasionally, however, we would like a function to be in a state. For example, in streaming, we define a message handler to accept a new message and return a result. For the message handler to return the average of a column for all the messages received, we can use partial application. def cumulativeAverage(mutable stat, newNum){ stat[0] = (stat[0] * stat[1] + newNum)/(stat[1] + 1) stat[1] += 1 return stat[0] } msgHandler = cumulativeAverage{0.0 0.0} each(msgHandler, 1 2 3 4 5) [1,1.5,2,2.5,3] 5\. RPC Programming ------------------- Remote Procedure Call (RPC) is one of the most commonly used functionalities in distributed systems. In DolphinDB, the implementation of the distributed file system, distributed database and distributed computing framework all utilize the built-in RPC system. * Can not only execute functions registered on a remote machine, but also serialize a local user-defined function to a remote node for execution. User privileges on remote machines are equivalent to those on local machines. * Parameters of a function can be either regular parameters such as scalar, vector, matrix, set, dictionary and table, or functions including user-defined functions. * Can use both exclusive connections between two nodes and shared connections between data nodes in a cluster. ### 5.1. Remote call with `remoteRun` We can use `xdb` to create a connection to a remote node. The remote node can be any node running DolphinDB and doesn't need to be in the current cluster. After the connection is created, we can use `remoteRun` to execute a function that is registered on the remote node or a locally defined function on the remote node. h = xdb("localhost", 8081) //run on the remote node remoteRun(h, "sum(1 3 5 7)"); 16 //the script above can be simplified as h("sum(1 3 5 7)"); 16 //at the remote node, run a function registered on the remote node h("sum", 1 3 5 7); 16 //run a user-defined function on the remote node def mysum(x) : reduce(+, x) h(mysum, 1 3 5 7); 16 //create a shared table 'sales' on the remote node (localhost:8081) h("share table(2018.07.02 2018.07.02 2018.07.03 as date, 1 2 3 as qty, 10 15 7 as price) as sales") //when we execute a user-defined function on a remote node, its dependent functions will be automatically serialized to the remote node. defg salesSum(tableName, d): select mysum(price*qty) from objByName(tableName) where date=d h(salesSum, "sales", 2018.07.02); 40 ### 5.2. Remote call with `rpc` Another way of remote procedure call is function `rpc`. Its parameters are the remote node alias, the function to be executed and the function parameters. Both the calling node and the remote node must be located in the same cluster. Otherwise, we need to use function `remoteRun`. Function `rpc` doesn't need a new connection. Instead, it reuses existing network connections. This way it saves network resources and the time to create new connections, especially when the cluster has many users. Please note that `rpc` can only execute one function on a remote node. To execute script with `rpc`, the script must be wrapped in a user-defined function. //'sales' is a shared table on a remote node 'nodeB' rpc("nodeB", salesSum, "sales",2018.07.02); 40 //when using `rpc`, we recommend using partial application to increase readability of the script rpc("nodeB", salesSum{"sales", 2018.07.02}); 40 //create a user on the controller node 'master' rpc("master", createUser{"jerry", "123456"}) //`rpc` can also call a built-in function or a user-defined function to run on a remote node rpc("nodeB", reduce{+, 1 2 3 4 5}); 15 The most powerful feature of built-in remote call funcions (`remoteRun` and `rpc`) is that we can send a local function that calls local user-defined functions to run on a remote node on the fly without compilation or deployment. The system automatically serializes the function definition and the definitions of all dependent functions together with necessary local data to remote nodes. In other systems we can only remote call functions without any user-defined dependent functions. ### 5.3. Indirect remote call with other functions DolphinDB provides numerous functions for indirect remote procedure calls. For examples, linear regression on distributed database uses `rpc` and `olsEx`; function `pnodeRun` calls a function on multiple remote nodes in parallel and then merges the returned results. `pnodeRun` is a quite useful tool in cluster management. //return the most recent 10 running or completed batch jobs on each node pnodeRun(getRecentJobs{10}) //return the most recent 10 SQL queries on 'nodeA' and 'nodeB' pnodeRun(getCompletedQueries{10}, `nodeA`nodeB) //clear caches on all data nodes pnodeRun(clearAllCache) ### 5.4. Distributed computing DolphinDB provides function `mr` for distributed computing and `imr` for iterative distributed computing based on MapReduce. Users only need specify a distributed data source and functions such as map function, reduce function, final function and terminate function. In the example below, we illustrate distributed median calculation and distributed linear regression. n=10000000 x1 = pow(rand(1.0,n), 2) x2 = norm(3.0:1.0, n) y = 0.5 + 3 * x1 - 0.5*x2 + norm(0.0:1.0, n) t=table(rand(10, n) as id, y, x1, x2) login(`admin,"123456") db = database("dfs://testdb", VALUE, 0..9) db.createPartitionedTable(t, "sample", "id").append!(t) Construct a function `myOLSEx` to conduct a distributed linear regression with user-defined map function `myOLSMap`, built-in reduce function `add`, user-defined final function `myOLSFinal`, and built-in function `mr`. //define a map function 'myOLSMap' def myOLSMap(table, yColName, xColNames){ x = matrix(take(1.0, table.rows()), table[xColNames]) xt = x.transpose() return xt.dot(x), xt.dot(table[yColName]) } //define a final function 'myOLSFinal' def myOLSFinal(result){ xtx = result[0] xty = result[1] return xtx.inv().dot(xty)[0] } def myOLSEx(ds, yColName, xColNames){ return mr(ds, myOLSMap{, yColName, xColNames}, +, myOLSFinal) } //calculate the linear regression coefficients on distributed data source with function 'myOLSEx' sample = loadTable("dfs://testdb", "sample") myOLSEx(sqlDS(), `y, 0.0 : 1.0, 0.001); -0.052973 //calculate the median of unpartitioned data with a built-in function `med` med(y); -0.052947 6\. Metaprogramming ------------------- Metaprogramming is a programming technique in which computer programs are treated as data. It means that a program can read, generate, analyze or transform other programs, or even modify itself while running. Metaprogramming usually serves 2 purposes: code generation and delayed execution. We can use metaprogramming to dynamically create script including function calls and SQL statements. Quite often job details are not known when the script is written. For example, in customized reporting, a complete SQL statement is determined only after the customer selects the table, columns and formats. Dynamic expressions are generated with a pair of '<>' or with built-in functions including `objByName`, `sqlCol`, `sqlColAlias`, `sql`, `expr`, `eval`, `partial` and `makeCall`. Delayed execution usually invloves the following scenarios: (1) to provide a callback function; (2) for system optimization; (3) when implementation of script is at a later stage than description stage. //generate dynamic expression with a pair of '<>' a = <1 + 2 * 3> a.typestr(); CODE a.eval(); 7 //generate dynamic expression with a built-in function a = expr(1, +, 2, *, 3) a.typestr(); CODE a.eval(); 7 The following example generates a customized report with metaprogramming. //Generate a SQL expression dynamicly to execute based on the given table, columns and formats. def generateReport(tbl, colNames, colFormat, filter){ colCount = colNames.size() colDefs = array(ANY, colCount) for(i in 0:colCount){ if(colFormat[i] == "") colDefs[i] = sqlCol(colNames[i]) else colDefs[i] = sqlCol(colNames[i], format{,colFormat[i]}) } return sql(colDefs, tbl, filter).eval() } //generate a table with 100 rows of simulated data t = table(1..100 as id, (1..100 + 2018.01.01) as date, rand(100.0, 100) as price, rand(10000, 100) as qty) //generate filter conditions with metaprogramming for a customized report generateReport(t, ["id","date","price","qty"], ["000","MM/dd/yyyy", "00.00", "#,###"], < id<5 or id>95 >); id date price qty --- ---------- ----- ----- 001 01/02/2018 50.27 2,886 002 01/03/2018 30.85 1,331 003 01/04/2018 17.89 18 004 01/05/2018 51.00 6,439 096 04/07/2018 57.73 8,339 097 04/08/2018 47.16 2,425 098 04/09/2018 27.90 4,621 099 04/10/2018 31.55 7,644 100 04/11/2018 46.63 8,383 Some DolphinDB built-in functions need to use metaprogramming for their parameters. For example, for window join function (`wj`), we need to specify one or more aggregate functions for the right table and the parameters for these functions. As description and execution of the task occur on different stages, we need to use metaprogramming to specify the aggregate functions and their parameters for delayed evaluation. t = table(take(`ibm, 3) as sym, 10:01:01 10:01:04 10:01:07 as time, 100 101 105 as price) q = table(take(`ibm, 8) as sym, 10:01:01+ 0..7 as time, 101 103 103 104 104 107 108 107 as ask, 98 99 102 103 103 104 106 106 as bid) wj(t, q, -2 : 1, < [max(ask), min(bid), avg((bid+ask)*0.5) as avg_mid]>, `time); sym time price max_ask min_bid avg_mid --- -------- ----- ------- ------- ------- ibm 10:01:01 100 103 98 100.25 ibm 10:01:04 101 104 99 102.625 ibm 10:01:07 105 108 103 105.625 We may also need metaprogramming to update in-memory partitioned tables without using SQL statements. //create an in-memory partitioned database with 'date' as the partitioning column db = database("", VALUE, 2018.01.02 2018.01.03) date = 2018.01.02 2018.01.02 2018.01.02 2018.01.03 2018.01.03 2018.01.03 t = table(`IBM`MSFT`GOOG`FB`IBM`MSFT as sym, date, 101 103 103 104 104 107 as price, 0 99 102 103 103 104 as qty) trades = db.createPartitionedTable(t, "trades", "date").append!(t) //delete the records with qty=0, and sort by volume in ascending order in each partition trades.erase!().sortBy!() //add a new column 'logPrice' trades[`logPrice]= //update column 'qty' for IBM trades[`qty, ]= 7\. Summary ----------- DolpinDB is specifically designed to process huge volumes of data by integrating the distributed database and the distributed computing framework. DolphinDB supports various programming paradigms, such as SQL programming, functional programming and metaprogramming. As a result, DolphinDB's scripting language is high expressive and enables rapid development. --- # Shell [Jump to main content](#wh_topic_body) Shell ===== This section mainly introduces the specific functions of the Shell interface, which contains the editor, variable browser, data browser, log browser, and database browser. To use the Shell interface, click **To log in** on databse browser or use the `login` function to log in. To log out, use the `logout` function or click the **Log out** button in the upper right corner. Note that when using `logout`, the system will automatically log users in again after a page refresh, while the **Log out** button enables users to completely exit the session. Editor ------ The script in the editor can be executed by clicking the ▶ icon in the upper left corner or pressing **Ctrl+E**. Select a part (or all) of the script, and click ▶ or press **Ctrl+E** to execute the selected script. If no script is selected, clicking ▶ will execute the entire script. You can also click a line and press **Ctrl+E** to execute only this line of script. ![](../../../images/Web%20Interface/shell_1.png) Note: A string exceeding 65,536 bytes cannot be printed here. For example, executing the script below will cause the web interface to be disconnected. You need to refresh the web page to re-establish the connection. a = array(STRING,10).append!(string(concat(take(`abcdefg01234,100005)))) print a Variable Browser ---------------- ![](../images/nb_ve_0.png) The variable browser in the web interface displays the name, data type, and size of variables in each data form. It is very similar to that in GUI. Their only difference is that you can right click variables in GUI to conduct operations such as undefine, check schema, etc, while the web interface doesn't offer these functionalities. Select a vector, matrix, or in-memory table in the variable browser to view the data. For a DFS table, you can check the data in the database browser. Data Browser ------------ The data browser in the web interface displays the execution results of the script in the editor. For example, execute the following script in the editor window: sym = `C`MS`BAC`BK`WFC price= 75.67 53.24 29.48 54.42 60.92 qty = 2200 1900 2100 3200 1800 t = table(sym, qty, price) select *, price*qty as volume from t ![](../../../images/Web%20Interface/shell_2_1.png) It is very similar to that in GUI except for the following differences: * In the GUI, you can right-click in the data browser to perform various operations such as copying, sorting, and exporting. In the web interface, you can copy text by selecting it, but sorting data isn't supported. To view results in a new window, click the "Open in new window" icon in the lower left corner. If you need to export data, click the "export CSV" icon. * ![](../../../images/Web%20Interface/shell_3_1.png) * To set decimal places of results displayed in data browser, you can click the "Settings" icon in the upper right corner. ![](../images/nb_db_2.png) Log Browser ----------- The log browser displays the completion time, elapsed time, execution results in scalar form, and error information of each execution. Execution results in scalar form will be displayed in the log browser. Results in other forms will be displayed in the data browser. In the following example, the result of `2 + 3`, as a scalar, is displayed in the log browser. The result of `2 + [1, 2, 3]`, as a vector, is displayed in the data browser ![](../../../images/Web%20Interface/shell_5.png) Note: * Currently messages in the log browser cannot be cleared. * The log browser can output up to 100,000 logs. Database Browser ---------------- The database browser features the following functionalities: * Create databases and tables; * Add columns (with comments) to a table; * Display information about the created databases and their tables. ![](../images/nb_db_3.png) In the DolphinDB web interface, users can create databases or tables by conditions or scripts. 1. Create databases and tables by conditions * Create databases Click the ![](../../../images/Web%20Interface/create.png) icon to open the **Create database** window. ![](../../../images/Web%20Interface/shell_4.png) Click **Clear** to clear the filled content or **Cancel** to cancel the creation. Click **Preview** to view the corresponding script: ![](../images/nb_db_5.png) Click **Cancel** to cancel the creation or **Go Back** to return to the **Create database** window page. Click **Execute** and the database is shown in the database browser if the script is correct, otherwise an error message is returned. * Create tables Hover over the database name in the browser and you can see the ![](../../../images/Web%20Interface/create_table.png) icon on the right. ![](../images/nb_db_6.png) Click the icon to open the **Create table** window: ![](../images/nb_db_7.png) Fill the required fields and click **Next** to view the corresponding script: ![](../images/nb_db_8.png) Click the ![](../../../images/Web%20Interface/copy.png) icon in the upper right corner to copy the script. Click **Go Back** to return to the **Create table** window or **Execute** to create the table. If the table is created successfully, the following message is shown: ![](../images/nb_db_9.png) Otherwise, the error message is displayed: ![](../images/nb_db_10.png) 2. Create databases and tables by scripts You can also create databases and tables directly by entering the corresponding script in the editor. After successful creation, click the 🔁 icon in the upper left corner of database browser to view the database/table. **Example**: execute the following script in the editor to create a DFS table _pt._ dbName="dfs://test_topic_partition" if(existsDatabase(dbName)){ dropDatabase(dbName) } db=database(dbName,VALUE,2012.01.01..2012.01.10) n=20 date=rand(2012.01.01..2012.01.10, n) sym=rand("A"+string(1..30), n) qty=rand(100, n) t=table(date, sym, qty) pt=db.createPartitionedTable(t, `pt, `date).append!(t) Click the ➕ icon before the table name to show the table schema, column names and types, and partitions. Click the ➖ icon to collapse the displayed information. ![](../images/nb_ddb_0.png) Click table _pt_ and the table is displayed in the data browser: ![](../images/nb_ddb_1.png) Hover over the **columns** and you can see the ![](../../../images/Web%20Interface/add.png) icon. Click the icon to add a new column: ![](../images/nb_ddb_2.png) Hover over a column of _pt_ and you can see the ![](../../../images/Web%20Interface/edit.png) icon. Click the icon to add comments to the column: ![](../images/nb_ddb_3.png) Click the **schema** of _pt_ and the table schema is displayed in the data browser: ![](../images/nb_ddb_4.png) Click **partitions**, select a partition, and the data of the partition is displayed in the data browser: ![](../images/nb_ddb_5.png) --- # Getting Started [Jump to main content](#wh_topic_body) Getting Started =============== Cluster Configuration --------------------- A cluster should be properly configured before being started. **Note**: The number of nodes, IP address, and port number in the following example are for illustration only. You can adjust the settings according to your actual deployment needs. * _controller.cfg_ (configuration file for the controller): Set the port number of the controller to 8900 and enable the performance monitoring. mode=controller logFile=ctl.log localSite=localhost:8900:ctl8900 perfMonitoring=1 * _agent.cfg_ (configuration file for the agent): Set the port number of the agent to 8988 and specify the controller (port number: 8900). mode=agent localSite=localhost:8988:agent controllerSite=localhost:8900:ctl8900 * _cluster.nodes_ (configuration file for nodes within the cluster): Specify the agent (port number: 8988) of the cluster, a data node (port number: 8811), and a compute node (port number: 8812). localSite,mode localhost:8988:agent,agent localhost:8811:DFS_NODE1,datanode localhost:8812:DFS_NODE2,computenode Login ----- * In standalone mode, after starting the DolphinDB server, enter the URL in the browser (default: [http://localhost:8848)](http://localhost:8848)/) to access the web interface. * In cluster mode, enter the URL and port number of the controller in the browser to access. If you fail to access the web interface, make sure that: * The DolphinDB server license is not expired; * The port of the node is not blocked by the firewall; * The web directory is under the server directory. In the Linux environment, if the port is blocked, try the following commands: 1. Query the firewall zones active for every network interface in your system. firewall-cmd --get-active-zones /* output: public interfaces: eth1 */ 2. Add the TCP port 8848 to the "public" firewall zone with permanent validity. sudo firewall-cmd --zone=public --permanent --add-port=8848/tcp If the command returns success, it indicates that port 8848 is open and ready for network communication. You can use the DolphinDB Web client without logging in, but some functions, such as viewing databases/ tables and creating distributed databases/tables, can only be used after logging in. To require users to log in before executing any script, add `webLoginRequired=true` to _server/dolphindb.cfg_ in standalone mode and to _controller.cfg_ and _cluster.cfg_ in cluster mode. After successful login, the username will be displayed in the upper right corner of the interface. If no command is executed for 10 minutes, the account will automatically log out. Cluster Startup --------------- ### Linux * **Method 1** Navigate to _/DolphinDB/server_ and execute the following Shell command (file permissions need to be modified upon the first startup): chmod +x dolphindb Navigate to _/DolphinDB/server/clusterDemo_ and execute the following Shell command to start the controller: sh startController.sh Start the agent: sh startAgent.sh Execute the following command to check whether the node starts successfully: ps aux|grep dolphindb * **Method 2** Open a command line window in the foreground mode and execute the following script to start the controller: ./dolphindb -mode controller -home data -config config/controller.cfg -clusterConfig config/cluster.cfg -logFile log/controller.log -nodesFile config/cluster.nodes Open another command line window and execute the following script to start the agent: nohup ./dolphindb -console 0 -mode agent -home data -config config/agent.cfg -logFile log/agent.log & * **Method 3** Open a command line window in the background mode and execute the following script to start the controller: nohup ./dolphindb -console 0 -mode controller -home data -config config/controller.cfg -clusterConfig config/cluster.cfg -logFile log/controller.log -nodesFile config/cluster.nodes & Open another command line window and execute the following script to start the agent: nohup ./dolphindb -console 0 -mode agent -home data -config config/agent.cfg -logFile log/agent.log & ### Windows * **Method 1** To start the controller and the agent in the foreground, double-click _startController.bat_ and _startAgent.bat_ under _C:\\DolphinDB\\server\\clusterDemo_. * **Method 2** To start the controller and the agent in the background, double-click _backgroundStartController.vbs_ and _backgroundStartAgent.vbs_ under _C:\\DolphinDB\\server\\clusterDemo_. * **Method 3** Open a command prompt and execute the following script to start the controller: dolphindb.exe -mode controller -home data -script config/dolphindb.dos -config config/controller.cfg -logFile log/controller.log -nodesFile config/nodes.cfg -clusterConfig config/cluster.cfg Open another command line window and execute the following script to start the agent: dolphindb.exe -mode agent -home data -script config/dolphindb.dos -config config/agent.cfg -logFile log/agent.log Node Startup ------------ In the web interface, you can start or stop data nodes and compute nodes and modify cluster configurations. Start the controller and the agent, and enter _:_ of the controller (which is configured in _controller.cfg_) in the address bar of the browser. ![](../../../images/Web%20Interface/getting_started_1.png) For details about how to start nodes in the cluster mode, see Cluster. **Note**: If the status of nodes doesn't refresh after clicking 🔁, you can manually refresh the web page (**Shift** + **F5**) or check the system logs in the **Log** tab to identify the specific reasons, such as: * expired license * node port occupied by other processes * metadata recovery in progress * license-specified maximum node count exceeded --- # Configuration [Jump to main content](#wh_topic_body) Configuration ============= On the controller, administrators can configure the following settings: * Controller configuration * Cluster node management * Cluster node configuration * Compute group configuration For detailed instructions, refer to the following sections. Controller Configuration ------------------------ Administrators can add, edit, or delete configuration parameters through the buttons on the **Controller Configuration** page in **Configuration.** ![](../../../images/Web%20Interface/Config_1.png) To add new controller configurations, follow the steps: 1. Click **Add controller config.** 2. Enter the configuration parameter name in the **key** column and the configuration value in the **Value** column. ![](../../../images/Web%20Interface/Config_2.png) **Note:** In the actual environment, the keys and values should be consistent with those supported by the DolphinDB Server version. For detailed configuration parameters names, refer to the [Parameter Configuration](https://docs.dolphindb.com/en/Database/Configuration/configuration.html) . 3. Click **Save**. Click **Edit** on the right side of the configuration parameter to modify it and click **Delete** to remove it. Cluster Node Management ----------------------- Administrators can add, edit, or delete cluster nodes on the **Cluster Node Management** page in **Configuration**. Since version 3.00.2, DolphinDB server supports configuring compute groups. ![](../images/config_1.png) ### Non-Compute Group Nodes For regular nodes that are not in a compute group, the following operations are supported: * **Create**: Click **Add new node** and fill in the alias, type (Datanode/Controller/Agent/Computenode), hostname/IP address, and port of the added node. Then, click **Save** to add the node to the cluster. * **Edit**: Click **Edit** next to an existing node to modify its configurations. * **Delete**: Click **Delete** next to an existing node to remove this node from the cluster. ### Compute Group Nodes **Create a Compute Group** Click **New Compute Group** and fill in the following fields: ![](../images/config_2.png) * Compute Group Name: ID of the group and prefix for all node aliases in the group. * Batch Add Nodes: Enter the alias, hostname/IP address, and port of each node. Alias must start with the compute group name. Nodes can share the same hostname/IP address, but the port must be unique. * Compute Group Configuration: Configure all nodes in the group with key-value pairs. Click **Complete** to create the compute group online. On the Cluster tab, you can enable/disable specific nodes in the group and monitor their status. **Add Nodes to a Compute Group** Click **Add new node** below an existing compute group and fill in the node’s alias (must start with the group name), type, hostname/IP address, and port to add a node to the group online. ![](../images/config_3.png) Click **Edit** next to a node to modify its configurations. Click **Delete Compute Group** to remove the entire group. Click **Delete** next to a node to remove the node. Cluster Node Configuration -------------------------- Administrators can add, edit, or delete cluster node configuration parameters for DolphinDB Server through the buttons on the **Cluster Node Configuration** page in **Configuration.** To add new cluster node configuration parameters, follow the steps: 1. Click **New config.** 2. Select a configuration parameter. If the parameter you're looking for is not listed in the dropdown, enter if manually. ![](../../../images/Web%20Interface/Config_5.png) **Note:** Manually entered configuration parameters will be categorized under **Others**. For detailed configuration item names, refer to the [Parameter Configuration](https://docs.dolphindb.com/en/Database/Configuration/configuration.html) . 3. \[Optional\] Set a qualifier for the configuration parameter. For example, enter dn1 or leave it blank. A qualifier can be the name of a specific node (e.g., dn1) to apply the configuration to that node, or it can be left blank to apply the configuration to all nodes. 4. Set the **Value**. Click **Edit** on the right side of the configuration parameter to modify it and click **Delete** to remove it. Compute Group Configuration --------------------------- On the Compute Group Configuration tab, you can view, modify, add, and delete configurations of existing compute groups. ![](../images/config_4.png) --- # streamEngineParser [Jump to main content](#wh_topic_body) streamEngineParser ================== Syntax ------ `streamEngineParser(name, metrics, dummyTable, outputTable, keyColumn, [timeColumn], [useSystemTime=false], [snapshotDir], [snapshotIntervalInMsgCount], [raftGroup], [filter], [keepOrder], [keyPurgeFilter], [keyPurgeFreqInSecond], [triggeringPattern='perBatch'], [triggeringInterval=1000], [lastBatchOnly=false], [contextByColumn], [updateTime], [useWindowStartTime=false], [roundTime=true], [forceTriggerTime], [sessionBegin], [sessionEnd], [mergeSessionEnd=false], [closed='left'], [fill='none'], [garbageSize], [forceTriggerSessionEndTime], [keyPurgeFreqInSec=-1])` Details ------- The stream engine parser provides a unified interface for computing complex metrics over stream data. These metrics often require logic involving cross-sectional computation, stateful computation, windowing calculations over time series, table joins, and more. To handle such computational tasks, multiple streaming engines need to be combined into a pipeline. With the stream engine parser, you don't need to manually define and connect engines in a specific order. Instead, you only need to rewrite your metrics following a set of rules. The stream engine parser analyzes the metric expressions and automatically constructs a computational pipeline by combining the appropriate engines. As the stream engine parser integrates DolphinDB's time-series, reactive state, and cross-sectional engines, it provides parameters for configuring each engine. Some parameters, like _keyColumn_, are shared across the engines. However, if you need to set different values for the shared parameters per engine, the stream engine parser cannot be used. In these cases, you'll need to manually build a cascade of engines. `streamEngineParser` returns a table object. It is the table returned by the first streaming engine in the auto-generated pipeline. Appending data to this table means that data enters the engine pipeline for processing. Note that there is no function to delete the entire pipeline at once. The engines created by the stream engine parser must be deleted individually by name (see **Arguments** -> _name_) after use. As mentioned earlier, the parser requires metrics to be written using specific functions or syntax rules. In this way, metrics can be translated into executable logic that is mapped to the appropriate streaming engines. For metrics with user-defined functions, the stream engine parser will analyze the function body logic. When parsing a metric, the parser follows the following rules: 1. **It looks for function signifiers indicating the engine type** * If a metric involves cross-sectional computation, the corresponding function must be a function signifier (often a function with the `row-` prefix). For example, to compute the sorted order of elements in each row, the `rowRank` function must be used to implement this layer of logic. Supported function signifiers for cross-sectional computation include: "byRow", "rowAvg", "rowCount", "count", "rowDenseRank", "rowMax", "rowMin", "rowProd", "rowRank", "rowSize", "rowSkew", "rowStd", "rowStdp", "rowSum", "rowSum2", "rowVar", "rowVarp", "rowAnd", "rowOr", "rowXor", "rowKurtosis", "rowCorr", "rowCovar", "rowBeta", "rowWsum", "rowWavg" For logic not expressible with `row-` functions or must be expressed with user-defined functions, you can use the `byRow` higher-order function. * If a metric involves windowing aggregation over time series, use the `rolling` higher-order function. For example, to calculate a rolling 3-element sum stepped by 3 elements on the "price" column using forward filling, specify ``rolling(sum,price,3,3,`ffill`)``. Syntax of `rolling`: rolling(func, funcArgs, window, [step=1], [fill='none']) * Built-in functions without the above function signifiers will be processed by the reactive state engine. 2. **Input table schema for intermediate engines** `streamEngineParser` allows you to specify the schema for only the input table of the first parsed engine through the _dummyTable_ parameter. The intermediate processing steps are abstracted away from the user. The stream engine parser uses specific naming conventions for the _dummyTables_ of intermediate engines. If you need to reference a column from an intermediate engine's _dummyTable_ in _metrics_ or other parameters, follow these naming conventions: * The first column of any intermediate _dummyTable_ is the time column. * The next columns are the ones specified by _keyColumn_. * After that come columns holding the calculation results of _metrics_. These metric columns are named "col\_0\_, col\_1\_, col\_2\_,..." and so on. Note: 1. The output tables for time-series engines and cross-sectional engines both contain a time column. However, reactive state engines do not output a time column. When the engine pipeline generates a reactive state engine, it will automatically add a time column to that engine's _metrics_. This enables the time column to be included in the _dummyTable_ passed to the next engine in the pipeline. In this scenario, if _useSystemTime_ is true, the auto-added time column is named "datetime". 2. Certain parameters for cross-sectional engines and reactive state engines may need column names from the intermediate _dummyTable_. For example: * _contextByColumn_ of cross-sectional engines * _filter_ and _keyPurgeFilter_ of reactive state engines To specify columns from an intermediate _dummyTable_, follow the aforementioned naming conventions, e.g., `contextByColumn = col_0_`. To specify the time column or a key column, simply use the column name. Required Arguments ------------------ **name** is a string specifying the prefix for the names of the engines in the parser. It can contain letters, numbers, and underscores, but must start with a letter. For example, if _name_ \= "test", the engine names will be "test0", "test1", "test2", etc. The number appended to each name indicates the engine's position within the parser, which is determined by the parsing results of _metrics_. **metrics** is metacode indicating the metrics to be computed by the parser. Tuples are supported. For more information, see [Functional Metaprogramming](../../Programming/Metaprogramming/functional_meta.html) . * _metrics_ can include built-in or user-defined functions, like <\[sum(qty), avg(price)\]>; expressions, like <\[avg(price1)-avg(price2)\]>; computation on columns, like <\[std(price1-price2)\]>. * Functions that return multiple values are also supported, such as (column names are optional). The metrics are typically complex, with multiple layers of computation logic. The parser analyzes each layer of logic and assigns it to the appropriate streaming engine (see **Details**). **dummyTable** is a table object. It must have the same schema as the stream table to which the engine subscribes. Whether it contains data or not doesn't matter. **outputTable** holds the computation results from the engine pipeline. It can be an in-memory table or a DFS table, but must be empty before calling `streamEngineParser`. The column names and data types in _outputTable_ must be specified upfront. The schema of _outputTable_ depends on the last engine in the pipeline: * For a time-series engine, the columns are ordered as: 1. Time column 1. The column type is either TIMESTAMP (when _useSystemTime_ \= true) or same type as _timeColumn_ (when _useSystemTime_ = false). 2. It displays window start (when _useWindowStartTime_ = true) or end times (when _useWindowStartTime_ = false). 2. Key column(s), if _keyColumn_ is specified. They are ordered the same as in _keyColumn_. 3. Metric computation result column(s). * For a reactive state engine, the columns are ordered as: 1. Key column(s), ordered the same as in _keyColumn_. 2. Metric computation result column(s). * For a cross-sectional engine: * If _contextByColumn_ is not specified, the columns of the output table are ordered as: 1. A TIMESTAMP column with system timestamps for each computation (or the corresponding values from the _timeColumn_, if this parameter is specified). 2. Metric computation result column(s). The data types must be consistent with the metric computation results. * If _contextByColumn_ is specified, the columns of the output table are ordered as: 1. A TIMESTAMP column with system timestamps for each computation (or the corresponding values from the _timeColumn_, if this parameter is specified). 2. The column specified in _contextByColumn_. 3. Metric computation result column(s). The data types must be consistent with the metric computation results. **keyColumn** is a string scalar or vector. * For time-series engines and reactive state engines, _keyColumn_ indicates the columns to group data by. If specified, computations are done per group, such as aggregations by stock. * For cross-sectional engines, _keyColumn_ indicates the column holding the unique keys. The engine will use only the latest record per key for each computation. Optional Arguments ------------------ **timeColumn** specifies the time column in the stream table to which the engine subscribes. It is required when _useSystemTime_ = false. * For time-series engines and reactive state engines, _timeColumn_ is can be a string scalar or a 2-element string vector. The vector specifies a DATE and a TIME/SECOND/NANOTIME column. The engine will concatenate the date and time values into a single time value for the output table. See concatDateTime() for details on the concatenated data type. * For cross-sectional engines, _timeColumn_ is a string. The specified column must be of TIMESTAMP type. **useSystemTime** indicates how the engine handles time when processing data. * For cross-sectional engines, _useSystemTime_ is an optional boolean parameter. If true (default), the engine will use the system time when each calculation starts and output it in the first column of the output table. If false, it will use the time values from the specified _timeColumn_ instead. * For time-series engines, _useSystemTime_ is an optional parameter specifying how data is windowed for processing: * When _useSystemTime_ = true, the engine will regularly window the streaming data at fixed time intervals for calculations according to the ingestion time (local system time with millisecond precision, independent of any time columns in the stream table) of each record. As long as a window contains data, the calculation will be performed automatically when the window ends. The first column in output table indicates the timestamps when the calculation occurred. Note: When _useSystemTime_ = true, _timeColumn_ must be left empty. * _useSystemTime_ = false (default): the engine will window the streaming data according to the _timeColumn_ in the stream table. The calculation for a window is triggered by the first record after the previous window. Please note that the record which triggers the calculation will not participate in this calculation. To enable snapshot in the streaming engines, specify parameters _snapshotDir_ and _snapshotIntervalInMsgCount_. **snapshotDir** is a string indicating the directory where the streaming engine snapshot is saved. * The directory must already exist, otherwise an exception is thrown. * If _snapshotDir_ is specified, the system checks whether a snapshot already exists in the directory when creating a streaming engine. If it exists, the snapshot will be loaded to restore the engine state. * Multiple streaming engines can share a directory where the snapshot files are named as the engine names. * The file extension of a snapshot can be: * _.tmp_: temporary snapshot * _.snapshot_: a snapshot that is generated and flushed to disk * _.old_: if a snapshot with the same name already exists, the previous snapshot is renamed to _.old_. **snapshotIntervalInMsgCount** is a positive integer indicating the number of messages to receive before the next snapshot is saved. **raftGroup** is an integer greater than 1, indicating ID of the raft group on the high-availability streaming subscriber specified by the configuration parameter _streamingRaftGroups_. Specify _raftGroup_ to enable high availability on the streaming engine. When an engine is created on the leader, it is also created on each follower and the engine snapshot is synchronized to the followers. When a leader is down, the raft group automatically switches to a new leader to resubscribe to the stream table. Note that _SnapShotDir_ must also be specified when specifying a raft group. The stream engine parser must be created on the leader of a raft group. Below are the parameters unique to a specific type of streaming engine: * reactive state engines ([createReactiveStateEngine](../c/createReactiveStateEngine.html) ): _filter_, _keepOrder_, _keyPurgeFilter_, _keyPurgeFreqInSecond_ * cross-sectional engines ([createCrossSectionalEngine](../c/createCrossSectionalEngine.html) ): _triggeringPattern_, _triggeringInterval_, _lastBatchOnly_, _contextByColumn_ * time-series engines ([createTimeSeriesEngine](../c/createTimeSeriesEngine.html) ): _updateTime_, _useWindowStartTime_, _roundTime_, _forceTriggerTime,_ _closed_, _fill_, _garbageSize_, _keyPurgeFreqInSec_ * daily time-series engines ([createDailyTimeSeriesEngine](../c/createDailyTimeSeriesEngine.html) ): _sessionBegin_, _sessionEnd_, _mergeSessionEnd_, _closed_, _fill_, _garbageSize_, _forceTriggerSessionEndTime_, _keyPurgeFreqInSec_ Note: If _triggeringPattern_\='keyCount', parameter _keepOrder_ must be set to true. Examples -------- This example implements Formula #1 from World Quant's 101 Alpha Factors on streaming data. The `rank` function performs a cross-sectional operation and is processed with a cross-sectional engine. `rank`'s parameters are computed using a reactive state engine which outputs to the cross-sectional engine. n = 100 sym = rand(`aaa`bbb`ccc, n) time = 2021.01.01T13:30:10.008 + 1..n maxIndex=rand(100.0, n) data = table(sym as sym, time as time, maxIndex as maxIndex) Alpha#001formula: rank(Ts_ArgMax(SignedPower((returns<0?stddev(returns,20):close), 2), 5))-0.5 //create a cross-sectional engine to calculate the rank of each stock dummy = table(1:0, `sym`time`maxIndex, [SYMBOL, TIMESTAMP, DOUBLE]) resultTable = streamTable(10000:0, `time`sym`factor1, [TIMESTAMP, SYMBOL, DOUBLE]) ccsRank = createCrossSectionalAggregator(name="alpha1CCS", metrics=<[sym, rank(maxIndex, percent=true) - 0.5]>, dummyTable=dummy, outputTable=resultTable, keyColumn=`sym, triggeringPattern='keyCount', triggeringInterval=3000, timeColumn=`time, useSystemTime=false) @state def wqAlpha1TS(close){ ret = ratios(close) - 1 v = iif(ret < 0, mstd(ret, 20), close) return mimax(signum(v)*v*v, 5) } //create a reactive state engine which outputs to the cross-sectional engine input = table(1:0, `sym`time`close, [SYMBOL, TIMESTAMP, DOUBLE]) rse = createReactiveStateEngine(name="alpha1", metrics=<[time, wqAlpha1TS(close)]>, dummyTable=input, outputTable=ccsRank, keyColumn="sym") rse.append!(data) dropStreamEngine("alpha1CCS") dropStreamEngine("alpha1") The computation can also be done by using the `streamEngineParser`: input = table(1:0, `sym`time`close, [SYMBOL, TIMESTAMP, DOUBLE]) resultTable = streamTable(10000:0, `time`sym`factor1, [TIMESTAMP, SYMBOL, DOUBLE]) //construc metrics metrics=<[sym, rowRank(wqAlpha1TS(close), percent=true)- 0.5]> streamEngine=streamEngineParser(name=`alpha1_parser, metrics=metrics, dummyTable=input, outputTable=resultTable, keyColumn=`sym, timeColumn=`time, triggeringPattern='keyCount', triggeringInterval=3000) streamEngine.append!(data) dropStreamEngine("alpha1_parser0") dropStreamEngine("alpha1_parser1") --- # Querying Tables [Jump to main content](#wh_topic_body) Querying Tables =============== In the DolphinDB web interface, users can query a table in two ways: Query Builder and Script Editor. After the query, users can preview and directly export the query result in CSV format. * Query Builder enables users to select columns from a table and set query conditions. Based on users' settings, it executes query conditions and dynamically generates the corresponding SQL scripts. * Script Editor enables users to query a table directly through the SQL scripts. Note: This function is only supported in a standalone mode, or on the data node or compute node in a cluster. Create Query ------------ Click the ![](images/01.png) icon on the right of the table name to open the query window. ![](images/02.png) Figure 1. Figure 1. Query Window Entry The query window is shown in the following figure. ![](images/03.png) Figure 2. Figure 2. Query Window Click the tab shown in the upper left corner to switch to the corresponding query pattern. ![](images/04.png) Figure 3. Figure 3. Query Patterns Query Builder ------------- Query Builder consists of two sections — **Select Columns** and **Set Query Conditions**. ![](images/05.png) Figure 4. Figure 4. Query Builder Interface ### Select Columns **Select Columns** section consists of two selection boxes and two operation buttons. To add columns for queries, users can select one or multiple columns in the box of unselected columns on the left, then click **Add Column**; to remove columns for queries, users can select one or multiple columns in the box of selected columns on the right, then click **Remove Column**. ![](images/06.png) Figure 5. Figure 5. Select Columns Section ### Set Query Conditions In this section, users can set conditions on partitioning columns and combine them with conditions on non-partitioning columns using the AND operator. Users can set one or multiple conditions. A query condition consists of three required items — column name, operator, and value. ![](images/07.png) Figure 6. Figure 6. Set Query Conditions Section #### Partitioning Columns If there is a partitioning column in the selected table, users should set at least 1 condition on the partitioning column. Conditions on partitioning columns are combined using the AND operator. If there is no partitioning column in the selected table, the section of partitioning columns will not be displayed. **How to Set a Condition on a Partitioning Column** The following steps describe how to set a condition on a partitioning column. 1. Select a partitioning column from the **Select column** dropdown, which contains all partitioning columns available for query in the selected table. Note that most column types are supported by Query Builder. For details, refer to the section "Supported Column Types". 2. Select an operator from the **Select operator** dropdown, which contains all operators available for the selected partitioning column. 3. Enter at least 1 value in the value box. For different operators, the value box may be a dropdown, a date table, or a manual input box. For details, refer to the section "Column Types and Operators". ![](images/08.png) Figure 7. Figure 7. How to Set a Query Condition **How to Add/Delete a Condition** To add a condition, click the ![](images/+.png) icon, which is combined with other conditions using the AND operator. To delete a condition, click the ![](images/-.png) icon. #### Non-Partitioning Columns In this section, users can set conditions on non-partitioning columns and combine them using the AND or OR operators. Non-partitioning columns are those available for query in the selected table, excluding partitioning columns. **Non-Partitioning Columns** section consists of condition items and condition groups. Users can create one or multiple condition groups and combine them using the OR operator. A condition group contains one or multiple condition items, combined using the AND operator. ![](images/09.png) Figure 8. Figure 8. Non-Partitioning Columns section To set conditions on non-partitioning columns or add/delete a condition item, follow the same steps as in the **Partitioning Columns** section. Click **Add Condition Group** to add a condition group, which is combined with other condition groups using the OR operator. Click the ![](images/bin.png) icon to delete a condition group. ![](images/10.png) Figure 9. Figure 9. How to Add/Delete a Condition Group #### Column Types and Operators **Supported Column Types** | Column Type | Examples | Category | Range | | --- | --- | --- | --- | | BOOL | 1b, 0b, true, false | Logical | 0~1 | | SHORT | 122h | Integral | \-2 15 +1~2 15 -1 | | INT | 21 | Integral | \-2 31 +1~2 31 -1 | | LONG | 22l | Integral | \-2 63 +1~2 63 -1 | | DATE | 2013.06.13 | Temporal | | | MONTH | 2012.06M | Temporal | | | TIME | 13:30:10.008 | Temporal | | | MINUTE | 13:30m | Temporal | | | SECOND | 13:30:10 | Temporal | | | DATETIME | 2012.06.13 13:30:10 or 2012.06.13T13:30:10 | Temporal | \[1901.12.13T20:45:53, 2038.01.19T03:14:07\] | | TIMESTAMP | 2012.06.13 13:30:10.008 or 2012.06.13T13:30:10.008 | Temporal | | | NANOTIME | 13:30:10.008007006 | Temporal | | | NANOTIMESTAMP | 2012.06.13 13:30:10.008007006 or 2012.06.13T13:30:10.008007006 | Temporal | \[1677.09.21T00:12:43.145224193, 2262.04.11T23:47:16.854775807\] | | FLOAT | 2.1f | Floating | Sig. Fig. 06-09 | | DOUBLE | 2.1 | Floating | Sig. Fig. 15-17 | | SYMBOL | | Literal | | | STRING | "Hello" or 'Hello' or \`Hello | Literal | | | DATEHOUR | 2012.06.13T13 | Temporal | | | DECIMAL32(S) | 3.1415926$DECIMAL32(3) | Decimal | (-1\*10^(9-S), 1\*10^(9-S)) | | DECIMAL64(S) | 3.1415926$DECIMAL64(3), , 3.141P | Decimal | (-1\*10^(18-S), 1\*10^(18-S)) | | DECIMAL128(S) | 3.1415926$DECIMAL128(3) | Decimal | (-1\*10^(38-S), 1\*10^(38-S)) | **Note**: * Avoid using single quotes (') when querying columns of SYMBOL/STRING type, otherwise the query may fail. * Filtering null values of columns is currently not supported by Query Builder and can only be achieved by Script Editor. For more information, refer to [Data Types](../../../Programming/DataTypesandStructures/DataTypes/DataTypes.dita) . **Supported Operators** | Column Type | Supported Operators | | --- | --- | | Temporal, numeric | \=, !=, >, <, >=, <= | | SYMBOL, STRING | like, not like, in, not in, =, != | | Others | \=, != | Instructions for LIKE/NOT LIKE pattern: The LIKE/NOT LIKE pattern must include the wildcard character %, which represents zero, one, or multiple characters of any type or length. Note that it is case-sensitive. For example: * 688% matches strings starting with "688", like "688101". * %SZ% matches strings that contain "SZ", such as "001SZ". * %6 matches strings ending with "6", like "abcd6". For more information, refer to [like](../../../Programming/SQLStatements/like.dita) . Instructions for IN/NOT IN operators: The value dropdown shows a subset of available values with a brief delay. Alternatively, enter one or more values manually in the field, pressing 'Enter' after each. For more information, refer to [in](../../../Programming/SQLStatements/in.dita) . For details about other operators, refer to [Operator Summary](../../../Programming/Operators/OperatorSummary.dita) . ### View Generated Query After column selection and condition setting, users can click **View Generated Query** to dynamically generate corresponding SQL scripts. To set the settings, users can click **Reset**. ![](images/11.png) Figure 10. Figure 10. View Generated Query The generated SQL scripts are as follows. Click **Go Back** to return to the Query Builder page to modify settings. Click the ![](images/copy.png) icon to copy the SQL scripts. ![](images/12.png) Figure 11. Figure 11. Dynamically Generated SQL Scripts ### Preview Data Click Preview Data to preview the result. ![](images/13.png) Figure 12. Figure 12. Preview Data The result will be previewed in table form, as shown below. **Note**: The preview page can display up to 20 million rows of data. If the full dataset exceeds 20 million rows, a partial preview will be displayed. ![](images/14.png) Figure 13. Figure 13. Preview in Table Form ### Export Data Click **Export Data** and enter the file name in the pop-up window to export the result in CSV format. **Note**: If the query returns no data or a dataset exceeding 500,000 rows, the export fails. ![](images/15.png) Figure 14. Figure 14. Export Data After the export, an "Export successfully" prompt pops up. Users can view the exported file in the browser's download queue. ![](images/16.png) Figure 15. Figure 15. Successful Export Prompt Script Editor ------------- Users can also directly enter SQL scripts in the **Script Editor** tab to create a query. By default, an example script of the selected table will be automatically generated and displayed in the **Script Editor** tab. After entering executable SQL scripts, click **Preview Data > Export Data** to export the result. ![](images/17.png) Figure 16. Figure 16. Script Editor Page --- # Cluster [Jump to main content](#wh_topic_body) Cluster ======= On the Compute Group Configuration tab, you can view, modify, add, and delete configurations of existing compute groups. ![](../images/cluster_1.png) Figure 1. Table Mode ![](../images/cluster_2.png) Figure 2. Card Mode Agent Panel ----------- The agent panel displays the name and state of agent nodes. ![](../images/cluster_3.png) Toolbar ------- ![](../images/cluster_4.png) * Click ⟳ to refresh the status of nodes. * Click ▶ to start the selected data nodes. * Click ⏹ to stop the selected data nodes. To filter nodes by their names, enter the text in the box to the right of 🔄 and then click **Enter**. Click **Controller Config** to view and modify the configuration parameters for the controller, which is equivalent to modifying the _controller.cfg_ file. Note the configuration takes effect only after the controller is rebooted. ![](../images/web_tb_1.png) Click **Nodes Setup** to check the cluster environment. You can add, delete or edit data nodes, which is equivalent to modifying the _cluster.nodes_ file. ![](../images/web_tb_2.png) Click **Nodes Config** to view, add, modify or delete the configuration parameters, which is equivalent to modifying the _cluster.cfg_ file. ![](../images/web_tb_3.png) Performance Monitoring Panel ---------------------------- The performance monitoring panel displays various performance indicators for the controller and data nodes. ![](../images/mt_0.png) To customize which indicators to display, right click the panel and click **Column Selection**. ![](../images/mt_1.png) ![](../images/mt_2.png) You can also drag to change the order of columns in the panel. To check the ServerLog or QueryLog of a particular node, click **view** in these 2 columns. ![](../../../images/Web%20Interface/cluster_2.png) --- # Job [Jump to main content](#wh_topic_body) Job === The Job tab in the web interface allows viewing, terminating, and deleting jobs as the Job Manager in GUI, but it does not support job filtering or batch termination. The Job tab divides jobs by status into three sections: running jobs, submitted jobs, and scheduled jobs. Each section contains the user ID, job ID, details, and other information, as shown below. ![](../../../images/Web%20Interface/job_1.png) * Running jobs: Contains information about ongoing jobs, such as their progress, priority, and total task number. Click ➕ on the left to display the details of each task; click ➖ to collapse the details; click **Stop** to cancel the job. * Submitted jobs: Contains information about submitted jobs (including ongoing and finished jobs), such as their status, start time, end time, and error message. Click **Stop** to cancel the running job. * Scheduled jobs: Contains information about jobs set to be performed at a specific time, such as their start date, end date, and execution frequency. Click **Delete** to cancel the job. --- # Quick Start [Jump to main content](#wh_topic_body) Quick Start =========== Let's get started learning the DolphinDB Python API with a quick introduction. Important: This documentation covers the latest version of DolphinDB Python API. If you are using version 1.30.21.2 or earlier of the DolphinDB Python API, please refer to the [previous API documentation](https://github.com/dolphindb/api_python3/blob/master/README.md) . --- # Quick Start [Jump to main content](#wh_topic_body) Quick Start =========== Let's get started learning the DolphinDB C++ API with a quick introduction. --- # Complex Event Processing [Jump to main content](#wh_topic_body) Complex Event Processing ======================== What is DolphinDB CEP --------------------- Starting from version 3.00.0, DolphinDB incorporated Complex Event Processing capabilities through the introduction of the DolphinDB CEP. It allows you to capture and match streams of events against predefined patterns, providing insights into ongoing occurrences. The CEP engine enables you to: * Specify the patterns that you want to detect and act upon in your stream; * Monitor streams of events to find particular events or patterns of interest; * Perform actions such as aggregation and transformation based on particular events or patterns; * Extract information from event streams and find out relationships between them. Differences between CEP engine and other streaming engines: | | CEP | Other Streaming Engines | | --- | --- | --- | | **Data Sources** | Receives events of various types. Each event can be defined with different schema. | Receives streams with same schema. | | **Data Processing** | Event-driven, all processing is triggered in response to events. | The same set of calculation rules applied on all incoming data. | | **Application** | Designed to continuously analyze real-time streams, extracting and identifying specific events or relationships between them. | Designed to process streams with aggregation, factor calculation, etc. | CEP Workflow ------------ The DolphinDB CEP is composed of the following main components: * **Stream Event Serializer**: Serializes real-time event streams from various data sources and writes them to a heterogeneous stream table in DolphinDB. * **Stream Event Deserializer**: Deserializes the events from the subscribed stream tables. It is implemented by the CEP engine internally. * **Event Dispatcher**: Distributes events to sub-engines based on specific attributes (defined within the CEP engine). * **CEP Engine**: Receives the latest event data in real-time by subscribing to the heterogeneous stream table. The CEP engine will implement event matching logic and act on selected events. The CEP engine can generate multiple sub-engines for parallel processing of events. Each sub-engine consists of: * **Event Input Queue**: Holds the incoming events for each sub-engine. * **Event Matcher**: Analyzes the events from the input queues to identify predefined patterns or conditions of interest. * **Monitors** **and Listeners**: Monitors define the processing logic for events and listeners listen for the events passed to the event matchers with specific patterns or conditions. When the event matchers identify events that match the patterns defined in the event listeners, the corresponding actions will be invoked. * **Event Output Queue**: Holds the output events from each sub-engine. The Stream Event Serializer serializes these output events and writes them to a heterogeneous stream table, which can then be subscribed to by APIs (clients) or plugins. The following figure illustrates the workflow of DolphinDB CEP. ![](../images/CEP_Architecture.png) --- # Stream [Jump to main content](#wh_topic_body) Stream ====== The Stream tab monitors the status of various stream computing tasks, including: * **Subscriber and Publisher Status**: Contains the status of workers on subscribers as well as the connection status between the local publisher and its subscribers. For details about each column, see [getStreamingStat().subWorkers](../../../Functions/g/getStreamingStat.html#details__table_cvz_41b_dzb) and [getStreamingStat().pubConns](../../../Functions/g/getStreamingStat.html#details__table_xkb_m1b_dzb) . * **Streaming Engine Status**: Contains the status of various streaming engines. For details about each column, see [getStreamEngineStat()](../../../Functions/g/getStreamEngineStat.html) . * **Stream Table Status**: Contains the status of non-persisted and persisted shared stream tables. For details about each column, see [getStreamTables](../../../Functions/g/getstreamtables.html) . ![](../../../images/Web%20Interface/stream_1.png) Click the tabs at the top of the page to switch to the corresponding section. Click **Refresh** to refresh the content of the current tab. Subscriber and Publisher Status ------------------------------- ### Status of Workers on Subscribers * Displays worker ID, subscription topic, queue depth, queue depth limit, last error message, last error timestamp, number of error messages, number of processed messages, last message ID, and last error message ID. * Other information is stored in the "More Info" column. Click **click to view** to check the information in the pop-up window. * Different queue depth is displayed with different colors: green indicates 0; orange indicates 1-10000; red indicates 10000 and above. * The status of workers on subscribers can be sorted in descending order according to "Queue Depth" or "Last Error Message". * Error messages are briefly displayed and users can click to view related details. ### Publisher Status * Subscriber: the IP address and port number of the subscriber node. * Queue Depth Limit: the maximum depth (number of records) of the message queue that is allowed on the publisher node. * Queue Depth: the current depth (number of records) of the message queue on the publisher node. Different queue depth is displayed with different colors: green indicates 0; orange indicates 1-10000; red indicates 10000 and above. * Table Name: the name of the persisted stream table on the publisher node. Streaming Engine Status ----------------------- Through this tab, users can view the information of all streaming engines created by the system and delete them in batches. ![](../../../images/Web%20Interface/stream_2.png) * Displays engine name, engine type, last error message, memory, number of groups, number of rows in a single table, number of rows in the left table, number of rows in the right table, garbage size, number of metrics, metacode of metrics, user, and status. * Other information is stored in the "More Info" column. Click **click to view** to check the information in the pop-up window. * Error messages are briefly displayed and users can click to view related details. * The status of streaming engines can be sorted in descending order according to "Memory". * Hover over the content in the "Metacode of Metrics" column to view the complete metacode. Stream Table Status ------------------- Through this tab, users can view the status of shared stream tables, and persistence workers. They can also delete stream tables in batches. ![](../../../images/Web%20Interface/stream_3.png) ### Non-persisted Shared Stream Table Status * Table Name: the name of the stream table. * Number of Rows: the number of rows in memory. * Number of Columns: the number of columns in the table. * Memory Size: the memory size of the table. ### Persisted Shared Stream Table Status * Table Name: the name of the stream table. * Load into Memory: whether the table is loaded to memory. * Number of Columns: the number of columns in the table. * Memory Size: the number of records in memory. * Total Number of Rows: the total number of records in the stream table. * Number of Rows in Memory: the number of rows loaded to memory. * Offset in Memory: the offset position of the first message in memory relative to all records in the stream table. memoryOffset = totalSize - sizeInMemory. * Number of Rows on Disk: the number of records that have been persisted to disk. * Offset on Disk: the offset position of the first message on disk relative to all records in the stream table. * Enable Asynchronous Persistence: whether to persist data with asynchronous mode. * Retention Time: how long (in terms of minutes) the log file will be kept. * Compress: whether to save compressed data. * Persistence Directory: the path to the persistent data. Click **details** to view the complete directory. * Number of Persistence Workers: the number of workers responsible for persisting stream tables to disk in asynchronous mode. * Raft Group: the ID of the raft group to which the high-availability stream table belongs. The value is empty for normal stream tables. * Raft Log Seq: the latest log sequence number of raft log. ### Persistence Worker Status When the parameter _asynWrite_ is set to true, the status of persistence workers will be displayed in this tab, which contains the following information: * Worker ID: the ID of the current persistence worker. * Queue Depth Limit: the maximum depth (number of records) of the message queue to persist a stream table to disk. * Queue Depth: the current depth (number of records) of the message queue to persist a stream table to disk. Different queue depth is displayed with different colors: green indicates 0; orange indicates 1-10000; red indicates 10000 and above. * Table Name: the name of the stream table that has been persisted to disk. --- # Stream for DolphinDB [Jump to main content](#wh_topic_body) Stream for DolphinDB ==================== Real-time stream processing handles a continuous "stream" of data as the data is published. The process includes real-time data collection, cleaning, analysis, warehousing, and displaying of results. The stream-based applications include trading, social networks, Internet of Things, system monitoring, and many other examples. DolphinDB's built-in stream processing framework is efficient and convenient, which supports stream publishing, subscription, data preprocessing, real-time in-memory calculation, complex window calculation, stream-table joins, anomaly detection, etc. The advantages of Stream for DolphinDB over other streaming analytics systems are: * High throughput, low latency, high availability * One-stop solution that is seamlessly integrated with time-series database and data warehouse * Natural support of stream-table duality and SQL statements Stream for DolphinDB provides numerous convenient features, such as: * Built-in time-series, cross-sectional, anomaly detection, and reactive state streaming engines. * High frequency data replay * Streaming data filtering 1\. Flowchart and Related Concepts ---------------------------------- Stream for DolphinDB uses the publish-subscribe model. Real-time data is ingested into the stream table and is then published to all subscribers. A data node, compute node or a third-party application can subscribe to and consume the streaming data through DolphinDB script or API. The DolphinDB streaming flowchart is as follows: ![image](images/DolphinDB_Streaming_Framework.jpg) Real-time data is ingested into a stream table on a publisher node. The streaming data can be subscribed by various types of objects: * Data warehouse for further analysis and reporting. * Streaming engine to perform calculations and to output the results to another stream table. The results can be displayed in real time in a platform such as Grafana and can also serve as a data source for event processing by another subscription. * API. For example, a third-party Java application can subscribe to streaming data with Java API. ### 1.1. Stream Table Stream table is a special type of in-memory table to store and publish streaming data. Stream table supports concurrent read and write. A stream table can be appended to, but the records of a stream table cannot be modified or deleted. SQL statements can be used to query stream tables. ### 1.2. Publish-Subscribe Stream for DolphinDB uses the classic publish-subscribe model. After new data is ingested to a stream table, all subscribers are notified to process the new data. A data node or compute node uses function `subscribeTable` to subscribe to the streaming data. ### 1.3. Streaming Engine The streaming engine refers to a module dedicated to real-time calculation and analysis of streaming data. DolphinDB provides dozens of [Streaming Engines](../Streaming/streaming_engines.html) for real-time calculation with streaming data. 2\. Core Functionalities ------------------------ To enable streaming, specify the configuration parameter _maxPubConnections_ on the publisher node and _subPort_ on the subscriber nodes. For details of streaming configuration, see [Reference](../Database/Configuration/reference.html) . ### 2.1. Publish Streaming Data Use function `streamTable` to define a stream table. Real-time data is ingested into the stream table and then published to all subscribers. A publisher usually has multiple subscribers in multiple sessions, so a stream table must be shared across all sessions with the command `share` before publishing streaming data. Define and share the stream table "pubTable": share streamTable(10000:0,`timestamp`temperature, [TIMESTAMP,DOUBLE]) as pubTable The stream table created by function `streamTable` may contain duplicate records. You can create a stream table with primary keys with function `keyedStreamTable`. For a stream table with primary keys, subsequent records with identical primary key values as an existing record will not be written to the stream table. share keyedStreamTable(`timestamp, 10000:0,`timestamp`temperature, [TIMESTAMP,DOUBLE]) as pubTable ### 2.2. Subscribe to Streaming Data Use function `subscribeTable` to subscribe to streaming data. Syntax of function `subscribeTable`: subscribeTable([server],tableName,[actionName],[offset=-1],handler,[msgAsTable=false],[batchSize=0],[throttle=1],[hash=-1],[reconnect=false],[filter],[persistOffset=false],[timeTrigger=false],[handlerNeedMsgId=false], [raftGroup]) Parameter Description: * Only parameters _tableName_ and _handler_ are required. All other parameters are optional. * The parameter _server_ is a string indicating the alias or the remote connection handle of a server where the stream table is located. If it is unspecified or an empty string (""), it means the local instance. There are 3 possibilities regarding the locations of the publisher and the subscriber: 1. Publisher and subscriber are on the same local data node: use empty string ("") for _server_ or leave it unspecified. subscribeTable(tableName="pubTable", actionName="act1", offset=0, handler=subTable, msgAsTable=true) 2. Publisher and subscriber are not on the same data node but in the same cluster: specify the publisher node's alias for _server_. subscribeTable(server="NODE2", tableName="pubTable", actionName="act1", offset=0, handler=subTable, msgAsTable=true) 3. The publisher is not in the same cluster as the subscriber: specify the remote connection handle of the publisher node for _server_. pubNodeHandler=xdb("192.168.1.13",8891) subscribeTable(server=pubNodeHandler, tableName="pubTable", actionName="act1", offset=0, handler=subTable, msgAsTable=true) * The parameter _tableName_ is a string indicating the shared stream table name. subscribeTable(tableName="pubTable", actionName="act1", offset=0, handler=subTable, msgAsTable=true) * The parameter _actionName_ is a string indicating the subscription task name. It can have letters, digits and underscores. It must be specified if multiple subscriptions on the same node subscribe to the same stream table. topic1 = subscribeTable(tableName="pubTable", actionName="realtimeAnalytics", offset=0, handler=subTable, msgAsTable=true) topic2 = subscribeTable(tableName="pubTable", actionName="saveToDataWarehouse", offset=0, handler=subTable, msgAsTable=true) Function `subscribeTable` returns the subscription topic. It is the combination of the alias of the node where the stream table is located, the stream table name, and the subscription task name (if _actionName_ is specified), separated by forward slash "/". If the local node alias is NODE1, the two topics returned by the example above are as follows: topic1: NODE1/pubTable/realtimeAnalytics topic2: NODE1/pubTable/saveToDataWarehouse If the subscription topic already exists, an exception is thrown. * The parameter _offset_ is an integer indicating the position of the first message where the subscription begins. A message is a row of the stream table. The _offset_ is relative to the first row of the stream table when it is created. If _offset_ is unspecified or -1, the subscription starts with the next new message. If _offset_\=-2, the system will get the persisted offset on disk and start subscription from there. If some rows were cleared from memory due to cache size limit, they are still included in determining where the subscription starts. The following example illustrates the role of _offset_. We write 100 rows to 'pubTable' and create 2 subscription topics: share streamTable(10000:0,`timestamp`temperature, [TIMESTAMP,DOUBLE]) as pubTable share streamTable(10000:0,`ts`temp, [TIMESTAMP,DOUBLE]) as subTable1 share streamTable(10000:0,`ts`temp, [TIMESTAMP,DOUBLE]) as subTable2 vtimestamp = 1..100 vtemp = norm(2,0.4,100) tableInsert(pubTable,vtimestamp,vtemp) topic1 = subscribeTable(tableName="pubTable", actionName="act1", offset=-1, handler=subTable1, msgAsTable=true) topic2 = subscribeTable(tableName="pubTable", actionName="act2", offset=50, handler=subTable2, msgAsTable=true) subTable1 is empty and subTable2 has 50 rows. When _offset_ is -1, the subscription will start from the next new message. * The parameter _handler_ is a unary function or a table. If _handler_ is a function, it is used to process the subscribed data. The only parameter of the function is the subscribed data, which can be a table or a tuple of the subscribed table columns. Quite often we need to insert the subscribed data into a table. For convenience _handler_ can also be a table, and the subscribed data is inserted into the table directly. The following example shows the two uses of _handler_. In the subscription of "act1", the subscribed data is written directly to subTable1; in the subscription of "act2", the subscribed data is filtered with the user-defined function `myHandler` and then written to subTable2. def myhandler(msg){ t = select * from msg where temperature>0.2 if(size(t)>0) subTable2.append!(t) } share streamTable(10000:0,`timestamp`temperature, [TIMESTAMP,DOUBLE]) as pubTable share streamTable(10000:0,`ts`temp, [TIMESTAMP,DOUBLE]) as subTable1 share streamTable(10000:0,`ts`temp, [TIMESTAMP,DOUBLE]) as subTable2 topic1 = subscribeTable(tableName="pubTable", actionName="act1", offset=-1, handler=subTable1, msgAsTable=true) topic2 = subscribeTable(tableName="pubTable", actionName="act2", offset=-1, handler=myhandler, msgAsTable=true) vtimestamp = 1..10 vtemp = 2.0 2.2 2.3 2.4 2.5 2.6 2.7 0.13 0.23 2.9 tableInsert(pubTable,vtimestamp,vtemp) The result shows that 10 messages are written to pubTable. SubTable1 receives all of them whereas subTable2 receives 9 messages as the row with vtemp=0.13 is filtered by `myhandler`. * The parameter _msgAsTable_ is a Boolean value indicating whether the subscribed data is ingested into handler as a table or as a tuple. If msgAsTable=true, the subscribed data is ingested into handler as a table and can be processed with SQL statements. The default value is false, which means the subscribed data is ingested into handler as a tuple of columns. def myhandler1(table){ subTable1.append!(table) } def myhandler2(tuple){ tableInsert(subTable2,tuple[0],tuple[1]) } share streamTable(10000:0,`timestamp`temperature, [TIMESTAMP,DOUBLE]) as pubTable share streamTable(10000:0,`ts`temp, [TIMESTAMP,DOUBLE]) as subTable1 share streamTable(10000:0,`ts`temp, [TIMESTAMP,DOUBLE]) as subTable2 topic1 = subscribeTable(tableName="pubTable", actionName="act1", offset=-1, handler=myhandler1, msgAsTable=true) topic2 = subscribeTable(tableName="pubTable", actionName="act2", offset=-1, handler=myhandler2, msgAsTable=false) vtimestamp = 1..10 vtemp = 2.0 2.2 2.3 2.4 2.5 2.6 2.7 0.13 0.23 2.9 tableInsert(pubTable,vtimestamp,vtemp) * The parameter _batchSize_ is an integer indicating the number of unprocessed messages to trigger the handler. If it is positive, the handler does not process messages until the number of unprocessed messages reaches batchSize. If it is unspecified or non-positive, the handler processes incoming messages as soon as each batch is ingested. In the following example, _batchSize_ is set to 11. share streamTable(10000:0,`timestamp`temperature, [TIMESTAMP,DOUBLE]) as pubTable share streamTable(10000:0,`ts`temp, [TIMESTAMP,DOUBLE]) as subTable1 topic1 = subscribeTable(tableName="pubTable", actionName="act1", offset=-1, handler=subTable1, msgAsTable=true, batchSize=11) vtimestamp = 1..10 vtemp = 2.0 2.2 2.3 2.4 2.5 2.6 2.7 0.13 0.23 2.9 tableInsert(pubTable,vtimestamp,vtemp) print size(subTable1) Write 10 messages to pubTable. Now the subscribing table subTable1 is empty. insert into pubTable values(11,3.1) print size(subTable1) After 1 more message is written to pubTable, the number of unprocessed messages reaches the value of _batchSize_ and triggers the handler. The subscribing table subTable1 receives 11 records. * The parameter _throttle_ is a floating-point number indicating the maximum waiting time (in seconds) before the handler processes the messages if the number of unprocessed messages is smaller than batchSize. The default value is 1. This optional parameter has no effect if batchSize is not specified. If throttle is less than the configuration parameter subThrottle/1000, throttle is effectively subThrottle/1000. There is little difference in the amount of time it takes to process one message versus a batch of messages (for example, 1000 messages). If each individual message triggers the handler, the speed of data ingestion may overwhelm the speed of data consumption. Moreover, data may pile up at the subscriber node and use up all memory. If _batchSize_ and _throttle_ are configured properly, we can improve throughput by adjusting the frequency at which the handler processes the messages. ### 2.3. Automatic reconnection To enable automatic reconnection after network disruption, the stream table must be persisted on the publisher. Please refer to "Streaming Data Persistence" for enabling persistence. When parameter _reconnect_ is set to true, the subscriber will record the offset of the streaming data. When the network connection is interrupted, the subscriber will automatically resubscribe from the offset. If the subscriber crashes or the stream table is not persisted on the publisher, the subscriber cannot automatically reconnect. ### 2.4. Filtering of Streaming Data on the Publisher Node Streaming data can be filtered at the publisher. It may significantly reduce network traffic. Use command `setStreamTableFilterColumn` on the stream table to specify the filtering column (as of now a stream table can have only one filtering column), then specify the parameter _filter_ in function `subscribeTable`. Only the rows with filtering column values in _filter_ are published to the subscriber. In the following example, the value of the parameter _filter_ is a vector. Only the data of 'IBM' and 'GOOG' are published to the subscriber: share streamTable(10000:0,`time`symbol`price, [TIMESTAMP,SYMBOL,INT]) as trades setStreamTableFilterColumn(trades, `symbol) trades_slave=table(10000:0,`time`symbol`price, [TIMESTAMP,SYMBOL,INT]) filter=symbol(`IBM`GOOG) subscribeTable(tableName="trades", actionName="trades_1", handler=append!{trades_1}, msgAsTable=true, filter=filter) In the following example, the value of the parameter _filter_ is a pair. share streamTable(10000:0,`time`symbol`price, [TIMESTAMP,SYMBOL,INT]) as trades setStreamTableFilterColumn(trades, `price) trades_1=table(10000:0,`time`symbol`price, [TIMESTAMP,SYMBOL,INT]) subscribeTable(tableName="trades", actionName="trades_1", handler=append!{trades_1}, msgAsTable=true, filter=1:100) In the following example, the value of the parameter _filter_ is a tuple. share streamTable(10000:0,`time`symbol`price, [TIMESTAMP,SYMBOL,INT]) as trades setStreamTableFilterColumn(trades, `price) trades_1=table(10000:0,`time`symbol`price, [TIMESTAMP,SYMBOL,INT]) subscribeTable(tableName="trades", actionName="trades_1", handler=append!{trades_1}, msgAsTable=true, filter=(10,1:5)) ### 2.5. Unsubscribe Each subscription is uniquely identified with a subscription topic. If a new subscription has the same topic as an existing subscription, the new subscription cannot be established. To renew a subscription, you need to cancel the existing subscription with command `unsubscribeTable`. Unsubscribe from a local table: unsubscribeTable(tableName="pubTable", actionName="act1") Unsubscribe from a remote table: unsubscribeTable(server="NODE_1", tableName="pubTable", actionName="act1") To delete a shared stream table, use command `undef`: undef("pubStreamTable", SHARED) ### 2.6. Streaming Data Persistence By default, the stream table keeps all streaming data in memory. streaming data can be persisted to disk for the following 3 reasons: * Backup and restore streaming data. When a node reboots, the persisted data can be automatically loaded into the stream table. * Avoid out-of-memory problem. * Resubscribe from any position. If the number of rows in the stream table reaches a preset threshold, the first half of the table will be persisted to disk. The persisted data can be resubscribed. To initiate streaming data persistence, we need to add a persistence path to the configuration file of the publisher node: persisitenceDir = /data/streamCache Then execute command `enableTableShareAndPersistence` to share the stream table and enable persistence of it. The following example shares pubTable as sharedPubTable and enables the persistence of it. The parameter _cacheSize_ is set to 1000000, and the default values of parameters _asynWrite_ and _compress_ are both true. When the stream table reaches 1 million rows, the first half of them are compressed to disk asynchronously. pudTable=streamTable(10000:0,`timestamp`temperature, [TIMESTAMP,DOUBLE]) enableTableShareAndPersistence(table=pubTable, tableName=`sharedPubTable, cacheSize=1000000) When `enableTableShareAndPersistence` is executed, if the persisted data of sharedPubTable already exists on the disk, the system will load the latest cacheSize=1000000 rows into the memory. Whether to enable persistence in asynchronous mode is a trade-off between data consistency and latency. If data consistency is the priority, we should use the synchronous mode. This ensures that data enters the publishing queue after the persistence is completed. On the other hand, if low latency is more important and we don't want disk I/O to affect latency, we should use asynchronous mode. The configuration parameter _persistenceWorkerNum_ only works when asynchronous mode is enabled. With multiple publishing tables to be persisted, increasing the value of _persistenceWorkerNum_ can improve the efficiency of persistence in asynchronous mode. Persisted data can be deleted with command `clearTablePersistence`. clearTablePersistence(pubTable) To turn off persistence, use command `disableTablePersistence`. disableTablePersistence(pubTable) Use function `getPersistenceMeta` to get detailed information regarding the persistence of a stream table: getPersistenceMeta(pubTable); The result is a dictionary with the following keys: //The number of rows in memory: sizeInMemory->0 //Whether asynchronous mode is enabled: asynWrite->true //The total number of rows of the stream table: totalSize->0 //Whether the data is persisted with compression: compress->true //The offset of data in memory relative to the total number of rows. memoryOffset = totalSize - sizeInMemory. memoryOffset->0 //The number of rows that have been persisted to disk: sizeOnDisk->0 //How long the log file will be kept in terms of minutes. The default value is 1440 minutes (one day). retentionMinutes->1440 //The path where the persisted data is saved persistenceDir->/hdd/persistencePath/pubTable //hashValue is the identifier of the worker for persistence of the stream table. hashValue->0 //The offset of the first of the persisted streaming messages on disk relative to the entire stream table. diskOffset->0 ### 2.7. Delete Stream Tables All subscriptions must be canceled before stream tables are deleted. To delete a shared stream table, use function `undef` or `dropStreamTable`. For example, delete table pubTable created above: undef(`pubTable, SHARED) dropStreamTable(`pubTable) Function `undef` can remove variables or function definitions from memory. For a persisted stream table, execute `dropStreamTable` and the table will be removed from the memory and disk. dropStreamTable(`pubTable); 3\. Streaming Engines --------------------- ### 3.1. Creating Streaming Engines DolphinDB [Streaming Engines](../Streaming/streaming_engines.html) can be used real-time stream processing. For example, you can create a reactive state engine: rse = createReactiveStateEngine(name="reactiveDemo", metrics =, dummyTable=tickStream, outputTable=result, keyColumn="sym", snapshotDir= "/home/data/snapshot", snapshotIntervalInMsgCount=20000) You can remove the definition of the streaming engine from memory with `dropStreamEngine`: dropStreamEngine("reactiveDemo") **Note**: DolphinDB versions 1.30.21/2.00.9 and earlier do not support concurrent access to the streaming engines. To use this feature, the server version must be higher than 1.30.21/2.00.9, and you must use the `share` statement/function to share the engine, for example, `share(engine, name)` or `share engine as name` (the shared name of the engine must be the same as the engine name). ### 3.2. Parallel Processing To process large volumes of data, you can specify the parameters _filter_ and _hash_ of function `subscribeTable` for parallel processing with multiple subscribers. The following example uses reactive state engines to calculate a factor in parallel. Suppose the configuration parameter _subExecutors_ is set to 4, 4 engines are thus created. Each engine subscribes to the stock data based on the hash value of the stockID and processes with a different subscription thread. The results generated by the 4 engines are output into one table. share streamTable(1:0, `sym`price, [STRING,DOUBLE]) as tickStream setStreamTableFilterColumn(tickStream, `sym) share streamTable(1000:0, `sym`factor1, [STRING,DOUBLE]) as resultStream for(i in 0..3){ rse = createReactiveStateEngine(name="reactiveDemo"+string(i), metrics =, dummyTable=tickStream, outputTable=resultStream, keyColumn="sym") subscribeTable(tableName=`tickStream, actionName="sub"+string(i), handler=tableInsert{rse}, msgAsTable = true, hash = i, filter = (4,i)) } **Note**: If multiple reactive state engines use one output table, the table must be a shared table. Otherwise, it is not thread-safe, and parallel writes to such table may cause system crashes. ### 3.3. Snapshot DolphinDB's built-in streaming engines (except join engines) support snapshots to save the engine state and restore the engine after system interruption. For example, the snapshot of a reactive state engine contains the latest processed message ID and the current state (i.e., the intermediate results) of the engine. If an exception occurs, the engine can be restored to the state of the last snapshot after rebooting, and the subscription will start from the next message once it has been processed. share streamTable(1:0, `sym`price, [STRING,DOUBLE]) as tickStream result = table(1000:0, `sym`factor1, [STRING,DOUBLE]) rse = createReactiveStateEngine(name="reactiveDemo", metrics =, dummyTable=tickStream, outputTable=result, keyColumn="sym", snapshotDir= "/home/data/snapshot", snapshotIntervalInMsgCount=20000) msgId = getSnapshotMsgId(rse) if(msgId >= 0) msgId += 1 subscribeTable(tableName=`tickStream, actionName="factors", offset=msgId, handler=appendMsg{rse}, handlerNeedMsgId=true) **To enable snapshot for a reactive state streaming engine:** **(1) Specify the parameters snapshotDir and snapshotIntervalInMsgCount of function `createReactiveStateEngine`.** The parameter _snapshotDir_ is used to specify the directory where the snapshot is saved, and snapshotIntervalInMsgCount indicates the number of messages between 2 snapshots. After a streaming engine is initialized, the system checks if a file named with the engine name and with a .snapshot extension is in the directory. For the above example, if the file _/home/data/snapshot/reactiveDemo.snapshot_ already exists, the snapshot is loaded to restore the engine. You can obtain the message ID of the latest snapshot with function `getSnapshotMsgId`. If not, -1 is returned. **(2) Set the following parameters of function `subscribeTable`:** * The message _offset_ (the position of the first message where the subscription begins) must be specified; * The parameter _handler_ must be `appendMsg` which can have 2 parameters passed: _msgBody_ and _msgId_; * The parameter _handlerNeedMsgId_ must be true. 4\. High Availability --------------------- ### 4.1. HA Streaming To implement uninterrupted streaming services, DolphinDB adopts a high-availability architecture of data replication based on Raft consensus algorithm to achieve high availability for streams. ### 4.2. HA Streaming Engines In addition to the HA streaming mentioned above, DolphinDB also supports HA streaming engines for uninterrupted stream processing. If high availability is enabled for an engine, after the engine is created on the leader, it will be created on each follower, and the snapshot will be synchronized to followers as well. When a leader is down, the raft group will automatically switch to a new leader and resubscribe to the table, and the engine will be restored to the latest snapshot for continuous real-time processing. The following example creates 2 high-availability stream tables and a high-availability reactive state engine, and then submits high-availability subscription on the raft group 2. haStreamTable(raftGroup=2, table=table(1:0, `sym`price, [STRING,DOUBLE]), tableName="haTickStream", cacheLimit=10000) haStreamTable(raftGroup=2, table=table(1:0, `sym`factor1, [STRING,DOUBLE]), tableName="result", cacheLimit=10000) ret = createReactiveStateEngine(name="haReact", metrics=, dummyTable=objByName("haTickStream"), outputTable=objByName("result"), keyColumn=`sym, snapshotDir= "/home/data/snapshot", snapshotIntervalInMsgCount=20000, raftGroup=2) subscribeTable(tableName="haTickStream", actionName="haFactors", offset=-1, handler=getStreamEngine("haReact"), msgAsTable=true, reconnect=true, persistOffset=true, handlerNeedMsgId=true, raftGroup=2) Note: To configure high availability for a streaming engine, parameters raftGroup, snapshotDir and snapshotIntervalInMsgCount must be specified. When submitting subscription with function `subscribeTable`: * To enable high availability for the subscribers, the parameter _raftGroup_ must be specified. The function can only be executed on the leader if _raftGroup_ is set. * To enable high availability for the streaming engine, _handlerNeedMsgId_ must be set to true and handler must be an engine object. The engine object is the handler returned when you create an engine, or it can be obtained with `getStreamEngine(engineName)`. * To automatically resubscribe to a high-availability stream table, set _reconnect_ to true to automatically reconnect to the new leader in case of a leader switch. * To avoid data loss on the subscribers, set _persistOffset_ to true when subscribing to a high-availability stream table. 5\. Streaming API ----------------- The consumer of streaming data may be a streaming engine, a third-party message queue, or a third-party program. DolphinDB provides streaming API for third-party programs to subscribe to streaming data. When new data arrives, API subscribers receive notifications on time. This allows DolphinDB streaming modules to be integrated with third-party applications. Currently, DolphinDB provides Java, Python, C++ and C# streaming API. 6\. Status monitoring --------------------- ### 6.1. Streaming As streaming calculations are conducted in the background, DolphinDB provides function `getStreamingStat` to monitor the streaming status. After calling function `subscribeTable`, you can check the subscription job with `getStreamingStat().pubTables`. When the worker thread begins data processing, the table subWorkers shows the corresponding thread status. ### 6.2. Streaming Engines Function `getStreamEngineStat` returns a dictionary. The key indicates the streaming engine type, and the value is a table containing metrics of the engine. For example, `getStreamEngineStat().DailyTimeSeriesEngine` returns a table as below: ![image](images/streaming/subconn.png) There is only one Daily Time Series Engine engine1 in the system, and it occupies about 32 KB memory. The memory usage of a streaming engine is mainly due to the continuous data ingested into the engine. When creating an engine, you can manage the frequency of historical data cleanup with parameter _garbageSize_ to control the memory usage. It is recommended to remove the engines that are no longer in use from memory with function `dropStreamEngine`. To delete an engine handle, set the handle to null. 7\. Performance Tuning ---------------------- If an extremely large amount of streaming data arrive in a short period of time, we may see an extremely high value of _queueDepth_ in table subWorkers on the subscriber node. When the message queue depth on the a subscriber node reaches _queueDepthLimit_, new messages from the publisher node are blocked from entering subscriber nodes. This leads to a growing message queue on the publisher node. When the message queue depth of the publisher node reaches _queueDepthLimit_, the system blocks new messages from entering the stream table on the publisher node. There are several ways to optimize streaming performance: 1. Adjust parameters _batchSize_ and _throttle_ to balance the cache on the publisher and the subscriber nodes. This can achieve a dynamic balance between streaming data ingestion speed and data processing speed. To take full advantage of batch processing, we can set _batchSize_ to wait for streaming data to accumulate to a certain amount before consumption. This, however, needs a certain amount of memory usage. If _batchSize_ is too large, it is possible that some data below the threshold of _batchSize_ get stuck in the buffer for a long time. To handle this issue, we can set parameter _throttle_ to consume the buffer data even if _batchSize_ is not met. 2. Increase parallelism by adjusting the configuration parameter _subExecutors_ to speed up message queue processing on the subscriber node. 3. When each executor processes a different subscription, if there is great difference in throughput of different subscriptions or the complexities of calculation, some executors may idle. If we set subExecutorPooling=true, all executors form a shared thread pool to process all subscribed messages together. All subscribed messages enter the same queue and multiple executors read messages from the queue for parallel processing. With the shared thread pool, however, there is no guarantee that messages will be processed in the order they arrive. Therefore parameter _subExecutorPooling_ should not be turned on when messages need to be processed in the order they arrive. By default, the system uses a hash algorithm to assign an executor to each subscription. If we would like to ensure the messages in two subscriptions are processed by the order as they arrive, we can specify the same value for parameter _hash_ in function `subscribeTable` of the two subscriptions, so that the same executor processes the messages of the two subscriptions. 4. If the stream table enables synchronous persistence, disk I/O maybe become a bottleneck. One way to handle this is to use the method in section 2.4 to persist data in asynchronous mode while setting up a reasonable persistence queue by specifying _maxPersistenceQueueDepth_ (default 10 million messages). Alternatively, we can also use SSD hard drives to improve write performance. 5. Publisher node may become the bottleneck. For example, there may be too many subscribing clients. We can use the following methods to handle this problem. First, we can reduce the number of subscriptions for each publisher node with multi-level cascading. Applications that can tolerate delay can subscribe to secondary or even third tier publisher nodes. Second, some parameters can be adjusted to balance throughput and latency. The parameter _maxMsgNumPerBlock_ sets the size of the messages batches. The default value is 1024. In general, larger batches increase throughput but increase network latency. 6. If the ingestion speed of streaming data fluctuates greatly and peak periods cause the consumption queue to reach the queue depth limit (default 10 million), then we can modify configuration parameters _maxPubQueueDepthPerSite_ and _maxSubQueueDepth_ to increase the maximum queue depth of the publisher and the subscriber. However, as memory consumption increases as queue depth increases, memory usage should be planned and monitored to properly allocate memory. 8\. Visualization ----------------- Streaming data visualization can be either real-time value monitoring or trend monitoring. * Real-time value monitoring: conduct rolling-window computation of aggregate functions with streaming data periodically. * Trend monitoring: generate dynamic charts with streaming data and historical data. Various data visualization platforms support real-time monitoring of streaming data, such as the popular open source data visualization framework Grafana. DolphinDB has implemented the interface between the server and client of Grafana. For details, please refer to the tutorial: https://github.com/dolphindb/grafana-datasource/blob/master/README.md 9\. Recommended Reading ----------------------- * [Streaming APIs](../API/chap_api.html "API in different languages for users to call DolphinDB from their programs") * [Historical Data Replay](historical_data_replay.html) --- # R API [Jump to main content](#wh_topic_body) R API ===== About ----- ### Description This is a R-Package which will make it easy for you to handle _DolphinDB_ with _R_. It is implemented by _R_ & _C++_, with the help of package `Rcpp`. The package supports : * Running scripts * Executing functions with arguments * Uploading variables on _DolphinDB_ server, while receiving object from server in the form and type of _R_. ### Dependency * R ( ≥ 3. 2. 0) * Rcpp ( ≥ 0. 12. 17) Install ------- ### Install _R_ Environment * For **Linux** user * `sudo apt-get install r-base` * **OR** download and install manually : https://www.r-project.org/ * For **Windows** user * Download and install the r base packages and the rtools at https://www.r-project.org/ ### Get into _R_ CMD * Type `R` in your terminal/console. * You should have set the correct **environment variable** and **path** for _R_. ### Install _devtools_ Package in _R_ * In _R_ CMD, type `install.packages("devtools")` to install package _devtools_. * Select the nearest mirror to download and install automatically. * Make sure the Internet is in good condition. ### Install _RDolphinDB_ Package by _devtools_ * In _R_ CMD, type `devtools::install_github("dolphindb/api-r")`. * This command will automatically download and install _RDolphinDB_ and its dependency package. * For **Windows** user * If the Installation failed with message: _Warning in system(cmd) : 'make' not found_. Just try: Sys.setenv(PATH = paste("*InstallDirectory*/Rtools/bin", Sys.getenv("PATH"), sep=";")) Sys.setenv(BINPREF = "*InstallDirectory*/Rtools/mingw_64/bin") * After installation, the package will be compiled and linked by _g++_ automatically. ### Use the _RDolphinDB_ Package * Assume you are running _DolphinDB_ server on **localhost:8848** * Then you can use `RDolphinDB` in the following way library(RDolphinDB) conn <- dbConnect(DolphinDB(), "localhost", 8848) if (conn@connected) { dbUpload(conn, c("val1", "val2"), list(3.5, c(1.3, 2.6, 3.7))) res_run <- dbRun(conn, "1 2 3") res_rpc <- dbRpc(conn, "size", list(c(1, 2, 3))) print(res_run) print(res_rpc) } dbClose(conn) Supported Data Types -------------------- R API supports the following data types of DolphinDB: | DolphinDB Type | DolphinDB Example | R Type | R Example | Description | | --- | --- | --- | --- | --- | | BOOL | false | Logical | FALSE | | | CHAR | 'A' | Integer | 65 | | | SHORT | 32 | Integer | 32 | | | INT | 1 | Integer | 1 | | | LONG | 100000 | Numeric | 10000 | The maximum integer in R is 2147483647 | | DATE | 2013.06.13 | Date | 2013-06-13 | | | MONTH | 2013.08M | Date | 2013-08-01 | The first day of the specified month | | TIME | 13:30:10.008 | POSIXct | 1970-01-01 13:30:10 | The specified timestamp on 1970.01.01 (accurate to seconds) | | MINUTE | 13:30m | POSIXct | 1970-01-01 13:30:00 | The specified timestamp on 1970.01.01 | | SECOND | 13:30:10 | POSIXct | 1970-01-01 13:30:10 | The specified timestamp on 1970.01.01 | | DATETIME | 2012.06.13T13:30:10 | POSIXct | 2012-06-13 13:30:10 | | | TIMESTAMP | 2012.06.13T13:30:10.008 | POSIXct | 2012-06-13 13:30:10 | | | NANOTIME | 13:30:10.008007006 | POSIXct | 1970-01-01 13:30:10 | The specified timestamp on 1970.01.01 (accurate to seconds) | | NANOTIMESTAMP | 2012.06.13T13:30:10.008007006 | POSIXct | 2012-06-13 13:30:10 | | | FLOAT | 2.1f | Numeric | 2.1 | | | DOUBLE | 2.1 | Numeric | 2.1 | | | STRING | "123" | character | "123" | | | SYMBOL | \["123", "123"\] | factor | \["123", "123"\] | | | BLOB | | Not supported | | | | DATEHOUR | | Not supported | | | Note: For SYMBOL data in DolphinDB, the R API parses it as factor. A SYMBOL column with few duplicate values may actually be stored as STRING type in DolphinDB, and the R API parses such column as character rather than factor. Documentation ------------- ### In _R_ CMD # About the package help(package = "RDolphinDB") # About the functions help("DolphinDB") help("dbConnect") help("dbRun") help("dbRpc") help("dbUpload") help("dbClose") ### Through our website https://dolphindb.com/ --- # JavaScript API [Jump to main content](#wh_topic_body) JavaScript API ============== Overview -------- The DolphinDB JavaScript API is a JavaScript library that encapsulates interactions with the DolphinDB database, such as connecting to the database, executing scripts, calling functions, uploading variables, etc. https://www.npmjs.com/package/dolphindb Features -------- * Communicates with the DolphinDB database using WebSocket and exchanges data in binary format, supporting real-time streaming data. * Supports running in both browser and Node.js environments. * Uses TypedArray in JavaScript such as Int32Array to handle binary data. * Supports serialized upload of up to 2 GB of data in a single call, with no limit on the amount of downloaded data. Usage ----- ### Connecting to DolphinDB #### Method 1: Use the built CDN version directly in the browser Save the following content to an `example.html` file and open it with a browser to run it. Press F12 to open the debug console and check the logs. DolphinDB #### Method 2: Install the npm package and import to the project ##### Installation (1) Install the latest version of Node.js and browser on your machine. * Windows: https://nodejs.org/en/download/prebuilt-installer/current * Linux: https://github.com/nodesource/distributions?tab=readme-ov-file#debian-and-ubuntu-based-distributions (2) (Optional) Create a new project using the following command. Skip this step if you already have a project. mkdir dolphindb-example cd dolphindb-example npm init --yes (3) Open the `package.json` file with an editor and add a line `"type": "module"` below `"main": "./index.js"` to enable ECMAScript modules. In the following code, you can use `import { DDB } from 'dolphindb'` to import npm package. (4) Install the npm package in the project. npm install dolphindb ##### Usage // 2.1 import in the browser environment import { DDB } from 'dolphindb/browser.js' // 2.1 import in the Node.js environment // import { DDB } from 'dolphindb' // For existing projects using CommonJS modules, import as follows: const { DDB } = await import('dolphindb') // 2.2 Initialize the instance to connect to DolphinDB using the WebSocket URL (no actual network connection is established) let ddb = new DDB('ws://127.0.0.1:8848') // Use HTTPS encryption // let ddb = new DDB('wss://dolphindb.com') // 2.3 Establish a connection to DolphinDB (requires DolphinDB database version no less than 1.30.16 or 2.00.4) await ddb.connect() #### Code completion and function prompt See https://cdn.dolphindb.cn/assets/docs.en.json #### Connection options let ddb = new DDB('ws://127.0.0.1:8848', { // Whether to automatically log in after establishing a connection, default is true autologin: true, // DolphinDB login username, default is 'admin' username: 'admin', // DolphinDB login password, default is '123456' password: '123456', // Set python session flag, default is false python: false, // Set the SQL standard to execute in the current session, use the SqlStandard enum, default is DolphinDB // sql: SqlStandard.MySQL, // sql: SqlStandard.Oracle, // Set this option for the database connection to be used only for streaming data, see `5. Streaming Data` for details streaming: undefined }) ### Calling functions #### `invoke` method ##### Code example const result = await ddb.invoke('add', [1, 1]) // TypeScript: const result = await ddb.invoke('add', [1, 1]) console.log(result === 2) // true In the example above, two parameters 1 (corresponding to the int type in DolphinDB) are uploaded to the DolphinDB database as parameters of the `add` function, and the result of the function call is received in `result`. `` is used for TypeScript to infer the type of the return value. ##### Method declaration /** Call a dolphindb function, passing in a native JS array as parameters, and return a native JS object or value (result after calling DdbObj.data()) */ async invoke ( /** Function name */ func: string, /** `[ ]` Call parameters, can be a native JS array */ args?: any[], /** Call options */ options?: { /** Urgent flag. Use urgent worker to execute, preventing being blocked by other jobs */ urgent?: boolean /** When setting node alias, send to the corresponding node in the cluster to execute (using DolphinDB's rpc method) */ node?: string /** When setting multiple node aliases, send to the corresponding multiple nodes in the cluster to execute (using DolphinDB's pnodeRun method) */ nodes?: string[] /** Required when setting the node parameter and the parameter array is empty, specify the function type, not passed in other cases */ func_type?: DdbFunctionType /** Optionally passed when setting the nodes parameter, not passed in other cases */ add_node_alias?: boolean /** Handle messages (DdbMessage) during this rpc */ listener?: DdbMessageListener } = { } ): Promise #### `call` method ##### Code example import { DdbInt } from 'dolphindb' const result = await ddb.call('add', [new DdbInt(1), new DdbInt(1)]) // TypeScript: const result = await ddb.call('add', [new DdbInt(1), new DdbInt(1)]) console.log(result.value === 2) // true #### Using DdbObj objects to represent data types in DolphinDB In the example above, two parameters `new DdbInt(1)`, corresponding to the INT type in DolphinDB, are uploaded to the DolphinDB database as arguments of the `add` function, and the result of the function call is received in `result`. `` is used by TypeScript to infer the type of the return value * result is a DdbInt, which is also a `DdbObj` * result.form is a DdbForm.scalar * result.type is a DdbType.int * result.value is data of number type in JavaScript (the value range and precision of INT can be accurately represented by JavaScript number type) It is recommended to first understand the concepts related to TypedArray in JavaScript, you can refer to: * https://stackoverflow.com/questions/42416783/where-to-use-arraybuffer-vs-typed-array-in-javascript * https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global\_Objects/TypedArray /** Can represent all data types in DolphinDB databases */ class DdbObj { /** is it little endian */ le: boolean /** DolphinDB data form */ form: DdbForm /** DolphinDB data type */ type: DdbType /** consumed length in buf parsed */ length: number /** table name / column name */ name?: string /** Lowest dimension - vector: rows = n, cols = 1 - pair: rows = 2, cols = 1 - matrix: rows = n, cols = m - set: the same as vector - dict: include keys, values vector - table: the same as matrix */ rows?: number /** 2nd dimension */ cols?: number /** the actual data. Different DdbForm, DdbType use different types in DdbValue to represent actual data */ value: T /** raw binary data, only top-level objects generated by parse_message when parse_object is false have this attribute */ buffer?: Uint8Array constructor (data: Partial & { form: DdbForm, type: DdbType, length: number }) { Object.assign(this, data) } } class DdbInt extends DdbObj { constructor (value: number) { super({ form: DdbForm.scalar, type: DdbType.int, length: 4, value }) } } // ... There are also many utility classes, such as DdbString, DdbLong, DdbDouble, DdbVectorDouble, DdbVectorAny, etc. type DdbValue = null | boolean | number | [number, number] | bigint | string | string[] | Uint8Array | Int16Array | Int32Array | Float32Array | Float64Array | BigInt64Array | Uint8Array[] | DdbObj[] | DdbFunctionDefValue | DdbSymbolExtendedValue enum DdbForm { scalar = 0, vector = 1, pair = 2, matrix = 3, set = 4, dict = 5, table = 6, chart = 7, chunk = 8, } enum DdbType { void = 0, bool = 1, char = 2, short = 3, int = 4, long = 5, // ... timestamp = 12, // ... double = 16, symbol = 17, string = 18, // ... } ##### Specifying form and type to manually create a DdbObj object If there is no shortcut class, you can also specify form and type to manually create a DdbObj object: // Created by the DdbDateTime shortcut class new DdbDateTime(1644573600) // Equivalent to manually creating an object of form = scalar, type = datetime through DdbObj const obj = new DdbObj({ form: DdbForm.scalar, type: DdbType.datetime, value: 1644573600, length: 0 }) // The corresponding type and value of value in js can refer to the result returned by ddb.eval (see the `eval` method declaration below) const obj = await ddb.eval('2022.02.11 10:00:00') console.log(obj.form === DdbForm.scalar) console.log(obj.type === DdbType.datetime) console.log(obj.value) // Another example is to create a set // refer to ddb.eval // const obj = await ddb.eval('set([1, 2, 3])') // console.log(obj.value) const obj = new DdbObj({ form: DdbForm.set, type: DdbType.int, value: Int32Array.of(1, 2, 3), length: 0 }) // easier to use shortcut classes const obj = new DdbSetInt( new Set([1, 2, 3]) ) ##### NULL object in scalar For the NULL object in the form of scalar, the value corresponding to DdbObj is null in JavaScript: ;(await ddb.eval('double()')).value === null // create NULL object new DdbInt(null) new DdbDouble(null) ##### Method declaration async call ( /** function name */ func: string, /** function arguments (The incoming native string and boolean will be automatically converted to DdbObj and DdbObj) */ args?: (DdbObj | string | boolean)[] = [ ], /** calling options */ options?: { /** Urgent flag. Use urgent worker to execute to prevent being blocked by other jobs */ urgent?: boolean /** When the node alias is set, the function is sent to the corresponding node in the cluster for execution (using the rpc method in DolphinDB) */ node?: string /** When setting multiple node aliases, send them to the corresponding multiple nodes in the cluster for execution (using the pnodeRun method in DolphinDB) */ nodes?: string[] /** It must be passed when setting the node parameter, the function type needs to be specified, and it is not passed in other cases */ func_type?: DdbFunctionType /** It may be passed when setting the nodes parameter, otherwise may not be passed */ add_node_alias?: boolean } = { } ): Promise ### Executing scripts #### `execute` method ##### Code example const result = await ddb.eval( 'def foo (a, b) {\n' + ' return a + b\n' + '}\n' + 'foo(1l, 1l)\n' ) // TypeScript: // import type { DdbLong } from 'dolphindb' // const result = await ddb.eval(...) console.log(result.value === 2n) // true In the preceding example, a script is uploaded through a string to the DolphinDB database for execution, and the execution result of the last statement `foo(1l, 1l)` is received. `` is used by TypeScript to infer the type of the return value * result is a `DdbLong`, which is also a `DdbObj` * result.form is `DdbForm.scalar` * result.type is `DdbType.long` * result.value is the native `bigint` in JavaScript (the precision of long cannot be accurately represented by JavaScript number, but it can be represented by bigint) As long as the WebSocket connection is not disconnected, the custom function `foo` will always exist in the subsequent session and can be reused, for example, you can use `await ddb.call('foo', [new DdbInt(1), new DdbInt(1)])` to call this custom function ##### Method declaration /** Execute DolphinDB script and return a native JS object or value (result after calling DdbObj.data()) */ async execute ( /** Script to execute */ script: string, /** Execution options */ options?: { /** Urgent flag, ensure the submitted script is processed using an urgent worker to prevent blocking by other jobs */ urgent?: boolean /** listener?: Handle messages (DdbMessage) during this rpc */ listener?: DdbMessageListener } ): Promise #### `eval` Method ##### Code example const result = await ddb.eval( 'def foo (a, b) {\n' + ' return a + b\n' + '}\n' + 'foo(1l, 1l)\n' ) // TypeScript: // import type { DdbLong } from 'dolphindb' // const result = await ddb.eval(...) console.log(result.value === 2n) // true In the example above, a script is uploaded as a string to the DolphinDB database for execution, and the result of the last statement `foo(1l, 1l)` is received in `result`. `` is used for TypeScript to infer the return value type. * result is a `DdbLong`, also `DdbObj` * result.form is `DdbForm.scalar` * result.type is `DdbType.long` * result.value is a native `bigint` in JavaScript (the precision of long cannot be accurately represented by JavaScript's number, but can be represented by bigint) ##### Method declaration /** Execute DolphinDB script and return a DdbObj object */ async eval ( /** Script to execute */ script: string, /** Execution options */ options: { /** Urgent flag, ensure the submitted script is processed using an urgent worker to prevent blocking by other jobs */ urgent?: boolean } = { } ): Promise ### Upload Variables #### Code example import { DdbVectorDouble } from 'dolphindb' let a = new Array(10000) a.fill(1.0) ddb.upload(['bar1', 'bar2'], [new DdbVectorDouble(a), new DdbVectorDouble(a)]) In the example above, two variables `bar1` and `bar2` are uploaded, with values being double vectors of length 10000. As long as the WebSocket connection is not disconnected, these variables `bar1` and `bar2` will always exist in subsequent sessions and can be reused. #### Method declaration async upload ( /** Variable names to upload */ vars: string[], /** Variable values to upload */ args: (DdbObj | string | boolean)[] ): Promise ### Other Examples import { nulls, DdbInt, timestamp2str, DdbVectorSymbol, DdbTable, DdbVectorDouble } from 'dolphindb' // Format timestamp in DolphinDB to string timestamp2str( ( await ddb.call('now', [false]) // TypeScript: await ddb.call>('now', [false]) ).value ) === '2022.02.23 17:23:13.494' // Create symbol vector new DdbVectorSymbol(['aaa', 'aaa', 'aaa', 'aaa', 'aaa', 'bbb']) // Create double vector with null values using native JavaScript array new DdbVectorDouble([0.1, null, 0.3]) // Create double vector more efficiently and memory-saving using JavaScript TypedArray let av = new Float64Array(3) av[0] = 0.1 av[1] = nulls.double av[2] = 0.3 new DdbVectorDouble(av) // Create DdbTable new DdbTable( [\ new DdbVectorDouble([0.1, 0.2, null], 'col0'),\ new DdbVectorSymbol(['a', 'b', 'c'], 'col1')\ ], 'mytable' ) ### Streaming Data // Create new streaming data connection configuration let sddb = new DDB('ws://192.168.0.43:8800', { autologin: true, username: 'admin', password: '123456', streaming: { table: 'name of the stream table to subscribe to', // Stream data processing callback, message type is StreamingMessage handler (message) { console.log(message) } } }) // Establish connection await sddb.connect() After the connection is established, the received streaming data will be called as the `message` parameter of the `handler`, and the message type is `StreamingMessage`, as follows: export interface StreamingParams { table: string action?: string handler (message: StreamingMessage): any } export interface StreamingMessage extends StreamingParams { /** The time when the server sends the message (nano seconds since epoch) std::chrono::system_clock::now().time_since_epoch() / std::chrono::nanoseconds(1) */ time: bigint /** Message ID */ id: bigint /** Subscription topic, i.e., the name of a subscription. It is a string composed of the alias of the node where the subscription table is located, the name of the stream table, and the name of the subscription task (if actionName is specified), separated by `/` */ topic: string /** Streaming data */ data: DdbTableData window: { /** Offset from the start of the connection, starting at 0, and increasing as the window moves */ offset: number /** Historical data */ data: TRows[] /** Array of objects received each time */ objs: DdbObj[] } /** If there is an error in parsing the message pushed after the successful subscription, the error is set and the handler is called */ error?: Error } Development Method ------------------ # Install the latest version of nodejs (see above) # Install pnpm package manager npm install -g pnpm git clone https://github.com/dolphindb/api-javascript.git cd api-javascript # Install project dependencies pnpm install # Copy .vscode/settings.template.json to .vscode/settings.json cp .vscode/settings.template.json .vscode/settings.json # Refer to the scripts in package.json # Build pnpm run build # Lint pnpm run lint # Test pnpm run test # Scan entries pnpm run scan # Manually complete untranslated entries # Run the scan again to update the dictionary file dict.json pnpm run scan --- # Application in Streaming [Jump to main content](#wh_topic_body) Application in Streaming ======================== Based on the DolphinDB streaming framework, CEP engine can achieve seamless integration of various streaming engines. The following example demonstrates how the CEP engine processes stock events. A time-series engine is used to calculate OHLC. The "streamMinuteBar\_1min" variable of the `MainMonitor` represents a table, which will be populated with the results of the time-series engine's calculations on the incoming stock events. class MarketData{ market :: STRING code :: STRING price :: DOUBLE qty :: INT eventTime :: TIMESTAMP def MarketData(m,c,p,q){ market = m code = c price = p qty = q eventTime = now() } } class MainMonitor{ streamMinuteBar_1min :: ANY // store the OHLC results tsAggrOHLC :: ANY // time-series engine def MainMonitor(){ colNames = ["time","symbol","open","max","min","close","volume","amount","ret","vwap"] colTypes = [TIMESTAMP, SYMBOL, DOUBLE, DOUBLE, DOUBLE, DOUBLE, INT, DOUBLE, DOUBLE, DOUBLE] streamMinuteBar_1min = table(10000:0,colNames, colTypes) } def updateMarketData(event) // define a listener and a time-series engine def onload(){ addEventListener(updateMarketData,'MarketData',,'all') colNames=["symbol","time","price","type","volume"] colTypes=[SYMBOL, TIMESTAMP, DOUBLE, STRING, INT] dummy = table(10000:0,colNames,colTypes) colNames = ["time","symbol","open","max","min","close","volume","amount","ret","vwap"] colTypes = [TIMESTAMP, SYMBOL, DOUBLE, DOUBLE, DOUBLE, DOUBLE, INT, DOUBLE, DOUBLE, DOUBLE] output = table(10000:0,colNames, colTypes) tsAggrOHLC = createTimeSeriesEngine(name="tsAggrOHLC", windowSize=60000, step=60000, metrics=<[first(price) as open ,max(price) as max,min(price) as min ,last(price) as close ,sum(volume) as volume ,wsum(volume, price) as amount ,(last(price)-first(price)/first(price)) as ret, (wsum(volume, price)/sum(volume)) as vwap]>, dummyTable=dummy, outputTable=streamMinuteBar_1min, timeColumn='time', useSystemTime=false, keyColumn="symbol", fill=`none) } def updateMarketData(event){ tsAggrOHLC.append!(table(event.code as symbol, event.eventTime as time, event.price as price, event.market as type, event.qty as volume)) } } dummy = table(array(STRING, 0) as eventType, array(BLOB, 0) as blobs) engine = createCEPEngine(name='cep1', monitors=, dummyTable=dummy, eventSchema=[MarketData]) --- # ops [Jump to main content](#wh_topic_body) ops === Starting from DolphinDB 1.30.19/2.00.7, you can use the "ops" module to perform database maintenance tasks such as canceling running jobs in a cluster, viewing disk usage and closing inactive sessions without having to write your own maintenance script. 1\. Environment --------------- The ops module is delivered with DolphinDB server 1.30.19/2.00.7 or higher. The module file _ops.dos_ is under the directory `server/modules`. You can also download the ops module [here](src/ops.dos) . Place the module file under the directory `[home]/modules` on the controller and data nodes in your cluster. The `[home]` directory is specified by the configuration parameter _home_, which you can check with the function `getHomeDir()`. For more information about DolphinDB modules, see [tutorial: Modules](modules.html) . 2\. Calling Module Functions ---------------------------- Import the ops module with the `use` keyword. There are 2 ways to call the module functions: * Refer to the module function directly use ops getAllLicenses() * Specify the full namespace of the function Use this option if other imported modules in the current session contain functions that have the same name. use ops ops::getAllLicenses() 3\. Function Reference ---------------------- ### 3.1. cancelJobEx **Syntax** cancelJobEx(id=NULL) **Arguments** * id: a string indicating a background job ID, which you can get with the server function `getRecentJobs()`. **Details** Cancels running background jobs in the cluster. If _id_ is specified, cancel the specified job; otherwise, cancel all the background jobs in the cluster. **Example** Create 3 background jobs: def testJob(n,id){ for(i in 0:n){ writeLog("demo"+id+"is working") sleep(1000) } } submitJob("demo1","test background job1",testJob,300,1); submitJob("demo2","test background job2",testJob,300,2); submitJob("demo3","test background job3",testJob,300,3); Cancel the job "demo1" and get the status of all background jobs on the data nodes and compute nodes: cancelJobEx("demo1") pnodeRun(getRecentJobs) The result shows that "demo1" is marked as "The task was canceled." | node | userID | jobId | rootJobId | jobDesc | priority | parallelism | receivedTime | startTime | endTime | errorMsg | | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | | comnode1 | admin | demo1 | 45c4eb71-6812-2b83-814e-ed6b22a99964 | test background job1 | 4 | 2 | 2022.08.29T17:20:47.061 | 2022.08.29T17:20:47.061 | 2022.08.29T17:22:15.081 | testJob: sleep(1000) => The task was canceled. | | comnode1 | admin | demo2 | 1c16dfec-7c5a-92b3-414d-0cfbdc83b451 | test background job2 | 4 | 2 | 2022.08.29T17:20:47.061 | 2022.08.29T17:20:47.062 | | | | comnode1 | admin | demo3 | e9dffcc1-3194-9181-8d47-30a325774697 | test background job3 | 4 | 2 | 2022.08.29T17:20:47.061 | 2022.08.29T17:20:47.062 | | | To cancel all jobs, run the following script: pnodeRun(getRecentJobs) ### 3.2. closeInactiveSessions **Syntax** closeInactiveSessions(hours=12) **Arguments** * hours: a numeric value indicating the session timeout period (in hours). The default value is 12. **Return** Returns a table containing information on all active sessions in the cluster. The table has the same schema as the table returned by the server function [getSessionMemoryStat](../Functions/g/getSessionMemoryStat.html) . **Details** If a session has been inactive for a time period longer than the specified _hours_, it is considered as timed out. Call this function to close all inactive sessions. Note: To check the last active time of a session, call server function [getSessionMemoryStat](../Functions/g/getSessionMemoryStat.html) . **Examples** getSessionMemoryStat() | userId | sessionId | memSize | remoteIP | remotePort | createTime | lastActiveTime | | --- | --- | --- | --- | --- | --- | --- | | admin | 1195587396 | 16 | 125.119.128.134 | 20252 | 2022.09.01T08:42:16.980 | 2022.09.01T08:45:23.808 | | guest | 2333906441 | 16 | 115.239.209.122 | 37284 | 2022.09.01T06:39:05.530 | 2022.09.01T08:42:17.127 | closeInactiveSessions(0.05) | userId | sessionId | memSize | remoteIP | remotePort | createTime | lastActiveTime | node | | --- | --- | --- | --- | --- | --- | --- | --- | | admin | 1195587396 | 16 | 125.119.128.134 | 20252 | 2022.09.01T08:42:16.980 | 2022.09.01T08:45:23.808 | DFS\_NODE1 | ### 3.3. getDDL **Syntax** getDDL(database, tableName) **Arguments** * database: a string indicating the path to a distributed database, e.g., "dfs://demodb". * tableName: a string indicating the name of a DFS table **Details** Returns the DDL statements that can be used to recreate the specified database and the DFS table, as well as the column names and column types of the DFS table. **Examples** ID=rand(10, n) x=rand(1.0, n) t=table(ID, x) db=database("dfs://rangedb", RANGE, 0 5 10) pt=db.createPartitionedTable(t, `pt, `ID) getDDL("dfs://rangedb", "pt") #output db = database("dfs://rangedb") colName = `ID`x colType = [INT,DOUBLE] tbSchema = table(1:0, colName, colType) db.createPartitionedTable(table=tbSchema,tableName=`pt,partitionColumns=`ID) ### 3.4. getTableDiskUsage **Syntax** getTableDiskUsage(database, tableName, byNode=false) **Arguments** * database: a string indicating the path to a distributed database, e.g., "dfs://demodb". * tableName: a string indicating the name of a DFS table. * byNode: is a Boolean indicating whether to display disk usage by node. The default value is false, i.e., to display the total disk usage of all nodes. **Details** Returns a table displaying the disk usage of the specified DFS table. It has the following columns: * node: a string indicating a node alias. It is returned only when _byNode = true_. * diskGB: a DOUBLE value indicating the disk usage of the specified DFS table. **Examples** getTableDiskUsage("dfs://rangedb", "pt", true) | node | diskGB | | --- | --- | | DFS\_NODE1 | 0.008498 | ### 3.5. dropRecoveringPartitions **Syntax** dropRecoveringPartitions(dbPath , tableName="") **Arguments** * dbPath:a string indicating the path to a distributed database, e.g., "dfs://demodb". * tableName: a string indicating the name of a DFS table. Specify it only when the database chunk granularity is at TABLE level (i.e.,[database](../Functions/d/database.html) : chunkGranularity = 'TABLE'). **Details** Deletes the partitions in RECOVERING status from the specified database. _tableName_ is a required parameter when the database chunk granularity is at TABLE level. **Example** First, get the metadata of all chunks in the cluster with the following server functions: rpc(getControllerAlias(), getClusterChunksStatus) | chunkId | file | size | version | vcLength | versionChain | state | replicas | replicaCount | lastUpdated | permission | | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | | 5c3bd88f-8a13-a382-2848-cb7c6e75d0fa | /olapDemo/20200905/61\_71/53R | 0 | 2 | 3 | 19752:0:2:7460 -> 19506:0:1:7214 -> 19506:0:0:7214 -> | RECOVERING | DFS\_NODE1:2:0:false:7494976728710525 | 1 | 2022.08.23T04:20:03.100 | READ\_WRITE | | 620526c7-6cf1-3c89-5444-de04f46aaa93 | /olapDemo/20200904/51\_61/53R | 0 | 2 | 3 | 19746:0:2:7454 -> 19495:0:1:7203 -> 19495:0:0:7203 -> | RECOVERING | DFS\_NODE1:2:0:false:7494976704543705 | 1 | 2022.08.23T04:20:02.564 | READ\_WRITE | The result suggests that both chunk files of the "olapDemo" database are in RECOVERING status. Execute `dropRecoveringPartitions` to force delete these two partitions: dropRecoveringPartitions(database("dfs://olapDemo")); ### 3.6. getAllLicenses **Syntax** getAllLicenses() **Arguments** None **Details** Returns a table displaying the license expiration date of all nodes in the cluster. It has the following columns: * nodeAlias: a string indicating a node alias. * endDate: a date indicating the expiration date. **Examples** getAllLicenses() | nodeAlias | endDate | | --- | --- | | DFS\_NODE1 | 2042.01.01 | | ctl18920 | 2042.01.01 | | agent | 2042.01.01 | ### 3.7. updateAllLicenses **Syntax** updateAllLicenses() **Arguments** None **Return** Returns a table displaying the license expiration date of all nodes in the cluster. It has the following columns: * nodeAlias: a string indicating a node alias. * endDate: a date indicating the expiration date. **Details** Note: Execute this function after you have replaced the license files on the nodes. Updates the license on all nodes in a cluster without a reboot. Return license expiration information. **Example** updateAllLicenses() | nodeAlias | endDate | | --- | --- | | DFS\_NODE1 | 2042.01.01 | | ctl18920 | 2042.01.01 | | agent | 2042.01.01 | ### 3.8. unsubscribeAll **Syntax** unsubscribeAll() **Arguments** None **Details** Cancels all subscriptions on the current node. **Examples** share streamTable(10:0, `id`val, [INT, INT]) as st t = table(10:0, `id`val, [INT, INT]) subscribeTable(tableName=`st, actionName=`sub_st, handler=append!{t}) undef(st, SHARED) #error All subscriptions to the shared stream table [st] must be canceled before it can be undefined. unsubscribeAll() undef(st, SHARED) ### 3.9. gatherClusterPerf **Syntax** gatherClusterPerf(monitoringPeriod=60, scrapeInterval=15, dir="/tmp") **Arguments** * monitoringPeriod: an integer indicating the time frame (in seconds) of monitoring. The default value is 60. * scrapeInterval: the interval (in seconds) at which the monitoring metrics are scraped. The default value is 15. * dir: a string indicating an existing directory to save the monitoring result. The default value is "/tmp". Note: For a Windows system, use absolute-path and forward slashes ("**/**") or double backslashes (\\) to separate the directories. **Details** Gets cluster performance monitoring metrics based on the specified monitoring period and scrape interval. Exports the result to the specified directory in a _statis.csv_ file. For more information about the monitoring metrics, see server function [getClusterPerf](../Functions/g/getClusterPerf.html) . **Examples** gatherClusterPerf(30, 3, "/tmp") // check the result in /tmp/statis.csv after 30 seconds ### 3.10. gatherStreamingStat **Syntax** gatherStreamingStat(subNode, monitoringPeriod=60, scrapeInterval=15, dir="/tmp") **Arguments** * subNode: a string indicating the alias of a subscriber node. * monitoringPeriod: an integer indicating the timeframe (in seconds) of the monitoring. The default value is 60. * scrapeInterval: the interval (in seconds) at which the monitoring metrics are scraped. The default value is 15. * dir: a string indicating an existing directory to save the monitoring result. The default value is "/tmp". **Note**: For a Windows system, use absolute-path and forward slashes ("**/**") or double backslashes (\\) to separate the directories. **Details** Gets the status of workers on the a subscriber node based on the specified monitoring period and scrape interval. Export the result to the specified directory in a _sub\_worker\_statis.csv_ file. For more information about the monitoring metrics, see server function [getStreamingStat](../Functions/g/getStreamingStat.html) . **Examples** gatherStreamingStat("subNode",30, 3, "/tmp") // check the result in /tmp/sub_worker_statis.csv after 30 seconds ### 3.11. getDifferentData **Syntax** getDifferentData(t1, t2) **Arguments** * t1 / t2: a handle to an in-memory table. **Details** Checks if the values of _t1_ and _t2_ are identical by calling server function [eqObj](../Functions/e/eqObj.html) . _t1_ and _t2_ must be of the same size. **Return** Returns the rows that are different in the two specified tables; otherwise, print "Both tables are identical". **Examples** t1=table(1 2 3 as id, 4 5 6 as val) t2=table(1 8 9 as id, 4 8 9 as val) t3=table(1 2 3 as id, 4 5 6 as val) for (row in getDifferentData(t1, t2)) print row #output id val -- --- 2 5 3 6 id val -- --- 8 8 9 9 getDifferentData(t1, t3) #output Both tables are identical ### 3.12. checkChunkReplicas **Syntax** checkChunkReplicas(dbName, tableName, targetChunkId) **Arguments** * dbName: a string indicating the path to a distributed database, e.g., "dfs://demodb". * tableName: a string indicating the name of a DFS table. * targetChunkId: a string indicating a chunk ID which you can get with server function [getTabletsMeta](../Functions/g/getTabletsMeta.html) . **Details** Checks if the two replicas of the specified chunk are identical. This function is available only when the configuration parameter _dfsReplicationFactor_ is set to 2 on the controller. **Return** A Boolean indicating whether the data of two chunk replicas are identical. **Examples** n=1000000 ID=rand(10, n) x=rand(1.0, n) t=table(ID, x) db=database("dfs://rangedb", RANGE, 0 5 10) pt=db.createPartitionedTable(t, `pt, `ID) pt.append!(t) checkChunkReplicas("dfs://rangedb", "pt", "af8268f0-151e-c18b-a84c-a77560b721e6") #output true Stop a data node with the `kill -9 PID` command: pt.append!(t) checkChunkReplicas("dfs://rangedb", "pt", "af8268f0-151e-c18b-a84c-a77560b721e6")// get the chunk ID with getTabletsMeta() #output checkChunkReplicas: throw "colFiles on two replicas are not same" => colFiles on two replicas are not same Reboot the data node. When the chunk recovery is complete, execute `checkChunkReplicas()` again: checkChunkReplicas("dfs://rangedb", "pt", "af8268f0-151e-c18b-a84c-a77560b721e6") // chunk ID can be checked with getTabletsMeta() #output true --- # create [Jump to main content](#wh_topic_body) create ====== The create statement is used to create a database or a table. The syntax is as follows: create DFS databases -------------------- The create statement only supports creating DFS databases. create database directory partitioned by partitionType(partitionScheme), [engine='OLAP'], [atomic='TRANS'], [chunkGranularity='TABLE'] Please refer to the related function [database](../../Functions/d/database.html) for details. The number of _partitionType_ indicates the partition levels. You can specify one to three _partitionType_ for a database and use more than one _partitionType_ to create a composite database. create tables ------------- This statement only supports creating regular in-memory tables and DFS tables. create table dbPath.tableName ( schema[columnDescription] ) [partitioned by partitionColumns], [sortColumns|primaryKey], [keepDuplicates=ALL], [sortKeyMappingFunction] [softDelete=false] [comment] **dbPath** is a string indicating the path of a dfs database. **tableName** is a string indicating the table name, or a vector indicating the table object. **schema** indicates the table schema, including two columns: _columnName_ and _columnType_. **columnDescription** uses keywords to add a description to a column, including: * comment adds a comment to a column; * compress specifies the compression method, which includes: "lz4", "delta", "zstd", "chimp". * Since version 3.00.1, indexes can be set for tables of TSDB (_keepDuplicates_\=ALL) or PKEY databases. For detailed instructions, refer to the [createPartitionedTable](../../Functions/c/createPartitionedTable.html) function. **partitionColumns** specifies the partitioning column(s). For a composite partition, this parameter is a STRING vector. It can be a column name, or a function call applied to a column (e.g. `partitionFunc(id)`). If a function call is specified, it must have exactly one column argument, and the other arguments must be constant scalars. In this case, the system will partition the table based on the result of the function call. Parameters for TSDB storage engine only: **sortColumns** is a required argument for TSDB engine. It is a string scalar/vector indicating the columns based on which the table is sorted. **keepDuplicates** specifies how to deal with records with duplicate sortColumns values. The default value is ALL. It can have the following values: * ALL: keep all records * LAST: only keep the last record * FIRST: only keep the first record **sortKeyMappingFunction** is a vector of unary functions. It has the same length as the number of sort keys. The specified mapping functions are applied to each sort key (i.e., the sort columns except for the temporal column) for dimensionality reduction.After the dimensionality reduction for the sort keys, records with a new sort key entry will be sorted based on the last column of _sortColumns_ (the temporal column). **Note**: * It cannot be specified when creating a dimension table. * Dimensionality reduction is performed when writing to disk, so specifying this parameter may affect write performance. * The functions specified in `sortKeyMappingFunction` correspond to each and every sort key. If a sort key does not require dimensionality reduction, leave the corresponding element empty in the vector. * If a mapping function is `hashBucket` AND the sort key to which it applies is a HASH partitioning column, make sure the number of Hash partitions is not divisible by `hashBucket().buckets` (or vice versa), otherwise the column values from the same HASH partition would be mapped to the same hash bucket after dimensionality reduction. **softDelete** determines whether to enable soft delete for TSDB databases. The default value is false. To use it, _keepDuplicates_ must be set to 'LAST'. It is recommended to enable soft delete for databases where the row count is large and delete operations are infrequent. **comment** is a STRING scalar for setting a comment for a DFS table. Parameters for PKEY storage engine only: **primaryKey** is a STRING scalar/vector that specifies the primary key column(s), uniquely identifying each record in a DFS table of the PKEY database. For records with the same primary key, only the latest one is retained. Note that: * _primaryKey_ must include all partitioning columns. * The primary key columns must be of Logical, Integral (excluding COMPRESSED), Temporal, STRING, SYMBOL, or DECIMAL type. * With more than one primary key column, a composite primary key is maintained. The composite primary key uses a Bloomfilter index by default (see the _indexes_ parameter for details). Please refer to the related function [createPartitionedTable](../../Functions/c/createPartitionedTable.html) / [createDimensionTable](../../Functions/c/createDimensionTable.html) for details. Create temporary in-memory tables --------------------------------- To create a temporary in-memory table, add keywords `local temporary` (case insensitive) to `create`: create local temporary table tableName( schema ) [on commit preserve rows] where, **tableName** is a string indicating the table name, or a variable of the table object **schema** is the table schema which contains 2 columns: columnName and columnType. **on commit preserve rows** (optional) specifies that the temporary table is session-specific. It is case-insensitive. Note: * In DolphinDB, the `create local temporary table` statement is equivalent to `create table` as it creates a local temporary in-memory table that is only valid in the current session. * Currently, global temporary tables and the keyword `on commit delete rows` are not supported. Examples -------- ### Creating an In-memory Table create table tb( id SYMBOL, val DOUBLE ) go; //Parse and run codes with the go statement first, otherwise an error of unrecognized variable tb will be reported. tb.schema() /* output partitionColumnIndex->-1 chunkPath-> colDefs-> name typeString typeInt comment ---- ---------- ------- ------- id SYMBOL 17 val DOUBLE 16 */ ### Creating an OLAP Database if(existsDatabase("dfs://test")) dropDatabase("dfs://test") create database "dfs://test" partitioned by VALUE(1..10), HASH([SYMBOL, 40]), engine='OLAP' **Creating a partitioned table** create table "dfs://test"."pt"( id INT, deviceId SYMBOL, date DATE[comment="time_col", compress="delta"], value DOUBLE, isFin BOOL ) partitioned by ID, deviceID, pt = loadTable("dfs://test","pt") pt.schema() /* output partitionSchema->([1,2,3,4,5,6,7,8,9,10],40) partitionSites-> partitionColumnType->[4,17] partitionTypeName->[VALUE,HASH] chunkGranularity->TABLE chunkPath-> partitionColumnIndex->[0,1] colDefs-> name typeString typeInt comment -------- ---------- ------- -------- id INT 4 deviceId SYMBOL 17 date DATE 6 time_col value DOUBLE 16 isFin BOOL 1 partitionType->[1,5] partitionColumnName->[id,deviceId] */ **Creating a dimension table** create table "dfs://test"."pt1"( id INT, deviceId SYMBOL, date DATE[comment="time_col", compress="delta"], value DOUBLE, isFin BOOL ) pt1 = loadTable("dfs://test","pt1") pt1.schema() /* output chunkPath-> partitionColumnIndex->-1 colDefs-> name typeString typeInt comment -------- ---------- ------- -------- id INT 4 deviceId SYMBOL 17 date DATE 6 time_col value DOUBLE 16 isFin BOOL 1 */ ### Creating a TSDB Database if(existsDatabase("dfs://test")) dropDatabase("dfs://test") create database "dfs://test" partitioned by VALUE(1..10), HASH([SYMBOL, 40]), engine='TSDB' **Creating a partitioned table** create table "dfs://test"."pt"( id INT, deviceId SYMBOL, date DATE[comment="time_col", compress="delta"], value DOUBLE, isFin BOOL ) partitioned by ID, deviceID, sortColumns=[`deviceId, `date], keepDuplicates=ALL pt = loadTable("dfs://test","pt") pt.schema() /* output engineType->TSDB keepDuplicates->ALL partitionColumnIndex->[0,1] colDefs-> name typeString typeInt extra comment -------- ---------- ------- ----- -------- id INT 4 deviceId SYMBOL 17 date DATE 6 time_col value DOUBLE 16 isFin BOOL 1 partitionType->[1,5] partitionColumnName->[id,deviceId] partitionSchema->([1,2,3,4,5,6,7,8,9,10],40) partitionSites-> partitionColumnType->[4,17] partitionTypeName->[VALUE,HASH] sortColumns->[deviceId,date] softDelete->false tableOwner->admin chunkGranularity->TABLE chunkPath-> */ **Creating a dimension table** create table "dfs://test"."pt1"( id INT, deviceId SYMBOL, date DATE[comment="time_col", compress="delta"], value DOUBLE, isFin BOOL ) sortColumns=[`deviceId, `date] pt1 = loadTable("dfs://test","pt1") pt1.schema() /* output sortColumns->[deviceId,date] softDelete->false tableOwner->admin engineType->TSDB keepDuplicates->ALL chunkGranularity->TABLE chunkPath-> partitionColumnIndex->-1 colDefs-> name typeString typeInt extra comment -------- ---------- ------- ----- -------- id INT 4 deviceId SYMBOL 17 date DATE 6 time_col value DOUBLE 16 isFin BOOL 1 */ ### Creating a PKEY Database if(existsDatabase("dfs://test")) dropDatabase("dfs://test") create database "dfs://test" partitioned by VALUE(1..10), engine="PKEY" **Creating a partitioned table** create table "dfs://test"."pt"( id INT, deviceId SYMBOL [indexes="bloomfilter"], date DATE [comment="time_col", compress="delta"], value DOUBLE, isFin BOOL ) partitioned by ID, primaryKey=`ID`deviceID **Creating a dimension table** create table "dfs://test"."dt"( id INT, deviceId SYMBOL [indexes="bloomfilter"], date DATE [comment="time_col", compress="delta"], value DOUBLE, isFin BOOL ) partitioned by ID, primaryKey=`ID`deviceID ### Creating a Temporary In-memory Table create local temporary table "tb" ( id SYMBOL, val DOUBLE ) on commit preserve rows tb.schema() partitionColumnIndex->-1 chunkPath-> colDefs-> name typeString typeInt extra comment ---- ---------- ------- ----- ------- id SYMBOL 17 val DOUBLE 16 ### Creating a Partitioned Table with User-Defined Rules For data with a column in the format `id_date_id` (e.g., ax1ve\_20240101\_e37f6), partition by date using a user-defined function: // Define a function to extract the date information def myPartitionFunc(str,a,b) { return temporalParse(substr(str, a, b),"yyyyMMdd") } // Create a database data = ["ax1ve_20240101_e37f6", "91f86_20240103_b781d", "475b4_20240101_6d9b2", "239xj_20240102_x983n","2940x_20240102_d9237"] tb = table(data as id_date, 1..5 as value, `a`b`c`d`e as sym) dbName = "dfs://testdb" if(existsDatabase(dbName)){ dropDatabase(dbName) } create database "dfs://testdb" partitioned by VALUE(2024.02.01..2024.02.02), engine='TSDB' create table "dfs://testdb"."pt"( date STRING, value INT, sym SYMBOL ) partitioned by myPartitionFunc(date, 6, 8) sortColumns="sym" // Use myPartitionFunc to process the data column pt = loadTable(dbName,"pt") pt.append!(tb) flushTSDBCache() select * from pt The queried data are read and returned by partition. The query result shows that table pt is partitioned by the date information extracted from the id\_date column. | id\_date | value | sym | | --- | --- | --- | | ax1ve\_20240101\_e37f6 | 1 | a | | 475b4\_20240101\_6d9b2 | 3 | c | | 239xj\_20240102\_x983n | 4 | d | | 2940x\_20240102\_d9237 | 5 | e | | 91f86\_20240103\_b781d | 2 | b | --- # Log [Jump to main content](#wh_topic_body) Log === You can view the log information for the current node via the log browser. ![](../images/web_log_0.png) --- # Historical Data Replay [Jump to main content](#wh_topic_body) Historical Data Replay ====================== In DolphinDB, we can import historical data into a stream table in chronological order as "real-time data" so that the same script can be used both for backtesting and real-time trading. Regarding streaming in DolphinDB please refer to [DolphinDB Streaming Tutorial](streaming_tutorial.html) . This article introduces functions `replay` and `replayDS` and then demonstrates the process of data replaying. 1\. Functions ------------- ### 1.1. `replay` replay(inputTables, outputTables, [dateColumn], [timeColumn], [replayRate], [absoluteRate=true], [parallelLevel=1]) Function `replay` injects data from specified tables or data sources into stream tables. * 'inputTables' is a table or a tuple. Each element of the tuple is an unpartitioned table or a data source generated by function `replayDS`. * 'outputTables' is a table or a tuple of tables, or a string or a string vector. The number of elements of outputTables must be the same as the number of elements of inputTables. If it is a vector, it is a list of the names of the shared stream tables where the replayed data of the corresponding tables of inputTables are saved. If it is a tuple, each element is a shared stream table where the replayed data of the corresponding table in inputTables are saved. The schema of each table in outputTables must be identical as the schema of the corresponding table in inputTables. * 'dateColumn' and 'timeColumn' are strings indicating the date column and time column in inputTables. If neither is specified, the first column of the table is chosen as 'dateColumn'. If there is a 'dateColumn', it must be one of the partitioning columns. If only 'timeColumn' is specified, it must be one of the partitioning columns. If information about date and time comes from the same column (e.g., DATETIME, TIMESTAMP), use the same column for both 'dateColumn' and 'timeColumn'. Data are replayed in batches determined by the smallest unit of time in 'timeColumn' or 'dateColumn' if 'timeColumn' is not specified. For examples, if the smallest unit of time in 'timeColumn' is second then all data in the same second are replayed in the same batch; if 'timeColumn' is not specified, then all data in the same day are replayed in the same batch. * 'replayRate' is a nonnegative integer indicating the number of rows to be replayed per second. If it is not specified, it means data are replayed at the maximum speed. * 'replayRate' is an integer. * 'absoluteRate' is a Boolean value. The default value is true. Regarding 'replayRate' and 'absoluteRate': (1) If 'replayRate' is a positive integer and absoluteRate=true, replay at the speed of 'replayRate' rows per second. (2) If 'replayRate' is a positive integer and absoluteRate=false, replay at 'replayRate' times the original speed of the data. For example, if the difference between the maximum and the minimum values of 'dateColumn' or 'timeColumn' is n seconds, then it takes n/replayRate seconds to finish the replay. (3) If 'replayRate' is unspecified or negative, replay at the maximum speed. * 'parallelLevel' is a positive integer. When the size of individual partitions in the data sources is too large relative to memory size, we need to use function `replayDS` to further divide individual partitions into smaller data sources. 'parallelLevel' indicates the number of threads loading data into memory from these smaller data sources simultaneously. The default value is 1. If 'inputTables' is a table or a tuple of tables, the effective 'parallelLevel' is always 1. ### 1.2. `replayDS` replayDS(sqlObj, [dateColumn], [timeColumn], [timeRepartitionSchema]) Function `replayDS` generates a group of data sources to be used as the inputs of function `replay`. It splits a SQL query into multiple subqueries based on 'timeRepartitionSchema' with 'timeColumn' within each 'dateColumn' partition. * 'sqlObj' is a table or metacode with SQL statements (such as , `date, `time, 08:00:00.000 + (1..10) * 3600000)\ replay(inputDS, outputTable, `date, `time, 1000, true, 2)\ \ ### 1.5. Replay multiple tables simultaneously using data sources\ \ To replay multiple tables simultaneously, assign a tuple of these table names to parameter 'inputTables' of function `replay` and specify the output tables. Each of the output tables corresponds to an input table and should have the same schema as the corresponding input table. All input tables should have identical 'dateColumn' and 'timeColumn'.\ \ ds1 = replayDS(, `date, `time, 08:00:00.000 + (1..10) * 3600000)\ ds3 = replayDS(, `date, `time, trs);\ \ (2) Define the output stream table 'outQuotes'.\ \ sch = select name,typeString as type from quotes.schema().colDefs\ share streamTable(100:0, sch.name, sch.type) as outQuotes\ \ (3) Define a dictionary for the ETF components weights and function `etfVal` to calculate ETF intrinsic value. For simplicity we use an ETF with only 6 component stocks.\ \ defg etfVal(weights,sym, price) {\ return wsum(price, weights[sym])\ }\ weights = dict(STRING, DOUBLE)\ weights[`AAPL] = 0.1\ weights[`IBM] = 0.1\ weights[`MSFT] = 0.1\ weights[`NTES] = 0.1\ weights[`AMZN] = 0.1\ weights[`GOOG] = 0.5\ \ (4) Define a streaming aggregator to subscribe to the output stream table 'outQuotes'. We specify a filtering condition for the subscription that only data with stock symbols of AAPL, IBM, MSFT, NTES, AMZN or GOOG are published to the aggregator. This significantly reduces unnecessary network overhead and data transfer.\ \ setStreamTableFilterColumn(outQuotes, `symbol)\ outputTable = table(1:0, `time`etf, [TIMESTAMP,DOUBLE])\ tradesCrossAggregator=createCrossSectionalAggregator("etfvalue", <[etfVal{weights}(symbol, ofr)]>, quotes, outputTable, `symbol, `perBatch)\ subscribeTable(tableName="outQuotes", actionName="tradesCrossAggregator", offset=-1, handler=append!{tradesCrossAggregator}, msgAsTable=true, filter=`AAPL`IBM`MSFT`NTES`AMZN`GOOG) \ \ (5) Start to replay data at the specified speed of 100,000 rows per second. The streaming aggregator conducts real-time calculation with the replayed data.\ \ submitJob("replay_quotes", "replay_quotes_stream", replay, [rds], [`outQuotes], `date, `time, 100000, true, 4)\ \ (6) Check ETF intrinsic values\ \ select top 15 * from outputTable\ \ | time | etf |\ | --- | --- |\ | 2019.06.04T16:40:18.476 | 14.749 |\ | 2019.06.04T16:40:19.476 | 14.749 |\ | 2019.06.04T16:40:20.477 | 14.749 |\ | 2019.06.04T16:40:21.477 | 22.059 |\ | 2019.06.04T16:40:22.477 | 22.059 |\ | 2019.06.04T16:40:23.477 | 34.049 |\ | 2019.06.04T16:40:24.477 | 34.049 |\ | 2019.06.04T16:40:25.477 | 284.214 |\ | 2019.06.04T16:40:26.477 | 284.214 |\ | 2019.06.04T16:40:27.477 | 285.68 |\ | 2019.06.04T16:40:28.477 | 285.68 |\ | 2019.06.04T16:40:29.478 | 285.51 |\ | 2019.06.04T16:40:30.478 | 285.51 |\ | 2019.06.04T16:40:31.478 | 285.51 |\ | 2019.06.04T16:40:32.478 | 285.51 |\ \ 4\. Performance testing\ -----------------------\ \ We tested data replaying in DolphinDB on a server with the following configuration:\ \ * Server: DELL PowerEdge R730xd\ * CPU: Intel Xeon(R) CPU E5-2650 v4(24cores, 48 threads, 2.20GHz)\ * RAM: 512 GB (32GB × 16, 2666 MHz)\ * Harddisk: 17T HDD (1.7T × 10, read speed 222 MB/s, write speed 210 MB/s)\ * Network: 10 Gigabit Ethernet\ \ DolphinDB script:\ \ sch = select name,typeString as type from quotes.schema().colDefs\ trs = cutPoints(09:30:00.000..16:00:00.001,60)\ rds = replayDS( inputDS = replayDS(sqlObj, `Date, `Time, 08:00:00.000 + (1..10) * 3600000) replay(inputDS, tickStream, `Date, `Time, 1000, true, 2) Additionally, batch processing can also be implemented by appending historical data in batches into streaming engines, which can bypass the publish-subscribe overhead. The following script writes tick trade for a day into a reactive state engine: tmp = select SecurityID, Date, Time, Price from loadTable("dfs://trade", "trade") where Date=2021.03.08 getStreamEngine("reactiveDemo").append!(tmp) Factor Expression Reuse ----------------------- In DolphinDB, factor expressions and function definitions describe semantics, while specific computations are handled by batch or stream computing engines. Streaming engines can reuse factor expressions in batch processing and return identical results, saving the efforts of code rewriting in production. This method streamlines validation in live applications through batch calculations, significantly reducing debugging costs. Example: Computing the daily ratio of buy volumes over total trade volumes Define a function `buyTradeRatio`. The `@state` decorator at line 1 is used to declare the function as a state function for optimization in stream processing, which can be omitted in batch processing. @state // Decorator for stream processing optimization def buyTradeRatio(buyNo, sellNo, tradeQty){ return cumsum(iif(buyNo>sellNo, tradeQty, 0))\\cumsum(tradeQty) } In batch processing, use SQL queries to leverage parallelism. factor = select SecurityID, Time, buyTradeRatio(BuyNo, SellNo, TradeQty) as Factor from loadTable("dfs://trade","trade") where Date=2020.12.31 and Time>=09:30:00.000 context by SecurityID csort Time                                                       In stream processing, specify the UDF `buyTradeRatio` (with an `@state` decorator) as the calculation metrics. The following script creates the demo reactive state engine grouping by SecurityID, with input message format matching the in-memory table tickStream: tickStream = table(1:0, `SecurityID`TradeTime`TradePrice`TradeQty`TradeAmount`BuyNo`SellNo, [SYMBOL,DATETIME,DOUBLE,INT,DOUBLE,LONG,LONG]) result = table(1:0, `SecurityID`TradeTime`Factor, [SYMBOL,DATETIME,DOUBLE]) factors = <[TradeTime, buyTradeRatio(BuyNo, SellNo, TradeQty)]> demoEngine = createReactiveStateEngine(name="demo", metrics=factors, dummyTable=tickStream, outputTable=result, keyColumn="SecurityID") --- # Performance Monitoring [Jump to main content](#wh_topic_body) Performance Monitoring ====================== Enable/Disable Performance Monitoring ------------------------------------- * If each node is started individually, the default setting for performance monitoring is False. To start performance monitoring, we need to set _perfMonitoring_\=true in the configuration file _dolphindb.cfg_. * If the cluster is started on the web interface, the default setting for performance monitoring is True. To stop performance monitoring on all the nodes started with a controller node, we need to set perfMonitoring=false in the configuration file dolphindb.cfg for the controller node. The job log file contains descriptive information of all the queries that have been executed for each node. The default folder for the job log file is the log folder. The default name of the job log file is _nodeAlias\_job.log_. We can set the path and name of the job log file with the parameter of _jobLog_ in the configuration file. Performance Monitoring Methods ------------------------------ DolphinDB provides the following 3 ways of performance monitoring: * With built-in functions. Performance monitoring must be enabled to run the following functions. Both of the following functions return various measures for performance monitoring: * [getPerf](../Functions/g/getPerf.html) : return performance monitoring measures for the local node. Can run on each node in a cluster. * [getClusterPerf](../Functions/g/getClusterPerf.html) : return performance monitoring measures for all the nodes in the cluster. Can only run on the controller. * [getJobStat](../Functions/g/getJobStat.html) : monitor the number of jobs and tasks that are are running or in the job queue. * On the web interface. The following are some of the performance monitoring metrics displayed on the web interface: * memUsed: memory used on the node * memAlloc: size of the current memory pool for DolphinDB on the node * medLast10QueryTime: the median execution time of the previous 10 finished queries * maxLast10QueryTime: the maximum execution time of the previous 10 finished queries * medLast100QueryTime: the median execution time of the previous 100 finished queries * maxLast100QueryTime: the maximum execution time of the previous 100 finished queries * maxRunningQueryTime: the maximum elapsed time of the queries that are currently running * Through API with third-party systems, such as Prometheus. We can take the following steps: 1. Download Prometheus following the instructions from: [https://prometheus.io/docs/prometheus/latest/getting\_started/](https://prometheus.io/docs/prometheus/latest/getting_started/) 2. Add the data nodes that we would like to monitor in the configuration file prometheus.yml (in Linux): static_configs: - targets: ['192.168.1.27:8503','192.168.1.27:8504'] 3. Start Prometheus (in Linux): ./prometheus -config.file=prometheus.yml --- # Single Sign-On [Jump to main content](#wh_topic_body) Single Sign-On ============== OAuth is an authorization protocol that supports SSO implementations with various grant types. DolphinDB supports three of them: authentication code grant (for Web interface), implicit grant (for Web interface), and client credentials grant (for APIs). To enable OAuth SSO, users need to obtain the permissions for both the third-party website and the DolphinDB server. After successful login, they can use the third-party services on the web interface. This section briefly introduces the procedure for single sign-on to GitHub. Step 1: Enable OAuth on the Third-Party Website ----------------------------------------------- Create an OAuh app on GitHub. For specific steps, refer to [Creating an OAuth app](https://docs.github.com/en/apps/oauth-apps/building-oauth-apps/creating-an-oauth-app) . ![](images/1.png) Figure 1. Figure 1 Configuring SSO on a Third-Party Website Once the OAuth app is successfully created, you will see the interface below, in which the "Client ID" and "Client secrets" (click **Generate a new client secret**) are displayed. ![](images/2.png) Figure 2. Figure 2 Interface for Successful Creation Step 2: Configure OAuth Parameters in DolphinDB ----------------------------------------------- Specify the client ID and password in DolphinDB Web interface as the above-mentioned "Client ID" and "Client secrets". DolphinDB provides a series of configuration parameters for OAuth SSO. Users can configure these parameters in the configuration file of DolphinDB server or in the Config tab of the web interface. **Note**: To use OAuth in a cluster, users need to add the relevant configuration parameters in _controller.cfg_ and _cluster.cfg_ on all nodes. Here is an example of setting OAuth configuration parameters: oauth = 1 oauthWebType = authorization code oauthAuthuri = "https://github.com/login/oauth/authorize" oauthClientld = xxxxxx oauthClientSecret = xxxxxx oauthTokenUri = "https://github.com/login/oauth/access_token" oauthUserUri = "https://api.github.com/user" oauthUserField = login ![](images/3.png) Figure 3. Figure 3 Config Interface on DolphinDB Web Step 3: Log into a Third-Party Account on DolphinDB Web ------------------------------------------------------- After the configuration is complete, when logging in from the Web interface, the page will redirect to the third-party account authorization interface. Once authorized, the page will redirect back to the original login page. ![](images/4.png) Figure 4. Figure 4 Third-Party Account Authorization Interface If the username displayed on the web interface is the authorized third-party username, the OAuth login is successful. --- # Graceful Shutdown [Jump to main content](#wh_topic_body) Graceful Shutdown ================= Before version 1.30.17/20.00.5, DolphinDB did not support a graceful shutdown. If the DolphinDB process is terminated by `pkill -9` or `kill -9`, the following problems may occur when restarting the DolphinDB service: 1. Some completed transactions may not be flushed to disk. When the data node restarts, redo log needs to be replayed to flush these transactions to disk. If there are multiple redo logs that need to be replayed, it might take a long time to restart the data node. 2. If there is a problem in replaying the redo log on the data node, the data node will not start up. Starting from version 1.30.17/20.00.5, DolphinDB supports graceful shutdown, which can avoid the above problems by flushing transactions to disk before the system shuts down. This section introduces the mechanism of graceful shutdown under linux system and the process of correctly disconnect from the DolphinDB service. Graceful Shutdown ----------------- A SIGTERM signal sent by the data node through the `kill -15` (`kill -TERM`) command will be captured by DolphinDB and processed as follows: * The data node stops sending heartbeats to the controller so that the controller knows this node is down. At this point, newly initiated tasks in the cluster such as read, write or recovery tasks will not be allocated to this node. * If there are ongoing transactions on the data node, wait for them to complete and flush to disk. * Transactions in the cache engine and the redo log are garbage-collected by the data node. Graceful Shutdown in Standalone Mode ------------------------------------ * Front-end interactive mode Type "quit" in the console to exit * In back-end interactive mode, kill command is required: `kill -15 process ID` or `pkill -15 dolphindb` Graceful Shutdown in Cluster Mode --------------------------------- 1. Shut down a data node: The following options are available: * Use the stop button in the Web-based Cluster Manager * Type the [stopDataNode](../Functions/s/stopDataNode.html) command in the Web-based Cluster Manager * Use the [stopDataNode](../Functions/s/stopDataNode.html) command on the GUI or other API clients 2. Shut down the agent node and controller: The command to graceful shut down the agent node and controller is the same as for standalone mode. Alternatively, you can run the stopAll.bat script to directly kill all processes. --- # Backup and Restore [Jump to main content](#wh_topic_body) Backup and Restore ================== Backup ------ In DolphinDB, data is backed up by partition. You can use function backup to back up partitions, tables, or a database. DolphinDB offers 2 types of backup: 1. Back up by copying files When you specify the parameter _dbPath_ for the backup function, the system copies the files to the target directory _backupDir/dbName/tbName/chunkID_. The metadata files _\_metaData_ and _domain_ are generated under the directory _backupDir/dbName/tbName_. The following example backs up table dfs://compoDB/pt by copying files: backup("/home/DolphinDB/backup","dfs://compoDB",true); 2. Back up with SQL statements When you specify the parameter _sqlObj_ for the backup function, the system serializes the data to a binary file and saves it as _/.bin_ to the directory _backupDir/dbName/tbName_. The metadata files _\_metaData_ and _domain_ are generated under the same directory. The following example backs up table dfs://compoDB/pt with SQL statements: backup("/home/DolphinDB/backup",,true); Specify where-conditions in SQL metacode to only back up the partitions with date>2017.08.10: backup("/home/DolphinDB/backup",