Multi-Container Deployment With Docker Compose

Docker Compose is a tool for defining and running multi-container Docker applications. This tutorial introduces how to use Docker Compose to deploy DolphinDB clusters on multiple containers.

Prerequisites and Environment Setup

Install Docker

Install Docker on Linux/Mac OS/Windows.

Install Docker Compose

We recommend you to download the compiled binary of Docker Compose from the official release. See Install Docker Compose CLI for installation instructions.

Host Information
- single server:


Host	IP	Docker Service	Mount Point
cnserver9	xx.xx.xx.122	dolphindb	/ddbdocker

multiple servers:


Host	IP	Docker Service	Mount Point
host1	xx.xx.xx.81	dolphindb controller1&dolphindb agent1	/host1
host2	xx.xx.xx.82	dolphindb controller2&dolphindb agent2	/host2
host3	xx.xx.xx.83	dolphindb controller3&dolphindb agent3	/host3

Cluster Architecture

A DolphinDB cluster consists of 4 types of nodes: controller, agent,data node, and compute node.

Note:

Any data node or compute node can be used as a client for data access, while a controller is only used for cluster management and coordination.
The IP address of a node must be an intranet IP. If an external network address is used, the network communication performance among nodes may be unstable. Inter-container communication is usually enabled by connecting the containers to a bridge network and assigning an IP address to each container.

Quick Starts (Single-Server Deployment)

The following example shows how to deploy a DolphinDB cluster using image dolphindb/dolphindb:v2.00.5 with Docker Compose. The DolphinDB image is the community edition of v2.00.5. As the community license limits the number of data nodes in a cluster, the following example deploys a cluster with 2 containers (one is a controller and the other contains an agent and a data node).

git clone https://github.com/dolphindb/dolphindb-k8s
cd dolphindb-k8s/docker-compose/ddb_cluster_quick && docker-compose up -d

Expected output:

[+] Running 2/2
 ⠿ ddb_controller Pulled                                                                                                 4.5s
 ⠿ ddb_agent1 Pulled                                                                                                     4.4s
[+] Running 3/3
 ⠿ Network dev_ddb           Created                                                                                     0.1s
 ⠿ Container ddb_controller  Started                                                                                     0.5s
 ⠿ Container ddb_agent1      Started                                                                                     0.8s

Enter "IP:port" (where port is 8900) in your browser and the page shows:

Production Environment (Single-Server High-Availability Deployment)

Prerequisites

This chapter will deploy a high-availability cluster with multiple controllers and data nodes on multiple containers of a single server using image dolphindb/dolphindb:v2.00.5. You can download a high-availability cluster with three controllers and three data nodes using Docker Compose. Note that an enterprise license is required to deploy a high-availability cluster.

Procedures

git clone https://github.com/dolphindb/dolphindb-k8s

Then check the tree-structured directory:

cd dolphindb-k8s/docker-compose/ddb_high_cluster && tree ./

Expected output:

./
├── cfg
│   ├── agent1.cfg
│   ├── agent2.cfg
│   ├── agent3.cfg
│   ├── cluster.cfg
│   ├── cluster.nodes
│   ├── controller1.cfg
│   ├── controller2.cfg
│   └── controller3.cfg
├── cluster
│   ├── agent
│   │   ├── data
│   │   └── log
│   └── controller
│       ├── data
│       └── log
├── docker-compose.yml
└── dolphindb.lic

8 directories, 10 files

The files/folders are explained below:


File/Folder	Description	Directory of Host	Directory of Container
docker-compose.yml	defining services, networks, and volumes for a Docker application	None	None
dolphindb.lic	license file of DolphinDB	./dolphindb.lic	/data/ddb/server/dolphindb.lic
cfg	storing the configuration files of nodes	./cfg/agent1.cfg; ./cfg/agent2.cfg; ./cfg/agent3.cfg; ./cfg/cluster.cfg; ./cfg/cluster.nodes; ./cfg/controller1.cfg; ./cfg/controller2.cfg; ./cfg/controller3.cfg	/data/ddb/server/clusterDemo/config/agent1.cfg; /data/ddb/server/clusterDemo/config/agent2.cfg; /data/ddb/server/clusterDemo/config/agent3.cfg; /data/ddb/server/clusterDemo/config/cluster.cfg; /data/ddb/server/clusterDemo/config/cluster.nodes; /data/ddb/server/clusterDemo/config/controller1.cfg; /data/ddb/server/clusterDemo/config/controller2.cfg; /data/ddb/server/clusterDemo/config/controller3.cfg;
cluster	storing DolphinDB data and logs	./cluster/controller/data; ./cluster/controller/log; ./cluster/agent/data; ./cluster/agent/log	/data/ddb/server/clusterDemo/data; /data/ddb/server/clusterDemo/log

Start Docker Compose service:

docker-compose up -d

Expected output:

[+] Running 7/7
 ⠿ Network dev_ddb            Created                                                                                    0.2s
 ⠿ Container ddb_controller3  Started                                                                                    1.0s
 ⠿ Container ddb_controller1  Started                                                                                    1.0s
 ⠿ Container ddb_controller2  Started                                                                                    0.9s
 ⠿ Container ddb_agent3       Started                                                                                    1.9s
 ⠿ Container ddb_agent1       Started                                                                                    1.9s
 ⠿ Container ddb_agent2       Started                                                                                    1.8s

Enter "IP:port" (where port is 8901) in your browser and the page shows:

Click the "guest" → "Log in" button in the upper right corner, and enter username "admin" and password "123456" to start the container services. See DolphinDB High-availability Cluster Deployment and Docker Documentationfor detailed instructions.

Note: For the single-node high-availability cluster deployed in this chapter, IP addresses within the virtual network domain are assigned to nodes. The cluster cannot be scaled out to a multi-node high-availability cluster.

Production Environment (Multi-Server High-Availability Deployment)

Prerequisites

This chapter will deploy a high-availability cluster with multiple controllers and data nodes on multiple containers of multiple servers using image dolphindb/dolphindb:v2.00.5. You can download a high-availability cluster with three controllers and three data nodes with Docker Compose. Note that an enterprise license is required to deploy a high-availability cluster.

Procedures

Clone the project by executing the following commands on each node. Before running the commands, make sure the three hosts can communicate with each other and prepare an enterprise license for the high-availability cluster. After cloning the repository, enter the corresponding host subdirectory under dolphindb-k8s.

host1:

git clone https://github.com/dolphindb/dolphindb-k8s  \
&& cd dolphindb-k8s/docker-compose/ddb_high_cluster_multi_machine/host1

host2:

git clone https://github.com/dolphindb/dolphindb-k8s  \
&& cd dolphindb-k8s/docker-compose/ddb_high_cluster_multi_machine/host2

host3:

git clone https://github.com/dolphindb/dolphindb-k8s  \
&& cd dolphindb-k8s/docker-compose/ddb_high_cluster_multi_machine/host3

Then check the tree-structured directory:

tree ./

Expected output:

host1:

./
├── cfg
│   ├── agent1.cfg
│   ├── cluster.cfg
│   ├── cluster.nodes
│   └── controller1.cfg
├── cluster
│   ├── agent
│   │   ├── data
│   │   └── log
│   └── controller
│       ├── data
│       └── log
├── docker-compose.yml
└── dolphindb.lic

8 directories, 6 files

host2:

./
├── cfg
│   ├── agent2.cfg
│   ├── cluster.cfg
│   ├── cluster.nodes
│   └── controller2.cfg
├── cluster
│   ├── agent
│   │   ├── data
│   │   └── log
│   └── controller
│       ├── data
│       └── log
├── docker-compose.yml
└── dolphindb.lic

8 directories, 6 files

host3:

./
├── cfg
│   ├── agent3.cfg
│   ├── cluster.cfg
│   ├── cluster.nodes
│   └── controller3.cfg
├── cluster
│   ├── agent
│   │   ├── data
│   │   └── log
│   └── controller
│       ├── data
│       └── log
├── docker-compose.yml
└── dolphindb.lic

8 directories, 6 files

The files/folders are explained below:


File/Folder	Description	Directory of Host	Directory of Container
docker-compose.yml	defining services, networks, and volumes for a Docker application	none	none
dolphindb.lic	license file of DolphinDB	./dolphindb.lic	/data/ddb/server/dolphindb.lic
cfg	storing the configuration files of nodes	./cfg/agent1.cfg; ./cfg/agent2.cfg; ./cfg/agent3.cfg; ./cfg/cluster.cfg; ./cfg/cluster.nodes; ./cfg/controller1.cfg; ./cfg/controller2.cfg; ./cfg/controller3.cfg	/data/ddb/server/clusterDemo/config/agent1.cfg; /data/ddb/server/clusterDemo/config/agent2.cfg; /data/ddb/server/clusterDemo/config/agent3.cfg; /data/ddb/server/clusterDemo/config/cluster.cfg; /data/ddb/server/clusterDemo/config/cluster.nodes; /data/ddb/server/clusterDemo/config/controller1.cfg; /data/ddb/server/clusterDemo/config/controller2.cfg; /data/ddb/server/clusterDemo/config/controller3.cfg;
cluster	storing DolphinDB data and logs	./cluster/controller/data; ./cluster/controller/log; ./cluster/agent/data; ./cluster/agent/log	/data/ddb/server/clusterDemo/data; /data/ddb/server/clusterDemo/log

The node and container information is shown below:


Container Name (Unique)	Node Type	Host IP	Node Name, Type, Port
ddb_controller1	controller	xx.xx.xx.81	controller1,controller,8901
ddb_controller2	controller	xx.xx.xx.82	controller2,controller,8902
ddb_controller3	controller	xx.xx.xx.83	controller3,controller,8903
ddb_agent1	agent & data node	xx.xx.xx.81	agent1,agent,8904 P1-node1,datanode,8905
ddb_agent2	agent & data node	xx.xx.xx.82	agent2,agent,8906 P2-node1,datanode,8907
ddb_agent3	agent & data node	xx.xx.xx.83	agent3,agent,8908 P3-node1,datanode,8909

Note:

The file mapping process overwrites the containers with the files on the host machine. It's necessary to confirm whether to keep the data, logs and configuration files when creating a container.
For IP address configuration on each server, see DolphinDB Multi-Machine Cluster Deployment. Before starting the services, replace the sample IP addresses 10.0.0.81, 10.0.0.82, and 10.0.0.83 with the actual intranet IP addresses of the hosts. Update cfg/cluster.nodes, localSite in each cfg/controller*.cfg, and localSite, controllerSite, and sites in each cfg/agent*.cfg. The controller information in controllerSite and sites must match localSite in the corresponding controller*.cfg, and cluster.nodes must be consistent across the three hosts. Also make sure host-to-host, container-to-container, and host-to-container network communication is available.

Start Docker Compose service by executing the following command on each server (where the docker-compose.yml resides):

docker-compose up -d

Expected output:

host1:

[+] Running 3/3
 ⠿ Network dev_ddb            Created                                                                                0.1s
 ⠿ Container ddb_controller1  Started                                                                                1.7s
 ⠿ Container ddb_agent1       Started                                                                                3.3s

host2:

[+] Running 3/3
 ⠿ Network dev_ddb            Created                                                                                0.1s
 ⠿ Container ddb_controller2  Started                                                                                1.4s
 ⠿ Container ddb_agent2       Started                                                                                3.2s

host3:

[+] Running 3/3
 ⠿ Network dev_ddb            Created                                                                                0.1s
 ⠿ Container ddb_controller3  Started                                                                                1.7s
 ⠿ Container ddb_agent3       Started                                                                                3.4s

Enter "IP:port" (where port is 8901) in your browser and the page shows:

FAQ

How to upgrade DolphinDB version?

For example, to upgrade the DolphinDB image to version dolphindb/dolphindb:v2.00.6:

To upgrade all containers, go to the .env file under the same directory as docker-compose.yml and modify the environment variables:

IMAGE=dolphindb/dolphindb:v2.00.6

Execute the following command and restart the services:

docker-compose down && docker-compose up -d

Expected output:

[+] Running 7/7
 ⠿ Container ddb_agent3       Removed                                                                                    1.6s
 ⠿ Container ddb_agent1       Removed                                                                                    1.6s
 ⠿ Container ddb_agent2       Removed                                                                                    1.3s
 ⠿ Container ddb_controller1  Removed                                                                                    2.7s
 ⠿ Container ddb_controller2  Removed                                                                                    2.6s
 ⠿ Container ddb_controller3  Removed                                                                                    2.6s
 ⠿ Network dev_ddb            Removed                                                                                    0.1s

Errors and Solutions

If the following error message is reported:

but no declaration was found in the volumes section.

It indicates that the data volume is not declared or that a relative path is not used to map the data volume.

Solution: Configure the data volume in the yaml file and use a relative path for file mapping.