Deployment on a New Server

Performance improvement of disk I/O is crucial for big data processing. Recommended hardware configurations can be found in section 1.1.3.

After physically installing a new hard drive or adding one via a cloud service provider, you can connect to the CentOS server via SSH. To add a new hard drive with a capacity of 50G via a virtual machine, follow these steps:

(1) Check the disk information by fdisk -l:

# fdisk -l
...

Disk /dev/sdb: 53.7 GB, 53687091200 bytes, 104857600 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size(logical/physical): 512 bytes/ 512 bytes
I/O size(minimum/optimal): 512 bytes / 512 bytes

...

It can be seen that /dev/sdb is the newly added 50G disk.

(2) Use fdisk to create and modify partitions:

# fdisk /dev/sdb
Welcome to fdisk (util-linux 2.23.2).

Changes will remain in memory only, until you decide to write them.
Be careful before using the write command.

Device does not contain a recognized partition table
Created a new DOS disklabel with disk identifier 0x7becd49e

Command(m for help): n  # create a new partition
Partition type:
   p   primary (0 primary, 0 extended, 4 free)
   e   extended
Select (default p): p
Partition number (1-4, default 1): 
First sector (2048-104857599, default 2048):
the default value 2048 is used
Last sector, +sectors or +size{K,M,G} (2048-104857599, default 104857599):
the default value 104857599 is used
Created a new partition 1 of type `Linux` and of size 50GiB

Command(m for help): q

(3) Use command mkfs.xfs to create an XFS file system:

# mkfs.xfs /dev/sdb
meta-data=/dev/sdb               isize=512    agcount=4, agsize=3276800 blks
         =                       sectsz=512   attr=2, projid32bit=1
         =                       crc=1        finobt=0, sparse=0
data     =                       bsize=4096   blocks=13107200, imaxpct=25
         =                       sunit=0      swidth=0 blks
naming   =version 2              bsize=4096   ascii-ci=0 ftype=1
log      =internal log           bsize=4096   blocks=6400, version=2
         =                       sectsz=512   sunit=0 blks, lazy-count=1
realtime =none                   extsz=4096   blocks=0, rtextents=0

(4) Create a mount point and mount the drive:

# mkdir -p /mnt/dev1
# mount /dev/sdb /mnt/dev1

(5) Check the file system type of the new disk:

# df -Th
Filesystem                Type      Size  Used  Avail Use% Mounted on
devtmpfs                devtmpfs  3.9G     0  3.9G    0% /dev
tmpfs                   tmpfs     3.9G     0  3.9G    0% /dev/shm
tmpfs                   tmpfs     3.9G  8.6M  3.9G    1% /run
tmpfs                   tmpfs     3.9G     0  3.9G    0% /sys/fs/cgroup
/dev/mapper/centos-root xfs        50G  1.6G   49G    4% /
/dev/sda1               xfs      1014M  150M  865M   15% /boot
/dev/mapper/centos-home xfs        41G   33M   41G    1% /home
tmpfs                   tmpfs     783M     0  783M    0% /run/user/0
/dev/sdb                xfs        50G   33M   50G    1% /mnt/dev1

(6) Set auto-mount at startup:

# blkid /dev/sdb
/dev/sdb: UUID="29ecb452-6454-4288-bda9-23cebcf9c755" TYPE="xfs"
# vi /etc/fstab
UUID=29ecb452-6454-4288-bda9-23cebcf9c755 /mnt/dev1             xfs     defaults        0 0

Note that the UUID is used to identify the new disk, which avoids identification errors when the disk is replaced or its location is changed.

core dump consists of the recorded state of the working memory of a computer program when the program has crashed or otherwise terminated abnormally. Therefore, if there is enough space, it is recommended to enable core dump. Below are steps to enable core dump:

(1) Create a path to store core files:

# mkdir /var/crash # The folder can be modified.

Note:

• Set aside enough space for the folder to hold core files, where the maximum value is about maxMemSize;
• Ensure that DolphinDB has write access to the folder.

(2) Enable core dump.

# vi /etc/security/limits.conf
* soft core unlimited

It takes effect for all processes. You can also specify the process name to enable core dump only for DolphinDB. Unlimited indicates unlimited size for core files.

(3) Set the output path and format for core files.

# vi /etc/sysctl.conf
kernel.core_pattern = /var/crash/core-%e-%s-%u-%g-%p-%t

Format:

• %e - the executable filename

• %s - number of signal causing dump

• %u - numeric real UID of dumped process

• %p - PID of dumped process

• %g - numeric real GID of dumped process

• %t - time of dump

• %h - hostname

(4) Reboot and check if the changes take effect.

# ulimit -c
unlimited

We recommend starting DolphinDB in single-node mode to verify whether the core dump configuration takes effect.

$ cd /path_to_dolphindb
$ chmod +x ./startSingle.sh
$ ./startSingle.sh
$ kill -11 dolphindb_pid
$ ls /var/crash/ | grep dolphindb_pid # Check if there is a core file in the folder.

The number of files open simultaneously when running DolphinDB may exceed the default maximum number 1024 on CentOS 7. It is recommended to configure the maximum number of open files to 102400. You can follow steps below:

(1) Check the maximum number of open files:

# ulimit -n # on a process level
1024
# cat /proc/sys/fs/file-max # on a kernel level
763964

It can be seen that the maximum number of file descriptors open in a process is 1024, and 763964 in a Linux host.

(2) Set the maximum limit to 102400 for the processes.

# vi /etc/security/limits.conf
* soft nofile 102400
* hard nofile 102400

It takes effect for all processes. You can also specify the process name to enable core dump only for DolphinDB.

(3) If the configured value for the maximum number of open files in a host is less than 102400, you need to change it to a value no less than 102400:

# vi /etc/sysctl.conf
fs.file-max = 102400

The maximum limit of open files in the host here is 763964, which is greater than 102400, so it is not modified.

(4) Reboot and check if the changes take effect.

# ulimit -n # on a process level
102400
# cat /proc/sys/fs/file-max # on a kernel level
763964

If a multi-machine cluster is deployed, it is necessary to configure NTPD (Network Time Protocol Daemon) for time synchronization to ensure the time order of transactions. We configure NTPD server and client on two virtual machines.

Configure with the following steps:

(1) Install the NTP package.

# yum install ntp

(2) Set the time of VM 1 in synchronization with cn.pool.ntp.org

# vi /etc/ntp.conf
...
# Hosts on local network are less restricted.
restrict 192.168.0.0 mask 255.255.254.0 nomodify notrap

# Use public servers from the pool.ntp.org project.
# Please consider joining the pool (http://www.pool.ntp.org/join.html).
server 0.cn.pool.ntp.org iburst
server 1.cn.pool.ntp.org iburst
server 2.cn.pool.ntp.org iburst
server 3.cn.pool.ntp.org iburst
...

The 4th line configures the IP subnet to request the NTP service on the local network with the address 192.168.0.0 and subnet mask 255.255.254.0. The nomodify option prevents client to modify the configuration of the server. The notrap option uses the trap facility for remote event logging.

(3) Enable the firewall on VM 1 and add the NTP service.

# firewall-cmd --add-service=ntp --permanent
# firewall-cmd --reload

(4) Set the time of VM 2 in synchronization with VM 1.

# vi /etc/ntp.conf
...
# Use public servers from the pool.ntp.org project.
# Please consider joining the pool (http://www.pool.ntp.org/join.html).
server 192.168.0.30 iburst
# server 0.centos.pool.ntp.org iburst
# server 1.centos.pool.ntp.org iburst
# server 2.centos.pool.ntp.org iburst
# server 3.centos.pool.ntp.org iburst
...

(5) Start NTPD on VM 1 and VM 2.

# systemctl enable ntpd
# systemctl restart ntpd

(6) Wait for a few seconds, and then check the status of NTPD.

VM 1:

# ntpstat
synchronised to NTP server (202.112.31.197) at stratum 2
   time correct to within 41 ms
   polling server every 64 s

VM 2:

# ntpstat
synchronised to NTP server (192.168.0.30) at stratum 3
   time correct to within 243 ms
   polling server every 64 s

Before running DolphinDB, you need to configure the firewall to open the ports of each node, or turn off the firewall.

Take 8900 ~ 8903 TCP ports for example:

# firewall-cmd --add-port=8900-8903/tcp --permanent
# firewall-cmd --reload
# systemctl restart firewalld

Turn off the firewall using the following commands if it is not required:

# systemctl stop firewalld
# systemctl disable firewalld.service

Swap space, also known as virtual memory, is used as an extension of RAM. If the system needs more memory resources and the RAM is full, inactive pages in memory are moved to the swap space. Swap space allows a computer to effectively utilize more memory than is physically available, but it can also cause performance degradation if it is used excessively. If system performance prioritize to memory, it is recommended to disable Swap with the following steps:

(1) Check available swap space.

# free -h
              total        used        free      shared  buff/cache   available
Mem:           7.6G        378M        7.1G        8.5M        136M        7.1G
Swap:          7.9G          0B        7.9G

It can be seen that the total swap memory available is 7.9G.

(2) Disable swap space temporarily.

# swapoff -a

(3) Check available swap space again.

# free -h
              total        used        free      shared  buff/cache   available
Mem:           7.6G        378M        7.1G        8.5M        136M        7.1G
Swap:            0B          0B          0B

The total swap memory available is 0B. The swap space has been disabled.

(4) To permanently disable swap space, comment out the line with the value of swap in the second column:

# vi /etc/fstab
...
# /dev/mapper/centos-swap swap                    swap    defaults        0 0
...

This section describes major differences between the different deployment options.

Docker is only a lightweight tool for resource isolation, and there is no significant performance difference between running DolphinDB in a Docker and a non-Docker environment. Solution of deployment depends on business needs.

In this section, we introduce how to configure each node of a high availability cluster for metadata and streaming data. Assuming all machines are configured with the same hardware, relative information is given in the table below:

The configuration files clusters.nodes and cluster.cfg are identical across the 3 machines, while controller.cfg and agent.cfg vary by server.

Note that only some important configurations are listed below.

• cluster.nodes

• cluster.cfg

diskIOConcurrencyLevel=0
node1.volumes=/ssd1/dolphindb/volumes/node1,/ssd2/dolphindb/volumes/node1 
node1.redoLogDir=/ssd1/dolphindb/redoLog/node1 
node1.chunkMetaDir=/ssd1/dolphindb/metaDir/chunkMeta/node1 
node1.TSDBRedoLogDir=/ssd1/dolphindb/tsdb/node1/redoLog
chunkCacheEngineMemSize=2
TSDBCacheEngineSize=2
...

• controller.cfg

localSite=175.178.100.3:8990:controller1
dfsMetaDir=/ssd1/dolphindb/metaDir/dfsMeta/controller1
dfsMetaDir=/ssd1/dolphindb/metaDir/dfsMeta/controller1
dataSync=1
...

• agent.cfg

localSite=175.178.100.3:8960:agent1
sites=175.178.100.3:8960:agent1:agent,175.178.100.3:8990:controller1:controller,119.91.229.229:8990:controller2:controller,175.178.100.213:8990:controller3:controller
...

Refer to the following table for information on configuration parameters above:

For detailed instructions of deployment procedures, you can refer to the following tutorials.

• Standalone Deployment

• Single-Server Cluster Deployment

• Multi-Machine Cluster Deployment

• High-availability Cluster Deployment

• Scale out a DolphinDB Cluster

Take the high-availability cluster configured in section 2.3 for example, place the corresponding configuration files in DolphinDB installation folder/clusterDemo/config on machines centos-1, centos-2, and centos-3 accordingly.

Note that the startup script uses the relative path, so you need to execute the script under the directory where the script is located.