Standalone Mode
In standalone mode, the configuration file is dolphindb.cfg. Configuration parameters can be specified either in dolphindb.cfg, or in the command line that starts DolphinDB.
Thread Types
DolphinDB provides the following types of threads:
-
worker: threads that handle regular interactive tasks.
Since version 1.30.22/2.00.10, DolphinDB has introduced a multilevel task assignment algorithm which divides the jobs and tasks (broken down from jobs) into different levels. The jobs and tasks are handled by the workers at the same levels. The levels of workers range from 0 to 5.
A job submitted to the node starts at level 0 and is handled by a level 0 worker. Based on the number of partitions involved in the job, the level 0 worker breaks it into an equivalent number of tasks. Local tasks are executed concurrently by the level 0 or level 1 workers, while remote tasks are sent to remote nodes by remote executors. Tasks sent to remote nodes are marked as level 1 and handled by level 1 workers there. If a task is further divided into subtasks, the remote subtasks are marked one level lower than their parent and handled by workers at the same level on the remote nodes.
With different levels of workers handling different levels of jobs and tasks, deadlock issues caused by circular dependencies across nodes can be effectively prevented.
The number of workers determines the concurrent computing capability of a system.
-
remote executor: an independent thread that sends tasks to remote computing nodes. The remote executor is failure tolerant. If one machine fails, the remote executor will send the task to another machine with the data necessary for the task.
-
batch job worker: threads that handle batch jobs. Batch jobs are submitted with function submitJob and are usually time-consuming jobs. A batch job worker will be destroyed automatically after being idle for over 60 seconds.
-
dynamic worker: if there are jobs in the queue of regular interactive jobs and no jobs are completed in the previous second, the system will create dynamic workers to process the jobs in the queue. A dynamic worker will automatically be destroyed after being idle for 60 seconds. Dynamic workers make the system more responsive and can solve deadlocks among multiple nodes.
-
web worker: threads that handle HTTP requests. DolphinDB provides a web-based cluster management interface to interact with DolphinDB nodes.
-
urgent worker: threads that only handle time-sensitive system-level jobs such as logins (
login
) and job cancellations (cancelJob
/cancelConsoleJob
). The configuration parameter urgentWorkerNum specifies the maximum number of urgent workers. The default is 1.
Parameter | Description |
---|---|
WorkerNum | The size of worker pool for regular interactive jobs. The default value is the number of CPU cores. |
remoteExecutors=1 | The number of remote executors. The default is 1. It is recommended to set it to the number of cluster nodes - 1. |
maxBatchJobWorker=4 | The maximum number of batch job workers. The default value is the value of workerNum. |
maxDynamicWorker=4 | The maximum number of dynamic workers. The default value is the value of workerNum. |
WebWorkerNum=1 | The size of the web worker pool to process HTTP requests. The default value is 1. |
urgentWorkerNum=1 | The maximum number of urgent workers. The default value is 1. |
maxCachedBatchJobNum=2048 | The maximum number of batch jobs in the batch job queue. The default value is 2048. |
jobParallelismHardLimit=false | A Boolean value indicating whether to set the limit on the maximum number of workers a task can use. The default value is false. If it is set to true, the maximum number of workers a task can use cannot exceed the specified parallelism. |
defaultJobParallelism=2 | A positive integer in range (0,64] indicating the default parallelism of the background task submitted by users. The default value is 2. Note that if the value of this parameter falls outside the specified range, the server fails to start. |
enableMultiThreadMerge | A Boolean value indicating whether the results of subqueries assigned to each node can be merged using multiple threads. The default value is false. When set to true, for multi-threaded merging to be effectively performed, the subquery result must contain at least two columns and have more than 50 million entries. This configuration parameter can provide a significant performance boost for queries involving a large number of partitions. |
Since version 2.00.11, a new configuration parameter compactWorkerNumPerVolume is added to tune the size of worker pool handling level file compaction, which provides control over resource usage, compaction speeds, and load balancing across volumes.
Parameter | Description |
---|---|
compactWorkerNumPerVolume | The size of worker pool for the level file compaction on each volume. The default value is 1. |
Memory Management
Parameter | Description |
---|---|
OLAPCacheEngineSize=0 Alias:chunkCacheEngineMemSize | The capacity of cache engine in units of GB. After cache engine is enabled, data is not written to disk until data in cache exceeds 30% of chunkCacheEngineMemSize. The default value is 0 indicating the cache engine is not enabled. To enable the cache engine, we must set chunkCacheEngineMemSize > 0 and dataSync = 1. |
maxMemSize=0 | The maximum memory (in units of GB) allocated to DolphinDB. If set to 0, it means no limits on memory usage. |
memoryReleaseRate=5 | The rate at which unused memory is released to the operating system. It is a floating point number between 0 and 10. The default value is 5. memoryReleaseRate = 0 means that unused memory will not be released actively; memoryReleaseRate = 10 means that memory will be released at the fastest speed. |
warningMemSize | When memory usage exceeds warningMemSize (in units of GB), the system will automatically clean up the cache of some databases to avoid OOM exceptions. The default value is 75% of MaxMemSize. |
reservedMemSize | A positive number specified in units of GB. When the available memory in DolphinDB is less than reservedMemSize, DolphinDB only allocates a memory block of limited size (as specifiedby maxBlockSizeForReservedMemory) for each memory request. reservedMemSize is provided to restrict the memory allocation for each memory request to improve the likelihood that there is enough memory for critical operations that use a small amount of memory (error reporting, rollbacks, etc.). For example, when data writes fail due to insufficient memory, the transactions can be rolled back to guarantee data consistency. If the parameter is not specified, the system sets reservedMemSize = 5% * maxMemSize and reservedMemSize must be between 64MB and 1GB. |
maxBlockSizeForReservedMemory | The maximum size (in units of KB) of the memory block that DolphinDB allocates for each memory request when its available memory is less than reservedMemSize. The default value is 64.It is not recommended to set it too high as exceptions or crashes may occur if there isn't enough memory left for critical database operations. |
TSDBCacheEngineSize | A positive number indicating the capacity (in GB) of the TSDB cache engine. The default value is 1. The memory used by the cache engine may reach twice the set value as an extra memoryblock will be allocated to cache the incoming data while data in the original memory is being flushed to disk. If the flush process is not fast enough, the newly allocated memory may also reach TSDBCacheEngineSize and thus block the writer thread. If the parameter is set too small, data in the cache engine may be flushed to disk too frequently, thus affecting the write performance; If set too high, a large volume of cached data is not flushed to disk until it reaches TSDBCacheEngineSize (or after 10 minutes). If power failure or shutdownoccurs in such cases, numerous redo logs are to be replayed when the system is restarting, causing a slow startup. |
TSDBLevelFileIndexCacheSize | The upper limit of the size (in GB) of level file metadata of the TSDB engine. It is of floating type. The default value is 5% of the maximum memory for DolphinDB (set by maxMemSize),and the minimum value is 0.1 (GB). |
TSDBCacheEngineCompression | Specifies whether to compress the data in TSDB cacheEngine. The default value is true, which means to compress the data. |
allowTSDBLevel3Compaction | Whether to allow compaction of level 3 files of TSDB storage engine. The default value is false. When set to true, for tables with keepDuplicates=FIRST/LAST, compaction of level 3 files is enabled. |
In prior versions, decimal numbers are parsed as DOUBLE floating point numbers by default. As of version 2.00.10.4, the parseDecimalAsFloatingNumber configuration parameter can specify the system default behavior for handling decimal numbers during parsing.
Parameter | Description |
---|---|
parseDecimalAsFloatingNumber =true | Whether decimal numbers are parsed as DOUBLE or DECIMAL64 type. True (default): parse decimals to DOUBLE floating point numbers. False: parse decimals to DECIMAL64 fixed point numbers. |
regularArrayMemoryLimit=2048 | The limit on the memory size (in
units of MB) of a regular array. Its value must be a power of 2. The
default value is 2048 (MB). The actual available memory is
min(regularArrayMemoryLimit,
maxMemSize/2) . |
appendTupleAsAWhole | Specifies how a tuple is appended
(append! ) or joined (join! ) to
another tuple. True (default): append or join the tuple as an
embedded tuple element. False: append or join each element of the
tuple independently. |
The TSDB engine caches SYMBOL base during read/write operations. An LRU (Least Recently Used) policy is employed to manage this cache, with two configuration parameters determining when to evict SYMBOL base cache based on time or capacity limits. Based on the assumption that entries that haven't been accessed for a long time are less likely to be used in the near future, LRU maintains a record of the order in which entries are accessed, and when cache size or retention time reaches specified limits, it evicts the entry that hasn't been accessed for the longest period first. Only SYMBOL base entries absent from both the cache engine and ongoing transactions will be evicted based on LRU.
Parameter | Description |
---|---|
TSDBSymbolBaseEvictTime=3600 | A positive integer indicating the amount of time (in seconds) to keep cached SYMBOL base entries before they are evicted. The default value is 3600. |
TSDBCachedSymbolBaseCapacity | A positive number indicating the maximum capacity (in GB) for cached SYMBOL base. It defaults to 5% of maxMemSize, with a 128MB minimum. When exceeded, the system removes cached SYMBOL base entriesuntil the cache size meets the limit. |
Hard Disk
To mitigate I/O bottlenecks, a data node can be configured with multiple volumes to improve the speed of parallel read and write.
Log files include metadata log file, redo log, query log, system log and batch job log. They have different requirements for disk performance. Generally, metadata log and redo log should be stored on SSDs while other logs can be stored on HDDs.
Parameter | Description |
---|---|
batchJobDir | The folder for batch job logs and results. The default value is <HomeDir>/batchJobs. |
batchJobFileRetentionTime | A floating-point number that sets the retention time (in days) for the files storing outputs (<job>.msg) and return values (<job>.obj) for batch and scheduled jobs. The default value 0 means these files will not be automatically deleted. |
chunkMetaDir | The folder for the metadata of data chunks on each data node. The default value is <HomeDir>/storage/CHUNK_METADATA. In a cluster, each data node should be configured with different chunkMetaDir. |
dataSync=0 | If dataSync=1, DolphinDB generates redo log when the database is being updated. It can avoid data loss in the event of a system crash or power outage. The default value of dataSync is 0. |
jobLogFile=jobLog.csv | The path and name of the job log file that contains descriptive information of all the queries that have been executed for each node. It must be a csv file. The default folder for the job log file is the log folder. The default name of the job log file is nodeAlias_job.log. |
jobLogRetentionTime=0 |
A floating-point number indicating the retention time (in days) to keep achived query logs. A query log on a node (<nodeAlias>_job.log) is archived if its size exceeds 1 GB. The default value is 0, indicating that archived logs are not to be deleted. |
enableDFSQueryLog=false | Different from parameter jobLogFile that only records executed queries, enableDFSQueryLog records all completed and ongoing queries and subqueries on DFS tables. The default value is false. To set it to true, parameter perfMonitoring must be enabled. When set to true, a query log named nodeAlias_query.log will be generated in the directory where nodeAlias_job.log is located. The query log contains the following fields: node, userId, sessionId, rootId, type, level, time, database, table, and jobDesc. Subqueries on DFS tables return a query log with fields "database" and "table" left empty. |
queryLogRetentionTime=0 | When the size of the query lognodeAlias_query.log exceeds 1 GB, the system archives the log file and subsequent logs will be written to a new file. This parameter is a floating point number indicating the retention time (in days) to keep archived query logs. The default value is 0, indicating that archived logs are not to be deleted. |
logFile=DolphinDBlog | The path and name of the log file. It displays the server configuration specifications, warnings and error messages. |
logLevel=INFO | The log file only keeps log information equal to higher than the specified level. From the lowest to the highest level, the possible values are DEBUG, INFO, WARNING andERROR. The default value is INFO. |
maxLogSize=1024 | The system will archive the server log after the log reaches the specified size limit (in MB). The default value is 1024 and the minimum value is 100. The archived log filename adds a prefix to the original log file name. The prefix is in the format of \date\seq, e.g. 20181109000. seq has 3 digits and starts with 000 |
redoLogDir | The directory of the redo log. The default value is /log/redoLog. |
TSDBRedoLogDir | The directory of the redo log of TSDB engine. The default value is <HomeDir>/log/TSDBRedo. TSDBMeta (including level file metadata) is stored in <HomeDir>/log, and transaction data is stored in <HomeDir>/storage/CHUNKS. To configure the parameter: Use the relative paths, i.e., do not start with '/'; Include <ALIAS> in the path: /home/xxx/<ALIAS>/redolog; Configure for each node separately: node1.TSDBRedoLogDir=/home/xxx/node1/redolog, node2.TSDBRedoLogDir=/home/xxx/node2/redolog. |
redoLogPurgeInterval=30 | Clear the redo log of transactions whose data have been persisted at intervals specified by redoLogPurgeInterval in terms of seconds. The default value is 30. |
redoLogPurgeLimit=4000 | Clear the redo log of transactions whose data have been persisted if the size of the redo log exceeds redoLogPurgeLimit in terms of MB. The default value is 4000. |
volumes=/hdd/hdd1/volumes, /hdd/hdd2/volumes, /hdd/hdd3/volumes, /hdd/hdd4/volumes | The folder where data files are saved in the distributed file system on a data node. The default value is <HomeDir>/<nodeAlias>/storage. |
diskIOConcurrencyLevel=4 | The number of threads for reading and writing to disk. The default value is 1. diskIOConcurrencyLevel = 0 means the threads handling the relevant tasks will read and write to the disk. When diskIOConcurrencyLevel > 0, the system will create the specified number of threads to read and write to the disk concurrently. For HDDs, it is recommended to set diskIOConcurrencyLevel to the number of volumes configured on the node (specified by the configuration parameter volumes). For SSDs, it is recommended to set diskIOConcurrencyLevel = 0 |
marketHolidayDir | Specifies the directory to the file of market
holidays. Either absolute or relative directory can be specified.
The system searches the relative directory in the following order:
home directory of the node, the working directory of the node, and
the directory with the DolphinDB executable. The default directory
is <HomeDir> /marketHoliday. The file
must be a single-column CSV file with DATE data. Based on the CSV
files under marketHolidayDir, a trading calendar is created with the
file name as its identifier. Note:
|
enableDfsRecoverRedo | Enable recovery redo log. |
recoverLogDir | The directory of the recovery redo log. It must be stored on a separate disk (preferably SSD) from the data files. |
enableAuditLog | Whether to enable logging DDL operations to a separate log file. The default value is false. When set to true, an audit log will be generated in the directory <logFile>/<ALIAS>_audit.log. The maximum size of an audit log file is 128M, beyond which another file prefixed with a timestamp (in second) will be automatically generated for continuous storage of DDL operations. |
auditLogRetentionTime | A floating-point number indicating the amount of time (in days) to keep an audit log before deleting it. For example, 0.5 means 12 hours. The default value is 0, indicating that audit log files are not deleted. |
IMOLTP
Parameter | Description |
---|---|
enableIMOLTPEngine | A Boolean value that determines whether to enable the IMOLTP engine. The default value is false. |
enableIMOLTPRedo | A Boolean value that determines whether to enable redo log mechanism for IMOLTP databases. The default value is true. |
IMOLTPRedoFilePath | A string that specifies the path where the redo log files will be stored. If a relative path is provided, the redo log files will be located under the <homeDir>/IMOLTP/ directory. The default value is <homeDir>/IMOLTP/im_oltp.redo. Note that the path must exist and be unique. |
IMOLTPSyncOnTxnCommit | This parameter only takes effect when enableIMOLTPRedo = true. It is a Boolean value that determines whether a transaction should be committed after the data is safely persisted to redo log files on disk. The default value is false, which means the transaction is committed after redo log is written to system buffer instead of disk. In cases of power outages or operating system crashes, data cannot be restored if it is false. |
enableIMOLTPCheckpoint | A Boolean value that determines whether to enable checkpoint mechanism for IMOLTP databases. The default value is true. |
IMOLTPCheckpointFilePath | A string that specifies the path where the checkpoint files will be stored. If a relative path is provided, the checkpoint files will be located under the <homeDir>/IMOLTP/ directory. The default value is <homeDir>/IMOLTP/im_oltp.ckp. Note that the path must exist and be unique. |
IMOLTPCheckpointThreshold | A LONG integer that sets the threshold for the redo logs (in MiB) to trigger a checkpoint. The default value is 100. |
IMOLTPCheckpointInterval | A LONG integer that sets the interval (in seconds) to trigger a checkpoint. The default value is 60. |
PKEY
Parameter | Description |
---|---|
enablePKEYEngine | A Boolean value that determines whether to enable the PKEY engine. The default value is true. |
PKEYMetaLogDir | A string that specifies the path where the PKEY meta log files will be stored. The default value is <ALIAS>/log/PKEYMeta. |
PKEYRedoLogDir | A string that specifies the path where the PKEY redo log files will be stored. The default value is <ALIAS>/log/PKEYRedo. |
PKEYCacheEngineSize | A positive number indicating the capacity (in GB) of the PKEY cache engine. The default value is 1. The memory used by the cache engine may reach twice the set value as an extra memory block will be allocated to cache the incoming data while data in the original memory is being flushed to disk. If the flush process is not fast enough, the newly allocated memory may also reach PKEYCacheEngineSize and thus block the writer thread. If the parameter is set too small, data in the cache engine may be flushed to disk too frequently, thus affecting the write performance; If set too high, a large volume of cached data is not flushed to disk until it reaches PKEYCacheEngineSize (or after 10 minutes). If power failure or shutdown occurs in such cases, numerous redo logs are to be replayed when the system is restarting, causing a slow startup. |
PKEYBlockCacheSize | A positive number indicating the capacity (in GB) of the PKEY block cache. The default value is 1. |
PKEYDeleteBitmapUpdateThreshold | A positive number indicating the buffer size threshold (in MB) for updating the PKEY delete bitmap. The default value is 100. The process of updating the PKEY delete bitmap occurs asynchronously in batches as a background operation. During the flushing process, the system extracts and temporarily stores the primary key and CID of the data in a buffer. This buffer is used to trigger updates to the delete bitmap. When this buffer reaches the size specified by PKEYDeleteBitmapUpdateThreshold, thedelete bitmap is updated and the buffer is cleared. This parameter should be set appropriately. If set too small, the delete bitmap will be updated too frequently, consuming excessive disk bandwidth. This can negatively impact queries, cache engine flushing, and level file compaction. If set too large, the de-duplication process during queries becomes more resource-intensive, potentially increasing query time. The system recovery process after a reboot may be significantly slower. |
PKEYStashedPrimaryKeyBufferSize | A positive number indicating the buffer size (in MB) for the PKEY engine. The default value is 1024. Under excessive write load, frequent disk flushing and delayed buffer clearing due to slow delete bitmap updates will result in the buffer to grow continuously. The buffer size may reach twice the set value. This parameter serves as a safeguard against potential OOM issues that could arise from an indefinitely growing staging buffer during periods of high write activity. |
PKEYBackgroundWorkerPerVolume | The size of worker pool for the PKEY level file compaction and delete bitmap updates on each volume. The default and minimum value is 1. |
PKEYCacheFlushWorkerNumPerVolume | The size of worker pool for the PKEY cache engine flushing on each volume. The default and minimum value is 1. |
User Login and Access Control
Parameter | Description |
---|---|
thirdPartyAuthenticator | Specifies a function view (which takes the same arguments as login) for third-party authentication. Users can connect to a third-party system through DolphinDB plugins such as HttpClient. Once specified, DolphinDB server authenticates the user login through the third-party system. |
enhancedSecurityVerification=false | A Boolean value indicating whether to enable
enhanced security verification, which includes password complexity
verification and password retry limit. Set to false (default) to
disable this function and true to enable. When it is enabled:
|
thirdPartyCreateUserCallback | Specifies one or multiple function views (split with
comma) to execute when the createUser has been
used. The callbacks will be invoked in the order they are specified,
from left to right. Note: Exceptions during the callback will be
caught and logged, but only the last one will be thrown after all
callbacks have finished executing. |
thirdPartyDeleteUserCallback | Specifies one or multiple function views (split with
comma) to execute when the deleteUser has been
used. The callbacks will be invoked in the order they are specified,
from left to right. Note: Exceptions during the callback will be
caught and logged, but only the last one will be thrown after all
callbacks have finished executing. |
strictPermissionMode | A Boolean value specifying whether to enable strict
permission mode on the current node. The default value is false.
When set to true, operations such as disk read/write are only
allowed for administrators. Related functions include:
saveTextFile , saveAsNpy ,
backup , restore ,
restoreDB , restoreTable ,
backupDB , backupTable ,
migrate , file ,
files , writeObject ,
readObject , loadPlugin ,
close , fflush ,
mkdir , rmdir ,
rm , writeLog ,
run , runScript ,
test , saveTable ,
savePartition ,
saveDualPartition ,
saveDatabase , saveText ,
loadText , loadModule ,
saveModule . |
enableShellFunction | A Boolean value indicating whether an administrator
can call the shell function. False (default) means
the shell function cannot be called by any user.
True means the permission to use the shell function
is granted for the administrator. |
Network
Parameter | Description |
---|---|
enableHTTPS=false | Whether to enable HTTPS protocol. The default value is false. |
localSite =localhost:8848:local8848 | Host address, port number and alias of the local node, separated by ":". In standalone model, the default value is localhost:8848:local8848. |
maxConnections=64 | The maximum number of connections (from GUI, API, other nodes, etc) to the local node. |
maxConnectionPerSite=2 | The maximum number of connections from the local node to remote nodes. |
tcpNoDelay=1 | Whether to enable the TCP_NODELAY socket option. The default value is false. |
DolphinDB provides the configuration parameter tcpUserTimeout to set the maximum hold time for data that has not been acknowledged or not been sent due to TCP Zero Window. When the specified timeout is exceeded, the TCP connection will be actively closed.
Parameter | Description |
---|---|
tcpUserTimeout=300000 | Set the socket option TCP_USER_TIMEOUT. The default value is 300000 (in ms). |
License Server
From DolphinDB 1.30.21/2.00.9 onwards, license server is supported for resource allocation. License server is only available with license of type 3. You may contact our technical support for more information.
Parameter | Description |
---|---|
licenseServerSite | Specifies the LAN information of the node where the license server is located. It is specified in the format of <host:port:alias>, where port must be the same as that specified in the license file. The license server is a standalone server that assigns hardware resources to each node (except the agent) in a cluster. If licenseServerSite is configured with the same value as localSite, license server runs on the current node; Otherwise the current node requests resources from the license server. If licenseServerSite is not configured, the current node uses the local license file. |
bindCores | Specifies the CPU cores (by core ID) for the node.
To specify multiple CPU cores, you can separate the core IDs with a
comma (","); To specify consecutive cores, you can use a hyphen
("-") to join the starting and ending IDs. For example, to bind CPU
cores numbered 1 to 4, it can be 1, 2, 3, 4 or 1-4. This parameter
only takes effect on the node that requests resources from the
license server.
|
Streaming
The required configuration parameters for a publisher node are maxPubConnections and persistenceDir.
Parameter | Description |
---|---|
maxMsgNumPerBlock=1024 | The maximum number of records in a message block. The default value is 1024. |
maxPersistenceQueueDepth= 10000000 | The maximum depth (number of records) of a message queue to persist a stream table to disk. The default value is 10,000,000. |
maxPubQueueDepthPerSite= 10000000 | The maximum depth (number of records) of a message queue on the publisher node. The default value is 10,000,000. |
maxPubConnections=0 | The maximum number of subscriber nodes that the publisher node can connect to. The default value is 0. For the node to server as a publisher, we must set maxPubConnections>0. |
persistenceDir= /home/DolphinDB/Data/Persistence | The directory where shared streaming tables are persisted to. To enable persistence, persistenceDir must be specified. In a cluster, each data node should be configured with different persistenceDir. |
persistenceWorkerNum=1 | The number of workers responsible for persisting streaming tables to disk in asynchronous mode. The default value is 0. |
Parameter | Description |
---|---|
MaxSubConnections=64 | The maximum number of publishers that the subscriber node can connect to. The default value is 64. |
MaxSubQueueDepth=10000000 | The maximum depth (number of records) of a message queue on the subscriber node. |
persistOffsetDir= /home/DolphinDB/streamlog | The directory to save the offset of the last subscribed message that has been processed. If persistOffsetDir is not specified, it will be saved under the directory specified by persistenceDir. If persistenceDir is not specified, it will be saved in the streamLog folder under the home directory. |
subExecutorPooling=true | A Boolean value indicating whether streaming executors use pooling mode. The default value is false. If it is set to 0, it means the thread can conduct message parsing and can also process messages. |
subExecutors=2 | The number of message processing threads in the subscriber node. Only when subscription is enabled is this parameter relevant. The default value is 1. If it is set to 0, it means the thread can conductmessage parsing and can also process messages. |
subPort=8000 | The port number that the subscription thread is listening on. Before version 2.00.9, this parameter must be specified to enable the node(s) to serve as subscriber(s). Since version 2.00.9, this parameter can be unspecified. |
subThrottle=1000 | A non-negative integer in milliseconds, indicating
the interval at which the system checks whether throttle in function
subscribeTable has been reached. The default
value is 1000. If the interval specified by the parameter throttle
in subscribeTable is less than the configuration
parameter subThrottle, the parameter handler in
subscribeTable will be triggered to process
messages at an interval of subThrottle. To set
throttle is less than 1 second, you need to modify the
configuration parameter subThrottle first. For example, to
set throttle =0.001 (second), please set subThrottle
=1 first. Note: This parameter is only valid if the parameter
batchSize is specified in function
subscribeTable . |
localSubscriberNum=1 | Sets the number of threads used to distribute the messages from the publish queue if the publisher and subscriber are on the same node. The default value is 1. If a number greater than 1 is specified, the system will allocate the threads to distribute messages in parallel to the processing threads. |
System Management
Parameter | Description |
---|---|
config=dolphindb.cfg | The configuration file. The default value is dolphindb.cfg. It can only be specified in command line. |
console=1 | A Boolean value indicating whether to start a DolphinDB console. The default value is 0. It can only be specified in command line. |
home=/home/DolphinDB/server | The DolphinDB home directory where the configuration files, the license file, the log file and other related files are located. It can only be specified in command line. |
logFile=dolphindb.log | The log file. The default value is dolphindb.log. It can only be specified in command line. |
maxPartitionNumPerQuery=65536 | The maximum number of partitions that a single query can search. The default value is 65536. |
checkExistingPartitionNumForQuery | Whether to use the number of existing partitions obtained from the controller to check if the number of partitions for a single query exceeds the maximum (configured by maxPartitionNumPerQuery). False (default) means that the number of partitions set in partitioning scheme is used. |
mode=single | single: standalone mode; datanode: data node; controller: controller node; agent: agent node. |
moduleDir=modules | The directory for the module files. The default value is the relative directory modules that needs to be created by the user. The system searches the relative directory modules in the following order: home directory of the node, the working directory of the node, and the directory with the DolphinDB executable. |
newValuePartitionPolicy=skip | Specifies how new data outside of existing VALUE
partitions are handled in databases. It can be "add", "skip", or
"fail".
|
allowMissingPartitions=true | Specifies the behavior when incoming data contains new partition values that do not match any existing partitions. true (default) means to write the data that belongs to existing partitions and ignore the out-of-scope data. false means not to write any incoming data to the database and throw an exception. Note: For VALUE partitions where newValuePartitionPolicy is set to "add" or "fail", data will be handled based on the corresponding rules. In those cases, allowMissingPartitions does not take effect. |
perfMonitoring=1 | Whether to enable performance monitoring. The default value is false for the standalone mode and true for the cluster mode. |
pluginDir=plugins | The directory for the plugin files. The default value is the relative directory plugins. The system searches the relative directory plugins in the following order: home directory of the node, the working directoryof the node, and the directory with the DolphinDB executable. |
preloadModules=plugins::mysql, system::log::fileLog | The modules or plugins that are loaded after the system starts. Use commas to separate multiple modules and plugins. |
init=dolphindb.dos | This file is executed when the system starts. The default file is <HomeDir> /dolphindb.dos. It usually contains definitions of system-level functions that are visible to all users and cannot be overwritten. |
startup=startup.dos | This file is executed after the system starts. The default file is <HomeDir> /startup.dos. It can be used to load plugins, load tables and share them, define and load stream tables, etc. |
postStart=postStart.dos | This file is executed after scheduled jobs are initialized to load scheduled jobs. The default file is <HomeDir> /postStart.dos. |
run=dailyJobs.dos | This file is executed after the startup script (startup.dos) is executed. The default folder is DolphinDB home directory. It can be used to schedule jobs. After this file is executed, the system terminates. |
tzdb=C:DolphinDBservertzdb | The directory of the time zone database. The default value is homeservertzdb. |
webRoot=/home/DolphinDB/server/web | The directory of the web server. The default value is <HomeDir> /web. |
webLoginRequired=false | Whether a user must log in to use the web-based cluster manager. The default value is false. |
useHardLink=true | Whether to use the hardlink feature of the file system. Set to true to use the system hardlink feature; Set to false to not use the hardlink feature. The default value is true. |
maxFileHandles | Sets the maximum number of file descriptors open in a DolphinDB process. |
enableConcurrentDimensionalTableWrite =false | Whether to allow conducting concurrent write or update/delete on dimension tables. The default value is false, indicating concurrent write and update/delete is disabled for dimension tables. |
removeSpecialCharInColumnName=false | Whether to allow column names to contain special
characters or to start without a letter. The default value is false,
which can be used to remove the special characters in the column
names generated by pivot by . Specify the value as
true to be compatible with version 1.30.14 and before. |
enableChunkGranularityConfig=false | Specify the chunk granularity to determine the level
of the lock of a DolphinDB transaction. When writing to a chunk, the
transaction locks it to prevent other transactions from writing to
it. Before version 1.30.16/2.00.4, the chunk granularity is set at
the database level, i.e., each partition in the database is a chunk.
In this case, concurrent writes to different tablets in the same
partition are not allowed. This configuration parameter was released
with version 1.30.16/2.00.4. The default value is false, which
indicates the chunk granularity is at the table level, i.e., each
tablet of a partition is a chunk. Concurrent writes to different
tables in the same partition are thus allowed. If it is set to true,
you can specify the chunk granularity with the parameter
chunkGranularity of function database . |
TSDBAsyncSortingWorkerNum=1 | A non-negative integer indicating the number of threads for asynchronous sorting in the TSDB cache engine. The default value is 1. Data written to the TSDB cache engine are sorted by sortColumns. The tasks of writing and sorting data can be processed either synchronously or asynchronously and the asynchronous mode optimizes writing performance. Set it to 0 to enable the synchronous mode. |
logRetentionTime | The amount of time to keep a log file before deleting it. The default value is 30 days (of floating-point data type). For example, 0.5 means 12 hours. If set to 0, log files are not deleted. |
traceLogDir | The SQL tracing log path. The default value is <HomeDir>/traces. |
oldChunkVersionRetentionTime= 60 | Sets the retention time (in minutes) for the old chunk versions. The upper limit is 240. When SQL update/upsert/delete is executed, the system generates a new replica of the chunk (named by "table_name_tid") to perform the update and delete operations. Each old chunk will be retained in the system for the specified retention time, and a maximum of 5 old chunks can be kept at the same time. |
enableLocalDatabase=true | A Boolean value indicating whether to enable the creation of a local database. The default value is true. Once configured, it takes effect for all nodes in the cluster. |
enableInsertStatementForDFSTable |
Whether to enable the |
Each persisted MVCC table has a log file. Operations (including adding, deleting and modifying) on an MVCC table are first written to the log. When the number of operations reaches mvccCheckpointThreshold, a log checkpoint is initiated, where the operations are performed on the table and the log is cleared.
The number of operations on an MVCC table are counted as below:
-
For adds (
append!
,tableInsert
,insert into
) and updates (update
): the number of applied rows multiplied by the number of applied columns; -
For deletes (
delete
): the number of applied rows.
loadMvccTable
, the log
file needs to be replayed. If the amount of data in the log is too large, it may
take a long time to replay, or even result in an OOM event.Parameter | Description |
---|---|
mvccCheckpointThreshold =5000000 | Sets the threshold for the number of operations on an MVCC table to trigger a checkpoint. The value can be set in [100,000, 2^31-1], and the default value is 5,000,000. |
Core Dumps
Parameter | Description |
---|---|
enableCoreDump=true | [Linux Only] Whether to check and enable core dumps when starting DolphinDB. The default value is true. true means that DolphinDB will check if core dumps are enabled during startup. If core dumps are disabled, it will try to enable them, provided the system's hard limit allows core dumps of a sufficient size. If core dumps are already enabled, no changes will be made. False means that DolphinDB will start up without checking or modifying the system's current core dump enablement state. |
disableCoreDumpOnShutdown =false | [Linux Only] Whether to disable core dumps during a graceful shutdown. The default value is false, meaning core dumps will not be disabled during a graceful shutdown. This parameter only takes effect for the local node. |
Compatibility
Parameter | Description |
---|---|
enableNullSafeJoin | Whether to match NULL values in join columns when
performing JOIN operations (including left join ,
left semi join , right join ,
equi join and full join ). The
default behavior, when it is set to false, is to disallow joining on
NULLs. This follows the ANSI SQL semantics, which is the same as the
behavior in version 2.00.10 and later. |
movingIndexedObjectByIndexedLabel | A Boolean value to specify whether operations of
moving functions on indexed matrices/vectors are performed
column-wise or row-wise. True (default) means performing by columns,
false means by rows. The m-functions affected by this configuration
parameter include: move , mcovar ,
mcorr , mbeta ,
mwavg , mwsum ,
mpercentile , mrank ,
mcount , mfirst ,
mlast , mavg ,
mmed , mprod ,
msum , msum2 ,
mstd , mvar ,
mstdp , mvarp ,
mskew , mkurtosis ,
mmin , and mmax . |
ignoreSpecialCharacterInPartitionId =true | A Boolean value to specify whether to ignore ":" and "." in partitioning columns when creating partition directories for value-partitioned tables with partitioning columns of STRING or SYMBOL type. The default value is true, which matches the behavior before version 2.00.11/1.30.23. For example, by default, keys like ".a:bc." and "abc" of the partitioning column will map to the same directory path since ":" and "." are ignored. So data written to partitions ".a:bc." and "abc" will both end up being stored under "abc". If set to false, "." and ":" will be included as part of the path, which means data written to the ".a:bc." and "abc" partitions will be separated into different paths. |
keepTupleInRowFunction | Specify whether row-based vector functions or
function byRow whose func is a vector
function return a columnar tuple if the input argument is a columnar
tuple. If it is set to true (default value), a columnar tuple is
returned; if it is set to false, an array vector is
returned. |
logicOrIgnoreNull=true | Specifies whether function or
ignores NULL values. When set to true (default):
|
nullAsMinValueForComparison=true | Whether a NULL value is treated as the minimum value in data comparison. The default value is true. If it is set to false, the result of comparison involving NULL value is NULL. |
The parameter decimalRoundingMode is introduced in 2.00.12 to set the rounding mode for the following conversion scenarios:
-
Parsing data of STRING/SYMBOL type to DECIMAL type (e.g., loading files with
loadText
). -
Converting floating-point numbers to DECIMAL type.
-
Converting DECIMAL value to Integral type.
-
Converting high-precision DECIMAL value to low-precision DECIMAL value.
Note: As of 2.00.10, the rounding mode for the first case was changed from truncation to rounding. For other scenarios, truncation was used.
Parameter | Description |
---|---|
decimalRoundingMode | Specifies the rounding mode for DECIMAL type. The default is trunc, meaning the decimal part is truncated. Set it to round to round the decimal part. |
DolphinDB Terminal
DolphinDB terminal is a command line interactive tool for connecting to a remote DolphinDB server to execute commands. It is supported in 1.10.4 and above.
Start the DolphinDB terminal in the Linux system with the following command:
rlwrap -r ./dolphindb -remoteHost 192.168.1.135 -remotePort 8848
Start the DolphinDB terminal in Windows with the following command:
dolphindb.exe -remoteHost 192.168.1.135 -remotePort 8848
After starting up successfully, the following is displayed:
DolphinDB Terminal 1.10.4 (Build:2020.04.03). Copyright (c) 2011~2020 DolphinDB, Inc.
To quit the terminal:
quit
The following are the parameters related to the DolphinDB terminal. These parameters can only be used in the command line.
Parameter | Description |
---|---|
remoteHost | IP address of the remote DolphinDB server. |
remotePort | Port number of the remote DolphinDB server. |
stdoutLog | Where to output the system log. The default value is 0 indicating file dolphindb.log; 1 means stdout; 2 means both stdout and file dolphindb.log. |
uid | User name of the remote DolphinDB server. |
pwd | Password of the remote DolphinDB server. |
run | The local DolphinDB script file that is sent to the remote DolphinDB server for execution when starting up the terminal. After the execution is completed, the terminal will automatically exit. |
If an exception occurs when running a local script with -run on a remote server, the exception return value is 255.
Resource Monitoring
DolphinDB provides the feature to track resource usage at the user level on data nodes or compute nodes, including CPU and memory usage, DFS SQL queries, number of rows read, and data volumes accessed.
Parameter | Default | Description |
---|---|---|
resourceSamplingInterval | -1 | An integer (in seconds) specifying the sampling interval of resource monitoring. The default value is -1, indicating resource tracking is disabled. To enable resource sampling, a positive integer must be set. |
resourceSamplingMaxLogSize | 1024 | A positive integer (in MB) specifying the maximum size threshold of the resource sampling log. The default value is 1024. Log rotation is applied to split the log once it reaches the specified size. The generated file name is prefixed with its generation timestamp. For example, the file 20231101162302_access.log is generated at 2023.11.01T16:23:02. |
resourceSamplingLogRetentionTime | -1 | An integer (in days) specifying the maximum retention period for the resource sampling logs. The default value -1 means logs never expire. |
resourceSamplingLogDir |
A string specifying the directory storing the resource sampling logs. The default value is <HomeDir>/resource. |