Memory Management

Execute getClusterPerf() on the controller to check the memory usage of each node in the cluster, specifically:

• memoryAlloc: The total memory allocated to DolphinDB on the node, which is roughly equal to the total amount of memory requested from the operating system.
• memoryUsed: The memory used by the node, including the memory consumed by the variables, DFS tables, message queues, etc.
• maxMemSize: The maximum amount of memory that can be used by the node.

Execute mem() to display the memory usage of the current node. It returns a dictionary where allocatedBytes indicates the size of the allocated memory, and freeBytes indicates the size of the available memory. The total size of memory used by the node can be calculated via mem().allocatedBytes - mem().freeBytes. (Note: For 1.20.0 or earlier versions, execute sum(mem().allocatedBytes - mem().freeBytes)).

Execute objs() to view the variables in the current session. It returns a table of variables where the "bytes" column indicates the memory usage by the variables. In addition to the memory usage by the variables in the current session, objs(true) also returns the memory usage by shared variables (including variables shared by other sessions).

Execute memSize() to view the size (in bytes) of memory occupied by an object. For example:

v=1..1000000
memSize(v)

The output is 4000000 (bytes).

Execute getSessionMemoryStat() to get the memory usage of each session on the current node. The result doesn't include memory usage of shared tables or DFS tables, as they don't belong to any particular user or session. As the result includes the "userId" for each session, you can get the memory usage of each user.

The functions introduced above do not take into account the memory usage by the jobs running in the background. You can use getRecentJobs() to view the background jobs on the local node, which may use a lot of memory. Currently, DolphinDB doesn't provide any function to get the memory usage of background jobs. You may need to estimate their memory usage. If the memory usage of a node is very high, consider cancelling some nonurgent background jobs.

If the memory usage of a node is close to maxMemSize, release all cached data on the node via clearAllCache(). If memory usage is still very high, check the following items:

• Check if there are sessions consuming too much memory via getSessionMemoryStat() and identify the corresponding users. Execute objs(true) to identify the variables (including the variables shared by other sessions) occupying large amount of memory in the current session.

• Check if there are background jobs or interactive jobs running longer than expected via getRecentJobs() and getConsoleJobs(). The memory usage of these jobs is not reflected in the result of getSessionMemoryStat().

• Check if the streaming data is consuming too much memory via getStreamingStat() (see section "Caching Streaming Data").

An OOM occurs often because a query requires more memory than the system can provide. Please first execute the following query to check the memory usage of all data nodes, where maxMemSize is the memory limit for the node, memoryUsed is the amount of memory consumed by the node, and memoryAlloc is the amount of memory allocated to the node.

select site, maxMemSize, memoryUsed, memoryAlloc from rpc(getControllerAlias(),getClusterPerf)

OOM exceptions usually have the following causes:

• The "where" clause doesn't specify any partitioning column or involves too many partitions. As a result, too much data needs to be loaded into memory.

Use sqlDS() to check how many partitions are involved in your query. For example:

ds=sqlDS(<select * from loadTable("dfs://demo","sensor")>)
ds.size()

To solve this issue, see Use partitioning columns in the "where" clause in section 2 "How to Optimize Memory Usage". For more information, see section 5 "Read Cache for DFS Databases".

• The write caches use too much memory.

getCacheEngineMemSize() //Check the memory consumed by the OLAP cache engine
getTSDBCacheEngineSize() //Check the memory consumed by the TSDB cache engine

For solution and more information, see section 6 "Write Cache for DFS Databases".

• Data is not evenly distributed across partitions. A query that attempts to load a large partition into memory will cause OOM if the partition is larger than the available memory.

Use getTabletsMeta() to check the size of each partition. For example:

getTabletsMeta("/demo/%", `sensor, true);

To solve this issue, see Carefully design your partitioning scheme in section 2 "How to Optimize Memory Usage". For more information, see section 5 "Read Cache for DFS Databases".

• A session contains variables that occupy a large amount of memory on the node, leaving little space for other objects or operations.

Use getSessionMemoryStat() to check the size of memory used by each session and the corresponding users.

Execute the following script to check the variables and their memory usage on each node:

pnodeRun(objs) //Return the memory occupied by the non-shared variables of the current session
pnodeRun(objs{true}) //Return the memory occupied by the variables, including the shared variables

To solve this issue, see Undefine large variables when they are no longer needed in section "How to Optimize Memory Usage". For more information, see section "Using Variables".

• The streaming engines use too much memory.

Use getStreamingStat() to check the queue depth of the publishing queue and subscribing queues. Use getStreamingEnghineStat() to check the memory usage by streaming engines.

To solve this issue, see Set a reasonable cache capacity for streaming data in section "How to Optimize Memory Usage". For more information, see section "Caching Streaming Data".

This issue is often due to the fact that the memory allocated to the node exceeds the system's physical memory limit and the operating system forces DolphinDB to quit. The Linux kernel has a mechanism called Out of Memory (OOM) Killer, which monitors the processes that occupy too much memory, especially the processes that suddenly take up a lot of memory. These processes will be automatically killed to prevent memory exhaustion. To check if DolphinDB exits because of the OOM killer, use the dmesg command:

dmesg -T|grep dolphindb

If you see "Out of Memory: Kill process" in the output, it means that the operating system killed the DolphinDB process (For more information, see the Linux kernel official documentation). The solution to this problem is to specify a memory limit for DolphinDB via the configuration parameter maxMemSize (for single node mode, modify the parameter in dolphindb.cfg; for cluster mode, modify it in cluster.cfg).

Specify the configuration parameter memoryReleaseRate to adjust the rate at which unused memory is released to the OS. memoryReleaseRate = 10 means that DolphinDB will return the idle memory to the OS at the fastest rate. The default value is 5. Setting this parameter is equivalent to setting the tcmallloc_release_rate parameter in TCMalloc.

Note: Execute mem() to display the memory usage of the current node. If the parameter freeUnusedBlocks = true, unused memory blocks will be released from memory.

Check if there's any information about the cache engine in the log. If there is, it means the amount of data that is intended to be loaded into memory by loadText() exceeds chunkCacheEngineMemSize (OLAP engine) or TSDBCacheEngineSize (TSDB engine), resulting in a stuck transaction. (For more information about the cache engines in DolphinDB, see Section 6 "Write Cache for DFS Databases" below.)

Try the following solutions:

(1) Specify a greater value for the configuration parameter chunkCacheEngineMemSize (OLAP engine) or TSDBCacheEngineSize (TSDB engine).

(2) For 2.00.4 and higher versions, use loadTextEx() instead and specify the atomic parameter to split the loading process into multiple transactions.

DolphinDB uses columnar storage. Only the relevant columns of a query are loaded into memory.

Example 4. Calculate the maximum value of the column "tag1" in the "2022.01.01" partition and check the memory usage. As the query involves only 1 column from 1 partition, only this column is loaded to memory. As explained in the section above, the size of a column in a partition is about 80MB.

The "2022.01.01" partition is stored on node1 (execute getClusterChunksStatus() on the controller to check the location of each partition). Connect to node1 and execute the following code:

select max(tag1) from loadTable(dbName,tableName) where day = 2022.01.01
mem().allocatedBytes - mem().freeBytes

The result is 84,267,136 (bytes), which is consistent with the expectation of 80MB.

Example 5. Query the first 100 records in the "2022.01.01" partition on node1 and check the memory usage.

Although only 100 records are selected, all columns in the partition (about 800 MB) are loaded because the smallest loading unit is a column of a partition.

Execute the following code on node1:

select top 100 * from loadTable(dbName,tableName) where day = 2022.01.01
mem().allocatedBytes - mem().freeBytes

The result is 839,255,392 (bytes), which is close to the expectation of 800MB.

A well-designed partitioning scheme helps to avoid OOM exceptions

As DolphinDB manages memory in units of a column of a partition, memory usage is closely related to how the database is partitioned. If data is not evenly distributed across partitions, and the size of a partition is even larger than the total memory size of your machine, then an OOM occurs when this partition is queried (when all columns are selected). As a rule of thumb, for a DFS table, the amount of data of a partition before compression is recommended to be between 100MB to 1GB. For example, if a table has 10 columns and each cell is 8 bytes, then each partition is recommended to store 1 - 2 million rows.

Moving large amount of data between nodes is time-consuming. When executing a query in a DFS database, DolphinDB sends the task to the nodes with the relevant data and collects the result from other nodes to the node that sends out the query. Raw data is moved across nodes only if the query cannot be completed otherwise.

Example 6. On node1, query the maximum value in the "tag1" columns from 2 partitions, "2022.01.02" and "2022.01.03". The "2022.01.02" partition is stored on node1 and "2022.01.03" is stored on node2. Execute the code below on node1 and node2 to check the memory usage.

select max(tag1) from loadTable(dbName,tableName) where day in [2022.01.02,2022.01.03]
mem().allocatedBytes - mem().freeBytes)

The result is 84,284,096 on node1 and 84,250,624 on node 2. This indicates that node1 loaded the relevant column from the "2022.01.02" partition to memory, and node2 loaded the relevant column from the "2022.01.03" partition to memory.

Example 7. Execute the following code on both nodes. We expect that the partition of "2022.01.02" would be loaded into memory on node1 and the partition of "2022.01.03" would be loaded into memory on node2. Memory usage is about 800MB on each node.

select top 100 * from loadTable(dbName,tableName) where day in [2022.01.02,2022.01.03]
mem().allocatedBytes - mem().freeBytes

The result on node1 is 839,279,968 (bytes) and the result on node2 is 839,246,496 (bytes), which is consistent with the expectation.

Note: Please avoid using select * statement without setting any filtering conditions, as it will load all data into memory and likely cause OOM with a large table. Especially for tables with many columns, it is recommended to load only the required columns.

For efficient use of memory, DolphinDB only stores a single copy of cached data for each partition across the entire cluster.

Example 8. Open two DolphinDB GUI windows and connect to node1 and node2. Run the following script on both nodes to get the data from the "2022.01.01" partition that is located on node1.

select * from loadTable(dbName,tableName) where date = 2022.01.01
mem().allocatedBytes - mem().freeBytes

The result on the node1 is 839,101,024 (bytes) whereas no memory is occupied on node2. The system loads the partition into memory of node1, and then transfers it to the memory of node2. After downloading the data to the client side, the system releases the memory from node2.

Example 9. Set maxMemSize = 10 (GB) and warningMemSize = 8 (GB) on the data nodes. Execute the following script to load 9 partitions (about 7.2GB in total) located on the same node to memory, one partition (about 800MB) at a time.

dbPath =  right(dbName,strlen(dbName)-5)
p = select top 9 * from rpc(getControllerAlias(),getClusterChunksStatus) 
    where file like dbPath +"%" and replicas like "node1%" 
    order by file
days = datetimeParse(t.file.substr(strlen(dbPath)+1,8),"yyyyMMdd")
for(d in days){
    select * from loadTable(dbName,tableName) where  date= d
    print mem().allocatedBytes - mem().freeBytes
}

The memory usage increases with the loading of each partition:

All partitions will be cached in the memory as long as the memory usage is below warningMemSize. If these partitions are queried by a subsequent query, they can be fetched from the memory instead of from the disk.

When the total memory usage reaches warningMemSize, DolphinDB will release the least recently used (LRU) data from memory to make room for the new data.

Example 10. In Example 9, we loaded only 9 partitions to memory. In this example, we load 15 partitions to memory one by one so eventually the total amount of the loaded data will exceed the warningMemSize. Let's see how memory usage evolves during the process:

As shown in the figure above, when the amount of cached data exceeds warningMemSize, DolphinDB will remove a part of the cached data to keep the total memory usage below warningMemSize.

Example 11. In this example, we show how memory usage changes when the cached data is close to the warningMemSize and a session object requests more memory.

First, check the current memory usage:

mem().allocatedBytes - mem().freeBytes

The result is 7,550,138,448 (MB) - a little over 7GB. The specified maximum memory usage is 10GB. Now we create a variable that takes 4GB of memory, then print the memory usage.

v = 1..1000000000
mem().allocatedBytes - mem().freeBytes

The result is 8,196,073,856, roughly 8GB. This implies that DolphinDB automatically removes cached data from memory to make room for new variables.