2.00.11

Added function fromStdJson for parsing standard JSON objects into DolphinDB variables. (2.00.11.6)

Added function sqlTuple to generate metacode with a tuple expression. (2.00.11.3)

Added functions adfuller and coint for cointegration testing. (2.00.11.2)

Added configuration parameter s3Endpoint for accessing AWS S3. (2.00.11.2)

Added function regexFindStr to find substrings that match a regular expression in a given string.(2.00.11.1)

Added functions highLong and lowLong to parse 16-byte data (such as INT128, UUID, IPADDR) to high-order and low-order 8-byte long integer data. (2.00.11.1)

Added configuration parameter enableDFSQueryLog to configure whether to log all DFS queries. (2.00.11.1)

Soft delete can be enabled for TSDB databases with newly added parameter softDelete. A soft delete marks a record as no longer active or valid without actually deleting it from the database. Soft deletes can improve performance, and can allow "deleted" data to be recovered. (2.00.11)

Added support for using the OVER clause with window functions which compute values over a particular window (group of rows) and return a single result for each row, including: sum, avg, std, stdp, var, varp, count, min, max, skew, kurtosis, med, rank, dense_rank, percent_rank, row_number, cume_dist, lead, lag, ntile, first_value, last_value, and nth_value. (2.00.11)

Implemented user-level resource management (2.00.11):

• Added function getUserHardwareUsage to get CPU and memory usage samples.

• Added function getUserTableAccessRecords to get SQL queries on DFS tables.

• Added functions enableResourceTracking and disableResourceTracking to enable and disable resource isolation.

• Added configuration parameters resourceSamplingInterval, resourceSamplingMaxLogSize, and resourceSamplingLogRetentionTime to set the collection interval, log splitting size threshold, and log retention time for usage sampling.

Added configuration parameter jobParallelismHardLimit to limit the maximum number of workers a job can use to the parallelism of that job. (2.00.11)

Added configuration parameter defaultJobParallelism to set the default parallelism for jobs submitted by users. (2.00.11)

Added configuration parameter datanodeRestartLevel to set the triggering condition for the controller to automatically restart the data/compute node. Added function setDatanodeRestartInterval to set the restart interval online and function getDatanodeRestartInterval to obtain the configured datanodeRestartInterval. (2.00.11)

Added configuration parameters thirdPartyCreateUserCallback and thirdPartyDeleteUserCallback to call back specified function views when using the createUser and deleteUser functions. (2.00.11)

Added functions RidgeCV, ElasticNetCV, and LassoCV for cross-validation of machine learning models. (2.00.11)

Added functions iminLast/imaxLast/miminLast/mimaxLast/rowIminLast/rowImaxLast to return the maximum index for multiple extreme values. (2.00.11)

Added financial functions bondDuration, bondDirtyPrice, and bondConvexity to calculate the bond duration, dirty price and convexity. (2.00.11)

Added function signbit to get the sign of ±0.0. (2.00.11)

Added function submitJobEx2 which has similar functionality as submitJobEx but supports passing in a callback function. (2.00.11)

Added functions polyFit and poly1d for polynomial fitting. (2.00.11)

Added function cubicSpline and cubicSplinePredict for cubic spline interpolation and prediction. (2.00.11)

Added function kendall to calculate the Kendall rank correlation coefficients. (2.00.11)

Added functions clip and clip! to truncate data outside a given range. These functions have similar functionality to numpy.clip. (2.00.11)

Added function tableUpsert which has the same functionality as upsert! but returns the number of records that have been updated and inserted. (2.00.11)

Added function brentq to find a root of a function in a bracketing interval using Brent's method. (2.00.11)

Added function snippet to return the results as a STRING scalar. (2.00.11)

Added function socp which minimizes a linear objective function within specified constraints for solving second-order cone programs. (2.00.11)

Added configuration parameter enableLocalDatabaseto enable or disable local database creation. (2.00.11)

Added maintenance function getTSDBDataStat to obtain statistics of the TSDB engine. (2.00.11)

Added function existsStreamTable to check if a stream table exists. (2.00.11)

Added function getStreamTables to get the information of all stream tables. (2.00.11)

Added function createRuleEngine to create a streaming rule engine which validates the inputs with user-defined rule sets. Added functions updateRule and deleteRule to update or delete the rule sets. (2.00.11)

Added function createNarrowReactiveStateEngine to create a reactive state engine that reorganizes the results into a narrow format. Added function addReactiveMetrics to add new metrics to the engine and function getReactiveMetrics to retrieve metrics. (2.00.11)

Added support for creating a matrix of STRING type. (2.00.11.4)

When invoking getCurrentSessionAndUser within the handler parameter of subscribeTable, it will return the "sessionID" of the session from which the subscribeTable operation was executed. (2.00.11.3)

When invoking getCurrentSessionAndUser within the filter parameter of streamFilter, it will return the "sessionID" of the session from which the stream filter engine was created. (2.00.11.3)

When parameter triggeringPattern of function createCrossSectionalEngine is set to “dataInterval“, the calculation will now be triggered only for intervals where new data has been received, instead of for all intervals. (2.00.11.3)

The args parameter of function makeUnifiedCall now supports input of metacode with a tuple expression. (2.00.11.3)

Function hex now can convert floating numbers to hexadecimal. (2.00.11.3)

Trading calendar now can specify weekends as trading days. (2.00.11.3)

Function backup supports backup of cold data. (2.00.11.3)

Partial application now supports using object methods. (2.00.11.3)

Function replayDS supports input of DECIMAL type. (2.00.11.2)

Non-key columns of a keyed stream table can be array vectors. (2.00.11.2)

Lambda expression supports object.method(...) usage. (2.00.11.2)

Specific elements in array vectors and columnar tuples can be modified by referring to their row and column indices. (2.00.11.2)

Optimized the performance of select, update, and delete when using makeKey. (2.00.11.2)

Optimized the precision of results returned by function format. (2.00.11.2)

The JIT version does not support chained operation with append!. (2.00.11.2)

A new column "owner" that displays the object creator is added to the table returned by function objs when querying shared variables. (2.00.11.1)

The following functions support specifying trading calendars when taking arguments of DURATION type: interval, wj, twindow, temporalAdd, duration, move, tmove, moving, tmovingWindowData and time-based moving functions. (2.00.11.1)

Added parameter method to function ols to allow users to construct coefficient matrices and inverse matrices or use singular value decomposition for ordinary least squares. (2.00.11.1)

Optimized the performance of temporalAdd when adding trading days to a temporal vector. (2.00.11.1)

Empty objects are allowed in the following higher-order functions: each, eachPre, eachPost, eachLeft, eachRight, cross, reduce, accumulate, any, all, loop, and ploop. (2.00.11.1)

The values in the dictionary passed to the transpose function can be specified as scalars. (2.00.11.1)

Added support for Copy-On-Write (COW) of dictionaries and tuples. (2.00.11.1)

Line breaks are supported before the member(.) operator in scripts. (2.00.11.1)

Integrated modules and function view functionality (2.00.11):

• Enabled namespace (module) for function view to prevent name conflicts. Any addition or deletion operations on function views will be synchronized and persisted to all nodes in a cluster.

• Modified VIEW_EXEC privilege to enable operations on all function views within the specified modules.

Enhanced access control on job cancellation. (2.00.11)

The value of a dictionary can take the DECIMAL type. (2.00.11)

window join supports special windows (= 0:0). The windows over the right table are determined by the current timestamp in the left table and its previous timestamp. (2.00.11)

Added return values for function schema (2.00.11):

• the database and table creator;

• compression algorithm for each column;

• whether soft delete is enabled.

The loadPlugin function allows directly loading the plugin by specifying the filePath parameter as the plugin name. (2.00.11)

The backup function now supports setting backupDir to a directory on AWS S3. (2.00.11)

Functions rowlmax and rowlmin now support passing array vectors as inputs. (2.00.11)

Enhanced functions transFreq, asFreq, and resample (2.00.11):

• The closed and label parameters have been relaxed to allow more flexible specification when the time precision of rule is larger than day.

• Added support for multiple trading days frequency for operations.

Function moving now supports passing a matrix as input (2.00.11).

Added support for online update for licenses with trial authorization. (2.00.11)

Function rowAt now supports taking columnar tuples as input. (2.00.11)

Function unpack now supports taking BLOB data as input. (2.00.11)

Enhanced higher-order functions byRow and byColumn: (2.00.11)

• Added support for symbols :H and :V which can be composed together.

• Added support for taking array vectors and columnar tuples as inputs.

Function transpose now supports taking array vectors and columnar tuples as inputs. (2.00.11)

Functions toJson and toStdJson support escape characters when converting BLOB types. (2.00.11)

Optimized the performance of functions scs and quadprog. (2.00.11)

The unionAll function supports NULL values for the first element when the input is a tuple. (2.00.11)

The memory taken up by the dimension table now will be released automatically. (2.00.11)

Function replaceColumns! supports replacing multiple columns of an in-memory table. (2.00.11)

Optimized query performance for the case when the TSDB table has more than two sortColumns and sort key hashing is enabled. (2.00.11)

For value-partitioned tables with partitioning columns of STRING or SYMBOL type, configuration parameter ignoreSpecialCharacterInPartitionId is added to specify whether to ignore ':' and '.' in partitioning columns when creating partition directories. (2.00.11)

Optimized the performance of function createAsofJoinEngine. (2.00.11)

Added parameter outputElapsedMicroseconds for the asof join engine and lookup join engine to output the elapsed time of each calculation. (2.00.11)

Functions createAsofJoinEngine, createSessionWindowEngine, and createLeftSemiJoinEngine now support passing array vectors as inputs. (2.00.11)

The calculation of time series engine now can be triggered by each incoming record with updateTime = 0. (2.00.11)

Added parameterskeyCapacity and parallelism for the reactive state engine to address the performance issue when calculating within a large number of groups. (2.00.11)

The state function defined in metrics of a reactive state engine now supports using for loops. (2.00.11)

Added parameter keepDuplicates for the lookup join engine to specify whether to keep all matching records in the right table. (2.00.11)

Added parameter subWindow for the time series engine to enable calculations in a subset time range within the window. (2.00.11)

Enhanced the streaming monitoring with function getStreamingStat for the status of workers of subscriber nodes. (2.00.11)

Function tmsum2 can be used as a state function in a reactive state engine. (2.00.11)

The cumTopN functions can be used as state functions in a reactive state engine. (2.00.11)

Added parameter closed for special windows of the reactive state engine to enable windows to be left-open and right-closed. (2.00.11)

Added option "dataInterval" for parameter triggeringPattern of function createCrossSectionalEngine to calculate at intervals based on timestamps in the data. (2.00.11)

Added option "saltedHash" for parameter dispatchType of function createStreamDispatchEngine to apply a salted hash algorithm on keyColumn and distribute records based on the hash result. (2.00.11)

Added parameter preciseRate of function replay to specify whether to replay at replayRate times the time difference between two adjacent records. (2.00.11)

Data of DECIMAL type now can be calculated in streaming engines. (2.00.11)

Enhanced job execution logs to include detailed tracking for UPDATE and DELETE statements. (2.00.11)

Optimized the query performance on partitioned tables which contain over 100000 rows or over 1000 partitions. (2.00.11)

• The GROUP BY is followed by a non-partitioned column and the SELECT clause uses a function that cannot be distributed, such as a UDF.

• The CONTEXT BY is followed by a non-partitioned column and the [HINT_KEEPORDER] keyword is not used.

For filtering conditions in a distributed query, DATE type can be compared with higher precision temporal types. (2.00.11)

Added size constraints for STRING, BLOB, and SYMBOL data written to distributed databases. (2.00.11)

Updated the exchange calendars for Year 2024. (2.00.11)

Fixed server crashes caused by SQL queries if a database path was not a string. (2.00.11.6)

Fixed an issue where replay tasks got stuck if the number of workers was significantly smaller than the number of partitions. (2.00.11.6)

Fixed an error in SQL queries with pivot by clause when the select/exec clause contained expressions. (2.00.11.6)

Fixed an error when restoring manual backup files from Amazon S3. (2.00.11.6)

Fixed server crashes when passing a tuple with no vector elements to nunique. (2.00.11.5)

Fixed an issue where data with a sort key was not properly split, leading to an excessively large block in a TSDB database when keepDuplicates = ALL. (2.00.11.5)

Fixed an issue where asynchronous replication occasionally got stuck. (2.00.11.5)

Fixed server crashes when passing a dictionary to at:R or at:L. (2.00.11.5)

Fixed occasional execution failures for tasks scheduled at around 24:00. (2.00.11.4)

Fixed occasional crashes when tracking resources were enabled. (2.00.11.4)

Fixed server crashes caused by stack overflow when using function mrank with excessively large data. (2.00.11.3)

Fixed a parsing issue for binary operator with function patterns, e.g +:A(1, 2 3 4). (2.00.11.3)

Fixed the error when using function eval to evaluate a lambda expression parsed by parseExpr. (2.00.11.3)

Fixed an issue where the value of CODE type returned from a user-defined function was incorrectly interpreted as of STRING type. (2.00.11.3)

Fixed the deserialization failure when calling a module function with the rpc function. (2.00.11.3)

Fixed the issue where data being asychronously replicated was removed when the controller failed to commit the transaction. This issue was introduced since version 2.00.11. (2.00.11.3)

Fixed server crashes that occurred when executing a SQL statement with vectorization applied to grouping columns and SELECT columns using the last() function without any arguments. (2.00.11.3)

Fixed a parsing issue that occurred when attempting to sort a non-existent column within a SQL statement. A validation check for the sorted column has been added, and an error will now be reported in such cases. (2.00.11.3)

Fixed data inconsistency during online recovery of replicas with large amount of different data. (2.00.11.2)

Fixed occasional server crashes caused by OOM. (2.00.11.2)

Fixed server crashes due to concurrent race conditions during remote streaming subscriptions. (2.00.11.2)

Fixed the issue where batch job queue got stuck due to job creation failures. (2.00.11.2)

Fixed a server startup failure due to deserialization error when a function view involved the update operations on shared tables. (2.00.11.1)

Fixed an issue where a transaction was stuck in a resolution state due to frequent resolution requests. (2.00.11.1)

Fixed an issue where the login or shutdown operations on a cluster via the web interface may get stuck under heavy load. (2.00.11.1)

Fixed a bug where the query scripts returned by function getUserTableAccessRecords were truncated by commas. (2.00.11.1)

Fixed occasional metadata errors arising from frequent OOM rollbacks across multiple transactions. (2.00.11.1)

Fixed incorrect results of each when the input was a partially-applied function (such as makeCall and binaryExpr). (2.00.11.1)

Fixed server crashes when the obj of upsert! was a keyed table with the original key column of STRING type and the corresponding newData column of SYMBOL type, and ignoreNull was set to true. (2.00.11.1)

Fixed incorrect results of rowImax, rowImin, rowImaxLast and rowIminLast when taking columnar tuples. (2.00.11.1)

Fixed incorrect results of add when taking BLOB vectors. (2.00.11.1)

Fixed incorrect results of update on DFS tables when aggregate functions, order-sensitive functions or user-defined functions were applied across partitions. (2.00.11.1)

Fixed incorrect query results when the where condition of a SQL query was specified as NULL. (2.00.11.1)

Fixed incorrect query results when a SQL query involved order by clause, and the where conditions used AND to specify multiple conditions containing aggregations. (2.00.11.1)

Fixed an error when a SQL order by clause specified variable names. This issue was introduced in versions 2.00.10 / 1.30.22. (2.00.11.1)

Fixed server crashes when using case when to evaluate an empty table in a SQL query. (2.00.11.1)

Fixed an error when using the find function in the order by clause. This issue was introduced in version 2.00.11. (2.00.11.1)

Fixed occasional server crashes during streaming subscription. (2.00.11.1)

Fixed an issue where multiple identical topics were generated after repeated streaming subscription and unsubscription. (2.00.11.1)

Fixed an error for createReactiveStateEngine when the parallelism parameter was set and metrics contained constant expressions. (2.00.11.1)

Fixed an error when sliceByKey accepted a shared keyed table as input. (2.00.11.1)

Fixed restore failures when attempting recovery for TSDB tables created without a specified sortKeyMappingFunction parameter or with sortKeyMappingFunction set to the asis function. (2.00.11.1)

Fixed an issue where users failed to delete redundant abnormal replicas through deleteReplicas. (2.00.11.1)

Fixed access denial errors when using nested function views. (2.00.11)

Fixed server crashes when appending NULL values to array vector columns in a window join engine. (2.00.11)

Fixed "No such file or directory" errors when reading empty dimension tables which were restored from backups after server restart. (2.00.11)

Fixed sporadic type mismatch errors during query execution for a TSDB table created with sortKeyMappingFunction when data was being flushed from cache engine to disk. (2.00.11)

Fixed server crashes when using ols on TSDB table columns without where conditions. (2.00.11)

Fixed occasional errors when renaming columns using rename!. (2.00.11)

Fixed startup failures when dataSync=1 was configured on the controller but chunkCacheEngineMemSize was not set on the compute nodes. (2.00.11)

Fixed server restart failure due to replay issues after writing strings over 64 KB to distributed tables. (2.00.11)

Fixed incorrect nunique results calculated over multiple fields. (2.00.11)

Fixed server crashes when using each on array vector columns in TSDB tables without where conditions. (2.00.11)

Fixed server crashes when the symbol :R (of high-order function eachRight) took an incorrect number of arguments. (2.00.11)

Fixed errors when querying the table imported with loadText from a non UTF-8 formatted CSV file. (2.00.11)

Fixed incorrect results when applying the string function with byRow on a matrix. (2.00.11)

Fixed incorrect align results on series with non-increasing labels that contain duplicates. (2.00.11)

Fixed incorrect results of the parallel computation of peach for ignoring NULL values. (2.00.11)

Fixed memory leaks when calling resample. (2.00.11)

Fixed a bug where the corrMatrix function modified values of input arguments. (2.00.11)

Fixed the following issues with fromJson (2.00.11):

• Server crashes caused by dictionary value being an empty JSON tuple.

• Incorrect results caused by dictionary value being nested JSON tuples.

Fixed occasional row count mismatch errors when specifying both partition and byColName for function unionAll. (2.00.11)

Fixed an issue where streamEngineParserfailed to create streaming engines due to parsing errors if row-based metrics invoked functions with multiple return values. (2.00.11)

Fixed an issue where the subscriber consuming messages at a slower pace compared with other subscribers failed to read data. (2.00.11)

Fixed server crashes caused by ingesting an empty table into the left table of a lookup join engine. (2.00.11)

Fixed deadlocks when two nodes simultaneously call unsubscribeTable to cancel subscriptions to the same table on a remote node. (2.00.11)

Fixed incorrect results of createTimeSeriesEngine when specifying multiple columns for timeColumn and calculating on one of the time columns. (2.00.11)

Fixed incomplete results when using the map keyword with context by clause. (2.00.11)

Fixed an issue where calculations were not conducted after sorting within groups when order-sensitive functions were used in the select clause with context by and csort. (2.00.11)

Fixed errors when using aggregate functions on vector slices in the select clause with group by. (2.00.11)

Fixed an issue where a variable with the same name as a function view was incorrectly parsed in a SQL where clause if the and operator was used. (2.00.11)

Fixed server crashes when the variable after drop table did not exist. (2.00.11)

Fixed incorrect results when using the zigzag function in the select clause with context by. (2.00.11)

Fixed server crashes on Windows when applying group by to long strings. (2.00.11)

Fixed an issue with sqlDS data sources whereby specified partitions from the right table were absent in the generated output when joining two distributed tables. (2.00.11)