3.00.0

Version: 3.00.0   Compatibility Level 2 with 2.00.11 & 1.30.23

For details, see Compatibility Changes.

Release Date: 2024-04-03

Available Downloads:

New Features

  • Added function fromStdJson for parsing standard JSON objects into DolphinDB variables. (3.00.0.2)

  • Added function parseInteger (alias parseInt) for converting strings into integers with specified numeral system. (3.00.0.2)

  • Added functions backupSettings and restoreSettings to back up and restore settings on users, permissions and function views for database migration. (3.00.0.2)

  • Added functions VaR and CVaR to calculate value at risk and conditional value at risk. (3.00.0.1)

  • Added functions nss and nssPredict to fit the yield curve and predict the bond yield. (3.00.0.1)

  • Introduced Python Parser for parsing Python scripts. It integrates Python syntax with DolphinDB scripting language, which supports:

    • Parsing native Python data structures, built-in functions, and most programming features including assignments, statements, function definitions, list comprehensions, classes, etc.

    • Using pandas DataFrame, Series, and Index (SingleIndex only) classes, and some methods.

    • Using DolphinDB data forms and built-in functions.

    • Using basic SQL functionalities, including creating databases and tables, appending and querying data, etc.

  • Added CEP (Complex Event Processing) Engine for aggregating, processing, and analyzing event streams in order to gain real-time insights from events as they occur. This engine allows:

    • Defining event types with Class.
    • Dynamically adding multiple event matching rules.
    • Customizing event handling functions.
    • Flexible event processing order.
    • Visualizing data in the CEP engine with Data View Engine.
    • Distributing events across threads for parallel processing.

  • Added support for Global Catalog to conveniently query DFS databases, cluster nodes and other resources.

  • Introduced IMOLTP storage engine, which enables:

    • Saving data in memory to avoid disk I/O overhead.

    • Adopting row-oriented storage format, suitable for OLTP scenarios.

    • Creating B+ Tree index (primary/secondary index) for frequent and high-concurrent updates and queries.

  • Added GPLearn Engine implementing genetic programming using GPUs for automated factor mining.

  • Added support for Class in DolphinDB scripting language.

  • The reactive state engine supports defining factors via OOP.

  • Added support for UDP in streaming publishing and subscription, which optimizes network bandwidth when multiple subscribers receive from a single publisher and increases data transferring efficiency.

  • Added function matrixRank to calculate the rank of a matrix.

  • Added function bondAccrInt to calculate the accrued interest of securities.

  • Added function pinverse to find the pseudoinverse of a matrix.

  • Added function geoWithin to determine whether a point lies inside or outside a polygon.

  • Added functions getDfsRecoveryConcurrency and resetDfsRecoveryConcurrency to get and set the maximum concurrency of data recovery tasks on the controller.

  • Added functions getDfsRebalanceConcurrency and resetDfsRebalanceConcurrency to get and set the maximum concurrency of data rebalancing tasks on the controller.

  • Added function getTSDBTableIndexCacheStatus to check the memory usage of the level file indices of TSDB tables loaded on the current node.

  • Added function rowGroupby for aggregation grouped by rows.

  • Added function wslr for weighted single linear regression.

  • Added function digitize to bucket data based on specific bin values.

  • Added function lowerBound to find the first element that is greater than or equal to a specific value and return its index in an increasing object.

  • Added function fmin to minimize a function using a Nelder-Mead simplex algorithm.

  • Added functions pdfF, pdfChiSquare, and pdfNormal to calculate probability density of F, chi-square, and normal distributions.

  • Added function createReactiveStateLessEngine which maintains states and outputs calculation results based on associated data for each data injection.

  • Added function kroghInterpolate for polynomial interpolation of a set of points.

  • Added mTopN function: mpercentileTopN.

  • Added m-functions: mfirstNot, mlastNot, mLowRange, mTopRange.

  • Added functions minIgnoreNull and maxIgnoreNull for obtaining the min and max values with NULLs ignored.

  • Added tm-functions: tmLowRange, tmTopRange.

  • Added support for ZSTD compression algorithm.

  • Added function setDynamicConfig to dynamically set a specified configuration parameter.

  • Added function getDynamicConfig to list all dynamically-modifiable configuration parameters.

  • Added configuration parameter decimalRoundingMode to set the rounding method for DECIMAL values.

  • Added configuration parameter checkExistingPartitionNumForQuery to use the number of existing partitions obtained from the controller for queries.

  • Added configuration parameter enableAuditLog to enable audit logs. Added function getAuditLog to query audit logs and configuration parameter auditLogRetentionTime to set audit log retention time.

  • Added configuration parameter enableNullSafeJoin to determine whether NULL values are allowed to match other NULL values during join operations.

  • Added function setStreamTableTimestamp to attach a timestamp column to a stream table to record the data ingestion time.

  • The following features are supported in DolphinDB JIT version:

    • Set and dictionary forms;
    • STRING and SYMBOL types;
    • Multiple return values;
    • Creating array vectors.

  • The use keyword can be used to call .dom module files in scripts.

  • Added support for creating a matrix of DECIMAL or STRING type.

  • Added support for using macro variables and column series in SQL statements.

Improvements

  • The parameter groupingCol of function groupby now supports specifying a tuple with vectors of the same length. (3.00.0.2)

  • The parameter sortDirections of function sortBy! can be specified for a keyed table. (3.00.0.2)

  • The function fixedLengthArrayVector now supports concatenating fixed-length array vectors. (3.00.0.2)

  • Added support for fitness functions groupby and contextby in the Shark GPLearn engine. (3.00.0.2)

  • Added support for helper functions std, stdp, skew, kurtosis, and rank in the Shark GPLearn engine. (3.00.0.2)

  • Added parameter keyFilter to function createCrossSectionalEngine to enable key selection with filtering conditions. (3.00.0.2)

  • Optimized the event matching performance of CEP engine. (3.00.0.2)

  • Added parameter verbose for createGPLearnEngine to set whether to output training information. (3.00.0.1)

  • Added parameter minimize for createGPLearnEngine to set whether to minimize or maximize fitness. (3.00.0.1)

  • More helper functions are supported for defining GPLearn Fitness functions. (3.00.0.1)

Issues Fixed

  • [D20-17505] Fixed occasional server crashes caused by LEFT JOIN. (3.00.0.3)

  • [D20-17716] Fixed incorrect results when using function ej on keyed in-memory tables. (3.00.0.3)

  • [D20-18554] Fixed an issue where a chunk path corresponded to multiple chunk IDs in extreme cases. (3.00.0.3)

  • [D20-18780] Fixed a system failure to skip cluster replication tasks with corrupted data, which resulted in endless retry attempts. (3.00.0.3)

  • Fixed server crashes caused by SQL queries when the first argument of interval was not a column. (3.00.0.2)

  • Fixed an issue where minPeriods of function mmad did not take effect. (3.00.0.2)

  • Fixed server crashes caused by SQL queries if a database path was not a string. (3.00.0.2)

  • Fixed incorrect results of distributed queries if context by clause involved more than two columns and the csort clause included these context-by columns. (3.00.0.2)

  • Fixed an issue where replay tasks got stuck if the number of workers was significantly smaller than the number of partitions. (3.00.0.2)

  • Fixed an error in SQL queries with pivot by clause when the select/exec clause contained expressions. (3.00.0.2)

  • Fixed an error when restoring manual backup files from Amazon S3. (3.00.0.2)

  • Fixed an error when querying an empty table where the select clause called function eachPre and the SQL query contained context by clause. This issue was introduced in version 2.00.11.1. (3.00.0.2)

  • Fixed precision loss of DECIMALs when writing streaming data to a DFS table with batchSize set to 1. (3.00.0.2)

  • Fixed occasional incorrect results of metaprogramming with macro variables when applying function patterns to multiple columns by group, particularly with aggregate or order-sensitive functions. (3.00.0.2)

  • Fixed an issue where the equal= sign in where conditions did not take effect if a STRING or SYMBOL partitioning column of a VALUE-partitioned table contained decimal points or other special characters. This issue was introduced in version 3.00.0. (3.00.0.2)

  • Fixed an error reported when aggregating joined columns. (3.00.0.2)

  • Fixed server crashes caused by SQL queries calling ols within the higher-order function moving. (3.00.0.2)

  • Fixed server crashes where Raft was not initialized while HA stream tables were already loaded during a node reboot. (3.00.0.2)

  • Fixed incorrect result of executionSet returned by getSlaveReplicationQueueStatus if a Leader was switched to Follower and then re-elected as Leader. (3.00.0.2)

  • Fixed Unrecognized column name error for queries using ej on DFS tables and external variables in where conditions. (3.00.0.2)

  • Fixed the error SQL context is not initialized yet. RefId: S02011 reported when specifying the condition of addEventListener to attributes of Monitor. (3.00.0.1)

  • Fixed server crashes on the ARM version caused by multi-threaded queries. (3.00.0.1)

  • Fixed server crashes caused by concurrent select and append! operations on in-memory tables after resource tracking was enabled. (3.00.0.1)

  • Fixed an issue where the execution of clearCachedModules in console did not take effect. (3.00.0.1)

  • Fixed server crashes when using the head function in the aggs parameter of wj. (3.00.0.1)

  • Fixed server crashes caused by compilation failure during recursive or nested calls to JIT functions. (3.00.0.1)

  • Fixed server crashes caused by a distributed query with the WHERE clause containing functions that generated temporary variables. (3.00.0.1)

  • Fixed an issue where the analytic function could not recognize the column used in multiple joins. (3.00.0.1)

  • Fixed an issue where the schema function returned an incorrect partitionSchema that contained partitions already dropped with function dropPartition(with deleteSchema= true specified).

  • Fixed an issue where the result of the iif function was incorrectly typed as a STRING when the output was intended to be metacode.

  • Fixed an error for function getFunctionViews when the returned function body was excessively large.

  • Fixed an issue where the result of each function call was not being properly removed when the function contained cast($) operations used to convert the data type to the same type.

  • Fixed server crashes when the results of lshift (<<) and rshift (>>) contained NULL values.

  • Fixed an issue where replicas of chunks could be copied to a compute node using the copyReplicas function.

  • Fixed an issue where the format function returned incorrect results when applying a specified format to DECIMAL objects.

  • Fixed server crashes when using function med with excessively large data.

  • Fixed server crashes when passing a tuple with non-vector elements to the nunique function.

  • Fixed incorrect results when both X and Y of the asof function input vectors containing only NULL values.

  • Fixed an issue where appending data to a vector ignored the maximum size limit. An error will now be reported in such cases.

  • Fixed an issue where the lj, aj and other join-related functions did not support keyword parameters.

  • Fixed an issue where the database function failed to create an in-memory database with enableLocalDatabase=false configured.

  • Fixed an issue with the sqlDS function where partition pruning was not applied when the SQL statement specified in the sqlObj parameter contained the IN keyword in the WHERE clause.

  • Fixed an issue in standalone mode, where transactions were incorrectly replayed due to the edit log being written before the redo log.

  • Fixed incorrect results returned by the reactive state engine when calculating the cummin function with the following types of data: UUID, INT128, IPADDR, COMPLEX and POINT.

  • Fixed an error occurred when comments were placed between the function name and the function body.

  • Fixed an issue where the cleanup process for expired files of persistent stream tables would get stuck in an infinite loop if it encountered a non-existent file.

  • Fixed an issue where the presence of stale partitions could potentially cause data from new partitions located under the same path to be garbage collected.

  • Fixed server crashes when calculating DECIMAL array vectors in a query with PIVOT BY clause.

  • Fixed server crashes when using both CASE and WHERE NOT keywords in a single query.

  • Fixed an error that reported when using three or more UNION ALL keywords in a single SELECT FROM statement.

  • Fixed inconsistent query results when using double quote ("") and single quotes ('') in the WHERE condition to filter the CHAR column of a DFS table.

  • Fixed incorrect query results on a sorted keyed table

  • Fixed an issue where assigning a matrix to multiple output columns in a query returned incorrect results.

  • Fixed server crashes when using exists keyword in a query containing joins of multiple partitioned tables.

  • Fixed an issue where using a non-aggregate function in the SELECT statement did not report an error when grouping value-partitioned columns using the GROUP BY clause. An error will now be reported in such cases.

  • Fixed an error that occurred when the query result contained an array vector and the PIVOT BY clause included sort keys.

  • Fixed an error that occurred when using a function with multiple return values in combination with a WHERE clause in a query.

  • Fixed server crashes due to an out-of-bounds error when accessing an array in metrics of a time-series engine with useSystemTime=true specified.

  • Fixed a parsing issue in time-series engines for functions that return array vectors.

  • Fixed an issue where a write failure to a persisted table could disrupt data consumption for the subscription.

  • Fixed server crashes when the number of metricNames did not match that of factors specified in metrics for function createNarrowReactiveStateEngine.

  • Fixed an error when using the nullFill! function in a reactive state engine. This error was introduced in version 2.00.11.2.

  • Fixed server crashes when the formal parameter of the JIT function had the same name as the built-in functions.

  • Fixed an issue where the newly appended element to a vector within a JIT function could not be accessed or modified by indexes.

  • Fixed incorrect results returned by the JIT function when calculating a vector that had NULL values appended or assigned to it.