From kdb+ to DolphinDB

Overview

This tutorial introduces how to use the DolphinDB kdb+ plugin to import data from kdb+ into DolphinDB. It also provides mappings between the data types, functions, keywords, and other features of kdb+ and their DolphinDB equivalents.

Importing Data from kdb+ to DolphinDB

Use the DolphinDB kdb+ plugin to import data from kdb+ to DolphinDB. You can find the plugin and a detailed tutorial on our GitHub page.

First, execute loadPlugin("/path/to/plugin/Pluginkdb.txt") to load the plugin in DolphinDB, so you can use its provided functions.

The plugin provides two importing options, importing from kdb+ online or from disk. Both options import the data into DolphinDB as an in-memory table.

  • Import from a live kdb+ server:

(1) Connect to the running kdb+ server with connect, which returns a connection handle. (2) Use loadTable to import the data into DolphinDB by passing in the connection handle, the path to the kdb+ table and optionally the path to the sym file for the table.

(3) After importing, close the connection.

// load plugin
loadPlugin("/home/DolphinDBPlugin/kdb/build/Pluginkdb.txt")

// make sure the subsequent code is executed after the plugin has been loaded
go

// connect to a kdb+ database. Leave username and password empty if the database does not require authentication
handle = kdb::connect("127.0.0.1", 5000, "admin:123456")

// specifiy file path
DATA_DIR="/home/kdb/data/kdb_sample"

// use loadTable to import data into DolphinDB
Daily = kdb::loadTable(handle, DATA_DIR + "/2022.06.17/Daily/", DATA_DIR + "/sym")
Minute = kdb::loadTable(handle, DATA_DIR + "/2022.06.17/Minute", DATA_DIR + "/sym")
Ticks = kdb::loadTable(handle, DATA_DIR + "/2022.06.17/Ticks/", DATA_DIR + "/sym")
Orders = kdb::loadTable(handle, DATA_DIR + "/2022.06.17/Orders", DATA_DIR + "/sym")

// close connection
kdb::close(handle)
  • Import from kdb+ data files on disk:

Use loadFile to read data from disk by specifying the path to the kdb+ table and optionally the path to the sym file for the table.

// load plugin
loadPlugin("/home/DolphinDBPlugin/kdb/build/Pluginkdb.txt")

// make sure the subsequent code is executed after the plugin has been loaded
go

// specifiy file path
DATA_DIR="/home/kdb/data/kdb_sample"

// use loadFile to import data into DolphinDB
Daily2 = kdb::loadFile(DATA_DIR + "/2022.06.17/Daily", DATA_DIR + "/sym")
Minute2= kdb::loadFile(DATA_DIR + "/2022.06.17/Minute/", DATA_DIR + "/sym")
Ticks2 = kdb::loadFile(DATA_DIR + "/2022.06.17/Ticks/", DATA_DIR + "/sym")
Orders2 = kdb::loadFile(DATA_DIR + "/2022.06.17/Orders/", DATA_DIR + "/sym")

DolphinDB Reference for kdb+ Users

This section explains the mapping between the data types, functions, keywords and other elements of kdb+ and the DolphinDB equivalents. For more information, see kdb+ and q documentation and DolphinDB user manual.

Datatypes

Basic Datatypes

kdb+ DatatypeDolphinDB Data TypeDolphinDB ExamplesSizeRange
VOIDNULL1
booleanBOOL1b, 0b, true, false10~1
byte
charCHAR'a', 97c1-2 7 +1~2 7 -1
shortSHORT122h2-2 15 +1~2 15 -1
intINT214-2 31 +1~2 31 -1
longLONG22, 22l8-2 63 +1~2 63 -1
realFLOAT2.1f4Sig. Fig. 06-09
floatDOUBLE2.18Sig. Fig. 15-17
dateDATE2013.06.134
monthMONTH2012.06M4
timeTIME13:30:10.0084
minuteMINUTE13:30m4
secondSECOND13:30:104
(datetime)DATETIME2012.06.13 13:30:10 or 2012.06.13T13:30:104
(datetime)TIMESTAMP2012.06.13 13:30:10.008 or 2012.06.13T13:30:10.0088
DATEHOUR2012.06.13T134
timespanNANOTIME13:30:10.0080070068
timestampNANOTIMESTAMP2012.06.13 13:30:10.008007006 or 2012.06.13T13:30:10.0080070068
symbolSYMBOL 4
symbolSTRING"Hello" or 'Hello' or `Hello
guidUUID5d212a78-cc48-e3b1-4235-b4d91473ee8716

Note:

  • There is no DolphinDB equivalent to the byte type of kdb+.
  • The char type of kdb+ is enclosed in double quotes whereas its DolphinDB equivalent, CHAR, is enclosed in single quotes. For example, "c" is treated as a string in DolphinDB.
  • The type indicator of long is "j" in kdb+ whereas the type indicator of LONG in DolphinDB is "l". If a long value has the trailing type indicator "j", for example, 42j, it will not be recognized by DolphinDB.
  • In kdb+, the type indicator of month is "m". In DolphinDB, the type indicator of MONTH is "M" and the type indicator of MINUTE is "m". Therefore, format such as 2006.07m is not accepted in DolphinDB.
  • kdb+ uses an 8-byte datetime type whereas DolphinDB uses a 4-byte DATETIME type.
  • The SYMBOL type of DolphinDB is a special STRING type and is equivalent to enumerated types in C++, Java and C#. SYMBOL stores strings as integers, which greatly improves query performance and storage efficiency. For more information, see DolphinDB user manual.
NULL and INF

Unlike kdb+, DolphinDB does not provide special lexical forms representing positive/negative infinity in various data types. To represent the lexical form for NULL values of integral, floating or temporal data type, you can use 00<data type symbol> . When a data type overflow occurs, DolphinDB treats the data as NULL value of this data type. When an assignment statement or expression uses a function without a return value, you will also get a NULL of VOID type.

In DolphinDB, you can use the isVoid function to check if a NULL value is VOID type. Use isNull or isValid to check the existence of NULL values of any type. If you are not concerned with the data type of a NULL value, it is recommended to use isNull or isValid.

For more information about the initialization, calculation, and usage in vectorized functions, aggregate functions and higher-order functions, see DolphinDB user manual.

Other Datatypes

kdb+ DatatypesDolphinDB Data Types / Data FormsDolphinDB Examples
listANY, matrix, dictionary(1,2,3)
enums
anymapANY DICTIONARY, dictionary{a:1,b:2}
dictionaryANY DICTIONARY, dictionary{a:1,b:2}
tablein-memory table

Note:

  • ANY DICTIONARY is the data type in DolphinDB for JSON.

  • In DolphinDB, a dictionary key must be a scalar while a value can be of any data type in any data forms. Nested dictionaries are supported. When a kdb+ dictinary is printed or iterated through, the system preservers the order in which keys are inserted. By contrast, whether a DolpinDB dictionary is ordered is determined by the ordered parameter during creation. When ordered is set to true, the created dictionary tracks the insertion order of the key-value pairs. By default, ordered is set to false.

  • In kdb+, a matrix is a list of lists all having the same count and is stored in row-major-order. In DolphinDB, MATRIX is an independent data form of which data is stored in column-major-order. To import a kdb+ matrix into DolphinDB, it must be transposed.

  • DolphinDB provides various data forms including scalar, vector, pair, matrix, set, dictionary and table with optimization for specific scenarios. For more information, see DolphinDB user manual.

Data Type Checking Functions

The DolphinDB built-in function typestr returns a string indicating the data type of an object and type returns an integer indicating the data type ID of the object.

Keywords by Category

control

kdb+DolphinDB
dodo-while
exit
ifif-else
whiledo-while

env

kdb+DolphinDB
getenvgetEnv
gtimegmtime
ltimelocaltime
setenv

interpret

kdb+DolphinDB
evaleval
parseparseExpr
reval
showprint
systemshell
valuevalues - Returns values of a dictionary or a tableprint - Prints variable contentsparseExpr - Converts string to metacode for executionexpr - Converts list to metacode for execution

join

Table Joinerskdb+DolphinDB
asof joinaj, aj0, ajf, ajf0, asofaj
equi joinejej, sej
inner joinij, ijfinner join
left joinlj, ljflj, lsj
plus joinpj
union joinuj, ujf
window joinwj, wj1wj, pwj

list

kdb+DolphinDB
countcount - number of non-NULL elementssize - number of all elements
mcountmcount
crossjoin:C, cross(join, X, Y)
cutcut
enlist[], array, bigArray
except
fillsffill, ffill!
firstfirst
lastlast
flipflip, transpose
groupgroups
inin
interintersection(set(X), set(Y))
nextnext
prevprev
xprevmove
razeflat - Converts a matrix or a list of vectors into a one dimensional vector
reversereverse
rotate
sublisthead, tail, subarray, subtuple
svconcat - Forms new string by concatenating strings/char:
tiltil
unionunion(set(X), set(Y)) or distinct(join(X, Y))
vssplit - Splits a string
where

logic

kdb+DolphinDB
allall
andand
anyany
notnot
oror

math

kdb+DolphinDB
absabs
coscos
acosacos
sinsin
asinasin
tantan
atanatan
avgavg, mean
avgscumavg
ceilingceil
corcorr
cov
scovcovar
deltasdeltas
devstdp
mdevmstdp
sdevstd
divdiv
emaema
expexp
xexppow
floorfloor
invinverse
loglog
xlog
lsq
mavgmavg
wavgwavg
maxmax
maxscummax
mmaxmmax
medmed
minmin
minscummin
mminmmin
mmudot
modmod
sumsum
sumscumsum
msummsum
wsumwsum
negneg
prdprod
prdscumprod
randrand
ratiosratios
reciprocalreciprocal
signumsignum
sqrtsqrt
withinbetween, in
varvarp
svarvar

Note:

  • Unlike kdb+, the TA-lib functions in DolphinDB ignore the NULL values at the beginning of the object in calculation and keep them in the output. The calculation with the sliding windows starts from the first non-NULL value.

    For count-based windows, the calculation will be performed only when elements fill the window completely. This means that the result of the first (window - 1) elements will be NULL.

  • In DolphinDB, the windowing logic of the moving (m-) functions is to return NULL for the first (window-1) windows by default. If you want the same result as the kdb+ equivalents, specify the parameter minPeriod as 1.

  • For the DolphinDB function ratios, if the input is a vector, the first element of the result is always NULL.

meta

kdb+DolphinDB
attr
nullisNull, isNothing, isVoid
tablesgetTables
typetype, typestr, form
view, views

query

kdb+DolphinDB
deletedelete, sqlDelete
execexec
fbycontextby
selectselect
updateUpdate records: update, sqlUpdate Append records: insert into Append columns: alter, addColumn

sort

kdb+DolphinDB
ascsort, sort!
iascisort, isort!
xascsortBy!
bin, binrbinsrch
descsort, sort!
idescisort, isort!
xdescsortBy!
differvalueChanged
distinctdistinct
rankrank
xbarbar
xrankbucket

table

kdb+DolphinDB
colscolumnNames
xcolrename!
xcolsreorderColumns!
csvCSV delimiter: ,
fkeys, key, keys, xkey
insertinsert into, tableInsert, append!, push!
metaschema
ungroup, xgroup
upsertinsert into, tableInsert, append!, push!
xascsortBy!
xdescsortBy!

text

kdb+DolphinDB
likelike, ilike
lowerlower
upperupper
trimtrim
ltrimltrim
rtrimrtrim
md5md5
ssregexFind
ssrregexReplace
stringstring

User-Defined Functions

kdb+DolphinDB
- An example of a signed lambda: f:{[x;y](x*x)+(y*y)+2*x*y}- An example of an unsigned lambda: {(x*x)+(y*y)+2*x*y}[20;4]- An example of a named function: def func(x, y) { return (x*x)+(y*y)+2*x*y; }- An example of an anonymous function: def (x, y) { return (x*x)+(y*y)+2*x*y; }- Examples of lambda expressions (a function definition with only one statement): -- def func(x,y): (x*x)+(y*y)+2*x*y;-- def(x,y): (x*x)+(y*y)+2*x*y;-- def(x,y)-> (x*x)+(y*y)+2*x*y;``x->x*x

Note:

  • In kdb+, function notation is also known as the lambda notation and the defined functions as lambdas. The term lambda is used to denote any function defined using the lambda notation. In DolphinDB, "lambda expression" only refers to a function definition with one statement.
  • In kdb+, functions with 3 or fewer arguments may omit the signature and instead use default argument names x, y and z to mean the first, second, and third arguments. In DolphinDB, functions must have an explicit list of parameters.
  • DolphinDB supports specifying default parameter values. The parameter with a default value is not mutable, and the default value must be a constant. Qualify a parameter by the "mutable" keyword if it can be modified within function body.
// default argument
def func(x=1): x+1;

// mutable argument
def i2t(mutable tb) { return tb.replaceColumn!(`time, time(tb.time/10); }
  • DolphinDB user-defined functions supports JIT compilation for performance optimization.
  • In DolphinDB, user-defined functions can call built-in functions in various forms:
    • standard function call format: <func>(parameters)
    • object-method call format: x.<func>(parameters) where x is the first parameter.
    • if the user-defined function has only one or two parameters, we can also use the infix notation.
def f(x, y): x+y+1;
p = 2;
q = 3;

f(p, q);   // standard notation

p.f(q);    // postfix notation

p f q;     // infix operator

Overloaded Glyphs and Operators

Like kdb+, there are two types of operators in DolphinDB: unary (performs an action with a single operand) and binary (performs an action with two operands). DolphinDB supports many types of basic operators, including arithmetic operators, Boolean operators, relational operators, membership operators, etc., as well as a variety of operational data types, including scalars, vectors, sets, matrices, dictionaries, and tables.

Evaluation Order: While kdb+ expressions are evaluated right-to-left with no rules for operator precedence, DolphinDB adopts the traditional notion of operator precedence adopted by most programming languages, and operators with the same precedence are evaluated from left to right.

. dot

kdb+ SemanticsDolphinDB
Applycall()
Index[]
Traptry-catch
Amend

@ at

kdb+ SemanticsDolphinDB
Apply Atcall()
Index At[]
Trap Attry-catch
Amend At

$ dollar

kdb+ SemanticsDolphinDB
Cast$, cast()
TokData type conversion functions can interpret a string directly
Enumerate
Padlpad(), rpad()
mmu**

! bang

kdb+ SemanticsDolphinDB
Dictdict()
Enkey, Unkey, Enumeration, Flip Splayed, internal
Displayprint()
Updateupdate(), update!()
Deleteerase!()

? query

kdb+ SemanticsDolphinDB
Findin(), find()
Rollrand()
Deal, Enum Extend
Selectselect
Exec, Simple Execexec
Vector Conditionaliif()

## hash

kdb+ SemanticsDolphinDB
Taketake()
Set Attribute

Operators

kdb+ Operatorkdb+ SemanticsDolphinDB
+Add+
-Substract-
*Multiply*
%Divide\
=Equals==
<>Not Equals!=
~Match
<Less Than<
<=Up To<=
>=At Least>=
>Greater Than>
| Greater
| OR
&Lessermin()
&AND&
_Cutcut()
_Dropdrop()
:Assign=
^Fillffill(), ffill!()
^Coalesceappend!()
,Join<-
'Compose
0: 1: 2:File Text, File Binary, Dynamic LoadloadText(), saveText()
0 ±1 ±2 ±nwrite to console, stdout, stderr, handle nprint()

Iterators

The iterators (once known as adverbs in kdb+) are built-in higher-order functions in DolphinDB, which can extend or enhance a function or an operator: they take a function and some objects as input. In general, the input data is dissembled into multiple pieces (which may or may not overlap) in a preset way at first; then the individual pieces of data are applied to the given function to produce results one at a time; finally all individual results are assembled into one object to return.

In both kdb+ and DolphinDB, the input data for a higher order function can be vectors, matrices, tables, scalars and dictionaries. In DolphinDB, a vector is dissembled into scalars, a matrix into columns (vectors), and a table into rows (represented by dictionaries). In the assembly phase, the results of scalar type are merged to form a vector, vectors to a matrix, and dictionaries to a table. A higher-order function iterates over a vector by elements, a matrix by columns, and a table by rows.

' quote

kdb+ SemanticsDolphinDB
Each, eacheach() or :E
Case
Each Parallel, peachpeach()
Each Prior, prioreachPre() or :P

Other Iterators

kdb+ Glyphskdb+ SemanticsDolphinDB
/:Each RighteachRight() or :R
\:Each LefteachLeft() or :L
/Over, overreduce() or :T
\Scan, scanaccumulate() or :A

Evaluation Control

kdb+kdb+ SemanticsDolphinDB
.[f;x;e]Traptry-catch
@[f;x;e]Trap-Attry-catch
:Returnreturn
'Signalthrow
do do-while
exit quit
while do-while
if if-else
$[x;y;z]Condif-else
x:yAssign<variable>=<object>
x[i]:yIndexed assign<variable>[index]=<object>
x op:y, op:[x;y] or x[i]op:y, op:[x i;y]Assign through operatorThe following assignments operators are supported:+=, -=. *=, /=, \=
\tTimertimer

Note:

  • DolphinDB supports multiple variable assignment, which allows us to assign multiple variables at the same time in one single line of code.

  • DolphinDB uses "&" to indicate assignment by reference. Assignment by reference does not make a new copy of the original value, but points to the original value to avoid unnecessary object copying. When swapping two variables, assignment by reference is more efficient than assignment by value.

    n=20000000;
    x=rand(200000.0, n);
    y=rand(200000.0, n);
    
    timer x, y = y, x;        // Time elapsed: 1240.119 ms
    timer {&t=x;&x=y;&y=t;}      // Time elapsed: 0.004 ms
  • In DolphinDB, you can manually release variables or functions from memory by using the undef command or the expression <variable>=NULL. For more information, see DolphinDB User Manual.