DolphinDB Backtesting Engine: Usage Guide and Performance Optimization

The backtesting engine is provided as a plugin. Figure 1-1 shows its logical architecture. The main backtesting workflow is as follows:

1. The backtesting engine receives a data stream replayed in chronological order and internally delivers it to the order matching simulator and the corresponding market data callback function;

2. The market data callback function processes the strategy logic and submits orders;

3. The backtesting engine performs risk checks on orders;

4. The backtesting engine sends orders that pass risk checks to the order matching simulator for order matching;

5. The backtesting engine tracks positions and funds in real time based on order executions. When strategy backtesting is complete, it returns information such as the strategy's returns and trade details.

Using the backtesting engine typically involves the five steps shown in Figure 1-2. First, the backtesting engine provides a range of event functions, including callbacks for strategy initialization and daily pre-market and post-market processing; handlers for tick, snapshot, and OHLC bar subscriptions; and functions that notify you of order change and execution. You can define indicators during strategy initialization and implement user-defined strategy logic in the appropriate callbacks. Next, configure the strategy's market data source, capital, order latency, fill ratio, and other settings. Then, create a backtesting engine based on the strategy and configuration. Next, replay the source data to run the backtesting engine. Finally, call the engine functions to retrieve the backtest results.

The following example demonstrates how to use the backtesting engine.

Implement the user-defined strategy logic:

@state
def pctChg(lastPrice, prevClosePrice){
 return lastPrice\prevClosePrice - 1
}
def initialize(mutable context){
 // Initialization callback
 print("initialize")
 // Subscribe to indicators calculated based on snapshot data
 d = dict(STRING,ANY)
 d["pctChg"] = <pctChg(lastPrice, prevClosePrice)>
 Backtest::subscribeIndicator(context["engine"], "snapshot", d) 
 context["maxPos"] = 500
}
def beforeTrading(mutable context){
 // Daily pre-market callback
 // Get the current trading date from context["tradeDate"];
 print("beforeTrading: " + context["tradeDate"])
 // Use Backtest::setUniverse to change the stock universe for the current trading day
 Backtest::setUniverse(context["engine"], ["000001.XSHE"])
 context["open"] = dict(STRING,BOOL)
}
def onSnapshot(mutable context, msg, indicator){ 
 // Query the current positions
 pos = Backtest::getPosition(context["engine"], msg.symbol)
 longPos = pos.longPosition
 if (indicator.pctChg > 0.01 and longPos <= context.maxPos and
 context["open"][istock] != true){
 Backtest::submitOrder(context["engine"], 
 (msg.symbol, context["tradeTime"], 5, msg.offerPrice[0], 100, 1), "buy")
 context["open"][istock] = true
 }
}
def onOrder(mutable context, msg){}
def onTrade(mutable context, msg){}
def afterTrading(mutable context){}
def finalize(mutable context){}

Configure the strategy's market data source, capital, and fill ratio.

// Configuration parameters
config = {
   startDate: 2022.04.11, // Start date of backtesting
   endDate: 2022.04.11,   // End date of backtesting
   strategyGroup: `stock,
   frequency: 0,
   cash: 100000000,       // Initial capital for the strategy
   commission: 0.00015,
   tax: 0.001,
   dataType: 1,          // Market data type; dataType = 1 indicates snapshot data
   msgAsTable: false,
   "context": {
          maxPos:0,
          open: {
                  A: true
                } 
             }
}

Once you have specified the strategy callback function, you can create a backtesting engine using createBacktester:

// Create a backtesting engine 
callbacks = {
        initialize: initialize,
        beforeTrading: beforeTrading,
        onSnapshot: onSnapshot,
        onOrder: onOrder,
        onTrade: onTrade,
        afterTrading: afterTrading,
        finalize: finalize
}
name = "BackTester1"
engine = Backtest::createBacktester(name, config, callbacks, false)

After obtaining the required market data, you can start strategy backtesting via appendQuotationMsg:

// Write data and start strategy backtesting. messageTable must contain data that meets the engine's requirements.
Backtest::appendQuotationMsg(engine, messageTable)

After the backtest is complete, you can get the results using different engine functions:

// Get backtest results
Backtest::getDailyPosition(engine)// Get daily position data
Backtest::getDailyTotalPortfolios(engine)// Get daily portfolio metrics
Backtest::getReturnSummary(engine)// Get summary for the returns
Backtest::getContextDict(engine)// Get the strategy's logical context
Backtest::getTradeDetails(engine)// Get the trade details table

The DolphinDB backtesting engine is built on DolphinDB's high-performance distributed storage and computing architecture. It supports developing and testing medium- and high-frequency strategies in DolphinScript, Python, or C++. Because the backtesting engine's JIT optimization depends on DolphinDB's native JIT support, currently only DolphinScript supports JIT optimization. When JIT optimization is enabled, strategy callback functions must not use default jit value or partial application. Global variables in the strategy must be declared in the context key of the config dictionary.

Starting from the 3.00.2.1 JIT version, the backtesting engine supports JIT optimization. When creating a backtesting engine, set the jit parameter of createBacktester to true.

Backtest::createBacktester(name, config, eventCallbacks, [jit=false], [securityReference])

On a non-JIT server, the backtesting engine does not support JIT optimization. Setting jit to true results in an error:

[PLUGIN::BACKTEST] This version of server doesn't support JIT function..

The backtesting engine's strategy event functions accept the following parameters: context for specifying global strategy variables, msg for market data, and indicator for market data indicators.

• context is a dictionary. To use JIT optimization, user-defined global variables in context cannot be tables; all other DolphinDB data types are supported.

• msg is a dictionary. In the onSnapShot and onTick functions for high-frequency strategy backtesting, use msg.lastPrice and msg.price to obtain the latest price. In the onBar function, the msg dictionary keys are instrument symbols, and the values are the corresponding market data. For example, use msg[istock].close and msg[istock].low to obtain the latest closing price and latest low price.

• The market data indicator's data type must match that of msg for the corresponding market data.

If your strategy needs to save certain market data in real time, you can use the following approach. In this example, the highest and lowest price indicators are saved in real time in the onSnapShot function.

def onSnapshot(mutable context, msg, indicator){
        context["highPrice"] = max(msg.highPrice, context["highPrice"])
        context["lowPrice"] = min(msg.lowPrice, context["lowPrice"])
} 
// The lifetime of context does not end after each callback completes;
// Variables stored in context remain available throughout the entire callback process unless they are modified by another callback function.

For better performance, the same object instance is used for msg in every event function call. Therefore, when using it in a strategy function, do not retain it for later use. Therefore, we do not recommend storing msg in an event function, as shown below.

def onBar(mutable context, msg, indicator){
        context["msg"] = msg
}

By default, the backtesting engine relies on DolphinDB’s order matching simulator, a high-precision system modeled on exchange matching practices that prioritizes orders by price and then by time. If your strategy logic does not require a high-precision order matching system—for example, some medium- and low-frequency strategies—and you want orders to be filled at the prices submitted by the strategy, set matchingMode to 3 when creating the engine.

config["matchingMode"] = 3

This setting executes orders at their submitted prices while still enforcing limit-up and limit-down restrictions. In other words, a buy order submitted at the limit-up price fails, and a sell order submitted at the limit-down price fails.

The backtesting engine supports not only complex real-time indicator calculation but also replaying pre-calculated indicators together with market data to implement strategy logic.

Backtest::subscribeIndicator(engine, marketDataType, metrics)

indicatorDict = dict(STRING, ANY)
indicatorDict["mavg"] = <mavg(lastPrice, 20)>

config["enableIndicatorOptimize"] = true // Enable optimization for indicator calculation
config["dataRetentionwindow"] = "ALL" // Retain all market data

@state
def myAtr(high, low, close, m = 14, n = 10) {
 prevClosePrice = prev(close)
 tr = rowMax([high - low,abs(high - prevClosePrice),abs(low - prevClosePrice)])
 atr = ema(tr, m)
 mAtr = mavg(atr,n)
 return atr,mAtr
}
def initialize(mutable contextDict){
 d = dict(STRING, ANY)
 d["ATR"] = <myAtr(signal[0], signal[1], signal[2], 14, 10)[0]>
 d["mATR"] = <myAtr(signal[0], signal[1], signal[2], 14, 10)[1]>
 d["RSI"] = <ta::rsi(signal[2], 14)>
 // Signal comes from the market data and is an array vector consisting of three columns of market data
 // fixedLengthArrayVector([backwardFactorHighPrice,backwardFactorLowPrice,backwardFactorClosePrice])
 Backtest::subscribeIndicator(contextDict["engine"], "kline", d)
}

update quoteTB set signal = fixedLengthArrayVector([ATR, prev(ATR), RSI]) context by symbol

def onBar(mutable contextDict, msg, indicator){
        keys = msg.keys()
        for(i in keys){
                price = msg[i].close
                signal = msg[i].signal
                atr = signal[0]
                matr = signal[1]
                rsi = signal[2]
        }
}

def onBar(mutable context, msg, indicator){ 
        closeList = context["closeList"]         // Record historical closing prices in real time
        lowPriceList = context["lowPriceList"]   // Record historical low prices in real time
        highPriceList = context["highPriceList"] // Record historical high prices in real time
        keys = msg.keys()
        for(i in keys){
                istock = msg[i].symbol
                price = msg[i].close
                if( type(closeList[istock]) == VOID){
                        closeList[istock] = array(DOUBLE, 0, 14)
                        lowPriceList[istock] = array(DOUBLE, 0, 14)
                        highPriceList[istock] = array(DOUBLE, 0, 14)
                }
                closeList[istock] = closeList[istock].append!(price)
                lowPriceList[istock] = lowPriceList[istock].append!(msg[i].low)
                highPriceList[istock] = highPriceList[istock].append!(msg[i].high)
                if(closeList[istock].size() >= 14){
                        n = size(closeList[istock])
                        closeList[istock] = closeList[istock][n - 14:]
                        lowPriceList[istock] = lowPriceList[istock][n - 14:]
                        highPriceList[istock] = highPriceList[istock][n - 14:]
                }
                atr_ = myatr(highPriceList[istock], lowPriceList[istock], 
            closeList[istock], 14, 10)
                atr = atr_[0]
                matr = atr_[1]
                rsi = RSI(closeList[istock], 14) 
        }
        context["closeList"] = closeList
        context["lowPriceList"] = lowPriceList
        context["highPriceList"] = highPriceList
}

When backtesting a strategy, the backtesting engine supports replaying market data in streaming mode. Batch replay can significantly improve performance. In this example, snapshot data is used as the market data. The strategy buys 100 shares when the latest price for the current day is 1% higher than the previous day's closing price. The maximum position is limited to 500 shares, and the strategy opens a position no more than once per day.

def onSnapshot(mutable context, msg, indicator){        
        // Query the current position
        pos = Backtest::getPosition(context["engine"], istock)
        if (indicator.pctChg > 0.01 and pos.longPosition <= context["maxPos"] and
        context["open"][istock] != true){
        // Backtest::submitOrder is the order placement function provided by the backtesting engine
                    Backtest::submitOrder(context["engine"], 
                    (msg.symbol, context["tradeTime"], 5, msg.offerPrice[0], 100, 1), "buy")
                    context["open"][istock] = true
        }
}

Based on the strategy above, we simulate the insertion of backtest data and compare the performance of writing 100,000 snapshot records one at a time with that of batch writing. Compared with writing records one at a time, batch writing improves performance by approximately fivefold or more. Therefore, batch writing is recommended during backtesting. When writing market data in batches, you can read cached data in batches to accelerate market data parsing.

n = 100000
t = table(take("000001.XSHE",n) as symbol, take("XSHE",n) as symbolSource, 
take(concatDateTime(2022.04.11, transFreq(10:00:00..11:00:00, "3S").distinct()), n) as timestamp, 
randUniform(7.0, 10.0, n) as lastPrice, randUniform(9.0, 10.0, n) as upLimitPrice, 
randUniform(7.0, 8.0, n) as downLimitPrice, take(10000, n) as totalBidQty, take(10000, n) as totalOfferQty,
take(arrayVector([10], [6.9, 6.8, 6.7, 6.6, 6.5, 6.4, 6.3, 6.2, 6.1,6.0]), n) as bidPrice,
take(arrayVector([10], [800, 900, 1000, 1100, 1200, 1000, 1000, 1000, 1000, 1000]), n) as bidQty, take(arrayVector([10],
[7.1, 7.2, 7.3, 7.4, 7.5, 7.6, 7.7, 7.8, 7.9, 8.0]), n) as offerPrice,
take(arrayVector([10], [1000, 1000, 1000, 1000, 1000, 1000, 1000, 1000, 1000, 1000]), n) as offerQty,
take(arrayVector([1], [double()]), n) as signal, randUniform(6.0, 7.0, n) as prevClosePrice)

messageTable.append!(t)
timer{Backtest::appendQuotationMsg(engine, messageTable)}
for (i in 0:messageTable.size()){
        tmp = select * from messageTable where rowNo(symbol) = i
        t = evalTimer(Backtest::appendQuotationMsg{engine2, tmp})
        tsRun = tsRun.append!(t)
}
try{Backtest::dropBacktestEngine(strategyName)}catch(ex){print ex}
tsRun = array(DOUBLE, 0, 10)
engine2 = Backtest::createBacktester(strategyName, userConfig, callbacks,, basicInfo)
// Write every 10 records
a = 0
for (i in 1..messageTable.size()){
        if(a >= size(messageTable)){continue}
        tmp = select * from messageTable where rowNo(symbol) >= a and rowNo(symbol) < (a + 10*i)
        t = evalTimer( Backtest::appendQuotationMsg{engine2,tmp})
        tsRun = tsRun.append!(t)
        a = a + 10*i
}
print(sum(tsRun))

DolphinDB and Python differ in how assignment with the equals sign works. In Python, '=' creates a reference: it simply makes one variable point to an existing variable, requiring neither additional memory allocation nor data copying. In DolphinDB, the assignment behavior of '=' depends on the object and its flags. When creating a dictionary, shallow copying is used by default. Therefore, assigning with '=' copies the object. If the transient flag is set, this becomes a deep copy that recursively copies all child objects under the object. In backtest callback functions, the market data msg uses deep copying by default. Assignment with '=' can therefore significantly reduce backtest speed, so avoid using '=' to directly reference the entire msg or its fields whenever possible (except indicator).

For most CTA strategies, high-precision order matching and real-time risk control are key to ensuring the strategy's success and effective execution. The strategy's primary logic may be based on minute-level OHLC bars, while order fills depend on a high-precision order matching engine or the strategy's take-profit and stop-loss logic depends on real-time, tick data. DolphinDB can implement a backtesting engine that uses real-time tick data as its input while triggering the onBar callback at the configured frequency. Strategies can use the high-frequency callback function onSnapshot to perform real-time take-profit and stop-loss operations. The relevant configuration items are callbackForSnapshot and frequency.

In addition, the backtesting engine includes built-in algorithmic orders for real-time take-profit and stop-loss execution. Set enableAlgoOrder to true to enable algorithmic orders:

Backtest::submitOrder(engine, msg, [label=""], [orderType = 0])

The msg tuple contains the instrument symbol, exchange symbol, time, order type, order price, stop-loss price, take-profit price, order quantity, buy/sell direction, slippage, order validity, and order expiration time. The valid values for orderType are:

• 0: Default value, indicating a standard order

• 5: Limit take-profit/stop-loss order

• 6: Market take-profit/stop-loss order

• 8: Two-sided quote order (supported for futures and options only)

• 9: Automatic order placement; the buy/sell direction can only be set to 1 or 2

• 10: Time-Weighted Average Price (TWAP) algorithmic order, which splits a large order evenly into multiple smaller orders for execution at regular time intervals

• 11: Volume-Weighted Average Price (VWAP) algorithmic order, which splits a large order into multiple smaller orders according to the distribution of trading volume

• 12: Trailing take-profit/stop-loss order

Detailed Implementation Example:

The following example uses an FX CTA trend-following strategy that uses Bollinger Band breakouts and RSI as entry signals to illustrate how to synthesize real-time market data from high-frequency quotes. This strategy calculates the Bollinger Bands and RSI indicators every hour and uses real-time tick data for risk monitoring and take-profit/stop-loss execution. Its logic is as follows:

• Open a long position: RSI is currently above 70, and the Bollinger Bands are moving upward and breaking through the upper band.

• Open a short position: The current RSI is below 30, and the Bollinger Bands are moving downward and breaking through the lower band.

If any pending long or short orders exist, use tick data to determine whether to cancel them or trigger take-profit and stop-loss operations.

Code Implementation:

First, subscribe to the indicators:

use ta
def initialize(mutable context){
        print("initialize")
        d = dict(STRING,ANY)
        d["rsi"] = <ta::rsi(lastPrice, 11)>
        d["bhigh"] = <ta::bBands(lastPrice, 20, 2, 2, 0)[0]>
        d["bmid"] = <ta::bBands(lastPrice, 20, 2, 2, 0)[1]>
        d["blow"] = <ta::bBands(lastPrice, 20, 2, 2, 0)[2]>
        Backtest::subscribeIndicator(context["engine"], "snapshot_kline", d)
        
}

Strategy logic: Buy when the RSI is above 70 and the Bollinger Bands are moving upward and breaking through the upper band. Sell when the RSI is below 30 and the Bollinger Bands are moving downward and breaking through the lower band. Submit algorithmic orders to perform real-time take-profit and stop-loss operations.

def onBar(mutable context, msg,indicator){
        istock=msg.keys()[0]
        if(indicator[istock].rsi <= 0){ return }
        position=Backtest::getPosition(context["engine"],istock)
        longpos = position.longPosition
        shortpos = position.shortPosition
        if(indicator[istock].rsi >70. and msg[istock].offerPrice[0]>indicator.bhigh and msg[istock].close>msg[istock].open){
                if(longpos <1){
                        orderId=Backtest::submitOrder(context.engine, (istock,msg[istock].symbolSource ,
                        context.barTime,5, round(msg[istock].offerPrice[0],5), 
                        msg[istock].offerPrice[0] -context.sl+context.Slippage , msg[istock].offerPrice[0]+ context.tp+context.Slippage, 
                        2, 1,context.Slippage, 0, context.barTime+36000000),"openBuy", 5)
                        return
                }
        }
        if(indicator[istock].rsi<30. and msg[istock].bidPrice[0]<indicator.blow and msg[istock].close<msg[istock].open){
                if(shortpos <1){
                        orderId=Backtest::submitOrder(context.engine, (istock,msg[istock].symbolSource,
                        context.barTime,5, round(msg[istock].bidPrice[0],5), 
                        msg[istock].bidPrice[0]+context.sl-context.Slippage, 
                        msg[istock].bidPrice[0] - context.tp-context.Slippage, 2, 2, context.Slippage , 0, 
                        context.barTime+36000000),"openSell", 5)
                        return
                }
        }
}

Performance test: In the example above, processing 29,317,091 market data records and 168 orders took 11 seconds.

When JIT optimization is enabled, strategy callback functions must not use default jit value or partial application. Global variables defined in the strategy must be declared in the context configuration item in config. JIT optimization does not support direct assignment to nested dictionary entries.

def onSnapshot(mutable context, msg, indicator = NULL){}// JIT compilation fails for this function

def onSnapshot(mutable context, msg, indicator, myParam){}// Add parameters to the strategy callback function
eventCallbacks["onSnapshot"] = onSnapshot{,,,myParam}

context = dict(STRING, ANY)
context["myParam"] = myParam
config["context"] = context

config = dict(STRING, ANY)
config["startDate"] = 2023.01.01 // Start date of backtesting
config["endDate"] = 2024.10.30 // End date of backtesting
config["strategyGroup"] = "stock" // Strategy type
// Configure global variables for the strategy
context = dict(STRING, ANY)
context["initPrice"] = 0.
context["feeRatio"] = 0.00002
context["alpha"] = 0.01
context["M"] = 0.01
context["highPrice"] = dict(STRING, ANY)
config["context"] = context

JIT: Data type does not match when assign value to attribute of ...

def onBar(context, msg, indicator){
 context["test"][istock] = msg.close
}

Attribute [("test","000001.XSHE")] doesn't exist

def onBar(context, msg, indicator){
 temp = context["test"]
 temp[istock] = msg.close
 temp["test"]=temp
}

def fun() {
    return 1., 2.
}
def onSnapshot(context,msg,indicator){
    // Unsupported method:
    a,b = fun()
    // Error: codegen for statement type MULTIASSIGN not supported: a, b fun()
    
    // Supported method:
    tmp = fun()
    a = tmp[0]
    a = tmp[1]
}