Implement High-Performance Backtesting in C++ with the Order Matching Simulator Plugin

Backtesting is a critical part of quantitative trading research and development. Before a quantitative strategy is applied to live trading, its performance on historical data must be evaluated through backtesting. In medium- and high-frequency strategy backtesting, you cannot simply assume that every order is filled in full at the current price or the end-of-day price. Instead, you need an order matching simulator to model the actual trading process, including whether an order can be filled, the execution price, trading volume, and market impact. DolphinDB provides an order matching simulator plugin that supports Level-2 tick-by-tick and snapshot market data from the Shanghai and Shenzhen stock exchanges. It delivers high-precision order matching consistent with exchange rules under the “price priority, time priority” principle, supports matching modes based on multiple types of market data, and offers extensive order matching configuration options to simulate real-world trading conditions. It also supports medium- and high-frequency quantitative trading strategy development and testing in the DolphinDB scripting language, Python, and C++, providing a high-performance and highly extensible backtesting solution.

Swordfish is a high-performance analytical computing library designed specifically for the financial industry, with excellent in-memory processing capabilities and optimized computational performance. In addition to a wide range of general-purpose computing functions, it supports real-time streaming data processing and user-defined functions to meet the needs of complex analytics and low-latency stream processing. Swordfish runs on any platform that supports C++, and users can invoke its APIs directly in C++ code for efficient computation.

The DolphinDB C++API can connect to the DolphinDB server and a C++ client, enabling bidirectional data transfer and remote script execution. It lets you conveniently use DolphinDB in C++ programs for data processing, analysis, modeling, and more, helping you accelerate these workloads with DolphinDB’s excellent computing performance and powerful storage capabilities.

This tutorial explains how to encapsulate the order matching simulator plugin in C++ to build a simple, extensible backtesting framework with high-precision order matching, which can be easily integrated into an existing C++ backtesting or simulation system. This tutorial defines a unified event-driven interface and provides two implementations: Swordfish and C++API. The applicable scenarios for the two approaches are as follows:

Table 1-1 Scenario comparison between the Swordfish and C++API backtesting framework implementations

Implementing a medium- and high-frequency quantitative trading strategy backtesting platform mainly involvesthe following three key parts:

• Market data replay

• Order matching simulation

• Strategy development and backtest performance evaluation

Based on this design, you can define the overall workflow for backtesting in C++ with DolphinDB as the data source:

1. Create and subscribe to a remote market data stream table.

2. Replay market data into the stream table.

3. Traverse and parse the data in the subscription callback, then write the parsed market data to the order matching simulator.

4. Trigger the market data callback, implement strategy logic in the callback, and place orders or perform other actions.

5. The order matching simulator outputs execution details and other information, triggers the relevant business callbacks, and lets you implement strategy logic in those callbacks.

6. Output performance metrics after the backtest ends.

The following diagram shows the system architecture for implementing user-defined backtesting with Swordfish:

The following diagram shows the system architecture for implementing user-defined backtesting with C++ API:

Based on the capabilities provided by DolphinDB’s order matching simulator interfaces, this tutorial defines market data and trading interfaces. The design mainly includes the following functional modules:

• Configuration module:Handles settings for backtesting framework instances, including remote connection settings, market data type settings, and concurrency-related settings.

• Active call interface module: Provides interfaces that you can call directly in code, including interfaces to create order matching simulator instances, replay data, place orders, and cancel orders.

• Callback interface module: Provides callback-based interfaces, including market data callbacks, order acknowledgments, and execution notifications.

Note:

Unless otherwise specified, the configuration items and interfaces introduced in this section can be used in both Swordfish and the C++ API.

Based on the system design and interface design, the following functional modules are designed to implement the backtesting and simulated trading system:

• Remote connection: Connect remotely to DolphinDB and execute scripts to implement interfaces for data replay, remote environment cleanup, and related operations.

• Plugin loading and usage: Load the order matching simulator plugin, encapsulate its script methods as C++ interfaces, and implement active interfaces such as order submission and cancellation, as well as callbacks for order notifications and trade notifications.

• Streaming data subscription, parsing, and callbacks: Subscribe to DolphinDB stream tables as data sources, parse streaming data, and write it to the order matching simulator to implement market data callbacks.

The following sections describe the implementation logic for Swordfish and the C++ API separately.

Based on the design and implementation described above, this section explains the implementation logic of a simple backtesting strategy through core code examples. Specifically, the strategy automatically generates a limit order after processing each item of snapshot data in real time. For the complete code, see the attached source files.

1. Define the interface class and implement the strategy

   Derive the CustomInterface class from the Interface class and override the onQuote method. When a piece of market data arrives, call getMatchEngine to get the engine instance, and then call submitOrder to place an order:

   class CustomInterface: public Interface {
     public:
       CustomInterface(unordered_map<string, string> config): Interface(config) {}
   
       void onQuote(const ConstantSP &quote) override {
           ...
           engine = getMatchEngine(new String(matchEngineName_));
           string symbol = quote->getMember("symbol")->getString();
           OrderField order;
           order.symbol = symbol;
           order.orderType = 5;
           order.price = bidPrice->getLong();
           order.orderQty = 200;
           order.direction = 1;
           submitOrder(engine, order);
       }
   };

2. Create the order matching simulator: In the main function, create an instance of the CustomInterface class, use the getExampleMatchEngineArgs method to get sample arguments for the order matching simulator, and then call createMatchEngine to create the order matching simulator.

   interface = new CustomInterface(config);
   auto args = interface->getExampleMatchEngineArgs();
   engine = interface->createMatchEngine(args);

3. Subscribe to market data: Use the subQuoteTable method to subscribe to market data thereby enabling the market data callback.

   interface->subQuoteTable();

4. Replay market data

   Use the replayQuoteToMatchEngine method to replay data for a specified stock symbol and date-time range into the engine

   interface->replayQuoteToMatchEngine(codes, startDate, endDate, engine, -1);

5. Wait for the backtest to finish

   Use the isBacktestEnd method to wait for the backtest to finish.

   interface->isBacktestEnd(true);

6. Display performance metrics

   For example, the getPosition method can be used to display positions. The CustomInterface class can also be used to implement and output user-defined metrics.

   cout << interface->getPosition() << endl;

The test environment used in this article is configured as follows:

• DolphinDB v3.00.2.2, single node

• Swordfish v3.00.2 ABI0

• CPU: Intel(R) Xeon(R) Silver 4216 CPU @ 2.10GHz, 64 cores

• Memory: 400 GB

• Disk: SSD, 12 Gbps

We measured the total time taken by the Swordfish-based backtesting framework to perform full-speed replay with multiple concurrent tasks. The order placement strategy submits one limit order for each snapshot quote. The performance statistics are as follows:

Table 6-1 Performance statistics for the backtesting framework implemented with Swordfish

As shown, multithreaded parallelism can effectively improve performance, but the improvement is not linear because the server still needs time to replay data into the stream table and send it to the client.

In the C++ API implementation, the order matching simulator runs on the remote DolphinDB server, so operations such as order placement and cancellation incur network latency. During accelerated replay, the server may already have replayed to a later point in time by the time the client operation is executed. Therefore, only real-time replay is supported, and no accelerated replay performance test can be performed.

This article demonstrates how to implement high-performance backtesting in C++ by using Swordfish or the C++ API to invoke the order matching simulator plugin, and how to encapsulate a unified interface for easy integration of DolphinDB backtesting logic into existing C++ backtesting systems. Among the two approaches, Swordfish can run the order matching simulator locally, offering greater flexibility, support for accelerated replay, and better overall performance.

CppApiBacktest.zip

SwordfishBacktest.zip

Note:To compile and run the Swordfish implementation, you must place the Swordfish-enabled license file dolphindb.lic in the asset folder.