Portfolio Backtesting with Weight Files

This strategy aims to simulate the process of actual portfolio weight adjustments at specific intervals (daily, weekly, monthly, or quarterly) to evaluate the return and risk performance of different weight allocation schemes in historical market environments. The core idea is that, at the beginning of each rebalancing period, adjust the position ratio of each stock according to predefined or weight files to align portfolio positions with target weights, thereby achieving systematic asset allocation and risk control.

In stock portfolio management and index enhancement strategies, portfolio rebalancing at fixed frequencies is a common and critical investment management technique. Unlike active trading strategies based on real-time signals, such passive portfolio rebalancing strategies do not rely on immediate market predictions but focus on verifying the long-term stability and diversification effects of the weight allocation itself. Researchers can independently evaluate the performance differences of weight schemes generated by different factor scoring, sector rotation, or optimization algorithms over historical periods, thereby providing empirical validation for asset allocation models or quantitative stock selection models.

To achieve the above objectives, this tutorial uses DolphinDB's high-performance backtesting engine to build a portfolio backtesting system. The system can efficiently load historical market data including multiple securities, parse external weight files, and execute rebalancing operations at specified frequencies. Through the engine's event-driven framework, the strategy can compare differences between current positions and target weights daily or at specified intervals, automatically submit buy/sell orders, and record the trade execution process. Combined with backtesting results, users can further analyze portfolio net value changes, volatility, maximum drawdown, Sharpe ratio, and other performance indicators, providing quantitative evaluation basis for the effectiveness and stability of different rebalancing strategies.

The implementation of medium-to-high-frequency quantitative trading strategy backtesting mainly includes three important components: sequential market data replay, order matching, and strategy development with backtesting performance evaluation. However, implementation often faces the following challenges:

First, massive medium-to-high-frequency trading data places extremely high demands on the query and computation performance of backtesting engines.

Second, to ensure effective strategy evaluation and optimization, the backtesting process should simulate actual trading conditions as closely as possible, including factors such as order execution, price, volume, and market impact.

Finally, backtesting engines should have flexible architectures that can support the implementation of various trading strategies and technical indicators and are easily extensible to adapt to different markets and trading requirements.

To address the above challenges, DolphinDB provides an easily extensible medium-to-high-frequency quantitative trading strategy backtesting solution based on its high-performance distributed storage and computing architecture. The solution implements three major functions: market data replay, simulated matching, and medium-to-high-frequency backtesting, supporting strategy development and testing in DLang, Python, or C++. Specifically, this solution includes the following three modules:

• Data replay: Supports replaying data from one or multiple distributed tables with different structures into streaming tables in strict time order or according to specified multi-column sorting orders, thus addressing the integration of research and production in factor computation and quantitative strategy development.
• Simulated matching: Supports Shanghai and Shenzhen exchange Level-2 tick-by-tick data and snapshot data. The matching engine follows the exchanges' "price priority, time priority" rules to achieve high-precision matching. And multiple matching modes are provided based on different market data types, along with rich configuration options to simulate real trading conditions.
• Backtesting: Supports strategy backtesting based on tick-by-tick, snapshot, minute, and daily market data, with customizable indicators to obtain backtesting returns, positions, trade details, and other information. Tick-by-tick and snapshot data support high-precision strategy backtesting, enabling integrated strategy verification for simulation and backtesting.

These three modules are highly compatible with external solutions. Even if users have already implemented solutions for certain components, DolphinDB's solutions can integrate with them to form a complete backtesting solution.

First, check whether securities meet trading conditions through user-defined functions (whether suspended on that day, whether hitting the upper or lower limit, whether trading volume and price are valid); then, conduct quantity verification according to market rules, rounding and correcting target shares according to the exchange's minimum trading unit and board lot sizes; finally, under capital constraints, calculate executable buy/sell quantities from the available funds to obtain actual order quantities.

def isBuyAble(data){
    if(data.volume <= 0.001 or (data.upLimitPrice == data.close)){
        return false 
    }
    return true
}
def isSellAble(data){
    if(data.volume <= 0.001 or (data.downLimitPrice == data.close)){
        return false 
    }
    return true
}
def checkBuyVolume(istock, num){
    if(num <= 0) {return 0 }
    if(istock.substr(0,2) == "68" and num >= 200){
        return num
    }
    else if(istock.substr(0,2) == "68"){ return 0}
    return floor(int(num)/100)*100
}
def getBuyAbleVolume(istock, cash, price){
    if(price <= 0. ) {return 0 }
    if(istock.substr(0,2) == "68" and floor(cash/price) >= 200){
        return floor(cash/price)
    }
    else if(istock.substr(0,2) == "68"){ return 0}
    return floor(floor(cash/price)/100)*100
}

Next, implement the strategy's main logic in the OHLC callback function onBar. The msg parameter in this function is a dictionary that receives OHLC data from the backtesting engine, with keys as security names and values as the corresponding market data. The strategy first obtains the current account's total equity through the getTotalPortfolios interface and current positions through getPosition. Based on equity and positions combined with target weights and current prices, weights are converted into target positions, and then rebalancing operations are executed in the order of "sell first, buy later" to realign portfolio positions with the target weight. The following code demonstrates the specific calculation and rebalancing logic:

def onBar(mutable context, msg, indicator){
	weightsToday = context["targetWeight"][date(context["tradeDate"])]
	idate = date(context["tradeDate"])
	if(typestr(weightsToday) == "VOID"){ return }	
	totalEquity = Backtest::getTotalPortfolios(context["engine"]).totalEquity[0]
	pos = select symbol, longPosition from Backtest::getPosition(context["engine"])
	pos = dict(pos.symbol,pos.longPosition)
	update weightsToday set targetMv = totalEquity*(1- 2*context["commission"]-context["tax"])*weight
	update weightsToday set open = each(getOpenPrice{msg},symbol)
	update weightsToday set targetPos = iif(open <= 0.,0,iif(symbol.substr(0,2) == "68"and floor(targetMv/open) >= 200,
	floor(targetMv/open),iif(symbol.substr(0,2) == "68",0,floor(floor(targetMv/open)/100)*100)))
	update weightsToday set Pos = nullFill(pos[symbol],0)
	sell = select symbol,Pos-targetPos as num from weightsToday where targetPos-Pos < 0
	sell = dict(sell.symbol,sell.num)
	for(istock in sell.keys()){
		data = msg[istock]
		num = sell[istock]
		if(isSellAble(data) and num > 0){
			Backtest::submitOrder(context["engine"], (istock, context["tradeTime"], 5, msg[istock].open, num, 3), "sell to close")	
		}
	}
	buy = select symbol,targetPos-Pos as num from weightsToday where targetPos-Pos>0
	buy = dict(buy.symbol,buy.num)
	for(istock in buy.keys()){
		data = msg[istock]
		num = buy[istock]
		cash = Backtest::getAvailableCash(context["engine"])
		price = data.open
		qty = getBuyAbleVolume(istock, cash*(1- context["commission"]), price)
		num = min(checkBuyVolume(istock, num), qty)
		if(isBuyAble(data) and num > 0){
			Backtest::submitOrder(context["engine"], (istock, context["tradeTime"], 5, price, int(num), 1), "buy to open")	
		}
	}	
}

The start and end dates of backtesting, initial capital, commission and stamp duty, market data type, order delay, etc., can all be flexibly configured through parameters to simulate different market conditions and trading strategy performance. Additionally, the strategy logic context can also set global variables in the strategy through configuration parameters.

For example, in this case, daily target weight data, commission and tax rate required for calculating target positions can be set through context. Thus, the strategy can directly reference these global variables during runtime to complete capital allocation and position adjustment. The following shows specific initial parameter configuration:

config = dict(STRING,ANY) 
config["startDate"] = 2012.02.01        
config["endDate"] = 2022.04.28       
config["strategyGroup"] = "stock"
config["cash"] = 100000000
config["commission"] = 0.0005
config["tax"] = 0.001 
config["dataType"]  = 4
config["matchingMode"] = 3
config["outputOrderInfo"] = true
w = select * from loadTable("dfs://dbweight",`dt) where weight >0. order by tradeDay
dailyWeight = dict(DATE,ANY,true)
for (idate in sort(distinct(w.tradeDay))){
	s = select symbol, weight from w where tradeDay = idate
	dailyWeight[idate] = s
}
context = dict(STRING,ANY)
context["targetWeight"] = dailyWeight
context["commission"] = config["commission"]
context["tax"] = config["tax"]
config["context"] = context	

After setting the engine name, configuration items, corresponding callback functions, basic contract information table, and other relevant parameters, create a backtesting engine instance through the createBacktester interface. The fourth parameter of createBacktester indicates whether to enable JIT optimization. The default value is false, meaning not enabled. If JIT optimization needs to be enabled, simply set this parameter to true.

strategyName = "Backtest_portfolio"
callbacks = dict(STRING,ANY)
callbacks["onBar"] = onBar
try{Backtest::dropBacktestEngine(strategyName)}catch(ex){print ex}
engine = Backtest::createBacktester(strategyName, config, callbacks,, )
timer Backtest::appendQuotationMsg(engine,  dailyData  ); 

After creating the backtesting engine through Backtest::createBacktester, users can execute backtesting in the following way, where dailyData is the corresponding daily market data.

name=["symbol","tradeTime","open","low","high","close","volume","amount","upLimitPrice","downLimitPrice","prevClosePrice","signal"]
type=["STRING","TIMESTAMP","DOUBLE","DOUBLE","DOUBLE","DOUBLE","LONG","DOUBLE","DOUBLE","DOUBLE","DOUBLE","DOUBLE[]"]
dailyData= loadText("./portfolioData.csv",schema = table(name as name,type as type))
timer Backtest::appendQuotationMsg(engine,  dailyData );

After the backtest is completed, users can obtain daily positions, daily equity, return overview, transaction details, and user-defined logic context in the strategy through corresponding interfaces. The following figure shows the transaction detail results obtained in this example: