Shark Symbolic Regression

Generate simulated daily market data for 20 stocks over 260 days. Each row represents one daily OHLC bar for a stock on a specific day.

def mockStock(symbol, startDate, nDays){
    tradeDate = startDate + 0..(nDays - 1)

    basePrice = 20 + rand(80.0, 1)[0]
    dailyRet = randNormal(0.0005, 0.015, nDays)
    close = basePrice * exp(cumsum(dailyRet))

    open = close * (1 + randNormal(0, 0.003, nDays))
    high = max(open, close) * (1 + rand(0.01, nDays))
    low = min(open, close) * (1 - rand(0.01, nDays))
    volume = long(1000000 + rand(9000000, nDays))
    amount = close * volume

    return table(take(symbol, nDays) as symbol, // Stock code
                 tradeDate, // Trade date
                 open, // Opening price
                 high, // High price
                 low, // Low price
                 close, // Closing price
                 volume, // Trading volume
                 amount) // Trading value
}

nStocks = 20
nDays = 260
symbols = format(1..nStocks, "000000") + ".XSHG"

quotes = each(mockStock{, 2025.01.02, nDays}, symbols).unionAll(false)
quotes = select * from quotes order by symbol, tradeDate

From the raw daily market data table quotes, calculate three stock factors that can serve as input variables for symbolic regression based on each stock's historical prices and trading volume.

• ret5: Return over the past 5 days.
• vol10: Volatility of returns over the past 10 days.
• volChg: The increase in trading volume relative to the average volume over the past 10 days.

factorData =
    select symbol,
           tradeDate,
           close,
           volume,
           close \ move(close, 5) - 1 as ret5,
           mstdp(close \ prev(close) - 1, 10) as vol10,
           volume \ mavg(volume, 10) - 1 as volChg
    from quotes
    context by symbol
    csort tradeDate
// Remove rows that contain NULL values, retaining only data where all three input variables are valid.
factorData =
    select *
    from factorData
    where isValid(ret5)
      and isValid(vol10)
      and isValid(volChg)
    order by symbol, tradeDate

Add the target variable nextRet to factorData to create the complete dataset, and split the dataset into training and test sets by date.

• dataset: The complete dataset containing input factors and the target variable.
• train: The training set, used to search for an expression during symbolic regression training.
• test: The test set, used to evaluate the expression.

Each sample row in dataset can be interpreted as follows: for a stock on a given day, ret5, vol10, and volChg are known, and the target is nextRet. Symbolic regression uses the training set to search for an optimal expression, such as f(ret5, vol10, volChg) ≈ nextRet. You can then use this expression to predict nextRet from the three factors ret5, vol10, and volChg.

dataset = select *, double(NULL) as nextRet from factorData
// Add noise to make the data closer to a real-world scenario.
noise = randNormal(0, 0.001, dataset.rows())

// Define a hidden relationship that Shark needs to learn in order to find the optimal expression.
dataset[`nextRet] =
      0.60 * dataset[`ret5] * dataset[`volChg] - 0.40 * dataset[`vol10] + 0.20 * dataset[`ret5] + noise

/*Split the data:
Data on or before 2025.06.30 is used as the training set
Data after 2025.06.30 is used as the test set*/
splitDate = 2025.06.30
train = select * from dataset where tradeDate <= splitDate order by symbol, tradeDate
test = select * from dataset where tradeDate > splitDate order by symbol, tradeDate

Use getGlobalSharkEngine() to obtain a handle to the global Shark engine in the environment. You can think of the engine as the entry point to Shark computing resources. Whether you train a symbolic regression expression or evaluate the expression on the test set, you must use this Shark engine. For details about symbolic regressor configuration, see the config parameter in the skSymbolRegressor documentation.

// Obtain the Shark engine
skengine = getGlobalSharkEngine()

// Configure the symbolic regressor
regressorConfig = {
    "functionSet": [add, sub, mul],
    "selection": {
        "name": "tournament",
        "param": {"size": 8}
    },
    "mutations": [
        {"name": "crossover", "weight": 70},
        {"name": "subtree", "weight": 10},
        {"name": "hoist", "weight": 5},
        {"name": "point", "weight": 10, "param": {"rate": 0.15}},
        {"name": "reproduction", "weight": 5}
    ],
    "verbose": true
}

symbolRegressor = skSymbolRegressor(skengine, "gplearn", regressorConfig)

Symbolic regression training enables Shark to automatically search for a mathematical expression among the three input variables ret5, vol10, and volChg, so that the expression's output is as close as possible to the target variable nextRet. For details about training configuration, see the config parameter in the skSymbolFit documentation.

/*Use mean squared error (MSE) as the fitness function.
A smaller MSE indicates that the expression's predicted values are closer to the target nextRet.*/
@gpu
def fit_mse(pred, y){
    err = pred - y
    return mean(err * err)
}

/*Extract columns from the training set as vectors.
The variable names ret5, vol10, and volChg are preserved in the final expression.*/
ret5 = train["ret5"]
vol10 = train["vol10"]
volChg = train["volChg"]

inputs = {
    "varTerminalSet": [ret5, vol10, volChg]
}

/*groupTerminalSet is not configured in this example.
groupTerminalSet is an optional set of grouping variables used to discover group factors with aggregate operators.
The length of the grouping variables must be the same as the length of the input variables in varTerminalSet. After groupTerminalSet is specified,
the length of target must equal the number of groups.

Shark performs aggregate factor discovery only when functionSet contains aggregate functions and groupTerminalSet is not empty.*/

target = train["nextRet"]

fitConfig = {
    "fitnessFunction": fit_mse,
    "initializer": {
        "name": "half",
        "param": {
            "minDepth": 2,
            "maxDepth": 5,
            "probConstant": 0.40
        }
    },
    "constantRange": {
        "floatRange": {"min": -1.0, "max": 1.0, "weight": 1},
        "intRange": {"min": -2.0, "max": 2.0, "weight": 1}
    },
    "epsilon": 1e-8,
    "parsimony": 0.0,
    "seed": long(20260625),
    "population": 2000,
    "generation": 60,
    "maxDepth": 7,
    "elites": 10,
    "simplify": true,
    "verbose": true
}

syms, info = skSymbolFit(symbolRegressor, inputs, target, fitConfig)

bestSymbol = syms[0]

print("Best expression:")
print(string(bestSymbol))

print("Training summary:")
print(info)

Evaluate the expression on the test set to obtain testPred, compare it with the previously simulated returns, and assess the expression's predictive performance. This example provides three commonly used evaluation metrics:

• MSE: Mean squared error. Smaller values are better. It measures the numerical distance between the predicted values and the true target values.
• Corr: The correlation coefficient. Values closer to 1 are better.
• Direction Accuracy: The directional hit rate. Higher values are better. It measures the proportion of samples where testPred and nextRet have the same sign, providing an intuitive indication of how often the up/down direction is predicted correctly.

A simple baseline is also calculated by using ret5 directly as the predicted value.

/*Compute the expression on the test set
The input table column names for skSymbolEval must match the training variable names: ret5, vol10, and volChg*/
evalContext = skSymbolEvalContext(skengine)

testInput = select ret5, vol10, volChg from test
testPred = skSymbolEval(evalContext, bestSymbol, testInput)
testY = test["nextRet"]


// Evaluate the predictive performance of the expression

testErr = testPred - testY
testMse = mean(testErr * testErr)
testCorr = corr(testPred, testY)
testDirectionAcc = mean(iif(testPred * testY > 0, 1.0, 0.0))

benchPred = test["ret5"]
benchErr = benchPred - testY
benchMse = mean(benchErr * benchErr)
benchCorr = corr(benchPred, testY)
benchDirectionAcc = mean(iif(benchPred * testY > 0, 1.0, 0.0))

metrics = table(
    ["SharkSymbol", "Benchmark_ret5"] as model,
    [testMse, benchMse] as mse,
    [testCorr, benchCorr] as corr,
    [testDirectionAcc, benchDirectionAcc] as directionAcc
)

The following example uses the results from one symbolic regression training run to explain the training output. The optimal expression obtained after training is as follows:

((vol10 * ((-0.271591 * ((ret5 - 1.000000) - -0.271591)) * -2.000000)) + ((-0.271591 * (ret5 * 1.000000)) * ((-0.711353 - (1.000000 * volChg)) - ((0.513379 + 0.708364) * (1.000000 * volChg)))))

Use the optimal expression to compute the variables in the test set and evaluate the results. The metrics table is as follows:

• MSE: SharkSymbol has a much smaller error than Benchmark_ret5. mse is the mean squared error, which represents the average squared difference between the predicted values and the true target, nextRet. A smaller value indicates that the predictions are closer to the target values.
• Corr: SharkSymbol is almost perfectly correlated with the target return. corr is the correlation coefficient between the predicted values and the target value nextRet. The closer it is to 1, the more consistent their movements are.
• Direction Accuracy: SharkSymbol predicts the direction with very high accuracy. 0.97875 means that the Shark expression correctly predicts the direction for 97.875% of the samples in the test set. By contrast, Benchmark_ret5 has a directional hit rate of only 64.875%.

The expression found by Shark almost reproduces the hidden formula used to construct nextRet, so it performs very well on the test set. In contrast, Benchmark_ret5 uses only a single momentum factor and does not incorporate the information from ret5 * volChg or vol10, resulting in a larger error, lower correlation, and significantly worse directional hit rate.