Timing of Back-Test¶
We show the runtime of a typical single-period optimization back-test.
This is similar to what was show in figure 7.8 of the paper.
Many elements matter in determining how fast a back-test can be run; here we present a few (size of risk model, choice of numerical solver and CVXPY flags, …) but many more are relevant and understanding of those comes only with (deep) expertise in optimization software (and computer systems).
One interesting feature of Cvxportfolio is that it enables for automatic caching of some expensive numerical procedures; one of them is estimation of large covariance matrices. Here we show the execution time difference when running the same back-test twice. The first time covariance matrices are estimated and saved on disk, the second time they are loaded. This especially matters when doing hyper-parameter optimization (the expensive calculation is only done once).
Finally, we show that cvxportfolio.result.BacktestResult does a good
job accounting for and reporting the time spent doing a back-test and in its
various components. You can expect it will do even more granular reporting in
future releases.
Note
To reproduce what is shown here you should make sure that the first
time this script is run there are no covariance matrices already saved
for the historical market data used here. If you run it from scratch, that
is OK, but if you re-run this script it will pick up the covariance
matrices already estimated. There is currently (Cvxportfolio 1.3.0) no
easy way to remove caches other than manually deleting files in
~/cvxportfolio_data, which you can always safely do.
import time
import matplotlib.pyplot as plt
import pandas as pd
import cvxportfolio as cvx
# same choice as in the paper
from .universes import SP500 as UNIVERSE
# changing these may have some effect on the solver time, but small
GAMMA_RISK = 1.
GAMMA_TRADE = 1.
GAMMA_HOLD = 1.
# the solve time grows (approximately) linearly with this. 15 is the same
# number we had in the paper examples
NUM_RISK_FACTORS = 15
# if you change this to 2 (quadratic model) the resulting problem is a QP
# and can be solved faster
TCOST_EXPONENT = 1.5
# you can add any constraint or objective
# term to see how it affects execution time
policy = cvx.SinglePeriodOptimization(
objective = cvx.ReturnsForecast()
- GAMMA_RISK * cvx.FactorModelCovariance(
num_factors=NUM_RISK_FACTORS)
- GAMMA_TRADE * cvx.StocksTransactionCost(exponent=TCOST_EXPONENT)
- GAMMA_HOLD * cvx.StocksHoldingCost(),
constraints = [
cvx.LeverageLimit(3),
],
# You can select any CVXPY solver here to see how it affects
# performance of your particular problem. This one is the default for
# this type of problems
solver='ECOS',
# this is a CVXPY compilation flag, it is recommended for large
# optimization problems (like this one) but not for small ones
ignore_dpp=True,
# you can add any other cvxpy.Problem.solve option
# here, see https://www.cvxpy.org/tutorial/advanced/index.html
)
# this downloads data for all the sp500
simulator = cvx.StockMarketSimulator(UNIVERSE)
# we repeat two times to see the difference due to estimation and saving
# of covariance matrices (the first run), and loading them from disk the
# second time
figures = {}
for run in ['first', 'second']:
# execution and timing, 5 years backtest
s = time.time()
result = simulator.backtest(
policy,
start_time=pd.Timestamp.today() - pd.Timedelta(f'{365.24*5}d'))
print('\n\n' + run.upper() + ' RUN')
print('BACK-TEST TOOK:', time.time() - s)
print(
'SIMULATOR + POLICY TIMES:',
result.simulator_times.sum() + result.policy_times.sum())
print(
'AVERAGE TIME PER ITERATION:',
result.simulator_times.mean() + result.policy_times.mean())
print('RESULT:')
print(result)
# plot; this method was introduced in Cvxportfolio 1.3.0
figures[run] = result.times_plot()
This is the output printed to screen when executing this script. You can see
the difference in timing between the two runs (all other statistics are
identical) because in the first run covariance matrices are estimated and
saved on disk, and the second time they are loaded. You can also see that
the time accounting done by cvxportfolio.simulator.MarketSimulator
is very accurate, and coincides (with a tiny difference) with what is reported
by Python’s time here in the script.
Updating data.......................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................
FIRST RUN; RESULT:
#################################################################
Universe size 502
Initial timestamp 2019-03-22 13:30:00+00:00
Final timestamp 2024-03-20 13:30:00+00:00
Number of periods 1258
Initial value (USDOLLAR) 1.000e+06
Final value (USDOLLAR) 2.898e+06
Profit (USDOLLAR) 1.898e+06
Avg. return (annualized) 52.0%
Volatility (annualized) 77.3%
Avg. excess return (annualized) 50.0%
Avg. active return (annualized) 50.0%
Excess volatility (annualized) 77.3%
Active volatility (annualized) 77.3%
Avg. growth rate (annualized) 21.3%
Avg. excess growth rate (annualized) 19.4%
Avg. active growth rate (annualized) 19.4%
Avg. StocksTransactionCost 0bp
Max. StocksTransactionCost 9bp
Avg. StocksHoldingCost 1bp
Max. StocksHoldingCost 5bp
Sharpe ratio 0.65
Information ratio 0.65
Avg. drawdown -47.0%
Min. drawdown -89.1%
Avg. leverage 300.0%
Max. leverage 403.0%
Avg. turnover 3.8%
Max. turnover 150.0%
Avg. policy time 0.232s
Avg. simulator time 0.094s
Of which: market data 0.016s
Total time 408.823s
#################################################################
BACK-TEST TOOK: 409.0286121368408
SIMULATOR + POLICY TIMES: 408.8231477737427
AVERAGE TIME PER ITERATION: 0.32523718995524475
SECOND RUN; RESULT:
#################################################################
Universe size 502
Initial timestamp 2019-03-22 13:30:00+00:00
Final timestamp 2024-03-20 13:30:00+00:00
Number of periods 1258
Initial value (USDOLLAR) 1.000e+06
Final value (USDOLLAR) 2.898e+06
Profit (USDOLLAR) 1.898e+06
Avg. return (annualized) 52.0%
Volatility (annualized) 77.3%
Avg. excess return (annualized) 50.0%
Avg. active return (annualized) 50.0%
Excess volatility (annualized) 77.3%
Active volatility (annualized) 77.3%
Avg. growth rate (annualized) 21.3%
Avg. excess growth rate (annualized) 19.4%
Avg. active growth rate (annualized) 19.4%
Avg. StocksTransactionCost 0bp
Max. StocksTransactionCost 9bp
Avg. StocksHoldingCost 1bp
Max. StocksHoldingCost 5bp
Sharpe ratio 0.65
Information ratio 0.65
Avg. drawdown -47.0%
Min. drawdown -89.1%
Avg. leverage 300.0%
Max. leverage 403.0%
Avg. turnover 3.8%
Max. turnover 150.0%
Avg. policy time 0.135s
Avg. simulator time 0.079s
Of which: market data 0.014s
Total time 268.317s
#################################################################
BACK-TEST TOOK: 268.54603695869446
SIMULATOR + POLICY TIMES: 268.3172538280487
AVERAGE TIME PER ITERATION: 0.2134584358218367
And these are the figures that are plotted. The first run, with longer
time spent in the policy (which does the covariance estimation,
in the cvxportfolio.forecast.HistoricalFactorizedCovariance
object).
This figure is made by the cvxportfolio.result.BacktestResult.times_plot() method.¶
And the second run, when covariances are loaded from disk. The time taken to load them is accounted for in the simulator times (there are a few spikes there, saved covariances are loaded at each change of universe).
This figure is made by the cvxportfolio.result.BacktestResult.times_plot() method.¶