Compare ANN to B&H, heatmap for outlier limit and lookback parameters

08Oct10

We have made some changes compared to the previous study.

1. Backtesting period:
On the one hand, we increased the backtesting period from 5 years to 12 years. This naturally, decreased the CAGR, because most of the gain was realized in the 2008 crash, because of the high volatility. As we increased the backtesting period, the avarage annualized return becomes less. It is more realistic this way.
One can argue that we should even exclude the 2008 crash period completely to have an even more realistic number. Yes, not a bad idea. That is the reason why I am more interested in the directional accuracy and not the actual return of the strategy. The directional accuracy (52%) is about constant in the whole period. However, in the highly volatile environment of 2008, that small directional accuracy boosted the overall return. That was a little unfair.

Our start date of the training samples is ‘1997-11-29’. That is the first day. In theory, we have daily data of the RUT from 1987, but we don’t want to use that. The reason is that later, we want to use the VIX as an input in the strategy. And the VIX data (from yahoo.finance.com) has a hole in it. There is no data for 1997-11-28. Therefore, we decided we extent our backtesting only to this date.

2. Buy&Hold return
The start date is 1997-11-29. Suppose we have a 200 day lookback window for ANN training. In that case, the first sample to test corresponds to the date 1998-07-16. (That is the 201 sample)
Let’s calculate the Buy&Hold return.
From 1998-07-16 till 2010-09-28, we have 12 years 2months. That is 12.3 years.
In that period, the RUT index went from 463.64 to 675.43 is x1.45; that is 45% TR (Total Return). That is 3.1% CAGR (yes, the last 10 years was the famous ‘lost decade’)
So, our ANN strategy should beat the 45% total return, that is 3.1% CAGR.
Luckily, it easily beats that.

3. Parameter tuning: outlier threshold and lookback window size.
We made 2 separate backtests for each parameter category. For example for the outlier limit = 5% (daily %gains greater than 5% were excluded) and lookback window size = 200 case, there are 2 measurements. We would like to emphasize that the ANN backtest is inherently random. This is because the weight initialization is random therefore different training of the ANN results different ANN. In theory, we can increase the number of epoch to infinity, still the different runs can result different ANN weights, because the teaching algorithm can stuck in local minima, instead of finding the global minima in the weight space.
However, note that as we increase the ensemble size, the random nature of the separate backtests diminish. In a production environment, this will be the way to decrease randomness. However, for our studies, we preferred faster execution time.

The fixed parameters:
nNeurons = 2;
nEnsembleMembers = 5;
nEpoch = 4;

The variable parameters:
lookbackWindowSize = 50..400;
testStartDateIndex = lookbackWindowSize + 1;
outlierFixLimit = infinity…0.02;

Without too much comment, here are the heatmaps for the 3 different statistics: directional accuracy, total return and annual return.

It reveals that we were lucky with our previous study, when we selected lookback size = 200 and outlier threshold = 5%. Those values are almost the optimal. The 18% CAGR and the 300% TR, it is too good to be true. It is more realistic to assume the returns in its vicinity of the optimal solution; that is about 14% CAGR and 150% TR.
Note that the Buy&Hold returned 3% CAGR and 45% TR during the same period.

Advertisements


No Responses Yet to “Compare ANN to B&H, heatmap for outlier limit and lookback parameters”

  1. Leave a Comment

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s


%d bloggers like this: