Combining the 3 inputs. 1 black box or 3 black boxes with confidence?

13Dec10

As a sidenote, we mention that it is probably our last post in 2010. We started this blog on 28th January 2010. In these 11 months, we wrote 44 different articles in this blog. This one is the 45th.

We didn’t aim to find the most profitable ANN (Artificial Neural Network) approach in the first year, but to learn the small nasty tricks that helps in understanding how the ANN black box work. Trading in real life based on decisions by a black box is emotionally very hard. Especially after some serious drawdown it is easier to deem the prediction of the ANN to be silly and stop trading the strategy. After one year of scrutinizing different ANN trainings, reading academic articles and books, tweaking parameters, visualizing inputs, outputs, we don’t believe any more that the ANN is a black box. Of course, its decisions are not disclosed in terms of rules, but we got familiar with them at the point that we can even predict their prediction.

Just roughly reminiscence the history of this blog. We started with the simplest of all, using only the previous day return for next day prediction. After that, we tried 2 Moving Averages inputs (short term, long term). Later we tried the VIX as an input. That didn’t work. We changed later the daily prediction to weekly hoping the less volatility will be better for us, but it didn’t work either. From the second half of the year, we started to investigate the day of the week as an input. This seemed to work. Later we tried the current day return as input and at last we moved on studying the Euro (FXE) return on the next day index return.

This post is also unusual, that it is the culmination of the last 3 main topics. The different inputs: day of the week, current day return, Euro daily return are combined finally. So, we can say this post is not only a chronological milestone, but it is a milestone content too.

As a preliminary of this article, you may visit another post here in which we combined the dayOfTheWeek and currDayChange inputs for the last 12 years. As you can see from that chart. I repeat here:

That combination was successful (130%-400% TR) for the 12 years backtest, but it treaded water in the last 4 years. (0% TR)

As the FXE started in December 2005, our prediction in this post starts in September 2006, because we use 200 days lookback period for ANN training. So, we cover about 4 years in these backtests.
In these tests we used:
2 bins for discretization of currDayChange and eurChange input (to keep it simple)
1 dimensional dayOfTheweek input

maxEpoch=100,
nNeurons=2 for the single ANN cases and nNeurons=3 for the combined ANN cases,
lookbackperiod=200
target outlier limit = 4%

1.
ANN(dayOfTheWeek)
In the previous post we showed how this input worked successfully with 10% CAGR in the last 12 years, but stopped (0% CAGR) working in the last 4 years.
An example Total Return chart:

We emphasize that it is only an ‘example’ chart. The ANN learning is random (if it is not random, it is wrong, or sub-optimal). The repeated backtest gave slightly different results. This is why we should average.
In another backtest, it gave 51.34% D_stat, 2.16% CAGR, and -7.86% TR. We conclude it didn’t work in the last 4 years. Probably, it is due to the 2008 market storms.

2.
ANN(currDayChange)
This is the best standalone strategy. Its result is 52.86% D_stat, 28.61% CAGR, 119.88% TR in the last 4 years. An example performance chart is presented here:

You may visit our previous post that shows its performance with different discretization levels for the last 12 years. However, note that those performances presented are for the 12 years. For example, the 2 bins case there you can find 13.8% CAGR for the last 12 years, but here we should 28.61% CAGR for the last 4 years part. This shows that how incredible lucky was this approach now, because of the 2008 crash and the volatility increase improved its return.

3.
ANN(eurChange)
Its result is 52.29% D_stat, 7.84% CAGR, 8.04% TR in the last 4 years. An example performance chart is presented here:

See our first study about it here and second study
Our conclusion that there were sub-periods when it worked, but overall in the 4 years there was a transition that turned the correlation upside down. This affected the performance negatively to the point that overall, it was not profitable.

4.
ANN (use all 3 inputs)
A blind combination of the 3 inputs gave the following result in average: 53.1% D_stat, 12.94% CAGR, 37.2%. (see the different experiments results in the table at the end of the post).
An example performance chart:

Note how the D_stat really improved and it is better than any of the standalone ANNs. However, its TR is not better than the TR of the best standalone ANN. The reason is that the best standalone ANN was very successful only in about 1 year, in the 2008 period. And it was not too successful in the other 3 years. The Combined ANN slightly better overall (in terms of D_stat), over the 4 years, but it is not especially good in that 2008 periods when most of the returns earned for the best standalone ANN. So, even if it predicts better overall, the profit is not increased.

In another point of view, one can say that the combined ANN is a ‘combination’ of the very good CurrDayChange ANN and the other two mediocre ANNs. So, it is natural to expect that the very good performance of the best ANN will be muted and the very bad performance of the other two ANNs will be lifted up. So, a combined performance should be between the best and the worst performances. The advantage of the combination is that the strategy is more robust, it is averagely good in every periods.
The 12.94% CAGR is quite decent considering how robust it is. In the long term, we should be happy to play it in real life without hesitation. However, we will show that we can do even better.

5.
3ANN, trade if all agree, ensemble: 1
The idea is that we trade only if all the standalone ANNs agree on the outcome. If all the three predicts +1, we play long, if all predicts -1, we play short and we are in cash otherwise. For measuring its performance we introduce another metrics. (see the table at the end) The WinFrequency is 19.75% that means we are right on 19.75% of the days. The lossFrequency is 16.00%, meaning we are wrong 16% of the days. On 64.25% of the days our profit is zero. Since there is not a single day in the 4 years history of RUT when its daily change was zero, it essentially means, we decided to be in cash 64.25% of the time. Note how the D_stat increased to a whopping 55.24%. (=19.75/16) The numbers show 12.18% CAGR and 47.26% TR. However, what we like about this strategy compared to other strategies is the low drawdown. See the performance chart:

6.
3ANN, trade if all agree, ensemble: 11
We simply repeated the previous experiments with 11 ensemble members instead of 1. However, note that conceptually, it is not a simple ensemble decision any more. The collective decision is not what you expect at the first sight.
We use the next rule for the ensemble forecast:
ensembleForecast = sum(sign(memberForecasts))
The sign of zero is zero, so those ensemble members are not participating. The ensembleForecast result is a positive or negative ‘integer’ number (not a float number). It is only the sign of the aggregated forecasts.
Later, the Trader subsystem uses the ‘ensembleForecast’, and if it is positive, it plays as long, if it is negative it plays as shorts. So, this 11 ensembling result can be positive if ANY of the 11 members (maybe only 1 out of the 11) agrees that all the 3 standalone ANNs has an Up vote.

Suppose that because of randomness, in the 1 member ensemble case (Case 5), instead of the 3 standalone ANNs giving 3 Up votes, there was only 2 Up votes. (because of pure chance) In that case the 1 member version gave a Cash signal. However, this 11 members version will likely give an Up signal (if it is the true case in life), because we repeat the same process 11 times.

When we repeat the test 11 times, the chances are good that we have 3 Up votes more frequently. You can see this effect in the performance table at the end. With the 11 ensemble case, the zeroFreq (that is the time we are in Cash) was reduced from 65% to 55% because of this. So, this strategy is in cash 55% of the time, nearly every second day. It trades only when it has high conviction about the direction. We really like this feature. The fallout of it is that we have less risk, because we are less exposed to the market.
So, we can interpret increasing the members to 11 not as the approach to decrease the randomness (as usual in other ANN networks), but as an approach to a little bit lessen the confidence, and to allow more days to participate in the forecast. Instead of 35% of the days as in the 1 members case, we trade 45% of the days. It is important to notice about this ensembling.
Note how the D_stat increased to 55.96%. (=24.80/19.51) The performance numbers show 19.03% CAGR and 84.79% TR. However, what we like about this strategy is the very low drawdown and the consistency. Higher highs and lower lows. It is very pretty. With almost 20% CAGR, you cannot wish more.

Finally, see the performance measurements table for all 6 strategies we talked about.

Discussion:
– Do we believe that in these 4 years currDayChange input was especially lucky and its exceptional performance will not continue in the future? Yes. Volatility is likely to decrease, that will trump its performance.
– Do we believe that these 4 years was unlucky for the dayOfTheWeek input, but its shine will return? Yes. It didn’t work, because of the 2008 crash. (it worked in the previous 12 years)
– Do we believe that these 4 years was unlucky for the EUR input, but it will perform better in the future? Yes. These 4 years contained a 180 degree transition period. That was unlucky. We expect the correlation between EUR and USD market will last longer than this (or the turn from positive to negative correlation will be not as sharp as it was now), so even this approach will be profitable in the future.
– As you can see, in the future we are pessimistic about our so far successful currDayChange ANN, and optimistic about the non performing dayOfTheWeek and Euro ANN. But to be honest, it is not too important what we believe. The market will decide. If the approach works (there is negative or positive correlation between any of those 3 inputs and the target), the ANN will adapt to it. Even if there is no correlation, and we have no edge in some ANNs, it is enough that we have an edge in only one of the three ANNs. If any of the 3 approaches work, the combination of them will bring up its strength in the long run.
– The 56% directional accuracy, the 19% CAGR and the 85% TR during the last 4 years is currently our best and most consistent result. It is on the market only 45% of the days. Playing by Ultra ETFs would be ideal, because we don’t hold it for long term. So, play with the thought of doubling that return to 38% per annum. (not mentioning the future improvements we plan) Pretty.
– However, the best thing about it is not the profitability. Far the best thing about the ‘trade if all agree’ combination is that it gives a very smooth return without significant drawbacks. Higher highs, lower lows. In real life, it is not difficult to play this strategy emotionally.

There are of course many varieties and many improvements we can do in the future (next year):
– Tweaking the number of bins of the discretization (the one presented here is only the 2 bins discretization. Mainly for illustration.)
– Tweak the weights of the different ANN outputs based on their last 50? days performance. Recently successful ANNs should be given higher weights. This is essentially a linear weighted combination of the ANNs on the meta level.
– Initializing a Meta-ANN as an overmind. This would make it possible a nonlinear combination of the standalone ANNs.

Advertisements


One Response to “Combining the 3 inputs. 1 black box or 3 black boxes with confidence?”


  1. 1 Ensemble strategy: Play if All Agree « Neural Sniffer

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s


%d bloggers like this: