Combining 3 inputs in one ANN

20May11

This is a 24 year backtest.
Our framework is best suited to backtest the variation of 2 parameters. That suits well to the human visual system that can spot ‘interesting’ areas mostly in 2D (as a 2D function projection to the third dimension). We have chosen the number of neurons (nNeurons) and the max epoch runs (maxEpoch) in this test.
We present performance measures PV (portfolio value), CAGR (annual growth), maxDD and PvMAPEfrMul (PV Mean Average Percentage Error From Multiplicative Approximation, the smaller the better) here. We used 200 days lookback periods.

ANN variations:
A. 1D inputs (weekDays), (T-0 %change), (T-1 %change)
B. 2D inputs: (weekDays, T-0 %change), (weekdays, T-1 %change), (T-0 %change, T-1 %change), (T-1 %change, T-0 %change)
C. 3D inputs: (weekDays, T-0 %change, T-1 %change)

These are 8 ANN types.
Each measurement used the following redundancy (randomness decrease)

nEnsembleMembers = 5
nEnsembleRepeat = 5;
nTest = 13;

It means that for each 8 tests, for each cell in the following excel tables, a PV chart was evaluated 13 times.
Considering, the excel table parameters (rows, columns) varied as
nNeurons: { 1, 2, 3, 4, };
maxEpoch: { 5, 13, 24, 49, 99 };

generating one excel table, required the evaluation of 13*4*5 = 260 PV charts.
One portfolio chart is 24 years, that is 24*250 = 6000 days, and on every day, we train 5 * 5 = 25 ANNs, so overall we have to train an ANN 260*6000*25= 39,000,000 times!
No wonder, it took about 8 hours to generate one excel table.
To run all the 8 experiments with different 1D, 2D, 3D inputs, it took 8 times 8 hours = 36 hours. As the computer is running only 16 hours per day (not at night), it took 4 days constant running on a 4-core Intel i7 950 machine.

Notes on backtesting method:
– using ANN alone (for only one day forecast) is not computation intensive (1 second), but backtesting anything with varying parameters… it takes an awful lot of time.
– we used a lot of redundancy: evaluating a forecast 5*5*13 times, but still, we couldn’t eliminate the randomness inherent in ANN.
– at first sight backesting the ANN(T-0 %change, T-1 %change) and ANN(T-1 %change, T-0 %change) versions appears to be a joke here. They should give equal results. We did we backtest them? We backtested them, because again: we wanted to be absolutely sure that our implementation is correct. If they don’t produce the same result, there is a bug somewhere.

PV (portfolio value):

CAGR (annual growth):

dStat:

maxDD:

PvMAPEfrMul (PV Mean Average Percentage Error From Multiplicative Approximation, the smaller the better):
(for explanation: see the definition in one of the previous post)

As in illustration, we show one of the best portfolio here generated by the 2D ANN(T-0, T-1) strategy.
Note that forecasts and portfolios are quite random.
So, don’t pay too much attention into this specific chart.
(click on the image to see it properly)

Notes:
– the dayOfTheweek hasn’t got too much forecast power. (we expected more, so it is a little disappointment). However, note that even if it is not an excellent strategy, it is not bad either. The dStat is about 54%, which is much better than a random monkey predictor. It can also achieve a 49% maxDD, which is quite good, because the Buy&Hold strategy on the RUT had 60% maxDD. So, even if its PV is the same as the 5.58 PV of the B&H, we may opt to use this ANN, because it is less risky than B&H. So, we contend that the weekDay input has ‘some’ predictive power, but not enough to play it profitable as a standalone strategy. However, it can be a good addition to another profitable system.

– the ANN(T-0 %change, T-1 %change) and ANN(T-1 %change, T-0 %change) versions give similar results. It is consistent with our expectation.

– 2D vs. 3D war:
+ the ANN(weekDays, T-0 %change, T-1 %change) is a little bit better (really, only slightly, but it is not worse) than the ANN(weekDays, T-0 %change), so, it seems that it was worth to add an additional input (T-1) to the 2 dimensional input. So, it seems that adding more input is better!!!
+ the ANN(weekDays, T-0 %change, T-1 %change) is worse than the ANN(T-0 %change, T-1 %change), so it seems that adding another additional input (weekDay) was not a good idea. So, it seems that adding more input is not better!!! (Note the contradiction to the previous point)
+ it is interesting to see that not the 3D input version is the winner. So, it is not always true that adding more input is better.

1D vs. 2D war: having T-0 input as a base, it seems that adding T-1 as an additional input is a good idea, but adding weekDays an additional input is a bad idea (it decreases the result). That is very interesting and difficult to explain. It seems that inputs form ‘natural groups’. There are inputs (like T-0, T-1) that are connected (by nature), they strengthen the prediction (one depends on the other). However, there are inputs that are independent (like weekDay, T-0). Suppose, you as a trader know that yesterday was a Down day. What helps you more?
+ if you additionally know that the day before yesterday was also a huge down day, you may think that the market is oversold, you can forecast tomorrow will be a bounce Up. The two inputs are dependant. Even if, T-1 per se, has no predictive power, combining it with something in which it can express a value worth taking. Yes, it is probably true that after having two Down days, an Up day is more probable (because markets are not totally, independently random; days are not independent)
+ if you additionally know that today is Friday, then what? You are not too much better in forecast, because the new input (weekDay) is independent from T-0, and unfortunately this independent input has no real predictive power.

– Based on this, we propose that we should group inputs into dependable groups. Probably the best way would be to use an ensemble that uses 2 ANNs. One uses the 2D(T-0, T-1) input the other using the 1D(weekDay) input. The ensemble forecast would be a combination of these 2 separate ANNs. This would add some stability. We will test this idea in the next post.

– We highlight our best strategy so far: the ANN(T-0 %change, T-1 %change).
PV: $865 (investing $1 in 1987), that is 35% CAGR. Imagine that using a double or triple ETF or other kind of leverage.
For example, playing with Ultra ETF (double leverage), and playing the equity curve by 50% additional leverage, we may expect 3*35=105% CAGR.
Even if you consider friction costs (commission, askbid), almost 100% CAGR is possible.
The directional accuracy is 56.51%. Remarkable. However, note the risk: In the backtest, we show 35% maxDD, but looking at the maxDD table, it is more realistic to expect at least 40% maxDD. Using Ultra ETFs (double leverage), that maxDD is 1-0.6*0.6= 64%. Note that playing the equity curve will not increase the maxDD, only the return.
Can we bear the pain of 64% maxDD? If we have 105% CAGR, the answer is: yes.
Compare it to the RUT Buy&Hold that has 5.5% CAGR and 60% maxDD during that period.

– nNeuron parameter: It is a general trend (in almost all 8 backtests) that using the less number of neurons is the best. The best performers are the nNeurons = 1, 2 cases. The 1 neuron was almost always better than the 2 neurons (that is strange, we expected that for high dimensional inputs (2D, 3D), the 2 neuron case will be the winner). The 4 neuron case was always the worst. Expected. It tends to overfit.

– maxEpoch parameter: it was weird to see that increasing the maxEpoch to infinity in general make the prediction instable. We can’t really explain that. In general, after more training, the NN should perform better. Considering the 3D input case, the 99 maxEpoch result is almost worse than the 13 maxEpoch case. Hardly explainable. Overfitting again, but why overfitting is the problem?
It suggests that we shouldn’t increase the maxEpoch. If we want better prediction (with the same inputs), we should decrease randomness not by over-training, but by redundancy (put more members in the ensemble).
‘Luckily’, in our best performing ANN, the ANN(T-0 %change, T-1 %change) and in the nNeurons 1, 2 case, the maxEpoch 99 is about the same (a little better) than the maxEpoch 99.

– best parameters based on this backtests: we would use the 2 neuron and 49 as maxEpoch. (albeit 1 neuron case a slightly, slightly better)

Conclusion:
The test shows us that combining inputs sometimes improve the result, sometimes not. It is difficult to know in advance, but we proposed a theory of dependent/independent inputs. We found that adding T-1 %change to the T-0 %change input (2D input case) gives the best performance. (unleveraged 35% CAGR, leveraged: 100% CAGR)

Advertisements


No Responses Yet to “Combining 3 inputs in one ANN”

  1. Leave a Comment

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s


%d bloggers like this: