De-trend and re-trend. Lookbackdays for forecasting the short term future market average


Notice that so far we have used 200 days lookback for estimating the day-of-the-week behavior, so we used a 200 days training set.
When we normalize the targets (next day %gain) in the training set, what we basically do is that we subtract the previous 200 days average daily %gain (and we may opt to multiply it to rescale the whole training set to have
However, after subtracting the average of the previous 200 days, the ANN predict a next day %gain with zero average. Considering that the market in the long run trends up (because of inflation and other factors), having an estimate of zero average, we predict that the next days will be about 50% up days and 50% down days. But, in reality, it is not the case. Markets go up in about 55% of the days and goes down about 45% of the days.
To compensate for this, we should add back the average %gain after the ANN makes its prediction. We say that we ‘de-trend’ the training set, but ‘re-trend’ the prediction. We subtract the long-term trend, but we insert the short term trend back.

Does it reveal the fine details of using the ANN?
Programming (creating the source code of) the ANN is science, but feeding the ANN and using it is more an art than science; as it is mentioned by many financial articles.
In this experiment, we use 200 days lookback for training samples.
We also use 200 days lookback for detrending the targets. (note in theory, we could use another lookback days or another strategy to detrend the training set. (like a running SMA(50) subtraction))

We study now what is the optimal lookback days for re-trending the prediction.
The parameters for this run:
nNeurons = 2;
nRepeatLearning = 20; % use the ensemble with this number of members
nEpoch = 4;
lookbackWindowSize = 200; % look back for day of the week anomaly samples
lookbackDeTrendWindowSize = 200; % we can detrend by another method, like subtracting the runnig local SMA(50)
lookbackReTrendWindowSize = 10..200; % we re-trend the prediction based on the near past

The result of our experiment is here: (click for larger image)

The deterministic k-NN like row shows the result for the strategy that uses the last 200 days samples, and forecast deterministically the next day %gain what was the average for that day (Mon,…Fri) in the past 200 days. It doesn’t use any ANN learning.
It achieves only 6.45% CAGR. (this is not bad). It shows that day of the week anomaly works, even without using ANN learning and smoothing. On the other hand, this 6.45% CAGR is given partly because of the momentum effect. If markets are down in the last 200 days, we forecast down day generally for all days from Monday to Friday. This is price momentum in effect.

The ‘no detrending’ row refers to the case when the ANN is given the raw 200 days lookback training samples, but the average is not subtracted.
It achieves 9% CAGR.

The third line removes the 200 days average from the training set, but adds the same 200 days average to the forecast. In theory, it should behave similar to the ‘no detrending’ case. However, it is a little bit better: 10.5% CAGR. It may be, because the ANN can use detrended traning set more efficiently, more accurately. It shows that even if it doesn’t seem to be sensible to detrend the input, we should do it, because it improves the ANN accuracy.

At the end of the list: as we decrease the lookback days to 60..10, the D_stat, GAGR% and TR% all becomes very random and chaotic. It cannot be considered reliable. For example the 30days lookback gives -0% CAGR, while the 20 days lookback gives +9% CAGR. It is just too volatile. We shouldn’t use that end of the spectrum.

That gives as the answer that we should use about 140 days lookback as an optimal. (or we can use any from 150 to 100)
Note also that it improved our overall result to 15% CAGR (not bad). (Compared to the case when we de-trend and re-trend with the 200 days lookback (that was 10% CAGR))
We will use this in the future.

In the far future we don’t want to do this kind of manual parameter optimization. We plan to build a controller layer that adaptively selects the best parameters based on past success.


No Responses Yet to “De-trend and re-trend. Lookbackdays for forecasting the short term future market average”

  1. Leave a Comment

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

%d bloggers like this: