Daily MR or FT: the data

25Nov10

Before moving fully to the 2 inputs ANN case (dayOfTheWeek, currDayChange), we regress back to a strategy we have already studied before without real success. Our input this time has only the currDayChange and we try to predict the nextDayChange. This is the classical daily Mean Reversion (MR) or Follow Through (FT) strategy. (Prediction of financial time series.) There are times when the index shows daily MR behaviour, other times daily FT behaviour. Different indices show different behaviour. For example, the SP500 is daily MR from about 1998. However, the same is not true for emerging markets (China, Russian index) or for small caps like the Russell 2000. They are usually FT. That is what other studies say. We use the Russell 2000 index, so we expect FT behaviour.

In this post we only visualize the data and try to make some conclusions. The previously studied dayOfTheWeek input has two very good properties that are not valid for the currDayChange input:
– dayOfTheWeek is a discrete data. Its values are 1, 2, 3, 4, 5.
– by and large, each discrete value has the same number of samples. The number of Mondays and the number of Tuesdays are about equal.
These two properties are very handy and facilitate training the ANN. The currDayChange is not this kind of data. Therefore we have very serious doubt that we can achieve the same kind of prediction power than in the dayOfTheWeek case. The currDayChange is a continuous data and it has a Gaussian distribution, so moving away from the mean, the number of samples decrease.

If forecasting continuous, Gaussian distributed function is not possible (we will try), or if we got very poor result, we may try to convert the continuous, Gaussian case to a discrete case. That is an idea to try. We sort the inputs into a fixed number of bins, so we achieve discrete input and we can assure that the bins contain approximately the same number of samples. As a last resort, we can try this prediction technique later.

In this post, we show 2 sequences of plots (actually 3). In the first sequence, we just average the nextDayChange values in each bin. The second sequence shows the same, but the greater than 4% nextDayChanges (the targets), the outliers are eliminated. The reason is that unfortunately the system is sensitive to outliers (the ANN approximates the mean and the mean is very sensitive to the outliers.) So, we got a better picture if we eliminate the outliers. And as we studied, eliminating outliers improves the prediction power. Our ANN will learn the outlier free data, so it is sensible to plot the outlier free data now.

F(currDayChange) = nextDayChange history (3200 days, 12 years).
The bins contain approximately equal number of samples. There are 2 bins, 4 bins, 6 bins, 10 bins, 20 bins versions.
Sequence 1: with outliers




Sequence 2: without Target Outliers;




Notes:
– the 2 bins shows that the Russell 2000 is a FT (rather than a MR index). Up days are followed by higher Up days and down days are followed by lower Up days. Ok, it is not strictly FT (in FT, down days are followed by down days), but it is not MR either. But it is more FT than MR.
– at the first sight, the 2 bins case charts can suggest that we cannot be clever enough, because our best forecast is always to vote for an Up next day. Because even Down days are followed by Up days. So, based on this data, a perfect predictor would always vote for Up day. Note that it is only true when we regard the whole 12 years period. However, our lookbackDays window is only 200 days. That was the optimal for the dayOfTheWeek case, we may find another lookback work better for this input. We contend that when we look only the last 200 day periods, we will find many times (especially in bear markets) that there is a real FT behaviour: Down days are followed by more Down days.
As we increase the number of bins, the randomness of the data is revealed gradually. That makes our life as a misery. While it is not too difficult to match a continuous sigm() or atan() function combinations to approximate the 2 bins, 4 bins, 6 bins versions, it is very difficult to approximate the 20 bins case with a smooth continuous function. At the end randomness trumps everything. We see why we are not very optimistic that predicting random continuous functions is possible at all with good accuracy. However, discretization of the input is one solution that may help in prediction. We are free to select the best data representation that we can find that helps the predictor to do its work. Using ANN is not a science, it is an art. Finding a good data representation need many creative ideas. One of the ideas is the discretization (others: dimension reduction, discretization modulo rotation, normalization, detrending, calculating higher order functions instead of raw data, etc.)
– To us, predicting the 4 bins or 6 bins version seems to be most promising. They reveal the nature of the function, but they don’t look too random to be impossible to approximate with a smooth continuous function.

What is the nature of the function to be predicted? (based on the 6 bins case chart: both with or without outliers)
– the discretization bins represent: very oversold, medium oversold, slightly oversold, slightly overbought, medium overbought, highly overbought conditions. The border between these categories are -1.47%, -0.65%, 0%, +0.65%, +1.47%. So, everything over 1.47% daily gain means overbought.
– when currDayChange is very overbought, next day seems to mean revert (down day). Investors reckon the previous day was too much.
– when currDayChange is mildly overbought that is the best for the next day return. This can be explained by the general bullishness of the market on that day. Investors wanted to buy stocks on the previous day, but because the market was up ‘mildly’, they thought the market was too extended on that day. So, they postponed their buying for the next day, hoping that prices will mean revert. However, the prices mean revert only if previous prices were higher than +1.47% on the previous day. So, investors wait in vain, and they drive up the prices big on the second day, because they are impatient.
– usually when prices are down (under 0%), the next day is not so good. It is not really FT in the long term 12 years case, but close to it.
– The difference between the outlier eliminated and non-eliminated case is the strongest in the very oversold case. If we don’t eliminate outliers, we have a very bullish Up day next day. This can be justified by a mean reverting explanation. If something is much oversold on a panic day, next day its price may be lift up. But note that this doesn’t happen in the outlier eliminated chart. When we eliminate outliers, we eliminate the non-normality of the market, we just have non-panic and non-happiness-madness days. In normal days, the rationality is that there is a real fundamental reason why the previous day was a very oversold down day (and not a panic reason) and this fundamental reason probably last for a long time implying FT and saying that the Down days should continue.
– The 20 bin case is very chaotic with the outliers, but without the outliers one can imagine the true nature of the underlying function. We try to show it with a red ‘smooth’ line in the following image. This is how we would approximate it.

– in outlier elimination we mean that when the TargetDailyChange is bigger than 4% change, we exclude the sample from the training. We call this TargetOutlierElimination. One can argue that we can similarly do an InputOutlierElimination.

It is exactly what we do in the 3rd Sequence. This charts are made by excluding all samples in which either the input or the target dailyChange is bigger than 4%.
Excluding only the targetOutliers eliminated 80 samples from 3267. (2.5%)
Excluding both the input and target outliers eliminated 142 samples from 3267. (4.3%)

Sequence 3: without Input and Target Outliers:




It is interesting that the second Sequence (with only Target outlier elimination) looks less random in the 20 bins case than the third sequence (with target and input outlier elimination). We should develop some theory why we want to predict that version, and not elimination outliers from the input only from the target. We will try to predict all 3 methods.

Advertisements


No Responses Yet to “Daily MR or FT: the data”

  1. Leave a Comment

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s


%d bloggers like this: