Weekly directional prediction: preparing input data


I want to find a simple example for which the NN (Neural Network) can prove that it works. For now to eliminate high frequency randomness I would go for weekly data instead of daily. Being simple means that the input to my NN should be simple. It should be 1 dimension input, or maximum 2 dimensions, so I can visually ‘debug’ its achievements.
Let’s formulate the prediction task as:

Predict the next week direction (only direction) (Up/Down) of the Russell 2000 (RUT) index based on
the previous week %change of the RUT and the VIX.

The VIX is the fear index, it measures the SPX monthly volatility (projected to be annual).
These data can be downloaded from yahooFinance

Everybody uses the SPY as a prediction, therefore I will work on the RUT.
See previous posts, why I prefer the RUT.
(for example I prefer to avoid the traffic jams and instead prefer to find abandoned roads; I may reach my goal faster.)

For a similar analysis please study this article on this blog.
Note that he used the SPX.

See his chart here:

My chart is this: (please click on the image to see it in a larger size)

The red dots are down weeks, the green dots are up weeks.

Unfortunately, this chart doesn’t reveal any visible pattern to the human eyes.
So, what can my NN learn if this seems to be totally random?

Maybe, it is not so random after all…
If I divide the input space to 4 quadrants and measure the
probability of next week Up day in each quadrant of the input space, I receive this statistics.


  • The period of inspection is 1997-2010 March. The reason is that the 1997-11-26 VIX data is missing from the yahoo database, so I restricted my inputs only after this date, because I wanted that the RUT and the VIX data be consistent.
  • the overall distribution skewed to the upside, since 55% of the total weeks finishes in the green (up weeks). That suits well to the overall tendency of the stock market to go Up (in the long term).
  • The most surprising finding is in the right top corner: if the previous week was RUT up week and VIX up week than the next week was Up 50 times and Down 26 times. The Up times occurs about 2 times more than Down times.
    The probability of the Up week in that case is 50/(50+26)=66%.
    (stop a little bit here: Isn’t it a little bit weird that there are 2 times more Up day than Down, and the probably is ‘only’ 66%. Isn’t it strange? Intuitively, someone would expect more probability, something like 75% probability . So, please, don’t trust your intuition when you can trust numbers. Trust your intuition only when numbers are not available.)
  • possible explanations for this unexpected quadrant statistics (I am not sure, but I try…)
    – RUT is a momentum (not mean-reverting), so when RUT is up, I expect that the next week it will be also Up
    – when the VIX is up this week, this usually means that the FEAR increased; that is usually a bad thing for the stock market, so, this should be bad for the next week RUT. However, it is not the case. It is very surprising. Why?

    1. One explanation is that the RUT maybe momentum, but the VIX maybe mean-reverting. So, if the VIX was up this week, it tends to be down next week. As the fear will diminish next week and RUT is still in its Up momentum, it will be a winner weak.
    2. Other explanation is that: If in the current week the RUT can increase in spite of the fact that the FEAR (VIX) increased this week, that means it is a very strong market now. It is so strong, it has so bullish sentiment that nothing can stop it to go higher next week.
    3. A third explanation: when the VIX increased this week that usually means that the fear increased; that means many speculator opened short positions for various indices, futures, etc. However, if the market doesn’t go down immediately in the beginning of the next week, these speculators sitting on large short positions panics and liquidate their short positions; Covering short positions leads the market (the RUT) higher next week.

    You see, I am quite inventive when I have to find explanations for past behaviours. Be aware that I am most probably wrong, and that humans in general very good at inventing narrative (and wrong) explanations afterthe fact happened.
    (like the daily articles on YahooFinance.com that try to explain the stock market behaviour After the market closed. My favourite example is a simple one in which the same fact can explain different/opposite stock market behaviour like, these two announcements:
    “Today the FED increased the interest rate. That is bad for stocks, therefore the market today went down”, but the same fact can explain the opposite:
    “Today the FED increased the interest rate, because the economy is in a very good shape. Therefore, the market went up today.”
    LOL. 🙂
    You see. Don’t read market commentators. Oh. Read it, but don’t believe. Ok, let’s go back to our topic.

  • you may argue that about 630 sample is not enough for making reliable statistics. You may be right.
    However, note that I use weekly data that tends to be less random than daily data.
    If somebody does a similar statistics like this using 640 Daily data, I would also question its reliability
    and I would think that his results is just ‘good luck’.
  • A real life strategy that can be played would look like this:
    – go long on the RUT for the next week if last week VIX was up and RUT was up
    – go short on the RUT for the next week if last week VIX was down and RUT was up
    (Please! Promise that you don’t play this… :))
  • Note that this statistics is only Directional statistics.
    I want to train my NN for directional answers now. So, for me, this will serve well as a crutch for my experiment.
    However, this statistics says nothing about the fact that if it was an Up day, how much up change it was.
    It can very possible be that the 50 up days achieved +0.1% gain in average, but the 26 down days gave -2.5% loss per day in average.
    In this case, going long is a disaster.

That is all for now. My input is ready. I have to teach this to a NN.
But I will cover that in the next post.


No Responses Yet to “Weekly directional prediction: preparing input data”

  1. Leave a Comment

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google+ photo

You are commenting using your Google+ account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )


Connecting to %s

%d bloggers like this: