Regimes; the adaptive NN: optimal lookback days


In the previous post, we hinted to make the method to be adaptive to regime changes. This regime changes can stem from various sources. A typical example of the regime change is that prior 1998 the USA market (SP500) was daily trend following, meaning up days were followed by up days and down days were followed by down days. After 1998 it changed to mean reversion (MR) regime: up days are followed by down days and vice versa. There were a TF (trend following) regime for 100 years and ‘suddenly’ now there is a MR regime in the last 14 years.

Any strategy that uses all the past information (100 years) for training a machine learning algorithm (NN, Kalman filter, SVN, k-NN) trains the TF style of the market and it performs poorly in the last 14 years regime.
Corollary: design the algorithm to be adaptive to the current regime. Even if it cost you a couple of %gain profit less in the overall backtested performance, you have the cozy feeling that your algorithm can withstand the washing waves of times. I would even sacrifice half of the potential gain for the assurance that as the regime changes, our algorithm adapts to it.
Btw. the reason for the daily TF to MR regime change is not explained.
-Some pundits think it was the rise of the Internet and the Netscape browser that made it possible that small investors could access the market directly from their home computer.
-Others suggest that the reason was that transaction costs changed significantly around that time. The ask-bid difference is now 1 penny only. Previously it was a percentage of a dollar (like 1/5rd of a dollar).
-Also the commission costs are decreased. Therefore now, people are more willing to take short term profit. So, in an up day, people are more willing to cash in the gain in the next day than before. Overall, it increases the trading volume.
-But situation can be more complex. There are the computerized trading algorithms that try to make the market more efficient. Can we contribute the MR regime to them?
-Or there is the derivatives market, the index following ETFs that trades at the end of the day based on the index. If the index goes up, ETFs buy more, driving it up further.
-Other reason can be the Efficient Portfolio Allocation methods. Hedging with PUT options. When the market goes down, we need more protection, buying more PUTs. But if we buy more PUTs, somebody has to sell it. That somebody usually sell the PUTs, but as his protection, he sells the underlying stocks driving the market even further down into a spiral.
We have many other potential explanations.
Interestingly, the Russian or emerging markets are still in TF regime, but as the volume increases on them, we should expect them to be MR as well.

Optimal lookback days
Considering our day of the week anomaly, the Mondays are bearish by and large in the overall sample.
See image for the average next day daily %gain for the last 15 years.
Note Monday is 1,…. Friday is 5. The next day of Friday is bearish (bearish Monday).

But there were periods when Mondays were very bullish (2010 spring). Or fathom this: sometimes, when Mondays are bearish for an extended period, investors notice it and start to play against it. The corollary is that Mondays stop being bearish, but the bearish day shifts to be either Friday or Tuesday. For example, if everybody ‘knows’ (expects) that the Monday is bearish, nobody wants to sell on Monday. They try to sell it on the previous Friday. Therefore the trend shifts to Friday to become the bearish day.

Adapting a strategy to regimes is unavoidable in a serious trading system. One parameter in our system is the number of days N we look back in time for considering training samples. In a fully automatic system (level 2 adoption by Varadi), this adaption parameter is determined also by the system. However, in our tentative approach now, we seek to fix this parameter. What can be a quasi optimal N look back days? 50 or 100 or 200 or 500?
Let’s try an experiment for estimating it.

In our experiment, we get rid of the NN learning. We use a method that estimates the next day %gain by only giving back the arithmetic mean (average) of the previous samples coming from the last N days.

pastSamples = allSamples(d where d > today – N days);
forecast(day T) = mean(pastSamples(d where d.dayOfTheWeek = T.dayOfTheWeek);

This is a deterministic algorithm. It has some reminiscences to the k-NN machine learning method (k-nearest neighbour), where the forecast is made from the neighbourhood of the input in the learning algorithm. It is difficult to name this algorithm machine learning algorithm, but in theory it is; it learns from the past. Albeit not a sophisticated one.

We plug in different values for the days N parameter.
We measure the D_stat(%) (directionally accuracy statistics) that is the winning bets vs. the losing bets in percentage, our estimated CAGR% (Compounded Annual Growth Rate that is based on the arithmetic mean of the daily returns projected for 250 days) and the TR% (total return in that period that is a geometric mean of the daily returns). We plot them:

From this, we conclude that optimal lookback period for this simplistic learning algorithm is somewhere around 200. That is the lookback days that we will use in the upcoming experiments.


No Responses Yet to “Regimes; the adaptive NN: optimal lookback days”

  1. Leave a Comment

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

%d bloggers like this: