Sensitivity Analysis on LookbackDays: 1D, 2D

23Oct11

See the 2 previous posts about the VXX estimation. We contended that there is only 1 parameter: the number of days we look back to get training samples.

To be frank however, there are more parameters:

–          whether selecting 1D or 2D case; or

–          the machine learning algorithm used: (Normal Equation or Gradient Descent, etc.), or

–          selecting the kind of instrument: VXX

However, let’s say that those parameters are not really parameters per se.

They were determined much earlier. They were determined by some other fundamental ideas we believe, and therefore we don’t optimize those parameters. For example just accept that we want to estimate daily VXX (not the RUT or AAPL). There is nothing really we can fine tune in it.

Therefore those parameters are not the focus of any sensitivity analysis.

After the prologue, let’s do some sensitivity analysis on the lookback days.

Note that we had to fix the startDay of those algorithms in theses backtest. Because we use maximum 200 days lookback, the first estimate can be calculated for the day: 201 (in the 1D case) and day: 202 (in the 2D case).

In this test, to make a fair competition between the different lookbackDays, we started all from day 201 or 202.

In theory, the 50 days lookback version can be started from day 51. However that would give extra advantage for the shorter period lookbackDays. (if they have larger period to play).

We want a fair comparison, so we cannot allow that.

Note that this is the reason, why for example in the previous post (1D) case we showed that lookbackDays = 50 was the best, achieving 10x multiplier.

This cannot be found here, because of the aforementioned reason.

 

Sensitivity Analysis (487 days = less than 2 years, assuming 250 trading days):

1. 1D case:

Let’s plot the final portfolioValue as a function of lookbackdays.

In the chart, the X axis is the lookbackDays – 1, so the chart is shifted by one, but it is OK.

(click on the image, if you want to see it proper original size)

based on that: the optimal lookback is somewhere between 30 to 60.                                                  Is it sensitive to the parameter? Yes, as it is usual. For example, the best parameter value gives 7x multiplier, the worst is 1x multiplier; so we can say it is quite sensitive to the parameter.

Note the range of 2-20 training samples: that is hardly enough samples; I wouldn’t consider those area to be useful at all, even if it shows a good performance.

So, the optimal value of the parameter is somewhere between 30 and 60. One strategy is (if we want to avoid parameter fine tuning) is just play middle: 45.

Do you see the danger here? Someone, who optimized the parameter and haven’t done any sensitivity analysis, thinks that it returns 7x per 2 years, and starts to play the strategy. But in real life, he can be unfortunate to have only 1-2x return in the future. (or he can be also lucky and get 14x return in the future.) The point her is that the future return should be expected less than it is shown in a fine tuned parameter backtests.

One idea to make it more stable:

Do different parameter runs (from 30 to 60): average their prediction; this may partially eliminate the parameter fine-tuning bias.

So, let’s define our UltimateEstimator by aggregating the decision of 30..60 lookbackdays.

The portfolio value curve of the UltimateEstimator gave PortfolioValue of 3.90:

UltimateEstimator is between the extremes: it is better than the worst (2x multiplier), but it is worse than the better 7x multiplier

However in real life, it is better to use this kind of estimator; it decreases the ‘lucky’/unlucky factor of the dependency of ‘parameter fine tuning’, concrete parameter selection.

Also, in general the aggregated Ultimate profit curve is smoother (less likely to contain DD), albeit, the -50% DD is still preset here, but even with that DD, it is smoother than the individual strategies

2. 2D case:

Let’s plot the final portfolioValue as a function of lookbackdays:

Based on that: the optimal lookback is somewhere from 70 to 115.

Is it sensitive to the parameter? Yes, as it is usual.

best parameter value: 7.5x multiplier, worst is 2x multiplier.

even with the unluckiest pick of the worst parameter, the profit was 2x (so, it is not a loss). That is good.

The only loss is in the range of 2-10, and 35-40. There are not enough training samples there.

Someone, who wants to avoid parameter fine tuning bias, may choose the middle of the range: 93.

Another idea to make it more stable: the same UltimateEstimator. Do different parameter runs (from 75 to 110): average their prediction; this may partially eliminate the parameter fine-tuning bias.

Aggregating the decision of 75..110 lookbackdays, the result UltimateEstimator gave PortfolioValue of 4.24

That is between the extremes: it is better than the worst (2x multiplier), but it is worse than the better 7x multiplier.   

3. Conclusion:

Note that with the Ultimate(75-110) version, we eliminated the parameter=fixLookbackdays, but we introduced 2 new parameters (instead of 1): 75 and 110.   :), so we again have some parameter bias; albeit note that we wanted to optimize the fixLookBackday parameters, but we haven’t ‘really’ optimized the range parameters: 75, 110.

The important note is that we introduced 2 new parameters, but the final result is not really sensitive to changing these 2 new parameters. Changing 75 to 76, hardly changes anything, while in the fixLookbackDays parameter case, changing that parameter from 93 to 94 had more significant effect on the final outcome.

 

This is the key message of this post: we cannot eliminate parameters, but what we can do is to assure that if we have parameters, the final outcome is not significantly sensitive on the parameters used.

Use 1D or 2D?

Comparing the 200 days long 1D vs. 2D sensitivity chart (not the Ultimate Portfolio Value chart), we prefer the 2D inputs case.

The maximum achieved is similar to the 1D case (max x7 multiplier was achieved: the 2D case achieved it about 3 occasions, the 1D only 1 times)

The minimum is better in the 2D case. In the 1D case, if we pick the wrong parameters, we can have a profit of 1x.

However, for the 2D case, even if we picked the wrong parameters, we can have a profit of 2x.

Comparing the range based Ultimate Portfolio Value chart,

The 2D case is better also, because for example the less DD. (see the big DD that we had in the last 3 weeks in the 1D case). The 2D case equity curve looks smoother too.

Advertisements


No Responses Yet to “Sensitivity Analysis on LookbackDays: 1D, 2D”

  1. Leave a Comment

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s


%d bloggers like this: