Matlab and automatic target normalization: mapminmax. Don’t trust your Matlab framework!

17Oct10

In this post we give a warning that the Matlab Neural Network Toolbax developer team made a grievous mistake. They tried to improve the usability, but they hide a potential erroneous problem behind the curtain.
In the new Matlab NN Toolbox, the team redesigned the usage of the ANN. This quote is from the official Matlab documentation:

For example, the following code returns a network that automatically preprocesses the inputs and targets and postprocesses the outputs:
net = newff(p,t,20);
net = train(net,p,t);
y = sim(net,p);
To create the same network in a previous release, you used the following longer code:
[p1,ps1] = removeconstantrows(p);
[p2,ps2] = mapminmax(p1);
[t1,ts1] = mapminmax(t);
pr = minmax(p2);
s2 = size(t1,1);
net = newff(pr,[20 s2]);
net = train(net,p2,t1);
y1 = sim(net,p2)
y = mapminmax(‘reverse’,y1,ts1);

Great! The inexperienced Matlab programmer is very happy with this development. (I was until I found the flaw.). Calling the ANN in the new Matlab version (7.9.0.5) is much simple now.
However, I simply contend in this post, that it is badly implemented. If we trust that the Matlab ANN automatically process our output, we get poor result. Albeit it works general, but in the limit case, the result is horrible.
Let’s try to forecast this function:

Here is the Matlab code:
TrainNumber=10;
x=[-2:0.01:2];
y=(exp(-1.9.*(x+0.5))).*sin(10*x);

xTest=[-2.005:0.01:2];
yTest=(exp(-1.9.*(xTest+0.5))).*sin(10*xTest);

multiplier = 0.00001;
y1 = y * multiplier;

err1Sum = 0;
err2Sum = 0;
testNumber = 100
for i = 1:testNumber
net=newff(x,y,20);
net.trainParam.epochs=TrainNumber;
net.trainParam.showWindow = 0;
net=train(net,x,y);
SimY=sim(net,xTest);

net1=newff(x,y1,20);
net1.trainParam.epochs=TrainNumber;
net1.trainParam.showWindow = 0;
net1=train(net1,x,y1);
SimY1=sim(net1,xTest) / multiplier;

hold on
plot(y,'black');
plot(SimY,'r');
plot(SimY1,'g');
hold off

err1Sum = err1Sum + mae(yTest - SimY);
err2Sum = err2Sum + mae(yTest - SimY1);
end
fprintf('\nerr1Avg = %.4f, err2Avg = %.4f,', err1Sum / testNumber, err2Sum /testNumber);

Note, I used the MAE = Mean Absolute Error for error measurement (usually, people use MSE, but it doesn’t matter).
The theory is that instead of ‘y’ as the output, I feed another network the same output scaled by a multiplier. For the Multiplier = 10000 case, you can see 100 plots here. (Note that for the Multiplier=100 case, you can see similar error patterns.) The red lines forecast the original series (without multiplier), while the green lines forecast the multiplied output, but the result is than scaled back to our problem space (by a dividing with the multiplier).
Overall, if the outputs are poorly scaled, the results can be (not always) horrible.

How can this happen? We trusted Matlab ANN Toolbox, that it does the automatic output normalization on behalf of us. What you can see in that plot is that it works ‘generally’, but not always. It is very risky to trust this automatic mechanism.

1.
What is the optimal multiplier?
See the table and the plot. The optimal multiplier is 0.01.

Note that if the Matlab team made a good work with this modification, in our previous table, all average MAE would have been equal, because all the problems are equivalent.

2.
Why is the 0.01 the optimal multiplier?
Our function Y range is [-16..+10]. Our theory is that if we multiply Y by only 0.1, the -16 will be mapped to -1.6, that is still not in the range of -1..+1. Note that the range -1..+1 is the perfect range for the TanSig transition function. So, we have to decrease the multiplier even more to 0.01 to get the most optimal result.

Conclusion:
The documentation of Matlab NN Toolbox states it handles the output normalization automatically by ‘mapminmax’. However, the prediction power of the ANN depends very much on which multiplier we use for the output (how we present the target data). So, the seemingly equal problems are not equal for the training. (We blame a little that Matlab did use the mapminmax as a cosmetic thing, but inside (we debugged the training code), the training algorithm, the performance calculation (MSE), and the mu, mu_max, etc. calculations still uses the un-normalized, original space, and not the normalized, mapminmax-ed space. This causes the difference in training.)

We cannot trust the Matlab automatic (mapminmax) output normalization. We have to scale the output manually, if we want to avoid that the learning algorithm runs amok. The optimal scaling maps the output into a space where even the outliers (+10, -16 in this case) are in the range of -1..+1. However, scaling it a little further, like into a range of -0.1..+0.1 worsen the result.

Advertisements


2 Responses to “Matlab and automatic target normalization: mapminmax. Don’t trust your Matlab framework!”

  1. 1 Ari

    That’s exactly correct and I wonder if there is any way to stop nn doing that by default.


Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s


%d bloggers like this: