Neural Network Toolbox

A Tutorial for the Course Einfuehrung in die Wissensverarbeitung
http://www.igi.tugraz.at/lehre/EW

Stefan Husler a

Institute for Theoretical Computer Science Ineldgasse 16b/I Abstract
This tutorial gives an introduction to the Matlab Neural Network Toolbox. The elements of matlab and the neural network toolbox are more easily understood when explained by an example. First a neural network will be used for a classication task. The second example will be a simple logical problem.

Matlab Files for SS07

Here you can nd the.m les for the rst practical lesson : Introduction to NNT.zip1
To make full use of this tutorial you have 1. to download the le nnt intro.zip2 which contains this tutorial and the accompanying Matlab programs. 2. Unzip nnt intro.zip which will generate a subdirectory named nnt intro where you can nd all the Matlab programs. 3. Add the path nnt intro to the matlab search path with a command like addpath(C:\Work\nnt intro) if you are using a Windows machine or addpath(/home/jack/nnt intro) if you are using a Unix/Linux machine.
The Neural Network Toolbox
The neural network toolbox makes it easier to use neural networks in matlab. The toolbox consists of a set of functions and structures that handle neural networks, so we do not need to write code for all activation functions, training algorithms, etc. that we want to use! The Neural Network Toolbox is contained in a directory called nnet. Type help nnet for a listing of help topics. A number of demonstrations are included in the toolbox. Each example states a problem, shows
1 Introduction_to_NNT.zip 2 http://www.igi.tugraz.at/lehre/CI/tutorials/nnt_intro.zip
the network used to solve the problem, and presents the nal results. Lists of the demos and applications scripts that are discussed in this guide can be found with help nndemos/.
The Structure of the Neural Network Toolbox
The toolbox is based on the network object. This object contains information about everything that concern the neural network, e.g. the number and structure of its layers, the conectivity between the layers, etc. Matlab provides high-level network creation functions, like newlin (create a linear layer), newp (create a perceptron) or new (create a feed-forward backpropagation network) to allow an easy construction of. As an example we construct a perceptron with two inputs ranging from -2 to 2: >> net = newp([-2 2;-2 2],1) First the architecture parameters and the subobject structures subobject structures: inputs: layers: outputs: targets: biases: inputWeights: layerWeights: {1x1 {1x1 {1x1 {1x1 {1x1 {1x1 {1x1 cell} cell} cell} cell} cell} cell} cell} of inputs of layers containing containing containing containing containing
1 output 1 target 1 bias 1 input weight no layer weights
are shown. The latter contains information about the individual objects of the network. Each layer consists of neurons with the same transfer function net.transferFcn and net input function net.netInputFcn, which are in the case of perceptrons hardlim and netsum. If neurons should have dierent transfer functions then they have to be arranged in dierent layers. The parameters net.inputWeights and net.layerWeights specify among other things the applied learning functions and their parameters. The next paragraph contains the training, initialization and performance functions. functions: adaptFcn: initFcn: performFcn: trainFcn: trains initlay mae trainc
The trainFcn and adaptFcn are used for the two dierent learning types batch learning and incremental or on-line learning. By setting the trainFcn parameter you tell Matlab which training algorithm should be used, which is in our case the cyclical order incremental training/learning function trainc. The ANN toolbox include almost 20 training functions. The performance function is the function that determines how well the ANN is doing its task. For a perceptron it is the mean absolute error performance function mae. For linear regression usually the mean squared error performance function mse is used. The initFcn is the function that initialized the weights and biases of the network. To get a list of the functions that are available type help nnet. To change one of these functions to another one in the toolbox or one that you have created, just assign the name of the function to the parameter, e.g. >> net.trainFcn = mytrainingfun; The parameters that concerns these functions are listed in the next paragraph.

parameters: adaptParam: initParam: performParam: trainParam:.passes (none) (none).epochs,.goal,.show,.time
By changing these parameters you can change the default behavior of the functions mentioned above. The parameters you will use the most are probably the components of trainParam. The most used of these are net.trainParam.epochs which tells the algorithm the maximum number of epochs to train, and net.trainParam.show that tells the algorithm how many epochs there should be between each presentation of the performance. Type help train for more information. The weights and biases are also stored in the network structure: weight and bias values: IW: {1x1 cell} containing 1 input weight matrix LW: {1x1 cell} containing no layer weight matrices b: {1x1 cell} containing 1 bias vector The.IW(i,j) component is a two dimensional cell matrix that holds the weights of the connection between the input j and the network layer i. The.LW(i,j) component holds the weight matrix for the connection from the network layer j to the layer i. The cell array b contains the bias vector for each layer.

A Classication Task

As example our task is to create and train a perceptron that correctly classies points sets belonging to three dierent classes. First we load the data from the le winedata.mat >> load winedata X C Each row of X represents a sample point whose class is specied by the corresponding element (row) in C. Further the data is transformed into the input/output format used by the Neural Network Toolbox >> P=X; where P(:,i) is the ith point. Since we want to classify three dierent classes we use 3 perceptrons, each for the classication of one class. The corresponding target function is generated by >> T=ind2vec(C); To create the perceptron layer with correct input range type >> net=newp(minmax(P),size(T,1));
The dierence between train and adapt
Both functions, train and adapt, are used for training a neural network, and most of the time both can be used for the same network. The most important dierence has to do with incremental training (updating the weights after the presentation of each single training sample) versus batch training (updating the weights after each presenting the complete data set).
Data Set 6 Class 1 Class 2 Class 3 5

Feature 2

13 Feature 1
Figure 1: Data set X projected to two dimensions.
Adapt First, set net.adaptFcn to the desired adaptation function. Well use adaptwb (from adapt weights and biases), which allows for a separate update algorithm for each layer. Again, check the Matlab documentation for a complete overview of possible update algorithms. >> net.adaptFcn = trains; Next, since were using trains, well have to set the learning function for all weights and biases: >> net.inputWeights{1,1}.learnFcn = learnp; >> net.biases{1}.learnFcn = learnp; where learnp is the Perceptron learning rule. Finally, a useful parameter is net.adaptParam.passes, which is the maximum number of times the complete training set may be used for updating the network: 4

>> net.adaptParam.passes = 1; When using adapt, both incremental and batch training can be used. Which one is actually used depends on the format of your training set. If it consists of two matrices of input and target vectors, like >> [net,y,e] = adapt(net,P,T); the network will be updated using batch training. Note that all elements of the matrix y are one, because the weights are not updated until all of the trainings set had been presented. If the training set is given in the form of a cell array >> for i = 1:length(P), P2{i} = P(:,i); T2{i}= T(:,i); end >> net = init(net); >> [net,y2,e2] = adapt(net,P2,T2); then incremental training will be used. Notice that the weights had to be initialized before the network adaption was started. Since adapt takes a lot more time then train we continue our analysis with second algorithm. Train When using train on the other hand, only batch training will be used, regardless of the format of the data (you can use both). The advantage of train is that it provides a lot more choice in training functions (gradient descent, gradient descent w/ momentum, Levenberg-Marquardt, etc.) which are implemented very eciently. So for static networks (no tapped delay lines) usually train is the better choice. We set >> net.trainFcn = trainb; for batch learning and >> net.trainFcn = trainc; for on-line learning. Which training parameters are present depends in general on your choice for the training function. In our case two useful parameters are net.trainParam.epochs, which is the maximum number of times the complete data set may be used for training, and net.trainParam.show, which is the time between status reports of the training function. For example, >> net.trainParam.epochs = 1000; >> net.trainParam.show = 100; We initialize and simulate the network with >> net = init(net); >> [net,tr] = train(net,P,T); The trainings error is calculated with >> Y=sim(net,P); >> train_error=mae(Y-T) train_error = 0.3801 So we see that the three classes of the data set were not linear seperable. The best time to stop learning would have been
>> [min_perf,min_epoch]=min(tr.perf) min_perf = 0.1948 min_epoch = 703
Figure 2: Performance of the learning algorithm train over 1000 epochs.

A Simple logical problem

The task is to create and train a neural network that solves the XOR problem. XOR is a function that returns 1 when the two inputs are not equal,

Construct a Feed-Forward Network
To solve this we will need a feedforward neural network with two input neurons, and one output neuron. Because that the problem is not linearly separable it will also need a hidden layer with two neurons.To create a new feed forward neural network use the command new. You have to enter the max and min of the input values, the number of neurons in each layer and optionally the activation functions. >> net = newff([0 1; 0 1],[2 1],{logsig,logsig}); The variable net will now contain an untrained feedforward neural network with two neurons in the input layer, two neurons in the hidden layer and one output neuron, exactly as we want it. The [0 1; 0 1] tells matlab that the input values ranges between 0 and 1. The logsig,logsig tells matlab that we want to use the logsig function as activation function in all layers. The rst parameter tells the network how many nodes there should be in the input layer, hence you do not have to specify this in the second parameter. You have to specify at least as many transfer functions as there are layers, not counting the input layer. If you do not specify any transfer function Matlab will use the default settings. First we construct a matrix of the inputs. The input to the network is always in the columns of the matrix. To create a matrix with the inputs 1 1, 1 0, and we enter: >> input = [0 0; 1 0] input = 1 1
Further we construct the target vector: >> target = [1 0] target = 0
Train the Network via Backpropagation
In this example we do not need all the information that the training algorithms shows, so we turn it of by entering: >> net.trainParam.show=NaN; Let us apply the default training algorithm Levenberg-Marquardt backpropagation trainlm to our network. An additional training parameters is.min grad. If the gradient of the performance is less than.min grad the training is ended. To train the network enter: >> net = train(net,input,target);
Because of the small size of the network, the training is done in only a second or two. Now we simulate the network, to see how it reacts to the inputs: >> output = sim(net,input) output = 0.0000 1.0000 1.0000 0.0000
That was exactly what we wanted the network to output! Now examine the weights that the training algorithm has set 7
>> net.IW{1,1} ans = 11.0358 16.8909

-9.5595 -17.5570

>> net.LW{2,1} ans = 25.9797

-25.7624

Graphical User Interface
Creat networks Enter data into the GUI Initialize, train, and simulate networks Export the training results from the GUI to the command line workspace Import data from the command line workspace to the GUI
A graphical user interface has been added to the toolbox. This interface allows you to:
To open the Network/Data Manager window type nntool.

Mathematical Notation for Equations and Figures

Basic Concepts

Scalars-small italic letters..a,b,c Vectors - small bold non-italic letters..a,b,c Matrices - capital BOLD non-italic letters..A,B,C

Language

Vector means a column of numbers.

Weight Matrices

Scalar Element w i, j ( t )
i - row, j - column, t - time or iteration
Matrix W ( t ) Column Vector w j ( t ) Row Vector iw ( t ).vector made of ith row of weight matrix W Bias Vector Scalar Element b i ( t ) Vector b ( t )

Layer Notation

A single superscript is used to identify elements of layer. For instance, the net input of layer 3 would be shown as n3. Superscripts k, l are used to identify the source (l) connection and the destination (k) connection of layer weight matrices ans input weight matrices. For instance, the layer weight matrix from layer 2 to layer 4 would be shown as LW4,2.

Input Weight Matrix IW

k, l k, l

Layer Weight Matrix LW

Figure and Equation Examples
The following figure, taken from Chapter 12 illustrates notation used in such advanced figures.

Inputs

Layers 1 and 2

Layer 3

Outputs

1 x (1*1)

a1(k) 4
a1(k) = tansig (IW1,1p1(k) +b1) a2(k)

1 y1(k)

3 x (2*2)

3 x (1*5)

a2(k) = logsig (IW2,1 [p1(k);p1(k-1) ]+ IW2,2p2(k-1)) a3(k)=purelin(LW3,3a3(k-1)+IW3,1 a1 (k)+b3+LW3,2a2 (k))
Mathematics and Code Equivalents
The transition from mathematics to code or vice versa can be made with the aid of a few rules. They are listed here for future reference. To change from mathematics notation to MATLAB notation, the user needs to: Change superscripts to cell array indices. For example, p p { 1 } Change subscripts to parentheses indices. For example, p 2 p ( 2 ) , and p 2 p { 1 } ( 2 ) Change parentheses indices to a second cell array index. For example, p ( k 1 ) p { 1, k 1 } Change mathematics operators to MATLAB operators and toolbox functions. For example, ab a*b The following equations illustrate the notation used in figures. n = w 1, 1 p 1 + w 1, 2 p 2 +. + w 1, R p R + b w 1, 1 w 1, 2 w 1, R W = w 2, 1 w 2, 2 w 2, R w S, 1 w S, 2 w S, R
Neural Network Design Book
Professor Martin Hagan of Oklahoma State University, and Neural Network Toolbox authors Howard Demuth and Mark Beale have written a textbook, Neural Network Design, published by the Brooks/Cole Publishing Company in 1996 (ISBN 0-534-94332-2). The book presents the theory of neural networks, discusses their design and application, and makes considerable use of MATLAB and the Neural Network Toolbox. Demonstration programs from the book are used in various chapters of this Guide. (You can find all the book demonstration programs in the Neural Network Toolbox by typing nnd.) The book has: An INSTRUCTORS MANUAL (ISBN 0-534-95049-3) for adopters and TRANSPARENCY OVERHEADS for class use. The overheads come one to a page for instructor use and three to a page for student use. To place an order for the book, call 1-800-354-9706. To obtain a copy of the INSTRUCTORS MANUAL, contact the Brooks/Cole Editorial Office, phone 1-800-354-0092. Ask specifically for an instructors manual if you are instructing a class and want one. You can go directly to the Brooks/Cole Neural Network Design page at

The neuron has a bias b, which is summed with the weighted inputs to form the net input n. This sum, n, is the argument of the transfer function f. n = w 1, 1 p 1 + w 1, 2 p 2 +. + w 1, R p R + b This expression can, of course, be written in MATLAB code as:

n = W*p + b

However, the user will seldom be writing code at this low level, for such code is already built into functions to define and simulate entire networks.
The figure of a single neuron shown above contains a lot of detail. When we consider networks with many neurons and perhaps layers of many neurons, there is so much detail that the main thoughts tend to be lost. Thus, the authors have devised an abbreviated notation for an individual neuron. This notation, which will be used later in circuits of multiple neurons, is illustrated in the diagram shown below.

Input p

Neuron a n

1x1 1x1

Where. R = number of elements in input vector

a = f(Wp +b)

Here the input vector p is represented by the solid dark vertical bar at the left. The dimensions of p are shown below the symbol p in the figure as Rx1. (Note that we will use a capital letter, such as R in the previous sentence, when referring to the size of a vector.) Thus, p is a vector of R input elements. These inputs post multiply the single row, R column matrix W. As before, a constant 1 enters the neuron as an input and is multiplied by a scalar bias b. The net input to the transfer function f is n, the sum of the bias b and the product Wp. This sum is passed to the transfer function f to get the neurons output a, which in this case is a scalar. Note that if we had more than one neuron, the network output would be a vector. A layer of a network is defined in the figure shown above. A layer includes the combination of the weights, the multiplication and summing operation (here realized as a vector product Wp), the bias b, and the transfer function f. The array of inputs, vector p, is not included in or called a layer. Each time this abbreviated network notation is used, the size of the matrices will be shown just below their matrix variable names. We hope that this notation will allow you to understand the architectures and follow the matrix mathematics associated with them.

net = newc([0 1; 0 1],2);
The weights are initialized to the center of the input ranges with the function midpoint. We can check to see these initial values as follows:
wts = net.IW{1,1} wts = 0.5000 0.5000 0.5000 0.5000
These weights are indeed the values at the midpoint of the range (0 to 1) of the inputs, as we would expect when using midpoint for initialization. The biases are computed by initcon, which gives

biases = 5.4366 5.4366

Now we have a network, but we need to train it to do the classification job. Recall that each neuron competes to respond to an input vector p. If the biases are all 0, the neuron whose weight vector is closest to p gets the highest net input and, therefore, wins the competition and outputs 1. All other neurons output 0. We would like to adjust the winning neuron so as to move it closer to the input. A learning rule to do this is discussed in the next section.
Kohonen Learning Rule (learnk)
The weights of the winning neuron (a row of the input weight matrix) are adjusted with the Kohonen learning rule. Supposing that the ith neuron wins, the elements of the ith row of the input weight matrix are adjusted as shown below.

iIW 1, 1

( q ) = iIW

( q 1 ) + ( p ( q ) iIW

(q 1))
The Kohonen rule allows the weights of a neuron to learn an input vector, and because of this it is useful in recognition applications. Thus, the neuron whose weight vector was closest to the input vector is updated to be even closer. The result is that the winning neuron is more likely to win the competition the next time a similar vector is presented, and less likely to win when a very different input vector is presented. As more and more inputs are presented, each neuron in the layer closest to a group of input vectors soon adjusts its weight vector toward those input vectors. Eventually, if there are enough neurons, every cluster of similar input vectors will have a neuron that outputs 1 when a vector in the cluster is presented, while outputting a 0 at all other times. Thus, the competitive network learns to categorize the input vectors it sees. The function learnk is used to perform the Kohonen learning rule in this toolbox.
Bias Learning Rule (learncon)
One of the limitations of competitive networks is that some neurons may not always get allocated. In other words, some neuron weight vectors may start out far from any input vectors and never win the competition, no matter how long the training is continued. The result is that their weights do not get to learn and they never win. These unfortunate neurons, referred to as dead neurons, never perform a useful function. To stop this from happening, biases are used to give neurons that only win the competition rarely (if ever) an advantage over neurons that win often. A positive bias, added to the negative distance, makes a distant neuron more likely to win. To do this job a running average of neuron outputs is kept. It is equivalent to the percentages of times each output is 1. This average is used to update the biases with the learning function learncon so that the biases of frequently

learning function chosen by the user. The function learngdm is recommended. Elman networks are not as reliable as some other kinds of networks because both training and adaption happen using an approximation of the error gradient. For an Elman to have the best chance at learning a problem it needs more hidden neurons in its hidden layer than are actually required for a solution by another method. While a solution may be available with fewer neurons, the Elman network is less able to find the most appropriate weights for hidden neurons since the error gradient is approximated. Therefore, having a fair number of neurons to begin with makes it more likely that the hidden neurons will start out dividing up the input space in useful ways. The function train trains an Elman network to generate a sequence of target vectors when it is presented with a given sequence of input vectors. The input vectors and target vectors are passed to train as matrices P and T. Train takes these vectors and the initial weights and biases of the network, trains the network using backpropagation with momentum and an adaptive learning rate, and returns new weights and biases. Let us continue with the example of the previous section, and suppose that we want to train a network with an input P and targets T as defined below
P = round(rand(1,8)) P = 1
T = [0 (P(1:end-1)+P(2:end) == 2)] T = 1 0
Here T is defined to be 0, except when two 1s occur in P, in which case T is 1. As noted previously, our network has five hidden neurons in the first layer.
We use trainbfg as the training function and train for 100 epochs. After training we simulate the network with the input P and calculate the difference between the target output and the simulated network output.
net = train(net,Pseq,Tseq); Y = sim(net,Pseq); z = seq2con(Y); z{1,1}; diff1 = T - z{1,1}
Note that the difference between the target and the simulated output of the trained network is very small. Thus, the network is trained to produce the desired output sequence on presentation of the input vector P. See Chapter 11 for an application of the Elman network to the detection of wave amplitudes.

Hopfield Network

The goal here is to design a network that stores a specific set of equilibrium points such that, when an initial condition is provided, the network eventually comes to rest at such a design point. The network is recursive in that the output is fed back as the input, once the network is in operation. Hopefully, the network output will settle on one of the original design points The design method that we present is not perfect in that the designed network may have undesired spurious equilibrium points in addition to the desired ones. However, the number of these undesired points is made as small as possible by the design method. Further, the domain of attraction of the designed equilibrium points is as large as possible. The design method is based on a system of first-order linear ordinary differential equations that are defined on a closed hypercube of the state space. The solutions exist on the boundary of the hypercube. These systems have the basic structure of the Hopfield model, but are easier to understand and design than the Hopfield model. The material in this section is based on the following paper: Jian-Hua Li, Anthony N. Michel and Wolfgang Porod, Analysis and synthesis of a class of neural networks: linear systems operating on a closed hypercube, IEEE Trans. on Circuits and Systems vol 36, no. 11, pp. 1405-22, November 1989. For further information on Hopfield networks, read Chapter 18 of the Hopfield Network [HDB96].

The architecture of the network that we are using follows.

a1(k-1)

LW1,1 a1(0) 1

Initial conditions

Symmetric saturated linear layer a1(0) = p and then for k = 1, 2,. a1(k) = satlins (LW1,1a1(k-1)) + b1)
As noted, the input p to this network merely supplies the initial conditions. The Hopfield network uses the saturated linear transfer function satlins.

a +1 n -+1 -1

a = satlins(n) Satlins Transfer Function
For inputs less than -1 satlins produces -1. For inputs in the range -1 to +1 it simply returns the input value. For inputs greater than +1 it produces +1. This network can be tested with one or more input vectors which are presented as initial conditions to the network. After the initial conditions are given, the network produces an output which is then fed back to become the input. This process is repeated over and over until the output stabilizes. Hopefully, each
output vector eventually converges to one of the design equilibrium point vectors that is closest to the input that provoked it.

Design (newhop)

Li et. al. [LiMi89] have studied a system that has the basic structure of the Hopfield network but is, in Lis own words, easier to analyze, synthesize, and implement than the Hopfield model. The authors are enthusiastic about the reference article, as it has many excellent points and is one of the most readable in the field. However, the design is mathematically complex, and even a short justification of it would burden this guide. Thus, we present the Li design method, with thanks to Li et al., as a recipe that is found in the function newhop. Given a set of target equilibrium points represented as a matrix T of vectors,
newhop returns weights and biases for a recursive network. The network is
guaranteed to have stable equilibrium points at the target vectors, but it could contain other spurious equilibrium points as well. The number of these undesired points is made as small as possible by the design method. Once the network has been designed, it can be tested with one or more input vectors. Hopefully those input vectors close to target equilibrium points will find their targets. As suggested by the network figure, an array of input vectors may be presented at one time or in a batch. The network proceeds to give output vectors that are fed back as inputs. These output vectors can be can be compared to the target vectors to see how the solution is proceeding. The ability to run batches of trial input vectors quickly allows you to check the design in a relatively short time. First you might check to see that the target equilibrium point vectors are indeed contained in the network. Then you could try other input vectors to determine the domains of attraction of the target equilibrium points and the locations of spurious equilibrium points if they are present. Consider the following design example. Suppose that we want to design a network with two stable points in a three-dimensional space.

net.biases{1} net.biases{3} net.inputWeights{1,1} net.inputWeights{2,1} net.inputWeights{2,2} net.layerWeights{3,1} net.layerWeights{3,2} net.layerWeights{3,3}
For example, typing net.biases{1} results in the following output.
ans = initFcn: learn: learnFcn: learnParam: '' 1 '' ''
size: 4 userdata: [1x1 struct]
Specify the weights tap delay lines in accordance with the network diagram, by setting each weights delays property.
net.inputWeights{2,1}.delays = [0 1]; net.inputWeights{2,2}.delays = 1; net.layerWeights{3,3}.delays = 1;

Network Functions

Type net and press return again to see the next set of properties.
functions: adaptFcn: initFcn: performFcn: trainFcn: (none) (none) (none) (none)
Each of these properties defines a function for a basic network operation. Set the initialization function to initlay so the network initializes itself according to the layer initialization functions that we have already set to initnw the Nguyen-Widrow initialization function.

net.initFcn = 'initlay';

This meets the initialization requirement of our network. Set the performance function to mse (mean squared error) and the training function to trainlm (Levenberg-Marquardt backpropagation) to meet the final requirement of the custom network.
net.performFcn = 'mse'; net.trainFcn = 'trainlm';

Weight and Bias Values

Before initializing and training the network, take a look at the final group of network properties (aside from the userdata property).
weight and bias values: IW: {3x2 cell} containing 3 input weight matrices
LW: {3x3 cell} containing 3 layer weight matrices b: {3x1 cell} containing 2 bias vectors
These cell arrays contain weight matrices and bias vectors in the same positions that the connection properties (net.inputConnect, net.layerConnect, net.biasConnect) contain 1s and the subobject properties (net.inputWeights, net.layerWeights, net.biases) contain structures. Evaluating each of the following lines of code reveals that all the bias vectors and weight matrices are set to zeros.

LS is a structure of the learning state that is updated for each call. (Use a null matrix [] the first time.) dW is the resulting S R weight change matrix. Your function is called as follows to update bias vector
[db,LS] = yourwblf(b,ones(1,Q),Z,N,A,T,E,gW,gA,D,LP,LS)
where: S is the number of neurons in the layer. b is a new S 1 bias vector. Your learning function must also provide information about itself using this calling format,

info = yourwblf(code)

where the correct information is returned for each of the following string codes: 'version' - Returns the Neural Network Toolbox version (3.0). 'deriv' - Returns the name of the associated derivative function. 'pdefaults' - Returns a structure of default performance parameters. To see how an example custom weight and bias initialization function works, type

help mywblf

type mywblf
You can use mywblf as a template to create your own weight and bias learning function.
Self-Organizing Map Functions
There are two kinds of functions that control how neurons in self-organizing maps respond. They are topology and distance functions.

Topology Functions

Topology functions calculate the positions of a layers neurons given its dimensions.
Once defined, you can assign your topology function to any layer of a network. For example, the following line of code assigns the topology function yourtopf to the second layer of a network.
net.layers{2}.topologyFcn = 'yourtopf';
Your topology function is used whenever your network is trained or adapts.
To be a valid topology function your function must calculate positions pos from dimensions dim as follows,
pos = yourtopf(dim1,dim2,.,dimN)
where: dimi is the number of neurons along the ith dimension of the layer. pos is an N S matrix of S position vectors, where S is the total number of neurons that is defined by the product dim1*dim1*.*dimN. The toolbox contains an example custom topology function called mytopf. Enter the following lines of code to see how it is used.

14-125

14logsig

Log sigmoid transfer function
A = logsig(N) info = logsig(code)
logsig is a transfer function. Transfer functions calculate a layers output from its net input. logsig(N) takes one input, N - S x Q matrix of net input (column) vectors.
and returns each element of N squashed between 0 and 1.
logsig(code) returns useful information for each code string:
Here is the code to create a plot of the logsig transfer function.
n = -5:0.1:5; a = logsig(n); plot(n,a)
You can create a standard network that uses logsig by calling newff or newcf.

14-126

To change a network so a layer uses logsig, set net.layers{i}.transferFcn to 'logsig'. In either case, call sim to simulate the network with purelin. See newff or newcf for simulation examples.
logsig(n) = 1 / (1 + exp(-n)) sim, dlogsig, tansig

14-127

Mean absolute error performance function
perf = mae(E,X,PP) perf = mae(E,net,PP) info = mae(code)
mae is a network performance function. mae(E,X,PP) takes from one to three arguments, E - Matrix or cell array of error vector(s). X - Vector of all weight and bias values (ignored). PP - Performance parameters (ignored).
and returns the mean absolute error. The errors E can be given in cell array form,
E - Nt x TS cell array, each element E{i,ts} is a Vi x Q matrix or[].

or as a matrix,

E - (sum of Vi) x Q matrix
Nt = net.numTargets TS = Number of time steps Q = Batch size Vi = net.targets{i}.size mae(E,net,PP) can take an alternate argument to X, net - Neural network from which X can be obtained (ignored). mae(code) returns useful information for each code string:
'deriv' - Name of derivative function. 'name' - Full name. 'pnames' - Names of training parameters. 'pdefaults' - Default training parameters.

14-128

Here a perceptron is created with a 1-element input ranging from -10 to 10, and one neuron.

net = newp([-10 10],1);

Here the network is given a batch of inputs P. The error is calculated by subtracting the output A from target T. Then the mean absolute error is calculated.
p = [-10 -5 10]; t = [1]; y = sim(net,p) e = t-y perf = mae(e)
Note that mae can be called with only one argument because the other arguments are ignored. mae supports those arguments to conform to the standard performance function argument list.

regression between each element of the network response and the corresponding target.
postreg(A,T) takes these inputs, A - 1 x Q array of network outputs. One element of the network output. T - 1 x Q array of targets. One element of the target vector.
M - Slope of the linear regression. B - Y intercept of the linear regression. R - Regression R-value. R=1 means perfect correlation.
In this example we normalize a set of training data with prestd, perform a principal component transformation on the normalized data, create and train a network using the pca data, simulate the network, unnormalize the output of the network using poststd, and perform a linear regression between the network outputs (unnormalized) and the targets to check the quality of the network training.
p = [-0.92 0.73 -0.47 0.74 0.29; -0.08 0.86 -0.67 -0.52 0.93]; t = [-0.08 3.4 -0.82 0.69 3.1]; [pn,meanp,stdp,tn,meant,stdt] = prestd(p,t); [ptrans,transMat] = prepca(pn,0.02); net = newff(minmax(ptrans),[5 1],{'tansig''purelin'},'trainlm'); net = train(net,ptrans,tn); an = sim(net,ptrans); a = poststd(an,meant,stdt); [m,b,r] = postreg(a,t);
Performs a linear regression between the network response and the target, and then computes the correlation coefficient (R-value) between the network response and the target.

premnmx, prepca

14-204

14poststd

Postprocess data which has been preprocessed by prestd
[P,T] = poststd(PN,meanp,stdp,TN,meant,stdt) [p] = poststd(PN,meanp,stdp)
poststd postprocesses the network training set that was preprocessed by prestd. It converts the data back into unnormalized units. poststd takes these inputs, PN - R x Q matrix of normalized input vectors. - R x 1 vector containing standard deviations for each P. - S x Q matrix of normalized target vectors. - S x 1 vector containing standard deviations for each T.
meanp - R x 1 vector containing standard deviations for each P. stdp TN
meant - S x 1 vector containing standard deviations for each T. stdt

Ni = net.numInputs Nl = net.numLayers Nt = net.numTargets ID = net.numInputDelays LD = net.numLayerDelays TS = Number of time steps Q

= Batch size

Ri = net.inputs{i}.size Si = net.layers{i}.size Vi = net.targets{i}.size
The columns of Pi, Pf, Ai, and Af are ordered from the oldest delay condition to the most recent:
The matrix format can be used if only one time step is to be simulated (TS = 1). It is convenient for networks with only one input and output, but can be used with networks that have more.

14-257

Pi - (sum of Ri) x (ID*Q) matrix. Ai - (sum of Si) x (LD*Q) matrix. Y E - (sum of Ui) x Q matrix. - (sum of Vi) x Q matrix
Pf - (sum of Ri) x (ID*Q) matrix. Af - (sum of Si) x (LD*Q) matrix.
If VV and TV are supplied they should be an empty matrix [] or a structure with the following fields:
VV.P, VV.T, TV.P TV.T - Validation/test inputs. - Validation/test targets, default = zeros.
VV.Pi, TV.Pi - Validation/test initial input delay conditions, default =

zeros.

VV.Ai, TV.Ai - Validation/test layer delay conditions, default = zeros.
The validation vectors are used to stop training early if further training on the primary vectors will hurt generalization to the validation vectors. Test vector performance can be used to measure how well the network generalizes beyond primary and validation vectors. If VV.T, VV.Pi, or VV.Ai are set to an empty matrix or cell array, default values will be used. The same is true for TV.T, TV.Pi, TV.Ai.
Here input P and targets T define a simple function which we can plot:
p = [8]; t = [0 0.84 0.91 0.14 -0.77 -0.96 -0.28 0.66 0.99]; plot(p,t,'o')
Here newff is used to create a two-layer feed-forward network. The network will have an input (ranging from 0 to 8), followed by a layer of 10 tansig neurons, followed by a layer with 1 purelin neuron. trainlm backpropagation is used. The network is also simulated.
net = newff([0 8],[10 1],{'tansig' 'purelin'},'trainlm');

14-258

y1 = sim(net,p) plot(p,t,'o',p,y1,'x')
Here the network is trained for up to 50 epochs to a error goal of 0.01, and then resimulated.
net.trainParam.epochs = 50; net.trainParam.goal = 0.01; net = train(net,p,t); y2 = sim(net,p) plot(p,t,'o',p,y1,'x',p,y2,'*')
train calls the function indicated by net.trainFcn, using the training parameter values indicated by net.trainParam.
Typically one epoch of training is defined as a single presentation of all input vectors to the network. The network is then updated according to the results of all those presentations. Training occurs until a maximum number of epochs occurs, the performance goal is met, or any other stopping condition of the function net.trainFcn occurs. Some training functions depart from this norm by presenting only one input vector (or sequence) each epoch. An input vector (or sequence) is chosen randomly each epoch from concurrent input vectors (or sequences). newc and newsom return networks that use trainr, a training function that presents each input vector once in random order.

'pnames'

- Names of training parameters.
You can create a standard network that uses trains for adapting by calling newp or newlin. To prepare a custom network to adapt with trains:
1 Set net.adaptFcn to 'trains'.
(This will set net.adaptParam to trainss default parameters.)
2 Set each net.inputWeights{i,j}.learnFcn to a learning function. 3 Set each net.layerWeights{i,j}.learnFcn to a learning function. 4 Set each net.biases{i}.learnFcn to a learning function. (Weight and bias
learning parameters will automatically be set to default values for the given learning function.)

14-327

To allow the network to adapt:
1 Set weight and bias learning parameters to desired values. 2 Call adapt.
See newp and newlin for adaption examples.
Each weight and bias is updated according to its learning function after each time step in the input sequence.
newp, newlin, train, trainb, trainc, trainr

14-328

14trainscg
Scaled conjugate gradient backpropagation
[net,TR,Ac,El] = trainscg(net,Pd,Tl,Ai,Q,TS,VV,TV) info = trainscg(code)
trainscg is a network training function that updates weight and bias values
according to the scaled conjugate gradient method.
trainscg(net,Pd,Tl,Ai,Q,TS,VV,TV) takes these inputs, net - Neural network. Pd Tl Ai Q TS VV TV - Delayed input vectors. - Layer target vectors. - Initial input delay conditions. - Batch size. - Time steps. - Either empty matrix [] or structure of validation vectors. - Either empty matrix [] or structure of test vectors.

14-329

Training occurs according to the trainscgs training parameters shown here with their default values:
net.trainParam.epochs net.trainParam.show net.trainParam.goal net.trainParam.time net.trainParam.min_grad net.trainParam.max_fail net.trainParam.sigma net.trainParam.lambda 0 inf 1e-6 5
Maximum number of epochs to train Epochs between showing progress Performance goal Maximum time to train in seconds Minimum performance gradient Maximum validation failures Determines change in weight for second derivative approximation. Parameter for regulating the indefiniteness of the Hessian.

If we then receive new inputs to apply to the trained network, we will use tramnmx to transform them first. Then the transformed inputs can be used to simulate the previously trained network. The network output must also be unnormalized using postmnmx.
p2 = [4 -7]; [p2n] = tramnmx(p2,minp,maxp); an = sim(net,pn); [a] = postmnmx(an,mint,maxt);
pn = 2*(p-minp)/(maxp-minp) - 1; premnmx, prestd, prepca, trastd, trapca

14-333

14trapca
Principal component transformation
[Ptrans] = trapca(P,transMat) trapca preprocesses the network input training set by applying the principal component transformation that was previously computed by prepca. This
function needs to be used when a network has been trained using data normalized by prepca. All subsequent inputs to the network need to be transformed using the same normalization.
trapca(P,transMat) takes these inputs, P - R x Q matrix of centered input (column) vectors. transMat - Transformation matrix.
Ptrans - Transformed data set.
p = [-1.5 -0.58 0.21 -0.96 -0.79; -2.2 -0.87 0.31 -1.4 -1.2]; t = [-0.08 3.4 -0.82 0.69 3.1]; [pn,meanp,stdp,tn,meant,stdt] = prestd(p,t); [ptrans,transMat] = prepca(pn,0.02); net = newff(minmax(ptrans),[5 1],{'tansig''purelin'},'trainlm'); net = train(net,ptrans,tn);
If we then receive new inputs to apply to the trained network, we will use trastd and trapca to transform them first. Then the transformed inputs can be used to simulate the previously trained network. The network output must also be unnormalized using poststd.
p2 = [1.5 -0.8;0.05 -0.3]; [p2n] = trastd(p2,meanp,stdp); [p2trans] = trapca(p2n,transMat) an = sim(net,p2trans); [a] = poststd(an,meant,stdt);

14-334

Ptrans = transMat*P; prestd, premnmx, prepca, trastd, tramnmx

14-335

14trastd
Preprocess data using a precalculated mean and standard deviation
[PN] = trastd(P,meanp,stdp) trastd preprocesses the network training set using the mean and standard deviation that were previously computed by prestd. This function needs to be used when a network has been trained using data normalized by prestd. All subsequent inputs to the network need to be transformed using the same normalization. trastd(P,T) takes these inputs, P - R x Q matrix of input (column) vectors. - R x 1 vector containing the original standard deviations for each
meanp - R x 1 vector containing the original means for each input. stdp

input. and returns,

p = [-0.92 0.73 -0.47 0.74 0.29; -0.08 0.86 -0.67 -0.52 0.93]; t = [-0.08 3.4 -0.82 0.69 3.1]; [pn,meanp,stdp,tn,meant,stdt] = prestd(p,t); net = newff(minmax(pn),[5 1],{'tansig' 'purelin'},'trainlm'); net = train(net,pn,tn);
If we then receive new inputs to apply to the trained network, we will use trastd to transform them first. Then the transformed inputs can be used to simulate the previously trained network. The network output must also be unnormalized using poststd.

weights and biases.

dead neurons - A competitive layer neuron that never won any competition
during training and so has not become a useful feature detector. Dead neurons do not respond to any of the training vectors.
decision boundary - A line, determined by the weight and bias vectors, for which the net input n is zero. delta rule - See the Widrow-Hoff learning rule. delta vector - The delta vector for a layer is the derivative of a networks output error with respect to that layers net input vector. distance - The distance between neurons, calculated from their positions with

a distance function.

distance function - A particular way of calculating distance, such as the Euclidean distance between two vectors. early stopping - A technique based on dividing the data into three subsets. The first subset is the training set used for computing the gradient and updating the network weights and biases. The second subset is the validation set. When the validation error increases for a specified number of iterations, the training is stopped, and the weights and biases at the minimum of the validation error are returned. The third subset is the test set. It is used to verify the network design. epoch - The presentation of the set of training (input and/or target) vectors to a network and the calculation of new weights and biases. Note that training vectors can be presented one at a time or all together in a batch. error jumping - A sudden increase in a networks sum-squared error during training. This is often due to too large a learning rate. error ratio - A training parameter used with adaptive learning rate and momentum training of backpropagation networks. error vector - The difference between a networks output vector in response to an input vector and an associated target output vector. feedback network - A network with connections from a layers output to that layers input. The feedback connection can be direct or pass through several layers. feedforward network - A layered network in which each layer only receives inputs from previous layers.
Fletcher-Reeves update - A method developed by Fletcher and Reeves for
computing a set of conjugate directions. These directions are used as search directions as part of a conjugate gradient optimization procedure.
function approximation - The task performed by a network trained to respond to inputs with an approximation of a desired function. generalization - An attribute of a network whose output for a new input vector tends to be close to outputs for similar input vectors in its training set. generalized regression network - Approximates a continuous function to an
arbitrary accuracy, given a sufficient number of hidden neurons.