Tik-61.261 Principles of Neural Computing
Raivio, Venna

Exercise 6
1. Construct a MLP network which is able to separate the two classes illustrated in Figure 1. Use two neurons both in the input and output layer and an arbitrary number of hidden layer neurons. The output of the network should be vector if the input vector belongs to class and if it belongs to class . Use nonlinear activation functions, namely McCulloch-Pitts model, for all the neurons and determine their weights by hand without using any specific learning algorithm.
1. What is the minimum amount of neurons in the hidden layer required for a perfect separation of the classes?
2. What is the maximum amount of neurons in the hidden layer?

2. The function

is approximated with a neural network. The activation functions of all the neurons are linear functions of the input signals and a constant bias term. The number neurons and the network architecture can be chosen freely. The approximation performance of the network is measured with the following error function:

where is the input vector of the network and is the corresponding response.
1. Construct a single-layer network which minimizes the error function.
2. Does the approximation performance of the network improve if additional hidden layers are included?

3. The MLP network of Figure 2 is trained for classifying two-dimensional input vectors into two separate classes. Draw the corresponding class boundaries in the -plane assuming that the activation function of the neurons is (a) sign, and (b) tanh.

4. Show that (a)

is the solution of the following difference equation:

where is a positive momentum constant. (b) Justify the claims 1-3 made on the effects of the momentum term in Haykin pp. 170-171.

5. Consider the simple example of a network involving a single weight, for which the cost function is

where , , and are constants. A back-propagation algorithm with momentum is used to minimize . Explore the way in which the inclusion of the momentum constant influences the learning process, with particular reference to the number of epochs required for convergence versus .

Jarkko Venna 2005-04-13