CIS 311: Neural Networks

Issues in Learning by Backpropagation

1. Issues in Connectionist Learning by Backpropagation

· While it may be more convenient and efficient to specify more than one layer of hidden units, additional layers do not add representational power to the discrimination.

Two-hidden-layer networks are more powerful, but one-hidden-layer networks may be sufficiently accurate for many tasks encountered in practice. It should be noted, however, that certain real world function can be modeled exactly by one-hidden-layer networks with prohibitively large number of hidden units.

One-hidden layer networks assume faster training.

· Training may require thousands of backpropagations;

· Backpropagation can get stuck or become unstable when varying the learning rate parameter, more precisely increasing too much of the learning parameter leads to unstable learning- errors decrease as well as increase during the training process;

· Backpropagation can be done in stages by adding for example new outcome nodes for desired outputs, so that the training continues starting from the previously learned weights and the added weights on the connections to the new outcome nodes;

· Backpropagation can be trained to learn several concepts simultaneously but this may be a slower process than the incremental addition of new outcome nodes;

· Excess weights lead to overfitting, which may be prevented by early stopping, network pruning, network growing, and applying regularization techniques.

Suggested Readings:

Bishop,C. (1995) "Neural Networks for Pattern Recognition",
Oxford University Press, Oxford, UK, pp.116-149.