311: Neural Networks

CIS 311: Neural Networks

On-line vs. Batch Backpropagation Algorithms

1. Revision by Epoch

From mathematical point of view the error derivatives should be computed after each epoch, i.e. after all examples in the training set have been processed.

This means that the error derivative is taken to be the sum of the error derivatives for all examples.

While this revision by epoch may have stronger theoretical motivation, revision by a particular example may yield better results and is more commonly used.

Revision by epoch is called Batch Learning!

Batch Learning version of the Backpropagation Algorithm

Initialization: Examples {( x_e, y_e)}_e=1^N, initial weights w_i set to small random values, learning rate h = 0.1
Repeat

for each training example ( x, y )

- forward propagate the example and calculate the outputs using the sigmoid function
- backward propagate the error to calculate the benefits

after processing all examples update the weights by the computed changes:
w = w + D w

until termination condition is satisfied.

2. Revision by Example

Revision by example is called On-line (incremental) Learning!

On-line Learning version of the Backpropagation Algorithm

Initialization: Examples {( x_e, y_e)}_e=1^N, initial weights w_i set to small random values, learning rate h = 0.1
Repeat

pick a training example ( x, y)

- forward propagate the example and calculate the outputs using the sigmoid function
- backward propagate the error to calculate the benefits
- update the weights by the computed changes:
w = w + D w

until termination condition is satisfied.

Suggested Readings:

Bishop,C. (1995) "Neural Networks for Pattern Recognition",
Oxford University Press, Oxford, UK, pp.140-149.

Haykin, Simon. (1999). "Neural Networks. A Comprehensive Foundation",
Second Edition, Prentice-Hall, Inc., New Jersey, 1999, pp. 178-184 (Section 4.6).