The simple single-layer Perceptrons with threshold or linear activation functions are not generalizable to more powerful learning mechanisms like multilayer neural networks. That is why, single-layer Perceptrons with sigmoidal activation functions are developed. The sigmoidal Perceptron produces output:

o = s( s ) = 1 / ( 1 + e^-s ), where: s= S_i=0^d w_i x_i

The Gradient descent rule for training sigmoidal Perceptrons is again:

w_i = w_i - h ¶E/¶w_i

The difference is in the error derivative ¶E/¶w_i, which due to the use of the sigmoidal function s( s ) becomes:

¶ E/¶w_i = ¶ ( ( ½ )S _e( y_e–o_e )²)/¶w_i

The Gradient descent training rule for training sigmoidal Perceptrons is:

w_i = w_i + h S_e( y_e–o_e ) s'( s ) x_ie

where: s'( s ) = s( s )( 1 - s( s ) ).

Gradient Descent Learning Algorithm for Sigmoidal Perceptrons

Initialization: Examples {( x_e, y_e)}_e=1^N, initial weights w_i set to small random values, learning rate parameter h = 0.1

Repeat

until termination condition is satisfied.

Example: Suppose an example of Perceptron which accepts two inputs x₁ and x₂, with weights w₁ = 0.5 and w₂ = 0.3 and w₀ = -1.

Let the following example is given: x₁ = 2, x₂ = 1, y = 0
The output of the Perceptron is :

o = s( -1 + 2 * 0.5 + 1 * 0.3 ) = s( 0.3 ) = 0.5744

The weight updates according to the gradient descent algorithm will be:

Dw₀ = ( 0 - 0.5744 ) * 0.5744 * ( 1 -0.5744 ) * 1 = - 0.1404

Dw₁ = ( 0 - 0.5744 ) * 0.5744 * ( 1 -0.5744 ) * 2 = - 0.2808

Dw₂ = ( 0 - 0.5744 ) * 0.5744 * ( 1 -0.5744 ) * 1 = - 0.1404

Let another example is given: x₁ = 1, x₂ = 2, y = 1

The output of the Perceptron is :

o = s( -1 + 1 * 0.5 + 2 * 0.3 ) = s( 0.1 ) = 0.525

The weight updates according to the gradient descent algorithm will be:

Dw₀ = - 0.1404 + ( 1 - 0.525 ) * 0.525 * ( 1 - 0.525 ) * 1 = -0.0219

Dw₁ = - 0.2808 + ( 1 - 0.525 ) * 0.525 * ( 1 - 0.525 ) * 1 = -0.1623

Dw₂ = - 0.1404 + ( 1 - 0.525 ) * 0.525 * ( 1 - 0.525 ) * 2 = 0.0966

If there are no more examples in the batch, the weights will be modified as follows:

w₀ = - 1 + ( -0.0219 ) = -1.0219

w₁ = 0.5 + ( -0.1623 ) = 0.3966

w₂ = 0.3 + 0.0966 = 0.3966

Incremental Gradient Descent Learning Algorithm for Sigmoidal Perceptrons

Initialization: Examples {( x_e, y_e)}_e=1^N, initial weights w_i set to small random values, learning rate parameter h = 0.1

Repeat

until termination condition is satisfied.

Suggested Readings:

Bishop,C. (1995) "Neural Networks for Pattern Recognition", Oxford University Press, Oxford, UK, pp.98-105.

Haykin, Simon. (1999). "Neural Networks. A Comprehensive Foundation", Second Edition, Prentice-Hall, Inc., New Jersey, 1999.