1. The Interpolation Problem

The multilayer perceptron(MLP) address the problem of stochastic approximation.

The radial-basis function networks (RBF) address the problem of curve-fitting, that is approximation in high-dimensional spaces. Learning in this case is equivalent to finding an interpolating surface in the multidimensional space that provides a best fit to the training data, measured by preselected statistical criteria.

The radial-basis function networks are developed for addressing the interpolation problem.

The curve-fitting or interpolation problem can be stated as follows:

Given a set of examples {( x_e, y_e )}_e=1^N find a function F: R^N® R¹that satisfies the interpolation condition: F( x_e ) = y_e for each e=1,2,...,N.

2. Radial-Basis Functions

The radial-basis functions technique suggests constructing of interpolation functions F of the following form:

F( x ) = S_i=1^N w_i j( || x - x_i || )

where: j( || x - x_i || ) is a set of nonlinear radial-basis functions, x_i are the centers of these functions, and ||.|| is the Euclidean norm.

The unknown weights can be found by solving the following linear matrix equation:

w = ( F^TF )^-1 F^T y

where: w is the weight vector, y is the response vector, and F is the design matrix

Several widely used basis functions are:

- multiquadratics: j( x ) = ( x² + c² )^1/2 for some c>0

- inverse multiquadratics: j( x ) = 1 / ( x² + c² )^1/2 for some c>0

- Gaussian: j( x ) = exp( - x² / 2s ² ) for some s >0

The multivariate Gaussian function provides also two important properties that make it a proper choice for building radial-basis functions: it is both translation and rotation invariant. The multivariate Gaussian function with these properties is called also Green's function:

G( x, x_i ) = exp( - || x - x_i ||² / 2s_i² )

3. Regularization Networks

The regularization network models the interpolation function F as a linear superposition (linear weighted sum ) of multivariate Gaussian functions which number is equal to the number of the given input examples N:

F( x ) = S_i=1^N w_i G( x, x_i ) or F( x ) = S_i=1^N w_i exp( - || x - x_i ||² / 2s_i² )

where: w_i are the weights.

The regularization network guarantees three important characteristics:
- it is a universal approximator in that it can approximate arbitrarily well any multivariate continuous function on a compact set, given a sufficiently large number of units;
- it has the best approximation property in that given an unknown nonlinear function there always exists a choice of coefficients that approximates the function better than all other choices;
- it produces optimal solutions that minimize the functional which measures how much the solution deviates from its true value as represented by the training examples.

4. Generalized RBF Networks

The one-to-one correspondence between the number of Green functions and the number of the tarining examples often become computationally inefficient in practice, in the sense that it may require combining a very large number of basis functions (since often in practice a large number of training examples are provided).

In real-world practical situations finding of the linear basis function weights needs to invert very large N ´ N matrices which may be ill-conditioned. In order to avoid such problems the network topology should be reduced.

Reducing of the network complexity means that we will attempt to find a solution that approximates the solution produced by the regularization network.

4.1. RBF Network Structure

An RBF network architecture consists of three layers:
- input layer: which passes the example vectors to the next layers
- hidden layer: applies a non-linear transformation function to the inputs and expandsthem in the usually very high-dimensional hidden space
- output layer: applies a linear transformation to the given vector

Note, however, that sometimes RBF is called one-layer network as it has only one hidden layer!

4.2. Function Mapping

The radial-basis function network models the interpolation function F as a linear superposition of a finite number of multivariate Gaussians as follows:

F( x ) = S_i=1ⁿ w_i exp( - || x - x_i ||² / 2s_i² )

where: w_i are the weights, and the number of basis functions n is less than the number of points in the training set n < N. The basis function centers x_i are determined in advance.

4.3. RBF Learning

Learning of the network weights w_i involves minimizing of the cost functional:

C = S_e=1^N ( y_e - F( x_e ))² + l S_j=1ⁿ w_j²

where: l is a regularization parameter.

The weight vector solution is given by the normal equation:

w = ( F^T F + lI )^-1 F^T y

where: I is n ´ n identity matrix which elements are all zero except these on the diagonal

y is the output vector given with the examples

F is the n ´ n design matrix of basis functions j = exp( - || x - x_i ||² / 2s_i² )

F= j11 j12 ... j1n
       j21 j22 ... j2n
       ... 
       jN1 jN2 ... jNn

RBF Training Algorithm

Initialization: Examples {( x_e, y_e )}_e=1^N

Training:

The radial-basis function centers x_i and variances s _i² may be trained along with the output weights but this takes a long time. That is why, they are often determined in advance.

Suggested Readings:

Bishop,C. (1995). Neural Networks for Pattern Recognition, Oxford University Press, Oxford, UK, pp.164-190 (Chapter 5).

Haykin, Simon. (1999). Neural Networks. A Comprehensive Foundation., Second Edition, Prentice-Hall, Inc., New Jersey, Section 5.