Nikolay Nikolaev's homepage

Nikolay Nikolaev and Hitoshi Iba

Adaptive Learning of Polynomial Networks:
Genetic Programming, Backpropagation and Bayesian Methods.

Series: Genetic and Evolutionary Computation
Springer, New York, 2006, XIV, 316 p. 62 illus., Hardcover
ISBN: 0-387-31239-0

(also available from Amazon.com)

Contents
1. Introduction 1.1 Inductive Learning 1.1.1 Learning and Regression
1.1.2 Polynomial Models
1.1.3 Inductive Computation Machinery 1.2 Why Polynomial Networks? 1.2.1 Advantages of Polynomial Networks
1.2.2 Multilayer Polynomial Networks 1.3 Evolutionary Search 1.3.1 STROGANOFF and Its Variants 1.4 Neural Network Training
1.5 Bayesian Inference
1.6 Statistical Model Validation
1.7 Organization of the Book
2. Inductive Genetic Programming (IGP) 2.1 Polynomial Neural Networks (PNN) 2.1.1 PNN Approaches
2.1.2 Tree-structured PNN 2.2 IGP Search Mechanisms 2.2.1 Sampling and Control Issues
2.2.2 Biological Interpretation 2.3 Genetic Learning Operators 2.3.1 Context-preserving Mutation
2.3.2 Crossover Operator
2.3.3 Size-biasing of the Genetic Operators
2.3.4 Tree-to-Tree Distance 2.4 Random Tree Generation
2.5 Basic IGP Framework
2.6 IGP Convergence Characteristics 2.6.1 Schema Theorem for IGP
2.6.2 Markov Model of IGP 2.7 Chapter Summary
3. Tree-like PNN Representations 3.1 Discrete Volterra Series
3.2 Mapping Capabilities of PNN
3.3 Errors of Approximation 3.3.1 Approximation Error Bounds
3.3.2 Empirical Risk 3.4 Linear Polynomial Networks 3.4.1 Horizontal PNN Models
3.4.2 Kernel PNN Models 3.5 Nonlinear Polynomial Networks 3.5.1 Block PNN Models
3.5.2 Orthogonal PNN Models
3.5.3 Trigonometric PNN Models
3.5.4 Rational PNN Models
3.5.5 Dynamic PNN Models 3.6 Chapter Summary
4. Fitness Functions and Landscapes 4.1 Fitness Functions 4.1.1 Static Fitness FunctionsP
4.1.2 Dynamic Fitness Functions
4.1.3 Fitness Magnitude 4.2 Fitness Landscape Structure
4.3 Fitness Landscape Measures 4.3.1 Statistical Correlation Measures
4.3.2 Probabilistic Measures
4.3.3 Information Measures
4.3.4 Quantitative Measures 4.4 Chapter Summary
5. Search Navigation 5.1 Reproduction Operator 5.1.1 Selection Strategies
5.1.2 Replacement Strategies
5.1.3 Implementing Reproduction 5.2 Advanced Search Control 5.2.1 Macroevolutionary Search
5.2.2 Memetic Search
5.2.3 Search by Genetic Annealing
5.2.4 Stochastic Genetic Hillclimbing
5.2.5 Coevolutionary Search
5.2.6 Distributed Search 5.3 Performance Examination 5.3.1 Fitness Evolvability
5.3.2 Convergence Measures
5.3.3 Diversity Measures
5.3.4 Measures of Self-Organization 5.4 Chapter Summary
6. Backpropagation Techniques 6.1 Multilayer Feed-forward PNN
6.2 First-Order Backpropagation (BP) 6.2.1 Gradient-Descent Search
6.2.2 First-Order Error Derivatives
6.2.3 Batch Backpropagation
6.2.4 Incremental Backpropagation
6.2.5 Control of the Learning Step
6.2.6 Regularized Delta Rule 6.3 Second-Order Backpropagation 6.3.1 Second-Order Error Derivatives
6.3.2 Newton's Method
6.3.3 Pseudo-Newton Training
6.3.4 Conjugate Gradients
6.3.5 Levenberg-Marquardt Method 6.4 Rational Backpropagation
6.5 Network Pruning 6.5.1 First-Order Network Pruning
6.5.2 Second-Order Network Pruning 6.6 Chapter Summary
7. Temporal Backpropagation 7.1 Recurrent PNN as State-Space Models
7.2 Backpropagation Through Time (BPTT) 7.2.1 Derivation of BPTT Algorithms
7.2.2 Ral-Time BPTT Algorithm
7.2.3 Epochwise BPTT Algorithm 7.3 Real-Time Recurrent Learning (RTRL)
7.4 Improved Dynamic Training 7.4.1 Teacher Enforced Training
7.4.2 Truncating in Time
7.4.3 Subgrouping
7.4.4 Common Temporal Training Problem 7.5 Second-Order Temporal BP
7.6 Recursive Backpropagation
7.7 Recurrent Network Optimization 7.7.1 Regularization
7.7.2 Recurrent Network Pruning 7.8 Chapter Summary
8. Bayesian Inference Techniques 8.1 Bayesian Error Function
8.2 Bayesian Neural Network Inference 8.2.1 Deriving Hyperparameters
8.2.2 Local vs. Global Regularization
8.2.3 Evidence Procedure for PNN Models
8.2.4 Predictive Data Distribution
8.2.5 Choosing a Weight Prior 8.3 Bayesian Network Pruning
8.4 Sparse Bayesian Learning
8.5 Recursive Bayesian Learning 8.5.1 Sequential Weight Estimation
8.5.2 Sequential Dynamic Hessian Estimation
8.5.3 Sequential Hyperparameter Estimation 8.6 Monte Carlo Training 8.6.1 Markov Chain Monte Carlo
8.6.2 Importance Sampling
8.6.2 Hybrid Sampling-Resampling 8.7 Chapter Summary
9. Statistical Model Diagnostics 9.1 Deviations of PNN Models
9.2 Residual Bootstrap Sampling
9.3 The Bias/Variance Dilemma 9.3.1 Statistical Bias and Variance
9.3.2 Measuring Bias and Variance 9.4 Confidence Intervals 9.4.1 Interval Estimation by the Delta Method
9.4.2 Bootstrapping Confidence Intervals 9.5 Prediction Intervals 9.5.1 Analytical Prediction Intervals
9.5.2 Empirical Learning of Prediction Bars 9.6 Bayesian Intervals 9.6.1 Analytical Bayesian Intervals
9.6.2 Empirical Bayesian Intervals 9.7 Model Validation Tests
9.8 Chapter Summary
10. Time Series Modelling 10.1 Time Series Modelling
10.2 Data Preprocessing
10.3 PNN vs. Linear ARMA Models
10.4 PNN vs. Genetically Programmed Functions
10.5 PNN vs. Statistical Learning Networks
10.6 PNN vs. Neural Network Models
10.7 PNN vs. Kernel Models
10.8 Recurrent PNN vs. Recurrent Neural Networks
10.9 Chapter Summary
11. Conclusions

References
Index