On Neural Network Topology Design for Nonlinear Control

On neural network topology designfor nonlinear controlJens Haecker and Stephan Rudolph*Institute for Statics and Dynamics of Aerospace StructuresUniversität Stuttgart,Pfaffenwaldring27,D-70569Stuttgart,GermanyProceedings SPIE Aerosense2001Conference On Applications and Science ofComputational Intelligence IV,Orlando,Florida,April16-20th,2001ABSTRACTNeural networks,especially in nonlinear system identi£cation and control applications,are typically considered to be black-boxes which are dif£cult to analyze and understand mathematically.Due to this reason,an indepth mathematical analysis offering insight into the different neural network transformation layers based on a theoretical transformation scheme is desired, but up to now neither available nor known.In previous works it has been shown how proven engineering methods such as dimensional analysis and the Laplace transform may be used to construct a neural network controller topology for time-invariant ing the knowledge of neural correspondencies of these two classical methods,the internal nodes of the network could also be successfully interpreted after training.As a further extension to these works,the paper describes the latest results of a theoretical interpretation framework de-scribing the neural network transformation sequences in nonlinear system identi£cation and control.This can be achieved by incorporation of the method of exact input-output linearization in the above mentioned two transformation sequences of dimen-sional analysis and the Laplace transformation.Based on these three theoretical considerations neural network topologies may be designed in special situations by a pure translation in the sense of a structural compilation of the known classical solutions into their correspondent neural topology.Based on known exemplary results,the paper synthesizes the proposed approach into the visionary goals of a structural compiler for neural networks.This structural compiler for neural networks is intended to automatically convert classical control formulations into their equivalent neural network structure based on the principles of equivalence between formula and operator, and operator and structure which are discussed in detail in this work.Keywords:modular neural networks,engineering principles,dimensional analysis,Laplace transform,exact input-output linearization1.INTRODUCTIONArti£cial neural networks(ANNs)are an alternative computational paradigm that calculates function approximations in a system of interconnected simple computational cells.1Neural networks of the feed-forward multilayer perceptron type that are in the scope of this work can be trained to represent virtually any function using data samples since they are universal function approximators.2Because of the simplicity of a single neuron,the complexity of the function represented by the whole system evolves from the interconnections between the cells.Typical applications of ANN in engineering are pattern recognition,data-based modelling where processes are not well known or understood,and/or optimization problems,when a near sub-optimal solution is acceptable.Neural networks are less suitable for numerical calculations,when high precision is required,or modeling,when there already is a good physical model available,nor optimization,when the global optimum is required.Although performance strongly depends on the topology of ANNs and the choice of network activation functions,no reliable methods for determining those aspects are known up to date.So engineers currently rely more or less on trial-and-error methods to£nd a suitable topology and a training method.Moreover,learned information after training in the neural network is typically distributed over the parameters of the whole network which limits the possibility of interpreting and explaining the results.On the other hand,in engineering a number of principles and design rules to simplify and solve certain(standard)problems are known.These are,among others the Fourier transform for frequency dependent phenomena,the Laplace transform to trans-form linear differential equations into algebraic equations,or even an input-output linearization3scheme for certain nonlinear *Correspondence:Email:rudolph@isd.uni-stuttgart.de;WWW:http://www.isd.uni-stuttgart.de/,Phone:+497116853799,Fax:+497116853706control problems.For some problems system speci£c knowledge and principles are known such as the necessary condition of dimensional homogeneity which holds for any physically correct functional relation.It is claimed that this a priori knowledge has to be included in the network topology design process to make neural networks more applicable and transparent.The main interest in neural networks is currently concentrated on the use in nonlinear control problems.Many engineering solutions are tailored to suit linear problems.Generally linear systems pose therefore no unsurmountable problems.In the last years neural networks have mainly been used to model nonlinear systems in control.ANNs can£nd simple suboptimal solutions to control problems and can be applied to systems where classical approaches based on system linearization do not work.Yet,ANNs lack methods for determining control stability or the possibility to interpret the results analytically.Various approaches and applications of neural control exist in the literature.4,5However from a scienti£c viewpoint it is still necessary to break the black-box structure of most network models.To include knowledge and interpretability into the neural network it is tried to identify neural correspondencies for engi-neering principles and to translate them into neural network modules or prede£ned sub-structures of the network.The goal is to de£ne a descriptor system that analyzes a problem and puts together the known parts of the network from prede£ned com-ponents.Some principles can be translated into primary data processing layers that can be attached to a core neural network with carefully devised degrees of freedom left in the network weights that are still to be learned.Thus,a simple translation of classical control theory solutions into a neural controller could be envisioned and constructed that results in a controller that performs at least as good as the classical or conventional approach.Further on,this controller could be made adaptive to adjust to a more complex real system using on-line training algorithms while being in service.The expected advantages are a transparent design scheme,an improved interpretation,and last not least more con£dence in the system in operation.WORK TOPOLOGYArti£cial neural networks of various kinds are known.Even though this examination is restricted to feed-forward multilayer perceptrons that have a comparatively simple structure,the freedom in the topology design allows various different architec-tures.Neural networks can be build up having any number of layers and neurons and virtually any mathematical operator as activation function in the neurons.The classical approach to use simple functions such as sigmoidal or hyperbolic functions in large networks has several disadvantages.For large networks training is slow and not reliable.Additionally,local minima of the optimization procedures interfere with the desire to£nd the global optimum,which is a major drawback of any multi parameter optimization.Another problem is the neural network’s currently limited ability to interpret and explain the training results,since the information learned is distributed over the whole network.2.1.Thoughts on neural operatorsFor neural operators there are all kinds of mathematical functions possible.Classically sigmoidal functions are used.They model the response functions found in biological neurons that jump from being inactive to an activated state when a certain threshold of excitation from the precedent neurons is exceeded.The sigmoidal function is similar to a step function,but is differentiable which is a property that is necessary for learning procedures like the backpropagation algorithm.Even though it is possible to model complex functions with nets of sigmoidal neurons,the trade-off is that the network structure will be large and complex.As stated,many problems with neural networks emerge from their size.However,this principle can be used in the opposite sense as a scheme to structure the neural network design process as explained in the following.The equivalence of a mathematical operator in its symbolic representation as a mathematical formula,and a network struc-ture computing the corresponding mathematical expression are straightforward.Looking at the exponential function e x in Figure1for instance,it can be observed that three different forms of representations are possible.The operator as one com-pact element(bottom left),its power series expansion as a formula(top)and a network consisting of operators and parameters (bottom right)are mathematically equivalent expressions.The operators in the shown network structure could be expanded the same way in sub-nets which encode the operators x i in an even simpler way.It is easy to see the correspondency looking at the mathematical operator on the left side and the symbolic representation as mathematical formula,or looking at the network structure on the right and the symbolic representation.From these equiv-alencies it is clear that a mathematical operator can be coded in a network using very simple operators in many different but equivalent forms.Usually these equivalences are used to expand complex operators into networks of simpler operators.However,large networks have also disadvantages.The number of local minima of the optimization procedures,e.g.during the training, depends on the number of degrees of freedom,that is the number of used neurons and internal connections(weights).For this reason the demand for something one could call operator compactness is desirable.Instead of complex networks with simple functions one could prefer to build up simpler networks with more complex operators instead.This would reduce the number of parameters in the net and would further improve the training and the physical interpretation of the nets later on.Figure1.Equivalence of a mathematical operator(bottom left),a symbolic representation as a mathematical formula(top), and a network structure(bottom right)at the example of the exponential function e x.Another property that could be exploited later is that all physically correct formulas belong to the so-called class of di-mensionally homogeneous functions.If it is attempted to encode physical equations into a neural network,the condition of dimensional homogeneity should therefore always be observed.2.2.Thoughts on network topology design using knowledgeIn the following the principles stated are visualized with a very simple physical system.In order to achieve this,a beam with a single vertical load P at the unsupported tip is considered.The de¤ection u of the tip due to bending of the beam can be analytically derived and expressed with the simple formula6u=13P l3EI,(1)where l is the length of the beam,E is Young’s modulus,and I is the moment of inertia of the beam about the axis of bending. It is a nonlinear time invariant static problem.Supposed the goal is to train a neural network to predict the de¤ection of the beam given the measurements of the sys-tem’s parameter vector,that is the static mapping u i=f NN(P i,l i,E i,I i).Therefore a certain number m of training vectors (P i,l i,E i,I i)and the system’s response(u i)are used to train a network for representing the formula given in equation(1). This can be achieved very easily.However,the interpretation of the learned data from the network is not trivial.One ends up with parameters of the network connections collected in weight matrices W ij.Supposed a neural net was analyzed to extract the formula represented by the net,dif£culties may occur as can be seen in the example network in Figure1(bottom right). The more speci£c and compact the used operators are,the better is the possible physical understanding and interpretation of the network structure after training.In the following several steps to gradually incorporate knowledge into the topology of the network will be shown by making use of the principles described above.As can be seen easily,this will improve the interpretation and even the approximation result.This is achieved through the reduction of the degrees of freedom leaving only those parameters for the network free to learn that are really unknown.This will be shown in the following example.The neural network to start with is of a classical conservative structure.The used neurons calculate the sum of their weighted inputs and have sigmoidal activation functions s(x).The example uses four input neurons for the input vector(P i,l i,E i,I i) and one for the output of(u i)as shown in Figure2a.The number of neurons in the intermediate layer is arbitrary.However, with a higher number of neurons dif£culties with the training of the network tend to increase.A closer examination of physical formulas reveals that most of them are of a certain structure.They tend to be of a product structure with exponents,e.g.p0=ν0pν11×pν22×...×pνm m(or sums thereof).This can be implemented directly in the neural network topology by simply using logarithmic functions in the£rst layer and exponential functions in the second layer.A net of that structure is shown in Figure2b.However,by mathematical analysis it is found that in that structure all neurons in the second layer are calculating redundant functions and could be replaced by only one neuron as shown in Figure3.The weight matrices W ij are replaced by parametersu=f NN(P,l,I,E)u= iν0i Pν1i lν2i Eν3i Iν4iFigure2.Neural network to predict the de¤ection of a beam.Typical black-box net(a)and structured network(b)with speci£c neuron functions that determine a de£ned goal function.that are directly interpretable as exponents of the function variables.The two steps considered in the Figures2and3can be automated by means of genetic algorithms that vary the neuron functions and the network topology as well as pruning techniques that remove neurons with small weighted connections as shown in previous works.7P1l3E−1I−1u=w0P w1l w2E w3I w4u=13work topology compaction by removing redundant neurons.From the trained network the exact function is obtained.After training the neural network shown in Figure3b is obtained.It calculates the exact function for the bending of the beam. Possible small deviations from the exact parameters,which are the exponents,could be corrected by introducing a constraint that allows only integer numbers(or fractions thereof).Most exponents in physical formulas are in this respect elements out of the interval[−4,4].An even harder constraint can be formulated using the relations between physical units that are subject of the Buckingham or Pi-theorem.8Only dimensionally homogeneous functions can be physically correct.That allows only certain combinations of physical quantities x that,according to equation(2),consist of a number{x}and a unit representation[x].Generally,for any physical quantity x holds:x={x}[x](2) For this reason,any relationship applies to the numerical values of x as well as to its dimensional representation which is expressed in Buckingham’s theorem.By applying the Buckingham theorem to the relevant variables x1...x n of the function, a set of dimensionless variablesπ1...πm is found that form a dimensionless function equivalent to the original one.The dependencies between the variables are given by this theorem and can be used as known weight connections between neurons. So only the remaining unknown parameters have to be adapted by a training process.Figure4b shows the new network. Obviously a core neural network is enclosed by a£rst and a last layer of neurons that are just a forward transformation layer from the physical quantities to dimensionless variables and a backward transformation,which are calledπ-transform and the π−1-transform respectively.6u=w0P w1l w2E w3I w4u=lν0πν11πν22=lν0 El2P ν1 I l4 ν2lπ−11π−12u=13Figure4.Transforming the network into a dimension homogeneous network by exploiting the relations between physical units.2.3.Thoughts related to network interpretation and generalizationIt is clear from the example that a systematic incorporation of knowledge into the network topology results in a more structured and less complex network topology.The degrees of freedom,i.e.the free parameters or connection weights in the neural net are reduced to a smaller number.Thus,the training algorithms,i.e.the optimization procedures that usually suffer from the curse of dimensionality,are more ef£cient due to the reduction of free parameters.Moreover,in this easy example the represented equation can be extracted directly from the trained neural network.Yet,the technique of dimensional analysis offers another advantage which lies in the generalization of the function repre-sented by the network in Figure4b.Generalization is the ability of a neural network to process correctly unknown data which are not contained in the training samples.This means that the network can reasonably interpolate between given samples as well as extrapolate out of the sample range.The state-of-the-art in testing the network performance and its generalizing capability typically is to calculate a sum-squared error for a set of training data samples and generalization test data sets presented to the neural network using a crossvalidation technique.Dimensionally homogeneous neural networks generalize correctly all data that are physically similar to known data points. The usage of the Pi-theorem to construct data not given in the training data explicitely as alternative validation data has been proposed.7A neural network trained with this data will represent not only known data samples but the underlying dimensionless functional relationship.This is an equivalent but more general representation of the functional relationship of the physical quantities based on the theoretical concept of similarity functions.63.MODULAR DESIGN OF NEURAL NETWORKSIn the last section it was shown how knowledge can be incorporated in a neural network for the beam de¤ection example.It was a very easy static mapping problem so a small neural network was suf£cient to model the functional dependencies describing the system.More complex problems require bigger neural networks which will be more dif£cult to understand.Moreover,the training ef£ciency in neural network learning strongly depends on the network size as well.Hence,it is proposed to divide neural networks up into smaller subnetworks or modules that perform local computation.It is known that in the human brain,which serves as the neural network role model for its arti£cial counterparts,local regions of physically close neurons perform speci£c computational tasks.Examples are Brocca’s or Wernike’s region that are specialized on the recognition of spoken language.9Naturally,researchers are still far from being able to model complex structures comparable to the human brain,but it is argued that complex behavior requires bringing together several different kinds of knowledge and processing paradigms,which seems not possible without structure,putational modularity.The modular approach permits to apply neural concepts to large scale networks.Even complex architectures could be greatly simpli£ed by identifying separate distinct subtasks in the problem and embedding them into a neural network structure. This way each task could be trained off-line and integrated later in a global architecture.Even hybrid models could be realized. Why should one wish to train a neural network for a task that is already completely described or modelled?This is an important question often ignored in the neural network community.It is seldom the case that from a process to be modelled nothing is known at all.Modular neural networks would allow the assembly of different simple plant models or to introduce a priori knowledge into the architecture by combining differently structured modules.The solution strategy when working with modular neural networks implies three steps according to Ronco.10First,the decomposition of a task into meaningful subtasks has to be mentioned.It is desirable that each module performs an explicit, interpretable and relevant function according to the mechanical and physical properties of the system.Second,the organization of the global modular architecture is important.Proposed architectures consist of several local system models and a gating system that switches between the models.Third and£nally,the integration of the inter-module communication is necessary.Signi£cant learning improvement is expected compared to a single big network.In small networks learning is faster and the learning results are easier to interpret.Moreover,modularity clari£es the overall presentation of the system because the activity of modules can be associated to certain operating regions in the plant.If those modules are linear in order to consequently connect the idea to control applications,the construction of good linear controllers with conventional techniques is possible,as well as a local analysis of their properties,such as stability.Another network can be used to trigger between adequate models and controllers,and the whole scheme can easily be coded into one global neural network as well.11A schematic example of such a gated modular neural network(GMNN)used as an identi£ed model of a dynamic system is shown in Figure5.Figure5.Neural network with a modular structure for the identi£cation of a plant.The neural network models NN2and NN3 are£tted for different plant operating regions shown in the plot on the right.NN1is a gating network responsible for detecting the appropriate system representation according to the input vector x and switching between the models.NN4can be used to either process the switching signal or even calculate combinations of the two model network outputs.104.NEURAL CONTROL DESIGN ASPECTSControl theory offers powerful tools from linear algebra to be used for system analysis and control as long as the system behaves linearly.Assumptions of system linearity have been made for this reason to develop a control theory on a solid mathematical basis.Control design from system linearization is a widely applied technique in industry.However,in reality most systems are nonlinear.It is the ability of neural networks to model nonlinear systems which is the feature most readily exploited in the synthesis of nonlinear controllers.Neural control techniques have successfully been applied to problems in robotics and other highly nonlinear systems.5A growing number of different neural control schemes exist that are£tted only for certain problems.However,the usage of neural networks in nonlinear control does not make sense per se.There are still many open research topics,such as the characterization of theoretical properties such as stability,controllability and observability or even the system identi£ability.It is not intended to give a survey on neural control methods here,since many of the basic principles are shown in the reports by Hunt12or Narendra.13The idea of a neural network structural compiler originates from the(re-)use of existing control theory applications,which intends the construction of mathematical controllers designed after classical theories and their representation in the form of neural networks.As indicated in Figure1,the calculations done in the neural network will be identical.Therefore it is claimed that it will be possible to design neural controllers at least as good as the classical ones.14However,by providing the network with additional degrees of freedom and applying training algorithms common in neural network computation,even an improved adaption could be achieved.Adaptivity is an important feature because the real world environment of the controller can be expected to be different from the simpli£ed linearized model used for the controller design.Analogous approaches to the idea promoted here already exist in techniques summarized under the term of intelligent control15which represents an attempt to bring together arti£cial intelligence techniques and control theory.Controllers are put together from prede£nes components in a structured design approach with a knowledge-based expert system as integration tool. This is realized for instance in the neuro-fuzzy control scheme.15,16A structure is provided by the fuzzy logic approach which builds up control laws from linguistic rules.Then the scheme is implemented in a neural network.The structure is determined after a simple algorithm from modules.Finally,the learning ability of the neural network is used to adapt the controller to the speci£c control situation by learning the controller’s parameter values.4.1.Neural correspondencies to classical control theoryClassical control theory utilizes a number of engineering principles that could be or are partly already applied to neural control. Preliminary studies done have concentrated on neural correspondences of engineering principles used in control and how these principles could be coded into a neural network scheme.14,17,18It is found that some realizations are clearly straightforward whereas others require more sophisticated procedures which can still be improved.In the following some of these principles will be explained in brief.Naturally,this list is far from being complete and should only indicate that neural correspondencies to classical engineering principles exist due to the equivalence between a neural topology and a mathematical formulation.These engineering principles are:•Dimensional analysis.14The power of the dimensional analysis has already been demonstrated for the beam de¤ection example and is visualized in Figure4.•The Laplace transform can be used to transform linear differential equations into algebraic equations and though is helpful for the analysis of dynamic systems.It is found that this transformation scheme can be transferred to a neural topology.17,14However,the Laplace transform is only applicable to linear systems.•The input-output linearization3scheme has already been applied to neural control.19,18The basic idea is to identify a feedback which linearizes the otherwise nonlinear behavior of the system.This way a system can be constructed which can be controlled like a linear system by using the standard classical approaches.The consequent combination of the above three principles results in a network with prede£ned layers that do some sort of data pre-and post-processing for a core network that can still be regarded as a black-box and which is the only part to be learned. Figure6shows exemplary a schematic diagram of this neural structure.It can be described as a network with a butter¤y topology which indicates that the information processed by the neural network is smaller than the original given data due to an intelligent selection of a data transformation sequence.The modular neural network shown in Figure6consists of the Pi-transform layers(Πand its inverseΠ−1),the Laplace layers(L and its inverse L−1),and the input-output linearization layers(I/O)that encapsulate a core neural network.It should be noted that this structure is not a standard feed-forward network,because the Pi-transform uses shortcuts from the£rst to the last layer and the linearization requires feedback connections.This structure is found to be suitable for the identi£cation of a dynamic system,which is a basis for many neural control applications.14Figure6.Modular neural representation of a nonlinear dynamic system.The butter¤y diagram:a core network encapsulated by transformation layers.From the inside:core neural network as black-box,input-output linearization layer(I/O),Laplace transform layer,and Pi-transform layer.Some thoughts should be given to common concepts in control such as the PID controller and the method of gain schedul-ing.20This method provides a linear controller for several linearized states or operational points of a system.It is straightfor-ward to implement gain scheduling in neural networks because the feedback gain coef£cient matrices usually are represented as look-up-tables and could as well be stored in neural networks.12,21Even complex designs such as LQR or LQG controllers22,20 can be implemented most readily into a neural control scheme.4.2.Neural Structure compiler conceptThis work aims at promoting the vision of a structural compiler for neural network topologies.Its function will be to assemble neural controllers from prede£ned modules based on a mathematical description as well as to disassemble the modules for system analysis.The modules are constructed from various engineering principles and classical control strategies by means of the identi£cation of neural correspondencies.Some examples of neural correspondencies to engineering principles have been shown in the previous section.For the example of system identi£cation using neural networks,studies have already been performed which investigate modular neural network topologies consisting of layers that perform forward-and backward-transformation around a core network with free parameters as shown in Figure6.The underlying principles and their justi£cation have been discussed in detail.。