Pages from EViews 6 Users Guide II

Appendix D. Estimation and Solution OptionsEViews estimates the parameters of a wide variety of nonlinear models, from nonlinear least squares equations, to maximum likelihood models, to GMM specifications. These types of nonlinear estimation problems do not have closed form solutions and must be estimatedusing iterative methods. EViews also solves systems of non-linear equations. Again, there are no closed form solutions to these problems, and EViews must use an iterative method to obtain a solution.Below, we provide details on the algorithms used by EViews in dealing with nonlinear esti-mation and solution, and the optional settings that we provide to allow you to control esti-mation.Our discussion here is necessarily brief. For additional details, we direct you to the quitereadable discussions in Press, et al. (1992), Quandt (1983), Thisted (1988), and Amemiya (1983).Setting Estimation OptionsWhen you estimate an equation in EViews, you enter specification information into theSpecification tab of the Equation Estimation dialog. Clicking on the Options tab displays a dialog that allows you to set various options to control the estimation procedure. The con-tents of the dialog will differ depending upon the options available for a particular estima-tion procedure.The default settings for the options will be taken from the global options (“EstimationDefaults” on page767), or from the options used previously to estimate the object.The Options tab for binarymodels is depicted here. Forother estimator and estimationtechniques (e.g. systems) thedialog will differ to reflect thedifferent estimation optionsthat are available.Starting Coefficient Val-uesIterative estimation proceduresrequire starting values for thecoefficients of the model. Thereare no general rules for select-626—Appendix D. Estimation and Solution Optionsing starting values for parameters. Obviously, the closer to the true values, the better, so if you have reasonable guesses for parameter values, these can be useful. In some cases, you can obtain starting values by estimating a restricted version of the model. In general, how-ever, you may have to experiment to find good starting values.EViews follows three basic rules for selecting starting values:•For nonlinear least squares type problems, EViews uses the values in the coefficient vector at the time you begin the estimation procedure as starting values.•For system estimators and ARCH, EViews uses starting values based upon preliminary single equation OLS or TSLS estimation. In the dialogs for these estimators, the drop-down menu for setting starting values will not appear.•For selected estimation techniques (binary, ordered, count, censored and truncated), EViews has built-in algorithms for determining the starting values using specific infor-mation about the objective function. These will be labeled in the Starting coefficientvalues combo box as EViews supplied.In the latter two cases, you may change this default behavior by selecting an item from the Starting coefficient values drop down menu. Y ou may choose fractions of the default start-ing values, zero, or arbitrary User Supplied.If you select User Supplied, EViews will use the values stored in the C coefficient vector at the time of estimation as starting values. T o see the starting values, double click on the coef-ficient vector in the workfile directory. If the values appear to be reasonable, you can close the window and proceed with estimating your model.If you wish to change the starting values, first make certain that the spreadsheet view of the coefficient vector is in edit mode, then enter the coefficient values. When you are finished setting the initial values, close the coefficient vector window and estimate your model.Y ou may also set starting coefficient values from the command window using the PARAM command. Simply enter the P ARAM keyword, followed by pairs of coefficients and theirdesired values:param c(1) 153 c(2) .68 c(3) .15sets C(1)=153, C(2)=.68, and C(3)=.15. All of the other elements of the coefficient vector are left unchanged.Lastly, if you want to use estimated coefficients from another equation, select Proc/Update Coefs from Equation from the equation window toolbar.For nonlinear least squares problems or situations where you specify the starting values,bear in mind that:Setting Estimation Options—627•The objective function must be defined at the starting values. For example, if your objective function contains the expression 1/C(1), then you cannot set C(1) to zero. Similarly, if the objective function contains LOG(C(2)), then C(2) must be greater than zero.• A poor choice of starting values may cause the nonlinear least squares algorithm to fail. EViews begins nonlinear estimation by taking derivatives of the objective func-tion with respect to the parameters, evaluated at these values. If these derivatives are not well behaved, the algorithm may be unable to proceed.If, for example, the starting values are such that the derivatives are all zero, you will immediately see an error message indicating that EViews has encountered a “Near Singular Matrix”, and the estimation procedure will stop.•Unless the objective function is globally concave, iterative algorithms may stop at a local optimum. There will generally be no evidence of this fact in any of the output from estimation.If you are concerned with the possibility of local optima, you may wish to select vari-ous starting values and see whether the estimates converge to the same values. One common suggestion is to estimate the model and then randomly alter each of the esti-mated coefficients by some percentage, then use these new coefficients as starting val-ues in estimation.Iteration and Convergence OptionsThere are two common iteration stopping rules: based on the change in the objective func-tion, or based on the change in parameters. The convergence rule used in EViews is based upon changes in the parameter values. This rule is generally conservative, since the change in the objective function may be quite small as we approach the optimum (this is how we choose the direction), while the parameters may still be changing.The exact rule in EViews is based on comparing the norm of the change in the parameters with the norm of the current parameter values. More specifically, the convergence test is:(40.13)where is the vector of parameters, is the 2-norm of , and tol is the specified toler-ance. However, before taking the norms, each parameter is scaled based on the largestobserved norm across iterations of the derivative of the least squares residuals with respect to that parameter. This automatic scaling system makes the convergence criteria more robust to changes in the scale of the data, but does mean that restarting the optimization from the final converged values may cause additional iterations to take place, due to slight changes in the automatic scaling value when started from the new parameter values.v i 1+()v i ()–2v i ()2----------------------------------tol £v x 2x628—Appendix D. Estimation and Solution OptionsThe estimation process achieves convergence if the stopping rule is reached using the toler-ance specified in the Convergence edit box of the Estimation Dialog or the EstimationOptions Dialog. By default, the box will be filled with the tolerance value specified in the global estimation options, or if the estimation object has previously been estimated, it will be filled with the convergence value specified for the last set of estimates.EViews may stop iterating even when convergence is not achieved. This can happen for two reasons. First, the number of iterations may have reached the prespecified upper bound. In this case, you should reset the maximum number of iterations to a larger number and try iterating until convergence is achieved.Second, EViews may issue an error message indicating a “Failure to improve”after a number of iterations. This means that even though the parameters continue to change, EViews could not find a direction or step size that improves the objective function. This can happen when the objective function is ill-behaved; you should make certain that your model is identified.Y ou might also try other starting values to see if you can approach the optimum from other directions.Lastly, EViews may converge, but warn you that there is a singularity and that the coeffi-cients are not unique. In this case, EViews will not report standard errors or t-statistics for the coefficient estimates.Derivative Computation OptionsIn many EViews estimation procedures, you can specify the form of the function for themean equation. For example, when estimating a regression model, you may specify an arbi-trary nonlinear expression in the coefficients. In these cases, when estimating the model, EViews will compute derivatives of the user-specified function.EViews uses two techniques for evaluating derivatives: numeric (finite difference) and ana-lytic. The approach that is used depends upon the nature of the optimization problem and any user-defined settings:•In most cases, EViews offers the user the choice of computing either analytic or numeric derivatives. By default, EViews will fill the options dialog with the globalestimation settings. If the Use numeric only setting is chosen, EViews will only com-pute the derivatives using finite difference methods. If this setting is not checked,EViews will attempt to compute analytic derivatives, and will use numeric derivativesonly where necessary.•EViews will ignore the numeric derivative setting and use an analytic derivative whenever a coefficient derivative is a constant value.•For some procedures where the range of specifications allowed is limited (e.g., V ARs, pools), EViews always uses analytic first and/or second derivatives, whatever the val-ues of these settings.Optimization Algorithms—629•The derivatives with respect to the AR coefficients in an ARMA specification are always computed analytically while those with respect to the MA coefficients arecomputed numerically.•In a limited number of cases, EViews will always use numeric derivatives. For exam-ple, selected GARCH (see “Derivative Methods” on page192) and state space modelsalways use numeric derivatives. As noted above, MA coefficient derivatives arealways computed numerically.•Logl objects always use numeric derivatives unless you provide the analytic deriva-tives in the specification.Where relevant, the estimationoptions dialog allows you tocontrol the method of takingderivatives. For example, theoptions dialog for standardregression allows you to over-ride the use of EViews analyticderivatives, and to choosebetween favoring speed oraccuracy in the computation ofany numeric derivatives (notethat the additional LS and TSLSoptions are discussed in detailin Chapter25. “AdditionalRegression Methods,” begin-ning on page23).Computing the more accurate numeric derivatives requires additional objective functionevaluations. While the algorithms may change in future versions, at present, EViews com-putes numeric derivatives using either a one-sided finite difference (favor speed), or using a four-point routine using Richardson extrapolation (favor precision). Additional details are provided in Kincaid and Cheney (1996).Analytic derivatives will often be faster and more accurate than numeric derivatives, espe-cially if the analytic derivatives have been simplified and carefully optimized to removecommon subexpressions. Numeric derivatives will sometimes involve fewer floating point operations than analytic, and in these circumstances, may be faster.Optimization AlgorithmsGiven the importance of the proper setting of EViews estimation options, it may prove useful to review briefly various basic optimization algorithms used in nonlinear estimation. Recall630—Appendix D. Estimation and Solution Optionsthat the problem faced in non-linear estimation is to find the values of parameters that optimize (maximize or minimize) an objective function .Iterative optimization algorithms work by taking an initial set of values for the parameters, say , then performing calculations based on these values to obtain a better set of param-eter values, . This process is repeated for , and so on until the objective func-tion no longer improves between iterations.There are three main parts to the optimization process: (1) obtaining the initial parameter values, (2) updating the candidate parameter vector at each iteration, and (3) determining when we have reached the optimum.If the objective function is globally concave so that there is a single maximum, any algo-rithm which improves the parameter vector at each iteration will eventually find this maxi-mum (assuming that the size of the steps taken does not become negligible). If the objective function is not globally concave, different algorithms may find different local maxima, but all iterative algorithms will suffer from the same problem of being unable to tell apart a local and a global maximum.The main thing that distinguishes different algorithms is how quickly they find the maxi-mum. Unfortunately, there are no hard and fast rules. For some problems, one method may be faster, for other problems it may not. EViews provides different algorithms, and will often let you choose which method you would like to use.The following sections outline these methods. The algorithms used in EViews may bebroadly classified into three types: second derivative methods, first derivative methods, and derivative free methods. EViews’ second derivative methods evaluate current parameter val-ues and the first and second derivatives of the objective function for every observation. First derivative methods use only the first derivatives of the objective function during the itera-tion process. As the name suggests, derivative free methods do not compute derivatives.Second Derivative MethodsFor binary, ordered, censored, and count models, EViews can estimate the model using Newton-Raphson or quadratic hill-climbing.Newton-RaphsonCandidate values for the parameters may be obtained using the method of Newton-Raphson by linearizing the first order conditions at the current parameter values, :(40.14)where is the gradient vector , and is the Hessian matrix .v F v ()v 0()v 1()v 2()v 3()F v v 1()∂F ∂v §v i ()g i ()H i ()v i 1+()v i ()–()+0=v i 1+()v i ()H i ()1–g i ()–=g ∂F ∂v §H ∂2F ∂v 2§Optimization Algorithms—631If the function is quadratic, Newton-Raphson will find the maximum in a single iteration. If the function is not quadratic, the success of the algorithm will depend on how well a local quadratic approximation captures the shape of the function.Quadratic hill-climbing (Goldfeld-Quandt)This method, which is a straightforward variation on Newton-Raphson, is sometimes attrib-uted to Goldfeld and Quandt. Quadratic hill-climbing modifies the Newton-Raphson algo-rithm by adding a correction matrix (or ridge factor) to the Hessian. The quadratic hill-climbing updating algorithm is given by:(40.15)where is the identity matrix and is a positive number that is chosen by the algorithm.The effect of this modification is to push the parameter estimates in the direction of the gra-dient vector. The idea is that when we are far from the maximum, the local quadraticapproximation to the function may be a poor guide to its overall shape, so we may be better off simply following the gradient. The correction may provide better performance at loca-tions far from the optimum, and allows for computation of the direction vector in cases where the Hessian is near singular.For models which may be estimated using second derivative methods, EViews uses qua-dratic hill-climbing as its default method. Y ou may elect to use traditional Newton-Raphson, or the first derivative methods described below, by selecting the desired algorithm in the Options menu.Note that asymptotic standard errors are always computed from the unmodified Hessian once convergence is achieved.First Derivative MethodsSecond derivative methods may be computationally costly since we need to evaluate the elements of the second derivative matrix at every iteration. Moreover, second derivatives calculated may be difficult to compute accurately. An alternative is to employ methods which require only the first derivatives of the objective function at the parameter values.For general nonlinear models (nonlinear least squares, ARCH and GARCH, nonlinear system estimators, GMM, State Space), EViews provides two first derivative methods: Gauss-New-ton/BHHH or Marquardt.Gauss-Newton/BHHHThis algorithm follows Newton-Raphson, but replaces the negative of the Hessian by an approximation formed from the sum of the outer product of the gradient vectors for each observation’s contribution to the objective function. For least squares and log likelihood v i 1+()v i ()H ˜i ()1–g i ()–=where H ˜i ()–H i ()–a I +=I a k k 1+()2§632—Appendix D. Estimation and Solution Optionsfunctions, this approximation is asymptotically equivalent to the actual Hessian when evalu-ated at the parameter values which maximize the function. When evaluated away from the maximum, this approximation may be quite poor.The algorithm is referred to as Gauss-Newton for general nonlinear least squares problems, and often attributed to Berndt, Hall, Hall and Hausman (BHHH ) for maximum likelihood problems.The advantages of approximating the negative Hessian by the outer product of the gradient are that (1) we need to evaluate only the first derivatives, and (2) the outer product is neces-sarily positive semi-definite. The disadvantage is that, away from the maximum, thisapproximation may provide a poor guide to the overall shape of the function, so that more iterations may be needed for convergence.MarquardtThe Marquardt algorithm modifies the Gauss-Newton algorithm in exactly the same manner as quadratic hill climbing modifies the Newton-Raphson method (by adding a correction matrix (or ridge factor) to the Hessian approximation).The ridge correction handles numerical problems when the outer product is near singular and may improve the convergence rate. As above, the algorithm pushes the updated param-eter values in the direction of the gradient.For models which may be estimated using first derivative methods, EViews uses Marquardt as its default method. Y ou may elect to use traditional Gauss-Newton via the Options menu.Note that asymptotic standard errors are always computed from the unmodified (Gauss-Newton) Hessian approximation once convergence is achieved.Choosing the step sizeAt each iteration, we can search along the given direction for the optimal step size. EViews performs a simple trial-and-error search at each iteration to determine a step size that improves the objective function. This procedure is sometimes referred to as squeezing or stretching .Note that while EViews will make a crude attempt to find a good step, is not actually optimized at each iteration since the computation of the direction vector is often moreimportant than the choice of the step size. It is possible, however, that EViews will be unable to find a step size that improves the objective function. In this case, EViews will issue an error message.EViews also performs a crude trial-and-error search to determine the scale factor for Mar-quardt and quadratic hill-climbing methods.l l aNonlinear Equation Solution Methods—633Derivative free methodsOther optimization routines do not require the computation of derivatives. The grid search is a leading example. Grid search simply computes the objective function on a grid of parame-ter values and chooses the parameters with the highest values. Grid search is computation-ally costly, especially for multi-parameter models.EViews uses (a version of) grid search for the exponential smoothing routine.Nonlinear Equation Solution MethodsWhen solving a nonlinear equation system, EViews first analyzes the system to determine if the system can be separated into two or more blocks of equations which can be solved sequentially rather than simultaneously. Technically, this is done by using a graph represen-tation of the equation system where each variable is a vertex and each equation provides a set of edges. A well known algorithm from graph theory is then used to find the strongly connected components of the directed graph.Once the blocks have been determined, each block is solved for in turn. If the block contains no simultaneity, each equation in the block is simply evaluated once to obtain values for each of the variables.If a block contains simultaneity, the equations in that block are solved by either a Gauss-Seidel or Newton method, depending on how the solver options have been set.Gauss-SeidelBy default, EViews uses the Gauss-Seidel method when solving systems of nonlinear equa-tions. Suppose the system of equations is given by:(40.16)where are the endogenous variables and are the exogenous variables.The problem is to find a fixed point such that . Gauss-Seidel employs an itera-tive updating rule of the form:.(40.17)to find the solution. At each iteration, EViews solves the equations in the order that they appear in the model. If an endogenous variable that has already been solved for in that iter-ation appears later in some other equation, EViews uses the value as solved in that iteration. For example, the k -th variable in the i -th iteration is solved by:x 1f 1x 1x 2ºx N z ,,,,()=x 2f 2x 1x 2ºx N z ,,,,()=x N f N x 1x 2ºx N z ,,,,()=x z x f x z ,()=x i 1+()f x i ()z ,()=634—Appendix D. Estimation and Solution Options.(40.18)The performance of the Gauss-Seidel method can be affected be reordering of the equations. If the Gauss-Seidel method converges slowly or fails to converge, you should try moving the equations with relatively few and unimportant right-hand side endogenous variables so that they appear early in the model.Newton's MethodNewton’s method for solving a system of nonlinear equations consists of repeatedly solving a local linear approximation to the system.Consider the system of equations written in implicit form:(40.19)where is the set of equations, is the vector of endogenous variables and is the vector of exogenous variables.In Newton’s method, we take a linear approximation to the system around some values and :(40.20)and then use this approximation to construct an iterative procedure for updating our current guess for :(40.21)where raising to the power of -1 denotes matrix inversion.The procedure is repeated until the changes in between periods are smaller than a speci-fied tolerance.Note that in contrast to Gauss-Seidel, the ordering of equations under Newton does not affect the rate of convergence of the algorithm.Broyden's MethodBroyden's Method is a modification of Newton's method which tries to decrease the calcula-tional cost of each iteration by using an approximation to the derivatives of the equation sys-tem rather than the true derivatives of the equation system when calculating the Newton step. That is, at each iteration, Broyden's method takes a step:(40.22)x k i ()f k x 1i ()x 2i ()ºx k 1–i ()x k i 1–()x k 1+i 1–()ºx ,,,N i 1–()z ,,,,,()=F x z ,()0=F x z x ∗z ∗F x z ,()F x ∗z ∗,()x ∂∂F x ∗z ∗,()x D +0==x x t 1+x t x ∂∂F x t z ∗,()1–F x t z ∗,()–=x x t 1+x t J t 1–F x t z ∗,()–=References—635where is the current approximation to the matrix of derivatives of the equation system.As well as updating the value of at each iteration, Broyden's method also updates the existing Jacobian approximation, , at each iteration based on the difference between the observed change in the residuals of the equation system and the change in the residuals pre-dicted by a linear approximation to the equation system based on the current Jacobianapproximation.In particular, Broyden's method uses the following equation to update :(40.23)where . This update has a number of desirable properties (see Chapter 8 of Dennis and Schnabel (1983) for details).In EViews, the Jacobian approximation is initialized by taking the true derivatives of the equation system at the starting values of . The updating procedure given above is repeated until changes in between periods become smaller than a specified tolerance. In some cases the method may stall before reaching a solution, in which case a fresh set of deriva-tives of the equation system is taken at the current values of , and the updating is contin-ued using these derivatives as the new Jacobian approximation.Broyden's method shares many of the properties of Newton's method including the fact that it is not dependent on the ordering of equations in the system and that it will generally con-verge quickly in the vicinity of a solution. In comparison to Newton's method, Broyden's method will typically take less time to perform each iteration, but may take more iterations to converge to a solution. In most cases Broyden's method will take less overall time to solve a system than Newton's method, but the relative performance will depend on the structure of the derivatives of the equation system.ReferencesAmemiya, Takeshi (1983). “Nonlinear Regression Models,” Chapter 6 in Z. Griliches and M. D. Intriliga-tor (eds.), Handbook of Econometrics, Volume 1, Amsterdam: Elsevier Science Publishers B.V . Dennis, J. E. and R. B. Schnabel (1983). “Secant Methods for Systems of Nonlinear Equations,” Numeri-cal Methods for Unconstrained Optimization and Nonlinear Equations . Prentice-Hall, London.Kincaid, David, and Ward Cheney (1996). Numerical Analysis , 2nd edition , Pacific Grove, CA: Brooks/Cole Publishing Company.Press, W . H., S. A. T eukolsky, W . T. Vetterling, and B. P. Flannery (1992). Numerical Recipes in C , 2ndedition , Cambridge University Press.Quandt, Richard E. (1983). “Computational Problems and Methods,” Chapter 12 in Z. Griliches and M.D. Intriligator (eds.), Handbook of Econometrics, Volume 1, Amsterdam: Elsevier Science Publish-ers B.V .Thisted, Ronald A. (1988). Elements of Statistical Computing , New Y ork: Chapman and Hall.J t x J t J J t 1+J t F x t 1+z ∗,()F x t z ∗,()–J t D x –()D x ¢D x ¢D x ----------------------------------------------------------------------------------------+=D x x t 1+x t –=x x x。