An Improved EMD Online Learning-Based Model for Gold Market Forecasting

An Improved EMD Online Learning-Based Model for Gold Market ForecastingShifei Zhou and Kin Keung LaiAbstract. In this paper, an improved EMD (Empirical Mode Decomposition) online learning-based model for gold market forecasting is proposed. First, we adopt the EMD method to divide the time series data into different subsets. Sec-ond, a back-propagation neural network model (BPNN) is used to function as the prediction model in our system. We update the online learning rate of BPNN in-stantly as well as the weight matrix. Finally, a rating method is used to identify the most suitable BPNN model for further prediction. The experiment results show that our system has a good forecasting performance.1 Introduction1.1 MotivationForecasting gold price becomes increasingly important. For a long history, the trading of gold in the international market is continuously active. The derivative of gold trading on the international gold market owns great variety. It mainly con-tains gold future, gold option, gold forward contracts, and so on[1]. Remarkably, since the price of gold varies within a limited range, this means gold is able to re-duce the effect of inflation, control the rise of price and help to carry out constric-tive monetary policy. Hence, gold becomes an important risk hedging tool as well as investment tool. Therefore, the works of predicting the price of gold market be-comes very significant and important to investors.However, the current forecasting algorithm has poor precision performance for non-linear problem. We will mainly focus on the artificial neural network (ANN) forecasting algorithm. First, the traditional ANN prediction algorithms employ a Shifei Zhou · Kin Keung LaiDepartment of Management Sciences, Collage of Business, City University of Hong Kong, Kowloon, Hong Konge-mail: shifzhou@.hkKin Keung LaiSchool of business and management, North China Electric Power University,Beijing, Chinae-mail: MSKKLAI@.hkJ. Watada et al. (Eds.): Intelligent Decision Technologies, SIST 10, pp. 75–84. © Springer-Verlag Berlin Heidelberg 201176 S. Zhou and K.K. Laiglobal fixed learning rate to change the weight matrix. This will cause two problems. On one hand, if the learning rate is too small, the weight matrix will change very slowly. Then, the network training process will take a long time to converge[2].Fig. 1(a) Algorithm Convergence of Small Learning Rate. Fig. 1(b) Algorithm Convergence of Large Learning Rate.On the other hand, if the learning rate is too large, we may miss the minimal optimal and cause the algorithm to diverge, like what has shown in Figure 1. Sec-ond, unlike the polynomial function, most of the non-linear problems have com-plex error surfaces. Hence, there will be many local minima. Accordingly, the training process may be trapped in such minima[3]. And the prediction precision will be affected.1.2 ContributionsIn this paper, we propose an improved empirical mode decomposition mod-el(IEMD) to forecast the trend and price of gold market. IEMD model can make efficient use of the history data to predict the future price. During the back-propagation neural network training process, we dynamically change the learning rate by using a global meta-learning rate. Our main contributions are listed as followings:(1) High Convergence Speed . When the trend of history data is downward, we will change the learning rate to increase the weight matrix in the same direction and speed up the decreasing speed of error function. Similarly, when the trend is upslope, the gradient will be increased. And the process will “climb the hill” of er-ror function effectively. Both of methods will speed up the convergence speed of neural network training process.(2) High Prediction Precision . By calculating the gradient of the error function, we minimize the mean square error through changing the weight matrix of predic-tion model. The novel system we propose is able to update the learning rate of back-propagation neural network instantly, and capture the trend of data series. As a result, our system is able to give a high prediction precision for gold market.An Improved EMD Online Learning-Based Model for Gold Market Forecasting 77(3) Significant Application. Gold plays an important role on the international market. It obtains an increasing attention from investors. Not only it owns the function of risk hedging, but also it can be stored without depreciation. A precise prediction of price on the gold market will help investors to save money. This, therefore, shows the potential significance of our system.2 Related Works2.1 Empirical Mode DecompositionWhat is empirical mode decomposition? The empirical mode decomposition method was firstly proposed by Huang et al[4]. This time series data decomposi-tion technique applies Hilbert transform to nonlinear and nonstationary time series data. The main idea of EMD is to decompose a time series into a sum of oscilla-tory functions.The EMD can make highly use of the history data. Since the EMD is able to di-vide the original time series into several subsets of data by using the intrinsic mode functions(IMF), the characteristics of different subsets of series can be iden-tified through training different BPNNs. We can use these trained BPNNs to pre-dict the future trend of time series and select one of the network model for future prediction.2.2 Online Learning AlgorithmWhat is online learning algorithm? In the online learning algorithm, the weight vectors of input data are updated immediately after the presentation of each data point[5]. Therefore, the online learning algorithm is able to adjust the weight of input data and capture the trend of time series instantly.Online learning algorithm has a number of advantages[5]. As the weight matrix is updated recursively, this algorithm can be use when there is no fixed training set and new data keeps coming in. Besides, as the local minima is a problem for gra-dient descent in nonlinear models, online learning can easily escape from local minima when a noise data is input. In this paper, we will use online learning algo-rithm to update the weight matrix. At the same time, we also update the learning rate to make sure the network model can be trained at a fast convergence speed.3 Improved Empirical Mode Decomposition Model (IEMD)3.1 IEMD Model StructureYu et al has proposed an EMD-based neural network ensemble learning model [6, 7]. In this model, presented as EMD-FNN-ALNN for shot, they use forward neural network (FNN) as the prediction model. The time series data is divided into several subsets by using IMF[4]. Each subset is used as an input data to the FNN model for training. The FNN model is finally used to predict future outcome. All78 S. Zhou and K.K. Laithe predicted results will be assigned a weight and combined together by using an Adaptive Linear Neural Network (ALNN). This model is pretty good to use the history data. However, prediction results will be greatly affected by the weight of each FNN model. Besides, if one of the FNN models has made a precise predic-tion, the final prediction may be imprecise because all the predicted results are summed together.To overcome these problems, we propose an improved EMD online learning-based back-propagation neural network model.Fig. 2 An Improve EMD Online Learning-based BPNN Model.The structure of IEMD-BPNN-PMR model is shown in Fig 2.To make efficient use of the history data, we adopt the EMD method to partition the data into several subsets. Each subset of data is used to train the back-propagation neural network. The time series are decomposed according to the sifting procedure proposed by Yu at el[6]. This procedure is repeated until all the data are divided and each sub-set of data has only one local minimal or maximal. At the end of the sifting proce-dure, the data series x(t) can be expressed as follows:1()()()ni n i x t c t r t ==+∑ (3.1)Where n is the number of IMFs, namely number of data subsets, c i (t) is the ith IMF, and r n (t) is the final residue of the procedure. Thus, any time series data can be decomposed by using this EMD method. And the frequency components which are contained in each frequency band are different. They also change the variance of data series x(t), while r n (t) represent the trend of the data series x(t).3.2 Back-Propagation Neural Network (BPNN)Back-propagation neural network is one type of artificial neural network[8], which is a class of typical intelligent learning paradigm and widely used in the field of data forecasting. In this paper, we use a three-layer neural network with error-back-propagation algorithm. And this network model is used as the predicting model of our system. There are many nodes in the hidden layer. Hence, there will be multiple combinations of the weights and data points.An Improved EMD Online Learning-Based Model for Gold Market Forecasting 79BPNN is able to provide a flexible mapping between inputs and outputs. In the work of Hornik[9], they had proved that a three-layer of feed-forward neural net-work with an identity transfer function in the output unit and logistic functions in the hidden layer can approximate any continuous function arbitrarily well. There-fore, in this paper, we use the three layer of BPNN as the forecasting model. Assume that y i represent the ith hidden node, then01ni i i i y w x w ==+∑ (3.2)where x i is the data input and w i is the corresponding weight. Meanwhile, y i is also called a net input, which will be used in the later calculation. After the hidden layer’s nodes are generated, we use the activation function y j =f(y i ) to transform the net input to output, where j =1,…,m and m represents the number of nodes in hid-den layer. By using this network model, we finally make the prediction for the time series.3.3 Prediction Model RatingWe have to ensemble the prediction results from different BPNN models. Thus, the prediction model rating (PMR) method is proposed to handle this problem. In this section, we will calculate the mean squared error E (MSE) of every output of the BPNN model.212()j E j j t y =− (3.3) Where t j is the target value of the time series; y j is the predicted outcome from the jth BPNN model; j ranges from 1 to m . Then, we compare all these E j and find the minimal one, and increase the rate of jth model by 1. This procedure is repeated throughout the training process. Finally, we will get a BPNN prediction model with largest rate for prediction.4 Improved Online Learning Algorithm4.1 Online Weight UpdateIn this section, an improved online learning algorithm is proposed. We will start with the mean squared error and make some mathematical methods to deduct the relationship between weight and MSE.First, we use the chain rule to decompose the gradient into two factors. Then, by using (3.3), we have the following result2'1()2()()j j j j j j i i ij j ijt y E y G y t f y x w y w ∂−∂∂===−∂∂∂ (4.1)80 S. Zhou and K.K. LaiWhere G represents the gradient of MSE. G is not necessary a positive value since f’(y i ) may be a negative value. Here, we assume G is positive. Then, if there is an increment in weight w ij , the error function will be accordingly increased. There-fore, this relationship helps us to modify the weight matrix to control the errors. Then, from Eqn (2.4) of chapter 2 in the works of Saad[5], we can calculate the increment of weight as follows.'()()j ij ij ij j j i i ij E w t y f y x w μμ∂Δ=−=−∂ (4.2)where ij μrepresents the online learning rate. We can see that the increment of weight will depend on the learning rate. If ij μis large, a single change to the weight will similarly large.The online weight update, that uses a local and time-varying learning rate for each weight, can be defined as follows.(1)()()()ij ij ij ij w t w t t w t μ+=+Δ (4.3) where ()ij w t Δ is the increment of weigh.4.2 Online Learning Rate UpdateAs discussed above, the online learning algorithm owns the ability of escaping from the local minimum. The noise in the stochastic error surface is likely to bounce the network out of local minima as long as they are not too severe.When the weight matrix is modified, we simultaneously change the local learn-ing rate by gradient descent method. In order to reduce error function, we will change the learning rate before changing the weights. Then, the gradient can be defined as follows.'(1)(1)(1)()()()(1)(1)()ij j j i i ij ij ij ij w t E t E t y t f y x w t t w t t μμ∂+∂+∂+==−Δ∂+∂+∂ (4.4) Then, we can define the online learning rate as follows.'()(1)()()()ij ij j j i i ij t t t y f y x w t μμλ=−+−Δ (4.5) λis a global meta-learning rate and is fixed globally. Since the mean average error and weight matrix are changed instantly with the series data input, the local learning rate is also dynamically adapted according to the change of weight matrix and the result of transfer function. As a result, this dynamic learning rate is able to change the step of learning and improve the performance of algorithm convergence.An Improved EMD Online Learning-Based Model for Gold Market Forecasting 815 Experiments5.1 Evaluation CriteriaIn this part, three classes of measurement criteria including level prediction, con-vergence speed and directional forecasting are employed to evaluate the perform-ance. Firstly, unlike the work in [10], we use the mean absolute error (MAE) [11] rather than the criteria of root mean squared error (RMSE)[10] because RMSE varies with the variability within the distribution of error magnitudes and MAE is a more natural measurement for the average error. Thus, we use MAE to compare the prediction precision of different forecasting model with our novel model.11ˆN ii i MAE y y N ==−∑ (5.1)Where N represents the total number of data points; ˆi yis the prediction result of ith data point; y i is the target value of ith data point. The smaller the value of MAE , the more precise the prediction model will be.Secondly, we will compare our model with other prediction models on the as-pect of convergence speed. Even though an algorithm can run with a high speed, this does not mean that the algorithm is a good one, because if the prediction result owns a large value of MAE, the algorithm will be valueless to investors. There-fore, we will consider the running time of the models together with the prediction precision.Finally, in order to measure the moving direction of time series accurately, we use a directional statistic (D stat ) [12], which is defined by the following equation.11100%Nstat i i D a N ==×∑ (5.2) where 1i a =if 11ˆ()()0i i i i yy y y ++−−>, otherwise,0i a =. N represents the total number of testing samples. The larger this factor is, the more proper direction the forecasting model will have.5.2 Experimental ResultsIn this section, we use the daily gold future price data from database of Global Fi-nancial Data (GFD). The database provides the most extensive time series data on commodities which is available anywhere. It includes historical data on about 100 different commodities. In this paper, we use the data spanning from 2 Jan 3 2006 to 13 Jul 2009, totally 820 observations as training data set. And we use the rest of the time series spanning from 14 Jul 2009 to 14 Jul 2010 for the evaluation of the prediction performance, which contains totally 252 observations. The training data and testing data are input into our model for performance evaluation. The empiri-cal results are the average performance measurement of the models because we82 S. Zhou and K.K. Laicompare the results by running different models on the data set several times. We finally count the results and get an average performance between all these models. In many literatures, researchers use the logistic sigmoid 1()1u f u e −=+as ac-tivation function for the units in hidden layer. This function has a small asymmet-ric range from 0 to 1. Then, this logistic sigmoid function will worsen the condi-tion of network. Accordingly, in our prediction model, we choose tanh as the activation function, which owns a symmetric range from -1 to 1 and is more suit-able for activation function for hidden units. Since'2tanh ()1tanh()u u =−, then we have '2()1i i f y y =−.Table 1 The Average MAE and Running Time Comparisons.Forecasting Model MAE Rank Time(m) RankFNN 4.27 5 94 1BPNN 2.53 4 109 3ARIMA 2.46 3 136 5EMD-FNN-ALNN 1.78 2124 4 IEMD-BPNN-PMR 0.92 1 102 2From Table 1, the MAE values of different models are different. FNN has large MAE of 4.27 and rank the last. Our model, IEMD-BPNN-PMR, ranks at the first place with a MAE value of 0.92 and is followed by EMD-FNN-ALNN model, ARIMA model, BPNN model and FNN model. This sequence represents a de-scending order of prediction precision. At the aspect of running time, we can see that the FNN model spend less time to converge. It takes 94 minutes in average, which is only 16.6% of the total time of all models. Our model ranks at the second place and takes 102 minutes to converge.The FNN model is a simple type of neural network. For a three layer of FNN, the model consists of a linear combination of the input data and the weight matrix and an activation function to transfer the combined results. Hence, the FNN model will take least running time to converge. However, the precision of this model is also the worst. Conversely, in our model, we adopt a BPNN model as the predic-tion model. Although it takes more time to get converged, but the prediction re-sults are more precise than the FNN model. Our model runs faster than the indi-vidual BPNN model. This is because we have employed an EMD method to decompose the input data, which ensures the network to get converged fast to some extent. Besides, the usage of improved online learning algorithm makes sure the weight matrix is changed instantly at each epoch. As a result, the training process has a high convergence speed.An Improved EMD Online Learning-Based Model for Gold Market Forecasting 83 Table 2 The D stat (%) Comparisons.Forecasting Model D stat (%) Rank4FNN 54.673BPNN 59.415ARIMA 51.86EMD-FNN-ALNN 61.58 2IEMD-BPNN-PMR 75.24 1The results of Table 2 show the comparisons of different models on the aspect of directional forecasting. Ranging at the first place is the IEMD-BPNN-PMR model, which is followed by EMD-FNN-ALNN model. The reason why our model can predict correctly the trend of data series most of the time is that we make some changes to the original BPNN model and use tanh as the activation function, which has a symmetric range from -1 to 1. This will help the activation function to capture the series trend flexibly and accurately.6 ConclusionsIn this paper, we proposed an improved EMD online learning-based model for gold market forecasting. We firstly use the EMD method to partition the time se-ries data into several subsets of data series, which will be used for training the neural network. Then, back-propagation neural network is employed to function as the prediction model in our novel system since BPNN is one of widely studied and used learning algorithms in the academy. By studying the essential process of BPNN mathematically, we give a deduction process to show that the online learn-ing can be updated instantly. As the learning rate is online updated, the weight will also be updated after every data input. This will ensure the BPNN model to be able to escape from the local minimum of the error function and, therefore, will show a good performance of quick convergence speed. The experiment results show that our system has performed a good prediction results as well as running time.Although we apply the novel algorithm to the gold market and obtain some im-pressive results, there is still some improvement we can make. The global meta-learning rate can be further studied to see whether there exists a relationship be-tween MAE and data samples. Besides, our model can also be applied to the stock and warrant market. There will be more interesting and practical results.References[1]Shafieea, S., Topalb, E.: An overview of global gold market and gold price forecast-ing. Resources Policy 35(3), 178–189 (2010)[2]Orr, G.B., Leen, T.K.: Using Curvature Information for Fast Stochastic Search. In:Neural Information Processing Systems, pp. 606–612 (1996)84 S. Zhou and K.K. Lai [3]Orr, G.B., Leen, T.K.: Learning in neural networks with local minima. Physical Re-view. A 46(8), 5221–5231 (1992)[4]Huang, N.E., Shen, Z., Long, S.R., Wu, M.C., Shih, H.H., Zheng, Q., Yen, N.C.,Tung, C.C., Liu, H.H.: The empirical mode decomposition and the Hilbert spectrum for nonlinear and nonstationary time series analysis. Proceedings of the Royal Society A: Mathematical, Physical & Engineering Sciences 454, 903–995 (1998)[5]Saad, D. (ed.): On-Line Learning in Neural Networks. Cambridge University Press,New York (1999)[6]Yu, L., Wang, S., Lai, K. K.: An EMD-Based Neural Network Ensemble LearningModel for World Crude Oil Spot Price Forecasting. In: Soft Computing Applications in Business 2008, pp. 261-271 (2008)[7]Yu, L., Lai, K. K., Wang, S., He, K.: Oil Price Forecasting with an EMD-Based Mul-tiscale Neural Network Learning Paradigm. In: International Conference on Compu-tational Science, vol. (3), pp. 925–932 (2007)[8]White, H.: Connectionist nonparametric regression: multilayer feedforward networkscan learn arbitrary mappings. Neural Networks 3, 535–549 (1990)[9]Hornik, K., Stinchocombe, M., White, H.: Multilayer feedforward networks are uni-versal approximators. Neural Networks 2, 359–366 (1989)[10]Bo, W., Wang, S., Lai, K.K.: A Hybrid ARCH-M and BP Neural Network Model ForGSCI Futures Price Forecasting. In: International Conference on Computational Sci-ence, vol. (3), pp. 917–924 (2007)[11]Willmott, C.J., Matsuura, K.: Advantages of the mean absolute error (MAE) over theroot mean square error (RMSE) in assessing average model performance. Climate Research 30(3), 79–82 (2005)[12]Yu, L., Wang, S., Lai, K. K.: A novel nonlinear ensemble forecasting model incorpo-rating GLAR and ANN for foreign exchange rates. Computers & OR 32, 2523–2541 (2005)。