Application of intelligent time series prediction method to dew point forecast

1.   Introduction

Weather forecasting is of great importance to human beings ^[1,2]. It means to analyze various atmospheric data, including air pressure, surrounding temperature, wind intensity and direction, rainfall, and so on. This study impacts multiple fields such as industry, agriculture ^[3], disaster protection ^[4] and energy management ^[5]. For instance, weather forecasting can be used to ensure storage as well as transportation in order to increase quantity and improve quality in agricultural planting and production. Moreover, by accurate prediction of local wind power, hydropower, and solar energy, we can make a better use of these renewable resources under sustainable strategy ^[6]. Certainly, the most important purpose is to reduce unnecessary losses of properties and lives as much as possible ^[7].

Among various parameters of weather is there an important parameter called dew point. The dew point temperature reflects the humidity of the air and is widely used in industrial production and weather forecasting. Additionally, in the pneumatic system, researches on dew point are beneficial to improvement of equipment reliability and service life ^[8]. As current dew point studies mainly rely on instrumental measurement in specific areas, equation solving remains the first choice. Most of these expressions are differential equations, whose results are almost determined by initial conditions. The prediction quality is greatly influenced even if there are few deviations from true values.

Traditional method for weather forecasting is the numerical weather prediction (NWP) ^[9]. Its core idea is to construct a general circulation model following the physical principles and perform numerical analysis using computers. Therefore, weather forecasting has often been troubled due to its complicated computation, large amounts of changeable parameters and latent continuity among data. As is discussed in ^[10], although better improving measures are undertaken, these problems still exist and call the appearance of machine learning.

Machine learning (ML), along with the subsequent deep learning (DL) as a great idea has multiple contributions to weather forecasting. Compared with classical NWP methods which take various complex equations and complicated computation into account, all that machine learning needs are almost the input data, a transmitting model and a modification step ^[11]. Accompanied by the huge improvement of computing power and increasingly big-data processing technology, machine learning is playing a significant role in weather forecasting, especially for one-dimension time series, since these series are more suitable for data with certain regularity and duplicate procedures ^[12,13].

As for time series, there are a good many forecasting models. Time series are a series of values observed in a sequence. They are characteristic of nonlinearity, correlation, and volatility ^[14,15,16]. Researchers have been devoted to its prediction for a long time. At the beginning, they choose linear fitting methods such as the autoregressive moving average (ARIMA) ^[17,18], the exponential smoothing model ^[19] and the naive method ^[20]. But most of them suffer from their trouble of solving nonlinear problems. Nowadays, with the development of ML and DL, nonlinearity is well resolved. Meanwhile, researchers can obtain better forecasting results benefiting from them. These methods include support vector machine (SVM) ^[21], fuzzy-series analysis ^[22], artificial neural networks (ANN) ^{[23,24,25,26]} and their variants. But when predicting dew point temperatures using SVM and random forest alike ^[27], they are still troubled by complex mathematical models.

ANN is a representative researching framework among all the methods due to its data-driven and big-data mining features ^[28]. By putting existing data into the ANN model to train, the hidden connection among data can be found, thus obtaining appropriate results. Its another advantage is adaptability ^[29]. The ANN can be adaptive based on settings of some updating parameters, whose modification can significantly improve forecasting results. Although the ANN reveals a good many advantages, the model performs not so well in the time series prediction due to their changeable features and unstable volatility under the circumstance of weather.

Multilayer Perceptron (MLP) ^[30] develops on the basis of perceptron, which is a one-layer classifier for the linear separable case. Given enough hidden layers, it can achieve high-level results. Nevertheless, it lacks flexibility when addressing problems about time series.

Elman ^[31] is a kind of ANN adding a context unit to basic neural networks, which makes it a BP-like forward mechanism. The context unit receives previous state output as the input of hidden units. The Elman network boasts nonlinearity and time-memory feature, which is sensitive to historical data and help construct required models. However, it is troubled by computational costs and overfitting problems.

Support Vector Regression (SVR) ^[32] is an application to series prediction based on SVM, which utilizes kernel functions, sparse solution to decide the number of support vectors. Although the dimension of the input data has no impact on the computing complexity, the absence of preprocessing leads to biased forecasts.

AR is the abbreviation of Autoregressive Recurrent Network. DeepAR ^[33], which is involved in the study, creatively combines conceptions of time-variability, continuity, and dynamicity. Moreover, it introduces the probability distribution and improves the accuracy of prediction results. However, when confronted with multi-step prediction, time cost is an unbearable problem. And taking the mean values as final results may result in distortion as well.

Below are other state-of-the-art models with memorability for time series. As a classical variant of recurrent neural network (RNN), long short-term memory (LSTM) is widely used due to its ability to avoid vanishing gradients in RNN ^[34]. The modification is to introduce more layers so as to alleviate multiplication of past time units. Existing researches show that LSTM performs well in financial time series ^[35]. Furthermore, for simplification, researchers proposed the gate recurrent unit (GRU) with less layers ^[36]. A few years later, Jaeger put forward a novel network called echo state network (ESN) ^[37]. The ESN develops based on RNN as well and absorbs the nonlinear high-dimensional transformation feature as the SVM does. ESN has widely applied to temperature time series ^[38] but suffer from unstable probability distributions which is harmful to its prediction. In addition, the three memorable algorithms above are time-consuming when series become longer.

Putting the above factors aside, original series are chaotic and disorganized, which needs a preprocessing method to reduce their adverse effects on predicting. A classical method is empirical mode decomposition (EMD) ^[39]. It is a mathematical method to separate a sequence into the sum of multiple subseries based on the Fourier transformation. This approach has strong interpretability but has little connection with basic features of the original data.

Given the situation above, we need a model concerning both data preprocessing and close relation with features of series. In recent studies, instead of traditional mathematical EMD, a novel decomposing method ^[40] based on sequence features called STL is used. The seasonal and trend elements of the original series are successfully extracted using the STL, an abbreviation for the seasonal-trend decomposition procedure based on LOESS. Then the original problem is simplified and become a prediction of the third subseries. Here, a model called dendritic neuron model (DNM) ^[41] inspired by biological excitement is introduced. By deciding which branch is activated, researchers can construct a neural network with complex relations. With the use of sigmoid functions and multiplicated units, nonlinearity is well solved. Furthermore, as time series are basically one-dimension, the network can be further improved to DNM* by simplifying several functions of DNM. So far, a STLDNM* approach is constructed in order to predict time series.

Considering the huge significance of weather forecasting, and the urgent need to overcome difficulties mentioned above, we propose the STLDNM* model to make a solution. The model successfully combines the decomposition technology STL and the machine learning model DNM, and experiments demonstrate that the proposed model performs better in predicting instant weather conditions than other cutting-edge methods. It excels in both aspects of accuracy and efficiency, thus has a promising prospect in weather forecasting in the real world.

The main contributions are summarized as follows:

• To tackle the problem of predicting dew point series, a new combination of the decomposing method STL and the forecasting model DNM is proposed. This combination boasts both consistency and accuracy in the application of weather forecasting.

• The DNM model is improved to be DNM* which is more suitable for time series. DNM* is better on both stability and accuracy and has potentials for more chaotic series. BP is chosen as the optimization method due to its simplicity. In this paper, DNM* manages to predict dew points of different cities nationwide.

2.   Methods

2.1.   Seasonal-trend decomposition based on LOESS (STL)

The STL is a method of decomposing time series data into three elements: the seasonal, trend and residual elements. Its formula is represented as follows:

where ${T}_{k}$, ${S}_{k}$, ${R}_{k}$ respectively denotes the trend, seasonal and residual element, for $k\in [1, M]$, $M$ is the length of time series.

LOESS, or locally weighted regression, is its full name. It aims to smooth the trend and seasonal parts of time series ^[42]. Two variables, ${x}_{i}$ and ${y}_{i}$ ($i = \mathrm{1, 2}, \dots, n$), are adopted to fit the regression curve, denoted by $\widehat{h}\left(x\right)$. Given $x$, fitting value of $\widehat{h}\left(x\right)$ is the smooth of $y$ along the scale of the independent variable. Therefore, we shall deal with datasets with missing values.

$\widehat{h}\left(x\right)$ is obtained considering two cases. Assume a positive integer $r$, then we shall launch a discussion below:

1) If $r\le n$, we choose the $r$ points from ${\{x}_{i}\}$ which stand closest to $x$. Then every point will be given a specific local weight according to how far it is from $x$. Below are two related equations. ${\lambda }_{r}\left(x\right)$ denotes how far $x$ is from the $r$ th farthest point ${x}_{i}$. $W$ is tricube weight function. ${v}_{i}\left(x\right)$ is the local weight of ${x}_{i}$ :

2) If $r > n$, the definition of ${\lambda }_{r}\left(x\right)$ becomes:

From these formulae above, we can see that ${x}_{i}$ closest to $x$ have the largest weights. As ${x}_{i}$ becomes farther from $x$, the weight decreases and eventually down to 0. This time ${x}_{i}$ is the $r$ th farthest point. The polynomial function of $\widehat{h}\left(x\right)$ is at last computed at the point $({x}_{i}, {y}_{i})$, then the locally-fitted value of $x$ is $\widehat{h}\left(x\right)$.

The STL embeds two iterated loops: the inner loop and the outer loop. The seasonal element is updated on the detrending series in the inner loop. Then the resulting series are robustly weighted and run into the outer loop, the weight of which is computed in the outer loop. After that, the method implements multiple times of cycles as mentioned above.

In this paper, python programmer is adopted and packaged STL module is imported. The specific periodicity during one observation ${n}_{p}$ is set to 365 and percentage of data used in LOESS regression is set to 0.6. Other pertinent parameters are set by the recommended settings.

2.2.   Dendritic neuron model

Dendritic neuron model (DNM) is the simulation of the real neuron, which contains four connecting layers. The first and the last layer utilizes a sigmoid function as their processing tools. The second layer has a multiplication unit for received inputs. The third layer is an addition unit for inputs of the second layer. The four layers are denoted as the synaptic layer, the dendrite layer, the membrane layer and the soma layer, respectively in order. Figure 1 shows the DNM's organizational structure. The model is explained in detail below:

2.2.1.   Synaptic layer

A synapse is the contact between neurons or inside the neuron itself. The original signal transfers from a presynaptic neuron to another one. By implementing the ionotropy, we determine whether the synapse is excitatory or inhibitory according to changes in the potential of postsynaptic. The synaptic layer serves to connects the $i$ th ($i = 1, 2, . . ., N$) synaptic input to the $j$ th ($j = 1, 2, . . ., N$) synaptic layer, shown as:

where ${x}_{i}\in [0, 1]$ denotes the input signals. ${Y}_{ij}$ represents the output of the $j$ th synaptic layer. And $k$ represents a constant parameter. The weight ${\omega }_{ij}$ and threshold ${\theta }_{ij}$ are connection parameters. Here four types of connection samples can be listed as follows:

• Case 1 (Constant 0 connection): under the condition of ${\omega }_{ij} < 0 < {\theta }_{ij}$ or $0 < {\omega }_{ij} < {\theta }_{ij}$, the input ${x}_{i}$ changes from 0 to 1, whereas the output result ${Y}_{ij}$ is roughly 0.

• Case 2 (Constant 1 connection): under the condition of ${\theta }_{ij} < {\omega }_{ij} < 0$ or ${\theta }_{ij} < 0 < {\omega }_{ij}$, t the input ${x}_{i}$ changes from 0 to 1, whereas the output result ${Y}_{ij}$ is roughly 1.

• Case 3 (Excitatory connection): under the condition of $0 < {\theta }_{ij} < {\omega }_{ij}$, while the input ${x}_{i}$ transforms from 0 to 1, input ${x}_{i}$ and the output ${Y}_{ij}$ are positively correlated.

• Case 4 (Inhibitory connection): under the condition of ${\omega }_{ij} < {\theta }_{ij} < 0$, the output ${Y}_{ij}$ is inversely proportional to the input, while the input ${x}_{i}$ transforms from 0 to 1.

As ${x}_{i}$ is normalized before entering the model, only the conforming part is worth paying attention to. Figure 2 displays four cases above in detail.

2.2.2.   Dendrite layer

In this layer, a multiplicative function is implemented on the outputs from the synaptic layer. Given the non-linearity of synapses transforming from 0 to 1, a multiplicative operation is introduced. This is equal to the logic AND operation, denoted in Figure 1 by the symbol ∏. Thus the output function for the $j$ th dendrite is:

2.2.3.   Membrane layer

Similar to the second layer, this layer implements an addition function of the outputs on each dendrite. This function makes it equal to the logic OR operation under normalization between 0 and 1. Then, the summed result is sent to the fourth layer. The function of the membrane layer is:

2.2.4.   Soma layer

At last, the soma layer utilizes another sigmoid function for the output $V$ of the membrane layer. Once $V$ exceeds its threshold, the neuron fires. Thus according to the following formula, the final result of the entire model is as follows:

2.3.   Combined model

The proposed model used in this paper is the combination of STL and DNM with their benefits. It makes the best adjustment for better forecasting results. Given the dataset, we implement the STL on it at first. Then, decomposed components are respectively processed. The seasonal element ${S}_{k}$ remains unchanged. ${T}_{k}$ is disposed using the least squares method (LSM). For the third part, the residual element ${R}_{k}$ is regarded as the input of DNM (*), where the sign* means there is a simplification of the model. At last, the three elements are added after being processed. Figure 3 is a clear illustration of the STLDNM (*).

As the seasonal element ${S}_{k}$ is an objective reflation of the original series, it tends to keep unchanged in this paper. As for ${T}_{k}$, a tricube continuous polynomial function is used to fit. Eq (9) describes the function:

where values of $p, q, u, v$ are obtained using LSM. Hence, the first two elements of the decomposed results are processed in an easy way.

2.4.   Improved DNM

According to our previous studies, DNM is broadly applied due to its advantage of high efficiency and theoretical explanation. For better prediction results, here we utilize a simplified DNM meant for time series according to our previous study ^[43]. Figure 4 is the structure of DNM*. In particular, the summation at the middle is omitted and becomes a line with a weight of one. Additionally, the sigmoid function reduces to a linear expression. The other two layers remain unchanged. The details are as follows:

Hence, the structure of proposed DNM* is greatly simplified. The performance of either fitting or calculation is significantly improved. By implementing a model of a single line, the DNM* performs a better effect for time series prediction.

As for time series, a phase space reconstruction (PSR) ^[44] is introduced to improve the experimenting results. It reconstructs the sequence into higher dimension. In this study, the embedding dimension and time delay are set to be 2 and 1 respectively. Thus, two adjacent values in the time series lay the foundation of later prediction. Generally, we can describe the input matrix as:

The corresponding target vector $T$ is

where $M$ represents the length of the series, and each dataset is divided into two parts: training part and testing part. The former possesses 70% and the latter is 30%.

2.5.   Back-propagation learning

When dealing with problems about time series, it is most suitable to take back-propagation as a choice during the process of supervised learning. BP is a method using gradient descent learning to update and adjust the parameters of the neural network, whose purpose is to reduce the error at the terminal. We define the error by Eq (16):

where the ${T}_{p}$ is the teaching signal, while ${O}_{p}$ is the real output of the model. In BP algorithm, we decrease the error by correcting the values of ${w}_{ij}$ and ${\theta }_{ij}$ of the DNM model, their variations are shown as:

The parameters of ${w}_{ij}$ and ${\theta }_{ij}$ are updated as follows:

where $\eta$ represents the learning rate of the BP, and $k$ represents the learning iteration. $\eta$ is set to be 0.1 in this paper. Then, the partial differentials of ${E}_{p}$ with respect to ${w}_{ij}$ and ${\theta }_{ij}$ are obtained using the chain rule:

Repeating the application of the chain rule, a list of partial differentials are displayed below:

The STL decomposing method is meant to extract the seasonal and trend elements to maintain the main and unique features of the original series, and the whole disposing procedures are implemented by the almost best appropriate measures. The three subseries are comprehensively considered as the global reflection of the original data, with no part neglected. Besides, the method has also robust properties due to its relatively stable parameters and computing steps. Therefore, the STL is an excellent choice to operate weather-related time series forecasting.

3.   Experiments

3.1.   Datasets selected

The data sets of this paper are the yearly dew point temperatures of four cities from 2010 to 2015 as shown in Table 1. Cities chosen are four different regions from mainland. Hence, the predicting result of STLDNM* can be fully experimented. In view of the same length of these sequences, original series are filtered before implementing STL.

In order to decide whether the original series can be short-termly forecasted or not, a maximum Lyapunov exponent (MLE) is computed before experiments ^[45]. This metric is obtained by the Wolf method. Given ${t}_{0}$ (the beginning time) and ${y}_{t0}$ (the reconstructed first phase point), we can get the minimum length between ${y}_{t0}$ and its nearest neighbor. This distance is called ${L}_{0}$. At a later time ${t}_{1}$, the initial distance becomes ${L}_{0}^{\text{'}} = ‖{y}_{t1}-{y}_{t0}‖ > ϵ$, where $ϵ$ is a positive threshold value. The ${L}_{0}^{\text{'}}$ will be replaced when another phase point ${y}_{t1}^{1}$ with ${L}^{1} = ‖{y}_{t1}-{y}_{t1}^{1}‖ < {L}_{0}^{\text{'}}$ appears. This calculation process is carried out until ${y}_{t}$ reaches ${y}_{N}$. As a result, we can obtain MLE as:

where $m$ is the generation. Table 1 also gives MLEs of four cities. As the fourth column shows, all these series are characterized by chaotic features as their MLEs are positive and less than one.

3.2.   Previous filtering

As the four chosen datasets are long, the original series are initially shortened. We firstly choose values on five or eight o'clock as they rarely involve invalidate data like "NaN". Then, from the initial point, each five values are averaged. Finally, a new series is constructed from each dataset.

Moreover, to reduce time cost, a normalization method is introduced to make inputs range from 0 to 1. We choose the simplest formula below:

where $MIN$ and $MAX$ manifest the minimal and maximal values of a series $x\left(t\right)$. Meanwhile, the teaching signal is going to be normalized under the same parameters. At last, outputs are disposed with similar rules.

3.3.   Parameter selection

Four user defined parameters affect predicting results of STLDNM*, three of which are parameters of the DNM model. Another important parameter is the ${n}_{p}$ of the STL, especially for series with relatively stable periods. The ${n}_{p}$ is decided by observing the sequence to be put into the STL. We make efforts to give a value closest to original period by using the scale line in the MATLAB toolbox. Besides, three structure parameters of DNM* are listed as 27 teams in Table 2. In order to provide a relatively comprehensive impression of the STLDNM*, three metrics mentioned above are included. Then, we execute a Friedman test ^[46] on the resulting values, thus obtaining the ranks of each dataset on each model as shown in Table 3. And Table 4 shows the final rank of each model, and the conclusion is that the Team 18 (${k}_{s} = 1, {\theta }_{s} = 1, \eta = 0.1$) performs the best under this evaluation.

4.   Results

Eight models are compared with STLDNM* in the experiment, including STLDNM*, STLDNM, DNM*, DNM, Elman, MLP, SVR, DeepAR and LSTM. As different conditions may provide different results, this paper chooses optimal parameters of each model shown in Table 5. Tables 6–8 shows the $MSE$, $MAPE$ and $MAE$ of each method on each dataset. These three evaluation metrics are simply described as follows:

To measure how well the proposed model and others perform, we utilize three criterions to assess each model's performances. These metrics are computed between targeted vectors and outputs as the following formulas:

• $MSE$ : Mean square error

• $MAPE$ : Mean absolute percentage error

• $MAE$ : Mean absolute error

Later we employ the Friedman test on all the eight models to display which ranks the highest. Table 9 gives the rank of each model under the standard of $MSE$, $MAPE$ and $MAE$. It concludes that STLDNM* performs the best. According to previous researches ^[47,48], to compensate the drawback that Friedman test can only judge whether there are differences among these models, Wilcoxon test is used for comparison between STLDNM* and other models. As can be seen from Table 10, STLDNM* outperforms its peers, except for MLP and SVM on MAPE. Besides, Table 11 shows the average time cost of each model. It can be seen that our model is the most efficient in terms of time cost. Tables 9–11 show that DNM* is better than other machine learning models, both in efficiency and precision.

The predicting results of four cities using several algorithms are shown in the Figures 5–8, respectively. Typically, we extract the Shenyang's averages of each algorithm and plot another graph as shown in Figure 9. Thus, a group of curves surrounding $"y = 0"$ line is generated. Although the STLDNM* has the least advantage on the SY dataset, it remains more stable than most algorithms except MLP. For the other three series, it maintains the minimum deviation value.

Finally, to verify the stability of each model, the convergence curve of each model is shown in Figure 10. Overall, DNM and STLDNM are worse than other models and converge rapidly after about 10 generations. While the improved STLDNM* and DNM* have obtained excellent performance, obviously superior to MLP and Elman. It is further verified that the improved DNM is suitable and efficient for time series.

Although DNM is less popular than SVR and DeepAR for weather forecasting, it stands out for its simplicity. The experimental results show that the combination of preprocessing technology, neural network, and classical learning algorithm can achieve excellent results.

5.   Conclusions

It is significantly important that a precise prediction of weather data reminds people to realize anormal phenomenon and make instant preparations, aiming to reduce loss to the least. However, forecasting consequences with high precision and efficiency is difficult due to various factors, such as variability and volatility.

In this paper, we present a new method combining decomposition method STL and prediction model DNM for dew point series. The STLDNM* model is applied to meteorology and the dew point of different cities in China is predicted successfully. Regarding STL as the preprocessing method, LSM and DNM* as the forecast method, BP as the learning algorithm, this combination has achieved excellent accuracy in the application of weather forecast.

DNM*, as a variation of DNM, is a single-input-single-output model that is more suitable to one-dimensional time series. The results show that the proposed model is superior to MLP, Elman, SVR, DeepAR in terms of time cost, convergence rate and prediction accuracy. And although the LSTM performs well in financial field, it does not fit weather data. In addition, Friedman test and Wilcoxon test were used to verify the above conclusions in terms of MSE, MAPE and MAE. Moreover, STLDNM* and DNM* are significantly better than DNM in stability. As a consequence, STLDNM* has great potential and prospects for development in more areas, whose performance in other types of series will be further discussed in the future.

Acknowledgments

This research was supported by the National Natural Science Foundation of China (NSFC) under Grant No. 12105120.

Conflict of interest

The authors declare there is no conflict of interest.