Multi-level stacking of LSTM recurrent models for predicting stock-market indices

1.   Introduction

Forecasting the next day's stock-market index (SMI) prices is crucial to investor success, and the research being applied to this problem is extensive. In financial fields, technical analyses exploit historical price data to make decisions, assuming that previous and current behaviours impact a stock's value evolution (Cervelló-Royo et al., 2015; Vargas et al., 2017). However, obtaining consistent forecasting accuracy is extremely challenging, owing to the non-linearity and uncertainty of future value. Thus, good prediction requires the elaboration of reliable methods to find stochastic relationships between present and future stock values using extant information.

Recently, machine-learning (ML) methods have shown great results in making statistically driven non-linear predictions based on the availability of copious historical data (Vargas et al., 2017; Li et al., 2021; Banerjee et al. 2021; Hassan et al., 2021). Moreover, ML models consistently outperform regression methods, especially with forecasting (Hassan et al., 2021; Wang et al., 2021; Weng et al., 2018). However, as implied, ML requires large amounts of historical data so that the stochastic relationships between changing variables can be "learned" (Jiao et al., 2017; Bahrammirzaee et al., 2010). As we will see, the availability of such dada is rare.

Numerous models have been used to predict SMI closing prices, including long short-term memories (LSTMs), gated recurrent units (GRUs), support vector regressors (SVRs), multi-layer perceptrons (MLPs), k-nearest-neighbour (KNN) algorithms (Li et al., 2021; Banerjee et al., 2021; Hassan et al., 2021; Wang et al., 2021; Weng et al., 2018; Bahrammirzaee et al., 2010; Gao et al., 2017; Heaton et al., 2017) and several others. For short-term closing-price prediction (Banerjee et al., 2020), the authors applied MLP models, LSTM, gated recurrent units (GRU), and bidirectional long short-term memory (BLSTM) to predict stock prices over short terms. They also compared the overall performance of their approaches against classic artificial neural networks (ANNs). They have addressed the problem of predicting the equity "close prices" for a short period. The authors reported that the prediction's average error, shown by MLP, LSTM, GRU, BLSTM, and BGRU models, varied between 9 and 10%. In (Weng et al., 2018), a financial expert system was designed, comprising a knowledge base module that captures historical stock prices and technical indicators, calculates sentiment scores of published news articles and mines trend data from Google searches. This information is transferred to an ML module to make informed predictions. The authors of (Jiao et al., 2017) compared the performance of four contemporary classification algorithms (i.e., random forest, gradient boosted trees, ANNs and logistic regression models) in predicting the movements of 463 Standard & Poor's (S & P) 500 stocks. They demonstrated that data quality and accurate feature selection are crucial to the success of SMI prediction and that immediate past information contains the most pertinent signals. In (Bahrammirzaee et al., 2010), ML methods outperformed statistical regression models when dealing with numerous financial problems, including portfolio management, financial prediction and credit evaluation. In (Gao et al., 2017), the authors proposed a recurrent neural network (RNN)-based LSTM model to predict stock closing prices. It applies optimisation techniques to compute the adaptive learning rates for each parameter in the neural models. The authors used as input all the available parameters in each data record of the training dataset, such as opening price (OP), low price (LO), high price (HI), closing price (CL), adjusted price (AD) and volume-of-trading (VO) metrics, showing superior performance over other ML methods. In (Heaton et al., 2017), the authors applied deep-learning techniques to forecast the directional daily movements of the S & P 500 index using technical indicators from financial news. The method was trained to recognise complex patterns and relationships among variables using RNNs and convolutional neural networks (CNNs) from the natural language processing field. The experiments reported in (Heaton et al., 2017) showed that a CNN model outperformed an RNN in detecting text semantics. However, RNNs are still better at recognising contextual information and modelling temporal features of SMI prediction.

Recently, other researchers have forwarded hybridised models consisting of two or more ML models (Chen et al., 2017; Lu et al., 2013). In (Chen et al., 2017), the authors proposed a hybridised framework consisting of a feature-weighted KNN model with a feature-weighted SVM to predict index values. Their method assigns weights to each feature based on its classification importance, computed from the information value gained. It then feeds historical data to the feature-weighted KNN to estimate prices. Their model was validated and tested on two Chinese. SMIs at different temporal horizons and resolutions. In (Lu et al., 2013), the authors suggested a hybrid model consisting of a nonlinear independent component analysis (NLICA) module, an SVR and a particle swarm optimisation (PSO) algorithm. In the first step, NLICA is applied to handle the nonlinearity expected in SMI data records so that the collection of features can be facilitated. Then, these features are used as inputs to train the SVR model, whose parameters are optimised by the PSO algorithm. The authors evaluated the performance of their approach in predicting the closing prices of the Taiwan Capitalisation Weighted Stock Index and the Shanghai Stock Exchange Composite Index. To reduce the dimensionality of the input vectors used for prediction, the authors of (Zhong et al., 2017) applied principal component analysis (PCA) and kernel-based PCA (KPCA) to restructure the original data. The authors then used an ANN model on the pre-processed data records to predict the daily direction of future market returns.

Practically, stability and consistency are two key challenges faced by researchers when building reliable prediction models, especially when dealing with time-series data (Li et al., 2021; Al-Hajj et al., 2018). A particular challenge for ML systems is the limited availability of datasets for training and validation. When lacking sufficient data, weak learners are often substituted, but they rarely produce operationally sound predictions. One method of remedying this weakness is to combine several basic weak-learner models so that their outcomes can be consolidated to produce a higher-resolution decision. For this purpose, stacking learning techniques have been developed. They consist of an ML meta-model that combines the outcomes of an ensemble of individual base learners (Al-Hajj et al., 2019; Sagi et al., 2018). This "ensemble learning" technique aggregates complementary outcomes from a set of weak learners to produce a final decision tuned to the input signal. Recently, several ensemble-learning approaches have been proposed to improve both short- and long-term time-series data predictions (Sagi et al., 2018; Hochreiter et al., 1997; Ren et al., 2015; Briscoe et al., 2011). Notably, not all of these methods use ML stacking for aggregation.

In this paper, we introduce an ensemble method that uses a multiple-level stacking technique of recurrent models to predict SMI closing prices using historical time-series data. The basic stacked models are LSTM variants (Hochreiter et al., 1997) and their predictions are combined using two MLP layers, also stacked. The base predictors are designed to predict closing prices one day ahead, using the index data from $n$ prior days. In this paper, we demonstrate the advantages of dividing index information across several base models and feeding their decisions to a post-prediction stage. Our experiments illustrate the superiority of the suggested strategy against global systems that receive and process all SMI information using a single model. We compare the performance of our suggested approach to several statistical and ML models (e.g., a moving average (MA) predictor, a regression SVM and more). These models were designed and implemented by another research group on a subset of S & P 500¹ index data from 2000 to 2016 (Gao et al., 2017). The suggested method is also validated and tested against the Nasdaq² Composite index. For each index, the multi-level stacking approach shows good improvement with respect to the individual predictors of the ensemble.

1 Yahoo Finance S & P 500 (https://finance.yahoo.com/quote/%5EGSPC/history?p=%5EGSPC)

2 Yahoo Finance NASDAQ Composite (https://finance.yahoo.com/quote/%5EIXIC?p=%5EIXIC)

The remainder of this paper is organised as follows. In Section 2, we introduce the ensemble methods and describe their advantages. In Section 3, we describe the MLP and LSTM methods. In Section 4, we explain our suggested multi-level stacking approach and the components thereof. The benchmark dataset and the preparation of the data records are illustrated in Section 5. In Section 6, we describe our experimental settings and their results and analyses. Finally, we conclude our study and draw future perspectives in Section 7.

2.   Ensemble method

An ML ensemble model combines multiple base learners trained to solve specific sub-problems of classification or regression. The outputs of the base models are then aggregated to produce final, consolidated prediction and classification responses. This approach decreases output variance and enhances overall performance (Al-Hajj et al., 2019; Sagi et al., 2018; Ren et al., 2016). Generally, constructing an ensemble technique requires a good understanding of the complementarity of both the aggregated metrics and the base-learner strengths. Figure 1 depicts a parallel aggregation ensemble technique.

The consolidation of outputs can use an averaging technique for regression or a voting technique for classification (Dietterich et al., 2000; Liu et al., 2021), depending on the problem being addressed. The choice of base models depends on many factors, such as the dimensionality of the problem space, the distribution hypothesis and the availability of training data. The base models may therefore receive similar or dissimilar features from a subset of the training data. Thus, they may incur high biases and variances individually because of their low or high degrees of model freedom. Notably, ensemble techniques have been found to decrease both bias and variance. The ensemble methods show good enhancement when the base models have considerable complementary information (Dietterich et al., 2000; Liu et al., 2021).

A stacking technique trains a machine learning meta-model to smartly generate a final decision based on those of the base models. The base models in a staking ensemble may receive similar or different features from a subset of training data. A stacking algorithm includes the following steps:

● Split the training data into two subsets: one for training the base models and another to validate their outputs and train the meta-model that combines the base models' outputs.

● Use the data of the first subset to train n selected individual base models.

● Each base model makes predictions for each observation in the validation subset.

● These predictions are used to train the meta-model for combination.

Figure 2 depicts the general structure of the stacking ensemble technique.

A multi-level stacking model involves multiple subsequent layers of machine learning meta-models of combination. The individual base models return their responses to their respective input data vectors. The first stacking layer produces a new dataset containing the individual base-model outputs, and the second layer takes those outputs as input to a meta-model that outputs the final prediction(s).

3.   MLP and LSTM

MLPs consist of sequentially interconnected layers of neurons (Haykin, 2011), which are individual units of computation (i.e., perceptrons). The output of a perceptron is determined by its activation function and the value of the input vector. A basic MLP uses three consecutive layers of perceptrons. The input layer collects input signals, the hidden layer receives the output of the first layer, and the output layer produces the final response. A feed-forward MLP consists of an arbitrary number of hidden layers that support complex calculations and improve the approximation of continuous functions. Generally, a set of labelled input-target pairs are used for training, as parameters are adjusted using back-propagation gradient descent optimisation (Haykin, 2011).

LSTMs are gated models designed to handle the temporal contexts of time-series data (Hochreiter et al., 1997). Their main components are depicted in Figure 3. The LSTM's input gate receives a combination of input signals and a hidden state. It applies a purposeful arrangement of internal gates (e.g., forget, output and arithmetic) that maintains (remembers) only the most important (discriminant) information to prevent the vanishing and exploding gradient problems that arise when back-propagation is used with gradient descent optimisation (Hochreiter et al., 1997; Haykin, 2011).

An LSTM cell receives a signal vector ${X}_{t}$ at instant $t$. The hidden state, ${h}_{t-1}$, represents the cell's output at instant $t-1$, and the cell state, ${c}_{t-1}$, output at $t-1$. The current hidden state, ${h}_{t}$, represents the LSTM's output at $t$, which becomes the next input alongside the cell state, ${c}_{t}$. The LSTM unit includes five sets of weights, defined as follows.

Variable D represents the dimensionality of the input vector, K represents the number of classes defined for a classification problem, and H represents the number of LSTM units in a layer. W(·) is the weight matrix, and b(·) denotes a bias vector that impacts the transformation of a particular gate's state. For a given input vector, its forward calculation is defined as follows.

The output of the forget gate is computed with Equation (6), the output of the input gate is computed with Equation (7), and the cell state is determined using Equations (8) and (9) in turn. Lastly, the output of the LSTM cell is found using Equations (10) and (11).

The outputs of the forget, input and output gates and cell memory are denoted with ${\boldsymbol{f}}_{\boldsymbol{t}}$, ${\boldsymbol{i}}_{\boldsymbol{t}}$, ${\boldsymbol{o}}_{\boldsymbol{t}}$ and, ${{č}}_{\boldsymbol{t}}$, respectively. The sigmoid function, $\boldsymbol{ \varphi }$, is defined by Equation (12) as the activation function, which denotes the element-wise product of two vectors to determine the cell response. The product of the LSTM unit includes its output value, ${h}_{t}$(), and its state memory, ${c}_{t}$, which are applied iteratively to the LSTM unit at the next time step. More details about LSTMs and other gated models can be found in (Hochreiter et al., 1997).

4.   Multi-level stacking model

The core multi-level stacking model uses three consecutive layers. The first consists of M LSTM units, each receiving successive values related to SMI closing prices of N previous days, including day $\boldsymbol{d}$ (current), to predict tomorrow's (d+1) SMI closing. The second layer contains S MLP meta-models, each collecting the outputs of the LSTM base models. The third layer contains a single MLP meta-model that combines the outputs of the meta-models. Note the clear distinction of our applied model in Figure 4 from the one previously described, in that ours applies an additional meta-model layer of multiple MLPs.

In our case study, we validate our approach on an S & P 500 data subset. Independent variables are analysed to determine the best inputs to the base prediction models and their LSTM configurations. All LSTMs use the same activation function with the same number of hidden layers. The back-propagation-through-time (BPTT) algorithm is applied for deep learning (Hochreiter et al., 1997), and all MLPs in the sub-meta-model layer use the same number of internal hidden layers, but with different numbers of perceptrons and activation functions. The final meta-model layer consists of one hidden layer and one perceptron to output the final prediction.

In this work, we implement and validate our stacked base-learner LSTM and MLP model using the Scikit-Learn toolkit (Pedregosa et al., 2011).

5.   Benchmark datasets

5.1.   SMIs

Training and validation experiments were conducted on a subset of the S & P 500 index extracted from Yahoo Finance, containing 505 common stocks maintained by 500 publicly traded companies covering about 80% of the U.S. equity market per capitalisation (Jiao et al., 2017). Daily OP, LO, HI, CL, AD and VO values between 3 January 2000 and 10 November 2016 (4,243 trading days), were analysed. We validated the advantages of our approach on a subset of the Nasdaq Composite index covering the period between 1 January 2004 and 31 December 2019 using the same horizon of one day ahead.

5.2.   Input sequence preparation

A sliding-window method (Figure 5) was used to pre-process the sequential input vectors for the LSTM base models. Data of $k$ days prior to day ${d}_{n}$ were used to predict SMI closing prices on day ${d}_{n+1}$. Consequently, the input sequence held records representing days ${d}_{n}$, ${d}_{n-1}$, ${d}_{n-2}$, …, ${d}_{n-k}.$ The output target was the closing price on day ${d}_{n+1}$.

Each LSTM in the first layer received one or a combination of two of the six parameter values listed above. For instance, one LSTM tooks the closing and opening prices of the previous $k$ days, while another took closing and high prices. In this work, we inspected the impact of each combination on LSTM performance, and the results are provided in Section 6.2.

6.   Experiments and results

6.1.   Statistical evaluation metrics

To evaluate our model performance, we applied root mean-square error (RMSE), mean bias error (MBE), mean absolute error (MAE), mean absolute percentage error (MAPE) and average MAPE (AMAPE) (Heaton et al., 2017; Al-Hajj et al., 2019). RMSE is commonly used to assess regression performance and is sensitive to large approximation errors. Therefore, it is useful when large estimation faults lead to higher costs than minor faults. The RMSE was computed using the following formula:

where ${\mathrm{M}}_{\mathrm{p}, \mathrm{I}}$ denotes the estimated value, and ${M}_{i}$ is the exact value.

MAE is the average of prediction errors, not counting their directions, and is evaluated as follows:

MBE computes the average bias of predictions, tallying under- and overestimations and allowing them to cancel one another. It is defined as follows:

MAPE evaluates the relative error between the predicted and measured values, as depicted in Equation (16):

AMAPE calculates the average MAPE between the estimated and measured values. It is computed by the following equation:

6.2.   Experiments

In our experiments, we divided the examined dataset into three parts, as follows:

1. One training subset for the base predictors,

2. One validation subset to train the array of meta-models in the second layer and the one in the output layer,

3. One testing subset to evaluate the LSTMs and MLPs. This latest subset was not seen by any of the individual or meta-models during the training and validation phases.

The ensemble of base STMs had similar structures and used the same activation function. Furthermore, the same BPTT algorithm and its parameters were used to train the LSTM-based predictors (Hochreiter et al., 1997). Each predictor received a sequence of SMI closing prices with a probably of one of the other available parameters' values of the stock for a day d and the $k$ preceding days (d-1, d-2, ……, d-k). The resultant LSTM models took the following inputs for the current day d and the k preceding days.

● LSTM-1: takes only the stock's closing prices (CL) for the current day d and the $k$ preceding days.

● LSTM-2: takes the stock's closing prices coupled with the opening prices (CL, OP) of the same stock for the current day d and the $k$ preceding days.

● LSTM-3: takes the stock's closing prices coupled with the highest prices (CL, HI) of the same stock for the current day d and the $k$ preceding days.

● LSTM-4: takes the stock's closing prices coupled with the lowest prices (CL, LO) of the same stock for the current day d and the $k$ preceding days.

This combination of parameters was found to be the best at providing good overall prediction performance. Moreover, we trained another LSTM (i.e., LSTM-5) to receive the sequence of all parameters to simulate a global approach. It was trained using the same BPTT algorithm. The objective was to compare the ensemble approach of LSTMs 1–4 with the global approach of LSTM-5. Table 1 provides an assessment of the ensemble approach vs. the global approach.

The second layer of the global stacking model contained three MLP models for stacking. Each included four inputs in its first layer and one hidden layer. The MLPs were varied by one or more of the following parameters: (1) the number of hidden neurons in the hidden layer, (2) the activation functions in the hidden and output layers, and (3) the learning rate type (i.e., constant or adaptive) (Pedregosa et al., 2011).

In some of the MLP models in this layer, we used the adaptive moment estimation optimizer (Adam). The resultant configuration was as follows,

● MLP-1: Nine hidden neurons, identity activation and constant learning rate with Adam optimisation (Pedregosa et al., 2011)

● MLP-2: 10 hidden neurons, identity activation and constant learning rate with Adam optimisation

● MLP-3: 10 hidden neurons, identity activation and adaptive learning with stochastic gradient descent optimisation (Pedregosa et al., 2011).

The meta-model in the final layer consisted of one MLP with three input units in the first layer. Its hidden layer included six hidden perceptrons using the identity activation function and a constant learning rate with Adam optimisation. The output layer contained a single output to represent the final prediction. The parameters were selected using five-fold cross-validation (Bergstra et al., 2012).

The overall results showed that the stacking-model results were superior to any single LSTM or MLP from internal phases. The MLP with the highest performance was MLP-3. Table 1 shows that MLP-3 decreased the average RMSEs of the base predictors (total 18.9127) by around 12.57%, according to Equation (17):

The third and fourth columns of Table 1 show good MAE and MBE scores for MLP-3 over the best-performing LSTM-3. The LSTM outputs provided complementary information, as their over- and underestimations cancelled out. This phenomenon is reflected by the alternating signs of the four main LSTM MBEs. The meta-model in the final layer outperformed all other MLPs and LSTMs. The scores in Table 1 show that MLP-4 decreased the average RMSE scores related to individual models (total 18.9127) by around 14.89%, as shown in Equation (18):

The overall multi-stacking model provided significantly better MBE scores than those of the constituent parts. Negative MBEs in the LSTMs reflect underestimated SMI closing prices. However, the relatively small MBEs in the MLPs indicate that they have good short-term performance. The results in Table 1 also illustrate the advantages of our stacked approach against LSTM-5, which received all SMI parameters. The stacked version outperformed LSTM-5 in nearly all evaluation metrics.

Figure 6 illustrates the LSTM daily predictions of SMI closing prices of the next day's S & P 500 values. The best was LSTM-3. For clarity, we show predictions for two consecutive months of September and October 2016.

Figure 7 presents the monthly average prediction errors of the next-day' SMI closing prices, as provided by the best LSTM model (LSTM-3), the best MLPs meta-model in the first stacking layer, namely the MLP-3, and the meta-model in the third layer. For clarity, we show the monthly average prediction errors for April-September 2016, inclusive.

Figure 7 shows that the proposed multi-level stacking model outperformed the best internal models in terms of the least prediction error deviation, which varied between −2 and 2. The related chart shows more stability than the other two charts.

Table 2 illustrates the advantages of our approach when predicting Nasdaq Composite SMI closing prices. The multi-level stacking model outperformed the best internal models on nearly all evaluation metrics.

Finally, to validate the utility of our multi-stacking approach, we compared its scores to those obtained by other statistical and ML models applied to the same S & P 500 dataset, covering the same period (Gao et al., 2017). These models included SVM, MA, exponential MA and stand-alone LSTM predictors. Table 3 compares the MAE, RMSE, MAPE and AMAPE scores of the comparison models (Gao et al., 2017) to those of LSTM-3 and our proposed multi-level stacking model.

In summary, for the considered prediction problem, we found that it was most efficient to divide the feature values from the SMI dataset into discriminant subsets and to input them in a complementary fashion into individual base models ensembled to provide output to consolidating meta-model layers. Doing so well minimizes hyperparameter adjustments during training and renders available datasets sufficient for training, validating, and testing the overall recurrent model.

The scores in the last row of Table 3 show that the proposed approach has well exploited the complementary outputs of base and sub meta-models to outperform all the described models in (Gao et al., 2017) for almost all the reported metrics.

7.   Conclusions

In this work, we addressed the problem of short-term predicting of SMI closing prices for market investment decisions. We introduced a novel framework that combines the capabilities of LSTM models in predicting time-series data results using MLP models to stack their outputs. The base predictors were designed to predict SMI closing prices using their information from the previous $n$ days, and the final decisions of the individual LSTM models were combined through two MLP stacking layers to produce a single prediction output.

We demonstrated the advantages of stacking on prediction accuracy, as the overall model outperformed all internal predictors by combining their complementary decisions. Furthermore, the suggested strategy showed good utility across multiple SMIs and outperformed other global systems specifically designed to ingest the same data to make the same predictions.

Applying the suggested method to new financial SMI datasets with different time horizons and resolutions is a problem for future study. Another opportunity would be to apply the same stacking technique using new gated models. Furthermore, the incorporation of sentiment analysis with financial SMI prediction is a major aspect of our future efforts.