Decision making in railway interlocking systems based on calculating the remainder of dividing a polynomial by a set of polynomials

1.   Introduction

In December 2019, a new, human-infecting SARS-Coronavirus-2, Coronavirus was recognized in Wuhan, China ^[1,2]. It was conveyed that the virus transferred between humans by droplets or close contact ^[3,4,5]. As of March 2020, the new SARS-Coronavirus-2 has above than 98,000 cases through 88 countries beyond China ^[6].

The virus is a pathogenic human coronavirus under the Beta coronavirus genus. The other two pathogenic species, including Severe Acute Respiratory Syndrome Coronavirus (SARS-Coronavirus) and the Middle East Respiratory Syndrome Coronavirus (MERS-Coronavirus) were outbreaks in China and the Middle East in 2002 and 2012 respectively ^[1]. On January 10, the full genome sequence of this massive RNA virus (SARS-Coronavirus-2) was first uncovered in China's laboratory ^[5,7] and placed in the NCBI GENBAND. Coronaviruses have enveloped viruses that contain a positive single-stranded ribonucleic acid (RNA) virus that affects humans and animals without segmentation ^[5,8]. The genomes of the Coronaviruses are formed of base pairs ranging from 26 kilobase-pairs (kbps) to 31 kbps, with GC contents fluctuating from 43 to 32%, and human-infecting coronaviruses including MERS-Coronavirus, HCoV-OC43, SARS-Coronavirus, HCoronavirus-NL63, HCoronavirus-229E, and HCoronavirus-HKU1 ^[9].

Coronaviruses are characterized by the ability to quickly evolve and adapt to various epidemiological conditions ^[10]. Each replication cycle of Coronavirus presents new genetic mutations, and its normal developmental rate is nearly from 4 to 10 nucleotide substitutions per site/ year ^[1]. During genomic information reproduction, SARS-Coronavirus-2 evolves ^[11]. The mutation happens because of certain errors when copying RNA to a new cell. SARS-Coronavirus-2 analyses can produce false-positive results if they are not targeted especially to SARS-Coronavirus-2, as the virus is difficult to distinguish from other Coronavirus organisms because of their genetic similarity. Therefore, it is essential to use improved diagnostic tools in order to correctly classify SARS Coronavirus-2 from other Coronaviruses.

Currently, Molecular approaches are commonly used to classify pathogens, such as quantitative real-time RT-PCR and nucleic acid sequencing methods ^[12]. However, owing to their comparatively new evolution and still not entirely understood characteristics, they have an overall unsatisfactory identification rate for this particular virus. Since coronaviruses are genetic diseases, viral sequencing techniques have been employed to detect the virus depending on its genome sequence and therefore avoiding the downsides of the classical diagnostic methods.

Classification using viral sequencing methods depends on strategies of alignment such as FASTA ^[13] and Basic Local Alignment Search Tool (BLAST) ^[14] algorithms. These methods count on the assumption that DNA sequences have common features, and their order predominates among various sequences ^[15,16]. However, there are limitations to these methods, which include the need for base sequences for the detection ^[17]. In addition, because viruses have a high level of mutation and most genomes do not have proper reference sequences, next-generation sequencing (NGS) genomic samples might not be identified by BLAST ^[18]. This method has also a negative impact of neglecting part of the vital information contained in the input sequence if it cannot completely fill a DNA sequence of fixed size.

Traditional machine learning approach using genomic signal processing technique was introduced to discern between Covid-19 and other Coronaviruses ^[19,20] by using their genomic sequences reported in the NCBI GENBAND in order to increase the accuracy of disease detection in lesser time. Some characteristics were then extracted, such as Discrete Fourier transformation, Discrete Cosine transformation, and incorporated into a classifier. However, this method needs to extract pre-selected features in order to recognize or classify the viral DNA sequences.

The deep learning approach has been recently introduced as various alternatives for DNA sequence recognition, as these strategies do not require pre-selected features to recognize or classify DNA sequences. Deep learning approaches ^[21,22] have evolved rapidly since they can be employed in the processing of large-scale genetic data, especially in the field of bioinformatics. Today, the size of these datasets exceeds the size of 10 million ^[23]. Studies in this area are focused on the analysis and classification of DNA and RNA sequences.

Deep learning to use Convolution Neural Networks (CNN) ^[24,25,26] have been efficiently used for the classification of DNA sequences. However, these methods encode the RNA sequences data into an input layer of the CNN therefore, numeric values have to be assigned to the different bases. In addition, CNN learns to recognize patterns across space while RNN helps in resolving temporal data problems.

Other alternative approaches focusing on deep learning have also been explored by slicing sequences into bits of fixed lengths, from 300 ^[16] to 3000 bps ^[23]. However, these methods ignore part of the data contained in the input sequence if it does not fill a piece of fixed size.

Because of the difficulties of differentiating between the SARS-Coronavirus-2 virus and other Coronavirus organisms (SARS-Coronavirus and MERS-Coronavirus) and respiratory infections because of their genetic similarity; it was aimed to develop an advanced system to correctly classify the virus based on its genomic sequence. Apart from the classical approaches, the proposed Deep Learning Bidirectional Recurrent Neural Networks (BRNNs) approach eliminates the need for pre-selected features to identify or classify the viral DNA sequences and also works on the full DNA input sequence as a whole thereby overcoming the problem of ignoring any data in the input sequence. It has also the advantages of rapid and high accuracy of disease detection and classification.

The rest of this article is organized as follows. The contribution of this research is presented in the next section. A background overview of BRNNs with different types of cell units is introduced in Section 4. The methodology, preparation of the data, and proposed algorithms are demonstrated in Section 5. Experimental results and a discussion of the results are presented in Section 6. Finally, concluding remarks are given in Section 7.

2.   Contribution

In this study, Recurrent Neural Networks (RNN) based models with different cell units were implemented for the purpose of classification of the SARS-Coronavirus-2 virus from other Coronavirus and respiratory infections based on its DNA sequence. Apart from the classical approaches, the proposed Deep Learning Bidirectional Recurrent Neural Networks (BRNNs) approach eliminates the need for pre-selected features and also works on the full DNA input sequence as a whole thereby overcoming the problem of ignoring any data in the input sequence. A comparison between the presented deep network models based on LSTM and GRU unit cells was performed to attain the most efficient model for virus classification. Also, an investigation of the hyper-parameters that worked best with the BRNN models was presented to select the highest accuracy classification model.

3.   Background

3.1.   Recurrent neural network (RNN) overview

In recent times, the RNN model has been an extremely favored architecture ^[27], especially for sequential data. The RNN structure is demonstrated in Figure 1. Each node at a time step comprises an input from the preceding node and uses a feedback loop to continue. By employing the current input and previous hidden state, each node produces a current hidden state and output as follows:

where $x$ is the input, y is the output sequence, $h$ is the hidden vector sequence at each time Step (t). $W$ denotes the weight matrices, and $b$ represents the bias for hidden and output states. $f$ is the activation function used at the hidden layers.

3.2.   Long short-term memory (LSTM) networks

The shared downside of the classical RNN models is that as the time step increases, the network fails to draw context from time steps of previous states far behind, such a phenomenon is known as long-term dependence. Moreover, exploding and vanishing gradient problems are often found quite commonly due to the deep layers of a network and repetitive behavior of the standard RNN,

To resolve this problem, LSTM models were implemented by positioning memory cells with multiple gates in a hidden layer ^[28]. Figure 2 displays a hidden layer block with an LSTM cell unit, and the three functions of gate controllers are demonstrated as follows:

■ Forget gate $\left({f}_{t}\right)$ (should be omitted): It chooses which fragment of the long-term state ${c}_{t}$ should be ignored.

■ Input gate (${i}_{t})$: It controls which part of c should be added to the long-term state${ c}_{t}$.

■ Output gate (${g}_{t})$: It defines which fragment of c should be fed to ${h}_{t}$and${o}_{t}$.

The following equations demonstrate the cell's long-term and short-term states and each layer's output in time steps.

where ${W}_{xf}, {W}_{x}i, {W}_{x}o, {W}_{x}g$ represent the weight matrices for the corresponding associated input vector and ${W}_{hf}, {W}_{h}i, {W}_{h}o, {W}_{h}g$ characterize the weight matrices of the short-term state of the previous time step, and ${b}_{f}, {b}_{i}, {b}_{o}, {b}_{g}$ are bias.

3.3.   Gated recurrent unit (GRU) network

GRUs presented in 2014 resemble LSTMs but they have fewer parameters ^[11]. Figure 3 shows the block of a hidden layer with a GRU cell unit. They also have gated units as LSTMs that regulate the flow of context within the device without having separate memory cells. GRU doesn't have an output barrier, unlike LSTM, thereby revealing its full content. Mathematically, GRU formulations are expressed by the following equations:

where ${W}_{xr}, {W}_{x}z, {W}_{x\tilde{o}}$ indicate the weight matrices for the consistently associated input vector and ${W}_{or}, {W}_{oz}, {W}_{o}$ characterize the weight matrices of the preceding time step, ${b}_{r}, {b}_{z}, b$ are biasing.

3.4.   Bidirectional recurrent neural network

Bidirectional Recurrent Neural Networks (BNN) is a version of the RNN that Schuster and Paliwal suggested in 1997 ^[17]. Standard RNN's use the previous context information, but Bidirectional RNNs explore also the future context. BRNN is a better alternative than plain RNN since the input sequence of bidirectional RNN processes has two RNN paths, one responsible for the forward states and the other for the backward states. Hence, Bidirectional RNN effectively increases the amount of information available to the network, improving the content available to the algorithm.

4.   Methodology

4.1.   Dataset

4.1.1.   Dataset selection

The DNA sequence information was derived from the Novel Coronavirus Resource 2019 repository (2019nCoVR) ^[29]. All the accessible sequences with the Query Nucleotide Completeness = "Complete", Sequence Quality = "High", Host = "Homo Sapiens", DNA Sequence lengths = (29,800–29,900 bps) nucleotides for each epidemic, and Data Source = "GenBank" for a total of 2500 complete genome samples. 500 random complete genomes for each epidemic (COVID-19, MERSA–CORONAVIRUS-2, Common cold, ARI, and SARS) were chosen.

4.1.2.   Data cleaning and preprocessing

To preprocess the data, all repeated sequences and sequences having a length that is out of the identified range (29,800–29,900 bps) were removed. Then the data was labelled and prearranged, as shown in Table 1. The data were grouped into 5 classes based on the disease type as follows:

■ First Class "COVID-19": Includes the genome virus [SARS-CoV-2]

■ Second Class "MERSA–CORONAVIRUS-2": Includes the genome virus [MERS-CoV]

■ Third Class "Common Cold": Includes the genome viruses [HCoV-OC43/HCoV-229E] which are placed in one category, as they are coronaviruses accountable for normal cold ^[30], and [HCoV-4408] was also added because it is a Beta Coronavirus as HCoV-OC43.

■ Fourth Class "ARI": Includes the genome viruses [HCoV-NL63/ HCoV-HKU1] in the same group as they are both accompanied by acute breathing infections ^[31].

■ Fifth Class "SARS": Includes the genome viruses [SARS-Coronavirus-P2/SARS-Coronavirus/SARS-Coronavirus GDH-BJH01/SARS-Coronavirus HKU-39849 ^[32]] which are categorized in the same class because they are all strains of SARS.

The samples were divided into two parts, 80% for training, 20% for testing, in tenfold cross-validation. The steps of implementation were summarized in Figure 4.

4.2.   Proposed BRNN architecture

In this section, the proposed BRNN classifier model architectures were performed and shown in Figure 5. The proposed model depends on RNN with enhanced cell units, LSTM and GRU, in their hidden states, while the training procedure is applied in a bidirectional manner. The model comprises five layers that are described as follows:

Layer 1 (Input sequence): After selecting and preprocessing the DNA sequences as described in Section 5.1, it was fed into the presented RNN models.

Layer 2 (k-mers model): The DNA sequence S was split into overlapping k-meters of length k (k = 4 is employed in this work) by the use of a sliding window with stride s, the size of the window is the current k-mer. As illustrated in Figure 6. For DNA sequence S, the windows are located covering the first k bases, and it moves one right character at a time to generate the next k-mer. Given a sequence of length $n, S = \left({S}_{1}, {S}_{2}, \dots, {S}_{n}\right)\text{where}{S}_{i}\in \{A, C, G, T\}$, $S$ is converted into $\tilde{n} = (n-{k}_{\text{high}}+1\text{)}$ k-mer's numbers.

It was observed that k-mer embedding for representing input sequences has several advantages over one-hot encoding, which was reported in previous works. Firstly, it improves the model performance, secondly; it reduces computational time and space required to execute models as compared to one-hot encoding because the word vector in the one-hot method may be a high dimensional vector since it encodes each word in a large number of text data.

Layer 3 (Word2vec algorithm): By using the word2vec algorithm ^[33], each k-mer is mapped into a d-dimensional vector space. Word2vec can utilize either Continuous Bag-Of-Words (CBOW) or Continuous Skip-Gram (CSG) to produce a distributed representation of words. A continuous bag-of-words (CBOW) was employed for all our experiments because the CBOW is faster to train than skip-gram and CBOW works well with a large amount of training data.

Layer 4 (Bidirectional RNN): It represents the suggested models based on Bidirectional RNN, where LSTM or GRU cells are used as the hidden blocks. The forward track traces the data section from left to right, while in both BLSTM and BGRU, the backward track traces the input from right to left. The forward recurrent sequence and backward hidden sequence can be formulated as follows:

where ${\mathrm{x}}_{t}$ is the input feature vector$, {\overrightarrow{\mathrm{h}}}_{t}\left({\overleftarrow{\mathrm{h}}}_{t}\right)$ is the activation vector on the forward (backward) hidden layer, ${\mathrm{W}}_{\mathrm{p}, \mathrm{q}}$ is the weight matrix$, {b}_{r}$ is the bias term$, f (\cdot)$ is the activation function on each node in the hidden layers, and y is the posterior probability vector of the output label.

Layer 5 (Output layer): A regular sigmoid function that is applied on the output layer in order to measure the expected probability of characters for each step of t and k in the alphabet. This performance is shown as follows:

4.3.   Hyper-parameters

While training a model, it is important to know which hyper-parameters to select. Hyper-parameters refer to the parameters that are set before the start of Bidirectional RNN training. The time required to train and test a model is dependent on the types of hyper-parameters selected. The tune-ability of an algorithm, hyper-parameters, or interacting hyper-parameters is a measure of how much performance can be gained by tuning it. In this research, an investigation of the hyper-parameters that work best on the RNN models was performed in order to attain the highest accuracy for DNA sequence classification. The hyper-parameters that were focused on in this work are the optimizer type and the number of unit cells.

Four different types of gradient descent algorithm were employed on both the BRNN models based on LSTM and GRU cells including SGD, Adam, Adagrad, and RMSprop optimizers and three different unit numbers of cells ((50, 80 and 100). The performance of different parameter settings was recorded and compared to get the optimum parameter settings.

Gradient descent (GD) is a class of algorithms that aims to find the minimum point on a function by following the gradient. Vanilla gradient descent just follows the gradient (scaled by the learning rate). Two common tools to improve gradient descent are the sum of gradient (first moment) and the sum of the gradient squared (second moment). The Momentum method uses the first moment with a decay rate to gain speed. AdaGrad uses the second moment with no decay to deal with sparse features. RMSProp uses the second moment with a decay rate to speed up from AdaGrad. Adam uses both first and second moments and is generally the best choice.

In our research, didn't use a Particle Swarm Optimization (PSO) because a global optimization method PSO does not use the gradient of the problem being optimized, which means PSO was less effective compared to GD in our application. In addition to that, we analyzed the effect of different embedding dimensions (numbers of cells), including 150 and 200. The model complexity will be increased by more weight parameters, which are caused by a larger d and need to be learned in the embedding layer.

4.4.   Performance evaluation

The performance of the proposed deep learning BRNN approach for SARS Coronavirus-2 virus classification was evaluated with the following classification performance measures which include accuracy, precision, recall, specificity, and F1-score and the corresponding confusion matrix was further developed. Confusion matrices represent a very important metric in machine learning evaluation since it provides further information on relations between classes and types of classification errors. The x-axis in the confusion matrix shows the predicted output (disease class) of the model and the y-axis shows the true class of the disease. The diagonal elements of the matrix represent the true positive values of the model, meaning all the correct classifications performed by the model and all other entries in the matrix show misclassifications. Off-diagonal elements for the row represent the probability that the disease of a certain class will be mis-classified as another class. The definition of the performance metrics are clarified as follows:

Accuracy: Represents the performance accuracy of the model

Precision: It identifies the proportion of correct positive classifications.

Recall (Sensitivity): It measures the proportion of actual positives correctly classified by the model.

Specificity: It identifies the proportion of correct negative classifications.

F1-score: It indicates the perfect balance between precision and recall values. It can be calculated as follows

Another significant metric is also assessed which is the area under the Receiver Operating Characteristic (ROC) curve. The Receiver Operating Characteristic (ROC) curve shows a relation between the True Positive Rate (TPR) and False Positive Rate (FPR) of the classification. The FPR is defined by:

Therefore, the ROC Curve Region (AUC) tests the efficiency of all potential classification thresholds in a given model and therefore shows the accuracy of the findings independently of that model.

5.   Results

In the first part of this section, experiments were conducted on two proposed BRNN models depending on two different types of cell units (GRU) and (LSTM) in order to attain a high-efficiency classification model based on viral DNA sequence. The RNN models were implemented using four different optimizers (SGD, Adam, Adagrad, and RMSprop) and the number of unit cells was altered between random values of 50, 80 and 100 to obtain the most suitable optimizer and the optimum number of cells that will lead to the highest classification efficiency.

A comparative study was performed and shown in Table 2 showing the classification accuracies of both models employing the four different types of optimizers and changing the unit number of cells in each optimizer. It was clear from the table that the highest accuracies were obtained when applying the SGD optimizer and the number of unit cells of 80 in both models. The results also revealed that the GRU BRNN model has achieved higher efficiencies than its counterpart model as it achieved an overall accuracy of 96.8 % while the LSTM RNN model achieved 95.8 %. This result can be attributed to the fact that the GRU cell in the BRNNs deploys a reset gate and an update gate in a hidden layer which is computationally more efficient than a usual LSTM network.

Furthermore, a plot was drawn and shown in Figure 7. In order to visualize the overall performance of the two models on all optimizers and applying the different number of unit cells. It was obvious from the plot that the performance of both models was nearly the same when applying a unit number of cells of 50. The highest classification accuracy achieved by the LSTM RNN model was 95.8% when applying the ADAM optimizer with a LSTM number of 50 and the SGD optimizer with a cell number of 80. However, the higher classification ability of the GRU was seen when applying a cell number of 80 and 100 for all optimizers reaching the highest value of 96.8% with the SGD optimizer and a GRU number of 80. The accuracy of both models didn't increase by increasing the number of unit cells concluding that the optimum number of cells was 80 and employing the SGD optimizer to achieve an efficient classification.

To evaluate the effectiveness of the implemented BRNN models employing the best hyper-parameters (SGD optimizer with a cell number of 80) and its ability to discriminate between SARS-Coronavirus-2 and other viruses, the confusion matrices for the two models and the ROC curves were developed and shown in Figures 8 and 9. The resulting confusion matrix of the LSTM BRNN model reveals that 77 out of the 83 SARS-Coronavirus-2 sequences were correctly categorized, while the other 6 sequences were incorrectly classified in other groups. In the other hand, the GRU BRNN model correctly identified 37 of the 39 SARS-Coronavirus-2 sequences, while the remaining two were misclassified. By comparing the two matrices, it can be seen that the GRU has fewer false-positive and fewer false negative values than the LSTM, which reveals the higher accuracy of prediction of the GRU model. Another measure that shows successful classification is the area under ROC curve (AUC value). Generally, ROC curve is defined as a probability curve in classification studies and the AUC values express the degree of class separability within these possibilities. The higher the AUC value, the higher the classification predictions of the employed model. As showed from Figures 8 and 9 that the average AUC value (0.98) of the GRU model was higher than that of the LSTM (0.96) which also confirms the results of the present study that proved the higher ability of classification of the GRU as compared to the LSTM model. These promising results can help in the efficient detection and identification of the SARS Coronavirus-2 and discriminate it from other Coronaviruses with high accuracy results which can further help contain its spread.

Furthermore, Tables 3 and 4 show the classification performance metrics for all five classes of disease. It was clear from the tables that the accuracy of prediction of the presented models was higher than 97% for all classes of viruses, showing the efficient performance of both models. However, when comparing the ability of both models to classify class 0 (SARS-Coronavirus-2 virus), it was seen that the GRU achieved a higher classification performance than the LSTM since it attained an accuracy of 98.4% and precision, Recall, and F1 Score values of 0.95 which were higher than the LSTM performance values. Hence, it can be concluded that the presented GRU BRNN model has a better prediction ability for COVID 19, thus providing an efficient tool to help doctors achieve better clinical decisions with high precision.

6.   Conclusions

Developing efficient diagnostic machine learning tools that target the genome in order to identify different classes of Corona Virus can help reduce the death rates and the time to identify the infection for proper clinical decisions. A Deep Learning Approach employing Bidirectional Recurrent Neural Networks (BRNN) based models was implemented to classify the SARS-Coronavirus-2 virus from other Coronaviruses and other respiratory infections based on its DNA sequence. It was aimed to propose a system with high-accuracy results that can classify the type of disease based on genome sequence leading to a rapid diagnosis of COVID-19, avoiding the disadvantages of the traditional diagnostic methods. An evaluation of the BRNN models depending on Long Short-Term Memory (LSTM) and Gated Recurrent Unit (GRU) cells was investigated to determine the most effective model for SARS Coronavirus-2 sequence classification. A study of the hyper-parameters that work best on the BRNN models was also performed, including the optimizer type and the number of unit cells. The results revealed that the best hyper-parameters producing the highest performance for both models were got when applying the SGD optimizer and an optimum number of unit cells of 80 in both models. Moreover, the GRU BRNN model was able to discriminate between SARS Coronavirus-2 and other viruses with a higher overall classification accuracy of 96.8% as compared to that of the LSTM BRNN model having a 95.8% overall classification accuracy. When comparing the ability of both models to classify all classes of viruses, the accuracy of prediction were all higher than 97% for all classes, indicating the efficient performance of both models. However, when comparing the ability of both models to classify class 0 (SARS-Coronavirus-2 virus), it was seen that the GRU achieved a higher classification performance than the LSTM since it attained higher accuracies, precision, recall, and F1 score values. Hence, it was concluded that the proposed GRU BRNN model has a better prediction ability for COVID-19, thus providing an efficient tool to help achieve better clinical decisions with high precision.

Conflict of interest

All authors declare that there is no conflicts of interest in this paper.