GNet: An integrated context-aware neural framework for transcription factor binding signal at single nucleotide resolution prediction

1.   Introduction

Transcription factors (TFs) are important molecules that control gene expression. They either activate or inhibit gene transcription by binding to specific DNA sequences ^[1,2]. Revealing how TFs recognize and bind to specific DNA sequences is crucial for understanding and further studying the function of cis-regulatory elements in regulatory genomics research. The DNA fragments that bind to specific TFs and are evolutionarily conserved are called transcription factor binding sites (TFBSs) ^[3]. Identifying TFBSs and their corresponding motifs are fundamental in the field of regulatory genomics ^[4]. With the rapid development of high-throughput sequencing technology, various approaches for quantifying TFBSs have emerged. For example, a protein-bound microarray (PBM) ^[5] allows for the study of DNA-protein interaction on the genome scale in a high-throughput manner. Systematic evolution of ligands by exponential enrichment coupled with massively parallel sequencing (SELEX-seq) ^[6] has become a preferred method to study the in vitro binding between TFs and DNA through large-scale mining of TF information. Chromatin immunoprecipitation sequencing (ChIP-seq) ^[7] provides the opportunity to study DNA-TF interactions on a genome-wide scale. An assay for transposase-accessible chromatin with high-throughput sequencing (ATAC-seq) ^[8] describes putative accessible regions of the genome that normally work with TFs and RNA polymerase. These methods provide us with a large amount of available data for studying TFBSs-associated tasks.

In recent years, multiple computational methods have been introduced for the identification of TFBSs and their motifs ^[9,10,11,12]. For example, Kmer SVM ^[9] and gkm-SVM ^[10] are support vector machine (SVM) models with K-mer and gapped K-mer characteristics of the DNA sequence to predict TFBSs. Multiple expectation maximations for motif elicitation (MEME) ^[11] and simple, thorough, rapid, enriched motif elicitation (STREME) ^[12] apply the expectation maximization (EM) algorithm and suffix trees to store data, and discover TF-DNA binding motifs by searching for repeated and ungapped fragments in sequences, respectively. Deep neural network (DNN) models ^[13] have rapidly become the mainstream algorithms in many fields due to its high efficiency and applicability, which has also been introduced to impressively predict TFBSs ^[14]. DeepBind uses deep learning to analyze how proteins bind to DNA and RNA ^[15]. DeepSEA is based on deep learning and was developed to predict sequence noncoding compilation effects and can also be used to predict TFBSs ^[16]. DanQ combines convolutional and bidirectional long short-term memory networks to build a de novo construction for predicting non-coding functions from sequences ^[17]. DeepGRN combines single attention modules and paired attention modules to predict TFBSs ^[18]. TFregulomeR is a computational tool that can reveal the context-specific characteristics and functions of transcription factors ^[19]. With one-hot encoding, BPNet ^[20] predicts base resolution ChIP-nexus binding profiles of pluripotency TFs by introducing expanded convolutional neural networks (CNN), and FCNsignal ^[21] constructs a fully convolutional neural network (FCN) model that can simultaneously predict base resolution signals, distinguish binding or non-binding regions, and predict corresponding TF-binding motifs. Although these DNN-based approaches have shown remarkable performances in the tasks of TFBSs prediction, location, and motifs recognition from genomic sequences, they seldom utilize the contextual features of DNA sequences and are limited to one-hot encoding feature representations and a onefold network.

The depth of the network is a major obstacle to training DNN models ^[22,23]. Deep highway neural network models ^[24] can effectively reduce the training difficulties caused by network depth with gated linear units ^[25], which greatly alleviates the vanishing gradient problems by applying a linear path for the gradients. The gated mechanism has been widely used in the study of biological sequences ^[26,27]. ACNet ^[28] utilizes a gated residual network to extract spatial information for a polyadenylation (Poly(A)) signal prediction based on co-occurrence embedding. PASNet ^[29] applies the gated convolution network to automatically extract the underlying patterns and identify polyadenylation signals (PAS) from the genome sequences. Deep motif (DeMo) ^[30] is a deep convolution/highway multilayer perceptron (MLP) framework that can classify TFBSs tasks in genomic sequences. Meanwhile, in recent years, attention mechanisms, which can extract more global spatial long-distance dependence and solve the problem of information overload by calculating the importance weight vector and allocating computing resources to more important tasks, have been introduced into many deep learning tasks such as computer vision and natural language processing ^[26,27]. Various types of attention mechanisms have emerged in different fields ^[31,32]; for example, self-attention ^[31] maps the input to three features-Q, K, and V-to consider the correlation between the different parts of one sample and obtain the weighted features. However, self-attention ignores potential correlations among different samples and has quadratic complexity. External attention ^[33] considers the correlations between all samples and generates a weighted output by referring to two cascaded linear layers as memory units, which has linear complexity. In order to derive the joint implicit information at different positions of nucleotide sequences, we improve the external attention mechanism and named this as a dual external attention (DEA) to obtain the relationships between segments within a sample and between samples.

In our work, we develop an integrated context-aware neural framework, GNet, based on a gated highway network to consider TF binding signals prediction at single nucleotide resolution, to determine the binding or non-binding regions at a sequence level, and understand motif recognition problems. GNet extracts spatial and temporal patterns based on attention and gated highway networks to obtain the co-occurrence embedding of the signals. Specifically, with one-hot coding, nucleotide chemical properties and nucleotide density as input features, while infiltrating the gated idea into the whole network and coupling with the improved DEA mechanism, GNet positively performs in three tasks, namely signal regression, sequence classification and motif recognition on ChIP-seq data, and chromatin accessibility ATAC-seq data, which outperforms several state-of-the-art methods. In addition, we conduct a cross-species studies on 15 human and 18 mouse TF data, and GNet also shows the best performance over the competition method.

2.   Materials and methods

2.1.   Data collection and preprocessing

ChIP-seq is a powerful tool for studying protein-DNA interactions in the body owing to its rapid and efficient genome-wide detection of DNA regions that interact with specific TFs. In this work, we collect 53 human TF ChIP-seq datasets from the ENCODE project ^[34] (https://www.encodeproject.org/), including 21 from the GM12878 cell line, 20 from the K562 cell line, and 12 from the HeLa-S3 cell line, as shown in Supplementary Table S1. To ensure data quality, we preferentially select data with biological replication. With the same operation as in ^[21], the peak of each sequence, as the center, is extended to a length of 1000 bp, and the obtained sequence and the corresponding signal (p-value) are regarded as "positive" samples. Sequences of the same length at 3000 bp upstream of the peak and the corresponding signals are selected as "negative" samples. We ensure that the signal value of the "negative" samples is smaller than that of the corresponding "positive" samples. Hg38 is used as the reference genome for selection. The signal values are normalized by ${\mathit{log}}_{10}(1+signal)$ to reduce the influence of large difference. We randomly select 20% of the positive and negative samples for each TF as test data, 10% of the remaining samples as validation data, and the rest as training data.

We also collect 6 chromatin accessibility ATAC-seq datasets from ENCODE, which are from A549, GM12878, HepG2, IMR90, K562, and MCF7 cell lines. For each dataset, we preprocess the above cell lines as ChIP-seq data. We download 18 mouse TF ChIP-seq datasets from ENCODE, which are from the same TF family as human datasets with mm10 as the reference genome. Supplementary Table S1 describes the details of these datasets.

In addition, we randomly select in vitro PBM data of 8 mouse TFs from the Dialogue for Reverse Engineering Assessments and Methods 5 (DREAM5) ^[35] project to further verify the effect of our improved DEA mechanism. The 8 datasets are from two distinct microarray designs, named HK and ME (the abbreviation of the designers' name). Each of these datasets contain more than 40,000 35 bp probe sequences with corresponding PBM probe intensities. We normalize the data according to the total signal strength.

2.2.   Feature representation

In our framework, each DNA sequence is represented as a matrix of dimension L^*8, where L is the length of the DNA sequence and 8 (4 + 3 + 1) corresponds to the sum of the dimensions of one-hot encoding, the 3D coordinates coding of nucleotide chemical features, and the nucleotide density coding of each nucleotide. For DNA sequence $N = \left\{{N}_{1}, {N}_{2}, \cdots , {N}_{L}\right\}$, the three encodings are as follows.

2.2.1.   One-hot encoding

DNA sequence N is encoded as an L^*4-matrix in one-hot form, where 4 is the dimension of the binary one-hot vector (A = [1,0,0,0], C = [0,1,0,0], G = [0,0,1,0], and T = [0,0,0,1]) corresponding to nucleotide {A, C, G, T} at that position.

2.2.2.   Nucleotide chemical property (RFHC)

Nucleotides have different chemical properties according to different ring structures, hydrogen bonds, and chemical functions, etc. ^[36]. For example, A and G contain two rings, while C and T contain one ring. G and C can form strong hydrogen bonds, while A and T form weak hydrogen bonds. A and C are classified as amino groups, while G and T are classified as ketones. We use a 3-dimension binary vector $\left({x}_{i}, {y}_{i}, {z}_{i}\right)$ to represent the chemistry of each nucleotide ${N}_{i}$, where ${x}_{i} = 1$ means that ${N}_{i}$ contains two rings and ${x}_{i} = 0$ shows that ${N}_{i}$ contains one rings; similarly, ${y}_{i}, {z}_{i}$ represents the characteristic of hydrogen bonds and chemical function of ${N}_{i}$, respectively. The specific formulas are as follows:

Therefore, each sequence is transformed into an L^*3-matrix according to the 3D binary encoding (A = [1,1,1], C = [0,0,1], G = [1,0,0], and T = [0,1,0]).

2.2.3.   Nucleotide density (ND)

ND takes into account the position and frequency information of nucleotides on the sequence, which is also an effective coding method ^[37,38]. The density D_i of nucleotide N_i at position i in sequence N is defined as follows:

where $||{L}_{i}||$ is the length of the sequence of the final nucleotide ${N}_{i}$ and $i$ stands for the position of the nucleotide ${N}_{i}$. Therefore, each sequence can be encoded as an L^*1 matrix.

2.3.   Network architecture

GNet is a DNN model with an encode-decode architecture, with details as follows.

2.3.1.   The gated highway network unit

As the depth of the neural network increases, the gradient back propagation of network training becomes more and more difficult ^[29], and the gated mechanism is an effective method to improve the efficiency of network training ^[24,39]. Gated highway neural models can control the information flow by capturing large context multi-level patterns using the linear shortcut connections with neither extra parameters nor computational cost, and thus can greatly alleviate the vanishing gradient problems since a linear path is provided for the gradients.

In this work, we infiltrate the idea of a gated mechanism into each component of GNet. Specifically, we introduce two gates, T and C, to control the information flow of the first input and the second input, respectively. For simplicity, we set $C = 1-T$. The specific formula of the gated network unit is as follows:

where ${X}_{first}^{\zeta }$ and ${X}_{\mathit{sec}ond}^{\zeta }$ are two different inputs of the $\zeta -th$ gated network unit, $\zeta \in \left\{1, 2, 3, 4\right\}$, $\otimes$ is the matrix multiplication, $\oplus$ is the element-wise addition operation, ${x}^{\zeta }$ is the input that controls the $\zeta -th$ gate, and ${f}_{S}\left( \right)$ is the sigmoid function for dynamically calculating the gate channel value. In particular,

In the subsequent encoding architecture, we design several gated units with different additional effects in convolution layers and gated recurrent unit (GRU) layers, see the encoder architecture section for details.

2.3.2.   Dual-external-attention mechanism

An attention mechanism is one of the common modules used to derive the global dependency pattern features in natural language processing, which can allocate resources according to the importance weight. An external-attention mechanism ^[33] was developed to solve the two major pain points of self-attention, that is, high computational complexity and the absence of relationships between samples ^[31]. It has linear complexity and an implicit consideration for the relationship between different samples. However, it ignores the relationship among the component elements in one sample. DEA, proposed in this paper, is an improved external-attention in which the relationship among component elements is considered by multiplying the input and its transpose, as shown in Figure 1. The specific formula of DEA is as follows:

Figure 1.  The framework of GNet. The three coding matrices of DNA sequence are spliced and fed into GNet with encoder-decoder-skip architecture, in which, the encoder part integrates the gated highway network to capture large context multi-level patterns and DEA mechanism to learn implicit relationships inter-sample and among samples. The decoder layer is connected with the corresponding encoder layer with a skip identity connection to make full use of the context information. GNet is applied for TF binding signal prediction under single nucleotide resolution, determination of binding or non-binding sequences, motif recognition and cross-species research between human and mouse.

DownLoad: Full-Size Img PowerPoint

where $X$ is the input, $Normal\left(\right)$ represents the normalization process, and $\otimes$ is the matrix multiplication. DEA can combine the advantages of self-attention and external attention to learn implicit relationships within a sample and among samples, and the computational cost is lower than that of self-attention.

2.3.3.   The encoder architecture

DNN can extract high-quality contextual information from DNA sequences without feature engineering, a variety of TFBSs-associated prediction models based on DNN have been proposed for extracting discerning patterns of DNA sequences automatically ^[17,18,19]. The feature extraction architecture of GNet consists of three blocks, each containing a convolution layer with a gated highway network unit, a maximum pooling layer, and a dropout layer ^[40]. CNN can capture the local spatial pattern features of the sequence, and the gated highway network increases the flexibility of contextual information extraction and alleviates the problem of gradient backflow obstruction. The gated convolution highway network unit in the first block is denoted as the formula:

where ${W}^{1}$, ${X}^{1}$, and ${b}^{1}$ are the weight matrix, input, and bias vector of the 1-st convolution, respectively, and $\mathit{\Delta} \left(\right)$ and $\mathit{\nabla} \left(\right)$ represent the first 1/2 and last 1/2 components of the convolutional layer output matrix, respectively. Here we take the first 1/2 components of the output matrix of the convolution layer as the input ${x}^{1}$ to generate the first gate.

The convolution gated network units in the second and the third blocks are as follows:

where ${W}^{\tau }$ and ${b}^{\tau }$ are the weight matrix and bias vector of the $\tau -th$ block of the encoder part, respectively, $\tau = 2$ and $3$, and ${X}^{\tau }$ is the output of the $\tau -1$ block. Here we take the output matrix of the $\tau -th$ convolution layer as the input ${x}^{\tau }$ to generate the $\tau -th$ gate. In this way, the information of the previous block can be fully considered to increase the multilinearity of the calculation and capture more context information.

In the third block, after the dropout layer, we add a bidirectional GRU ^[41] with a gated highway network unit:

where $\overrightarrow{GRU}\left(\right)$ and $\overleftarrow{GRU}\left(\right)$ are the GRU forward and backward operations, respectively. The output matrix of the forward operation of the GRU is used as the input to generate the fourth gate. A GRU with a gated mechanism can better extract the long-term dependence of nucleotides. The output of the GRU is fed into the DEA layer, which allows the model to focus on elements that contribute more to the prediction by assigning different weights to each position. Finally, the global context information of the sequence is captured by the global average pooling layer.

2.3.4.   The decoder architecture with gated mechanism

The decoder architecture that contains four blocks is the process of up-sampling, where each block contains an upper sampling layer with a linear interpolation algorithm, a batch normalization (BN) layer, and a CNN layer. In each of the three blocks, there is a skip identity connection with the corresponding encoder layer to make full use of the context information. We apply up-sampling to recover the down-sampled eigenmatrix, and then integrate the positional information in the encoded process with the gated identity connection. The last layer of convolution is used to convert the features into predictable signals of dimension $1\times l$. The process for this part is as follows:

where $linear\left(\right)$ represents a linear interpolation operation, ${{X}_{in}}^{\zeta }$ is the input of the network, and ${X}^{\zeta }$ is the position information derived from the corresponding encoded part.

2.4.   The implementation of the model

GNet has been executed on Tesla P40 using python 3.6, cuda10.0 with pytorch1.1 backend. GNet is an integrated framework, in which the regression task at a single nucleotide resolution is fundamental, so the mean square error (MSE) at a single nucleotide resolution is used as the loss function:

where N and L are the number of samples and the length of the sequence, respectively, and ${\widehat{y}}_{ij}$ and ${y}_{ij}$ are the predicted signal value and the real value, respectively. In order to avoid overfitting, ${L}_{2}$ regularization (||θ||₂) is adopted, where λ represents the regularization parameter. In our work, we apply an Adam optimizer ^[42] to train our network, with the minimum batch size 500 and a learning rate decay with a decay rate 0.9 for every 10 epochs. The learning rate, β value of Adam, and regularization parameters are randomly selected from {0.001, 0.0001}, {0.9, 0.99, 0.999}, and {0, 0.001}, respectively, and model with the best performance on the validation data set is saved for testing. The source code for GNet is available at https://github.com/keke0419/GNet-main.

2.5.   Evaluation indicators

As an integrated framework, GNet is applied to handle three different tasks, so we use different indicators for evaluation. For signal modeling tasks at a single nucleotide resolution, the mean square error (MSE) and Pearson correlation coefficient (PCC) are chosen as evaluation indexes. For the determination task of binding or non-binding regions, we use the area under the receiver operating characteristic curve (AUC) and the area under the precision recall rate curve (AUPRC) as evaluation indicators. Additionally, for the motif recognition and prediction task, $-{log}_{2}(p-value)$, $-{log}_{2}(E-value)$, and $-{log}_{2}(q-value)$ of the testing are used as evaluation indicators.

3.   Experimental results

We tested GNet on three tasks with 53 TF ChIP-seq datasets and 6 chromatin accessibility ATAC-seq datasets, as well as cross-species studies between human and mouse TFS on 15 human and 18 mouse TF ChIP-seq datasets.

3.1.   Predicting base resolution signals on ChIP-seq data

We conducted a regression analysis on 53 human TF ChIP-seq datasets from ENCODE to predict the base resolution signal and compared GNet with two of the most advanced competitive approaches, BPNet ^[20] and FCNsignal ^[21], on the test sets of 53 TF ChIP-seq data with MSE and PCC as evaluation metrics. BPNet ^[20] takes a one-hot encoding of the sequence as an input, and constructs a dilated CNN model to predict the base resolution ChIP-nexus binding profiles of pluripotency TFs. FCNsignal ^[21] constructs a CNN-dominated encoding and decoding architecture with a one-hot encoding of sequences as an input to predict TF-DNA binding signals at base resolution level. Figure 2 and Supplementary Table S2 shows the performance comparison of the three algorithms.

Figure 2.  The performance of GNet in TF binding signal prediction at single nucleotide level on 53 ChIP-seq datasets compared with BPNet and FCNsignal. (a), (b) show the MSE values of the three methods, the smaller the value, the better the model. (c), (d) display the PCC values of the three methods, the higher value is better.

DownLoad: Full-Size Img PowerPoint

GNet performed remarkable on these 53 TF ChIP-seq datasets, with the lowest average MSE of 0.121, compared with a value of 0.133 for BPNet and a value of 0.127 for FCNsignal, and the largest average PCC of 0.798, in contrast to a value of 0.768 for BPNet and a value of 0.780 for FCNsignal. This indicates that, compared with the two models that use only one-hot encoding as features and are dominated by CNN, infiltrating the idea of a gated mechanism into the whole network, and considering the physicochemical and the density properties of nucleotide is effective for identifying TFBSs.

In addition, to demonstrate the performance of our model in predicting TF signals at a single nucleotide resolution, we visualized TF signal values predicted by GNet and the two competitive methods and compared them with real signal values. Figure 3(a) shows a comparison of two randomly chosen TFs. GNet thoroughly fits the TF signal values, and the fitting degree is better than the other competitive methods.

Figure 3.  Visualization of model effects and predicted motifs. (a) Comparison of TF signals predicted by models at single nucleotide resolution. We compare GNet with two competitive methods and the real experimental signals on randomly selected 2 ChIP-seq datasets. (b) Comparison of motifs predicted by different models, and compared with the experimental verification on randomly selected 3 ChIP-seq datasets.

DownLoad: Full-Size Img PowerPoint

3.2.   Determining the binding or non-binding sequence on ChIP-seq data

The problem here is how to use the predicted signal values at a base resolution to determine either binding or non-binding sequences. In ^[21], the authors confirmed that the maximum value of signals could reflect the openness of TF-DNA binding; the higher the signal value, the higher the degree of openness. Based on the same strategy, we distinguished binding or non-binding sequences with GNet.

We compared our model with the two competitive methods mentioned in Section 3.1 with AUC and AUPRC for evaluation. As shown in Figure 4 and Supplementary Table S2, the average AUC and AUPRC of GNet on 53 datasets are greater than the two competitive methods with an AUC of 0.951 and an AUPRC of 0.954. In particular, the AUC values of GNet are higher than FCNsignal and BPNet on 46 and 48 datasets, respectively, and the AUPRC values are higher than FCNsignal and BPNet on 44 and 50 datasets, respectively.

Figure 4.  The performance of GNet in binding or non-binding sequence determination on 53 ChIP-seq datasets compared with BPNet and FCNsignal. (a), (b) present the AUC values of the three algorithms. (c), (d) show the AUPRC values of the three methods.

DownLoad: Full-Size Img PowerPoint

3.3.   Predicting motifs on ChIP-seq data

We adopt the same motif identification strategy as ^[21,43]; the position with the maximum predicted signal value in the sequence is locked and extended 49 bp forward and 50 bp backward to get a 100 bp segment as a possible binding region. We use the training weight of the first convolution layer in the coding architecture to the possible binding region with a sliding window to get value and select the sequence segment of the region with the maximum value; these sequence segments are aligned to calculate the corresponding position frequency matrix (PFM) to match with experimentally verified motifs in the standard database HOCOMOCO ^[44].

In addition to the two competitive methods mentioned above, we also select two motif recognition tools, MEME ^[11] and STREME ^[12], as the benchmark methods, where MEME uses the EM algorithm to discover a new, unencapsulated motif by searching for recurring, ungapped sequence fragments in the sequence, and STREME stores data with suffix trees, and uses it to efficiently count matches to a PWM of a candidate motif, to identify rich or relatively rich ungapped motifs of fixed length in the sequence. We use the negative logarithm with base 2 of $p-value$, $E-value$, and $q-value$ generated in TOMTOM ^[45] as our evaluation indicators. The results are shown in Figure 5(a) and Table S3.

Figure 5.  The performance of GNet in motif identification on 53 ChIP-seq datasets compared with BPNet, FCNsignal, MEME and STREME, and cross-species studying between human and mouse on corresponding TFs compared with BPNet and FCNsignal. (a) shows the results of hypothesis testings in motif identification, the larger the values of the three indexes are, the closer the predicted motif is to the standard motif. (b) Heat map displays the results of GNet trained on human ChIP-seq data and tested on corresponding TF mouse data compared with the two competitive methods. (c) Results of GNet trained on mouse ChIP-seq data and tested on corresponding TF mouse and human test datasets respectively.

DownLoad: Full-Size Img PowerPoint

In 53 ChIP-seq datasets, GNet outperforms the four competitive methods with the lowest average $p-value$, $E-value$, and $q-value$, so that the average values of the three indicators are all higher than the collective. Moreover, GNet found the largest number of motifs, specifically 9, 5, 7, and 4 more than the competitive methods MEME, STREME, BPNet, and FCNsignal, respectively. Figure 3(b) shows partial TF motifs predicted with the five methods. The motifs predicted by GNet are closer to the experimentally verified motifs than that predicted by other methods. Figures 1 and S1 shows the visualization of part of predicted motifs.

3.4.   The performance of GNet on ATAC-seq data

ATAC-seq data are commonly used to test chromatin accessibility and to determine gene expression regulation mechanisms. In this study, to verify the overall performance of GNet, we downloaded 6 ATAC-seq datasets from ENCONE as independent evaluation data, on which we tested our model in signal prediction at a single nucleotide resolution and the binding or non-binding sequences prediction were compared with the two competitive methods FCNsignal and BPNet. The evaluation indexes are MSE, PCC, AUC, and AUPRC. The comparation results are shown in Table 1.

Table 1.  Comparison of GNet, BPNet and FCNsignal in regression and classification tasks on 6 ATAC-seq datasets.

Model	indicator	K562	MCF7	IMR90	HepG2	GM12878	A549	Mean
GNet	MSE	0.221	0.281	0.254	0.217	0.318	0.495	0.298
	PCC	0.804	0.818	0.816	0.805	0.814	0.757	0.802
	AUC	0.944	0.951	0.957	0.951	0.956	0.937	0.949
	AUPRC	0.949	0.954	0.958	0.952	0.958	0.944	0.953
FCNsignal	MSE	0.216	0.278	0.217	0.206	0.318	0.479	0.286
	PCC	0.787	0.818	0.804	0.784	0.797	0.736	0.788
	AUC	0.943	0.946	0.943	0.945	0.954	0.933	0.944
	AUPRC	0.948	0.949	0.952	0.947	0.958	0.941	0.949
BPNet	MSE	0.215	0.240	0.244	0.246	0.296	0.412	0.276
	PCC	0.767	0.818	0.771	0.763	0.796	0.730	0.774
	AUC	0.925	0.950	0.929	0.916	0.942	0.941	0.934
	AUPRC	0.931	0.954	0.931	0.919	0.946	0.949	0.938
Note: Bold numbers are the best results.

---

 | Show Table

DownLoad: CSV

The average PCC, AUC, and AUPRC of GNet are all higher than those of the competitive methods; in particular, GNet achieves the best PCC, AUC, and AUPRC values on the 5 datasets, except for A456t. This shows that GNet has a good robustness, even on ATAC-seq data.

3.5.   The performance on cross-species studies

It was demonstrated that the genes of humans and mice are more than 85 percent similar, and about 80 percent of the proteins encoded by genes are homologous. In recent years, many human and mouse genome databases have been developed, among which HOCOMOCO ^[44] is specially designed for the study of human and mouse TFs. We downloaded 15 human TFBSs ChIP-seq datasets and 18 mouse datasets of the same corresponding TF families from ENCODE. For the same TF family, we trained GNet on human TFBSs datasets and tested on mouse datasets; the results of base-resolution signal prediction and the binding or non-binding sequence distinction tasks are shown in Figure 5(b).

Compared with the results tested on human TFBSs test data, the results of GNet tested on mouse datasets are generally good, and even for some TFs, the results are better than those tested on human datasets, such as for mouse TF CTCF, GNet obtains a PCC of 0.95 compared to the value of 0.941 for the human test set. The average PCC, AUC, and AUPRC values of GNet are higher than those of the competitive methods on these two cross-species tasks.

We also trained GNet on mouse data and tested on human data. As shown in Figure 5(c), the models trained on mouse TFBSs data performed relatively good on human data, and PCC values are even 30% greater than those tested on mouse data. This indicates that GNet performs well in a cross-species study of human and mouse TF.

4.   Discussion

In this section, we conducted ablation analyses for the experiment by comparing the effects of an improved DEA and the other attention mechanisms, the effect of a network with and without the gated mechanism, and the influence of nucleotide density and nucleotide chemical features added in this architecture.

4.1.   The effects of dual external attention mechanism

We compared the performance of our model in signal prediction at a single nucleotide resolution and a binding or non-binding sequences prediction with attention-free, self-attention ^[31], external attention, and DEA mechanisms, respectively, on several randomly selected datasets (7 ChIP-seq datasets and 2 ATAC-seq datasets) without changing the other settings in the architecture. The results are shown in Table 2, from which we can see that GNet with a built-in DEA obtains the best performance. In addition, the models with an external attention and attention-free behave similarly, that is, an external attention mechanism which focuses on the relationship between different samples does not only contribute much to feature extraction of TFBS-associated tasks. The DEA mechanism that can obtain the implicit relationships within and among samples has a lower computation complexity and higher accuracy than self-attention. This indicates that DEA is more effective in feature extraction of TFBS-associated tasks.

Table 2.  Comparison of the models with attention-free, self-attention, external attention and DEA respectively.

Model	indicator	HeLa-S3		K562		GM12878		ATAC-seq		Mean
		E2F1	ELK1	NFYA	MXI1	CEBPB	ZEB1	K562	GM12878
DEA	MSE	0.132	0.063	0.092	0.104	0.058	0.112	0.221	0.318	0.1375
	PCC	0.838	0.691	0.841	0.732	0.696	0.791	0.804	0.814	0.7759
	AUC	0.993	0.974	0.983	0.954	0.917	0.961	0.944	0.956	0.9603
	AUPRC	0.993	0.986	0.985	0.959	0.928	0.960	0.949	0.958	0.9648
Self-attention	MSE	0.131	0.068	0.094	0.121	0.085	0.121	0.196	0.291	0.1384
	PCC	0.837	0.680	0.833	0.725	0.684	0.789	0.806	0.816	0.7713
	AUC	0.993	0.971	0.969	0.956	0.911	0.961	0.943	0.956	0.9575
	AUPRC	0.992	0.984	0.978	0.960	0.919	0.961	0.947	0.958	0.9624
External-attention	MSE	0.127	0.061	0.105	0.131	0.059	0.102	0.228	0.298	0.1389
	PCC	0.842	0.684	0.82	0.702	0.672	0.795	0.786	0.816	0.7646
	AUC	0.993	0.965	0.974	0.939	0.875	0.961	0.94	0.957	0.9505
	AUPRC	0.993	0.981	0.978	0.947	0.892	0.959	0.945	0.959	0.9568
Attention-free	MSE	0.136	0.070	0.095	0.110	0.057	0.112	0.191	0.341	0.1390
	PCC	0.824	0.676	0.836	0.709	0.698	0.780	0.805	0.81	0.7673
	AUC	0.993	0.971	0.982	0.950	0.917	0.956	0.945	0.958	0.9590
	AUPRC	0.993	0.984	0.984	0.956	0.926	0.954	0.949	0.961	0.9634
Note: Bold numbers are the best results.

---

 | Show Table

DownLoad: CSV

To further verify the effect of our improved DEA mechanism, we randomly downloaded PBM data of 8 mice TFs from DREAM5 project and applied DeepBind ^[18] as the evaluation model to evaluate the performance of the DEA mechanism by predicting the probe strength; R² (reflecting the proportion that all variations of dependent variables can be explained by independent variables through a regression relationship, the closer it is to 1, the better the interpretation of independent variables to dependent variables) and PCC are selected as the evaluation indexes. We trained DeepBind with self-attention, external attention, and DEA mechanisms via 5-fold cross validation, and the results are shown in Supplementary Table S4. The average effect of DeepBind combined with DEA is better than that of the model with an external attention mechanism and is as good as that of the combined mechanism with self-attention. This confirms that our improved DEA mechanism takes less time and works not worse than self-attention, at least in TFBS-associated tasks.

4.2.   The effect of introducing the gated highway unit and different types of features

We compared the effects of our model with and without the gated highway unit in signal prediction at a single nucleotide resolution and the binding or non-binding sequences prediction on the same datasets as the last subsection, without changing the other settings in the architecture. As shown in Table 3, GNet with a built-in gated highway unit performs better than the model without a gated mechanism. This indicates that, without an extra computational cost, our gated highway neural unit can efficiently capture large contextual multilevel patterns.

Table 3.  The comparison results of the models with or without gated highway units, and only one-hot encoding.

Model	indicator	HeLa-S3		K562		GM12878		ATAC-seq		Mean
		E2F1	ELK1	NFYA	MXI1	CEBPB	ZEB1	K562	GM12878
GNet	MSE	0.132	0.063	0.092	0.104	0.058	0.112	0.221	0.318	0.1375
	PCC	0.838	0.691	0.841	0.732	0.696	0.791	0.804	0.814	0.7759
	AUC	0.993	0.974	0.983	0.954	0.917	0.961	0.944	0.956	0.9603
	AUPRC	0.993	0.986	0.985	0.959	0.928	0.960	0.949	0.958	0.9648
No gated mechanism	MSE	0.136	0.066	0.094	0.108	0.075	0.115	0.217	0.292	0.1379
	PCC	0.831	0.679	0.828	0.723	0.68	0.792	0.801	0.795	0.7661
	AUC	0.991	0.964	0.971	0.955	0.897	0.96	0.943	0.95	0.9539
	AUPRC	0.991	0.981	0.978	0.959	0.909	0.959	0.947	0.953	0.9596
Only one-hot	MSE	0.131	0.063	0.088	0.108	0.063	0.110	0.238	0.405	0.1508
	PCC	0.840	0.691	0.844	0.733	0.676	0.787	0.801	0.816	0.7735
	AUC	0.992	0.960	0.976	0.955	0.914	0.956	0.945	0.958	0.9570
	AUPRC	0.991	0.980	0.980	0.958	0.922	0.955	0.950	0.961	0.9621
FCNsignal	MSE	0.141	0.068	0.098	0.117	0.058	0.108	0.216	0.318	0.1443
	PCC	0.825	0.671	0.822	0.700	0.678	0.789	0.787	0.797	0.7586
	AUC	0.992	0.961	0.978	0.944	0.873	0.960	0.943	0.954	0.9506
	AUPRC	0.991	0.982	0.983	0.945	0.883	0.959	0.948	0.958	0.9561
Note: Bold numbers are the best results.

---

 | Show Table

DownLoad: CSV

At the same time, we also compared GNet with the model using only a one-hot encoding as the input. Table 3 shows that our model integrating a one-hot encoding, nucleotide density, and nucleotide chemical features as the input performs better. A one-hot coding is not sufficient to cover all intrinsic features of DNA sequences; density, as well as chemical properties of sequences, are also important in TFBS-associated tasks.

5.   Conclusions

In this work, we propose an integrate context-aware neural framework, named GNet, based on a gated mechanism and improved external attention to consider TF binding signals prediction at a single nucleotide resolution, determination of binding or non-binding regions at the sequence level and motif recognition problems respectively. Most previous studies have only performed regression or classification tasks at the sequence level, rarely at the single nucleotide level, and few model studies have integrated multiple tasks such as a regression and classification. On 53 human TF ChIP-seq datasets and 6 chromatin accessibility ATAC-seq datasets, the experimental results show that GNet has an excellent performance in the three tasks, is superior to several competitive methods, and the results of cross-species studies on 15 human and 18 mouse TF data show that GNet also shows the best performance in cross-species prediction over the competitive methods.

GNet performs well in these three tasks and cross-species studies, which can extract spatial and temporal patterns benefiting from the following aspects: the addition of two new features, gated mechanism including gated highway network unit and the skip identity connection in decoder architecture, and improved attention mechanisms. RFHC coding considers the three chemical properties of nucleotides, and ND coding considers the position and frequency information of nucleotides, adding these two types of features effectively avoids the loss of information. At the same time, the gated mechanism can increase the flexibility of the model, extract more effective contextual information and reduce the gradient backflow blocking problem. Combined with the improved attention mechanism DEA, the model can be trained to learn the implicit relationships inter-sample and among samples to improve the model performance.

Although GNet model has achieved excellent performance, there are still some limitations, such as not deeply constructing the decoder framework. In future work, we will consider introducing K-mer or Word2vec ^[46] embedding to consider the dependence between nucleotides. At the same time, complementary sequences, reverse sequences and complementary reverse sequences of DNA have been proved to play a certain role in TFBSs prediction ^[47]. Deconvolution has also been widely used in various decoding networks ^[48,49]. Meanwhile, the algorithm of learning how to embed DNA sequence and TF tag into the same space is also generated ^[50], which provides new visions and opportunities for our subsequent studying.

Use of AI tools declaration

The authors declare they have not used Artificial Intelligence (AI) tools in the creation of this article.

Acknowledgments

The authors thank those who contributed to this paper, as well as the reviewers for their careful reading and valuable suggestions.

Conflict of interest

The authors declare there is no conflict of interest.