Iterator-Net: sinogram-based CT image reconstruction

1.   Introduction

Computed tomography (CT) is one of the major imaging techniques that is widely used in clinical practice. Different types of imaging modalities have both similarities and characteristics in their tomographic image reconstruction theories and techniques. Recently, molecular imaging, small animal imaging and the new CT devices have become the focus of international peers. The success of new imaging devices is inseparable from the development of tomographic image reconstruction technology. In the past 40 years, a variety of tomographic reconstruction algorithms have been developed. These algorithms are divided into three categories: mathematical reconstruction algorithms, deep learning reconstruction algorithms and unrolled dynamics (UD) reconstruction method ^[1].

The mathematical reconstruction algorithm is modeled according to imaging principles and has a reliable interpretation, but it does not make use of the prior knowledge of the data. Deep learning makes full use of the prior knowledge of the data, but it is not interpretable. The UD reconstruction algorithm combines the advantages of deep learning techniques and traditional reconstruction algorithms. It has been widely used in medical image processing ^{[2,3,4,5,6,7,8]}. ISTA-net ^[9] is a structured deep network that learns a nonlinear transform in an end-to-end manner. This transform replaces the complex transform associated with the proximal mapping. ISTA-NET++, another end-to-end ISTA-unfolding deep network, handles a multi-ratio task using a single model. This kind of method improves the interpretability of the deep learning model and opens up a new way for medical image reconstruction.

The application of UD methods in CT image reconstruction is also gradually increasing ^[1,11,12]. Dong et al. proposed the meta-inversion network (MetaInv-Net) ^[1] which is a representative UD method. Its backbone network architecture is composed of many submodules related to the conjugate gradient (CG) algorithm. Instead of imitating the iterative method, Jin et al. applied a convolutional neural network (CNN) to an inverse problem and proposed FBPConvNet ^[10]. It is another UD method for CT reconstruction, which is used to explore an advanced CNN architecture. Hu et al. ^[11] designed a UD method, called learned experts' assessment-based reconstruction network (LEARN). They unfolded an iterative reconstruction scheme into many submodules of a neural network. However, these UD methods all have a common disadvantage. They all simulate regularization operators in optimization problems without the iterative algorithm itself. Therefore, this paper attempts to explore a deep neural network to learn the map of the iterative algorithm itself.

In this paper, we propose a novel UD method for CT reconstruction, which is fundamentally different from the methods mentioned above. Instead of using neural networks to simulate the approximation operator in iterative algorithms, we design a model to simulate the map between the former and the latter in maximum-likelihood expectation maximization (MLEM) and we call it iNet. We train the network to approximate the real map as closely as possible. This model improves the quality of reconstructed CT images and does not require explicit iterative expressions. This method also has strong generalization performance. Namely, we do not have to study numerical solutions for optimization problems and can use neural networks to learn an iterative algorithm to solve the corresponding optimization problems. This work can provide new inspiration for UD methods.

2.   Related work

2.1.   MLEM algorithm

The image reconstruction problem is modeled as follows:

where $A\in {R}^{m*n}$ denotes the system matrix of the inverse problem, ${a}_{ij}$ represents the element in row $i$ and column $j$ of $A$, $X = {[{x}_{1}, \dots, {x}_{n}]}^{T}$ denotes the unknown image, and ${P = [{p}_{1}, \dots, {p}_{m}]}^{T}$ denotes the known measured data. Usually, we give an estimate of $X$, and then the estimate of the corresponding measured value is expressed as ${Y = [{y}_{1}, \dots, {y}_{m}]}^{T}$, where ${y}_{i} = \sum _{j = 1}^{n}{a}_{ij}{x}_{j}$. Supposing $P$ obeys Gaussian distribution with an expectation ${\mu }_{i} = {y}_{i}$ and the same variance $\sigma$, we can get the probability density of $P$ distribution:

The likelihood function is established as follows:

Take the logarithm of this likelihood function:

According to the previous assumptions, we know that the above equation is equivalent to the following equation:

To find the extreme value of $\mathit{ln}\left(Prob\right)$, we find the partial derivative of the above Eq (5) with respect to ${x}_{l}$,

Bring $\frac{\partial {y}_{i}}{\partial {x}_{l}} = \frac{\partial }{\partial {x}_{l}}\left[\sum _{i = 1}^{m}{a}_{ij}{x}_{j}\right] = {a}_{il}$ into Eq (6), we get the following formula:

By the corresponding Kuhn-Tucker (KT) conditions, we know that

By the fixed point theory, we achieve the update rule:

Then, the final MLEM algorithm is obtained as follows:

where ${X}^{k}$ and ${X}^{k+1}$ represent the results of the $k$ -th and $(k+1)$ -th iteration of $X$. To avoid the case where the denominator in Eq (10) is zero, we usually introduce an arbitrary small positive number $\varepsilon$, and obtain the general form of Eq (10) as follows:

2.2.   Universal approximation theorem

Deep learning models are used to learn mapping relations in the data space, including linear and nonlinear mapping relations. Initially, it was thought that learning nonlinear mappings required specialized nonlinear models. Fortunately, the universal approximation theorem shows that a feedforward neural network with a linear output layer and at least one layer containing any kind of "squeezed" activation function can approximate a Borel measurable function from one finite dimensional space to another with arbitrary accuracy. Specifically, the neural network built by the hidden layers with arbitrary width and depth approximates continuous functions with arbitrary accuracy ^[12,13]. Moreover, neural networks with a single hidden layer, as shown in these works ^[14,16], can accurately approximate any nonlinear continuous function.

3.   Methods

3.1.   Overview

We propose a kind of iterator network to simulate the solution form of the MLEM algorithm and call it iNet. This model improves the quality of reconstructed CT images and does not need explicit iterative expressions. To understand the iNet, we present the conceptual explanation and the architecture of iNet in this section.

3.2.   iNet

We know that the iterative form of the MLEM algorithm is as follows:

Observing the above expression, we obviously find the following conclusions:

Where f represents the map of the former and the latter. We train the iNet to learn the map f instead of manually computing the explicit expression of the MLEM algorithm.

3.3.   Network architecture

Figure 1 shows the detailed network structure of the proposed iNet. It is easy to find that the iNet consists of a convolution part, deconvolution part and skip connections. The convolution part consists of five convolution layers followed by Rectified Linear Unit (ReLU) activation function. The number of convolution kernels is 96, the size of all convolution kernels used is 3 × 3 and the fill of the convolution is "valid". Therefore, the number and the dimension of feature maps after each convolution are 96 and reduced by 4, respectively. The deconvolution part consists of a deconvolution layer, three ReLU activation functions followed by a deconvolution layer, and a deconvolution layer followed by ReLU. We use two skip connections between the convolution and deconvolution, which can avoid the loss of image detail. On the other hand, the skip connection also solves the problems caused by overly deep convolutional layers. It's worth noting that the number of convolution kernels of the final layer is 1. To make the network structure clearer, we explained these symbols of "Adding of pixel level" by adding the Figure 2.

4.   Experimental descriptions

To verify the feasibility of our iNet, we conduct experiments on a CT dataset and report qualitative and quantitative results. Firstly, we introduce the dataset in detail. Then, we selected some reconstructed images for display. Finally, we calculate the peak signal-to-noise ratio (PSNR) and structure similarity index measure (SSIM) of the reconstructed image.

4.1.   Dataset

The dataset is stored in the cancer imaging archive (TCIA). It contains clinical CT images of 149 patients scanned using the Lightspeed VCT scanner (GE Healthcare, Waukesha, WI). The data of each patient contains normal-dose and simulated low-dose projection images, normal-dose CT images, and the corresponding clinical information. We used data from 5 patients as the training set and data from 1 patient as the testing set. A scanned image of each patient includes about 300 2D slices, which can be found at the official website of TCIA. These 2D slices were then projected using a systematic matrix forward projection to generate corresponding sinogram data. Then the sinogram data was used to generate corresponding CT images with the MLEM algorithm. Figure 3 shows the schematic diagram of the experimental data generated using a slice. Figure 4 shows the dataset used to train and test the proposed method.

Taking one of the slices as an example, we show the data reconstruction in Figure 3:

1). For each CT slice of 512 × 512 size, interpolation is performed using resize function to obtain a CT slice of 256 × 256.

2). The above slice was then projected using a systematic matrix forward projection to generate corresponding sinogram data.

3). Finally, the CT image was generated using the above sinogram with the MLEM algorithm.

4.2.   Evaluation methods

To evaluate the effect of iNet, we compared the reconstructed image generated by iNet with by the three representative algorithms. They are the alternating direction method of multipliers (ADMM), filtered back projection (FBP) and MLEM.

4.3.   Quantitative analysis

To quantitatively compare different CT reconstruction methods, PSNR and SSIM are calculated of the reconstructed CT PSNR is defined as:

where ${u}_{max}$ is the maximum pixel value of the image, and MSE indicates the mean square error between the reconstructed result and the target image. SSIM is defined as:

where $x$ and $y$ represent the reconstructed image and the target image, ${u}_{x}$ and ${u}_{y}$ are the mean values of $x$, $y$, ${\sigma }_{x}$ and ${\sigma }_{y}$ are the standard deviations of $x$, $y$, respectively. ${c}_{1}$, ${c}_{2}$ are two very small constants to avoid the denominator to be zero. SSIM measures the similarity between two images, and the closer the value is to 1, the more similar the two images are to each other.

5.   Experimental results

We report and discuss the performance of our iNet according to experimental results. Figure 5 is the visualization of the three reconstructed images. The first column is the real image. The second, third, fourth, and last columns are the results generated by ADMM, FBP, MLEM, and the proposed iNet. The parts indicated by red and blue rectangle boxes represent regions of interest (ROIs) and the corresponding zoomed, respectively. The reconstructed image in Figure 5(b) generated by the ADMM algorithm has obvious blurring. The reconstructed images in Figure 5(c)-(e) have better performance. Compared with other methods mentioned in this work, the proposed method generated more reliable reconstructed images, and we also proved this conclusion with quantitative values.

We calculated the PSNR and SSIM of reconstructed images shown in Figure 5 and listed the results in Table 1. It is easy to find that the PSNR and SSIM values of the image reconstructed by iNet are the highest. Additionally, Figure 6-8 show the profiles of the reconstructed images shown in Figure 5. The blue curves correspond to the label (reference) and the orange curves correspond to the different reconstruction methods. It can be seen that the reconstructed images of our iNet are closer to the reference images.

6.   Discussion and conclusions

In this work, we proposed the iNet, which is inspired by the recently popular UD reconstruction method. Guided by the universal approximation theorem, we show that nonlinear functional relationships in iterative reconstruction algorithms can be simulated using a neural network model. The key to this framework is its potential to learn nonlinear maps in iterative reconstruction algorithms, which makes it suitable for extension to any iterative algorithm. In addition, the proposed method in this work brings new ideas to the research of interpretability of deep learning methods.

As a first attempt, we evaluated the performance of iNET on a clinical CT dataset. This improvement over standard ADMM, FBP, and MLEM is significant and promises to be used for further applications and theoretical studies, which we leave for future work. The applications include the reconstruction of larger size images, and smaller and sparser real data. The theoretical research includes systematizing such methods and proving them in mathematical terms in the future.

Acknowledgments

This work was supported by the Fundamental Research Funds for the Central Universities (N2019006 and N180719020).

Conflict of interest

The authors declare there is no conflict of interest.