We present a joint model based on deep learning that is designed to inpaint the missing-wedge sinogram of electron tomography and reduce the residual artifacts in the reconstructed tomograms. Traditional methods, such as weighted back projection (WBP) and simultaneous algebraic reconstruction technique (SART), lack the ability to recover the unacquired project information as a result of the limited tilt range; consequently, the tomograms reconstructed using these methods are distorted and contaminated with the elongation, streaking, and ghost tail artifacts. To tackle this problem, we first design a sinogram filling model based on the use of Residual-in-Residual Dense Blocks in a Generative Adversarial Network (GAN). Then, we use a U-net structured Generative Adversarial Network to reduce the residual artifacts. We build a two-step model to perform information recovery and artifacts removal in their respective suitable domain. Compared with the traditional methods, our method offers superior Peak Signal to Noise Ratio (PSNR) and the Structural Similarity Index (SSIM) to WBP and SART; even with a missing wedge of 45°, our method offers reconstructed images that closely resemble the ground truth with nearly no artifacts. In addition, our model has the advantage of not needing inputs from human operators or setting hyperparameters such as iteration steps and relaxation coefficient used in TV-based methods, which highly relies on human experience and parameter fine turning.
Guanglei Ding and Yitong Liu contributed equally.
The reconstruction of tomography images or tomograms has great significances for physical, materials, medical sciences because it offers capabilities to investigate the internal structures of a non-transparent object without having to dissect or disrupt it. Tomography is performed by taking a series of projection images of a three-dimensional (3D) object around a fixed tilt axis to form a sinogram. By inverse Radon transform the obtained sinogram, a tomogram, i.e. the cross-sectional images, showing the density and morphological structure inside an object can be reconstructed. However, in many practical applications, it is difficult or not possible to obtain a complete set of projection images with full rotations from −180° to +180°, due to limitations on hardware conditions, radiation dose, or the state of the object being imaged. In transmission electron microscopes (TEM), for example, the distance between the electromagnetic lenses is only a few millimeters. Given the TEM samples are typically 3 mm in size, the limited space imposes a physical limitation on the tilt range. Even when a specialized high-tilt sample holder is used, projection images can only be recorded from −70° to +70° and projection information of a 40° tilt range are not accessible[
Graph: Figure 1The missing-wedge problem in electron tomography.
At the present, one of the major challenges in all tomography techniques, including electron tomography, is the incomplete or insufficient sampling in the angular/radon space, which makes the inversion problem mathematically ill-posed, i.e. there is insufficient number of linear equations to solve the linear algebraic problem, which leads to artifacts and reduction in reconstruction quality and resolution. To solve this problem, many methods have been proposed to mitigate the artifacts of inverse Radon transform or back projection. For example, the weighted back projection (WBP) method corrects back projection by applying a ramp filter that dumps the low-frequency information and enhances the high-frequency ones. WBP is an efficient and non-parameter method; however, it performs well only when there are sufficient projections available. When the angular sampling is sparse or there is a missing wedge, the WBP method introduces streaking, elongation and ghost tail artifacts. Improved upon WBP, simultaneous algebraic reconstruction technique (SART)[
In recent years, with the explosive development of deep neural networks, many creative algorithms based on deep learning have been developed in the computer vision field, such as image transformation, object detection, segmentation, edge detection, image restoration and sharpening[
Compared with other methods, GAN produces more realistic images with more details and higher image quality, especially in inpainting applications. In this paper, we introduce a two-step deep GAN model to tackle the missing-wedge problem. We first design a sinogram filling model based on the use of a super-resolution reconstruction GAN[
Graph: Figure 2The details of Original images, missing wedge reconstruction in WBP, SART, and our method.
In this section, we present the construction of the two-step joint model that can efficiently recover the missing-wedge of information without introducing visible artifacts in the reconstructed sinogram. We firstly present a sinogram filling network based on Residual in Residual Dense Block (RRDB)[
The working pipeline of the two-step model is shown in Fig. 3. For the missing wedge inpainting process, we perform Radon transform on a library of images to create sinograms with and without the missing wedge. The complete sinograms are used as the ground truth and the missing-wedge sinograms are used as the input of the inpainting model.
Graph: Figure 3Schematics of the entire working pipeline of the joint model proposed in this article.
For the training of the de-artifacts model, we collect reconstructed tomograms of the missing-wedge sinograms, the ground-truth sinograms, and the inpainted sinograms. We use them as the input of de-artifacts model and the original cross-sectional images as the ground truth to compute the loss.
Figure 4 shows the structure of inpainting GAN model. During training, the generator learns to generate inpainted sinograms that more and more resembles the ground-truth sinograms. We compute a part of the joint loss, mean square error (MSE), by using ground-truth sinograms and inpainted sinograms. The other part of the joint loss is GAN loss. For this GAN loss, there is another technique in used, which is called CGAN[
Graph: Figure 4The structure of the sinogram inpainting network.
To create a training library for the inpainting model, we first create a library of cross-sectional images. The library comprises of simulated images and images acquired from open datasets including ImageNet[
Image augmentation.
dataset\processing Pad Resize Radom Rotation Radom Flip Radom Affine Random Noise Size ImageNet √ √ 10,000 Random shape √ √ √ 15,000 MGH √ √ √ √ √ 15,000 NBIA √ √ √ √ √ 15,000
Sinograms with and without the missing wedge were created by Radon transforming the library of cross-sectional images (see Supplementary Materials for detail). Figure 5 shows examples of a brain CT image from the library (Fig. 5a), the ground-truth sinogram (Fig. 5b), and sinogram missing 45 degrees of projections Fig. 5c,d).
Graph: Figure 5The ground-truth sinogram and missing-wedge sinogram. (a) The cross-sectional brain image, (b) complete sinogram, (c) missing-wedge sinogram, (d) missing-wedge sinogram with the missing projections padded with zeros.
The generative model of the inpainting GAN is showed in Fig. 6. The generative network structure is mainly based on the RRDB model proposed by Xintao Wang[
Graph: Figure 6Inpainting model generator structure.
Our discriminator model uses a classic convolutional layer stack, while the difference is that dual feature extraction is used-the large receptive field slow path and the small receptive field fast path. These two paths are used to extract global and local image features, and to ensure the overall and local output quality of the model. However, due to the difference in width and height of the input data, the convolution kernel of the input layer is asymmetric. At the same time, we also use dilated convolution to increase the receptive field of the model that will give more gradient information to guide the generator. As for nominalization layer, we use Group Nominalization[
Graph: Figure 7Discriminator Structure details and convolution kernel information.
We use a joint loss function consisting of MSE and GAN loss. For GAN loss, we used the least squares GAN loss[
The total training epochs are 30. During the training, we set the ratio of the discriminative model and the generative model training frequency as 1:1. For the first three epochs, we set learning rates as 1e-4, 2e-4, and 4e-4, for both the generative and the discriminative models. Then the rate decays at the 20th and 28th epochs multiplied by 0.1. The optimizers and hyper-parameters are shown in Table 2. We set minibatch size 8 and using two Nvidia 1080TI GPUs. After each epoch training, we validate the training process by validation dataset, and then evaluate the SNR and SSIM score.
The training optimizer and hyper-parameters.
Model\hyper params Optimizer learning rate weight decay betas momentum alpha Generator Adam 4e-4 1e-4 (0.9, 0.999) / / Discriminator RMSprop 4e-4 1e-4 / 0 0.99
The inpainted sinogram is expected to provide improved reconstruction quality because of the recovery of the missing wedge of information. However, when we use WBP or SART to reconstruct the tomograms, there are still residual streaking and ghost tail artifacts in the reconstruction. The residual artifacts are a result of any small deviations of the inpainted sinogram from the ground truth. So, the goal of this model is to reduce the residual artifacts in the final tomogram.
Our training dataset consists of the following four subsets. The total size is 45000. We randomly choose 5000 samples as the verification dataset. The detail is shown in Table 3.
- Tomograms reconstructed from the missing-wedge sinograms using WBP. Size is 10,000.
- Tomograms reconstructed from the complete sinograms using WBP. Size is 7,500.
- Tomograms reconstructed from the missing-wedge sinograms using SART. Size is 7,500.
- Tomograms reconstructed from the output of inpainting model using WBP. Size is 20,000.
De-artifacts model training dataset.
Condition of sinogram WBP SART Missing wedge 10,000 7,500 Complete 7,500 / Inpainted 20000 /
The primary purpose of the de-artifacts model is to remove the artifacts yielding from the reconstruction process after filling the sinogram. So, the images generated via inpainting model is the core of the training data set. The inpainting models at different checkpoints are used to generate inpainted sinograms, followed by WBP reconstruction as shown in Fig. 8. By choosing a few different checkpoints of the inpainting model, we can obtain multiple different levels of inpainting effect (The later the checkpoint is, the stronger the inpainting effect is) of sinogram to improve the robustness of the de-artifact model.
Graph: Figure 8The training data of the denoising model.
A small number of missing-wedge sinograms are directly transformed by WBP and SART. It will generate slightly different artifact patterns to improve the robustness on the de-artifact model. We also included tomograms reconstructed by WBP from the complete sinograms. There are fewer artifacts in these tomograms. By using these images in the training dataset, over-de-artifacting can be prevented. In other words, the false positive rate is reduced. It will also prevent the model from overfitting.
The generation model in the denoising GAN is a standard U-net structure[
As for discriminator, it has a similar structure with the discriminator in inpainting GAN, as shown in Supplementary Fig. S2. This a standard convolution layer stack. We keep using dilated convolution and set dilation equal to two. Before the output layer, the last convolution layer uses Max pooling rather than Average pooling.
However, this time we replace Group Normalization with Batch Normalization. Because this model requires less memory. So, we can set much larger minibatch size to attenuate the noise yield from BN. The training details are in Supplementary Table S1.
Figure 9 shows the tomograms reconstructed by our joint model and other benchmarking methods. The result shows that our method readily fills the missing wedge of information and near perfectly reconstruct the image of random geometrical shapes. On the other hand, the missing wedge leads to prominent artifacts in the tomograms reconstructed by SART or WBP methods. It is worth noting that our method is capable of filling the missing wedge information up to the high spatial frequencies, which is partly lost in the SART reconstruction (Fig. 9).
Graph: Figure 9The comparing of reconstruction images and fast Fourier transformed (FFT) images. From left to right are the original image, Inpainting-de-artifacted (our method) image, complete sinogram with SART, missing-wedge sinogram with SART, missing-wedge sinogram with WBP.
Despite the outstanding performance of our method in reconstructing random geometrical shapes, some of the experimental images can be far more complex and details-abundant, and therefore to reconstruct these images requires the reconstruction algorithms to self-adapt to the sceneries such requirements renders conventional methods or even some of the state-of-the-art TVM methods ineffective. Figure 10 shows the comparison of reconstruction tomograms of such complex scenarios from ImageNet and MGH by our joint model and the benchmark methods. It is visually obvious that our method provides superior reconstruction results. The outstanding results suggest our model is highly robust and can self-adapt to different scenarios without having to choose hyperparameters which is a known to be the strength of deep GAN models.
Graph: Figure 10The comparing of reconstruction effect of complete sinogram (C_), missing-wedge sinogram (M_) in WBP, SART and our method.
To systematically evaluate the performance of our method compared with other reconstruction approaches, we investigate the Peak Signal to Noise Ratio (PSNR), the Structural Similarity Index (SSIM), and Perceptual Index (PI)[
Figure 11 is the PSNR vs. SSIM plot of the different methods[
Graph: Figure 11Plot of SSIM and PSNR. Upper right is better than lower left.
Peak Signal to Noise Ratio (PSNR) and Structural Similarity Index (SSIM) of the joint model and the benchmarking methods plotted in Fig. 11.
Method PSNR SSIM missing_wbp 13.07 0.2804 missing_sart 18.55 0.3124 missing_tvm 20.09 0.3283 complete_wbp 24.84 0.4499 complete_sart 0.5522 complete_tvm 24.81 0.7130
We also benchmarked out method using the Perceptual Index (PI) and Root Mean Square Error (RMSE). The perceptual quality is judged by the non-reference measurements of Ma's score[
Graph
In Fig. 12, our method has the lowest PI compared to that of other reconstruction methods in all conditions, and the quantitative numbers are listed in Table 5. It means that our method has the best perceptual quality in the reconstruction of missing-wedge sinograms, which is even better than the quality of SART reconstruction of the complete sinograms because both SART and WBP involve algebraic operation, leading to the artifacts that cannot be removed by themselves even with complete sinograms. However, our approach can easily eliminate the artifacts and achieve better perceptual image quality. As for RMSE, which represents the quantitative deviation of the reconstructed tomograms from the ground truth images, our method also shows outstanding performance, and the RMSE of our jointed model is only slightly higher than that of the SART reconstruction of the complete sinograms. In conclusion, by using quantitative measurements (e.g. PSNR/SSIM) and Perceptual Index, we show that that our joint model presents the highest perceptual reconstruction quality and a markable objective quality score among all the benchmarking reconstruction methods.
Graph: Figure 12Perceptual Index and RMSE of tomograms reconstructed by our joint model and the benchmarking methods from the missing-wedge sinograms (M_) and complete sinograms (C_).
Perceptual Index and RMSE of the joint model and the benchmarking methods plotted in Fig. 12.
Method Perceptual Index RMSE missing_tvm (M_tvm) 8.9767 25.2336 complete_tvm (C_tvm) 8.3549 18.2653 missing_sart (M_sart) 7.4593 29.0784 missing_wbp (M_wbp) 7.4409 50.6803 complete_wbp (C_wbp) 7.0195 21.453 complete_sart (C_sart) 7.0032
Further, we explore how these two models, the inpainting network and the de-artifacts network, work separately and jointly. We find that the inpainting process make the de-artifacts process more robust and easier to recover the details. As shown in Fig. 13, using only the de-artifacts model leads to blurred boundaries (show in the green boxes) and a poor intensity recovery (shown in the red boxes). For sinogram inpainting model alone, the reconstruction still has residual artifacts because the information filling is done in the sinogram space where the weighting of the errors is different from that of the tomogram space. But if two models work jointly, the inpainting output can make the de-artifacts process more robust both in terms of edge recovery and intensity accuracy (Fig. 13).
Graph: Figure 13Comparison of the reconstruction results of the de-artifacts, inpainting, and joint model.
Finally, our model is built based on simulated data. So, we tested our method using experimental data of gold nanorods and layered cathode materials. The results are shown in Fig. 14, even though these data have never been used in training, our joint model clearly outperforms other methods.
Graph: Figure 14Tomograms of gold nanorod and layered cathode material reconstructed by WBP, SART, TVM and the joint model.
The reconstruction artifacts of limited-tilt range tomography are largely due to loss of information in the missing wedge. The lost of information is also manifested in the sinogram—a range of projection information is unavailable making the tomography inverse problem ill-posed. In this paper, we show that the unacquried projection information can be effectively recovered in the sinogram domain using an inpainting GAN model through learning from thousands of sinograms. However, the imperfection of the inpainted information can still lead to artifacts. To fully resolve the problem, we designed a second GAN network that removes residual artifacts in the tomogram domain. By combining the two networks into a joint model, it achieves remarkable tomography reconstruction quality for missing-wedge sinograms with a missing angle as large as 45 degrees. The improved performance of our model stems from the fact that we decouple the problem into two separate domains. In each domain, a unique solution can be learned efficiently. In addition, our method is parameter free. Its performance is independent of parameters turning, prior knowledge, or the human operator's experience.
This research is supported by the University of California, Irvine.
H.L.X. conceived the idea and led the research. G.D., Y.L., R.Z., H.L.X. designed the model, performed the computational work, and wrote the manuscript.
The datasets generated and analyzed during the current study are available from the corresponding author on reasonable request.
The code generated during the current study are available from the corresponding author on reasonable request.
The authors declare no competing interests.
Graph: Supplementary info
Supplementary information accompanies this paper at 10.1038/s41598-019-49267-x.
Publisher's note: Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
By Guanglei Ding; Yitong Liu; Rui Zhang and Huolin L. Xin
Reported by Author; Author; Author; Author