基于残差网络的结直肠内窥镜图像超分辨率重建方法 -

姓名
邮箱
手机号码
标题
留言内容
验证码

基于残差网络的结直肠内窥镜图像超分辨率重建方法

doi:10.37188/CO.2022-0247

郑跃坤^{1, 2,},
葛明锋^2,,,
常智敏²,
董文飞^{1, 2,}

1.
中国科学技术大学生物医学工程学院(苏州)生命科学与医学部, 安徽合肥 230026
2.
中国科学院苏州生物医学工程技术研究所, 江苏苏州 215163

基金项目:国家重点研发计划（No. 2021YFB3602200）；苏州市科技计划项目（No. SZS201903）

详细信息

作者简介:
郑跃坤（1998—），男，广东汕头人，硕士研究生，2020年于中山大学获得学士学位，主要从事图像处理、深度学习等方面的研究。E-mail：645352858@qq.com
葛明锋（1987—），男，江苏南通人，博士，副研究员，硕士生导师，主要从事高光谱、荧光显微成像方面研究。E-mail：gemf@sibet.ac.cn
董文飞（1975—），男，吉林长春人，博士，研究员，博士生导师，1999年于中国科学院长春应用化学研究所获得高分子物理化学专业硕士学位，2004年于德国马普胶体界面所和波兹坦大学获得自然科学博士学位，主要从事纳米材料和技术在生物医用光子学领域的应用基础研究。E-mail:wenfeidong@126.com

中图分类号:TP391.41
计量
- 文章访问数:278
- HTML全文浏览量:191
- PDF下载量:180
- 被引次数:0
出版历程
- 收稿日期:2022-11-29
- 录用日期:2023-03-15
- 修回日期:2022-12-23
- 网络出版日期:2023-04-04

Super-resolution reconstruction for colorectal endoscopic images based on a residual network

1.
School of Biomedical Engineering (Suzhou), Division of Life Sciences and Medicine, University of Science and Technology of China, Hefei 230026, China
2.
Suzhou Institute of Biomedical Engineering and Technology, Chinese Academy of Science, Suzhou 215163, China

Funds:Supported by the National Key R & D Program of China (No. 2021YFB3602200); Suzhou Science and Technology Plan Project (No.SZS201903)

More Information

Corresponding author:gemf@sibet.ac.cn

摘要

摘要:
针对结直肠镜图像分辨率偏低、纹理信息偏少和细节模糊等缺点，提出了一种基于残差注意力网络的图像超分辨率重建算法SMRAN，选取结直肠息肉内窥镜图像数据集PolypsSet中的部分图像作为原始数据进行实验。首先，使用卷积网络提取低分辨率图像的浅层特征；其次，设计Res-Sobel结构对图像边缘特征进行增强；然后，通过引入不同大小的卷积核，设计多尺度特征融合模块(Multi-Scale feature Extraction Block, MEB)，自适应地提取不同尺度的特征，从而得到有效的图像信息，并通过残差注意力网络将Res-Sobel模块和多尺度特征融合模块MEB进行连接；最后，通过亚像素卷积层对图像进行重建，得到最终的高分辨率图像。在尺度因子为×4时，网络在测试集上的测试结果如下: 峰值信噪比PSNR为34.25 dB，结构相似性SSIM为0.8675。实验结果表明，与传统的双三次插值算法及常用的SRCNN、RCAN等深度学习算法相比，本文提出的SMRAN对结直肠内窥镜图像具有更好的超分辨率重建效果。
- 内窥镜图像/
- 超分辨率重建/
- 残差结构/
- 注意力机制/
- 多尺度特征融合/
- 索贝尔算子
Abstract:
In this paper, an image super-resolution reconstruction multi-scale algorithm based on a residual attention network (SMRAN) is proposed to solve the problems caused by low resolutions, less texture information and blurred details in colorectal endoscopic images. Images from the colorectal polyp endoscope image dataset PolypsSet are selected as the raw data for these experiments. A convolutional network is built to extract the shallow features of the low-resolution image and a Res-Sobel block is designed to enhance its edge features. A multi-scale feature fusion block MEB is designed by introducing convolution kernels of different sizes to adaptively extract image features of different scales and obtain effective image information. The Res-Sobel block and multi-scale feature fusion module block MEB are connected through the residual attention network. Finally, a high-resolution image is reconstructed at the sub-pixel convolution layer. When the amplification factor is ×4, the performance of the proposed algorithm on the test set are as follows: the peak signal-to-noise ratio (PSNR) is 34.25 dB and the structural similarity (SSIM) is 0.8675. Compared with the traditional bicubic interpolation algorithm and commonly used deep learning algorithms such as SRCNN and RCAN, the proposed SMRAN algorithm shows better super-resolution reconstruction results on colorectal endoscopic images.
- endoscopic image/
- super-resolution reconstruction/
- residual structure/
- attention mechanism/
- multi-scale feature extraction/
- Sobel operator

HTML全文

图 1SMRAN架构

Figure 1.Architecture of SMRAN

下载: 全尺寸图片幻灯片

图 2Res-Sobel模块

Figure 2.Res-Sobel Block

下载: 全尺寸图片幻灯片

图 3多尺度特征融合模块(MEB)

Figure 3.Multi-scale feature extraction block(MEB)

下载: 全尺寸图片幻灯片

图 4CBAM注意力机制

Figure 4.CBAM attention mechanism

下载: 全尺寸图片幻灯片

图 5SMRAB结构

Figure 5.SMRAB Structure

下载: 全尺寸图片幻灯片

图 6验证集的PSNR曲线

Figure 6.PSNR curve of validation set

下载: 全尺寸图片幻灯片

图 7验证集的SSIM曲线

Figure 7.SSIM curve of validation set

下载: 全尺寸图片幻灯片

图 8采用不同超分辨率算法的结直肠息肉内窥图像的重建效果对比图

Figure 8.Comparison of reconstruction effects of endoscopic images of colorectal polyps using different super-resolution algorithms

下载: 全尺寸图片幻灯片

图 9SMRAN模型对光学分辨率的提升效果

Figure 9.Improvement of optical resolution by SMRAN model

下载: 全尺寸图片幻灯片

表 1测试集上不同算法的PSNR值

Table 1.PSNR values of different algorithms on the testing set (Unit: dB)

算法	PSNR(dB)
算法	×2	×3	×4
Bicubic	33.85	31.94	29.91
SRCNN	36.70	34.53	32.04
FSRCNN	37.63	35.22	32.23
EDSR	37.34	35.25	32.13
ESPCN	36.75	34.78	31.38
RCAN	39.04	35.63	33.86
本文算法	39.69	36.92	34.25

下载: 导出CSV

表 2测试集上不同算法的SSIM值

Table 2.SSIM values of different algorithms on the testing set

算法	SSIM
算法	×2	×3	×4
Bicubic	0.9121	0.8824	0.8103
SRCNN	0.9400	0.8983	0.8642
FSRCNN	0.9382	0.9132	0.8660
EDSR	0.9325	0.9158	0.8401
ESPCN	0.9392	0.9003	0.8566
RCAN	0.9483	0.9182	0.8667
本文算法	0.9559	0.9249	0.8675

下载: 导出CSV

表 3不同损失函数的PSNR和SSIM值

Table 3.PSNR and SSIM values for different loss functions

损失函数	PSNR(dB)	SSIM
${L_1}$	34.27	0.8664
${L_1}$+MS_SSIM	34.25	0.8675

下载: 导出CSV

表 4各模块对性能的影响

Table 4.The impact of each module on performance

Res-Sobel Block	MEB	CBAM	PSNR(dB)/SSIM
	√	√	33.33/0.8577
√		√	33.39/0.8601
√	√		33.47/0.8609
√	√	√	34.25/0.8675

下载: 导出CSV

表 5Kvasir-SEG数据集上不同算法的PSNR值

Table 5.PSNR values of different algorithms on the Kvasir-SEG dataset (Unit: dB)

算法	PSNR(dB)
算法	×2	×3	×4
Bicubic	37.35	33.86	31.78
SRCNN	39.98	35.98	33.23
FSRCNN	40.62	36.57	33.68
EDSR	40.94	36.69	33.99
ESPCN	39.71	35.91	33.01
RCAN	41.58	37.62	34.23
本文算法	41.80	37.81	34.56

下载: 导出CSV

表 6Kvasir-SEG数据集上不同算法的SSIM值

Table 6.SSIM values of different algorithms on the Kvasir-SEG dataset

算法	SSIM
算法	×2	×3	×4
Bicubic	0.9776	0.9469	0.9079
SRCNN	0.9833	0.9658	0.9242
FSRCNN	0.9859	0.9672	0.9256
EDSR	0.9867	0.9669	0.9219
ESPCN	0.9886	0.9710	0.9396
RCAN	0.9859	0.9702	0.9447
本文算法	0.9884	0.9714	0.9456

下载: 导出CSV

参考文献 (28)

[1]	刘爽, 田兆星, 李浩然, 等. 一种基于改进YOLOv5s网络的结直肠腺瘤实时检测方法[J]. 河北大学学报(自然科学版),2022,42(3):327-336. LIU SH, TIAN ZH X, LI H R,et al. A real-time method for colorectal adenoma detection based on an improved YOLOv5s network[J].Journal of Hebei University(Natural Science Edition), 2022, 42(3): 327-336. (in Chinese)
[2]	白瑞峰, 江山, 孙海江, 等. 基于编码解码结构的微血管减压图像实时语义分割[J]. 中国光学(中英文),2022,15(5):1055-1065.doi:10.37188/CO.2022-0120 BAI R F, JIANG SH, SUN H J,et al. Real-time semantic segmentation of microvascular decompression images based on encoder-decoder structure[J].Chinese Optics, 2022, 15(5): 1055-1065. (in Chinese)doi:10.37188/CO.2022-0120
[3]	陈晓冬, 艾大航, 张佳琛, 等. Gabor滤波融合卷积神经网络的路面裂缝检测方法[J]. 中国光学,2020,13(6):1293-1301.doi:10.37188/CO.2020-0041 CHEN X D, AI D H, ZHANG J CH,et al. Gabor filter fusion network for pavement crack detection[J].Chinese Optics, 2020, 13(6): 1293-1301. (in Chinese)doi:10.37188/CO.2020-0041
[4]	左超, 陈钱. 分辨率、超分辨率与空间带宽积拓展——从计算光学成像角度的一些思考[J]. 中国光学(中英文),2022,15(6):1105-1166.doi:10.37188/CO.2022-0105 ZUO CH, CHEN Q. Resolution, super-resolution and spatial bandwidth product expansion——some thoughts from the perspective of computational optical imaging[J].Chinese Optics, 2022, 15(6): 1105-1166. (in Chinese)doi:10.37188/CO.2022-0105
[5]	GU X G, ZHOU F X, CHEN R F,et al. . Endoscopic single image super-resolution based on transformer and convolutional neural network[C].7th International Conference on Intelligent Life System Modelling, Image Processing and Analysis, Springer, 2021: 24-32.
[6]	YANG X R, CHEN Y, TAO R,et al. . Endoscopic image deblurring and super-resolution reconstruction based on deep learning[C].2020 International Conference on Artificial Intelligence and Computer Engineering (ICAICE), IEEE, 2020: 168-172.
[7]	TURAN M. A generative adversarial network based super-resolution approach for capsule endoscopy images[J].Medicine Science, 2021, 10(3): 1002-1007.doi:10.5455/medscience.2021.06.218
[8]	TAŞ M, YILMAZ B. Super resolution convolutional neural network based pre-processing for automatic polyp detection in colonoscopy images[J].Computers&Electrical Engineering, 2021, 90: 106959.
[9]	DONG CH, LOY C C, HE K M,et al. Image super-resolution using deep convolutional networks[J].IEEE Transactions on Pattern Analysis and Machine Intelligence, 2016, 38(2): 295-307.doi:10.1109/TPAMI.2015.2439281
[10]	雷茂, 郭锋, 秦明伟. 基于改进POCS算法的太赫兹图像超分辨率重建[J]. 传感器与微系统,2022,41(3):122-125.doi:10.13873/J.1000-9787(2022)03-0122-04 LEI M, GUO F, QIN M W. Super-resolution reconstruction for terahertz images based on improved POCS algorithm[J].Transducer and Microsystem Technologies, 2022, 41(3): 122-125. (in Chinese)doi:10.13873/J.1000-9787(2022)03-0122-04
[11]	孙琰玥, 何小海, 陈为龙. 小波局部适应插值的图像超分辨率重建[J]. 计算机工程,2010,36(13):183-185.doi:10.3969/j.issn.1000-3428.2010.13.065 SUN Y Y, HE X H, CHEN W L. Image super-resolution reconstruction of wavelet local-adaptation interpolation[J].Computer Engineering, 2010, 36(13): 183-185. (in Chinese)doi:10.3969/j.issn.1000-3428.2010.13.065
[12]	庞博, 张旭东, 徐小红. 自适应图像插值在超分辨率图像重建中的应用[J]. 合肥工业大学学报(自然科学版),2006,29(7):825-829. PANG B, ZHANG X D, XU X H. Super-resolution image reconstruction using adaptive interpolation method[J].Journal of Hefei University of Technology(Natural Science), 2006, 29(7): 825-829. (in Chinese)
[13]	陈楠, 张标. 多尺度半耦合卷积稀疏编码的遥感影像超分辨率重建[J]. 计算机辅助设计与图形学学报,2022,34(3):382-391. CHEN N, ZHANG B. Multi-scale semi-coupled convolutional sparse coding for the super-resolution reconstruction of remote sensing image[J].Journal of Computer-Aided Design&Computer Graphics, 2022, 34(3): 382-391. (in Chinese)
[14]	李方玗, 贾晓芬, 赵佰亭, 等. 高效多注意力特征融合的图像超分辨率重建算法[J/OL]. 小型微型计算机系统: 1-9[2022-11-25]. http://kns.cnki.net/kcms/detail/21.1106.TP.20221111.1852.006.html. LI F Y, JIA X F, ZHAO B T,et al. . Efficient multi-attention feature fusion for image super-resolution reconstruction algorithms[J/OL].Journal of Chinese Computer Systems: 1-9[2022-11-25]. http://kns.cnki.net/kcms/detail/21.1106.TP.20221111.1852.006.html. (in Chinese)
[15]	林敏强, 赵曈, 祝明欣. 基于残差注意力的合成孔径雷达图像超分辨率算法[J]. 电子产品世界,2022,29(11):33-34. LIN M Q, ZHAO T, ZHU M X. SAR image super-resolution algorithm based on residual attention[J].Electronic Engineering&Product World, 2022, 29(11): 33-34. (in Chinese)
[16]	须颖, 刘帅, 邵萌, 等. 一种多尺度GAN的低剂量CT超分辨率重建方法[J]. 西安电子科技大学学报,2022,49(2):228-236.doi:10.19665/j.issn1001-2400.2022.02.026 XU Y, LIU SH, SHAO M,et al. Multi-scale generation antagonistic network for the low-dose CT images super-resolution reconstruction algorithm[J].Journal of Xidian University, 2022, 49(2): 228-236. (in Chinese)doi:10.19665/j.issn1001-2400.2022.02.026
[17]	LIM B, SON S, KIM H,et al. . Enhanced deep residual networks for single image super-resolution[C].Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops, IEEE, 2017: 1132-1140.
[18]	KIM J, LEE J K, LEE K M. Accurate image super-resolution using very deep convolutional networks[C].Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, IEEE, 2016: 1646-1654.
[19]	ZHANG Y L, LI K P, LI K,et al. . Image super-resolution using very deep residual channel attention networks[C].Proceedings of the 15th European Conference on Computer Vision, Springer, 2018: 294-310.
[20]	LEDIG C, THEIS L, HUSZÁR F,et al. . Photo-realistic single image super-resolution using a generative adversarial network[C].Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, IEEE, 2017: 105-114.
[21]	SIMONYAN K, ZISSERMAN A. Very deep convolutional networks for large-scale image recognition[C].3rd International Conference on Learning Representations, ICLR, 2015.
[22]	王玉玺, 陈健美. 基于自适应Canny算子和多方向Sobel算子的虹膜边缘检测算法[J]. 计算机与数字工程,2020,48(11):2744-2749.doi:10.3969/j.issn.1672-9722.2020.11.040 WANG Y X, CHEN J M. Iris edge detection algorithm based on adaptive canny operator and multi-directional Sobel operator[J].Computer&Digital Engineering, 2020, 48(11): 2744-2749. (in Chinese)doi:10.3969/j.issn.1672-9722.2020.11.040
[23]	LIU S R, TANG X Y, WANG D. Facial expression recognition based on Sobel operator and improved CNN-SVM[C].2020 IEEE 3rd International Conference on Information Communication and Signal Processing (ICICSP), IEEE, 2020: 236-240.
[24]	HAO F, XU D SH, CHEN D L,et al. Sobel operator enhancement based on eight-directional convolution and entropy[J].International Journal of Information Technology, 2021, 13(5): 1823-1828.doi:10.1007/s41870-021-00770-3
[25]	WOO S, PARK J, LEE J Y,et al. . CBAM: convolutional block attention module[C].Proceedings of the 15th European Conference on Computer Vision (ECCV), Springer, 2018: 3-19.
[26]	SHI W ZH, CABALLERO J, HUSZÁR F,et al. . Real-time single image and video super-resolution using an efficient sub-pixel convolutional neural network[C].Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, IEEE, 2016: 1874-1883.
[27]	LI K D, FATHAN M I, PATEL K,et al. Colonoscopy polyp detection and classification: Dataset creation and comparative evaluations[J].PLoS One, 2021, 16(8): e0255809.doi:10.1371/journal.pone.0255809
[28]	JHA D, SMEDSRUD P H, RIEGLER M A,et al. . Kvasir-SEG: a segmented polyp dataset[C].26th International Conference on Multimedia Modeling, Springer, 2020: 451-462.