基于插值超分辨的双目三维重建方法

刘宇豪; 吴福培; 吴树壮; 王瑞

doi:10.37188/CO.2023-0214

基于插值超分辨的双目三维重建方法

doi: 10.37188/CO.2023-0214

cstr: 32171.14.CO.2023-0214

汕头大学工学院机械工程系, 广东汕头 515063

基金项目: 国家自然科学基金（No. 61573233）；广东省自然科学基金（No. 2021A1515010661）；广东省普通高校创新团队资助项目（No. 2020KCXTD012）

详细信息

作者简介:
刘宇豪（1999—），男，重庆人，硕士，工程师，2024年于汕头大学机械工程专业获得硕士学位，现就职于联得自动化装备股份有限公司，担任视觉工程师。研究方向：双目视觉与三维测量。E-mail：1848266286@qq.com

吴福培（1980—），男，广西玉林人，博士，教授， 2009年于华南理工大学机械工程专业获得博士学位，现就职于汕头大学机械工程系，主要研究方向为自动光学检测和3D测量。E-mail：fpwu@stu.edu.cn

中图分类号: TP394.1
计量
- 文章访问数: 131
- HTML全文浏览量: 71
- PDF下载量: 77
- 被引次数: 0
出版历程
- 收稿日期: 2023-11-29
- 修回日期: 2024-01-09
- 网络出版日期: 2024-05-20

Binocular 3D reconstruction method based on interpolation super-resolution

Department of Mechanical Engineering, College of Engineering, Shantou University, Shantou 515063, China

Funds: Supported by National Natural Science Foundation of China (No. 61573233); National Natural Science Foundation of Guangdong, China (No. 2021A1515010661); the Guangdong Provincial University Innovation Team Project (No. 2020KCXTD012)

More Information

Corresponding author: fpwu@stu.edu.cn

摘要

摘要:
基于双目立体匹配重建物体表面三维形貌时，其匹配精度往往受限于传感器尺寸、镜头焦距和光源环境等物理条件。针对此问题，本文提出了一种基于插值超分辨的双目表面三维重建方法。首先，在图像预处理阶段，建立基于小波变换与双直方图均衡融合的图像增强方法，克服传统双目视觉受限于复杂环境光干扰等问题；其次，构建基于拉格朗日与三次多项式插值的超分辨算法，提升图像像素密度，为双目匹配代价计算阶段增加图像细节，从而提高匹配精度；最后，基于SLIC算法对目标图像进行分割，并针对各分割区域分别做二次曲面拟合，进而获得与物体实际表面更为贴合的高度曲线，从而降低重建误差并可提高重建精度。实验结果表明：5组测量样品的全局高度测量平均相对误差为±2.3%，实验平均测量时长为1.882 8 s，最大时长为1.936 2 s，较传统方法有明显提升。实验分析结果验证了本文所提方法的有效性。
- 双目视觉 /
- 三维测量 /
- 插值超分辨 /
- 超像素分割
Abstract:
The reconstruction of the three-dimensional surface morphology of objects based on binocular stereo matching is constrained by physical conditions such as sensor size, lens focal length, and environmental light. A binocular surface three-dimensional reconstruction method based on interpolation super-resolution is proposed in response to this issue. First, at the image preprocessing stage, an image enhancement method based on wavelet transform and dual histogram equalization fusion is established to overcome the problems of traditional binocular vision limited by complex environmental light interference. Second, a super-resolution algorithm based on Lagrange and cubic polynomial interpolation is constructed to increase the image’s pixel density and add image details to the binocular matching cost calculation stage, thereby improving the matching accuracy. Finally, a simple linear iterative clustering (SLIC) method is used to segment the target image, and a secondary surface fitting is performed for each segmented area to obtain a height curve that is more closely aligned with the actual surface of the object, which can reduce the reconstruction error and improve the reconstruction accuracy. The experimental results show that the average relative error of the global height measurement of 5 sets of measurement samples is ±2.3%, the average measurement time of the experiment is 1.8828 s, and the maximum time is 1.9362 s. This is a significant improvement over traditional methods. Experimental analysis results verify the effectiveness of the proposed algorithm.
- binocular vision /
- 3D measurement /
- interpolation super-resolution /
- superpixel segmentation

HTML全文

图 1 双目相机深度估计

Figure 1. Depth estimation by binocular cameras

下载: 全尺寸图片幻灯片

图 2 拉格朗日与三次多项式插值结合的算法框图

Figure 2. Block diagram of composed Lagrange and cubic interpolations algorithm

下载: 全尺寸图片幻灯片

图 3 拉格朗日与三次多项式插值组合图。（a）拉格朗日多项式内插；（b）三次多项式内插

Figure 3. Combination graph of Lagrange and cubic polynomial interpolations. (a) Lagrange polynomial interpolation; (b) cubic polynomial interpolation

下载: 全尺寸图片幻灯片

图 4 基于小波变换与双直方图均衡融合的图像增强算法流程

Figure 4. Flowchart of image enhancement algorithm based on wavelet transform and dual histogram equalization fusion

下载: 全尺寸图片幻灯片

图 5 图像增强实验结果。（a）原始图像；（b）直方图均衡结果；（c）小波变换结果；（d）本文算法结果

Figure 5. Experimental results of image enhancement. (a) Original images; (b) the results of histogram equalization; (c) the results of wavelet transform; (d) the results of the proposed algorithm

下载: 全尺寸图片幻灯片

图 6 采集图像增强结果。（a）采集图像；（b）增强图像

Figure 6. Enhanced results of acquired images. (a) Acquired images; (b) enhanced images

下载: 全尺寸图片幻灯片

图 7 （a）二维高斯分布图及（b）高斯加权卷积块

Figure 7. (a) Two-dimensional Gaussian distribution diagram and (b) the Gaussian weighted convolution block

下载: 全尺寸图片幻灯片

图 8 基于插值超分辨的邻域窗口像素值变换结果。（a）左视图邻域窗口；（b）右视图邻域窗口；（c）邻域窗口内像素值变换结果

Figure 8. Pixel value transformation results of neighborhood windows based on interpolation super-resolution. (a) Neighborhood windows of left image; (b) neighborhood windows of right image; (c) pixel value transformation results in neighborhood window

下载: 全尺寸图片幻灯片

图 9 SLIC算法结果。（a）采集图像；（b）超像素分割图；（c）边缘图像

Figure 9. Result of SLIC algorithm. (a) Captured image; (b) superpixel segmentation image; (c) edge image

下载: 全尺寸图片幻灯片

图 10 算法处理流程图

Figure 10. Algorithm processing flowchart

下载: 全尺寸图片幻灯片

图 11 （a）采集图像及（b）实验平台

Figure 11. (a) Acquiring images and (b) the experimental platform

下载: 全尺寸图片幻灯片

表 1 3种图像增强算法对标准图像的处理结果对比

Table 1. Comparison of processing results of three image enhancement algorithms with respect to the standard image

Name	Method	DE	NCC	CII	PSNR	SSIM
Cameraman	Histogram	6.7703	0.9848	0.8039	19.2229	0.6916
	Wavelet	4.4301	0.9888	0.7131	17.6452	0.4708
	Proposed algorithm	7.1149	0.9980	1.0132	24.5250	0.8788
Lena	Histogram	7.3383	0.9862	0.9051	19.3935	0.7784
	Wavelet	5.0992	0.9846	0.7341	17.0583	0.5613
	Proposed algorithm	7.4317	0.9921	1.0088	22.1287	0.8171
Barbara	Histogram	7.3816	0.9892	0.8492	18.2246	0.8147
	Wavelet	6.1788	0.9824	0.7005	17.9306	0.5767
	Proposed algorithm	7.5339	0.9908	1.0132	22.2251	0.8343

下载: 导出CSV

表 3 三维重建实验结果

Table 3. Experimental results of three-dimensional reconstruction

	Convex surface	Trapezoidal surface	Angular surface	Semicircular surface	Concave surface
Sample image
Acquired image
Reconstructed image

下载: 导出CSV

表 2 实验参数

Table 2. Experimental parameters

Parameter	Value
Sample size	16 mm×8 mm×6.5 mm
FOV of CCD camera	30 mm×20 mm
Image resolution	640 pixels×480 pixels
Focal length f	6 mm
Object distance	180 mm
Baseline distance	49.38 mm

下载: 导出CSV

表 4 3种重建方法的检测结果比较

Table 4. Comparison of measurement results for three reconstruction methods

Name of Curves	Method in this paper			Method in Ref. [27]			Method in Ref. [28]
Name of Curves	Maximum error/mm	Average error/mm	Detection time/s	Maximum error/mm	Average error/mm	Detection time/s	Maximum error/mm	Average error/mm	Detection time/s
Convex	0.2047	0.1464	1.8551	0.2756	0.2229	1.3287	0.3856	0.2684	2.5932
Trapezoidal	0.1468	0.1152	1.9173	0.4078	0.2615	1.3813	0.2378	0.1915	2.7367
Angular	0.1064	0.0768	1.8436	0.2178	0.1525	1.3102	0.1787	0.1425	2.4353
Semicircular	0.2538	0.1675	1.9362	0.3612	0.2717	1.3922	0.4101	0.3017	2.8577
Concave	0.1512	0.1278	1.8619	0.2766	0.2105	1.3247	0.2166	0.1705	2.6019
Average	0.1726	0.1267	1.8828	0.3078	0.2238	1.3474	0.2858	0.2149	2.6450

下载: 导出CSV

表 5 样本高度测量误差

Table 5. Sample height measurement error rates

Name of Curves	CD	IoU	MAE/(mm)	RMSE/(mm)
Convex	0.2634	0.6443	0.1150	0.2742
Trapezoidal	0.2317	0.6987	0.1050	0.2791
Angular	0.1448	0.7328	0.0882	0.2488
Semicircular	0.3163	0.6166	0.1403	0.4348
Concave	0.2215	0.7065	0.1057	0.2710
Average	0.2355	0.6798	0.1108	0.3016

下载: 导出CSV

map

参考文献(28)

[1]	WU F P, ZHU S K, YE W L. A single image 3D reconstruction method based on a novel monocular vision system[J]. Sensors, 2020, 20(24): 7045. doi: 10.3390/s20247045
[2]	NASRULLOH A V, WILLCOCKS C G, JACKSON P T G, et al. Multi-scale segmentation and surface fitting for measuring 3-D macular holes[J]. IEEE Transactions on Medical Imaging, 2018, 37(2): 580-589. doi: 10.1109/TMI.2017.2767908
[3]	WANG Y Y, ZHANG H T, XUE J SH, et al. Measurement of three-dimensional pavement morphology using binocular reconstruction improved by second-order fusion algorithm[J]. Automation in Construction, 2024, 157: 105152. doi: 10.1016/j.autcon.2023.105152
[4]	WANG Y Z, JIA H R, JIA P X, et al. A novel algorithm for three-dimensional shape reconstruction for microscopic objects based on shape from focus[J]. Optics & Laser Technology, 2024, 168: 109931.
[5]	DU F, WANG K, ZHANG G J, et al. Damage characteristics of coal under different loading modes based on CT three-dimensional reconstruction[J]. Fuel, 2022, 310: 122304. doi: 10.1016/j.fuel.2021.122304
[6]	尹仕斌, 任永杰, 刘涛, 等. 机器视觉技术在现代汽车制造中的应用综述[J]. 光学学报,2018,38(8):11-22. YIN SH B, REN Y J, LIU T, et al. Review on application of machine vision in modern automobile manufacturing[J]. Acta Optica Sinica, 2018, 38(8): 11-22. (in Chinese).
[7]	GUO X ZH, SHI ZH Y, YU B, et al. 3D measurement of gears based on a line structured light sensor[J]. Precision Engineering, 2020, 61: 160-169. doi: 10.1016/j.precisioneng.2019.10.013
[8]	SIDKI H M, AMER M. Evaluation of CMM for flatness measurements[J]. Metrology and Measurement Systems, 2008, 15(4): 585-593.
[9]	GUILLORY J, TRUONG D, WALLERAND J P, et al. An SI-traceable multilateration coordinate measurement system with half the uncertainty of a laser tracker[J]. Measurement Science and Technology, 2023, 34(6): 065016. doi: 10.1088/1361-6501/acc26a
[10]	LIN J R, XIN R K, SHI SH D, et al. An accurate 6-DOF dynamic measurement system with laser tracker for large-scale metrology[J]. Measurement, 2022, 204: 112052. doi: 10.1016/j.measurement.2022.112052
[11]	HUANG J Q, SUN M ZH, MA J J, et al. Super-resolution image reconstruction for high-density three-dimensional single-molecule microscopy[J]. IEEE Transactions on Computational Imaging, 2017, 3(4): 763-773. doi: 10.1109/TCI.2017.2699425
[12]	SHI CH Y, LUO X B, GUO J, et al. Three-dimensional intravascular reconstruction techniques based on intravascular ultrasound: a technical review[J]. IEEE Journal of Biomedical and Health Informatics, 2018, 22(3): 806-817. doi: 10.1109/JBHI.2017.2703903
[13]	WANG ZH ZH. Robust three-dimensional face reconstruction by one-shot structured light line pattern[J]. Optics and Lasers in Engineering, 2020, 124: 105798. doi: 10.1016/j.optlaseng.2019.105798
[14]	张新荣, 王鑫, 王瑶, 等. 基于转动式二维扫描仪和多传感器的三维重建方法[J]. 中国光学(中英文),2023,16(3):663-672. doi: 10.37188/CO.2022-0159 ZHANG X R, WANG X, WANG Y, et al. 3D reconstruction method based on a rotating 2D laser scanner and multi-sensor[J]. Chinese Optics, 2023, 16(3): 663-672. (in Chinese). doi: 10.37188/CO.2022-0159
[15]	BEVACQUA M T, SCAPATICCI R. A compressive sensing approach for 3D breast cancer microwave imaging with magnetic nanoparticles as contrast agent[J]. IEEE Transactions on Medical Imaging, 2016, 35(2): 665-673. doi: 10.1109/TMI.2015.2490340
[16]	SUN SH L, WANG X Y, CHEN Y J. Retracted: the application value of CT three-dimensional microscope reconstruction technology in the diagnosis of cervical cancer[J]. Scanning, 2022, 2022: 5648195.
[17]	WANG ZH ZH, ZHOU Q, SHUANG Y C. Three-dimensional reconstruction with single-shot structured light dot pattern and analytic solutions[J]. Measurement, 2020, 151: 107114. doi: 10.1016/j.measurement.2019.107114
[18]	MOONEY S W J, MARLOW P J, ANDERSON B L. The perception and misperception of optical defocus, shading, and shape[J]. eLife, 2019, 8: e48214.
[19]	TIAN X, LIU R, WANG ZH Y, et al. High quality 3D reconstruction based on fusion of polarization imaging and binocular stereo vision[J]. Information Fusion, 2022, 77: 19-28.
[20]	LIU L, ZHU P, GUAN J X, et al. A binocular reconstruction method fused with laplacian image information for pavement texture evaluation[J]. Measurement, 2021, 185: 110039. doi: 10.1016/j.measurement.2021.110039
[21]	LI Y, FU Y J, ZHONG K J, et al. A virtual binocular line-structured light measurement method based on a plane mirror[J]. Optics Communications, 2022, 510: 127974.
[22]	TSENG K K, ZHANG Y Q, ZHU Q L, et al. Semi-supervised image depth prediction with deep learning and binocular algorithms[J]. Applied Soft Computing, 2020, 92: 106272. doi: 10.1016/j.asoc.2020.106272
[23]	BOUKHTACHE S, BLAYSAT B, GRÉDIAC M, et al. Alternatives to bicubic interpolation considering FPGA hardware resource consumption[J]. IEEE Transactions on Very Large Scale Integration (VLSI) Systems, 2021, 29(2): 247-258. doi: 10.1109/TVLSI.2020.3032888
[24]	李佳星, 赵勇先, 王京华. 基于深度学习的单幅图像超分辨率重建算法综述[J]. 自动化学报,2021,47(10):2341-2363. LI J X, ZHAO Y X, WANG J H. A review of single image super-resolution reconstruction algorithms based on deep learning[J]. Acta Automatica Sinica, 2021, 47(10): 2341-2363. (in Chinese).
[25]	刘艳, 李庆武, 霍冠英, 等. 结合局部二进制表示和超像素分割求精的立体匹配[J]. 光学学报,2018,38(6):0615003. doi: 10.3788/AOS201838.0615003 LIU Y, LI Q W, HUO G Y, et al. Local binary description combined with superpixel segmentation refinement for stereo matching[J]. Acta Optica Sinica, 2018, 38(6): 0615003. (in Chinese). doi: 10.3788/AOS201838.0615003
[26]	冯维, 徐仕楠, 王恒辉, 等. 逐像素调制的高反光表面三维测量方法[J]. 中国光学,2022,15(3):488-497. doi: 10.37188/CO.2021-0220 FENG W, XU SH N, WANG H H, et al. Three-dimensional measurement method of highly reflective surface based on per-pixel modulation[J]. Chinese Optics, 2022, 15(3): 488-497. (in Chinese). doi: 10.37188/CO.2021-0220
[27]	吴福培, 余冠霖, 乐聪, 等. 微型相机模组贴装过程画胶表面三维测量方法[J]. 光子学报,2022,51(4):0412006. doi: 10.3788/gzxb20225104.0412006 WU F P, YU G L, LE C, et al. Three-dimensional measurement method of glue surface during the mounting process of micro-camera module[J]. Acta Photonica Sinica, 2022, 51(4): 0412006. (in Chinese). doi: 10.3788/gzxb20225104.0412006
[28]	平茜茜, 刘勇, 董欣明, 等. 基于偏振双目视觉的无纹理高反光目标三维重构[J]. 红外与毫米波学报,2017,36(4):432-438. PING X X, LIU Y, DONG X M, et al. 3-D reconstruction of textureless and high-reflective target by polarization and binocular stereo vision[J]. Journal of Infrared and Millimeter Waves, 2017, 36(4): 432-438. (in Chinese).