Article Contents

Chinese Optics> 2023> Accepted Manuscript

ZHU Xin-jun, ZHAO Hao-miao, WANG Hong-yi, SONG Li-mei, SUN Rui-qun. Structured light depth and phase estimation with light self-limited attention for a hybrid network[J]. Chinese Optics. doi: 10.37188/CO.2023-0066

Citation:

PDF( 5480 KB)

Structured light depth and phase estimation with light self-limited attention for a hybrid network

doi:10.37188/CO.2023-0066

School of Artificial Intelligence, Tiangong University, Tianjin 300387, China

Funds:Supported by National Natural Science Foundation of China (No. 61905178); Science & Technology Development Fund of Tianjin Education Commission for Higher Education (No. 2019KJ021)

More Information

Corresponding author:xinjunzhu@tiangong.edu.cn
Available Online:18 Sep 2023

Abstract

Abstract

Phase retrieval and depth estimation are vital to three-dimensional measurement using structured light. Currently, conventional methods used for structured light analysis have limited efficiency and produce unreliable results. To enhance the reconstruction effect of structured light, this paper proposes a hybrid network for structured light phase and depth estimation based on Light Self-Limited Attention (LSLA). Specifically, a CNN-Transformer hybrid module is constructed and integrated into a U-shaped structure to harness the complementary benefits of CNN and Transformer. The network is assessed comparatively with other networks in tasks related to structured light phase estimation and depth estimation. The outcome of the experimental indicates that the suggested network achieves finer detail processing in phase and depth estimation compared to other networks. Specifically, in structured light phase anddepth estimation, its accuracy improves by 31% and 26%, respectively. Therefore, the proposed network improves the accuracy of deep neural networks in the aforementioned areas.
- Structured light,
- deep learning,
- self-attention mechanism,
- fringe analysis,
- depth estimation

FullText(HTML)

References (23)

References

[1]	左超, 张晓磊, 胡岩, 等. 3D真的来了吗?—三维结构光传感器漫谈[J]. 红外与工程,2020,49(3):0303001. doi:10.3788/IRLA202049.0303001 ZUO CH, ZHANG X L, HU Y, et al. Has 3D finally come of age?——An introduction to 3D structured-light sensor[J]. Infrared and Laser Engineering, 2020, 49(3): 0303001. (in Chinese) doi:10.3788/IRLA202049.0303001
[2]	王永红, 张倩, 胡寅, 等. 显微条纹投影小视场三维表面成像技术综述[J]. 中国光学,2021,14(3):447-457. doi:10.37188/CO.2020-0199 WANG Y H, ZHANG Q, HU Y, et al. 3D small-field surface imaging based on microscopic fringe projection profilometry: a review[J]. Chinese Optics, 2021, 14(3): 447-457. (in Chinese) doi:10.37188/CO.2020-0199
[3]	冯世杰, 左超, 尹维, 等. 深度学习技术在条纹投影三维成像中的应用[J]. 红外与工程,2020,49(3):0303018. doi:10.3788/IRLA202049.0303018 FENG SH J, ZUO CH, YIN W, et al. Application of deep learning technology to fringe projection 3D imaging[J]. Infrared and Laser Engineering, 2020, 49(3): 0303018. (in Chinese) doi:10.3788/IRLA202049.0303018
[4]	SU X Y, CHEN W J. Fourier transform profilometry: a review[J]. Optics and Lasers in Engineering, 2001, 35(5): 263-284. doi:10.1016/S0143-8166(01)00023-9
[5]	ZHENG D L, DA F P, KEMAO Q, et al. Phase-shifting profilometry combined with Gray-code patterns projection: unwrapping error removal by an adaptive median filter[J]. Optics Express, 2017, 25(5): 4700-4713. doi:10.1364/OE.25.004700
[6]	AN Y T, HYUN J S, ZHANG S. Pixel-wise absolute phase unwrapping using geometric constraints of structured light system[J]. Optics Express, 2016, 24(16): 18445-18459. doi:10.1364/OE.24.018445
[7]	GHIGLIA D C, ROMERO L A. Robust two-dimensional weighted and unweighted phase unwrapping that uses fast transforms and iterative methods[J]. Journal of the Optical Society of America A, 1994, 11(1): 107-117. doi:10.1364/JOSAA.11.000107
[8]	FENG SH J, CHEN Q, GU G H, et al. Fringe pattern analysis using deep learning[J]. Advanced Photonics, 2019, 1(2): 025001.
[9]	NGUYEN H, WANG Y Z, WANG ZH Y. Single-shot 3D shape reconstruction using structured light and deep convolutional neural networks[J]. Sensors, 2020, 20(13): 3718. doi:10.3390/s20133718
[10]	VAN DER JEUGHT S, DIRCKX J J J. Deep neural networks for single shot structured light profilometry[J]. Optics Express, 2019, 27(12): 17091-17101. doi:10.1364/OE.27.017091
[11]	张钊, 韩博文, 于浩天, 等. 多阶段深度学习单帧条纹投影三维测量方法[J]. 红外与工程,2020,49(6):20200023. doi:10.3788/irla.12_2020-0023 ZHANG ZH, HAN B W, YU H T, et al. Multi-stage deep learning based single-frame fringe projection 3D measurement method[J]. Infrared and Laser Engineering, 2020, 49(6): 20200023. (in Chinese) doi:10.3788/irla.12_2020-0023
[12]	RANFTL R, BOCHKOVSKIY A, KOLTUN V. Vision transformers for dense prediction[C]. Proceedings of IEEE/CVF International Conference on Computer Vision, IEEE, 2021.
[13]	YANG G L, TANG H, DING M L, et al. Transformer-based attention networks for continuous pixel-wise prediction[C]. Proceedings of 2021 IEEE/CVF International Conference on Computer Vision, IEEE, 2021.
[14]	QI F, ZHAI J Z, DANG G H. Building height estimation using Google Earth[J]. Energy and Buildings, 2016, 118: 123-132. doi:10.1016/j.enbuild.2016.02.044
[15]	ZHU X J, HAN ZH Q, YUAN M K, et al. Hformer: hybrid CNN-transformer for fringe order prediction in phase unwrapping of fringe projection[J]. arXiv:, 2112, 06759: 2021. (查阅网上资料, 请核对文献类型及格式). ZHU X J, HAN ZH Q, YUAN M K, et al . Hformer: hybrid CNN-transformer for fringe order prediction in phase unwrapping of fringe projection[J]. arXiv:, 2112 , 06759 : 2021 . (查阅网上资料, 请核对文献类型及格式).
[16]	GENG J. Structured-light 3D surface imaging: a tutorial[J]. Advances in Optics and Photonics, 2011, 3(2): 128-160. doi:10.1364/AOP.3.000128
[17]	GUO J Y, HAN K, WU H, et al. CMT: convolutional neural networks meet vision transformers[C]. Proceedings of 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition, IEEE, 2022.
[18]	ZHENZHE H C, WEI H. ViT-LSLA: Vision Transformer with Light Self-Limited-Attention[J]. Computer Science, 2022. (查阅网上资料, 请核对文献类型及格式). ZHENZHE H C, WEI H . ViT-LSLA: Vision Transformer with Light Self-Limited-Attention[J]. Computer Science, 2022 . (查阅网上资料, 请核对文献类型及格式).
[19]	RONNEBERGER O, FISCHER P, BROX T. U-Net: convolutional networks for biomedical image segmentation[C]. Proceedings of the 18th International Conference on Medical Image Computing and Computer-Assisted Intervention, Springer, 2015.
[20]	WANG L, LU D Q, QIU R W, et al. 3D reconstruction from structured-light profilometry with dual-path hybrid network[J]. EURASIP Journal on Advances in Signal Processing, 2022, 2022(1): 14. doi:10.1186/s13634-022-00848-5
[21]	袁梦凯, 朱新军, 侯林鹏. 基于R2U-Net的单帧投影条纹图深度估计[J]. 与光电子学进展,2022,59(16):1610001. YUAN M K, ZHU X J, HOU L P. Depth estimation from single-frame fringe projection patterns based on R2U-Net[J]. Laser & Optoelectronics Progress, 2022, 59(16): 1610001. (in Chinese)
[22]	FAN CH M, LIU T J, LIU K H. SUNet: swin transformer UNet for image denoising[C]. Proceedings of 2022 IEEE International Symposium on Circuits and Systems, IEEE, 2022.
[23]	ZHU X J, ZHANG ZH ZH, HOU L P, et al. Light field structured light projection data generation with Blender[C]. Proceedings of 2022 3rd International Conference on Computer Vision, Image and Deep Learning & International Conference on Computer Engineering and Applications, IEEE, 2022.