基于轻型自限制注意力的结构光相位及深度估计混合网络

朱新军; 赵浩淼; 王红一; 宋丽梅; 孙瑞群

doi:10.37188/CO.2023-0066

Volume 17 Issue 1

Jan. 2024

Turn off MathJax

Article Contents

Chinese Optics > 2024 > 17(1): 118-127.

ZHU Xin-jun, ZHAO Hao-miao, WANG Hong-yi, SONG Li-mei, SUN Rui-qun. A hybrid network based on light self-limited attention for structured light phase and depth estimation[J]. Chinese Optics, 2024, 17(1): 118-127. doi: 10.37188/CO.2023-0066

Citation:

ZHU Xin-jun, ZHAO Hao-miao, WANG Hong-yi, SONG Li-mei, SUN Rui-qun. A hybrid network based on light self-limited attention for structured light phase and depth estimation[J]. Chinese Optics, 2024, 17(1): 118-127. doi: 10.37188/CO.2023-0066

Citation:

ZHU Xin-jun, ZHAO Hao-miao, WANG Hong-yi, SONG Li-mei, SUN Rui-qun. A hybrid network based on light self-limited attention for structured light phase and depth estimation[J]. Chinese Optics, 2024, 17(1): 118-127. doi: 10.37188/CO.2023-0066

PDF( 5444 KB)

A hybrid network based on light self-limited attention for structured light phase and depth estimation

doi: 10.37188/CO.2023-0066

School of Artificial Intelligence, Tiangong University, Tianjin 300387, China

Funds: Supported by National Natural Science Foundation of China (No. 61905178); Science & Technology Development Fund of Tianjin Education Commission for Higher Education (No. 2019KJ021)

More Information

Corresponding author: xinjunzhu@tiangong.edu.cn
Received Date: 14 Apr 2023
Rev Recd Date: 15 May 2023

Available Online: 18 Sep 2023

Abstract

Abstract

Phase retrieval and depth estimation are vital to three-dimensional measurement using structured light. Currently, conventional methods for structured light phase retrieval and depth estimation have limited efficiency and are lack of robustness in their results and so on. To improve the reconstruction effect of structured light by deep learning, we propose a hybrid network for structured light phase and depth estimation based on Light Self-Limited Attention (LSLA). Specifically, a CNN-Transformer hybrid module is constructed and integrated into a U-shaped structure to realize the advantages complementary of CNN and Transformer. The proposed network is experimentally compared with other networks in structured light phase estimation and structured light depth estimation. The experimental results indicate that the proposed network achieves finer detail processing in phase and depth estimation compared to other networks. Specifically, for structured light phase and depth estimation, its accuracy improves by 31% and 26%, respectively. Therefore, the proposed network improves the accuracy of deep neural networks in the structured light phase and depth estimation areas.
- structured light,
- deep learning,
- self-limited attention,
- phase estimation,
- depth estimation

FullText(HTML)

References(23)

References

[1]	左超, 张晓磊, 胡岩, 等. 3D真的来了吗?—三维结构光传感器漫谈[J]. 红外与激光工程,2020,49(3):0303001. doi: 10.3788/IRLA202049.0303001 ZUO CH, ZHANG X L, HU Y, et al. Has 3D finally come of age?——An introduction to 3D structured-light sensor[J]. Infrared and Laser Engineering, 2020, 49(3): 0303001. (in Chinese) doi: 10.3788/IRLA202049.0303001
[2]	王永红, 张倩, 胡寅, 等. 显微条纹投影小视场三维表面成像技术综述[J]. 中国光学,2021,14(3):447-457. doi: 10.37188/CO.2020-0199 WANG Y H, ZHANG Q, HU Y, et al. 3D small-field surface imaging based on microscopic fringe projection profilometry: a review[J]. Chinese Optics, 2021, 14(3): 447-457. (in Chinese) doi: 10.37188/CO.2020-0199
[3]	冯世杰, 左超, 尹维, 等. 深度学习技术在条纹投影三维成像中的应用[J]. 红外与激光工程,2020,49(3):0303018. doi: 10.3788/IRLA202049.0303018 FENG SH J, ZUO CH, YIN W, et al. Application of deep learning technology to fringe projection 3D imaging[J]. Infrared and Laser Engineering, 2020, 49(3): 0303018. (in Chinese) doi: 10.3788/IRLA202049.0303018
[4]	SU X Y, CHEN W J. Fourier transform profilometry: a review[J]. Optics and Lasers in Engineering, 2001, 35(5): 263-284. doi: 10.1016/S0143-8166(01)00023-9
[5]	ZHENG D L, DA F P, KEMAO Q, et al. Phase-shifting profilometry combined with Gray-code patterns projection: unwrapping error removal by an adaptive median filter[J]. Optics Express, 2017, 25(5): 4700-4713. doi: 10.1364/OE.25.004700
[6]	AN Y T, HYUN J S, ZHANG S. Pixel-wise absolute phase unwrapping using geometric constraints of structured light system[J]. Optics Express, 2016, 24(16): 18445-18459. doi: 10.1364/OE.24.018445
[7]	GHIGLIA D C, ROMERO L A. Robust two-dimensional weighted and unweighted phase unwrapping that uses fast transforms and iterative methods[J]. Journal of the Optical Society of America A, 1994, 11(1): 107-117. doi: 10.1364/JOSAA.11.000107
[8]	FENG SH J, CHEN Q, GU G H, et al. Fringe pattern analysis using deep learning[J]. Advanced Photonics, 2019, 1(2): 025001.
[9]	NGUYEN H, WANG Y Z, WANG ZH Y. Single-shot 3D shape reconstruction using structured light and deep convolutional neural networks[J]. Sensors, 2020, 20(13): 3718. doi: 10.3390/s20133718
[10]	VAN D J S, DIRCKX J J J. Deep neural networks for single shot structured light profilometry[J]. Optics Express, 2019, 27(12): 17091-17101. doi: 10.1364/OE.27.017091
[11]	张钊, 韩博文, 于浩天, 等. 多阶段深度学习单帧条纹投影三维测量方法[J]. 红外与激光工程,2020,49(6):20200023. doi: 10.3788/irla.12_2020-0023 ZHANG ZH, HAN B W, YU H T, et al. Multi-stage deep learning based single-frame fringe projection 3D measurement method[J]. Infrared and Laser Engineering, 2020, 49(6): 20200023. (in Chinese) doi: 10.3788/irla.12_2020-0023
[12]	RANFTL R, BOCHKOVSKIY A, KOLTUN V. Vision transformers for dense prediction[C]. Proceedings of IEEE/CVF International Conference on Computer Vision, IEEE, 2021.
[13]	YANG G L, TANG H, DING M L, et al. Transformer-based attention networks for continuous pixel-wise prediction[C]. Proceedings of 2021 IEEE/CVF International Conference on Computer Vision, IEEE, 2021.
[14]	QI F, ZHAI J Z, DANG G H. Building height estimation using Google Earth[J]. Energy and Buildings, 2016, 118: 123-132. doi: 10.1016/j.enbuild.2016.02.044
[15]	ZHU X J, HAN ZH Q, YUAN M K, et al. Hformer: hybrid CNN-transformer for fringe order prediction in phase unwrapping of fringe projection[J]. Optical Engineering, 2022, 61(9): 093107.
[16]	GENG J. Structured-light 3D surface imaging: a tutorial[J]. Advances in Optics and Photonics, 2011, 3(2): 128-160. doi: 10.1364/AOP.3.000128
[17]	GUO J Y, HAN K, WU H, et al. CMT: convolutional neural networks meet vision transformers[C]. Proceedings of 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition, IEEE, 2022.
[18]	CHEN ZH ZH, HANG W, ZHAO Y X. ViT-LSLA: Vision Transformer with Light Self-Limited-Attention[J]. arXiv:2210.17115.
[19]	RONNEBERGER O, FISCHER P, BROX T. U-Net: convolutional networks for biomedical image segmentation[C]. Proceedings of the 18th International Conference on Medical Image Computing and Computer-Assisted Intervention, Springer, 2015.
[20]	WANG L, LU D Q, QIU R W, et al. 3D reconstruction from structured-light profilometry with dual-path hybrid network[J]. EURASIP Journal on Advances in Signal Processing, 2022, 2022(1): 14. doi: 10.1186/s13634-022-00848-5
[21]	袁梦凯, 朱新军, 侯林鹏. 基于R2U-Net的单帧投影条纹图深度估计[J]. 激光与光电子学进展,2022,59(16):1610001. YUAN M K, ZHU X J, HOU L P. Depth estimation from single-frame fringe projection patterns based on R2U-Net[J]. Laser & Optoelectronics Progress, 2022, 59(16): 1610001. (in Chinese)
[22]	FAN CH M, LIU T J, LIU K H. SUNet: swin transformer UNet for image denoising[C]. Proceedings of 2022 IEEE International Symposium on Circuits and Systems, IEEE, 2022.
[23]	ZHU X J, ZHANG ZH ZH, HOU L P, et al. Light field structured light projection data generation with Blender[C]. Proceedings of 2022 3rd International Conference on Computer Vision, Image and Deep Learning & International Conference on Computer Engineering and Applications, IEEE, 2022.