留言板

尊敬的读者、作者、审稿人, 关于本刊的投稿、审稿、编辑和出版的任何问题, 您可以本页添加留言。我们将尽快给您答复。谢谢您的支持!

姓名
邮箱
手机号码
标题
留言内容
验证码

基于插值超分辨的双目三维重建方法

刘宇豪 吴福培 吴树壮 王瑞

刘宇豪, 吴福培, 吴树壮, 王瑞. 基于插值超分辨的双目三维重建方法[J]. 中国光学(中英文), 2024, 17(4): 862-874. doi: 10.37188/CO.2023-0214
引用本文: 刘宇豪, 吴福培, 吴树壮, 王瑞. 基于插值超分辨的双目三维重建方法[J]. 中国光学(中英文), 2024, 17(4): 862-874. doi: 10.37188/CO.2023-0214
LIU Yu-hao, WU Fu-pei, WU Shu-zhuang, WANG Rui. Binocular 3D reconstruction method based on interpolation super-resolution[J]. Chinese Optics, 2024, 17(4): 862-874. doi: 10.37188/CO.2023-0214
Citation: LIU Yu-hao, WU Fu-pei, WU Shu-zhuang, WANG Rui. Binocular 3D reconstruction method based on interpolation super-resolution[J]. Chinese Optics, 2024, 17(4): 862-874. doi: 10.37188/CO.2023-0214

基于插值超分辨的双目三维重建方法

cstr: 32171.14.CO.2023-0214
基金项目: 国家自然科学基金(No. 61573233);广东省自然科学基金(No. 2021A1515010661);广东省普通高校创新团队资助项目(No. 2020KCXTD012)
详细信息
    作者简介:

    刘宇豪(1999—),男,重庆人,硕士,工程师,2024年于汕头大学机械工程专业获得硕士学位,现就职于联得自动化装备股份有限公司,担任视觉工程师。研究方向:双目视觉与三维测量。E-mail:1848266286@qq.com

    吴福培(1980—),男,广西玉林人,博士,教授, 2009年于华南理工大学机械工程专业获得博士学位,现就职于汕头大学机械工程系,主要研究方向为自动光学检测和3D测量。E-mail:fpwu@stu.edu.cn

  • 中图分类号: TP394.1

Binocular 3D reconstruction method based on interpolation super-resolution

Funds: Supported by National Natural Science Foundation of China (No. 61573233); National Natural Science Foundation of Guangdong, China (No. 2021A1515010661); the Guangdong Provincial University Innovation Team Project (No. 2020KCXTD012)
More Information
  • 摘要:

    基于双目立体匹配重建物体表面三维形貌时,其匹配精度往往受限于传感器尺寸、镜头焦距和光源环境等物理条件。针对此问题,本文提出了一种基于插值超分辨的双目表面三维重建方法。首先,在图像预处理阶段,建立基于小波变换与双直方图均衡融合的图像增强方法,克服传统双目视觉受限于复杂环境光干扰等问题;其次,构建基于拉格朗日与三次多项式插值的超分辨算法,提升图像像素密度,为双目匹配代价计算阶段增加图像细节,从而提高匹配精度;最后,基于SLIC算法对目标图像进行分割,并针对各分割区域分别做二次曲面拟合,进而获得与物体实际表面更为贴合的高度曲线,从而降低重建误差并可提高重建精度。实验结果表明:5组测量样品的全局高度测量平均相对误差为±2.3%,实验平均测量时长为1.882 8 s,最大时长为1.936 2 s,较传统方法有明显提升。实验分析结果验证了本文所提方法的有效性。

     

  • 图 1  双目相机深度估计

    Figure 1.  Depth estimation by binocular cameras

    图 2  拉格朗日与三次多项式插值结合的算法框图

    Figure 2.  Block diagram of composed Lagrange and cubic interpolations algorithm

    图 3  拉格朗日与三次多项式插值组合图。(a)拉格朗日多项式内插;(b)三次多项式内插

    Figure 3.  Combination graph of Lagrange and cubic polynomial interpolations. (a) Lagrange polynomial interpolation; (b) cubic polynomial interpolation

    图 4  基于小波变换与双直方图均衡融合的图像增强算法流程

    Figure 4.  Flowchart of image enhancement algorithm based on wavelet transform and dual histogram equalization fusion

    图 5  图像增强实验结果。(a)原始图像;(b)直方图均衡结果;(c)小波变换结果;(d)本文算法结果

    Figure 5.  Experimental results of image enhancement. (a) Original images; (b) the results of histogram equalization; (c) the results of wavelet transform; (d) the results of the proposed algorithm

    图 6  采集图像增强结果。(a)采集图像;(b)增强图像

    Figure 6.  Enhanced results of acquired images. (a) Acquired images; (b) enhanced images

    图 7  (a)二维高斯分布图及(b)高斯加权卷积块

    Figure 7.  (a) Two-dimensional Gaussian distribution diagram and (b) the Gaussian weighted convolution block

    图 8  基于插值超分辨的邻域窗口像素值变换结果。(a)左视图邻域窗口;(b)右视图邻域窗口;(c)邻域窗口内像素值变换结果

    Figure 8.  Pixel value transformation results of neighborhood windows based on interpolation super-resolution. (a) Neighborhood windows of left image; (b) neighborhood windows of right image; (c) pixel value transformation results in neighborhood window

    图 9  SLIC算法结果。(a)采集图像;(b)超像素分割图;(c)边缘图像

    Figure 9.  Result of SLIC algorithm. (a) Captured image; (b) superpixel segmentation image; (c) edge image

    图 10  算法处理流程图

    Figure 10.  Algorithm processing flowchart

    图 11  (a)采集图像及(b)实验平台

    Figure 11.  (a) Acquiring images and (b) the experimental platform

    表  1  3种图像增强算法对标准图像的处理结果对比

    Table  1.   Comparison of processing results of three image enhancement algorithms with respect to the standard image

    Name Method DE NCC CII PSNR SSIM
    Cameraman Histogram 6.7703 0.9848 0.8039 19.2229 0.6916
    Wavelet 4.4301 0.9888 0.7131 17.6452 0.4708
    Proposed algorithm 7.1149 0.9980 1.0132 24.5250 0.8788
    Lena Histogram 7.3383 0.9862 0.9051 19.3935 0.7784
    Wavelet 5.0992 0.9846 0.7341 17.0583 0.5613
    Proposed algorithm 7.4317 0.9921 1.0088 22.1287 0.8171
    Barbara Histogram 7.3816 0.9892 0.8492 18.2246 0.8147
    Wavelet 6.1788 0.9824 0.7005 17.9306 0.5767
    Proposed algorithm 7.5339 0.9908 1.0132 22.2251 0.8343
    下载: 导出CSV

    表  3  三维重建实验结果

    Table  3.   Experimental results of three-dimensional reconstruction

    Convex surface Trapezoidal surface Angular surface Semicircular surface Concave surface
    Sample image
    Acquired image
    Reconstructed image
    下载: 导出CSV

    表  2  实验参数

    Table  2.   Experimental parameters

    ParameterValue
    Sample size16 mm×8 mm×6.5 mm
    FOV of CCD camera30 mm×20 mm
    Image resolution640 pixels×480 pixels
    Focal length f6 mm
    Object distance180 mm
    Baseline distance49.38 mm
    下载: 导出CSV

    表  4  3种重建方法的检测结果比较

    Table  4.   Comparison of measurement results for three reconstruction methods

    Name of Curves Method in this paper Method in Ref. [27] Method in Ref. [28]
    Maximum error/mm Average error/mm Detection time/s Maximum error/mm Average error/mm Detection time/s Maximum error/mm Average error/mm Detection time/s
    Convex 0.2047 0.1464 1.8551 0.2756 0.2229 1.3287 0.3856 0.2684 2.5932
    Trapezoidal 0.1468 0.1152 1.9173 0.4078 0.2615 1.3813 0.2378 0.1915 2.7367
    Angular 0.1064 0.0768 1.8436 0.2178 0.1525 1.3102 0.1787 0.1425 2.4353
    Semicircular 0.2538 0.1675 1.9362 0.3612 0.2717 1.3922 0.4101 0.3017 2.8577
    Concave 0.1512 0.1278 1.8619 0.2766 0.2105 1.3247 0.2166 0.1705 2.6019
    Average 0.1726 0.1267 1.8828 0.3078 0.2238 1.3474 0.2858 0.2149 2.6450
    下载: 导出CSV

    表  5  样本高度测量误差

    Table  5.   Sample height measurement error rates

    Name of Curves CD IoU MAE/(mm) RMSE/(mm)
    Convex 0.2634 0.6443 0.1150 0.2742
    Trapezoidal 0.2317 0.6987 0.1050 0.2791
    Angular 0.1448 0.7328 0.0882 0.2488
    Semicircular 0.3163 0.6166 0.1403 0.4348
    Concave 0.2215 0.7065 0.1057 0.2710
    Average 0.2355 0.6798 0.1108 0.3016
    下载: 导出CSV
  • [1] WU F P, ZHU S K, YE W L. A single image 3D reconstruction method based on a novel monocular vision system[J]. Sensors, 2020, 20(24): 7045. doi: 10.3390/s20247045
    [2] NASRULLOH A V, WILLCOCKS C G, JACKSON P T G, et al. Multi-scale segmentation and surface fitting for measuring 3-D macular holes[J]. IEEE Transactions on Medical Imaging, 2018, 37(2): 580-589. doi: 10.1109/TMI.2017.2767908
    [3] WANG Y Y, ZHANG H T, XUE J SH, et al. Measurement of three-dimensional pavement morphology using binocular reconstruction improved by second-order fusion algorithm[J]. Automation in Construction, 2024, 157: 105152. doi: 10.1016/j.autcon.2023.105152
    [4] WANG Y Z, JIA H R, JIA P X, et al. A novel algorithm for three-dimensional shape reconstruction for microscopic objects based on shape from focus[J]. Optics & Laser Technology, 2024, 168: 109931.
    [5] DU F, WANG K, ZHANG G J, et al. Damage characteristics of coal under different loading modes based on CT three-dimensional reconstruction[J]. Fuel, 2022, 310: 122304. doi: 10.1016/j.fuel.2021.122304
    [6] 尹仕斌, 任永杰, 刘涛, 等. 机器视觉技术在现代汽车制造中的应用综述[J]. 光学学报,2018,38(8):11-22.

    YIN SH B, REN Y J, LIU T, et al. Review on application of machine vision in modern automobile manufacturing[J]. Acta Optica Sinica, 2018, 38(8): 11-22. (in Chinese).
    [7] GUO X ZH, SHI ZH Y, YU B, et al. 3D measurement of gears based on a line structured light sensor[J]. Precision Engineering, 2020, 61: 160-169. doi: 10.1016/j.precisioneng.2019.10.013
    [8] SIDKI H M, AMER M. Evaluation of CMM for flatness measurements[J]. Metrology and Measurement Systems, 2008, 15(4): 585-593.
    [9] GUILLORY J, TRUONG D, WALLERAND J P, et al. An SI-traceable multilateration coordinate measurement system with half the uncertainty of a laser tracker[J]. Measurement Science and Technology, 2023, 34(6): 065016. doi: 10.1088/1361-6501/acc26a
    [10] LIN J R, XIN R K, SHI SH D, et al. An accurate 6-DOF dynamic measurement system with laser tracker for large-scale metrology[J]. Measurement, 2022, 204: 112052. doi: 10.1016/j.measurement.2022.112052
    [11] HUANG J Q, SUN M ZH, MA J J, et al. Super-resolution image reconstruction for high-density three-dimensional single-molecule microscopy[J]. IEEE Transactions on Computational Imaging, 2017, 3(4): 763-773. doi: 10.1109/TCI.2017.2699425
    [12] SHI CH Y, LUO X B, GUO J, et al. Three-dimensional intravascular reconstruction techniques based on intravascular ultrasound: a technical review[J]. IEEE Journal of Biomedical and Health Informatics, 2018, 22(3): 806-817. doi: 10.1109/JBHI.2017.2703903
    [13] WANG ZH ZH. Robust three-dimensional face reconstruction by one-shot structured light line pattern[J]. Optics and Lasers in Engineering, 2020, 124: 105798. doi: 10.1016/j.optlaseng.2019.105798
    [14] 张新荣, 王鑫, 王瑶, 等. 基于转动式二维激光扫描仪和多传感器的三维重建方法[J]. 中国光学(中英文),2023,16(3):663-672. doi: 10.37188/CO.2022-0159

    ZHANG X R, WANG X, WANG Y, et al. 3D reconstruction method based on a rotating 2D laser scanner and multi-sensor[J]. Chinese Optics, 2023, 16(3): 663-672. (in Chinese). doi: 10.37188/CO.2022-0159
    [15] BEVACQUA M T, SCAPATICCI R. A compressive sensing approach for 3D breast cancer microwave imaging with magnetic nanoparticles as contrast agent[J]. IEEE Transactions on Medical Imaging, 2016, 35(2): 665-673. doi: 10.1109/TMI.2015.2490340
    [16] SUN SH L, WANG X Y, CHEN Y J. Retracted: the application value of CT three-dimensional microscope reconstruction technology in the diagnosis of cervical cancer[J]. Scanning, 2022, 2022: 5648195.
    [17] WANG ZH ZH, ZHOU Q, SHUANG Y C. Three-dimensional reconstruction with single-shot structured light dot pattern and analytic solutions[J]. Measurement, 2020, 151: 107114. doi: 10.1016/j.measurement.2019.107114
    [18] MOONEY S W J, MARLOW P J, ANDERSON B L. The perception and misperception of optical defocus, shading, and shape[J]. eLife, 2019, 8: e48214.
    [19] TIAN X, LIU R, WANG ZH Y, et al. High quality 3D reconstruction based on fusion of polarization imaging and binocular stereo vision[J]. Information Fusion, 2022, 77: 19-28.
    [20] LIU L, ZHU P, GUAN J X, et al. A binocular reconstruction method fused with laplacian image information for pavement texture evaluation[J]. Measurement, 2021, 185: 110039. doi: 10.1016/j.measurement.2021.110039
    [21] LI Y, FU Y J, ZHONG K J, et al. A virtual binocular line-structured light measurement method based on a plane mirror[J]. Optics Communications, 2022, 510: 127974.
    [22] TSENG K K, ZHANG Y Q, ZHU Q L, et al. Semi-supervised image depth prediction with deep learning and binocular algorithms[J]. Applied Soft Computing, 2020, 92: 106272. doi: 10.1016/j.asoc.2020.106272
    [23] BOUKHTACHE S, BLAYSAT B, GRÉDIAC M, et al. Alternatives to bicubic interpolation considering FPGA hardware resource consumption[J]. IEEE Transactions on Very Large Scale Integration (VLSI) Systems, 2021, 29(2): 247-258. doi: 10.1109/TVLSI.2020.3032888
    [24] 李佳星, 赵勇先, 王京华. 基于深度学习的单幅图像超分辨率重建算法综述[J]. 自动化学报,2021,47(10):2341-2363.

    LI J X, ZHAO Y X, WANG J H. A review of single image super-resolution reconstruction algorithms based on deep learning[J]. Acta Automatica Sinica, 2021, 47(10): 2341-2363. (in Chinese).
    [25] 刘艳, 李庆武, 霍冠英, 等. 结合局部二进制表示和超像素分割求精的立体匹配[J]. 光学学报,2018,38(6):0615003. doi: 10.3788/AOS201838.0615003

    LIU Y, LI Q W, HUO G Y, et al. Local binary description combined with superpixel segmentation refinement for stereo matching[J]. Acta Optica Sinica, 2018, 38(6): 0615003. (in Chinese). doi: 10.3788/AOS201838.0615003
    [26] 冯维, 徐仕楠, 王恒辉, 等. 逐像素调制的高反光表面三维测量方法[J]. 中国光学,2022,15(3):488-497. doi: 10.37188/CO.2021-0220

    FENG W, XU SH N, WANG H H, et al. Three-dimensional measurement method of highly reflective surface based on per-pixel modulation[J]. Chinese Optics, 2022, 15(3): 488-497. (in Chinese). doi: 10.37188/CO.2021-0220
    [27] 吴福培, 余冠霖, 乐聪, 等. 微型相机模组贴装过程画胶表面三维测量方法[J]. 光子学报,2022,51(4):0412006. doi: 10.3788/gzxb20225104.0412006

    WU F P, YU G L, LE C, et al. Three-dimensional measurement method of glue surface during the mounting process of micro-camera module[J]. Acta Photonica Sinica, 2022, 51(4): 0412006. (in Chinese). doi: 10.3788/gzxb20225104.0412006
    [28] 平茜茜, 刘勇, 董欣明, 等. 基于偏振双目视觉的无纹理高反光目标三维重构[J]. 红外与毫米波学报,2017,36(4):432-438.

    PING X X, LIU Y, DONG X M, et al. 3-D reconstruction of textureless and high-reflective target by polarization and binocular stereo vision[J]. Journal of Infrared and Millimeter Waves, 2017, 36(4): 432-438. (in Chinese).
  • 加载中
图(11) / 表(5)
计量
  • 文章访问数:  127
  • HTML全文浏览量:  69
  • PDF下载量:  70
  • 被引次数: 0
出版历程
  • 收稿日期:  2023-11-29
  • 修回日期:  2024-01-09
  • 网络出版日期:  2024-05-20

目录

    /

    返回文章
    返回