A Model Development Approach Based on Point Cloud Reconstruction and Mapping Texture Enhancement

Boyang You; Barmak Honarvar Shakibaei Asli

doi:10.3390/bdcc8110164

journal article Open Access Nov 20, 2024

A Model Development Approach Based on Point Cloud Reconstruction and Mapping Texture Enhancement

Boyang You

Barmak Honarvar Shakibaei Asli

Big Data and Cognitive Computing Vol. 8 No. 11 pp. 164 · MDPI AG

View at Publisher Save 10.3390/bdcc8110164

Abstract

To address the challenge of rapid geometric model development in the digital twin industry, this paper presents a comprehensive pipeline for constructing 3D models from images using monocular vision imaging principles. Firstly, a structure-from-motion (SFM) algorithm generates a 3D point cloud from photographs. The feature detection methods scale-invariant feature transform (SIFT), speeded-up robust features (SURF), and KAZE are compared across six datasets, with SIFT proving the most effective (matching rate higher than 0.12). Using K-nearest-neighbor matching and random sample consensus (RANSAC), refined feature point matching and 3D spatial representation are achieved via antipodal geometry. Then, the Poisson surface reconstruction algorithm converts the point cloud into a mesh model. Additionally, texture images are enhanced by leveraging a visual geometry group (VGG) network-based deep learning approach. Content images from a dataset provide geometric contours via higher-level VGG layers, while textures from style images are extracted using the lower-level layers. These are fused to create texture-transferred images, where the image quality assessment (IQA) metrics SSIM and PSNR are used to evaluate texture-enhanced images. Finally, texture mapping integrates the enhanced textures with the mesh model, improving the scene representation with enhanced texture. The method presented in this paper surpassed a LiDAR-based reconstruction approach by 20% in terms of point cloud density and number of model facets, while the hardware cost was only 1% of that associated with LiDAR.

Topics

No keywords indexed for this article. Browse by subject →

References

63

[1]

Digital twin modeling

Fei Tao, Bin Xiao, Qinglin Qi et al.

Journal of Manufacturing Systems 2022 10.1016/j.jmsy.2022.06.015

[2]

Gong "Advancements in digital twin modeling for underground spaces and lightweight geometric modeling technologies" Autom. Constr. (2024) 10.1016/j.autcon.2024.105578

[3]

Wu, H., Ji, P., Ma, H., and Xing, L. (2023). A comprehensive review of digital twin from the perspective of total process: Data, models, networks and applications. Sensors, 23. 10.3390/s23198306

[4]

Elaksher, A., Ali, T., and Alharthy, A. (2023). A quantitative assessment of LiDAR data accuracy. Remote Sens., 15. 10.3390/rs15020442

[5]

Meyer "Accuracy (trueness and precision) of a dual-structured light facial scanner and interexaminer reliability" J. Prosthet. Dent. (2020) 10.1016/j.prosdent.2019.10.010

[6]

Frangez "Assessment and improvement of distance measurement accuracy for time-of-flight cameras" IEEE Trans. Instrum. Meas. (2022) 10.1109/tim.2022.3167792

[7]

Bi, S., Gu, Y., Zou, J., Wang, L., Zhai, C., and Gong, M. (2021). High precision optical tracking system based on near infrared trinocular stereo vision. Sensors, 21. 10.3390/s21072528

[8]

Wang, Y., Funk, N., Ramezani, M., Papatheodorou, S., Popović, M., Camurri, M., Leutenegger, S., and Fallon, M. (June, January 30). Elastic and efficient LiDAR reconstruction for large-scale exploration tasks. Proceedings of the 2021 IEEE International Conference on Robotics and Automation (ICRA), Xi’an, China. 10.1109/icra48506.2021.9561736

[9]

Zhang, J., Zhang, F., Kuang, S., and Zhang, L. (2024, January 20–27). Nerf-lidar: Generating realistic lidar point clouds with neural radiance fields. Proceedings of the AAAI Conference on Artificial Intelligence, Vancouver, BC, Canada. 10.1609/aaai.v38i7.28546

[10]

Wang "Review of real-time three-dimensional shape measurement techniques" Measurement (2020) 10.1016/j.measurement.2020.107624

[11]

Wang "Three-dimensional reconstruction with single-shot structured light dot pattern and analytic solutions" Measurement (2020) 10.1016/j.measurement.2019.107114

[12]

Liu "Lightweight Structured Line Map Based Visual Localization" IEEE Robot. Autom. Lett. (2024) 10.1109/lra.2024.3387137

[13]

Cao "Simultaneous calibration of hand-eye and kinematics for industrial robot using line-structured light sensor" Measurement (2023) 10.1016/j.measurement.2023.113508

[14]

Liang "3D reconstruction of weld pool surface in pulsed GMAW by passive biprism stereo vision" IEEE Robot. Autom. Lett. (2019) 10.1109/lra.2019.2924844

[15]

Li "RGB line pattern-based stereo vision matching for single-shot 3-D measurement" IEEE Trans. Instrum. Meas. (2020)

[16]

Jing, J., Li, J., Xiong, P., Liu, J., Liu, S., Guo, Y., Deng, X., Xu, M., Jiang, L., and Sigal, L. (2023, January 2–6). Uncertainty guided adaptive warping for robust and efficient stereo matching. Proceedings of the IEEE/CVF International Conference on Computer Vision, Paris, France. 10.1109/iccv51070.2023.00307

[17]

Hu "A new microscopic telecentric stereo vision system-calibration, rectification, and three-dimensional reconstruction" Opt. Lasers Eng. (2019) 10.1016/j.optlaseng.2018.09.011

[18]

Berra, E., and Peppa, M. (2020, January 21–26). Advances and challenges of UAV SFM MVS photogrammetry and remote sensing: Short review. Proceedings of the 2020 IEEE Latin American Grss & ISPRS Remote Sensing Conference (Lagirs), Santiago, Chile. 10.1109/lagirs48042.2020.9285975

[19]

Gao, L., Zhao, Y., Han, J., and Liu, H. (2022). Research on multi-view 3D reconstruction technology based on SFM. Sensors, 22. 10.3390/s22124366

[20]

Wang, J., Rupprecht, C., and Novotny, D. (2023, January 2–6). Posediffusion: Solving pose estimation via diffusion-aided bundle adjustment. Proceedings of the IEEE/CVF International Conference on Computer Vision, Paris, France. 10.1109/iccv51070.2023.00896

[21]

Pan, L., Baráth, D., Pollefeys, M., and Schönberger, J.L. (October, January 29). Global Structure-from-Motion Revisited. Proceedings of the European Conference on Computer Vision (ECCV), Milan, Italy.

[22]

Barath, D., Mishkin, D., Eichhardt, I., Shipachev, I., and Matas, J. (2021, January 20–25). Efficient initial pose-graph generation for global sfm. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, Nashville, TN, USA. 10.1109/cvpr46437.2021.01431

[23]

Liu, S., Jiang, S., Liu, Y., Xue, W., and Guo, B. (2022). Efficient SfM for Large-Scale UAV Images Based on Graph-Indexed BoW and Parallel-Constructed BA Optimization. Remote Sens., 14. 10.3390/rs14215619

[24]

Bond, Y.L., Ledwell, S., Osornio, E., and Cruz, A.C. (2023, January 25–27). Efficient Scene Reconstruction for Unmanned Aerial Vehicles. Proceedings of the 2023 Fifth International Conference on Transdisciplinary AI (TransAI), Laguna Hills, CA, USA. 10.1109/transai60598.2023.00036

[25]

Barath, D., Noskova, J., Eichhardt, I., and Matas, J. (2022, January 21–24). Pose-graph via Adaptive Image Re-ordering. Proceedings of the BMVC, London, UK.

[26]

Tolias "Fine-tuning CNN image retrieval with no human annotation" IEEE Trans. Pattern Anal. Mach. Intell. (2018)

[27]

Wei, X., Zhang, Y., Li, Z., Fu, Y., and Xue, X. (2020, January 23–28). Deepsfm: Structure from motion via deep bundle adjustment. Proceedings of the Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK. 10.1007/978-3-030-58452-8_14

[28]

Schonberger, J.L., and Frahm, J.M. (2016, January 27–30). Structure-from-motion revisited. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Las Vegas, NV, USA. 10.1109/cvpr.2016.445

[29]

Li, Z., Luo, S., Zeng, W., Guo, S., Zhuo, J., Zhou, L., Ma, Z., and Zhang, Z. (2022, January 19–21). 3d reconstruction system for foot arch detecting based on openmvg and openmvs. Proceedings of the 2022 5th International Conference on Pattern Recognition and Artificial Intelligence (PRAI), Chengdu, China. 10.1109/prai55851.2022.9904285

[30]

Lyra, V.G.d.M., Pinto, A.H., Lima, G.C., Lima, J.P., Teichrieb, V., Quintino, J.P., da Silva, F.Q., Santos, A.L., and Pinho, H. (2020, January 7–10). Development of an efficient 3D reconstruction solution from permissive open-source code. Proceedings of the 2020 22nd Symposium on Virtual and Augmented Reality (SVR), Virtual. 10.1109/svr51698.2020.00044

[31]

Wu, C. (July, January 29). Towards linear-time incremental structure from motion. Proceedings of the 2013 International Conference on 3D Vision-3DV, Seattle, WA, USA. 10.1109/3dv.2013.25

[32]

Zhou "Point cloud denoising review: From classical to deep learning-based approaches" Graph. Model. (2022) 10.1016/j.gmod.2022.101140

[33]

Surface Reconstruction From Point Clouds: A Survey and a Benchmark

Zi Huang, Yuxin WEN, Zihao Wang et al.

IEEE Transactions on Pattern Analysis and Machine... 2024 10.1109/tpami.2024.3429209

[34]

Azinović, D., Martin-Brualla, R., Goldman, D.B., Nießner, M., and Thies, J. (2022, January 18–24). Neural rgb-d surface reconstruction. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, New Orleans, LA, USA. 10.1109/cvpr52688.2022.00619

[35]

You, C.C., Lim, S.P., Lim, S.C., San Tan, J., Lee, C.K., and Khaw, Y.M.J. (2020, January 17–19). A survey on surface reconstruction techniques for structured and unstructured data. Proceedings of the 2020 IEEE Conference on Open Systems (ICOS), Penang, Malaysia. 10.1109/icos50156.2020.9293685

[36]

SACF-Net: Skip-Attention Based Correspondence Filtering Network for Point Cloud Registration

Yue Wu, Xidao Hu, Yue Zhang et al.

IEEE Transactions on Circuits and Systems for Vide... 2023 10.1109/tcsvt.2023.3237328

[37]

Lu "Deep feature-preserving normal estimation for point cloud filtering" Comput. Aided Des. (2020) 10.1016/j.cad.2020.102860

[38]

Zhang "Hypergraph spectral analysis and processing in 3D point cloud" IEEE Trans. Image Process. (2020) 10.1109/tip.2020.3042088

[39]

Ren, D., Ma, Z., Chen, Y., Peng, W., Liu, X., Zhang, Y., and Guo, Y. (2024). Spiking pointnet: Spiking neural networks for point clouds. arXiv.

[40]

Hao "An improved PointNet++ point cloud segmentation model applied to automatic measurement method of pig body size" Comput. Electron. Agric. (2023) 10.1016/j.compag.2022.107560

[41]

Hermosilla, P., Ritschel, T., and Ropinski, T. (November, January 27). Total denoising: Unsupervised learning of 3D point cloud cleaning. Proceedings of the IEEE/CVF International Conference on Computer Vision, Seoul, Republic of Korea.

[42]

Liu "Implicit surface reconstruction with radial basis functions via PDEs" Eng. Anal. Bound. Elem. (2020) 10.1016/j.enganabound.2019.09.021

[43]

Dai, P., Xu, J., Xie, W., Liu, X., Wang, H., and Xu, W. (August, January 27). High-quality surface reconstruction using gaussian surfels. Proceedings of the ACM SIGGRAPH 2024 Conference Papers, Denver, CO, USA. 10.1145/3641519.3657441

[44]

Comi "Touchsdf: A deepsdf approach for 3d shape reconstruction using vision-based tactile sensing" IEEE Robot. Autom. Lett. (2024) 10.1109/lra.2024.3396054

[45]

Gatys, L., Ecker, A., and Bethge, M. (2015). Texture synthesis and the controlled generation of natural stimuli using convolutional neural networks. arXiv. 10.1109/cvpr.2016.265

[46]

Zhang, Y., Huang, N., Tang, F., Huang, H., Ma, C., Dong, W., and Xu, C. (2023, January 17–24). Inversion-based style transfer with diffusion models. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, Vancouver, BC, Canada. 10.1109/cvpr52729.2023.00978

[47]

Lin "GAN-based day-to-night image style transfer for nighttime vehicle detection" IEEE Trans. Intell. Transp. Syst. (2020) 10.1109/tits.2019.2961679

[48]

Gatys, L.A., Ecker, A.S., Bethge, M., Hertzmann, A., and Shechtman, E. (2017, January 21–26). Controlling perceptual factors in neural style transfer. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Honolulu, HI, USA. 10.1109/cvpr.2017.397

[49]

Tang, H., Liu, S., Lin, T., Huang, S., Li, F., He, D., and Wang, X. (2023, January 17–24). Master: Meta style transformer for controllable zero-shot and few-shot artistic style transfer. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, Vancouver, BC, Canada. 10.1109/cvpr52729.2023.01758

[50]

Zhang, C., Xu, X., Wang, L., Dai, Z., and Yang, J. (2024, January 20–27). S2wat: Image style transfer via hierarchical vision transformer using strips window attention. Proceedings of the AAAI Conference on Artificial Intelligence, Vancouver, BC, Canada. 10.1609/aaai.v38i7.28529

Showing 50 of 63 references

Metrics

1

Citations

63

References

Details

Published: Nov 20, 2024
Vol/Issue: 8(11)
Pages: 164
License: View

Authors

B

Boyang You

College of Mechanical and Electrical Engineering, Nanjing University of Aeronautics and Astronautics, Nanjing 210016, China

B

Barmak Honarvar Shakibaei Asli

Centre for Life-Cycle Engineering and Management, Faculty of Engineering and Applied Sciences, Cranfield University, Bedford MK43 0AL, UK

Cite This Article

Boyang You, Barmak Honarvar Shakibaei Asli (2024). A Model Development Approach Based on Point Cloud Reconstruction and Mapping Texture Enhancement. Big Data and Cognitive Computing, 8(11), 164. https://doi.org/10.3390/bdcc8110164

A Model Development Approach Based on Point Cloud Reconstruction and Mapping Texture Enhancement

You May Also Like