Open software for hierarchical localization of robots on a sparsed 3d map using multimodal place recognition

In intelligent transport, when constructing 3D maps, there is often a need to improve the quality of global localization on a pre-built map, for example, for initial localization when turning on a robot without additional assistance in the form of manual setting of its position. To solve this problem, this paper proposes an original architecture of the hierarchical localization approach, characterized by a multimodal model of place recognition based on video camera and LiDAR data and a high-speed neural network algorithm for point cloud registration. A software implementation of the proposed architecture has been developed and open sources as a part of OpenPlaceRecognition library. Experiments have been conducted with open datasets of real mobile robots NCLT and ITLP-Campus, which demonstrated an operating time of less than 100 ms and a sufficient level of quality on an embedded Nvidia Jetson AGX Xavier device. The prospects of the approach for practical application are shown.

1. Sarlin P. E. et al. From coarse to fine: Robust hierarchical localization at large scale //Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. – 2019. – С. 12716-12725.

2. Garg S., Fischer T., Milford M. Where Is Your Place, Visual Place Recognition? //Proceedings of the Thirtieth International Joint Conference on Artificial Intelligence (IJCAI-21). – International Joint Conferences on Artificial Intelligence, 2021. – С. 4416-4425.

3. Zhang Y. X. et al. A comprehensive survey and taxonomy on point cloud registration based on deep learning //Proceedings of the Thirty-Third International Joint Conference on Artificial Intelligence. – 2024. – С. 8344-8353.

4. Belkin, I. V., Abramenko, A. A., Bezuglyi, V. D., Yudin, D. A. Localization of mobile robot in prior 3D LiDAR maps using stereo image sequence. Компьютерная оптика. – 2024. – № 48(3) – 406-417.

5. Yudin D. et al. Hpointloc: Point-based indoor place recognition using synthetic rgb-d images //International Conference on Neural Information Processing. – Cham : Springer International Publishing, 2022. – С. 471-484.

6. Kirilenko D. et al. Vector symbolic scene representation for semantic place recognition //2022 International Joint Conference on Neural Networks (IJCNN). – IEEE, 2022. – С. 1-8.

7. J. Komorowski, M. Wysoczańska, and T. Trzcinski, MinkLoc++: Lidar and Monocular Image Fusion for Place Recognition, in 2021 International Joint Conference on Neural Neworks (IJCNN), Jul. 2021, pp. 1–8. doi: 10.1109/IJCNN52387.2021.9533373.

8. H. Lai, P. Yin, and S. Scherer, AdaFusion: Visual-LiDAR Fusion With Adaptive Weights for Place Recognition, IEEE Robotics and Automation Letters, vol. 7, no. 4, pp. 12038–12045, Oct. 2022, doi: 10.1109/LRA.2022.3210880.

9. X. Yu, B. Zhou, Z. Chang, K. Qian, and F. Fang, MMDF: Multi-Modal Deep Feature Based Place Recognition of Mobile Robots With Applications on Cross-Scene Navigation, IEEE Robotics and Automation Letters, vol. 7, no. 3, pp. 6742–6749, Jul. 2022, doi: 10.1109/LRA.2022.3176731.

10. Melekhin, A., Yudin, D., Petryashin, I., Bezuglyj, V. Mssplace: multi-sensor place recognition with visual and text semantics // arXiv preprint arXiv:2407.15663, 2024

11. Muravyev, K., Melekhin, A., Yudin, D., Yakovlev, K. PRISM-TopoMap: online topological mapping with place recognition and scan matching // IEEE Robotics and Automation Letters. 2025.

12. M. A. Fischler and R. C. Bolles, Random sample consensus: a paradigm for model fitting with applications to image analysis and automated cartography, Commun. ACM, vol. 24, no. 6, pp. 381–395, Jun. 1981, doi: 10.1145/358669.358692.

13. Z. Zhang, Iterative point matching for registration of free-form curves and surfaces, Int J Comput Vision, vol. 13, no. 2, pp. 119–152, Oct. 1994, doi: 10.1007/BF01427149.

14. R. Kümmerle, G. Grisetti, H. Strasdat, K. Konolige, and W. Burgard, G2o: A general frame-work for graph optimization, in 2011 IEEE International Conference on Robotics and Automation, May 2011, pp. 3607–3613. doi: 10.1109/ICRA.2011.5979949.

15. H. Yang, J. Shi, and L. Carlone, TEASER: Fast and Certifiable Point Cloud Registration, IEEE Transactions on Robotics, vol. 37, no. 2, pp. 314–333, Apr. 2021, doi: 10.1109/TRO.2020.3033695.

16. Qin Z. et al. Geotransformer: Fast and robust point cloud registration with geometric transformer //IEEE Transactions on Pattern Analysis and Machine Intelligence. – 2023. – Т. 45. – №. 8. – С. 9806-9821.

17. Lu F. et al. Hregnet: A hierarchical network for large-scale outdoor lidar point cloud registration //Proceedings of the IEEE/CVF International Conference on Computer Vision. – 2021. – С. 16014-16023.

18. Cao H., Wang Y., Li D. Dms: Low-overlap registration of 3d point clouds with double-layer multi-scale star-graph //IEEE Transactions on Visualization and Computer Graphics. – 2024.

19. Geiger A., Lenz P., Urtasun R. Are we ready for autonomous driving? the kitti vision bench-mark suite //2012 IEEE conference on computer vision and pattern recognition. – IEEE, 2012. – С. 3354-3361.

20. OpenPlaceRecognition. URL: https://github.com/OPR-Project/OpenPlaceRecognition

21. Carlevaris-Bianco N., Ushani A. K., Eustice R. M. University of Michigan North Campus long-term vision and lidar dataset //The International Journal of Robotics Research. – 2016. – Т. 35. – №. 9. – С. 1023-1035.

22. Melekhin, A., Bezuglyj, V., Petryashin, I., Muravyev, K., Linok, S., Yudin, D., Panov, A. ITLP-Campus: A Dataset for Multimodal Semantic Place Recognition. In International Conference on Intelligent Information Technologies for Industry. – 2024. – pp. 185-195.

23. ROS-2 реализация разработанной библиотеки OpenPlaceRecognition. https://github.com/OPR-Project/OpenPlaceRecognition-ROS2.