inttransИнтеллектуальный транспортIntelligent transport3033-6007АО «НИИАС»inttrans-14Research ArticleСтатьиОткрытый программный модуль иерархической локализации роботов на разреженной 3d-карте с применением мультимодального распознавания местаOpen software for hierarchical localization of robots on a sparsed 3d map using multimodal place recognitionЮдинД. А.YudinD. A.

к.т.н., зав. лабораторией

Mосква

Ph.D., Head of the laboratory

Moscow

yudin.da@mipt.ruМелехинА. А.MelekhinA. A.

инженер

Mосква

engineer

Moscow

melekhin.aa@mipt.ruЛинокС. А.LinokS. A.

мл.научный сотрудник

Mосква

Junior Researcher

Moscow

linok.sa@phystech.eduMФТИ, AIRIРоссияMIPT, AIRIRussian FederationMФТИРоссияMIPTRussian Federation20251109202601(33)3748Copyright © Юдин Д.А., Мелехин А.А., Линок С.А., 20262026Юдин Д.А., Мелехин А.А., Линок С.А.Yudin D.A., Melekhin A.A., Linok S.A.Данная работа распространяется под лицензией Creative Commons Attribution 4.0.This work is licensed under a Creative Commons Attribution 4.0 License.https://www.intelligent-transport.ru/jour/article/view/14

В интеллектуальном транспорте при построении 3D-карт часто возникает необходимость повышения качества глобальной локализации на построенной заранее карте, например, для первоначальной локализации при включении робота без дополнительной помощи в виде ручной установки его положения. Для решения этой проблемы в настоящей статье предлагается оригинальная архитектура подхода иерархической локализации, отличающаяся мультимодальной моделью распознавания места по данным видеокамер и лидара и быстродействующим нейросетевым алгоритмом регистрации облаков точек. Разработана и размещена в открытый доступ программная реализация предложенной архитектуры, которая включена в открытую библиотеку OpenPlaceRecognition. Проведены эксперименты с открытыми наборами данных реальных мобильных роботов NCLT и ITLP-Campus, которые продемонстрировали время работы менее 100 мс и достаточный уровень качества на встраиваемом устройстве Nvidia Jetson AGX Xavier. Показана перспективность подхода для практического применения.

In intelligent transport, when constructing 3D maps, there is often a need to improve the quality of global localization on a pre-built map, for example, for initial localization when turning on a robot without additional assistance in the form of manual setting of its position. To solve this problem, this paper proposes an original architecture of the hierarchical localization approach, characterized by a multimodal model of place recognition based on video camera and LiDAR data and a high-speed neural network algorithm for point cloud registration. A software implementation of the proposed architecture has been developed and open sources as a part of OpenPlaceRecognition library. Experiments have been conducted with open datasets of real mobile robots NCLT and ITLP-Campus, which demonstrated an operating time of less than 100 ms and a sufficient level of quality on an embedded Nvidia Jetson AGX Xavier device. The prospects of the approach for practical application are shown.

интеллектуальное транспортное средствоиерархическая локализация3D-картараспознавание местанейронная сетьмультимодальностьintelligent vehiclehierarchical localization3D mapplace recognitionneural networkmultimodalityРабота выполнена при поддержке Фонда содействия инновациям для реализации проекта “Открытая программная библиотека мультимодальной нейросетевой глобальной локализации транспортного средства” Договор 34ГУКодИИС12-D7/81485 от 07.12.2022.ReferencesSarlin P. E. et al. From coarse to fine: Robust hierarchical localization at large scale //Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. – 2019. – С. 12716-12725.Sarlin P. E. et al. From coarse to fine: Robust hierarchical localization at large scale //Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. – 2019. – С. 12716-12725.Garg S., Fischer T., Milford M. Where Is Your Place, Visual Place Recognition? //Proceedings of the Thirtieth International Joint Conference on Artificial Intelligence (IJCAI-21). – International Joint Conferences on Artificial Intelligence, 2021. – С. 4416-4425.Garg S., Fischer T., Milford M. Where Is Your Place, Visual Place Recognition? //Proceedings of the Thirtieth International Joint Conference on Artificial Intelligence (IJCAI-21). – International Joint Conferences on Artificial Intelligence, 2021. – С. 4416-4425.Zhang Y. X. et al. A comprehensive survey and taxonomy on point cloud registration based on deep learning //Proceedings of the Thirty-Third International Joint Conference on Artificial Intelligence. – 2024. – С. 8344-8353.Zhang Y. X. et al. A comprehensive survey and taxonomy on point cloud registration based on deep learning //Proceedings of the Thirty-Third International Joint Conference on Artificial Intelligence. – 2024. – С. 8344-8353.Belkin, I. V., Abramenko, A. A., Bezuglyi, V. D., Yudin, D. A. Localization of mobile robot in prior 3D LiDAR maps using stereo image sequence. Компьютерная оптика. – 2024. – № 48(3) – 406-417.Belkin, I. V., Abramenko, A. A., Bezuglyi, V. D., Yudin, D. A. Localization of mobile robot in prior 3D LiDAR maps using stereo image sequence. Компьютерная оптика. – 2024. – № 48(3) – 406-417.Yudin D. et al. Hpointloc: Point-based indoor place recognition using synthetic rgb-d images //International Conference on Neural Information Processing. – Cham : Springer International Publishing, 2022. – С. 471-484.Yudin D. et al. Hpointloc: Point-based indoor place recognition using synthetic rgb-d images //International Conference on Neural Information Processing. – Cham : Springer International Publishing, 2022. – С. 471-484.Kirilenko D. et al. Vector symbolic scene representation for semantic place recognition //2022 International Joint Conference on Neural Networks (IJCNN). – IEEE, 2022. – С. 1-8.Kirilenko D. et al. Vector symbolic scene representation for semantic place recognition //2022 International Joint Conference on Neural Networks (IJCNN). – IEEE, 2022. – С. 1-8.J. Komorowski, M. Wysoczańska, and T. Trzcinski, MinkLoc++: Lidar and Monocular Image Fusion for Place Recognition, in 2021 International Joint Conference on Neural Neworks (IJCNN), Jul. 2021, pp. 1–8. doi: 10.1109/IJCNN52387.2021.9533373.J. Komorowski, M. Wysoczańska, and T. Trzcinski, MinkLoc++: Lidar and Monocular Image Fusion for Place Recognition, in 2021 International Joint Conference on Neural Neworks (IJCNN), Jul. 2021, pp. 1–8. doi: 10.1109/IJCNN52387.2021.9533373.H. Lai, P. Yin, and S. Scherer, AdaFusion: Visual-LiDAR Fusion With Adaptive Weights for Place Recognition, IEEE Robotics and Automation Letters, vol. 7, no. 4, pp. 12038–12045, Oct. 2022, doi: 10.1109/LRA.2022.3210880.H. Lai, P. Yin, and S. Scherer, AdaFusion: Visual-LiDAR Fusion With Adaptive Weights for Place Recognition, IEEE Robotics and Automation Letters, vol. 7, no. 4, pp. 12038–12045, Oct. 2022, doi: 10.1109/LRA.2022.3210880.X. Yu, B. Zhou, Z. Chang, K. Qian, and F. Fang, MMDF: Multi-Modal Deep Feature Based Place Recognition of Mobile Robots With Applications on Cross-Scene Navigation, IEEE Robotics and Automation Letters, vol. 7, no. 3, pp. 6742–6749, Jul. 2022, doi: 10.1109/LRA.2022.3176731.X. Yu, B. Zhou, Z. Chang, K. Qian, and F. Fang, MMDF: Multi-Modal Deep Feature Based Place Recognition of Mobile Robots With Applications on Cross-Scene Navigation, IEEE Robotics and Automation Letters, vol. 7, no. 3, pp. 6742–6749, Jul. 2022, doi: 10.1109/LRA.2022.3176731.Melekhin, A., Yudin, D., Petryashin, I., Bezuglyj, V. Mssplace: multi-sensor place recognition with visual and text semantics // arXiv preprint arXiv:2407.15663, 2024Melekhin, A., Yudin, D., Petryashin, I., Bezuglyj, V. Mssplace: multi-sensor place recognition with visual and text semantics // arXiv preprint arXiv:2407.15663, 2024Muravyev, K., Melekhin, A., Yudin, D., Yakovlev, K. PRISM-TopoMap: online topological mapping with place recognition and scan matching // IEEE Robotics and Automation Letters. 2025.Muravyev, K., Melekhin, A., Yudin, D., Yakovlev, K. PRISM-TopoMap: online topological mapping with place recognition and scan matching // IEEE Robotics and Automation Letters. 2025.M. A. Fischler and R. C. Bolles, Random sample consensus: a paradigm for model fitting with applications to image analysis and automated cartography, Commun. ACM, vol. 24, no. 6, pp. 381–395, Jun. 1981, doi: 10.1145/358669.358692.M. A. Fischler and R. C. Bolles, Random sample consensus: a paradigm for model fitting with applications to image analysis and automated cartography, Commun. ACM, vol. 24, no. 6, pp. 381–395, Jun. 1981, doi: 10.1145/358669.358692.Z. Zhang, Iterative point matching for registration of free-form curves and surfaces, Int J Comput Vision, vol. 13, no. 2, pp. 119–152, Oct. 1994, doi: 10.1007/BF01427149.Z. Zhang, Iterative point matching for registration of free-form curves and surfaces, Int J Comput Vision, vol. 13, no. 2, pp. 119–152, Oct. 1994, doi: 10.1007/BF01427149.R. Kümmerle, G. Grisetti, H. Strasdat, K. Konolige, and W. Burgard, G2o: A general frame-work for graph optimization, in 2011 IEEE International Conference on Robotics and Automation, May 2011, pp. 3607–3613. doi: 10.1109/ICRA.2011.5979949.R. Kümmerle, G. Grisetti, H. Strasdat, K. Konolige, and W. Burgard, G2o: A general frame-work for graph optimization, in 2011 IEEE International Conference on Robotics and Automation, May 2011, pp. 3607–3613. doi: 10.1109/ICRA.2011.5979949.H. Yang, J. Shi, and L. Carlone, TEASER: Fast and Certifiable Point Cloud Registration, IEEE Transactions on Robotics, vol. 37, no. 2, pp. 314–333, Apr. 2021, doi: 10.1109/TRO.2020.3033695.H. Yang, J. Shi, and L. Carlone, TEASER: Fast and Certifiable Point Cloud Registration, IEEE Transactions on Robotics, vol. 37, no. 2, pp. 314–333, Apr. 2021, doi: 10.1109/TRO.2020.3033695.Qin Z. et al. Geotransformer: Fast and robust point cloud registration with geometric transformer //IEEE Transactions on Pattern Analysis and Machine Intelligence. – 2023. – Т. 45. – №. 8. – С. 9806-9821.Qin Z. et al. Geotransformer: Fast and robust point cloud registration with geometric transformer //IEEE Transactions on Pattern Analysis and Machine Intelligence. – 2023. – Т. 45. – №. 8. – С. 9806-9821.Lu F. et al. Hregnet: A hierarchical network for large-scale outdoor lidar point cloud registration //Proceedings of the IEEE/CVF International Conference on Computer Vision. – 2021. – С. 16014-16023.Lu F. et al. Hregnet: A hierarchical network for large-scale outdoor lidar point cloud registration //Proceedings of the IEEE/CVF International Conference on Computer Vision. – 2021. – С. 16014-16023.Cao H., Wang Y., Li D. Dms: Low-overlap registration of 3d point clouds with double-layer multi-scale star-graph //IEEE Transactions on Visualization and Computer Graphics. – 2024.Cao H., Wang Y., Li D. Dms: Low-overlap registration of 3d point clouds with double-layer multi-scale star-graph //IEEE Transactions on Visualization and Computer Graphics. – 2024.Geiger A., Lenz P., Urtasun R. Are we ready for autonomous driving? the kitti vision bench-mark suite //2012 IEEE conference on computer vision and pattern recognition. – IEEE, 2012. – С. 3354-3361.Geiger A., Lenz P., Urtasun R. Are we ready for autonomous driving? the kitti vision bench-mark suite //2012 IEEE conference on computer vision and pattern recognition. – IEEE, 2012. – С. 3354-3361.OpenPlaceRecognition. URL: https://github.com/OPR-Project/OpenPlaceRecognitionOpenPlaceRecognition. URL: https://github.com/OPR-Project/OpenPlaceRecognitionCarlevaris-Bianco N., Ushani A. K., Eustice R. M. University of Michigan North Campus long-term vision and lidar dataset //The International Journal of Robotics Research. – 2016. – Т. 35. – №. 9. – С. 1023-1035.Carlevaris-Bianco N., Ushani A. K., Eustice R. M. University of Michigan North Campus long-term vision and lidar dataset //The International Journal of Robotics Research. – 2016. – Т. 35. – №. 9. – С. 1023-1035.Melekhin, A., Bezuglyj, V., Petryashin, I., Muravyev, K., Linok, S., Yudin, D., Panov, A. ITLP-Campus: A Dataset for Multimodal Semantic Place Recognition. In International Conference on Intelligent Information Technologies for Industry. – 2024. – pp. 185-195.Melekhin, A., Bezuglyj, V., Petryashin, I., Muravyev, K., Linok, S., Yudin, D., Panov, A. ITLP-Campus: A Dataset for Multimodal Semantic Place Recognition. In International Conference on Intelligent Information Technologies for Industry. – 2024. – pp. 185-195.ROS-2 реализация разработанной библиотеки OpenPlaceRecognition. https://github.com/OPR-Project/OpenPlaceRecognition-ROS2.ROS-2 реализация разработанной библиотеки OpenPlaceRecognition. https://github.com/OPR-Project/OpenPlaceRecognition-ROS2.

The authors declare that there are no conflicts of interest present.