inttransИнтеллектуальный транспортIntelligent transport3033-6007АО «НИИАС»10.24412/3033-6007-2026-339-4-22inttrans-113Research ArticleИСКУССТВЕННЫЙ ИНТЕЛЛЕКТ И МАШИННОЕ ОБУЧЕНИЕСравнительный анализ методов повышения точности RAG-систем на специализированных корпусах нормативно-технической документацииComparative Analysis of Methods for Improving RAG System Accuracy on Specialized Corpora of Regulatory and Technical DocumentationАгафоновА. А.AgafonovA. A.

аспирант, младший научный сотрудник

Postgraduate Student, Junior Researcher

agafonov.a@spcras.ruПономаревА. В.PonomarevA. V.

к.т.н., доцент, старший научный сотрудник

Candidate of Technical Sciences, Associate Professor, Senior Researcher

ponomarev@iias.spb.suСанкт-Петербургский Федеральный исследовательский центр Российской академии наук (СПб ФИЦ РАН)РоссияSt. Petersburg Federal Research Center of the Russian Academy of Sciences (SPC RAS)Russian Federation202627092026103(39)422Copyright © Агафонов А.А., Пономарев А.В., 20262026Агафонов А.А., Пономарев А.В.Agafonov A.A., Ponomarev A.V.Данная работа распространяется под лицензией Creative Commons Attribution 4.0.This work is licensed under a Creative Commons Attribution 4.0 License.https://www.intelligent-transport.ru/jour/article/view/113

В условиях цифровизации промышленности возрастает потребность в интеллектуальных системах поддержки принятия решений, способных эффективно извлекать знания из массивов нормативно-технической документации. В статье рассматриваются интеллектуальные вопрос-ответные системы на основе технологии генерации, дополненной поиском (Retrieval-Augmented Generation, RAG), применительно к специализированному корпусу нормативно-технических документов железнодорожной отрасли. Проводится сравнительный анализ ряда методов повышения точности RAG-систем, включая стратегии обработки запросов (HyDE, Step-Back, Multi-Query, Decomposition, Pseudo-Relevance Feedback, Recursive Refinement), двухэтапное извлечение с использованием кросс-энкодера текстовых фрагментов (вопроса и нормативного пункта), а также дообучение моделей эмбеддинга текстовых фрагментов на синтетических парах «вопрос - нормативный пункт». Эксперименты показывают, что наиболее существенный прирост качества поиска обеспечивает предметное дообучение моделей эмбеддинга. Применение кросс-энкодера также дает положительный эффект, особенно для моделей с изначально менее точным ранжированием. В то же время продвинутые стратегии обработки запросов не приводят к значимому повышению качества извлечения по сравнению с базовым подходом. Сформированные корпус документов, наборы вопросов и программный код размещены в открытом доступе.

The digital transformation of industry drives a growing demand for intelligent decision support systems capable of efficiently extracting knowledge from regulatory and technical documentation. This paper examines Retrieval-Augmented Generation (RAG) technology applied to a specialized corpus of railway regulatory documents. A comparative analysis is conducted for several methods aimed at improving RAG accuracy, including query processing strategies (HyDE, Step-Back, Multi-Query, Decomposition, Pseudo-Relevance Feedback, Recursive Refinement), two-stage retrieval with a cross-encoder, and fine-tuning of embedding models on synthetic question-regulatory clause pairs. The experiments demonstrate that domain-specific fine-tuning of embedding models yields the most significant improvement in retrieval quality. Cross-encoder reranking also provides a positive effect, particularly for models with initially less accurate ranking. At the same time, advanced query processing strategies do not lead to a substantial improvement in retrieval quality compared to the baseline approach. The constructed document corpus, question sets, and source code are made publicly available.

RAGбольшие языковые моделисемантический поискмодели эмбеддингакросс-энкодернормативно-техническая документацияжелезнодорожный транспортдообучениеRAGlarge language modelssemantic searchembedding modelscross-encoderregulatory and technical documentationrailway transportfine-tuningReferencesRetrieval-Augmented Generation for Knowledge-Intensive NLP Tasks / P. Lewis, E. Perez, A. Piktus [et al.] // Advances in Neural Information Processing Systems. – 2020. – Vol. 33. – P. 9459–9474.Lewis, P., Perez, E., Piktus, A., Petroni, F., Karpukhin, V., Goyal, N., et al. (2020). Retrievalaugmented generation for knowledge-intensive NLP tasks. Advances in Neural Information Processing Systems, 33, 9459–9474.Retrieval-Augmented Generation for Large Language Models: A Survey / Y. Gao, Y. Xiong, X. Gao [et al.] // arXiv. – 2023. – arXiv:2312.10997. – DOI 10.48550/arXiv.2312.10997.Gao, Y., Xiong, Y., Gao, X., Jia, K., Pan, J., Bi, Y., et al. (2023). Retrieval-augmented generation for large language models: A survey (arXiv:2312.10997). arXiv. https://doi.org/10.48550/a rXiv.2312.10997A Review of Large Language Models in Medical Education, Clinical Decision Support, and Healthcare Administration / J. Vrdoljak, Z. Boban, M. Vilovi´c [et al.] // Healthcare. – 2025. – Vol. 13, № 6. – Art. 603. – DOI 10.3390/healthcare13060603.Vrdoljak, J., Boban, Z., Vilović, M., Kumrić, M., & Božić, J. (2025). A review of large language models in medical education, clinical decision support, and healthcare administration. Healthcare, 13(6), Article 603. https://doi.org/10.3390/healthcare13060603A Comprehensive Survey of Retrieval-Augmented Large Language Models for Decision Making in Agriculture: Unsolved Problems and Research Opportunities / A. Vizniuk, G. Diachenko, I. Laktionov [et al.] // Journal of Artificial Intelligence and Soft Computing Research. – 2025. – Vol. 15, № 2. – P. 115–146. – DOI 10.2478/jaiscr-2025-0007.Vizniuk, A., Diachenko, G., Laktionov, I., Siwocha, A., Xiao, M., & Smoląg, J. (2025). A comprehensive survey of retrieval-augmented large language models for decision making in agriculture: Unsolved problems and research opportunities. Journal of Artificial Intelligence and Soft Computing Research, 15(2), 115–146. https://doi.org/10.2478/jaiscr-2025-0007Ершов, М. А. Интеллектуальная СППР для оценки конкурентоспособности в электронной коммерции: методология интеграции RAG-системы / М. А. Ершов // Экономика строительства. – 2025. – № 11. – С. 518–521.Ershov, M. A. (2025). Intelligent DSS for competitiveness assessment in e-commerce: Methodology of RAG-system integration. Ekonomika stroitelstva, (11), 518–521. (in Russian)Самофалов, Г. С. Интеграция больших языковых моделей (LLM) и RAG в контуры поддержки принятия решений (DSS) для задач ценообразования и управления спросом в розничной торговле / Г. С. Самофалов // Интернаука. – 2026. – № 16–1. – С. 38–42.Samofalov, G. S. (2026). Integration of large language models (LLM) and RAG into decision support systems (DSS) for pricing and demand management in retail. Internauka, (16-1), 38–42. (in Russian)Cioplea, R. Memory in the Age of AI: How LLMs Remember, Forget, and Leak : Technical Report, ver. 3.0. – 2026. – 37 p. – DOI 10.17605/OSF.IO/9WXG3.Cioplea, R. (2026). Memory in the age of AI: How LLMs remember, forget, and leak (Technical Report ver. 3.0). https://doi.org/10.17605/OSF.IO/9WXG3RAGAS: Automated Evaluation of Retrieval-Augmented Generation / S. Es, J. James, L. E. Anke, S. Schockaert // Proceedings of the 18th Conference of the European Chapter of the Association for Computational Linguistics: System Demonstrations. – 2024. – P. 150–158. – DOI 10.18653/v1/2024.eacl-demo.16.Es, S., James, J., Anke, L. E., & Schockaert, S. (2024). RAGAS: Automated evaluation of retrieval-augmented generation. In Proceedings of the 18th Conference of the European Chapter of the Association for Computational Linguistics: System Demonstrations (pp. 150–158). https://doi.org/10.18653/v1/2024.eacl-demo.16Precise Zero-Shot Dense Retrieval without Relevance Labels / L. Gao, X. Ma, J. Lin, J. Callan // Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). – 2023. – P. 1762–1777. – DOI 10.18653/v1/2023.acl-long.99.Gao, L., Ma, X., Lin, J., & Callan, J. (2023). Precise zero-shot dense retrieval without relevance labels. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) (pp. 1762–1777). https://doi.org/10.18653/v1/2023.acl-long.99Take a Step Back: Evoking Reasoning via Abstraction in Large Language Models / H. S. Zheng, S. Mishra, X. Chen [et al.] // International Conference on Learning Representations (ICLR 2024). – 2024.Zheng, H. S., Mishra, S., Chen, X., Cheng, H.-T., Chi, E. H., Le, Q. V., & Zhou, D. (2024). Take a step back: Evoking reasoning via abstraction in large language models. In International Conference on Learning Representations (ICLR 2024).DMQR-RAG: Diverse Multi-Query Rewriting for RAG / Z. Li [et al.] // arXiv. – 2024. – arXiv:2411.13154. – DOI 10.48550/arXiv.2411.13154.Li, Z., et al. (2024). DMQR-RAG: Diverse multi-query rewriting for RAG (arXiv:2411.13154). arXiv. https://doi.org/10.48550/arXiv.2411.13154Ammann, P. J. L. Question Decomposition for Retrieval-Augmented Generation / P. J. L. Ammann, J. Golde, A. Akbik // Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 4: Student Research Workshop). – 2025. – P. 497–507. – DOI 10.18653/v1/2025.acl-srw.32.Ammann, P. J. L., Golde, J., & Akbik, A. (2025). Question decomposition for retrievalaugmented generation. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 4: Student Research Workshop) (pp. 497–507). https: //doi.org/10.18653/v1/2025.acl-srw.32Salton, G. Improving Retrieval Performance by Relevance Feedback / G. Salton, C. Buckley // Journal of the American Society for Information Science. – 1990. – Vol. 41, № 4. – P. 288–297.Salton, G., & Buckley, C. (1990). Improving retrieval performance by relevance feedback. Journal of the American Society for Information Science, 41(4), 288–297.Mackie, I. Generative relevance feedback with large language models / I. Mackie, S. Chatterjee, J. Dalton // Proceedings of the 46th International ACM SIGIR Conference on Research and Development in Information Retrieval. – 2023. – P. 2026–2031.Mackie, I., Chatterjee, S., & Dalton, J. (2023). Generative relevance feedback with large language models. In Proceedings of the 46th International ACM SIGIR Conference on Research and Development in Information Retrieval (pp. 2026–2031).Wang, Y. MaFeRw: Query Rewriting with Multi-Aspect Feedbacks for Retrieval-Augmented Large Language Models / Y. Wang // arXiv. – 2024. – arXiv:2408.17072. – DOI 10.48550/arXiv.2408.17072.Wang, Y. (2024). MaFeRw: Query rewriting with multi-aspect feedbacks for retrieval-augmented large language models (arXiv:2408.17072). arXiv. https://doi.org/10.48550/arXiv.2408.17072Nogueira, R. Passage Re-Ranking with BERT / R. Nogueira, K. Cho // arXiv. – 2019. – arXiv:1901.04085. – DOI 10.48550/arXiv.1901.04085.Nogueira, R., & Cho, K. (2019). Passage re-ranking with BERT (arXiv:1901.04085). arXiv. https://doi.org/10.48550/arXiv.1901.04085M3-Embedding: Multi-Linguality, Multi-Functionality, Multi-Granularity Text Embeddings Through Self-Knowledge Distillation / J. Chen [et al.] // arXiv. – 2024. – arXiv:2402.03216. – DOI 10.48550/arXiv.2402.03216.Chen, J., et al. (2024). M3-Embedding: Multi-linguality, multi-functionality, multi-granularity text embeddings through self-knowledge distillation (arXiv:2402.03216). arXiv. https://doi.org/10.48550/arXiv.2402.03216Reimers, N. Sentence-BERT: Sentence Embeddings Using Siamese BERT-Networks / N. Reimers, I. Gurevych // Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP). – 2019. – P. 3982–3992. – DOI 10.18653/v1/D19-1410.Reimers, N., & Gurevych, I. (2019). Sentence-BERT: Sentence embeddings using Siamese BERT-networks. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP) (pp. 3982–3992). https://doi.org/10.18653/v1/D19-1410Text Embeddings by Weakly-Supervised Contrastive Pre-Training / L. Wang, N. Yang, X. Huang [et al.] // arXiv. – 2022. – arXiv:2212.03533. – DOI 10.48550/arXiv.2212.03533.Wang, L., Yang, N., Huang, X., Jiao, B., Yang, L., Jiang, D., et al. (2022). Text embeddings by weakly-supervised contrastive pre-training (arXiv:2212.03533). arXiv. https: //doi.org/10.48550/arXiv.2212.03533MTEB: Massive Text Embedding Benchmark / N. Muennighoff, N. Tazi, L. Magne, N. Reimers // arXiv. – 2022. – arXiv:2210.07316. – DOI 10.48550/arXiv.2210.07316.Muennighoff, N., Tazi, N., Magne, L., & Reimers, N. (2022). MTEB: Massive text embedding benchmark (arXiv:2210.07316). arXiv. https://doi.org/10.48550/arXiv.2210.07316The Russian-focused embedders’ exploration: ruMTEB benchmark and Russian embedding model design / A. Snegirev [et al.] // arXiv. – 2024. – arXiv:2408.12503. – DOI 10.48550/arXiv.2408.12503.Snegirev, A., et al. (2024). The Russian-focused embedders’ exploration: ruMTEB benchmark and Russian embedding model design (arXiv:2408.12503). arXiv. https://doi.org/10.48550/arXiv.2408.12503Multilingual E5 Text Embeddings: A Technical Report / L. Wang [et al.] // arXiv. – 2024. – arXiv:2402.05672. – DOI 10.48550/arXiv.2402.05672.Wang, L., et al. (2024). Multilingual E5 text embeddings: A technical report (arXiv:2402.05672). arXiv. https://doi.org/10.48550/arXiv.2402.05672Паршин, К. А. Исследование применения признаков терминов железнодорожной отрасли при формировании классификатора / К. А. Паршин, М. С. Подгорный // Теория и практика современной науки. – 2019. – № 3. – С. 229–232.Parshin, K. A., & Podgornyy, M. S. (2019). A study of the application of railway industry term features in the formation of a classifier. Teoriya i praktika sovremennoy nauki, (3), 229–232. (in Russian)Qwen3 Technical Report / A. Yang [et al.] // arXiv. – 2025. – arXiv:2505.09388. – DOI 10.48550/arXiv.2505.09388.Yang, A., et al. (2025). Qwen3 technical report (arXiv:2505.09388). arXiv. https://doi.org/10.48550/arXiv.2505.09388

The authors declare that there are no conflicts of interest present.