TY - JOUR

T1 - An Approach to Information Extraction from Texts of a Limited Subject Domain Based on a Chain of Large Language Models

AU - Sidorova, E. A.

AU - Ivanov, A. I.

AU - Ilina, D. V.

AU - Ovchinnikova, K. A.

AU - Osmushkin, N. M.

AU - Sery, A. S.

PY - 2025

Y1 - 2025

N2 - The paper considers the approach for extracting the information from texts of limited domains of knowledge based on a chain of neural language models. The task is represented in the form of three subtasks solved sequentially: (1) term extraction and classification; (2) coreference resolution; (3) extraction of relations of entities named with the terms. The dataset was based on texts on computational linguistics from the Habr forum. In the markup for term classification and relation extraction, 17 classes of terms and 51 relations were used in accordance with the ontology of computational linguistics. Prompt chain-based methods were used to apply LLMs, where each next query to the LLM is based on the results of the previous step. Six types of prompt templates were developed: for extracting, classifying, verifying terms, extracting coreferential relations, relations specified by the ontology, and a specialized template for relations linking entities of the same class. Sentence-BERT, GPT-4 and Mistral-based models were used at different steps of the study; a comparison with the SFT approach (ruRoBERTa) was made; hybrid approaches that have shown the best results were also developed. For term extraction and classification, F1=0.77 was obtained, for coreference resolution—F1=0.897, and for relation extraction—F1=0.847.

AB - The paper considers the approach for extracting the information from texts of limited domains of knowledge based on a chain of neural language models. The task is represented in the form of three subtasks solved sequentially: (1) term extraction and classification; (2) coreference resolution; (3) extraction of relations of entities named with the terms. The dataset was based on texts on computational linguistics from the Habr forum. In the markup for term classification and relation extraction, 17 classes of terms and 51 relations were used in accordance with the ontology of computational linguistics. Prompt chain-based methods were used to apply LLMs, where each next query to the LLM is based on the results of the previous step. Six types of prompt templates were developed: for extracting, classifying, verifying terms, extracting coreferential relations, relations specified by the ontology, and a specialized template for relations linking entities of the same class. Sentence-BERT, GPT-4 and Mistral-based models were used at different steps of the study; a comparison with the SFT approach (ruRoBERTa) was made; hybrid approaches that have shown the best results were also developed. For term extraction and classification, F1=0.77 was obtained, for coreference resolution—F1=0.897, and for relation extraction—F1=0.847.

KW - извлечение информации

KW - извлечение терминов

KW - извлечение отношений

KW - обработка естественного языка

KW - машинное обучение

KW - большие языковые модели

KW - онтология предметной области

KW - цепочки инструкций

KW - information extraction

KW - term extraction

KW - relation extraction

KW - natural language processing

KW - machine learning

KW - large language models

KW - ontology of domain of knowledge

KW - prompt chaining

UR - https://www.scopus.com/pages/publications/105039287545

UR - https://www.mendeley.com/catalogue/3644d634-3c69-32e6-bae1-8e5a15cf643e/

U2 - 10.28995/2075-7182-2025-23-361-373

DO - 10.28995/2075-7182-2025-23-361-373

M3 - Conference article

SP - 103

EP - 116

JO - Компьютерная лингвистика и интеллектуальные технологии

JF - Компьютерная лингвистика и интеллектуальные технологии

SN - 2221-7932

IS - 23

T2 - International Conference “Dialogue 2025”: Computational Linguistics and Intellectual Technologies

Y2 - 23 April 2025 through 25 April 2025

ER -