TY - JOUR

T1 - Contrastive fine-tuning to improve generalization in deep NER

AU - Bondarenko, Ivan

N1 - The work is supported by the Mathematical Center in Akademgorodok under the agreement No. 075-15-2022—282 with the Ministry of Science and Higher Education of the Russian Federation.

PY - 2022/6/15

Y1 - 2022/6/15

N2 - A novel algorithm of two-stage fine-tuning of a BERT-based language model for more effective named entity recognition is proposed. The first stage is based on training BERT as a Siamese network using a special contrastive loss function, and the second stage consists of fine-tuning the NER as a "traditional" sequence tagger. Inclusion of the contrastive first stage makes it possible to construct a high-level feature space at the output of BERT with more compact representations of different named entity classes. Experiments have shown that this fine-tuning scheme improves the generalization ability of named entity recognition models fine tuned from various pre-trained BERT models. The source code is available under an Apache 2.0 license and hosted on GitHub https://github.com/bond005/runne_contrastive_ner

AB - A novel algorithm of two-stage fine-tuning of a BERT-based language model for more effective named entity recognition is proposed. The first stage is based on training BERT as a Siamese network using a special contrastive loss function, and the second stage consists of fine-tuning the NER as a "traditional" sequence tagger. Inclusion of the contrastive first stage makes it possible to construct a high-level feature space at the output of BERT with more compact representations of different named entity classes. Experiments have shown that this fine-tuning scheme improves the generalization ability of named entity recognition models fine tuned from various pre-trained BERT models. The source code is available under an Apache 2.0 license and hosted on GitHub https://github.com/bond005/runne_contrastive_ner

KW - распознавание именованных сущностей

KW - сопоставительное обучение

KW - Сиамские нейронные сети

KW - BERT

KW - NER

KW - named entity recognition

KW - contrastive learning

KW - Siamese neural networks

KW - BERT

KW - NER

UR - https://www.dialog-21.ru/media/5847/_-dialog2022scopus.pdf

UR - https://www.scopus.com/record/display.uri?eid=2-s2.0-85148050734&origin=inward&txGid=c1a2f8910ed0346faabe1a4929c6b7e2

UR - https://www.mendeley.com/catalogue/53c2698e-400d-312c-945d-b0e44dece9ed/

U2 - 10.28995/2075-7182-2022-21-70-80

DO - 10.28995/2075-7182-2022-21-70-80

M3 - Article

SP - 70

EP - 80

JO - Компьютерная лингвистика и интеллектуальные технологии

JF - Компьютерная лингвистика и интеллектуальные технологии

SN - 2221-7932

IS - 21

M1 - 8

ER -