TY - JOUR

T1 - Wav2vec2 Without Attention: Do You Need Hopfield Networks for Self-Supervised Learning of Speech Representations?

AU - Grebenkin, D.

AU - Bondarenko, I.

PY - 2024/10

Y1 - 2024/10

N2 - In this work, we consider the possibility of replacing multi-head attention with dense associative memory (DAM) layers in the wav 2vec2 automatic speech recognition algorithm. We examine the hypothesis that the concept of modern Hopfield networks is more suitable for restoration of missing fragments of the audio signal task and speech-to-text task than multi-head attention. Our experiments indicate that the model with the new architecture allows to improve the quality of speech recognition and can be used for pretraining the models on a large amount of audio data.

AB - In this work, we consider the possibility of replacing multi-head attention with dense associative memory (DAM) layers in the wav 2vec2 automatic speech recognition algorithm. We examine the hypothesis that the concept of modern Hopfield networks is more suitable for restoration of missing fragments of the audio signal task and speech-to-text task than multi-head attention. Our experiments indicate that the model with the new architecture allows to improve the quality of speech recognition and can be used for pretraining the models on a large amount of audio data.

UR - https://www.scopus.com/record/display.uri?eid=2-s2.0-85208801879&origin=inward&txGid=e7bf0a610121a8caecfc1191f85cf728

UR - https://www.mendeley.com/catalogue/27435cff-6d49-3e53-b0c3-31a36aed60c8/

U2 - 10.1007/s10958-024-07420-6

DO - 10.1007/s10958-024-07420-6

M3 - Article

VL - 285

SP - 28

EP - 35

JO - Journal of Mathematical Sciences (United States)

JF - Journal of Mathematical Sciences (United States)

SN - 1072-3374

IS - 1

ER -