TY - JOUR

T1 - Efficient Nonlinear Distortion Compensation Algorithms Using Machine Learning for Telecommunication Systems

AU - Redyuk, A. A.

AU - Shevelev, E. I.

AU - Sidelnikov, O. S.

AU - Danilko, V. R.

AU - Bazarov, T. O.

AU - Senko, M. A.

AU - Samodelkin, L. A.

AU - Nanii, O. E.

AU - Treshchikov, V. N.

AU - Fedoruk, M. P.

N1 - Redyuk, A., Shevelev, E., Sidelnikov, O. et al. Efficient Nonlinear Distortion Compensation Algorithms Using Machine Learning for Telecommunication Systems. Bull. Lebedev Phys. Inst. 52 (Suppl 11), S1172–S1187 (2025). https://doi.org/10.3103/S106833562560456X The work by A.A. Redyuk, E.I. Shevelev, O.S. Sidelnikov, and M.P. Fedoruk on the development of nonlinear distortion compensation algorithms, training algorithm parameters, assessing their computational complexity, and performing calculations (Chapters 2, 3, 4, and 6) was supported by the Russian Science Foundation (project no. 25-61-00010, https://rscf.ru/project/25-61-00010/). The work on reviewing the literature on nonlinear distortion compensation methods (Section 1) was supported by the state assignment for fundamental research (no. FSUS-2025-0010).

PY - 2026/2/27

Y1 - 2026/2/27

N2 - Compensation for nonlinear signal distortions is a key challenge for ensuring high throughput and increasing the range of modern fiber-optic communication systems. Nonlinear effects significantly limit data transmission quality, especially with increasing signal power and propagation over long distances. Therefore, the development of accurate, robust, and computationally efficient compensation methods incorporating modern machine learning approaches is particularly relevant in optical telecommunications engineering. The paper presents a comparative analysis of efficient nonlinear distortion compensation algorithms in fiber-optic communication systems using machine learning methods. Modifications of the classical digital backpropagation (DBP) method are discussed: learned DBP (LDBP), enhanced DBP (EnDBP), a perturbation-based model (PBM), a hybrid PB-DBP scheme, and an approach using convolutional neural networks (CNNs). Using experimental data from a laboratory prototype of a 2000-km-long optical transmission line, the capabilities of parameter learning are demonstrated and the effectiveness of the methods in terms of compensation accuracy and computational complexity is compared. The methods demonstrate a significant increase in signal-to-noise ratio and allow a balance between accuracy and load on the digital signal processing module to be optimized.

AB - Compensation for nonlinear signal distortions is a key challenge for ensuring high throughput and increasing the range of modern fiber-optic communication systems. Nonlinear effects significantly limit data transmission quality, especially with increasing signal power and propagation over long distances. Therefore, the development of accurate, robust, and computationally efficient compensation methods incorporating modern machine learning approaches is particularly relevant in optical telecommunications engineering. The paper presents a comparative analysis of efficient nonlinear distortion compensation algorithms in fiber-optic communication systems using machine learning methods. Modifications of the classical digital backpropagation (DBP) method are discussed: learned DBP (LDBP), enhanced DBP (EnDBP), a perturbation-based model (PBM), a hybrid PB-DBP scheme, and an approach using convolutional neural networks (CNNs). Using experimental data from a laboratory prototype of a 2000-km-long optical transmission line, the capabilities of parameter learning are demonstrated and the effectiveness of the methods in terms of compensation accuracy and computational complexity is compared. The methods demonstrate a significant increase in signal-to-noise ratio and allow a balance between accuracy and load on the digital signal processing module to be optimized.

KW - digital signal processing

KW - fiber-optic communication lines

KW - machine learning

KW - nonlinear distortion compensation

KW - nonlinear signal distortion

KW - nonlinearity

KW - нелинейные искажения сигнала

KW - нелинейность

KW - волоконно-оптические линии связи

KW - компенсация нелинейных искажений

KW - машинное обучение

KW - цифровая обработка сигналов

UR - https://www.mendeley.com/catalogue/c7e66031-afa8-3b58-a969-c2c151d4f81b/

UR - https://www.scopus.com/pages/publications/105031502464

U2 - 10.3103/S106833562560456X

DO - 10.3103/S106833562560456X

M3 - Article

VL - 52

SP - S1172-S1187

JO - Bulletin of the Lebedev Physics Institute

JF - Bulletin of the Lebedev Physics Institute

SN - 1068-3356

IS - Suppl 11

ER -