TY - JOUR

T1 - ML-Assisted Particle Swarm Optimization of a Perturbation-Based Model for Nonlinearity Compensation in Optical Transmission Systems

AU - Redyuk, Alexey

AU - Shevelev, Evgeny

AU - Danilko, Vitaly

AU - Fedoruk, Mikhail

N1 - This work was supported by the Analytical Center for the Government of the Russian Federation in accordance with the subsidy agreement (identifier 000000D730324P540002) and the agreement with the Novosibirsk State University dated December 27, 2023 No. 70-2023-001318.

PY - 2024/1/1

Y1 - 2024/1/1

N2 - Nonlinear signal distortions are one of the primary factors limiting the capacity and reach of optical transmission systems. Currently, several approaches exist for compensating nonlinear distortions, but for practical implementation, algorithms must be simultaneously accurate, fast, and robust against various interferences. One established approach involves applying perturbation theory methods to the nonlinear Schrödinger equation, which enables the determination of the relation between transmitted and received symbols. In most studies, gradient methods are used to find perturbation coefficients by minimizing the mean squared error between symbols. However, the main parameter characterizing the quality of information transmission is the bit error rate. We propose a modification of the conventional perturbation-based approach for fiber nonlinearity compensation in the form of a two-stage scheme for calculating perturbation coefficients. In the first stage, the coefficients are computed using a single 1D convolutional layer by minimizing the mean squared error. In the second stage, the obtained solution is used as an initial approximation for minimizing the bit error rate using the particle swarm optimization method. In numerical experiments, using the nonlinearity compensation algorithm based on the proposed scheme, we achieved a 0.8 dB gain in the signal-to-noise ratio for a 16QAM 20× 100km link with a channel net rate of 222 Gbit/s and demonstrated improved accuracy compared to the single-stage scheme. We estimated computational complexity of the algorithm and demonstrated the relation between its complexity and accuracy. Additionally, we developed a method for learning perturbation coefficients without relying on ideal symbols from the transmitter, instead using the received symbols after hard decision detection.

AB - Nonlinear signal distortions are one of the primary factors limiting the capacity and reach of optical transmission systems. Currently, several approaches exist for compensating nonlinear distortions, but for practical implementation, algorithms must be simultaneously accurate, fast, and robust against various interferences. One established approach involves applying perturbation theory methods to the nonlinear Schrödinger equation, which enables the determination of the relation between transmitted and received symbols. In most studies, gradient methods are used to find perturbation coefficients by minimizing the mean squared error between symbols. However, the main parameter characterizing the quality of information transmission is the bit error rate. We propose a modification of the conventional perturbation-based approach for fiber nonlinearity compensation in the form of a two-stage scheme for calculating perturbation coefficients. In the first stage, the coefficients are computed using a single 1D convolutional layer by minimizing the mean squared error. In the second stage, the obtained solution is used as an initial approximation for minimizing the bit error rate using the particle swarm optimization method. In numerical experiments, using the nonlinearity compensation algorithm based on the proposed scheme, we achieved a 0.8 dB gain in the signal-to-noise ratio for a 16QAM 20× 100km link with a channel net rate of 222 Gbit/s and demonstrated improved accuracy compared to the single-stage scheme. We estimated computational complexity of the algorithm and demonstrated the relation between its complexity and accuracy. Additionally, we developed a method for learning perturbation coefficients without relying on ideal symbols from the transmitter, instead using the received symbols after hard decision detection.

KW - machine learning

KW - nonlinear signal distortions

KW - nonlinearity compensation

KW - optical transmission system

KW - particle swarm optimization

KW - perturbation-based model

UR - https://www.mendeley.com/catalogue/bff5c374-86c9-3b73-9d61-8ae9176cedf1/

UR - https://www.scopus.com/record/display.uri?eid=2-s2.0-85208236242&origin=inward&txGid=5715f1dc2698fb576450761e21309546

U2 - 10.1109/JLT.2024.3487204

DO - 10.1109/JLT.2024.3487204

M3 - Article

SP - 1

EP - 8

JO - Journal of Lightwave Technology

JF - Journal of Lightwave Technology

SN - 0733-8724

ER -