TY - JOUR

T1 - Using x86 mode switching for program code protection

AU - Lebedev, R. K.

N1 - Публикация для корректировки.

PY - 2023

Y1 - 2023

N2 - A novel program code obfuscation approach involving the x86 mode switching is proposed in the paper. The details and existing applications of x86 mode switching are reviewed, as well as the possible consequences of using this switching to the reverse engineering tools. Based on this approach, a few specific methods are proposed and evaluated against the most popular reverse engineering tools of various purposes, including disassemblers, decompilers, binary instrumentation and symbolic execution tools. A method of seamless integration of these machine code level obfuscations to the C, C++ and possibly other compilers is also proposed.

AB - A novel program code obfuscation approach involving the x86 mode switching is proposed in the paper. The details and existing applications of x86 mode switching are reviewed, as well as the possible consequences of using this switching to the reverse engineering tools. Based on this approach, a few specific methods are proposed and evaluated against the most popular reverse engineering tools of various purposes, including disassemblers, decompilers, binary instrumentation and symbolic execution tools. A method of seamless integration of these machine code level obfuscations to the C, C++ and possibly other compilers is also proposed.

KW - code protection

KW - decompilation

KW - disassembly

KW - obfuscation

KW - reverse engineering

KW - symbolic execution

KW - x86 mode switching

UR - https://www.scopus.com/record/display.uri?eid=2-s2.0-85179049819&origin=inward&txGid=b648c7c9542d909e371b1081528382ca

UR - https://www.mendeley.com/catalogue/385e5532-5aac-30f2-bb9c-9973d12463a9/

U2 - 10.17223/20710410/61/6

DO - 10.17223/20710410/61/6

M3 - Article

VL - 61

SP - 104

EP - 120

JO - Прикладная дискретная математика

JF - Прикладная дискретная математика

SN - 2071-0410

ER -