Lateral Movement Attack Detection using Variational Autoencoders

Shabani, Mostafa; Tafazoli, Tala

doi:10.22042/isecure.2026.242099

Lateral Movement Attack Detection using Variational Autoencoders

Articles in Press

Document Type : Research Article

Authors

Mostafa Shabani ¹

Tala Tafazoli ²

¹ Department of Industrial Engineering, Iran University of Science and Technology

² ICT Security Faculty, ICT Research Institute (ITRC)

10.22042/isecure.2026.242099

Abstract

Lateral movement, a sophisticated cyberattack strategy, enables adversaries to stealthily infiltrate networks following an initial breach. Detecting such maneuvers is exceptionally challenging, as they are designed to seamlessly blend with legitimate system operations and network traffic, rendering traditional signature-based defenses ineffective. Supervised machine learning approaches, while promising, are constrained by their dependence on pre-labeled datasets of known attack patterns. To overcome these limitations, this study introduces a novel hybrid deep learning framework that integrates a Variational Autoencoder (VAE) for robust feature extraction, coupled with a supervised classifier to identify lateral movement. Through meticulous feature engineering on the LMD dataset, the VAE is trained exclusively on normative system and network behavior, constructing a probabilistic representation of legitimate activity. Anomalies, detected via reconstruction error, signal potential malicious intrusions. Empirical evaluation demonstrates the framework’s superior performance, achieving a detection time of 00:00:02:54 and an AUC of 99.6983%, reflecting exceptional class separation and computational efficiency. This hybrid architecture delivers a scalable, high-accuracy solution, establishing the VAE as a pivotal tool for combating advanced persistent threats with unparalleled precision and operational viability.

Keywords

Lateral Movement Attack

Variational Auto Encoder

hybrid learning

Anomaly Detection

[1] Vu Dinh Phai Le Quy Don Qi Shi Nathan Shone, Tran Nguyen Ngoc. A deep learning approach to network intrusion detection. IEEE Transactions on Emerging Topics in Computational Intelligence, 2(1):41–50, 2018.

[2] Mohammed Al-Habib Kamal Al-Sabahi Majjed Al-Qatf, Yu Lasheng. Deep learning approach combining sparse autoencoder with svm for network intrusion detection. IEEE Access, pages 52843–52856, 2018.

[3] Jinoh Kim Sang C. Suh Ikkyun Kim Kuinam J. Kim Donghwoon Kwon, Hyunjoo Kim. A survey of deep learning-based network anomaly detection. IEEE Transactions on Emerging Topics in Computational Intelligence, 22:949–961, 2019.

[4] Cheah Wai Shiang Johari Abdullah Farhan Ahmad Zeeshan Ahmad, Adnan Shahid Khan. Deep learning approaches for anomaly-based intrusion detection systems: A survey, taxonomy, and open issues. Transactions on emerging telecommunications technologies, 31(1):949–961, 2021.

[5] Ahmed Z. Emam Arwa Aldweesh, Abdelouahid Derhab. Deep learning approaches for anomaly-based intrusion detection systems: A survey, taxonomy, and open issues. Transactions on emerging telecommunications technologies, 189(C), 2020.

[6] Mohammad A. Salahuddin Abbas Abou Daya Noura Limam Raouf Boutaba Tim Bai, Haibo Bian. Rdp-based lateral movement detection using machine learning. Computer Communications, 165(1):9–19, 2021.

[7] Konstantia Barbatsalou Christos Smiliotopoulos, Georgios Kambourakis. On the detection of lateral movement through supervised machine learning and an open-source tool to create turnkey datasets from sysmon logs. International Journal of Information Security, 22:1893–1919, 2023.

[8] M. Roshni Thanka Ashwathy Anda Chacko, Bijolin Edwin. Detecting the lateral movement in cyberattack at the early stage using machine learning techniques. Disruptive Technologies for Big Data and Cloud Applications, 2021.

[9] R. Zhang X. Wang, Z. Yan and P. Zhang. Attacks and defenses in user authentication systems: A survey. Journal of Network and Computer Applications, 188.

[10] D. Dimov and Y. Tzonev. Pass-the-hash. Proceedings of the 18th International Conference on Computer Systems and Technologies, 3.

[11] X. Chen S. Yu Q. Xuan J. Zhou, J. Yao and X. Yang. Lateral movement detection via timeaware subgraph classification on authentication logs.

[12] A. Kucukelbir D. M. Blei and J. D. McAuliffe. Variational inference: A review for statisticians. J Am Stat Assoc, 112(518):859–877, 2017.

[13] E. A. Barros da Silva L. Pinheiro Cinelli, M. Arau´jo Marins and S. Lima Netto. Variational Autoencoder in Variational Methods for Machine Learning with Applications to Deep Networks. Cham: Springer International Publishing, 2021.

[14] K. Dvijotham S. Gowal T. Cemgil, S. Ghaisas and P. Kohli. The autoencoding variational autoencoder. Advances in Neural Information Processing Systems.

[15] G. Kambourakis C. Smiliotopoulos and K. Barbatsalou. On the detection of lateral movement through supervised machine learning and an open-source tool to create turnkey datasets from sysmon logs. Int J Inf Secur, 22(6):1893–1919, 2023.

[16] G. Kaiafas et al. Detecting malicious authentication events trustfully. In NOMS 2018 - 2018 IEEE/IFIP Network Operations and Management Symposium.

[17] G.-H. Syu C.-M. Chen and Z.-X. Cai. Analyzing system log based on machine learning model. International Journal of Network Security, 22(6).

[18] A. Fawaz A. Bohara, M. A. Noureddine and W. H. Sanders. An unsupervised multi-detector approach for identifying malicious lateral movement. In Proceedings of the IEEE Symposium on Reliable Distributed Systems.

[19] J. Liu B. Jiang L. Su M. Chen, Y. Yao and Z. Lu. A novel approach for identifying lateral movement attacks based on network embedding. In IEEE Intl Conf on Parallel Distributed Processing with Applications, Ubiquitous Computing Communications, Big Data Cloud Computing, Social Computing Networking, Sustainable Computing Communications (ISPA/IUCC/BDCloud/SocialCom/SustainCom).

[20] A. Alva R. Sreedhar M. Bhadkamkar H. Pal Singh Bhasin, E. Ramsdell and H. Pal Singh. Data center application security: Lateral movement detection of malware using behavioral models.

[21] B. A. Powell. Role-based lateral movement detection with unsupervised learning. Intelligent Systems with Applications, 16.

[22] M. A. Salahuddin N. Limam A. A. Daya H. Bian, T. Bai and R. Boutaba. Uncovering lateral movement using authentication logs. IEEE Transactions on Network and Service Management, 18 (1):1049–1063, 2021.

[23] A. A. Daya M. A. Salahuddin N. Limam T. Bai, H. Bian and R. Boutaba. A machine learning approach for rdp-based lateral movement detection. In IEEE 44th Conference on Local Computer Networks (LCN).