ProAPT: Projection of APTs with Deep Reinforcement Learning

Dehghan, Motahareh; Sadeghiyan, Babak; Khosravian, Erfan; Sedighi Moghadam, Alireza; Nooshi, Farshid

doi:10.22042/isecure.2024.428569.1052

ProAPT: Projection of APTs with Deep Reinforcement Learning

Document Type : Research Article

Authors

Motahareh Dehghan ¹

Babak Sadeghiyan ²

Erfan Khosravian ³

Alireza Sedighi Moghadam ²

Farshid Nooshi ²

¹ Department of Industrial and Systems Engineering, Tarbiat Modares University, Tehran, Iran

² Computer Engineering Department, Amirkabir University of Technology, Tehran, Iran

³ Mechanical Engineering Department, Payame Noor University, Lashkarak Highway, Tehran, Iran

https://doi.org/10.22042/isecure.2024.428569.1052

Abstract

The highest level in Endsley's situation awareness model is called projection when the status of elements in the environment is shortly predicted. In cybersecurity situation awareness, the projection for an Advanced Persistent Threat (APT) requires to predict the next step of the APT.
The threats are constantly changing and becoming more complex. As supervised and unsupervised learning methods require APT datasets for projecting the next step of APTs, they cannot identify unknown APT threats.
In reinforcement learning methods, the agent interacts with the environment, which might project the next step of known and unknown APTs. So far, reinforcement learning has not been used to project the next step of APTs.
In reinforcement learning, the agent uses the previous states and actions to approximate the best action of the current state. When the number of states and actions is abundant, the agent employs a neural network to approximate the best action of each state.
This paper presents a deep reinforcement learning system to project the next step of APTs. As there exists some relation between attack steps, we employ the Long Short Term Memory method to approximate the best action of each state. In our proposed system, based on the current situation, we project the next steps of APT threats.
We have evaluated our proposed system on the DAPT2020 dataset. Based on the evaluations performed on the mentioned dataset, six criteria F1, accuracy, precision, recall, loss, and average time were obtained, which are 0.9533, 0.9736, 0.9352, 0.97, 0.0143, and 0.05749(seconds), respectively.

Keywords

Situation Awareness

Advanced Persistent Threats

Projection

Deep Reinforcement Learning

LSTM

DAPT2020

SCVIC-APT-202

[1] John M Flach. Situation awareness: Context matters! a commentary on endsley. Journal of Cognitive Engineering and Decision Making, 9(1):59–72, 2015.
[2] Mica R Endsley. Toward a theory of situation awareness in dynamic systems. Human factors, 37(1):32–64, 1995.
[3] Martin Hus´ak, Jana Kom´arkov´a, Elias Bou-Harb and Pavel ˇCeleda. Survey of attack projection, prediction, and forecasting in cyber security. IEEE Communications Surveys & Tutorials, 21(1):640–660, 2018.
[4] Richard S Sutton, Andrew G Barto, et al. Introduction to reinforcement learning. vol. 135, 1998.
[5] Fred Cohen. Simulating cyber attacks, defences, and consequences. Computers & Security, 18(6):479–518, 1999.
[6] Anirban Chakrabarti and Govindarasu Manimaran. Internet infrastructure security: A taxonomy. IEEE network, 16(6):13–21, 2002.
[7] Shanchieh J Yang, Adam Stotz, Jared Holsopple, Moises Sudit, and Michael Kuhl. High level information fusion for tracking and projection of multistage cyber attacks. Information Fusion, 10(1):107–121, 2009.
[8] Martin Hus´ak and Jaroslav Kaˇspar. Aida framework: real-time correlation and prediction of intrusion detection alerts. In Proceedings of the 14th international conference on availability, reliability and security, pages 1–8, 2019.
[9] Nikolaos Polatidis, Elias Pimenidis, Michalis Pavlidis, and Haralambos Mouratidis. Recommender systems meeting security: From product recommendation to cyber-attack prediction. In Engineering Applications of Neural Networks: 18th International Conference, EANN 2017, Athens, Greece, August 25–27, 2017, Proceedings, pages 508–519. Springer, 2017.
[10] Nikolaos Polatidis, Elias Pimenidis, Michalis Pavlidis, Spyridon Papastergiou, and Haralambos Mouratidis. From product recommendation to cyber-attack prediction: Generating attack graphs and predicting future attacks. Evolving Systems, 11:479–490, 2020.
[11] Ahmet Okutan, Shanchieh Jay Yang, and Katie McConky. Predicting cyber attacks with bayesian networks using unconventional signals. In Proceedings of the 12th Annual Conference on Cyber and Information Security Research, pages
1–4, 2017.
[12] Kaixing Huang, Chunjie Zhou, Yu-Chu Tian, Shuanghua Yang, and Yuanqing Qin. Assessing the physical impact of cyberattacks on industrial cyber-physical systems. IEEE Transactions on Industrial Electronics, 65(10):8153–8162, 2018.
[13] Daniel S Fava, Stephen R Byers, and Shanchieh Jay Yang. Projecting cyberattacks through variable-length markov models. IEEE Transactions on Information Forensics and Security, 3(3):359–369, 2008.
[14] Dongmei Zhao, Hongbin Wang, and Shixun Geng. Compound attack prediction method based on improved algorithm of hidden markov model. Journal of web engineering, 19(7–8):1213–1238, 2020.
[15] Sanjana Ingale, Milind Paraye, and Dayanand Ambawade. Enhancing multi-step attack prediction using hidden markov model and naive bayes. In 2020 International Conference on Electronics and Sustainable Communication Systems(ICESC), pages 36–44. IEEE, 2020.
[16] Timothy Chadza, Konstantinos G Kyriakopoulos, and Sangarapillai Lambotharan. Analysis of hidden markov model learning algorithms for the detection and prediction of multi-stage network attacks. Future generation computer systems,
108:636–649, 2020.
[17] Ibrahim Ghafir, Konstantinos G Kyriakopoulos, Sangarapillai Lambotharan, Francisco J Aparicio-Navarro, Basil Assadhan, Hamad Binsalleeh, and Diab M Diab. Hidden markov models and alert correlations for the prediction of advanced persistent threats. IEEE Access, 7:99508–99520, 2019.
[18] Mohamed Abdlhamed, Kashif Kifayat, Qi Shi, and William Hurst. A system for intrusion prediction in cloud computing. In Proceedings of the International Conference on Internet of things and Cloud Computing, pages 1–9, 2016.
[19] Radek P´ıbil, Viliam Lis`y, Christopher Kiekintveld, Branislav Boˇsansk`y, and Michal Pˇechouˇcek. Game theoretic model of strategic honeypot selection in computer networks. In Decision and Game Theory for Security: Third International Conference, GameSec 2012, Budapest, Hungary, November 5-6, 2012. Proceedings 3, pages 201–220. Springer, 2012.
[20] Fannv He, Yuqing Zhang, Donghang Liu, Ying Dong, Caiyun Liu, and Chensi Wu. Mixed wavelet-based neural network model for cyber security situation prediction using modwt and hurst exponent analysis. In Network and System Security: 11th International Conference, NSS 2017, Helsinki, Finland, August 21–23, 2017, Proceedings 11, pages 99–111. Springer, 2017.
[21] Xiaorong Cheng and Su Lang. Research on network security situation assessment and prediction. In 2012 Fourth international conference on computational and information sciences, pages 864–867. IEEE, 2012.
[22] Gaya K Jayasinghe, J Shane Culpepper, and Peter Bertok. Efficient and effective realtime prediction of drive-by download attacks. Journal of Network and Computer Applications, 38:135–149, 2014.
[23] Solomon Ogbomon Uwagbole, William J Buchanan, and Lu Fan. Applied machine learning predictive analytics to sql injection attack detection and prevention. In 2017 IFIP/IEEE Symposium on Integrated Network and Service Management (IM), pages 1087–1090. IEEE, 2017.
[24] Ouissem Ben Fredj, Alaeddine Mihoub, Moez Krichen, Omar Cheikhrouhou, and Abdelouahid Derhab. Cybersecurity attack prediction: a deep learning approach. In 13th international conference on security of information and networks,
pages 1–6, 2020.
[25] Xing Fang, Maochao Xu, Shouhuai Xu, and Peng Zhao. A deep learning framework for predicting cyber attacks rates. EURASIP Journal on Information security, 2019:1–11, 2019.
[26] Yu-Beng Leau and Selvakumar Manickam. A novel adaptive grey verhulst model for network security situation prediction. International Journal of Advanced Computer Science and Applications, 7(1), 2016.
[27] Ornella Lucresse Soh. Advanced persistent threat detection using anomaly score calibration and multi-class classification. PhD thesis, 2023.
[28] Soo Yin Yi, Manmeet Mahinderjit Singh, Gian Chand Sodhy, and Thulfiqar Jabar. Fingerprinting generation for advanced persistent threats (apt) detection using machine learning techniques. In 2023 13th International Conference on Information Technology in Asia (CITA), pages 31–36. IEEE, 2023.
[29] Abel Yeboah-Ofori, Haralambos Mouratidis, Umar Ismai, Shareeful Islam, and Spyridon Papastergiou. Cyber supply chain threat analysis and prediction using machine learning and ontology. In Artificial Intelligence Applications and Innovations: 17th IFIP WG 12.5 International Conference, AIAI 2021, Hersonissos, Crete, Greece, June 25–27, 2021, Proceedings 17,
pages 518–530. Springer, 2021.
[30] Omar Azib Alkhudaydi, Moez Krichen, and Ans D Alghamdi. A deep learning methodology for predicting cybersecurity attacks on the internet of things. Information, 14(10):550, 2023.
[31] Surjeet Dalal, Poongodi Manoharan, Umesh Kumar Lilhore, Bijeta Seth, Deema Mohammed alsekait, Sarita Simaiya, Mounir Hamdi and Kaamran Raahemifar. Extremely boosted neural network for more accurate multi-stage cyber attack prediction in cloud computing environment. Journal of Cloud Computing, 12(1):14, 2023.
[32] Moustafa Mahmoud, Mohammad Mannan, and Amr Youssef. Apthunter: Detecting advanced persistent threats in early stages. Digital Threats: Research and Practice, 4(1):1–31, 2023.
[33] Nguyen Cong Luong, Dinh Thai Hoang, Shimin Gong, Dusit Niyato, Ping Wang, Ying-Chang Liang, and Dong In Kim. Applications of deep reinforcement learning in communications and networking: A survey. IEEE communications surveys & tutorials, 21(4):3133–3174, 2019.
[34] Ian Goodfellow. Deep Learning. MIT Press, 2016.
[35] Ali Ahmadian Ramaki, Abbas Ghaemi-Bafghi, and Abbas Rasoolzadegan. Towards event aggregation for reducing the volume of logged events during ikc stages of apt attacks. arXiv preprint arXiv:2109.14303, 2021.
[36] Yussuf Ahmed, A Taufiq Asyhari, and Md Arafatur Rahman. A cyber kill chain approach for detecting advanced persistent threats. Computers, Materials and Continua, 67(2):2497–2513, 2021.
[37] Targeted attack lifecycle.
[38] Sowmya Myneni, Ankur Chowdhary, Abdulhakim Sabur, Sailik Sengupta, Garima Agrawal, Dijiang Huang, and Myong Kang. Dapt 2020-constructing a benchmark dataset for advanced persistent threats. In Deployable Machine Learning for Security Defense: First International Workshop, MLHat 2020, San Diego, CA, USA, August 24, 2020, Proceedings 1, pages 138–163. Springer, 2020.
[39] Jinxin Liu, Yu Shen, Murat Simsek, Burak Kantarci, Hussein T Mouftah, Mehran Bagheri, and Petar Djukic. A new realistic benchmark for advanced persistent threats in network traffic. IEEE Networking Letters, 4(3):162–166, 2022.
[40] Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. Advances in neural information processing systems, 27, 2014.
[41] Zhaoqing Pan, Weijie Yu, Xiaokai Yi, Asifullah Khan, Feng Yuan, and Yuhui Zheng. Recent progress on generative adversarial networks(gans): A survey. IEEE access, 7:36322–36333, 2019.
[42] Lei Xu, Maria Skoularidou, Alfredo CuestaInfante, and Kalyan Veeramachaneni. Modeling tabular data using conditional gan. Advances in neural information processing systems, 32, 2019.
[43] Synthetic Data Vault (SDV). Copulagan model.
[44] DARPA. n.d. Transparent computing.
[45] The bot-iot dataset, 2019.
[46] Abderahman Rejeb, Karim Rejeb, Steve Simske, Horst Treiblmaier, and Suhaiza Zailani. The big picture on the internet of things and the smart city: a review of what we know and what we need to know. Internet of Things, 19:100565, 2022.
[47] DEFCON. Hacking conference home.