An LSTM-DBSCAN Approach for Interpretable Insider Threat Detection via Behavioural Anomaly Analysis

Mohammadi, Mohammad; Bannaye Zahmati, Moein; Noferesti, Morteza

doi:10.22042/isecure.2026.241277

An LSTM-DBSCAN Approach for Interpretable Insider Threat Detection via Behavioural Anomaly Analysis

Articles in Press

Document Type : Research Article

Authors

Mohammad Mohammadi

Moein Bannaye Zahmati

Morteza Noferesti

Department of Computer Engineering, Bozorgmehr University of Qaenat, Qaen, South Khorasan, Iran.

10.22042/isecure.2026.241277

Abstract

Insider threats pose a significant cybersecurity risk, as authorised users can exploit legitimate access to compromise sensitive systems and data. This paper proposes an integrated behavioural anomaly detection approach to address three critical challenges in AI-driven insider threat detection: lack of interpretability, misleading evaluation metrics, and misalignment with operational taxonomies. Our approach employs a three-stage pipeline: (1) an LSTM autoencoder to detect temporal anomalies in login patterns, (2) DBSCAN clustering to identify suspicious file access and device usage during anomalous sessions, and (3) DBSCAN-based URL analysis to uncover exfiltration patterns. By analysing behaviour across time, location, and web activity, this framework builds actionable threat chains mapped to MITRE ATT&CK techniques including T1078, T1005, T1204.002, T1567.002. It bridges the gap between theoretical models and the daily work of a Security Operations Center (SOC). In the data exfiltration scenario on the CERT R6.2 insider threat dataset, the proposed approach achieved a recall of 83.3% and an accuracy of 91.7% in classifying malicious days. The framework also provides interpretable alerts and maintains operational efficiency.

Keywords

Insider threat detection

LSTM

DBSCAN clustering

MITRE ATT&‌‌‌‌‌CK

Behavioural anomaly analysis

[1] Xiangrui Cai, Yang Wang, Sihan Xu, Hao Li, Ying Zhang, Zheli Liu, and Xiaojie Yuan. Lan: learning adaptive neighbors for real-time insider threat detection. IEEE Transactions on Information Forensics and Security, 2024.

[2] Usman Rauf, Fadi Mohsen, and Zhiyuan Wei. A taxonomic classification of insider threats: Existing techniques, future directions & recommendations. Journal of Cyber Security and Mobility, 12(2):221–252, 2023.

[3] Shuhan Yuan and Xintao Wu. Deep learning for insider threat detection: Review, challenges and opportunities. Computers & Security, 104:102221, 2021.

[4] Fatima Rashed Alzaabi and Abid Mehmood. A review of recent advances, challenges, and opportunities in malicious insider threat detection using machine learning methods. IEEE Access, 12:30907–30927, 2024.

[5] Chunrui Zhang, Shen Wang, Dechen Zhan, Tingyue Yu, Tiangang Wang, and Mingyong Yin. Detecting insider threat from behavioral logs based on ensemble and self-supervised learning. Security and Communication Networks, 2021(1):4148441, 2021.

[6] S Asha, D Shanmugapriya, and G Padmavathi. Maliciousinsiderthreatdetectionusingvariation of sampling methods for anomaly detection in cloud environment. Computers and Electrical Engineering, 105:108519, 2023.

[7] Brian Lindauer. Insider Threat Test Dataset. 9 2020.

[8] Bader Al-Sada, Alireza Sadighian, and Gabriele Oligeri. Mitre att&ck: State of the art and way forward. ACM Computing Surveys, 57(1):1–37, 2024.

[9] P Rajesh, Mansoor Alam, Mansour Tahernezhadi, A Monika, and Gm Chanakya. Analysis of cyber threat detection and emulation using mitre attack framework. In 2022 International Conference on Intelligent Data Science Technologies and Applications (IDSTA), pages 4–12. IEEE, 2022.

[10] Vector Guo Li, Matthew Dunn, Paul Pearce, Damon McCoy, Geoffrey M. Voelker, and Stefan Savage. Reading the tea leaves: A comparative analysis of threat intelligence. In 28th USENIX Security Symposium (USENIX Security 19), pages 851–867, Santa Clara, CA, August 2019. USENIX Association.

[11] Haitao Xiao, Yan Zhu, Bin Zhang, Zhigang Lu, Dan Du, and Yuling Liu. Unveiling shadows: A comprehensive framework for insider threat detection based on statistical and sequential analysis. Computers & Security, 138:103665, 2024.

[12] Motahareh Dehghan, Babak Sadeghian, Erfan Khosravian, Alireza Sedighi Moghaddam, and Farshid Nooshi. Proapt: Projection of apts with deep reinforcement learning. ISeCure, 17(1), 2025.

[13] P Lavanya, H Anila Glory, and VS Shankar Sriram. Mitigating insider threat: a neural network approach for enhanced security. IEEE Access, 2024.

[14] Preetam Pal, Pratik Chattopadhyay, and MayankSwarnkar. Temporalfeatureaggregation with attention for insider threat detection from activity logs. Expert Systems with Applications, 224:119925, 2023.

[15] Duc C Le, Nur Zincir-Heywood, and Malcolm Heywood. Training regime influences to semisupervised learning for insider threat detection. In 2021 IEEE Security and Privacy Workshops (SPW), pages 13–18. IEEE, 2021.

[16] Raghav Bhardwaj, Morteza Noferesti, Madeline Janecek, and Naser Ezzati-Jivan. Emd-scs: A dynamic behavioral approach for early malware detection with sonification of system call sequences. In 2023 IEEE 22nd International Conference on Trust, Security and Privacy in Computing and Communications (TrustCom), pages 1728–1737. IEEE, 2023.

[17] Yue Guan, Morteza Noferesti, and Naser EzzatiJivan. A two-tiered framework for anomaly classification in iot networks utilizing cnn-bilstm model. Software Impacts, 20:100646, 2024.

[18] Vipin Kumar and Basant Subba. A tfidfvectorizer and svm based sentiment analysis framework for text data corpus. In 2020 national conference on communications (NCC), pages 1–6. IEEE, 2020.