Feature Map Correlations in Video Face Forgery Detection

Mazhar, Seyede Fateme; Mansouri, Azadeh

doi:10.22042/isecure.2026.247463

Feature Map Correlations in Video Face Forgery Detection

Articles in Press

Document Type : Research Article

Authors

Seyede Fateme Mazhar

Azadeh Mansouri

Department of Electrical and Computer Engineering, Faculty of Engineering, Kharazmi University, Tehran, Iran

10.22042/isecure.2026.247463

Abstract

Face manipulation techniques have raised widespread public concern. Although conventional convolutional neural networks (CNNs) demonstrate satisfactory performance, they exhibit limitations in capturing the critical features that are directly affected by manipulation artifacts. Fake faces often exhibit unnatural correlations between feature maps, particularly in local patterns. The Gram matrix can illustrate these irregularities in feature map relationships and support discrimination between real and fake images. In this paper, we show that the relationships among convolutional neural network feature maps typically display natural and coherent patterns; in contrast, manipulated faces tend to disrupt this consistency, which can be effectively captured by analysing the Gram matrix of the feature maps. Experimental results demonstrate that the proposed correlation-based features achieve satisfactory performance in both single-dataset and cross-dataset validation.

Keywords

Deepfake

Deep Learning

Fake Video Detection

Feature Map

Gram Matrix

Video Manipulation

[1] Diederik P Kingma, Max Welling, et al. Autoencoding variational bayes, 2013.

[2] Ian J Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair,AaronCourville,andYoshuaBengio. Generative adversarial nets. Advances in neural information processing systems, 27, 2014.

[3] Harshika Goyal, Mohammad Saif Wajid, Mohd Anas Wajid, Akib Mohi Ud Din Khanday, Mehdi Neshat, and Amir Gandomi. Stateof-the-art ai-based learning approaches for deepfake generation and detection, analyzing opportunities, threading through pros, cons, and future prospects. arXiv preprint arXiv:2501.01029, 2025.

[4] Luisa Verdoliva. Media forensics and deepfakes: an overview. IEEE journal of selected topics in signal processing, 14(5):910–932, 2020.

[5] Darius Afchar, Vincent Nozick, Junichi Yamagishi, and Isao Echizen. Mesonet: a compact facial video forgery detection network. In 2018 IEEE international workshop on information forensics and security (WIFS), pages 1–7. IEEE, 2018.

[6] Andreas Rossler, Davide Cozzolino, Luisa Verdoliva, Christian Riess, Justus Thies, and Matthias Nießner. Faceforensics++: Learning to detect manipulated facial images. In Proceedings of the IEEE/CVF international conference on computer vision, pages 1–11, 2019.

[7] David Güera and Edward J Delp. Deepfake video detection using recurrent neural networks. In 2018 15th IEEE international conference on advanced video and signal based surveillance (AVSS), pages 1–6. IEEE, 2018.

[8] Irene Amerini, Leonardo Galteri, Roberto Caldelli, and Alberto Del Bimbo. Deepfake video detection through optical flow based cnn. In Proceedings of the IEEE/CVF international conference on computer vision workshops, pages 0–0, 2019.

[9] Roberto Caldelli, Leonardo Galteri, Irene Amerini, and Alberto Del Bimbo. Optical flow based cnn for detection of unlearnt deepfake manipulations. Pattern Recognition Letters, 146:31– 37, 2021.

[10] Ali Bou Nassif, Qassim Nasir, Manar Abu Talib, and Omar Mohamed Gouda. Improved optical flow estimation method for deepfake videos. Sensors, 22(7):2500, 2022.

[11] Akash Chintha, Aishwarya Rao, Saniat Sohrawardi, Kartavya Bhatt, Matthew Wright, and Raymond Ptucha. Leveraging edges and optical flow on faces for deepfake detection. In 2020 IEEE international joint conference on biometrics (IJCB), pages 1–10. IEEE, 2020.

[12] Zongmei Chen, Xin Liao, Xiaoshuai Wu, and Yanxiang Chen. Compressed deepfake video detection based on 3d spatiotemporal trajectories. In 2024 Asia Pacific Signal and Information Processing Association Annual Summit and Conference (APSIPA ASC), pages 1–8. IEEE, 2024.

[13] Abdelwahab Almestekawy, Hala H Zayed, and Ahmed Taha. Deepfake detection: Enhancing performance with spatiotemporal texture and deep learning feature fusion. Egyptian Informatics Journal, 27:100535, 2024.

[14] Bilal Yousaf, Muhammad Usama, Waqas Sultani, Arif Mahmood, and Junaid Qadir. Fake visual content detection using two-stream convolutional neural networks. Neural Computing and Applications, 34(10):7991–8004, 2022.

[15] Chuangchuang Tan, Yao Zhao, Shikui Wei, Guanghua Gu, Ping Liu, and Yunchao Wei. Frequency-aware deepfake detection: Improving generalizability through frequency space domain learning. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pages 5052–5060, 2024.

[16] Yang Yu, Rongrong Ni, Yao Zhao, Siyuan Yang, Fen Xia, Ning Jiang, and Guoqing Zhao. Msvt: Multiple spatiotemporal views transformer for deepfake video detection. IEEE Transactions on Circuits and Systems for Video Technology, 33(9):4462–4471, 2023.

[17] MuhammadJaved,ZhaohuiZhang,FidaHussain Dahri, and Asif Ali Laghari. Real-time deepfake videodetectionusingeyemovementanalysiswith a hybrid deep learning approach. Electronics, 13(15):2947, 2024.

[18] Amir Hossein Bakhtiari and Azadeh Mansouri. Feature maps correlation-based video quality assessment. Multimedia Tools and Applications, 83(23):63309–63328, 2024.

[19] Alexandros Haliassos, Konstantinos Vougioukas, Stavros Petridis, and Maja Pantic. Lips don’t lie: A generalisable and robust approach to face forgery detection. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 5039–5049, 2021.

[20] Dong-Keon Kim and Kwang-Su Kim. Generalized facial manipulation detection with edge region feature extraction. In Proceedings of the IEEE/CVF winter conference on applications of computer vision, pages 2828–2838, 2022.

[21] Chengrui Wang and Weihong Deng. Representative forgery mining for fake face detection. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 14923–14932, 2021.

[22] Leon A Gatys, Alexander S Ecker, and Matthias Bethge. Image style transfer using convolutional neural networks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 2414–2423, 2016.

[23] Narges Honarjoo, Fatemeh Taher, and Azadeh Mansouri. Exploring feature map correlations for effective fake face detection. In 2024 11th International Symposium on Telecommunications (IST), pages 200–204. IEEE, 2024.

[24] H Schwerdtfeger. Introduction to Linear Algebra and the Theory of Matrices. P. Noordhoff, Groningen, 1950.

[25] Petros Drineas and Michael W Mahoney. Approximating a gram matrix for improved kernelbased learning. In International Conference on Computational Learning Theory, pages 323–337. Springer, 2005.

[26] Yuezun Li, Xin Yang, Pu Sun, Honggang Qi, and Siwei Lyu. Celeb-df: A large-scale challenging dataset for deepfake forensics. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 3207–3216, 2020.

[27] Kaipeng Zhang, Zhanpeng Zhang, Zhifeng Li, and Yu Qiao. Joint face detection and alignment using multitask cascaded convolutional networks. IEEE signal processing letters, 23(10):1499–1503, 2016.

[28] Yaning Zhang, Qiufu Li, Zitong Yu, and Linlin Shen. Distilled transformers with locally enhanced global representations for face forgery detection. Pattern Recognition, 161:111253, 2025.

[29] Rui Yang, Kang You, Cheng Pang, Xiaonan Luo, and Rushi Lan. Cstan: A deepfake detection network with cst attention for superior generalization. Sensors, 24(22):7101, 2024.

[30] Weinan Guan, Wei Wang, Jing Dong, and Bo Peng. Improving generalization of deepfake detectors by imposing gradient regularization. IEEE Transactions on Information Forensics and Security, 19:5345–5356, 2024.

[31] Bosheng Yan, Chang-Tsun Li, and Xuequan Lu. Jrc: Deepfake detection via joint reconstruction and classification. Neurocomputing, 598:127862, 2024.

[32] Xueqi Qiu, Xingyu Miao, Fan Wan, Haoran Duan, Tejal Shah, Varun Ojha, Yang Long, and Rajiv Ranjan. D2fusion: Dual-domain fusion with feature superposition for deepfake detection. Information Fusion, 120:103087, 2025.

[33] Shichao Dong, Jin Wang, Renhe Ji, Jiajun Liang, Haoqiang Fan, and Zheng Ge. Implicit identity leakage: The stumbling block to improving deepfake detection generalization. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 3994–4004, 2023.

[34] Rui Shao, Tianxing Wu, Liqiang Nie, and Ziwei Liu. Deepfake-adapter: Dual-level adapter for deepfake detection. International Journal of Computer Vision, 133(6):3613–3628, 2025.