The ISC International Journal of Information Security

Abstract Distributed Denial-of-Service (DDoS) attacks are among the most critical security threats to distributed network infrastructures, including blockchain systems. These attacks degrade performance, cause congestion, and disrupt service delivery or transaction processing. Traditional mitigation techniques have undergone extensive development. However, they often fail to intelligently detect and manage traffic patterns and struggle to adapt to dynamic conditions in decentralized environments. This paper proposes a reinforcement learning-based congestion control (CC) method that dynamically adjusts congestion window (CWND) following traditional TCP principles based on signals such as delay and packet loss. What distinguishes our approach is that the RL-agent interprets persistent or abnormal congestion patterns as potential indicators of adversarial high-load conditions (e.g., DDoS-induced congestion) and adapts CWND adjustments more intelligently to reduce their adverse. Leveraging the Q-learning algorithm, the proposed approach adapts dynamically to fluctuating traffic and conditions. Its learning capability enables continuous monitoring of behavior and timely responsiveness to anomalies, including sustained congestion patterns often associated with adversarial traffic surges. Simulation results across various DDoS scenarios—evaluated against conventional CC algorithms—demonstrate considerable improvements in key performance indicators such as reduced latency, enhanced bandwidth utilization, improved stability, decreased packet loss, and increased throughput. The proposed Q-learning-based CC operates at the peer-to-peer layer, regulating flow among blockchain nodes. It is independent of consensus mechanisms while indirectly improving consensus efficiency by reducing message delays and packet loss. This method offers a scalable and intelligent solution for cc under adversarial conditions, thereby contributing to improved robustness and efficiency in both general distributed systems and blockchain networks.