Application-layer distributed denial-of-service (L7 DDoS) attacks, by exploiting application-layer requests to overwhelm functions or components of victim servers, has become a major rising threat to today's Internet. However, because the traffic from an L7 DDoS attack appears totally legitimate in transport and network layers, it is difficult to detect and defend against an L7 DDoS attack with traditional DDoS solutions.

In this paper, we propose a new, reinforcement-learning-based approach to detecting and mitigating L7 DDoS attacks. By continuously monitoring and analyzing the system load of the victim server, the dynamic behaviors of clients, and the network load of the victim server, our approach can choose one of the most suitable mitigation actions, such as blocking DDoS upstream, blocking DDoS locally, or postponing L7 requests, thus achieving the best mitigation efficacy of the L7 DDoS attack. Moreover, with the help of a new multi-objective reward function, when a L7 DDoS attack is overwhelming the reinforcement learning agent can selectively sacrifice legitimate requests to keep the victim server functioning, and when otherwise the agent affects little legitimate requests.

Our evaluation results show that our approach can protect a victim server from L7 DDoS attacks effectively by detecting 98.73% of the L7 DDoS traffic flows at the peak system load, with most of them mitigated.