Emerging Threats to Online Security: Securing Systems Against Unauthorized Automation and Web Bots

Abstract

Internet services increasingly suffer from unwanted automation through bots, which pose significant challenges including financial losses, security breaches, and diminished user trust. The ability of bots to convincingly emulate human interactions complicates detection efforts, particularly as advancements in machine learning enable increasingly sophisticated automated agents. Bots are used to carry out a wide range of attacks, including credential stuffing, web scraping, and distributed denial-of-service attacks. Traditional countermeasures, such as CAPTCHAs, have become increasingly ineffective due to advances in artificial intelligence, highlighting the need for alternative detection approaches.

This cumulative dissertation addresses key challenges in bot detection, analysis, and prevention, with the goal of mitigating bot-related risks through novel, non-intrusive, and scalable solutions.

For bot detection, approaches are introduced that leverage the interaction behaviors of humans with web-based services, such as mouse movements, typing patterns, and website navigation. Unlike traditional methods that rely on static identifiers (e.g., IP addresses) or explicit user challenges (e.g., CAPTCHAs), these methods passively distinguish humans from bots by analyzing user interaction patterns using machine learning based detection models trained on behavioral data. Evaluations of synthetic and human behavior demonstrate the effectiveness of these approaches.

Effective bot defense also requires the analysis of bot software to uncover operational strategies and vulnerabilities. A method is presented to accelerate the reverse engineering of closed-source applications, a critical yet resource-intensive task. Specifically, dynamic binary instrumentation is employed to systematically identify and prioritize critical code segments (Points-of-Interest) related to sensitive data, as specified by the analyst (Items-of-Interest). Empirical validation on complex malware, including ransomware and peer to peer botnets, demonstrates substantial efficiency improvements and reliable identification of key functionalities.

To address the scalability challenge posed by automated bots, a preventive strategy is proposed that targets API-based automation. Existing obfuscation techniques primarily hinder the initial creation of bots but fail to prevent subsequent large-scale deployments. The proposed method obfuscates client-server communication protocols by assigning distinct protocols to each client. Consequently, attackers are required to reverse engineer each instance individually, significantly increasing the cost and complexity of large-scale bot operations without disrupting legitimate user interactions.

This thesis presents methods to enhance bot defenses across detection, analysis, and prevention. While it advances all three areas, challenges remain. As defenses improve, a shift toward UI-based bots that circumvent APIs is expected. Moreover, ongoing progress in AI will further blur the distinction between human users and automated agents, undermining current detection techniques. A promising direction lies in strong, yet privacy-preserving authentication mechanisms that robustly bind virtual identities to human identities.