While online social networks (OSNs) provide application programming interfaces (APIs) to enable the development of OSN applications, some of these applications, unfortunately, can be malicious. They can be running on the devices for OSN users throughout the Internet, causing security, privacy, and liability concerns to the network service providers of these OSN users.

In this thesis, we study how a network service provider may inspect its network traffic to detect network flows from malicious API-based OSN applications. In particular, we devise a deep learning based methodology to detect network flows generated by malicious API-based OSN applications. We implement this methodology on a testbed, and show that our solution is effective and can accurately label 97.6% network flows from the malicious OSN applications, with only 1.6% false positives.