Inference and Analysis of Formal Models of Botnet Command and Control Protocols

[PDF] [PS] [PS.BZ2]

@article{cho10botnets,
  author = {Chia Yuan Cho and Domagoj Babi\'c and 
    Richard Shin and Dawn Song},
  title = {Inference and Analysis of Formal Models of 
    Botnet Command and Control Protocols},
  booktitle = {CCS'10: Proceedings of the 2010 ACM 
    Conference on Computer and Communications Security},
  year = {2010},
  publisher = {ACM},
  pages = {426--440},
  location = {Chicago, Illinois, USA},
}

Abstract: We propose a novel approach to infer protocol state machines in the realistic high-latency network setting, and apply it to the analysis of botnet Command and Control (C&C) protocols. Our proposed techniques enable an order of magnitude reduction in the number of queries and time needed to learn a botnet C&C protocol compared to classic algorithms (from days to hours for inferring the MegaD C&C protocol). We also show that the computed protocol state machines enable formal analysis for botnet defense, including finding the weakest links in a protocol, uncovering protocol design flaws, inferring the existence of unobservable communication back-channels among botnet servers, and finding deviations of protocol implementations which can be used for fingerprinting. We validate our technique by inferring the protocol state-machine from Postfix's SMTP implementation and comparing the inferred state-machine to the SMTP standard. Further, our experimental results offer new insights into MegaD's C&C, showing our technique can be used as a powerful tool for defense against botnets.