Classification of Malware Using Structured Control Flow

Cesare, S. and Xiang, Y.

Malware is a pervasive problem in distributed computer and network systems. Identification of malware variants provides great benefit in early detection. Control flow has been proposed as a characteristic that can be identified across variants, resulting in flowgraph based malware classification. Static analysis is widely used for the classification but can be ineffective if malware undergoes a code packing transformation to hide its real content. This paper proposes a novel algorithm for constructing a control flow graph signature using the decompilation technique of structuring. Similarity between structured graphs can be quickly determined using string edit distances. To reverse the code packing transformation, a fast application level emulator is proposed. To demonstrate the effectiveness of the automated unpacking and flowgraph based classification, we implement a complete system and evaluate it using synthetic and real malware. The evaluation shows our system is highly effective in terms of accuracy in revealing all the hidden code, execution time for unpacking, and accuracy in classification.

Cite as: Cesare, S. and Xiang, Y. (2010). Classification of Malware Using Structured Control Flow. In Proc. Eighth Australasian Symposium on Parallel and Distributed Computing (AusPDC 2010) Brisbane, Australia. CRPIT, 107. Chen, J. and Ranjan, R. Eds., ACS. 61-70

(from crpit.com) (local if available)