Edinburgh Napier University

  Despite the continuous updating of anti-detection systems for malicious programs (malware), malware has moved to an abnormal threat level; it is being generated and spread faster than before. One of the most serious challenges faced by anti-detection malware programs is an automatic mutation in the code; this is called polymorphic malware via the polymorphic engine. In this case, it is difficult to block the impact of signature-based detection. Hence new techniques have to be used in order to analyse modern malware. One of these techniques is machine learning algorithms in a virtual machine (VM) that can run the packed malicious file and analyse it dynamically through automated testing of the code. Moreover, recent research used image processing techniques with deep learning framework as a hybrid method with two analysis types and extracting a feature engineering approach in the analysis process in order to detect polymorphic malware efficiently. This paper presents a brief review of the latest applied techniques against this type of malware with more focus on the machine learning method for analysing and detecting polymorphic malware. It will discuss briefly the merits and demerits of it.