This is a very interesting paper from some smart people at Columbia University. Here is my layman's summary for the terminally lazy:
We are going to model the feasibility of modeling polymorphic shellcode to see if we can rely on antivirus heuristics and behavioral detection techniques.
Examine a ton of models & do some math that makes my head hurt.
"Our empirical results demonstrate the difficulty of modeling polymorphic behavior. We briefly summarized the achievements of the shellcoder community in making their code polymorphic and examined ways to improve some of these techniques. We presented analytical methods that can help assess the capabilities of polymorphic engines and applied them to some state-of-the-art engines. We explained why signature–based modeling works in some cases and confirmed that the viability of such approaches matches the intuitive belief that polymorphism will eventually defeat these methodologies. The strategy of modeling malicious behavior leads to an unending arms race with an attacker. Alternatively, whitelisting normal content or behavior patterns (perhaps in randomized ways in order to defend against blending attacks) might ultimately be safer than blacklisting arbitrary and highly varied malicious behavior or content."
Welcome to In Depth Defense. In Depth Defense LLC is a privately owned Information Security Consulting company owned and operated by Mark Baggett. In Depth Defense specializes in Penetration Testing and Incident Response. At this time In Depth Defense is not accepting any new client work, but we are happy to speak to you and point you to other resources in the community.
Mark Baggett has been active in Information Security for 18+ years. I've served in a variety of roles from software developer to CISO. You can find archives of older blog entries below and read my newer posts on http://www.pauldotcom.com, http://isc.sans.edu and http://pen-testing.sans.org