From CastleCopsWiki

[edit] About on access vs on demand scans

For most part if it cannot detect on demand, on access doesn't help.

Yes the results should be the same. While some AV's do not scan archives or some types of compressed files, but once the baddie is unpacked and executed it will still detect it. The performance degradation from scanning compressed files can be extreme and the malware in a copmressed file can not hurt you any way, so the resident scanner/on access/on execution scanner of most AV's do not scan archives.

BTW On access scanner or Resident scanner are the two most common descriptors although some people do call it an on execution scanner.

However...

This is not always true : for some AVs, some options that are available with the on-access scanner are not available for the resident monitor. For example, Norman Virus Control can use it's sandbox for on-demand scans, for scanning incoming files (email, instant messaging or network shares) but not for the on-access scanner. Most probably because it would cause an excessive slow down.

On the other hand, on-access scanners can sometimes catch malwares that on-demand ones would miss. This situation happens with malware droppers : it is possible that the dropped malware is detected by the on-access scanner while its dropper is not. An some* on-demand scanner will not see the dropped file.

• Note that some on-demand scanners may see the dropped files :

- some low-tech scanners like ClamAV will look for search strings (signatures) through the entire file. If the dropped file is not encrypted or otherwise encoded (this is often the case), ClamAV may find a signature in a file located e.g. in the resource section of the scanned executable, without even noticing it. - some high-tech scanners will emulate through the file, see what files would be dropped by the executable, and recursively scan such files. Bitdefender and Norman are known to do that, it is likely that others do it also. - It might also be possible to look explicitely for the pattern MZ...PE inside every scanned executable (or its resources, if any), or every executable that imports functions for writing something to the disk/launching another executable. But I don't know if any scanner use this technique, since it would probably be quite slow and not improve the detection rate much (?).

Another area is memory scans.

http://web.archive.org/web/20030221025639/www.extremetech.com/article2/0,3973,367093,00.asp

[edit] Memory scanner

True Memory scanners are sometimes available only on demand (not on access), though you will need to execute the sample , then run an on demand memory scan (just normally scanning the file won't dete ct it). Some AT, automatically do a memory scan when a new process starts though, so in this case, the on access scan will detect the sample seconds after it is executed while a scan of the file won't.

There are several types of memory scanners:

1. Fake Memory Scanners

They scan any program running in memory ... with the help of a file scanner.

2. Process Memory Scanners

Real memory scanner but only scans processes (e.g., TDS-3).

3. Process and Module Memory Scanners

Real memory scanner which scans not only processes but also modules (DLLs).

4. "Complete" Memory Scanners

Such scanners scan the entire memory and can also detect code-injecting trojans like Flux. There are various topics on this trojan both in this forum and in our forum. (Again AV/AT software developers got pretty upset when I mentioned that several scanners had difficulties to detect this nasty. But in the meantime, most of them have reacted.)

4) If I have a 'real memory scanner' why should I worry about whether the scanner can do generic unpacking or static unpacking? Won't the memory scanner pick it up when i try to run it ?

As stated by others, the risk is that the scanner may scan the memory before the malware is unpacked (bringing no improvement compared with a file scanner) or after it has been (partly) executed (and may therefore have already performed some malcicious activities). Notice also that during execution of the unpacked samples, some zones of the memory containing the code will be modified, which means that a signature used for this kind of scans should be chosen with care.

True in the sense that no av currently has a real time memory scanner. all av's i'm aware of( with memory scanners, whether true memory scanners, or filedumpers )perform the memory scan only on demand. and then its too late Honestly, I cannot imagine any other way of doing it. You cannot scan the memory of a process after every instruction performed (as the last instruction may just have been the one that finishes the unpacking and starts the unpacked program). Well, you could if you turned all the running applications into full emulation (or tracing), but that would slow the computer down by order of (lots of) magnitudes. So, either you scan the file before you allow it to execute (either using packed signatures, or specific unpackers, or some kind of generic unpackers/emulators), or you let it run and scan the memory afterwards using some "real memory scanner" - but you simply risk activating the malware in the later case.

This wouldn't make any difference in my opinion. I'd say the difference is the following: either the program scans the real content of the memory (call it "real" memory scanner), or it simply enumerates the list of loaded modules (EXE, DLL, ...) and scans the corresponding files on disk. The difference, obviously, is that when scanning the files on disk, they may be packed by various executable packers, wheras when scanning the real memory, they will (usually) be already unpacked.

Unfortunately, Norman doesn't use its Sandbox as a "generic unpacking engine", i.e. it apparently doesn't scan the unpacked process/file after sandboxed execution via signatures, but instead uses only heuristic methods. In contrast to that, Nod32's 'Advanced Heuristic' (based on emulation) obviously does use signature scan, and therefore serves as a reasonable "generic unpacker".

[edit] problems with emultaion

About emulation.

Is it able to emulate through a given packer ? Is it able to emulate the malware itself ? Is it able to flag the sample "behind" the packer as a malware ?

Nod32's 'Advanced Heuristic' (based on emulation) obviously does use signature scan, and therefore serves as a reasonable "generic unpacker".

This is probably a good idea, but not so obvious. The big problem is that, for some packers, you will never have a complete version of the malware unpacked in memory at the same time (this is the case for Armadillo, afaik), or that some parts of its code may have been moved around by the packer (ASProtect "stolen bytes"). For the other packers, the problem is to find when emulation should be stopped: if stopped too early (i.e. before the original entry point), the malware will not be completely unpacked, and stopping it too late results in a loss of time (and in this case, signature has to be chosen carefully). Last but not least : emulating the instructions used to unpack the program is much more time consuming than executing it. Anyway, it is a generic unpacker, not a universal one.

Kaspersky & clones: best static unpacking engine.

Pro: supports MANY packers/compressors/crypters/protectors.

Con: can be easily fooled by minor variations of the unpacking stub.

2. NOD32, Ewido: generic unpacking engine (emulation).

Pro (in theory): no exact signatures of unpacking stub are required in order to determine the packer and apply the matching static unpacking routine. By contrast, it is sufficient that the emulation detects that the target is compressed/crypted etc. Consequently, it is not possible to "fool" an emulation by a minor modification of the unpacking stub of a well-known compressor like UPX.

Con: emulation does not handle as many packers as Kaspersky's static unpacking engine. Tobias Graf (from Ewido) will speak at VB2005 about the concept of a generic unpacking engine ( http://www.virusbtn.com/conference/v...echFri1400.xml ). Hopefully, he will also tell the people how an emulation can handle:

- loops (which exploit the speed disadvantage of an emulation) - anti-emulation code (which stops the execution of a target if it detects that it is executed in a virtual environment) - and much more ...

3. In addition to the use of a static or generic unpacking engine scanners may use heuristics (in order to detect suspicious modifications of a compressed target etc.) or signatures from uncompressed parts of a target (e.g., the resource section). You may also try to pick additional signatures from compressed targets if you do not have an unpacking engine at all ... ;-) And, of course, you may use a memory scanner. Reply With Quote

KAV has a pretty weak detection of stubs, that is true but that is not necessarily a disadvantage for every static unpacker. Interesting enough, KAV fails to detect ASProtect and Armadillo reliable even if it's able to unpack that variant of the protector (unmodified samples).

The main disadvantage is the speed and the only reason Ewido is fast is that they are using assembler and only work on x86 32 bit Intel platforms. An emulation with the same speed on 64 bit SPARC Solaris would impress me much more. ;-) Ewido fails to unpack almost every anti-debugging protector I tested (PE-Shield, telock, PE-Lock32, UPC, Krypton, YodaProt and so on. The problem is not code that detects emulation, actually no one is doing that. The problem is to 100% mirror the OS structures/behaviour like SEH, PEB, KERNEL32.DLL. And you can easily increase the difficulty. Try to emulate Armadillo, SVKP or Molebox. The malware authors seem to have noticed this, in the last few weeks and months you could notice a slow increase in usage of the mentioned protectors. Which will basically makes emulation useless at some point if the trend continues. The AV scan engines are loosing the race already I fear.

speed is one of the disadvantages of generic unpacking using emulation. It was a real problem until we improved performance quite a lot by adding dynamic code translation, but scanning performance on packed samples is still slower than when using static unpacking. But generic unpacking is much more versatile, for example there is a good chance that newer versions of executable packers or protectors can be unpacked correctly with no changes at all while static unpacking would require to add new decompression routines.

Well, emulation has proven to be very efficient for unpacking ordinary packers (designed only to reduce executable size) and low to middle level protectors. Heavyweight protectors like SVKP or XPROTECTOR are a real problem. But they are very inconvenient for the users too (use drivers - sometimes cause bluescreens, also they only work for the users with administrator privileges). Any program protected with such monsters is a potential malware so maybe antivirus can issue some alert in this case and let the user decide what to do.

Emulation is a method used for detecting some polymorphic viruses and, sometimes, for unpacking. Polymorphic viruses are composed of : - a small decryptor, that should alwas be different in two different samples of the virus. - an encrypted portion of code, that performs all the virus tasks (including the creation of mutated decrypors).

When the virus is launched, the execution is passed to the decryptor and the virus code is decrypted in memory. The decryptor then transfers the execution to the begining of this newly decrypted code.

Emulating through the decryptor is a way to let the virus decrypt its own code while keeping the code execution in a secure environnment (the instructions are not executed on the real CPU).

Runtime packed executables are also composed of two parts, the unpacker and the packed code. When the program is launched, the "unpacker" code is executed first and the execution is then transfered to the beginning of the newly unpacked code.

Generic unpacking consists in : 1/ emulating the "unpacker" part of the executable 2/ detecting when the execution is transfered to the original code 3/ launching a signature scan on the resulting piece of code

There are other methods for doing generic unpacking (all are based on the same principle : let the program unpack itself): - Single-stepping. This method consists in letting the program execute itself on the real CPU one instruction at a time, which allows to test, between two instructions of the program, if the execution has been transfered to the original code. This is technique uses techniques employed by the debuggers. However, there are two major problems with this method : it's awfully slow and it is not safe (the execution could run out of control). It is therefore not used in antiviruses. You can find the source code of such a generic unpacker here : http://wave.prohosting.com/mackt/pro...uw/guw32b8.zip However, remember that it must not be used to unpack malware (moreover, I think that it will not work under XP)

- Dynamic translation. Instead of emulating all the instruction, it is possible to rewrite the instructions in such a way that they will not interact directlty with your OS and to run these "translated" instructions on the real CPU afterwards. "Translating" an instruction is not significantly faster than emulating it. The main advantage is that each instruction is translated only once, whereas some instructions would have been emulated several time during the execution of the program (every time a routine is called/for every iterations of a loop). As a result, dynamic translation can be much faster than emulation. I must say that I did not understand yet how it can proceed safely with self-modifying code. Which is the heart of the problem. This topic will be adressed during the next Virus Bulletin conference : http://www.virusbtn.com/conference/v...chWeds1700.xml

- There are several "more or less generic" simple methods that can be used when manually unpacking samples (e.g. breaking into some windows API that are almost always called when a program has been created with some compilers), but these method are not secure and they let the code run on the real CPU and interact with the OS. They cannot be used by antiviruses, or should be coupled with emulation, etc...

NOD's heuristic engine uses emulation and/or dynamic translation in order to safely observe the execution flow (which windows API functions are called, etc.) It would be better to say that NOD's heuristic and generic unpacking engines both use the same emulation/translation primitives.

3) What is the difference between heuristics and behaviour blocker. Also Would behaviour blocking involve emulation as well?

Behaviour blockers let the code run normally on the CPU. They put some filters between the OS functions and the calling programs. When some of these functions are called (i.e. when creating a registry key, writing to a file, etc...) with parameters that look suspicioous (the registry key will allow a program to run at next boot, the file is an executable file), the behaviour blocker raises a warning and stops the execution of the program.

However, some heuristic engines observe the behavior of the program under emulation, and especially which APIs are called, with which parameters, to decide whether the executable should be classified as a malware. This is for example the method used by Norman's Sandbox : http://sandbox.norman.no/pdf/03_sand...whitepaper.pdf In this case, the OS "reactions" also have to be emulated, which makes the problem more difficult.

eatures:

Listed features are related to proactive/early detection (or blocking, reporting) of malicious Win32 PE executables. Based on the behavior of the malwares rather than on signatures/patterns.

• Behavior-based heuristics: Code of analyzed executables is emulated by the

antivirus. Suspicious actions are recorded and compared with heuristic rules. Norman's Sandbox is the typical example.

• On execution behavioral detection: Similar to above, but the file is

actually executed on the real system, and it's actual actions are logged. If the application is classified as malicious, its execution is stopped. There may be some funcrtionnalites to mitigate the impact of actions already performed by the malware.

• Live reporting of 0days: The application provides an utility to submit

automatically to the virus lab malwares that have been detected on a client's computer using a proactive method.

• ACLs on specific actions/API calls: The application provides the ability

to block some specific actions that are commonly performed by some kind of malwares (e.g. execute files from temp directory, alter some registry keys, etc.). This includes typical HIPS functionnalities.

• Outbreak protection (throttle): The application is able to recognize an

abnormal behaviour of the system (such as sending mails at a high rate) and take adequate measures to mitigate the risk due to this behavior. The application may not identify what executable or script caused the abnormal behaviour.