AntiVirus features
From CastleCopsWiki
| Caution | The article below is currently in beta and has not been reviewed for factual errors. |
Contents |
[edit] Introduction
What exactly should you be looking out for in a good AntiVirus? For certain factors like price, stability, user friendliness , resource usage are important features. But beyond those features that we all take for granted, most users would agree that the real worth of an Antivirus comes from it's brute ability to detect malware and clean infections.
Of these two abilities, the former (detection of malware) usually takes precedence in the minds of most people, because if your Antivirus is able to protect you by detecting and zapping malware in a timely fashion, infection never actually has a chance to occur.
Cleaning ability is important of course for people seeking help , but for those of you seeking a Antivirus to protect your clean system from protection, detection rates should be your key focus.
[edit] Measuring Detection ability
What should we then decide to use to judge if an antivirus has good detection rates? Let us first start with factors we shouldn't use.
[edit] anecdotal stories
We all hear stories about how someone used to trust Antivirus X, until it let some malware through, and now they use Antivirus Y instead. While such stories might be truth it still doesn't allow one to have a reasonable grounds to judge that Antivirus X is worse. For example it might be possible that if such a person was using Antivirus Y in the first place, he might get infected by something that Antivirus X would have blocked! So we definitely need more evidence.
[edit] Size of signature base or number of signatures
Some people like to claim that antiviruses with more signatures are better because they can detect more baddies. Unfortunately, antiviruses signatures are not all equal and such simple minded focus on numbers fails to take into account signature quality. Some antiviruses can detect many variants of the same type of malware with one signature, while other antiviruses need several signatures.
For a simple made up example, to an antivirus that does not support UPX unpacking, each and every malware packed that way requires a separate signature before detection! So such an antivirus would have more signatures despite not having better detection,We will have more to say about signature quality later.
Here are a couple of factors that *might* have some bearing on detection ability
[edit] Frequency of updates
Some antiviruses have a regular policy to push new antivirus signature updates several times a day (the top ones do it once an hour), while others have a policy of longer periods (up to once a week), and yet others don't follow any regular update cycle. In many cases, updates are spaced apart not because the analysts are working slower, but rather, to reduce server load. Given that antiviruses are literally worthless without signatures it would seem the faster you get regular updates the better. Of course, even for antiviruses that maintain a longer cycle period, they will often push critical signature updates if a fast spreading threat is spotted regardless of schedule.
[edit] Reaction speed of vendor to major threats
Arguably, it is pretty pointless to get quick and constant signature updates to rare and 'zoo viruses' while being unprotected for days against a common fast spreading threat.This is a reason advanced by some security vendors to explain why they don't put a priority on zoo viruses. Following this line of reasoning, it might be interesting to track how fast antiviruses add signatures to fast spreading threats like Soebig. Such tests are not perfect of course, beyond because it is complicated by issues such as heuristics. Certain antiviruses might already be able to detect new strains of Soebig without any updates at all so they are under less of a constrain to push out signature updates.
In this test, AVG actually does very well compared to other antiviruses, despite it usually being far worse in traditional antivirus tests, this probably points out to the fact that AVG focuses more resources on handling common in the wild threats than other rare zoo malware.
Of course, when I say measuring the detection rate of antiviruses, most people would instantly think of a traditional test, where you gather together a large test bed of antiviruses and run the antivirus on demand against the test bed and see the percentage detected*. In the next section we will discuss how such tests should be interpreted, the pitfalls involved in running such tests, and exploitating them to the real world.
- For obvious reasons testing the onaccess scan is pretty unthinkable, as you would have to execute each sample one by one! In most cases it doesn't matter if the on demand scanner cannot detect the sample, the on access scanner won't as well. The major exception is if the malware is in a file archive (not real time packer) the antivirus cannot scan though, but most test beds use fairly standard file archives like zip/rar/7z. Another exception is that true memory scanners and/or behavior monitoring type features can pick up malware that the on demand scan can't, but this would be after the fact detection where the malware already runs.
[edit] Why Antivirus tests are not as easy as they look
On first sight, it doesn't seem to take much skill to run an antivirus test. All it takes is familiarity with antivirus software (so you don't make any mistakes in the configuration) and access to a good test bed of malware.
Don't be fooled, the science of antivirus testing is very demanding , and very very few people are qualified to do so. To see why this is so ,let us consider what a good antivirus test has to require.
[edit] Good knowledge of antivirus
This one might seem to be a no brainer, but testers have often misconfigured antiviruses while testing, leading to inaccurate results. Then there is a question of testing with "optional settings" vs "out of box settings".
[edit] Good selection of testbed
This one is a biggie.
Firstly off, the malware selected for testing must be as far as possible a random ,unbiased sample. This seems obvious, but it is a rule broken in many "independent tests" where the tester uses all the threats detected by the signature database of one product as the testbed to test all other competitors. In such a case, obviously one product will score 100% (the testbed is after all chosen from it's signatures), while the best the rivals can do is to equal that, and in almost all cases, they will fall below. Does that mean the rival scanners are bad? Of course not, if the tables were turned the rival scanner would score 100% and the others would look bad too. See screenshot (venn diagram showing this)
You might think you know all this, but how many times have we seen a bar charts proporting to show the superiority of a certain product by showing that other competitors detect 50%, or 70% or even 90% of the threats they do?
- hmm, i wonder if i can add a screenshot of one without offending any one??***
Secondly, intuitively , we want the sample to be as large as possible (but see later for a special kind of test), otherwise we might as well fall back on anecdotal stories. Here we run into two problems.
Firstly is the problem of getting good quality working malware to form a quality testbed. This might seem to be an oxymoron, but many virus testbases you download from 'black' Vxer sites contains a lot of junk, broken malware that just doesn't work and hence are harmless. Very very few people have sufficient contacts in the antivirus industry to obtain good samples of malware (and most of these people are not independent). Given the sample sizes we are talking about (easily ten of thousands), sifting them out from working functional malware is not trival. This wouldn't be a problem if it affected all antiviruses equally, but depending on the detection methods used it sometimes makes a difference. A heavily altered malware (in an attempt to avoid detection) might be broken and hence an antivirus using emultation would recognise that and stop. However another antivirus based on a static unpacker might unpack the malware anyway, and detect it as a threat based on strings in the sample. This penalties one antivirus unfairly.
Even if we can assure ourselves that all the samples are working as they should be, we run into another problem of deciding whether everything in it is a legitimate target. On a small scale, some antivirus detect only malware that is a direct threat, they ignore virus/hacker tools, dangerous tools/applications, jokes, simulators(test malware), etc. Then there are greyware type of programs that have legitimate purposes but are also often used by hackers. Examples include Netcat, remote admin type programs like VNC, pcanywhere, even IRC. Similarly it is arguable whether antivirus running on windows machines needs to detect otherOS type malware, or even legacy dos viruses that don't work anymore on modern XP machines.
Lastly, good antivirus tests also have a test set of clean files to run against for a false positive test. Otherwise , someone could just submit a 'antivirus entry' that flagged everything as a virus and score 100%!
There are indeed many many problems at difficulties in running a standard antivirus test, so much so that most people are not qualified to do it. Certainly you should take tests done by hobbyist or even magazine tests (if not sublet out to a professional outfit) with a pinch of salt. See this for a further discussion.
Even in the rare cases where there is good control over samples, careful interpretion is still called for.
For one thing, even if we have a totally random unbiased sampling, arguably this does not fairly represent the real world conditions we are going to meet.
In the real world, a few malware (mainly worms) spread like wild fire, while the vast majority of malware are mainly curiousities that someone wrote for fun and loaded up on a forum and aren't found on any real world machine.
Shouldn't missed detection of the former count for more than missing detection of the former? This brings us to the debate over In the wild (ITW) and zoo malware.
Earlier we discussed a question of legitimate targeting policy, here we have a variant of the same problem. In this case no one (or rather few people) are saying that antiviruses shouldn't detect zoo viruses, but rather it is less important that they do so.
In formally, a threat is said to be in the wild if it is being actively spread in the real world, as opposed to zoo viruses which are confined in some virus lab /zoo as a curiosity.
More formally the term "in the wild" is defined by the WildList Organization International as
When a virus is reported to us by two or more Reporters, it's a pretty good indication that the virus is out there, spreading, causing real problems to users. We consider such a virus to be 'In the Wild'.
As far as where is 'out there', we like the definition given by Paul Ducklin of Sophos, PLC in his paper 'Counting Viruses': For a virus to be considered In the Wild, it must be spreading as a result of normal day-to-day operations on and between the computers of unsuspecting users. This means viruses which merely exist but are not spreading are not considered 'In the Wild'.
Similarly, for a trojan to be considered "In the Wild", it must be found on the computers of unsuspecting users, in the course of normal day-to-day operations.
The WildList Organization International is responsible for keeping up to date the list (typically there are a 500-600 entries as opposed to zoo viruses where people have used testbeds that are easily 50,000 and upwards) and every month a test is carried out and antiviruses that manage to detect all the samples (with no false positive) get the VB100% award.
Like any test, there are of course criticisms of the VB100 test, see for example this one and this one, but mainly they focus on whether reporting procedures which determines what gets entered in the wild list is robust enough. Once this happens the question becomes whether their classification of 'in the wild' has any value.
Indeed, many critics would say that there is no clear line between a zoo virus and in the wild viruses and all antiviruses should detect all known malware. It is easier said then done given the amount of malware out there, and limited malware analyst time (creating signatures unfortunately is more an art than a science and cannot be easily automated as we shall see). As mentioned before, some companies have decided to give priority to detecting in the wild threats, so as to achieve maximum protection for most number of users, however that is cold comfort to the ones hit by a supposedly zoo virus .
Given all these problems I almost hesitate to suggest good test sites but the 2 most well known besides VB100 are viruscomparitives and avtest.otg.
[edit] Other type of tests
Retrospective tests
Some tests have attempted to measure the strength of heuristics in antiviruses, by testing a older version of a signature base of the AV against new threats released after it was last updated. In theory, the samples detected by the AV then must be based on heuristics alone and *not* on signatures. In practice things are not so simple (difficulty in knowing what exactly is already in the signatures for one) but it still gives a rough gauge.
Another type of antivirus test differs from both the traditional on demand test, and retrospective test in that it uses only a small number of samples. these tests don't claim to test how good avs are at detecting a broad range of malware, but rather how easy it is to beat them by modifying a well known threat to avoid detection.
Typically it involves someone taking a well known trojan that is initially detected by the antivirus, and modifying it in some way maybe using a packer, do some hext editing or running some sort of hacker tool to change it (or all 3) and seeing if it can fool the antivirus. Here's an example of such a test.
Such tests claim to directly test AV quality (quality of signature and robustness of detection methods) but understanding and interpret ions of such tests is often very difficult because besides possessing technical knowledge of the generalities it requires specific deep knowledge of the internals of the antivirus engine , something the developers are notorious closed mouth about such information.
For example, even trying to figure out whether an antivirus supports a certain real time packer by packing a well known trojan and scanning it to see if it is detected doesn't work. Because it might be possible that signatures of the packed samples are included. A very likely scenario if we are talking about a popular trojan. There are dozens of other problems some of which are summarized here
In the next section we will indeed try to discuss some more arcane technical aspects of antiviruse. Bold text
