Authentium Virus Blog

A friend forwarded me a link to an article on NPR: “Cyberwarrior Shortage Threatens U.S. Security“.

Personally, I am a pacifist. I abhor violence in any form for any reason. I dislike war for the same reason. I am however a pragmatist. I believe that you should have the ability to defend yourself. I also believe that sometimes the best defense is a well timed attack. This is contradictory, but it is what I believe.

I also believe that being a warrior should be an honorable profession. It requires integrity, compassion, experience and perseverance. Not quite the skill set you will find in a 16 year old script-kiddie calling himself a security expert. Or somebody that would adjust his own scores by cheating.

Too many people call themselves security experts these days. The vast majority of them don’t have the qualifications, the integrity or the ability. The whole field of computer security is surrounded by way too much myth and hype for this to change in the near future.

One statement is true: True security experts are a rare breed and it is very hard and expensive to recruit them. Finding twenty to thirty thousand of them is unrealistic. You will be lucky to find two hundred of them. Obviously it would be better for society if this changes, but it will take a significant investment in math and science related fields.

Personally I am not applying, for the simple reason that I am a pacifist. But I do hope that the people chosen to be the warriors of the future live up to the expectations that we as society have for the defenders of our liberty and our lives.

There has been a lot of excitement about AMTSO and what it is all about. This specific posting was inspired by “The edge of reason(ableness)…“.

Some disclaimers:

I have previously expressed my opinion on this blog that AMTSO is a good thing. I have also repeatedly expressed my opinion about bad testing.

Some assumptions from my side:

The major reason I am personally not more involved in AMTSO is basically a lack of bandwidth: I don’t have the time.

What I have seen is that what they have done has basically been positive. Their intentions seems to be good.

Have they been able to achieve everything I would have wished? I don’t think so. But they are making progress. They have the major players involved in trying to make sense of a constantly changing, complex and specialized field. They have an open invitation for any organization that feel that they can make a contribution to join and improve AMTSO and help it in its work.

Will testing be perfect after AMTSO is finished with their job? Firstly I dare you to define perfect testing, secondly I don’t think that they will ever be finished. The field changes too rapidly for any decisions taken today to be valid for too long.

I am not going to comment in any detail on the contents of the Kevin Townsend article that started all of this. I have to wonder about “false authority syndrome” ? What I will say is that taking a test collection of 2 samples are statistically irrelevant. Using a public multiscanner as a method to test products is also extremely dangerous. What is also quite funny is his references to Sophos and ducks. You have to understand something about Sophos to get that inside joke.

The Wildlist is not perfect. But you would be amazed at how much it has changed over the last year, and there are some exciting changes planned for the near future. I do however have an open invitation to anybody that can think of a better way to create a better, relevant, consistent and reproducible test set to document and implement it.

Testing costs money, and the better the testing, the more money it costs. Who should pay for it? The vendors definitely should not. Where should the money come from to create this perfect testing infrastructure and process?

I also think everybody is taking this way too seriously. I think criticism can be good and if it is constructive should be used to improve matters. If it is just negative and contributes nothing of value then it should just be ignored. Nothing is to be gained to respond angrily to any type of criticism. Either ignore it or respond in an unemotional way.

I understand that AMTSO has a lot on its plate, and there are a significant number of very contentious issues being debated by highly skilled people. They don’t have an easy job to do, and doing it will take time. They may also not get it right according to everybody, but hopefully they will get it right according to most people most of the time.

This will be my third blog posting about the dangers of multiscanners. The previous ones are: Malware Naming Confusion and Multiscanners: The good, the bad and the ugly.

This time it is about how garbage can spread and infect definition files.

What I am talking about is somebody is detecting some file just because somebody else is. Or people demand that we detect a file just because somebody else is.

We accidentally did a test that was similar to the one that Kaspersky did that lead to my first blog entry on this subject. The original test was that they created several executables and added detection for it. These executables were then submitted to a public multiscanning service and they monitored detection of these samples. After a few weeks a large number of vendors detected the samples with virtually identical names.

We have a file that we detect as W32/TestSample. It is a simple Windows executable that does 2 things. It opens a handle to itself, which always fails, and it displays a message box stating that it is a test sample. This allows our OEM customers to test our memory and registry scanning routines and to test the functionality of the scan engine in a safe environment. It definitely is not malware.

Or so I thought.

Obviously if 13 products detect a file it must be malware. Interestingly enough some of the major brands detect this file. I would love to name and shame, but I am confident there must be some similar garbage in our own collection. What is really funny is that only one other vendor at least tried to copy our name for this. Two other vendors called it Adware, 3 detected it heuristically and 6 others detected it accurately by name using different names.

This specific sample is at least in one large testing organizations test set that is used to determine malware detection rates.

It is easy to be critical towards these other companies for including detection for this but in the end I think I understand how it happened. Either a mutual client insisted that these samples must be detected or they saw the samples in the test set of some antivirus tester and decided to add detection for it because it is easier than to dispute the sample. Or they saw that we detected it and therefore added detection for it without analyzing the actual sample.

Just be careful how you interpret the results of a multiscanner. Not everything detected by antivirus products are really malware. There are too many non-technical influences on what should be detected for the results of a multiscanner to be valuable.

What is a multiscanner?

It is a system where multiple AV products are used to scan files and provide a report about the files.

What is the good about a multiscanner?

What is bad about a multiscanner?

Before I go on to the ugly I need to explain the previous statements. When a scanner detects something the only information that you have is that some scanner detected that file. It does not imply that the file is malware.

That may sound surprising but the reality is that not all scanners were created equally. Some scanners have heuristics that is so paranoid that they trigger on virtually every second file. Today I spent some time looking at about 200 files gathered by one of our monitoring systems. The files were from three different sources and about 90% of them were detected by at least one scanner. Strangely enough they were all variants of three totally legitimate products and should not have been detected. Had I trusted the multiscanner I would have duplicated their mistake.

Not all scanners are meant to be used this way. Some scanners are focused on scanning email or gateway based traffic. This enables them to specialize and tweak their heuristics in such a way that they do an amazing job at the gateway but would be disastrous to use on the desktop. They have virtually no false positives when used as they were designed, but when used in a multiscanner or as part of a desktop product they would behave in very unexpected ways.

Some scanners will automatically detect that it is being used to scan a collection of malware and change their behavior. This can make it difficult to trust in a multiscanner environment as its behavior is not consistent and the results can be surprising.

I am not going to name any specific vendors but completely trusting the information you get from a multiscanner is quite dangerous. The information gathered from a multiscanner about a file have to be added to other information about the file before a determination can be made whether the file is malware or not.

What is ugly about a multiscanner?

This article was triggered by a highly amusing article by Kaspersky: On the way to better testing. Responses by ESET and PC Magazine. What amused me is to know that they are in the same boat as we are. It is hard to convince a customer that some file that some so-called “tester” magically conjured up from some disreputable spot is garbage if 10 other scanners detect it. What is less amusing is if one of your own products are detected by 10 other products and you try to get it white listed or detection for it removed.

Both these situations are the real ugly of multiscanners. Not only the ones in use by every AV company out there but also the public ones.

I am not trying to point any fingers nor am I saying that any individual scanner or multiscanner should not be trusted. All I am saying is that the context in which a product or products are used should be understood. The risks and the value of the information provided by any source should be investigated and understood before a decision based on the information is made.