There has been a lot of excitement about AMTSO and what it is all about. This specific posting was inspired by “The edge of reason(ableness)…“.
Some disclaimers:
Authentium is not an AMTSO member and I have not been involved with AMTSO.
Authentium is a vendor, but we generally don’t make it into the worst tests because we are relatively unknown.
I am a Wildlist reporter.
I have previously expressed my opinion on this blog that AMTSO is a good thing. I have also repeatedly expressed my opinion about bad testing.
Some assumptions from my side:
Testing malware correctly is hard
Malware is a very specialized field
There is a significant amount of money to be made or lost in the anti-malware/security field
The only constant about malware is that it changes all the time
Nobody has an infinite amount of money
The major reason I am personally not more involved in AMTSO is basically a lack of bandwidth: I don’t have the time.
What I have seen is that what they have done has basically been positive. Their intentions seems to be good.
Have they been able to achieve everything I would have wished? I don’t think so. But they are making progress. They have the major players involved in trying to make sense of a constantly changing, complex and specialized field. They have an open invitation for any organization that feel that they can make a contribution to join and improve AMTSO and help it in its work.
Will testing be perfect after AMTSO is finished with their job? Firstly I dare you to define perfect testing, secondly I don’t think that they will ever be finished. The field changes too rapidly for any decisions taken today to be valid for too long.
I am not going to comment in any detail on the contents of the Kevin Townsend article that started all of this. I have to wonder about “false authority syndrome” ? What I will say is that taking a test collection of 2 samples are statistically irrelevant. Using a public multiscanner as a method to test products is also extremely dangerous. What is also quite funny is his references to Sophos and ducks. You have to understand something about Sophos to get that inside joke.
The Wildlist is not perfect. But you would be amazed at how much it has changed over the last year, and there are some exciting changes planned for the near future. I do however have an open invitation to anybody that can think of a better way to create a better, relevant, consistent and reproducible test set to document and implement it.
Testing costs money, and the better the testing, the more money it costs. Who should pay for it? The vendors definitely should not. Where should the money come from to create this perfect testing infrastructure and process?
I also think everybody is taking this way too seriously. I think criticism can be good and if it is constructive should be used to improve matters. If it is just negative and contributes nothing of value then it should just be ignored. Nothing is to be gained to respond angrily to any type of criticism. Either ignore it or respond in an unemotional way.
I understand that AMTSO has a lot on its plate, and there are a significant number of very contentious issues being debated by highly skilled people. They don’t have an easy job to do, and doing it will take time. They may also not get it right according to everybody, but hopefully they will get it right according to most people most of the time.