How’d we pick the products for the anti-phishing test?

I’ve gotten several inquiries about how we selected the products we tested in the anti-phishing technology evaluation. That’s a fair question; some companies are unhappy that they were included, and some that they weren’t.

When we defined the parameters for the testing, we selected the vendors that had either browser-based toolbar add-ons or built-in anti-phishing technology in the browser as of May 2006 and that (in our opinion or by market data) had a significant usage presence. There are dozens of products that meet the first test, but not that many that meet the second. We picked the top 8 based on our understanding of actual usage and deployment. I didn’t want to include payware products because the original objective was for us to help Microsoft understand how well IE 7 worked compared to its biggest competitors– and in this market segment, payware products are at a disadvantage.

Would we have preferred to test all the products? Sure. The team at Carnegie Mellon that did a similar study (with a smaller list of products and a smaller set of URLs) said the same thing. However, we had to draw the line somewhere. When we redo the tests, we’ll probably change the product mix around; I’d expect to see Firefox 2.0 included, and maybe some of the commercial products.

To address Symantec’s complaint, I’d make two points. First, Norton Confidential wasn’t announced until June, so how could we have included it? You’re making the Firefox argument. We only tested products that were publicly available at the start of our time period; we excluded Norton Internet Security 2006 because it was commercial (and I suspect that if we’d tested the 2006 version, we’d be hearing that we should’ve tested the 2007 version instead. Sic transit gloria annual releases…)

Second, it’s pretty worthless to have a blog but not allow comments or trackbacks. That’s not a blog, it’s a monologue. Whatever you think of the quality of Microsoft’s products (including IE), you have to admit that they have aggressively embraced blogging as a way to communicate directly with customers– something I’d like to see more security companies emulate.

Update: fixed the link to McAfee’s SiteAdvisor blog.

Technorati Tags: ,

Comments Off on How’d we pick the products for the anti-phishing test?

Filed under Security

Comments are closed.