CMU releases updated anti-phishing report

Dr. Lorrie Cranor of CMU and her team have recently released a new version of their own anti-phishing analysis. It makes for interesting reading, as its methodology is slightly different from 3Sharp’s (and quite different from the Firefox team’s methdodology). Cranor’s team used an automated system to feed phish in and record the tool responses, which is much more scalable than the human-driven system that 3Sharp and Firefox both used; there are a few other methodological improvements detailed in the study as well. (Interestingly, they too chose to include SiteAdvisor, which scored very poorly.)
Table 4 of the study is the big burrito; it lists both false positive and catch rates for the ten technologies they tested. SpoofGuard had the best catch rate, but it also scored a whopping 38% false positive rate… oops. EarthLink came in second, followed by Netcraft, Google, IE7, and Cloudmark.
What does this mean? I’m not sure. The CMU study used a data feed only from Phishtank, which means its results should line up with what the Firefox team found. However, CMU didn’t test Firefox 2.0, so there’s no way to make a direct comparison. The URLs they tested were gathered over a 3-day period, which IMHO is too short to give a good baseline. However, I like the automated testbed that CMU used, and the discussion of toolbar exploits is really interesting stuff that I hope all the toolbar vendors are paying careful attention to.

Technorati Tags: 3sharp, phishing