Security update numbers, apples to apples

Ed posted comparing IBM and Microsoft’s security update records. He missed a few important details, though that’s understandable given that he’s not a security dude. Just to set the record straight, though, I wanted to point out something that security folks learn pretty quickly: simplistic comparisons that claim that “vendor X has better security than vendor Y based on patches” are worthless. Any time you see one, there are some hard questions you should be asking.

First, what products are included? We don’t know what criteria McAfee used to make their pretty graphs. Did they include Office updates? Updates for Windows 2000 before it went EOL? Windows Media Player? Who knows? Reputable researchers and vendors will always include their source data; if you don’t see it, you should be wary.

Second, what basis of comparison is being used? Most broad-based comparisons of vendors are flawed because they mix dissimilar items, usually applications and OSes. You can say “Microsoft had to issue more patches than IBM”, but that’s meaningless unless you’re talking about specific products. A more interesting question would be to ask something like “Who had more patches to install: an Exchange 2003 admin on Windows 2003, or a Lotus Domino 6.5 admin on RHEL?” Well, according to Secunia, the numbers break down like this:

All of a sudden the comparison doesn’t favor IBM quite so much! A more proper comparison might leave the operating system out of it (after all, there are more Notes seats on Windows than on Linux), but even then there’s still room for argument: Secunia doesn’t break down Domino R6 vs 6.5, so the vuln count of 22 may include some items that aren’t relevant.

Third, counting patches alone leaves out some important dimensions. It’s like counting the money in your wallet by counting bills and ignoring denominations– would you rather have 10 $1 bills or 1 $100? Other factors to evaluate include the severity of the vulnerability and how long between its emergence (or disclosure) before the vendor gets a patch out– the so-called “days of risk” model.

Fourth, not all vendors tell the truth. More kindly, not all vendors tell the whole truth and nothing but. For example, IBM doesn’t include severity ratings on its security page, so you can’t judge the severity of a reported vuln unless you’re already pretty knowledgeable. Oracle is flat-out dishonest in some of its security patch release notes. When you’re comparing vendor security, you should include the nature, frequency, and accuracy of their security-related disclosures and communications.



Filed under Security

10 responses to “Security update numbers, apples to apples

  1. I like Microsoft’s nomenclature and transparency with regards to bug fixes. Anytime you get into a “rating” scenario there is always the potential for subjectivity, but at least Microsoft takes a stance on what they think the impact is. For IBM it’s anybody’s guess and you need extremely deep domain knowledge of an exploit to decide how much — or even if — it concerns you.
    Regarding the statement “after all, there are more Notes seats on Windows than on Linux”, would you care to share your data? After all: “Reputable researchers and vendors will always include their source data; if you don’t see it, you should be wary.”
    I’m [mostly] joking about that. I get tired of the Microsoft fans trying to paint the picture that Notes and Domino are most popular on Windows. “Popular” is subjective and I’ve not seen any data that shows Domino server distribution by OS. Please stop spreading that bit of misinformation. It’s not explicitly harmful but it is disingenuous.

  2. The Secure Windows Initiative (SWI) had 4 pillars: secure by design, secure by default, secure in deployment, and communications. You can certainly argue about whether they’ve achieved their goals on the first three, but I think any fair observer would agree that Microsoft’s security communications are head and shoulders above most other companies’.
    As for the seats on Windows/seats on Linux debate: first, since there’s no shipping Linux Notes client, I’m right by default 🙂 What I should’ve said is that there are more total seats hosted on Windows than on Linux. Even though IBM doesn’t break revenue up by platform, I’m not alone in thinking this– I had a conversation with Peter O’Kelly of Burton about it, and I trust him as an impartial source. Anecdotally, I can say that I’ve never seen a Domino site running primarily on Linux, but have seen dozens on Windows, but I know anecdotes != data.

  3. What does the number of Notes seats have to do with Domino vulnerabilities, anyway? Did you mean to say “after all, there are more Domino servers on Windows than on Linux”?
    By the way, there is a shipping Notes client for Linux, part number C75LXNA.

  4. Overall, number of seats is the commonly accepted measurement for market share, which is why I used it here. If you prefer to think of it in terms of server counts, that’s probably a better way to make my point, which is that slagging Windows doesn’t necessarily help arguments like Ed’s.
    As for the shipping client bit: you’re right and I’m wrong. I didn’t realize that the Linux client was shipping already.

  5. Bill Brown

    How many of those RHEL patches need to be installed on as box running *ONLY* Domino. If you don’t load Xwindows, there goes a whole bunch of patches to be installed. Not running a browser on your server (always a bad idea, esp in Windows), no need to install the handful of Firefox updates.
    (I wanted to see exactly what advisories Secunia issued, but the pages were coming up blank for me.)

  6. Beats me; I don’t have a list of Domino’s prerequisite packages. I’d suspect that most of the Linux Domino servers out there have X11 on them, though, based on my own informal experience with other Linux servers. It’s much easier to administer the local machine if you have a windowing environment on it.

  7. Anonymous

    For shame. Just look at the “212” vulnerabilities for RHES. Imagemagik? Squid? squirrelmail? gpdf? gimp?
    Perhaps this will someday hold water when someone lists all the Microsoft vulnerabilities for Acrobat Reader, Photoshop, Symantec Antivirus, McAfee VirusScan, Microsoft Paint, Microsoft Outlook, Microsoft Word, Microsoft Excel, Microsoft PowerPoint, Microsoft Publisher, Microsoft Encarta, Google Toolbar, Google Picassa, EudoraMail, Microsoft Visio, Winzip, PowerZip, LimeWire, Bearshare, eDonkey, Kazaa, KazaaLite, Napster, and every freeware CD burner application, and all the spyware bundled with AOL on the Microsoft side.
    What you are spreading is more FUD. Which is a shame, because if it was done objectively, the numbers might actually be closer, and relevant, and useful to someone truly concerned about their server security. Instead, the ignorant will consume your fluff undigested, and continue to irresponsibly administer their servers in the mistaken belief that Microsoft actually cares about their security. The enlightened will calmly reject your “information” henceforth, and listen no more. Both sides are getting a disservice.

  8. Anonymous,
    Last I was aware, Microsoft doesn’t bundle any third-party software — or even any additional Microsoft software — with their server operating systems. I can’t find a single SKU to buy that includes any of the products you mentioned. In order to get any of those products on a Microsoft server, the user either has to install them manually *or* buy a server from a hardware manufacturer (like Dell or HP) that includes a pre-installed OEM version of the OS image.
    Does Microsoft include stuff in their retail desktop packages? Yup, they do — a limited amount. Most of that junk, though, is again added in *by hardware manufacturers* using OEM versions. Still, I’ll grant you that your complaint would have been a valid criticism *iff* Paul had been discussing desktop OSs.
    Red Hat, on the other hand, does include a ton of software in their RHEL distributions that you can install on the machine as part of the initial installation (and in fact you have to aggressively prune software from a default RHEL installation, in my experience, if you want to nail a RHEL machine down to a well-defined set of roles that you can properly harden).

  9. Clicked “Post” too soon. My bad.
    As I was going to say before I so rudely interrupted myself, what Paul is spreading is not in fact FUD. It is an honest attempt to take a look at the difficulties in doing these kinds of comparisons by the numbers alone.
    As was pointed out in the post, even the reported “8 vs. 22” comparison between Exchange and Domino is less than compelling *just on the numbers* precisely because the group doing the reporting fails to take Domino’s versioning into account and break down the results by version. How many of those 22 patches does a Domino admin need to install on their 6.5 system? No way to tell from the data at hand.
    Let us, then, address the meat of your attack: “in the mistaken belief that Microsoft actually cares about their security.” This statement reveals your entire thrust, your bias. You choose not to believe that Microsoft cares about security — it doesn’t care about security for its products, it doesn’t care about security for its customers.
    It’s trivially easy to dispute that contention. Let me assume for the sake of the argument that you are one of the people who thinks that Microsoft is a monolithic entity that cares only about one thing: profit. Microsoft clearly spends a lot of money — an ever-increasing amount, actually — on security. They’ve spent a large amount of time and effort retolling their internal processes to ensure that security is addressed at all stages of the development process. Even if Microsoft is only the dollar-grubbing company that you and many others seem to feel it is, then Microsoft cares about security *because* customers care about security and are willing to spend their dollars on it.

  10. Wow, I love anonymous commenters! I’m beginning to think that I should follow Ed’s lead and just ban ’em. If you’re not confident enough of your arguments to make them with your name attached, then perhaps you should just keep quiet.