Microsoft announced today that they are making significant changes to the way that Exchange 2013 updates are developed, packaged, and released. These changes come on the heels of some notable, and embarrassing, quality problems with previous Exchange rollup updates (RU). In fairness, the recent quality problems spring from the inclusion of Oracle-provided code in Exchange, code which needed to be updated to fix security vulnerabilities. (I hope the lesson here is clear: Microsoft, don’t ever include software from Oracle in your products, ever again, for any reason.) Anyway, even laying all of the blame for the original problem at Oracle’s doorstep, the fact is that the problematic RUs directly affected Exchange customers and besmirched the reputation of the Exchange team for delivering robust, high-quality code.
To help prevent this kind of problem from ever occurring again, Microsoft is taking some significant steps. First, the Exchange Sustaining Engineering team is going away, and its engineers are being integrated into the regular product development team. This sort of “one team, one fight” approach has been successful in many other places: rather than having a separate team working on sustaining older versions of the product while the main engineering effort goes toward new versions, management is free to pick the best available engineers for each update or new feature, regardless of whether that work is going toward an old version or a new version. There has already been a similar unification of the teams that develop Exchange on-premises and Office 365. In fact, that unification serves as the model for another important change Microsoft is making: exactly the same code will be deployed in Office 365 as will be delivered to Exchange on-premises customers. Note that this does not mean that the features between those two versions will be identical; there are already many cmdlets, parameters, and assorted capabilities that work on one side but not the other, especially in Exchange 2013. But the code will be the same. With a single code base deployed across millions of Office 365 (and, presumably, Live@EDU) mailboxes and millions more customer mailboxes, Microsoft’s job of testing, fixing, and sustaining Exchange should become quite a bit simpler. In addition, I expect that Microsoft will continue to rigorously test updates before they go live on Office 365, and this testing will complement the existing release testing to hopefully give us more robust updates. The CU release plan is to release each CU to on-premises and hybrid customers after it has already been deployed on the production Office 365 network. That means that by the time you and I get a CU, it’s already been tested by the Exchange team, tested by the Office 365 team before deployment into production, and tested and validated by running in Office 365 production.
It’s important to note, of course, that Office 365 is a monolithic environment in many respects; it doesn’t necessarily represent the very wide range of configurations that customers actually have deployed out in the field. It remains to be seen whether future updates will run into problems because they passed tests in Microsoft’s environments, then failed under certain unusual configurations out in customer-land. More likely, Microsoft will establish support boundaries covering specific configurations and scenarios that they test the CUs against, although these boundaries may not necessarily be made public.
The biggest change in the servicing model, however, is that the existing system of rollup updates will go away. Instead, we will get quarterly cumulative updates (CU). This model is familiar to OCS and Lync administrators, who are accustomed to getting Lync CUs every so often instead of service packs (although the Lync product group still has a separate sustaining engineering team).
Cumulative updates package all of the rollups since RTM into a single update, so that applying the latest CU always brings you up-to-date on all product patches released to that point. This simplifies things for administrators, but it also means that you are essentially required to reinstall the product and do a build-to-build upgrade with each CU release– an action that cannot be reversed. It will take some time to figure out the best strategy for dealing with CU installation problems, should they arise.
CU packages may include features, too. In the past, customers have balked at the inclusion of features in updates outside of service packs, so it’s not clear where Microsoft will draw the line on how big a feature is OK to include in a CU. They have also remained mum about the future of service packs, although I expect to see occasional SP releases that combine the latest CU with major feature releases and documentation updates. Microsoft has explicitly said that CUs may include schema updates, eliminating at least one potential distinction between CUs and SPs.
Security patches will be included in CUs but may also be installed separately. That is, when CU1 ships, it will contain a set of security updates that were the latest available at whatever point in time the CU is frozen. You’ll install a CU, then add any security updates that were released between that freeze date and the current date.
There’s another major change, too: the support timeline for CUs is changing rather dramatically. Microsoft will support a CU for 3 months after the next CU ships. In other words, you have six months from the date a CU ships before it becomes unsupported. Suppose that CU1 is released on 3/2 and CU2 is released on 6/1. CU1 will age out of support on 9/1. Is 3 months enough time for customers to test and deploy CU2? That remains to be seen. I have the sense that a noticeable fraction of Exchange customers will balk at this timeline. In complex environments (where “complex” implies complexity of infrastructure, business or legal requirements, and/or politics) it may not be feasible to test, certify, and globally deploy a CU within that time window.
Finally, the other change I think noteworthy is that CUs won’t be offered through Windows Update; you’ll have to download them from the Microsoft Download Center. This is a good move because it reduces the risk that you’ll accidentally get an update that breaks something in your environment.
On balance, are these changes a good idea?
On one hand, having a regular cadence for CU releases is a great thing. We’ve seen the positive impact of having security fixes released on a predictable schedule: it’s easier to plan for, test, and deploy releases when you know what they fix and when they’re coming. Unfortunately, Microsoft has done a poor job in the past with documenting exactly what fixes are included in each rollup; there are many fixes that are not listed in the RU documentation, and there is generally no comprehensive public list for a given RU showing everything that’s fixed. This is different than what happens with security updates, the documentation for which tends to be pretty explicit about what’s fixed and which files are updated by the fix. I hope that we’ll see more detailed information about included fixes when the CU system gets rolling.
On the other hand, the schedule pressure caused by having a regular release schedule for CUs means that Microsoft may still have to deal with the case where a needed fix isn’t ready in the scheduled CU release timeframe. That means we’ll probably still be seeing hot fixes and security patches released outside the normal CU schedule. That’s fine; we all know how to deal with those. I am a little concerned that schedule pressure may lead to the release of fixes or features in a CU that have not been tested thoroughly enough for release; the reason I’m only a little concerned is because the visibility of releasing updates to both on-premises and Office 365 Exchange means that there will be a lot of pressure to ensure that updates are robust before release– and that will benefit everyone involved.
On that same other hand, I don’t think the 3-month support timeframe is going to sit well with customers. I freely admit that I don’t know this for sure (after all, Microsoft just announced it today), but that’s how it strikes me based on what I see when discussing Exchange servicing with existing customers.
Microsoft hasn’t said when we should expect CU1 for Exchange 2013, although I expect it to be soon. When it arrives, we’ll have to see how the update process itself works; on an ongoing basis, examining the timeliness and quality of future CUs will be the only way to see whether this change turns out to be good for Exchange.
Paul's Down-Home Page 