Paul's Down-Home Page

It’s here! 

 Microsoft today released Service Pack 3 for Exchange 2010 (available from http://www.microsoft.com/en-us/download/details.aspx?id=36768).

SP3 enables full coexistence between Exchange 2010 and Exchange 2013, plus it adds a couple of new features, such as Windows Server 2012 support. See the full feature list here.

After you’ve upgraded allof your 2010 servers to SP3, you will be able to install Exchange 2013 in your forest. Opinions differ on whether you should– for example, fellow MVPs Tony Redmond, Michael B. Smith, and Jason Sherry all agree that you should wait. For my part, I’m OK with deploying 2013 into production; it does have some rough edges, but I think CU1 is right around the corner, so adding 2013 in your environment now is certainly defensible.

Microsoft announced today that they are making significant changes to the way that Exchange 2013 updates are developed, packaged, and released. These changes come on the heels of some notable, and embarrassing, quality problems with previous Exchange rollup updates (RU). In fairness, the recent quality problems spring from the inclusion of Oracle-provided code in Exchange, code which needed to be updated to fix security vulnerabilities. (I hope the lesson here is clear: Microsoft, don’t ever include software from Oracle in your products, ever again, for any reason.) Anyway, even laying all of the blame for the original problem at Oracle’s doorstep, the fact is that the problematic RUs directly affected Exchange customers and besmirched the reputation of the Exchange team for delivering robust, high-quality code.

To help prevent this kind of problem from ever occurring again, Microsoft is taking some significant steps. First, the Exchange Sustaining Engineering team is going away, and its engineers are being integrated into the regular product development team. This sort of “one team, one fight” approach has been successful in many other places: rather than having a separate team working on sustaining older versions of the product while the main engineering effort goes toward new versions, management is free to pick the best available engineers for each update or new feature, regardless of whether that work is going toward an old version or a new version. There has already been a similar unification of the teams that develop Exchange on-premises and Office 365. In fact, that unification serves as the model for another important change Microsoft is making: exactly the same code will be deployed in Office 365 as will be delivered to Exchange on-premises customers. Note that this does not mean that the features between those two versions will be identical; there are already many cmdlets, parameters, and assorted capabilities that work on one side but not the other, especially in Exchange 2013. But the code will be the same. With a single code base deployed across millions of Office 365 (and, presumably, Live@EDU)  mailboxes and millions more customer mailboxes, Microsoft’s job of testing, fixing, and sustaining Exchange should become quite a bit simpler. In addition, I expect that Microsoft will continue to rigorously test updates before they go live on Office 365, and this testing will complement the existing release testing to hopefully give us more robust updates. The CU release plan is to release each CU to on-premises and hybrid customers after it has already been deployed on the production Office 365 network. That means that by the time you and I get a CU, it’s already been tested by the Exchange team, tested by the Office 365 team before deployment into production, and tested and validated by running in Office 365 production.

It’s important to note, of course, that Office 365 is a monolithic environment in many respects; it doesn’t necessarily represent the very wide range of configurations that customers actually have deployed out in the field. It remains to be seen whether future updates will run into problems because they passed tests in Microsoft’s environments, then failed under certain unusual configurations out in customer-land. More likely, Microsoft will establish support boundaries covering specific configurations and scenarios that they test the CUs against, although these boundaries may not necessarily be made public.

The biggest change in the servicing model, however, is that the existing system of rollup updates will go away. Instead, we will get quarterly cumulative updates (CU). This model is familiar to OCS and Lync administrators, who are accustomed to getting Lync CUs every so often instead of service packs (although the Lync product group still has a separate sustaining engineering team).

Cumulative updates package all of the rollups since RTM into a single update, so that applying the latest CU always brings you up-to-date on all product patches released to that point. This simplifies things for administrators, but it also means that you are essentially required to reinstall the product and do a build-to-build upgrade with each CU release– an action that cannot be reversed. It will take some time to figure out the best strategy for dealing with CU installation problems, should they arise.

CU packages may include features, too. In the past, customers have balked at the inclusion of features in updates outside of service packs, so it’s not clear where Microsoft will draw the line on how big a feature is OK to include in a CU. They have also remained mum about the future of service packs, although I expect to see occasional SP releases that combine the latest CU with major feature releases and documentation updates. Microsoft has explicitly said that CUs may include schema updates, eliminating at least one potential distinction between CUs and SPs.

Security patches will be included in CUs but may also be installed separately. That is, when CU1 ships, it will contain a set of security updates that were the latest available at whatever point in time the CU is frozen. You’ll install a CU, then add any security updates that were released between that freeze date and the current date.

There’s another major change, too: the support timeline for CUs is changing rather dramatically. Microsoft will support a CU for 3 months after the next CU ships. In other words, you have six months from the date a CU ships before it becomes unsupported. Suppose that CU1 is released on 3/2 and CU2 is released on 6/1. CU1 will age out of support on 9/1. Is 3 months enough time for customers to test and deploy CU2? That remains to be seen. I have the sense that a noticeable fraction of Exchange customers will balk at this timeline. In complex environments (where “complex” implies complexity of infrastructure, business or legal requirements, and/or politics) it may not be feasible to test, certify, and globally deploy a CU within that time window.

Finally, the other change I think noteworthy is that CUs won’t be offered through Windows Update; you’ll have to download them from the Microsoft Download Center. This is a good move because it reduces the risk that you’ll accidentally get an update that breaks something in your environment.

On balance, are these changes a good idea?

On one hand, having a regular cadence for CU releases is a great thing. We’ve seen the positive impact of having security fixes released on a predictable schedule: it’s easier to plan for, test, and deploy releases when you know what they fix and when they’re coming. Unfortunately, Microsoft has done a poor job in the past with documenting exactly what fixes are included in each rollup; there are many fixes that are not listed in the RU documentation, and there is generally no comprehensive public list for a given RU showing everything that’s fixed. This is different than what happens with security updates, the documentation for which tends to be pretty explicit about what’s fixed and which files are updated by the fix. I hope that we’ll see more detailed information about included fixes when the CU system gets rolling.

On the other hand, the schedule pressure caused by having a regular release schedule for CUs means that Microsoft may still have to deal with the case where a needed fix isn’t ready in the scheduled CU release timeframe. That means we’ll probably still be seeing hot fixes and security patches released outside the normal CU schedule. That’s fine; we all know how to deal with those. I am a little concerned that schedule pressure may lead to the release of fixes or features in a CU that have not been tested thoroughly enough for release; the reason I’m only a little concerned is because the visibility of releasing updates to both on-premises and Office 365 Exchange means that there will be a lot of pressure to ensure that updates are robust before release– and that will benefit everyone involved.

On that same other hand, I don’t think the 3-month support timeframe is going to sit well with customers. I freely admit that I don’t know this for sure (after all, Microsoft just announced it today), but that’s how it strikes me based on what I see when discussing Exchange servicing with existing customers.

Microsoft hasn’t said when we should expect CU1 for Exchange 2013, although I expect it to be soon. When it arrives, we’ll have to see how the update process itself works; on an ongoing basis, examining the timeliness and quality of future CUs will be the only way to see whether this change turns out to be good for Exchange.

I was very surprised to learn today that you cannot uninstall a single Exchange 2013 role from a serverb.

Quick review: one of the major changes in Exchange 2013 is that the number of server roles has decreased from 5 (hub, mailbox, edge, UM, CAS) to 2 (mailbox and CAS). This approach offers lots of potential benefits, but it also represents a major change to the internals of Exchange.

In Exchange 2007 and Exchange 2010, you could easily combine multiple roles on a single server, adding and removing roles more or less at will. This was fully supported and worked well: running setup with the /roles switch and the appropriate mode would do the trick.

For what I’m sure are excellent reasons, this is no longer supported; attempting to remove either the CAS or mailbox role from a server that has both will produce an error message indicating “invalid command line arguments”. In fairness, this is documented in the release notes, right up at the top– but I missed that the first time around and wanted to mention it here as a reference. 

The workaround: if you want to remove a role, you must remove Exchange entirely from the target server, then reinstall only the role you do want.

Fellow MVP and all-around ace Michael B. Smith just published an article accurately titled “Exchange Server 2013 Gotchas.” Rather than editorialize, I’ll just ask that you read the article and consider the list of issues he’s found.

I don’t agree that all the things he lists as faults are actually faults; for example, I have no problem with Outlook 2003 no longer being supported, and it doesn’t matter to me that installation is slow. Many of the other items he lists have fairly low impact, such as the absence of the Edge Transport role (which very few of my customers use) or the fact that there is no longer a separate UM role– very few organizations have enough concurrent calls to warrant investing in a dedicated UM server. No one likes to see functionality disappear, such as the loss of S/MIME support in OWA, but at the same time we’ve gotten a ton of new functionality (including offline mode and touch mode in OWA, to name two possible reasons why S/MIME was deprioritized.)

Having said that, there are a number of his points that I agree with wholeheartedly. The documentation is clearly not finished; there are many missing pieces, and this is compounded by Microsoft’s still-unresolved decision to change the way TechNet URLs work. The Exchange 2013 message hygiene functionality is useful but limited, and the fact that there is still no way to deploy Exchange 2013 into an existing Exchange 2007/2010 organization is difficult to swallow– Microsoft must know that this is a huge bottleneck for deployments, so there must be some very good reason why the needed coexistence patches have not yet been released.

I don’t think I’d agree with his opinion that “Exchange 2013 is not ready for prime time,” though. Shipping is a feature in itself, and while that’s no excuse to rush a poorly designed or unstable product out the door, I think that Microsoft generally does a good job of balancing market demand for releases against the engineering and support effort required to prepare those releases. With Kevin Allison’s many public statements about getting to a more stable, predictable release cadence for Exchange (both as an on-premises product and a service), I think the new normal is going to be seeing more frequent, more incremental releases. Exchange 2010 RTM, of course, lacked a number of features (such as the ability to manage public folders) that were added in SP1, so the idea of shipping what’s ready to hit a schedule and fleshing it out in an incremental release is nothing new.

Michael’s larger point is well taken: before you deploy Exchange 2013, you should be very sure that you understand how it differs from previous versions and how those differences may affect your messaging operations. As Tony and I work on Exchange 2013 Inside Out, we are taking careful note of the kinds of issues Michael points out, as well as how Microsoft responds to them; we’ll keep doing so throughout the release of Exchange 2010 SP3 and, eventually, Exchange 2013 SP1, to make sure that the book reflects the best available knowledge when it’s released later this year.

In possibly related news, if you’re interested in software engineering and release management issues, I suspect Steven Sinofsky’s new blog, Learning by Shipping, might be of interest to you.

Now, back to the unified messaging chapter… and go Seahawks!