Tag Archives: Exchange 2013

Microsoft Exchange engineering and cloud-scale

The Exchange team (or at least Perry Clarke, its fearless leader) has been known to describe Exchange Online as “the gateway drug to the cloud.” But how did that come to pass?

This week at Ignite, I was lucky enough to have dinner with some folks from the Exchange product team and a very, very large customer where we discussed the various ways in which Exchange engineering has blazed a trail the rest of Microsoft’s server products have eventually followed. After a bracing Twitter discussion this afternoon with @swiftonsecurity and some of her other followers, I thought it would be fun to put together a partial list of some of the things we discussed to illustrate how the Exchange team has built a stairway to heaven, or an elevator to the cloud, or something like that.

Let’s start with PowerShell. Love it or hate it, it is here, so we all have to deal with it. In 2007, the idea that Exchange would be built on PS was both revolutionary and, to many, revolting, but it allowed Microsoft to do several important things (not all of which shipped in Exchange 2007, but all of which are critical to cloud operations):

  • Greatly improve testability, both for the developers themselves but also for administrators, who now got a suite of protocol and endpoint-related tests they could run as part of troubleshooting– critically important when you have to troubleshoot in a global network of data centers hosting tens of millions of mailboxes
  • Fully enable role-based access control, also critical for cloud deployments where customers want to control who can do what with their data
  • Finally decouple the presentation layer of the UI (EMC, EAC, etc) from business logic
  • Massively improve the tools for scripting, including enabling very large-scale bulk operations– an obvious requirement for a cloud-scale service

Requiring PowerShell was a bold move by the Exchange team but one which has both paid off hugely and one that’s been echoed by the Windows, SharePoint, SQL Server and Skype teams, all of whom depend on it for managing their own cloud services. (See also: the Microsoft Graph APIs.)

Then there’s storage performance. In ancient days, getting scale from Exchange pretty much required the use of SANs due to Exchange’s IO requirements. Now, thanks to the IOPS diet imposed by Exchange engineering, it doesn’t. Tony does his usual excellent job of summarizing the actual reductions. Summary: Exchange 2016 requires roughly 96% fewer IOPS than Exchange 2003 did. There have been a ton of storage performance improvements in Exchange’s sister products (notably SQL) but those have their own stories that I’m not competent to tell. The relentless drive to cut IOPS requirements was one of the biggest enablers for Exchange Online, since controlling storage provisioning costs is critical for any type of scaled cloud service.

Of course, data protection is critical too. Exchange moved from having a single monolithic database to one with separate property and MIME databases (Exchange 2000) then to having software-based database replication with clustering (Exchange 2007) to shared-nothing, fully-replicated active/passive database replication (Exchange 2010 and later). Keeping multiple separate database copies (including lagged copies) enables all sorts of DR and HA scenarios that previously had required SANs. The ability to reliably use cheap JBOD disks, which thanks to Moore’s Law have embiggened nicely during Exchange’s lifetime, has been a key enabler for Exchange Online.

Then there’s a bunch of other architectural changes and improvements that are really only interesting to Exchange nerds. For the latest example, I present “read from passive,” but there’s also all the stuff covered by the Preferred Architecture.

Oh, I almost forgot: managed availability gives ExO a fair degree of self-healing, although its behavior sometimes surprises on-prem admins who see it do things on their behalf unexpectedly.

Oh, and let’s not forget the conversion of all the Exchange codebase to managed code– that was an important accelerator for the move to the cloud, as well as serving as a lighthouse for other product groups with code of similar vintage.

There are more examples, I’m sure, but these should get the point across– there’s been a steady stream of architectural changes in the nearly 20 years since Exchange 4.0 shipped that have led directly to the capability, power, and reliability of Exchange Online– which really has been the gateway drug for getting Microsoft’s customers to Office 365.



Leave a comment

Filed under UC&C

Microsoft rolls out Clutter admin improvements

Back in November, I wrote about my early experience with the Office 365 Clutter feature. I’ve been using it on and off– mostly off, due to a rare bug that surfaced because my mailbox is actually hosted on a portion of the Office 365 cloud that descends from the old Exchange Labs “friends and family” tenant. The bug kept Clutter from correctly moving clutter messages automatically; once it was fixed things returned to normal after I re-enabled the Clutter feature, and I’ve been happily using it since.

One of the big advantages of Office 365 is that the service team can develop and release new features much faster than they can for on-premises services. Sure enough, Microsoft today announced three new features for Clutter.

The biggest of these is the ability to create transport rules that flag messages, or senders, as exempt from Clutter processing. This is exactly the same thing as specifying safe senders for message hygiene filtering, although the implementation is a little different. You’ll create a transport rule that has the conditions and exceptions you want, but with an action that adds a header value of “ClutterBypassedByTransportRuleOverride: TRUE”, as described here. I have not personally had even a single false positive from Clutter since I’ve been testing it, and I haven’t seen any complaints about false positive problems from other users, MVPs, or customers. Having said that, Microsoft was smart to include a way to exempt certain messages from processing, as this will soothe some users and tenant administrators who are worried about the potential to have important messages be misdirected.

Second, the Clutter folder can now be managed by retention policies. This is an eminently logical thing to do, and it nicely highlights the flexibility of Exchange’s messaging records management system.

Rounding out the trio, you now have a very limited ability to customize the message that users see when they enable Clutter for their mailboxes: you can change the display name that the notification appears to be from, and you’ll soon be abe to change the logo. Frankly, this is weak sauce; there’s no way to customize the text of the notification, add custom URLs to it, or otherwise modify it in a useful way. Long-time Exchange administrators will recognize a familiar pattern exemplified by customizable delivery status notifications (DSNs), quota warning messages, and MailTips in previous versions of Exchange: first Microsoft delivered a useful feature with no customization capability, then they enabled limited customization, then (after prolonged complaining from customers) they broadened the range of things that could be customized. Let’s hope that pattern holds here.

There’s still one weak spot in the Clutter feature set: it still requires individual users to opt in (or out). While it’s true that users would likely be alarmed by the sudden forceful application or removal of the Clutter feature from their mailboxes, it’s also true that Office 365 as a whole needs to provide better controls for administrators to regulate which service features users have access to. I am hopeful that we’ll see better admin controls (and reporting) for this feature in the future.

While these improvements aren’t necessarily earth-shaking, they do add some welcome utility to what is already a valuable feature. Clutter is a great example of a feature that can make a measurable positive difference in users’ satisfaction with the service, and I look forward to more improvements in the feature.

1 Comment

Filed under Office 365, UC&C

The difference between supportability and patching

I’m at the annual MVP Summit this week, and everything we hear and see is pretty much NDA (except for pictures of Flat Tony). However, we just had a really interesting discussion that I think is safe to abstract here.

A couple years ago I wrote a post about what it means to be supported or unsupported. What I wrote then still stands: when Microsoft says something is unsupported, there can be multiple reasons for that label, and you do whatever-it-is at your own risk.

Microsoft’s support policy for Exchange 2013 can be summed up as “N-1”: when they release a new cumulative update (CU) or service pack, that version and the previous version are considered to be supported. So, in the fullness of time, when we get Exchange 2013 CU7, then CU6 and CU7 will be the officially supported versions.

It’s very clear that there’s a lot of confusion about what “supported” means in this context. Microsoft product support will always support you if you call for help with a product that’s within its lifecycle window. Call them today and ask how to configure Exchange ActiveSync on Exchange 2010 RU2, they’ll help you. Call to ask about an issue you’re seeing with DAG failover in Exchange 2013 CU2, they’ll help you. Call for help with Exchange 2003, and they may even help you on a best-effort basis.

What they won’t do is create fixes for bugs or problems in unsupported versions.

If you call them and say “hey, I’m having this problem with Exchange 2013 SP1,” they will help you troubleshoot it. If it’s a known problem, they may tell you “update to CU5 or later”– but Microsoft will not create a hotfix or IU that fixes that problem in SP1, or any other older version that’s outside that N-1 boundary.

So: help always, bug fixes only within the support boundary. Tell your friends.


1 Comment

Filed under Office 365, UC&C

Moving to Summit 7 Systems

It must be the season or something. Like several of my peers (e.g. Paul, Phoummala, and Michael, to name 3), I’m moving on from my current position to a unique new challenge. In my case, I’m taking the role of Principal Architect at Summit 7 Systems.

Astute readers may remember that, just about a year ago, I joined Dell’s global services organization as a global principal consultant. I was fortunate to work with a large group of extremely smart and talented people, including several MCMs (Todd, Dave, Andrew, Ron, and Alessandro, y’all know who I’m talking about!) Working for a large company has both its benefits and challenges, but I was happy with the work I was doing and the people I was working with. However, then this happened.

Scott Edwards, cofounder of Summit 7 and a longtime friend from my prior time in Huntsville, told me that he wanted to grow Summit 7’s very successful business, previously focused on SharePoint and business process consulting, to expand into Office 365, Lync, and Exchange. Would I be interested in helping? Yes, yes, I would. Summit 7 is already really well known in the SharePoint world, with customers such as NASA, Coca-Cola, Nucor Steel, and the State of Minnesota. SharePoint consulting is a very different world in many ways from what I’m used to, so it will be interesting, challenging, and FUN to carry the Lync/Exchange/365 torch into a new environment.

In my new role, I’ll be building a practice essentially from scratch, but I’ll be able to take advantage of Summit 7’s deep bench of project management, business process consulting, marketing, and sales talent. I’m excited by the opportunity, which is essentially the next step forward from my prior work as a delivery specialist. I am not yet taking over the role of Summit 7’s corporate pilot, but that’s on my to-do list as well. (A couple of folks have already asked, and the answer is: yes, I will be flying myself occasionally to customer gigs, something that Dell explicitly forbade. Can’t wait!)

This is an exciting opportunity for me and I relish the chance to get in and start punching. Stay tuned! (Meanwhile, you can read the official Summit 7 press release here.)


Filed under UC&C

Mailbox-level backups in Office 365

Executive summary: there aren’t any, so plan accordingly.

Recently I was working with a customer (let’s call him Joe, as in “Joe Customer”) who was considering moving to Office 365. They went to our executive briefing center in Austin, where some Dell sales hotshots met and briefed them, then I joined in via Lync (with video!) for a demo. The demo went really well, and I was feeling good about our odds of winning the deal… until the Q&A period.

“How does Office 365 provide mailbox-level backups?” Joe asked.

“Well, it doesn’t,” I said. “Microsoft doesn’t give you direct access to the mailbox databases. Instead, they give you deleted item retention, plus you can use single-item retention and various types of holds.” Then I sent him this link.

“Let me tell you why I’m asking,” Joe retorted after skimming the link. “A couple of times we’ve lost our CIO’s calendar. He uses an Outlook add-in that prints out his calendar every day, and sometimes it corrupts calendar items. We need to be able to do mailbox-level backups so that we can restore any damaged items.”

At that point I had to admit to being stumped. Sure enough, there is no Office 365 feature or capability that protects against this kind of logical corruption. You can’t use New-MailboxExportRequest or the EAC to export the contents of Office 365 mailboxes to PST files. You obviously can’t run backup tools that run on the Exchange server against your Office 365 mailbox databases; there may exist tools that use EWS to directly access a mailbox and make a backup copy, but I don’t know of any that are built for that purpose.

I ran Joe’s query past a few folks I know on the 365 team. Apart from the (partially helpful) suggestion not to run Outlook add-ins that are known to corrupt data, none of them had good answers either.

While it’s tempting to view the inability to do mailbox-level backups as a limitation, it’s perfectly understandable. Microsoft spent years trying to get people not to run brick-level backups using MAPI. The number of use cases for this feature is getting smaller each year as both the data-integrity and retention features of Exchange get better. In fact, one of the major reasons that we now have single-item recovery in its current form is because customers kept asking for expanded tools to recover deleted items, either after an accidental deletion or a purge. Exchange also incorporates all sorts of infrastructure to protect against data loss, both for stored data and data in transit, but nothing really helps in this case: the corrupt data comes from the client, and Exchange is faithfully storing and replicating what it gets from the client. In fairness, we have seen business logic added to Exchange in the past to protect against problems caused by malformed calendar entries created by old versions of Outlook, but clearly Microsoft can’t do that for every random add-in that might stomp on a user’s calendar.

A few days after the original presentation, I sent Joe an email summarizing what I’d found out and telling him that, if mailbox-level backup was an absolute requirement, he probably shouldn’t move those mailboxes to Office 365.

The moral of this story, to an extent that there is one, is that Microsoft is engineering Office 365 for the majority of their users and their needs. Just as Word (for instance) is supplemented by specialized plugins for reference and footnote tracking, mathematical typesetting, and chemistry diagrams, Exchange has a whole ecosystem of products that connect to it in various ways, and Office 365 doesn’t support every single one of those. The breadth and diversity of the Exchange ecosystem is one of the major reasons that I expect on-premises Exchange to be with us for years to come. Until it finally disappears, don’t forget to do some kind of backups.


Filed under Office 365, UC&C

Exchange Server and Azure: “not now” vs “never”

Wow, look what I found in my drafts folder: an old post.

Lots of Exchange admins have been wondering whether Windows Azure can be used to host Exchange. This is to be expected for two reasons. First, Microsoft has been steadily raising the volume of Azure-related announcements, demos, and other collateral material. TechEd 2014 was a great example: there were several Azure-related announcements, including the availability of ExpressRoute for private connections to the Azure cloud and several major new storage improvements. These changes build on their aggressive evangelism, which has been attempting, and succeeding, to convince iOS and Android developers to use Azure as the back-end service for their apps. The other reason, sadly, is why I’m writing: there’s a lot of misinformation about Exchange on Azure (e.g. this article from SearchExchange titled “Points to consider before running Exchange on Azure”, which is wrong, wrong, and wrong), and you need to be prepared to defuse its wrongness with customers who may misunderstand what they’re potentially getting into.

On its face, Azure’s infrastructure-as-a-service (IaaS) offering seems pretty compelling: you can build Windows Server VMs and host them in the Azure cloud. That seems like it would be a natural fit for Exchange, which is increasingly viewed as an infrastructure service by customers who depend on it. However, there are at least three serious problems with this approach.

First: it’s not supported by Microsoft, something that the “points to consider” article doesn’t even mention. The Exchange team doesn’t support Exchange 2010 or Exchange 2013 on Azure or Amazon EC2 or anyone else’s cloud service at present. It is possible that this will change in the future, but for now any customer who runs Exchange on Azure will be in an unsupported state. It’s fun to imagine scenarios where the Azure team takes over first-line support responsibility for customers running Exchange and other Microsoft server applications; this sounds a little crazy but the precedent exists, as EMC and other storage companies did exactly this for users of their replication solutions back in Exchange 5.5/2000 times. Having said that, don’t hold your breath. The Azure team has plenty of other more pressing work to do first, so I think that any change in this support model will require the Exchange team to buy in to it. The Azure team has been able to get that buy-in from SharePoint, Dynamics, and other major product groups within Microsoft, so this is by no means impossible.

Second: it’s more work. In some ways Azure gives you the worst of the hosted Exchange model: you have to do just as much work as you would if Exchange were hosted on-premises, but you’re also subject to service outages, inconsistent network latency, and all the other transient or chronic irritations that come, at no extra cost, with cloud services. Part of the reason that the Exchange team doesn’t support Azure is because there’s no way to guarantee that any IaaS provider is offering enough IOPS, low-enough latency, and so on, so troubleshooting performance or behavior problems with a service such as Azure can quickly turn into a nightmare. If Azure is able to provide guaranteed service levels for disk I/O throughput and latency, that would help quite a bit, but this would probably require significant engineering effort. Although I don’t recommend that you do it at the moment, you might be interested in this writeup on how to deploy Exchange on Azure; it gives a good look at some of the operational challenges you might face in setting up Exchange+Azure for test or demo use.

Third: it’s going to cost more. Remember that IaaS networks typically charge for resource consumption. Exchange 2013 (and Exchange 2010, too) is designed to be “always on”. The workload management features in Exchange 2013 provide throttling, sure, but they don’t eliminate all of the background maintenance that Exchange is more-or-less continuously performing. These tasks, including GAL grammar generation for Exchange UM, the managed folder assistant, calendar repair, and various database-related tasks, have to be run, and so IaaS-based Exchange servers are continually going to be racking up storage, CPU, and network charges. In fairness, I haven’t estimated what these charges might be for a typical test-lab environment; it’s possible that they’d be cheap enough to be tolerable, but I’m not betting on it, and no doubt a real deployment would be significantly more expensive.

Of course, all three of these problems are soluble: the Exchange team could at any time change their support policy for Exchange on Azure, and/or the Azure team could adjust the cost model to make the cost for doing so competitive with Office 365 or other hosted solutions. Interestingly, though, two different groups would have to make those decisions, and their interests don’t necessarily align, so it’s not clear to me if or when we might see this happen. Remember, the Office 365 team at Microsoft uses physical hardware exclusively for their operations.

Does that mean that Azure has no value for Exchange? On the contrary. At TechEd New Orleans in June 2013, Microsoft’s Scott Schnoll said they were studying the possibility of using an Azure VM as the witness server for DAGs in Exchange 2013 CU2 and later. This would be a super feature because it would allow customers with two or more physically separate data centers to build large DAGs that weren’t dependent on site interconnects (at the risk, of course, of requiring always-on connectivity to Azure). The cost and workload penalty for running an FSW on Azure would be low, too. In August 2013, the word came down: Azure in its present implementation isn’t suitable for use as an FSW. However, the Exchange team has requested some Azure functionality changes that would make it possible to run this configuration in the future, so we have that to look forward to.

Then we have the wide world of IaaS capabilities opened up by Windows Azure Active Directory (WAAD), Azure Rights Management Services, Azure Multi-Factor Authentication, and the large-volume disk ingestion program (now known as the Azure Import/Export Service). As time passes, Microsoft keeps delivering more, and better, Azure services that complement on-premises Exchange, which has been really interesting to watch. I expect that trend to continue, and there are other, less expensive ways to use IaaS for Exchange if you only want it for test labs and the like. More on that in a future post….


Filed under General Tech Stuff, UC&C

Speaking at Exchange Connections 2014

I’m excited to say that I’ll be presenting at Exchange Connections 2014, coming up this fall at the Aria in Las Vegas.

Tony posted the complete list of speakers and session titles a couple of days ago. I’m doing three sessions:

  • “Who Wears the Pants In Your Datacenter: Taming Managed Availability”: an all-new session in which the phrase “you’re not the boss of me” will feature prominently. You might want to prepare by reading my Windows IT Pro article on MA, sort of to set the table.
  • “Just Like Lemmings: Mass Migration to Office 365”: an all-new session that discusses the hows and whys of moving large volumes of mailbox and PST data into the service, using both Microsoft and third-party tools. (On the sometimes-contentious topic of public folder migration, I plead ignorance; see Sigi Jagott’s session if you want to know more). There is a big gap between theory and practice here and I plan to shine some light into it.
  • “Deep Dive: Exchange 2013 and Lync 2013 Integration” covers the nuts and bolts of how to tie Lync and Exchange 2013 together. Frankly, if you saw me present on this topic at DellWorld, MEC, or Lync Conference, you don’t need to attend this iteration. However, every time I’ve presented it, the room has been packed to capacity, so there’s clearly still demand for the material!

Exchange Connections always has a more relaxed, intimate feeling about it than the bigger Microsoft-themed conferences. This is in part because it’s not a Microsoft event and in part because it is considerably smaller. As a speaker, I really enjoy the chance to engage more deeply with the attendees than is possible at mega-events. If you’re planning to be there, great— and, if not, you should change your plans!

1 Comment

Filed under Office 365, UC&C