Tag Archives: Exchange 2013

Microsoft Exchange engineering and cloud-scale

The Exchange team (or at least Perry Clarke, its fearless leader) has been known to describe Exchange Online as “the gateway drug to the cloud.” But how did that come to pass?

This week at Ignite, I was lucky enough to have dinner with some folks from the Exchange product team and a very, very large customer where we discussed the various ways in which Exchange engineering has blazed a trail the rest of Microsoft’s server products have eventually followed. After a bracing Twitter discussion this afternoon with @swiftonsecurity and some of her other followers, I thought it would be fun to put together a partial list of some of the things we discussed to illustrate how the Exchange team has built a stairway to heaven, or an elevator to the cloud, or something like that.

Let’s start with PowerShell. Love it or hate it, it is here, so we all have to deal with it. In 2007, the idea that Exchange would be built on PS was both revolutionary and, to many, revolting, but it allowed Microsoft to do several important things (not all of which shipped in Exchange 2007, but all of which are critical to cloud operations):

  • Greatly improve testability, both for the developers themselves but also for administrators, who now got a suite of protocol and endpoint-related tests they could run as part of troubleshooting– critically important when you have to troubleshoot in a global network of data centers hosting tens of millions of mailboxes
  • Fully enable role-based access control, also critical for cloud deployments where customers want to control who can do what with their data
  • Finally decouple the presentation layer of the UI (EMC, EAC, etc) from business logic
  • Massively improve the tools for scripting, including enabling very large-scale bulk operations– an obvious requirement for a cloud-scale service

Requiring PowerShell was a bold move by the Exchange team but one which has both paid off hugely and one that’s been echoed by the Windows, SharePoint, SQL Server and Skype teams, all of whom depend on it for managing their own cloud services. (See also: the Microsoft Graph APIs.)

Then there’s storage performance. In ancient days, getting scale from Exchange pretty much required the use of SANs due to Exchange’s IO requirements. Now, thanks to the IOPS diet imposed by Exchange engineering, it doesn’t. Tony does his usual excellent job of summarizing the actual reductions. Summary: Exchange 2016 requires roughly 96% fewer IOPS than Exchange 2003 did. There have been a ton of storage performance improvements in Exchange’s sister products (notably SQL) but those have their own stories that I’m not competent to tell. The relentless drive to cut IOPS requirements was one of the biggest enablers for Exchange Online, since controlling storage provisioning costs is critical for any type of scaled cloud service.

Of course, data protection is critical too. Exchange moved from having a single monolithic database to one with separate property and MIME databases (Exchange 2000) then to having software-based database replication with clustering (Exchange 2007) to shared-nothing, fully-replicated active/passive database replication (Exchange 2010 and later). Keeping multiple separate database copies (including lagged copies) enables all sorts of DR and HA scenarios that previously had required SANs. The ability to reliably use cheap JBOD disks, which thanks to Moore’s Law have embiggened nicely during Exchange’s lifetime, has been a key enabler for Exchange Online.

Then there’s a bunch of other architectural changes and improvements that are really only interesting to Exchange nerds. For the latest example, I present “read from passive,” but there’s also all the stuff covered by the Preferred Architecture.

Oh, I almost forgot: managed availability gives ExO a fair degree of self-healing, although its behavior sometimes surprises on-prem admins who see it do things on their behalf unexpectedly.

Oh, and let’s not forget the conversion of all the Exchange codebase to managed code– that was an important accelerator for the move to the cloud, as well as serving as a lighthouse for other product groups with code of similar vintage.

There are more examples, I’m sure, but these should get the point across– there’s been a steady stream of architectural changes in the nearly 20 years since Exchange 4.0 shipped that have led directly to the capability, power, and reliability of Exchange Online– which really has been the gateway drug for getting Microsoft’s customers to Office 365.

 

 

Leave a comment

Filed under UC&C

Microsoft rolls out Clutter admin improvements

Back in November, I wrote about my early experience with the Office 365 Clutter feature. I’ve been using it on and off– mostly off, due to a rare bug that surfaced because my mailbox is actually hosted on a portion of the Office 365 cloud that descends from the old Exchange Labs “friends and family” tenant. The bug kept Clutter from correctly moving clutter messages automatically; once it was fixed things returned to normal after I re-enabled the Clutter feature, and I’ve been happily using it since.

One of the big advantages of Office 365 is that the service team can develop and release new features much faster than they can for on-premises services. Sure enough, Microsoft today announced three new features for Clutter.

The biggest of these is the ability to create transport rules that flag messages, or senders, as exempt from Clutter processing. This is exactly the same thing as specifying safe senders for message hygiene filtering, although the implementation is a little different. You’ll create a transport rule that has the conditions and exceptions you want, but with an action that adds a header value of “ClutterBypassedByTransportRuleOverride: TRUE”, as described here. I have not personally had even a single false positive from Clutter since I’ve been testing it, and I haven’t seen any complaints about false positive problems from other users, MVPs, or customers. Having said that, Microsoft was smart to include a way to exempt certain messages from processing, as this will soothe some users and tenant administrators who are worried about the potential to have important messages be misdirected.

Second, the Clutter folder can now be managed by retention policies. This is an eminently logical thing to do, and it nicely highlights the flexibility of Exchange’s messaging records management system.

Rounding out the trio, you now have a very limited ability to customize the message that users see when they enable Clutter for their mailboxes: you can change the display name that the notification appears to be from, and you’ll soon be abe to change the logo. Frankly, this is weak sauce; there’s no way to customize the text of the notification, add custom URLs to it, or otherwise modify it in a useful way. Long-time Exchange administrators will recognize a familiar pattern exemplified by customizable delivery status notifications (DSNs), quota warning messages, and MailTips in previous versions of Exchange: first Microsoft delivered a useful feature with no customization capability, then they enabled limited customization, then (after prolonged complaining from customers) they broadened the range of things that could be customized. Let’s hope that pattern holds here.

There’s still one weak spot in the Clutter feature set: it still requires individual users to opt in (or out). While it’s true that users would likely be alarmed by the sudden forceful application or removal of the Clutter feature from their mailboxes, it’s also true that Office 365 as a whole needs to provide better controls for administrators to regulate which service features users have access to. I am hopeful that we’ll see better admin controls (and reporting) for this feature in the future.

While these improvements aren’t necessarily earth-shaking, they do add some welcome utility to what is already a valuable feature. Clutter is a great example of a feature that can make a measurable positive difference in users’ satisfaction with the service, and I look forward to more improvements in the feature.

1 Comment

Filed under Office 365, UC&C

The difference between supportability and patching

I’m at the annual MVP Summit this week, and everything we hear and see is pretty much NDA (except for pictures of Flat Tony). However, we just had a really interesting discussion that I think is safe to abstract here.

A couple years ago I wrote a post about what it means to be supported or unsupported. What I wrote then still stands: when Microsoft says something is unsupported, there can be multiple reasons for that label, and you do whatever-it-is at your own risk.

Microsoft’s support policy for Exchange 2013 can be summed up as “N-1”: when they release a new cumulative update (CU) or service pack, that version and the previous version are considered to be supported. So, in the fullness of time, when we get Exchange 2013 CU7, then CU6 and CU7 will be the officially supported versions.

It’s very clear that there’s a lot of confusion about what “supported” means in this context. Microsoft product support will always support you if you call for help with a product that’s within its lifecycle window. Call them today and ask how to configure Exchange ActiveSync on Exchange 2010 RU2, they’ll help you. Call to ask about an issue you’re seeing with DAG failover in Exchange 2013 CU2, they’ll help you. Call for help with Exchange 2003, and they may even help you on a best-effort basis.

What they won’t do is create fixes for bugs or problems in unsupported versions.

If you call them and say “hey, I’m having this problem with Exchange 2013 SP1,” they will help you troubleshoot it. If it’s a known problem, they may tell you “update to CU5 or later”– but Microsoft will not create a hotfix or IU that fixes that problem in SP1, or any other older version that’s outside that N-1 boundary.

So: help always, bug fixes only within the support boundary. Tell your friends.

 

1 Comment

Filed under Office 365, UC&C

Moving to Summit 7 Systems

It must be the season or something. Like several of my peers (e.g. Paul, Phoummala, and Michael, to name 3), I’m moving on from my current position to a unique new challenge. In my case, I’m taking the role of Principal Architect at Summit 7 Systems.

Astute readers may remember that, just about a year ago, I joined Dell’s global services organization as a global principal consultant. I was fortunate to work with a large group of extremely smart and talented people, including several MCMs (Todd, Dave, Andrew, Ron, and Alessandro, y’all know who I’m talking about!) Working for a large company has both its benefits and challenges, but I was happy with the work I was doing and the people I was working with. However, then this happened.

Scott Edwards, cofounder of Summit 7 and a longtime friend from my prior time in Huntsville, told me that he wanted to grow Summit 7’s very successful business, previously focused on SharePoint and business process consulting, to expand into Office 365, Lync, and Exchange. Would I be interested in helping? Yes, yes, I would. Summit 7 is already really well known in the SharePoint world, with customers such as NASA, Coca-Cola, Nucor Steel, and the State of Minnesota. SharePoint consulting is a very different world in many ways from what I’m used to, so it will be interesting, challenging, and FUN to carry the Lync/Exchange/365 torch into a new environment.

In my new role, I’ll be building a practice essentially from scratch, but I’ll be able to take advantage of Summit 7’s deep bench of project management, business process consulting, marketing, and sales talent. I’m excited by the opportunity, which is essentially the next step forward from my prior work as a delivery specialist. I am not yet taking over the role of Summit 7’s corporate pilot, but that’s on my to-do list as well. (A couple of folks have already asked, and the answer is: yes, I will be flying myself occasionally to customer gigs, something that Dell explicitly forbade. Can’t wait!)

This is an exciting opportunity for me and I relish the chance to get in and start punching. Stay tuned! (Meanwhile, you can read the official Summit 7 press release here.)

4 Comments

Filed under UC&C

Mailbox-level backups in Office 365

Executive summary: there aren’t any, so plan accordingly.

Recently I was working with a customer (let’s call him Joe, as in “Joe Customer”) who was considering moving to Office 365. They went to our executive briefing center in Austin, where some Dell sales hotshots met and briefed them, then I joined in via Lync (with video!) for a demo. The demo went really well, and I was feeling good about our odds of winning the deal… until the Q&A period.

“How does Office 365 provide mailbox-level backups?” Joe asked.

“Well, it doesn’t,” I said. “Microsoft doesn’t give you direct access to the mailbox databases. Instead, they give you deleted item retention, plus you can use single-item retention and various types of holds.” Then I sent him this link.

“Let me tell you why I’m asking,” Joe retorted after skimming the link. “A couple of times we’ve lost our CIO’s calendar. He uses an Outlook add-in that prints out his calendar every day, and sometimes it corrupts calendar items. We need to be able to do mailbox-level backups so that we can restore any damaged items.”

At that point I had to admit to being stumped. Sure enough, there is no Office 365 feature or capability that protects against this kind of logical corruption. You can’t use New-MailboxExportRequest or the EAC to export the contents of Office 365 mailboxes to PST files. You obviously can’t run backup tools that run on the Exchange server against your Office 365 mailbox databases; there may exist tools that use EWS to directly access a mailbox and make a backup copy, but I don’t know of any that are built for that purpose.

I ran Joe’s query past a few folks I know on the 365 team. Apart from the (partially helpful) suggestion not to run Outlook add-ins that are known to corrupt data, none of them had good answers either.

While it’s tempting to view the inability to do mailbox-level backups as a limitation, it’s perfectly understandable. Microsoft spent years trying to get people not to run brick-level backups using MAPI. The number of use cases for this feature is getting smaller each year as both the data-integrity and retention features of Exchange get better. In fact, one of the major reasons that we now have single-item recovery in its current form is because customers kept asking for expanded tools to recover deleted items, either after an accidental deletion or a purge. Exchange also incorporates all sorts of infrastructure to protect against data loss, both for stored data and data in transit, but nothing really helps in this case: the corrupt data comes from the client, and Exchange is faithfully storing and replicating what it gets from the client. In fairness, we have seen business logic added to Exchange in the past to protect against problems caused by malformed calendar entries created by old versions of Outlook, but clearly Microsoft can’t do that for every random add-in that might stomp on a user’s calendar.

A few days after the original presentation, I sent Joe an email summarizing what I’d found out and telling him that, if mailbox-level backup was an absolute requirement, he probably shouldn’t move those mailboxes to Office 365.

The moral of this story, to an extent that there is one, is that Microsoft is engineering Office 365 for the majority of their users and their needs. Just as Word (for instance) is supplemented by specialized plugins for reference and footnote tracking, mathematical typesetting, and chemistry diagrams, Exchange has a whole ecosystem of products that connect to it in various ways, and Office 365 doesn’t support every single one of those. The breadth and diversity of the Exchange ecosystem is one of the major reasons that I expect on-premises Exchange to be with us for years to come. Until it finally disappears, don’t forget to do some kind of backups.

8 Comments

Filed under Office 365, UC&C

Exchange Server and Azure: “not now” vs “never”

Wow, look what I found in my drafts folder: an old post.

Lots of Exchange admins have been wondering whether Windows Azure can be used to host Exchange. This is to be expected for two reasons. First, Microsoft has been steadily raising the volume of Azure-related announcements, demos, and other collateral material. TechEd 2014 was a great example: there were several Azure-related announcements, including the availability of ExpressRoute for private connections to the Azure cloud and several major new storage improvements. These changes build on their aggressive evangelism, which has been attempting, and succeeding, to convince iOS and Android developers to use Azure as the back-end service for their apps. The other reason, sadly, is why I’m writing: there’s a lot of misinformation about Exchange on Azure (e.g. this article from SearchExchange titled “Points to consider before running Exchange on Azure”, which is wrong, wrong, and wrong), and you need to be prepared to defuse its wrongness with customers who may misunderstand what they’re potentially getting into.

On its face, Azure’s infrastructure-as-a-service (IaaS) offering seems pretty compelling: you can build Windows Server VMs and host them in the Azure cloud. That seems like it would be a natural fit for Exchange, which is increasingly viewed as an infrastructure service by customers who depend on it. However, there are at least three serious problems with this approach.

First: it’s not supported by Microsoft, something that the “points to consider” article doesn’t even mention. The Exchange team doesn’t support Exchange 2010 or Exchange 2013 on Azure or Amazon EC2 or anyone else’s cloud service at present. It is possible that this will change in the future, but for now any customer who runs Exchange on Azure will be in an unsupported state. It’s fun to imagine scenarios where the Azure team takes over first-line support responsibility for customers running Exchange and other Microsoft server applications; this sounds a little crazy but the precedent exists, as EMC and other storage companies did exactly this for users of their replication solutions back in Exchange 5.5/2000 times. Having said that, don’t hold your breath. The Azure team has plenty of other more pressing work to do first, so I think that any change in this support model will require the Exchange team to buy in to it. The Azure team has been able to get that buy-in from SharePoint, Dynamics, and other major product groups within Microsoft, so this is by no means impossible.

Second: it’s more work. In some ways Azure gives you the worst of the hosted Exchange model: you have to do just as much work as you would if Exchange were hosted on-premises, but you’re also subject to service outages, inconsistent network latency, and all the other transient or chronic irritations that come, at no extra cost, with cloud services. Part of the reason that the Exchange team doesn’t support Azure is because there’s no way to guarantee that any IaaS provider is offering enough IOPS, low-enough latency, and so on, so troubleshooting performance or behavior problems with a service such as Azure can quickly turn into a nightmare. If Azure is able to provide guaranteed service levels for disk I/O throughput and latency, that would help quite a bit, but this would probably require significant engineering effort. Although I don’t recommend that you do it at the moment, you might be interested in this writeup on how to deploy Exchange on Azure; it gives a good look at some of the operational challenges you might face in setting up Exchange+Azure for test or demo use.

Third: it’s going to cost more. Remember that IaaS networks typically charge for resource consumption. Exchange 2013 (and Exchange 2010, too) is designed to be “always on”. The workload management features in Exchange 2013 provide throttling, sure, but they don’t eliminate all of the background maintenance that Exchange is more-or-less continuously performing. These tasks, including GAL grammar generation for Exchange UM, the managed folder assistant, calendar repair, and various database-related tasks, have to be run, and so IaaS-based Exchange servers are continually going to be racking up storage, CPU, and network charges. In fairness, I haven’t estimated what these charges might be for a typical test-lab environment; it’s possible that they’d be cheap enough to be tolerable, but I’m not betting on it, and no doubt a real deployment would be significantly more expensive.

Of course, all three of these problems are soluble: the Exchange team could at any time change their support policy for Exchange on Azure, and/or the Azure team could adjust the cost model to make the cost for doing so competitive with Office 365 or other hosted solutions. Interestingly, though, two different groups would have to make those decisions, and their interests don’t necessarily align, so it’s not clear to me if or when we might see this happen. Remember, the Office 365 team at Microsoft uses physical hardware exclusively for their operations.

Does that mean that Azure has no value for Exchange? On the contrary. At TechEd New Orleans in June 2013, Microsoft’s Scott Schnoll said they were studying the possibility of using an Azure VM as the witness server for DAGs in Exchange 2013 CU2 and later. This would be a super feature because it would allow customers with two or more physically separate data centers to build large DAGs that weren’t dependent on site interconnects (at the risk, of course, of requiring always-on connectivity to Azure). The cost and workload penalty for running an FSW on Azure would be low, too. In August 2013, the word came down: Azure in its present implementation isn’t suitable for use as an FSW. However, the Exchange team has requested some Azure functionality changes that would make it possible to run this configuration in the future, so we have that to look forward to.

Then we have the wide world of IaaS capabilities opened up by Windows Azure Active Directory (WAAD), Azure Rights Management Services, Azure Multi-Factor Authentication, and the large-volume disk ingestion program (now known as the Azure Import/Export Service). As time passes, Microsoft keeps delivering more, and better, Azure services that complement on-premises Exchange, which has been really interesting to watch. I expect that trend to continue, and there are other, less expensive ways to use IaaS for Exchange if you only want it for test labs and the like. More on that in a future post….

5 Comments

Filed under General Tech Stuff, UC&C

Speaking at Exchange Connections 2014

I’m excited to say that I’ll be presenting at Exchange Connections 2014, coming up this fall at the Aria in Las Vegas.

Tony posted the complete list of speakers and session titles a couple of days ago. I’m doing three sessions:

  • “Who Wears the Pants In Your Datacenter: Taming Managed Availability”: an all-new session in which the phrase “you’re not the boss of me” will feature prominently. You might want to prepare by reading my Windows IT Pro article on MA, sort of to set the table.
  • “Just Like Lemmings: Mass Migration to Office 365”: an all-new session that discusses the hows and whys of moving large volumes of mailbox and PST data into the service, using both Microsoft and third-party tools. (On the sometimes-contentious topic of public folder migration, I plead ignorance; see Sigi Jagott’s session if you want to know more). There is a big gap between theory and practice here and I plan to shine some light into it.
  • “Deep Dive: Exchange 2013 and Lync 2013 Integration” covers the nuts and bolts of how to tie Lync and Exchange 2013 together. Frankly, if you saw me present on this topic at DellWorld, MEC, or Lync Conference, you don’t need to attend this iteration. However, every time I’ve presented it, the room has been packed to capacity, so there’s clearly still demand for the material!

Exchange Connections always has a more relaxed, intimate feeling about it than the bigger Microsoft-themed conferences. This is in part because it’s not a Microsoft event and in part because it is considerably smaller. As a speaker, I really enjoy the chance to engage more deeply with the attendees than is possible at mega-events. If you’re planning to be there, great— and, if not, you should change your plans!

1 Comment

Filed under Office 365, UC&C

Microsoft updates Recoverable Items quota for Office 365 users

Remember when I posted about the 100GB limit for Personal Archive mailboxes in Office 365? It turns out that there was another limit that almost no one knew about, primarily because it involves mailbox retention. As of today, when you put an Office 365 mailbox on In-Place Hold, the size of the Recoverable Items folder is capped at 30GB. This is plenty for the vast majority of customers because a) not many customers use In-Place Hold in the first place and b) not many users have mailboxes that are large enough to exceed the 30GB quota. Multiply two small numbers together and you get another small number.

However, there are some customers for whom this is a problem. One of the most interesting things about Office 365 to me is the speed at which Microsoft can respond to their requests by changing aspects of the service architecture and provisioning. In this case, the Exchange team is planning to increase the size of the Recoverable Items quota to 100GB. Interestingly, they’re actually starting by increasing the quota for user mailboxes that are now on hold— so from now until July 2014, they’ll be silently increasing the quota for those users. If you put a user on hold today, however, their quota may not be set to 100GB until sometime later.

If you need an immediate quota increase, or if you’re using a dedicated tenant, you’ll still have to use the existing mechanism of filing a support ticket to have the quota increased.

There’s no public post on this yet, but I expect one shortly. In the meantime, bask in the knowledge that with a 50GB mailbox, 100GB Personal Archive, and 100GB Recoverable Items quota, your users probably aren’t going to run out of mailbox space any time soon.

2 Comments

Filed under Office 365, UC&C

Two-factor authentication for Outlook and Office 2013 clients

I don’t usually put on my old man hat, but indulge me for a second. Back in February 2000, in my long-forgotten column for TechNet, here’s what I said about single-factor passwords:

I’m going to let you in on a secret that’s little discussed outside the security world: reusable passwords are evil.

I stand by the second half of that statement: reusable passwords are still evil, 14 years later, but at least the word is getting out, and multi-factor authentication is becoming more and more common in both consumer and business systems. I was wrong when I assumed that smart cards would become ubiquitous as a second authentication factor; instead, the “something you have” role is increasingly often filled by a mobile phone that can receive SMS messages. Microsoft bought into that trend with their 2012 purchase of PhoneFactor, which is now integrated into Azure. Now Microsoft is extending MFA support into Outlook and the rest of the Office 2013 client applications, with a few caveats. I attended a great session at MEC 2014 presented by Microsoft’s Erik Ashby and Franklin Williams that both outlined the current state of Office 365-integrated MFA and outlined Microsoft’s plans to extend MFA to Outlook.

First, keep in mind that Office 365 already offers multi-factor authentication, once you enable it, for your web-based clients. You can use SMS-based authentication, have the service call you via phone, or use a mobile app that generates authentication codes, and you can define “app passwords” that are used instead of your primary credentials for applications— like Outlook, as it happens— that don’t currently understand MFA. You have to enable MFA for your tenant, then enable it for individual users. All of these services are included with Office 365 SKUs, and they rely on the Azure MFA service. You can, if you wish, buy a separate subscription to Azure MFA if you want additional functionality, like the ability to customize the caller ID that appears when the service calls your users.

With that said, here’s what Erik and Franklin talked about…

To start with, we have to distinguish between the three types of identities that can be used to authenticate against the service. Without going into every detail, it’s fair to summarize these as follows:

  • Cloud identities are homed in Azure Active Directory (AAD). There’s no synchronization with on-premises AD because there isn’t one.
  • Directory sync (or just “dirsync”) uses Microsoft’s dirsync tool, or an equivalent third-party tool, to sync an on-premises account with AAD. This essentially gives services that consume AAD a mostly-read-only copy of your organization’s AD.
  • Federated identity uses a federation broker or service such as Active Directory Federation Services (AD FS), Okta, Centrify, and Ping to allow your organization’s AD to answer authentication queries from Office 365 services. In January 2014 Microsoft announced a “Works With Office 365 – Identity” logo program, so if you don’t want to use AD FS you can choose another federation toolset that better meets your requirements.

Client updates are coming to the Office 2013 clients: Outlook, Lync, Word, Excel,  PowerPoint, and SkyDrive Pro. With these updates, you’ll see a single unified authentication window for all of the clients, similar (but not necessarily identical) to the existing login window you get on Windows when signing into a SkyDrive or SkyDrive Pro library from within an Office client. From that authentication window, you’ll be able to enter the second authentication factor that you received via phone call, SMS, or authentication app. During the presentation, Franklin (or maybe Erik?) said “if you can authenticate in a web browser, you can authenticate in Office clients”— very cool. (PowerShell will be getting MFA support too, but it wasn’t clear to me exactly when that was happening).

These client updates will also provide support for two specific types of smart cards: the US Department of Defense Common Access Card (CAC) and the similar-but-civilian Personal Identity Verification (PIV) card. Instead of using a separate authentication token provided by the service, you’ll plug in your smart card, authenticate to it with your PIN, and away you go.

All three of the identity types of these methods provide support for MFA; federated identity will gain the ability to do true single sign-on (SSO) jn Office 2013 clients, which will be a welcome usability improvement. Outlook will get SSO capabilities with the other two identity types, too.

How do the updates work? That’s where the magic part comes in. The Azure Active Directory Authentication Library (ADAL) is being extended to provide support for MFA. When the Office client makes a request to the service the service will return a header that instructs the client to visit a security token service (STS) using OAuth. At that point, Office uses ADAL to launch the browser control that displays the authentication page, then, as Erik puts it, “MFA and federation magic happens transparent to Office.” If the authentication succeeds, Office gets security tokens that it caches and uses for service authentication. (The flow is described in more detail in the video from the session, which is available now for MEC attendees and will be available in 60 days or so for non-attendees).

There are two important caveats that were a little buried in the presentation. First is that MFA in Outlook 2013 will require the use of MAPI/HTTP. More seriously, MFA will not be available to on-premises Exchange 2013 deployments until some time in the future. This aligns with Microsoft’s cloud-first strategy, but it is going to aggravate on-premises customers something fierce. In fairness, because you need the MFA infrastructure hosted in the Microsoft cloud to take advantage of this feature, I’m not sure there’s a feasible way to deliver SMS- or voice-based MFA for purely on-prem environments, and if you’re in a hybrid, then you’re good to go.

Microsoft hasn’t announced a specific timeframe for these updates (other than “second half calendar 2014”), and they didn’t say anything about Mac support, though I would imagine that the rumored v.next of Mac Office would provide this same functionality. The ability to use MFA across all the Office client apps will make it easier for end users, reducing the chance that they’ll depend solely on reusable passwords and thus reducing the net amount of evil in the world— a blessing to us all.

1 Comment

Filed under Office 365, UC&C

Script to download MEC 2014 presentations

Yay for code reuse! Tom Arbuthnot wrote a nifty script to download all the Lync Conference 2014 presentations, and since Microsoft used the same event management system for MEC 2014, I grabbed his script and tweaked it so that it will download the MEC 2014 session decks and videos. It only works if you are able to sign into the MyMEC site, as only attendees can download the presentations and videos at this time. I can’t guarantee that the script will pull all the sessions but it seems to be working so far— give it a try. (And remember, the many “Unplugged” sessions weren’t recorded so you won’t see any recordings or decks for them). If the script works, thank Tom; if it doesn’t, blame me.

Download the script

3 Comments

Filed under UC&C

The value of lagged copies for Exchange 2013

Let’s talk about… lagged copies.

For most Exchange administrators, the subject of lagged database copies falls somewhere between “the Kardashians’ shoe sizes” and “which of the 3 Stooges was the funniest” in terms of interest level. The concept is easy enough to understand: a lagged copy is merely a passive copy of a mailbox database where the log files are not immediately played back, as they are with ordinary passive copies. The period between the arrival of a log file and the time when it’s committed to the database is known as the lag interval. If you have a lag interval of 24 hours set to a database, a new log for that database generated at 3pm on April 4th won’t be played into the lagged copy until at least 3pm on April 5th (I say “at least” because the exact time of playback will depend on the copy queue length). The longer the lag interval, the more “distance” there is between the active copy of the mailbox database and the lagged copy.

Lagged copies are intended as a last-ditch “goalkeeper” safety mechanism in case of logical corruption. Physical corruption caused by a hardware failure will happen after Exchange has handed the data off to be written, so it won’t be replicated. Logical corruption introduced by components other than Exchange (say, an improperly configured file-level AV scanner) that directly write to the MDB or transaction log files wouldn’t be replicated in any event, so the real use case for the lagged copy is to give you a window in time during which logical corruption caused by Exchange or its clients hasn’t yet been replicated to the lagged copy. Obviously the size of this window depends on the length of the lag interval, and whether or not it is sufficient for you to a) notice that the active database has become corrupted b) play the accumulated logs forward into the lagged copy and c) activate the lagged copy depends on your environment.

The prevailing sentiment in the Exchange world has largely been “ I do backups already so lagged copies don’t give me anything.” When Exchange 2010 first introduced the notion of a lagged copy, Tony Redmond weighed in on it. Here’s what he said back then:

For now, I just can’t see how I could recommend the deployment of lagged database copies.

That seems like a reasonable stance, doesn’t it? At MEC this year, though, Microsoft came out swinging in defense of lagged copies. Why would they do that? Why would you even think of implementing lagged copies? It turns out that there are some excellent reasons that aren’t immediately apparent. (It may help to review some of the resiliency and HA improvements delivered in Exchange 2013; try this this excellent omnibus article by Microsoft’s Scott Schnoll if you want a refresher.) Here are some of the reasons why Microsoft has begun recommending the use of lagged copies more broadly.

1. Lagged copies are better in 2013

Exchange 2013 includes a number of improvements to the lagged copy mechanism. In particular, the new loose truncation feature introduced in SP1 means that you can prevent a lagged copy from taking up too much log space by adjusting the the amount of log space that the replay mechanism will use; when that limit is reached the logs will be played down to make room. Exchange 2013 (and SP1) also make a number of improvements to the Safety Net mechanism (discussed fully in Chapter 2 of the book), which can be used to play missing messages back into a lagged copy by retrieving them from the transport subsystem.

2. Lagged copies are continuously verified

When you back up a database, Exchange checks the page checksum of every page as it is backed up by computing the checksum and comparing it to the stored checksum; if that check fails, you get the dreaded JET_errReadVerifyFailure (-1018) error. However, just because you can successfully complete the backup doesn’t mean that you’ll be able to restore it when the time comes. By comparison, the Exchange log playback mechanism will log errors immediately when they are encountered during log playback. If you’re monitoring event logs on your servers, you’ll be notified as soon as this happens and you’ll know that your lagged copy is unusable now, not when you need to restore it. If you’re not monitoring your event logs, then lagged copies are the least of your problems.

3. Lagged copies give you more flexibility for recovery

When your active and passive copies of a database become unusable and you need to fall back to your lagged copy, you have several choices, as described in TechNet. You can easily play back every log that hasn’t yet been committed to the database, in the correct order, by using Move-ActiveMailboxDatabase. If you’d rather, you can play back the logs up to a certain point in time by removing the log files that you don’t want to play back. You can also play messages back directly from Safety Net into the lagged copy.

4. There’s no hardware penalty for keeping a lagged copy

Some administrators assume that you have to keep lagged copies of databases on a separate server. While this is certainly supported, you don’t have to have a “lag server” or anything like unto it. The normal practice in most designs has been to store lagged copies on other servers in the same DAG, but you don’t even have to do that. Microsoft recommends that you keep your mailbox databases no bigger than 2TB. Stuff your server with a JBOD array of the new 8TB disks (or, better yet, buy a Dell PowerVault MD1220) and you can easily put four databases on a single disk: the active copy of DB1, the primary passive copy of DB2, the secondary passive copy of DB3, and the lagged copy of DB4. This gives you an easy way to get the benefits of a 4-copy DAG while still using the full capacity of the disks you have: the additional IOPS load of the lagged copy will be low, so hosting it on a volume that already has active and passive copies of other databases is a reasonable approach (one, however, that you’ll want to test with jetstress).

It’s always been the case that the architecture Microsoft recommends when a new version of Windows or Exchange is released evolves over time as they, and we, get more experience with it in the real world. That’s clearly what has happened here; changes in the product, improvements in storage hardware, and a shift in the economic viability of conventional backups mean that lagged copies are now much more appropriate for use as a data protection mechanism than they were in the past. I expect to see them deployed more and more often as Exchange 2013 deployments continue and our collective knowledge of best practices for them improves.

3 Comments

Filed under UC&C

MEC 2014 wrapup

BLUF: it was a fantastic conference, far and away the best MEC I’ve attended. The quality of the speakers and technical presentations was very high, and the degree of community interaction and engagement was too.

I arrived in Austin Sunday afternoon and went immediately to dinner at County Line on the Lake, a justly famous Austin BBQ restaurant, to put on a “thank you” dinner for some of the folks who helped me with my book. Unfortunately, the conference staff had scheduled a speakers’ meeting at the same time, and a number of folks couldn’t attend due to flight delays or other last-minute intrusions. Next time I’ll poll invitees for their preferred time, and perhaps that will help. However, the dinner and company were both excellent, and I now have a copy of the book signed by all in attendance as a keepsake— a nice reversal of my usual pattern of signing books and giving them away.

Monday began with the keynote. If you follow me (or any number of other Exchange MVPs) on Twitter, you already know what I think: neither the content of the presentation nor its actual presentation was up to snuff when compared either to prior MEC events or other events such as Lync Conference. At breakfast Monday, Jason Sherry and I were excitedly told by an attendee that his Microsoft account rep insisted that he attend the keynote, and for the life of me I couldn’t figure out why until the tablet giveaway. That raised the energy level quite a bit! I think that for the next MEC, Julia White should be handed the gavel and left to run the keynote as she sees fit; I can guarantee that would result in a more lively and informative event.  (For another time: a review of the Venue 8 Pro, which I like a great deal based on my use of it so far). One area where the keynote excelled, though, was in its use of humor. The video vignette featuring Greg Taylor and David Espinoza was one of the funniest such I’ve ever seen, and all of the other bits were good as well— check them out here. The keynote also featured a few good-natured pokes at the community, such as this:

Ripped

For the record, although I’ve been lifting diligently, I am not (yet) built like the guy who’s wearing my face on screen… but there’s hope.

I took detailed notes on each of the sessions I attended, so I’ll be posting about the individual sessions over the next few days. It’s fair to say that I learned several valuable things at each session, which is sort of the point behind MEC. I found that the quality of the “unplugged” sessions I attended varied a bit between sessions; the worst was merely OK, while the best (probably the one on Managed Availability) was extremely informative. It’s interesting that Tony and I seemed to choose very few of the same sessions, so his write-ups and mine will largely complement each other. My Monday schedule started with Kamal Janardhan’s session on compliance and information protection. Let me start by saying that Kamal is one of my favorite Microsoft people ever. She is unfailingly cheerful, and she places a high value on transparency and openness. When she asks for feedback on product features or futures, it’s clear that she is sincerely seeking honest feedback, not just saying it pro forma. Her session was great; from there, I did my two back-to-back sessions, both of which went smoothly. I was a little surprised to see a nearly-full room (I think there were around 150 people) for my UM session, and even more surprised to see that nearly everyone in the room had already deployed UM on either Exchange 2010 or 2013. That’s a significant change from the percentage of attendees deploying UM at MEC 2012. I then went to the excellent “Unplugged” session on “Exchange Top Issues”, presented by the supportability team and moderated by Tony. After the show closed for the day, I was fortunate to be able to attend the dinner thrown by ENow Software for MVPs/MCMs and some of their key customers. Jay and Jess Gundotra, as always, were exceptional hosts, the meal (at III Forks) was excellent, and the company and conversation were delightful. Sadly I had to go join a work conference call right after dinner, so I missed the attendee party.

Tuesday started with a huge surprise. On my way to the “Exchange Online Migrations Technical Deep Dive” session (which was good but not great; it wasn’t as deep as I expected), I noticed the picture below flashing on the hallway screens. Given that it was April Fool’s Day, I wasn’t surprised to see the event planners playing jokes on attendees, I just wasn’t expecting to be featured as part of their plans. Sadly, although I’m happy to talk to people about migrating to Office 365, the FAA insists that I do it on the ground and not in the air. For lunch, I had the good fortune to join a big group of other Dell folks (including brand-new MVP Andrew Higginbotham, MCM Todd Hawkins, Michael Przytula, and a number of people from Dell Software I’d not previously met) at Iron Works BBQ. The food and company were both wonderful, and they were followed by a full afternoon of excellent sessions. The highlight of my sessions on Tuesday was probably Charlie Chung’s session on Managed Availability, which was billed as a 300-level session but was more like a 1300-level. I will definitely have to watch the recording a few times to make sure I didn’t miss any of the nuances.

Surprise!

This is why I need my commercial pilot’s license— so I can conduct airborne sessions at the next MEC.

Tony has already written at length about the “Exchange Oscars” dinner we had Tuesday night at Moonshine. I was surprised and humbled to be selected to receive the “Hall of Fame” award for sustained contributions to the Exchange community; I feel like there are many other MVPs, current and past, who deserve the award at least as much, if not more. It was great to be among so many friends spanning my more than 15 years working with Exchange; the product group turned out en masse and the conversation, fellowship, and celebration was the high point of the entire conference for me. I want to call out Shawn McGrath, who received the “Best Tool” award for the Exchange Remote Connectivity Analyzer, which became TestExchangeConnectivity.com. Shawn took a good idea and relentlessly drove it from conception to implementation, and the whole world of Exchange admins has benefited from his effort.

Wednesday started with the best “Unplugged” session I attended: it covered Managed Availability and, unlike the other sessions I went to, featured a panel made mostly of engineers from the development team. There were a lot of deep technical questions and a number of pointed roadmap discussions (not all of which were at my instigation). The most surprising session I attended, I think, was the session on updates to Outlook authentication— turns out that true single sign-on (SSO) is coming to all the Office 2013 client applications, and fairly soon, at least for Office 365 customers. More on that in my detailed session write-ups. The MVPs were also invited to a special private session with Perry Clarke. I can’t discuss most of what we talked about, but I can say that I learned about the CAP theorem (which hadn’t even been invented when I got my computer science degree, sigh), and that Perry recognizes the leadership role Exchange engineering has played in bringing Microsoft’s server products to high scale. Fun stuff!

Then I flew home: my original flight was delayed so they put me on one leaving an hour earlier. The best part of the return trip might have been flying on one of American’s new A319s to Huntsville. These planes are a huge improvement over the nasty old MD80s that AA used to fly DFW-HSV, and they’re nicer than DL’s ex-AirTran 717s to boot. So AA is still in contention for my westbound travel business.

A word about the Hilton Austin Downtown, the closest hotel to the conference center: their newly refurbished rooms include a number of extremely practical touches. There’s a built-in nightlight in the bathroom light switch, and each bedside table features its own 3-outlet power strip plus a USB port, and the work desk has its own USB charging ports as well. Charging my phone, Kindle, Venue 8 Pro, and backup battery was much simpler thanks to the plethora of outlets. The staff was unfailingly friendly and helpful too, which is always welcome. However, the surrounding area seemed to have more than its share of sirens and other loud noises; next time I might pick a hotel a little farther away.

I’ll close by saying how much I enjoyed seeing old friends and making new ones at this conference. I don’t have room (or a good enough memory) to make a comprehensive list, but to everyone who took the time to say hello in the hall, ask good questions in a session, wave at me across the expo floor, or pass the rolls at dinner— thank you.

Now to get ready for TechEd and Exchange Connections…

Leave a comment

Filed under UC&C

Getting ready for MEC 2014

Wow, it’s been nearly a month since my last post here. In general I am not a believer in posting stuff on a regular schedule, preferring instead to wait until I have something to say. All of my “saying” lately has been on behalf of my employer though. I have barely even had time to fly. For another time: a detailed discussion of the ins and outs of shopping for an airplane. For now, though, I am making my final preparations to attend this year’s Microsoft Exchange Conference (MEC) in Austin! My suitcase is packed, all my devices are charged, my slides are done, and I am prepared to overindulge in knowledge sharing, BBQ eating, and socializing.

It is interesting to see the difference in flavor between Microsoft’s major enterprise-focused conferences. This year was my first trip to Lync Conference, which I would summarize as being a pretty even split between deeply technical sessions and marketing focused around the business and customer value of “universal communications”. In reviewing the session attendance and rating numbers, it was no surprise that the most-attended sessions and the highest-rated sessions tended to be 400-level technical sessions such as Brian Ricks’ excellent deep-dive on Lync client sign-in behavior. While I’ve never been to a SharePoint Conference, from what my fellow MVPs say about it, there was a great deal of effort expended by Microsoft on highlighting the social features of the SharePoint ecosystem, with a heavy focus on customization and somewhat less attention directed at SharePoint Online and Office 365. (Oh, and YAMMER YAMMER YAMMER YAMMER YAMMER.) Judging from reactions in social media, this focus was well-received but inevitably less technical given the newness of the technology.

That brings us to the 2014 edition of MEC. The event planners have done something unique by loading the schedule with “Unplugged” panel discussions, moderated by MVP and MCM/MCSM experts and consisting of Microsoft and industry experts in particular technologies. These panels provide an unparalleled opportunity to get, and give, very candid feedback around individual parts of Exchange and I plan on attending as many of them as I can. This is in no way meant to slight the many other excellent sessions and speakers that will be there. I’d planned to summarize specific sessions that I thought might be noteworthy, but Tony published an excellent post this morning that far outdoes what I had in mind, breaking down sessions by topic area and projected attendance. Give it a read.

I’m doing two sessions on Monday: Exchange Unified Messaging Deep Dive at 245p and Exchange ActiveSync: Management Challenges and Best Practices at 1145a. The latter is a vendor session with the folks from BoxTone, during which attendees both get lunch (yay) and the opportunity to see BoxTone’s products in action. They’re also doing a really interesting EAS health check, during which you provide CAS logs and they run them through a static analysis tool that, I can almost guarantee, will tell you things you didn’t know about your EAS environment. Drop by and say hello!

Leave a comment

Filed under UC&C

Office 365 Personal Archives limited to 100GB

There’s a bit of misinformation, or lack of information, floating around about the use of Office 365 Personal Archives. This feature, which is included in the higher-end Office 365 service plans (including E3/E4 and the corresponding A3/A4 plans for academic organizations), is often cited as one of the major justifications for moving to Office 365. It’s attractive because of the potential savings from greatly reducing PST file use and eliminating (or at least sharply reducing) the use of on-premises archiving systems such as Enterprise Vault.

Some Microsoft folks have been spreading the good news that archives are unlimited (samples here and here), and so have many consultants, partners, and vendors– including me. In fact, I had a conversation with a large customer last week in which they expressed positive glee about being able to get their data out of on-prem archives and into the cloud.

The only problem? Saying the archives are unlimited isn’t quiiiiite true.

If you read the service description for Exchange Online (which we all should be doing regularly anyway, as it changes from time to time), you’ll see this:

Clip from Nov 2013 O365 service description

Clip from Nov 2013 O365 service description

See that little “3”? Here’s its text:

Each subscriber receives 50 GB of storage in the primary mailbox, plus unlimited storage in the archive mailbox. A default quota of 100 GB is set on the archive mailbox, which will generally accommodate reasonable use, including the import of one user’s historical email. In the unlikely event that a user reaches this quota, a call to Office 365 support is required. Administrators can’t increase or decrease this quota.

So as an official matter, there is no size limit. As a practical matter, the archive is soft-limited to 100GB, and if you want to store more data than that, you’ll have to call Microsoft support to ask for a quota increase. My current understanding is that 170GB is the real limit, as that is the maximum size to which the quota can currently be increased. I don’t know if Microsoft has stated this publicly anywhere yet but it’s certainly not in the service descriptions. That limit leads me to wonder what the maximum functional size of an Office 365 mailbox is– that is, if Microsoft didn’t have the existing 100GB quota limit in place, how big a mailbox could they comfortably support? (Note that this is not the same as asking what size mailbox Outlook can comfortably support, and I bet those two numbers wouldn’t match anyway.) I suppose that in future service updates we’ll find out, given that Microsoft is continuing to shovel mailbox space at users as part of its efforts to compete with Google.

Is this limit a big deal? Not really; the number of Office 365 customers who will need more than 100GB of archive space for individual user mailboxes is likely to be very small. The difference between “unlimited” and “so large that you’ll never encounter the limit” is primarily one of semantics. However, there’s always a danger that customers will react badly to poor semantics, perhaps because they believe that what they get isn’t what they were promised. While I would like to see more precision in the service descriptions, it’s probably more useful to focus on making sure that customers (especially those who are heavy users of on-premises archives or PST files) know that there’s currently a 100GB quota, which is why I wrote this post.

For another time: a discussion of how hard, or easy, it is to get large volumes of archive data into Office 365 in the first place. That’s one of the many topics I expect to see explored in great depth at MEC 2014, where we’ll get the Exchange team’s perspective, and then again at Exchange Connections 2014, where I suspect we’ll get a more nuanced view.

5 Comments

Filed under Office 365, UC&C

Microsoft, encryption, and Office 365

So the gloves are starting to come off: Microsoft general counsel Brad Smith wrote a long blog post this morning discussing how Microsoft plans to protect its customers’ data from unlawful interception by “unauthorized government access”. He never specifically mentions NSA, GCHQ, et al, but clearly the Five Eyes partners are who he’s talking about. Many other news outlets have dissected Smith’s post in detail, so I wanted to focus on a couple of lesser-known aspects.

First is that Microsoft is promising to use perfect forward secrecy (PFS) when it encrypts communications links. Most link-encryption protocols, including IPsec and SSL, use a key exchange algorithm known as Diffie-Hellman to allow  the two endpoints can agree on a temporary session key by using their longer-term private/public key pairs. The session key is usually  be renegotiated for each conversation. If Eve the eavesdropper or Mallet the man-in-the-middle intercept the communications, they may be able to decrypt it if they can guess or obtain the session key. Without PFS, an attacker who can intercept and record a communication stream now and can guess or obtain the private key of either endpoint can decrypt the stream. Think of this like finding a message in a bottle written in an unknown language, then next year seeing Rosetta Stone begin to offer a course in the language. PFS protects an encrypted communication stream now from future attack by changing the way the session keys are generated and shared. Twitter, Google, and a number of other cloud companies have already deployed PFS (Google, in fact, started in 2011) so it is great to see Microsoft joining in this trend. (A topic for another day: under what conditions can on-premises Exchange and Lync use PFS? Paging Mark Smith…)

Second is that Microsoft is acknowledging that they use data-at-rest encryption, and will be using it more often. Probably more than any other vendor, Microsoft is responsible for democratizing disk encryption by including BitLocker in Windows Vista and its successors, then steadily improving it. (Yes, I know that TrueCrypt and PGP predated BitLocker, but their installed bases are tiny by comparison.) Back in 2011 I wrote about some of the tradeoffs in using BitLocker with Exchange, and I suspected that Microsoft was using BitLocker in their Office 365 data centers, a suspicion that was confirmed recently during a presentation by some of the Office 365 engineering team and, now, by Smith’s post. Having said that, data-at-rest encryption isn’t that wonderful in the context of Office 365 because the risk of an attacker (or even an insider) stealing data by stealing/copying physical disks from an Office 365 data center is already low. There are many layers of physical and procedural security that help keep this risk low, so encrypting the stored data on disk is of relatively low value compared to encrypting the links over which that data travels.

The third aspect is actually something that’s missing from Smith’s post, expressed as one word: Skype. Outlook.com, Office 365, SkyDrive, and Azure are all mentioned specifically as targets for improved encryption, but nothing about Skype? That seems like a telling omission, especially given Microsoft’s lack of prior transparency about interception of Skype communications. Given the PR benefits that the company undoubtedly expects from announcing how they’re going to strengthen security, the fact that Smith was silent on Skype indicates, at least to suspicious folks like me, that for  now they aren’t making any changes. Perhaps the newly-announced transparency centers will provide neutral third parties an opportunity to inspect the Skype source code to verify its integrity.

Finally, keep in mind that nothing discussed in Smith’s post addresses targeted operations where the attacker (or government agency, take your pick) mounts man-in-the-middle attacks (QUANTUM/FOXACID)  or infiltrates malware onto a specific target’s computer. That’s not necessarily a problem that Microsoft can solve on its own.

Leave a comment

Filed under Office 365, UC&C