Tag Archives: Exchange ActiveSync

Getting ready for MEC 2014

Wow, it’s been nearly a month since my last post here. In general I am not a believer in posting stuff on a regular schedule, preferring instead to wait until I have something to say. All of my “saying” lately has been on behalf of my employer though. I have barely even had time to fly. For another time: a detailed discussion of the ins and outs of shopping for an airplane. For now, though, I am making my final preparations to attend this year’s Microsoft Exchange Conference (MEC) in Austin! My suitcase is packed, all my devices are charged, my slides are done, and I am prepared to overindulge in knowledge sharing, BBQ eating, and socializing.

It is interesting to see the difference in flavor between Microsoft’s major enterprise-focused conferences. This year was my first trip to Lync Conference, which I would summarize as being a pretty even split between deeply technical sessions and marketing focused around the business and customer value of “universal communications”. In reviewing the session attendance and rating numbers, it was no surprise that the most-attended sessions and the highest-rated sessions tended to be 400-level technical sessions such as Brian Ricks’ excellent deep-dive on Lync client sign-in behavior. While I’ve never been to a SharePoint Conference, from what my fellow MVPs say about it, there was a great deal of effort expended by Microsoft on highlighting the social features of the SharePoint ecosystem, with a heavy focus on customization and somewhat less attention directed at SharePoint Online and Office 365. (Oh, and YAMMER YAMMER YAMMER YAMMER YAMMER.) Judging from reactions in social media, this focus was well-received but inevitably less technical given the newness of the technology.

That brings us to the 2014 edition of MEC. The event planners have done something unique by loading the schedule with “Unplugged” panel discussions, moderated by MVP and MCM/MCSM experts and consisting of Microsoft and industry experts in particular technologies. These panels provide an unparalleled opportunity to get, and give, very candid feedback around individual parts of Exchange and I plan on attending as many of them as I can. This is in no way meant to slight the many other excellent sessions and speakers that will be there. I’d planned to summarize specific sessions that I thought might be noteworthy, but Tony published an excellent post this morning that far outdoes what I had in mind, breaking down sessions by topic area and projected attendance. Give it a read.

I’m doing two sessions on Monday: Exchange Unified Messaging Deep Dive at 245p and Exchange ActiveSync: Management Challenges and Best Practices at 1145a. The latter is a vendor session with the folks from BoxTone, during which attendees both get lunch (yay) and the opportunity to see BoxTone’s products in action. They’re also doing a really interesting EAS health check, during which you provide CAS logs and they run them through a static analysis tool that, I can almost guarantee, will tell you things you didn’t know about your EAS environment. Drop by and say hello!

Leave a comment

Filed under UC&C

Android 4.4/KitKat Exchange ActiveSync problems; fixed in 4.4.1?

Apple’s iOS has gotten a deservedly bad reputation for its Exchange ActiveSync implementation. But, to their credit, things seem to be fairly stable with the latest iOS 7.0.4 update. On the other hand, Google seems to have largely gotten a free pass on the quality of its EAS implementation; in fact, for quite some time Android didn’t include EAS functionality, although some individual vendors did. The latest release, 4.4 (or “KitKat”, a particularly nasty type of candy, at least in the US), includes EAS as part of the core OS, but it appears to have some bugs, including at least one that I am still trying to get a good understanding of.

First, there appears to be a problem with client certificate authentication, i.e. it doesn’t work. To Google’s credit, they maintain a public bug-tracking system where everyone can see the bug report and status, at least of this particular bug. Imagine a world where Microsoft and Apple were similarly transparent about bugs in their major products… OK, back to reality; Google of course doesn’t do the same for their proprietary products, just for open-source efforts such as Android. On the other hand, this kind of public reporting lets people show their ignorance; check out this thread, where a couple of engineers for a competing product show that they haven’t read the protocol specs in detail (hint: see this discussion of WindowSize to spot the flaw in their argument).

Anyway, Tony pointed out this particular problem to the Exchange community just before Thanksgiving. Recently I was contacted by a customer who was seeing another widespread KitKat issue: devices persistently pounding the server with EAS Sync commands, over and over and over and… well, you get the idea. Although I haven’t seen a clear cause identified, Google claims to have fixed this problem in the 4.4.1 update (see the reply by Ersher in page 24 of this thread), so the question becomes whether all the users claiming to be affected by this bug have upgraded.

Actually, the question becomes at what point Exchange administrators begin to proactively block new mobile device OS releases! While I’m not quite ready to declare a fatwa on all new device releases, it is beginning to look at though organizations with diverse BYOD populations might be well served to establish some kind of criteria for staging support of new releases. Apple, Microsoft, and Google all offer developer access to new OS releases, often months in advance, so one possibility is to establish a pool of test devices for new OS releases— something which many sites already do with new desktop OS releases. The logistics of working out such a program might be challenging, but I think the effort might be well worth it if it prevents unpleasant surprises caused by device-side EAS misbehavior.

There’s another, perhaps less palatable, option on the horizon. Now that we have OWA for Devices (known colloquially as Mobile OWA, or MOWA, within Microsoft), if you were so inclined you could block all iOS device access and require your users to use MOWA. Since there’s no MOWA version for Android yet (and there may never be; Microsoft hasn’t given any hints), this wouldn’t be a comprehensive solution, and it would likely aggravate users to a high degree… but as improvements in MOWA performance and capability roll out, it might become a more viable option.

(side note: speaking of aggravation, it’s amazing how aggravated Google’s customers get when they don’t receive an official answer from Google in the time frame they expect. At least Google gives official answers in their support forums, something you are unlikely to see happen much in the support fora offered for iOS and Windows Phone!)

One thing I’d like to see emerge is something akin to collaborative spam filtering— when I report a message as spam to my filtering service, that message is filtered for other subscribers too. It seems like BoxTone or another company might be able to offer a subscription service to customers that gives them early alerts to wide-scale problems reported by other customers, such as regional outages in a carrier network or a pattern of sync misbehavior for a specific device family. I know I’d be happy to pay money for a service that would give me early warning of apparent problems with new device software releases— what about you?


Filed under UC&C

iOS 7 Exchange ActiveSync problems revisited

Back in September I posted an article about a problem that occurred when synchronizing iOS 7 devices against Exchange 2010 SP2. The wheels of justice grind slowly, but Microsoft has released a KB article and accompanying hotfix that describe the symptoms precisely.

I also got an odd report from a large enterprise customer; they had several hundred iOS 7.0.2 devices, all on Verizon in one specific region, that were having synchronization problems. The issue here turned out to be a network configuration issue on Verizon’s network that required some action from them to fix.

Now you’re probably starting to see the value in solutions like those from BoxTone





Filed under UC&C

Odd iOS 7.0x Exchange ActiveSync problem

from the oops-they-may-have-done-it-again department…

I just got an e-mail from a former coworker reporting a problem with synchronizing some, but not all, iOS 7.0.x devices with Exchange 2010 SP2. There are four users (Alex, Eric, James, and Peter, let’s say) with shiny new iPhone 5s devices. Two of them get the same error when syncing: the Provision verb is returning a status of 110 and throwing an exception from Microsoft.Exchange.Security.Compliance.MessageDigestForNonCryptographicPurposes.HashCore. This seems to point to a problem with crypto negotiation with the devices, but I haven’t been able to look at a trace of the conversation between the device and the server to check.

James’ device works fine. Alex’s device works fine. Peter’s device does not work, either with his own account or Alex’s. Eric’s device does not work with his account; no other accounts have been tested. This seems to indicate that the problem is not (necessarily) with the account. Peter and Eric have both wiped their devices, deleted their Exchange accounts, rebooted the devices, and done all the other stuff you might try when faced with this problem, but to no avail.

This Apple support forum thread seems to indicate that a few others who have the same problem, but none of the recommended fixes have worked for Alex or Peter. My working theory is that this is due to an unwanted interaction of some kind between Exchange 2010 SP2 and iOS 7.x, but I can’t prove that yet. As far as I can tell, Exchange 2013 CU2 doesn’t have the same problem.

I’m posting this in hope that it might come to the attention of anyone else who’s having a similar problem so I can get a sense of its scope and nature.

More news when there is news…


Filed under UC&C

Microsoft releases new OWA apps for iPhone, iPad

Well, this is gonna be fun: Microsoft just released a new native mail/calendar/contacts app (which they’re calling “OWA”) for the iPhone and iPad. A few quick notes:

  • It is only supported with Office 365 wave 15 mailboxes. It may, or may not, work against on-premises Exchange 2013 mailboxes. (Update 130716 1509: Microsoft has in fact committed to on-prem support, but haven’t said when.)
  • It is a native app, with separate versions for iPhone (iPhone 4 and later) and iPad (iPad 2 and later). Both versions require iOS 6. Making a native app rather than just a bound web control means that the app can include some other cool features.. including gesture controls and voice control.
  • It supports Information Rights Management (and, yay, reading signed S/MIME messages). Oh, and it supports delegate access too. Oh, and online Personal Archives… and shared calendars, too!
  • No support for public folders, I’m afraid.
  • It uses Exchange Web Services, not EAS; to the Exchange CAS and mailbox roles, OWA on a device looks almost exactly like OWA in a browser.
  • It has full offline functionality, powered by a local sqlite database.
  • When you request a remote wipe, the wipe request removes the app and all the data from its device but leaves the rest of the device untouched. This is a huge feature.

Of course, I’ll have full coverage of the app (and how to administer and manage it) in the clients chapter of Exchange 2013 Inside Out: Clients, Connectivity, and Unified Messaging. Until then, grab the client and play with it! I was able to download, install, and use it on my iPad3 without any trouble, but the App Store refused to allow me to download it to an iPhone 4. Stay tuned…


1 Comment

Filed under UC&C

Excessive transaction log growth with iOS 6.1 devices

Well, it appears that Apple has done it again: reports are starting to surface of runaway transaction log growth when mobile devices running iOS 6.1 synchronize with Exchange Server. Tony has a good synopsis here.

Those of you who have been administering Exchange for a while may think this sounds familiar– that’s because there was a very similar problem with Microsoft Entourage back in the day, as detailed by Jeremy Kelly here. Remarkably, a couple of years later, we got the same bug in a slightly different guise, as described in KB 935848. In both cases, the problem was that the client was too stupid to detect certain types of failures, so it would keep retrying the failed operation, which would keep failing. This endless loop quickly resulted in large volumes of transaction log files on the Exchange server.

Luckily, Exchange 2010 and 2013 include throttling to prevent misbehaving clients from using up an excessive share of resources. However, the throttling controls available regulate EAS based on the amount of time user requests take, the number of concurrent connections, or the number of device partnerships. None of these parameters are useful in preventing the iOS 6.1-related problem; it’s not that the individual requests take up an excessive amount of time, it’s that there are so many requests that they generate an excessive log volume. (This video may provide a useful explanation for the phenomenon.)

Exchange 2013 includes the ability to specifically block misbehaving Exchange ActiveSync devices based on “suspicious” behavior. I will have a lot more to say about that in the near future, although that spiffy feature doesn’t help anyone now suffering the problem. For now, all we can do is the following:

  • Block iOS 6.1 devices using an Exchange ActiveSync device access rule
  • Discourage your users from upgrading, although I expect this to be an ineffective strategy
  • If you have a support relationship with Apple, report this problem to them. If you’re a developer, file a RADAR issue. If you have enterprise technical support with Apple, use it. I’ve seen reports that the ordinary consumer-level technical support (i.e. the $49 pay-per-incident support, as well as AppleCare) doesn’t have any way to report this particular problem in an actionable way.

Thoughts for another time: the rapid adoption rate of iOS devices has many benefits for users, including largely avoiding the fragmentation problems that plague Android with issues (like this “smishing” fix that virtually no one has). However, when Apple ships a buggy update, which is common, that rapid adoption multiplies the pain of the bug.

Update 1535 CST 8 Feb: Ina Fried at AllThingsD is reporting that Vodafone is telling iPhone 4S users not to upgrade to iOS 6.1.

1 Comment

Filed under UC&C

Microsoft wins UK case vs Motorola Mobility/Google

Earlier this year I had the unique (to me) opportunity to serve as a technical expert witness in a court case in the UK. Tony’s already written about the case but I wanted to add my perspective.

I was contacted by Bird and Bird to see if I might be willing and able to act as a technical expert in a court case; that’s all they said at first. The nature of the questions they were asking soon clued me in that the case involved Exchange ActiveSync and multiple-point-of-presence (MPOP) support for presence publishing– two completely separate technologies which Motorola/Google had lumped together in this case.  .

My role was to perform a wire-level analysis of the protocols in question: EAS, SIP/SIMPLE as implemented in Lync, and the Windows Live Messenger protocol. For each of these protocols, my job was to produce an accurate, annotated packet capture showing exactly what happened when multiple devices synchronized with the same account, and when the status on one device changed.

This isn’t what most people think of when they think of expert testimony; in courtroom dramas and books, it always seems like the expert is being asked to provide an opinion, or being cross-examined on the validity of their opinion. No one wanted my opinion in this case (which is perfectly normal), just for me to to accurately and impartially report what was happening on the wire.

This proved to be incredibly interesting from a technical standpoint. Like most administrators, it had never really occurred to me to look into the depths of the EAS protocol itself to see exactly what bits were being passed on the wire. After a great deal of study of the ActiveSync protocol documentation and many a late night slaving away over Wireshark and Network Monitor captures, I’d produced a report that showed the actual network traffic that passed between client(s) and server for a variety of test scenarios, along with an explanation of the contents of the packets and how they related to user action on the device.

Along the way, I gained a new appreciation for the economy of design of these protocols– it’s surprising how efficient they are when you look at them at such a low level. (And a shout out to Eric Lawrence for his incredibly useful Fiddler tool, which made it much easier for me to get the required data into a usable format.) I found a few bugs in Wireshark, learned more than I wanted to about SSL provisioning on Windows Phone 7.5 devices, and generally had a grand time. I particularly enjoyed working with the attorneys at Bird and Bird, who were quite sharp and had lovely accents to boot. (I’m not sure they enjoyed my accent quite as much, but oh well.)

When I finished my report, I submitted it to Bird and Bird and that was the last I heard of the case until today, when Mr. Justice Arnold issued his ruling. It was submitted as part of Microsoft’s justification explaining why their implementations did not infringe on Motorola’s patent; the purpose of having an annotated set of packet captures was to clearly illustrate the differences between the claimed innovations in the patent and Microsoft’s implementation to show why Microsoft wasn’t infringing.

Florian Mueller has a good summary of the case that highlights something I didn’t know: the patent at issue is the only one on which an Android manufacturer is currently enforcing an injunction against Apple. I am no patent attorney, but it would seem that Apple might have grounds to have this injunction lifted. It will be interesting to see what happens in the related German court cases that Muller cites, but it’s hard for this layman to see any other likely result besides a Microsoft win… but we will see.

Leave a comment

Filed under General Tech Stuff, UC&C

Apple releases fix for EAS calendar hijacking bug

Good news: Apple just released iOS 6.0.1, the release notes for which say that the update fixes “a bug affecting Exchange meetings.” More details here.

Leave a comment

Filed under UC&C

Postel’s Law, Exchange ActiveSync, and iOS 6

Over the last week the Exchange community has learned a bit more about problems with Apple’s Exchange ActiveSync implementation in iOS 6; Microsoft has released a KB article outlining the problem and suggesting some workarounds, and Tony Redmond this morning pointed to a TUAW article that I hadn’t previously seen which asserts that the hijacking problem is a known issue with previous versions of iOS.

Tony’s article is titled “The emerging need for more supervision over ActiveSync implementations.” It’s certainly hard to disagree with that basic premise. Just over 18 months ago, Microsoft launched an Exchange ActiveSync logo certification program with the goal of inducing third parties who sell EAS-compatible devices or software to verify that they properly implement the client side of EAS. A quick check of this page maintained by Microsoft’s legal department shows just over four dozen EAS licensees, but only 3 vendors are listed on the EAS logo page itself. There are several possible explanations, ranging from lack of vendor interest in getting certified to a Microsoft failure to update the contents of the page.

Whatever the reason, I think Tony is right that Microsoft needs to be more proactive in requiring their EAS licensees to perform more robust testing on their clients.

Furthermore, I’m starting to think that the certification and testing program isn’t sufficient in and of itself, which brings me to Postel’s Law.

Some years ago, I worked for a company that made e-mail encryption software. This was during a time when even basic SMTP interoperability between different vendors’ systems was not assured– the Electronic Mail Association had biannual “plugfests” where vendors such as Lotus, Microsoft, and Netscape got together to ensure that their products could exchange SMTP mail properly. Whenever we ran into an interop problem, my boss would cite Postel’s Law:

Be conservative in what you do, be liberal in what you accept from others.

It seems pretty clear that Apple’s iOS EAS implementation has bugs– long-standing ones, if the TUAW article is right– and that Apple has a responsibility to fix them. I am coming to the opinion, though, that the server-side EAS implementation is too lenient about what it accepts. For years Exchange administrators had to suffer through a dance that went something like this:

  1. A new version of Outlook would ship with a bug that would occasionally corrupt calendar or message items in some interesting way.
  2. The corrupted items would cause the client to crash; in more severe cases, they could also crash the store process or cause backups, mailbox exports, etc. to fail.
  3. The Exchange team would release a hotfix to keep that particular flavor of badly formed item from causing such a problem.
  4. The Outlook team would release a hotfix that would keep that version of Outlook from emitting that particular flavor of corruption.
  5. In the next release of Exchange, the team would add business logic to reject items with the specific flaws generated by Outlook.
  6. GOTO 1

Over time, the Exchange business logic got better, as did Outlook’s track record of not creating bad items in the first place. This approach has served Exchange administrators and users pretty well, and it’s clear that the lessons learned from this painful process have been applied in Exchange Web Services. For example, EWS will reject attempts to create calendar items with end dates that come before start dates– something that Outlook used to occasionally do out of sheer perversity.

However, it looks as though, at least in this specific case, EAS doesn’t have business logic in place to catch the modifications that lead to the hijacking behavior. That’s something that Microsoft can fix in three steps.

  • First, they can update the EAS spec so that it clearly defines which fields of each data item are to be considered read-only. This gives implementers a fair shot to understand where they may need to change their clients.
  • Second, they can update Exchange itself so that it rejects client requests that violate the updated spec.
  • Third, they can vigorously “encourage” their major ISV partners to test more aggressively against the updated spec. I have to wonder, for example, how much, if any, testing Apple has done against Exchange 2013…

I’ve been putting the blame on Apple for this problem, and while I don’t think that doing so is unfair or inaccurate, I also believe that the Exchange team can do a better job on the server side of blocking bad client requests, and I hope they’re madly planning how to do so in Exchange 2013 SP1.

Leave a comment

Filed under UC&C

More on bugs in iOS6 Exchange ActiveSync

As I hoped they would (but couldn’t mention in my previous post), Microsoft has weighed in with a KB article on the iOS 6 “hijacked meeting” bug. The article ID is 2768774, and it makes a couple of cogent points: first, that Microsoft is aware of, and is actively investigating, the bug. Second, that “Microsoft cannot mitigate this issue”; they recommend contacting Apple. The interesting part to me was the section on workarounds. Microsoft’s suggestions include:

  • telling users not to make calendar changes using the iOS app
  • recommending that users not update to iOS6 (as if! iOS device users are legendarily quick to update to new versions)
  • blocking iOS 6 devices using Exchange ActiveSync allow/block/quarantine
  • blocking users who have delegates, or who are granted delegate access, from using EAS
  • using POP/IMAP instead of EAS (this made me laugh out loud– really!)

I have been testing Savvy Software’s Agenda app recently and like it quite a bit. However, it, like almost all the other calendar apps of which I’m aware,  it uses the underlying iOS calendar data store and sync mechanism, so I’d expect it to fall victim to the same bug. NitroDesk has ported its Touchdown app to iOS (though its App Store reviews are pretty poor), and Emtrace has the MoxierMail EAS client (which I still need to review.) So pick your poison: none of Microsoft’s suggested workarounds are great, but their hands are tied. It’s pretty clear this is an Apple problem and it will be incumbent on Apple to fix it… which I am sure they will, eventually. 




Filed under UC&C

iOS 6 and Exchange ActiveSync misbehavior

Brace yourself for a surprise: there’s at least one major bug in Apple’s Exchange ActiveSync client in iOS 6.

cue shocked silence…

This is not surprising, of course; EAS is a fairly complex protocol and Apple has displayed a somewhat cavalier attitude towards verifying that their EAS clients behave properly. If you don’t believe me, all you have to do is look at this list of known EAS issues with third-party devices and see how many of them involve iOS.

Anyway, the latest widely reported  issue is that meetings sometimes appear to be hijacked– Alice will send out a meeting invitation to Bob and Carole, but somehow Carole will end up as the meeting organizer, thus gaining the ability to cancel or change meetings.

The hijacking bug isn’t the only one; users have reported a few other iOS 6 EAS issues, including its apparent failure to handle the case where the user’s primary SMTP address is different from the user name (e.g. an SMTP address of paul@robichaux.net coupled with a UPN of paul.robichaux@robichaux.net, for example), but these other issues all have easy workarounds; meeting hijacking is the most pestiferous.

The usual pattern for these bugs is fairly predictable:

  1. Apple (or another EAS licensee) releases an update.
  2. People start complaining in various fora that some EAS-related functionality is broken. However, these reports are only rarely made directly to either Microsoft or the vendor.
  3. A critical mass of reports accumulates and begins to draw attention. This is often accelerated by the opening of support cases with either Microsoft or the ISV.
  4. The source of the problem is identified, a resolution is developed, and everything is fixed.
  5. GOTO 1

Now, stop laughing. That really is what usually happens. Note that Microsoft’s hands are somewhat tied during this process. Until they get feedback from customers that something is broken, they can’t very well investigate it. I imagine that it is very frustrating for the EAS team to see people blaming Exchange for what end up being bugs in the EAS client implementation. But I digress.

Tony points out a few nuances of how the process works, including suggestions for figuring out which devices are actually in use. (Note that one improvement in iOS 6 is that it reports a value for the DeviceOS property returned by Get-ActiveSyncDeviceStatistics; older versions just left that field blank.) Leaving that aside, though, it’s interesting to consider what’s known about the calendar hijacking bug. The best explanation I’ve seen, ironically, comes from the Z-Push development team. (Z-Push, you may recall, is an open-source EAS implementation that has nothing to do whatsoever with Exchange.) However, they are apparently first to market with a public explanation of the problem with iOS 6 that causes meeting hijacking. I won’t repeat it here; it’s worth reading the original. The root of the problem appears to be that iOS 6 emits meetings with zero attendees, and that Exchange accepts these as valid. I’m not sure whether Exchange’s acceptance is a desired behavior or not but I’m pretty sure that the device should never be emitting a zero-attendee meeting. It’s possible that there are cases where this is not true, which is why the Z-Push folks are holding their patch in QA for now instead of pushing it into the main tree of their product.

The tricky issue here, of course, is how to get the problem fixed for those of us who aren’t using Z-Push. Microsoft could conceivably make a change to Exchange’s business logic for calendar items, preventing ill-formed meeting items from being propagated. Apple could likewise fix their client so that it doesn’t send out ill-formed items in the first place. Both sides have an interest in providing a smooth EAS experience for iOS users, but each side has a different set of engineering and delivery constraints that make the process of actually getting the fix out to customers a challenge.

Microsoft hasn’t publicly said much about this bug, other than that it is being investigated. (And Apple, AFAIK, has said absolutely nothing about it, which is regrettably typical.) Your best bet is to keep an eye on KB 2563324 for updates so you’ll know when Microsoft believes they have a solid understanding of the problem and the best way to fix it.

Meanwhile, the Z-Push team claims that turning off the Exchange calendar attendant feature would eliminate the problem, at the cost of some useful functionality. The iOS 6 bug is rare enough in most environments that I’d advise living with it rather than giving up the attendant functionality, but that’s a choice you’ll have to make based on your users and their needs.

I wonder whether Emtrace’s MoxierMail client has this problem? I’ve got an evaluation copy but haven’t been able to evaluate it yet; might be time to move that up a couple of notches on the priority scale….


Filed under UC&C