Exchange 2010 DAGs support encryption and compression of the mail data they pass around as part of the seeding and replication process. How do these capabilities work?
The short answer is “it’s a Windows thing.” Exchange uses the encryption and signing methods provided by the underlying Windows OS whenever possible. In the case of SSL/TLS, Exchange can do much of its own certificate management (including generating its own self-signed certificates), but all of the actual crypto is done by Windows.
In the case of DAG encryption, Exchange uses the Kerberos implementation included in the Windows Kerberos security support provider (SSP). The SSP provides routines that allow clients to encrypt, sign, verify, and decrypt messages containing arbitrary application-specific data. Once the two endpoints have concluded a handshake and key exchange, they can both make calls to process the data blocks.
Exchange always tries to pick the strongest possible algorithm from the list of those supported by the host OS. For Windows Server 2008 and 2008 R2, that means AES 256-bit, ideally with SHA-1 used as an HMAC.
On the compression front, DAGs use the familiar LZ77 algorithm. Microsoft internally calls their implementation (described on page 45 here) “XPRESS”, for some reason that I haven’t been able to figure out.
Outlook uses many of these same mechanisms; for example, MAPI RPC compression between Exchange and Outlook uses XPRESS too, and MAPI RPC encryption can use the Kerberos SSP (or the NTLM SSP, but that’s not available for DAGs.) That’s no big surprise, given that these capabilities are built into Windows. Why reinvent the wheel?
In terms of how you control these options in a live DAG, Set-DatabaseAvailabilityGroup is your friend. By default, newly created DAGs will use encryption only when replicating across different subnets, with no encryption used during seeding operations. You can change this with the NetworkEncryption switch; just set it to Enabled instead of InterSubnetOnly. In like manner, compression is normally enabled only for cross-subnet replication, but the NetworkCompression switch will let you fix that right up.
Note that these properties are associated with the DAG itself, not the underlying DAG network. For that reason, you can only set these properties when the DAG (and its underlying WFC nodes) are up– the settings are node-level properties.
What about using IPsec with DAGs? That’s a topic for another post!