Tag Archives: Exchange Hybrid

It’s a Century | Remote mailbox move to Office 365 fails

Today is a special day. Yes it is a Century indeed.

This post marks the 100th post in my technical blog and it has been an absolute pleasure to see that the community is benefiting from my experience in cloud & related technologies. I must thank all my fellow comrades in technical communities worldwide who always encourage me to share the new stuff to promote awareness. Also my MVP colleagues, You guys rock! Kudos to you all for being my role models. Years ago I too was an apprentice but the community paved the way for me to become a MVP and ignite the same passion for knowledge that I have throughout the world.

And by the way 100th post is break-fix solution so let’s see how we can troubleshoot an Exchange Hybrid deployment with Office 365.

I’ve encountered a strange issue during a Exchange 2013 + Office 365 hybrid deployment few days back. After a successful completion of a hybrid configuration I decided to move few mailboxes to office 365. I chose a mailbox which is less than 200 MB in size but even after 4 hours it was still in syncing state.

Looking at the mailbox move log I could see below error is happening all the time.

​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​10/19/2015 4:25:56 AM [TY1PR04MB0893] ” created move request.
10/19/2015 4:27:20 AM [HK2PR04MB1010] The Microsoft Exchange Mailbox Replication service ‘HK2PR04MB1010.apcprd04.prod.outlook.com’ (15.1.300.10 caps:7FFF) is examining the request.
10/19/2015 4:27:20 AM [HK2PR04MB1010] Connected to target mailbox ‘xxxx.onmicrosoft.com\6be5af13-590d-4461-b6bf-07a39d6767c9 (Primary)’, database ‘APCPR04DG064-db006’, Mailbox server ‘HK2PR04MB1010.apcprd04.prod.outlook.com’ Version 15.1 (Build 300.0).
10/19/2015 4:27:21 AM [HK2PR04MB1010] Connected to source mailbox ‘xxxx.onmicrosoft.com\6be5af13-590d-4461-b6bf-07a39d6767c9 (Primary)’, database ‘XXXX’, Mailbox server ‘XXXX’ Version 15.0 (Build 1076.0), proxy server ‘XXXX’ 15.0.1076.6 caps:1F7FFFFFCB07FFFF.
10/19/2015 4:27:23 AM [HK2PR04MB1010] Request processing started.
10/19/2015 4:27:23 AM [HK2PR04MB1010] Source mailbox information:
Regular Items: 2075, 139.8 MB (146,642,417 bytes)
Regular Deleted Items: 1447, 122.6 MB (128,550,697 bytes)
FAI Items: 48, 246.9 KB (252,781 bytes)
FAI Deleted Items: 0, 0 B (0 bytes)
10/19/2015 4:27:23 AM [HK2PR04MB1010] Cleared sync state for request 6be5af13-590d-4461-b6bf-07a39d6767c9 due to ‘CleanupOrphanedMailbox’.
10/19/2015 4:27:23 AM [HK2PR04MB1010] Mailbox signature will not be preserved for mailbox ‘xxxx.onmicrosoft.com\6be5af13-590d-4461-b6bf-07a39d6767c9 (Primary)’. Outlook clients will need to restart to access the moved mailbox.
10/19/2015 4:27:25 AM [HK2PR04MB1010] Stage: CreatingFolderHierarchy. Percent complete: 10.
10/19/2015 4:27:26 AM [HK2PR04MB1010] Initializing folder hierarchy from mailbox ‘xxxx.onmicrosoft.com\6be5af13-590d-4461-b6bf-07a39d6767c9 (Primary)’: 56 folders total.
10/19/2015 4:27:26 AM [HK2PR04MB1010] Folder creation progress: 0 folders created in mailbox ‘xxxx.onmicrosoft.com\6be5af13-590d-4461-b6bf-07a39d6767c9 (Primary)’.
10/19/2015 4:28:30 AM [HK2PR04MB1010] Folder hierarchy initialized for mailbox ‘xxxx.onmicrosoft.com\6be5af13-590d-4461-b6bf-07a39d6767c9 (Primary)’: 55 folders created.
10/19/2015 4:28:30 AM [HK2PR04MB1010] Stage: CreatingInitialSyncCheckpoint. Percent complete: 15.
10/19/2015 4:28:30 AM [HK2PR04MB1010] Initial sync checkpoint progress: 0/56 folders processed. Currently processing mailbox ‘xxxx.onmicrosoft.com\6be5af13-590d-4461-b6bf-07a39d6767c9 (Primary)’.
10/19/2015 4:29:10 AM [HK2PR04MB1010] Initial sync checkpoint completed: 49 folders processed.
10/19/2015 4:29:10 AM [HK2PR04MB1010] Stage: LoadingMessages. Percent complete: 20.
10/19/2015 4:30:06 AM [HK2PR04MB1010] Messages have been enumerated successfully. 3569 items loaded. Total size: 262.4 MB (275,191,505 bytes).
10/19/2015 4:30:06 AM [HK2PR04MB1010] Stage: CopyingMessages. Percent complete: 25.
10/19/2015 4:30:06 AM [HK2PR04MB1010] Copy progress: 0/3569 messages, 0 B (0 bytes)/262.4 MB (275,191,505 bytes), 32/56 folders completed.
10/19/2015 4:33:06 AM [HK2PR04MB1010] Transient error DataExportTransientException has occurred. The system will retry (1/60).
10/19/2015 4:34:41 AM [HK2PR04MB1010] The Microsoft Exchange Mailbox Replication service ‘HK2PR04MB1010.apcprd04.prod.outlook.com’ (15.1.300.10 caps:7FFF) is examining the request.
10/19/2015 4:34:42 AM [HK2PR04MB1010] Connected to target mailbox ‘xxxx.onmicrosoft.com\6be5af13-590d-4461-b6bf-07a39d6767c9 (Primary)’, database ‘APCPR04DG064-db006’, Mailbox server ‘HK2PR04MB1010.apcprd04.prod.outlook.com’ Version 15.1 (Build 300.0).
10/19/2015 4:34:49 AM [HK2PR04MB1010] Connected to source mailbox ‘xxxx.onmicrosoft.com\6be5af13-590d-4461-b6bf-07a39d6767c9 (Primary)’, database ‘XXXX’, Mailbox server ‘XXXX’ Version 15.0 (Build 1076.0), proxy server ‘XXXX’ 15.0.1076.6 caps:1F7FFFFFCB07FFFF.
10/19/2015 4:34:51 AM [HK2PR04MB1010] Request processing continued, stage LoadingMessages.
10/19/2015 4:35:48 AM [HK2PR04MB1010] Messages have been enumerated successfully. 3569 items loaded. Total size: 262.4 MB (275,191,505 bytes).
10/19/2015 4:35:48 AM [HK2PR04MB1010] Stage: CopyingMessages. Percent complete: 25.
10/19/2015 4:35:48 AM [HK2PR04MB1010] Copy progress: 0/3569 messages, 0 B (0 bytes)/262.4 MB (275,191,505 bytes), 32/56 folders completed.
10/19/2015 4:39:04 AM [HK2PR04MB1010] Transient error DataExportTransientException has occurred. The system will retry (1/60).

Surprisingly when I create a new mailbox in Exchange on-premises, initiate a directory synchronization and try to move that mailbox to the Office 365 tenant no issues were found. But whenever I tried to move existing mailboxes the issue is persistent. I though of reducing the mailbox size but in this scenario it was not an option.

Solution | Tweaking the MsExchangeMailboxReplication.exe.config

According to Microsoft Transient errors are usually connectivity issues to the MRSProxy. The remote move is a pull request from Office 365 tenant to on-premises to be exact. So any timeout might break the operation and will result in a never ending retry loop like this.

The MRSProxy config file is available in C:\Program Files\Microsoft\Exchange Server\V15\Bin\MsExchangeMailboxReplication.exe.config contains several parameter that we can tune to avoid such issues.

  • Stop the Microsoft Exchange Mailbox Replication Service in all CAS/MBX servers. Without that you may not be allowed perform the next step.
  • Edit the MsExchangeMailboxReplication.exe.config as below.

<MRSProxyConfiguration

IsEnabled=”true”

MaxMRSConnections=”100″

DataImportTimeout=”00:01:00″ />

  • Change the above DataImportTimeout value to “00:10:00” (10 minutes)
  • Start the Microsoft Exchange Mailbox Replication Service
  • Perform iisreset in CAS servers

Optionally you can set the ExportBufferSizeOverrideKB to 7500 in the same config file though it’s an optional optimization .  This is more of an optimization though and should not be necessary. This reduces the number of migration calls, especially for larger mailboxes and reduces the time spent in network latency. Be mindful that you need to have at least Exchange 2013 SP1 in your CAS servers to be able to edit this value.

If your WAN network traffic is highly utilized and unstable, above changes can save you a good number of troubleshooting hours for remote moves in your hybrid Exchange setup.

EWS not working in Exchange Hybrid

Deploying a hybrid Exchange environment should be a carefully planned process though you might encounter hiccups here and there unexpectedly. In a recent hybrid effort after executing the HCW for Exchange 2013 I have noticed that two tedious issues happening in the environment.

  1. Free/Busy information is not visible in Outlook Clients (any version). Scheduling Assistant throws an error saying that it cannot locate the server. Outlook Web App works just fine.
  2. Unable to create a migration batch from the EAC. It throws below error.

The ExchangeRemote endpoint settings could not be determined from the autodiscover response. No MRSProxy was found running at <FQDN of CAS server>.

I decided to have a look on the properties of EWS virtual directory for CAS servers. EWS is responsible for remote moves & free/busy information.

Enable MRS Proxy endpoint option was selected as it should be. Also I could access the EWS service from outside. So the URL is correct.

EWS (1)

When I checked the authentication section the Integrated Windows Authentication option was not enabled so I enabled same. This option allows domain joined computers to use session credentials of their respective windows logins with Outlook clients.

EWS (2)After doing iisreset on all CAS servers I noticed that Free/Busy information is now available via Outlook and I can successfully create and complete migration batches as well.

This TechNet article explains different scenarios in an Exchange Hybrid deployment which can result in such an error and worth a look if you cannot see free/busy information from one tenant to the other (Exhcnage Online Users cannot see Exchange On-Premises free/busy info and vice versa)

Outlook 2007 Clients are disconnected in Exchange Hybrid

Deploying an Exchange Hybrid setup requires lot of effort specially around DNS records. In a recent deployment I noticed that Outlook 2007 clients are in disconnected state once the Hybrid feature has been enabled for an Exchange 2013 single forest organization.

We were using a new Subject Alternative Name (SAN) Certificate from a third party CA which included AutoDiscover.domain.com record. An autodicover record is vital to share free/bust information between Exchange Online & On-premises organization, as well as clients to differentiate where there mailboxes reside.

The problem was not present in Outlook 2010/2013 clients and creating new user profiles was not an option and was not a remedy. Renewing the SSL certificate has always been troubling if we don’t do it the proper way.

Solution

I decided to take a look on autodiscover virtual directories first. To do that open Exchange PowerShell as administrator and executed below cmdlet.

Get-AutoDiscoverVirtualDirectory | fl internalurl,externalurl

The virtual directory was not pointing to correct URL. Therefore below cmdlet has been used to rectify same.

Get-AutodiscoverVirtualDirectory -server <Server Name> | Set-AutodiscoverVirtualDirectory -ExternalUrl ‘https://<Server FQDN>/Autodiscover/Autodiscover.xml’ -InternalUrl ‘https://<Server FQDN>/Autodiscover/Autodiscover.xml’

I had three MBX/CAS servers to run this cmdlet. The <Server Name> placeholder represents same. Now in IIS Manager of each MBX/CAS server, right click the MSExchangeAutoDiscoverAppPool application pool and click Recycle. This will recycle the AutoDiscover application pool.

When I checked the connectivity of Outlook 2007 clients most of them are showing connected and e-mails were downloading without any issue. For those clients that didn’t respond we had to reconfigure Outlook profiles but it was quite a few.

Recommendation

I’m not really comfortable with Outlook 2007, specially if it involves Office 365. If you intend to use Outlook 2007 with this kind of a setup I strongly suggest that you update to the latest service pack or if possible update it to Outlook 2010 at minimum. Service Packs play a vital role and in this case none of those Outlook clients were patched.