It’s a Century | Remote mailbox move to Office 365 fails

Today is a special day. Yes it is a Century indeed.

This post marks the 100th post in my technical blog and it has been an absolute pleasure to see that the community is benefiting from my experience in cloud & related technologies. I must thank all my fellow comrades in technical communities worldwide who always encourage me to share the new stuff to promote awareness. Also my MVP colleagues, You guys rock! Kudos to you all for being my role models. Years ago I too was an apprentice but the community paved the way for me to become a MVP and ignite the same passion for knowledge that I have throughout the world.

And by the way 100th post is break-fix solution so let’s see how we can troubleshoot an Exchange Hybrid deployment with Office 365.

I’ve encountered a strange issue during a Exchange 2013 + Office 365 hybrid deployment few days back. After a successful completion of a hybrid configuration I decided to move few mailboxes to office 365. I chose a mailbox which is less than 200 MB in size but even after 4 hours it was still in syncing state.

Looking at the mailbox move log I could see below error is happening all the time.

​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​10/19/2015 4:25:56 AM [TY1PR04MB0893] ” created move request.
10/19/2015 4:27:20 AM [HK2PR04MB1010] The Microsoft Exchange Mailbox Replication service ‘HK2PR04MB1010.apcprd04.prod.outlook.com’ (15.1.300.10 caps:7FFF) is examining the request.
10/19/2015 4:27:20 AM [HK2PR04MB1010] Connected to target mailbox ‘xxxx.onmicrosoft.com\6be5af13-590d-4461-b6bf-07a39d6767c9 (Primary)’, database ‘APCPR04DG064-db006’, Mailbox server ‘HK2PR04MB1010.apcprd04.prod.outlook.com’ Version 15.1 (Build 300.0).
10/19/2015 4:27:21 AM [HK2PR04MB1010] Connected to source mailbox ‘xxxx.onmicrosoft.com\6be5af13-590d-4461-b6bf-07a39d6767c9 (Primary)’, database ‘XXXX’, Mailbox server ‘XXXX’ Version 15.0 (Build 1076.0), proxy server ‘XXXX’ 15.0.1076.6 caps:1F7FFFFFCB07FFFF.
10/19/2015 4:27:23 AM [HK2PR04MB1010] Request processing started.
10/19/2015 4:27:23 AM [HK2PR04MB1010] Source mailbox information:
Regular Items: 2075, 139.8 MB (146,642,417 bytes)
Regular Deleted Items: 1447, 122.6 MB (128,550,697 bytes)
FAI Items: 48, 246.9 KB (252,781 bytes)
FAI Deleted Items: 0, 0 B (0 bytes)
10/19/2015 4:27:23 AM [HK2PR04MB1010] Cleared sync state for request 6be5af13-590d-4461-b6bf-07a39d6767c9 due to ‘CleanupOrphanedMailbox’.
10/19/2015 4:27:23 AM [HK2PR04MB1010] Mailbox signature will not be preserved for mailbox ‘xxxx.onmicrosoft.com\6be5af13-590d-4461-b6bf-07a39d6767c9 (Primary)’. Outlook clients will need to restart to access the moved mailbox.
10/19/2015 4:27:25 AM [HK2PR04MB1010] Stage: CreatingFolderHierarchy. Percent complete: 10.
10/19/2015 4:27:26 AM [HK2PR04MB1010] Initializing folder hierarchy from mailbox ‘xxxx.onmicrosoft.com\6be5af13-590d-4461-b6bf-07a39d6767c9 (Primary)’: 56 folders total.
10/19/2015 4:27:26 AM [HK2PR04MB1010] Folder creation progress: 0 folders created in mailbox ‘xxxx.onmicrosoft.com\6be5af13-590d-4461-b6bf-07a39d6767c9 (Primary)’.
10/19/2015 4:28:30 AM [HK2PR04MB1010] Folder hierarchy initialized for mailbox ‘xxxx.onmicrosoft.com\6be5af13-590d-4461-b6bf-07a39d6767c9 (Primary)’: 55 folders created.
10/19/2015 4:28:30 AM [HK2PR04MB1010] Stage: CreatingInitialSyncCheckpoint. Percent complete: 15.
10/19/2015 4:28:30 AM [HK2PR04MB1010] Initial sync checkpoint progress: 0/56 folders processed. Currently processing mailbox ‘xxxx.onmicrosoft.com\6be5af13-590d-4461-b6bf-07a39d6767c9 (Primary)’.
10/19/2015 4:29:10 AM [HK2PR04MB1010] Initial sync checkpoint completed: 49 folders processed.
10/19/2015 4:29:10 AM [HK2PR04MB1010] Stage: LoadingMessages. Percent complete: 20.
10/19/2015 4:30:06 AM [HK2PR04MB1010] Messages have been enumerated successfully. 3569 items loaded. Total size: 262.4 MB (275,191,505 bytes).
10/19/2015 4:30:06 AM [HK2PR04MB1010] Stage: CopyingMessages. Percent complete: 25.
10/19/2015 4:30:06 AM [HK2PR04MB1010] Copy progress: 0/3569 messages, 0 B (0 bytes)/262.4 MB (275,191,505 bytes), 32/56 folders completed.
10/19/2015 4:33:06 AM [HK2PR04MB1010] Transient error DataExportTransientException has occurred. The system will retry (1/60).
10/19/2015 4:34:41 AM [HK2PR04MB1010] The Microsoft Exchange Mailbox Replication service ‘HK2PR04MB1010.apcprd04.prod.outlook.com’ (15.1.300.10 caps:7FFF) is examining the request.
10/19/2015 4:34:42 AM [HK2PR04MB1010] Connected to target mailbox ‘xxxx.onmicrosoft.com\6be5af13-590d-4461-b6bf-07a39d6767c9 (Primary)’, database ‘APCPR04DG064-db006’, Mailbox server ‘HK2PR04MB1010.apcprd04.prod.outlook.com’ Version 15.1 (Build 300.0).
10/19/2015 4:34:49 AM [HK2PR04MB1010] Connected to source mailbox ‘xxxx.onmicrosoft.com\6be5af13-590d-4461-b6bf-07a39d6767c9 (Primary)’, database ‘XXXX’, Mailbox server ‘XXXX’ Version 15.0 (Build 1076.0), proxy server ‘XXXX’ 15.0.1076.6 caps:1F7FFFFFCB07FFFF.
10/19/2015 4:34:51 AM [HK2PR04MB1010] Request processing continued, stage LoadingMessages.
10/19/2015 4:35:48 AM [HK2PR04MB1010] Messages have been enumerated successfully. 3569 items loaded. Total size: 262.4 MB (275,191,505 bytes).
10/19/2015 4:35:48 AM [HK2PR04MB1010] Stage: CopyingMessages. Percent complete: 25.
10/19/2015 4:35:48 AM [HK2PR04MB1010] Copy progress: 0/3569 messages, 0 B (0 bytes)/262.4 MB (275,191,505 bytes), 32/56 folders completed.
10/19/2015 4:39:04 AM [HK2PR04MB1010] Transient error DataExportTransientException has occurred. The system will retry (1/60).

Surprisingly when I create a new mailbox in Exchange on-premises, initiate a directory synchronization and try to move that mailbox to the Office 365 tenant no issues were found. But whenever I tried to move existing mailboxes the issue is persistent. I though of reducing the mailbox size but in this scenario it was not an option.

Solution | Tweaking the MsExchangeMailboxReplication.exe.config

According to Microsoft Transient errors are usually connectivity issues to the MRSProxy. The remote move is a pull request from Office 365 tenant to on-premises to be exact. So any timeout might break the operation and will result in a never ending retry loop like this.

The MRSProxy config file is available in C:\Program Files\Microsoft\Exchange Server\V15\Bin\MsExchangeMailboxReplication.exe.config contains several parameter that we can tune to avoid such issues.

  • Stop the Microsoft Exchange Mailbox Replication Service in all CAS/MBX servers. Without that you may not be allowed perform the next step.
  • Edit the MsExchangeMailboxReplication.exe.config as below.

<MRSProxyConfiguration

IsEnabled=”true”

MaxMRSConnections=”100″

DataImportTimeout=”00:01:00″ />

  • Change the above DataImportTimeout value to “00:10:00” (10 minutes)
  • Start the Microsoft Exchange Mailbox Replication Service
  • Perform iisreset in CAS servers

Optionally you can set the ExportBufferSizeOverrideKB to 7500 in the same config file though it’s an optional optimization .  This is more of an optimization though and should not be necessary. This reduces the number of migration calls, especially for larger mailboxes and reduces the time spent in network latency. Be mindful that you need to have at least Exchange 2013 SP1 in your CAS servers to be able to edit this value.

If your WAN network traffic is highly utilized and unstable, above changes can save you a good number of troubleshooting hours for remote moves in your hybrid Exchange setup.

  • Dave Thompson

    Pardon my ignorance – how did you get the detailed move log above? I thought it must be easy, but I can’t find anything like the level of detail you are getting above. I have an issue with mailbox moves from hybrid on-premise Exchange 2013 and O365 and would appreciate more than a one-line error. Thanks in advance – very useful post.

    • jeet rajkumar

      Get-MoveRequestStatistics “username” -IncludeReport | fl or Get-MoveRequestStatistics “username” -IncludeReport | fl >username.txt

  • Dave Thompson

    Ah – okay, I found it sort of, from the Move Request window in the EMC of Exchange 2010, then Properties. Although I’m still curious where to find this in the Exchange 2013 ECP ?

    • Hi Dave, actually you can find this log in the migration batch itself for Exchange 2013. When you click the failed/completed/in-progress migration batch there is a link in right side called “view details” which directs you to the log file.

  • Roshan

    wow superb……you saved my life 🙂

  • Ramesh Dasari

    Hi,
    We did these changes but Mailbox Replication Service is not starting after making this change. Any idea on that ?