It’s a Century | Remote mailbox move to Office 365 fails

Today is a special day. Yes it is a Century indeed.

This post marks the 100th post in my technical blog and it has been an absolute pleasure to see that the community is benefiting from my experience in cloud & related technologies. I must thank all my fellow comrades in technical communities worldwide who always encourage me to share the new stuff to promote awareness. Also my MVP colleagues, You guys rock! Kudos to you all for being my role models. Years ago I too was an apprentice but the community paved the way for me to become a MVP and ignite the same passion for knowledge that I have throughout the world.

And by the way 100th post is break-fix solution so let’s see how we can troubleshoot an Exchange Hybrid deployment with Office 365.

I’ve encountered a strange issue during a Exchange 2013 + Office 365 hybrid deployment few days back. After a successful completion of a hybrid configuration I decided to move few mailboxes to office 365. I chose a mailbox which is less than 200 MB in size but even after 4 hours it was still in syncing state.

Looking at the mailbox move log I could see below error is happening all the time.

​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​10/19/2015 4:25:56 AM [TY1PR04MB0893] ” created move request.
10/19/2015 4:27:20 AM [HK2PR04MB1010] The Microsoft Exchange Mailbox Replication service ‘HK2PR04MB1010.apcprd04.prod.outlook.com’ (15.1.300.10 caps:7FFF) is examining the request.
10/19/2015 4:27:20 AM [HK2PR04MB1010] Connected to target mailbox ‘xxxx.onmicrosoft.com\6be5af13-590d-4461-b6bf-07a39d6767c9 (Primary)’, database ‘APCPR04DG064-db006’, Mailbox server ‘HK2PR04MB1010.apcprd04.prod.outlook.com’ Version 15.1 (Build 300.0).
10/19/2015 4:27:21 AM [HK2PR04MB1010] Connected to source mailbox ‘xxxx.onmicrosoft.com\6be5af13-590d-4461-b6bf-07a39d6767c9 (Primary)’, database ‘XXXX’, Mailbox server ‘XXXX’ Version 15.0 (Build 1076.0), proxy server ‘XXXX’ 15.0.1076.6 caps:1F7FFFFFCB07FFFF.
10/19/2015 4:27:23 AM [HK2PR04MB1010] Request processing started.
10/19/2015 4:27:23 AM [HK2PR04MB1010] Source mailbox information:
Regular Items: 2075, 139.8 MB (146,642,417 bytes)
Regular Deleted Items: 1447, 122.6 MB (128,550,697 bytes)
FAI Items: 48, 246.9 KB (252,781 bytes)
FAI Deleted Items: 0, 0 B (0 bytes)
10/19/2015 4:27:23 AM [HK2PR04MB1010] Cleared sync state for request 6be5af13-590d-4461-b6bf-07a39d6767c9 due to ‘CleanupOrphanedMailbox’.
10/19/2015 4:27:23 AM [HK2PR04MB1010] Mailbox signature will not be preserved for mailbox ‘xxxx.onmicrosoft.com\6be5af13-590d-4461-b6bf-07a39d6767c9 (Primary)’. Outlook clients will need to restart to access the moved mailbox.
10/19/2015 4:27:25 AM [HK2PR04MB1010] Stage: CreatingFolderHierarchy. Percent complete: 10.
10/19/2015 4:27:26 AM [HK2PR04MB1010] Initializing folder hierarchy from mailbox ‘xxxx.onmicrosoft.com\6be5af13-590d-4461-b6bf-07a39d6767c9 (Primary)’: 56 folders total.
10/19/2015 4:27:26 AM [HK2PR04MB1010] Folder creation progress: 0 folders created in mailbox ‘xxxx.onmicrosoft.com\6be5af13-590d-4461-b6bf-07a39d6767c9 (Primary)’.
10/19/2015 4:28:30 AM [HK2PR04MB1010] Folder hierarchy initialized for mailbox ‘xxxx.onmicrosoft.com\6be5af13-590d-4461-b6bf-07a39d6767c9 (Primary)’: 55 folders created.
10/19/2015 4:28:30 AM [HK2PR04MB1010] Stage: CreatingInitialSyncCheckpoint. Percent complete: 15.
10/19/2015 4:28:30 AM [HK2PR04MB1010] Initial sync checkpoint progress: 0/56 folders processed. Currently processing mailbox ‘xxxx.onmicrosoft.com\6be5af13-590d-4461-b6bf-07a39d6767c9 (Primary)’.
10/19/2015 4:29:10 AM [HK2PR04MB1010] Initial sync checkpoint completed: 49 folders processed.
10/19/2015 4:29:10 AM [HK2PR04MB1010] Stage: LoadingMessages. Percent complete: 20.
10/19/2015 4:30:06 AM [HK2PR04MB1010] Messages have been enumerated successfully. 3569 items loaded. Total size: 262.4 MB (275,191,505 bytes).
10/19/2015 4:30:06 AM [HK2PR04MB1010] Stage: CopyingMessages. Percent complete: 25.
10/19/2015 4:30:06 AM [HK2PR04MB1010] Copy progress: 0/3569 messages, 0 B (0 bytes)/262.4 MB (275,191,505 bytes), 32/56 folders completed.
10/19/2015 4:33:06 AM [HK2PR04MB1010] Transient error DataExportTransientException has occurred. The system will retry (1/60).
10/19/2015 4:34:41 AM [HK2PR04MB1010] The Microsoft Exchange Mailbox Replication service ‘HK2PR04MB1010.apcprd04.prod.outlook.com’ (15.1.300.10 caps:7FFF) is examining the request.
10/19/2015 4:34:42 AM [HK2PR04MB1010] Connected to target mailbox ‘xxxx.onmicrosoft.com\6be5af13-590d-4461-b6bf-07a39d6767c9 (Primary)’, database ‘APCPR04DG064-db006’, Mailbox server ‘HK2PR04MB1010.apcprd04.prod.outlook.com’ Version 15.1 (Build 300.0).
10/19/2015 4:34:49 AM [HK2PR04MB1010] Connected to source mailbox ‘xxxx.onmicrosoft.com\6be5af13-590d-4461-b6bf-07a39d6767c9 (Primary)’, database ‘XXXX’, Mailbox server ‘XXXX’ Version 15.0 (Build 1076.0), proxy server ‘XXXX’ 15.0.1076.6 caps:1F7FFFFFCB07FFFF.
10/19/2015 4:34:51 AM [HK2PR04MB1010] Request processing continued, stage LoadingMessages.
10/19/2015 4:35:48 AM [HK2PR04MB1010] Messages have been enumerated successfully. 3569 items loaded. Total size: 262.4 MB (275,191,505 bytes).
10/19/2015 4:35:48 AM [HK2PR04MB1010] Stage: CopyingMessages. Percent complete: 25.
10/19/2015 4:35:48 AM [HK2PR04MB1010] Copy progress: 0/3569 messages, 0 B (0 bytes)/262.4 MB (275,191,505 bytes), 32/56 folders completed.
10/19/2015 4:39:04 AM [HK2PR04MB1010] Transient error DataExportTransientException has occurred. The system will retry (1/60).

Surprisingly when I create a new mailbox in Exchange on-premises, initiate a directory synchronization and try to move that mailbox to the Office 365 tenant no issues were found. But whenever I tried to move existing mailboxes the issue is persistent. I though of reducing the mailbox size but in this scenario it was not an option.

Solution | Tweaking the MsExchangeMailboxReplication.exe.config

According to Microsoft Transient errors are usually connectivity issues to the MRSProxy. The remote move is a pull request from Office 365 tenant to on-premises to be exact. So any timeout might break the operation and will result in a never ending retry loop like this.

The MRSProxy config file is available in C:\Program Files\Microsoft\Exchange Server\V15\Bin\MsExchangeMailboxReplication.exe.config contains several parameter that we can tune to avoid such issues.

  • Stop the Microsoft Exchange Mailbox Replication Service in all CAS/MBX servers. Without that you may not be allowed perform the next step.
  • Edit the MsExchangeMailboxReplication.exe.config as below.

<MRSProxyConfiguration

IsEnabled=”true”

MaxMRSConnections=”100″

DataImportTimeout=”00:01:00″ />

  • Change the above DataImportTimeout value to “00:10:00” (10 minutes)
  • Start the Microsoft Exchange Mailbox Replication Service
  • Perform iisreset in CAS servers

Optionally you can set the ExportBufferSizeOverrideKB to 7500 in the same config file though it’s an optional optimization .  This is more of an optimization though and should not be necessary. This reduces the number of migration calls, especially for larger mailboxes and reduces the time spent in network latency. Be mindful that you need to have at least Exchange 2013 SP1 in your CAS servers to be able to edit this value.

If your WAN network traffic is highly utilized and unstable, above changes can save you a good number of troubleshooting hours for remote moves in your hybrid Exchange setup.

EWS not working in Exchange Hybrid

Deploying a hybrid Exchange environment should be a carefully planned process though you might encounter hiccups here and there unexpectedly. In a recent hybrid effort after executing the HCW for Exchange 2013 I have noticed that two tedious issues happening in the environment.

  1. Free/Busy information is not visible in Outlook Clients (any version). Scheduling Assistant throws an error saying that it cannot locate the server. Outlook Web App works just fine.
  2. Unable to create a migration batch from the EAC. It throws below error.

The ExchangeRemote endpoint settings could not be determined from the autodiscover response. No MRSProxy was found running at <FQDN of CAS server>.

I decided to have a look on the properties of EWS virtual directory for CAS servers. EWS is responsible for remote moves & free/busy information.

Enable MRS Proxy endpoint option was selected as it should be. Also I could access the EWS service from outside. So the URL is correct.

EWS (1)

When I checked the authentication section the Integrated Windows Authentication option was not enabled so I enabled same. This option allows domain joined computers to use session credentials of their respective windows logins with Outlook clients.

EWS (2)After doing iisreset on all CAS servers I noticed that Free/Busy information is now available via Outlook and I can successfully create and complete migration batches as well.

This TechNet article explains different scenarios in an Exchange Hybrid deployment which can result in such an error and worth a look if you cannot see free/busy information from one tenant to the other (Exhcnage Online Users cannot see Exchange On-Premises free/busy info and vice versa)

Outlook 2007 Clients are disconnected in Exchange Hybrid

Deploying an Exchange Hybrid setup requires lot of effort specially around DNS records. In a recent deployment I noticed that Outlook 2007 clients are in disconnected state once the Hybrid feature has been enabled for an Exchange 2013 single forest organization.

We were using a new Subject Alternative Name (SAN) Certificate from a third party CA which included AutoDiscover.domain.com record. An autodicover record is vital to share free/bust information between Exchange Online & On-premises organization, as well as clients to differentiate where there mailboxes reside.

The problem was not present in Outlook 2010/2013 clients and creating new user profiles was not an option and was not a remedy. Renewing the SSL certificate has always been troubling if we don’t do it the proper way.

Solution

I decided to take a look on autodiscover virtual directories first. To do that open Exchange PowerShell as administrator and executed below cmdlet.

Get-AutoDiscoverVirtualDirectory | fl internalurl,externalurl

The virtual directory was not pointing to correct URL. Therefore below cmdlet has been used to rectify same.

Get-AutodiscoverVirtualDirectory -server <Server Name> | Set-AutodiscoverVirtualDirectory -ExternalUrl ‘https://<Server FQDN>/Autodiscover/Autodiscover.xml’ -InternalUrl ‘https://<Server FQDN>/Autodiscover/Autodiscover.xml’

I had three MBX/CAS servers to run this cmdlet. The <Server Name> placeholder represents same. Now in IIS Manager of each MBX/CAS server, right click the MSExchangeAutoDiscoverAppPool application pool and click Recycle. This will recycle the AutoDiscover application pool.

When I checked the connectivity of Outlook 2007 clients most of them are showing connected and e-mails were downloading without any issue. For those clients that didn’t respond we had to reconfigure Outlook profiles but it was quite a few.

Recommendation

I’m not really comfortable with Outlook 2007, specially if it involves Office 365. If you intend to use Outlook 2007 with this kind of a setup I strongly suggest that you update to the latest service pack or if possible update it to Outlook 2010 at minimum. Service Packs play a vital role and in this case none of those Outlook clients were patched.

Office 365 woes | Password Sync doesn’t synchronize with Azure AD Connect

Microsoft has retired DirSync tool and introduced one tool to synchronize your on-premises identities to Microsoft Azure Active Directory. This tool “Azure AD Connect” can be used in both Office 365 only & Azure AD without O365 scenarios to extend your local AD to the cloud. One of the key advantages is the ability to synchronize the password hash of local AD account with Azure AD where the users will have a single identity & password rather than two different sets of credentials. This is a hash of the password hash stored in your local AD not clear text password. This feature is heavily used in Office 365 deployments to provide SSO (Single/Same Sign On) capabilities to the users to provide single consolidated identity.

I’m working on an Exchange 2013 Hybrid setup these days where one of the prerequisites is to setup Directory Synchronization. We have identified the password hash sync is also required in this situation. When I completed the Azure AD Connect setup wizard I got below error at first. If you are unfamiliar with configuring AAD connect you can refer here.

Now surprisingly I cannot connect to Azure AD via PowerShell and got the same error. My credentials were correct and I decided to check that with my laptop (different ISP) and it was a success. Surprisingly the local ISP has some connectivity issues with Office 365 IP ranges and frequent timeouts were evident and then we changed the network path to a different ISP to resolve the issue.

Since the wizard was completed and no accounts were synced, I decided to go for a manual force sync with PowerShell. Notice the path for DirSync binaries; it hasn’t been changed in AAD Connect as well. The option Initial will do a full synchronization where the option delta will do a delta sync only.

C:\Program Files\Microsoft Azure AD Sync\Bin> .\DirectorySyncClientCmd.exe Initial

All the user accounts were properly synced this time but when I tried to login to Office 365 using an AD account it always says that the password is incorrect.

Investigation

When I tried to review the existing configuration for Azure AD connect I noticed that it reports the password sync as disabled. Uninstalling the tool, installing again and performing couple of full sync were useless and still I was getting the same issue when I’m trying to login.

  1. Verified password sync is disabled via using PowerShell

    Following cmdlets have been used to verify above and I noticed password sync was in fact disabled although I checked the option in Azure AD connect setup.

    Import-Module ADSync

    Get-ADSyncAADPasswordSyncConfiguration -SourceConnector <‘LOCAL DOMAIN NAME>

  2. Enabled password sync via PowerShell

    Set-ADSyncAADPasswordSyncConfiguration -SourceConnector <‘LOCAL DOMAIN NAME> -TargetConnector <‘xxxxxxx.onmicrosoft.com – AAD’> -Enable $true

After executing above cmdlets and another full synchronization I noticed below error has been logged in Directory Synchronization event log, still with no luck with passwords.

In the Synchronization Service Manager (C:\Program Files\Microsoft Azure AD Sync\UIShell\miisclient.exe) displays status of “stopped-extension-dll-exception” for operations on the Microsoft Azure Active Directory Connector.

Solution

  1. The “stopped-extension-dll-exception” error was due to the fact that password policy of Azure AD Sync account, set to expire after a certain number of days. This account is created automatically once you run the Azure AD Connect wizard.

    To set the password to never expire following cmdlets were used.

    PS C:\> Connect-MsolService

    Provide your O365 admin credentials to connect to Azure AD service.

    PS C:\> Set-MsolUser -UserPrincipalName <Sync Account UPN> -PasswordNeverExpires $true

  2. Make sure the sync account is a global administrator in your office 365 tenant. In my case it was not and I have assign the user role Global Administrator to that account. You can refer the original Microsoft Office 365 support article on how to do that here.
  3. To overcome the “Password Synchronization has not been activated for this company” error is logged in Event Viewer I executed below cmdlet again in PowerShell.

Set-ADSyncAADPasswordSyncState -ConnectorName < ‘xxxxxxx.onmicrosoft.com – AAD’> -Enable $True

Morale of the Story

When I ran the Azure AD Connect wizard for the first time I selected Start the synchronization process ASAP option as below. Due to the issues I have faced with internet connectivity it wasn’t much of a success.

Also I didn’t select the Use an existing service account option during installation. By default, Azure AD Connect will create a local service account for the synchronization services to use in that case the account that wasn’t a global admin. The user role (which should be global admin), password expiration policy & password is unknown in this automatically created account. Therefore I highly recommend to use an existing global administrator account in your O365 tenant to overcome this.

Also always use the Customize option where you have control over additional configuration options such as above service account while setting up Azure AD Connect.