Exchange migration “Couldn’t switch the mailbox into Sync Source mode”

Exchange

The issue

During one of my Exchange Online migration projects I encountered the following error on several mailboxes:

“Message : Error: Couldn’t switch the mailbox into Sync Source mode.
This could be because of one of the following reasons:
Another administrator is currently moving the mailbox.
The mailbox is locked.
The Microsoft Exchange Mailbox Replication service (MRS) doesn’t have the correct permissions.
Network errors are preventing MRS from cleanly closing its session with the Mailbox server.
If this is the case, MRS may continue to encounter this error for up to 2 hours – this duration is controlled by the TCP KeepAlive settings on the Mailbox server.
Wait for the mailbox to be released before attempting to move this mailbox again.”

The environment where I encountered this was an Exchange Server 2010 environment in a Database Availability Group (DAG) configuration. There where three DAG servers and two Client Access servers.
We placed two Exchange Server 2016 servers in front for the Hybrid connection with Exchange Online.

The cause

This was a uncommon error for me, so I did some research first before proceeding with the suggested action to alter the TCP settings and request that the entire Exchange Server environment is restarted.

I found several blogs with different solutions, rather then changing the TCP value. One of the suggestions was to run the mailbox repair option. I did this without success. I also tried an internal move request of the mailbox and it failed on the same error.

After reading a blog by Brad Hughes on the topic, I found a interesting remark about the cause of the error.

“When moving a mailbox, the Mailbox Replication Service (MRS) sets an “InTransitStatus” flag in the source mailbox to make sure other moves don’t try to act on this source mailbox at the same time.  This flag is really just held in memory in the source Information Store (Store) process (Store.exe for 2010 and Microsoft.Exchange.Store.Worker.exe for 2013 and 2016). “

The solution for me

So my conclusion on this was:

If it’s held in memory, what options do I have to ‘reset’ this, without restarting the servers?

Because the source mailbox databases were placed in a DAG, I asked the customer to failover the mailbox database, which contained the error user. After the failover (which caused downtime for less then a minute), I recreated the move request again and this time the mailbox was synced to Office 365.

I find this solution less disrupting and quicker than a reboot of an entire Exchange Server.

The initial error was for a primary mailbox, but this error can also occur when a user has an in-place archive. In this case, for me, it was also enough to dismount and mount the mailbox database.
Yes, you read it correctly; dismount and mount the mailbox database. The archive databases weren’t part of the DAG configuration. This action also triggers a memory reset.

This is not an error that happens when you start a migration; I also got it five times during synchronization of mailboxes. The solution here was the same, failover or dismount/mount the mailbox database.

Conclusion

I would suggest to not always go blindly with the Microsoft suggestion for the fix. Investigate the root cause is of the problem.
Yes, the Microsoft suggestion would have fixed it, but for the suggested fix to work, you need to restart all Exchange servers. This will cause a failover or dismount/mount action of the mailbox database.

For me it was just enough to just failover or dismount/mount the mailbox database(s) to get the synchronization starting or continuing. Without disrupting the rest of the organization, and aiding me in meeting my deadlines as a consultant.

I hope this helps for you, too.

TOOL: My own Exchange Move Request Report script

As a consultant I do a lot of migrations or assist in them. In this case I wrote my own script for processing the information generated by the Mailbox Replication Service. This service is used in Microsoft Exchange on-premises and in the cloud for migrating the mailboxes between databases or environments.

After multiple times doing the administration and processing of information manually. I decided to create my own processing script to retrieve the desired information and make it re-usable in Excel or other tooling, to export gathered information to CSV file format. I named the script Get Move Request Report script.

The script is provide as-is and maybe be used at your own risk.

Versions

Version 1: This is the first version and it’s basic. No advanced switch options or logging.

About the Exchange Move Request Report Script

The script consist of three functions:

1. Creates an overview of the Bad Items it found in the move request report.
2. Creates an overview of the basic move information of the move request statistics.
3. Creates an overview of the extended information of the move request statistics.
4. Creates an overview of information that quickly can be used for reports to stakeholders.

Parameters

When you execute the script and no parameters are given. It generates a bad items overview, basic move information and extended move information output of all existing move request in scope.

.\Get-MoveRequestReport.ps1

To generate only the outputs for all InProgress move requests, give the following command:

.\Get-MoveRequestReport.ps1 -Inprogress $true -AllMoves $false

To generate only the outputs for all Synced move requests, give the following command:

.\Get-MoveRequestReport.ps1 -Synced $true -AllMoves $false

To generate only the outputs for all Completed move requests, give the following command:

.\Get-MoveRequestReport.ps1 -Completed $true -AllMoves $false

Requirements

To use the script, you already need to have an Exchange PowerShell session open to the target environment (on-premises or cloud), where the move request are created.

Exchange Move Request Report Script

The script is displayed below for your review:

[CmdletBinding()]
param (
    [bool]$CompletedMoves = $false, 
    [bool]$Synced = $false,    
    [bool]$Inprogress = $false,
    [bool]$AllMoves = $true,
    [bool]$IncludeAllBadItems = $true,
    [bool]$MoveDataBasic = $true,
    [bool]$MoveDataFull = $true,
    [bool]$GenereExportInfo = $true
)

#region Functions
function generateBadItemInformation ($BadItem,$User)
{
    $ExportEntry = New-Object PSObject
    $ExportEntry | Add-Member -MemberType "NoteProperty" -Name Identity -Value $User.Alias
    $ExportEntry | Add-Member -MemberType "NoteProperty" -Name BatchName -Value $User.Batchname
    $ExportEntry | Add-Member -MemberType "NoteProperty" -Name DisplayName -Value $User.DisplayName
    $ExportEntry | Add-Member -MemberType "NoteProperty" -Name Date -Value $BadItem.Date
    $ExportEntry | Add-Member -MemberType "NoteProperty" -Name Subject -Value $BadItem.Subject
    $ExportEntry | Add-Member -MemberType "NoteProperty" -Name Kind -Value $BadItem.Kind
    $ExportEntry | Add-Member -MemberType "NoteProperty" -Name FolderName -Value $BadItem.FolderName
    
    
    $ExportEntry   
}

function generateBasicMoveInformation ($User)
{
    $ExportEntry = New-Object PSObject
    $ExportEntry | Add-Member -MemberType "NoteProperty" -Name Identity -Value $User.Alias
    $ExportEntry | Add-Member -MemberType "NoteProperty" -Name BatchName -Value $User.Batchname
    $ExportEntry | Add-Member -MemberType "NoteProperty" -Name DisplayName -Value $User.DisplayName
    $ExportEntry | Add-Member -MemberType "NoteProperty" -Name Status -Value $User.Status
    $ExportEntry | Add-Member -MemberType "NoteProperty" -Name SyncStage -Value $User.SyncStage
    $ExportEntry | Add-Member -MemberType "NoteProperty" -Name RemoteDatabase -Value $User.RemoteDatabase
    $ExportEntry | Add-Member -MemberType "NoteProperty" -Name RemoteHostName -Value $User.RemoteHostName
    $ExportEntry | Add-Member -MemberType "NoteProperty" -Name Message -Value $User.Message
    $ExportEntry | Add-Member -MemberType "NoteProperty" -Name RecipientTypeDetails -Value $User.RecipientTypeDetails
    $ExportEntry | Add-Member -MemberType "NoteProperty" -Name TotalMailboxSize -Value $User.TotalMailboxSize
    $ExportEntry | Add-Member -MemberType "NoteProperty" -Name TotalMailboxItemCount -Value $User.TotalMailboxItemCount
    $ExportEntry | Add-Member -MemberType "NoteProperty" -Name TotalArchiveSize -Value $User.TotalArchiveSize
    $ExportEntry | Add-Member -MemberType "NoteProperty" -Name TotalArchiveItemCount -Value $User.TotalArchiveItemCount
    $ExportEntry | Add-Member -MemberType "NoteProperty" -Name TotalPrimarySize -Value $User.TotalPrimarySize
    $ExportEntry | Add-Member -MemberType "NoteProperty" -Name TotalPrimaryItemCount -Value $User.TotalPrimaryItemCount
    $ExportEntry | Add-Member -MemberType "NoteProperty" -Name BytesTransferred -Value $User.BytesTransferred

    $ExportEntry   
}

function generateFullMoveInformation ($User)
{
    $ExportEntry = New-Object PSObject
    $ExportEntry | Add-Member -MemberType "NoteProperty" -Name Identity -Value $User.Alias
    $ExportEntry | Add-Member -MemberType "NoteProperty" -Name BatchName -Value $User.Batchname
    $ExportEntry | Add-Member -MemberType "NoteProperty" -Name DisplayName -Value $User.DisplayName
    $ExportEntry | Add-Member -MemberType "NoteProperty" -Name Status -Value $User.Status
    $ExportEntry | Add-Member -MemberType "NoteProperty" -Name SyncStage -Value $User.SyncStage
    $ExportEntry | Add-Member -MemberType "NoteProperty" -Name RemoteDatabase -Value $User.RemoteDatabase
    $ExportEntry | Add-Member -MemberType "NoteProperty" -Name RemoteHostName -Value $User.RemoteHostName
    $ExportEntry | Add-Member -MemberType "NoteProperty" -Name Message -Value $User.Message
    $ExportEntry | Add-Member -MemberType "NoteProperty" -Name RecipientTypeDetails -Value $User.RecipientTypeDetails
    $ExportEntry | Add-Member -MemberType "NoteProperty" -Name TotalMailboxSize -Value $User.TotalMailboxSize
    $ExportEntry | Add-Member -MemberType "NoteProperty" -Name TotalMailboxItemCount -Value $User.TotalMailboxItemCount
    $ExportEntry | Add-Member -MemberType "NoteProperty" -Name TotalArchiveSize -Value $User.TotalArchiveSize
    $ExportEntry | Add-Member -MemberType "NoteProperty" -Name TotalArchiveItemCount -Value $User.TotalArchiveItemCount
    $ExportEntry | Add-Member -MemberType "NoteProperty" -Name TotalPrimarySize -Value $User.TotalPrimarySize
    $ExportEntry | Add-Member -MemberType "NoteProperty" -Name TotalPrimaryItemCount -Value $User.TotalPrimaryItemCount
    $ExportEntry | Add-Member -MemberType "NoteProperty" -Name BytesTransferred -Value $User.BytesTransferred
    $ExportEntry | Add-Member -MemberType "NoteProperty" -Name OverallDuration -Value $User.OverallDuration
    $ExportEntry | Add-Member -MemberType "NoteProperty" -Name TotalSuspendedDuration -Value $User.TotalSuspendedDuration
    $ExportEntry | Add-Member -MemberType "NoteProperty" -Name TotalFailedDuration -Value $User.TotalFailedDuration
    $ExportEntry | Add-Member -MemberType "NoteProperty" -Name TotalQueuedDuration -Value $User.TotalQueuedDuration
    $ExportEntry | Add-Member -MemberType "NoteProperty" -Name TotalInProgressDuration -Value $User.TotalInProgressDuration
    $ExportEntry | Add-Member -MemberType "NoteProperty" -Name StatusDetail -Value $User.StatusDetail
    $ExportEntry | Add-Member -MemberType "NoteProperty" -Name SourceVersion -Value $User.SourceVersion
    $ExportEntry | Add-Member -MemberType "NoteProperty" -Name FailureCode -Value $User.FailureCode
    $ExportEntry | Add-Member -MemberType "NoteProperty" -Name FailureType -Value $User.FailureType
    $ExportEntry | Add-Member -MemberType "NoteProperty" -Name FailureSide -Value $User.FailureSide
    $ExportEntry | Add-Member -MemberType "NoteProperty" -Name QueuedTimestamp -Value $User.QueuedTimestamp
    $ExportEntry | Add-Member -MemberType "NoteProperty" -Name StartTimestamp -Value $User.StartTimestamp
    $ExportEntry | Add-Member -MemberType "NoteProperty" -Name LastUpdateTimestamp -Value $User.LastUpdateTimestamp
    $ExportEntry | Add-Member -MemberType "NoteProperty" -Name LastSuccessfulSyncTimestamp -Value $User.LastSuccessfulSyncTimestamp
    $ExportEntry | Add-Member -MemberType "NoteProperty" -Name InitialSeedingCompletedTimestamp -Value $User.InitialSeedingCompletedTimestamp
    $ExportEntry | Add-Member -MemberType "NoteProperty" -Name TotalStalledDueToContentIndexingDuration -Value $User.TotalStalledDueToContentIndexingDuration
    $ExportEntry | Add-Member -MemberType "NoteProperty" -Name TotalStalledDueToMdbReplicationDuration -Value $User.TotalStalledDueToMdbReplicationDuration
    $ExportEntry | Add-Member -MemberType "NoteProperty" -Name TotalStalledDueToMailboxLockedDuration -Value $User.TotalStalledDueToMailboxLockedDuration
    $ExportEntry | Add-Member -MemberType "NoteProperty" -Name TotalStalledDueToReadThrottle -Value $User.TotalStalledDueToReadThrottle
    $ExportEntry | Add-Member -MemberType "NoteProperty" -Name TotalStalledDueToWriteThrottle -Value $User.TotalStalledDueToWriteThrottle
    $ExportEntry | Add-Member -MemberType "NoteProperty" -Name TotalStalledDueToReadCpu -Value $User.TotalStalledDueToReadCpu
    $ExportEntry | Add-Member -MemberType "NoteProperty" -Name TotalStalledDueToWriteCpu -Value $User.TotalStalledDueToWriteCpu
    $ExportEntry | Add-Member -MemberType "NoteProperty" -Name TotalStalledDueToReadUnknown -Value $User.TotalStalledDueToReadUnknown
    $ExportEntry | Add-Member -MemberType "NoteProperty" -Name TotalStalledDueToWriteUnknown -Value $User.TotalStalledDueToWriteUnknown
    $ExportEntry | Add-Member -MemberType "NoteProperty" -Name TotalTransientFailureDuration -Value $User.TotalTransientFailureDuration

    
    $ExportEntry   
}

function GenerateExcelInformation ($BadItems,$User)
{
    $ExportEntry = New-Object PSObject
    $ExportEntry | Add-Member -MemberType "NoteProperty" -Name Identity -Value $User.Alias
    $ExportEntry | Add-Member -MemberType "NoteProperty" -Name BatchName -Value $User.Batchname
    $ExportEntry | Add-Member -MemberType "NoteProperty" -Name DisplayName -Value $User.DisplayName
    $ExportEntry | Add-Member -MemberType "NoteProperty" -Name Status -Value $User.Status
    $ExportEntry | Add-Member -MemberType "NoteProperty" -Name SyncStage -Value $User.SyncStage
    $ExportEntry | Add-Member -MemberType "NoteProperty" -Name RemoteDatabase -Value $User.RemoteDatabase
    $ExportEntry | Add-Member -MemberType "NoteProperty" -Name RemoteHostName -Value $User.RemoteHostName
    $ExportEntry | Add-Member -MemberType "NoteProperty" -Name Message -Value $User.Message
    $ExportEntry | Add-Member -MemberType "NoteProperty" -Name RecipientTypeDetails -Value $User.RecipientTypeDetails
    $ExportEntry | Add-Member -MemberType "NoteProperty" -Name TotalMailboxSize -Value $User.TotalMailboxSize
    $ExportEntry | Add-Member -MemberType "NoteProperty" -Name TotalMailboxItemCount -Value $User.TotalMailboxItemCount
    $ExportEntry | Add-Member -MemberType "NoteProperty" -Name BadItems -Value $BadItems.Count
    $ExportEntry | Add-Member -MemberType "NoteProperty" -Name BytesTransferred -Value $User.BytesTransferred
    
    $ExportEntry   
}


#endregion

#region Global Vales
$LogPath = "C:\Scripts\MRS\Reports\"
$FileTimeStamp = (Get-Date -Format yyyyMMdd-HHmmss).ToString()
$ExportBadItems = @()
$ExportBasicInformation = @()
$ExportFullInformation = @()
$ExportExcelInfo = @()
$Moves = @()
#endregion


#region Collect Movedata

If($AllMoves){
    $Moves = Get-MoveRequest | Get-MoveRequestStatistics -IncludeReport
}

If($CompletedMoves){
    $Moves = Get-MoveRequest -MoveStatus Completed | Get-MoveRequestStatistics -IncludeReport
}

If($Synced){
    $Moves += Get-MoveRequest -MoveStatus Synced | Get-MoveRequestStatistics -IncludeReport
}

If($Inprogress){
    $Moves += Get-MoveRequest -MoveStatus Inprogress | Get-MoveRequestStatistics -IncludeReport
}

#endregion

#region Process moves Data

If($IncludeAllBadItems){

    Foreach ($Move in $Moves){
        $Baditems = ""
        $Baditems = ($Move.Report).Baditems
          
            If($Baditems -eq ""){
                Write-Host "No Errors found for " -BackgroundColor Green
            }
                Else{
                    Foreach($BadItem in $Baditems){
                        $ExportBadItems += generateBadItemInformation -BadItem $Baditem -User $Move
                    }
                }
    }

}

If($GenereExportInfo){

    Foreach ($Move in $Moves){
        $Baditems = ""
        $Baditems = ($Move.Report).Baditems
          
            If($Baditems -eq ""){
                Write-Host "No Errors found for " -BackgroundColor Green
            }
            Else{
                $ExportExcelInfo += GenerateExcelInformation -BadItems $Baditems -User $Move
            }
    }

}



If($MoveDataBasic){

     Foreach ($Move in $Moves){
        $ExportBasicInformation += generateBasicMoveInformation -User $Move
     }           
}


If($MoveDataFull){

    Foreach ($Move in $Moves){
        $ExportFullInformation += generateFullMoveInformation -User $Move           
    }
}

#endregion

#region Export data
If($IncludeAllBadItems){
    $ExportBadItems | ogv
    $FileName = $LogPath+"BadItems"+$FileTimeStamp+".csv"
    $ExportBadItems | Export-Csv -Path $FileName -Delimiter ";" -NoTypeInformation
}

If($MoveDataBasic){
    $ExportBasicInformation | ogv
    $FileName = $LogPath+"BasicMoveData"+$FileTimeStamp+".csv"
    $ExportBasicInformation |Export-Csv -Path $FileName -Delimiter ";" -NoTypeInformation
}

If($MoveDataFull){
    $ExportFullInformation | ogv
    $FileName = $LogPath+"FullMoveData"+$FileTimeStamp+".csv"
    $ExportFullInformation | Export-Csv -Path $FileName -Delimiter ";" -NoTypeInformation
}


if($ExportExcelInfo){
    $ExportExcelInfo | ogv
    $FileName = $LogPath+"ExcelData"+$FileTimeStamp+".csv"
    $ExportExcelInfo | Export-Csv -Path $FileName -Delimiter ";" -NoTypeInformation
}

#endregion

I hope this is also useful for your migrations and have fun with the script. As time goes and report demands change. I will update the script on this blog to keep you up-to-date.