Field Notes: DKIM and missing selector records

Reading Time: 3 minutes

During a project with one of my customers, I was tasked to look at a non-delivery report (NDR) for a mail message. The bounce error was pretty confusing, but after reviewing the headers, we noticed that the DKIM check had failed. This was a bit of a surprise, because the message was sent from Microsoft Exchange Online. The root cause was that the selector mentioned in de header wasn’t resolving. Apparently, Microsoft did not provide the answer to validate the record.

The situation

The organization uses Microsoft Exchange Online and has setup DKIM signing with the accepted domains listed in the tenant. This particular organization owns 70+ domains. For every accepted domain, they have created the required CNAME records. During the validation, roughly 20 domains had a failed active selector entry.

I raised a case with Microsoft Office 365 Support, which confirmed that is was a bug and resolved the issue for my customer. They also informed the product team of this error.

How to setup DKIM

By default Exchange Online signs outgoing message with a DKIM signature. More information can be found here. The validation information recorded in the header points to one of the following records:

  • selector1-[tenantname]-onmicrosoft-com._domainkey.[tenantname].onmicrosoft.com
  • selector2-[tenantname]-onmicrosoft-com._domainkey.[tenantname].onmicrosoft.com

image

To enable DKIM you will also need to setup the following CNAME records in the domain:

  • selector1._domainkey
  • selector2._domainkey

Using Windows PowerShell, you can get the full value to use for these DNS records. You can retrieve the information through the Exchange Online PowerShell module. The recommended administrative permission is Security Admin.

Follow these steps:

  1. Connect to Exchange Online using the Exchange Online Powershell module.
  2. Sign in with your cloud administrator credentials.
  3. Give the following command for all existing domains:
    Get-DkimSigningConfig | FL Identity,Selector1CNAME,Selector2CNAME

    image

  4. Give the following command for one specific domain:
    Get-DkimSigningConfig –identity [domainname]| Format-List Identity,Selector1CNAME,Selector2CNAME

    image

  5. Register the new CNAME records with the Selector[1/2]Cname as value.
  6. After succesfully creating the DNS records, you can activate the DKIM signing configuration. This can be done via the following command:
    Set-DkimSigningConfig –identity [domainname] –Enabled:$true

Note: If the domain is missing, you can create the DkimSigningConfig entry through this powershell command:

New-DkimSigningConfig -KeySize 2048 –DomainName [domainname] –Enabled:$false

Only one selector returns data

With the initial creation of your DKIM configuration, Microsoft populates both selectors in their DNS. After a while, though, only the active selector will be resolvable and the other selector is not. This is by Microsoft's design. The record should be visible again when the key rotates.

How to validate the existing records

The challenge for my customer was to validate the records for all 70+ domains. We can do this manually, but where it the fun in that!? We decided to do it through PowerShell.

The below Windows PowerShell script requires an active session to Exchange Online through PowerShell. The recommended administrative permissions is Security Admin.

We ran the following script:

$domains = Get-DkimSigningConfig |Where-Object {$_.Enabled -like "True"}
$Datenow = Get-date
$FailedLookupslog = "C:\Temp\FailedDKIMrecords.log"
$CheckDNS = @()
$failed = @()


ForEach($domain in $domains){

    $activeSelector1 = $domain.SelectorBeforeRotateOnDate
    $activeSelector2 = $domain.SelectorAfterRotateOnDate


    If($($domain.RotateOnDate.ToUniversalTime()) -gt $($datenow.ToUniversalTime()) ){
         $tempattribute = "$($activeSelector1)Cname"
         $CheckDNS += $domain."$tempattribute"
     }
     else{
         $tempattribute = "$($activeSelector2)Cname"
         $CheckDNS += $domain."$tempattribute"
     }
}


foreach($entry in $CheckDNS){
     $works = Resolve-DnsName -Name $entry -Type txt -erroraction 'silentlycontinue'


     If($works){}
     else{
         Write-host "$entry is not working" -ForegroundColor yellow
         $Failed += $entry
     }

     $works = ""
}

"Domain: , selector:" | Out-File -FilePath $FailedLookupslog

Foreach($fail in $failed){

    if($fail -like "selector1*"){
         $temp = $domains | where{$_.selector1Cname -like $fail}
     }
     else{
    $temp = $domains | where{$_.selector2Cname -like $fail}

   }
    "$($temp.Domain) , $fail" | Out-File -FilePath $FailedLookupslog -Append

}

The failed active selector entries were listed in the logfile.

Concluding

Organizations that enable the DKIM feature rely on Microsoft to maintain the information and keep it up-to-date.  As a consumer of this feature you need to trust, but verify that it’s still works as designed.

By using the script in the blog, you can automate, verify and let it report to your ticketing system for review and follow-up actions.

Security Officer: Please block the iOS native mail app (for) now!

Reading Time: 5 minutesLast week an announcement was made: The native mail app in Apple's iOS has zero-day vulnerabilities, deemed critical. No patch is available at this time.

More information about the vulnerability can be found here.

For you as IT admin this means that you probably have work to do. The main questions you may be facing from management or your security officer are:

  1. How many people use the native mail app in our organization?
  2. Can you block access to people using the native mail app?

I'll answer these two questions in this blog post, for the on-premises environment and for Office 365.

How many people use the native iOS mail app?

To gain insights into the current usage of the native mail app, we need to use Windows PowerShell.

Note:
Although the official support for Exchange Server 2010 ends October 2020, I 'm also adding the commands for Exchange Server 2010 to this blogpost. Organizations are (unfortunately) still using it.

To execute the commands you will need to be connected through the Exchange Management Shell. This applies to all versions of Exchange. The minimum permissions needed are Recipient Administrator.

Query Microsoft Exchange Server 2010

For Micrososft Exchange Server 2010, we need the Get-ActiveSyncDevice and Get-ActiveSyncDeviceStatistics Windows PowerShell cmdlets.

Use the below lines of Windows PowerShell to query Microsoft Exchange Server 2010 for usage of the native iOS mail app:

$AlliOSDevices = Get-ActiveSyncDevice -Filter {(DeviceOs -like "iOS*") -and (ClientType -eq "EAS")}

$QueryResults = $AlliOSDevices | %{Get-ActiveSyncDeviceStatistics -Identity $_.Identity} | Sort-Object -Descending -Property LastSuccessSync

$QueryResults | Export-Csv C:\temp\iOSDevicesResult.csv -Delimiter ","

$QueryResults | Out-GridView

image

Note: the warning shown in the example, is caused by the fact that for Microsoft Exchange Server 2013 and higher a different command is recommended. See Query Microsoft Exchange Server 2013 or Later for more information.

The results are now available on the C:\temp folder on the server. Load the results in Microsoft Excel and the last successful synchronization is shown on top.

Query Microsoft Exchange Server 2013, or later, and Microsoft Exchange Online

For Microsoft Exchange Server 2013, newer versions of Microsoft Exchange Server and for Microsoft Exchange Online (part of Microsoft Office 365 and Microsoft 365), we need the Get-MobileDevice and Get-MobileDeviceStatistics Windows PowerShell Cmdlets.

Use the below lines of Windows PowerShell to query Microsoft Exchange Server 2013, and later, for usage of the native iOS mail app:

$AlliOSDevices = Get-MobileDevice -Filter {(DeviceOs -like "iOS*") -and (ClientType -eq "EAS")}

$QueryResults = $AlliOSDevices | %{Get-MobileDeviceStatistics -Identity $_.Identity} | Sort-Object -Descending -Property LastSuccessSync

$QueryResults | Export-Csv C:\temp\iOSDevicesResult.csv -Delimiter ","

$QueryResults | Out-GridView

image

The results are now available on the C:\temp folder on the server. Load the results in Microsoft Excel and the last successful synchronization is shown on top.

How to block the iOS native mail app

Using the previous scripts, we known who is using the native iOS mail app. The service desk and/or security officer can use direct communications to these employees. Their message should be that use of the native mail app is (about to be) disabled. The employees should configure the Microsoft Outlook app for iOS to regain access to e-mail on their mobile devices.

Block Access

To create the block rules for iPad and iOS, we are using Windows PowerShell with an active management connection to the Microsoft Exchange Server Environment. We recommend to be be logged in as Organization Administrator of the Exchange Server environment.

In Exchange Online you'll need to have the Exchange Administrator role. The Windows PowerShell cmdlet that we are going to use is New-ActiveSyncDeviceAccessRule.

To block access for iPad and iPhones, please use the following lines of Windows PowerShell:

New-ActiveSyncDeviceAccessRule -Characteristic DeviceType -QueryString "iPhone" -AccessLevel Block

New-ActiveSyncDeviceAccessRule -Characteristic DeviceType -QueryString "iPad" -AccessLevel Block

image

Remove block rule

To rollback or remove the rules, you can use the following lines of Windows PowerShell from the Exchange Management connection. The Windows PowerShell cmdlet we are going to use is: Remove-ActiveSyncDeviceAccessRule.

Remove-ActiveSyncDeviceAccessRule "iPhone (DeviceType)" -confirm:$false

Remove-ActiveSyncDeviceAccessRule "iPad (DeviceType)" -confirm:$false

image

Is it working?

After applying the policy, it is a recommended practice to review mobile devices. In the past, a device may be manually allowed and will continue to function after activating the block rules.

To gain insight in the block status and the excluded devices, we are going to use the same Windows PowerShell cmdlets as before. This time, we're adding an additional where filter: We are going to filter out the ‘DeviceAccessStateReason’ with the global value. These devices will be automatically blocked. Below, I'll show the example for Microsoft Exchange Server 2010 and for Microsoft Exchange Server 2013 and later.

Is it working on Microsoft Exchange Server 2010?

Use the following lines of Windows PowerShell to figure out if the block rule is working on Microsoft Exchange Server 2010:

$AlliOSDevices = Get-Activesyncdevice -Filter {(DeviceOs -like "iOS*") -and (ClientType -eq "EAS")}

$QueryResults = $AlliOSDevices | %{Get-ActivesyncDeviceStatistics -Identity $_.Identity} | where {$_.DeviceAccessStateReason -ne "Global" }| Sort-Object -Descending -Property LastSuccessSync

$QueryResults | Export-Csv C:\temp\iOSDevicesBlockedAllowed.csv -Delimiter ","

$QueryResults | Out-GridView

image

Is it working on Microsoft Exchange Server 2013, or later, and Microsoft Exchange Online?

Use the following lines of Windows PowerShell to figure out if the block rule is working on Microsoft Exchange Server 2013, newer versions of Microsoft Exchange Server and for Microsoft Exchange Online (part of Microsoft Office 365 and Microsoft 365), :

$AlliOSDevices = Get-MobileDevice -Filter {(DeviceOs -like "iOS*") -and (ClientType -eq "EAS")}

$QueryResults = $AlliOSDevices | %{Get-MobileDeviceStatistics -Identity $_.Identity} |where {$_.DeviceAccessStateReason -ne "Global" }| Sort-Object -Descending -Property LastSuccessSync

$QueryResults | Export-Csv C:\temp\iOSDevicesBlockedAllowed.csv -Delimiter ","

$QueryResults | Out-GridView

image

Concluding

Don't panic when you get difficult questions from management and\or the security officer. The answer is Yes, thanks to Windows PowerShell.

We can see who is using the native app and we can block access. With a few lines of Windows PowerShell we gain insight and take control.

I hope that the above information is helpful to you in regaining security, until the zero day in the iOS native mail app is fixed.

Valimail Monitor for Office 365: Your Free DMARC Reporting Tool

Reading Time: 3 minutesOn their security blog on the 3rd of June 2019, Microsoft announced that Valimail Monitor for Office 365 is available. This option enables organizations using Exchange Online from Office 365 for their company mail to leverage DMARC.

The Road to securing E-Mail

Cyberattacks are common these days. These attacks can be actively targeting your organization over the internet or through incoming emails.

Reputation of your name and mail on the internet are important these days. Reputation attacks via email are achieved by spoofing; sending e-mail messages on behalf of your domain. To counter this, you can:

  • Enable SPF (Sender Policy Framework) records, and;
  • Enable DKIM (DomainKeys Identified Mail)

This is a common practice. However, after you have enabled this, you don't get any feedback about the attacks or invalid sources. To gain this insight you will need to activate DMARC (Domain-based Message Authentication, Reporting and Conformance). After you’ve enabled DMARC, via a simple DNS TXT record, you will start receiving automated mail messages with an XML file as attachment on the e-mail address listed in the TXT record.

In short: if you want to gain the insight, who is using your domain on the internet, start using DMARC.

Valimail to the rescue!

With Microsoft’s announcement, you get access to Valimail; a free tool to gain these insights.

Stop processing the XML files by hand or scripting tools. We all love (free) automation, right?

Requirements

To gain access to this information, you already need to have setup the following:

    • Existing SPF record containing all the authoritative mail sources
    • Enable DKIM on your mail flow (activated by default in Office 365).
      For outgoing mail, a transport agent can be installed on the on-premises Exchange Server or activated as an option on your anti-spam solution.
    • Activated a basic DMARC record in your DNS domain, for example:
      "v=DMARC1; p=none; rua=mailto:reports@example.com".
      This example shows you’re using DMARC1 and you monitor existing connections. Please report findings to reports@example.com.

How to set it up

Follow these steps to set it up:

  • First go to the following website: https://go.valimail.com/microsoft.html
  • Fill in the required information.
    image
  • Now wait for response from the Office 365 team of Valimail.
    image
  • Update your DNS record with the requested entry and test the record.
    image
  • Wait for your initial invite to create a login account.
    image
  • When the invite is sent, accept the invite and configure a password for your account.
    image

Tip!
Don’t forget to enable 2-factor authentication on your account or configure Azure AD single sign-on as described in Enable Valimail Single Sign-On with Azure Active Directory.

Conclusion

I have discussed DMARC before with customers and it's a valuable option to gain insight who is sending e-mail messages on behalf of your DNS domain. The only problem was, how to translate the XML files.

Yes, other tools are available, but for most, you will need to pay a fee to use.

ValiMail is free for organizations using Office 365, so why not use it? Regain control over your mail domain, today.

Further reading

Below are some articles that explain SPF, DKIM and DMARC in more depth:

Exchange migration “Couldn’t switch the mailbox into Sync Source mode”

Reading Time: 3 minutesExchange

The issue

During one of my Exchange Online migration projects I encountered the following error on several mailboxes:

"Message : Error: Couldn’t switch the mailbox into Sync Source mode.
This could be because of one of the following reasons:
Another administrator is currently moving the mailbox.
The mailbox is locked.
The Microsoft Exchange Mailbox Replication service (MRS) doesn’t have the correct permissions.
Network errors are preventing MRS from cleanly closing its session with the Mailbox server.
If this is the case, MRS may continue to encounter this error for up to 2 hours – this duration is controlled by the TCP KeepAlive settings on the Mailbox server.
Wait for the mailbox to be released before attempting to move this mailbox again."

The environment where I encountered this was an Exchange Server 2010 environment in a Database Availability Group (DAG) configuration. There where three DAG servers and two Client Access servers.
We placed two Exchange Server 2016 servers in front for the Hybrid connection with Exchange Online.

The cause

This was a uncommon error for me, so I did some research first before proceeding with the suggested action to alter the TCP settings and request that the entire Exchange Server environment is restarted.

I found several blogs with different solutions, rather then changing the TCP value. One of the suggestions was to run the mailbox repair option. I did this without success. I also tried an internal move request of the mailbox and it failed on the same error.

After reading a blog by Brad Hughes on the topic, I found a interesting remark about the cause of the error.

"When moving a mailbox, the Mailbox Replication Service (MRS) sets an "InTransitStatus" flag in the source mailbox to make sure other moves don't try to act on this source mailbox at the same time.  This flag is really just held in memory in the source Information Store (Store) process (Store.exe for 2010 and Microsoft.Exchange.Store.Worker.exe for 2013 and 2016). "

The solution for me

So my conclusion on this was:

If it's held in memory, what options do I have to 'reset' this, without restarting the servers?

Because the source mailbox databases were placed in a DAG, I asked the customer to failover the mailbox database, which contained the error user. After the failover (which caused downtime for less then a minute), I recreated the move request again and this time the mailbox was synced to Office 365.

I find this solution less disrupting and quicker than a reboot of an entire Exchange Server.

The initial error was for a primary mailbox, but this error can also occur when a user has an in-place archive. In this case, for me, it was also enough to dismount and mount the mailbox database.
Yes, you read it correctly; dismount and mount the mailbox database. The archive databases weren't part of the DAG configuration. This action also triggers a memory reset.

This is not an error that happens when you start a migration; I also got it five times during synchronization of mailboxes. The solution here was the same, failover or dismount/mount the mailbox database.

Conclusion

I would suggest to not always go blindly with the Microsoft suggestion for the fix. Investigate the root cause is of the problem.
Yes, the Microsoft suggestion would have fixed it, but for the suggested fix to work, you need to restart all Exchange servers. This will cause a failover or dismount/mount action of the mailbox database.

For me it was just enough to just failover or dismount/mount the mailbox database(s) to get the synchronization starting or continuing. Without disrupting the rest of the organization, and aiding me in meeting my deadlines as a consultant.

I hope this helps for you, too.

TOOL: My own Exchange Move Request Report script

Reading Time: 5 minutesAs a consultant I do a lot of migrations or assist in them. In this case I wrote my own script for processing the information generated by the Mailbox Replication Service. This service is used in Microsoft Exchange on-premises and in the cloud for migrating the mailboxes between databases or environments.

After multiple times doing the administration and processing of information manually. I decided to create my own processing script to retrieve the desired information and make it re-usable in Excel or other tooling, to export gathered information to CSV file format. I named the script Get Move Request Report script.

The script is provide as-is and maybe be used at your own risk.

Versions

Version 1: This is the first version and it’s basic. No advanced switch options or logging.

About the Exchange Move Request Report Script

The script consist of three functions:

1. Creates an overview of the Bad Items it found in the move request report.
2. Creates an overview of the basic move information of the move request statistics.
3. Creates an overview of the extended information of the move request statistics.
4. Creates an overview of information that quickly can be used for reports to stakeholders.

Parameters

When you execute the script and no parameters are given. It generates a bad items overview, basic move information and extended move information output of all existing move request in scope.

.\Get-MoveRequestReport.ps1

To generate only the outputs for all InProgress move requests, give the following command:

.\Get-MoveRequestReport.ps1 -Inprogress $true -AllMoves $false

To generate only the outputs for all Synced move requests, give the following command:

.\Get-MoveRequestReport.ps1 -Synced $true -AllMoves $false

To generate only the outputs for all Completed move requests, give the following command:

.\Get-MoveRequestReport.ps1 -Completed $true -AllMoves $false

Requirements

To use the script, you already need to have an Exchange PowerShell session open to the target environment (on-premises or cloud), where the move request are created.

Exchange Move Request Report Script

The script is displayed below for your review:

[CmdletBinding()]
param (
    [bool]$CompletedMoves = $false, 
    [bool]$Synced = $false,    
    [bool]$Inprogress = $false,
    [bool]$AllMoves = $true,
    [bool]$IncludeAllBadItems = $true,
    [bool]$MoveDataBasic = $true,
    [bool]$MoveDataFull = $true,
    [bool]$GenereExportInfo = $true
)

#region Functions
function generateBadItemInformation ($BadItem,$User)
{
    $ExportEntry = New-Object PSObject
    $ExportEntry | Add-Member -MemberType "NoteProperty" -Name Identity -Value $User.Alias
    $ExportEntry | Add-Member -MemberType "NoteProperty" -Name BatchName -Value $User.Batchname
    $ExportEntry | Add-Member -MemberType "NoteProperty" -Name DisplayName -Value $User.DisplayName
    $ExportEntry | Add-Member -MemberType "NoteProperty" -Name Date -Value $BadItem.Date
    $ExportEntry | Add-Member -MemberType "NoteProperty" -Name Subject -Value $BadItem.Subject
    $ExportEntry | Add-Member -MemberType "NoteProperty" -Name Kind -Value $BadItem.Kind
    $ExportEntry | Add-Member -MemberType "NoteProperty" -Name FolderName -Value $BadItem.FolderName
    
    
    $ExportEntry   
}

function generateBasicMoveInformation ($User)
{
    $ExportEntry = New-Object PSObject
    $ExportEntry | Add-Member -MemberType "NoteProperty" -Name Identity -Value $User.Alias
    $ExportEntry | Add-Member -MemberType "NoteProperty" -Name BatchName -Value $User.Batchname
    $ExportEntry | Add-Member -MemberType "NoteProperty" -Name DisplayName -Value $User.DisplayName
    $ExportEntry | Add-Member -MemberType "NoteProperty" -Name Status -Value $User.Status
    $ExportEntry | Add-Member -MemberType "NoteProperty" -Name SyncStage -Value $User.SyncStage
    $ExportEntry | Add-Member -MemberType "NoteProperty" -Name RemoteDatabase -Value $User.RemoteDatabase
    $ExportEntry | Add-Member -MemberType "NoteProperty" -Name RemoteHostName -Value $User.RemoteHostName
    $ExportEntry | Add-Member -MemberType "NoteProperty" -Name Message -Value $User.Message
    $ExportEntry | Add-Member -MemberType "NoteProperty" -Name RecipientTypeDetails -Value $User.RecipientTypeDetails
    $ExportEntry | Add-Member -MemberType "NoteProperty" -Name TotalMailboxSize -Value $User.TotalMailboxSize
    $ExportEntry | Add-Member -MemberType "NoteProperty" -Name TotalMailboxItemCount -Value $User.TotalMailboxItemCount
    $ExportEntry | Add-Member -MemberType "NoteProperty" -Name TotalArchiveSize -Value $User.TotalArchiveSize
    $ExportEntry | Add-Member -MemberType "NoteProperty" -Name TotalArchiveItemCount -Value $User.TotalArchiveItemCount
    $ExportEntry | Add-Member -MemberType "NoteProperty" -Name TotalPrimarySize -Value $User.TotalPrimarySize
    $ExportEntry | Add-Member -MemberType "NoteProperty" -Name TotalPrimaryItemCount -Value $User.TotalPrimaryItemCount
    $ExportEntry | Add-Member -MemberType "NoteProperty" -Name BytesTransferred -Value $User.BytesTransferred

    $ExportEntry   
}

function generateFullMoveInformation ($User)
{
    $ExportEntry = New-Object PSObject
    $ExportEntry | Add-Member -MemberType "NoteProperty" -Name Identity -Value $User.Alias
    $ExportEntry | Add-Member -MemberType "NoteProperty" -Name BatchName -Value $User.Batchname
    $ExportEntry | Add-Member -MemberType "NoteProperty" -Name DisplayName -Value $User.DisplayName
    $ExportEntry | Add-Member -MemberType "NoteProperty" -Name Status -Value $User.Status
    $ExportEntry | Add-Member -MemberType "NoteProperty" -Name SyncStage -Value $User.SyncStage
    $ExportEntry | Add-Member -MemberType "NoteProperty" -Name RemoteDatabase -Value $User.RemoteDatabase
    $ExportEntry | Add-Member -MemberType "NoteProperty" -Name RemoteHostName -Value $User.RemoteHostName
    $ExportEntry | Add-Member -MemberType "NoteProperty" -Name Message -Value $User.Message
    $ExportEntry | Add-Member -MemberType "NoteProperty" -Name RecipientTypeDetails -Value $User.RecipientTypeDetails
    $ExportEntry | Add-Member -MemberType "NoteProperty" -Name TotalMailboxSize -Value $User.TotalMailboxSize
    $ExportEntry | Add-Member -MemberType "NoteProperty" -Name TotalMailboxItemCount -Value $User.TotalMailboxItemCount
    $ExportEntry | Add-Member -MemberType "NoteProperty" -Name TotalArchiveSize -Value $User.TotalArchiveSize
    $ExportEntry | Add-Member -MemberType "NoteProperty" -Name TotalArchiveItemCount -Value $User.TotalArchiveItemCount
    $ExportEntry | Add-Member -MemberType "NoteProperty" -Name TotalPrimarySize -Value $User.TotalPrimarySize
    $ExportEntry | Add-Member -MemberType "NoteProperty" -Name TotalPrimaryItemCount -Value $User.TotalPrimaryItemCount
    $ExportEntry | Add-Member -MemberType "NoteProperty" -Name BytesTransferred -Value $User.BytesTransferred
    $ExportEntry | Add-Member -MemberType "NoteProperty" -Name OverallDuration -Value $User.OverallDuration
    $ExportEntry | Add-Member -MemberType "NoteProperty" -Name TotalSuspendedDuration -Value $User.TotalSuspendedDuration
    $ExportEntry | Add-Member -MemberType "NoteProperty" -Name TotalFailedDuration -Value $User.TotalFailedDuration
    $ExportEntry | Add-Member -MemberType "NoteProperty" -Name TotalQueuedDuration -Value $User.TotalQueuedDuration
    $ExportEntry | Add-Member -MemberType "NoteProperty" -Name TotalInProgressDuration -Value $User.TotalInProgressDuration
    $ExportEntry | Add-Member -MemberType "NoteProperty" -Name StatusDetail -Value $User.StatusDetail
    $ExportEntry | Add-Member -MemberType "NoteProperty" -Name SourceVersion -Value $User.SourceVersion
    $ExportEntry | Add-Member -MemberType "NoteProperty" -Name FailureCode -Value $User.FailureCode
    $ExportEntry | Add-Member -MemberType "NoteProperty" -Name FailureType -Value $User.FailureType
    $ExportEntry | Add-Member -MemberType "NoteProperty" -Name FailureSide -Value $User.FailureSide
    $ExportEntry | Add-Member -MemberType "NoteProperty" -Name QueuedTimestamp -Value $User.QueuedTimestamp
    $ExportEntry | Add-Member -MemberType "NoteProperty" -Name StartTimestamp -Value $User.StartTimestamp
    $ExportEntry | Add-Member -MemberType "NoteProperty" -Name LastUpdateTimestamp -Value $User.LastUpdateTimestamp
    $ExportEntry | Add-Member -MemberType "NoteProperty" -Name LastSuccessfulSyncTimestamp -Value $User.LastSuccessfulSyncTimestamp
    $ExportEntry | Add-Member -MemberType "NoteProperty" -Name InitialSeedingCompletedTimestamp -Value $User.InitialSeedingCompletedTimestamp
    $ExportEntry | Add-Member -MemberType "NoteProperty" -Name TotalStalledDueToContentIndexingDuration -Value $User.TotalStalledDueToContentIndexingDuration
    $ExportEntry | Add-Member -MemberType "NoteProperty" -Name TotalStalledDueToMdbReplicationDuration -Value $User.TotalStalledDueToMdbReplicationDuration
    $ExportEntry | Add-Member -MemberType "NoteProperty" -Name TotalStalledDueToMailboxLockedDuration -Value $User.TotalStalledDueToMailboxLockedDuration
    $ExportEntry | Add-Member -MemberType "NoteProperty" -Name TotalStalledDueToReadThrottle -Value $User.TotalStalledDueToReadThrottle
    $ExportEntry | Add-Member -MemberType "NoteProperty" -Name TotalStalledDueToWriteThrottle -Value $User.TotalStalledDueToWriteThrottle
    $ExportEntry | Add-Member -MemberType "NoteProperty" -Name TotalStalledDueToReadCpu -Value $User.TotalStalledDueToReadCpu
    $ExportEntry | Add-Member -MemberType "NoteProperty" -Name TotalStalledDueToWriteCpu -Value $User.TotalStalledDueToWriteCpu
    $ExportEntry | Add-Member -MemberType "NoteProperty" -Name TotalStalledDueToReadUnknown -Value $User.TotalStalledDueToReadUnknown
    $ExportEntry | Add-Member -MemberType "NoteProperty" -Name TotalStalledDueToWriteUnknown -Value $User.TotalStalledDueToWriteUnknown
    $ExportEntry | Add-Member -MemberType "NoteProperty" -Name TotalTransientFailureDuration -Value $User.TotalTransientFailureDuration

    
    $ExportEntry   
}

function GenerateExcelInformation ($BadItems,$User)
{
    $ExportEntry = New-Object PSObject
    $ExportEntry | Add-Member -MemberType "NoteProperty" -Name Identity -Value $User.Alias
    $ExportEntry | Add-Member -MemberType "NoteProperty" -Name BatchName -Value $User.Batchname
    $ExportEntry | Add-Member -MemberType "NoteProperty" -Name DisplayName -Value $User.DisplayName
    $ExportEntry | Add-Member -MemberType "NoteProperty" -Name Status -Value $User.Status
    $ExportEntry | Add-Member -MemberType "NoteProperty" -Name SyncStage -Value $User.SyncStage
    $ExportEntry | Add-Member -MemberType "NoteProperty" -Name RemoteDatabase -Value $User.RemoteDatabase
    $ExportEntry | Add-Member -MemberType "NoteProperty" -Name RemoteHostName -Value $User.RemoteHostName
    $ExportEntry | Add-Member -MemberType "NoteProperty" -Name Message -Value $User.Message
    $ExportEntry | Add-Member -MemberType "NoteProperty" -Name RecipientTypeDetails -Value $User.RecipientTypeDetails
    $ExportEntry | Add-Member -MemberType "NoteProperty" -Name TotalMailboxSize -Value $User.TotalMailboxSize
    $ExportEntry | Add-Member -MemberType "NoteProperty" -Name TotalMailboxItemCount -Value $User.TotalMailboxItemCount
    $ExportEntry | Add-Member -MemberType "NoteProperty" -Name BadItems -Value $BadItems.Count
    $ExportEntry | Add-Member -MemberType "NoteProperty" -Name BytesTransferred -Value $User.BytesTransferred
    
    $ExportEntry   
}


#endregion

#region Global Vales
$LogPath = "C:\Scripts\MRS\Reports\"
$FileTimeStamp = (Get-Date -Format yyyyMMdd-HHmmss).ToString()
$ExportBadItems = @()
$ExportBasicInformation = @()
$ExportFullInformation = @()
$ExportExcelInfo = @()
$Moves = @()
#endregion


#region Collect Movedata

If($AllMoves){
    $Moves = Get-MoveRequest | Get-MoveRequestStatistics -IncludeReport
}

If($CompletedMoves){
    $Moves = Get-MoveRequest -MoveStatus Completed | Get-MoveRequestStatistics
}

If($Synced){
    $Moves += Get-MoveRequest -MoveStatus Synced | Get-MoveRequestStatistics
}

If($Inprogress){
    $Moves += Get-MoveRequest -MoveStatus Inprogress | Get-MoveRequestStatistics
}

#endregion

#region Process moves Data

If($IncludeAllBadItems){

    $Moves2 = get-migrationuser | where{$_.SkippedItemCount -gt 0} | Get-MoveRequestStatistics -IncludeReport
    Foreach ($Move in $Moves2){
        $Baditems = ""
        $Baditems = ($Move.Report).Baditems
          
            If($Baditems -eq ""){
                Write-Host "No Errors found for " -BackgroundColor Green
            }
                Else{
                    Foreach($BadItem in $Baditems){
                        $ExportBadItems += generateBadItemInformation -BadItem $Baditem -User $Move
                    }
                }
    }

}

If($GenereExportInfo){

    Foreach ($Move in $Moves){
        $Baditems = ""
        $Baditems = ($Move.Report).Baditems
          
            If($Baditems -eq ""){
                Write-Host "No Errors found for " -BackgroundColor Green
            }
            Else{
                $ExportExcelInfo += GenerateExcelInformation -BadItems $Baditems -User $Move
            }
    }

}



If($MoveDataBasic){

     Foreach ($Move in $Moves){
        $ExportBasicInformation += generateBasicMoveInformation -User $Move
     }           
}


If($MoveDataFull){

    Foreach ($Move in $Moves){
        $ExportFullInformation += generateFullMoveInformation -User $Move           
    }
}

#endregion

#region Export data
If($IncludeAllBadItems){
    $ExportBadItems | ogv
    $FileName = $LogPath+"BadItems"+$FileTimeStamp+".csv"
    $ExportBadItems | Export-Csv -Path $FileName -Delimiter ";" -NoTypeInformation
}

If($MoveDataBasic){
    $ExportBasicInformation | ogv
    $FileName = $LogPath+"BasicMoveData"+$FileTimeStamp+".csv"
    $ExportBasicInformation |Export-Csv -Path $FileName -Delimiter ";" -NoTypeInformation
}

If($MoveDataFull){
    $ExportFullInformation | ogv
    $FileName = $LogPath+"FullMoveData"+$FileTimeStamp+".csv"
    $ExportFullInformation | Export-Csv -Path $FileName -Delimiter ";" -NoTypeInformation
}


if($ExportExcelInfo){
    $ExportExcelInfo | ogv
    $FileName = $LogPath+"ExcelData"+$FileTimeStamp+".csv"
    $ExportExcelInfo | Export-Csv -Path $FileName -Delimiter ";" -NoTypeInformation
}

#endregion

I hope this is also useful for your migrations and have fun with the script. As time goes and report demands change. I will update the script on this blog to keep you up-to-date.

The mysterious case of a failed account recovery and orphaned mailbox

Reading Time: 5 minutes

In this blogpost I want to address two real-life cases that I encountered in the same Microsoft Office 365 tenant. The reason why I address the two issues in this one blog is because the errors and steps to resolution, were identical for both issues.

Background Information

The issues occurred in a cloud-only tenant. The tenant has multiple custom domains configured, and in use. The tenant consist of multiple user accounts and shared mailboxes.  There where no external scripts or data sources that are feeding into the Azure AD tenant with account information or automated management tasks.

I was called in to resolve the issues. Names and domains are anonymized for the purpose of this blogpost.

The Issues

Issue 1

The first issue occurred after a user account deletion and recovery.  There were two accounts, that were converted to shared mailboxes.

Mailbox 1: edatorial@custdomainA.com (Primary email) – UPN: edatorial@custdomainA.com – Created in 2017
Mailbox 2: edatorial@custdomainB.com (Primary email) – UPN: edatorial@custdomain.onmicrosoft.com – Created in 2018

The issue here was that mailbox 1 was accidentally deleted. We used the recovery page in the Office 365 Admin Portal to restore this account.
When we did this, we couldn't change the primary address of both shared mailboxes. We hit an error stating that the proxy address already existed on the other account. On both accounts it was listed as a proxy address in Exchange Online and in Azure AD.
It should be impossible within Exchange Online to have the same proxy address on multiple accounts.

Issue 2

The second issue was that the customer requested a shared mailbox was to be deleted , but the customer asked for a empty shared mailbox with the same name some days later.  This mailbox was created, full access rights were delegated and people started working with the mailbox.

Mailbox 1: tooling@custdomainA.com (Primary email) – UPN: tooling@custdomainA.com

When I was handed the case, the customer reported that they couldn't access the mailbox anymore. When I looked in Exchange Online, I saw the mailbox still listed on the Shared Mailboxes page.
In the Office 365 Admin Portal, I didn't see the user account. Instead, it was listed on the Deleted Accounts page. We performed an account restore. This was successful, but not the solution to get it working again.

 

Resolving Issue 1

The information we started with for resolving the issues was that both accounts/mailboxes are visible in the Office 365 Admin Portal and in Exchange Online on the Shared Mailboxes page.

Observations and Symptoms

When both accounts were visible and active again, we tried to manage both accounts from the Exchange Online portal. Mailbox 1 gave an error in the management website; the account wasn’t located on the Domain Controller. Mailbox 2 gave an error, when we tried to alter the proxy addresses; the proxy address already exists on Mailbox 1.

I opened an Exchange Management Shell connection to the tenant, and tried to change the information there. I received the same errors as in the web interface; User not found and proxy address already exists.

"Could this account be incorrectly mapped?"

I checked if the accounts in Azure AD were correctly mapped to the Exchange Online accounts, by changing their display name. Within five minutes the information was updated in Exchange Online. So we know that the mailboxes are correctly mapped to the Azure AD accounts.

Then I remembered the behavior of Exchange Online, that it always wants to add the userPrincipalName (UPN) as an alias on the mailbox and cannot be removed, as long as the UPN is set. But as given in the description the UPNs already were different…

So I listed the mailbox information through the Exchange Online management shell.  Here I discovered that on both mailboxes the attributes WindowsLiveID, and MicrosoftOnlineServicesID, contained the same UPN, edatorial@custdomain.onmicrosoft.com.

Solution

Fixing Mailbox 2

Based on that discovery, I decided to update the UPN of both accounts. First I altered the UPN of mailbox 2, because this mailbox was already set to edatorial@custdomain.onmicrosoft.com . I updated the UPN of Mailbox 2 to edatorial@custdomainB.com and waited on the internal sync of Azure AD and Exchange Online. After five minutes, I checked the attributes WindowsLiveID and MicrosoftOnlineServicesID on Mailbox 2; these where updated to the new UPN information. Then I removed the edatorial@custdomain.onmicrosoft.com as an alias on Mailbox 2. This was successful and no errors were shown.

Mailbox 1 wasn’t fixed after this.

I decided to perform the same action on mailbox 1 as I did on mailbox 2. First I changed the UPN from edatorial@custdomainA.com to edatorial@custdomain.onmicrosoft.com in the Office 365 Admin Portal. Also I changed the display name back to how it was, to see, when the account was updated in Exchange Online. When this was changed, I changed the UPN back from edatorial@custdomain.onmicrosoft.com to edatorial@custdomainA.com in the Office 365 Admin Portal. After five minutes I checked the WindowsLiveID and MicrosoftOnlineServicesID attributes on Mailbox 1; these were updated to the new UPN information. Also it was now possible to manage the mailbox again.

And then…

Something curious happened 15 minutes later, though. Mailbox 1 was deleted again from Exchange Online and Azure AD. When I looked on the Deleted Users page in the Office 365 Admin Portal, the account was listed there again. We initiated a recovery once again and this worked as designed. Now the account was usable and working again. In the audit log of Azure AD, I couldn’t find the delete action, so determining the root cause of that spontaneous deletion was impossible.

Resolving Issue 2

The information for resolving started with that the account was restored in the Azure AD Portal. The mailbox was already visible in Exchange Online.

Observations and Symptoms

After the Azure AD account was restored, I checked if I could manage the mailbox again from the Exchange Online admin page. I only found an error; the object couldn’t be found on the Domain Controller.

As with Issue 1, I checked if the account was correctly mapped to the mailbox. I updated the display name and five minutes later I saw the change in Exchange Online. So I confirmed that the objects were mapped to each other. Based on the experience with Issue 1, I checked if the attributes: WindowsLiveID and MicrosoftOnlineServicesID were the same. This was not the case. The attributes were pointing to tooling@custdomain.onmicrosoft.com instead of tooling@custdomainA.com .

Solution

As solution to this problem I decided to change the userPrincipalName (UPN) from tooling@custdomainA.com to tooling@custdomain.onmicrosoft.com. This time, the change wasn’t picked up by Exchange Online. We already checked the integration, so I decided to delete the user one more time from the Office 365 Admin Portal. Also I waited on Exchange Online to see if the mailbox was deleted from their side. This was the case. So now both the Azure AD account and Mailbox where in a soft delete state.

Going from soft-delete to restored state

Now I restored the Azure AD account from the Office 365 Admin Portal and five minutes later the mailbox was also recovered. This time we could manage the mailbox again. So as the last step in the solution I changed the UPN one more time from tooling@custdomain.onmicrosoft.com to tooling@custdomainA.com  and this was now processed by Exchange Online. The attributes WindowsLiveID and MicrosoftOnlineServicesID were the same as the UPN in Azure AD.

Unknown Root Causes

At time of writing this blog, I still don't know what caused both issues. 

All management tasks of the tenant are done through the Office 365 Admin Portal and Exchange Online.
The actions I took to resolve Issue 1 were on January 9th. When I was called in to resolve Issue 2, two days later, I saw that this account was deleted on January 9th.

Concluding

If I were to guess, the problem may lay in the  automated recovery procedure and automatic health tasks within Azure AD. I’m still trying to reproduce the issues, to point to a probable cause.

I hope that this blog was informative and useful in the future, when you might come across similar issues.