In this blogpost I want to address two real-life cases that I encountered in the same Microsoft Office 365 tenant. The reason why I address the two issues in this one blog is because the errors and steps to resolution, were identical for both issues.
Background Information
The issues occurred in a cloud-only tenant. The tenant has multiple custom domains configured, and in use. The tenant consist of multiple user accounts and shared mailboxes. There where no external scripts or data sources that are feeding into the Azure AD tenant with account information or automated management tasks.
I was called in to resolve the issues. Names and domains are anonymized for the purpose of this blogpost.
The Issues
Issue 1
The first issue occurred after a user account deletion and recovery. There were two accounts, that were converted to shared mailboxes.
Mailbox 1: edatorial@custdomainA.com (Primary email) – UPN: edatorial@custdomainA.com – Created in 2017
Mailbox 2: edatorial@custdomainB.com (Primary email) – UPN: edatorial@custdomain.onmicrosoft.com – Created in 2018
The issue here was that mailbox 1 was accidentally deleted. We used the recovery page in the Office 365 Admin Portal to restore this account.
When we did this, we couldn't change the primary address of both shared mailboxes. We hit an error stating that the proxy address already existed on the other account. On both accounts it was listed as a proxy address in Exchange Online and in Azure AD.
It should be impossible within Exchange Online to have the same proxy address on multiple accounts.
Issue 2
The second issue was that the customer requested a shared mailbox was to be deleted , but the customer asked for a empty shared mailbox with the same name some days later. This mailbox was created, full access rights were delegated and people started working with the mailbox.
Mailbox 1: tooling@custdomainA.com (Primary email) – UPN: tooling@custdomainA.com
When I was handed the case, the customer reported that they couldn't access the mailbox anymore. When I looked in Exchange Online, I saw the mailbox still listed on the Shared Mailboxes page.
In the Office 365 Admin Portal, I didn't see the user account. Instead, it was listed on the Deleted Accounts page. We performed an account restore. This was successful, but not the solution to get it working again.
Resolving Issue 1
The information we started with for resolving the issues was that both accounts/mailboxes are visible in the Office 365 Admin Portal and in Exchange Online on the Shared Mailboxes page.
Observations and Symptoms
When both accounts were visible and active again, we tried to manage both accounts from the Exchange Online portal. Mailbox 1 gave an error in the management website; the account wasn’t located on the Domain Controller. Mailbox 2 gave an error, when we tried to alter the proxy addresses; the proxy address already exists on Mailbox 1.
I opened an Exchange Management Shell connection to the tenant, and tried to change the information there. I received the same errors as in the web interface; User not found and proxy address already exists.
"Could this account be incorrectly mapped?"
I checked if the accounts in Azure AD were correctly mapped to the Exchange Online accounts, by changing their display name. Within five minutes the information was updated in Exchange Online. So we know that the mailboxes are correctly mapped to the Azure AD accounts.
Then I remembered the behavior of Exchange Online, that it always wants to add the userPrincipalName (UPN) as an alias on the mailbox and cannot be removed, as long as the UPN is set. But as given in the description the UPNs already were different…
So I listed the mailbox information through the Exchange Online management shell. Here I discovered that on both mailboxes the attributes WindowsLiveID, and MicrosoftOnlineServicesID, contained the same UPN, edatorial@custdomain.onmicrosoft.com.
Solution
Fixing Mailbox 2
Based on that discovery, I decided to update the UPN of both accounts. First I altered the UPN of mailbox 2, because this mailbox was already set to edatorial@custdomain.onmicrosoft.com . I updated the UPN of Mailbox 2 to edatorial@custdomainB.com and waited on the internal sync of Azure AD and Exchange Online. After five minutes, I checked the attributes WindowsLiveID and MicrosoftOnlineServicesID on Mailbox 2; these where updated to the new UPN information. Then I removed the edatorial@custdomain.onmicrosoft.com as an alias on Mailbox 2. This was successful and no errors were shown.
Mailbox 1 wasn’t fixed after this.
I decided to perform the same action on mailbox 1 as I did on mailbox 2. First I changed the UPN from edatorial@custdomainA.com to edatorial@custdomain.onmicrosoft.com in the Office 365 Admin Portal. Also I changed the display name back to how it was, to see, when the account was updated in Exchange Online. When this was changed, I changed the UPN back from edatorial@custdomain.onmicrosoft.com to edatorial@custdomainA.com in the Office 365 Admin Portal. After five minutes I checked the WindowsLiveID and MicrosoftOnlineServicesID attributes on Mailbox 1; these were updated to the new UPN information. Also it was now possible to manage the mailbox again.
And then…
Something curious happened 15 minutes later, though. Mailbox 1 was deleted again from Exchange Online and Azure AD. When I looked on the Deleted Users page in the Office 365 Admin Portal, the account was listed there again. We initiated a recovery once again and this worked as designed. Now the account was usable and working again. In the audit log of Azure AD, I couldn’t find the delete action, so determining the root cause of that spontaneous deletion was impossible.
Resolving Issue 2
The information for resolving started with that the account was restored in the Azure AD Portal. The mailbox was already visible in Exchange Online.
Observations and Symptoms
After the Azure AD account was restored, I checked if I could manage the mailbox again from the Exchange Online admin page. I only found an error; the object couldn’t be found on the Domain Controller.
As with Issue 1, I checked if the account was correctly mapped to the mailbox. I updated the display name and five minutes later I saw the change in Exchange Online. So I confirmed that the objects were mapped to each other. Based on the experience with Issue 1, I checked if the attributes: WindowsLiveID and MicrosoftOnlineServicesID were the same. This was not the case. The attributes were pointing to tooling@custdomain.onmicrosoft.com instead of tooling@custdomainA.com .
Solution
As solution to this problem I decided to change the userPrincipalName (UPN) from tooling@custdomainA.com to tooling@custdomain.onmicrosoft.com. This time, the change wasn’t picked up by Exchange Online. We already checked the integration, so I decided to delete the user one more time from the Office 365 Admin Portal. Also I waited on Exchange Online to see if the mailbox was deleted from their side. This was the case. So now both the Azure AD account and Mailbox where in a soft delete state.
Going from soft-delete to restored state
Now I restored the Azure AD account from the Office 365 Admin Portal and five minutes later the mailbox was also recovered. This time we could manage the mailbox again. So as the last step in the solution I changed the UPN one more time from tooling@custdomain.onmicrosoft.com to tooling@custdomainA.com and this was now processed by Exchange Online. The attributes WindowsLiveID and MicrosoftOnlineServicesID were the same as the UPN in Azure AD.
Unknown Root Causes
At time of writing this blog, I still don't know what caused both issues.
All management tasks of the tenant are done through the Office 365 Admin Portal and Exchange Online.
The actions I took to resolve Issue 1 were on January 9th. When I was called in to resolve Issue 2, two days later, I saw that this account was deleted on January 9th.
Concluding
If I were to guess, the problem may lay in the automated recovery procedure and automatic health tasks within Azure AD. I’m still trying to reproduce the issues, to point to a probable cause.
I hope that this blog was informative and useful in the future, when you might come across similar issues.
Thanks