Field notes: Make the actual source client IP visible for a load-balanced SMTP service

Reading Time: 5 minutes

A typical headache for Microsoft Exchange Server administrators is to setup load-balancing with port 25. They face the issue that, typically in these setups, they no longer see the source client IPs of the actual clients; All traffic is translated and is presented to the SMTP service with the load balancer IP as source. This makes working with receive connectors quite difficult. Many organizations have opted to use multiple virtual IPs and/or different port numbers for the SMTP service…

I think there is a better solution. I think the Direct Server Return option on load balancers provides a better solution; This option forces the load balancer to present the original client IP to the back-end server, in our case: the SMTP service of Microsoft Exchange Server. The back-end server sends response packets directly to the client instead of the load balancer (no Server Network Address Translation (NAT)).

This works fine if you have a single subnet or no firewall(s) between the client, the load balancer and the back-end server. However, if a firewall plays a role in the communications flow, then the firewall will drop the connections because of asymmetric traffic flows; The firewall notices that the TCP information does not match with what it knows.

The (not so) secret sauce

The basic trick is to have the Windows Server respond with the Virtual IP of the load balancer. This way, the TCP packet looks like it’s coming from the load balancer, but actually it’s coming from the back-end server. Most firewalls are happy and allow the traffic.

Note: The communication flow must go back and forth through the same devices. Otherwise, you still experience asymmetric traffic flow and traffic will be blocked.

Another hurdle you may encounter is the need to disable ARP responses to the actual Virtual IP addresses used by the systems. When firewalls come into play, you need to look at the communications path of the traffic. Firewalls can manipulate traffic or do additional translation, that may interfere with what you want to achieve.

How to make the actual source IP address visible to a load-balanced SMTP service

Below, I'll share the steps I follow to setup my solution.

Create a lookback adapter on Windows Server

First, we need to create a lookback adapter on the Windows Server that hosts the SMTP service. Perform the following steps with an account that has administrator privileges:

Click Start, then type cmd in the search box.
When cmd.exe appears, right-click it and choose Run as administrator.
The Administrator: Command Prompt window opens.
On the command prompt, type hdwwiz.exe and press Enter.
Click Next >.
Select Install the hardware that I manually select from a list (Advanced):
Click Next >:
Select Network adapters, then click Next >:
Select Microsoft as the manufacturer. Then, select Microsoft KM-TEST Loopback Adapter:
Click Next >.
Select Next > to confirm the installation:
Click Finish to complete the installation.

Configure the Loopback adapter on Windows Server

Now, we need to configure the network adapters on the Windows Server that hosts the SMTP service. Perform the following steps with an account that has administrator privileges:

Click Start > Control Panel > Network and Sharing Center.
Follow the Change adapter settings link in the left navigation pane.
Select the newly created connection and rename it to Loopback.
Right-click the Loopback connection and choose Properties from the menu.
Confirm that Microsoft Loopback Adapter or Microsoft Loopback Adapter # is displayed in the Connect Using: field. If it is not, return to step 2 and retry the Properties for another adapter.
Deselect every item in the list below This connection uses the following items:, except Internet Protocol (TCP/IP):
Select Internet Protocol (TCP/IP), and click Properties to open the Internet Protocol Version 4 (TCP/IPv4) Properties window.
Select the Use the following IP address option.
Fill in the IP address and Subnet mask fields of the Virtual Service that you configured on your load balancer:
Click the Advanced… button to open the Advanced TCP/IP Settings window.
On the IP Settings tab of the Advanced TCP/IP Settings window, deselect Automatic Metric and provide the value 254:
Switch to the WINS tab.
Select the Disable NetBIOS over TCP/IP option:
Click OK to close the Advanced TCP/IP Settings window.
Click OK to close the Internet Protocol Version 4 (TCP/IPv4) Properties window.
Click OK to close the connection properties.
Switch back to the Administrator: Command Prompt window.
On the command prompt, issue the following command with the LAN interface in scope (the interface that the load balancer points to):
netsh interface ipv4 set interface "LAN" weakhostreceive=enabled
On the command prompt, issue the following commands, too. This time, with the Loopback adapter as the target: (Repeat if you have multiple loopback interfaces)
netsh interface ipv4 set interface "loopback" weakhostsend=enabled
netsh interface ipv4 set interface "loopback" weakhostreceive=enabled
Close the Administrator: Command Prompt window.

The configuration on the Windows Server is now complete.

Configure the load balancer and firewalls

When you have the above configuration in place on the Windows Server that hosts the SMTP service, you can enable the Direct Server Return option in your load balancer.

Concluding

I personally try to implement the above solution in every project, that require me to build a new Microsoft Exchange Server environment with high-availability and load balancing. I've got the Direct Server Return option working with Citrix Netscaler and KEMP Loadmaster implementations.

Reclaim control of your SMTP configuration and regain insight to the source client IP for SMTP traffic and build secure receive connectors again, with the information above.

Test before going into production.

Field Notes: DKIM and missing selector records

Reading Time: 3 minutes

During a project with one of my customers, I was tasked to look at a non-delivery report (NDR) for a mail message. The bounce error was pretty confusing, but after reviewing the headers, we noticed that the DKIM check had failed. This was a bit of a surprise, because the message was sent from Microsoft Exchange Online. The root cause was that the selector mentioned in de header wasn’t resolving. Apparently, Microsoft did not provide the answer to validate the record.

The situation

The organization uses Microsoft Exchange Online and has setup DKIM signing with the accepted domains listed in the tenant. This particular organization owns 70+ domains. For every accepted domain, they have created the required CNAME records. During the validation, roughly 20 domains had a failed active selector entry.

I raised a case with Microsoft Office 365 Support, which confirmed that is was a bug and resolved the issue for my customer. They also informed the product team of this error.

How to setup DKIM

By default Exchange Online signs outgoing message with a DKIM signature. More information can be found here. The validation information recorded in the header points to one of the following records:

selector1-[tenantname]-onmicrosoft-com._domainkey.[tenantname].onmicrosoft.com
selector2-[tenantname]-onmicrosoft-com._domainkey.[tenantname].onmicrosoft.com

To enable DKIM you will also need to setup the following CNAME records in the domain:

selector1._domainkey
selector2._domainkey

Using Windows PowerShell, you can get the full value to use for these DNS records. You can retrieve the information through the Exchange Online PowerShell module. The recommended administrative permission is Security Admin.

Follow these steps:

Connect to Exchange Online using the Exchange Online Powershell module.
Sign in with your cloud administrator credentials.

Give the following command for all existing domains:

Get-DkimSigningConfig | FL Identity,Selector1CNAME,Selector2CNAME

Give the following command for one specific domain:

Get-DkimSigningConfig –identity [domainname]| Format-List Identity,Selector1CNAME,Selector2CNAME

Register the new CNAME records with the Selector[1/2]Cname as value.
After succesfully creating the DNS records, you can activate the DKIM signing configuration. This can be done via the following command:
```
Set-DkimSigningConfig –identity [domainname] –Enabled:$true
```

Note: If the domain is missing, you can create the DkimSigningConfig entry through this powershell command:

New-DkimSigningConfig -KeySize 2048 –DomainName [domainname] –Enabled:$false

Only one selector returns data

With the initial creation of your DKIM configuration, Microsoft populates both selectors in their DNS. After a while, though, only the active selector will be resolvable and the other selector is not. This is by Microsoft's design. The record should be visible again when the key rotates.

How to validate the existing records

The challenge for my customer was to validate the records for all 70+ domains. We can do this manually, but where it the fun in that!? We decided to do it through PowerShell.

The below Windows PowerShell script requires an active session to Exchange Online through PowerShell. The recommended administrative permissions is Security Admin.

We ran the following script:

$domains = Get-DkimSigningConfig |Where-Object {$_.Enabled -like "True"}
$Datenow = Get-date
$FailedLookupslog = "C:\Temp\FailedDKIMrecords.log"
$CheckDNS = @()
$failed = @()

ForEach($domain in $domains){

    $activeSelector1 = $domain.SelectorBeforeRotateOnDate
    $activeSelector2 = $domain.SelectorAfterRotateOnDate

    If($($domain.RotateOnDate.ToUniversalTime()) -gt $($datenow.ToUniversalTime()) ){
         $tempattribute = "$($activeSelector1)Cname"
         $CheckDNS += $domain."$tempattribute"
     }
     else{
         $tempattribute = "$($activeSelector2)Cname"
         $CheckDNS += $domain."$tempattribute"
     }
}

foreach($entry in $CheckDNS){
     $works = Resolve-DnsName -Name $entry -Type txt -erroraction 'silentlycontinue'

     If($works){}
     else{
         Write-host "$entry is not working" -ForegroundColor yellow
         $Failed += $entry
     }

     $works = ""
}

"Domain: , selector:" | Out-File -FilePath $FailedLookupslog

Foreach($fail in $failed){

    if($fail -like "selector1*"){
         $temp = $domains | where{$_.selector1Cname -like $fail}
     }
     else{
     $temp = $domains | where{$_.selector2Cname -like $fail}
    }
     "$($temp.Domain) , $fail" | Out-File -FilePath $FailedLookupslog -Append
}

The failed active selector entries were listed in the logfile.

Concluding

Organizations that enable the DKIM feature rely on Microsoft to maintain the information and keep it up-to-date. As a consumer of this feature you need to trust, but verify that it’s still works as designed.

By using the script in the blog, you can automate, verify and let it report to your ticketing system for review and follow-up actions.

Field Notes: Meeting the requirements for Interoperability between Microsoft Teams and Microsoft Exchange Server

Reading Time: 3 minutes

In this blog post, I want to walk you through my experiences with setting up and enable the interoperability between Microsoft Teams and on-premises Microsoft Exchange Server environments.

Since the beginning of this year, Microsoft Teams adoption has seen a tremendous uptick in usage. Organizations needed to adopt Microsoft Teams as their Unified Communications (UC) platform, but the other cloud migrations went on the back burner or were not even in scope anymore for this year. All is fine, until somebody asks: "I don’t see my calendar in Microsoft Teams… Can you fix this?"

Sure we can. In this blog post, I'll walk you through the process of enabling the interoperability between the Microsoft Teams service and the on-premises Microsoft Exchange Server environment.

Requirements

Let's start with listing the Microsoft requirements and what they actually mean in real life:

Known requirements

We start with the known requirements as listed here. Summarized these are:

The employee identities need to be synchronized to Azure AD;
The organization needs to have the Exchange Hybrid option in Azure AD Connect;
OAuth authentication between your on-premises Exchange and Exchange Online organizations needs to be configured;
Microsoft Teams scheduling of meetings by delegates require an additional partner application;
The employees need to have to be licensed: the Microsoft Teams license to be specific;
The on-premises Microsoft Exchange Server environment needs to run Microsoft Exchange Server 2016 CU3 or a newer version of Microsoft Exchange Server.
Good to know: The Microsoft hybrid Agent (Modern Exchange Hybrid) cannot be used for the Microsoft Teams calendar integration. It is listed here in the constraints of the Microsoft Hybrid Agent.

My approach

When a customer asks “I want to have my calendar visible is Microsoft Teams, is this possible for us and what do we need to do?”, then this is my approach:

To see if it’s even possible to realize the interoperability, we need to validate the current on-premises Microsoft Exchange Server environment in terms of the version: It needs to run at least Microsoft Exchange Server 2016 with CU3. The importance of this question is in the availability of the new AutodiscoverV2 and the new REST-based application programming interface (API) in the Exchange Web Services (EWS) capabilities. But that is not all: In order to benefit from these new capabilities, the mailbox needs to be homed on a Exchange Server 2016 CU3 or higher mailbox server. If this is all true, then I validate if Autodiscover service and Exchange Web Services of Exchange Server 2016 is accessible from the internet. If not, the organizations must be willing to allow this.

Initial checks

This results in the following checks:

Autodiscover and Exchange Web Services (EWS) need to be accessible from the internet and pointing to a Microsoft Exchange Server that runs at least Microsoft Exchange Server 2016 CU3;
The mailboxes for the employees in scope need to be homed on a Microsoft Exchange Server hat runs at least Microsoft Exchange Server 2016 CU3;
The employees must have Microsoft Teams licenses assigned;
Good to Know: Microsoft Teams will not use the configured hybrid configuration within Exchange Online, but trusts the native Autodiscover functionality. Via autodiscover.domain.tld it is pointed to the on-premises environment or via SRV records it needs to point to the on-premises environment.
Good to Know: Autodiscover (http) redirect will not work for the Microsoft Teams integration. This might impact the current public certificates that an organization might currently use.

If any of requirements are not met or not possible to achieve, then the integration is a NO GO.

Further implementation information

When the initial requirements are all met, we can ask the following questions:

Are there any Microsoft Exchange Server left in the organization that run Microsoft Exchange Server 2010?
If a Microsoft Exchange Server 2010 is present in the environment, then the Exchange Hybrid configuration wizard will not create the OAuth configuration with Azure AD during the setup and we need to do it manually.
Is a Exchange Hybrid configuration desired and on the roadmap for the organization?
Exchange Hybrid is a requirement from Microsoft for the interoperability between Microsoft Exchange Server and Microsoft Teams. If we cannot use the Exchange Hybrid configuration wizard (which is the preferred option), we need to configure the Oauth configuration manually.
Is it required that delegates need to be able to make appointments on behalf of the delegator?
If so, we need to perform steps 2 and 3 of the instructions listed here. The steps will provision the trust for the Skype for Business online integration.
Can we configured the Exchange Hybrid Writeback permission on the AD Connector account in Active Directory for Azure AD Connect?
Within Azure AD Connect we need to enable the synchronization option for the Exchange Hybrid feature. This requires that the configured service account has the correct writeback permissions for Exchange attributes. The list of required attributes to which we need write permissions is listed here.

Concluding

Compared to what is documented in the Microsoft docs and the real world, some parts are missing in the explanation or the requirements list of Microsoft. In this blog post I want to give you my past experience and help you with the little details to enable organizations to implement the calendar integration.