Most Microsoft-based Hybrid Identity implementations use Active Directory Federation Services (AD FS) Servers, Web Application Proxies and Azure AD Connect installations. In this series, labeled Hardening Hybrid Identity, we’re looking at hardening these implementations, using recommended practices.
In this part of the series, we’ll look at best practices to handle time synchronization on non-domain-joined Web Application Proxy servers.
Note:
This blogpost assumes you’re running Web Application Proxies as non-domain-joined Server Core Windows Server 2016 installations.
If your Web Application Proxy servers are domain-joined, the default Windows Time hierarchy takes care of time synchronization. Reasons to domain-join Web Application Proxies are:
- Centralized management policies dictating every Windows Server is domain-joined
- You want to publish web applications using Windows Integrated Authentication, leveraging Kerberos Constrained Delegation (KCD). (Web Application Proxies need to be joined to the same Active Directory domain as the AD FS Servers)
Most Web Application Proxy server implementations result in non-domain-joined servers, that’s because:
- Most implementations need to adhere to a rule in the standards framework that internet-facing servers are not domain-joined or may not have a direct connection to the Internet (read: they need to be proxied). With the Web Application Proxy role acting both as an AD FS Proxy and a fancy reverse proxy, this role is caught in the middle.
- Most frameworks dictate that Internet-facing servers need to be placed on a perimeter network (also known as a DMZ) and organizations are not too fond of opening up the perimeter network to the internal network, or the Domain Controller (virtual) network for all the UDP and TCP network ports that are necessary to domain-join Windows Servers.
Why look at time synchronization for Web Application Proxies
Hybrid Identity deployments are often long and costly implementations. As they are built using on-premises systems, they are intended to provide the functionality for the full duration of the economic lifetime of these systems (4-5 years).
Having reliable time throughout the Hybrid Identity implementation is important.
Possible negative impact (What could go wrong?)
When time synchronization is overlooked for Web Application Proxies, their time may drift. Depending on the amount of time drift, functionality breaks:
- People will no longer be able to log on, when outside the organization’s network, due to time mismatches with AD FS Servers (resulting in token time-outs) and/or intervening load balancers.
- Logs from Web Application Proxies will no longer bear proper time stamps.
- Certificate errors may occur when certificate validity can no longer be evaluated reliably.
- Web Application Proxies will no longer be able to get Windows Updates.
Some of this behavior can be detected in an easy way, as people will complain when their functionality is negatively impacted. Other symptoms may be overlooked, until it’s too late.
Three solutions for time synchronization
There are three solutions to reliably implement time synchronization for non-domain-joined Web Application Proxies:
- Synchronize time with a reliable time source off the network
- Synchronize time with a reliable time source on the network
- Synchronize time with a hypervisor host
(in the case of virtualized Web Application Proxies)
Synchronize time with a reliable time source off the network
This solution takes clues from the default method any non-domain-joined Windows installation synchronizes time with time.windows.com.
time.windows.com may not be the most reliable time source on the Internet, but may just work and offer sufficiently reliable time. Most organizations opt to synchronize systems with pool.ntp.org, but my recommendations for this solution are to:
- Use a combination of DNS names and IP addresses, to avoid failures in time synchronization during DNS outages and reduce the risk of DNS poisoning.
For this practice, it’s important to periodically check the validity of the IP addresses entered. - Open any necessary firewall ports for NTP (UDP 123) from the Web Application Proxy Server(s) to the designated time sources on the Internet.
Synchronize time with a reliable time source on the network
Many large organizations already have a GPS-backed NTP appliance on their network. Some have even equipped each of their datacenters with one. These appliances are placed on the perimeter network or offer their services from a shared network segment to all other network segments, including the perimeter network (DMZ).
When configuring time synchronization with the appliance, make a risk assessment to connect to the IP address or the DNS host name; in large organizations you might not get notified of changes to this particular piece of infrastructure and you might end up with faulty time synchronization. Again, when using IP addresses, it’s important to periodically check the validity of the IP addresses entered.
While GPS-based attacks do exist, there are currently no examples known of successful GPS spoofing attacks. When choosing a GPS-based appliance, it might be wise to include the requirement that countermeasures have been taken to prevent GPS spoofing attacks as laid out in the US Department of Homeland Security’s whitepaper "Improving the Operation and Development of Global Positioning System (GPS) Equipment Used by Critical Infrastructure" dated November 12, 2017.
Synchronize time with a hypervisor host
As discussed previously for VMware-based scenarios, virtual machines running as Web Application Proxy servers may benefit from time synchronization with vSphere hosts.
Hyper-V-based en XenServer-based virtual machines running as Web Application Proxy servers may benefit from time synchronization with their respective hypervisor hosts.
On the above situations, no network ports need to be opened between networks or even to the Internet. The Integration Components (known in the VMware world as VMware Tools) take care of the communication between the hypervisor host and the virtual machine running as Web Application Proxy server. Of course, the hypervisor hosts themselves need to synchronize time with a reliable time source, which may require expanded network connectivity.
Getting Ready
To handle time synchronization on non-domain-joined Web Application Proxies, make sure to meet the following requirements:
System requirements
Make sure the Web Application Proxies are installed with the latest cumulative Windows Updates. This makes sure they have the latest information on time zone changes.
Privilege requirements
Make sure to sign in with an account that has local administrator privileges on each of the Web Application Proxies. Your Web Application Proxies all have different local administrator passwords, right?
Network requirements
When synchronizing time with a reliable time source on the Internet or on the network, make sure the Web Application Proxies can reach the time source using UDP port 123. Open up firewalls to allow this traffic.
When synchronizing time with the hypervisor platform, make sure the hypervisor hosts can either independently reach the time source using UDP port 123, or make sure the Domain Controller holding the PDCe FSMO role for the Active Directory domain the Hyper-V hosts are a member of can reach the time source using UDP port 123.
Who to communicate to
As the Web Application Proxies operate in a process, notify all stakeholders in the process. This means sending a heads-up to the load balancer guys and gals, the networking guys and gals, the rest of the Active Directory team and the teams that are responsible for Azure AD, Office 365 and cloud applications. It’s also a good idea to talk to the people responsible for backups, restores and disaster recovery (as Web Application Proxy don’t store any dynamic data, in some organizations these hosts are not part of regular backup jobs but are backed up using an on-demand job after changes).
One of the challenges you can easily avoid through communications is that multiple persons and/or teams make changes to the configuration. When it breaks, you don’t want to roll-back a bunch of changes, just the one that broke it. Make sure you have the proper freeze/unfreeze moments to achieve that.
How to do it
To handle time synchronization on non-domain-joined Web Application Proxies, perform these actions:
Synchronize time with a reliable time source
To synchronize time with a reliable time source, be sure to check the connection to the time host on the Internet or the internal network, then set it.
Checking the connection to the time source
Run the following line on the Command Prompt (cmd.exe), when logged on to the Web Application Proxy to test the pool.ntp.org address as a reliable time source:
w32tm.exe /stripchart /computer:pool.ntp.org /dataonly /samples:5
Perform the above command line for all DNS names and IP addresses you want to test as a reliable time source. This command also comes in handy when you want to evaluate the IP addresses you may or may not enter as reliable time sources.
Configuring Time Sources
Run the following two lines on the Command Prompt (cmd.exe), when logged on with an account that has local administrative privileges on the Web Application Proxy to set time sources:
w32tm.exe /config /manualpeerlist: "europe.pool.ntp.org time.nist.gov 192.43.244.18 193.67.79.202" /syncfromflags:manual /reliable:yes /update
net stop w32time && net start w32time
Make sure to separate any DNS names and IP addresses for the /manualpeerlist switch with a space. This is the appropriate way to set multiple time sources.
Note:
The above DNS names and IP addresses are for illustration purposes. They may not operate any longer.
Synchronize time with a hypervisor host
To synchronize time with the hypervisor host, install the integration components for the hypervisor platform on the virtual machine that operates as Web Application Proxy. This should be sufficient as default settings on all hypervisor platforms is to have virtual machines synchronize time with the hypervisor.
Concluding
Synchronize time with a reliable time source on non-domain-joined Web Application Proxies to avoid issues.
Related blogposts
Active Directory Time Sync (broken by default)
Managing Active Directory Time Synchronization on VMware vSphere
Further reading
Configure WAP for KCD with Web App
Check a NTP Server Date and Time Using the Windows Command Prompt
Windows Time Service Tools and Settings
“It’s Simple!” – Time Configuration in Active Directory
Login