It's more likely you're going to implement your Server Core boxes inside a legacy environment, than implement them inside a new Windows Vista, Windows Server 2008 only environment.

I'll try to explain the 'Next Generation IP Stack' which was introduced with Windows Vista and is also present in Windows Server 2008, both in Full and Server Core installations. I'll show you how to tweak down some default global TCP settings, that are part of what Microsoft calls the "Scalable Network Initiative".

Tweaking down will ensure you can access your Server Core boxes without delays from Windows 2000, Windows XP and Windows Server 2003.

The stuff in this post will also be very useful when you deploy Server Core inside a networking environment based on 100Mb/s (or slower) infrastructure, since most of the default tweaks in the "Scalable Network Initiative" assume you have a Gigabit Network Interface Controller.

Note:
For copper-based Network Interface Controllers Windows Server 2008 ships with Gigabit Network Interface Controller drivers only. When you want to use your 10Mb/s or 100Mb/s Network Interface Controllers with Windows Server 2008 you need to manually add drivers.

Default settings

The default settings for the Transmission Control Protocol (TCP) part of TCP-IP are easily readable in Server Core installations of Windows Server 2008.

The command to use is:

netsh interface tcp show global

This will show you the settings for:

Receive-Side Scaling

Receive-Side Scaling is a technology which allows a Windows Vista and later Microsoft Operating Systems to utilize multiple processors to handle the network stack. Technically it allows large file transmissions to be processed in a parallel manner. Packets are balanced between your processors (or cores) through the use of a software hash codes and hardware hash codes for each connection.

The origin or Receive-Side Scaling

In our multi-core and even multi-processor systems not having Receive-Side Scaling might mean one processor would be utilizing 100% CPU, while the other would be virtually doing nothing. In the case of a File Server you might want both processors to be used and the load to be balanced.

The problem with Receive-Side Scaling

The problem with Receive-Side Scaling is it's susceptible to packets that were changed before they were received. The hashes don't match in that case and the connection is dropped. Microsoft acknowledges this is a problem when NAT, ICS, ISA Server or other firewall products are used. If your anti-virus software includes a firewall (and most do) this might present a problem.

How to turn off Receive-Side Scaling

Using the command line

To turn off Receive Side Scaling on your Server Core installation of Windows Server 2008 simply type the following command:

netsh interface tcp set global rss=disabled

Using the registry

Alternatively you can edit the settings for Receive-Side Scaling in the registry, although this will require a reboot, since the values to modify exist or need to exist in the following place in the registry:

HKLM\SYSTEM\CurrentControlSet\Services\Tcpip\Parameters

By default you wouldn't find a REG_DWORD value for EnableRSS in this registry key. If you want to disable Receive-Side Scaling though, you need to create it manually. After you created the new value you can set the data to 0, which means disabled. If the value is not present, create it. If you decide to enable it afterwards you can change the value to 1, which meant enabled.

For more information on Receive-Side Scaling please read the Scalable Networking with RSS  Whitepaper on the Microsoft Website.

Chimney Offloading

TCP-IP Chimney Offloading is a pretty neat technology that takes advantage of the current generation (Gigabit) Network Interface Controllers that have been equipped with a TCP Offload Engine (TOE). These Network Interface Cards are capable of managing connections and traffic on their own. By offloading a lot of calculations of this traffic from the CPU to the Network Interface Card (NIC) you can use your CPU to calculate other things.

The word chimney is derived from the Chimney Architecture and finds its origin in the (OSI) Stack. The purpose of the chimney is to isolate traffic between the top and the bottom of the chimney. Vents in the chimney (that is, special function calls for manipulating the offloaded control plane) allow the host stack to query, update, invalidate, and terminate the offload state.

Microsoft has acknowledged some problems with Chimney Offloading.

How to turn off Chimney Offloading

Using the command line

If you don't have a Network Interface Card with a TCP Offload Engine (TOE) or just want to ensure proper network traffic flow you can disable TCP Chimney Offloading using the following command:

netsh interface tcp set global chimney=disabled

Using the registry

To disable Chimney Offloading using the registry you can browse to the following key:

HKLM\SYSTEM\CurrentControlSet\Services\Tcpip\Parameters

You should see a REG_DWORD registry value named EnableTCPChimney. The data for this registry entry would read 1 by default, which would mean enabled. To disable TCP Chimney Offloading change the data to 0, which resembles disabled. If the value is not present, create it.

You need to reboot your Server Core box to make your registry changes take effect.

Receive Window Auto-Tuning

When you have a broad network connection you might want to use it efficiently and that is exactly what the IETF had in mind when they worked on RFC 1323 back in 1992. One of the new things in RFC 1323 is Receive Window Auto-Tuning, which allows the receiving end of a TCP connection to advertise a larger 'window' in TCP-IP packets to transmit data in. This allows for a higher percentage of data in relation to headers in TCP-IP packets, increasing the effective bandwidth on these connections.

There are 5 modes for Receive Window Auto-Tuning:

1. Disabled
   Fix the receive window at its default value. (65535 bytes)
2. HighlyRestricted
   Allow the receive window to grow beyond its default value, but do so very conservatively.
3. Restricted
   Allow the receive window to grow beyond its default value, but limit such
   growth in some scenarios.
4. Normal
   Allow the receive window to grow to accomodate almost all scenarios.
5. Experimental
   Allow the receive window to grow to accommodate extreme scenarios.
   WARNING: This can dramatically degrade performance in common scenarios and should only be used for research purposes.

Unfortunately not every device has RFC 1323 implemented in a way that allows compatibility. Microsoft mentions examples of these devices inside KnowledgeBase Article 934430.

How to turn off Auto-Tuning

To disable Receive Window auto-Tuning on your Server Core installation of Windows Server 2008 type the following command:

netsh interface tcp set global autotuning=disabled

Alternatively you can choose one of the remaining 3 modes for Receive Window Auto-Tuning to change the default 'normal' setting. Setting Receive Window Auto-Tuning to 'Restricted' should best fit your situation.

Concluding

In Windows Server 2008 the Network Interfaces are tweaked by default. You can change a couple of global settings to undo these tweaks so you can enjoy Windows Server 2008 inside legacy networks. To do so, issue the following commands:

netsh interface tcp set global rss=disabled
netsh interface tcp set global chimney=disabled
netsh interface tcp set global autotuning=disabled

Further reading

Microsoft Whitepaper – Scalable Networking with RSS
Microsoft Whitepaper – Introducing TCP Chimney
Scalable Networking Pack: Frequently Asked Questions
You cannot host TCP connections when Receive Side Scaling is enabled
A Windows Server 2003-based computer may stop responding during shutdown
"General Network error," "Communication link failure,", "A transport-level error"
Network connectivity may fail when you try to use Windows Vista behind a firewall device
When putting a Vista computer on a SBS 2003 network
Find your bandwidth in Vista really slow? Here is a simple hack for you
Vista: TCP window scaling not compatible with some network hardware
Network connectivity may fail when you try to use Windows Vista behind a firewall device The Complete Windows Vista vs. Windows XP Networking Performance Comparison