Hyper-V Synthetic Networking Is Much Faster

I decided it was time to look at upgrading my home broadband, mostly to get better-than-128KB upload speeds.

After the ISP-side change had eventually wound its way through, I was interested to see that while my upload speeds had improved, my download speeds still seemed capped at about 25Mbits/s (rule of thumb: divide by 8 to get megabytes per sec, so about 3MBytes/sec), despite a new modem and speeds that should’ve been “up to 100Mbit/sec”.

Before launching a tirade at the ISP, I re-plugged my laptop directly into the cable modem, and got around 120MBit/sec test results from speedtest.net, so knew it wasn’t the ISP-side plan itself rate-limiting me any more. And thus began the investigation.

A packet walks into a bar

My setup looks basically like this:

(regular internal networks on 10.x subnets) -> [ TMG -> DMZ -> Smoothwall ->] Cable

With a “side path” for the Xbox, which goes (10.x) -> Smoothwall -> Cable, to benefit from the UPNP mapping feature of Smoothwall.

Technically, any UPnP device or app can use the same side path, but “regular” browsing by WPAD-enabled stuff can benefit from TMG’s web proxy cache to some extent.

Any testing I performed through Smoothwall led me back to the 25MBit capped rate. I checked QoS was off, which it was, and then decided I’d eliminate Smoothwall and try pushing the laptop through TMG directly (bypassing the Smoothwall) as a test.

The test gave me ~115MBit/sec+. Good enough!

Leaping to conclusions

The complicating factor is that everything to the right of the internal networks above is virtualized on my WS2012 R2 Hyper-V box, and I guessed that the rate limiting might have something to do with the legacy network adapters in the Smoothwall VM (one each for internal and external, plus a perimeter NIC for fun). My TMG VM uses synthetic NICs only, so I assumed that it was probably the key performance improvement when using it. The CPU on the router VM didn’t seem to be breaking 0% (externally; 4% is 100% of one core), so I guessed that was a sign the cause might’ve been semi-external, or in the hypervisor.

Rather than work out what performance counters I could use to prove this, I figured I’d try to get the legacy NICs upgraded to the new Hyper-V synthetic NICs instead.

And after reading the vast array of “here’s how you compile XYZ into your kernel” articles on it, I figured that it sounded way too hard (especially as I was using a now-legacy copy of SmoothWall Express 3.0) and finding some kind of distro which worked with Hyper-V enlightenments was probably the easiest route! No pun intended.

How do you test that? Easy, you just leave a VM at default settings – which include a new NIC – and see if it can detect the adapter. If it can, great! If it can’t, try another!

My Little Firewall (packets are magic)

First, I did a little research as to what might support Hyper-V natively, and found pfSense 2.2, based on FreeBSD 10.2, which has inbuilt Hyper-V integration services. I installed it, it found the new NICs, and after minimal faffing around, I had a working firewall, which speed tested up around 115-120MBit/sec. Boosh!

pfSense looks really capable and has lots of new options to explore, but I wondered whether upgrading to SmoothWall 3.1 (which I’d been putting off) would have yielded similar results? I don’t have a huge rule set to migrate, so I started building a fresh Smoothwall Express 3.1 VM from the 200MB ISO, and after configuring my networks from scratch, tried that… also a nice, fast result with synthetic NICs.

So I now have a choice of external firewalls (and whether to bridge the cable modem again, which I like: fewer moving parts to manage), with speedy integrated networking components, and I’m generally quite happy that non-Windows distros seem to be integrating the Hyper-V components for faster-than-legacy networking!

[Update and aside 2017-03-03 – I eventually added a scooter computer to the lineup as an always-on low-power host, and put PfSense onto it “bare metal”. But it quickly became apparent that the Realtek NIC driver included in the box was pretty suboptimal, and would drop out under mild load. So I’m now running Server 2016 on the tiny host, and PfSense in a VM on it, and it’s fast and reliable using the Windows Realtek NIC driver virtualized through Hyper-V. Ironic fun!]

Come to the Windows 8 and Windows Server 2012 Premier Roadshow

Australian Premier Support customers: Join us for an overview of the new stuff in Windows Server 2012 and Windows 8!

This series of events will run for the entire day in each city and showcase 4 sessions of about 90 minutes, on a range of Windows Server 2012 and Windows 8 client topics. All topics will be presented by the best Premier Field Engineers across Australia and New Zealand.

Except they got me for Sydney! I’m covering the new Networking features.

Windows Server 2012 – Networking

Connect from anywhere, more working and less waiting, better network management via cost-aware networking. Sound interesting? This session provides a general overview, including many of the improvements to DirectAccess, BranchCache, and general networking improvements in Windows 8 and Server 2012.

Details and signup:

http://blogs.msdn.com/b/shyam/archive/2012/08/16/windows-8-and-server-2012-road-show.aspx

 

And if you’re at Tech.Ed 2012 AU, you can catch Darth Chad and I presenting on the enhancements to Windows Server 2012 DirectAccess.

 

Configuring Kerberos for SharePoint farms – a generic gotchas list

Recently, I worked on a Kerberos configuration issue with a customer; these are my notes from the visit.

You’ll see some common themes with Kerbie Goes Bananas, and it puts much of that into practice. Speaking of, I must redo Kerbie with SetSPN -S  (shameface)

 

1. DNS should use an A record to refer to the load balancing IP, not a CNAME

This configuration step avoids an Internet Explorer behaviour whereby IE resolves a CNAME into an A record, and requests a ticket by building an SPN for the A record, instead of the CNAME.

More information is available at http://support.microsoft.com/kb/938305 . In most cases, adjusting the behaviour of Internet Explorer across all machines is harder than adjusting the DNS entry involved.

2. SPNs must be registered against the Application Pool Account

Note: use the Windows 2008 (or later) version of SetSPN to identify problems such as duplicates when updating SPNs. Any existing document using SETSPN -A should be updated to use SETSPN -S.

Only two SPNs are required for Kerberos to function against a farm – the FQDN, and the short hostname.

These must be applied to the account used by the Application Pool receiving the user request, which practically means that in most cases, only one account is usable per hostname (pair).

SPNs to be registered are:

HTTP/farm
HTTP/farm.example.com

Against the user identity of the Application Pool the user is connecting to – say, DOMAIN\SPAccount. This must be a domain account when used in a Farm scenario.

Note that no port number is used for the default port, and that these SPNs are also used for TLS/SSL.

SETSPN -S HTTP/farm DOMAIN\SPAccount
SETSPN -S HTTP/farm.example.com DOMAIN\SPAccount

If the individual hostname is to be used occasionally (e.g. for troubleshooting), http/machinename and http/machineFQDN should be registered against that account as well.

This should result in a list of SPNs as shown:

setspn -l DOMAIN\SPAccount

Registered ServicePrincipalNames for CN=SharePoint App Pool Account,OU=Service Accounts,DC=example,DC=com:

HTTP/farm

HTTP/farm.example.com

3. The App Pool Account must be used for authentication

In a web farm scenario, a domain account must be used as the application pool identity. Once a suitable domain account is configured as the application pool identity (DOMAIN\SPAccount in this example), Kernel-Mode Authentication must be disabled, or the configuration’s useAppPoolCredentials property must be set to true (both may be used).

If this step is not performed, the app pool will not be able to decrypt the Kerberos ticket supplied by the client.

To disable Kernel-mode Authentication

Open InetMgr (IIS Manager), browse to Authentication for the site, click Windows Authentication and open Advanced Settings (Actions pane on the right), and untick “Use Kernel-mode Authentication”.

Reference: http://technet.microsoft.com/en-us/library/cc754628(WS.10).aspx

To set useAppPoolCredentials to true:

Open a CMD window as Administrator, then:

CD %windir%\system32\inetsrv

appcmd.exe set config -section:system.webServer/security/authentication/windowsAuthentication -useAppPoolCredentials:true

Note: one line (wrapped), with no space after any dash (-) character.

Reference: http://technet.microsoft.com/en-us/library/dd759186.aspx

4. Performance – Kerberos and NTLM

Use of Kerberos should significantly reduce traffic between WFEs and Domain Controllers.

Every NTLM-authenticated connection requires the server to make a connection to a DC to complete authentication. The number of connections available to a DC simultaneously is governed by MaxConcurrentApi registry value.

Kerberos allows the client to authenticate to a DC once for the website, and to continue to use the ticket for the ticket lifetime (10 hours by default), across multiple connections, without necessarily needing to interact with the DC again.

References

MaxConcurrentApi
http://support.microsoft.com/kb/326040 (original article)
http://support.microsoft.com/kb/975363 (now supports 150)

Kerberos vs NTLM authentication with ISA Server (same concepts apply with Sharepoint or any Web app)
http://technet.microsoft.com/en-us/library/bb984870.aspx

And a third-party performance comparison of Kerb and NTLM authentication with kernel-mode authentication and without was found here (not overall site performance, just basic RPS).

http://blog.michelbarneveld.nl/michel/archive/2009/12/02/kernel-mode-authentication-performance-benefits.aspx

TMG SP2 now out there

There I was, blathering away about Kerberos and SetSPN and sleeping – sleeping! – while the long-awaited-but-unnanounced TMG SP2 was released. And announced, I guess.

The documentation’s still being updated (the release notes haven’t made it up yet), but you can try it out from here:

Microsoft Forefront Threat Management Gateway (TMG) 2010 Service Pack 2

And topically:

New Reports
• The new Site Activity report displays a report showing
the data transfer between users and specific websites for any
user.

Error Pages
• A new look and feel has been created for
error pages.
• Error pages can be more easily customized and can include
embedded objects.

Kerberos Authentication
• You can now use
Kerberos authentication when you deploy an array using network load balancing
(NLB).

Enjoy!

PSA: You really need to update your Kerberos setup documentation with SetSPN -S!

Hi!

You might remember me from such posts as Kerbie Goes Bananas, and SetSPN improvements for Windows 2008. Or something.

I’m here with a public service announcement! Excitement!

It’s been long enough since Windows 2008 (and the downlevel release of SetSPN) that I feel comfortable respectfully asking you to please:

Search and Replace SetSPN -A with SetSPN -S.

In your organization, if you ever happen to run across a document that describes a procedure that looks anything like this:

SetSPN -A http/yourwebfarm DOMAIN\YourFarmAccount

Please:

  •  mail the author, or
  •  file a bug against the content, or
  •  use the Community Content feature if it’s somewhere on Technet, or
  •  mail anyone and everyone responsible for upkeep or implementation of that document

to change the SETSPN -A command to a SETSPN -S.

You may need to include a foreword describing where to get the 2008 version of SetSPN (I think I may have just spoiled it for you) if you’re still strongly a 2003/XP shop, with no newer SetSPN-toting OSs available.

Why the change?

Because it’ll hurt you less in the long run.

The original release of SetSPN was strongly account-centric. Given a Windows account, it would let you:

  • Add an SPN to that account
  • Remove an SPN from that account
  • List the SPNs associated with that account

Unfortunately, this makes it very easy to add the same SPN to multiple accounts – creating a duplicate SPN. This is a very bad thing.

The same SPN can’t easily be added more than once to the same user account, but the original tool does nothing to prevent the same SPN being added to multiple user accounts – and unfortunately, that’s exactly the situation you’re trying to avoid.

BAD EXAMPLES BAD BAD DO NOT USE BAD

Any of

  • SETSPN -A http/farm DOMAIN\FarmUser
  • SETSPN -A http/farm DOMAIN\FarmComputer$

or

  • SETSPN -A http/farm DOMAIN\FarmComputer1$
  • SETSPN -A http/farm DOMAIN\FarmComputer2$

or

  • SETSPN -A http/farm ANYTHING followed by
  • SETSPN -A http/farm ANYTHING_ELSE

breaks kerberos for http://farm.

To restate the rule: One SPN can be associated with precisely one account.

So please, use SetSPN -S

And that’s exactly what SETSPN -S is designed to prevent. SETSPN -S performs a quick check for duplicates before adding an SPN – which is the best possible time at which to catch the problem. So yay-the-Windows-2008-AD-team.

Duplicates! Gotta Catch ‘Em All 2011 Edition

If you suspect you have duplicate SPNs in your environment, well, why just suspect? Run

  • SETSPN -X

To be told explicitly what duplicates you have kicking around in AD (there are forestwide switches you can use too). Yep, that used to be a nasty LDIFDE export with an LDAP filter expression; much simpler now!

 

ISA 2000: The End Draws Near

While updating some documentation today and noticing it’s 2011 (when, exactly, did that happen?), I dug up the ISA Server 2000 Lifecycle information.

Paraphrasing the table here:

  Availability Mainstream Support Ends Extended Support Ends

Internet Security and Acceleration Server 2000 Enterprise Edition

18/03/2001 11/04/2006 12/04/2011

That’s right, kids, it expires on April 12 this year. (The date format is the *cough* correct *cough* UK/AU format above, naturally)

I have fond memories of ISA Server 2000. Actually, now I remember it, the memories were less “fond” and more around being confused by the task pane (I’m a right-click kinda guy), and the documentation, and whether packet filters were something applicable to publishing rules or not. Experience counted for a lot with it, and when it was released, it was a whole lotta new for everyone involved in using and supporting it.

ISA 2000 was where we originally derived the “two minute rule” from for ISA support (at least in Australia): When you’ve made a change, and you’re testing it, give it two minutes. (Saying that caused most type-A admin people to give it at least a minute, and a minute was usually enough for a change to percolate through the system).

I’d been a keen user of Proxy 2.0 at home and at work, on a very early cable modem implementation in Australia (see also: The Lane Cove Effect), and our geeky household upgraded through Windows 2000 betas with Proxy 2.0 patches, until finally ISA 2000 betas became available. Not too long after that, the release version was installed, glistening, on the low-spec former-work-desktop 486 we were using for routing and cheapie IIS hosting duties.

ISA 2000, you served us well. But your time is well and truly past. Bon voyage on the sea of retirement.

If you’re still using ISA 2000, and you’d like to try our new hotness, please try Forefront TMG 2010. The documentation’s better (and most ISA 2004/2006 documentation still applies), and it installs on current Windows versions. Thanks!

Autoproxy might still be broken in current Java runtimes

A customer battling automatic proxy configuration issues with ISA/TMG, and PAC/WPAD.DAT pointed me at the following bug:

http://bugs.sun.com/bugdatabase/view_bug.do;jsessionid=e70c81c1a56f7d856f2e50539c708?bug_id=6887492

Which, if I’m interpreting it right, is Closed. In Connect-speak, that would mean “not being worked on”. (If it was, or is, and a newer version fixes this, please let me know).

From the TMG perspective, a possible workaround is to install the firewall client (or TMG Client as it’s known these days) on the client computer, and turn off the proxy settings at the Java level – this allows the TMG Client to transparently authenticate at the Windows Sockets (winsock) level, bypassing the proxy autodetection script entirely.