Website Security Suggestion: Get rid of cruft! (script included)

Right: One of my pet hates is cruft on a production website.

Cruft is stuff – files – which has accumulated because nobody’s paying attention. Cruft includes sampleware. Developer experiments. Readmes. Sample configs. Backups of files which never get cleaned up. Just general accumulated stuff. It’s website navel lint. Hypertext hairballs.

Cruft. Has. No. Place. On. A. Production. Website!

Worst-case, it might actually expose security-sensitive information. (That’s the worst type of cruft!).

Want to find cruft? Well, easiest way to start is:

D:\WebContent> dir /s *.txt

That’s a good start. For every Readme.txt, add 10 points. For every web.config.txt, add 1000 points (why? That’s a potentially huge problem – .config is blocked by Request Filtering by default (with certain exceptions), but .config.txt: no problem! Download away.)

If you score more than 10 points, you need to rethink your strategy.

  • There is no reason for files like readme.txt to exist within your production website
    • Okay, there’s one reason and that’s when you’re providing one you know about, and have vetted, for download.
      • I mean, obviously if the site is there to provide readme.txt s for apps people are downloading, great! But if it’s the readme for some developer library which has been included wholesale, bad pussycat.
  • There is no reason for files like web.config.bak to exist within your production website.
    • Luckily, .bak files aren’t servable with the default StaticFileHandler behaviour. But that doesn’t mean an app (or * scriptmap…) can’t be convinced to hand you one…
  • If you have web.config.bak.txt files, you’re asking for trouble.
    • Change your operational process. Don’t risk leaking usernames and passwords this way.

The Core Rationale

Web developers and site designers should be able to explain the presence of every single file on your website.

I don’t care if it’s IIS or Apache or nginx or SuperCoolNewTechnologyX… the developers should be responsible for every single file deployed to production.

And before the admins (Hi!) get smug and self-satisfied (you still can, you just need to check you’re not doing the next thing…), just check that when you deploy new versions of Site X, you’re not backing up the last version of Site X to a servable content area within the new version of Site X.

For example, your content is in F:\Websites\CoolNewSite\ with the website pointed to that location…

  • It’s safe to back up to F:\Backups\CoolNewSite\2016-11-13 because it’s outside the servable website
  • It’s not cool to back up to F:\Websites\CoolNewSite\2016-11-13 because that’s part of the website.

How Do I Know If I’m Crufty?

As I do, I started typing this rant a while ago, and then thought: You know what? I should script that!

I had a bunch of DIR commands I was using, and sure, could’ve just made a CMD, but who does that these days? (Says my friend. (Singular))

Then {stuff}… but it finally bubbled to the top of my to-do list… So I wrote a first draft Get-CruftyWebFiles script.

I’ve lots of enhancement ideas from here, but wanted to get something which basically worked. I think this basically works!

Sure, there’s potential duplication if sites and apps overlap (i.e. the same file might be listed repeatedly) (which is fine; I figure you weed that out in post production), and if your site is self-referential it might get caught in a loop (hit Ctrl+C if you think/know that’s you, and *stop doing that*)

So, feel free if you want to see how crufty your IIS 7.5+ (assumed? Tested on 8.5) sites are:

The Script: https://github.com/TristankMS/IIS-Junk

Usage (roughly):

Copy to target web server. Then from an Admin PS prompt:

  • .\Get-CruftyWebFiles.ps1   # scans all web content folders linked from Sites, and outputs to .\crufty.csv
  • .\Get-CruftyWebFiles.ps1 -WebSiteName “Default Web Site”     # limits to just the one website.
  • .\Get-CruftyWebFiles.ps1 -DomainName “YOURDOMAIN”    # checks for that text string used in txt / xml files as well

Pull the CSV into Excel, Format as Table, and get sorting and filtering. Severity works on a lower-is-more-critical basis. Look at anything with a zero first.

Todo: Cruft Scoring (severity’s already in there), more detections/words, general fit and finish. Also considering building a cruft module for a security scanner, or just for the script, to check what’s findable on a website given some knowledge of the structure.

* oh! No I’m not

App Pool Recycling Defaults: Why 1740 minutes?

Without doubt, one of the most FAQ when discussing Application Pools in IIS Admin and Troubleshooting workshops!

Scott Forsyth shares Wade’s answer

In Wade’s words: “you don’t get a resonate pattern”.

then follows up with useful advice on establishing your own best recycling interval:

First off, I think 29 hours is a good default. For a situation where you don’t know the environment, which is the case for a default setting, having a non-resonate pattern greater than one day is a good idea.

However, since you likely know your environment, it’s best to change this.

Lots of useful information and commentary in a post that’s sure to become a genre classic. Really. No sarcasmo.

IIS 7: But why do I get a 500.19 – Cannot add duplicate collection entry- with 0x800700b7 !?

(Because I’m sure that was your exact exclamation when you hit it!)

Also applies to IIS 7.5 (Windows Server 2008 R2), IIS 8.0 (Windows Server 2012), IIS 8.5 (Windows Server 2012 R2) and IIS 10 (Windows Server 2016).

The Background

This week, I was out at a customer site performing an IIS Health Check, and got pulled into a side meeting to look at an app problem on a different box.

The customer was migrating a bunch of apps from older servers onto some shiny new IIS 7.5 servers, and had hit a snag while testing one of these with their new Windows 7 build.

To work around that, they were going to use IE in compatibility mode (via X-UA-Compatible), but adding HTTP response headers caused the app to fail completely and instantly with a 500.19 configuration error.

We tested with a different header (“X-UA-Preposterous”) and it had the same problem, so we know it’s not the header itself.

“Now that’s interesting!”

At first I thought it was app failure, but as it turns out…

The Site Layout

This becomes important – remember I noted that the app was being migrated from an old box to a new one?

Well, on the old box, it was probably one app of many. But the new model is one app per site, so a site was being created for each app.

The old location for the app was (say) http://oldserver/myapp/, but the contents were copied to the root of http://newsite/ on the new server.

To allow the app to run without modification to all its paths, a virtual directory was created for /myapp/ (to mimic the old path) which also pointed to the root of newsite.

image

So myApp above points to c:\inetpub\wwwroot , and so does Default Web Site.

Setting up the problem

So, using the GUI, I set the X-UA-Compatible header to IE=EmulateIE7. The GUI wrote this to the web.config file, as you can see in the status bar at the bottom:

image

Browsing to a file in the root of the website works absolutely fine. No problem at all.

Browsing to anything in the /myApp/ vdir, though, is instantly fatal:

image

If you try to open HTTP Response Headers in the /myApp/ virtual directory, you also get a configuration error:

image

What does that tell us? It tells us that the configuration isn’t valid… interesting… because it’s trying to add a unique key for X-UA-Compatible twice.

Why twice? Where’s it set? We’re at the site level, so we checked the Server level HTTP Response Headers… blank.

But… it’s set in a web.config file, as we can see above. And the web.config file is in the same location as the path.

Lightbulb moment

Ah. So we’re probably processing the same web.config twice, once for each segment of the url!

So, when the user requests something in the root of the site, like http://website/something.asp:

1. IIS (well, the W3WP configuration reader) opens the site’s web.config file, and creates a customheaders collection with a key of X-UA-Compatible

2. IIS Serves the file

And it works. But when the user requests something from the virtual directory as well – like http://website/myApp/something.asp

1. IIS opens the site web.config file, and creates a customheaders collection with a key of X-UA-Compatible

2. IIS opens the virtual directory web.config file (which is the same web.config file) and tries to create the key again, but can’t, because it’s not unique

3. IIS can’t logically resolve the configuration, so responds with an error

Options for Fixing It

1. Don’t use a virtual directory

(or rather, don’t point the virtual directory to the root of the website)

This problem exclusively affects a “looped” configuration file, so if you move the site contents into a physical directory in that path, it’ll just work.

There will be one configuration file per level, the GUI won’t get confused, and nor will the configuration system.

Then you just use a redirecting default.asp or redirect rules to bounce requests from the root to /myApp/ .

2. Clear the collection

You can add a <clear /> element to the web.config file, and that’ll fix it for any individual collection, as shown here:

<customHeaders>
      <clear />
<add name=”X-UA-Compatible” value=”IE=EmulateIE7″ />
</customHeaders>

The clear element tells IIS to forget what it thinks it knows already, and just go with it. (When you break inheritance of a collection in the GUI, this is what it does under the covers).

The problem with this approach is that you need to do it manually, and you need to do it for every collection.

In our case, we had Failed Request Tracing rules as well which failed with the same type of error, promptly after fixing the above problem, confirming the diagnosis.

3. Move the configuration

And this splits into two possible approaches:

3a. Editing Applicationhost.config by hand

You can remove the web.config and use a <location path=”New Site/myApp”> tag in applicationhost.config to store configuration, and that’ll work until someone uses web.config again.

3b. Using Feature Delegation

If you do want to prevent web.config being used, you can use the Feature Delegation option to make all properties you don’t want people to set in web.config files Read-Only. (aka “lock” those sections). “Not Delegated” would also work.

image

This can be done per-Site using Custom Site Delegation, or globally.

And! This has the added happy side-benefit of making the GUI target applicationhost.config, rather than creating or modifying web.config files for that site.

 

Hope that helps you if you hit it…

TMG Rollup 3 out now; so’s Mod_Security for IIS

TMG SP2 Update Rollup 3

As the ISA Blog mentions, Rollup 3 for TMG Service Pack 2 is now available:

We are happy to announce the availability of Rollup 3 for Forefront Threat Management Gateway (TMG) 2010 Service Pack 2 (SP2). TMG SP2 Rollup 3 is available for download here: Rollup 3 for Forefront Threat Management Gateway (TMG) 2010 Service Pack 2

Please see KB Article ID: 2735208 for details of the fixes included in this rollup.

The Build Number for this update is: 7.0.9193.575

Fair number of new fixes included and it looks like a worthwhile update. I’m putting it on my home TMG box tonight. As a reminder, the hotfix rollups are cumulative for a given Service Pack, so if you’re already at Service Pack 2 (and you should be) you just need SP2UR3 if you skipped UR1 or UR2.

Mod_Security for IIS

In other security-related news, mod_security for IIS hit a stable release at 2.7.2, as the SRD blog notes:

We are pleased to announce the release of a stable version of the open source web application firewall module ModSecurity IIS 2.7.2. Since the announcement of availability of the beta version in July 2012, we have been working very hard to bring the quality of the module to meet the enterprise class product requirements. In addition to numerous reliability improvements, we have introduced following changes since the first beta version was released:

  • optimized performance of request and response body handling
  • added “Include” directive, relative path and wildcard options to the configuration files
  • re-written installer code to avoid .NET Framework dependency and added installation error messages to system event log
  • integrated OWASP Core Rule Set in the MSI installer with IIS-specific configuration
  • fixed about 10 functional bugs reported by ModSecurity IIS users.

Microsoft also released recently a TechNet article entitled "Security Best Practices to Protect Internet Facing Web Servers", which explains in details benefits of deploying a WAF module on a web server.

The Technet article referenced above is worth a read if you’re charged with delivering IIS web server security for random applications!

 

Where’s Waldo?

I’m spending more time editing the MSPFE blog than here at the moment, so if you’re missing my quippy, irreverent style… tough! (But I still love you. Happy Valentine’s day! (No gifts for you this year.))

IE10 Compat Inspector

There’s an very handy-looking new tool that can be used to quickly determine a site’s compatibility with IE10.

I set it up with Fiddler on one of my machines, and can now enable a pop-up item in Fiddler under the Rules menu.

I wasn’t familiar with Fiddler’s rules engine before… more investigation is needed!

IUSR vs Application Pool Identity – Why use either?

(pasted from my email clippings. I’m on holiday right now, catching up on paperwork!)

The TLDR version is: using AppPoolIdentity as both the App Pool Account and Anonymous user account lets you have multiple isolated anonymous websites on one box.

IIS 7.x and upwards (as of Win2008 R2 and Windows 2008 SP2, also in IIS 8.x in Windows Server 2012 and IIS 10.x in Windows Server 2016) supports a new Application Pool account type, called an ApplicationPoolIdentity. This low-privileged account can be used to isolate distinct sets of anonymous website content, without requiring the administrator to set up a unique account for each website manually.

So whereas the default IUSR anonymous account is per-server, an ApplicationPoolIdentity is per-app-pool, and IIS creates one app pool per site, by default when the GUI is used to create a site.

So by setting the ApplicationPoolIdentity as the anonymous user account for a site, you can isolate content and configuration for that site so that no other sites on the same box can access it, even if it’s an anonymous site.

And now, the long version!

 

Before I start: Terminology disambiguation corner (because App Pool Identity is a horribly overloaded term nowadays):

  • Application Pool Account = the account used to run the App Pool, whether custom user, NetworkService, LocalService, AppPoolIdentity or LocalSystem
  • ApplicationPoolIdentity = the new account type that has a unique App Pool Name-based identity SID (S-1-5-82-SHA1{App Pool Name})

Also, a reminder that process identity is the basic “RevertToSelf” identity for a process, and that thread identity can be different from process identity via impersonation or explicit logon.

So, any or all of the threads in a process might be someone other than the process identity, but if any call RevertToSelf or somehow lose their token, they’ll snap back to acting as the process identity. (Which is the ultra-short version of why you don’t want that being LocalSystem or another privileged account.)

 

App Pool Account:

The when-not-impersonating/process identity; used to start the app pool and to read web.config files; pretty much needs permissions to everything.

On IUSR vs Application Pool Account as anonymous:

IUSR

  • IUSR has the same SID on every machine.
  • IUSR is appropriate if you run one anonymous website on the computer.
  • You secure your content to IUSR with NTFS permissions, and that website can access it.
  • If you run two websites with the anonymous account as IUSR, they can access each other’s content.
  • For low-security applications and intranet sites, that might be OK.

App Pool Account as Anonymous

The alternative is to use an App Pool Account as the Anonymous account (so a thread doesn’t bother putting on its IUSR clothes on anonymous requests)

  • ApplicationPoolIdentity has the same SID on every machine with a common config (because the SID is a hash of the name), so has the same benefit as IUSR for content security, only specific to the app.
  • It’s an appropriate choice if you run multiple anonymous websites and need isolation of content.
  • Other appropriate choice: creating an explicit user account for each App Pool and using that as anonymous.
  • (i.e. the anonymous Coke application should never be able to read the Pepsi application’s files) (arguably always the case with multiple anon websites on the one box)

Using the App Pool Account as anonymous is a good idea because it allows you to secure your content at the NTFS level for just COMPUTER\Coke or IIS AppPool\Pepsi, and be assured that Windows file system security will prevent one company’s anonymous app from reading (or otherwise affecting) its competitor’s anonymous content.

Using the AppPoolIdentity as the App Pool Account in this case is just a simple, no-hassle way of having a common user account on all machines that share the IIS configuration (or at least the name of the app pool), without having to faff about creating or replicating Windows users and worrying about their permission level.

The bit I’m less confident on but still fairly sure I’m right:

When it gets to off-box (eg database) resources, you’re out of IIS-land and into app framework (ASP.Net)-land; short version is that if your token isn’t delegable (for eg, comes from an NTLM auth), it’ll fail to be passed to the next hop, and you’ll end up with process identity and any limitations/benefits it incurs.

Configuring Kerberos for SharePoint farms – a generic gotchas list

Recently, I worked on a Kerberos configuration issue with a customer; these are my notes from the visit.

You’ll see some common themes with Kerbie Goes Bananas, and it puts much of that into practice. Speaking of, I must redo Kerbie with SetSPN -S  (shameface)

 

1. DNS should use an A record to refer to the load balancing IP, not a CNAME

This configuration step avoids an Internet Explorer behaviour whereby IE resolves a CNAME into an A record, and requests a ticket by building an SPN for the A record, instead of the CNAME.

More information is available at http://support.microsoft.com/kb/938305 . In most cases, adjusting the behaviour of Internet Explorer across all machines is harder than adjusting the DNS entry involved.

2. SPNs must be registered against the Application Pool Account

Note: use the Windows 2008 (or later) version of SetSPN to identify problems such as duplicates when updating SPNs. Any existing document using SETSPN -A should be updated to use SETSPN -S.

Only two SPNs are required for Kerberos to function against a farm – the FQDN, and the short hostname.

These must be applied to the account used by the Application Pool receiving the user request, which practically means that in most cases, only one account is usable per hostname (pair).

SPNs to be registered are:

HTTP/farm
HTTP/farm.example.com

Against the user identity of the Application Pool the user is connecting to – say, DOMAIN\SPAccount. This must be a domain account when used in a Farm scenario.

Note that no port number is used for the default port, and that these SPNs are also used for TLS/SSL.

SETSPN -S HTTP/farm DOMAIN\SPAccount
SETSPN -S HTTP/farm.example.com DOMAIN\SPAccount

If the individual hostname is to be used occasionally (e.g. for troubleshooting), http/machinename and http/machineFQDN should be registered against that account as well.

This should result in a list of SPNs as shown:

setspn -l DOMAIN\SPAccount

Registered ServicePrincipalNames for CN=SharePoint App Pool Account,OU=Service Accounts,DC=example,DC=com:

HTTP/farm

HTTP/farm.example.com

3. The App Pool Account must be used for authentication

In a web farm scenario, a domain account must be used as the application pool identity. Once a suitable domain account is configured as the application pool identity (DOMAIN\SPAccount in this example), Kernel-Mode Authentication must be disabled, or the configuration’s useAppPoolCredentials property must be set to true (both may be used).

If this step is not performed, the app pool will not be able to decrypt the Kerberos ticket supplied by the client.

To disable Kernel-mode Authentication

Open InetMgr (IIS Manager), browse to Authentication for the site, click Windows Authentication and open Advanced Settings (Actions pane on the right), and untick “Use Kernel-mode Authentication”.

Reference: http://technet.microsoft.com/en-us/library/cc754628(WS.10).aspx

To set useAppPoolCredentials to true:

Open a CMD window as Administrator, then:

CD %windir%\system32\inetsrv

appcmd.exe set config -section:system.webServer/security/authentication/windowsAuthentication -useAppPoolCredentials:true

Note: one line (wrapped), with no space after any dash (-) character.

Reference: http://technet.microsoft.com/en-us/library/dd759186.aspx

4. Performance – Kerberos and NTLM

Use of Kerberos should significantly reduce traffic between WFEs and Domain Controllers.

Every NTLM-authenticated connection requires the server to make a connection to a DC to complete authentication. The number of connections available to a DC simultaneously is governed by MaxConcurrentApi registry value.

Kerberos allows the client to authenticate to a DC once for the website, and to continue to use the ticket for the ticket lifetime (10 hours by default), across multiple connections, without necessarily needing to interact with the DC again.

References

MaxConcurrentApi
http://support.microsoft.com/kb/326040 (original article)
http://support.microsoft.com/kb/975363 (now supports 150)

Kerberos vs NTLM authentication with ISA Server (same concepts apply with Sharepoint or any Web app)
http://technet.microsoft.com/en-us/library/bb984870.aspx

And a third-party performance comparison of Kerb and NTLM authentication with kernel-mode authentication and without was found here (not overall site performance, just basic RPS).

http://blog.michelbarneveld.nl/michel/archive/2009/12/02/kernel-mode-authentication-performance-benefits.aspx

PSA: You really need to update your Kerberos setup documentation with SetSPN -S!

Hi!

You might remember me from such posts as Kerbie Goes Bananas, and SetSPN improvements for Windows 2008. Or something.

I’m here with a public service announcement! Excitement!

It’s been long enough since Windows 2008 (and the downlevel release of SetSPN) that I feel comfortable respectfully asking you to please:

Search and Replace SetSPN -A with SetSPN -S.

In your organization, if you ever happen to run across a document that describes a procedure that looks anything like this:

SetSPN -A http/yourwebfarm DOMAIN\YourFarmAccount

Please:

  •  mail the author, or
  •  file a bug against the content, or
  •  use the Community Content feature if it’s somewhere on Technet, or
  •  mail anyone and everyone responsible for upkeep or implementation of that document

to change the SETSPN -A command to a SETSPN -S.

You may need to include a foreword describing where to get the 2008 version of SetSPN (I think I may have just spoiled it for you) if you’re still strongly a 2003/XP shop, with no newer SetSPN-toting OSs available.

Why the change?

Because it’ll hurt you less in the long run.

The original release of SetSPN was strongly account-centric. Given a Windows account, it would let you:

  • Add an SPN to that account
  • Remove an SPN from that account
  • List the SPNs associated with that account

Unfortunately, this makes it very easy to add the same SPN to multiple accounts – creating a duplicate SPN. This is a very bad thing.

The same SPN can’t easily be added more than once to the same user account, but the original tool does nothing to prevent the same SPN being added to multiple user accounts – and unfortunately, that’s exactly the situation you’re trying to avoid.

BAD EXAMPLES BAD BAD DO NOT USE BAD

Any of

  • SETSPN -A http/farm DOMAIN\FarmUser
  • SETSPN -A http/farm DOMAIN\FarmComputer$

or

  • SETSPN -A http/farm DOMAIN\FarmComputer1$
  • SETSPN -A http/farm DOMAIN\FarmComputer2$

or

  • SETSPN -A http/farm ANYTHING followed by
  • SETSPN -A http/farm ANYTHING_ELSE

breaks kerberos for http://farm.

To restate the rule: One SPN can be associated with precisely one account.

So please, use SetSPN -S

And that’s exactly what SETSPN -S is designed to prevent. SETSPN -S performs a quick check for duplicates before adding an SPN – which is the best possible time at which to catch the problem. So yay-the-Windows-2008-AD-team.

Duplicates! Gotta Catch ‘Em All 2011 Edition

If you suspect you have duplicate SPNs in your environment, well, why just suspect? Run

  • SETSPN -X

To be told explicitly what duplicates you have kicking around in AD (there are forestwide switches you can use too). Yep, that used to be a nasty LDIFDE export with an LDAP filter expression; much simpler now!

 

DebugDiag 1.2 (64-bit capable, .Net 2.0+ compatible) released

Great news, everybody! Wait, that was Farnsworth-y – no, really, it’s great news!

DebugDiag 1.2 (or to give it its full title, the Debug Diagnostic Toolkit) has been released to the web and is available from the Microsoft Download Center.

Notes from the email that described this release:

Analysis:

· .Net 2.0 and higher analysis integrated to the Crash Hang analysis.

· SharePoint Analysis Script.

· Performance Analysis Script.

· .NET memory analysis script (beta).

· Native heap analysis for all supported operating systems

Collection:

· Generate series of Userdumps.

· Performance Rule.

· IIS ETW hang detection. 

· .NET CLR 4.0 support.

· Managed Breakpoint Support.

· Report Userdump generation to the Event log.

Deployment

· Import/Export of rules and configuration, including ‘Direct Push’ to remote servers.

· Enterprise deployment support using XCopy and Register.bat.

Non-supported items

· x64 userdump analysis on x86 systems.

· Installing x86 DebugDiag on x64 systems.

· Installing 1.2 and 1.1 DebugDiag on the same system.

· 1.2 Memory leak analysis of 1.1 leaktrack.

· Analysis of x86 Userdumps generated by x64 debugger.

Notes about this release:

– Uninstall all previous DebugDiag versions before you install DebugDiag 1.2.

DebugDiag makes analysis of IIS (and sometimes random process) user mode memory dumps a cinch. This long-awaited release should be helpful to any IIS admin or developer trying to troubleshoot problems with their web site or server.