TMG Large Logging Queue: No More SQL Lockdowns?

What you say!?


The new logging system in TMG 2010 is seriously cool, and it’s designed to cope with extended instances of SQL Server going away. Extended meaning multi-hour, but depending on disk space, it could be multi-day.


Short Version


There’s a good detailed description of it here, which I’ll try to crystallize:


The most likely reason for the Firewall Service to stop, and for TMG 2010 to enter Lockdown mode, is a lack of disk space on the TMG box.


Yep. If the LLQ files can’t be created, that’s when the Firewall Service will be stopped. If SQL goes away for a few days… well, that’s okay. We got that covered*.


Long But-Not-That-Long Version


But why does that help avoid lockdown mode? More crystallization required:


The kernel-mode FWENG component is now responsible for logging. When everything’s going swimmingly, it logs to a buffer in memory.



The logs in this memory buffer are the logs that might be lost if the server experiences a hard crash – like a blue screen – and that’s why the registry key names governing it are appropriately pessimistic (both in HKLM\System\CurrentControlSet\Services):



(1) Fweng\Parameters\LogQueueMaxLossCount 
(2) Fweng\Parameters\LogQueueMaxLossTimeInSeconds


To get the FWENG buffers out to SQL Server, or MSDE/SQL Express, or text files (more on that later), the Microsoft Forefront TMG Control Service (the service name is still IsaCtrl, by the way) executes a Log Formatter.


The Log Formatter is responsible for taking the raw log information from memory or disk (we’ll get to that), transforming it into the desired output format, and storing it wherever you’ve specified. (The Firewall Service used to do that.)


If FWENG starts exceeding its thresholds – meaning that either the FWENG memory buffers fill up (1) or have remained in memory for Long Enough (2) and haven’t been pulled and stored by IsaCtrl – FWENG will start pushing its Log Queue buffers to disk instead, in .LLQ files.


When the Control service restores its connection to SQL or MSDE, and operations start succeeding again, the logs will be formatted and stored, and the LLQ files deleted as they’re dealt with.


How cool is that? SQL logging here we come!


You mentioned Text Files, but they seem kinda robust?


Well, the log formatter is responsible for transforming log queue records to text format as well, so a text formatting failure would have a similar effect to someone unplugging SQL.


But the reasons I could dream up for that failing are pretty much the same as they currently would be for text logging to fail either way – the disk is full, or very heavily fragmented, or perhaps broken, or Antivirus hasn’t been properly excluded from the path and/or process.


* – pending traffic and disk space. If you can generate 100GB of logs in an afternoon, you’re unlikely to last days before SQL comes back, aren’t ya? Unless you’ve got a billion gig to play with.

More Network Inspection System updates

image


A new Vuln (vulnerability) NIS definition for Outlook Express / Windows Mail MS10-030 joins the recent Expl (exploit) definition for the Sharepoint XSS issue (currently an Advisory).


The other type of signature is a Policy signature – not an exploit or a vulnerability per se, but a security feature an Administrator might want to enable.


NIS is one of the aspects of TMG 2010 I’m most speculatively excited about- imagine being able to more-or-less automatically protect a whole subnet of vulnerable machines while patches are being produced or deployed – and the MMPC have been steadily releasing definition updates that protect TMG-protected machines from known exploits since launch.


NIS updates typically coincide with a bulletin release – but recently, we’ve seen that they’ve been produced for at least one Advisory as well.


NIS updates aren’t particularly discoverable right now, though – I can’t find another way to get notifications of NIS updates other than through the TMG interface.


And NIS entries are not currently searchable in the MMPC Encyclopedia either – they’re linked directly from the definitions in the MMC above.

TmgAdConfig (aka ADConfig, ADConfigPack)

To avoid you tearing your hair out trying to find it:

The tool TMGADCONFIG.exe is included in the ADCONFIGPACK.exe download, available from this location, which extracts to the Program Files(x86)\Forefront TMG Tools\ADCONFIG folder by default.

I was chasing it down with great vengeance and furious anger cos the examples on the web indicated that the AD Marker was stored in the Configuration partition (i.e. at the root of a given domain – one setting for the domain (cue indignant “well what about site awareness!?” on my part)).

The AD Marker, incidentally, is a new method through which the Firewall Client (TMG Client) tries to detect available TMG Servers – it’s intended to be more secure than the existing WPAD.DAT/WSPAD.DAT mechanisms (though they should still work unless you turn them off).

Happily, the –? story differs from the other published works (and included .doc file), and implies much site-aware joy, so I thought I’d republish the detailed help for all of it here, for your viewing pleasure.

TmgAdConfig -?

Forefront TMG Auto-Discovery Configuration Tool
Usage:
    TmgAdConfig.exe add -site <site-name> -type winsock -url <service-url> [-f]
    TmgAdConfig.exe add -default -type winsock -url <service-url> [-f]

    TmgAdConfig.exe del -site <site-name> -type winsock
    TmgAdConfig.exe del -default -type winsock

    TmgAdConfig.exe list [-default] [-site [<site-name>]]

    TmgAdConfig.exe <any-command> -help

Example:
TmgAdConfig.exe add -site My-Site -type winsock -url
http://contoso.com:8080/wspad.dat
    Register the given URL as the winsock proxy service for site My-Site.
TmgAdConfig.exe list -site My-Site
    Print all service markers registered for site My-Site.

 

TmgAdConfig add -help

Forefront TMG Auto-Discovery Configuration Tool
Description:
    Add (register) a new marker in Active Directory.

Usage:
    TmgAdConfig.exe add -site <site-name> -type winsock -url <service-url> [-f]
    TmgAdConfig.exe add -default -type winsock -url <service-url> [-f]

Parameters:
    -site     Active Directory site associated with the new marker.
    -default  Makes the new marker the default for all sites.
    -type     Service type of the marker.
    -url      URL from which to retrieve the service configuration file.
    -f        Force overwrite if the marker already exists.

Example:
TmgAdConfig.exe add -site My-Site -type winsock -url
http://contoso.com:8080/wspad.dat
    Register the given URL as the winsock proxy service for site My-Site.

 

TmgAdConfig del -help

Forefront TMG Auto-Discovery Configuration Tool
Description:
    Remove (unregister) a marker previously registered in Active Directory.

Usage:
    TmgAdConfig.exe del -site <site-name> -type winsock
    TmgAdConfig.exe del -default -type winsock

Parameters:
    -site     Active Directory site from which to remove the marker.
    -default  Remove the site-default marker.
    -type     Service type of the marker to remove.

Example:
TmgAdConfig.exe del -site My-Site -type winsock
    Unregister the current winsock proxy service registered for site My-Site.

 

TmgAdConfig list -help

Forefront TMG Auto-Discovery Configuration Tool
Description:
    List registered markers.

Usage:
    TmgAdConfig.exe list [-site [<site-name>]] [-default]

Parameters:
    -site     List site markers (use without a value to list all sites).
    -default  List the default markers.

Examples:
TmgAdConfig.exe list
    Print all registered service markers.
TmgAdConfig.exe list -default
    Print all default service markers.

 Enjoy!

Slow IO?

(aka IOlwayscallsmethat. OK, horrendous joke based on a bad Dad Joke. Geneva has been notified.)

Exciting news over at the NTDebugging blog – a new Storport.sys update that enables ETW tracing of slow IO, for Windows 2008 and up.

It’s more involved than a registry-key-and-go, but if you’re facing what could be external delays on a SAN-connected box, it’s an excellent way of determining that (since it measures it essentially from the adapter on out).

Xbox Live vs TMG

Foreword – Added 2011-03-08

As far as I’m aware, nothing significant has changed since the blog linked here – ISA Server is now TMG, sure, but XBox Live and TMG don’t officially support one another. This blog post captures something that seems to work for me, but may not work for you. (If you find a better or more reliable way, I’m all ears).

Again – easiest way to ensure XBox happiness is with a compatible non-Strict-NAT router. I don’t have one of them; all I have is an enterprise-security-and-firewalling product as my SOHO router, so I made enterprise-security-and-firewall-ade with it.

Xboooox
(If you just want the “how to set it up” bit without the commentary, skip to the next heading).

Continuing in the tradition of trying to get my game on through ISA Server , I decided to try out the Halo Reach beta tonight, and was promptly stumped when I couldn’t access my Account History (I didn’t need to, as it turns out, but I couldn’t, so it was a challenge , so I wasn’t about to let my girlfiend (not a misspelling) watch TV until I’d fixed it).

The Xbox generally worked fine for games, but frequently in the Marketplace, bad stuff would happen (i.e. an error saying something about not being able to access the marketplace now, but sometimes a retry would work, extremely weirdly).

The logs showed that TMG was intercepting the traffic, running it through the Web Proxy Filter, and noticing that it wasn’t (how to put it nicely) valid , so dumping it, with an error message indicating 13 – The Data Is Invalid. (cue indignant hmph )

With the help of Jim Harrison and Bala Natarajan, I ran through some reconfiguration steps; here’s what I ended up with that works:

How I set it up
Toolbox Objects:

Computers:

  • Xbox (just a name for the XBox’s IP address – you still have to know the IP for publishing rules (each time; can’t just use the computer object), so the Xbox IP should be static/reserved.)

Protocol Definitions:

  • Xbox HTTPTCP/80 Outbound , not based on HTTP base definition, not bound to the Web Filter . That’s important.
  • The next three from before, which seem to work pretty reliably* for online play:
  • Xbox – TCP/3074 Outbound, and 3074 UDP Send and Receive.
  • Xbox TCP Server – TCP/3074 Inbound
  • Xbox UDP Server – UDP 3074 Receive Send

Rules (in this order)

  • 1. Xbox In TCP – Server Publish Xbox IP, using protocol XBOX TCP Server, on External IP
  • 2. Xbox In UDP – Server Publish Xbox IP, using protocol Xbox UDP Server, on External IP
  • 3. Xbox HTTP – Access Rule, Allow only Xbox HTTP, to External, from Xbox IP
  • 4. Xbox Deny Special Rule – Access Rule, Deny only HTTP (that’s normal HTTP , not our special new HTTP), to External, from Xbox IP
  • 5. My general allow/deny rules , including a quite-high-up rule allowing Xbox access to any protocol outbound anywhere (I have that set for all computers, but if you want to be sure, make a special rule just for the Xbox allowing All Outbound to External .) Any {Allow All Outbound} rules must be ordered after that special HTTP Deny rule.

Notes:

  • The reason you need a special Deny rule for regular ol’ HTTP – despite the unbinding of the web filter from the XBox HTTP custom protocol, and being quite specific about the protocol you’re allowing – has to do with the way protocols are collapsed and dealt with by the Firewall engine. For more information, check out Why do I need a deny rule to make an allow rule for a custom protocol work correctly? at the always-amazing Formerly-Known-As-ISA Blog.
  • knowledgeable/nitpicky/interested observers may note that the publishingrules that I have first can actually be pretty much anywhere; I just keep the Xbox rules grouped so they’re all in the one spot. And at the front so they’re processed as quickly as possible; lag bad.
    • Aside: If shifting individual rules a long way up or down, don’t just right-click yourself into RSI – remember you can multi-select rules that are in the way, then right-click and move all of them up or down above or below the rule you’re wanting to shift. It’s not drag-and-drop conweenyent, but then it’s not as susceptible to “Oops I dragged that OU into Domain Controllers” Syndrome either.

Other Settings I’d twiddled but may or may not be relevant:

  • I excluded the Xbox from compression using HTTP Compression exclusions (this shouldn’t be relevant any more with the Deny rule above, as the HTTP filter won’t be inspecting that traffic)
  • I excluded the Xbox from NIS using NIS exceptions (unsure if NIS still fires for tcp/80 when the Web Proxy Filter is out of the way. I guess I could look. Yeah, I’ll do that. After a kill or two. Or eight.)

There. That’s my word count for the month. Ooh, a non-code-locked Blur demo too! I’ve left my console unloved for too long.

More Notes:

  • I’ve seen mention of other ports being required inbound; I haven’t tried them. As far as I can tell, I can do everything through this setup, voice, host games, the works. With the All Outbound allow rule following the special stuff at the top, I haven’t experienced a problem (that I know about)
  • The connection test still reports “Strict NAT” as if it’s a bad thing. That’s OK, I just ignore that.

 

OneNote 2010 vs Mesh

image I installed Office 2010 on my main home machine last night, and on my work PCs today. I love it, it’s very shiny. (And as an aside, while I’m at it – docs.com is amazing).

What was my point again? Oh yes, Mesh. I’ve used either Mesh beta or Foldershare to sync OneNote notebooks since before Office 2007 came out, and it’s generally been excellent, barring the odd bizarre occurrence. I don’t think it’s officially supported (and Mesh being beta, nothing is officially supported) but it does the job.

I had another one of them today: After installing Office 2010 at work, I noticed some replication occurring as the Mesh stargate-y thing rotated away. I had OneNote open at home on my desktop, so I Remote Desktop Mesh-ed into it and closed the Moe icon, but it looked like it was reoccurring any time I had OneNote open anywhere, and all notes in the whole meshed folder (about 20MB) were being pushed up and back, and loads of Replication Conflict messages were being displayed (note to Mesh folk: if you do one thing better next time around, please, make that dialog box better – for eg, “this computer is current right now, make all the other prompts go away based on that.)

Anyhoo, I was poking around in the options and found the Notebook Properties page, closed Mesh everywhere, killed all open instances of OneNote on all three machines, then upgraded the notebook to 2010 format on my work desktop. Since doing that, and firing up Mesh and OneNote on the other computers again, no more replication nastiness.

Just thought I’d mention it in case anyone’s noticing a lot of replication straight after an upgrade – just close everything, change everything on the “master”, and then let that one replicate out, and things should snap back to normal. It seems. So far. All good!

Bamboo-zled!

From the Your Puns Are Without Peer (i.e. they have no friends) department, I noticed that since yesterday’s reboot (the first in a long, long time), the pen had stopped working on the Wacom Bamboo I use with my desktop PC.

I’ve installed a fair wadge of malware recently, just before the last reboot (oh, okay, it was Office 2010, which is brilliant, and Visual Studio 2010, which I really should try at some point, but I’m scared. So scared.) and wondered if one of the above (*cough* Visual Studio *cough*) might’ve updated a runtime or something.

Off to Wacom’s site, and yep, there’s a brand-spanking-new (4th April anyway) driver available, and it works brilliantly again after that (and a restart for good measure). Maybe that’s all I needed anyway. Maybe I didn’t even need that. But I have it, and now we’re very happy together, thanks for asking.

Are you funny enough? Let us help.

I was forwarded a link to the Education Competencies site. What beauty. What simplicity. What abject terror! (seriously, anything with “Competencies” in it means HR involvement, and HR involvement means Catbert, and Catbert is widely understood to be purrr(e) evil).

It had the following gem:

“Timing. There is a time for everything and sometimes humor is not appropriate. Since you are reading this because you or others don’t think you are good at using humor, the best technique is to follow the lead of others.”

Giggle. Er, sorry, it’s not funny. Or is it? Maybe I need to write it down to remember it…

“Being funnier. […] Jot down funny things that happen around you so you can remember them.”

Aww, you’ll make a stand-up comedian of me yet, Microsoft. Hugs.