ZFS VHDX Growing on Hyper-V 2019

I fiddle about with a charming little PFSense firewall at home, which I recently managed to mangle, so I decided to rebuild it on my new Windows Server 2019 Hyper-V virtualization host. Excitement!

Having thought that at least some of my problems were to do with file corruption, I did some very, very basic research, and decided this new-fangled ZFS thing sounds like the way to go!

It was great until it wasn’t! Here’s a snapshot of my drive size over time:

02/02/2019  08:27 PM     5,674,893,312 PFSense2019ZFS.vhdx
03/02/2019 02:21 PM 7,923,040,256 PFSense2019ZFS.vhdx
03/02/2019 02:30 PM 7,956,594,688 PFSense2019ZFS.vhdx
03/02/2019 02:51 PM 8,090,812,416 PFSense2019ZFS.vhdx
03/02/2019 02:52 PM 8,090,812,416 PFSense2019ZFS.vhdx
03/02/2019 02:52 PM 8,090,812,416 PFSense2019ZFS.vhdx
04/02/2019 03:19 PM 11,244,929,024 PFSense2019ZFS.vhdx
04/02/2019 03:29 PM 11,244,929,024 PFSense2019ZFS.vhdx
04/02/2019 03:59 PM 11,312,037,888 PFSense2019ZFS.vhdx
18/02/2019 03:14 PM 57,248,055,296 PFSense2019ZFS.vhdx
19/02/2019 05:40 PM 60,536,389,632 PFSense2019ZFS.vhdx

So over about 2 weeks, I’d used ~60GB of space(!) Unfortunately the drive size available on this host is only 120GB, so I’m already unable to export a copy locally. Grumble.

So I’m unsure what the moral of the story is. Sorry about that, I was hoping for a stronger conclusion too!

Does ZFS not work nicely with VHDX expanding disks? Does PFSense 2.4.4. chew disk space that it doesn’t report within the OS itself? Can I squeeze another question in before returning to my drive-switch-a-roo export and rebuild?

How To (quickly) Tell If You’re 5 Years Out Of Date On Security Updates

There’s a fun indicator you can use to quickly evaluate whether you’ve been missing security updates for the last five years (ish) on older Operating Systems (i.e. Win2008-2008 R2), and it’s the build number. Not infallible, but then not often wrong.

Helpful Table Of Problem Versions

If you’d rather skip my rambling – and let’s face it, you should – here’s the list of build number indicators which might mean you have an update problem.

  • 10.0.14393.0 – Windows Server 2016 ships with a broken servicing stack which can’t talk to WSUS. (15 months out of date)
  • 6.3.9600.16xxx – (18xxx is current) means Windows Server 2012 R2 or Windows 8.1 without 2919355 applied (~3 years missing updates)
  • 7.6.7600 – Windows Server 2008 R2 or Windows 7 without Service Pack 1 (5 years missing updates)
  • 6.2.6001 – Windows Server 2008 Service Pack 1 (5 years missing updates)

By comparison, good build numbers (as of Sep 2017) are:

  • 10.0.14393.1670 – Win10 or Win 2016 with Sep 2017 CU (anything later than .187 is probably OK)
  • 6.3.9600.18xxx – Win2012 R2 post-2919355
  • 7.6.7601 – Win2008 R2 SP1 / Win7 SP1
  • 6.2.6002 – Win2008 SP2

You can use the WSUS console (yes, even if you’re using SCCM, though you probably have cooler methods available) to quickly evaluate build numbers across your fleet.
In a pinch, you can use AD Users and Computers if you’re just evaluating the third number in the sequence (i.e. doesn’t work for 2919355 or for Windows 2016 boxes).

Back In The Day, Build Numbers Were Even More Useful

Very helpfully, the Windows Vista era introduced incremental build numbers for Operating System versions when Service Packs were applied. So when it shipped, Windows Vista – which you’ll recall came out almost a year ahead of the server equivalent, Windows Server 2008 – shipped with the build number 6000.

Windows Server 2008 shipped with “Windows Vista” Service Pack 1 inbuilt, as it were, and so Vista SP1 and Windows Server 2008 SP1 (i.e. RTM) have the same build number, 6001.

Service Pack 2 followed, again incrementing the build number for both to 6002.

For the Windows 7 era, things were a bit more straightforward. Windows 7 and Windows Server 2008 R2 shipped at about the same time, as build 7600.

When Service Pack 1 was released for both, the build number incremented to 7601.

Quite a few of our Premier Security Assessments pull OS information using WMI from targets, and I sort by the self-reported build number to quickly identify groups of hosts which might not have a Service Pack. It’s very, very infrequently wrong. You could equally do the same by whether “Service Pack X” appears in the CSDVersion, but the build number is a nice, straightforward way of identifying this if you’re collecting it widely.

(AD Computer objects track what appears to be the same information, so querying AD might be a viable option if you’re reasonably certain that the computer objects there are still “live”).

What can you do with this information?

Well, you can say for sure that anything which self-reports as being build 7600 – i.e. not 7601 – probably hasn’t had any Windows security updates since about 2013.

The Support Lifecycle site notes that without SP1, Windows Server 2008 R2 (7600) exited support in April 2013. That’s the point after which security updates stop applying, because they require SP1 (7601), which isn’t installed.

Likewise, if you’ve a Windows Server 2008 (6001) Server, it hit End Of Support at the same time (and Service Pack 2 (6002) is required for any updates beyond that point).

If you haven’t got the relevant Service Pack approved in WSUS (or SCCM), the computers won’t even see updates beyond this point as being applicable. So it might seem like you’ve a bunch of completely updated and compliant servers, (on closer inspection finding lots of updates aren’t applicable to them) but if they haven’t taken the Service Pack, they’re only as updated as they self-report. And they know the newer updates aren’t for them.

In this case, “newer” means “pretty much everything since mid 2013”

What should you do?

So here’s what to do: Pull a report of the OS versions reported by servers within your environment. Clients too, if you think it’s possible some don’t have Win7 SP1.

You could do something like:

  • Start, Run, WinVer on a suspect PC (if it doesn’t say Service Pack X, problem)
  • PS:   get-adcomputer -Filter ‘(OperatingSystemVersion -like “*7600*”) -or (OperatingSystemV
    ersion -like “*6001*”)’ -Properties OperatingSystemVersion,OperatingSystemServicePack | export-csv NoServicePack.csv           #  (a blank NoServicePack.csv = good)
  • Or    wmic /node:servername os get version     – if WMI (RPC) is enabled to the target (in which case, extra bonus security points lost unless you’re using a PAW or management host – you should be firewalling!)
  • Or use WSUS: Turn on the Version column in the All Computers view in the WSUS console, then Group By (or just Sort by) Version and look at the build numbers reported. (Don’t forget to filter by Any)

If there are 7600s or 6001s found, check a few out, and just confirm that they’re not relevant-Service-Pack-less. (Best-case outcome: they’re being misreported.) If they are, try to work out and address the root cause – for eg, the Service Pack update wasn’t approved, or the WSUS catalog doesn’t include the update, or the PC isn’t in the right SCCM update group, or… whatever it is.

As a note, if you’re in that bucket, you’re likely to have many updates to apply, which will likely take some time and disk space to chew through. (If it’s simpler to redeploy an OS with a current build than update an older one, consider that).


And if you’ve found some unpatched boxes as a result of reading this, a) phew, lucky we found them now, and b) really think about that root cause. Mistakes in any human-driven process are predictable: does your process allow for mistakes and have any built-in correction for them? Update management isn’t always easy, but many update policies are geared towards fragility and failure, due to excessive process being required for an update to make it to the target box. A process failure without a corrective phase might result in updates being missed for years.

In some cases, what we hear is that some set of updates are initially rejected (or “deferred”) due to issues or concerns, which is fair enough – but then the decision doesn’t get revisited for months or years afterwards – sometimes never, until the update state is compared with Windows Update. If you don’t look back and check your assumptions – really test what updates are deployed and what you’re still vulnerable to – then things can rapidly and near-invisibly deteriorate, until suddenly, one day you’re looking back at 5 years of unpatched systems.

Core question: If the participants in your existing update process/policy had “just” been pointed directly at Windows Update and set to update weekly, how many Critical and Important updates might have been applied in the interim? Would the outcomes have been better?

And And: an afterthought for 2012 R2

I haven’t got into 2919355 yet, but it’s the 2012 R2 (and Windows 8.1) equivalent of a Service Pack, and as of late 2014, it became the mandatory update on which all other 2012 R2 (and 8.1) updates depended.

If you haven’t installed it, as with the older OSes above, updated would have stopped in – let’s say 2015. So you may be a couple of years behind by now.

I don’t know if it’s as simple as a build check for that one (it might be visible though the detailed build reported by  the WSUS console – I don’t have one to check right now), but it’s the other key update we find missing when evaluating update state using MBSA manually.

From a quick bit of KB spelunking, I figure there might be a way to tell from the WSUS reported client version (but it’d always be a “soft” confirmation) – check out the difference between the file information in the pre- and post-2919355 articles for the same update (while still in the grace period)

Pre (i.e. the version for computers without 2919355)

For all supported x64-based versions of Windows 8.1 and Windows Server 2012 R2

Post (i.e. the version information for computers with 2919355 installed already)

For all supported x64-based versions of Windows 8.1 and Windows Server 2012 R2

So I’ll hazard an ultra-hazardous guess, which is that if you have computers self-reporting in WSUS as being 6.3.9600.16xxx , they might have stalled at pre-2919355, so need 2919355 (or a descendent or prerequisite) approved, and then I assume the build number will be 17xxx or higher. MBSA can help you identify what Windows Update would think was missing, so you can search WSUS for approval states by KB ID.

Disassembling my Notebook Optical Mouse

Unhelpful Preamble

I converted to the Notebook Optical Mouse as a desktop mouse about eighteen months ago, and haven’t looked back. I find that using a heavier mouse is now actively challenging for me – I find the larger type sluggish and unresponsive.

The added battery weight of the Notebook Wireless Optical is enough to put me off – that’s how light this little sucker is. As a result, I own five of them. One for home, two for work, two for the laptop (and just as well, as my girlfriend’s mum’s bird – a cockatiel – gradually chewed through the USB cable while on holiday a year ago, while warming itself near the laptop fan). I’m a big fan. I recommend them heartily.


Fixing the Scroll

The scroll wheel on the home mouse has been getting a bit non-scrolly recently – you can scroll down all you want, but sometimes scrolling up just stuck, and it would take a couple of attempts.

After coming up blank with searches like “disassemble notebook optical mouse” and “open notebook optical mouse” and “break open mouse,” etc, I resigned myself to buying another one. Then pulled out the tool kit. (Hey, there’s no risk; I’m buying another one anyway, right?)

Cutting a short story long, remembering there are no user serviceable parts, if you do decide to void your warranty (*I guess; didn’t bother looking) by opening one, here’s how to avoid my mildly-finish-damaging initial mistakes in opening it:

  • There’s a screw.

Once you understand that, everything else becomes easy! (A lot easier than breaking the glued-on red side panels off with a screwdriver, while looking for any magic unlock clips. Hmm.)

The screw is under the large black skid pad at the bottom of the mouse.

The last place I looked – I did actually check the front feet, and the hologram logo as they looked like the most likely screw-hiders. But peel the black thing off with something sharp (mine re-stuck with no problem), and once the screw is removed, you can lift the top from the Microsoft logo bit – there are a couple of clip/hinge/catch things, one each side of the mouse cable. Once the screw is gone, it’s easy.