Reclaim Thin Provisioned Space – Punchzero

Thin provisioning of virtual disks has been a quiet benefit to IT administrators working in virtual environments.  In the recent past of thick provisioned storage as the only option (still common in non-virtualized, local storage environments), administrators made educated guesses about the amount of disk space their systems would need.  Only in the 2008 Server family did Microsoft even include operating system tools to expand disk partitions, an incentive to over-provision storage.

With Thin Provisioned disks, administrators were able to make extremely generous space allocations at the operating system level, while holding off on physical disk space allocation until it was actually used.  This over-provisioning practice often comes with the silent promise that operating system level storage usage would be monitored to see what the steady-state usage becomes.  The administrator might even make a silent promise to no one in particular to examine that usage level and shrink the disk to the steady-state level plus a reasonable safety margin.

I’ll set aside the quite-obvious danger of over-provisioning the physically available capacity aside, and focus on the issue of the creeping high-water mark.  Administrators sometimes make the erroneous assumption that thin provisioned disks will automatically shrink in size allocated by the vSphere hypervisor when files are deleted.  While the operating system might consider a file deleted (removing all pointers in the file allocation table in Windows, for example), the hypervisor doesn’t know about this status change, and thus doesn’t reclaim the space.  Unfortunately, this can lead to situations where a thin disk has a temporary spike in the high-water mark, followed by a drop to the steady-state level, leaving the thin disk allocated at the spiked level.  Or perhaps the operating system isn’t strict about only using previously deleted blocks for new allocation (and why would it be?) so that new space usage doesn’t always over-write previously deleted space. So how can we reclaim thin provisioned space?

There are options that involve storage vMotion (only available at the Enterprise Plus license level in 5.x) or vSphere Converter (make a copy of the workload and disk in a new location) which can result in awkwardly sorting out the naming.  Fortunately there’s actually a native way to reclaim that deleted space. It just requires comfort with and access to the ESXi shell.

vmkfstools [-K | --punchzero ]

Under the hood, this will remove all zeroed blocks from the vmdk. Unfortunately, because of the previously discussed operating system behavior, deleted files don’t necessarily result in zeroed blocks. To maximize the effectiveness of this technique, you should use operating system tools to overwrite all free space in the guest with zeroes. Keep in mind that this will expand the thin disk to it’s maximum. Make sure you have enough space to do it.

For Windows, the Sdelete tool is what you want. You can get Sdelete from TechNet individually or as part of the Sysinternals package.

Since you read my post on using Chocolatey, just install it with a single command.

c:\> cinst sysinternals

And if you’re in a Linux guest, use dd to do the same thing.

dd if=/dev/zero of=/[path]/zeroes bs=4096 && rm -f /[path]/zeroes
# [path] is located on target disk


Keep in mind that this will inflate the disk to it’s maximum size. After the disk has been filled with zeroed blocks, shut down the guest and connect to the ESXi shell. (Remember how to enable the ESXi Shell?)

From the shell, navigate to the directory with the virtual disk you’re re-thinning, then  vmkfstools –punchzero. Here’s the vmdkfstools documentation.

vmkfstools --punchzero [disk-name].vmdk

It’s not ideal to have to schedule downtime to clean up after thin provisioned disk space, but it has to be done [EDIT: Barring Storage vMotion with alternate storage locations]. Monitor your usage and make sure you have enough space to inflate the thin provisioned disk to a fully-allocated state.

Photo by AMagill