During some performance and configuration testing of a Nutanix cluster, I came across an issue where I couldn’t delete a Nutanix datastore (container) from the cluster. I could successfully unmount it from all hosts in the cluster, but I couldn’t actually delete it.

Now, you won’t run into this issue very often, but a clear case where you would is when performing the Nutanix Diagnostics test. This test involves creating a container, mounting it on the ESXi-hosts in the cluster, and deploying several virtual machines on to the datastore.

So far so good, the test will run and you’ll get some nice performance indicators on how your cluster is performing. But during cleanup (thankfully, there is such an option in the test), not everything will be cleaned up, and you’ll be left with a ‘NTNX-NFS-DEFAULT’ container that can’t be deleted from the cluster. Prism, the Nutanix HTML5-based GUI, will keep throwing an error about vDisks not being marked for deletion. This indicates there’s still stuff left on the NFS datastore, obviously, so let’s clean out the datastore of any leftover VMs and VMDKs and try again. No luck. It took me a few minutes to realize that a vDisk isn’t just VMs and VMDKs, but basically any file over 512KB on the NDFS filesystem. Can you think of a file on a VMFS datastore, not a VM or VMDK and larger than 512KB? A quick look with the datastore browser solves the mystery!

protectedlist

While my specific error was caused by the ‘protectedlist’, a part of VMware HA / FDM, the issue applies to (at least) all files over 512KB that remain on the datastore. It might even apply to all files that are left on the datastore, but I haven’t checked.

The general, and easy, solution to this issue: make sure you delete everything (not just files encapsulated in vDisks on the NDFS, but you know, everything) from the datastore before unmounting and deleting the container.

One thing to mention: while looking under the hood with the Nutanix CLI (or ncli for short) to correlate the vDisk ID shown in the original error message to a VM or VMDK, nothing came up. The vDisk ID wasn’t listed with ‘ncli vdisk ls’, and I couldn’t find another way to search for the vDisk…
I’m hoping one of you guys knows of a way to get these files to show up using ncli, or any general way to browse the NDFS filesystem to troubleshoot these kinds of issues server-side…