Balaji Ramamoorthy
Download CV

VMware: "Virtual machine disks consolidation is needed"

Real issue · 2 min read

How to find affected VMs with PowerCLI, check for locks and backup proxy disks, and consolidate safely.

Share on LinkedIn

Symptom

vSphere shows the warning "Virtual machine disks consolidation is needed" on a VM. The snapshot list may look empty, but the VM folder on the datastore still holds -delta.vmdk or -000001.vmdk files, and the VM keeps growing and slowing down.

Where it happened

Virtual machines protected by a backup product, and virtual machines where a snapshot was deleted while the datastore was nearly full or a file was locked.

What I checked

First, how many VMs are affected:

Get-VM | Where-Object { $_.ExtensionData.Runtime.ConsolidationNeeded } | Select-Object Name, VMHost

Then, for the one VM:

  • Free space on the datastore. Consolidation needs room to merge the files.
  • Whether a backup job is running or has failed on that VM.
  • Whether the base disks are attached to a backup proxy VM, which can happen when hot-add backup fails to clean up.
  • Whether a file is locked by another host.

Root cause

A snapshot delete removes the snapshot from the list first, then merges its changes into the base disk. If the merge fails, because of a lock, a lack of space or a backup tool that did not release the disk, the disk chain is left behind and vSphere flags that consolidation is needed.

The fix

  1. Make sure there is free space on the datastore, and no backup job is running on the VM.
  2. Right-click the VM, choose Snapshots, then Consolidate. Or from PowerCLI:
(Get-VM vm-app01).ExtensionData.ConsolidateVMDisks()
  1. If it fails with a file lock error, find the owner of the lock. On the ESXi host, run this against the locked file. The output shows the MAC address of the host that holds the lock:
vmkfstools -D /vmfs/volumes/datastore1/vm-app01/vm-app01-000001.vmdk
  1. If the MAC address belongs to a backup proxy VM, look at that VM's settings for disks that belong to the protected VM. Remove them from the proxy with the option to keep the files, not the option that deletes them from disk. Then run Consolidate again.
  2. If the lock belongs to another ESXi host, restarting the management agents on that host or vMotioning the VM may free it. Do this only with a change record.

How to stop it happening again

  • Alert on the consolidation flag. A scheduled PowerCLI report is enough.
  • Keep datastores below the free-space threshold set in your standards.
  • Do not leave manual snapshots in place for more than a few days, and give each one an owner.
  • Check that the backup tool removes hot-added disks after each job.