vSAN All-Flash Disk Group Isolation During a Live Change Window: Anatomy of a P1 Recovery

After-Action Review / Issue · 4 min read

When a cache-tier NVMe failed during off-peak maintenance and triggered controller queue exhaustion, here is how we prevented split-brain, evacuated components safely, and restored 100% data integrity.

Share on LinkedIn

During a live off-peak maintenance window across a 16-node VMware vSAN all-flash cluster hosting tier-1 SQL Server and ERP databases, an unexpected hardware failure turned a standard ESXi rolling patch cycle into a high-stakes storage recovery operation.

A single NVMe cache-tier drive suffered an unrecoverable controller failure, but instead of failing fast and gracefully dropping offline, the device began throwing intermittent SCSI sense timeouts, leading to controller queue exhaustion and partial disk group isolation.

Here is how we stabilized the cluster, prevented split-brain data corruption, and safely evacuated and rebuilt the disk group with zero data loss and zero guest OS crashes.


1. The Incident Scenario

  • Estate: 16 Dell PowerEdge R740xd nodes, dual Intel Xeon Gold, 768 GB RAM per host, two all-flash vSAN disk groups per host (1x 1.6 TB NVMe write-cache + 4x 3.84 TB SAS SSD capacity tier).
  • Storage Policy: RAID-6 (Erasure Coding, FTT=2) with checksum validation enabled.
  • Trigger: Host ESX-NODE-07 entered Maintenance Mode with Ensure Accessibility. While virtual machines were migrating off the host, host ESX-NODE-03 suddenly reported: vSAN Disk group 0 on host ESX-NODE-03 in state: Degraded / Error Storage Object Health: Reduced Availability with no per-object tolerance

Because one host was already in maintenance mode and another host experienced a cache tier failure, the cluster had two concurrent fault domains compromised!


2. Immediate Diagnostic Triage

vSAN architecture mandates that if a cache tier device fails, the entire disk group (including its 4 capacity drives) is taken offline to maintain cache coherency.

Connecting via SSH to the affected host:

# Query vSAN physical disk status and health
esxcli vsan storage list

# Inspect physical disk response times and queue depth
esxtop -> press 'd' (disk adapter view)

The output showed:

  • NVMe cache drive naa.5002538c00000001 had average response times exceeding 18,500 milliseconds (18.5 seconds)!
  • The LSI SAS/NVMe controller queue was 100% saturated with queued abort commands.
  • The drive was failing to respond to SCSI SYNCHRONIZE CACHE (10) primitives, but had not yet tripped the controller's hardware death detection.

This is the most dangerous state in storage engineering: the "grey failure" (limping device), where the hardware neither works nor dies, starving adjacent I/O queues.


3. The Recovery Protocol: Step-by-Step

Step 1: Prevent Guest I/O Stalls by Isolating the Rogue Device

Rather than waiting for the hardware timeout threshold to be reached, we forcibly detached the failing disk group from the ESXi kernel to release the controller queue locks:

# Forcibly unmount the degraded disk group from vSAN
esxcli vsan storage diskgroup unmount -d naa.5002538c00000001

Instantly, the queue latency across ESX-NODE-03 dropped from 18,500ms back to sub-1ms. Virtual machines on the remaining disk group on ESX-NODE-03 resumed normal I/O immediately without guest OS BSODs or file system corruption.

Step 2: Mitigate Clustered Risk (Cancel Pending Maintenance)

With two hosts down, the cluster was operating on the edge of its failure tolerance. We immediately cancelled the maintenance mode on ESX-NODE-07, verifying its disk groups rejoined the cluster quorum cleanly before proceeding:

# Verify vSAN cluster membership and quorum state
esxcli vsan cluster get
# Sub-Cluster Member Count: 16
# Health: Healthy (Quorum established)

Step 3: Evacuate Objects and Replace the Cache NVMe

Once quorum was solidified:

  1. Replaced the failed NVMe drive using hot-swap carrier.
  2. Verified the new drive was detected by the perccli utility: /opt/MegaRAID/perccli/perccli64 /c0 show
  3. Re-created the disk group in vSAN:
# Recreate disk group with new cache NVMe and clean capacity disks
esxcli vsan storage diskgroup add -s naa.5002538c_new_cache -d naa.5002538d_cap1 -d naa.5002538d_cap2 -d naa.5002538d_cap3 -d naa.5002538d_cap4

Step 4: Component Resynchronization & Scrubbing

vSAN immediately triggered background resynchronization. To ensure the rebuild did not impact production database IOPS, we throttled resync bandwidth via the vSAN performance policy until peak business hours concluded.


4. Architectural Lessons for High-Availability Storage

  1. RAID-6 (FTT=2) Saved the Data: If the cluster had been configured with standard RAID-1 (FTT=1), a concurrent maintenance mode and cache failure would have resulted in immediate data unavailability. Always use FTT=2 for clusters of 8 nodes or more.
  2. Beware the "Limping" NVMe Drive: A dead drive is easy to handle; a drive responding in 18 seconds will drag down an entire storage controller. Learn how to forcibly unmount degraded devices quickly.
  3. Always Verify Quorum Before Performing Maintenance: Never enter maintenance mode on host N+1 until host N has fully exited and resynchronized all degraded components.