Balaji Ramamoorthy
Download CV

VMware vMotion fails at 14 or 21 percent: check the MTU first

Real issue · 2 min read

A vMotion that starts and then times out is usually an MTU or network path problem. The vmkping tests that prove it.

Share on LinkedIn

Symptom

A vMotion starts, then stops partway through, often at 14 or 21 percent. The task ends with an error such as "Timed out waiting for migration data" or "The vMotion failed because the destination host did not receive data from the source host".

Where it happened

A cluster of ESXi hosts with a dedicated vMotion network, after a network change, a new host being added, or a change to jumbo frames.

What I checked

  1. That each host has a VMkernel adapter with the vMotion service enabled, on the expected subnet.
  2. That the source and destination vMotion addresses can reach each other.
  3. That the MTU is the same on the VMkernel adapter, the virtual switch and every physical switch port in the path.
esxcli network ip interface list
esxcli network ip interface ipv4 get

Test from the source host with the vMotion network stack, using the destination host's vMotion address:

vmkping ++netstack=vmotion -I vmk2 192.168.50.12
vmkping ++netstack=vmotion -I vmk2 -d -s 1472 192.168.50.12
vmkping ++netstack=vmotion -I vmk2 -d -s 8972 192.168.50.12

The -d option forbids fragmentation. Use -s 1472 to test a path with an MTU of 1500, and -s 8972 to test a path with an MTU of 9000. A small ping that works while the large one fails points to an MTU mismatch.

If the host does not use the separate vMotion stack, leave out ++netstack=vmotion.

Root cause

The most common cause is an MTU mismatch. The small packets used to start the migration get through, so the task begins, and then the large data packets are dropped somewhere along the path. Other causes are a firewall or access list between the vMotion subnets, a wrong VLAN on one host, or a duplicate IP address on the vMotion network.

The fix

Make the MTU the same everywhere. If the design uses jumbo frames, the VMkernel adapter, the virtual switch or distributed switch and every physical port must all be set to 9000. If you cannot change the physical network now, set the VMkernel adapter and virtual switch back to 1500 as a temporary measure. Check VLAN settings and look for a duplicate address with the ping test. Run the vMotion again.

If the ping tests pass and the vMotion still fails, read the logs on both hosts around the time of the failure: /var/log/vmkernel.log and /var/log/hostd.log, and the vmware.log file in the VM folder.

How to stop it happening again

  • Write the MTU and VLAN for each network in the design document, and check them after every network change.
  • Add a vMotion test between every pair of hosts to the checklist for adding a host.
  • Keep the vMotion network on its own VLAN and subnet, and record the addresses in one place.