Locked files 101

Locked files ?

Since  a few months we see more and more “consolidation needed messages” in our vCenter environment, we’re still searching for the reason. Of course we used some VMware articles to get in a bit deeper. But for a successful we need to remove the file lock. I made a little procedure which worked best for us. The information also can be found in the VMware KB’s but sometimes it’s a bit confusing because you have different approaches in different versions.

Investigating virtual machine file locks on ESXi/ESX (10051)Unable to delete the virtual machine snapshot due to locked files (2017072)
Unable to perform operations on a virtual machine with a locked disk (1003397)
Investigating hosted virtual machine lock files (1003857)

Find locked files

It’s easiest to power off the VM. So you’re sure there shouldn’t be any lock anymore. If you’re not sure or have a lot of disks it can be a b*tch to check for locks using the vmkfstools -D. It’s doable, but I found a easier way using the touch command.

Start a SSH session to one of the ESXi hosts in the cluster where the VM resides you want to investigate.

Move to the location of the VM on the datastore: # cd /vmfs/volumes/<datastore>/<VMname>
Now use the touch * command to change the datestamp of all file in this directory. If you have  a lock you will get output like :

touch: VM1-flat.vmdk: Device or resource busy
touch: VM1_3-flat.vmdk: Device or resource busy

Who owns me?

We know now that the 2 files above are in some kind of lock and cannot be written to. Now we can move on using the vmkfstools command to see which ESXi host owns the file.

# vmkfstools -D VM1-flat.vmdk
Lock [type 10c00001 offset 13957120 v 448, hb offset 3510272
gen 41, mode 2, owner 00000000-00000000-0000-000000000000 mtime 845230
num 1 gblnum 0 gblgen 0 gblbrk 0]

We now see 2 things Mode 2 which indicates the kind of lock. See the list below with the kind of modes there can be:

• mode 0 = no lock
• mode 1 = is an exclusive lock (vmx file of a powered on VM, the currently used disk (flat or delta), *vswp, etc.)
• mode 2 = is a read-only lock (e.g. on the ..-flat.vmdk of a running VM with snapshots)
• mode 3 = is a multi-writer lock (e.g. used for MSCS clusters disks or FT VMs)

We see mode 2 which indicates a file lock. Normally there also is an owner, but this shows only zeroes so we can’t use this information.

In the few lines below we see this information about the Read Only Owner:

RO Owner[0] HB Offset 3510272 53baaace-0e95cf36-46a4-441ea15dea84
Addr <4, 5, 119>, gen 269, links 1, type reg, flags 0, uid 0, gid 0, mode 600
len 34359738368, nb 32768 tbz 7056, cow 0, newSinceEpoch 32768, zla 3, bs 1048576

The read only owner of this lock is MAC address 441ea15dea84 which converts to 44:1e:a1:5d:ea:84.

Find the MAC

A fast and easy way to find the corresponding MAC address is with powerCLI. Because we know the cluster where it happens I came up with the command line below:

get-cluster *001*|Get-VMHost|Get-VMHostNetworkAdapter|where {$_.Mac -eq "<strong>44:1e:a1:5d:ea:84</strong>"}|select vmhost,,mac

VMHost                                                                                  Mac
——                                                                                          —
ESX.host.net                              44:1e:a1:5d:ea:84
Now we put this host in maintanance and reboot. Normally the file lock should be gone now and we can consolidate the VM.