Root disk full WTF?

Root disk full  WTF?

Suddenly somehow we got a virtual machine which couldn’t be powered on, vCenters events showed messages like:

Power On virtual machine
<Computer>
A general system error occurred: Unknown error

Mmm strange, but seen it before. I used the articles in each section to find the solution

Part 1

I started to look at the /var/spool/snmp directory, but this one was empty so it could not be the cause of filling the /root disk.

  1. Connect to the ESXi host using SSH. For more information, see Using ESXi Shell in ESXi 5.0 and 5.1 (2004746).
  2. Check if SNMP is creating too many .trp files in the /var/spool/snmp directory on the ESXi host by running the command:ls /var/spool/snmp | wc -lNote: If the output indicates that the value is 2000 or more, this may be causing the full inodes.
  3. Delete the .trp files in the /var/spool/snmp/ directory by running the commands:# cd /var/spool/snmp
    # for i in $(ls | grep trp); do rm -f $i;done
  4. Change directory to /etc/vmware/ and back up the snmp.xml file by running the commands:# cd /etc/vmware
    # mv snmp.xml snmp.xml.bkup
  5. Create a new file named snmp.xml and open it using a text editor. For more information, see Editing files on an ESX host using vi or nano (1020302).
  6. Copy and paste these contents to the file:<?xml version="1.0" encoding="ISO-8859-1"?>
    <config>
    <snmpSettings><enable>false</enable><port>161</port><syscontact></syscontact><syslocation></syslocation>
    <EnvEventSource>indications</EnvEventSource><communities></communities><loglevel>info</loglevel><authProtocol></authProtocol><privProtocol></privProtocol></snmpSettings>
    </config>
  7. Save and close the file.
  8. Reconfigure SNMP on the affected host by running the command:# esxcli system snmp set –-enable=true
  9. To confirm the SNMP services are running normally again, run the command:# esxcli system snmp getHere is an example of the output:
    /etc/vmware # esxcli system snmp get
    Authentication: Communities: Enable: true Engineid: 00000063000000a10a0121cf Hwsrc: indications Loglevel: info Notraps: Port: 161 Privacy: Remoteusers: Syscontact: Syslocation: Targets: Users: V3targets:

But to be sure I disabled SNMP like mentioned in the article.
ESXi 5.1 host becomes unresponsive when attempting a vMotion migration or a configuration change (2040707)

Part 2

In the first article there is also a hint that HP Gen8 hardware could have similar issues with another resolution. HAH! We use that hardware, so I went on to remove the hpHelper.log file. With the steps below

To remove the hpHelper.log file:

  1. Log in to the ESXi host as the root user.
  2. Stop the HP Helper management agent by running the command:/etc/init.d/hp-ams.sh stop
  3. To remove the hpHelper.log file, run the command:rm /var/log/hpHelper.log
  4. To restart the HP Helper management agent, run the command:/etc/init.d/hp-ams.sh start

ESXi ramdisk full due to /var/log/hpHelper.log file size (2055924)

After I started everything and mentioned the HPHelper.log file was super small I tried again. DAMN! Same error, guess there is something else which is filling the root disk, so my search continues.

Part 3

Somehow I got a clear moment and thought about the sfcdb-watchdog service. After a search with sfcbd and root disk full I found this article :

ESXi 5.x host is disconnected from vCenter Server due to sfcbd exhausting inodes (2037798)

With the help of the command vdf -h I noticed that there was only 8 MB Free on /root

Ramdisk                   Size      Used Available
root                           32M        24M         8M
etc                             28M     224K     27M
tmp                         192M       40K    191M
hostdstats            834M        3M    830M

Mmm that’s not much.

To free up inodes:
  1. Connect to the ESXi host using SSH.
  2. To stop the sfcbd service, run the command:/etc/init.d/sfcbd-watchdog stop
  3. Manually delete the files in the var/run/sfcb directory to free inodes.To remove the files, run the commands:cd /var/run/sfcb
    rm [0-2]*
    rm [3-6]*
    rm [7-9]*
    rm [a-c]*
    rm [d-f]*
  4. To restart the sfcbd service, run the command:/etc/init.d/sfcbd-watchdog startNote: You may need to restart management agents after step 4. For more information, see Restarting the Management agents on an ESXi or ESX host (1003490).

After doing this I had like 30MB free on /root.

This issue should be resolved in update below. Unfortunatly it was not possible for us to update these hosts before with the latest patches.

To manually resolve this issue, you can prevent the root file system from being impacted

  1. Connect to the ESXi host using SSH.
  2. Run this command to stop the sfcbd service:/etc/init.d/sfcbd-watchdog stop
  3. Enable this command to be run at boot time:esxcli system visorfs ramdisk add –name sfcbtickets –min-size 0 –max-size 1024 –permissions 0755 –target /var/run/sfcbFor more information, see Modifying the rc.local or local.sh file in ESX/ESXi to execute commands while booting (2043564).
  4. Run this command to start the sfcbd service:/etc/init.d/sfcbd-watchdog start
  5. Run this command and verify if the new ramdisk exists:esxcli system visorfs ramdisk list
Note: This issue may also occur if the SNMP agent is enabled. For more information, see vCenter Server and vmkernel logs report the message: The ramdisk ‘root’ is full (2033073).
After doing this I was able to succesfully start up the virtual machine.
Now it’s time to make sure we do the updates asap!
Another handy article to gather file system information

One comment

Leave a Reply

Your email address will not be published. Required fields are marked *

This site uses Akismet to reduce spam. Learn how your comment data is processed.