Root ramdisk is full and Cannot Write to /var/log/run

Recently I ran into a problem with a couple of ESXi hosts (HP Proliant BL465c Gen8) while trying to vMotion some guests to another host. I noticed an error under ‘Tasks & Events’ which to be honest didn’t really give much away, other than “A general system error occurred”!

Upon further investigation I found that the underlying message was that the host was out of disk space whilst trying to vMotion guests.

The error that alerted me to this was “/var/log/vmware/journal/xxx error writing file. There is no space left on the device.”

Now HPs fix is to update all drivers to ensure that they have the latest hp-ams VIB installed to avail the latest fixes. But we all know this is easier said than done! Here is what I did to get over it.

Troubleshooting Steps

> You will need to enable SSH on the host with this problem, go to the Configuration tab on the host in the vCenter or vSphere client, Security Profile, Properties link on the Services section.

> Scroll down to find SSH, options and start the SSH service.

> Use your preferred terminal software to SSH to host (if the root ramdisk is in fact full you will possibly get a connection refused error and you will need to iLO or KVM to the host.

> Enable ESXi Shell, press ALT-F1 to change to the management shell and login as root.

> Run ‘vdf -h’ and look for the root filesystem, it should look like:


Ramdisk                   Size      Used Available Use% Mounted on
root                       32M        2M       29M   7% --

> When running the above command if it returns 0M and 100% used then that’s your problem! We need to clear up some space.

cd /var/log/ ls -la

> Check the size of the hpHelper.log it is likely that it will be quite large, remove the file if this is the case.

To remove the hpHelper.log follow these steps:

> Stop the HP Helper management agent by running the command:

/etc/init.d/hp-ams.sh stop

> To remove the hpHelper.log file, run the command:

rm /var/log/hpHelper.log

> To restart the HP Helper management agent, run the command:

/etc/init.d/hp-ams.sh start

> In the event of a host reboot, run the command:

chkconfig hp-ams.sh off

The hosts are now back online and working normally, once I had completed the above the hosts started to vMotion guests around on its own accord. It seems that on this occasion the hpHelper.log file had grown in doing so filled the ramdisk which was causing issues.

Resources
VMware ESXi 5.1 – Ramdisk is Full and Cannot Write to /var/log/run