/var runs full on VMware ESXi 6.7 on HP Gen8 Server

Today two of my ESXi 6.7 servers running on HP DL360p Gen8 didn’t behave as expected. VMotion and remote consoles quit with internal error.

In vspere syslog I found:

The ramdisk 'var' is full. As a result, the file /var/log/EMU/mili/mili2d.log could not be written.

On ESXi vdf -h revealed a full file system. The milid2.log was the problem and dumped the following errors:

CRITICAL:backend_init:OneConnect Adapter Not Found.
ERROR:rename all the configuration files!
ERROR:MILI_enumerate_elxiscsi:Failed to initialize User Init with status = 19
ERROR:MILI_enumerate_elx_nics:Failed to initialize USer Init with status = 19
ERROR:could not open device node /vmfs/devices/char/vmkdriver/be_esx_nic

In both servers no emulex hardware was installed.  So I removed 3 packages form those ESXis with a reboot between each. The last one solved the problem!

esxcli software vib remove --vibname elxnet

esxcli software vib remove --vibname elxiscsi

esxcli software vib remove --vibname elx-esx-libelxima.so

On my DL360 Gen9 this problem did not occur.

Update: As Gunnar pointed out in a comment this could also happen on a Gen9 on newer vSphere 6.7 Updates.

Update 2 (Thanks Pete): VMware resolved this problem with vSphere 6.7 U2 or Patch ESXi670-201904211-UG

16 thoughts on “/var runs full on VMware ESXi 6.7 on HP Gen8 Server”

  1. Just had the same problem on our Gen8 server after trying to vMotion a system onto it, this failed with:

    A general system error occurred: Failed to write header to “/var/log/vmware/journal/1536303045.0: Error while writing to file. There is no space left on the device

    I deleted /var/log/EMU/mili/mili2d.log to free up the necessary space, and uninstalled the packages you mention.

    Thanks!

  2. Thanks for this helpful post!

    We had an issue while trying to vMotion a VM to a Gen8 host:

    A general system error occurred: Failed to write header to “/var/log/vmware/journal/1536303045.0: Error while writing to file. There is no space left on the device

    After deleting /var/log/EMU/mili/mili2d.log the migration worked again.

    We’ve uninstalled the mentioned packages to prevent further issues.

  3. I have the exact same problem. I was able to reset the size of the mili2d.log file back to 0 bytes and the problems disappeared, a reboot will also clear the mili2d.log file, i am a little hesitant to remove th sofware bundles, i also dont have any emulex nic’s or scsi adapters.

    Running this command will clear the contents of the mili2d.log file.
    >mili2d.log (ran at the location of the file in my case is \var\log\EMU\mili\

  4. Just ran into this issue with both dl360p and a ML350p using the latest ESXi 6.7U2 version HPE custom version. Thanks for the great post

  5. Same problem with HPE Gen9 DL380 Servers since 6.7 U2 (May 2019) – same with 6.7 U3.

    esxcli software vib remove –vibname elx-esx-libelxima.so

    solved my problems – also no emulex Hardware is installed.

  6. Same problem with DL380 Gen10 – HPE OEM Image (ESXi 6.7 U1)

    Is this a problem only with hpe oem images?

  7. thanks.
    had the same problem with 6.7 u3 and dl380 gen10. after deleting the milid2.log the ramdisk still showed 100% in use and vmotion was not possible. Restarting the MGMT agents released the space on ramdisk.

    Regards

  8. Thank you so much!!!
    I also had to restart the management agents.
    Resolved the issue for me.

Leave a Reply to Perseus Cancel reply

Your email address will not be published. Required fields are marked *

This site uses Akismet to reduce spam. Learn how your comment data is processed.