Today two of my ESXi 6.7 servers running on HP DL360p Gen8 didn’t behave as expected. VMotion and remote consoles quit with internal error.
In vspere syslog I found:
The ramdisk 'var' is full. As a result, the file /var/log/EMU/mili/mili2d.log could not be written.
On ESXi vdf -h revealed a full file system. The milid2.log was the problem and dumped the following errors:
CRITICAL:backend_init:OneConnect Adapter Not Found. ERROR:rename all the configuration files! ERROR:MILI_enumerate_elxiscsi:Failed to initialize User Init with status = 19 ERROR:MILI_enumerate_elx_nics:Failed to initialize USer Init with status = 19 ERROR:could not open device node /vmfs/devices/char/vmkdriver/be_esx_nic
In both servers no emulex hardware was installed. So I removed 3 packages form those ESXis with a reboot between each. The last one solved the problem!
esxcli software vib remove --vibname elxnet esxcli software vib remove --vibname elxiscsi esxcli software vib remove --vibname elx-esx-libelxima.so
On my DL360 Gen9 this problem did not occur.
Update: As Gunnar pointed out in a comment this could also happen on a Gen9 on newer vSphere 6.7 Updates.
Update 2 (Thanks Pete): VMware resolved this problem with vSphere 6.7 U2 or Patch ESXi670-201904211-UG
Just had the same problem on our Gen8 server after trying to vMotion a system onto it, this failed with:
A general system error occurred: Failed to write header to “/var/log/vmware/journal/1536303045.0: Error while writing to file. There is no space left on the device
I deleted /var/log/EMU/mili/mili2d.log to free up the necessary space, and uninstalled the packages you mention.
Thanks!
Thanks for this helpful post!
We had an issue while trying to vMotion a VM to a Gen8 host:
A general system error occurred: Failed to write header to “/var/log/vmware/journal/1536303045.0: Error while writing to file. There is no space left on the device
After deleting /var/log/EMU/mili/mili2d.log the migration worked again.
We’ve uninstalled the mentioned packages to prevent further issues.
I have the exact same problem. I was able to reset the size of the mili2d.log file back to 0 bytes and the problems disappeared, a reboot will also clear the mili2d.log file, i am a little hesitant to remove th sofware bundles, i also dont have any emulex nic’s or scsi adapters.
Running this command will clear the contents of the mili2d.log file.
>mili2d.log (ran at the location of the file in my case is \var\log\EMU\mili\
Thanks a lot.
Top notch!
Just ran into this issue with both dl360p and a ML350p using the latest ESXi 6.7U2 version HPE custom version. Thanks for the great post
Same problem with HPE Gen9 DL380 Servers since 6.7 U2 (May 2019) – same with 6.7 U3.
esxcli software vib remove –vibname elx-esx-libelxima.so
solved my problems – also no emulex Hardware is installed.
Thanks, I updated my post.
Same problem with DL380 Gen10 – HPE OEM Image (ESXi 6.7 U1)
Is this a problem only with hpe oem images?
I think so. At least on my machines I use the oem image.
VMware has a patch for this now.
https://docs.vmware.com/en/VMware-vSphere/6.7/rn/vsphere-esxi-67u2-release-notes.html
List under section ESXi670-201904211-UG
Or search for “elx-esx-libelxima.so”
Thanks. Post is updated now.
Thanks.
Same problem here with DL380 Gen10 on 6.7 U3…
So much for solving this problem in 6.7u2 …
thanks.
had the same problem with 6.7 u3 and dl380 gen10. after deleting the milid2.log the ramdisk still showed 100% in use and vmotion was not possible. Restarting the MGMT agents released the space on ramdisk.
Regards
Thank you so much!!!
I also had to restart the management agents.
Resolved the issue for me.