Since a couple of weeks we let vSphere monitor our VMs in addition to the nagios monitoring we have had from the beginning.
The vShield Manager VM started to show odd behavior: CPU spikes to 100% every 10 to 30 minutes, then returning to normal. Operation worked as normal, nothing showed in the log files (although: you don’t really see the system log files in the interface).
Somewhere I read something about the mySQL database having problems, and a reboot would fix that. So, reboot… After all, what could go wrong, right? The single firewall VMs are working independent from the Manager, so nobody should even notice… so far so good.
vShield Manager VM did not boot up properly again. You could not login via the web interface. On the console you’d see only the message “System startup is not complete. Please logout and log back in after a few minutes.”
After some hours still the same. On the console (you don’t get a full bash, just a stripped down management shell that does not much) not much was showing. On thing made us curious though: “show filesystems” did say something about mount file could not be read.
A bad foreboding struck my colleague and me: Filesystem corrupt? Without a bash there is not much you can do. So we downloaded a live linux cd, rebooted the VM into that livelinux, and saw it in an instance: The filesystem simply was full. 0 bytes free. Ok, that definitely is a reason for mySQL and tomcat not to start, and definitely a reason why nothing shows up in the log files.
Now with a bash at hand, we investigated, and our first thought where the log files. But we found only 20 MB worth of log files in /var/log. Everything rotated, gzipped and deleted as supposed. So what was going on?
Turns out vmware forgot about the tomcat logfiles. Tomcat saves its own logfiles into its own directory. And that directory was full of catalina-logfiles, going back to the very day the system was installed.
Simple fix: Delete the old junk no one needs, and the vShield Manager is rebooting like a charm.
So, vmware, I’d guess you have fixed that in a newer version already, although I have not checked yet. If not: Shame on you. Leaving admins out with no proper bash or other good way to determine what is going on. That is a shortfall on your side!
P.S.: No hard feelings though, vmware. I still love you 🙂