Sometimes things happen for no reason. I am working in a new, inherited environment these days and a lot of my time is spent investigating, gathering data and information, and of course putting out fires here and there. The last system administrator had left before I started and so I am flying blind a bit, but that is alright.
The other morning I got alerts that vCloud was down. I started investigating, but did not have all the pieces together yet. The in-house vCloud website had an error message, but it was not very detailed. I googled the error message, but there were not many search results. The ones that I found talked about NTP settings, IPv6 settings, DNS problems, and some other stuff but none of the suggested fixes did the trick. But with each piece of extra information I came things a bit closer. I finally had all the moving pieces together and checked the individual pieces of my inherited vCloud environment.
So, it turned out that my vCloud database server (MS SQL) had run out of space. The system was brought up with the database recovery model set to Full with no functioning management/cleanup job associated with it. The transaction log file had ballooned and filled up the disk space. I changed the database recovery model to “Simple”, ran a full backup, and then shrank the database and log files. Things started working fine.
About half an hour later 2 users reported that their browser based console for their vCloud VMs/vApps had stopped working. All they showed was connecting. I started searching for a fix, but all I could find on the Internet was issues related to Windows 8 and the VMware VMRC. Apparently there was an issue with Internet Explorer 10, Windows 8, and the older version of the vCenter client. The fix was to install a new version of the vCenter client. While this did not apply to me, I upgraded anyway as there was a newer built of the vCenter Client (5.1 | Build 941893) available for download. Of course this did not fix the problem.
My Google searches were not good and so I started looking around internally. The log files did not show anything and then I used Fiddler Tool (a web proxy traffic inspection tool), but that did not show much either. However, it showed some HTTPS related stuff and I started wondering if maybe this is just a certificate issue. I opened up Firefox and went into vCloud. I accepted the different SSL certificate exceptions it threw at me. Then I opened the VMRC console and everything worked just fine.
I went back into Internet Explorer and checked the Trusted Site settings. I added the SSL URL for my vCloud to the trusted sites in IE and then tried vCloud again via Internet Explorer. The VMRC console started up, connected fine, and all was good.