Posted by mattr on April 15, 2010 at 10:59 pm
With the advent of vSphere my company has been looking at how best to make the move from ESX 3. One obvious path under consideration was to look at migrating to ESXi, both from the perspective of reduced patching (the default view being a lower attack surface with the removal of the Service Console) and the fact that roadmaps have often alluded to it being the final destination for ESX anyway.
With that in mind, I took to having a look at ESX4i in more depth to fathom where it would fit into our environment. We are very much an HP house, housing both standard and blade servers, so one of the core management systems we use is HP Systems Insight Manager. Its primary purpose is hardware monitoring – knowing when the hardware is either failing or failed being a must for our service assurance.
It was therefore a surprise when I first managed to add my test ESXi server to our existing SIM server (version 5.3), that certain components were not showing up. Most concerning was the lack of a disk subsystem. No array information, no disk breakdown – nothing.
Now I don’t know about other companies, but by far the most common failure in a server for us tends to be a disk. Therefore for SIM not to see it appeared somewhat dangerous. Puzzled by this, I investigated further – my first thought was that perhaps it was a GUI issue and that the underlying monitor was present but not displayed.
So I wandered up to the test server in the CER and disengaged one of the drives to see what happened. The HP SIM display did not change – no email alert, no status change on the GUI – even after ten minutes. By contrast the default alerts inside my test vCenter server, which was seeing the entire disk array inside the Hardware Status tab, immediately started sending alerts and changing status on screen.
This was supremely confusing – obviously for the vCenter to be able to detect the disk and any alerts the correct CIM information was being produced on the ESXi host (I was using the correct HP flavour of ESXi) yet SIM was oblivious to a problem.
Sorry to say after some more fiddling with authentication on SIM, trawling through numerous HP forums and even starting a Google Wave on the subject, I had to call it a day and put the ESXi question on ice.
Then back in late March, I saw something on Twitter announcing the release of SIM version 6 – there were various references made to improved support for ESX4 so I was anxious to see if the monitoring components had been upgraded.
So a swift guest build later (SIM 6 supports 64 bit so a Windows 2008 R2 was my choice) I was busy installing the new version. Now SIM is still not that intuitive when it comes to ESXi – it uses WBEM as a method to query the system so the authentication needs to be set there.
Before setting up a discovery job, a change needed to be made to the event filtering. Under Options > Events > Event Filter Options the ‘Accept unregistered events’ needs to be enabled. Don’t ask me why but the HP document suggests it needs to be there for ESXi.
I then set up a discovery job to look for the server (note the port used for access below – you will need to add this under the Advanced protocols credentials > WBEM/WMI tab:
Once this had been saved the job was started– the feedback is vastly improved in version 6, and you see each type of discovery it is trying and whether it is good or bad. The one you are looking for is the WBEM entry.
Once complete I switched to the All Systems view to see if the server had appeared. It had and I was greeted with the following extra display window.
NB. The auto-discovery is also much improved in SIM 6 – if you discover a host it automatically works out what is hosted on it and adds it as well if required. Very nice.
On clicking any of these categories I was pushed to a status window for the host where I was delighted to see the following – noting the comparison between v6 and 5.3 …
 |
 |
| SIM 6 |
SIM 5.3 |
Success! Full access to the complete hardware layer in ESXi through SIM … looks like we can put ESXi back on the upgrade table 
Continue Reading
Posted by mattr on March 18, 2010 at 9:58 pm
VMware has so many new shiny gadgets it is sometimes easy to forget the simplicity it can add to an administrators life. One such example was a few weeks ago.
I was upgrading the memory in a Windows 2000 VM to 8GB, and had to alter the boot.ini to use more than 4GB. Unfortunately I was doing this remotely and during the save process my VPN died. When I rebooted the server I was faced with the dreaded ‘NTBTLDR.EXE is missing’. Now I knew that the problem must lie with the boot.ini, but how to get at it?
Because I was remote I did not have a Windows 2000 disk to get a recovery console up and I needed to get the server up and running again. I then realised that I did not need to boot from a CD to get to the VM’s hard drive.
I shut the VM off again, and jumped on to another server on the same host. In Edit Settings, I simply added the vmdk of the troubled host to the other machine and rescanned for the disk in Disk Management. Lo and behold the disk appeared and I could access the boot.ini file – I had managed to add a space at the beginning of the file during the save process which was causing the problem.
I rectified the file, removed the disk from the spare server and then the vmdk from the client. I crossed my fingers and restarted the VM … and gratefully saw the windows splash screen appear.
If this had been a physical box I would not have had a chance of fixing this remotely, and even on site there would have been much more time expended on finding a chassis with spare slots, array controller configurations etc. so it goes to show even the most basic functions of VMware are worth the effort of putting it in in the first place
Continue Reading
Posted by mattr on January 26, 2010 at 10:43 pm
I have been battling to try and find out where my HP servers store and provide serial numbers for vCentre in the Hardware Status plugin – to no avail I am sad to say so far. However while investigating I have had to work out how to use Wireshark to decode the SSL traffic between the vCentre and an ESX host, which is very useful for troubleshooting connectivity or other issues between a VC and ESX host, so I thought I would record the process for reference …
For Wireshark (Download the latest version here http://www.wireshark.org/download.html ) to be able to decode the SSL traffic between a VC and host, it needs the private key of the SSL certificate. To get this you will need to copy it from the host via an SCP client ..
The file you are looking for is rui.key and is located in /etc/vmware/ssl folder.
Once you have copied the file to your local machine you can fire up Wireshark and head to the Edit > Preferences.
In there under the Protocols menu on the left hand side, select SSL and you will see the following fields:
The RSA keys list field is where you tell Wireshark to look for the server source i.p. (the ESX host),port,protocol and location of the private key you want to use to decrypt.
In this example I am using the same key to decrypt both standard 443 and 5989 (Secure CIM) traffic – to do more that one you simply use a semi-colon to separate the string:
10.0.0.1,5989,wbem-https,c:\temp\rui.key;10.0.0.1,443,https,c:\temp\rui.key
The SSL debug file field allows Wireshark to write out how it is using the key to aid troubleshooting.
Once you have this you can load up a capture from your vCentre and look for some SSL traffic – you should see in the lower frame something like this:
As you expect the output is unreadable in the standard tab, but if you look at the Decrypted SSL data tab you should see the data magically become a lot more useful:
Wireshark helpfully will now also display a context menu called Follow SSL Stream, once it can decrypt the traffic, and will piece all the traffic it can find back together and pop up a window with the whole transaction in one place.
Once you get into the underlying transactions it is great to see what is going back and forth between the vCentre and hosts – I hope this helps you figure out whatever problem you may be having …
Continue Reading
