vCenter Operations Manager – SSL Certificate issues

So during a recent deployment of vCenter Operations Manager (5.8.2) at a customer site I encountered the following error whilst trying to pair the vCOPs vApp to their vCenter Server.

vcops ssl cert

“Unable to get vCenter Server certificate chain”

This was the first time I had encountered this issue deploying vCOPs, fortunately given how much exposure I got to SSL certifications during a previous project I knew it could be down to one of 2 things….. either the SSL certificate had expired, or that it was not generated with the correct parameters.

Note: Quickest way to look at a vCenter Server’s SSL certificate is to just open a browser and point it at the vCenter’s IP address, then view the certificate…..
vcops ssl 1 vcops ssl 2
(Left – IE, Right – Chrome)

or if it’s a Windows deployment of vCenter 4.1 or later, you can find the certificate here: C:\ProgramData\VMware\VMware VirtualCenter\SSL\rui.crt (Note that c:\ProgramData is a hidden folder!).

 

It seemed that the SSL certificate was valid (expiry date was 2022), however I noticed that the public key certificate was weak as the key length was only 512 bits!!
What had happened was a previous partner had upgraded them from VI3.5 to vSphere 4.0 to vSphere 5.0 and had forgotten to re-generate the SSL certificates!
Prior to vCenter Server 4.1, by default VMware self-signed their SSL certificates with a public key length of 512 bits! So when they upgraded they kept the same SSL certificates.

Post vCenter Server 4.1, if you installed from scratch the public key length is set to RSA 2048 bits by default.
vcops ssl 3
So because the public key length was only 512 bits, vCOPs could not authenticate the vCenter Servers’ certificate (I believe it has to be a minimum of 1024 bits)!
More info from VMware’s KB here: http://kb.vmware.com/kb/2037082 and Microsoft’s KB here: http://support.microsoft.com/kb/2661254

 

As it was a production environment and they couldn’t afford to regenerate their SSL certificates, I had to ‘inject’ the vCenter Server certificate into the vCOPs VMs keystores as follows:

  1. Copy the rui.crt file (the SSL certificate) on the vCenter Server into the tmp drive of the vCOPs UI VM. (This can be easily achieved using WinSCP).
  2. Login to the console of the UI VM as root.
  3. Change to the directory where the certificate keystore is located: /usr/lib/vmware-vcops/user/conf
  4. Issue this command to add the vCenter Server certificate to the certificate store: keytool -importcert -file /tmp/rui.crt -alias https://<VC FQDN or IP>/sdk -keystore truststore -storepass oxygen
  5. Issue this command to verify that the certificate is in the certificate store: keytool -list -keystore truststore -storepass oxygen
  6. Issue this command to copy the truststore file from the UI virtual machine and paste it to the Analytics virtual machine: scp truststore secondvm-external:/usr/lib/vmware-vcops/user/conf/
  7. Restart all services with the su – admin -c “vcops-admin restart” command, or reboot the vApp from the vCOPs admin page.

Once the SSL certificate was injected into the vCOPs VMs keystore it was plain sailing and we could continue with the setup wizard.

 

Ideally if you still have weak certificates in your environment, you should really be replacing them by generating new SSL certs! =)

Advertisements

Site Recovery Manager SSL error

Recently had to regenerate the SSL certificates on my vCenter Server Appliance as I had noticed they were originally created with localhost.localdom as the DNS name.

Quite easy to regenerate the SSL certs, just log onto the admin page of the vCSA (http://ip-address:5480) and log in as root. Click on the Admin tab and by Certificate regeneration enabled select Yes.
When you next reboot the vCSA, the SSL certs will be regenerated – just be sure to change it back to No after it’s complete otherwise it will regenerate a new certificate every time you reboot!
srmerror3

A problem I discovered after regenerating the SSL certificate of my vCSA was that I could no longer connect to Site Recovery Manager using vSphere Client.
It would try to connect and then fail with a Connection Error.
srmerror1

As I knew this was working prior to regenerating the SSL certificate, I guessed that SRM was still trying to authenticate with the vCenter Server using the old SSL certificate.

Checking through the SRM logs (\ProgramData\VMware\VMware vCenter Site Recovery Manager\Logs\) confirmed that my assumption was correct:
srmerror2

The SRM logs show a certificate error:
Failed to connect: std::exception 'class Vmacore::Ssl::SSLVerifyException' "SSL Exception: Verification parameters:
.......
The remote host certificate has these problems:
.......

Quickest way I could think of in order to solve my issue was to modify the installation of SRM to update the certificates.
Log into the SRM server, open up Programs and Features from the Windows Control Panel. Select the entry for VMware vCenter Site Recovery Manager and click Change.
At the SRM wizard, select Modify.
srmerror4

You won’t be able to change the vCenter Server details but you will be able to change the authentication method. Regenerate the certificate by selecting Automatically generate certificate.
Ensure you select Use existing database, otherwise you will lose all your protection groups and recovery plans.

Once the SRM installer was finished, I was able to reconnect to SRM using vSphere client.

Installing SSL Certificates in vCenter Server

So when I started this blog one of the posts I was planning to write was how to install SSL certificates in vCenter Server in order to replace the self-signed VMware ones.

I kind of realised that it would be far simpler just to refer everyone to the great articles written by Derek Seaman which I’ve used time and again! After all, why re-invent the wheel?!? I’ve read so many blogs out there where people have just re-worded Derek’s instructions….. plagiarism is so rife on the internet!

So here it is:

http://www.derekseaman.com/2012/09/vmware-vcenter-51-installation-part-1.html

His blog post covers the whole process of replacing SSL certificates for vCenter Server components, and even goes into detail about how to create the correct Certificate Templates in a Microsoft CA (I’m assuming you know how to setup a Microsoft Active Directory Certificate Services?? If not then visit Microsofts TechNet: http://technet.microsoft.com/en-us/library/cc772393(v=ws.10).aspx).

It’s a 15 part blog, with sub-blogs about setting up the correct CA template and generating CSRs, pre-staging SSL certs….. he’s even written a 4 part article on how to use the new VMware vCenter Certificate Automation Tool – which is VMware’s first stab at trying to tackle this complicated and tedious process (http://www.derekseaman.com/2013/04/using-vmware-vcenter-certificate.html)!

 

Anyways, I’m currently running through the process of deploy signed SSL certs for the latest update to vCenter Server (5.1 u1). Hopefully it’ll be painless like my past deployments! =)

I’m hoping to put together a post on changing SSL certs for Site Recovery Manager as this is usually the next VMware product that people tend to deploy with signed-certs…. and I’ve yet to discover a blog like Derek’s which covers the process is such clarity!

Back to Blogging……

So I know I said I wasn’t going to blog much in the coming weeks, but giving the fact that my jury service has been cancelled next week (court case was cancelled so Jury was dismissed due to no other court cases running) and also the fact that my current work project has been cancelled (client cancelled the contract with my company), I pretty much have quite a bit of free time!

Not to mention that I had a sleepless night as all I could think about was that I NEED to blog some of the stuff that’s floating around my head regarding VMware – just so I can put my brain at rest!

So hopefully in the upcoming weeks, I intend to blog about the experiences I’ve had over the past couple of months touching upon:

  • Changing the SSL certificates of VMware products (away from the self-signed VMware ones to a CA certified one).
  • Transact-SQL scripts for creating databases for VMware products.
  • Loadbalancing workflow that I wrote recently to automate the deployment of a loadbalancer in vCD (and hope to generalise so that others can use it).

That should basically fill out my blog for a couple of weeks due to the vast amount of information to get down on paper (or in this case on screen).

First up tomorrow (yes, procrastination doesn’t disappear even when you have some free time!) will be a brief look at how you manually setup a loadbalancer within vCD, and then hopefully I can delve into how the vCO actions can be used for each manual step and what I’ve learnt.

Oh, and as for the job hunting part….. I’m quite thankful that at the moment it seems recruitment agents are calling me up rather than me desperately calling them up! I’m positive that I will be able to find another role that will allow me to continue my VMware journey! (and if you’re a potential employer, or recruitment agent reading this – please contact me if you have any opportunities of interest!)

^_^