VMware has published information about upgrading to 5.1 and the required length of host certificates. In my case, I verified before hand that my certificates were the appropriate length so I proceeded, but as it turns out, my vCenter certificates were not the proper length.
Our vCenter was installed first with version 4, which auto-generated self-signed certificates which we never bothered to replace. These certificates had a key length of 512-bits. When we upgraded to 5.0 and 5.1, these certificates were maintained. As it turns out, this isn’t compatible with the new VMware Single Sign on service. In my case the troubleshooting process led us to reinstall Single Sign On, Inventory service and vCenter 5.1.
Prior to upgrading, verify the public key length is at least 2048 bits, otherwise replace the certificate.
If your certificate is the appropriate minimum length, your upgrade should proceed normally and vCenter should function as expected.