Sometimes little things slip through, especially when you’re in a rush. We recently discovered excessive CPU latency was affecting performance on a number of SQL servers on what should have been otherwise sufficient hosts. The cause wasn’t immediately clear, but since we began monitoring the servers with Veeam Management Pack for VMware (review to come soon!) we did know that we had to isolate why only some of the hosts in the cluster exhibited the issue. After searching around for a bit we found a thread where a user talked about using the HP “High Performance” power policy in the BIOS and experiencing latency, while setting it to “OS Managed” effectively eliminated the latency.
We embarked on the journey – maintenance mode the host, reboot and check the BIOS setting when the horrifying reality became clear: It wasn’t high performance at all, but rather the HP Default: “Balanced”. It certainly was not our standard configuration which was in fact, to set high performance, so it must have been missed during the last hardware refresh when the person deploying the server set up the server.
After changing the setting to “OS Managed”, then in ESXi set the policy to high performance and the difference was staggering.
Prior to 3/11 at 5PM
Why such a setting is the default out of box given the nature of servers I don’t know. I can appreciate green initiatives but with the advent of virtualization and modern operating systems I can’t imagine that having the default selection (to be managed at the BIOS level only) is one that effectively and intentionally cripples the performance of your shiny new servers.
The latency did seem to have a real impact on CPU intensive processes as well – we discovered the daily report that ran from 5:00 AM for 90 minutes began taking only 60 minutes after making the change.
Moral of the story? Review configurations before launching into production and use monitoring tools to alert you to things you may have missed. Check out this great post from VMware discussing and comparing power management on ESXi.