Including the HP Offline Bundle as part of an upgrade to ESXi 4.1 U2

If you’re running your vSphere deployment on HP kit then there’s a pretty good chance you use the HP Customized ISO Image for installation, for example this one for ESXi 4.1 U1. These customised images typically contain HP management tools and drivers and are great for saving time during the installation process. Naturally you will be upgrading ESXi at some point, but it’s important that you also keep the HP part up-to-date too. To accompany the release of ESXi 4.1 U2 there is a corresponding release of the HP Offline Bundle. The release notes for this version do not suggest any enhancements or bug fixes, only that it is the version that is supported with 4.1 U2.

Once you have downloaded the HP Offline Bundle, it is possible to deploy it via a number of methods. The installation notes suggest installing with the vihostupdate utility, however in this post I’m going to show how it can be installed via vCenter Update Manager.

1) First of all the HP Offline Bundle needs to be imported into the Update Manager Patch Repository. Navigate to the Patch Repositoy and select Import Patches.

2. Enter the path to the HP Offline Bundle. (Note: the GUI does not display the full path, just the filename)

3. The patches will then be uploaded to the Patch Repository.

4. Finally, finish off the wizard.

5. Once the HP Offline Bundle has been imported a Host Extension Patch Baseline needs to be created. Navigate to Baselines and Groups and create a new Baseline, with the type changed from the default Host Patch to Host Extension.

6. Select the Extension to add to the Baseline. The best thing to do here is enter HP into the search box to reduce the number of items to choose from.

7. On the final page confirm the HP Offline Bundle has been added.

Alternatively, once the HP Offline Bundle has been uploaded into the Patch Repository, it’s much easier to create this baseline via the vCenter Update Manger PowerCLI cmdlets.   Note that we need to use the Extension parameter since the baseline will be of type Host Extension.

New-PatchBaseline -Name "HP Offline Bundle for ESXi 4.1 U2 " -Description "HP Offline Bundle for ESXi 4.1 U2" -IncludePatch (Get-Patch -SearchPhrase "HP ESXi 4.1 Bundle 1.2-25") -TargetType Host -Static -Extension

The main advantage for me of deploying the HP Offline Bundle with Update Manager is that we can take advantage of Baseline Groups. As the name suggests Baseline Groups enable you to group together multiple and different types of baselines. Consequently in this instance we can place the ESXi and HP Offline Bundle upgrade into a Baseline Group and carry out the upgrade to 4.1 U2 in a single remediation task.

There are similar upgrade packages for ESXi 5 from HP, so this process could also be used for that upgrade.

Configuring HP EVA Recommended Settings for ESXi via PowerCLI

The HP Enterprise Virtual Array Family with VMware vSphere 4.0 , 4.1 AND 5.0 Configuration Best Practises Guide, available here, contains many recommendations for ESXi configuration. There are a number of recommended settings in this document to enhance the storage performance, a subset of which I have picked as appropriate for the environment and then needed to configure them on all ESXi hosts.

They can be implemented via PowerCLI and the below script demonstrates how these different types of settings can be configured. The most interesting one for me was setting the default Path Selection Policy to VMW_PSP_RR. The guide recommends you use the following command from the ESXi console:

esxcli nmp satp setdefaultpsp -satp VMW_SATP_ALUA -psp VMW_PSP_RR

With the introduction of the Get-ESXCLI cmdlet we can now carry out the equivalent from PowerCLI. Get-EsxCLI requires a direct connection to the ESXi host rather than from vCenter, so all the settings in this script are configured via a direct connection to the ESXi host

Warning: Before carrying out any of these types of changes make sure you talk to your Storage Adminstrator to confirm what is appropriate for your own environment. Other array vendors offer different recommendations so be sure to check their documentation for similar settings.

<#
.SYNOPSIS
Implement HP Recommended Settings for EVA SAN

.DESCRIPTION
Implement HP Recommended Settings for EVA SAN

.PARAMETER  HostName
Name of the ESXi host to configure the settings on

.EXAMPLE
PS C:\> ./Set-HPEVAESXiConfig.ps1 -Hostname ESX01

.EXAMPLE
PS C:\> Get-Content ESXServers.txt | ./Set-HPEVAESXiConfig.ps1

.NOTES
Author: Jonathan Medd
Date: 21/12/2011
#>

[CmdletBinding()]
param(
[Parameter(Position=0,Mandatory=$true,HelpMessage="Name of the ESXi host to configure the settings on",
ValueFromPipeline=$True,ValueFromPipelineByPropertyName=$True)]
[Alias('IPAddress','Server','ComputerName')]
[System.String]
$HostName
)

begin {

Write-Host "Please enter credentials to connect to the ESXi hosts" -ForegroundColor:Yellow
$Credential = Get-Credential
$UserName = $Credential.GetNetworkCredential().UserName
$Password = $Credential.GetNetworkCredential().Password
}

process {

Connect-VIServer $Hostname -User $Username -Password $Password | Out-Null

Write-Host "Setting Disk.DiskMaxIOSize Advanced Option"
Set-VMHostAdvancedConfiguration -VMHost $Hostname -Name Disk.DiskMaxIOSize -Value 128 | Out-Null

Write-Host "Changing any LUNs with MultipathPolicy set to MostRecentlyUsed to be RoundRobin instead"
Get-ScsiLun -VmHost $Hostname -LunType "disk" | Where-Object {$_.MultipathPolicy –eq "MostRecentlyUsed"} | Set-ScsiLun -MultipathPolicy "RoundRobin" | Out-Null

Write-Host "Setting the default PSP to be VMW_PSP_RR"
$esxCli = Get-EsxCli –Server $Hostname
$esxCli.nmp.satp.setdefaultpsp("VMW_PSP_RR", "VMW_SATP_ALUA")

Write-Host "Disconnecting from Host"
$DefaultVIServer | Disconnect-VIServer -Confirm:$false
}

Issue Patching ESXi 4.1 U1 installed on USB / SD Media

Whilst attemping to install the Dell ESXi Agent and upgrade the ESXi revision on a new VMware host, I hit the following issue.

Dell Agent – Error encountered:
Description - I/O Error (17) on file /var/tmp: [Errno 17] File exists: '/var/tmp'
Message - Unable to create, write or read a file as expected.I/O Error (17) on file /var/tmp: [Errno 17]
File exists: '/var/tmp'

ESXi upgrade with Update Manager – The host returns esxupdate error codes: 10. Check the Update Manager log files and esxupdate log files for more details

Turns out the is a known issue with the scratch partition in ESXi 4.1 U1 when it is installed non-persitent storage, i.e. USB or SD card.

http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=1037190

This issue is fixed in the latest firmware release for ESXi 4.1 U1 (381591), but obviously you need to be able to patch the system up to that level first. There are a couple of workaruonds mentioned in the article, but one not mentioned is that if you reboot the host the issue will go away temporarily, so allowing enough time for you to patch it up to the revision with the fix.

Luckily, on my way to work on the morning I had the issue, I had already heard it discussed on the Hands on Virtualization Podcast with Barry Coombs and Jonathan Franconi so was well prepared for the problem.

The Case of the Missing UserVars.CIMOEMProvidersEnabled Value in ESXi 4.1

In ESXi 4.0 the Advanced Software Setting UserVars.CIMOEMProvidersEnabled is used to enable an agent such as the Dell OpenManage offline bundle for ESXi which provides hardware management for Dell PowerEdge servers. This is supplied as a vSphere Installation Bundle (VIB) and Alan Renouf has a great post on his blog on how to install a VIB like this one via PowerCLI.

Whilst testing out getting the same Dell Agent installed into ESXi 4.1 there are a number of changes you need to be aware of:

1) UserVars.CIMOEMProvidersEnabled is no more

It has been replaced by UserVars.CIMoemProviderEnabled – note the lower case oem and the dropped s from Providers. Why you would do this between versions, I have no idea! Developers, eh? It’s a case sensitive value when setting from PowerCLI so be careful.

2) UserVars.CIMoemProviderEnabled Is Not Visible in a Fresh Install

OK, so you have found the name change, now to set it. However, when you go to look for it in a fresh install it doesn’t appear to exist!

Wot no UserVars.CIMoemProviderEnabled ? (If it was there, it would be the last value in the list)

You have the same issue if trying to set it with PowerCLI. The below fails with an error that the value doesn’t exist:

Set-VMHostAdvancedConfiguration -VMHost ESX1 -Name UserVars.CIMoemProviderEnabled -Value 1

It turns out that it is not possible to set the value until the Dell ESXi Management Agent has been installed, this is a change from 4.0. I guess this kind of makes sense when you think about it, i.e. there is not much point setting it if no agent is installed. So, installing the Dell agent and then rebooting the host will then see the value appear in the GUI and also be configurable via PowerCLI.

3) Version 6.4 is required for ESXi 4.1

Back in ESXi 4.0 the version of the Dell agent was 6.2. The supported version for ESXi 4.1 is 6.4 which can be downloaded here. This version of the agent requires that you be running version 6.4 also of the Dell OpenManage Server Administrator Managed Node software so you may need to upgrade that too. This version has been given a smarten up and has a nicer looking appearance over the previous version.

ESXi 4.0 Slow Boot Times When Hosting Passive MSCS Nodes With RDM LUNs

During the initial stages of an upgrade of a number of VMware hosts from ESX 3.5 U5 to ESXi 4.0 U2 the boot times rose from the normal few mins (most of which is Dell Hardware checks) to around 12 mins.

In particular it was appearing to hang for 5 mins, whilst on the screen the below was displayed:

Loading module multiextent

This would only happen after the install was completed and the host connected back to the fibre channel SAN, otherwise boot times were normal. It was also fine on ESX 3.5 U5 when connected to the SAN.

Some research led me to the below blog post which describes that this can occur when the hosts are part of a cluster which contain Passive MSCS Nodes with RDM LUNs.

http://www.vstable.com/tag/slow/

I made the recommendation to modify the Scsi.UWConflictRetries Advanced Setting to the minimum value of 80 and the boot time dropped to around 5 mins, slighty longer than before, but much better.

Of course you could also make this change in PowerCLI using the below:

Get-VMHost test01 | Set-VMHostAdvancedConfiguration -Name  Scsi.UWConflictRetries -Value 80

Watch out because the name of the Advanced Setting appears to be case sensitive.

Update: 09/02/11

In ESXi 4.1 the value Scsi.UWConflictRetries has been removed and has been replaced with Scsi.CRTimeoutDuringBoot , see updated KB article http://kb.vmware.com/kb/1016106 . The article recommends setting Scsi.CRTimeoutDuringBoot to 1. You can do this with PowerCLI like this:

Get-VMHost test01 | Set-VMHostAdvancedConfiguration -Name  Scsi.CRTimeoutDuringBoot -Value 1