ESXi 4.0 Slow Boot Times When Hosting Passive MSCS Nodes With RDM LUNs

During the initial stages of an upgrade of a number of VMware hosts from ESX 3.5 U5 to ESXi 4.0 U2 the boot times rose from the normal few mins (most of which is Dell Hardware checks) to around 12 mins.

In particular it was appearing to hang for 5 mins, whilst on the screen the below was displayed:

Loading module multiextent

This would only happen after the install was completed and the host connected back to the fibre channel SAN, otherwise boot times were normal. It was also fine on ESX 3.5 U5 when connected to the SAN.

Some research led me to the below blog post which describes that this can occur when the hosts are part of a cluster which contain Passive MSCS Nodes with RDM LUNs.

http://www.vstable.com/tag/slow/

I made the recommendation to modify the Scsi.UWConflictRetries Advanced Setting to the minimum value of 80 and the boot time dropped to around 5 mins, slighty longer than before, but much better.

Of course you could also make this change in PowerCLI using the below:

Get-VMHost test01 | Set-VMHostAdvancedConfiguration -Name  Scsi.UWConflictRetries -Value 80

Watch out because the name of the Advanced Setting appears to be case sensitive.

Exploring Extended Properties in PowerCLI

I was asked recently via Twitter how to find the CpuFeatureMask property of a VM using PowerCLI. When running a basic

Get-VM Test01

the below properties are outputted to the console:

It is possible to view more properties and values by runnning:

Get-VM Test01 | Format-List *

Unfortunately this still does not reveal the CpuFeatureMask property.  However, if we pipe the Get-VM command through to Get-View we will get back a .NET view object for the VM – by saving this into a variable we can then drill down through the various levels and look for the property we need.

$vm = Get-VM | Get-View
$vm

Below is the top level of information which is returned:

This may look initially like a bewildering amount of info, however if you look at the first few entries you could treat them like categories of information. Since CpuFeatureMask is a configuration property it would seem like a good guess to try looking in the Config category:

You can essentially browse this category by entering the following

$vm.config

and looking down the list of properties returned you will see CpuFeatureMask

On this particular VM CpuFeatureMask is not set, but you get the idea. To retrieve this property for all of your VMs is a simple one liner. With Select-Object we can pick one of the standard properties Name by simply specifiying it; we can use another technique to create our own property with a label and expression for the CpuFeatureMask.

Get-VM | Get-View | Select-Object Name,@{Name="CpuFeatureMask";Expression={$_.config.CpuFeatureMask}}

You could apply a similar technique to other cmdlets like Get-Host or Get-Cluster to retrieve non-standard properties.

Reporting on VMware Update Manager Baselines with PowerCLI

I’ve mentioned on this blog before that I’ve been using VMware Update Manager a lot recently – and wrote about some of my experiences here. Today I was really pleased to see that Carter Shanklin’s team released some cmdlets for PowerCLI to cover Update Manager which had only previously been available back as a beta in the VI Toolkit days.

They arrived just in time because I am currently preparing for a round of ESX patching and I needed to provide a report of hotfixes I was intending to deploy for a particular version of ESX. In the Update Manager GUI I had already created my baseline and scanned it against a host to produce a compliance report of hotfixes we would need to deploy this time.

You can see below that it produces a nice report for me, but I needed to export that information to a format whereby I can give that information to someone else.

One of the new cmdlets is Get-Baseline. I pointed this at my test baseline and with the code below was quickly able to get the information I needed out into a CSV file. I knew from the above report that I just needed to select any patches since 29/12/2009. One of the properties of the patches returned by Get-Baseline is the date it was published so first of all I set a date for which I could query after and stored it in the $BeginDate variable, I then queried the baseline using that date as a starting point.

$BeginDate = (Get-Date).adddays(-65)
Get-Baseline Test | Select-Object -ExpandProperty currentpatches | Where-Object {$_.'releasedate' -gt $BeginDate} | Select-Object Name,IDByVendor,Description,@{n='Product';e={$_.product | Select-Object -expandproperty Version}},ReleaseDate | Export-Csv patches.csv -NoTypeInformation

Which produces an output like this:

You’ll notice that I make use of the ExpandProperty parameter for Select-Object which makes it nice and easy to get to properties which are returned in an array, otherwise although they look fine in the console, when you export them to CSV you will not get what you hope for.

It’s days like today when I’m especially glad I started using PowerShell and very thankful that the vendors of technologies I’m using make this stuff so simple by providing cmdlets for managing their products.

Update:

Initially I tried to use the Get-Compliance cmdlet to find these patches rather than by date, however it only seemed to return a status of Compliant or Not Compliant. Thankfully following a post on the communities it has been pointed out that Get-Compliance has a Detailed parameter which returns a lot more information. Consequently there is no need to mess around with dates, instead you can query for NotCompliantPatches.

$ComplianceStatus = Get-Compliance -Entity 'Server1' -Detailed
$ComplianceStatus.NotCompliantPatches | Select-Object Name,IDByVendor,Description,@{n='Product';e={$_.product | Select-Object -expandproperty Version}},ReleaseDate | Export-Csv patches.csv -NoTypeInformation

Bug in Cluster mem.usage.average Statistic in vSphere 4.0 U1

A while back I posted a script on a basic capacity report I run each month to get an overview of CPU and Memory usage in our various clusters. Since upgrading to vSphere 4.0 U1 I noticed some strange behavior in the results for memory, i.e. they came back at pretty close to 0% (typically between 0.05 and 0.06%) for the average memory usage in a cluster which typically were quite heavily used. On investigating further this also appeared the same in the GUI.

Using Get-Stat, mem.usage.average comes back at 0% where it should be a significant value:

The same in the GUI, the other cluster memory stats have significant values, but Memory Usage Average flatlines along the bottom of the graph:

This stat shows normal behaviour against host machines. I reported it to VMware who confirmed it to be a bug in 4.0 U1.

Thought I would share in case anyone else spends some time scratching their head and checking maths in scripts like I did. I’ll post back when there is a fix.

Update 19/05/2010:

Just got a follow up email from VMware Support that this bug will be fixed in VC 4.0 U3 – no release date for that yet.

Slide Deck from Nov 2009 London VMUG

After my PowerCLI session at yesterday’s London VMUG a few people asked me for the content. I believe the content from all sessions will soon be posted to http://www.box.net/londonug, but in the meantime you can get my slides from the below link.

PowerCLI Workshop London VMUG.pptx

Thanks to all who chatted to me afterwards, its always nice to know that someone got something out of a session you put on.

vSphere 4.0 Quick Start Guide – Now Available From Amazon

I recieved a preview copy of the vSphere 4.0 Quick Start Guide a few weeks back from my good friend and PowerCLI expert Alan Renouf . It is a great read and because of its size is really handy fo carrying around and referring to without needing to lug a 700 page book around with you. (Having said that I do currently have Scott Lowe’s Mastering vSphere in my bag at the moment!)

Alan is a contributor to the book and has provided examples of PowerCLI throughout to accompany various sections.

The final release of the book is now available from Amazon.com.

PowerCLI workshop at London VMUG 24th November

I have been lucky enough to be invited to run a PowerCLI  pre-show workshop before the main event of the next London VMUG on 24th November. A couple of VMUG’s back Alan Renouf ran a similar session on how to get started with PowerCLI. I thought this time I would move things one step on so the kind of topics I am likely to cover are reporting scripts and how you can make practical use of them, oneliners to get you great information and take a look at the VESI.

I will be giving away two e-book copies of Hal Rottenberg’s Managing VMware Infrastructure with Windows PowerShell TFM which he has kindly donated. I’m most likely to give these away to those who contribute the best during the workshop so there is an extra incentive to make it interactive.

Details to sign up can be found here.