Reporting on VMware Update Manager Baselines with PowerCLI

I’ve mentioned on this blog before that I’ve been using VMware Update Manager a lot recently – and wrote about some of my experiences here. Today I was really pleased to see that Carter Shanklin’s team released some cmdlets for PowerCLI to cover Update Manager which had only previously been available back as a beta in the VI Toolkit days.

They arrived just in time because I am currently preparing for a round of ESX patching and I needed to provide a report of hotfixes I was intending to deploy for a particular version of ESX. In the Update Manager GUI I had already created my baseline and scanned it against a host to produce a compliance report of hotfixes we would need to deploy this time.

You can see below that it produces a nice report for me, but I needed to export that information to a format whereby I can give that information to someone else.

One of the new cmdlets is Get-Baseline. I pointed this at my test baseline and with the code below was quickly able to get the information I needed out into a CSV file. I knew from the above report that I just needed to select any patches since 29/12/2009. One of the properties of the patches returned by Get-Baseline is the date it was published so first of all I set a date for which I could query after and stored it in the $BeginDate variable, I then queried the baseline using that date as a starting point.

$BeginDate = (Get-Date).adddays(-65)
Get-Baseline Test | Select-Object -ExpandProperty currentpatches | Where-Object {$_.'releasedate' -gt $BeginDate} | Select-Object Name,IDByVendor,Description,@{n='Product';e={$_.product | Select-Object -expandproperty Version}},ReleaseDate | Export-Csv patches.csv -NoTypeInformation

Which produces an output like this:

You’ll notice that I make use of the ExpandProperty parameter for Select-Object which makes it nice and easy to get to properties which are returned in an array, otherwise although they look fine in the console, when you export them to CSV you will not get what you hope for.

It’s days like today when I’m especially glad I started using PowerShell and very thankful that the vendors of technologies I’m using make this stuff so simple by providing cmdlets for managing their products.

Update:

Initially I tried to use the Get-Compliance cmdlet to find these patches rather than by date, however it only seemed to return a status of Compliant or Not Compliant. Thankfully following a post on the communities it has been pointed out that Get-Compliance has a Detailed parameter which returns a lot more information. Consequently there is no need to mess around with dates, instead you can query for NotCompliantPatches.

$ComplianceStatus = Get-Compliance -Entity 'Server1' -Detailed
$ComplianceStatus.NotCompliantPatches | Select-Object Name,IDByVendor,Description,@{n='Product';e={$_.product | Select-Object -expandproperty Version}},ReleaseDate | Export-Csv patches.csv -NoTypeInformation

Bug in Cluster mem.usage.average Statistic in vSphere 4.0 U1

A while back I posted a script on a basic capacity report I run each month to get an overview of CPU and Memory usage in our various clusters. Since upgrading to vSphere 4.0 U1 I noticed some strange behavior in the results for memory, i.e. they came back at pretty close to 0% (typically between 0.05 and 0.06%) for the average memory usage in a cluster which typically were quite heavily used. On investigating further this also appeared the same in the GUI.

Using Get-Stat, mem.usage.average comes back at 0% where it should be a significant value:

The same in the GUI, the other cluster memory stats have significant values, but Memory Usage Average flatlines along the bottom of the graph:

This stat shows normal behaviour against host machines. I reported it to VMware who confirmed it to be a bug in 4.0 U1.

Thought I would share in case anyone else spends some time scratching their head and checking maths in scripts like I did. I’ll post back when there is a fix.

New Simple Talk Article – Using VMware vCenter Update Manager to keep your vSphere Hosts Up-To-Date with Patching

I’m pretty obsessive about patching systems whatever the OS; keeping servers up-to-date and consistent across your environment is very important. VMware ship a great tool vCenter Update Manager to help you with this task for ESX / ESXi systems – you can find out more about it in a new article I have written for Simple-Talk.

Slide Deck from Nov 2009 London VMUG

After my PowerCLI session at yesterday’s London VMUG a few people asked me for the content. I believe the content from all sessions will soon be posted to http://www.box.net/londonug, but in the meantime you can get my slides from the below link.

PowerCLI Workshop London VMUG.pptx

Thanks to all who chatted to me afterwards, its always nice to know that someone got something out of a session you put on.

vSphere 4.0 Quick Start Guide – Now Available From Amazon

I recieved a preview copy of the vSphere 4.0 Quick Start Guide a few weeks back from my good friend and PowerCLI expert Alan Renouf . It is a great read and because of its size is really handy fo carrying around and referring to without needing to lug a 700 page book around with you. (Having said that I do currently have Scott Lowe’s Mastering vSphere in my bag at the moment!)

Alan is a contributor to the book and has provided examples of PowerCLI throughout to accompany various sections.

The final release of the book is now available from Amazon.com.

Extending the System Disk in Windows Server 2008

As a system administrator it is often not the latest and greatest big new features of a new operating system which you end up finding the most useful, sometimes its the small improvements which really make your life easier. I’ve been spending a lot more time with Windows Server 2008 recently (OK, I know R2 is out, but it takes time for large organisations to move away from older OS’s and applications.)

Obviously there are great new headline features with things like Server Core and Read-Only Domain Controllers, but I have found a new feature I love which is the abilty to extend a system disk without having to go through some kind of convoluted process. It is particularly easy if it is a VM and you have enough space on the existing datastore to extend the VM’s disk without requesting more storage from your SAN team.

In VSphere 4.0 it even lets me extend the size of the vmdk with the server powered on, I don’t remember that being possible in 3.5, but I might be wrong. The big advantage for this for me is the fact that you do not need downtime to carry out this whole process, either extending the vmdk or the system volume!

Update: I just read in Mastering vSphere by Scott Lowe that the above is a new feature in vSphere 4.0 and the VM is required to be hardware version 7.

Take the following steps to carry it out:

1) Edit the settings of the 2008 VM and increase the size of the disk.

2) Within the Disk Management MMC, kick off a ‘Rescan Disks’

3) You will now see the additional space available at the end of the current disk.

4) Right-click the C: drive and choose ‘Extend Volume’

5) Follow the wizard through to add the additonal space. On completion your system disk will now have the additional space added – all with no downtime!

For completeness there are a number of ways you could do this in Window Server 2003, the way I have typically used is the following. (Make sure you have a good backup of this VM before going ahead)

1) Power down the VM. Edit the settings of the VM to grow the VMDK. Do not power on the VM!

2) Find another Windows Server 2003 VM which is Powered On and has access to the same datastores. Edit the settings of the VM, add an additional hard disk, select the option to use an existing virtual disk, browse to the initial VM’s vmdk and select it.

3) In Disk Management Rescan the disks and you will see an additional drive with unallocated space available other than your current drives.

4) Use the command line utility diskpart to extend this drive.

• Enter diskpart to start
• List Volume will show available volumes on the machine
• Check the additional volume has not been listed as a System partition
• Select that volume with Select Volume volumenumber
• Enter Extend to increase the size of the drive with all available space
• List Volume will let you check it was successful
• Exit to finish

Go back into Disk Management and you should be able to see a larger drive.

5) Edit the settings of the VM and remove the additional disk – make sure you do not select the option to delete the files from disk!

6) Power the initial VM back on. After logging back in the first time you will most likely be prompted to restart the server. Once logged back in, check Disk Management to see the larger system disk.

PowerCLI workshop at London VMUG 24th November

I have been lucky enough to be invited to run a PowerCLI  pre-show workshop before the main event of the next London VMUG on 24th November. A couple of VMUG’s back Alan Renouf ran a similar session on how to get started with PowerCLI. I thought this time I would move things one step on so the kind of topics I am likely to cover are reporting scripts and how you can make practical use of them, oneliners to get you great information and take a look at the VESI.

I will be giving away two e-book copies of Hal Rottenberg’s Managing VMware Infrastructure with Windows PowerShell TFM which he has kindly donated. I’m most likely to give these away to those who contribute the best during the workshop so there is an extra incentive to make it interactive.

Details to sign up can be found here.

Find the BIOS Version of an ESX Host

A while back I needed to confirm what level of BIOS firmware a bunch of ESX hosts were at. Unfortunately I ran out of time to look properly, but today I discovered how to do it whilst looking through the VMware SDK for something else. Turns out it is very simple:

$VMHost = Get-VMHost 'Server01' | Get-View
$VMHost.Hardware.BiosInfo

This will give you results along the lines:

BiosVersion                   ReleaseDate                   DynamicType                   DynamicProperty
———–                           ———–                          ———–                              —————
2.5.0                         12/09/2008 00:00:00

Average CPU and Memory Use Per ESX Cluster

More stats for my capacity report, this time I want to know on a typical day in the month what is the average CPU and memory use like across my hosts and clusters. Note: this post is not aimed at troubleshooting performance issues, rather at a high level gives me a reasonable idea of the CPU and memory use in each cluster during peak and non-peak hours. By running this each month I can look at possible trends and where extra resource may be required.

Get-Stat is a very useful cmdlet in PowerCLI to extract performance data across multiple hosts or VM’s for which it is not really easy to do the same kind of thing in the VI client when looking at multiple machines. By using this script we can collect the data for each host and then aggregate it across them all to get an average figure.

Connect-VIServer virtualcenter | Out-Null

$Clusters = Get-Cluster

$PeakStart = Get-Date -year 2009 -month 8 -day 27 -hour 8 -minute 0 -sec 0
$PeakEnd = Get-Date -year 2009 -month 8 -day 27 -hour 18 -minute 0 -sec 0
$OffPeakStart = Get-Date -year 2009 -month 8 -day 27 -hour 18 -minute 0 -sec 0
$OffPeakEnd = Get-Date -year 2009 -month 8 -day 28 -hour 8 -minute 0 -sec 0

foreach ($Cluster in $Clusters){

    $VMHosts = Get-Cluster $Cluster | Get-VMHost

    $PeakCPUStatTotal = 0
    $OffPeakCPUStatTotal = 0
    $PeakMemoryStatTotal = 0
    $OffPeakMemoryStatTotal = 0

    foreach ($VMHost in $VMHosts){

    $PeakCPUStat = $VMhost | Get-Stat -Start $PeakStart -Finish $PeakEnd -stat cpu.usage.average -IntervalSecs 4500
    $PeakCPUStatAverage = $PeakCPUStat | Measure-Object value -ave
    $PeakCPUStatTotal = $PeakCPUStatTotal + $PeakCPUStatAverage.average

    $OffPeakCPUStat = $VMhost | Get-Stat -Start $OffPeakStart -Finish $OffPeakEnd -stat cpu.usage.average -IntervalSecs 4500
    $OffPeakCPUStatAverage = $OffPeakCPUStat | Measure-Object value -ave
    $OffPeakCPUStatTotal = $OffPeakCPUStatTotal + $OffPeakCPUStatAverage.average

    $PeakMemoryStat = $VMhost | Get-Stat -Start $PeakStart -Finish $PeakEnd -stat mem.usage.average -IntervalSecs 4500
    $PeakMemoryStatAverage = $PeakMemoryStat | Measure-Object value -ave
    $PeakMemoryStatTotal = $PeakMemoryStatTotal + $PeakMemoryStatAverage.average

    $OffPeakMemoryStat = $VMhost | Get-Stat -Start $OffPeakStart -Finish $OffPeakEnd -stat mem.usage.average -IntervalSecs 4500
    $OffPeakMemoryStatAverage = $OffPeakMemoryStat | Measure-Object value -ave
    $OffPeakMemoryStatTotal = $OffPeakMemoryStatTotal + $OffPeakMemoryStatAverage.average

    }

    $PeakCPUStatResult = [math]::round(($PeakCPUStatTotal / $VMHosts.count), 0)
    $OffPeakCPUStatResult = [math]::round(($OffPeakCPUStatTotal / $VMHosts.count), 0)
    $PeakMemoryStatResult = [math]::round(($PeakMemoryStatTotal / $VMHosts.count), 0)
    $OffPeakMemoryStatResult = [math]::round(($OffPeakMemoryStatTotal / $VMHosts.count), 0)

    Write-Host "$Cluster has a Peak CPU Stat of $PeakCPUStatResult %"
    Write-Host "$Cluster has an Off Peak CPU Stat of $OffPeakCPUStatResult %"
    Write-Host "$Cluster has a Peak Memory Stat of $PeakMemoryStatResult %"
    Write-Host "$Cluster has an Off Peak Memory Stat of $OffPeakMemoryStatResult %"

}

Update:

Thanks to Glenn Sizemore for the suggestion of using the cluster level counters. I have re-visited the script and updated based on those counters and can confirm it runs a lot quicker.  Since the cluster CPU counter result is in MHz and my original script was based around percentages, we  first of all have to calculate the total amount of MHz available in the cluster, after that its just a question of expressing the cpu.usagemhz.average as a percentage of the total. Memory is easier because the valule reported by the cluster with mem.usage.average is already a percentage.

Connect-VIServer virtualcenter | Out-Null

$Clusters = Get-Cluster

$PeakStart = Get-Date -year 2009 -month 8 -day 27 -hour 8 -minute 0 -sec 0
$PeakEnd = Get-Date -year 2009 -month 8 -day 27 -hour 18 -minute 0 -sec 0
$OffPeakStart = Get-Date -year 2009 -month 8 -day 27 -hour 18 -minute 0 -sec 0
$OffPeakEnd = Get-Date -year 2009 -month 8 -day 28 -hour 8 -minute 0 -sec 0

foreach ($Cluster in $Clusters){

$TotalCPUHZ = 0
$TotalCPUMHZ = 0

    $VMHosts = $Cluster | Get-VMHost

    foreach ($VMHost in $VMHosts){

    $View = $VMHost | Get-View

    $NumCpuCores = $View.Hardware.CpuInfo.NumCpuCores
    $Hz = $View.Hardware.CpuInfo.Hz
    $CPUHZ = $NumCpuCores * $Hz

    $TotalCPUHZ = $TotalCPUHZ +$CPUHZ
    }

$TotalCPUMHZ = $TotalCPUHZ / 1000000

 $PeakCPUStatAverage = $cluster | Get-Stat -Start $PeakStart -Finish $PeakEnd -Stat cpu.usagemhz.average -IntervalSecs 4500 | Measure-Object value -ave
 $PeakCPUStatResult = [math]::round(($PeakCPUStatAverage.average / $TotalCPUMHZ *100), 0)

 $OffPeakCPUStatAverage = $cluster | Get-Stat -Start $OffPeakStart -Finish $OffPeakEnd -Stat cpu.usagemhz.average -IntervalSecs 4500 | Measure-Object value -ave
 $OffPeakCPUStatResult = [math]::round(($OffPeakCPUStatAverage.average / $TotalCPUMHZ *100), 0)

 $PeakMemoryStatAverage = $cluster | Get-Stat -Start $PeakStart -Finish $PeakEnd -Stat mem.usage.average -IntervalSecs 4500 | Measure-Object value -ave
 $PeakMemoryStatResult = [math]::round($PeakMemoryStatAverage.average, 0)

 $OffPeakMemoryStatAverage = $cluster | Get-Stat -Start $OffPeakStart -Finish $OffPeakEnd -Stat mem.usage.average -IntervalSecs 4500  | Measure-Object value -ave
 $OffPeakMemoryStatResult = [math]::round($OffPeakMemoryStatAverage.average, 0)

    Write-Host "$Cluster has a Peak CPU Stat of $PeakCPUStatResult %"
    Write-Host "$Cluster has an Off Peak CPU Stat of $OffPeakCPUStatResult %"
    Write-Host "$Cluster has a Peak Memory Stat of $PeakMemoryStatResult %"
    Write-Host "$Cluster has an Off Peak Memory Stat of $OffPeakMemoryStatResult %"
}

Of course the original script is still valid if you don’t have you hosts clustered, rather you simply have them grouped into folders. Although uncommon, this can occur if you have a particular workload spread across hosts which you wish to evaluate, but they were not clustered for whatever reason, cost being a typical exaxmple.

In this instance instead of:

$Clusters = Get-Cluster

Replace it with

$VMHosts = Get-Folder 'Non-Clustered Hosts' | Get-VMHost

Thanks again to Glenn for his tip about the cluster stats.

How many hosts and VM’s in Virtual Center

More stats for my capacity report, this time numbers of VM’s in total in Virtual Center and average per host in each cluster. Obviously performance is not based on the numbers of VM’s per host, but its an interesting figure to keep track of.

Connect-VIServer virtualcenter | Out-Null

# Total number of hosts
$TotalVMHosts = Get-VMHost
$TotalVMHostsCount = $TotalVMHosts.count
Write-Host "There are $TotalVMHostsCount Hosts in $DefaultVIServer"

# Total number of guests
$TotalVMs = Get-VM
$TotalVMsCount = $TotalVMs.count
Write-Host "There are $TotalVMsCount Virtual Machines in $DefaultVIServer"

$Clusters = Get-Cluster

foreach ($Cluster in $Clusters){

    $VMHosts = Get-Cluster $Cluster | Get-VMHost
    $VMs = Get-Cluster $Cluster | Get-VM

    $AverageVMsPerCluster = [math]::round(($VMs.count / $VMHosts.count), 1)
    Write-Host "$Cluster has $AverageVMsPerCluster guests per VMware host"

}