PowerCLI workshop at London VMUG 24th November

I have been lucky enough to be invited to run a PowerCLI  pre-show workshop before the main event of the next London VMUG on 24th November. A couple of VMUG’s back Alan Renouf ran a similar session on how to get started with PowerCLI. I thought this time I would move things one step on so the kind of topics I am likely to cover are reporting scripts and how you can make practical use of them, oneliners to get you great information and take a look at the VESI.

I will be giving away two e-book copies of Hal Rottenberg’s Managing VMware Infrastructure with Windows PowerShell TFM which he has kindly donated. I’m most likely to give these away to those who contribute the best during the workshop so there is an extra incentive to make it interactive.

Details to sign up can be found here.

Find the BIOS Version of an ESX Host

A while back I needed to confirm what level of BIOS firmware a bunch of ESX hosts were at. Unfortunately I ran out of time to look properly, but today I discovered how to do it whilst looking through the VMware SDK for something else. Turns out it is very simple:

$VMHost = Get-VMHost 'Server01' | Get-View
$VMHost.Hardware.BiosInfo

This will give you results along the lines:

BiosVersion                   ReleaseDate                   DynamicType                   DynamicProperty
———–                           ———–                          ———–                              —————
2.5.0                         12/09/2008 00:00:00

Average CPU and Memory Use Per ESX Cluster

More stats for my capacity report, this time I want to know on a typical day in the month what is the average CPU and memory use like across my hosts and clusters. Note: this post is not aimed at troubleshooting performance issues, rather at a high level gives me a reasonable idea of the CPU and memory use in each cluster during peak and non-peak hours. By running this each month I can look at possible trends and where extra resource may be required.

Get-Stat is a very useful cmdlet in PowerCLI to extract performance data across multiple hosts or VM’s for which it is not really easy to do the same kind of thing in the VI client when looking at multiple machines. By using this script we can collect the data for each host and then aggregate it across them all to get an average figure.

Connect-VIServer virtualcenter | Out-Null

$Clusters = Get-Cluster

$PeakStart = Get-Date -year 2009 -month 8 -day 27 -hour 8 -minute 0 -sec 0
$PeakEnd = Get-Date -year 2009 -month 8 -day 27 -hour 18 -minute 0 -sec 0
$OffPeakStart = Get-Date -year 2009 -month 8 -day 27 -hour 18 -minute 0 -sec 0
$OffPeakEnd = Get-Date -year 2009 -month 8 -day 28 -hour 8 -minute 0 -sec 0

foreach ($Cluster in $Clusters){

    $VMHosts = Get-Cluster $Cluster | Get-VMHost

    $PeakCPUStatTotal = 0
    $OffPeakCPUStatTotal = 0
    $PeakMemoryStatTotal = 0
    $OffPeakMemoryStatTotal = 0

    foreach ($VMHost in $VMHosts){

    $PeakCPUStat = $VMhost | Get-Stat -Start $PeakStart -Finish $PeakEnd -stat cpu.usage.average -IntervalSecs 4500
    $PeakCPUStatAverage = $PeakCPUStat | Measure-Object value -ave
    $PeakCPUStatTotal = $PeakCPUStatTotal + $PeakCPUStatAverage.average

    $OffPeakCPUStat = $VMhost | Get-Stat -Start $OffPeakStart -Finish $OffPeakEnd -stat cpu.usage.average -IntervalSecs 4500
    $OffPeakCPUStatAverage = $OffPeakCPUStat | Measure-Object value -ave
    $OffPeakCPUStatTotal = $OffPeakCPUStatTotal + $OffPeakCPUStatAverage.average

    $PeakMemoryStat = $VMhost | Get-Stat -Start $PeakStart -Finish $PeakEnd -stat mem.usage.average -IntervalSecs 4500
    $PeakMemoryStatAverage = $PeakMemoryStat | Measure-Object value -ave
    $PeakMemoryStatTotal = $PeakMemoryStatTotal + $PeakMemoryStatAverage.average

    $OffPeakMemoryStat = $VMhost | Get-Stat -Start $OffPeakStart -Finish $OffPeakEnd -stat mem.usage.average -IntervalSecs 4500
    $OffPeakMemoryStatAverage = $OffPeakMemoryStat | Measure-Object value -ave
    $OffPeakMemoryStatTotal = $OffPeakMemoryStatTotal + $OffPeakMemoryStatAverage.average

    }

    $PeakCPUStatResult = [math]::round(($PeakCPUStatTotal / $VMHosts.count), 0)
    $OffPeakCPUStatResult = [math]::round(($OffPeakCPUStatTotal / $VMHosts.count), 0)
    $PeakMemoryStatResult = [math]::round(($PeakMemoryStatTotal / $VMHosts.count), 0)
    $OffPeakMemoryStatResult = [math]::round(($OffPeakMemoryStatTotal / $VMHosts.count), 0)

    Write-Host "$Cluster has a Peak CPU Stat of $PeakCPUStatResult %"
    Write-Host "$Cluster has an Off Peak CPU Stat of $OffPeakCPUStatResult %"
    Write-Host "$Cluster has a Peak Memory Stat of $PeakMemoryStatResult %"
    Write-Host "$Cluster has an Off Peak Memory Stat of $OffPeakMemoryStatResult %"

}

Update:

Thanks to Glenn Sizemore for the suggestion of using the cluster level counters. I have re-visited the script and updated based on those counters and can confirm it runs a lot quicker.  Since the cluster CPU counter result is in MHz and my original script was based around percentages, we  first of all have to calculate the total amount of MHz available in the cluster, after that its just a question of expressing the cpu.usagemhz.average as a percentage of the total. Memory is easier because the valule reported by the cluster with mem.usage.average is already a percentage.

Connect-VIServer virtualcenter | Out-Null

$Clusters = Get-Cluster

$PeakStart = Get-Date -year 2009 -month 8 -day 27 -hour 8 -minute 0 -sec 0
$PeakEnd = Get-Date -year 2009 -month 8 -day 27 -hour 18 -minute 0 -sec 0
$OffPeakStart = Get-Date -year 2009 -month 8 -day 27 -hour 18 -minute 0 -sec 0
$OffPeakEnd = Get-Date -year 2009 -month 8 -day 28 -hour 8 -minute 0 -sec 0

foreach ($Cluster in $Clusters){

$TotalCPUHZ = 0
$TotalCPUMHZ = 0

    $VMHosts = $Cluster | Get-VMHost

    foreach ($VMHost in $VMHosts){

    $View = $VMHost | Get-View

    $NumCpuCores = $View.Hardware.CpuInfo.NumCpuCores
    $Hz = $View.Hardware.CpuInfo.Hz
    $CPUHZ = $NumCpuCores * $Hz

    $TotalCPUHZ = $TotalCPUHZ +$CPUHZ
    }

$TotalCPUMHZ = $TotalCPUHZ / 1000000

 $PeakCPUStatAverage = $cluster | Get-Stat -Start $PeakStart -Finish $PeakEnd -Stat cpu.usagemhz.average -IntervalSecs 4500 | Measure-Object value -ave
 $PeakCPUStatResult = [math]::round(($PeakCPUStatAverage.average / $TotalCPUMHZ *100), 0)

 $OffPeakCPUStatAverage = $cluster | Get-Stat -Start $OffPeakStart -Finish $OffPeakEnd -Stat cpu.usagemhz.average -IntervalSecs 4500 | Measure-Object value -ave
 $OffPeakCPUStatResult = [math]::round(($OffPeakCPUStatAverage.average / $TotalCPUMHZ *100), 0)

 $PeakMemoryStatAverage = $cluster | Get-Stat -Start $PeakStart -Finish $PeakEnd -Stat mem.usage.average -IntervalSecs 4500 | Measure-Object value -ave
 $PeakMemoryStatResult = [math]::round($PeakMemoryStatAverage.average, 0)

 $OffPeakMemoryStatAverage = $cluster | Get-Stat -Start $OffPeakStart -Finish $OffPeakEnd -Stat mem.usage.average -IntervalSecs 4500  | Measure-Object value -ave
 $OffPeakMemoryStatResult = [math]::round($OffPeakMemoryStatAverage.average, 0)

    Write-Host "$Cluster has a Peak CPU Stat of $PeakCPUStatResult %"
    Write-Host "$Cluster has an Off Peak CPU Stat of $OffPeakCPUStatResult %"
    Write-Host "$Cluster has a Peak Memory Stat of $PeakMemoryStatResult %"
    Write-Host "$Cluster has an Off Peak Memory Stat of $OffPeakMemoryStatResult %"
}

Of course the original script is still valid if you don’t have you hosts clustered, rather you simply have them grouped into folders. Although uncommon, this can occur if you have a particular workload spread across hosts which you wish to evaluate, but they were not clustered for whatever reason, cost being a typical exaxmple.

In this instance instead of:

$Clusters = Get-Cluster

Replace it with

$VMHosts = Get-Folder 'Non-Clustered Hosts' | Get-VMHost

Thanks again to Glenn for his tip about the cluster stats.

How many hosts and VM’s in Virtual Center

More stats for my capacity report, this time numbers of VM’s in total in Virtual Center and average per host in each cluster. Obviously performance is not based on the numbers of VM’s per host, but its an interesting figure to keep track of.

Connect-VIServer virtualcenter | Out-Null

# Total number of hosts
$TotalVMHosts = Get-VMHost
$TotalVMHostsCount = $TotalVMHosts.count
Write-Host "There are $TotalVMHostsCount Hosts in $DefaultVIServer"

# Total number of guests
$TotalVMs = Get-VM
$TotalVMsCount = $TotalVMs.count
Write-Host "There are $TotalVMsCount Virtual Machines in $DefaultVIServer"

$Clusters = Get-Cluster

foreach ($Cluster in $Clusters){

    $VMHosts = Get-Cluster $Cluster | Get-VMHost
    $VMs = Get-Cluster $Cluster | Get-VM

    $AverageVMsPerCluster = [math]::round(($VMs.count / $VMHosts.count), 1)
    Write-Host "$Cluster has $AverageVMsPerCluster guests per VMware host"

}

How much SAN storage are my ESX hosts using?

As part of some monthly stats I need to collate, I had to find out how much SAN storage was being used by my ESX hosts. Luckily we have a pretty good naming convention for our datastores so it’s pretty easy to filter out local storage figures. We simply query Virtual Centre for datastores, filter out any local ones, total up their capacity and then convert the figure into Terabytes.

(Note this gives you the total amount of storage allocated to you by your storage team, not how much is actually in use as part of that. If that’s what you need then use the ‘FreeSpaceMB figure to work it out)

Connect-VIServer virtualcenter | Out-Null

    $datastores = Get-Datastore |  Where-Object {$_.name -notlike "*local*"}
    $sum = $datastores | Measure-Object capacitymb -sum
    $result = [math]::round(($sum.sum * 1MB / 1GB) / 1024,1)

Write-Host 'Total SAN Storage' $result 'Terabytes'

(Note: I don’t normally use Write-Host for output, but for the purposes of this example it works)

If you don’t have such a clear naming convention for your datastores then Carter Shanklin has a post over on the PowerCLI blog which will return only those on SAN storage.

More stat collections to follow…….

This month I shall mostly be wearing…..an ESX hat.

During the last month whilst some colleagues have been away on their holidays, I’ve been spending a lot more time with our Virtual Infrastructure than normal and I thought I’d share a few tips I’ve picked up along the way. They are probably no brainers for your hardcore VMware administrators, but for those like me who aren’t 100% dedicated to one specific area, and particularly have been long time Windows admins, there’s nothing like getting really involved with something for a few weeks to really get to grips with how this stuff works.

The kind of tasks and troubleshooting you carry out in the VI client are pretty intuitive, but when you have to delve a bit deeper and venture into the Service Console of ESX its time to thank that well spent time learning PowerShell, the PowerCLI toolkit and generally being proficient with command line based troubleshooting.

1) Kill a VM which won’t power off

Following attempts to shutdown a virtual machine and then use the Power Off button, it would not power off at all. Use the Service Console commands:

vm-support -x   (to get the ID of VM’s on the host)  and

vm-support -X <vmid> to kill the VM in question.

http://www.yellow-bricks.com/2009/04/15/the-basics-how-to-kill-a-vm-thats-stuck-during-shutdown/

http://www.techhead.co.uk/vmware-esx-how-to-clear-a-hung-vm

2) Switch the Service Console NIC

One of the ports in a 4 port NIC failed and it was the port being used by the Service Console, consequently the host was not available in Virtual Center although all the VM’s were still running.  I had another NIC available so re-patched, but obviously couldn’t use the VI client to change the Service Console NIC.

Check your available nics:
“esxcfg-nics -l”
Check which is connected to service console switch (look at “Uplinks” of vSwitch0):
“esxcfg-vswitch -l”
unlink actual vmnic from vSwitch0:
“esxcfg-vswitch -U vmnic0 vSwitch0″
link another nic to vSwitch0:
“esxcfg-vswitch -L vmnic1 vSwitch0″

http://www.experts-exchange.com/Software/VMWare/Q_23027712.html

3) Vmotion hangs at 10% or 94%

Moving a stack of VM’s around with VMotion, a couple got stuck at 94% and all the others stayed queued behind them, cancelling these moves in the VI client didn’t do anything, other than changing the status of the queued VMotions to ‘cancelled’. The couple which were stuck at 94% actually seemed to have completed successfully, i.e. they were on the new hosts. To clear these errors and allow further VMotions to take place I had to carry out the following on both the source and destination hosts: (before running the first command you need to check you do not have Automatic Startup / Shutdown enabled)

service mgmt-vmware restart

and

service vmware-vpxa restart

http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=1003490

The first time I just went with

service vmware-vpxa restart

only, but then subsequent VMotion’s got stuck at 10%.

I’m not sure why this happened, a subsequent rule of thumb seemed to be don’t submit more than 4 Vmotions requests at at time, but I don’t have any technical reasons to back this up.

4) VM Configuaration files not in same datastore as vmdk disk

Before some SAN work I noticed on a report that some VM’s config files were located in a folder which was on a different datastore to the VMDK – this was going to make the work more tricky. Since I was able to power off the VM’s in question I could use the Migrate wizard and the Advanced button on the disk page to move the configuration files into the same folder on the same datastore as the vmdk file – much tidier!

5) Find all VM’s with an RDM

Need to find which of your VM’s has an RDM connected  – a great PowerShell script from LucD.

http://communities.vmware.com/message/1063909;jsessionid=791355B6149788C1EBDA7F25B2A2B270

6) Run Dell DSET utility to produce hardware fault report.

If you have a hardware fault with one of their servers Dell may require you to run a DSET report and send them the output. Download the Linux version and copy it to your ESX server – an easy way to transfer the files is to use a utility like the great (and free)  Veeam FastSCP 3.0.  Then from the Service Console navigate your way to where you copied the file and change the permissions so that you can execute it, since most likely by default you won’t be able to:

chmod +x delldset_v1.7.0.119.bin

then run the utility

./delldset_v1.7.0.119.bin

Option 2 to run the report and once complete you can copy the output file using FastSCP to somewhere you can easily get to it.

7) Updating the BIOS of a Dell 2950 running ESX

OK so first thought was since it’s not running Windows I’ll just boot up with a floppy BIOS update like the old days, but since these servers have no floppy drive and no USB one to hand, I went the better way really, downloaded the Linux version and it was pretty easy.

http://www.tonywilko.net/blog/?p=3

8 Unable to format a LUN

The storage team provisioned some new LUN’s for me, but when I tried to configure them with VMFS through the Add Storage Wizard some of them were blank on the Available field and on the subsequent screen I recieved the error ‘Unable to read partition information’ and was not able to add the storage.

Luckily my colleague Alan Renouf had returned from holiday and pointed me in the direction of his blog with a post where he had experienced the same issue – it worked for me too.

If I pick up anymore tips before they get back, I’ll be sure to let you know.

Virtualising Citrix on VMware

I was lucky enough to take on a project initially started and blogged about by my co-host on the Get-Scripting podcast Alan Renouf.

In summary, his posts were mainly around the design decision of whether to go for VM’s with one or two vCPU’s and how many Citrix users you could support per VM.  Following on from his initial testing using Citrix Edgesight we ran a pilot with a few different scenarios and it turned out that the best performance with the highest number of Citrix users per VM came out to be a VM with 2 x vCPU’s; a conclusion which didn’t really match the initial testing, I guess you can’t beat real users doing real work and the sometimes crazy things they get up to pushing the boundries of performance.

A number of other decisions were also made at this time, most of which contributed to other significant cost savings on top of those we were going to achieve simply by reducing the number of physical boxes used to host the Citrix environment.

Something else which came out of the pilot was a decision to store the VM’s on local storage not SAN. Whilst this obviously reduces the flexibility offered by a virtualisation solution with shared storage, which gives options like VMotion, DRS etc, the cost savings gained by using local storage were very significant. Not only did we have none of the charges associated with SAN storage (fibre cards, cabling, switch ports, SAN disk) we could deploy the hosts with ESX Foundation licenses. From a redundancy and maintenance point of view we designed it so that we could afford to lose more than one host for a period of time and still have enough capacity to provide a good service.

We deployed four VM’s per ESX host, i.e. 8 cores available to 8 x vCPU’s. (Note: I have recently read Duncan Epping’s post around how many cores you should specify when using CPU affinity. It makes for interesting reading, thankfully we are not currently seeing any issues around what might arise from this)

During the pilot and the early part of the rollout we found we were able to happily achieve around 45 users per VM, i.e. up to 180 per ESX host with CPU levels on the host comfortably averaging below the 75% mark. As the rollout progressed and we retired the physical Citrix boxes the levels attracted by each VM were more typically around the 40 users mark, i.e. approx 160 users per physical host.

This was because we were able to replace three physical Citrix boxes with one ESX host containing four Citrix VM’s, so a 3 – 1 reduction physical, but a 25% increase in the number of Citrix servers which obviously means you naturally attaract less users per Citrix server with a consistent number of users.  However, since we deployed 2 x vCPU machines it also meant cost savings with half the required Windows VM’s over the original plan to deploy 1x vCPU VM’s which would have meant eight Citrix VM’s per host.

One issue we did experience was that of vCPU peaks from rogue user processes which would hog all the CPU for significant periods of time and give a bad experience to other users on that VM. This was believed to happen previously in the physical Citrix deployemnt, but was more easily masked by the availability of physical cores. Most typically these processes would be Internet Explorer, quite often accessing Flash based content. To mitigate this issue we used some application threading software on each VM to set maximum levels for CPU usage per user process. This performed very well by limiting these processes to a certain amount of vCPU and consequently not impacting other Citrix users’ performance – the decision to use 2 x vCPU’s in a VM helped here too, the 1x vCPU VM’s in the pilot really suffered with this problem.

UK User Group Events in May

So May looks like a great month for some of the user groups I regularly attend.

First up we have the VMware user group in London on Thursday May 14th. This is an excellent event for VMware administrators to attend and has a great mix of vendor and community contributions. In particular this time check out Alan Renouf’s pre-show PowerShell workshop. This is before the usual start time and should be great if you are new to PowerShell or already using the VI Toolkit.

Sign up details are here.

Next up is the PowerShell user group at Microsoft in Reading on Tuesday 19th May. This time we have a real treat in store, PowerShell MVP Dmitry Sotnikov from Quest will be stopping by on his whistle stop tour of the UK to tell us about PowerGUI and the AD cmdlets, in particular recent updates. This was a great event when he presented for us last year and is a real opportunity if you use either of these products to get your questions answered or put forward any suggestions for improvements. You also get to meet a really cool guy and see if he manages to blue screen his machine again during a PowerShell demo by stopping all the services, having forgot to use the ‘whatif’ paramter! (I told him I’d get him back for making fun of me when I interviewed him for the Get-Scripting podcast)

Sign up details are here. You need to contact Richard if you wish to attend in person.

Finally on Thursday 28th May the Windows Server and Vista Squad user groups join the Active Directory user group for a joint Windows 7 event. Full content is yet to be confirmed, but that’s the first Windows 7 event I’ve heard of in this country.

Full details here.

Hope to see you at one of these events.

Get-Scripting Guys Take Over the March UK Powershell User Group

Myself and Alan Renouf from the Get-Scripting Podcast will be presenting this month at the UK Powershell User Group on Thursday 26th March at Microsoft in Reading.

First up on the night will be Richard Siddaway talking to us about using Regular Expressions in Powershell. This was requested at a previous event and I know that Richard is really looking forward to talking about that subject

Then the Get-Scripting guys will take over:

I will be talking about some of the features that are part of Active Directory in Server 2008 R2 , currently in beta, in particular the native AD Powershell cmdlets which will ship as part of that product and what I have learned about them so far.

vExpert Alan Renouf will then tell us about the VI toolkit, which for those of you not in the know are the cmdlets shipped by VMware to enable you to manage their ESX product.

A good mixture I think of general scripting skills, third-party added value to your Powershell learning and future technologies.

If you can’t make the event in person then as usual it will be broadcast by LiveMeeting (details here). It wouldn’t be a UK Powershell event if the LiveMeeting part doesn’t go 100% to plan, but we will endeavour to ensure we avoid the usual trick of the microphone going on mute.

If you wish to attend the event in person you should contact Richard via his blog so that he can have a badge for you at the welcome desk. If the prospect of us presenting isn’t enough to attract you then as always there will be pizza at half-time

Today’s UK VMware User Group

Today’s UK VMware User Group was a great community content event. Of course there was a sponsor presentation (Veeam) without whom these type of events can’t be put on, but there were also a lot of contributions from people in the group.

We had:

1) Veeam talking about their reporting and backup products.

2) Mike Laverick from RTFM education talking about Site Recovery Manager and not the VMware view of it, rather real world struggles – warts and all as he put it. Its great to see this kind of content because you get to find out how these products really function in reality. He also gave away a few copies of his book on Site Recovery Manager which I’m sure is a must read for this topic.

3) Get-Scripting’s very own vExpert Alan Renouf gave us a debrief from VMworld Europe in Cannes. By the sounds of it a great event, but possibly could be extended to include an extra day and maybe needs a bigger venue given the issues some people had getting into sessions.

4) A presentation from another community member about technology futures, ‘the cloud’, advances in mobile devices, stuff like that, and how they may affect the corporate world. This is now the second event I have been to which has talked about employee owned IT, i.e. you turn up to a job with your own laptop and corporate IT supply you either with a VM to run corporate applications or all delivered through a browser. The demand coming from cool, young, hip people (like me ) who won’t stand for the constraints of a corporate environment.

“Why can’t I run facebook / second life / quake / twitter on my macbook / ipod/ pink netbook and just run your apps on the same machine too?”

5) Steve Bruck and Alan Renouf (again) with an update about the recently released VI toolkit V1.5 . Some good examples, also lots of interest in Alan’s getting started guide.

6) Finally finished off with more community content, this time about the upcoming vCenter Heartbeat product.

It was great to see slightly more audience participation than normal this time, which I think might have come from the slightly smaller room and more, but shorter sessions. I really enjoyed it anyway.

If you want to hear REAL people talking about how they really use VMware based products then this is the event for you.

Next one 14th May! More details can be found here.