Average CPU and Memory Use Per ESX Cluster

More stats for my capacity report, this time I want to know on a typical day in the month what is the average CPU and memory use like across my hosts and clusters. Note: this post is not aimed at troubleshooting performance issues, rather at a high level gives me a reasonable idea of the CPU and memory use in each cluster during peak and non-peak hours. By running this each month I can look at possible trends and where extra resource may be required.

Get-Stat is a very useful cmdlet in PowerCLI to extract performance data across multiple hosts or VM’s for which it is not really easy to do the same kind of thing in the VI client when looking at multiple machines. By using this script we can collect the data for each host and then aggregate it across them all to get an average figure.

Connect-VIServer virtualcenter | Out-Null

$Clusters = Get-Cluster

$PeakStart = Get-Date -year 2009 -month 8 -day 27 -hour 8 -minute 0 -sec 0
$PeakEnd = Get-Date -year 2009 -month 8 -day 27 -hour 18 -minute 0 -sec 0
$OffPeakStart = Get-Date -year 2009 -month 8 -day 27 -hour 18 -minute 0 -sec 0
$OffPeakEnd = Get-Date -year 2009 -month 8 -day 28 -hour 8 -minute 0 -sec 0

foreach ($Cluster in $Clusters){

    $VMHosts = Get-Cluster $Cluster | Get-VMHost

    $PeakCPUStatTotal = 0
    $OffPeakCPUStatTotal = 0
    $PeakMemoryStatTotal = 0
    $OffPeakMemoryStatTotal = 0

    foreach ($VMHost in $VMHosts){

    $PeakCPUStat = $VMhost | Get-Stat -Start $PeakStart -Finish $PeakEnd -stat cpu.usage.average -IntervalSecs 4500
    $PeakCPUStatAverage = $PeakCPUStat | Measure-Object value -ave
    $PeakCPUStatTotal = $PeakCPUStatTotal + $PeakCPUStatAverage.average

    $OffPeakCPUStat = $VMhost | Get-Stat -Start $OffPeakStart -Finish $OffPeakEnd -stat cpu.usage.average -IntervalSecs 4500
    $OffPeakCPUStatAverage = $OffPeakCPUStat | Measure-Object value -ave
    $OffPeakCPUStatTotal = $OffPeakCPUStatTotal + $OffPeakCPUStatAverage.average

    $PeakMemoryStat = $VMhost | Get-Stat -Start $PeakStart -Finish $PeakEnd -stat mem.usage.average -IntervalSecs 4500
    $PeakMemoryStatAverage = $PeakMemoryStat | Measure-Object value -ave
    $PeakMemoryStatTotal = $PeakMemoryStatTotal + $PeakMemoryStatAverage.average

    $OffPeakMemoryStat = $VMhost | Get-Stat -Start $OffPeakStart -Finish $OffPeakEnd -stat mem.usage.average -IntervalSecs 4500
    $OffPeakMemoryStatAverage = $OffPeakMemoryStat | Measure-Object value -ave
    $OffPeakMemoryStatTotal = $OffPeakMemoryStatTotal + $OffPeakMemoryStatAverage.average

    }

    $PeakCPUStatResult = [math]::round(($PeakCPUStatTotal / $VMHosts.count), 0)
    $OffPeakCPUStatResult = [math]::round(($OffPeakCPUStatTotal / $VMHosts.count), 0)
    $PeakMemoryStatResult = [math]::round(($PeakMemoryStatTotal / $VMHosts.count), 0)
    $OffPeakMemoryStatResult = [math]::round(($OffPeakMemoryStatTotal / $VMHosts.count), 0)

    Write-Host "$Cluster has a Peak CPU Stat of $PeakCPUStatResult %"
    Write-Host "$Cluster has an Off Peak CPU Stat of $OffPeakCPUStatResult %"
    Write-Host "$Cluster has a Peak Memory Stat of $PeakMemoryStatResult %"
    Write-Host "$Cluster has an Off Peak Memory Stat of $OffPeakMemoryStatResult %"

}

Update:

Thanks to Glenn Sizemore for the suggestion of using the cluster level counters. I have re-visited the script and updated based on those counters and can confirm it runs a lot quicker.  Since the cluster CPU counter result is in MHz and my original script was based around percentages, we  first of all have to calculate the total amount of MHz available in the cluster, after that its just a question of expressing the cpu.usagemhz.average as a percentage of the total. Memory is easier because the valule reported by the cluster with mem.usage.average is already a percentage.

Connect-VIServer virtualcenter | Out-Null

$Clusters = Get-Cluster

$PeakStart = Get-Date -year 2009 -month 8 -day 27 -hour 8 -minute 0 -sec 0
$PeakEnd = Get-Date -year 2009 -month 8 -day 27 -hour 18 -minute 0 -sec 0
$OffPeakStart = Get-Date -year 2009 -month 8 -day 27 -hour 18 -minute 0 -sec 0
$OffPeakEnd = Get-Date -year 2009 -month 8 -day 28 -hour 8 -minute 0 -sec 0

foreach ($Cluster in $Clusters){

$TotalCPUHZ = 0
$TotalCPUMHZ = 0

    $VMHosts = $Cluster | Get-VMHost

    foreach ($VMHost in $VMHosts){

    $View = $VMHost | Get-View

    $NumCpuCores = $View.Hardware.CpuInfo.NumCpuCores
    $Hz = $View.Hardware.CpuInfo.Hz
    $CPUHZ = $NumCpuCores * $Hz

    $TotalCPUHZ = $TotalCPUHZ +$CPUHZ
    }

$TotalCPUMHZ = $TotalCPUHZ / 1000000

 $PeakCPUStatAverage = $cluster | Get-Stat -Start $PeakStart -Finish $PeakEnd -Stat cpu.usagemhz.average -IntervalSecs 4500 | Measure-Object value -ave
 $PeakCPUStatResult = [math]::round(($PeakCPUStatAverage.average / $TotalCPUMHZ *100), 0)

 $OffPeakCPUStatAverage = $cluster | Get-Stat -Start $OffPeakStart -Finish $OffPeakEnd -Stat cpu.usagemhz.average -IntervalSecs 4500 | Measure-Object value -ave
 $OffPeakCPUStatResult = [math]::round(($OffPeakCPUStatAverage.average / $TotalCPUMHZ *100), 0)

 $PeakMemoryStatAverage = $cluster | Get-Stat -Start $PeakStart -Finish $PeakEnd -Stat mem.usage.average -IntervalSecs 4500 | Measure-Object value -ave
 $PeakMemoryStatResult = [math]::round($PeakMemoryStatAverage.average, 0)

 $OffPeakMemoryStatAverage = $cluster | Get-Stat -Start $OffPeakStart -Finish $OffPeakEnd -Stat mem.usage.average -IntervalSecs 4500  | Measure-Object value -ave
 $OffPeakMemoryStatResult = [math]::round($OffPeakMemoryStatAverage.average, 0)

    Write-Host "$Cluster has a Peak CPU Stat of $PeakCPUStatResult %"
    Write-Host "$Cluster has an Off Peak CPU Stat of $OffPeakCPUStatResult %"
    Write-Host "$Cluster has a Peak Memory Stat of $PeakMemoryStatResult %"
    Write-Host "$Cluster has an Off Peak Memory Stat of $OffPeakMemoryStatResult %"
}

Of course the original script is still valid if you don’t have you hosts clustered, rather you simply have them grouped into folders. Although uncommon, this can occur if you have a particular workload spread across hosts which you wish to evaluate, but they were not clustered for whatever reason, cost being a typical exaxmple.

In this instance instead of:

$Clusters = Get-Cluster

Replace it with

$VMHosts = Get-Folder 'Non-Clustered Hosts' | Get-VMHost

Thanks again to Glenn for his tip about the cluster stats.