These VMs were on Hardware Version 7 with the latest VMware Tools installed. Both of the above hardware components were using two of the HW7 only drivers, PVSCSI and VMXNET3.  The below KB article explains how to disable the HotPlug capability so that the ‘Safely Remove Hardware’ message no longer appears in the system tray.

KB article 1012225

This worked successfully on Windows Server 2003 VMs. On Windows Server 2008 VMs the message was only appearing when logged in with administrative privileges, again the KB article changes worked. (Actually they didn’t on 2008 – see below update)

One curiosity with the KB article was that it stated this change would not work with the VMXNET3 driver, however in both Windows 2003 and 2008 it was successful. Looking back at the KB today to make this post the article seems to have been updated (July 19th) to remove this caveat and now only states the more vague:

“Note: You can disable the HotPlug capability for PCI devices such as e1000 or vmxnet2 NICs”

Update: Actually the devices.hotplug = “false” suggestion in the KB article does not seem to work on Windows Server 2008 or 2008 R2.

Even with the above setting applied the below is still available from ‘Safely Remove Hardware’ in Windows Server 2008 and 2008 R2.

I thought I would test out and see what happens if you try to eject either device. Attempting to remove the disk results in the below, i.e. it won’t actually let you do it.

However, ejecting the network card sees no warning or blocking prompts and the network is consequently disconnected.

The network card is now no longer available to be removed.

Looking at the configuration of the VM you will see that the network adapater has been removed.

However, it is possible to add it back whilst powered on and the drivers are installed.

I’ll post a further update after my support ticket with VMware is answered.

Exchange 2010 is supported on hardware virtualisation platforms provided the conditions in this Technet article are met. Whilst looking at various options for a possible Exchange 2010 deployment for a user base in the hundreds (it obviously made sense to look at what possibilities are available if deciding to virtualise the mailbox server role) I stumbled across this blog post which suggested that whilst Exchange 2010 was supported as virtual, when running a DAG it was not supported if part of a virtualised cluster.

Given that most people looking to deploy Exchange 2010 DAG solutions virtually, may well already have an existing cluster and do not wish to purchase standalone virtual hosts just for this solution it seemed a bit dissapointing to read that Microsoft had taken this stance. I contacted MS support to clarify exactly what was and wasn’t supported.

Whilst going back and forth with the support guy I read this Technet Magazine Article which suggested that the above stance might have changed recently and in fact a DAG would be supported within a virtualised cluster, provided that all virtualisation HA features would be disabled for the Exchange 2010 DAG VMs. This seemed to reflect the below quote from the virtualisation support article

“DAGs are supported in hardware virtualization environments provided that the ….. clustered root servers have been configured to never failover or automatically move mailbox servers that are members of a DAG to another root server.”

Eventually they confirmed that this was correct and whilst it would be supported to deploy a DAG in a virtualised cluster with these features turned off (HA and DRS in VMware ESX) there was a strong emphasis that this would not be recommended. Exactly why it was not recommended was difficult to ascertain, the impression I got from the call was:

• As Aidan mentions in his blog post he speculates that it hasn’t been tested enough to be recommended yet. There was also a lot of emphasis on the call on large scale deployments and not much demand yet for testing smaller deployment scenarios.
• Fixing VMs to particular hosts does mean that the hosts themselves become a management burden, it was mentioned that this could be a high cost.

Whilst it does increase the level of management, purchasing modern physical hardware to run an Exchange 2010 DAG could mean that the servers are not highly utilised for smaller deployments – consequently the hardware cost of doing so can be comparatively high to the cost of a VM.

The main purpose in this deployement  to consider virtualising Exchange was hardware cost, not the HA features it could bring – Exchange itself will bring application HA.

The upshot of this post is essentially that an Exchange 2010 DAG in a virtualised cluster is supported by Microsoft provided the HA features are turned off, but not recommended by them. Consequently you can take that information into your design process and consider if it makes sense for your deployment.

Update:

If running VMware virtualisation, you must also consider that prior to vSphere 4.0 U1 running MSCS clusters within HA/DRS clusters was not supported, this changed with the release of U1. Since an Exchange 2010 DAG relies on Windows Failover Clustering then you must be on at least U1 to be supported – however, again HA / DRS must be disabled for the VMs in question. Virtual Kenneth has some very useful information about this in a blog post.

In particular it was appearing to hang for 5 mins, whilst on the screen the below was displayed:

Loading module multiextent

This would only happen after the install was completed and the host connected back to the fibre channel SAN, otherwise boot times were normal. It was also fine on ESX 3.5 U5 when connected to the SAN.

Some research led me to the below blog post which describes that this can occur when the hosts are part of a cluster which contain Passive MSCS Nodes with RDM LUNs.

http://www.vstable.com/tag/slow/

I made the recommendation to modify the Scsi.UWConflictRetries Advanced Setting to the minimum value of 80 and the boot time dropped to around 5 mins, slighty longer than before, but much better.

Of course you could also make this change in PowerCLI using the below:

Get-VMHost test01 | Set-VMHostAdvancedConfiguration -Name  Scsi.UWConflictRetries -Value 80

Watch out because the name of the Advanced Setting appears to be case sensitive.

Get-VM Test01

the below properties are outputted to the console:

It is possible to view more properties and values by runnning:

Get-VM Test01 | Format-List *

Unfortunately this still does not reveal the CpuFeatureMask property.  However, if we pipe the Get-VM command through to Get-View we will get back a .NET view object for the VM – by saving this into a variable we can then drill down through the various levels and look for the property we need.

$vm = Get-VM | Get-View
$vm

Below is the top level of information which is returned:

This may look initially like a bewildering amount of info, however if you look at the first few entries you could treat them like categories of information. Since CpuFeatureMask is a configuration property it would seem like a good guess to try looking in the Config category:

You can essentially browse this category by entering the following

$vm.config

and looking down the list of properties returned you will see CpuFeatureMask

On this particular VM CpuFeatureMask is not set, but you get the idea. To retrieve this property for all of your VMs is a simple one liner. With Select-Object we can pick one of the standard properties Name by simply specifiying it; we can use another technique to create our own property with a label and expression for the CpuFeatureMask.

Get-VM | Get-View | Select-Object Name,@{Name="CpuFeatureMask";Expression={$_.config.CpuFeatureMask}}

You could apply a similar technique to other cmdlets like Get-Host or Get-Cluster to retrieve non-standard properties.

They arrived just in time because I am currently preparing for a round of ESX patching and I needed to provide a report of hotfixes I was intending to deploy for a particular version of ESX. In the Update Manager GUI I had already created my baseline and scanned it against a host to produce a compliance report of hotfixes we would need to deploy this time.

You can see below that it produces a nice report for me, but I needed to export that information to a format whereby I can give that information to someone else.

One of the new cmdlets is Get-Baseline. I pointed this at my test baseline and with the code below was quickly able to get the information I needed out into a CSV file. I knew from the above report that I just needed to select any patches since 29/12/2009. One of the properties of the patches returned by Get-Baseline is the date it was published so first of all I set a date for which I could query after and stored it in the $BeginDate variable, I then queried the baseline using that date as a starting point.

$BeginDate = (Get-Date).adddays(-65)
Get-Baseline Test | Select-Object -ExpandProperty currentpatches | Where-Object {$_.'releasedate' -gt $BeginDate} | Select-Object Name,IDByVendor,Description,@{n='Product';e={$_.product | Select-Object -expandproperty Version}},ReleaseDate | Export-Csv patches.csv -NoTypeInformation

Which produces an output like this:

You’ll notice that I make use of the ExpandProperty parameter for Select-Object which makes it nice and easy to get to properties which are returned in an array, otherwise although they look fine in the console, when you export them to CSV you will not get what you hope for.

It’s days like today when I’m especially glad I started using PowerShell and very thankful that the vendors of technologies I’m using make this stuff so simple by providing cmdlets for managing their products.

Update:

Initially I tried to use the Get-Compliance cmdlet to find these patches rather than by date, however it only seemed to return a status of Compliant or Not Compliant. Thankfully following a post on the communities it has been pointed out that Get-Compliance has a Detailed parameter which returns a lot more information. Consequently there is no need to mess around with dates, instead you can query for NotCompliantPatches.

$ComplianceStatus = Get-Compliance -Entity 'Server1' -Detailed
$ComplianceStatus.NotCompliantPatches | Select-Object Name,IDByVendor,Description,@{n='Product';e={$_.product | Select-Object -expandproperty Version}},ReleaseDate | Export-Csv patches.csv -NoTypeInformation

Using Get-Stat, mem.usage.average comes back at 0% where it should be a significant value:

The same in the GUI, the other cluster memory stats have significant values, but Memory Usage Average flatlines along the bottom of the graph:

This stat shows normal behaviour against host machines. I reported it to VMware who confirmed it to be a bug in 4.0 U1.

Thought I would share in case anyone else spends some time scratching their head and checking maths in scripts like I did. I’ll post back when there is a fix.

Update 19/05/2010:

Just got a follow up email from VMware Support that this bug will be fixed in VC 4.0 U3 – no release date for that yet.

PowerCLI Workshop London VMUG.pptx

Thanks to all who chatted to me afterwards, its always nice to know that someone got something out of a session you put on.

Alan is a contributor to the book and has provided examples of PowerCLI throughout to accompany various sections.

The final release of the book is now available from Amazon.com.

Obviously there are great new headline features with things like Server Core and Read-Only Domain Controllers, but I have found a new feature I love which is the abilty to extend a system disk without having to go through some kind of convoluted process. It is particularly easy if it is a VM and you have enough space on the existing datastore to extend the VM’s disk without requesting more storage from your SAN team.

In VSphere 4.0 it even lets me extend the size of the vmdk with the server powered on, I don’t remember that being possible in 3.5, but I might be wrong. The big advantage for this for me is the fact that you do not need downtime to carry out this whole process, either extending the vmdk or the system volume!

Update: I just read in Mastering vSphere by Scott Lowe that the above is a new feature in vSphere 4.0 and the VM is required to be hardware version 7.

Take the following steps to carry it out:

1) Edit the settings of the 2008 VM and increase the size of the disk.

2) Within the Disk Management MMC, kick off a ‘Rescan Disks’

3) You will now see the additional space available at the end of the current disk.

4) Right-click the C: drive and choose ‘Extend Volume’

5) Follow the wizard through to add the additonal space. On completion your system disk will now have the additional space added – all with no downtime!

For completeness there are a number of ways you could do this in Window Server 2003, the way I have typically used is the following. (Make sure you have a good backup of this VM before going ahead)

1) Power down the VM. Edit the settings of the VM to grow the VMDK. Do not power on the VM!

2) Find another Windows Server 2003 VM which is Powered On and has access to the same datastores. Edit the settings of the VM, add an additional hard disk, select the option to use an existing virtual disk, browse to the initial VM’s vmdk and select it.

3) In Disk Management Rescan the disks and you will see an additional drive with unallocated space available other than your current drives.

4) Use the command line utility diskpart to extend this drive.

• Enter diskpart to start
• List Volume will show available volumes on the machine
• Check the additional volume has not been listed as a System partition
• Select that volume with Select Volume volumenumber
• Enter Extend to increase the size of the drive with all available space
• List Volume will let you check it was successful
• Exit to finish

Go back into Disk Management and you should be able to see a larger drive.

5) Edit the settings of the VM and remove the additional disk – make sure you do not select the option to delete the files from disk!

6) Power the initial VM back on. After logging back in the first time you will most likely be prompted to restart the server. Once logged back in, check Disk Management to see the larger system disk.