Safely Remove Hardware Appears on HW v7 VMs

Recently I noticed an issue with some Citrix VMs which were displaying the Safely Remove Hardware option to users logged in to that server.

These VMs were on Hardware Version 7 with the latest VMware Tools installed. Both of the above hardware components were using two of the HW7 only drivers, PVSCSI and VMXNET3.  The below KB article explains how to disable the HotPlug capability so that the ‘Safely Remove Hardware’ message no longer appears in the system tray.

KB article 1012225

This worked successfully on Windows Server 2003 VMs. On Windows Server 2008 VMs the message was only appearing when logged in with administrative privileges, again the KB article changes worked. (Actually they didn’t on 2008 – see below update)

One curiosity with the KB article was that it stated this change would not work with the VMXNET3 driver, however in both Windows 2003 and 2008 it was successful. Looking back at the KB today to make this post the article seems to have been updated (July 19th) to remove this caveat and now only states the more vague:

“Note: You can disable the HotPlug capability for PCI devices such as e1000 or vmxnet2 NICs”

Update: Actually the devices.hotplug = “false” suggestion in the KB article does not seem to work on Windows Server 2008 or 2008 R2.

Even with the above setting applied the below is still available from ‘Safely Remove Hardware’ in Windows Server 2008 and 2008 R2.

I thought I would test out and see what happens if you try to eject either device. Attempting to remove the disk results in the below, i.e. it won’t actually let you do it.

However, ejecting the network card sees no warning or blocking prompts and the network is consequently disconnected.

The network card is now no longer available to be removed.

Looking at the configuration of the VM you will see that the network adapater has been removed.

However, it is possible to add it back whilst powered on and the drivers are installed.

I’ll post a further update after my support ticket with VMware is answered.

Update: 20/08/2010

VMware Support acknowledged the issue and have provided a workaround for vSphere 4.0 and said that is was already fixed in 4.1. The (long!) workaround for 4.0 is listed below and is VM specific, i.e. the change would need to be made manually on any affected VM. I have successfully tested this workaround on both Windows Server 2008 and 2008 R2. They indicated that the KB article would be updated to reflect this, but so far no change to it – this was about three weeks ago, not sure how long these changes take to filter through.

vSphere 4.0 Workaround Begin

Verifying the ID of the NIC device you want to remove from being shown as a Removable Device

1. Click Start > Run and type devmgmt.msc to launch Device Manager.
2. Expand the appropriate device category (for example, Network Adapters or Storage Controllers).
3. Right-click the device and go to Properties, then click the Details tab.
4. In Windows XP and Server 2003, the Device Instance ID is selected by default. In Windows 7 and Server 2008, select Device Instance Path from the Property drop-down list.

The ID/Path looks similar to PCI\VEN_1022&DEV_2000&SUBSYS_20001022&REV_10\4&47B7341&0&1088.

Note: The Device ID/Path may be different, depending on the device type of the virtual machine

5. Highlight the instance ID and copy it to a notepad as you need it later in this section.

Modifying the Windows registry to hide the device from the Windows Removable Devices

1. Click Start > Run and type regedit to launch Registry Editor.
2. Navigate to HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Enum.
3. Use the sections of the ID to locate the device. For example: PCI, then VEN_1022&DEV_2000&SUBSYS_20001022&REV_10 then 4&47B7341&0&1088.
4. By default, you likely do not have permission to modify the value, so right-click on the key (folder), go to Permissions and give your account Full Control.
5. Inside that key is a value named Capabilities, which has a value of 4 or 6 or higher.
6. Right-click the value and click Modify, then subtract 4 from the value and click OK. For example: 6 to 2, 4 to 0, etc.

Note: It may take Windows a moment to hide the NIC from the Safely Remove Hardware list.

In Windows 2008, the local Administrator account does not have write permissions in the section of registry this workaround edits. To workaround this, the user level can be temporarily raised to run as the Local System and the appropriate permissions then applied to the registry. To do this:

1. Click Start > Run and type cmd to launch Command Prompt.
2. Take note of the current time. In the command window, enter the following (where 16:30 is replaced with a time one or two minutes ahead of the current time):
at 16:30 /interactive cmd.exe
3. This spawns a cmd window that is running as the Local System account. Type regedit.exe to open Regedit as Local System.
4. Navigate to the necessary key, apply Full Control permissions to the local Administrator account.
5. Close Regedit and the cmd window.

Jonathan’s Note: There is an easier way to achieve running regedit with local system permissions using the sysinternals tool psexec.

psexec -i -s regedit

Exporting and editing the Windows registry keys

1. In the Windows Registry, click File > Export and save the key on your computer.
2. Open the key with a text editor and remove all the extra subkeys and values that you do not want to change. In the end it should look similar to:

Windows Registry Editor Version 5.00
[HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Enum\PCI\VEN_1022&DEV_2000&SUBSYS_20001022&REV_10\4&47b7341&0&1088]
“Capabilities”=dword:00000002

Creating a Windows batch script to make the registry changes

1. Open Notepad.
2. Enter the following line (update the path for where you saved the file and what you named it):

regedit /s “C:\Startup\vmnicregedit.reg”

3. Save the file with a .bat extension.

Creating a local policy on the system to run the batch script at start up

1. Click Start > Run and type gpedit.msc.
2. Expand Computer Configuration.
3. Expand Windows Settings.
4. Select Scripts (Startup/Shutdown) in the left window pane.
5. Double-click Startup in the right window pane.
6. Click Add.
7. Click Browse and browse to the .bat file created above.

Testing the workaround

After applying all the changes above, reboot the virtual machine. The network adapter should not show up as a removable device.

vSphere 4.0 Workaround End

vSphere 4.1

After I tested the above workaround I then fired up a Windows Server 2008 VM on vSphere 4.1 and without any modifications the issue was still present. This time though when I added the devices.hotplug = “false” configuration to the VM, the Safely Remove Hardware icon was completly removed from the task bar – so the support engineer was correct, it is resolved in 4.1. See below – there is no Safely Remove Hardware icon.

It’s amost worth upgrading to vSphere 4.1 just for that rather than having to follow the lengthy workaround! I did quiz the engineer as to whether we would see a fix in 4.0. He said he hoped so, but it was not guaranteed and would be fed back to engineering, so I guess we wait and see.

Exchange 2010 DAG Support Within Virtualised Clusters

One of the major new features within Exchange 2010 is the Database Availabilty Group (DAG). This replaces High Availability options from previous versions of Exchange such as SCR and CCR – it essentially works by having multiple copies of the same Exchange databases replicated across multiple Exchange servers.

Exchange 2010 is supported on hardware virtualisation platforms provided the conditions in this Technet article are met. Whilst looking at various options for a possible Exchange 2010 deployment for a user base in the hundreds (it obviously made sense to look at what possibilities are available if deciding to virtualise the mailbox server role) I stumbled across this blog post which suggested that whilst Exchange 2010 was supported as virtual, when running a DAG it was not supported if part of a virtualised cluster.

Given that most people looking to deploy Exchange 2010 DAG solutions virtually, may well already have an existing cluster and do not wish to purchase standalone virtual hosts just for this solution it seemed a bit dissapointing to read that Microsoft had taken this stance. I contacted MS support to clarify exactly what was and wasn’t supported.

Whilst going back and forth with the support guy I read this Technet Magazine Article which suggested that the above stance might have changed recently and in fact a DAG would be supported within a virtualised cluster, provided that all virtualisation HA features would be disabled for the Exchange 2010 DAG VMs. This seemed to reflect the below quote from the virtualisation support article

“DAGs are supported in hardware virtualization environments provided that the ….. clustered root servers have been configured to never failover or automatically move mailbox servers that are members of a DAG to another root server.”

Eventually they confirmed that this was correct and whilst it would be supported to deploy a DAG in a virtualised cluster with these features turned off (HA and DRS in VMware ESX) there was a strong emphasis that this would not be recommended. Exactly why it was not recommended was difficult to ascertain, the impression I got from the call was:

• As Aidan mentions in his blog post he speculates that it hasn’t been tested enough to be recommended yet. There was also a lot of emphasis on the call on large scale deployments and not much demand yet for testing smaller deployment scenarios.
• Fixing VMs to particular hosts does mean that the hosts themselves become a management burden, it was mentioned that this could be a high cost.

Whilst it does increase the level of management, purchasing modern physical hardware to run an Exchange 2010 DAG could mean that the servers are not highly utilised for smaller deployments – consequently the hardware cost of doing so can be comparatively high to the cost of a VM.

The main purpose in this deployement  to consider virtualising Exchange was hardware cost, not the HA features it could bring – Exchange itself will bring application HA.

The upshot of this post is essentially that an Exchange 2010 DAG in a virtualised cluster is supported by Microsoft provided the HA features are turned off, but not recommended by them. Consequently you can take that information into your design process and consider if it makes sense for your deployment.

Update:

If running VMware virtualisation, you must also consider that prior to vSphere 4.0 U1 running MSCS clusters within HA/DRS clusters was not supported, this changed with the release of U1. Since an Exchange 2010 DAG relies on Windows Failover Clustering then you must be on at least U1 to be supported – however, again HA / DRS must be disabled for the VMs in question. Virtual Kenneth has some very useful information about this in a blog post.

ESXi 4.0 Slow Boot Times When Hosting Passive MSCS Nodes With RDM LUNs

During the initial stages of an upgrade of a number of VMware hosts from ESX 3.5 U5 to ESXi 4.0 U2 the boot times rose from the normal few mins (most of which is Dell Hardware checks) to around 12 mins.

In particular it was appearing to hang for 5 mins, whilst on the screen the below was displayed:

Loading module multiextent

This would only happen after the install was completed and the host connected back to the fibre channel SAN, otherwise boot times were normal. It was also fine on ESX 3.5 U5 when connected to the SAN.

Some research led me to the below blog post which describes that this can occur when the hosts are part of a cluster which contain Passive MSCS Nodes with RDM LUNs.

http://www.vstable.com/tag/slow/

I made the recommendation to modify the Scsi.UWConflictRetries Advanced Setting to the minimum value of 80 and the boot time dropped to around 5 mins, slighty longer than before, but much better.

Of course you could also make this change in PowerCLI using the below:

Get-VMHost test01 | Set-VMHostAdvancedConfiguration -Name  Scsi.UWConflictRetries -Value 80

Watch out because the name of the Advanced Setting appears to be case sensitive.

Exploring Extended Properties in PowerCLI

I was asked recently via Twitter how to find the CpuFeatureMask property of a VM using PowerCLI. When running a basic

Get-VM Test01

the below properties are outputted to the console:

It is possible to view more properties and values by runnning:

Get-VM Test01 | Format-List *

Unfortunately this still does not reveal the CpuFeatureMask property.  However, if we pipe the Get-VM command through to Get-View we will get back a .NET view object for the VM – by saving this into a variable we can then drill down through the various levels and look for the property we need.

$vm = Get-VM | Get-View
$vm

Below is the top level of information which is returned:

This may look initially like a bewildering amount of info, however if you look at the first few entries you could treat them like categories of information. Since CpuFeatureMask is a configuration property it would seem like a good guess to try looking in the Config category:

You can essentially browse this category by entering the following

$vm.config

and looking down the list of properties returned you will see CpuFeatureMask

On this particular VM CpuFeatureMask is not set, but you get the idea. To retrieve this property for all of your VMs is a simple one liner. With Select-Object we can pick one of the standard properties Name by simply specifiying it; we can use another technique to create our own property with a label and expression for the CpuFeatureMask.

Get-VM | Get-View | Select-Object Name,@{Name="CpuFeatureMask";Expression={$_.config.CpuFeatureMask}}

You could apply a similar technique to other cmdlets like Get-Host or Get-Cluster to retrieve non-standard properties.

Reporting on VMware Update Manager Baselines with PowerCLI

I’ve mentioned on this blog before that I’ve been using VMware Update Manager a lot recently – and wrote about some of my experiences here. Today I was really pleased to see that Carter Shanklin’s team released some cmdlets for PowerCLI to cover Update Manager which had only previously been available back as a beta in the VI Toolkit days.

They arrived just in time because I am currently preparing for a round of ESX patching and I needed to provide a report of hotfixes I was intending to deploy for a particular version of ESX. In the Update Manager GUI I had already created my baseline and scanned it against a host to produce a compliance report of hotfixes we would need to deploy this time.

You can see below that it produces a nice report for me, but I needed to export that information to a format whereby I can give that information to someone else.

One of the new cmdlets is Get-Baseline. I pointed this at my test baseline and with the code below was quickly able to get the information I needed out into a CSV file. I knew from the above report that I just needed to select any patches since 29/12/2009. One of the properties of the patches returned by Get-Baseline is the date it was published so first of all I set a date for which I could query after and stored it in the $BeginDate variable, I then queried the baseline using that date as a starting point.

$BeginDate = (Get-Date).adddays(-65)
Get-Baseline Test | Select-Object -ExpandProperty currentpatches | Where-Object {$_.'releasedate' -gt $BeginDate} | Select-Object Name,IDByVendor,Description,@{n='Product';e={$_.product | Select-Object -expandproperty Version}},ReleaseDate | Export-Csv patches.csv -NoTypeInformation

Which produces an output like this:

You’ll notice that I make use of the ExpandProperty parameter for Select-Object which makes it nice and easy to get to properties which are returned in an array, otherwise although they look fine in the console, when you export them to CSV you will not get what you hope for.

It’s days like today when I’m especially glad I started using PowerShell and very thankful that the vendors of technologies I’m using make this stuff so simple by providing cmdlets for managing their products.

Update:

Initially I tried to use the Get-Compliance cmdlet to find these patches rather than by date, however it only seemed to return a status of Compliant or Not Compliant. Thankfully following a post on the communities it has been pointed out that Get-Compliance has a Detailed parameter which returns a lot more information. Consequently there is no need to mess around with dates, instead you can query for NotCompliantPatches.

$ComplianceStatus = Get-Compliance -Entity 'Server1' -Detailed
$ComplianceStatus.NotCompliantPatches | Select-Object Name,IDByVendor,Description,@{n='Product';e={$_.product | Select-Object -expandproperty Version}},ReleaseDate | Export-Csv patches.csv -NoTypeInformation

Bug in Cluster mem.usage.average Statistic in vSphere 4.0 U1

A while back I posted a script on a basic capacity report I run each month to get an overview of CPU and Memory usage in our various clusters. Since upgrading to vSphere 4.0 U1 I noticed some strange behavior in the results for memory, i.e. they came back at pretty close to 0% (typically between 0.05 and 0.06%) for the average memory usage in a cluster which typically were quite heavily used. On investigating further this also appeared the same in the GUI.

Using Get-Stat, mem.usage.average comes back at 0% where it should be a significant value:

The same in the GUI, the other cluster memory stats have significant values, but Memory Usage Average flatlines along the bottom of the graph:

This stat shows normal behaviour against host machines. I reported it to VMware who confirmed it to be a bug in 4.0 U1.

Thought I would share in case anyone else spends some time scratching their head and checking maths in scripts like I did. I’ll post back when there is a fix.

Update 19/05/2010:

Just got a follow up email from VMware Support that this bug will be fixed in VC 4.0 U3 – no release date for that yet.

New Simple Talk Article – Using VMware vCenter Update Manager to keep your vSphere Hosts Up-To-Date with Patching

I’m pretty obsessive about patching systems whatever the OS; keeping servers up-to-date and consistent across your environment is very important. VMware ship a great tool vCenter Update Manager to help you with this task for ESX / ESXi systems – you can find out more about it in a new article I have written for Simple-Talk.

Slide Deck from Nov 2009 London VMUG

After my PowerCLI session at yesterday’s London VMUG a few people asked me for the content. I believe the content from all sessions will soon be posted to http://www.box.net/londonug, but in the meantime you can get my slides from the below link.

PowerCLI Workshop London VMUG.pptx

Thanks to all who chatted to me afterwards, its always nice to know that someone got something out of a session you put on.

vSphere 4.0 Quick Start Guide – Now Available From Amazon

I recieved a preview copy of the vSphere 4.0 Quick Start Guide a few weeks back from my good friend and PowerCLI expert Alan Renouf . It is a great read and because of its size is really handy fo carrying around and referring to without needing to lug a 700 page book around with you. (Having said that I do currently have Scott Lowe’s Mastering vSphere in my bag at the moment!)

Alan is a contributor to the book and has provided examples of PowerCLI throughout to accompany various sections.

The final release of the book is now available from Amazon.com.

Extending the System Disk in Windows Server 2008

As a system administrator it is often not the latest and greatest big new features of a new operating system which you end up finding the most useful, sometimes its the small improvements which really make your life easier. I’ve been spending a lot more time with Windows Server 2008 recently (OK, I know R2 is out, but it takes time for large organisations to move away from older OS’s and applications.)

Obviously there are great new headline features with things like Server Core and Read-Only Domain Controllers, but I have found a new feature I love which is the abilty to extend a system disk without having to go through some kind of convoluted process. It is particularly easy if it is a VM and you have enough space on the existing datastore to extend the VM’s disk without requesting more storage from your SAN team.

In VSphere 4.0 it even lets me extend the size of the vmdk with the server powered on, I don’t remember that being possible in 3.5, but I might be wrong. The big advantage for this for me is the fact that you do not need downtime to carry out this whole process, either extending the vmdk or the system volume!

Update: I just read in Mastering vSphere by Scott Lowe that the above is a new feature in vSphere 4.0 and the VM is required to be hardware version 7.

Take the following steps to carry it out:

1) Edit the settings of the 2008 VM and increase the size of the disk.

2) Within the Disk Management MMC, kick off a ‘Rescan Disks’

3) You will now see the additional space available at the end of the current disk.

4) Right-click the C: drive and choose ‘Extend Volume’

5) Follow the wizard through to add the additonal space. On completion your system disk will now have the additional space added – all with no downtime!

For completeness there are a number of ways you could do this in Window Server 2003, the way I have typically used is the following. (Make sure you have a good backup of this VM before going ahead)

1) Power down the VM. Edit the settings of the VM to grow the VMDK. Do not power on the VM!

2) Find another Windows Server 2003 VM which is Powered On and has access to the same datastores. Edit the settings of the VM, add an additional hard disk, select the option to use an existing virtual disk, browse to the initial VM’s vmdk and select it.

3) In Disk Management Rescan the disks and you will see an additional drive with unallocated space available other than your current drives.

4) Use the command line utility diskpart to extend this drive.

• Enter diskpart to start
• List Volume will show available volumes on the machine
• Check the additional volume has not been listed as a System partition
• Select that volume with Select Volume volumenumber
• Enter Extend to increase the size of the drive with all available space
• List Volume will let you check it was successful
• Exit to finish

Go back into Disk Management and you should be able to see a larger drive.

5) Edit the settings of the VM and remove the additional disk – make sure you do not select the option to delete the files from disk!

6) Power the initial VM back on. After logging back in the first time you will most likely be prompted to restart the server. Once logged back in, check Disk Management to see the larger system disk.