Jonathan Medd's Blog

Tivoli Monitoring, WMI and Server Buffers Full

If you run Tivoli Monitoring 6.2 to monitor Windows Server systems and use other applications to query WMI, e.g. PowerShell and Get-WmiObject, then you may receive the error ‘Server buffers are full and data cannot be accepted’.

Restarting the WMI service will temporaily clear it, but the issue is liable to come back again. This can occur because of a file handle leak in the ITM Windows OS agent when collecting “Processor Information” attribute group.

There is a fix for this issue available from the below website.

http://www-01.ibm.com/support/docview.wss?rs=2292&context=SSRM2J&dc=DB550&uid=swg1IZ51505&loc=en_US&cs=UTF-8&lang=en&rss=ct2292tivoli

Safely Remove Hardware Appears on HW v7 VMs

Recently I noticed an issue with some Citrix VMs which were displaying the Safely Remove Hardware option to users logged in to that server.

These VMs were on Hardware Version 7 with the latest VMware Tools installed. Both of the above hardware components were using two of the HW7 only drivers, PVSCSI and VMXNET3.  The below KB article explains how to disable the HotPlug capability so that the ‘Safely Remove Hardware’ message no longer appears in the system tray.

KB article 1012225

This worked successfully on Windows Server 2003 VMs. On Windows Server 2008 VMs the message was only appearing when logged in with administrative privileges, again the KB article changes worked. (Actually they didn’t on 2008 – see below update)

One curiosity with the KB article was that it stated this change would not work with the VMXNET3 driver, however in both Windows 2003 and 2008 it was successful. Looking back at the KB today to make this post the article seems to have been updated (July 19th) to remove this caveat and now only states the more vague:

“Note: You can disable the HotPlug capability for PCI devices such as e1000 or vmxnet2 NICs”

Update: Actually the devices.hotplug = “false” suggestion in the KB article does not seem to work on Windows Server 2008 or 2008 R2.

Even with the above setting applied the below is still available from ‘Safely Remove Hardware’ in Windows Server 2008 and 2008 R2.

I thought I would test out and see what happens if you try to eject either device. Attempting to remove the disk results in the below, i.e. it won’t actually let you do it.

However, ejecting the network card sees no warning or blocking prompts and the network is consequently disconnected.

The network card is now no longer available to be removed.

Looking at the configuration of the VM you will see that the network adapater has been removed.

However, it is possible to add it back whilst powered on and the drivers are installed.

I’ll post a further update after my support ticket with VMware is answered.

Update: 20/08/2010

VMware Support acknowledged the issue and have provided a workaround for vSphere 4.0 and said that is was already fixed in 4.1. The (long!) workaround for 4.0 is listed below and is VM specific, i.e. the change would need to be made manually on any affected VM. I have successfully tested this workaround on both Windows Server 2008 and 2008 R2. They indicated that the KB article would be updated to reflect this, but so far no change to it – this was about three weeks ago, not sure how long these changes take to filter through.

vSphere 4.0 Workaround Begin

Verifying the ID of the NIC device you want to remove from being shown as a Removable Device

1. Click Start > Run and type devmgmt.msc to launch Device Manager.
2. Expand the appropriate device category (for example, Network Adapters or Storage Controllers).
3. Right-click the device and go to Properties, then click the Details tab.
4. In Windows XP and Server 2003, the Device Instance ID is selected by default. In Windows 7 and Server 2008, select Device Instance Path from the Property drop-down list.

The ID/Path looks similar to PCI\VEN_1022&DEV_2000&SUBSYS_20001022&REV_10\4&47B7341&0&1088.

Note: The Device ID/Path may be different, depending on the device type of the virtual machine

5. Highlight the instance ID and copy it to a notepad as you need it later in this section.

Modifying the Windows registry to hide the device from the Windows Removable Devices

1. Click Start > Run and type regedit to launch Registry Editor.
2. Navigate to HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Enum.
3. Use the sections of the ID to locate the device. For example: PCI, then VEN_1022&DEV_2000&SUBSYS_20001022&REV_10 then 4&47B7341&0&1088.
4. By default, you likely do not have permission to modify the value, so right-click on the key (folder), go to Permissions and give your account Full Control.
5. Inside that key is a value named Capabilities, which has a value of 4 or 6 or higher.
6. Right-click the value and click Modify, then subtract 4 from the value and click OK. For example: 6 to 2, 4 to 0, etc.

Note: It may take Windows a moment to hide the NIC from the Safely Remove Hardware list.

In Windows 2008, the local Administrator account does not have write permissions in the section of registry this workaround edits. To workaround this, the user level can be temporarily raised to run as the Local System and the appropriate permissions then applied to the registry. To do this:

1. Click Start > Run and type cmd to launch Command Prompt.
2. Take note of the current time. In the command window, enter the following (where 16:30 is replaced with a time one or two minutes ahead of the current time):
at 16:30 /interactive cmd.exe
3. This spawns a cmd window that is running as the Local System account. Type regedit.exe to open Regedit as Local System.
4. Navigate to the necessary key, apply Full Control permissions to the local Administrator account.
5. Close Regedit and the cmd window.

Jonathan’s Note: There is an easier way to achieve running regedit with local system permissions using the sysinternals tool psexec.

psexec -i -s regedit

Exporting and editing the Windows registry keys

1. In the Windows Registry, click File > Export and save the key on your computer.
2. Open the key with a text editor and remove all the extra subkeys and values that you do not want to change. In the end it should look similar to:

Windows Registry Editor Version 5.00
[HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Enum\PCI\VEN_1022&DEV_2000&SUBSYS_20001022&REV_10\4&47b7341&0&1088]
“Capabilities”=dword:00000002

Creating a Windows batch script to make the registry changes

1. Open Notepad.
2. Enter the following line (update the path for where you saved the file and what you named it):

regedit /s “C:\Startup\vmnicregedit.reg”

3. Save the file with a .bat extension.

Creating a local policy on the system to run the batch script at start up

1. Click Start > Run and type gpedit.msc.
2. Expand Computer Configuration.
3. Expand Windows Settings.
4. Select Scripts (Startup/Shutdown) in the left window pane.
5. Double-click Startup in the right window pane.
6. Click Add.
7. Click Browse and browse to the .bat file created above.

Testing the workaround

After applying all the changes above, reboot the virtual machine. The network adapter should not show up as a removable device.

vSphere 4.0 Workaround End

vSphere 4.1

After I tested the above workaround I then fired up a Windows Server 2008 VM on vSphere 4.1 and without any modifications the issue was still present. This time though when I added the devices.hotplug = “false” configuration to the VM, the Safely Remove Hardware icon was completly removed from the task bar – so the support engineer was correct, it is resolved in 4.1. See below – there is no Safely Remove Hardware icon.

It’s amost worth upgrading to vSphere 4.1 just for that rather than having to follow the lengthy workaround! I did quiz the engineer as to whether we would see a fix in 4.0. He said he hoped so, but it was not guaranteed and would be fed back to engineering, so I guess we wait and see.

TrainSignal – Exchange 2010 Training DVDs

I was lucky enough recently to receive a review copy of the recently released Exchange Server 2010 Training DVDs from TrainSignal. Not all IT pros are fortunate enough to be able to go out on training courses for everything they want or need to learn about for their job, so I was very interested to see how this form of instructor based DVD training would work.

I had an excellent first impression of this experience because whilst I was waiting for the DVDs to ship from the US I was informed that the same training course was also available for me online so I could start straight away! I thought this was a fantastic idea and a terrific surprise I was not expecting. If you’re like me, then once you decide to get into a topic you want to get on with it as soon as possible, so not having to wait for the DVDs to arrive was great.

The topics are split into 40 different videos and cover everything from the real basics for those who haven’t worked with any version of Exchange before through to the more advanced topics such as high availabilty and disaster recovery. The advantage over on-site course based training I found of having all the videos available was that if there was a section you felt you already knew about or was not particularly relevant for your environment, then it could be skipped through rather than having to sit through at an on-site course. Naturally, if you found you ever did need it then it would be possible to go back to it at a later date.

The videos are a mix of instructor led Powerpoint type slides intermixed with lots of real demonstrations. The instructor takes the course at a good pace, slow enough for new Exchange administrators, but not too slow that experienced admins would get frustrated. I also felt that the instructor had real world experience tips to add, despite Exchange 2010 still being a relatively new product.

The DVD package contains 3 discs. The first two contain the entire course split over the 2 discs and can be watched on any PC or Mac. The instructor notes that you see during the different videos are also available on this DVD as a pdf. Tip: You need to install the TechSmith Screen Capture Codec first to watch the videos, which the DVD helpfully prompts you to do if you don’t have it installed. If like me you missed that on the top of the help page and spent 5 mins trying to install plugins for Firefox then you probably shouldn’t really be left responsible for important IT systems.

The third disc contains the entire course on different formats for loading onto mobile devices. There are video formats in .mv4 for iPod and .wmv for other devices. Also available are MP3s which might be useful if you enjoying learning just by listening or maybe as revision repeats if going for the Exchange 2010 certification.

I found the different available sources of the course fantastic and really helped to get through the material quicker than I would if only one was available. For instance:

• I used the online course at lunchtime whilst at work
• The iPod material whilst commuting
• The DVD or online course again whilst at home

Overall I was really impressed by this set of DVDs and would definitely consider other series in the future. Previously if I was not able to attend a course for something I needed to learn about then typically I would buy a book instead, but in future I think I would be looking to see if Trainsignal had a series for it first.

You can find out more about the other courses they have available at www.trainsignal.com or follow them on Twitter. Currently there are courses for lots of Windows products, VMware and Cisco technologies.

Exchange 2010 DAG Support Within Virtualised Clusters

One of the major new features within Exchange 2010 is the Database Availabilty Group (DAG). This replaces High Availability options from previous versions of Exchange such as SCR and CCR – it essentially works by having multiple copies of the same Exchange databases replicated across multiple Exchange servers.

Exchange 2010 is supported on hardware virtualisation platforms provided the conditions in this Technet article are met. Whilst looking at various options for a possible Exchange 2010 deployment for a user base in the hundreds (it obviously made sense to look at what possibilities are available if deciding to virtualise the mailbox server role) I stumbled across this blog post which suggested that whilst Exchange 2010 was supported as virtual, when running a DAG it was not supported if part of a virtualised cluster.

Given that most people looking to deploy Exchange 2010 DAG solutions virtually, may well already have an existing cluster and do not wish to purchase standalone virtual hosts just for this solution it seemed a bit dissapointing to read that Microsoft had taken this stance. I contacted MS support to clarify exactly what was and wasn’t supported.

Whilst going back and forth with the support guy I read this Technet Magazine Article which suggested that the above stance might have changed recently and in fact a DAG would be supported within a virtualised cluster, provided that all virtualisation HA features would be disabled for the Exchange 2010 DAG VMs. This seemed to reflect the below quote from the virtualisation support article

“DAGs are supported in hardware virtualization environments provided that the ….. clustered root servers have been configured to never failover or automatically move mailbox servers that are members of a DAG to another root server.”

Eventually they confirmed that this was correct and whilst it would be supported to deploy a DAG in a virtualised cluster with these features turned off (HA and DRS in VMware ESX) there was a strong emphasis that this would not be recommended. Exactly why it was not recommended was difficult to ascertain, the impression I got from the call was:

• As Aidan mentions in his blog post he speculates that it hasn’t been tested enough to be recommended yet. There was also a lot of emphasis on the call on large scale deployments and not much demand yet for testing smaller deployment scenarios.
• Fixing VMs to particular hosts does mean that the hosts themselves become a management burden, it was mentioned that this could be a high cost.

Whilst it does increase the level of management, purchasing modern physical hardware to run an Exchange 2010 DAG could mean that the servers are not highly utilised for smaller deployments – consequently the hardware cost of doing so can be comparatively high to the cost of a VM.

The main purpose in this deployement  to consider virtualising Exchange was hardware cost, not the HA features it could bring – Exchange itself will bring application HA.

The upshot of this post is essentially that an Exchange 2010 DAG in a virtualised cluster is supported by Microsoft provided the HA features are turned off, but not recommended by them. Consequently you can take that information into your design process and consider if it makes sense for your deployment.

Update:

If running VMware virtualisation, you must also consider that prior to vSphere 4.0 U1 running MSCS clusters within HA/DRS clusters was not supported, this changed with the release of U1. Since an Exchange 2010 DAG relies on Windows Failover Clustering then you must be on at least U1 to be supported – however, again HA / DRS must be disabled for the VMs in question. Virtual Kenneth has some very useful information about this in a blog post.

PowerShell 2.0: One Cmdlet at a Time #107 Add-Type

Continuing the series looking at new cmdlets available in PowerShell 2.0. This time we look at the Add-Type cmdlet.

What can I do with it?

Imbed code from modern programming languages into your PowerShell session or scripts. The list of valid languages are: C#, C# 3.0, VisualBasic and JScript – C# is the default. Use the Language parameter to specify one if it is not C#.

Example:

Within a PowerShell session use some C# code to create a TakeAway class and create a static method Minus. Use the Add-Type cmdlet to add the class to the session and then call the TakeAway class and Minus static method.

$csharp = @"
public class TakeAway
{
    public static int Minus(int a, int b)
    {
        return (a - b);
    }
}
"@
Add-Type -TypeDefinition $csharp
[TakeAway]::Minus(10,7)

You will see that we get the expected answer of 3:

How could I have done this in PowerShell 1.0?

PowerShell 1.0 did not support adding C# or other code into PowerShell scripts, you could however have created your own cmdlet which I’m sure would have been very straightforward for most sysadmins

1000 things 1% better!

PowerShell 2.0: One Cmdlet at a Time #106 Import-LocalizedData

Continuing the series looking at new cmdlets available in PowerShell 2.0. This time we look at the Import-LocalizedData cmdlet.

What can I do with it?

Enable text in scripts displayed to users to be presented in their own language. The cmdlet uses the automatic variable $PSUICulture to determine the language to use and alternate text is stored within .psd1 files in subdirectories of the folder that the script is stored.

Example:

In a script called RegionalTest.ps1 use the ConvertFrom-StringData cmdlet to create a series of text messages to display to the user. Import-LocalizedData will retrieve the value of the $PSUICulture automatic variable, get the contents of the RegionalTest.psd1 file in the es-ES directory (assume the user is Spanish) and store the data within the variable designated by the BindingVariable parameter. Then display the Welcome text.

$UserMessages = Data {
    # culture="en-US"
    ConvertFrom-StringData @'
    Welcome = Welcome to the application
    Error1 = You have entered an incorrect username
    Error2 = You have entered an incorrect password
'@
    }

Import-LocalizedData -BindingVariable $UserMessages

$UserMessages.Welcome

The contents of the RegionalTest.psd1 file for Spanish would look like (apologies for any bad translation!)

ConvertFrom-StringData @'
    Welcome = Recepción al uso
    Error1 = Usted ha incorporado un username incorrecto
    Error2 = Usted ha incorporado una contraseña incorrecta
'@

and be stored in the es-ES folder below C:\Scripts where RegionalTest.ps1 lives

When run on the Spanish user’s machine the Spanish text would be displayed rather than the original English.

How could I have done this in PowerShell 1.0?

Script Internationalisation features were introduced in PowerShell 2.0 and not supported in version 1.0 – more info here.

1000 things 1% better!

ESXi 4.0 Slow Boot Times When Hosting Passive MSCS Nodes With RDM LUNs

During the initial stages of an upgrade of a number of VMware hosts from ESX 3.5 U5 to ESXi 4.0 U2 the boot times rose from the normal few mins (most of which is Dell Hardware checks) to around 12 mins.

In particular it was appearing to hang for 5 mins, whilst on the screen the below was displayed:

Loading module multiextent

This would only happen after the install was completed and the host connected back to the fibre channel SAN, otherwise boot times were normal. It was also fine on ESX 3.5 U5 when connected to the SAN.

Some research led me to the below blog post which describes that this can occur when the hosts are part of a cluster which contain Passive MSCS Nodes with RDM LUNs.

http://www.vstable.com/tag/slow/

I made the recommendation to modify the Scsi.UWConflictRetries Advanced Setting to the minimum value of 80 and the boot time dropped to around 5 mins, slighty longer than before, but much better.

Of course you could also make this change in PowerCLI using the below:

Get-VMHost test01 | Set-VMHostAdvancedConfiguration -Name  Scsi.UWConflictRetries -Value 80

Watch out because the name of the Advanced Setting appears to be case sensitive.

PowerShell 2.0: One Cmdlet at a Time #105 Set-StrictMode

Continuing the series looking at new cmdlets available in PowerShell 2.0. This time we look at the Set-StrictMode cmdlet.

What can I do with it?

Configure strict mode for the current scope. An error will be generated when the content of an expression, script or script block violates coding rules. Note: it is possible to use the Version parameter to pick which coding rules to use. The PowerShell help lists the current possible options as:

1.0

– Prohibits references to uninitialized variables, except for uninitialized variables in strings.

2.0

– Prohibits references to uninitialized variables (including uninitialized variables in strings).

– Prohibits references to non-existent properties of an object.

– Prohibits function calls that use the syntax for calling methods.

– Prohibits a variable without a name (${}).

Latest

–Selects the latest (most strict) version available. Use this value to assure that scripts use the strictest available version, even when new versions are added to Windows PowerShell.

Example:

Examine what happens when you add the undefined $b to the undefined $a with strict mode off. Next, turn on strict mode using Version 1.0 and run the same test.

$a + $b
Set-StrictMode -Version 1.0
$a + $b

Note the error message generated with strict mode on because $a has not been initialised.

Examine what happens when you define $a to be a numerical value and attempt to reference a property with strict mode off. Next, turn on strict mode using Version 2.0 and run the same test.

$a = 32
$a.Time
Set-StrictMode -Version 2.0
$a.Time

Note the error message generated with strict mode on because the Time property does not exist.

How could I have done this in PowerShell 1.0?

You could have used Set-PSDebug, however Set-Strictmode applies only to the current scope or child scopes and does not impact the global scope. For more information on scopes in PowerShell look here.

1000 things 1% better!

PowerShell 2.0: One Cmdlet at a Time #104 Trace-Command

Continuing the series looking at new cmdlets available in PowerShell 2.0. This time we look at the Trace-Command cmdlet.

What can I do with it?

Begin a trace of a command or expression.

Example:

Examine debug info for Parameter Binding when piping a string through to Get-Service.

Trace-Command -Name ParameterBinding -Option All -Expression {'winmgmt' | Get-Service} -PSHost

You will see it is possible to work through the debug info to find out what is happening:

Note: it is also possible to output the debug info to a file, simply remove the PSHost parameter and use FilePath instead.

Trace-Command -Name ParameterBinding -Option All -Expression {'winmgmt' | Get-Service} -FilePath C:\Debug.txt

The resulting debug info is now easily viewable in Notepad.

How could I have done this in PowerShell 1.0?

You could have used Set-TraceSource , but Trace-Command applies the trace only to the specified command.

1000 things 1% better!

PowerShell 2.0: One Cmdlet at a Time #103 Update-List

Continuing the series looking at new cmdlets available in PowerShell 2.0. This time we look at the Update-List cmdlet.

What can I do with it?

Add, Remove or Replace items from a property value of an object. This cmdlet can only update a property when it supports the IList interface. So far this does not include any of the core Windows PowerShell cmdlets – however it does include some of the cmdlets that ship with Exchange 2007 and later.

Example:

Add additional email addresses to the Test1 user’s mailbox using the Add parameter of Update-List.

Get-Mailbox Test1  | Update-List -Property EmailAddresses -Add admin@contoso.com,webmaster@contoso.com
| Set-Mailbox

How could I have done this in PowerShell 1.0?

Shay Levy has a great blog post on dealing with AD / Mailbox accounts with multi-valued attributes.

1000 things 1% better!