Tag Archives: esxi

Issue with Nested ESXi and Multiple VMkernel Ports

While working with Nested ESXi in my lab, I had an issue where I could communicate with the IP address on vmk0, but after adding multiple additional VMkernel Ports could not communicate with any of the additional IP addresses. It’s a simple network for testing, everything on the same subnet and no VLANs involved.

I hadn’t done too much reading on the subject before, other than knowing I needed to implement Promiscuous Mode for the Port Group on the physical ESXi servers. It seemed strange that I could communicate with one of the addresses, but not the rest. I tracked down the following posts, but both suggested that only Promiscuous Mode need be enabled.

http://blog.paulregan.co.uk/2013/04/nested-esxi-install-second-vmkernel.html

http://vinf.net/2009/06/12/vsphere-esxi-as-a-vm-vmkernel-traffic-not-working/

I was running a Distributed Switch on the physical ESXi servers, so I tested moving one of the VMkernel ports to a Standard Switch with Promiscuous Mode enabled on the Port Group. It worked fine there, so was naturally curious why.

This communitites posting showed that Forged Transmits also needed to be enabled. The difference between the Standard and Distributed switches is that Forged Transmits is Accepted by default on a Standard switch

NestedESXiVMkernel01

and Rejected by default on a Distributed switch

NestedESXiVMkernel02

hence my experience above.

For more information check out these two posts from William Lam and Chris Wahl who are about two years ahead of me on this 😉

http://www.virtuallyghetto.com/2013/11/why-is-promiscuous-mode-forged.html

http://wahlnetwork.com/2013/04/29/how-the-vmware-forged-transmits-security-policy-works/

 

Enabling NFS VAAI Support in Synology 5.1

Synology enabled VAAI support for NFS in version 5.1 of their DSM software. In order to take advantage of this technology from ESXi hosts we need to do two things:

  • Upgrade DSM to at least version 5.1-5004 (2014/11/06)
  • Install the Synology NFS Plug-in for VMware VAAI

DSM

DSM can be upgraded from within the Control Panel application. Head to the Update & Restore section, check for and install updates. This will likely require a reboot so ensure anything or anyone using it is shutdown or notified.

NFSNAAI07

 

ESXi

Prior to installing the NFS plugin my two NFS datastores don’t have the Hardware Acceleration support.

NFSNAAI02

 

From the 5.1-5004 Release Notes:

VMware VAAI NAS
Added NFS support for two primitives: Full File Clone and Reserve Space.
Please note that you should install the Synology NFS Plug-in for VMware VAAI and read the instructions in README.txt to make sure installation is successful.

Once the plugin has been downloaded it is possible to use either VMware Update Manager or esxcli to install the vib. For the purposes of my home lab without Update Manager I’m going to show you the esxcli way.

Upload the vib to a datastore all hosts can access, then the command to install the vib is:

esxcli software vib install -v /path to vib/esx-nfsplugin.vib

Once installed, the ESXi host will require a reboot

NFSNAAI04

After the reboot you can check all was successful by running:

esxcli software vib list | grep nfs

NFSNAAI05

and examining the NFS datastores

NFSNAAI06

Synology DSM: Enabling iSCSI Multiple Sessions

During setting up an iSCSI LUN on my new Synology box (DSM 4.3), I encountered an issue where the first ESXi box would connect successfully to the iSCSI target, but not the second. So no devices showing on the second node after configuring the iSCSI adapter, despite no apparent errors during the config process.

SyniSCSI00

Looking at the Synology box showed the first ESXi node connected (172.20.0.210), but not the second.

SyniSCSI01

I noticed on the above screen Multiple Sessions – Disabled. I hadn’t seen any mention of that while setting up the iSCSI LUN so looked into the config.

SyniSCSI02

So it appears that multiple sessions are disabled by default, accompanied by a warning that your OS must be cluster aware. Comfortable that ESXi meets that requirement, I decided to proceed.

SyniSCSI03

Multiple sessions are now available.

SyniSCSI04

Rescanning the iSCSI hba and the device now appears.

SyniSCSI05

Now the Synology box shows multiple sessions connected.

SyniSCSI06

 

Problems With VM After Failed Snapshot-Based Backup – Unable to Access File Since It Is Locked

Whatever backup solution you use to backup your virtual infrastructure with, you may sometimes end up with VM snapshots that need to be cleaned up. After a backup failure alert, I use the following PowerCLI one-liners to quickly identify and remove snapshots left behind (by say Netapp SMVI).

Get-VM | Get-Snapshot | Where-Object {$_.Name -like 'smvi*'} | ft VM,Name,Created -AutoSize

Get-VM | Get-Snapshot | Where-Object {$_.Name -like 'smvi*'} | Remove-Snapshot -RunAsync -Confirm:$false

Recently I had an instance where post a backup failure the snapshot failed to remove with the error Unable to communicate with the remote host, since it is disconnected.

SnapshotIssue01

From what I could ascertain, following the backup failure the VM had been knocked offline and marked as (Invalid) in vCenter registration. So step 1 was to remove the VM from vCenter and re-add back to the inventory via the right-clicking on the VMX file. Once back in vCenter I was able to remove the snapshot.

However, the VM was then marked as Virtual machine disks consolidation is needed.SnapshotIssue02Taking the Consolidate option from the Snapshot menu however failed with the error Unable to access file since it is locked.

SnapshotIssue03

There are loads of posts out there on how this can happen if you are using a backup solution that mounts vmdks into the backup appliance and that removing the disk from the backup appliance and retrying should resolve the issue, including this VMware KB article.

I’m not sure if Netapp SMVI even works that way, but I didn’t have any additional disks on the VM used for that purpose. I powered off the SMVI VM anyway, tried the consolidate again, but still no luck.

The VM was now registered on a different host than it had been during the backup failure, but the file lock was still present. I decided to place the original host in maintenance mode in case that was still maintaining a lock on the file. The host failed to get into maintenance mode and hung at 83% for ages after migrating all VMs off. A restart of the ESXi management agents resolved this and I was then able to place the host and maintenance mode and restarted it for good measure.

Following the host restart and with the SMVI VM still powered off, the consolidate was then successful.

Storage vMotion Fails with VM Hardware Version 4

Having recently enabled Storage DRS in a vSphere 5.1 environment we began to see a lot of the following errors in vCenter:

The device or operation specified at index ‘x’ is not supported for the current virtual machine version ‘vmx-04’. A minimum version of ‘vmx-06’ is required for this operation to succeed

svMotionSDRS

The host(s) running the VM(s) in question contained the error matched in this VMware KB article:

[2009-07-10 14:13:41.632 F638BB90 info ‘vm:/vmfs/volumes/4a56e6c2-9319e3df-f1af-001e0bea4030/RVHOLS029/RVHOLS029.vmx’] Upgrade is required for virtual machine, version: 4

The VMs in question were all quite old, were still running VM Hardware Version 4 and required upgrading to a later version before Storage vMotion would move them.

This can sure generate a lot of failure alerts with Storage DRS turned on, a thin-provisioned datastore going over the specified capacity threshold and many VMs attempting to be moved to clear some space back 😉

I’m sure you don’t need me to tell you that keeping VM Hardware Versions, VMware Tools, OS patches etc up-to-date in your environment is one of those necessary maintenance tasks to keep smooth running systems.

You can identify which VMs are on a particular Hardware Version with a simple PowerCLI command:

Get-VM | Where-Object {$_.Version -eq 'v4'} | Sort-Object Name | Format-Table Name,Version -AutoSize

Even better, if you are regularly running the vCheck report then you would have already picked up on this since one of the checks is for VM Hardware Versions prior to a value you have specified 😉

Get-Credential No Longer Prepends Username With a Backslash in PowerShell v3

This one tripped me up earlier in the week, so thought it was worth sharing in case you hit the same issue sometime. In PowerShell v2 and earlier when using Get-Credential to save credentials into a variable and NOT using a full Windows domain credential, e.g. something like:

Get-Credential01

 

instead of a more Windows style credential:

Get-Credential03

 

then the resultant stored credential prepends a \ in front of the username:

$cred = Get-Credential

PS C:\> $cred

UserName Password
-------- --------
\root System.Security.SecureString

PS C:\> $cred.UserName.Length
5

This doesn’t cause any issues when working with the credential object, for instance to connect directly to an ESXi host instead of to vCenter using the credential object with root as the username.

Connect-VIServer ESXi01 -Credential $cred

However, if you need to pick out the username from the credential object and use it for something else this can be a problem. This is what I tripped over this week, not sure how I’ve never hit it before. I was taking out the username to use as part of a connection string for pscp.exe .

$ESXiUsername = $Credential.UserName
$ESXiSSLPath = "[email protected]$($DNSName):/etc/vmware/ssl"

Unfortunately, this made the connection string

\[email protected]:/etc/vmware/ssl

not:

[email protected]:/etc/vmware/ssl

and so the connection failed.

You can get round this by using the TrimStart method, like in the following:

$ESXiUsername = $Credential.UserName.TrimStart('\')

The reason I discovered this was I had written the code in PowerShell v3 and then run it from a v2 box. In v3 Get-Credential no longer prepends the \ when a non-Windows domain credential is used

$cred = Get-Credential

PS C:\> $cred

UserName Password
-------- --------
root System.Security.SecureString

PS C:\> $cred.UserName.Length
4

I hadn’t seen any mention of this with the release and the documentation doesn’t seem to have been updated either,

“If you enter a user name without a domain, Get-Credential inserts a backslash before the name.”

so I filed a Connect item.

Cisco UCS C210 M2 ESXi 5.1 Stuck At ‘Initializing scheduler….’

After upgrading a Cisco UCS C210 M2 rack mount server to ESXi 5.1 and then ESXi patches from 25/07/2013 the host was stuck at ‘Initializing scheduler….’

InitialisingScheduler

I had checked my firmware version was satisfactory for ESXi 5.1

InitialisingScheduler00

but found reports suggesting this (intermittent) issue has been around for a while with earlier versions of ESXi, different versions of UCS models and firmware and maybe HP models too.

Before trying the suggested workaround of disabling legacy USB support, I decided to get the box up to the latest firmware.

To do this in a rack mount which is not managed by UCS Manager, download the firmware from Cisco.com (requires registration) and boot from the ISO

InitialisingScheduler02

 

Existing firmware levels and the level to move to are displayed

InitialisingScheduler03

The firmware packages progress will be updated…..

InitialisingScheduler04

Once complete, restart the server

InitialisingScheduler06

Since applying the update 1.4(3u) I have not seen the issue occur again, yet…….

Make your vmnics blink with ethtool

A colleague of mine demonstrated this for me yesterday while we were troubleshooting which physical NIC out of 8 in a rack mount server matched up to which vminc (0 – 7) in vSphere. Tougher than you might think when vmnic6 and 7 mapped to the onboard physical NICs 1 and 2!

NickBlink

Using the command line tool ethtool you can make each vmnic blink for a specified period of seconds allowing you to identify which port on the back of the server it maps to. So to make vmnic0 and 4 blink for 5 seconds, use the following:

ethtool -p vmnic0 5

ethtool -p vmnic4 5

ethtool

Using PowerCLI to set ESXi Dump Collector Settings

I needed to check the configuration of all hosts in an environment for their ESXi Dump Collector Settings and then ensure they were all set to the correct values. I was using the handy ESXi Dump Collector which ships as part of the vCenter 5.1 package.

ESXi Dump Collector

There are no PowerCLI cmdlets for doing this (yet) so I made my own, Get-VMHostDumpCollector and Set-VMHostDumpCollector. Since you can get and set this item via esxcli I used the Get-ESXCli cmdlet for most of the work. Usage is pretty straightforward. To query the settings for all hosts.

Get-VMHost | Get-VMHostDumpCollector | Format-Table VMHost,HostVNic,NetworkServerIP,NetworkServerPort,Enabled -Auto

To configure all hosts with the same settings

Get-VMHost | Set-VMHostDumpCollector -HostVNic "vmk0" -NetworkServerIP "192.168.0.100" -NetworkServerPort 6500

Here are the functions:

function Get-VMHostDumpCollector {
<#
 .SYNOPSIS
 Function to get the Dump Collector config of a VMHost.

 .DESCRIPTION
 Function to get the Dump Collector config of a VMHost.

 .PARAMETER VMHost
 VMHost to configure Dump Collector settings for.

.INPUTS
 String.
 System.Management.Automation.PSObject.

.OUTPUTS
 System.Management.Automation.PSObject.

.EXAMPLE
 PS> Get-VMHostDumpCollector -VMHost ESXi01

 .EXAMPLE
 PS> Get-VMHost ESXi01,ESXi02 | Get-VMHostDumpCollector
#>
[CmdletBinding()][OutputType('System.Management.Automation.PSObject')]

Param
 (

[parameter(Mandatory=$true,ValueFromPipeline=$true)]
 [ValidateNotNullOrEmpty()]
 [PSObject[]]$VMHost
 )

begin {

 $DumpCollectorObject = @()
 }

 process {

 foreach ($ESXiHost in $VMHost){

try {

if ($ESXiHost.GetType().Name -eq "string"){

 try {
 $ESXiHost = Get-VMHost $ESXiHost -ErrorAction Stop
 }
 catch [Exception]{
 Write-Warning "VMHost $ESXiHost does not exist"
 }
 }

 elseif ($ESXiHost -isnot [VMware.VimAutomation.ViCore.Impl.V1.Inventory.VMHostImpl]){
 Write-Warning "You did not pass a string or a VMHost object"
 Return
 }

# --- Get Dump Collector Config via ESXCli

$ESXCli = Get-EsxCli -VMHost $ESXiHost

$DumpCollector = $ESXCli.System.Coredump.Network.Get()

$hash = @{

 VMHost = $ESXiHost
 HostVNic = $DumpCollector.HostVNic
 NetworkServerIP = $DumpCollector.NetworkServerIP
 NetworkServerPort = $DumpCollector.NetworkServerPort
 Enabled = $DumpCollector.Enabled
 }
 $Object = New-Object PSObject -Property $hash
 $DumpCollectorObject += $Object

}
 catch [Exception]{

 throw "Unable to get Dump Collector config"
 }
 }
 }
 end {

 Write-Output $DumpCollectorObject
 }
}

function Set-VMHostDumpCollector {
<#
 .SYNOPSIS
 Function to set the Dump Collector config of a VMHost.

 .DESCRIPTION
 Function to set the Dump Collector config of a VMHost.

 .PARAMETER VMHost
 VMHost to configure Dump Collector settings for.

.PARAMETER HostVNic
 VNic to use

.PARAMETER NetworkServerIP
 IP of the Dump Collector

.PARAMETER NetworkServerPort
 Port of the Dump Collector

.INPUTS
 String.
 System.Management.Automation.PSObject.

.OUTPUTS
 None.

.EXAMPLE
 PS> Set-VMHostDumpCollector -HostVNic "vmk0" -NetworkServerIP "192.168.0.100" -NetworkServerPort 6500 -VMHost ESXi01

 .EXAMPLE
 PS> Get-VMHost ESXi01,ESXi02 | Set-VMHostDumpCollector -HostVNic "vmk0" -NetworkServerIP "192.168.0.100" -NetworkServerPort 6500
#>
[CmdletBinding()]

Param
 (

[parameter(Mandatory=$true,ValueFromPipeline=$true)]
 [ValidateNotNullOrEmpty()]
 [PSObject[]]$VMHost,

[parameter(Mandatory=$true,ValueFromPipeline=$false)]
 [ValidateNotNullOrEmpty()]
 [String]$HostVNic,

[parameter(Mandatory=$true,ValueFromPipeline=$false)]
 [ValidateNotNullOrEmpty()]
 [String]$NetworkServerIP,

[parameter(Mandatory=$true,ValueFromPipeline=$false)]
 [ValidateNotNullOrEmpty()]
 [int]$NetworkServerPort
 )

begin {

 }

 process {

 foreach ($ESXiHost in $VMHost){

try {

if ($ESXiHost.GetType().Name -eq "string"){

 try {
 $ESXiHost = Get-VMHost $ESXiHost -ErrorAction Stop
 }
 catch [Exception]{
 Write-Warning "VMHost $ESXiHost does not exist"
 }
 }

 elseif ($ESXiHost -isnot [VMware.VimAutomation.ViCore.Impl.V1.Inventory.VMHostImpl]){
 Write-Warning "You did not pass a string or a VMHost object"
 Return
 }

# --- Set the Dump Collector config via ESXCli
 Write-Verbose "Setting Dump Collector config for $ESXiHost"
 $ESXCli = Get-EsxCli -VMHost $ESXiHost

$ESXCli.System.Coredump.Network.Set($null, $HostVNic, $NetworkServerIP, $NetworkServerPort) | Out-Null
 $ESXCli.System.Coredump.Network.Set($true) | Out-Null

Write-Verbose "Successfully Set Dump Collector config for $ESXiHost"
 }
 catch [Exception]{

 throw "Unable to set Dump Collector config"
 }
 }
 }
 end {

 }
}

Including the HP Offline Bundle as part of an upgrade to ESXi 4.1 U2

If you’re running your vSphere deployment on HP kit then there’s a pretty good chance you use the HP Customized ISO Image for installation, for example this one for ESXi 4.1 U1. These customised images typically contain HP management tools and drivers and are great for saving time during the installation process. Naturally you will be upgrading ESXi at some point, but it’s important that you also keep the HP part up-to-date too. To accompany the release of ESXi 4.1 U2 there is a corresponding release of the HP Offline Bundle. The release notes for this version do not suggest any enhancements or bug fixes, only that it is the version that is supported with 4.1 U2.

Once you have downloaded the HP Offline Bundle, it is possible to deploy it via a number of methods. The installation notes suggest installing with the vihostupdate utility, however in this post I’m going to show how it can be installed via vCenter Update Manager.

1) First of all the HP Offline Bundle needs to be imported into the Update Manager Patch Repository. Navigate to the Patch Repositoy and select Import Patches.

2. Enter the path to the HP Offline Bundle. (Note: the GUI does not display the full path, just the filename)

3. The patches will then be uploaded to the Patch Repository.

4. Finally, finish off the wizard.

5. Once the HP Offline Bundle has been imported a Host Extension Patch Baseline needs to be created. Navigate to Baselines and Groups and create a new Baseline, with the type changed from the default Host Patch to Host Extension.

6. Select the Extension to add to the Baseline. The best thing to do here is enter HP into the search box to reduce the number of items to choose from.

7. On the final page confirm the HP Offline Bundle has been added.

Alternatively, once the HP Offline Bundle has been uploaded into the Patch Repository, it’s much easier to create this baseline via the vCenter Update Manger PowerCLI cmdlets. 🙂  Note that we need to use the Extension parameter since the baseline will be of type Host Extension.

New-PatchBaseline -Name "HP Offline Bundle for ESXi 4.1 U2 " -Description "HP Offline Bundle for ESXi 4.1 U2" -IncludePatch (Get-Patch -SearchPhrase "HP ESXi 4.1 Bundle 1.2-25") -TargetType Host -Static -Extension

The main advantage for me of deploying the HP Offline Bundle with Update Manager is that we can take advantage of Baseline Groups. As the name suggests Baseline Groups enable you to group together multiple and different types of baselines. Consequently in this instance we can place the ESXi and HP Offline Bundle upgrade into a Baseline Group and carry out the upgrade to 4.1 U2 in a single remediation task.

There are similar upgrade packages for ESXi 5 from HP, so this process could also be used for that upgrade.