Learning Points From PowerShell Scripting Games Event #3

Event 3 for the PowerShell Scripting Games 2013 has closed, here are a few learning points I picked up on from entries submitted.

1) Filter to the left

Some cmdlets in PowerShell have their own filtering capabilities, which can make queries of large data sets more efficient. However, not all cmdlets do have this capability and you will need to pipe the results to Where-Object instead. It’s always worth checking the help and examples for a cmdlet first to see the best way to filter and if it has an option to do so then use it!

I’ve seen quite a few entries do this

Get-WmiObject Win32_LogicalDisk -ComputerName "localhost" | Where {$_.DriveType -eq 3}

This will get you the right results, but better is to use the filter parameter of Get-WmiObject

Get-WmiObject Win32_LogicalDisk -Filter "DriveType=3"

2) ConvertTo-Html: PreContent and PostContent Parameters

Quite a few entries made nice use of these parameters to add additional content to the HTML output generated. For instance you could add an h2 header before the <table> with PreContent

Get-WmiObject Win32_LogicalDisk -filter "DriveType=3" | Select-Object Name,Size | ConvertTo-Html -PreContent "<h2>Local Fixed Disk Report</h2>" | Out-File Report.html

It’s possible to include PowerShell code as part of these parameters, so to fulfill one of the other requirements to place the date and time at the bottom of the report you could do something like this:

Get-WmiObject Win32_LogicalDisk -filter "DriveType=3" | Select-Object Name,Size | ConvertTo-Html -PreContent "<h2>Local Fixed Disk Report</h2>" -PostContent "<br><hr><p> $(Get-Date)" | Out-File Report.html

Learning Points From PowerShell Scripting Games Event #2

Event 2 for the PowerShell Scripting Games 2013 has closed, here are a few learning points I picked up on from entries submitted.

1) Win32_Processor

This event is a bit of a sneaky one and if you haven’t been affected by the issue before then you may not know it. The particular issue I am referring to  here is that “The number of physical hyperthreading-enabled processors or the number of physical multicore processors is incorrectly reported in Windows Server 2003“. The issue is essentially this:

“Before you apply this hotfix, the WMI classes and the WMI properties exhibit the following behavior.

Win32_ComputerSystem

• The NumberOfLogicalProcessors property is not available.
• The NumberOfProcessors property returns the number of logical processors that are available on the system.

Win32_Processor

• The number of Win32_Processor instances that are returned is equal to the number of logical processors that are available on the system.
• The NumberOfCores property is not available.
• The NumberOfLogicalProcessors property is not available.

After you apply this hotfix, the WMI classes and the WMI properties exhibit the following behavior.

Win32_ComputerSystem

• The NumberOfProcessors property returns the number of physical processors that are available on the system.
• The NumberOfLogicalProcessors property returns the number of logical processors that are available on the system.

Win32_Processor

• The NumberOfLogicalProcessors property returns the number of logical processors on the current instance.
• The NumberOfCores property returns the number of cores on the current instance.
• The number of Win32_Processor instances that are returned is equal to the number of physical processors that are available on the system.

”

This tripped me up once when I received back inconsistent results from a large server estate where I was querying via WMI the number of processors per server. Luckily in that environment I was able to deploy the hotfix everywhere, but you might not be able to and potentially you may have further inconsistencies with Windows 2000 servers that can’t be fixed. This event contains a Windows 2000 server, so you may need to code around it.

The initial thought would be to check the OS version, but then you would have to detect if the hotfix was installed if it was a 2003 box – probably too much work. So what we could do is check for the NumberofCores property on a WMI query for Win32_Processor and if it doesn’t exist calculate the number of cores and sockets via other means. I haven’t tested this on Windows 2003 minus hotfix or 2000 yet, but you should be able to count the number of unique SocketDesignations returned to determine the number of sockets.

$Processors = Get-WmiObject Win32_Processor
if ($NoOfCores = $Processors.NumberOfCores){

$NoOfSockets = ($Processors | Measure-Object).Count
}
else {

$NoOfCores = 'N/A'
 $NoOfSockets = ($Processors | Select-Object SocketDesignation -Unique | Measure-Object).count
}

2) Using Add-Member when creating custom objects

Event 2 requires you to output results and it’s good to see many people doing this by creating their own objects to output. A number of entries use the Add-Member cmdlet to add properties to their custom object. While this is a perfectly valid way to do it and was the technically prescribed way  in PowerShell v1, I prefer a couple of different approaches and in this previous article explain why – mainly around performance. So I would do something like this (note that it does require PowerShell v3)

[pscustomobject] @{
 Name = $_.Name
 NoOfCores = $NoOfCores
 NoOfSockets = $NoOfSockets
#etc........
}

Hope this helps!

Learning Points From PowerShell Scripting Games Event #1

Event 1 for the PowerShell Scripting Games 2013 has closed, here are a few learning points I picked up on from entries submitted.

1) Get-ChildItem -Recurse

When you need to retrieve files from paths with subfolders the Recurse parameter of Get-ChildItem makes this really easy. For instance

Get-ChildItem -Path C:\Application\Log -Filter *.log -Recurse

is a really easy way to return everything below C:\Application\Log. In the specific instance of this event, this is OK because you only have three subfolders, but potentially there could be a lot more and some of them might not be relevant.

So a better way to do this might be to use wildcards in your path. For instance here we know that all of the subfolders that we are interested in contain the string ‘app’ so we could use something like the below:

Get-ChildItem -Path C:\Application\Log\*app*\*.log

Note you can use the * wildcard not only for part of the filename, but also the directory and you can combine multiple wildcards.

2) Copy-Item followed by Remove-Item

The goal of the event is to archive files from the expensive storage to cheaper, archived storage. Some examples used a two-step process to do this with:

Copy-Item.....

Remove-Item.....

(and some did not even include the Remove-Item, so the files are duplicated) No need to do that, you can use Move-Item to make it a one step process.

3) Maintaining the Folder Structure at the Destination

Make sure you read all of the requirements for the event. One of which was to maintain the folder structure at the destination archive. So something like the following will simply end up with all of the log files in one unmanageable folder.

Get-ChildItem -Path C:\Application\Log\*app*\*.log -file | Move-Item -Destination C:\Archive

Since we are not using the Recurse parameter in the initial query, an attempt to move the file will fail because the path does not exist. Instead we can do something similar to the touch command in Unix to first create an empty file, then overwrite it with the file move.

Get-ChildItem -Path C:\Application\Log\*app*\*.log | ForEach-Object {New-Item -ItemType File -Path "C:\Archive\$($_.Directory.Name)\$($_.Name)" -Force; Move-Item $_ -Destination "C:\Archive\$($_.Directory.Name)" -Force}

Looking forward to seeing entries for the next event. Remember you can still join in and there are plenty of prizes to be won!

PowerShell Scripting Games 2013

Looking for a good way to start learning PowerShell or fancy testing yourself with some more advanced problems to solve? It’s time again for the annual Scripting Games and this year the PowerShell community are running the event, ably supported by the Microsoft Scripting Guy.

There are separate Beginner and Advanced tracks and plenty of prizes to be won in each event. I’ll be helping out with the other community judges to highlight some of the entries. The Scripting Games are a great way to get started learning PowerShell, in fact it was one of the main resources I used when I first started.

Full details on how to enter and other information here.

Downloading Single or Entire Ranges of Lego Set Instructions with PowerShell

I’ve always enjoyed Lego and it’s currently experiencing a resurgence in our house thanks to strategic hinting encouragement that my children would find it fun too. (It seems like I’m not the only one)

What does tend to happen though is that as sets are pulled apart played with we often need to dig out the instructions to put entire sets pieces back together again. Although I carefully file any instructions from sets we purchase, often it’s easier to download them as pdfs from the Lego website and view them on a tablet and sometimes we buy them 2nd hand without instructions so need to download them anyway.

The official Lego website is pretty good for making these available, even for sets which were released a long time ago, but I particularly like BrickInstructions.com since they make it very easy to find links on the official Lego site to the particular instructions you need and also display the instructions on their own site.

As our collection of sets grew I was manually downloading a copy of the instructions for each one to store for quick reference, then I figured why not just download the whole lot for the particular ranges we are interested in, so that I’ve always got any of them. Naturally I was not going to do that manually……

So the following two PowerShell functions will enable you to download either single, multiple or an entire range of Lego set instructions via the links provided by BrickInstructions.com. The download requests are submitted via a single BITS Transfer job so make good use of available bandwidth, handle network interruptions etc…

Note: Some of the code in these functions requires PowerShell v3 

If you already know the Lego set numbers you can use the first function Get-LegoSetInstructions like this:

9488,7655,7656 | Get-LegoSetInstructions -DestinationFolder "C:\Lego\Instructions\Star Wars" -Verbose

The files hosted on  the Lego website are named with a generic number code, so I change this with each download to the name and number of the set so that they are easy to find once downloaded:

function Get-LegoSetInstructions {
<#
 .SYNOPSIS
 Function to download Lego Set Instructions.

 .DESCRIPTION
 Function to download Lego Set Instructions.

 .PARAMETER ID
 Lego set to download instructions for

.PARAMETER Link
 Link to Lego set instructions page

.PARAMETER DestinationFolder
 Folder to save the instructions in

.PARAMETER Force
 Overwrite an existing instructions file if it already exists

.INPUTS
 String

.OUTPUTS
 None.

.EXAMPLE
 PS> Get-LegoSetInstructions -ID 9488 -DestinationFolder "C:\Lego\Instructions"

.EXAMPLE
 PS> 9488,7655,7656 | Get-LegoSetInstructions -DestinationFolder "C:\Lego\Instructions"

.EXAMPLE
 PS> Get-LegoSetInstructions -Link "http://lego.brickinstructions.com/instructions.php?code=10212&amp;set=Imperial Shuttle" -DestinationFolder "C:\Lego\Instructions"

.NOTES
 Version: 1.0 - First draft
 Date: 17/04/2013
 Tag: lego,download,BITS
#>
[CmdletBinding(DefaultParametersetName="ID")]

Param (

 [parameter(Mandatory=$true,ValueFromPipeline=$true,ParameterSetName="ID")]
 [ValidateNotNullOrEmpty()]
 [String[]]$ID,

[parameter(Mandatory=$true,ValueFromPipelineByPropertyName=$true,ParameterSetName="Link")]
 [ValidateNotNullOrEmpty()]
 [String[]]$Link,

 [parameter(Mandatory=$true)]
 [ValidateNotNullOrEmpty()]
 [String]$DestinationFolder,

[parameter(Mandatory=$false)]
 [Switch]$Force
 )

 begin {

$ErrorActionPreference = 'Stop'
 Write-Debug $MyInvocation.MyCommand.Name
 $LegoSetObject = @()

# --- Test for the existence of the BITS transfer module
 if (!(Get-Module BitsTransfer -ListAvailable)){

 throw "BITS Transfer module not available. Run from a system with this module"
 }

# --- Test for the existence of the Destination Folder
 if (!(Test-Path $DestinationFolder)){

 throw "Destination Folder $DestinationFolder does not exist"
 }
 }

process {

switch ($PsCmdlet.ParameterSetName)
 {
 "ID" {

 foreach ($Set in $ID){

 # --- Retrieve the web page for the Lego instructions
 $url = Invoke-WebRequest "http://lego.brickinstructions.com/instructions.php?code=$Set"

# --- Find links that contain pdfs. There may be multiple for different print formats on newer modelss
 if ($PDFLink = $url.Links | Where-Object {$_.outerHTML -like '*pdf*'} | Select-Object -First 1){

$Description = $url.ParsedHtml.nameProp
 $Output = Join-Path -Path $DestinationFolder -Childpath ($Description + ".pdf")

$Object = [pscustomobject]@{

 LegoSet = $Set
 Description = $Description
 DownloadLink = $PDFLink.href
 Output = $Output
 }
 $LegoSetObject += $Object

}
 else {
 Write-Warning "No instructions for Lego Set $Set were found"
 }
 }
 }

"Link" {

foreach ($href in $Link){

 # --- Retrieve the web page for the Lego instructions
 $url = Invoke-WebRequest $href

 # --- Pick out the set number from the link
 $href -match ".*?(?<Set>\d+)" | Out-Null
 $Set = $Matches.Set

# --- Find links that contain pdfs. There may be multiple for different print formats on newer modelss
 if ($PDFLink = $url.Links | Where-Object {$_.outerHTML -like '*pdf*'} | Select-Object -First 1){

$Description = $url.ParsedHtml.nameProp
 $Output = Join-Path -Path $DestinationFolder -Childpath ($Description + ".pdf")

$Object = [pscustomobject]@{

 LegoSet = $Set
 Description = $Description
 DownloadLink = $PDFLink.href
 Output = $Output
 }
 $LegoSetObject += $Object
 }
 else {
 Write-Warning "No instructions for Lego Set $Set were found"
 }
 }
 }
 }
 }

end {
 # --- If no Force parameter, then filter the downloads to those that don't already exist
 if (-not ($PSBoundParameters.ContainsKey('Force'))){

$LegoSetObject = $LegoSetObject | Where-Object {-not (Test-Path $_.Output)}
 }

# ---
 $Job = Start-BitsTransfer -Source $LegoSetObject.DownloadLink -Destination $LegoSetObject.Output -Asynchronous

 Write-Verbose "Checking the status of the BITS transfer download job...."

while (($Job.JobState -eq "Transferring") -or ($Job.JobState -eq "Connecting")){

 Write-Verbose "BITS Transfer job is still downloading...."
 Start-Sleep 10}

switch ($Job.JobState){

"Transferred" {Complete-BitsTransfer -BitsJob $Job}
 "Error" {Write-Warning "BITS transfer has failed..." }
 default {Write-Warning "Please investigate download for Lego Set "}
 }
 Write-Verbose "BITS transfer download job has completed...."

Write-Output $LegoSetObject
 }
}

If you don’t know the set number or want an entire range then you can use the Get-LegoSetLink function to generate links to the pages on the BrickInstructions.com website for all of the Lego sets in that range. The list of possible ranges are on the left-hand side of the site.

So to get the Harry Potter range:

Get-LegoSetLink -Range "Harry Potter"

The Get-LegoSetInstructions function will also take hyperlinks as input, so to get an entire range of instructions downloaded you can do this:

Get-LegoSetLink -Range "Harry Potter" | Get-LegoSetInstructions -DestinationFolder "C:\Lego\Instructions\Harry Potter" -Verbose

function Get-LegoSetLink {
<#
 .SYNOPSIS
 Function to retrieve Lego Set Links from supplied Lego range.

 .DESCRIPTION
 Function to retrieve Lego Set Links from supplied Lego range.

 .PARAMETER Range
 Range of Lego sets to retrieve links for

.INPUTS
 String

.OUTPUTS
 None.

.EXAMPLE
 PS> Get-LegoSetLink -Range "Star Wars"

.NOTES
 Version: 1.0 - First draft
 Date: 18/04/2013
 Tag: lego,link
#>
[CmdletBinding()]

Param (

 [parameter(Mandatory=$true,ValueFromPipeline=$true)]
 [ValidateNotNullOrEmpty()]
 [String]$Range

)

begin {

$ErrorActionPreference = 'Stop'
 Write-Debug $MyInvocation.MyCommand.Name
 $LegoLinkObject = @()
 }

process {

 # --- Retrieve the web page for the Lego instructions
 $EndUrl = ("lego_" + ($Range -replace " ", "_") + ".php").ToLower()
 $url = Invoke-WebRequest "http://lego.brickinstructions.com/$EndUrl"

# --- Find the Lego set links
 if ($LegoSets = $url.links | Where-Object {$_.href -like '*set=*'}){
 foreach ($LegoSet in $LegoSets){

$Object = [pscustomobject]@{

 Description = $LegoSet.innerHTML
 Link = "http://lego.brickinstructions.com/" + $LegoSet.href
 }
 $LegoLinkObject += $Object
 }

# If there are additional pages, e.g. for large ranges, cycle through each page to find all of the sets
 if ($AdditionalPages = $url.links | Where-Object {$_.href -like '*page=*'} | Where-Object {$_.innerhtml -match "\d+"} | Select-Object innerhtml,href -Unique){

foreach ($Page in $AdditionalPages){

$AdditionalPageURL = "http://lego.brickinstructions.com" + $Page.href
 $url = Invoke-WebRequest $AdditionalPageURL

$LegoSets = $url.links | Where-Object {$_.href -like '*set=*'}

foreach ($LegoSet in $LegoSets){

$Object = [pscustomobject]@{

 Description = $LegoSet.innerHTML
 Link = "http://lego.brickinstructions.com/" + $LegoSet.href
 }
 $LegoLinkObject += $Object
 }
 }
 }
 }
 else {

Write-Warning "No sets are available for this range"
 }
 }
 end {
 Write-Output $LegoLinkObject
 }

}

Accio Instructions!

Start-BitsTransfer – Submitting greater than 60 asynchronous jobs generates error

Start-BitsTransfer enables you to download multiple files using Windows’ Background Intelligent Transfer Service , including the ability to have them processed as background jobs.

I encountered an issue when submitting these jobs if the number of files where greater than 60, the 61st and onwards would fail to submit until the existing jobs had been completed or cleared.

Start-BitsTransfer : Object reference not set to an instance of an object.
At line:1 char:1
+ Start-BitsTransfer -Source "http://intranet.server01.local/downlo ...
+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
 + CategoryInfo : NotSpecified: (:) [Start-BitsTransfer], NullReferenceException
 + FullyQualifiedErrorId : System.NullReferenceException,Microsoft.BackgroundIntelligentTransfer.Management.NewBitsTransferCommand

There is a better way to do this anyway since the source and destination paramaters will take multiple files, as long as you match them up in the right order. So you can create a single job with multiple files and so far I have not hit a limit, e.g.

Start-BitsTransfer -Source $MyObject.DownloadLink -Destination $MyObject.Output -Asynchronous

I filed the original issue on Connect since it appears to be a bug and should at least produce a better error message.

https://connect.microsoft.com/PowerShell/feedback/details/785266/start-bitstransfer-submitting-60-asynchronous-jobs-generates-error

Issue with PowerCLI: Not authenticated and session timeout

A colleague of mine experienced this issue recently where after making a PowerCLI connection to a vCenter and instantly running a command such as Get-VM, he would be prompted by the error:

Get-VM. Not authenticated. Possible cause of this error is that the connection was left unused for a while and session has timed out.

Checking he was connected to a vCenter appeared to indicate that he was, i.e.

$global:defaultVIServer

returned a value. Seems like this may be an issue with PowerCLI 5.1 since other similar reports indicate reverting to 4.1 does not have the issue.

We didn’t have that option, so in this instance took the recommendation to amend the PowerCLI timeout as follows, which seemed to help in our case:

Set-PowerCLIConfiguration -WebOperationTimeoutSeconds -1

Writing PowerShell Code on OS X using Sublime

The majority of my PowerShell code is written in a Windows VM where all the typical native PowerShell tools are available. However, occasionally I may want to quickly view or make small changes to some code in OS X. It’s possible via the built-in TextEdit application, but that’s kind of the equivalent of using Notepad on Windows, i.e. a pretty basic experience.

I recently read this review of Text Editors on The Register and decided to check out Sublime after discovering PowerShell could be added as an additional language to support syntax highlighting.

First of all download and install Sublime (available for OS X, Windows and Linux). To add PowerShell syntax highlighting:

1) Download the zip file from https://github.com/SublimeText/PowerShell

2) Unzip and move the contents to Users\Username\Library\Application Support\Sublime Text 2\Packages

Note: I couldn’t track this down at first, the instructions stated to copy to the Packages folder, but not exactly where that would be. Fortunately, there is a menu item which will take you straight there:

3) Restart Sublime and now PowerShell v2 will be listed as an available syntax

Now you can work with PowerShell (and other) code in a nice interface

PowerShell Quick Tip: Accessing the ProgramFiles(x86) Environment Variable

Accessing environment variables in PowerShell is easy, either:

dir env:

to view all of them, or:

dir env:ProgramFiles

to access a specific one. However, if you try that for the ProgramFiles(x86) environment variable you are greeted with the following error:

PS C:\> dir env:ProgramFiles(x86)
x86 : The term 'x86' is not recognized as the name of a cmdlet, function, script file, or operable program. Check the spelling of the name, or if a path was included, verify that the
path is correct and try again.

There are a few ways around this:

1)

dir env:ProgramFiles`(x86`)

2)

dir "env:ProgramFiles(x86)"

3)

${Env:ProgramFiles(x86)}

4)

[Environment]::GetEnvironmentVariable("ProgramFiles(x86)")

Learn Active Directory Management in a Month of Lunches – 50% off promo code

Wanted to let readers know that I have a 50% off promo code for an upcoming book from Richard Siddaway, ‘Learn Active Directory Management in a Month of Lunches’. It’s now available in Manning’s Early Access Program and I have a 50% of promo code which is valid until Feb 26, 2013 12 midnight EST. Use the code below when ordering:

learnadmau

Here’s the book abstract:

Active Directory is the heart of a Windows network, providing a centralized location for administration, security, and other core management functions. For example, Active Directory authenticates all users in a Windows network and enforces policies for managing your desktop estate. If you’re new to Active Directory administration—or if you find yourself unexpectedly thrust into that role—you need to get a handle on Active Directory quickly. This is the book for you.

Learn Active Directory Management in a Month of Lunches is a practical, hands-on tutorial designed for IT pros new to Active Directory. It skips the theory and concentrates on the day to day administration tasks you need to know to keep your network running smoothly. Just set aside an hour a day for a month—lunchtime would be perfect—and you’ll be comfortable and productive with Active Directory before you know it.

This book makes no assumptions about your background and starts by introducing Active Directory and walking you through its basic features. You’ll learn how to administer AD both from the GUI tools built into Windows and by using PowerShell and the AD cmdlets. Along the way, you’ll touch on best practices for managing user access, setting good group policies, automating backup processes, and more. The examples in the book use Windows Server 2012 version of Active Directory and point out any important differences from earlier versions.