Another Orchestrator/PowerShell hang issue

It has come to my attention that some times the scripts will still hang after using the trick from a previous post to fix an issue with standard input when running PowerShell in Orchestrator workflows. After further research there appears to be an issue with stderr hanging the scripts in a similar way to stdin. To be safe you should do something with the output from stderr (where error messages are sent).

You can do at least three things

//Throw it away with 2>NUL
command = new Command("cmd.exe /c powershell.exe -File c:\\orchestrator\\test.ps1 < NUL 2>NUL");
//Log it to a new file with 2> filename
command = new Command("cmd.exe /c powershell.exe -File c:\\orchestrator\\test.ps1 < NUL 2>c:\\orchestrator\\stderr.log");
//Send it wherever standard output goes using 2>&1
command = new Command("cmd.exe /c powershell.exe -File c:\\orchestrator\\estest.ps1 < NUL 2>&1");

For further reading on the topic of standard input, output, and error please see

Running PowerShell in VMware Orchestrator – updated

Updated 7/29/2010 – I have added a new follow up post this post regarding another related hanging issue. Please make sure to check it out.

This is a copy of my comment in the VMware Communities, but some times the Google search results for items in this blog show up and the ones for the communities do not.

The original problem is that PowerShell scripts hang when running them in Orchestrator.

I have spent some time on this and have a way to make it work. For some reason when run this way the $input variable in Powershell is expecting a pipeline and I believe it just waits for pipelined input.

The $input variable is of type System.Management.Automation.Runspaces.Pipeline instead of the standard System.Collections.ArrayList+ArrayListEnumeratorSimple when running scripts. I have tried using the methods in the Pipeline class to clear things up to no avail. I have also tried using the input property on the Orchestrator Command class, but I could not get it to work. I am just getting started with Orchestrator though, so it may be my lack of knowledge.

As a workaround you can input NUL to the script or command using cmd.exe.

// Run a command
command = new Command("cmd /c powershell.exe -Command dir variable: >> c:\\orchestrator\\input.out < NUL");
//Run a script
command = new Command("cmd /c powershell.exe -File c:\\orchestrator\\input.ps1 < NUL");

Adding VMware PowerCLI to your standard PowerShell environment

In order to get access to PowerCLI when using another PowerShell shortcut or environment you can run the following commands which is what happens when you run the VMware provided shortcut.

# Adds the base cmdlets
Add-PSSnapin VMware.VimAutomation.Core
# Add the following if you want to do things with Update Manager
Add-PSSnapin VMware.VumAutomation
# This script adds some helper functions and sets the appearance. You can pick and choose parts of this file for a fully custom appearance.
. "C:\Program Files\VMware\Infrastructure\vSphere PowerCLI\Scripts\Initialize-VIToolkitEnvironment.ps1"

**UPDATE 2/7/2013**
As Paul noted in the comments, the new initialization script is called Initialize-PowerCLIEnvironment.ps1. Also, if you are on a 64 bit system the files will be in “Program Files (x86)”

Note on the last line that there is a space between the “.” and the path to the script. That space is very important as it means we are including or “dot sourcing” the file.

If you want this all to load every time you launch PowerShell you can add these commands to your profile startup scripts. This is a great place to add functions or aliases that you use all the time. The profile is run once when the PowerShell environment is launched.

PowerCLI Examples for VMUG Presentation 7/20/2010

These are some examples that I will be reviewing at an upcoming Central Ohio VMware User’s Group.

# List all VMs
# Get one VM
Get-VM ubuntu01
# Get VMs that match
Get-VM ubuntu*
# List VMs in a Datacenter
Get-Datacenter dc001 | Get-VM
# List VMs in a cluster
Get-Cluster cluster001 | Get-VM
# Get 2 CPU or larger VMs
Get-VM | Where-Object {$_.NumCPU -ge 2}
# Get 2 GB RAM or larger VMs
Get-VM | Where-Object {$_.MemoryMB -ge 2048}
# Getting information about what you can do
Get-Help Get-VM
Get-VM | Get-Member
# Save a report of VM CPU/MEM configuration information
Get-VM | Select-Object Name, NumCPU, MemoryMB | Export-Csv "VM_CPU_Memory_20100720.csv" -NoTypeInformation
# List VMs with connected CD-ROMs
Get-VM | Where-Object {$_ | Get-CDDrive | Where-Object {$_.ConnectionState.Connected}}
# Disconnect all connected CD-ROMs
Get-VM | Get-CDDrive | Where-Object {$_.ConnectionState.Connected} | Set-CDDrive -Connected $false -Confirm:$false
# Migrate a template to another host
Get-Template ubuntu | Set-Template -ToVM | Move-VM -Destination (Get-VMHost esx001.lab.local) | Set-VM -ToTemplate -Confirm:$false
# Report on Storage
Get-Datastore | Select-Object Name, FreeSpaceMB, CapacityMB | Export-Csv "Datastores_20100720.csv" -NoTypeInformation
# Storage VMotion ALL VMs from one LUN to another (All at the same time) COULD BE BAD
Get-Datastore "cla_nonrep_01" | Get-VM | Move-VM -Datastore (Get-Datastore "cla_nonrep_00")
# Storage VMotion ALL VMs from one LUN to another (One at a time)
Get-Datastore "cla_nonrep_01" | Get-VM | ForEach-Object {Move-VM -VM $_ -Datastore (Get-Datastore "cla_nonrep_00")}
# Set all limited CPU reservations to unlimted
Get-VM | Get-VMResourceConfiguration | Where-Object { $_.CpuLimitMhz -ne -1} | Set-VMResourceConfiguration -CpuLimitMhz $null
# Set all limited memory reservations to unlimited
Get-VM | Get-VMResourceConfiguration | Where-Object { $_.MemLimitMB -ne -1} | Set-VMResourceConfiguration -MemLimitMB $null
# Configure a new network on a host
Get-VirtualSwitch -VMHost esx003.lab.local -Name vSwitch0 | New-VirtualPortGroup -Name "VGT Test" -VLanId 4095
# Configure a new network on a cluster
Get-VirtualSwitch -VMHost (Get-Cluster cluster001 | Get-VMHost) -Name vSwitch0 | New-VirtualPortGroup -Name "VGT Test" -VLanId 4095
# Generate a diagnostic log bundle
Get-Log -VMHost esx001.lab.local -Bundle -DestinationPath c:\hostlogs
# List the entries in vmkwarning
(Get-Log -VMHost esx001.lab.local -Key vmkwarning).Entries
# Find something in a log
(Get-Log -VMHost esx001.lab.local -Key vmkernel).Entries | Select-String "Error"
# What logs are available?]
Get-LogType -VMHost esx001.lab.local
# Find VMs with snapshots (Pretty slow - could be optimized)
Get-VM | Where-Object {$_ | Get-Snapshot}
#Verify advanced settings
# http://wannemacher.us/?p=135
# Change advanced settings
# http://wannemacher.us/?p=104
# Change the root password across multiple hosts
# http://wannemacher.us/?p=304

Understanding VMware VMkernel Traffic Routing

As a basis for an upcoming post on splitting vmkernel traffic across over layer 3 boundaries I wanted to describe how vmkernel traffic is routed on an ESX host. There seems to be a lot of confusion in this area and hopefully this will help to clear it up.

If you need a refresher on IP addresses, network masks, or subnets check out this Cisco article.

Directly Connected Networks

If a host is directly connected to a subnet it will use that interface to talk to devices in that subnet. For example if I have an interface with the IP NETMASK, that interface will be used to talk to anything on the network. This applies to every directly connected interface.

If I have three vmkernel port groups defined with the following IP information

Then vmk0 will be used to talk to everything on, vmk1 for, and vmk2 for

Remote Networks

So, what happens when the device I am talking to is on a subnet that I am not directly connected to? This is where the routing table really comes into play so let’s take a look at it using:

vicfg-route –list

VMkernel Routes:
Network             Netmask             Gateway         Local Subnet         Local Subnet         Local Subnet

We see the directly connected networks with a Gateway of Local Subnet. This describes the direct communication that we discussed in Directly Connected Networks.

The last line is a result of our configuration of the “VMkernel Default Gateway” when setting up the vmkernel port group. What it says is send everything else to the router at

The router is in the network and since vmk0 is directly connected to that subnet we know that it will be used for all non local traffic.

A point of clarification

I have seen some confusing statements out there to the effect of “The vmkernel port group with the default gateway assigned will be used to send traffic.” As we have seen, this is not quite true.

All vmkernel ports use the same default gateway so there is no specific assignment per port group. The vmkernel port group that is directly connected to the specified gateway will be used. Unless specific routes are added that means the vmknic in the same subnet as the default gateway will be used for all routed vmkernel traffic.

The routing table can be customized using the vicfg-route command, but should be done rarely. I will discuss one reason you want to do that in my post on splitting vmkernel traffic when crossing layer 3 boundaries.

Side Note: Service Console vs. VMkernel
On the non ESXi versions of vSphere the service console and vmkernel each have their own TCP/IP stacks and therefore have their own IP configuration including routing tables. This means that any IP configuration of one has no effect on the other. The service console’s routing table can be viewed with the command “route” or “route -n”.

Export and Import Customization Specs Using PowerCLI

Here are a couple of functions to help you export and import OS customization specs in vCenter. Be aware that if you do this between vCenters or the certificate changes then you will have to set any encrypted passwords to their correct value after the import.

These functions and the standard *-OSCustomizationSpec cmdlets should allow you to replicate your customization specs between multiple vCenters fairly easily.

Function Export-OSCustomizationSpec {
    param (
        [string]$exportFile = "$specname.xml"
    $csmgr = Get-View CustomizationSpecManager
    if ($csmgr.DoesCustomizationSpecExist($specName)) {
        $spec = $csmgr.GetCustomizationSpec($specName)
        $csmgr.CustomizationSpecItemToXml($spec) | Out-File $exportFile
    else {
        throw "Spec $specName not found"
Function Import-OSCustomizationSpec {
    param (
        [string]$specName #Use to change the spec name from that defined in the file
    $specXml = Get-Content $importFile
    $csmgr = Get-View CustomizationSpecManager
    $spec = $csmgr.XmlToCustomizationSpecItem($specXml)
    # Change the name if a new one was given.
    if ($specName) {
        $spec.Info.Name = $specName
    if ($csmgr.DoesCustomizationSpecExist($spec.Info.Name)) {
        throw "Spec $specName already exists."
    else {

Notify All Active vSphere Users (PowerCLI version)

Steve Jin recently posted How to Notify All the Active vSphere Users on his DoubleCloud blog. Below are some functions to make this easy to do via PowerCLI.

Function Get-VIServiceMessage {
    $sessionManager = Get-View SessionManager
Function Set-VIServiceMessage {
    param (
        [String]$message = $(throw "Must specify a message.")
    $sessionManager = Get-View SessionManager
Function Remove-VIServiceMessage {

Summarize PowerCLI Cmdlets by Noun

Here is some simple code to generate a reference list of the PowerCLI cmdlet objects(nouns) and what actions(verbs) you can perform on them.

This can come in handy for people new to PowerCLI to show them what the cmdlets will handle for them.

$commandTable = @{}
$commandList = Get-Command -Module VMware.VimAutomation.Core -CommandType Cmdlet
foreach ($command in $commandList) {
    if (!$commandTable.ContainsKey($command.Noun)) {
        $commandTable[$command.Noun] = @()
    $commandTable[$command.Noun] += $command.Verb
foreach ($command in $commandTable.Keys | Sort-Object) {
    $verbs = $commandTable[$command] | Sort-Object
    $command | Select-Object @{Name="Noun"; Expression={$command}}, 
                                 @{Name="Verbs"; Expression={[String]::Join(",", $verbs)}}

Here is the output for PowerCLI version 4 update 1.

Unable to install ESX400-201002401 or ESX400-200912401

Update Manager 4.0 Update 1 was refusing to install either of these two patches. At first I was confused because I did not have Nexus deployed, but this was looking similar to the problem that Update 1 was supposed to fix.

I downloaded the bundle from VMware’s web site and tried to install the patch manually and got this error:

The following problems were encountered trying to resolve dependencies:
   cross_emulex-cim-provider_400. provides 'emulex-cim-provider
   >= 400.' (required by rpm_vmware-esx-cim_4.0.0-1.11.236512@i386), but
   is obsoleted by the host

The culprit is the HBAnywhere installation I loaded for HBA firmware management. I uninstalled HBAnywhere and the patches worked just fine.

$ sudo rpm -qa | grep elx
$ sudo rpm -e elxvmwarecorekit-esx40-4.0a44-1

I originally had to follow this Emulex KB article to get the application installed.

I went to Emulex’s web site to find a HBAnywhere update package for ESX 4 U1 and now I can not find a download for the original version let alone an updated one. The firmware update manual still lists HBAnywhere as a utility for VMware though.

If anyone knows of an updated package or another way to do firmware updates online I would appreciate letting me know in the comments.

UPDATE 5/6/2010
The current version of HBAnywhere resolves this issue. Here are the newest downloads for VMware.