PowerShell Asynchronous Job Queueing

Here is some sample code to have a variable size work queue for asynchronous jobs in PowerShell. The sample code uses Start-Job, but it was written as a skeleton to be able to run multiple PowerCLI tasks at once.

The variable length is beneficial in case the work should be split up, but there is a chance of overwhelming the system if too many concurrent jobs are started at once.

Write-Host "Script Starting."
 
$concurrentTasks = 3 # How many tasks can run at once.
$waitTime = 500 # Duration in ms the script waits to check for job status changes.
 
# $jobs holds pertinent job information. This is a test script so these are
# just simple values to track script progress.
$jobs = @("a", "b", "c", "d", "e", "f", "g")
$jobIndex = 0 #Which job is up for running
$workItems = @{} # Items being worked on
$workComplete = $false # Is the script done with what it needs to do?
 
while (!$workComplete) {
	# Process any finished jobs.
	foreach ($key in @() + $workItems.Keys) {
		# Write-Host "Checking job $key."
		if ($workItems[$key].State -eq "Completed") {
			"$key is done."
			$result = Receive-Job $workItems[$key]
			$workItems.Remove($key)
			"Result: $result"
		}
	}
 
	# Start new jobs if there are open slots.
	while (($workItems.Count -lt $concurrentTasks) -and 
		   ($jobIndex -lt $jobs.Length)) {
			# These jobs don't do anything other than wait a variable amount of time
			# and print an output message.
			$workTime = Get-Random -Minimum 2000 -Maximum 10000
			$job = $jobs[$jobIndex]
			"Starting job {0}." -f $job
			$workItems[$job] = Start-Job -ArgumentList $workTime, $job -ScriptBlock {Start-Sleep -Milliseconds $args[0]; "{0} processed." -f $args[1]}
			$jobIndex += 1
	}
 
	# If all jobs have been processed we are done.
	if ($jobIndex -eq $jobs.Length -and $workItems.Count -eq 0) {
		$workComplete = $true
	}
 
	# Wait between status checks
	Start-Sleep -Milliseconds $waitTime
}
 
Write-Host "Script Finished."

Comments (10)

  1. Brett Mack

    Wow, exactly what I was looking for. I had written similar code for threading in C#, and was faced with moving it to powershell.

    Thanks, perfect!

  2. Eric (Post author)

    Glad it helped. Let me know if you run into any issues.

  3. Anonymous

    I agree. I wrote a script that makes several hundred wmi calls to individual servers that takes forever to run. Exactly what I needed!

  4. David

    Thanks a lot for this post, it really helped me better understand how start-job operates and your code is nice and elegant. Well done.

  5. H.stultiens

    Hello, I think this is a very usefull framework but I have a hard time figuring out where to put the actual worker data, I have several ps1 scripts that I want to execute and want to use this mechanism to speed things up, in the background. Can you show me where I would put .ps1 files to execute?

  6. Eric (Post author)

    @H.stultiens
    You would put the call to your script in the Start-Job line. Be aware that everything in Start-Job will run in it’s own environment, so you may need to pass some information / objects / connections through the argument list.

  7. Prashant

    kudos to you! You made my day -:)

  8. SamB

    You would put the call to your script in the Start-Job line

    Could you provide Exsample of Placent ?

  9. SamB

    Placement

    Its Late!

  10. Stephan

    Do the threads have access to global variables?

Leave a Comment

Your email address will not be published.