PowerShell Asynchronous Job Queueing

Write-Host "Script Starting."
 
$concurrentTasks = 3 # How many tasks can run at once.
$waitTime = 500 # Duration in ms the script waits to check for job status changes.
 
# $jobs holds pertinent job information. This is a test script so these are
# just simple values to track script progress.
$jobs = @("a", "b", "c", "d", "e", "f", "g")
$jobIndex = 0 #Which job is up for running
$workItems = @{} # Items being worked on
$workComplete = $false # Is the script done with what it needs to do?
 
while (!$workComplete) {
	# Process any finished jobs.
	foreach ($key in @() + $workItems.Keys) {
		# Write-Host "Checking job $key."
		if ($workItems[$key].State -eq "Completed") {
			"$key is done."
			$result = Receive-Job $workItems[$key]
			$workItems.Remove($key)
			"Result: $result"
		}
	}
 
	# Start new jobs if there are open slots.
	while (($workItems.Count -lt $concurrentTasks) -and 
		   ($jobIndex -lt $jobs.Length)) {
			# These jobs don't do anything other than wait a variable amount of time
			# and print an output message.
			$workTime = Get-Random -Minimum 2000 -Maximum 10000
			$job = $jobs[$jobIndex]
			"Starting job {0}." -f $job
			$workItems[$job] = Start-Job -ArgumentList $workTime, $job -ScriptBlock {Start-Sleep -Milliseconds $args[0]; "{0} processed." -f $args[1]}
			$jobIndex += 1
	}
 
	# If all jobs have been processed we are done.
	if ($jobIndex -eq $jobs.Length -and $workItems.Count -eq 0) {
		$workComplete = $true
	}
 
	# Wait between status checks
	Start-Sleep -Milliseconds $waitTime
}
 
Write-Host "Script Finished."

Comments (10)

1. Brett Mack April 15, 2011 at 8:50 am

   Wow, exactly what I was looking for. I had written similar code for threading in C#, and was faced with moving it to powershell.

   Thanks, perfect!
2. Eric (Post author)April 26, 2011 at 12:04 pm

   Glad it helped. Let me know if you run into any issues.
3. Anonymous May 23, 2011 at 7:09 pm

   I agree. I wrote a script that makes several hundred wmi calls to individual servers that takes forever to run. Exactly what I needed!
4. David November 7, 2011 at 7:27 pm

   Thanks a lot for this post, it really helped me better understand how start-job operates and your code is nice and elegant. Well done.
5. H.stultiens October 16, 2012 at 4:16 am

   Hello, I think this is a very usefull framework but I have a hard time figuring out where to put the actual worker data, I have several ps1 scripts that I want to execute and want to use this mechanism to speed things up, in the background. Can you show me where I would put .ps1 files to execute?
6. Eric (Post author)November 17, 2012 at 1:47 pm

   @H.stultiens
   You would put the call to your script in the Start-Job line. Be aware that everything in Start-Job will run in it’s own environment, so you may need to pass some information / objects / connections through the argument list.
7. Prashant December 6, 2012 at 6:39 am

   kudos to you! You made my day -:)
8. SamB April 8, 2013 at 10:50 pm

   You would put the call to your script in the Start-Job line

   Could you provide Exsample of Placent ?
9. SamB April 8, 2013 at 10:50 pm

   Placement

   Its Late!
10. Stephan December 18, 2015 at 8:10 am

    Do the threads have access to global variables?