Here is some sample code to have a variable size work queue for asynchronous jobs in PowerShell. The sample code uses Start-Job, but it was written as a skeleton to be able to run multiple PowerCLI tasks at once.
The variable length is beneficial in case the work should be split up, but there is a chance of overwhelming the system if too many concurrent jobs are started at once.
Write-Host "Script Starting." $concurrentTasks = 3 # How many tasks can run at once. $waitTime = 500 # Duration in ms the script waits to check for job status changes. # $jobs holds pertinent job information. This is a test script so these are # just simple values to track script progress. $jobs = @("a", "b", "c", "d", "e", "f", "g") $jobIndex = 0 #Which job is up for running $workItems = @{} # Items being worked on $workComplete = $false # Is the script done with what it needs to do? while (!$workComplete) { # Process any finished jobs. foreach ($key in @() + $workItems.Keys) { # Write-Host "Checking job $key." if ($workItems[$key].State -eq "Completed") { "$key is done." $result = Receive-Job $workItems[$key] $workItems.Remove($key) "Result: $result" } } # Start new jobs if there are open slots. while (($workItems.Count -lt $concurrentTasks) -and ($jobIndex -lt $jobs.Length)) { # These jobs don't do anything other than wait a variable amount of time # and print an output message. $workTime = Get-Random -Minimum 2000 -Maximum 10000 $job = $jobs[$jobIndex] "Starting job {0}." -f $job $workItems[$job] = Start-Job -ArgumentList $workTime, $job -ScriptBlock {Start-Sleep -Milliseconds $args[0]; "{0} processed." -f $args[1]} $jobIndex += 1 } # If all jobs have been processed we are done. if ($jobIndex -eq $jobs.Length -and $workItems.Count -eq 0) { $workComplete = $true } # Wait between status checks Start-Sleep -Milliseconds $waitTime } Write-Host "Script Finished." |
Wow, exactly what I was looking for. I had written similar code for threading in C#, and was faced with moving it to powershell.
Thanks, perfect!
Glad it helped. Let me know if you run into any issues.
I agree. I wrote a script that makes several hundred wmi calls to individual servers that takes forever to run. Exactly what I needed!
Thanks a lot for this post, it really helped me better understand how start-job operates and your code is nice and elegant. Well done.
Hello, I think this is a very usefull framework but I have a hard time figuring out where to put the actual worker data, I have several ps1 scripts that I want to execute and want to use this mechanism to speed things up, in the background. Can you show me where I would put .ps1 files to execute?
@H.stultiens
You would put the call to your script in the Start-Job line. Be aware that everything in Start-Job will run in it’s own environment, so you may need to pass some information / objects / connections through the argument list.
kudos to you! You made my day -:)
You would put the call to your script in the Start-Job line
Could you provide Exsample of Placent ?
Placement
Its Late!
Do the threads have access to global variables?