Ever had multi core system and some kind of work load which you glued in
unix shell that could use parallelization? I did, so i wrote a poor mans parallelization technique in
bash.
parallel(){
# number of processes you want
num=$1
shift
# loop through commands
for i; do
# make sure at most $num processes are active
until ((`jobs | wc -l` < num)); do
sleep 1
done
# execute the current command in background
$i &
done
# wait for all the remaining processes
wait
}
The following snippet demonstrates how to run 3 commands, up to 2 in parallel.
$ time parallel 2 "sleep 1" "sleep 2" "sleep 3" 2> /dev/null
real 0m4.080s
user 0m0.024s
sys 0m0.036s
What happens is the following:
- executes sleep 1
- executes sleep 2
- waits 1 second (for the first one to finish)
- executes sleep 3
- once it runs out of commands just waits for everything to finish
The ones that we are waiting for here are steps 1 and 3, so they add up to 4 seconds we see.
- sed and grep tips (almirkaric.com)
- bash builtins (almirkaric.com)