Advanced Bash: Parallel Execution in DevOps

Wed, 02 Sep 2026 00:00:00 +0000

Parallel Execution in Bash

In the previous post, we explored collecting system metrics and sending automated alerts. But there’s a common problem with monitoring scripts: when you need to check 50 servers simultaneously, sequential execution can take minutes—or even hours if each server takes a few seconds to respond.

Parallel Execution solves this: instead of waiting for server A to finish before moving to server B, you run both at the same time. Bash provides multiple mechanisms for parallelism—from simple &/wait, to convenient xargs -P, to powerful GNU parallel. This article covers leveraging parallelism to optimize DevOps script performance.

Basic Parallel Mechanisms in Bash

Background Processes with `&`

The & symbol at the end of a command runs it in the background, allowing Bash to continue executing the next statement:

1
2
3
4
5
6
7
8
9


# Sequential: takes ~9 seconds
sleep 3
sleep 3
sleep 3

# Parallel with &: takes ~3 seconds
sleep 3 &
sleep 3 &
sleep 3 &

Waiting with `wait`

wait blocks until all background processes complete:

1
2
3
4
5
6


echo "Starting..."
sleep 3 &
sleep 2 &
sleep 1 &
wait
echo "All done!"

Checking Running Processes with `jobs`

1
2
3
4
5
6
7
8


# List background jobs
jobs -l

# Count only running jobs
jobs -r | wc -l

# Count only stopped jobs
jobs -s | wc -l

PID Management

Each background process has a unique PID. You can save the PID for control:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17


# Save PID
sleep 60 &
PID=$!
echo "Started background job with PID: $PID"

# Check if job is still running
if kill -0 "$PID" 2>/dev/null; then
 echo "Job $PID is still running"
else
 echo "Job $PID has finished"
fi

# Stop the job
kill "$PID"

# Force stop
kill -9 "$PID"

Limiting Concurrency

Running too many background processes simultaneously can cause overload. The most common way to limit concurrency is using PID arrays or job counting:

Method 1: PID Array

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20


MAX_PARALLEL=3

pids=()
for i in $(seq 1 10); do
 echo "Starting task $i..."
 sleep 2 & # Simulate task
 pids+=($!)

 # Wait when limit reached
 if [[ ${#pids[@]} -ge $MAX_PARALLEL ]]; then
 wait "${pids[0]}"
 pids=("${pids[@]:1}") # Remove first PID
 fi
done

# Wait for remaining
for pid in "${pids[@]}"; do
 wait "$pid"
done
echo "All tasks completed."

Method 2: Job Counting

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13


MAX_PARALLEL=3

for server in server1 server2 server3 server4 server5; do
 ssh "$server" "uptime" &

 # Wait when too many jobs running
 while [[ $(jobs -r | wc -l) -ge $MAX_PARALLEL ]]; do
 sleep 0.1
 done
done

wait
echo "All servers checked."

xargs — Parallel with `-P`

xargs is a powerful tool for running commands on input lists. The -P flag enables parallel execution:

1
2
3
4
5
6
7
8


# Run ssh on 3 servers in parallel
printf "%s\n" server1 server2 server3 | xargs -n 1 -P 3 -I {} ssh user@{} "hostname"

# Check disk on multiple servers
printf "%s\n" server{1..10} | xargs -n 1 -P 5 -I {} bash -c '
 disk=$(ssh user@{} "df / | tail -1 | awk \"{print \$5}\"" 2>/dev/null)
 echo "{}: ${disk:-FAIL}"
'

Flag explanation:

-n 1: Take only 1 argument from input per invocation.
-P 3: Maximum 3 concurrent processes.
-I {}: Placeholder to insert input values into the command.

Practical Example: Parallel Database Backups

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11


DATABASES=("db_users" "db_orders" "db_inventory" "db_logs" "db_analytics")

printf "%s\n" "${DATABASES[@]}" | xargs -n 1 -P 3 -I {} bash -c '
 echo "Backing up {}..."
 mysqldump --single-transaction {} > /backup/{}_$(date +%Y%m%d).sql
 if [[ $? -eq 0 ]]; then
 echo "SUCCESS: {}"
 else
 echo "FAILED: {}"
 fi
'

GNU parallel — The Most Powerful Tool

GNU parallel is a dedicated parallel execution tool, more powerful than xargs with many features:

Installation

1
2
3
4
5
6
7
8


# Debian/Ubuntu
apt-get install -y parallel

# CentOS/RHEL
yum install -y parallel

# macOS
brew install parallel

Basic Usage

1
2
3
4
5
6
7
8


# Run ssh on multiple servers
parallel -j 3 ssh user@{} "df -h /" ::: server1 server2 server3 server4

# Run with multiple arguments
parallel -j 3 echo "Processing {1} with {2}" ::: A B C ::: 1 2 3

# Read input from file
parallel -j 5 < server_list.txt ssh {} "uptime"

Advanced Features

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14


# Show progress
parallel --progress -j 3 sleep {} ::: 1 2 3 4 5

# Show ETA
parallel --eta -j 3 sleep {} ::: 1 2 3 4 5

# Results in order
parallel --keep-order -j 3 echo {} ::: C B A

# Write output to separate files
parallel --results /tmp/results/ -j 3 echo {} ::: task1 task2 task3

# Dry-run (print commands without executing)
parallel --dry-run -j 3 ssh user@{} "apt update" ::: server1 server2 server3

Example: Parallel SSH Execution on Production

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30


#!/usr/bin/env bash
# parallel-ssh.sh — Run commands on multiple servers in parallel with GNU parallel

set -euo pipefail

readonly SERVERS_FILE="${1:-servers.txt}"
readonly COMMAND="${2:?Usage: $0 <servers.txt> <command>}"
readonly MAX_JOBS="${MAX_JOBS:-5}"
readonly SSH_OPTS="-o ConnectTimeout=10 -o StrictHostKeyChecking=accept-new"

if [[ ! -f "$SERVERS_FILE" ]]; then
 echo "Error: Server file $SERVERS_FILE not found" >&2
 exit 1
fi

echo "Running on $(grep -cvE '^\s*#|^\s*$' "$SERVERS_FILE") servers with max $MAX_JOBS parallel jobs..."

grep -vE '^\s*#|^\s*$' "$SERVERS_FILE" | parallel -j "$MAX_JOBS" --keep-order --joblog /tmp/parallel-ssh.log '
 result=$(ssh '"$SSH_OPTS"' {} "'"${COMMAND}"'" 2>&1)
 exit_code=$?
 if [[ $exit_code -eq 0 ]]; then
 echo "[OK] {}"
 else
 echo "[FAIL] {} (exit: $exit_code)"
 fi
 echo " Output: $result"
'

echo ""
echo "Job log saved to /tmp/parallel-ssh.log"

Race Conditions and File Locking

When multiple processes write to the same file, race conditions can occur—data gets overwritten or interleaved. flock solves this with file locking:

The Race Condition Problem

1
2
3
4
5
6


# DANGEROUS: Multiple processes writing to same log → race condition
for i in $(seq 1 10); do
 echo "Task $i completed" >> /tmp/shared.log &
done
wait
# Result: May lose lines or be out of order

Solution with flock

1
2
3
4
5
6
7
8
9


# Safe: Use flock to lock file before writing
for i in $(seq 1 10); do
 (
 flock 200 # Lock file descriptor 200
 echo "Task $i completed at $(date '+%H:%M:%S.%N')" >> /tmp/shared.log
 ) 200>/tmp/shared.log &
done
wait
# Result: Correct order, no lost lines

Embedding flock in Scripts

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17


#!/usr/bin/env bash
# safe-concurrent-write.sh — Safe concurrent logging with flock

exec 200>/tmp/app.lock # Open file descriptor for locking

for task in task1 task2 task3 task4 task5; do
 (
 flock 200
 echo "[$(date '+%Y-%m-%d %H:%M:%S')] Processing $task" >> /tmp/app.log
 sleep $((RANDOM % 3)) # Simulate task
 echo "[$(date '+%Y-%m-%d %H:%M:%S')] Completed $task" >> /tmp/app.log
 ) &
done

wait
echo "All tasks finished. Log:"
cat /tmp/app.log

Practical Example: Parallel System Health Check

 1
 2
 3
 4
 5
 6
 7
 8
 9
 10
 11
 12
 13
 14
 15
 16
 17
 18
 19
 20
 21
 22
 23
 24
 25
 26
 27
 28
 29
 30
 31
 32
 33
 34
 35
 36
 37
 38
 39
 40
 41
 42
 43
 44
 45
 46
 47
 48
 49
 50
 51
 52
 53
 54
 55
 56
 57
 58
 59
 60
 61
 62
 63
 64
 65
 66
 67
 68
 69
 70
 71
 72
 73
 74
 75
 76
 77
 78
 79
 80
 81
 82
 83
 84
 85
 86
 87
 88
 89
 90
 91
 92
 93
 94
 95
 96
 97
 98
 99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116


#!/usr/bin/env bash
# ==============================================================================
# Script : parallel-healthcheck.sh
# Purpose: Health check multiple servers in parallel with aggregated results
# ==============================================================================

set -euo pipefail

readonly SERVERS_FILE="${SERVERS_FILE:-/etc/healthcheck/servers.txt}"
readonly MAX_JOBS="${MAX_JOBS:-10}"
readonly TIMEOUT="${TIMEOUT:-10}"
readonly LOG_DIR="/var/log/healthcheck"
readonly TIMESTAMP=$(date '+%Y%m%d_%H%M%S')
readonly RESULT_FILE="${LOG_DIR}/result_${TIMESTAMP}.txt"
readonly FAIL_FILE="${LOG_DIR}/failures_${TIMESTAMP}.txt"

mkdir -p "$LOG_DIR"

# Check functions
check_ssh() {
 local server="$1"
 ssh -o ConnectTimeout="$TIMEOUT" -o BatchMode=yes \
 "deploy@${server}" "echo OK" 2>/dev/null
}

check_disk() {
 local server="$1"
 ssh -o ConnectTimeout="$TIMEOUT" -o BatchMode=yes \
 "deploy@${server}" "df / | tail -1 | awk '{print \$5}' | tr -d '%'" 2>/dev/null
}

check_service() {
 local server="$1" service="$2"
 ssh -o ConnectTimeout="$TIMEOUT" -o BatchMode=yes \
 "deploy@${server}" "systemctl is-active --quiet ${service} && echo OK || echo DOWN" 2>/dev/null
}

# Worker function for parallel
check_server() {
 local server="$1"
 local status="OK"
 local details=""

 # Check SSH
 if ! check_ssh "$server" >/dev/null 2>&1; then
 echo "FAIL ${server} SSH connection failed"
 return 1
 fi

 # Check disk
 local disk
 disk=$(check_disk "$server")
 if [[ -n "$disk" && "$disk" -gt 85 ]]; then
 details+="disk:${disk}% "
 status="WARN"
 fi

 # Check services
 for svc in nginx docker sshd; do
 local svc_status
 svc_status=$(check_service "$server" "$svc")
 if [[ "$svc_status" == "DOWN" ]]; then
 details+="${svc}:down "
 status="WARN"
 fi
 done

 echo "${status} ${server} ${details}"
}

export -f check_ssh check_disk check_service check_server
export TIMEOUT

main() {
 if [[ ! -f "$SERVERS_FILE" ]]; then
 echo "Error: $SERVERS_FILE not found" >&2
 exit 1
 fi

 local total
 total=$(grep -cvE '^\s*#|^\s*$' "$SERVERS_FILE")

 echo "Checking $total servers with max $MAX_JOBS parallel jobs..."
 echo ""

 grep -vE '^\s*#|^\s*$' "$SERVERS_FILE" | \
 parallel -j "$MAX_JOBS" --keep-order --joblog "${LOG_DIR}/joblog_${TIMESTAMP}.log" \
 check_server {} 2>/dev/null | tee "$RESULT_FILE"

 # Summary
 local ok_count warn_count fail_count
 ok_count=$(grep -c "^OK" "$RESULT_FILE" 2>/dev/null || echo 0)
 warn_count=$(grep -c "^WARN" "$RESULT_FILE" 2>/dev/null || echo 0)
 fail_count=$(grep -c "^FAIL" "$RESULT_FILE" 2>/dev/null || echo 0)

 echo ""
 echo "========================================"
 echo " SUMMARY"
 echo " Total: $total"
 echo " OK: $ok_count"
 echo " WARN: $warn_count"
 echo " FAIL: $fail_count"
 echo "========================================"

 # List failures
 if [[ "$fail_count" -gt 0 ]]; then
 echo ""
 echo "Failures:"
 grep "^FAIL" "$RESULT_FILE" | tee "$FAIL_FILE"
 fi

 # Exit with error if any failures
 [[ "$fail_count" -gt 0 ]] && exit 1
}

main "$@"

Sample Output

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21


Checking 8 servers with max 5 parallel jobs...

OK web-01
OK web-02
WARN db-01 disk:87% docker:down
OK web-03
OK worker-01
OK worker-02
FAIL bastion SSH connection failed
OK web-04

========================================
 SUMMARY
 Total: 8
 OK: 6
 WARN: 1
 FAIL: 1
========================================

Failures:
FAIL bastion SSH connection failed

Deployment Notes & Best Practices

Always limit concurrency: Running too many SSH sessions or HTTP requests simultaneously can overload the network or source server. Always set MAX_JOBS appropriately—typically 5-10 for SSH, 20-50 for HTTP requests.
Race conditions when logging: When parallel scripts write to the same file, always use flock or have each process write to a separate file and aggregate afterward.
Test sequentially before parallelizing: Always run the script in sequential mode (MAX_JOBS=1) first to confirm logic is correct, then increase parallelism.
GNU parallel vs xargs: xargs -P is sufficient for most simple use cases. GNU parallel excels when you need: ordered output, progress/ETA display, complex error handling, or multi-host execution.
Timeouts: Always set timeouts for SSH and HTTP requests to prevent scripts from hanging indefinitely when a server is unresponsive.

Conclusion

Parallel Execution is an advanced but extremely useful skill in Bash scripting. From simple background processes with &/wait, to convenient xargs -P, to powerful GNU parallel—you can dramatically reduce processing time for large tasks. Combined with flock to prevent race conditions, you’ll have a safe and efficient parallel execution system.

In the next post, we will explore Bash and REST APIs: calling APIs with curl, parsing JSON with jq, sending bearer auth tokens, and automating workflows with GitHub/Slack APIs.

Xargs on Hoang Duong