SEPTEMBER 02, 2026 · 10 MIN READ · HOÀNG DƯƠNG ·

Advanced Bash: Parallel Execution in DevOps

Comprehensive guide to parallel execution in Bash: background processes, wait, jobs, xargs -P, GNU parallel, concurrency management, race conditions, and file locking with flock in DevOps.

DEVOPS BASH

Parallel Execution in Bash

In the previous post, we explored collecting system metrics and sending automated alerts. But there’s a common problem with monitoring scripts: when you need to check 50 servers simultaneously, sequential execution can take minutes—or even hours if each server takes a few seconds to respond.

Parallel Execution solves this: instead of waiting for server A to finish before moving to server B, you run both at the same time. Bash provides multiple mechanisms for parallelism—from simple &/wait, to convenient xargs -P, to powerful GNU parallel. This article covers leveraging parallelism to optimize DevOps script performance.

Basic Parallel Mechanisms in Bash

Background Processes with `&`

The & symbol at the end of a command runs it in the background, allowing Bash to continue executing the next statement:

1
2
3
4
5
6
7
8
9
# Sequential: takes ~9 seconds
sleep 3
sleep 3
sleep 3

# Parallel with &: takes ~3 seconds
sleep 3 &
sleep 3 &
sleep 3 &

Waiting with `wait`

wait blocks until all background processes complete:

1
2
3
4
5
6
echo "Starting..."
sleep 3 &
sleep 2 &
sleep 1 &
wait
echo "All done!"

Checking Running Processes with `jobs`

1
2
3
4
5
6
7
8
# List background jobs
jobs -l

# Count only running jobs
jobs -r | wc -l

# Count only stopped jobs
jobs -s | wc -l

PID Management

Each background process has a unique PID. You can save the PID for control:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
# Save PID
sleep 60 &
PID=$!
echo "Started background job with PID: $PID"

# Check if job is still running
if kill -0 "$PID" 2>/dev/null; then
    echo "Job $PID is still running"
else
    echo "Job $PID has finished"
fi

# Stop the job
kill "$PID"

# Force stop
kill -9 "$PID"

Limiting Concurrency

Running too many background processes simultaneously can cause overload. The most common way to limit concurrency is using PID arrays or job counting:

Method 1: PID Array

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
MAX_PARALLEL=3

pids=()
for i in $(seq 1 10); do
    echo "Starting task $i..."
    sleep 2 &  # Simulate task
    pids+=($!)

    # Wait when limit reached
    if [[ ${#pids[@]} -ge $MAX_PARALLEL ]]; then
        wait "${pids[0]}"
        pids=("${pids[@]:1}")  # Remove first PID
    fi
done

# Wait for remaining
for pid in "${pids[@]}"; do
    wait "$pid"
done
echo "All tasks completed."

Method 2: Job Counting

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
MAX_PARALLEL=3

for server in server1 server2 server3 server4 server5; do
    ssh "$server" "uptime" &

    # Wait when too many jobs running
    while [[ $(jobs -r | wc -l) -ge $MAX_PARALLEL ]]; do
        sleep 0.1
    done
done

wait
echo "All servers checked."

xargs — Parallel with `-P`

xargs is a powerful tool for running commands on input lists. The -P flag enables parallel execution:

1
2
3
4
5
6
7
8
# Run ssh on 3 servers in parallel
printf "%s\n" server1 server2 server3 | xargs -n 1 -P 3 -I {} ssh user@{} "hostname"

# Check disk on multiple servers
printf "%s\n" server{1..10} | xargs -n 1 -P 5 -I {} bash -c '
    disk=$(ssh user@{} "df / | tail -1 | awk \"{print \$5}\"" 2>/dev/null)
    echo "{}: ${disk:-FAIL}"
'

Flag explanation:

-n 1: Take only 1 argument from input per invocation.
-P 3: Maximum 3 concurrent processes.
-I {}: Placeholder to insert input values into the command.

Practical Example: Parallel Database Backups

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
DATABASES=("db_users" "db_orders" "db_inventory" "db_logs" "db_analytics")

printf "%s\n" "${DATABASES[@]}" | xargs -n 1 -P 3 -I {} bash -c '
    echo "Backing up {}..."
    mysqldump --single-transaction {} > /backup/{}_$(date +%Y%m%d).sql
    if [[ $? -eq 0 ]]; then
        echo "SUCCESS: {}"
    else
        echo "FAILED: {}"
    fi
'

GNU parallel — The Most Powerful Tool

GNU parallel is a dedicated parallel execution tool, more powerful than xargs with many features:

Installation

1
2
3
4
5
6
7
8
# Debian/Ubuntu
apt-get install -y parallel

# CentOS/RHEL
yum install -y parallel

# macOS
brew install parallel

Basic Usage

1
2
3
4
5
6
7
8
# Run ssh on multiple servers
parallel -j 3 ssh user@{} "df -h /" ::: server1 server2 server3 server4

# Run with multiple arguments
parallel -j 3 echo "Processing {1} with {2}" ::: A B C ::: 1 2 3

# Read input from file
parallel -j 5 < server_list.txt ssh {} "uptime"

Advanced Features

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
# Show progress
parallel --progress -j 3 sleep {} ::: 1 2 3 4 5

# Show ETA
parallel --eta -j 3 sleep {} ::: 1 2 3 4 5

# Results in order
parallel --keep-order -j 3 echo {} ::: C B A

# Write output to separate files
parallel --results /tmp/results/ -j 3 echo {} ::: task1 task2 task3

# Dry-run (print commands without executing)
parallel --dry-run -j 3 ssh user@{} "apt update" ::: server1 server2 server3

Example: Parallel SSH Execution on Production

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
#!/usr/bin/env bash
# parallel-ssh.sh — Run commands on multiple servers in parallel with GNU parallel

set -euo pipefail

readonly SERVERS_FILE="${1:-servers.txt}"
readonly COMMAND="${2:?Usage: $0 <servers.txt> <command>}"
readonly MAX_JOBS="${MAX_JOBS:-5}"
readonly SSH_OPTS="-o ConnectTimeout=10 -o StrictHostKeyChecking=accept-new"

if [[ ! -f "$SERVERS_FILE" ]]; then
    echo "Error: Server file $SERVERS_FILE not found" >&2
    exit 1
fi

echo "Running on $(grep -cvE '^\s*#|^\s*$' "$SERVERS_FILE") servers with max $MAX_JOBS parallel jobs..."

grep -vE '^\s*#|^\s*$' "$SERVERS_FILE" | parallel -j "$MAX_JOBS" --keep-order --joblog /tmp/parallel-ssh.log '
    result=$(ssh '"$SSH_OPTS"' {} "'"${COMMAND}"'" 2>&1)
    exit_code=$?
    if [[ $exit_code -eq 0 ]]; then
        echo "[OK] {}"
    else
        echo "[FAIL] {} (exit: $exit_code)"
    fi
    echo "  Output: $result"
'

echo ""
echo "Job log saved to /tmp/parallel-ssh.log"

Race Conditions and File Locking

When multiple processes write to the same file, race conditions can occur—data gets overwritten or interleaved. flock solves this with file locking:

The Race Condition Problem

1
2
3
4
5
6
# DANGEROUS: Multiple processes writing to same log → race condition
for i in $(seq 1 10); do
    echo "Task $i completed" >> /tmp/shared.log &
done
wait
# Result: May lose lines or be out of order

Solution with flock

1
2
3
4
5
6
7
8
9
# Safe: Use flock to lock file before writing
for i in $(seq 1 10); do
    (
        flock 200  # Lock file descriptor 200
        echo "Task $i completed at $(date '+%H:%M:%S.%N')" >> /tmp/shared.log
    ) 200>/tmp/shared.log &
done
wait
# Result: Correct order, no lost lines

Embedding flock in Scripts

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
#!/usr/bin/env bash
# safe-concurrent-write.sh — Safe concurrent logging with flock

exec 200>/tmp/app.lock  # Open file descriptor for locking

for task in task1 task2 task3 task4 task5; do
    (
        flock 200
        echo "[$(date '+%Y-%m-%d %H:%M:%S')] Processing $task" >> /tmp/app.log
        sleep $((RANDOM % 3))  # Simulate task
        echo "[$(date '+%Y-%m-%d %H:%M:%S')] Completed $task" >> /tmp/app.log
    ) &
done

wait
echo "All tasks finished. Log:"
cat /tmp/app.log

Practical Example: Parallel System Health Check

  1
  2
  3
  4
  5
  6
  7
  8
  9
 10
 11
 12
 13
 14
 15
 16
 17
 18
 19
 20
 21
 22
 23
 24
 25
 26
 27
 28
 29
 30
 31
 32
 33
 34
 35
 36
 37
 38
 39
 40
 41
 42
 43
 44
 45
 46
 47
 48
 49
 50
 51
 52
 53
 54
 55
 56
 57
 58
 59
 60
 61
 62
 63
 64
 65
 66
 67
 68
 69
 70
 71
 72
 73
 74
 75
 76
 77
 78
 79
 80
 81
 82
 83
 84
 85
 86
 87
 88
 89
 90
 91
 92
 93
 94
 95
 96
 97
 98
 99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
#!/usr/bin/env bash
# ==============================================================================
# Script : parallel-healthcheck.sh
# Purpose: Health check multiple servers in parallel with aggregated results
# ==============================================================================

set -euo pipefail

readonly SERVERS_FILE="${SERVERS_FILE:-/etc/healthcheck/servers.txt}"
readonly MAX_JOBS="${MAX_JOBS:-10}"
readonly TIMEOUT="${TIMEOUT:-10}"
readonly LOG_DIR="/var/log/healthcheck"
readonly TIMESTAMP=$(date '+%Y%m%d_%H%M%S')
readonly RESULT_FILE="${LOG_DIR}/result_${TIMESTAMP}.txt"
readonly FAIL_FILE="${LOG_DIR}/failures_${TIMESTAMP}.txt"

mkdir -p "$LOG_DIR"

# Check functions
check_ssh() {
    local server="$1"
    ssh -o ConnectTimeout="$TIMEOUT" -o BatchMode=yes \
        "deploy@${server}" "echo OK" 2>/dev/null
}

check_disk() {
    local server="$1"
    ssh -o ConnectTimeout="$TIMEOUT" -o BatchMode=yes \
        "deploy@${server}" "df / | tail -1 | awk '{print \$5}' | tr -d '%'" 2>/dev/null
}

check_service() {
    local server="$1" service="$2"
    ssh -o ConnectTimeout="$TIMEOUT" -o BatchMode=yes \
        "deploy@${server}" "systemctl is-active --quiet ${service} && echo OK || echo DOWN" 2>/dev/null
}

# Worker function for parallel
check_server() {
    local server="$1"
    local status="OK"
    local details=""

    # Check SSH
    if ! check_ssh "$server" >/dev/null 2>&1; then
        echo "FAIL ${server} SSH connection failed"
        return 1
    fi

    # Check disk
    local disk
    disk=$(check_disk "$server")
    if [[ -n "$disk" && "$disk" -gt 85 ]]; then
        details+="disk:${disk}% "
        status="WARN"
    fi

    # Check services
    for svc in nginx docker sshd; do
        local svc_status
        svc_status=$(check_service "$server" "$svc")
        if [[ "$svc_status" == "DOWN" ]]; then
            details+="${svc}:down "
            status="WARN"
        fi
    done

    echo "${status} ${server} ${details}"
}

export -f check_ssh check_disk check_service check_server
export TIMEOUT

main() {
    if [[ ! -f "$SERVERS_FILE" ]]; then
        echo "Error: $SERVERS_FILE not found" >&2
        exit 1
    fi

    local total
    total=$(grep -cvE '^\s*#|^\s*$' "$SERVERS_FILE")

    echo "Checking $total servers with max $MAX_JOBS parallel jobs..."
    echo ""

    grep -vE '^\s*#|^\s*$' "$SERVERS_FILE" | \
        parallel -j "$MAX_JOBS" --keep-order --joblog "${LOG_DIR}/joblog_${TIMESTAMP}.log" \
        check_server {} 2>/dev/null | tee "$RESULT_FILE"

    # Summary
    local ok_count warn_count fail_count
    ok_count=$(grep -c "^OK" "$RESULT_FILE" 2>/dev/null || echo 0)
    warn_count=$(grep -c "^WARN" "$RESULT_FILE" 2>/dev/null || echo 0)
    fail_count=$(grep -c "^FAIL" "$RESULT_FILE" 2>/dev/null || echo 0)

    echo ""
    echo "========================================"
    echo "  SUMMARY"
    echo "  Total:  $total"
    echo "  OK:     $ok_count"
    echo "  WARN:   $warn_count"
    echo "  FAIL:   $fail_count"
    echo "========================================"

    # List failures
    if [[ "$fail_count" -gt 0 ]]; then
        echo ""
        echo "Failures:"
        grep "^FAIL" "$RESULT_FILE" | tee "$FAIL_FILE"
    fi

    # Exit with error if any failures
    [[ "$fail_count" -gt 0 ]] && exit 1
}

main "$@"

Sample Output

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
Checking 8 servers with max 5 parallel jobs...

OK web-01
OK web-02
WARN db-01 disk:87% docker:down
OK web-03
OK worker-01
OK worker-02
FAIL bastion SSH connection failed
OK web-04

========================================
  SUMMARY
  Total:  8
  OK:     6
  WARN:   1
  FAIL:   1
========================================

Failures:
FAIL bastion SSH connection failed

Deployment Notes & Best Practices

Always limit concurrency: Running too many SSH sessions or HTTP requests simultaneously can overload the network or source server. Always set MAX_JOBS appropriately—typically 5-10 for SSH, 20-50 for HTTP requests.
Race conditions when logging: When parallel scripts write to the same file, always use flock or have each process write to a separate file and aggregate afterward.
Test sequentially before parallelizing: Always run the script in sequential mode (MAX_JOBS=1) first to confirm logic is correct, then increase parallelism.
GNU parallel vs xargs: xargs -P is sufficient for most simple use cases. GNU parallel excels when you need: ordered output, progress/ETA display, complex error handling, or multi-host execution.
Timeouts: Always set timeouts for SSH and HTTP requests to prevent scripts from hanging indefinitely when a server is unresponsive.

Conclusion

Parallel Execution is an advanced but extremely useful skill in Bash scripting. From simple background processes with &/wait, to convenient xargs -P, to powerful GNU parallel—you can dramatically reduce processing time for large tasks. Combined with flock to prevent race conditions, you’ll have a safe and efficient parallel execution system.

In the next post, we will explore Bash and REST APIs: calling APIs with curl, parsing JSON with jq, sending bearer auth tokens, and automating workflows with GitHub/Slack APIs.

TAGS:

#bash #shell-script #devops #parallel-execution #xargs #gnu-parallel #concurrency #performance

RESPONSES & DISCUSSION