JULY 05, 2026 · 11 MIN READ · HOÀNG DƯƠNG ·

Error Handling in Bash: Exception Traps, Retry Patterns, and Failure Recovery for DevOps

Complete guide to Bash error handling for DevOps: exit codes, safe cleanup with trap EXIT/ERR, set -e pitfalls, automated retry patterns, and resilient scripting.

DEVOPS BASH

Error Handling in Bash for DevOps

In the previous post, we explored essential debugging techniques to diagnose failures after they happen. However, in production environments and automated CI/CD pipelines, reactive debugging is not enough. Production-grade DevOps scripts must proactively handle errors as they occur at runtime.

Intermittent network hiccups, exhausted disk capacity, slow-starting dependent services, and permission misconfigurations are inevitable realities. Without robust error handling, automation scripts risk:

Silently continuing execution in an invalid state, causing cascading data corruption.
Abruptly crashing without releasing locks or cleaning up temporary scratch directories, blocking subsequent runs.
Failing immediately on transient glitches that could have succeeded after a short retry.

This guide provides a comprehensive framework for exception handling in Bash: from mastering Exit Codes and avoiding set -e traps, to deterministic resource cleanup with trap, building reusable Retry Patterns, and standardizing log severity levels.

Understanding Exit Codes and Status Conventions

Every command or program executed in a Linux/Unix environment returns an integer between 0 and 255 upon completion, known as its Exit Status or Exit Code. This value is captured in the special shell parameter $?.

0: Indicates that the command completed successfully.
1 - 255: Indicates that an error or anomalous condition occurred.

Standard Linux Exit Codes

Linux and POSIX shell environments adhere to standardized status conventions that every DevOps engineer should know:

Exit Code	Standard Meaning	Common Cause
`0`	Success	Command completed without errors
`1`	General error	Catch-all for general failures (e.g., division by zero)
`2`	Misuse of shell built-in	Missing mandatory arguments for internal commands
`126`	Command invoked cannot execute	Missing execute permissions (`chmod +x`)
`127`	Command not found	Binary name typo or missing directory in `$PATH`
`128`	Invalid argument to exit	Calling `exit 3.14` (only integer values permitted)
`128 + N`	Process terminated by signal `N`	`130` (128+2) for `Ctrl+C` (SIGINT), `137` (128+9) for `SIGKILL`
`255`	Exit status out of range	Status value exceeded 0–255 range

Defining Custom Exit Codes

To enable downstream CI/CD pipelines and monitoring agents to distinguish between distinct failure domains, define explicit exit constants at the top of your scripts (values between 10 and 99 are recommended to avoid conflicting with system codes):

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
# Script error constants
readonly EXIT_SUCCESS=0
readonly EXIT_INVALID_ARGS=10
readonly EXIT_CONFIG_MISSING=11
readonly EXIT_NETWORK_TIMEOUT=12
readonly EXIT_DISK_FULL=13
readonly EXIT_DEPENDENCY_ERROR=14

validate_config() {
    local config_file="$1"
    if [[ ! -f "$config_file" ]]; then
        echo "[ERROR] Configuration file $config_file does not exist." >&2
        exit "$EXIT_CONFIG_MISSING"
    fi
}

Execution Flow Control with Short-Circuiting and set -e

Inline Error Handling with Short-Circuit Operators

The boolean operators && (AND) and || (OR) provide concise one-line conditionals without requiring bulky if...fi structures:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
# Execute subsequent command only if previous command SUCCEEDS
mkdir -p /opt/backup && echo "Backup directory created."

# Execute fallback or exit immediately if command FAILS
cd /opt/app/releases || {
    echo "[FATAL] Cannot enter release directory." >&2
    exit 1
}

# Verify file presence or copy default template
[[ -f ".env" ]] || cp ".env.example" ".env"

Fast-Fail Execution with set -e

By default, Bash continues running subsequent lines even when earlier commands fail. Enabling set -e (or set -o errexit) instructs the shell to terminate immediately if any pipeline returns a non-zero exit code.

1
2
3
4
5
6
7
8
9
#!/usr/bin/env bash
set -e

echo "Preparing archive..."
tar -czf backup.tar.gz /data/db-dump/
# If the tar command fails (e.g., permission denied), execution halts immediately,
# preventing accidental deletion of unbacked data below!
rm -rf /data/db-dump/*
echo "Done."

Hidden Pitfalls of set -e in Practice

While set -e is indispensable for fail-fast safety, it possesses subtle nuances that can surprise even experienced engineers.

Pitfall 1: Arithmetic Evaluation Returning Zero

In Bash arithmetic evaluations (( ... )), a calculated result of 0 evaluates to an exit code of 1 (following C-style boolean conventions where zero represents false):

1
2
3
4
5
6
7
8
#!/usr/bin/env bash
set -e

count=0
# Post-increment returns original value (0), giving exit status 1 -> SCRIPT TERMINATES!
(( count++ )) 

echo "This message will NEVER be displayed!"

Workaround:

1
2
3
4
5
# Approach 1: Pre-increment evaluation
(( ++count ))

# Approach 2: Explicit variable assignment
count=$(( count + 1 ))

Pitfall 2: Test Commands Returning False by Design

Utilities like grep, diff, or test intentionally return 1 when no match or differences are found. Under set -e, this causes sudden termination:

1
2
3
4
5
#!/usr/bin/env bash
set -e

# If no WARNING entries exist, grep returns 1 -> Script crashes!
grep "WARNING" /var/log/app.log

Workaround:

1
2
3
4
5
6
7
8
9
# Approach 1: Append || true when failure is non-fatal
grep "WARNING" /var/log/app.log || true

# Approach 2: Evaluate inside an if condition (set -e is suspended in test expressions)
if grep -q "WARNING" /var/log/app.log; then
    echo "Warnings detected in log."
else
    echo "No warnings found."
fi

Pitfall 3: Pipelines Masking Intermediate Errors

In a command chain cmd1 | cmd2 | cmd3, Bash by default only inspects the exit code of cmd3. If cmd1 crashes while cmd3 succeeds, the script continues unnoticed:

1
2
# If mysqldump fails due to auth errors, gzip still compresses empty input and returns 0!
mysqldump -u root -p invalid_db | gzip > dump.sql.gz

Workaround: Always combine with set -o pipefail:

1
set -euo pipefail

Trapping Errors and Safe Resource Cleanup with trap

A foundational rule of infrastructure reliability is: Whether a script completes successfully, encounters a fatal error, or is manually interrupted with Ctrl+C, all ephemeral resources must be cleanly released.

The trap command allows you to intercept OS signals and shell lifecycle events to execute cleanup logic.

Essential Signals and Events

Signal / Event	Trigger Condition
`EXIT` (or `0`)	Fired whenever the script terminates (normal exit or via `exit`)
`ERR`	Fired when any simple command returns a non-zero exit code
`SIGINT` (Signal 2)	Fired when the user sends an interrupt (`Ctrl+C`)
`SIGTERM` (Signal 15)	Fired when receiving a termination signal from `kill` or Docker engine

Standard Cleanup Pattern

Below is a production-grade template for deterministic resource cleanup:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
#!/usr/bin/env bash
set -euo pipefail

# Allocate scratch storage and process lock
readonly WORK_DIR=$(mktemp -d -t deploy-workspace-XXXXXX)
readonly LOCK_FILE="/tmp/my-service-deploy.lock"

cleanup() {
    local exit_code=$?
    echo "[CLEANUP] Releasing temporary resources..."
    
    # Remove temporary scratch workspace
    if [[ -d "$WORK_DIR" ]]; then
        rm -rf "$WORK_DIR"
        echo "[CLEANUP] Removed temporary directory: $WORK_DIR"
    fi
    
    # Release process lock
    if [[ -f "$LOCK_FILE" ]]; then
        rm -f "$LOCK_FILE"
        echo "[CLEANUP] Removed process lock."
    fi
    
    if [[ $exit_code -ne 0 ]]; then
        echo "[CLEANUP] Script terminated with error status: $exit_code" >&2
    else
        echo "[CLEANUP] Cleanup completed successfully."
    fi
    
    exit "$exit_code"
}

# Register cleanup handler across normal exit and interrupt signals
trap cleanup EXIT SIGINT SIGTERM

Building an Automated Retry Pattern

In distributed cloud architectures, transient network failures, temporary rate limits, or services warming up are routine occurrences. Rather than failing immediately, a resilient script employs a configurable Retry Pattern with backoff delays.

Here is a universal retry helper capable of executing arbitrary commands with bounded attempts:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
# ==============================================================================
# Function: retry
# Arguments:
#   $1: Maximum attempt count (max_attempts)
#   $2: Interval between attempts in seconds (delay_seconds)
#   $3+: Command to execute with arguments
# ==============================================================================
retry() {
    local max_attempts="$1"
    local delay="$2"
    shift 2
    local command=("$@")
    
    local attempt=1
    until "${command[@]}"; do
        local exit_code=$?
        if (( attempt >= max_attempts )); then
            echo "[ERROR] Command '${command[*]}' failed after $attempt attempts (Exit code: $exit_code)." >&2
            return "$exit_code"
        fi
        
        echo "[WARN] Command '${command[*]}' failed (attempt $attempt/$max_attempts). Retrying in ${delay}s..." >&2
        sleep "$delay"
        (( attempt++ ))
    done
    
    return 0
}

Practical DevOps Use Cases for Retry

1
2
3
4
5
# Wait for database health endpoint to respond (up to 5 attempts, 3s delay)
retry 5 3 curl -fsS "http://localhost:5432/healthz"

# Download remote deployment artifact with retry
retry 3 5 aws s3 cp "s3://my-bucket/artifacts/app.tar.gz" "./app.tar.gz"

Fail-Fast Strategy and Structured Logging

Balancing resilience requires knowing when to fail fast versus when to recover gracefully:

Fail-Fast: For deterministic, irrecoverable errors (e.g., missing secrets, unreadable configs, full filesystem), abort immediately with descriptive error messages.
Graceful Fallback: When a secondary data source or cache is available, log a warning and proceed along the fallback execution path.

Standardized Severity Logging Framework

Standardize your script output with ISO 8601 timestamps and appropriate file descriptor routing (stdout for informational logs, stderr for warnings/errors):

1
2
3
4
log_info()  { echo "[$(date -u +'%Y-%m-%dT%H:%M:%SZ')] [INFO]  $*" ; }
log_warn()  { echo "[$(date -u +'%Y-%m-%dT%H:%M:%SZ')] [WARN]  $*" >&2 ; }
log_error() { echo "[$(date -u +'%Y-%m-%dT%H:%M:%SZ')] [ERROR] $*" >&2 ; }
log_fatal() { echo "[$(date -u +'%Y-%m-%dT%H:%M:%SZ')] [FATAL] $*" >&2 ; exit 1 ; }

Practical Example: Resilient Database Backup and Sync Script

The following complete DevOps script ties together Strict Mode, concurrency locking, signal trapping, automated retry synchronization, and custom exit status codes:

  1
  2
  3
  4
  5
  6
  7
  8
  9
 10
 11
 12
 13
 14
 15
 16
 17
 18
 19
 20
 21
 22
 23
 24
 25
 26
 27
 28
 29
 30
 31
 32
 33
 34
 35
 36
 37
 38
 39
 40
 41
 42
 43
 44
 45
 46
 47
 48
 49
 50
 51
 52
 53
 54
 55
 56
 57
 58
 59
 60
 61
 62
 63
 64
 65
 66
 67
 68
 69
 70
 71
 72
 73
 74
 75
 76
 77
 78
 79
 80
 81
 82
 83
 84
 85
 86
 87
 88
 89
 90
 91
 92
 93
 94
 95
 96
 97
 98
 99
100
101
102
103
104
105
106
107
108
109
110
111
112
#!/usr/bin/env bash
# ==============================================================================
# Script: backup-and-sync.sh
# Purpose: Resilient data backup and remote synchronization with auto-recovery
# ==============================================================================

set -euo pipefail
IFS=$'\n\t'

# --- Exit Code Constants ---
readonly EXIT_OK=0
readonly EXIT_LOCK_FAILED=20
readonly EXIT_BACKUP_FAILED=21
readonly EXIT_SYNC_FAILED=22

# --- Path Configurations ---
readonly SOURCE_DIR="/opt/production/data"
readonly LOCK_FILE="/tmp/backup-sync.lock"
readonly WORK_DIR=$(mktemp -d -t backup-stage-XXXXXX)
readonly TIMESTAMP=$(date +%Y%m%d_%H%M%S)
readonly ARCHIVE_NAME="backup-${TIMESTAMP}.tar.gz"

# --- Structured Logging ---
log_info()  { echo "[$(date +'%Y-%m-%d %H:%M:%S')] [INFO]  $*"; }
log_warn()  { echo "[$(date +'%Y-%m-%d %H:%M:%S')] [WARN]  $*" >&2; }
log_error() { echo "[$(date +'%Y-%m-%d %H:%M:%S')] [ERROR] $*" >&2; }

# --- Deterministic Cleanup ---
cleanup() {
    local status=$?
    log_info "Initiating cleanup routines..."
    
    if [[ -d "$WORK_DIR" ]]; then
        rm -rf "$WORK_DIR"
        log_info "Cleaned up temporary workspace: $WORK_DIR."
    fi
    
    if [[ -f "$LOCK_FILE" ]]; then
        rm -f "$LOCK_FILE"
        log_info "Released process lock ($LOCK_FILE)."
    fi
    
    if [[ $status -ne 0 ]]; then
        log_error "Backup workflow terminated with error code: $status"
    fi
}
trap cleanup EXIT SIGINT SIGTERM

# --- Retry Helper ---
retry_command() {
    local max_tries="$1"
    local wait_sec="$2"
    shift 2
    local cmd=("$@")
    
    local count=1
    until "${cmd[@]}"; do
        local code=$?
        if (( count >= max_tries )); then
            log_error "Command '${cmd[*]}' failed after $count attempts."
            return "$code"
        fi
        log_warn "Operation failed (attempt $count/$max_tries). Retrying in ${wait_sec}s..."
        sleep "$wait_sec"
        (( count++ ))
    done
    return 0
}

# --- Main Execution Flow ---
main() {
    log_info "Starting database backup process..."

    # 1. Concurrency check (Prevent overlapping runs)
    if [[ -f "$LOCK_FILE" ]]; then
        local running_pid
        running_pid=$(cat "$LOCK_FILE" 2>/dev/null || echo "")
        if [[ -n "$running_pid" ]] && kill -0 "$running_pid" 2>/dev/null; then
            log_error "Another backup process (PID: $running_pid) is actively running. Aborting!"
            exit "$EXIT_LOCK_FAILED"
        fi
    fi
    echo $$ > "$LOCK_FILE"

    # 2. Source path validation
    if [[ ! -d "$SOURCE_DIR" ]]; then
        log_error "Source directory $SOURCE_DIR does not exist!"
        exit "$EXIT_BACKUP_FAILED"
    fi

    # 3. Create compressed archive in scratch directory
    log_info "Archiving data from $SOURCE_DIR..."
    tar -czf "$WORK_DIR/$ARCHIVE_NAME" -C "$SOURCE_DIR" . || {
        log_error "Archive creation failed!"
        exit "$EXIT_BACKUP_FAILED"
    }
    log_info "Created archive: $ARCHIVE_NAME ($(du -h "$WORK_DIR/$ARCHIVE_NAME" | cut -f1))"

    # 4. Sync to remote backup destination with automatic retry
    log_info "Synchronizing archive to remote backup storage..."
    
    mkdir -p "/opt/remote-storage/backups"
    if ! retry_command 3 2 cp "$WORK_DIR/$ARCHIVE_NAME" "/opt/remote-storage/backups/"; then
        log_error "Failed to synchronize backup archive after multiple attempts!"
        exit "$EXIT_SYNC_FAILED"
    fi

    log_info "Archive successfully synchronized to remote storage."
    log_info "Backup workflow completed successfully!"
}

main "$@"

Deployment Notes & Best Practices

Keep Cleanup Handlers Simple: Logic within trap handlers must be lightweight and defensive. Always check for directory existence before calling rm -rf to prevent recursive errors within the handler itself.
Capture $? Immediately: When saving exit status (local status=$?), capture it as the very first operation inside error handlers. Any intermediate command (even echo) will overwrite $? with its own status.
Docker Signal Handling: When running scripts as container ENTRYPOINT, ensure the process runs as PID 1 (or wrap execution with tini / dumb-init) so SIGTERM signals from docker stop reach your shell trap cleanly.
Differentiate Stdout and Stderr: Always direct operational diagnostics and failure logs to >&2 (stderr). This ensures downstream data pipelines or subshell captures (output=$(script.sh)) receive only pure payload data on stdout.

Conclusion

Proactive exception handling is what distinguishes fragile scripts from production-ready automation software. By combining set -euo pipefail, deterministic trap resource cleanup, standardized error codes, and robust retry patterns, you can build reliable automation that withstands operational turbulence.

In the next article, we will examine Post 11 — Integrating Bash Scripts into CI/CD Pipelines: structuring secure, non-interactive workflows in GitHub Actions and GitLab CI.

TAGS:

#bash #shell-script #devops #error-handling #linux #automation #reliability

RESPONSES & DISCUSSION