Debugging Bash Scripts: Practical Troubleshooting and Error Detection for DevOps

Debugging Bash Scripts for DevOps

In the previous post, we covered automated task scheduling using cron. When a script runs in the background on remote servers or within CI/CD pipelines, silent failures with ambiguous logs are among the leading causes of production outages.

Unlike compiled languages or rich runtime environments (such as Go or Python), Bash by default tends to continue executing even when a command fails. A misspelled variable name, a missing target directory, or an undetected pipe error can turn an innocent maintenance script into an unintended data loss catastrophe.

This article covers comprehensive debugging techniques in Bash—from dry-run syntax validation and execution tracing, custom PS4 formatting, and Strict Mode enforcement, to automated trap ERR call stacks and static code analysis with ShellCheck.

Fast Syntax Validation with bash -n

Before running a script that modifies files or triggers infrastructure mutations, always validate its syntax first using the -n (noexec) flag:

1

bash -n deploy.sh

The -n flag reads and parses the script file to verify grammatical correctness (such as unclosed fi, missing braces }, or invalid case...esac branches) without executing any commands.

For example, given a script with an unclosed conditional block:

1
2
3
4
5

#!/usr/bin/env bash

if [ "$ENV" = "production" ]; then
    echo "Deploying to production..."
# Missing fi

Running the syntax check immediately flags the problem:

1
2

$ bash -n deploy.sh
deploy.sh: line 6: syntax error: unexpected end of file

In CI/CD pipelines, you can validate all shell scripts across a repository in one command:

1

find . -type f -name "*.sh" -exec bash -n {} +

Step-by-Step Execution Tracing with bash -x and set -x

The most widely used debugging technique in Bash is Execution Tracing via the -x (xtrace) flag. When enabled, Bash prints each command after performing parameter expansion, command substitution, and word splitting, right before execution.

Approach 1: Run the entire script in trace mode

1

bash -x script.sh [arguments]

Or add the -x flag directly to the Shebang line:

1

#!/usr/bin/env bash -x

Approach 2: Selectively toggle tracing within a script

If a script contains hundreds of lines and you only need to inspect a specific block, use the set -x (enable) and set +x (disable) pair:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12

#!/usr/bin/env bash

echo "Initializing deployment environment..."

# Trace only this specific critical section
set -x
target_dir="/opt/app/releases/$(date +%Y%m%d)"
mkdir -p "$target_dir"
cp -r ./dist/* "$target_dir/"
set +x

echo "File transfer completed."

When executed, only lines between set -x and set +x are traced with the + prefix:

1
2
3
4
5
6
7

Initializing deployment environment...
++ date +%Y%m%d
+ target_dir=/opt/app/releases/20260702
+ mkdir -p /opt/app/releases/20260702
+ cp -r ./dist/app.bin ./dist/config.json /opt/app/releases/20260702/
+ set +x
File transfer completed.

Enhancing Trace Output with the PS4 Prompt Variable

By default, Bash prints a simple + sign before each traced command. When your script invokes multiple nested functions or sources external libraries, identifying the exact file, function name, and line number becomes challenging.

The PS4 (Prompt String 4) environment variable allows you to customize the trace prefix. Key variables include:

Configure PS4 at the beginning of your script or pass it directly from the terminal:

1

export PS4='+ [${BASH_SOURCE[0]##*/}:${LINENO}] ${FUNCNAME[0]:+${FUNCNAME[0]}(): }'

Example demonstration:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15

#!/usr/bin/env bash
export PS4='+ [${BASH_SOURCE[0]##*/}:${LINENO}] ${FUNCNAME[0]:+${FUNCNAME[0]}(): }'
set -x

build_package() {
    local version="$1"
    echo "Packaging release version $version..."
}

main() {
    local app_version="v2.4.0"
    build_package "$app_version"
}

main

The resulting trace provides crystal-clear context for every line:

1
2
3
4
5

+ [build.sh:15] main(): local app_version=v2.4.0
+ [build.sh:16] main(): build_package v2.4.0
+ [build.sh:8] build_package(): local version=v2.4.0
+ [build.sh:9] build_package(): echo 'Packaging release version v2.4.0...'
Packaging release version v2.4.0...

Bash Strict Mode for Robust DevOps Automation

In DevOps workflows, silent failures are dangerous. Place standard Strict Mode configuration at the top of every automation script:

1
2

set -euo pipefail
IFS=$'\n\t'

Detailed breakdown of each flag:

• set -e (errexit): Immediately exits the script if any simple command returns a non-zero exit code.
• set -u (nounset): Treats unset variables as an error and exits immediately (preventing catastrophes like rm -rf "$UNSET_DIR/*" expanding to rm -rf "/*").
• set -o pipefail: By default, pipeline cmd1 | cmd2 | cmd3 only returns the exit code of cmd3. This option ensures that if any command fails in the pipeline, the overall pipeline returns the first non-zero status.
• IFS=$'\n\t': Sets the Internal Field Separator strictly to newlines and tabs, avoiding unintended word splitting on whitespace.

Handling Expected Non-Zero Exits with set -e

When using set -e, commands that naturally return non-zero codes on negative results (such as grep finding no matches) will terminate the script prematurely. Handle them cleanly:

1
2
3
4
5
6
7
8
9

# Approach 1: Append || true when failure is non-fatal
grep "ERROR" /var/log/app.log || true

# Approach 2: Wrap in an if conditional (set -e is suspended during test expressions)
if grep -q "ERROR" /var/log/app.log; then
    echo "Errors detected in log."
else
    echo "Log is clean."
fi

Catching Failures and Printing Stack Traces with trap ERR

To capture rich diagnostic context when unexpected failures occur, bind a custom error handler to the ERR signal via trap:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42

#!/usr/bin/env bash
set -euo pipefail

# Error handler that outputs a structured call stack
handle_error() {
    local exit_code="$1"
    local line_no="$2"
    local command="$3"
    
    echo "==========================================" >&2
    echo "CRITICAL ERROR DETECTED IN SCRIPT!" >&2
    echo "Failed Command : $command" >&2
    echo "Line Number    : $line_no" >&2
    echo "Exit Code      : $exit_code" >&2
    echo "Call Stack:" >&2
    
    local frame=0
    while caller $frame; do
        ((frame++))
    done >&2
    echo "==========================================" >&2
}

# Trap ERR signal to invoke handler
trap 'handle_error $? $LINENO "$BASH_COMMAND"' ERR

step_one() {
    echo "Executing step 1..."
}

step_two() {
    echo "Executing step 2: triggering failure..."
    ls /nonexistent-system-path-for-demo
}

main() {
    step_one
    step_two
    echo "Completed successfully."
}

main

When executed, failure at line 31 immediately produces actionable diagnostics:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13

Executing step 1...
Executing step 2: triggering failure...
ls: cannot access '/nonexistent-system-path-for-demo': No such file or directory
==========================================
CRITICAL ERROR DETECTED IN SCRIPT!
Failed Command : ls /nonexistent-system-path-for-demo
Line Number    : 31
Exit Code      : 2
Call Stack:
31 step_two script.sh
36 main script.sh
40 main script.sh
==========================================

Static Code Analysis with ShellCheck

ShellCheck is the premier static analysis tool (linter) for shell scripts. It identifies subtle logic flaws, unquoted variable expansions, POSIX portability traps, and syntax antipatterns.

Installing ShellCheck

1
2
3
4
5

# On Ubuntu / Debian
sudo apt-get update && sudo apt-get install -y shellcheck

# On macOS via Homebrew
brew install shellcheck

Running ShellCheck

1

shellcheck deploy.sh

For instance, given an unquoted variable in a critical command:

1
2
3

#!/usr/bin/env bash
filename=$1
rm -rf /tmp/data/$filename

ShellCheck immediately warns against globbing hazards:

1
2
3

In script.sh line 3:
rm -rf /tmp/data/$filename
                 ^-- SC2086 (info): Double quote to prevent globbing and word splitting.

Integrating ShellCheck into pre-commit hooks and CI pipelines guarantees baseline quality across your entire engineering organization.

Practical Example: A Resilient Deployment Script

Here is a production-ready DevOps deployment script incorporating dry-run compatibility, Strict Mode, dynamic PS4 tracing, trap ERR handling, and input validation:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55

#!/usr/bin/env bash
# ==============================================================================
# Script: deploy-service.sh
# Purpose: Safe application release automation with comprehensive debugging
# ==============================================================================

set -euo pipefail
IFS=$'\n\t'

# Enable verbose tracing only when DEBUG=1 is set
if [[ "${DEBUG:-0}" == "1" ]]; then
    export PS4='+ [${BASH_SOURCE[0]##*/}:${LINENO}] ${FUNCNAME[0]:+${FUNCNAME[0]}(): }'
    set -x
fi

# Cleanup on exit
cleanup() {
    local exit_code=$?
    if [[ $exit_code -ne 0 ]]; then
        echo "[ERROR] Deployment failed with exit code: $exit_code" >&2
    fi
}
trap cleanup EXIT

# Trap runtime command failures
handle_error() {
    local exit_code="$1"
    local line_no="$2"
    local cmd="$3"
    echo "[CRITICAL] Command '$cmd' at line $line_no failed (exit code: $exit_code)" >&2
}
trap 'handle_error $? $LINENO "$BASH_COMMAND"' ERR

# Validate arguments
APP_NAME="${1:-}"
RELEASE_VERSION="${2:-}"

if [[ -z "$APP_NAME" || -z "$RELEASE_VERSION" ]]; then
    echo "Usage: $0 <app_name> <release_version>" >&2
    exit 1
fi

DEPLOY_DIR="/opt/apps/$APP_NAME/releases/$RELEASE_VERSION"

echo "==> Deploying $APP_NAME version $RELEASE_VERSION..."
mkdir -p "$DEPLOY_DIR"

echo "==> Generating release metadata..."
echo "version=$RELEASE_VERSION" > "$DEPLOY_DIR/version.env"
echo "deployed_at=$(date -u +"%Y-%m-%dT%H:%M:%SZ")" >> "$DEPLOY_DIR/version.env"

echo "==> Switching current release symlink..."
ln -sfn "$DEPLOY_DIR" "/opt/apps/$APP_NAME/current"

echo "==> Deployment for $APP_NAME completed successfully!"

Execute in debug mode:

1

DEBUG=1 ./deploy-service.sh web-api v1.0.0

Deployment Notes & Best Practices

• Protecting Secrets Under Trace Mode: The set -x flag outputs all expanded variable values to the terminal/log stream. Disable tracing with set +x before reading passwords or tokens, and re-enable it only after sensitive operations finish.
• Isolating Debug Streams: In production or CI/CD, route xtrace output (file descriptor 2 / stderr) to dedicated debug log files without cluttering stdout: BASH_XTRACEFD=3 bash -x script.sh 3> /var/log/script-trace.log.
• Automate ShellCheck in CI: Integrate a ShellCheck step in GitHub Actions or GitLab CI to catch common syntax antipatterns before code merges.
• Pair set -e with Cleanup Traps: Halting on error with set -e is vital, but always attach a trap ... EXIT handler to clean up temporary files, lock files, and hanging network sockets.

Conclusion

Debugging is an essential discipline for transforming ad-hoc scripts into resilient, enterprise-grade automation tools. By combining bash -n dry runs, set -x execution tracing, customized PS4 prefixes, Strict Mode, and automated ShellCheck linting, you gain complete visibility and control over your Bash workloads.

In the next post, we will explore Post 10 — Advanced Error Handling in Bash: deep diving into custom exit codes, retry patterns, and automated resource cleanup.