System Management with Bash: Monitoring Resources and Controlling Processes in DevOps

Sat, 29 Aug 2026 00:00:00 +0000

System Management with Bash in DevOps

In the previous post, we explored integrating Bash into CI/CD pipelines—from enforcing Strict Mode to protecting secrets and preventing Command Injection. Now, we return to the server itself—where Bash demonstrates its most immediate power: system management and resource monitoring.

When operating DevOps infrastructure, you cannot always sit in front of a screen watching servers. Sometimes a process consumes RAM abnormally, a disk fills up and disrupts log rotation, or a critical service crashes in the middle of the night. In these moments, a well-crafted Bash script can detect issues before you even receive an emergency call, and even automatically intervene to restore service.

This article covers the core commands for monitoring system resources (CPU, RAM, Disk, Network), managing processes and services, and includes a practical DevOps script with cron integration for continuous server monitoring.

System Resource Monitoring

CPU and System Load

The top command is the first tool you reach for when inspecting CPU activity:

1
2


# Run top once and exit (suitable for scripts)
top -bn1 | head -5

To extract the actual CPU usage percentage, use grep and awk:

1
2
3
4


# CPU usage (calculated from idle)
cpu_idle=$(top -bn1 | grep "Cpu(s)" | awk '{print $8}' | tr -d ',')
cpu_usage=$(echo "100 - $cpu_idle" | bc)
echo "CPU usage: ${cpu_usage}%"

Alternatively, use mpstat (from the sysstat package) for more precise measurement:

1
2


# Average CPU usage over 1 second
mpstat 1 1 | tail -1 | awk '{print "CPU usage: " 100 - $NF "%"}'

uptime provides a quick snapshot of system load average over the last 1, 5, and 15 minutes:

1

uptime | awk -F'load average:' '{print "Load average:" $2}'

Note on load average: If the load average exceeds the number of CPU cores, the system is overloaded. For example, a 4-core server with an average load of 5.2 indicates processes are waiting to be executed.

RAM and Swap

The free command displays memory status at a glance:

1

free -h

To calculate the percentage of RAM in use:

1
2


# RAM usage percentage
free | awk '/Mem:/ {printf "RAM: %.1f%% used\n", $3/$2*100}'

Check whether swap is being consumed—a warning sign when the system is running low on RAM:

1
2
3
4
5


# Check swap usage
swap_used=$(free | awk '/Swap:/ {print $3}')
if [[ "$swap_used" -gt 0 ]]; then
 echo "Warning: Using ${swap_used}KB of swap"
fi

Disk and Inode

df displays disk space utilization:

1
2


# Partition usage by percentage
df -h --output=target,pcent | tail -n +2

Detect partitions exceeding the 80% threshold:

1
2
3
4
5
6


while read -r mount usage; do
 pct=$(echo "$usage" | tr -d '%')
 if [[ "$pct" -ge 80 ]]; then
 echo "Warning: $mount is using $usage disk space"
 fi
done < <(df -h --output=target,pcent | tail -n +2)

du helps identify which directories consume the most space:

1
2


# Top 10 largest directories from /
du -h --max-depth=1 / 2>/dev/null | sort -rh | head -10

Check inodes—you may have free disk space but exhausted inodes:

1

df -i --output=target,ipcent | tail -n +2

Process Management

Listing and Finding Processes

ps is the most fundamental command for viewing running processes:

1
2
3
4
5


# List all processes sorted by CPU
ps aux --sort=-%cpu | head -15

# Find a process by name
ps aux | grep -E '[n]ginx' | awk '{print $2, $4"%CPU", $11}'

Tip: Use [n]ginx instead of nginx so grep does not match its own command line.

To retrieve the PID of a specific process:

1
2
3
4


# Get PID of the Nginx master process
pidof nginx
# Or
pgrep -f "nginx: master"

Controlling Processes

kill sends signals to processes. Understanding these signals enables precise intervention:

Signal	Number	Meaning	When to Use
SIGTERM	15	Graceful termination request	Try first; allows process to clean up resources
SIGKILL	9	Immediate termination	Use only when SIGTERM fails
SIGHUP	1	Reload configuration	Restart Nginx, Apache without stopping the process
SIGSTOP	19	Pause	Pause a process for investigation

1
2
3
4
5
6
7
8


# Attempt graceful termination first
kill "$PID"

# Force kill after 5 seconds if still alive
sleep 5
if kill -0 "$PID" 2>/dev/null; then
 kill -9 "$PID"
fi

pkill stops multiple processes matching a name pattern:

1
2


# Stop all old python processes (excluding current script)
pkill -f "python.*old_script"

Real-Time Process Monitoring

top or htop (if installed) provides real-time observation:

1
2


# Run top with 3 updates, 1 second apart
top -bn3 -d1 | head -30

If your server is CLI-only and you want a more user-friendly interface than top, htop is the popular choice:

1
2


# Install htop on Debian/Ubuntu
apt-get install -y htop

Service Management with systemctl

On modern Linux systems (systemd), systemctl is the primary command for controlling services:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11


# Check service status
systemctl status nginx

# Start / stop / restart
systemctl start nginx
systemctl stop nginx
systemctl restart nginx

# Enable / disable auto-start on boot
systemctl enable nginx
systemctl disable nginx

List all running services:

1

systemctl list-units --type=service --state=running

Find failed services:

1

systemctl list-units --state=failed

View service logs with journalctl:

1
2
3
4
5
6
7
8


# Last 50 lines of Nginx logs
journalctl -u nginx -n 50 --no-pager

# Real-time log streaming (similar to tail -f)
journalctl -u nginx -f

# Logs from 1 hour ago
journalctl -u nginx --since "1 hour ago"

Checking Ports and Network Connections

ss (Socket Statistics) is the modern replacement for netstat:

1
2
3
4
5
6
7
8


# List all listening ports
ss -tlnp

# Check a specific port (e.g., 80)
ss -tlnp | grep ':80'

# Connection summary by state
ss -s

lsof helps find which process is using a file or port:

1
2
3
4
5


# Find the process holding port 443
lsof -i :443

# Find which process has a specific file open
lsof /var/log/syslog

Practical Example: System Monitoring and Auto-Remediation Script

Below is a complete script combining the commands above, designed to run periodically via cron, detect issues, and automatically intervene when necessary:

 1
 2
 3
 4
 5
 6
 7
 8
 9
 10
 11
 12
 13
 14
 15
 16
 17
 18
 19
 20
 21
 22
 23
 24
 25
 26
 27
 28
 29
 30
 31
 32
 33
 34
 35
 36
 37
 38
 39
 40
 41
 42
 43
 44
 45
 46
 47
 48
 49
 50
 51
 52
 53
 54
 55
 56
 57
 58
 59
 60
 61
 62
 63
 64
 65
 66
 67
 68
 69
 70
 71
 72
 73
 74
 75
 76
 77
 78
 79
 80
 81
 82
 83
 84
 85
 86
 87
 88
 89
 90
 91
 92
 93
 94
 95
 96
 97
 98
 99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130


#!/usr/bin/env bash
# ==============================================================================
# Script : system-monitor.sh
# Purpose: Monitor system resources, alert, and auto-remediate
# Schedule via cron: */5 * * * * /opt/scripts/system-monitor.sh
# ==============================================================================

set -euo pipefail
IFS=$'\n\t'

readonly LOG_FILE="/var/log/system-monitor.log"
readonly MAX_LOG_SIZE=10485760 # 10MB
readonly CPU_THRESHOLD="${CPU_THRESHOLD:-90}"
readonly RAM_THRESHOLD="${RAM_THRESHOLD:-85}"
readonly DISK_THRESHOLD="${DISK_THRESHOLD:-80}"

log() {
 local level="$1"; shift
 echo "[$(date '+%Y-%m-%d %H:%M:%S')] [$level] $*" | tee -a "$LOG_FILE" 2>/dev/null

 # Rotate log if too large
 if [[ -f "$LOG_FILE" ]]; then
 local size
 size=$(stat -f%z "$LOG_FILE" 2>/dev/null || stat -c%s "$LOG_FILE" 2>/dev/null || echo 0)
 if [[ "$size" -gt "$MAX_LOG_SIZE" ]]; then
 mv "$LOG_FILE" "${LOG_FILE}.1"
 touch "$LOG_FILE"
 fi
 fi
}

# Check CPU — kill the top CPU consumer if threshold exceeded
check_cpu() {
 local cpu_idle cpu_usage top_pid
 cpu_idle=$(top -bn1 | grep "Cpu(s)" | awk '{print $8}' | tr -d ',' || echo "0")
 cpu_usage=$(echo "100 - $cpu_idle" | bc 2>/dev/null || echo "0")

 if [[ "$(echo "$cpu_usage > $CPU_THRESHOLD" | bc 2>/dev/null || echo 0)" -eq 1 ]]; then
 log "WARN" "CPU usage: ${cpu_usage}% (threshold: ${CPU_THRESHOLD}%)"

 # Get the highest CPU consuming PID
 top_pid=$(ps -eo pid,%cpu --sort=-%cpu | awk 'NR==2 {print $1}')
 if [[ -n "$top_pid" && "$top_pid" != "PID" ]]; then
 local top_cmd
 top_cmd=$(ps -p "$top_pid" -o comm= 2>/dev/null || echo "unknown")
 log "WARN" "Killing PID $top_cmd ($top_pid) — high CPU"

 # Try SIGTERM first, wait 5s then SIGKILL
 kill "$top_pid" 2>/dev/null || true
 sleep 5
 if kill -0 "$top_pid" 2>/dev/null; then
 kill -9 "$top_pid" 2>/dev/null || true
 log "WARN" "Force killed PID $top_pid"
 fi
 fi
 else
 log "INFO" "CPU: ${cpu_usage}% — OK"
 fi
}

# Check RAM
check_ram() {
 local ram_pct
 ram_pct=$(free | awk '/Mem:/ {printf "%.1f", $3/$2*100}')

 if [[ "$(echo "$ram_pct > $RAM_THRESHOLD" | bc 2>/dev/null || echo 0)" -eq 1 ]]; then
 log "WARN" "RAM usage: ${ram_pct}% (threshold: ${RAM_THRESHOLD}%)"

 # Drop page cache (safe, no data loss)
 sync
 echo 3 > /proc/sys/vm/drop_caches 2>/dev/null || true
 log "INFO" "Dropped page cache"
 else
 log "INFO" "RAM: ${ram_pct}% — OK"
 fi
}

# Check Disk
check_disk() {
 while read -r mount usage; do
 local pct
 pct=$(echo "$usage" | tr -d '%')
 if [[ "$pct" -ge "$DISK_THRESHOLD" ]]; then
 log "WARN" "Disk $mount usage: ${pct}% (threshold: ${DISK_THRESHOLD}%)"

 # Auto-clean compressed logs older than 30 days
 find /var/log -name "*.gz" -mtime +30 -delete 2>/dev/null || true
 log "INFO" "Cleaned old compressed logs (>30 days)"
 fi
 done < <(df -h --output=target,pcent 2>/dev/null | tail -n +2)
}

# Check critical services
check_services() {
 local critical_services=("nginx" "docker" "sshd")
 for svc in "${critical_services[@]}"; do
 if ! systemctl is-active --quiet "$svc" 2>/dev/null; then
 log "WARN" "Service $svc is not running — attempting restart"
 systemctl restart "$svc" 2>/dev/null || true
 sleep 2
 if systemctl is-active --quiet "$svc" 2>/dev/null; then
 log "INFO" "Service $svc restarted successfully"
 else
 log "ERROR" "Failed to restart service $svc"
 fi
 fi
 done
}

# Check critical ports
check_ports() {
 local critical_ports=(22 80 443)
 for port in "${critical_ports[@]}"; do
 if ! ss -tln | grep -q ":${port} " 2>/dev/null; then
 log "WARN" "Port $port is not listening"
 fi
 done
}

main() {
 log "INFO" "=== System monitoring started ==="
 check_cpu
 check_ram
 check_disk
 check_services
 check_ports
 log "INFO" "=== System monitoring completed ==="
}

main "$@"

Cron Configuration for Periodic Execution

1
2
3


# Run every 5 minutes
crontab -e
*/5 * * * * /opt/scripts/system-monitor.sh >> /var/log/system-monitor-cron.log 2>&1

Sample Output

1
2
3
4
5
6
7
8
9


[2026-08-29 10:00:01] [INFO] === System monitoring started ===
[2026-08-29 10:00:01] [INFO] CPU: 23.5% — OK
[2026-08-29 10:00:01] [INFO] RAM: 61.2% — OK
[2026-08-29 10:00:02] [WARN] Disk / usage: 87% (threshold: 80%)
[2026-08-29 10:00:02] [INFO] Cleaned old compressed logs (>30 days)
[2026-08-29 10:00:02] [INFO] Service nginx — running
[2026-08-29 10:00:02] [INFO] Service docker — running
[2026-08-29 10:00:02] [INFO] Service sshd — running
[2026-08-29 10:00:02] [INFO] === System monitoring completed ===

Deployment Notes & Best Practices

Avoid killing critical processes: Never kill -9 a process without fully identifying what it is. Always review ps aux output carefully before intervening. In the example script above, we only kill the highest CPU consumer—but in production, add a whitelist to protect important PIDs (databases, application servers).
Log rotation: The script auto-rotates logs when the file exceeds 10MB. In practice, integrate with logrotate for centralized log management and to retain necessary historical data.
Configurable thresholds: Use environment variables (CPU_THRESHOLD, RAM_THRESHOLD, DISK_THRESHOLD) instead of hardcoding values, so you can adjust per server without modifying the script.
Consider alternative solutions: Bash is well-suited for simple monitoring scripts. As your infrastructure grows (hundreds of servers), combine it with Prometheus + Grafana or dedicated monitoring tools—but Bash remains a fast choice for creating emergency intervention scripts.
Test before deployment: Always run scripts on staging or debug with bash -x before adding them to cron in production.

Conclusion

System management does not demand complex tools—often, a clean Bash script combining ps, top, free, df, systemctl, and ss is sufficient to keep servers running reliably. From monitoring CPU/RAM, checking disk usage, tracking service status, to automatically remediating detected issues, Bash remains a trustworthy companion for the DevOps Engineer.

In the next post, we will explore automating SSH with Bash: managing multiple servers simultaneously, executing remote commands, and transferring files securely with ssh, scp, and rsync.

Process-Management on Hoang Duong