Fix ‘Out of Memory: Killed Process’ — the Linux OOM Killer

A service dies with no error in its own logs. Hours later — or days later — you find the real culprit in the kernel log:

Out of memory: Killed process 31504 (apache2) total-vm:1482712kB, anon-rss:214380kB

That line is the OOM killer: the kernel’s last resort when physical memory and swap are both exhausted. It picked a process, sent it SIGKILL with no chance to clean up, and moved on. The process it killed is not necessarily the one that leaked — it is simply the one whose death would free the most memory. Fixing this properly means reading the kill report, finding the real consumer, and making sure the next spike bends the machine instead of breaking it.

sudo dmesg -T | grep -iE "out of memory|oom-killer" | tail -5

What the OOM Killer Actually Does

Linux overcommits memory: it lets processes allocate more than physically exists, betting they will not all use it at once. When the bet fails — every page of RAM in use, swap full or absent — the kernel cannot fulfil a memory request and has two options: freeze entirely, or kill something. It kills something.

Each process carries an oom_score, driven mostly by how much memory it holds. When the killer fires, the highest score dies. That is why the biggest single process — often a database — gets killed even when the memory was eaten by a swarm of smaller processes that individually score low.

Step 1: Confirm the Kill and Identify the Victim

The kernel log is the source of truth. With human-readable timestamps:

sudo dmesg -T | grep -iE "out of memory|oom-killer" | tail -5

If the ring buffer has rotated, ask the journal for kernel messages instead:

sudo journalctl -k --since "3 days ago" | grep -iE "out of memory|oom"

For a service that mysteriously stopped, systemd records the cause directly:

systemctl status apache2 --no-pager

An OOM-killed unit shows Active: failed (Result: oom-kill). Note the timestamp — a unit can sit in this state for days while the rest of the machine looks perfectly healthy, which is exactly why an OOM kill on a quiet server often goes unnoticed until someone asks why the site is down.

Step 2: Read the Kill Report

Right above the “Killed process” line, the kernel dumps a table of every process at the moment of the kill:

sudo dmesg -T | grep -B 40 "Out of memory" | tail -45

The column that matters is rss — resident memory, in 4 KB pages. Multiply by 4 for kilobytes. Scan it and ask one question: was the memory eaten by one huge process, or by many copies of the same program?

The distinction decides your fix. One 900 MB Java process on a 1 GB host is a heap-size problem. Forty Apache workers at 70 MB each is a worker-limit problem — no single worker looks alarming, but together they are 2.8 GB trying to fit where it cannot. Death by a thousand workers is the classic pattern on small VPSes, and it is usually triggered by a traffic spike or a bot wave hitting an expensive endpoint like a login page.

Step 3: Get the Service Back Up

An OOM-killed service does not come back on its own. Clear the failed state and start it:

sudo systemctl reset-failed apache2
sudo systemctl start apache2
systemctl is-active apache2

If it refuses to start for reasons beyond memory, that is a different investigation — our guide to systemd services that fail to start covers reading the unit’s journal properly.

Step 4: Fix What Ate the Memory

Web server worker limits

Apache’s prefork MPM defaults to MaxRequestWorkers 150. With mod_php, each worker holds roughly 60–90 MB once warm. On a 1 GB machine the safe budget is the arithmetic, not the default:

free -m
ps -o rss= -C apache2 | awk '{sum+=$1; n++} END {printf "%d workers, avg %d MB\n", n, sum/n/1024}'

Take about 60–70% of total RAM, divide by average worker size, and set that in /etc/apache2/mods-available/mpm_prefork.conf. On 1 GB that lands around 10–12 workers — requests queue briefly under load instead of the whole host collapsing. The same math applies to PHP-FPM’s pm.max_children and nginx unit workers.

Databases

MySQL’s InnoDB buffer pool defaults to 128 MB, but tuning guides written for big servers get pasted onto small ones. Check what yours is set to:

mysql -e "SELECT @@innodb_buffer_pool_size/1024/1024 AS buffer_mb;"

On a 1 GB host sharing space with a web server, 128–256 MB is realistic. Anything set to multiple gigabytes on such a machine is a standing invitation to the OOM killer.

Runaway or leaking applications

When one process genuinely leaks, cap it at the systemd level so the kernel kills only that unit, cleanly, instead of choosing a victim host-wide:

sudo systemctl edit myapp.service

Then in the override:

[Service]
MemoryMax=512M
Restart=on-failure
RestartSec=5s

The service now dies alone at 512 MB and comes back five seconds later — an incident that pages nobody instead of an outage.

Containers

A container with no memory limit competes with the host itself. Give every production container a ceiling (docker run --memory=512m, or resource limits in compose/Kubernetes). A container killed by its own limit exits with code 137 — that is the contained, correct version of this failure.

Add Swap as a Shock Absorber

Many VPS images ship with no swap at all, which means the gap between “memory pressure” and “processes dying” is zero. Check, then add 2 GB:

swapon --show
sudo fallocate -l 2G /swapfile
sudo chmod 600 /swapfile
sudo mkswap /swapfile
sudo swapon /swapfile
echo '/swapfile none swap sw 0 0' | sudo tee -a /etc/fstab

Keep the kernel from using it eagerly — swap should absorb spikes, not host your working set:

sudo sysctl vm.swappiness=10
echo 'vm.swappiness=10' | sudo tee -a /etc/sysctl.conf

Swap does not fix the underlying consumer. What it buys is time and signal: the machine slows down measurably instead of executing processes, and slow shows up in monitoring before dead does.

Verify the Fix

Watch memory behave under real load for a few minutes:

free -m
vmstat 5 6

In the vmstat output, si/so (swap in/out) should be zero or near it at rest. Then confirm the kernel has stayed quiet since your fix:

sudo dmesg -T | grep -i "out of memory" | tail -1

The last entry should predate your changes.

Prevent It From Recurring

Three habits close this class of incident out:

Auto-restart anything critical. Restart=on-failure in the unit turns a future kill into a blip. A service that can sit dead for a week is a monitoring gap wearing a systemd uniform.

Budget memory explicitly. Worker count × worker size + database buffers + OS overhead must fit inside physical RAM with room to spare. Defaults are written for bigger machines than yours.

Alert on pressure, not just on death. Any uptime monitor catches the outage; a free memory check (or simply alerting when swap usage grows) catches the week before it.

Frequently Asked Questions