Docker: No Space Left on Device — Reclaim Disk Safely

A build or pull dies partway through:

write /var/lib/docker/tmp/...: no space left on device

Docker accumulates. Every image layer you have ever pulled, every stopped container, every dangling build-cache entry stays on disk until something removes it. On a busy build host it is routine to find tens of gigabytes of material nothing references. The trick is clearing it without deleting a volume that holds a database.

docker system df

Step 1: Confirm It Is Actually Docker

Check overall disk usage first — the full filesystem may have nothing to do with Docker:

df -h /var/lib/docker

If Use% is at or near 100%, look at how Docker is spending it:

docker system df

You get a table of Images, Containers, Local Volumes, and Build Cache, each with a total size and a RECLAIMABLE column. That last column is the important one — it tells you how much you can recover without touching anything in use.

For detail on which specific items are large:

docker system df -v

One more possibility worth ruling out early: you can exhaust inodes while disk space remains. If df -h looks fine but writes still fail, check:

df -i /var/lib/docker

Step 2: Clear the Safe Things First

Start with the operation that cannot destroy data:

docker system prune

This removes stopped containers, unused networks, dangling images, and dangling build cache. It prompts before acting, and it leaves named volumes and tagged images alone. On most machines this alone recovers the majority of the space.

If the report showed build cache as the biggest consumer — common on CI runners and development laptops — target it directly:

docker builder prune

To clear only cache older than a week, keeping recent layers so your next build stays fast:

docker builder prune --filter until=168h

Step 3: Unused Images

Dangling images are untagged leftovers from rebuilds. Remove them alone:

docker image prune

To remove every image not currently used by a container, including tagged ones:

docker image prune -a

That second command deletes images you will have to pull again. On a machine with slow bandwidth or images built locally and never pushed to a registry, that can be expensive. Check what would go first:

docker images --format "table {{.Repository}}\t{{.Tag}}\t{{.Size}}" | sort -k3 -h

To clear images older than a month while keeping recent ones:

docker image prune -a --filter "until=720h"

Step 4: Volumes — Handle With Care

This is where people lose data. Named volumes hold databases, uploads, and anything else meant to outlive a container. Pruning them is irreversible.

List what exists and which are unreferenced:

docker volume ls -f dangling=true

Before deleting any of them, inspect what a volume contains:

docker run --rm -v SOME_VOLUME:/data alpine ls -la /data

A volume shows as dangling simply because no container currently references it — which is exactly what a stopped database’s volume looks like between deployments. Only after confirming a volume is genuinely disposable should you remove it by name:

docker volume rm SOME_VOLUME

Avoid docker system prune --volumes and docker volume prune on any machine holding real data. These delete every unreferenced volume at once, with no way back. The convenience is not worth the risk.

Step 5: Container Logs

A long-running container with a chatty application can produce a log file many gigabytes in size. Docker does not rotate these by default. Find the offenders:

sudo du -sh /var/lib/docker/containers/* | sort -h | tail -5

Fix it globally by configuring rotation in /etc/docker/daemon.json:

{
  "log-driver": "json-file",
  "log-opts": { "max-size": "10m", "max-file": "3" }
}

Validate the file before restarting, since a malformed daemon.json stops Docker from starting at all:

sudo python3 -m json.tool /etc/docker/daemon.json
sudo systemctl restart docker

Note this applies to newly created containers, not existing ones. Recreate long-running containers for the setting to take effect on them.

Verify the Fix

docker system df
df -h /var/lib/docker

Then prove the original operation works rather than assuming:

docker pull alpine:latest

If pulls succeed but builds still fail, the build cache is on a different filesystem — check docker info | grep "Docker Root Dir" to see where Docker actually stores data, which is not always /var/lib/docker.

Prevent It From Recurring

Add log rotation as above — it is the single highest-value change on any server running long-lived containers.

Schedule a conservative weekly cleanup that never touches volumes:

sudo tee /etc/cron.weekly/docker-prune > /dev/null <<'EOF'
#!/bin/sh
docker container prune -f --filter "until=168h"
docker image prune -f
docker builder prune -f --filter "until=168h"
EOF
sudo chmod +x /etc/cron.weekly/docker-prune

Every command there is scoped and safe: no -a, no --volumes. On build servers, giving /var/lib/docker its own partition means a runaway build fills that filesystem instead of taking the whole host down with it — a full root partition breaks logging, SSH, and package management all at once.

Frequently Asked Questions