Some of my docker containers were complaining that they didn’t have enough drive space. This looked odd – so i logged in to the host and checked around:

[email protected]:/$ df -h
Filesystem Size Used Avail Use% Mounted on
udev 2.0G 0 2.0G 0% /dev
tmpfs 394M 12M 382M 3% /run
/dev/sda1 26G 25G 0 100% /
tmpfs 2.0G 2.0M 2.0G 1% /dev/shm
tmpfs 5.0M 0 5.0M 0% /run/lock
tmpfs 2.0G 0 2.0G 0% /sys/fs/cgroup
//fs.cynexia.net/largeappdata 2.7T 285G 2.4T 11% /mnt/largeappdata
//fs.cynexia.net/video 5.4T 3.0T 2.4T 56% /mnt/video
//fs.cynexia.net/appdata 2.5T 52G 2.4T 3% /mnt/appdata

All space used up. Huh. Wonder why? I did a quick check to see what’s using most space:

[email protected]:/$ sudo du -xh / | grep '[0-9\.]\+G'
8.0K /var/lib/docker/aufs/diff/1dba1b90260105df03d0147c535c104cca0dd24fcc9273f0bc27b725c7cc676f/usr/local/crashplan/jre/lib/locale/zh.GBK/LC_MESSAGES
12K /var/lib/docker/aufs/diff/1dba1b90260105df03d0147c535c104cca0dd24fcc9273f0bc27b725c7cc676f/usr/local/crashplan/jre/lib/locale/zh.GBK
19G /var/lib/docker/aufs/diff
19G /var/lib/docker/aufs
2.2G /var/lib/docker/containers/8c5725f63f681e012fcc479e78133f31ab1c760b7d8d4e0a7e150d213face41f
2.3G /var/lib/docker/containers
21G /var/lib/docker
21G /var/lib
22G /var
25G /

clearly /var/lib/docker/aufs/diff is what’s causing it. Let’s clean that up:

[email protected]:/$ docker rmi $(docker images -aq --filter=dangling=true)
Untagged: mnbf9rca/[email protected]:9846b7570b5ba6d686be21623446cec8abd9db04cf55a39ce45cabfaa0d63f9f
Deleted: sha256:011bf974552570c536f8f98c73e0ed7d09ef9e2bfcbc7b3f3e02e19682b7480e
Deleted: sha256:a0637dd0588be6aee9f4655260176e6da802fcd92347cdf789ae84f3503322c3
Deleted: sha256:6e21a0999ad14a1cc0ccc8e31611b137793e3614338e01f920e13bfeb4128fdc
Deleted: sha256:b98c7813439119c3d2f859060fe11bf10151f69587f850a48448cae0fa4d9305
Untagged: mnbf9rca/[email protected]:ad493202d196dfae418769428ba6dea4d576ce1adec7ebe90837d0b965fe9b42
Deleted: sha256:b8df5a1ffa1eedd7be03d4a2a37549bf81699cc6fa1586c1d3510d90d4e9e562
...
Deleted: sha256:07c09e3cb65b3cec786933f882a08d5b0a34cd94f6922ada0d6f0cf779482ee0

Let’s check now…

[email protected]:/$ sudo du -xh / | grep '[0-9\.]\+G'
8.0K /var/lib/docker/aufs/diff/1dba1b90260105df03d0147c535c104cca0dd24fcc9273f0bc27b725c7cc676f/usr/local/crashplan/jre/lib/locale/zh.GBK/LC_MESSAGES
12K /var/lib/docker/aufs/diff/1dba1b90260105df03d0147c535c104cca0dd24fcc9273f0bc27b725c7cc676f/usr/local/crashplan/jre/lib/locale/zh.GBK
4.8G /var/lib/docker/aufs/diff
4.8G /var/lib/docker/aufs
2.2G /var/lib/docker/containers/8c5725f63f681e012fcc479e78133f31ab1c760b7d8d4e0a7e150d213face41f
2.3G /var/lib/docker/containers
7.4G /var/lib/docker
7.6G /var/lib
8.3G /var
11G /

much better! It turns out there are a few great cleanup agents e.g. docker-gc-cron which will do the job for me.