Log rotation is not optional
A server I had not touched in months stopped accepting writes on a Sunday evening. The disk was full. Not the database, not uploads — container logs. Eleven gigabytes of them, from a service that logs one line per request and had been doing so, cheerfully, since spring.
The default is the problem
Docker's default logging driver keeps every line forever. There is no size cap, no age cap, and nothing warns you. On a machine with a fifty-gigabyte disk that is a countdown timer, and its length depends entirely on how chatty your application happens to be.
The fix, in one file
Four lines in /etc/docker/daemon.json, then restart the
daemon:
{
"log-driver": "json-file",
"log-opts": { "max-size": "10m", "max-file": "3" }
}That caps each container at thirty megabytes. Existing containers keep their old settings until they are recreated, which is the part people miss — set it and then actually redeploy, or nothing changes and you will be back here in the spring.
While you are there
Check journalctl --disk-usage too. The systemd journal has
its own cap, but the default on some images is generous enough to matter
on a small disk. SystemMaxUse=200M in
journald.conf has never once let me down.
None of this is clever. It is the kind of thing that is obvious in hindsight and invisible until the disk fills on a Sunday.