I’ve got Pi-hole running in a Docker container on my Synology DS415+ for several years now. No issues at all.
However, after the recent 7.1.1 Update 9, suddenly, my local DNS entries in Pi-hole stopped getting resolved. It always resorted to external DNSes for those. As I host a few services at home, I’m using this feature to point devices inside my home network directly to the Traefik endpoint instead of routing everything into the Internet and back.
I’ve verified that the /etc/pihole/hosts/custom.list gets created fine inside the Docker container and contained the correct entries. Pi-hole itself was running just fine and did its thing. I was even able to update the local records from the GUI and they would pop up in the custom.list file. No error message. But whenever I tried to query one of those, I always got the public IP – not my internal one.
Digging deeper, I found this weird error message in the /var/log/pihole/pihole.log on every startup:
dnsmasq[55]: failed to create inotify for /etc/pihole/hosts: No space left on deviceCode language: HTTP (http)It was weird because /etc/pihole is a bind-mount and the underlying storage still had 2.2TB free space. And why would inotify need disk space anyways? Turns out, this is an effect of the “everything is a file” philosophy on Linux. There’s a limit of how many inotify watchers are allowed per user and as every watcher is basically an entry below /proc/, i.e. a “file”, when that watcher can’t be created, it seems to throw that error message about No space left on device.
And it seems to be that this limit has been vastly reduced with Synology’s latest update. Checking the /proc/sys/fs/inotify/max_user_watches showed a limit of only 1000 entries for me. And as I have several containers running on that box – including a SyncThing – I could see how those get used up very quickly.
And to make matters worse, the failure mode of Pi-hole in that case seems to be to not ingest the custom local DNS entries at all.
Speaking of SyncThing – that’s exactly the piece of software other Synology owners had issues with inotify as well. And the best solution (which also survives further OS updates) seems to be to create a small Scheduled Task that gets triggered on bootup and sets the limit to the desired value.
If it works, it works, I guess? And it does work!
And if you want to investigate WHICH process is occupying all those inotify slots, there’s this StackOverflow post.