Marcelo Laia (HE12026-07-27): > I'm designing a logging system on Debian Testing and would appreciate > some advice regarding a failure scenario that I recently became aware > of. > > My initial idea is quite simple: > > - Debian Testing > - rsyslog > - Log files stored on an external USB SSD > - The root filesystem should remain as untouched as possible, while the > SSD receives all log writes. > > While discussing this design elsewhere, someone pointed out an issue > that I had not considered. > > If the USB SSD is unexpectedly disconnected, the USB bus resets, or the > filesystem becomes unmounted for any reason, the mount point directory > still exists. As I understand it, applications may continue writing to > that directory, which now belongs to the underlying root filesystem. On > systems using flash storage, this could eventually wear out the internal > storage without immediately being noticed. > > My first thought was to periodically check /proc/mounts (or > /proc/self/mountinfo) with a small script and immediately stop rsyslog > if the filesystem is no longer mounted. This would probably work, but it > feels like I might be reinventing something that Linux or Debian already > provides. > > So my questions are: > > 1. Is there a standard or recommended Debian/Linux approach to prevent > applications from silently writing to the underlying directory after > a mounted filesystem disappears? > 2. Are there existing mechanisms in systemd, mount units, udev, rsyslog, > or elsewhere that are intended to handle this situation?
systemd can guarantee that the rsyslog service will only run when a certain filesystem is mounted. You will need to find your way of twisty little directives (Requires, WanteBy, PartsOf…), all alike, but it is possible. But if the bus or drive has a problem while the filesystem is mounted, that will have no effect, as the filesystem does not get unmounted, it only goes into I/O error. IIRC, even the device itself does not disappear, so a systemd dependency on the device will not work. It is probable you get udev events when unplugging a device with a mounted filesystem (I do not want to test it right now on my computer), I suggest you run `udevadm monitor` and do the experiment. Then you can write a rule that stops the service and unmounts the filesystem. But then, you might lose logs that were sent during the event. And there are logs sent during the event, starting with the logs that tell you that the event happened and why. You want those. Rather than stopping rsyslog, you probably should mount a tmpfs over it and force syslog to reopen its files. Then you can handle the failure, repair the drive, remount it, suspend rsyslog, flush the tmpfs to disk and resume rsyslog. > 3. How would you implement a robust solution for this type of logging > system? Good question, but not an easy one. Maybe the kernel has some kind of fault-tolerant overlay filesystem. Otherwise, that requires thought. Regards, -- Nicolas George
