Re: Read-only rootfs on systemd [SOLVED]
Thanks for your help and the replies. So this issue is now resolved. Summary of Issue: Mounting root as read-only as documented in (https://wiki.debian.org/ReadonlyRoot) because rootfs is busy. Summary of Solution: 1. 'lsof +L1' showed cupsd getting stuck on /etc/passwd (deleted). Looking at systemd journalctl, a custom service on this server was creating users blindly on every startup due to a bug in the code logic of this custom user management application. This caused /etc/passwd to keep changing. By fixing the bug, the /etc/passwd (deleted) was no longer an issue. 2. After reading the debian wiki on ReadOnlyRoot, there is a section on samba creating a file /etc/samba/dhcp.conf everytime dhclient was called. This server has a dual ethernet port with one port not plugged. As a result, dhclient kept trying to get a lease for this eth1 port and thus samba kept creating that file in /etc and rootfs was always busy. Not sure how this issue was solved but by adding the following line to /etc/fstab, I don't see the repeated dhclient messages anymore: tmpfs /tmptmpfs nodev,nosuid 0 0 -- To UNSUBSCRIBE, email to debian-user-requ...@lists.debian.org with a subject of "unsubscribe". Trouble? Contact listmas...@lists.debian.org Archive: https://lists.debian.org/loom.20140307t202848-...@post.gmane.org
Re: Read-only rootfs on systemd
On Thu 06 Mar 2014 at 23:05:20 +, Amit wrote: > 0) After reboot and running 'lsof +L1': > COMMAND PID USER FD TYPE DEVICE SIZE/OFF NLINK NODE NAME > cupsd935 root8r REG8,1 1392 0 132095 > /etc/passwd (deleted) I upgraded my wheezy install to jessie and cannot reproduce this. > 1) Shutting down cups: > sudo service cups stop > > No /etc/passwd in 'lsof +L1' output > > 2) Starting cups: > $ sudo /etc/init.d/cups start > [ ok ] Starting cups (via systemctl): cups.service. > > 3) No /etc/passwd in 'lsof +L1' output It would appear to be relevant to investigate why starting cups at boottime produces a different outcome. It might also be worthwhile using another init system to determine whether 'lsof +L1' gives the same as above. > 4) fuser output shows cupsd process using /etc/passwd but no output in lsof > showing '(deleted)'. I get that. -- To UNSUBSCRIBE, email to debian-user-requ...@lists.debian.org with a subject of "unsubscribe". Trouble? Contact listmas...@lists.debian.org Archive: https://lists.debian.org/07032014181859.8901992de...@desktop.copernicus.demon.co.uk
Re: Read-only rootfs on systemd
On Thu, Mar 06, 2014 at 11:05:20PM +, Amit wrote: > 0) After reboot and running 'lsof +L1': > COMMAND PID USER FD TYPE DEVICE SIZE/OFF NLINK NODE NAME > cupsd935 root8r REG8,1 1392 0 132095 > /etc/passwd (deleted) So it's reproducible. > 1) Shutting down cups: > sudo service cups stop > > No /etc/passwd in 'lsof +L1' output Since cupsd is the only one who writes in there, and you've just stopped cupsd - that's expected. > 2) Starting cups: > $ sudo /etc/init.d/cups start > [ ok ] Starting cups (via systemctl): cups.service. Oops. That's something I've forgot. I expected sysvinit compatibility layer to take care of cupsd starting. > > 3) No /etc/passwd in 'lsof +L1' output > > 4) fuser output shows cupsd process using /etc/passwd but no output in lsof > showing '(deleted)'. Weird. Just checked again, and on my Wheezy install nobody is using /etc/passwd. Are you running Jessie? > At this point remounting as 'ro' works. As it should be, as nobody is writing to the / filesystem now. As a workaround to all this, you probably can just restart cups on system's boot, and remount '/' read-only after that. As a real solution to the problem, I suggest you to fill a bug report to http://bugs.debian.org Reco -- To UNSUBSCRIBE, email to debian-user-requ...@lists.debian.org with a subject of "unsubscribe". Trouble? Contact listmas...@lists.debian.org Archive: https://lists.debian.org/20140307092141.GA26358@x101h
Re: Read-only rootfs on systemd
Reco gmail.com> writes: > > Can you do the following, please: > > 1) Shutdown cups by systemd's way (systemctl blahblah …). > > 2) Start it by /etc/init.d/cups start. > > 3) Confirm with lsof whenever /etc/passwd is kept open. > > 4) While you're at it, invoke 'fuser /etc/passwd' to ensure that nobody > else is using it. > > Reco > 0) After reboot and running 'lsof +L1': COMMAND PID USER FD TYPE DEVICE SIZE/OFF NLINK NODE NAME cupsd935 root8r REG8,1 1392 0 132095 /etc/passwd (deleted) 1) Shutting down cups: sudo service cups stop No /etc/passwd in 'lsof +L1' output 2) Starting cups: $ sudo /etc/init.d/cups start [ ok ] Starting cups (via systemctl): cups.service. 3) No /etc/passwd in 'lsof +L1' output 4) fuser output shows cupsd process using /etc/passwd but no output in lsof showing '(deleted)'. At this point remounting as 'ro' works. -- To UNSUBSCRIBE, email to debian-user-requ...@lists.debian.org with a subject of "unsubscribe". Trouble? Contact listmas...@lists.debian.org Archive: https://lists.debian.org/loom.20140307t000512-...@post.gmane.org
Re: Read-only rootfs on systemd
On Thu, 6 Mar 2014 18:35:06 + (UTC) Amit wrote: > Reco gmail.com> writes: > > > > > https://wiki.debian.org/ReadonlyRoot#cups says: > > > > CUPS stores any kind of state files under /etc (classes.conf, > > cupsd.conf, printers.conf subscriptions.conf) and upstream is against > > any modification. > > > > Personally I worked around similar problem by moving /etc/cups > > to /var/opt, and symlinking /var/opt/cups to /etc. > > > > Thanks for the reply. The problem doesn't seem to be the statefiles, it > seems to be '/etc/passwd'. I don't think I can safely move this to /var > right? IMO it's not safe indeed to move /etc/passwd somewhere else. But, I've checked cups on my print-server and it does not keep open /etc/passwd at all. And, given what cups should do, I find it unlikely that it should behave that way. Can you do the following, please: 1) Shutdown cups by systemd's way (systemctl blahblah …). 2) Start it by /etc/init.d/cups start. 3) Confirm with lsof whenever /etc/passwd is kept open. 4) While you're at it, invoke 'fuser /etc/passwd' to ensure that nobody else is using it. Reco -- To UNSUBSCRIBE, email to debian-user-requ...@lists.debian.org with a subject of "unsubscribe". Trouble? Contact listmas...@lists.debian.org Archive: https://lists.debian.org/20140306234036.27ebda625335038c18fb3...@gmail.com
Re: Read-only rootfs on systemd
Brian cityscape.co.uk> writes: > > On Thu 06 Mar 2014 at 01:21:03 +, Amit wrote: > > > I need cups, so is there a way around this? > > This doesn't answer your question but I have a spare Wheezy with > separate /, /home, and /var. I installed systemd, made the rootfs > ro in fstab and booted with init=/lib/systemd/systemd. The rootfs > was mounted ro. cupsd is also running. > > Thanks for testing. I am running jessie at the moment. I kept a backup of sysvinit as recommended by wiki.debian.org/systemd and used that. I see the same issue of cups using /etc/passwd (deleted). So doesn't seem to be a systemd issue, more of a cups issue I guess. -- To UNSUBSCRIBE, email to debian-user-requ...@lists.debian.org with a subject of "unsubscribe". Trouble? Contact listmas...@lists.debian.org Archive: https://lists.debian.org/loom.20140306t193742-...@post.gmane.org
Re: Read-only rootfs on systemd
Robin gmail.com> writes: > > Just a suggestion have you tried a re-install of cups since fresh > install of systemd > Thanks for the reply. Yes, the first thing I did was install systemd and then all the other packages but anyways I tried reinstalling again but no luck. -- To UNSUBSCRIBE, email to debian-user-requ...@lists.debian.org with a subject of "unsubscribe". Trouble? Contact listmas...@lists.debian.org Archive: https://lists.debian.org/loom.20140306t193534-...@post.gmane.org
Re: Read-only rootfs on systemd
Reco gmail.com> writes: > > https://wiki.debian.org/ReadonlyRoot#cups says: > > CUPS stores any kind of state files under /etc (classes.conf, > cupsd.conf, printers.conf subscriptions.conf) and upstream is against > any modification. > > Personally I worked around similar problem by moving /etc/cups > to /var/opt, and symlinking /var/opt/cups to /etc. > Thanks for the reply. The problem doesn't seem to be the statefiles, it seems to be '/etc/passwd'. I don't think I can safely move this to /var right? Thanks, Amit -- To UNSUBSCRIBE, email to debian-user-requ...@lists.debian.org with a subject of "unsubscribe". Trouble? Contact listmas...@lists.debian.org Archive: https://lists.debian.org/loom.20140306t193345-...@post.gmane.org
Re: Read-only rootfs on systemd
Hi. On Thu, 6 Mar 2014 14:49:30 + Brian wrote: > On Thu 06 Mar 2014 at 01:21:03 +, Amit wrote: > > > I need cups, so is there a way around this? > > This doesn't answer your question but I have a spare Wheezy with > separate /, /home, and /var. I installed systemd, made the rootfs > ro in fstab and booted with init=/lib/systemd/systemd. The rootfs > was mounted ro. cupsd is also running. https://wiki.debian.org/ReadonlyRoot#cups says: CUPS stores any kind of state files under /etc (classes.conf, cupsd.conf, printers.conf subscriptions.conf) and upstream is against any modification. Personally I worked around similar problem by moving /etc/cups to /var/opt, and symlinking /var/opt/cups to /etc. Reco -- To UNSUBSCRIBE, email to debian-user-requ...@lists.debian.org with a subject of "unsubscribe". Trouble? Contact listmas...@lists.debian.org Archive: https://lists.debian.org/20140306191540.8ed53c576ecd29e1043cb...@gmail.com
Re: Read-only rootfs on systemd
On 6 March 2014 01:21, Amit wrote: > Amit gmail.com> writes: > > [snip] > >> >> However, setting up a fresh install of systemd, the readonly does not >> have any effect. The rootfs is still mounted as rw. All I did was >> changed /etc/fstab. Based on the systemd man pages, this should be >> enough. >> >> How do I go about debugging/fixing this issue? >> > > The problem is cupsd. Doing an 'lsof +L1': > > COMMAND PID USER FD TYPE DEVICE SIZE/OFF NLINK NODE NAME > > cupsd 825 root8r REG8,1 1392 0 132009 /etc/passwd > (deleted) > > Stopping cups service and re-mounting as 'ro' now works. > > I need cups, so is there a way around this? > > Thanks, > Amit > > Just a suggestion have you tried a re-install of cups since fresh install of systemd -- rob -- To UNSUBSCRIBE, email to debian-user-requ...@lists.debian.org with a subject of "unsubscribe". Trouble? Contact listmas...@lists.debian.org Archive: https://lists.debian.org/CAOZWb-qJMcCFK5VF9S-m4UQ7T_Domh=psv2rttsxw29vnpp...@mail.gmail.com
Re: Read-only rootfs on systemd
On Thu 06 Mar 2014 at 01:21:03 +, Amit wrote: > I need cups, so is there a way around this? This doesn't answer your question but I have a spare Wheezy with separate /, /home, and /var. I installed systemd, made the rootfs ro in fstab and booted with init=/lib/systemd/systemd. The rootfs was mounted ro. cupsd is also running. -- To UNSUBSCRIBE, email to debian-user-requ...@lists.debian.org with a subject of "unsubscribe". Trouble? Contact listmas...@lists.debian.org Archive: https://lists.debian.org/06032014144345.2d9a7ff52...@desktop.copernicus.demon.co.uk
Re: Read-only rootfs on systemd
Amit gmail.com> writes: [snip] > > However, setting up a fresh install of systemd, the readonly does not > have any effect. The rootfs is still mounted as rw. All I did was > changed /etc/fstab. Based on the systemd man pages, this should be > enough. > > How do I go about debugging/fixing this issue? > The problem is cupsd. Doing an 'lsof +L1': COMMAND PID USER FD TYPE DEVICE SIZE/OFF NLINK NODE NAME cupsd 825 root8r REG8,1 1392 0 132009 /etc/passwd (deleted) Stopping cups service and re-mounting as 'ro' now works. I need cups, so is there a way around this? Thanks, Amit -- To UNSUBSCRIBE, email to debian-user-requ...@lists.debian.org with a subject of "unsubscribe". Trouble? Contact listmas...@lists.debian.org Archive: https://lists.debian.org/loom.20140306t021826-...@post.gmane.org
