[Bug 811823] Re: idmapd upstart job ends in an inconsistent state if /usr is a separate partition
This bug was fixed in the package nfs-utils - 1:1.2.2-1ubuntu1.2 --- nfs-utils (1:1.2.2-1ubuntu1.2) maverick-proposed; urgency=low * debian/nfs-common.idmapd.upstart: don't use a script unnecessarily for our job when we can exec directly - making the job more resilient in the face of races with /usr being mounted. LP: #811823. * Drop rpc_pipefs.conf; this has gotten far more complicated than it should be, just do the mount in-line in each of the gssd and idmapd jobs. -- Steve Langasek steve.langa...@ubuntu.com Sun, 17 Jul 2011 22:45:15 -0700 ** Changed in: nfs-utils (Ubuntu Maverick) Status: Fix Committed = Fix Released ** Changed in: nfs-utils (Ubuntu Lucid) Status: Fix Committed = Fix Released -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/811823 Title: idmapd upstart job ends in an inconsistent state if /usr is a separate partition To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/nfs-utils/+bug/811823/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 811823] Re: idmapd upstart job ends in an inconsistent state if /usr is a separate partition
** Tags removed: verification-done-lucid verification-needed -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/811823 Title: idmapd upstart job ends in an inconsistent state if /usr is a separate partition To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/nfs-utils/+bug/811823/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 811823] Re: idmapd upstart job ends in an inconsistent state if /usr is a separate partition
This bug was fixed in the package nfs-utils - 1:1.2.0-4ubuntu4.2 --- nfs-utils (1:1.2.0-4ubuntu4.2) lucid-proposed; urgency=low * debian/nfs-common.idmapd.upstart: don't use a script unnecessarily for our job when we can exec directly - making the job more resilient in the face of races with /usr being mounted. LP: #811823. * Drop rpc_pipefs.conf; this has gotten far more complicated than it should be, just do the mount in-line in each of the gssd and idmapd jobs. -- Steve Langasek steve.langa...@ubuntu.com Sun, 17 Jul 2011 22:51:42 -0700 -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/811823 Title: idmapd upstart job ends in an inconsistent state if /usr is a separate partition To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/nfs-utils/+bug/811823/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 811823] Re: idmapd upstart job ends in an inconsistent state if /usr is a separate partition
I have a report from a user on Natty that this resolved a boot time NFS race condition for them. This did the trick. I applied the 5.1 patch on the machine where we run our nfs client , upon rebooting the nfs client machine, the nfs share was properly mounted. ** Tags added: verification-done-natty -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/811823 Title: idmapd upstart job ends in an inconsistent state if /usr is a separate partition To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/nfs-utils/+bug/811823/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 811823] Re: idmapd upstart job ends in an inconsistent state if /usr is a separate partition
** Tags added: verification-done -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/811823 Title: idmapd upstart job ends in an inconsistent state if /usr is a separate partition To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/nfs-utils/+bug/811823/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 811823] Re: idmapd upstart job ends in an inconsistent state if /usr is a separate partition
This bug was fixed in the package nfs-utils - 1:1.2.2-4ubuntu5.1 --- nfs-utils (1:1.2.2-4ubuntu5.1) natty-proposed; urgency=low * debian/nfs-common.idmapd.upstart: don't use a script unnecessarily for our job when we can exec directly - making the job more resilient in the face of races with /usr being mounted. LP: #811823. * Drop rpc_pipefs.conf; this has gotten far more complicated than it should be, just do the mount in-line in each of the gssd and idmapd jobs. -- Steve Langasek steve.langa...@ubuntu.com Sun, 17 Jul 2011 22:21:57 -0700 ** Changed in: nfs-utils (Ubuntu Natty) Status: Fix Committed = Fix Released -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/811823 Title: idmapd upstart job ends in an inconsistent state if /usr is a separate partition To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/nfs-utils/+bug/811823/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 811823] Re: idmapd upstart job ends in an inconsistent state if /usr is a separate partition
** Tags removed: verification-done-natty -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/811823 Title: idmapd upstart job ends in an inconsistent state if /usr is a separate partition To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/nfs-utils/+bug/811823/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 811823] Re: idmapd upstart job ends in an inconsistent state if /usr is a separate partition
** Tags added: testcase -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/811823 Title: idmapd upstart job ends in an inconsistent state if /usr is a separate partition To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/nfs-utils/+bug/811823/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
Re: [Bug 811823] Re: idmapd upstart job ends in an inconsistent state if /usr is a separate partition
Hi Steve, On Thu, Sep 08, 2011 at 09:12:53PM -, Steve Atwell wrote: Steve, I'm looking at your change to gssd.conf and idmapd.conf, and it looks like you've got a race condition. You add the following to the pre-start script of both: if ! mountpoint -q $PIPEFS_MOUNTPOINT then mount -t rpc_pipefs rpc_pipefs $PIPEFS_MOUNTPOINT fi The problem is that the filesystem could get mounted between when you check with mountpoint and when you call mount. And because mount is the last command in the pre-start script, if it fails then the pre-start will fail, preventing the job from starting. I haven't actually seen this issue, but since two jobs are running this same code around the same time, it definitely seems like a possibility. Probably a simple || true on the mount command would be sufficient. If the mountpoint isn't actually there when rpc.gssd or rpc.idmapd starts, that process will exit and upstart will retry. You're right that this is not atomic; however, I believe the worst-case scenario is that the kernel fs will be mounted twice at the same mount point, which should not cause significant problems. (I have seen this outcome in practice on my machine while testing.) -- Steve Langasek Give me a lever long enough and a Free OS Debian Developer to set it on, and I can move the world. Ubuntu Developerhttp://www.debian.org/ slanga...@ubuntu.com vor...@debian.org -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/811823 Title: idmapd upstart job ends in an inconsistent state if /usr is a separate partition To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/nfs-utils/+bug/811823/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 811823] Re: idmapd upstart job ends in an inconsistent state if /usr is a separate partition
You're right that this is not atomic; however, I believe the worst-case scenario is that the kernel fs will be mounted twice at the same mount point, which should not cause significant problems. (I have seen this outcome in practice on my machine while testing.) Hm, that's not the behavior that I see. If I try to mount rpc_pipefs twice, mount exits with an error on the second attempt. root@satwelltest1:~# mount -t rpc_pipefs rpc_pipefs /var/lib/nfs/rpc_pipefs root@satwelltest1:~# mount -t rpc_pipefs rpc_pipefs /var/lib/nfs/rpc_pipefs mount: rpc_pipefs already mounted or /var/lib/nfs/rpc_pipefs busy mount: according to mtab, rpc_pipefs is already mounted on /var/lib/nfs/rpc_pipefs root@satwelltest1:~# echo $? 32 -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/811823 Title: idmapd upstart job ends in an inconsistent state if /usr is a separate partition To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/nfs-utils/+bug/811823/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 811823] Re: idmapd upstart job ends in an inconsistent state if /usr is a separate partition
Steve, I'm looking at your change to gssd.conf and idmapd.conf, and it looks like you've got a race condition. You add the following to the pre-start script of both: if ! mountpoint -q $PIPEFS_MOUNTPOINT then mount -t rpc_pipefs rpc_pipefs $PIPEFS_MOUNTPOINT fi The problem is that the filesystem could get mounted between when you check with mountpoint and when you call mount. And because mount is the last command in the pre-start script, if it fails then the pre-start will fail, preventing the job from starting. I haven't actually seen this issue, but since two jobs are running this same code around the same time, it definitely seems like a possibility. Probably a simple || true on the mount command would be sufficient. If the mountpoint isn't actually there when rpc.gssd or rpc.idmapd starts, that process will exit and upstart will retry. -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/811823 Title: idmapd upstart job ends in an inconsistent state if /usr is a separate partition To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/nfs-utils/+bug/811823/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 811823] Re: idmapd upstart job ends in an inconsistent state if /usr is a separate partition
Marking verification-done-lucid per this comment: https://bugs.launchpad.net/ubuntu/+source/nfs- utils/+bug/643289/comments/27 ** Tags added: verification-done-lucid -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/811823 Title: idmapd upstart job ends in an inconsistent state if /usr is a separate partition To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/nfs-utils/+bug/811823/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 811823] Re: idmapd upstart job ends in an inconsistent state if /usr is a separate partition
** Changed in: nfs-utils (Ubuntu Natty) Status: Fix Committed = Fix Released -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/811823 Title: idmapd upstart job ends in an inconsistent state if /usr is a separate partition To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/nfs-utils/+bug/811823/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 811823] Re: idmapd upstart job ends in an inconsistent state if /usr is a separate partition
** Changed in: nfs-utils (Ubuntu Natty) Status: Fix Released = Fix Committed -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/811823 Title: idmapd upstart job ends in an inconsistent state if /usr is a separate partition To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/nfs-utils/+bug/811823/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 811823] Re: idmapd upstart job ends in an inconsistent state if /usr is a separate partition
** Branch linked: lp:ubuntu/natty-proposed/nfs-utils -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/811823 Title: idmapd upstart job ends in an inconsistent state if /usr is a separate partition To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/nfs-utils/+bug/811823/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 811823] Re: idmapd upstart job ends in an inconsistent state if /usr is a separate partition
Hello Steve, or anyone else affected, Accepted nfs-utils into natty-proposed, the package will build now and be available in a few hours. Please test and give feedback here. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Thank you in advance! ** Changed in: nfs-utils (Ubuntu Natty) Status: In Progress = Fix Committed ** Tags added: verification-needed ** Changed in: nfs-utils (Ubuntu Maverick) Status: In Progress = Fix Committed -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/811823 Title: idmapd upstart job ends in an inconsistent state if /usr is a separate partition To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/nfs-utils/+bug/811823/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 811823] Re: idmapd upstart job ends in an inconsistent state if /usr is a separate partition
** Changed in: nfs-utils (Ubuntu Lucid) Status: New = Triaged ** Changed in: nfs-utils (Ubuntu Maverick) Status: New = Triaged ** Changed in: nfs-utils (Ubuntu Natty) Status: New = Triaged ** Changed in: nfs-utils (Ubuntu Lucid) Importance: Undecided = Medium ** Changed in: nfs-utils (Ubuntu Maverick) Importance: Undecided = Medium ** Changed in: nfs-utils (Ubuntu Natty) Importance: Undecided = Medium ** Changed in: nfs-utils (Ubuntu Lucid) Assignee: (unassigned) = Steve Langasek (vorlon) ** Changed in: nfs-utils (Ubuntu Maverick) Assignee: (unassigned) = Steve Langasek (vorlon) ** Changed in: nfs-utils (Ubuntu Natty) Assignee: (unassigned) = Steve Langasek (vorlon) -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/811823 Title: idmapd upstart job ends in an inconsistent state if /usr is a separate partition To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/nfs-utils/+bug/811823/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 811823] Re: idmapd upstart job ends in an inconsistent state if /usr is a separate partition
This bug was fixed in the package nfs-utils - 1:1.2.2-4ubuntu8 --- nfs-utils (1:1.2.2-4ubuntu8) oneiric; urgency=low * debian/nfs-common.idmapd.upstart: don't use a script unnecessarily for our job when we can exec directly - making the job more resilient in the face of races with /usr being mounted. LP: #811823. * Drop rpc_pipefs.conf; this has gotten far more complicated than it should be, just do the mount in-line in each of the gssd and idmapd jobs. -- Steve Langasek steve.langa...@ubuntu.com Sun, 17 Jul 2011 02:23:01 -0700 ** Changed in: nfs-utils (Ubuntu) Status: Triaged = Fix Released -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/811823 Title: idmapd upstart job ends in an inconsistent state if /usr is a separate partition To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/nfs-utils/+bug/811823/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 811823] Re: idmapd upstart job ends in an inconsistent state if /usr is a separate partition
** Branch linked: lp:ubuntu/nfs-utils -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/811823 Title: idmapd upstart job ends in an inconsistent state if /usr is a separate partition To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/nfs-utils/+bug/811823/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 811823] Re: idmapd upstart job ends in an inconsistent state if /usr is a separate partition
** Changed in: nfs-utils (Ubuntu Natty) Status: Triaged = In Progress -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/811823 Title: idmapd upstart job ends in an inconsistent state if /usr is a separate partition To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/nfs-utils/+bug/811823/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 811823] Re: idmapd upstart job ends in an inconsistent state if /usr is a separate partition
** Description changed: Because the idmapd upstart job uses a script line instead of directly exec'ing the server, in the case where /usr is a separate partition and not yet mounted at the time the job is started (e.g., because an NFS mount is attempted in parallel to an fsck of /usr), the job will fail to start and get wedged in a 'start/running' state with no associated PID (upstart bug #545673). If we exec the daemon directly, upstart will detect the exec() failure and respawn as needed. This partially addresses the problems described in bug #643289, and is worth SRUing in its own right even though it doesn't provide a complete solution for all users. + + SRU justification: race conditions in the nfs-utils upstart jobs cause + NFS client startup to be unreliable in certain configurations in lucid + and above, a regression vs. pre-upstart releases of Ubuntu. + + Although this upload includes the removal of the rpc_pipefs job (moving + its functionality into the gssd and idmapd jobs for simplicity), the + real risk of regression here is very small. There are no changes to the + start/stop conditions to the gssd or idmapd jobs, only a change to the + script rule for the idmapd job. This should have *no effect* except in + the case where /usr/sbin/rpc.idmapd is missing at the time the job runs + - i.e., in the case when /usr has not yet been mounted. + + TEST CASE: + 1. configure a system with /usr on a separate partition. + 2. install nfs-common from the release (or -updates) pocket. + 3. configure an NFSv4 mount in /etc/fstab to trigger idmapd to start before /usr is mounted. (The nfsv4 mount does not have to actually be mountable. Here is an example entry: + borges:/ /home/devel nfs4sec=krb5i,proto=tcp 0 0 + ) + 4. boot and verify with 'service idmapd' that the idmapd service is left in state 'start/running', with no associated process. + 5. install nfs-common from the -proposed pocket and reboot. + 6. verify with 'service idmapd' that the idmapd service has been successfully started, and is in state 'start/running' with an associated pid. -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/811823 Title: idmapd upstart job ends in an inconsistent state if /usr is a separate partition To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/nfs-utils/+bug/811823/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 811823] Re: idmapd upstart job ends in an inconsistent state if /usr is a separate partition
** Changed in: nfs-utils (Ubuntu Maverick) Status: Triaged = In Progress ** Changed in: nfs-utils (Ubuntu Lucid) Status: Triaged = In Progress -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/811823 Title: idmapd upstart job ends in an inconsistent state if /usr is a separate partition To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/nfs-utils/+bug/811823/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
