Josh Boyer <jwbo...@redhat.com> writes: > On Fri, Feb 08, 2013 at 01:19:49PM -0500, Josh Boyer wrote: >> On Thu, Feb 07, 2013 at 07:35:01PM -0500, Josh Boyer wrote: >> > On Thu, Feb 07, 2013 at 02:15:02PM -0800, Andrew Morton wrote: >> > > On Thu, 7 Feb 2013 16:57:42 -0500 >> > > Josh Boyer <jwbo...@redhat.com> wrote: >> > > >> > > > Hi All, >> > > > >> > > > We've hit a weird error in Fedora using the 3.8-rcX kernels. It seems >> > > > the mock tool is getting back ENOMEM when doing very simple things that >> > > > normally just work. The 3.7 kernels on the same userspace work just >> > > > fine. It seems just running 'mock init -v' is enough to cause the >> > > > failure. >> > > >> > > I assume you're not seeing the "page allocation failure" message and >> > > backtrace. This means that either >> > >> > Right. If I disable our debug options, I see no backtraces at all and >> > the python app still gets ENOMEM returned. (See below for those >> > interested). >> > >> > > a) it's a __GFP_NOWARN callsite. This is rare. Or >> > > >> > > b) it's actually a different error but someone went and overwrote a >> > > callee's return value with -ENOMEM. We do this a lot and it sucks. >> > >> > We do it in copy_io :\. >> > >> > > > At first glance it seems copy_io is failing (possibly because >> > > > get_task_io_context fails), and then the above fallout is printed. The >> > > > warning seems fairly valid, but I don't think that is the root of the >> > > > problem. >> > > >> > > yes, get_task_io_context() might be the place. Tried adding a few >> > > error-path printks in there to see what's happening? >> > >> > Yeah, that's my next step. I guess I know what I'll be doing tomorrow. >> > >> > > I can't see anything around there which leaves interrupts disabled >> > > though. It's quite likely that there's some code with is forgetting to >> > > reenable interrupts on a rarely-tested error path, and that ENOMEM is >> > > tickling the bug. >> > >> > Right, agreed. As I said, I think that is mostly a secondary issue. >> > Hopefully it will be easy to fix once we figure out why we're getting >> > the ENOMEM error. >> > >> > Python backtrace below. Seems to be failing on forking a umount command >> > after init'ing the chroot. I can put the full output somewhere if >> > people are interested. >> >> OK. I've bisected this down to: >> >> 50804fe3737ca6a5942fdc2057a18a8141d00141 is the first bad commit >> commit 50804fe3737ca6a5942fdc2057a18a8141d00141 >> Author: Eric W. Biederman <ebied...@xmission.com> >> Date: Tue Mar 2 15:41:50 2010 -0800 >> >> pidns: Support unsharing the pid namespace. >> >> >> I haven't really gotten much farther than that yet, but the bisect was >> pretty straight forward. Eric, is there anything specific I can gather >> or do to help figure out why that is causing mock to get such a weird >> error? I can provide the bisect log if you'd like. > > I took a look at what mock was doing and it was mostly very simple > stuff. The two exceptions were that it was calling unshare, then doing > some file checks and I/O, and then calling fork to exec off some helper > things. Up until the point it fails, the forks work and the children go > do whatever it is they were supposed to do. I've CC'd Clark Williams > just in case people have questions on mock itself, but I'm not sure that > will be needed.
Our emails crossed paths. You have just confirmed my suspicion about what was going wrong. The practical question is why mock is calling unshare(CLONE_NEWPID) because it clearly seems not to understand how to unshare the pid namespace and use it that way. Except for forgeting to reenable irqs in the failure path of alloc_pid the behavior is exactly correct and is how the pid namespace is designed to behave in the case of unshare. > which is consistent with what mock is seeing. If I comment out the call > to unshare, it seems to always work. It seems to consistently fail with > ENOMEM after the first 3-5 forked children, but it varies within that > range. If you add a waitpid or space out your forks you will see that it always fails after your first child in the pid namespace has exited. We don't allow children in a pid namespace after fork has exited. Eric -- To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
