From: Nathaniel Smith <[email protected]<mailto:[email protected]>> Date: Sunday, July 10, 2016 at 11:08 AM To: Vitaly Kruglikov <[email protected]<mailto:[email protected]>> Cc: "[email protected]<mailto:[email protected]>" <[email protected]<mailto:[email protected]>> Subject: Re: [Wheel-builders] error: signalfd.h not found when building manylinux wheel
On Jul 10, 2016 10:20 AM, "Vitaly Kruglikov" <[email protected]<mailto:[email protected]>> wrote: > > > > On 7/9/16, 11:15 PM, "Nathaniel Smith" > <[email protected]<mailto:[email protected]>> wrote: > > >On Fri, Jul 8, 2016 at 10:33 PM, Vitaly Kruglikov > ><[email protected]<mailto:[email protected]>> wrote: > >> > >> > >> On 7/8/16, 10:07 PM, "Nathaniel Smith" > >> <[email protected]<mailto:[email protected]>> wrote: > >> > >>>On Fri, Jul 8, 2016 at 3:23 PM, Vitaly Kruglikov > >>><[email protected]<mailto:[email protected]>> > >>>wrote: > >>>> I am attempting to build a manylinux wheel from the nupic.core project > >>>>in > >>>> github. I am using the docker image > >>>> quay.io/pypa/manylinux1_x86_64<http://quay.io/pypa/manylinux1_x86_64>. > >>>> nupic.core builds and statically links against the capnproto library, > >>>>which > >>>> relies on signalfd.h. Unfortunately, the docker image > >>>> quay.io/pypa/manylinux1_x86_64<http://quay.io/pypa/manylinux1_x86_64> > >>>> does not provide signalfd.h, so my > >>>>build > >>>> fails like this: > >>>> > >>>> Linking CXX static library libkj.a > >>>> [ 27%] Built target kj > >>>> [ 29%] Building CXX object src/kj/CMakeFiles/kj-async.dir/async.c++.o > >>>> [ 30%] Building CXX object > >>>>src/kj/CMakeFiles/kj-async.dir/async-unix.c++.o > >>>> > >>>>/nupic.core/build/scripts/ThirdParty/Source/CapnProto/src/kj/async-unix > >>>>.c > >>>>++:36:26: > >>>> fatal error: sys/signalfd.h: No such file or directory > >>>> #include <sys/signalfd.h> > >>>> > >>>> What is the recommended solution for this problem? > >>> > >>>man signalfd says: > >>> > >>>VERSIONS > >>> signalfd() is available on Linux since kernel 2.6.22. Working > >>>support > >>> is provided in glibc since version 2.8. The signalfd4() > >>>system > >>>call > >>> (see NOTES) is available on Linux since kernel 2.6.27. > >>> > >>>CentOS 5 ships glibc 2.5, so signalfd is simply not available to > >>>manylinux1 wheels. I guess the recommended solution is to modify your > >>>code so that it doesn't require signalfd? It looks you aren't the only > >>>person to run into this this week... > >>> > >>> https://groups.google.com/forum/#!topic/capnproto/OpH9RtOBcZU > >>> > >>>and the suggestion there is to use -DKJ_USE_EPOLL=0 to tell capnproto > >>>not to use signalfd. > >>> > >>>This is an interesting data point for the benefits of defining a > >>>CentOS-6-based manylinux2, though... > >>> > >>>-n > >>> > >>>-- > >>>Nathaniel J. Smith -- https://vorpus.org > >> > >> > >> Thanks Nathaniel, unfortunately it is not as simple as that. > >> Unfortunately, capnproto is not my code, so I am somewhat limited in > >>what > >> I can do with it. You also replied to a similar question concerning > >>pipe2, > >> SOCK_NONBLOCKING, etc. Those are actually all tied together. I also > >>tried > >> building capnproto with -DKJ_USE_EPOLL=0 to get around the signalfd > >> dependency, and that¹s what triggered the pipe2, SOCK_NONBLOCKING, etc. > >> was not declared in this scope compiler errors. It turns out that pipe2, > >> etc. are not available on CentOS-5 either. So, -DKJ_USE_EPOLL=0 or not, > >> the compilation fails due to missing headers or symbols in CentOS-5. > >> > >> I am now trying to create a patch for captnproto to enable the build to > >>go > >> through, but it is messy, as other parts of capnproto rely on some of > >>the > >> related headers. What a mess! > > > >Sorry to hear that :-/. > > > >Unfortunately, there's not really anything the manylinux1 image can do > >to help, because the manylinux1 spec says that manylinux1 wheels must > >work with CentOS 5, so by definition code that requires signalfd > >and/or pipe2 cannot be built into a manylinux1 wheel. > > > >If it helps, pipe2(pipefds, <something>) can be straightforwardly > >replaced by > > > > pipe(pipefds) > > fcntl(pipefds[0], F_SETFD, <something>) > > fcntl(pipefds[1], F_SETFD, <somthing>) > > > >(The pipe2 version is atomic with respect to forks, which is why it > >was added, but if you don't have pipe2 then the fcntl version is > >generally going to be fine outside of some fairly obscure cases.) > > > >Your other option is to lie and cheat :-). Technically you *can* build > >on CentOS 6 and then manually rename your .whl file to claim to be a > >manylinux1 wheel, and PyPI will accept it and pip will install it. > >(auditwheel will scream at you, but nothing forces you to listen to > >auditwheel.) Officially I can't recommend this. Unofficially, if you > >go ahead and do it anyway, then please let us know how it goes, > >because your user's reactions will be a very useful data point for > >designing the manylinux2 spec ;-). > > > >(A "manylinux1" wheel that was built on CentOS 6 should work for > >almost every user; AFAIK the one exception is users on CentOS 5. > >Unfortunately it turns out that we don't have any data on how many > >such users are out there [1], but they're certainly fairly rare...) > > > >-n > > > >[1] https://github.com/pypa/pip/pull/3836 > > > >-- > >Nathaniel J. Smith -- https://vorpus.org > > > Thanks, Nathaniel, for following up here and also in > https://github.com/pypa/manylinux/issues/75. I think that your suggestion > about using CentOS 6 would involve forking the > https://github.com/pypa/manylinux repo, replacing centos:5.11 with > centos:6 and building a new image, including running > build_scripts/build.sh, right? That would be a reasonable approach, yeah. You'll probably find you need to tweak a few other things, like I think the toolchain repo we're using is also centos5 specific. Good luck and let us know how it goes! -n Hi Nathaniel, here is the update you asked for: I ended up following your recommendation and created my own manylinux docker image from centos-6.8. by forking the manylinux github repo into Numenta’s github repo https://github.com/numenta/manylinux. The README there explains in more detail. After addressing the initial build issues, I ran into several additional problems that initially prevented cross-platform execution of the python extension: 1. Runtime symbol preemption and 2. c++ ABI incompatibility. The problem details are described in the comment https://discourse.numenta.org/t/segmentation-fault-while-running-basic-swarm/877/23?u=vkruglikov, and my solution is described in https://discourse.numenta.org/t/segmentation-fault-while-running-basic-swarm/877/24?u=vkruglikov. So far, after building nupic.bindings extension using my custom manylinux docker image (having solved those additional symbol preemption and c++ ABI compatibility problems), the wheel was able to pass all nupic tests on centos-6.8, Ubuntu 14.04, and Ubuntu 16.04. FYI and thanks for your support, Vitaly
_______________________________________________ Wheel-builders mailing list [email protected] https://mail.python.org/mailman/listinfo/wheel-builders
