Hm, lets have this SIG-Build meeting as scheduled and then have another follow-up later probably around 9am PST, 6pm Europe, 1am Singapore. Does that time work for everyone? (Date TBD).
My take on this whole thing is that it sounds a lot like re-implementing an entire distro complete with package manager inside pip just because pip is not sufficient for what we need. My longer term goal is to fix things up so TensorFlow can just be packaged directly in distro package repos and most users would go that route. This would definitely not be a universal solution and we'd still need to have a pip package anyway. I think we should leave CUDA out of the discussion initially and see if we can get the cpu-only wheel working correctly. Hopefully cpu-only is viable on manylinux2014 then we can tackle CUDA afterwards. On Tue, 5 Feb 2019 at 00:36, Uwe L. Korn <xho...@gmail.com> wrote: > > I think that problem is whether this would get merged. Conda was created > after a meeting with Guido van Rossum and other folks at a PyCon quite some > years ago where the final call was that this is not a problem of the core > Python ecosystem and that the scientific Python community has to roll their > own solution. > > @Wes McKinney or someone else: Were you at this meeting and can outline why > it was declined back then? > > Uwe > > Am Mo., 4. Feb. 2019 um 17:33 Uhr schrieb Manuel Klimek <kli...@google.com>: >> >> On Mon, Feb 4, 2019 at 5:32 PM Uwe L. Korn <xho...@gmail.com> wrote: >>> >>> Just as a heads-up: I would like to also join the meeting but am also >>> located in Europe. >>> >>> I have spent quite some time with the packaging of wheels for pyarrow and >>> turbodbc thus I would like to also give input on this. For Apache Arrow, I >>> see newer manylinux2014 standard as a possible way to go. I'm not so fond >>> of rolloing lib(std)c++ packages inside of pip. It's sadly the case that >>> the features of pip don't allow a good dependency resolution, also with >>> taking CUDA into account, a dependency resolution that differs between >>> source and binary builds of a package. For this case, exactly conda was >>> developed because it was considered out-of-scope for the core Python >>> packaging system. I'm not sure whether we actually can fit all the >>> requirements of the packages that take part in this mail thread into pip >>> without simply reimplementing conda inside of pip. >> >> >> One question is probably: what would that entail, and why would it be bad? :) >> >>> >>> >>> Uwe >>> >>> Am Mo., 4. Feb. 2019 um 16:34 Uhr schrieb Jason Zaman <ja...@perfinion.com>: >>>> >>>> yeah that's expected. The timing is complicated with people spread all >>>> over. We will post notes after the meeting on the SIG-Build mailing >>>> list and I'd also be up for organizing a separate call with europe >>>> folks if that would be of interest. >>>> >>>> On Mon, 4 Feb 2019 at 19:29, 'Manuel Klimek' via SIG Build >>>> <bu...@tensorflow.org> wrote: >>>> > >>>> > +Dmitri Gribenko >>>> > >>>> > Dmitri has experience with EasyBuild, which seems to be used by the HPC >>>> > community to solve the bootstrap problem and could be used to build a >>>> > toolchain image & pip package. >>>> > >>>> > Unfortunately we'll not be able to join the meeting as it's at midnight >>>> > CEST - looking forward to the conclusions from the meeting! >>>> > >>>> > On Mon, Feb 4, 2019 at 6:00 AM Jason Zaman <ja...@perfinion.com> wrote: >>>> >> >>>> >> Hey all, >>>> >> >>>> >> We're having the TensorFlow SIG-Build meeting on 5th Feb 3pm PST >>>> >> (https://www.timeanddate.com/worldclock/fixedtime.html?iso=20190205T15&p1=224). >>>> >> Agenda is linked from: >>>> >> https://groups.google.com/a/tensorflow.org/forum/#!topic/build/akyPcGoBIy4 >>>> >> >>>> >> I'd like to invite everyone from this thread to join the call if at >>>> >> all possible. The agenda for this meeting will spend most of the time >>>> >> focusing on the manylinux issue and hopefully we can get together to >>>> >> flesh out a decent plan on how to tackle this. >>>> >> >>>> >> Thanks, >>>> >> Jason >>>> >> >>>> >> On Wed, 30 Jan 2019 at 23:34, 'Manuel Klimek' via SIG Build >>>> >> <bu...@tensorflow.org> wrote: >>>> >> > >>>> >> > On Wed, Jan 30, 2019 at 4:21 PM Antoine Pitrou <anto...@python.org> >>>> >> > wrote: >>>> >> >> >>>> >> >> >>>> >> >> Le 30/01/2019 à 16:09, Manuel Klimek a écrit : >>>> >> >> > >>>> >> >> > On Wed, Jan 30, 2019 at 3:51 PM Antoine Pitrou <anto...@python.org >>>> >> >> > <mailto:anto...@python.org>> wrote: >>>> >> >> > >>>> >> >> > >>>> >> >> > Le 30/01/2019 à 15:35, Manuel Klimek a écrit : >>>> >> >> > > >>>> >> >> > > Am I reading you wrong, or are you actually proposing to >>>> >> >> > package another >>>> >> >> > > libstdc++ version as a Python wheel? >>>> >> >> > > >>>> >> >> > > >>>> >> >> > > That would be the idea. >>>> >> >> > > >>>> >> >> > > >>>> >> >> > > If so, are you going to claim that the given wheel is >>>> >> >> > > manylinux-compatible? >>>> >> >> > > >>>> >> >> > > >>>> >> >> > > That is my question :) Why wouldn't it be? (I'd link it >>>> >> >> > against >>>> >> >> > > manylinux libc and other C-only system libs) >>>> >> >> > >>>> >> >> > The problem is when you are loading two modules that link >>>> >> >> > against >>>> >> >> > different libstdc++ versions in the same process. >>>> >> >> > Incidentally, it's >>>> >> >> > the problem which prompted this discussion. >>>> >> >> > >>>> >> >> > >>>> >> >> > Sure, I'm aware :) I think as long as the requirement that all >>>> >> >> > libraries >>>> >> >> > that want to exchange runtime-ABI-compatible versions are compiled >>>> >> >> > with >>>> >> >> > the same toolchain, we can provide a way to mangle the symbols >>>> >> >> > differently. >>>> >> >> >>>> >> >> Ah, I see... Indeed, mangling the symbols may work for this. >>>> >> >> >>>> >> >> That said, if you're looking to create a de facto standard, why >>>> >> >> can't it >>>> >> >> be proposed as a manylinux iteration? >>>> >> > >>>> >> > >>>> >> > I'd have thought because it doesn't change the system requirements, >>>> >> > while manylinux seems to be all about system requirements. >>>> >> > The idea is that that toolchain would still work on any manylinux >>>> >> > compatible machine. >>>> >> > >>>> >> > >>>> >> > >>>> >> >> >>>> >> >> >>>> >> >> Regards >>>> >> >> >>>> >> >> Antoine. >>>> >> > >>>> >> > -- >>>> >> > You received this message because you are subscribed to the Google >>>> >> > Groups "SIG Build" group. >>>> >> > To unsubscribe from this group and stop receiving emails from it, >>>> >> > send an email to build+unsubscr...@tensorflow.org. >>>> >> > Visit this group at >>>> >> > https://groups.google.com/a/tensorflow.org/group/build/. >>>> > >>>> > -- >>>> > You received this message because you are subscribed to the Google >>>> > Groups "SIG Build" group. >>>> > To unsubscribe from this group and stop receiving emails from it, send >>>> > an email to build+unsubscr...@tensorflow.org. >>>> > Visit this group at >>>> > https://groups.google.com/a/tensorflow.org/group/build/.
