Re: Dealing with flit -- a simplified packaging of python modules
On Sat, Sep 26, 2015 at 5:21 AM, Donald Stufft wrote stuff Thanks for the summary, it sounds like the Python community is slowly moving towards a setup that is more closely aligned with Debian's values and setup. -- bye, pabs https://wiki.debian.org/PaulWise
Re: Dealing with flit -- a simplified packaging of python modules
On September 26, 2015 at 5:30:35 AM, Paul Wise (p...@debian.org) wrote: > On Fri, 2015-09-25 at 19:25 -0400, Donald Stufft wrote: > > > Because the way Python packaging currently is and historically has > > been, binary packages are not something that is widely available or > > viable. > > Hmm, I thought eggs have been around for ages (seems about 8 years)? > Eggs are problematic and hardly anyone publishes them. - Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA
Re: Dealing with flit -- a simplified packaging of python modules
On Fri, 2015-09-25 at 19:25 -0400, Donald Stufft wrote: > Because the way Python packaging currently is and historically has > been, binary packages are not something that is widely available or > viable. Hmm, I thought eggs have been around for ages (seems about 8 years)? -- bye, pabs https://wiki.debian.org/PaulWise signature.asc Description: This is a digitally signed message part
Re: Dealing with flit -- a simplified packaging of python modules
On September 25, 2015 at 8:21:39 PM, Ben Finney (ben+deb...@benfinney.id.au) wrote: > Donald Stufft writes: > > > On September 25, 2015 at 7:24:30 PM, Paul Wise (p...@debian.org) wrote: > > > Why are end users using source packages instead of binary packages > > > and then complaining that the source tarballs aren't ready-to-run > > > binary packages? > > > > Because the way Python packaging currently is and historically has > > been, binary packages are not something that is widely available or > > viable. > > So, commendable effort has gone toward making that problem reduce, > perhaps eventually to go away. Congratulations are deserved for getting > us this far, and thanks for continuing to push for feasible binary > distributions of Python packages on all supported OSen. > > It seems reasonable, then, to also put effort toward making source > distributions more targeted toward being bundles of the released > *source* for the project version, and reducing the justification for > bundling non-source files in the source distribution. I think distutils/setuptools is to blame for a lot of this, they don't make it easy to have generated files that aren't either compiled C code or py(c|o). Ideally in the future distutils/setuptools will no longer be special (except as a historical and legacy fallback) and we'll have something far more pluggable in the future. This will ideally make it a lot easier and sane to have custom build steps that create other generated files as part of the building of a package. > > Can we expect the value of “but people expect non-source stuff bundled > in the source distrubution” to diminish, as some kind of objection to > making proper source distributions? It's certainly a goal of mine that we move as much of the upstream Python packaging onto binary packages so that most people are not installing from sdists or really, "installing" from an sdist being an outdated concept and long term, the only thing we do with an sdist is turn it into a binary package, even if that build step is taken on the end user's computer. I of course, can't speak to everyone so it would not surprise me if some people simply will never be willing to not distributed generated files as part of their sdist. I suspect this to be a long road though, and certainly the nature of *nix makes it harder on us, since it's hard to make a binary package that works for say, all of Linux due to ABI problems and such. The more cases where ``pip install`` has end users pulling from a sdist, the more resistence I suspect you'll find from authors to making sdist have more complicated build steps. I suspect you'll have an easier time convinving authors of pure Python things that include some generated files (translations, js, whatever) to rely entirely on wheels than you will have convincing authors of Python things that include some C or other compiled code that precludes universal(ish) Wheels. - Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA
Re: Dealing with flit -- a simplified packaging of python modules
Donald Stufft writes: > On September 25, 2015 at 7:24:30 PM, Paul Wise (p...@debian.org) wrote: > > Why are end users using source packages instead of binary packages > > and then complaining that the source tarballs aren't ready-to-run > > binary packages? > > Because the way Python packaging currently is and historically has > been, binary packages are not something that is widely available or > viable. So, commendable effort has gone toward making that problem reduce, perhaps eventually to go away. Congratulations are deserved for getting us this far, and thanks for continuing to push for feasible binary distributions of Python packages on all supported OSen. It seems reasonable, then, to also put effort toward making source distributions more targeted toward being bundles of the released *source* for the project version, and reducing the justification for bundling non-source files in the source distribution. Can we expect the value of “but people expect non-source stuff bundled in the source distrubution” to diminish, as some kind of objection to making proper source distributions? -- \“If we ruin the Earth, there is no place else to go. This is | `\not a disposable world, and we are not yet able to re-engineer | _o__) other planets.” —Carl Sagan, _Cosmos_, 1980 | Ben Finney
Re: Dealing with flit -- a simplified packaging of python modules
On September 25, 2015 at 7:24:30 PM, Paul Wise (p...@debian.org) wrote: > On Thu, Sep 24, 2015 at 1:17 PM, Thomas Kluyver wrote: > > > That's my point ;-). From our upstream point of view, it's not a bug > > that the distributions we put on PyPI contain generated/bundled files - > > we do it that way deliberately, so that end users can install without > > needing Javascript developer tools to build those files. If you want a > > pure source tarball without generated files, that's available from > > Github. > > Why are end users using source packages instead of binary packages and > then complaining that the source tarballs aren't ready-to-run binary > packages? > Because the way Python packaging currently is and historically has been, binary packages are not something that is widely available or viable. - Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA
Re: Dealing with flit -- a simplified packaging of python modules
On Thu, Sep 24, 2015 at 1:17 PM, Thomas Kluyver wrote: > That's my point ;-). From our upstream point of view, it's not a bug > that the distributions we put on PyPI contain generated/bundled files - > we do it that way deliberately, so that end users can install without > needing Javascript developer tools to build those files. If you want a > pure source tarball without generated files, that's available from > Github. Why are end users using source packages instead of binary packages and then complaining that the source tarballs aren't ready-to-run binary packages? -- bye, pabs https://wiki.debian.org/PaulWise
Re: Dealing with flit -- a simplified packaging of python modules
On Thu, Sep 24, 2015, at 03:30 AM, Paul Wise wrote: > Source tarballs containing generated/bundled files is a bug that > should be fixed. That's my point ;-). From our upstream point of view, it's not a bug that the distributions we put on PyPI contain generated/bundled files - we do it that way deliberately, so that end users can install without needing Javascript developer tools to build those files. If you want a pure source tarball without generated files, that's available from Github. Thomas
Re: Dealing with flit -- a simplified packaging of python modules
On Sun, Sep 20, 2015 at 12:47 PM, Thomas Kluyver wrote: > However, my hope in that sentence was that other packaging will come not > to rely on Python sdists containing a setup.py file. Using sdists for > Debian packaging is already somewhat dubious, because they can contain > generated and bundled files (we do both for Jupyter Notebook sdists). Source tarballs containing generated/bundled files is a bug that should be fixed. -- bye, pabs https://wiki.debian.org/PaulWise
Re: Dealing with flit -- a simplified packaging of python modules
On Sat, Sep 19, 2015, at 09:36 AM, Barry Warsaw wrote: > There have been countless attempts at moving the Python packaging > infrastructure to a declarative syntax over the years. I remember > talking to > Tarek at a Pycon *many* years ago about this. Maybe this time it'll > catch > on. :) I think the introduction of wheels makes this goal more practical, as now there's a defined package format that tools can build, rather than relying on executing a script at install time. I'm also making the problem easier using the 80/20 rule: flit doesn't aim to replace all setup.py Python packaging. It's limited to a single module/package of pure Python code, which I think accounts for a large majority of what people do with setup.py. > flit doesn't build sdists, so I guess the current toolchain which starts > with > .orig.tar.gz won't work with flitted packages. The README says: > > "People may also want a traditional sdist for other reasons, such as > Linux > distro packaging. I hope that these problems will diminsh over time." > > I'm not sure which problem you hope will diminish! People wanting > traditional > sdists, the problem of Linux distro packaging or needing sdists > for > downstream consumers like Debian. There are certainly times when I wish distro packaging would focus on maintaining a much smaller set of core infrastructure packages really well. Take Jupyter, for instance: as upstream, we feel no investment in Linux distro packaging of it, and if we're helping people who got it from apt, we're likely to recommend that they uninstall that and use Anaconda or pip to install a recent version. I don't think distro packaging works for user-facing applications, and AFAIK it's not widely used for web app deployment either. However, my hope in that sentence was that other packaging will come not to rely on Python sdists containing a setup.py file. Using sdists for Debian packaging is already somewhat dubious, because they can contain generated and bundled files (we do both for Jupyter Notebook sdists). The way I'd like to see things working is for the canonical source for a release to be the tag in the VCS. Popular code hosting sites like Github make the source tree at this tag readily available as a downloadable tarball, e.g.: https://github.com/jupyter/testpath/archive/0.2.tar.gz The source tree contains metadata about the Python package in the repo. Different tools can use that to build wheels for PyPI, conda packages, Debian packages, or whatever. Of course, only the first of those is implemented yet, and it's no doubt more complex to build a .deb package, but that's where I'd like things to go. Thomas
Re: Dealing with flit -- a simplified packaging of python modules
On Sep 19, 2015, at 12:35 AM, Thomas Kluyver wrote: >By the way, I am also upstream for flit, and I'm prepared to help build >some tooling to use it for distro packaging. I know it will cause some >inconvenience in the short term because there's infrastructure around >setup.py packaging, but ultimately I think that having declarative >metadata should be an advantage. There have been countless attempts at moving the Python packaging infrastructure to a declarative syntax over the years. I remember talking to Tarek at a Pycon *many* years ago about this. Maybe this time it'll catch on. :) flit doesn't build sdists, so I guess the current toolchain which starts with .orig.tar.gz won't work with flitted packages. The README says: "People may also want a traditional sdist for other reasons, such as Linux distro packaging. I hope that these problems will diminsh over time." I'm not sure which problem you hope will diminish! People wanting traditional sdists, the problem of Linux distro packaging or needing sdists for downstream consumers like Debian. OTOH, as wheels get more popular for upstream package distributions, I do think we need a good story for turning wheels into debs. There's some work ongoing to turn debs back into wheels: https://github.com/paulproteus/dirtbike Cheers, -Barry
Re: Dealing with flit -- a simplified packaging of python modules
On Sat, Sep 19, 2015, at 01:05 AM, Julien Puydt wrote: > Yes, you're also upstream for ptyprocess and terminado :-P Guilty as charged ;-) I work for Jupyter/IPython, so there are several dependencies from that that I'm responsible for. > I have nothing against declarative -- it's just that I suspect we will > need something like a --buildsystem=flit or some such, and I have no > clue how to do something like this I don't know how to define a new buildsystem either. For the first package, I hope that we can do without that, at the cost of a longer Debian/rules file that specifies more stuff explicitly, and then work out what bits of that can be abstracted into tooling. Thomas
Re: Dealing with flit -- a simplified packaging of python modules
Hi, Le samedi 19 sept. 2015 à 00:35:49 (-0700), Thomas Kluyver a écrit : > On Fri, Sep 18, 2015, at 11:56 PM, Julien Puydt wrote: > > here is a new way to package modules for Python: > > https://github.com/takluyver/flit > > > > It means that something packaged using it doesn't use a setup.py or some > > such, but a flit.ini ; see for example: > > https://github.com/jupyter/testpath > > > > I'm not sure how to package something like this (and testpath is a > > depends for IPython's tests), so I think asking here is better. > > By the way, I am also upstream for flit, and I'm prepared to help build > some tooling to use it for distro packaging. I know it will cause some > inconvenience in the short term because there's infrastructure around > setup.py packaging, but ultimately I think that having declarative > metadata should be an advantage. > > Thomas > Yes, you're also upstream for ptyprocess and terminado :-P I have nothing against declarative -- it's just that I suspect we will need something like a --buildsystem=flit or some such, and I have no clue how to do something like this... Snark on #debian-python
Re: Dealing with flit -- a simplified packaging of python modules
On Fri, Sep 18, 2015, at 11:56 PM, Julien Puydt wrote: > here is a new way to package modules for Python: > https://github.com/takluyver/flit > > It means that something packaged using it doesn't use a setup.py or some > such, but a flit.ini ; see for example: > https://github.com/jupyter/testpath > > I'm not sure how to package something like this (and testpath is a > depends for IPython's tests), so I think asking here is better. By the way, I am also upstream for flit, and I'm prepared to help build some tooling to use it for distro packaging. I know it will cause some inconvenience in the short term because there's infrastructure around setup.py packaging, but ultimately I think that having declarative metadata should be an advantage. Thomas
Dealing with flit -- a simplified packaging of python modules
Hi, here is a new way to package modules for Python: https://github.com/takluyver/flit It means that something packaged using it doesn't use a setup.py or some such, but a flit.ini ; see for example: https://github.com/jupyter/testpath I'm not sure how to package something like this (and testpath is a depends for IPython's tests), so I think asking here is better. Snark on #debian-python
