Re: Current state of packaging Python software for Debian
On Jul 04, 2011, at 08:20 AM, Robert Collins wrote: >unittest2 is still a unittest runner at heart; the basic model is >sound to scale up to N processes and so forth (see tox or >testrespository for instance), but compatibility with arbitrary ways >of running is pretty tricky. See for instance the guts of the >following three runners: trial, django and zc.testing.testrunner. > >All I'm saying is don't hold your breath: those communities could have >plugged into the original unittest compatibly but didn't - I think it >needs to be massively more clear *how* to do so, and on older Pythons >for those communities - they don't live on the edge ;) - for >one-runner-with-plugins to reach that point. > >As a data point, in the java world multiple runners is still the case, >with some common interop on output format. I'm actually fine with all that diversity. I'd just like to promote the interface `python setup.py test` (or `pysetup test`) as the preferred way of running the test suite for a package. It doesn't even have to run the full test suite (e.g like Python's own -uall flag), but it should provide a decent smoke test for any package. Sure, there will be packages that can't or won't support this standard, but I think the majority of packages can, and should. A carrot approach here would attach some kind of gold star to a package for those that have such a test suite and pass it at package build time. The stick approach would entail having the Debian tools default to running that command and FTBFS any package that fails its test suite. We can always special case the special cases. -Barry signature.asc Description: PGP signature
Re: Current state of packaging Python software for Debian
On Sat, Jul 2, 2011 at 2:07 AM, Barry Warsaw wrote: >>In some sub-communities, py.test or nosetests are the standard, not >>setup.py test. > > Yes, but if I understand where Michael is taking unittest2, those can just be > refactored to be plugins instead of separate standards. Then `python setup.py > test` can be the preferred and documented standards, while allowing those > other command lines to also exist. unittest2 is still a unittest runner at heart; the basic model is sound to scale up to N processes and so forth (see tox or testrespository for instance), but compatibility with arbitrary ways of running is pretty tricky. See for instance the guts of the following three runners: trial, django and zc.testing.testrunner. All I'm saying is don't hold your breath: those communities could have plugged into the original unittest compatibly but didn't - I think it needs to be massively more clear *how* to do so, and on older Pythons for those communities - they don't live on the edge ;) - for one-runner-with-plugins to reach that point. As a data point, in the java world multiple runners is still the case, with some common interop on output format. -Rob -- To UNSUBSCRIBE, email to debian-python-requ...@lists.debian.org with a subject of "unsubscribe". Trouble? Contact listmas...@lists.debian.org Archive: http://lists.debian.org/CAJ3HoZ1mOqYGcMfn0Lf514dBmmuBOD5GL78=mvfrewqxvsc...@mail.gmail.com
Re: Current state of packaging Python software for Debian
Hi, Le 01/07/2011 16:07, Barry Warsaw a écrit : > On Jun 15, 2011, at 05:07 PM, Éric Araujo wrote: >> Yes, last summer’s GSoC added a test command, which defaults to >> unittest(2) test discovery and can be configured to use any test runner >> on any test suite. It runs tests against the modules in the build >> directory, to be able to work with code converted at build time with >> 2to3 (and soon, to be able to access the PEP 376 dist-info files). >> Barry, I’m waiting for reports about the problems you ran into :) > IIRC, they were problems with doctests not getting properly converted, but I > forget the details at the moment. I'll try Python 3.3/packaging again soon > and definitely file bugs for anything I find. Can I use the Python tracker > for those? Certainly, using the Distutils2 component and 3.3 version. Tarek, Alexis and I will get auto-nosied. >> Regarding Sphinx, I don’t think it would be appropriate to add a command >> for it in the stdlib. We already have upload_docs, which can upload any >> set of HTML files. > Well, almost. I think there are some problems with that in Python 2, such as > if the Sphinx docs aren't laid out in the particular way expected by > `upload_docs`. E.g. say you don't have an index.html file but you do have a > README.html. upload_docs merely complies with the Cheeseshop rules: http://wiki.python.org/moin/PyPiHostingDocumentation >> However, it’s much easier to add custom command in packaging¹, so Sphinx’ >> distutils-based build² command can be used with pysetup. > That probably makes sense given that Sphinx is a separate project, but > whatever we can do to make it easy to use the de-facto standard, the better. > For example, `python setup.py build_sphinx` is a pretty lousy command name, > but that's Georg's problem. ;) OTOH, if packaging supports command > extensions, then `pysetup build` should Just Work to build the docs too, when > Sphinx is installed and docs are written in reST. Yes, automatic registration of commands and subcommands is one of the goals of Tarek. Cheers -- To UNSUBSCRIBE, email to debian-python-requ...@lists.debian.org with a subject of "unsubscribe". Trouble? Contact listmas...@lists.debian.org Archive: http://lists.debian.org/4e0f3163.6090...@netwok.org
Re: Current state of packaging Python software for Debian
On Jun 15, 2011, at 02:39 AM, Zygmunt Krynicki wrote: >I agree in the value but I don't want to assume that it is a default practice >or requirement. I package/hack on the webapp (django) side of tests and after >being spoiled with the goodness of python-testtools and python-testscenarios >I ventured to create equivalents (compatible wrappers really) for >django. This means that my packages build depend on 5 additional packages >just for testing. I really hope we could take an 80/20 approach here. IOW, do by default that which *most* packages will benefit from, which IMHO is installing the tests by default. Those packages which have additional dependencies, huge test data, or complicated tests can always add overrides to separate things into a -tests package. Or, make the common things trivial and the difficult things possible. :) -Barry signature.asc Description: PGP signature
Re: Current state of packaging Python software for Debian
On Jun 15, 2011, at 05:07 PM, Éric Araujo wrote: >Yes, last summer’s GSoC added a test command, which defaults to >unittest(2) test discovery and can be configured to use any test runner >on any test suite. It runs tests against the modules in the build >directory, to be able to work with code converted at build time with >2to3 (and soon, to be able to access the PEP 376 dist-info files). >Barry, I’m waiting for reports about the problems you ran into :) IIRC, they were problems with doctests not getting properly converted, but I forget the details at the moment. I'll try Python 3.3/packaging again soon and definitely file bugs for anything I find. Can I use the Python tracker for those? >Regarding Sphinx, I don’t think it would be appropriate to add a command >for it in the stdlib. We already have upload_docs, which can upload any >set of HTML files. Well, almost. I think there are some problems with that in Python 2, such as if the Sphinx docs aren't laid out in the particular way expected by `upload_docs`. E.g. say you don't have an index.html file but you do have a README.html. >However, it’s much easier to add custom command in packaging¹, so Sphinx’ >distutils-based build² command can be used with pysetup. That probably makes sense given that Sphinx is a separate project, but whatever we can do to make it easy to use the de-facto standard, the better. For example, `python setup.py build_sphinx` is a pretty lousy command name, but that's Georg's problem. ;) OTOH, if packaging supports command extensions, then `pysetup build` should Just Work to build the docs too, when Sphinx is installed and docs are written in reST. >In some sub-communities, py.test or nosetests are the standard, not >setup.py test. Yes, but if I understand where Michael is taking unittest2, those can just be refactored to be plugins instead of separate standards. Then `python setup.py test` can be the preferred and documented standards, while allowing those other command lines to also exist. Cheers, -Barry signature.asc Description: PGP signature
Re: Current state of packaging Python software for Debian
Apologies for the delayed response. On Jun 15, 2011, at 01:03 AM, Zygmunt Krynicki wrote: >W dniu 15.06.2011 00:04, Barry Warsaw pisze: >> On Jun 14, 2011, at 05:53 PM, Zygmunt Krynicki wrote: >> >>> Can please we have standardized hooks to build sphinx documentation and run >>> setup.py test tests? >> >> I'd like to see the packaging folks address this. Eric is subscribed to >> this list and can probably speak to packaging's take on it, but my >> preferences would be that >> >> $ pysetup test > >I have no practical knowledge of python3 (are we there yet?) so I cannot >comment but... Depends on where "there" is :). Yes, you can write real things in Python 3, and you can fairly easily write code that runs in both Python 3 and 2. The catch of course is whether your entire dependency stack is available for Python 3. We're only going to get better at that over time. > In Python< 3.3, using >> setuptools/distribute/distutils2, this should be the standard interface: >> >> $ python setup.py test > >I would also like this to become the de-facto standard. Can we somehow make >it happen? (debian policy, python something?)3.3). I think there are a couple of steps. First, make sure all your own code supports this standard, and encourage the authors of any code you package or run across to do the same. The other part is to strongly encourage use of this interface by supporting tools, in Python, Debian, Ubuntu and elsewhere. For example, I'd love to see a "quality measure" on PyPI that simply ran the tests using this interface and giving a GOOD/BAD/DUNNO state right on the main page. This way, we don't have to wield a heavy stick to get people to adopt the convention because developers will want to turn the red or yellow status to green. We could also hook the Debian build system to run `python setup.py test` on build by default and complain when that's missing, or fail when the tests don't pass. >Below I cut most of the discussion where we agree on sphinx documentation and >python. > >> In a setup.py world: >> >> $ python setup.py build_sphinx > >[cut] > >This is all fine and pretty (thanks to python). On the debian side I always >have to copy-paste the same-looking code over and over again (symlink jquery, >don't compress .js and .css, build-depend on sphinx and recommend jquery on >-doc package. Always the same boring and useless text in -doc-base files. All >begs for automation. Yep! >> Georg (the upstream Sphinx maintainer) makes a good point, which is that he >> really can't be expected to test Sphinx with any version of jquery than the >> one he ships. Operating systems (Debian/Ubuntu) are the integrators, and >> as Jakub points out in that issue, if Debian deviates from upstream by >> replacing Sphinx's version of jquery, it's incumbent on Debian to ensure it >> works properly. > >I agree, still, in the end there are only two possible choices: > >1) We stop replacing the bundled jquery and recommend this as best practice. > >2) We keep stripping jquery and replacing it with a symlink to libjs-jquery >but we make the process less cumbersome and manual. I think #2 will always be problematic. The advantage of the current approach is that it's easier to manage for security purposes. The disadvantage is that someone needs to ensure the same version of the library works across every package that links to it. Seems to me that could be an intractable problem. -Barry signature.asc Description: PGP signature
Re: Current state of packaging Python software for Debian
* Vincent Bernat , 2011-06-15, 07:31: On the debian side I always have to copy-paste the same-looking code over and over again (symlink jquery, If you use dh_link in your debian/rules (most likely you do), you need only a single line in debian/.links to do that. Don't you need to remove jquery.js from your package as well? Nope, dh_links does this for you as well. -- Jakub Wilk -- To UNSUBSCRIBE, email to debian-python-requ...@lists.debian.org with a subject of "unsubscribe". Trouble? Contact listmas...@lists.debian.org Archive: http://lists.debian.org/20110615220522.gc8...@jwilk.net
Re: Current state of packaging Python software for Debian
Le 15/06/2011 10:18, Zygmunt Krynicki a écrit : > W dniu 15.06.2011 03:28, Ben Finney pisze: If we are talking from a perspective of upstream developers that also maintain their packages then I would *love* to see setup.py sdist-test and would use it each day. >>> >>> ;-) >> >> How would a putative ‘sdist_test’ differ from ‘test’? Why is this an >> argument for a new command, and not an argument to improve what is done >> by ‘test’ anyway? > > Pure test runs the test on the check-out of the code and is usually > invoked by the upstream developer. It can (and often does) run on a > superset of files that are distributed with sdist). > > In contrast, the theoretical sdist_test would first create a release > tarball with sdist, unpack it to some temporary directory and run > `setup.py test` there. I’m sympathetic to the idea (one nit: it would be a new option to the test command, not a new command). It can be done today by manually unpacking sdist, going to that dir and running test, but wrapping that in a packaging/distutils2 command would not harm. > Such a command would be useful for checking that project manifest file > contains everything desired to make the program run correctly (I often > miss something and only realise it's missing when I start packaging). This has bitten so many people that I’m definitely +1 on doing something here. We already have a check command to perform some checks on the contents of setup.cfg (or setup.py in distutils 2.7 and 3.x); I’d bet a standard way to verify the sdist between build and upload would become a favorite command very fast. On the other hand, we cannot and should not try do too much in a packaging command; people want to check unpacking, run the test suite, check installation, and possibly other things which IMO fall under the scope of tools like tox. But trying to import all modules, checking the list of files and running tests are definitely things that we can discuss for the stdlib. On a related note, a virtualenv-style isolation feature is planned for Python 3.3. With the building blocks provided in the packaging module and that feature, it should be easy to make a tox-like tool to perform checks on sdists. Regards -- To UNSUBSCRIBE, email to debian-python-requ...@lists.debian.org with a subject of "unsubscribe". Trouble? Contact listmas...@lists.debian.org Archive: http://lists.debian.org/4df8cedc.4080...@netwok.org
Re: Current state of packaging Python software for Debian
Hi, [Barry Warsaw] > [Zygmunt Krynicki] >> Can please we have standardized hooks to build sphinx documentation and run >> setup.py test tests? > I'd like to see the packaging folks address this. Eric is subscribed to this > list and can probably speak to packaging's take on it, but my preferences > would be that > > $ pysetup test > > would run all the tests in Python 3.3 and beyond (and in a distutils2 world). > I don't think this is supported (yet?), and my limited testing required some > 2to3 hackery that isn't quite working. Yes, last summer’s GSoC added a test command, which defaults to unittest(2) test discovery and can be configured to use any test runner on any test suite. It runs tests against the modules in the build directory, to be able to work with code converted at build time with 2to3 (and soon, to be able to access the PEP 376 dist-info files). Barry, I’m waiting for reports about the problems you ran into :) Note that our implementation of test and 2to3 conversion is quite different from setuptools/distribute’s. For one thing, we use regular command options in the config file or on the command line, not global setup() arguments (and now setup() is gone anyway). I’m greatly responsible for that, because I believe it’s much cleaner. Regarding Sphinx, I don’t think it would be appropriate to add a command for it in the stdlib. We already have upload_docs, which can upload any set of HTML files. However, it’s much easier to add custom command in packaging¹, so Sphinx’ distutils-based build² command can be used with pysetup. ¹ http://docs.python.org/dev/packaging/setupcfg#global-options ² Why isn’t it named build_docs?! [Ben Finney] > AFAICT it *is* the de facto standard. Perhaps you mean that you want it > to be the de jure standard? What change are you wanting on this? In some sub-communities, py.test or nosetests are the standard, not setup.py test. Regards -- To UNSUBSCRIBE, email to debian-python-requ...@lists.debian.org with a subject of "unsubscribe". Trouble? Contact listmas...@lists.debian.org Archive: http://lists.debian.org/4df8cab9.8040...@netwok.org
Re: Current state of packaging Python software for Debian
On Wed, Jun 15, 2011 at 10:18:25AM +0200, Zygmunt Krynicki wrote: > W dniu 15.06.2011 03:28, Ben Finney pisze: > > >>>If we are talking from a perspective of upstream developers that > >>>also maintain their packages then I would *love* to see setup.py > >>>sdist-test and would use it each day. > >> > >>;-) > > > >How would a putative ‘sdist_test’ differ from ‘test’? Why is this an > >argument for a new command, and not an argument to improve what is done > >by ‘test’ anyway? > > Pure test runs the test on the check-out of the code and is usually > invoked by the upstream developer. It can (and often does) run on a > superset of files that are distributed with sdist). > > In contrast, the theoretical sdist_test would first create a release > tarball with sdist, unpack it to some temporary directory and run > `setup.py test` there. > > Such a command would be useful for checking that project manifest > file contains everything desired to make the program run correctly > (I often miss something and only realise it's missing when I start > packaging). > > IMHO it should be a separate command to ensure that 1) stuff that > works today keeps working 2) there is no performance penalty for > running setup.py test that would be required by setup.py sdist_test. I think what you really want to do is test an installation. I.e. run the tests from the source code on an installation. Note that the installation should not necessarily have to contain the tests. Theres some code and a spec out there to standardize this for Debian source/binary packages, see the recent debian-devel thread [1]. I've done this for some python-zope.* packages, but I need to use a very non-standard testrunner incantation to run the tests. I'm also forced to install the tests. Ideal would be some kind of standard: $ pysetup test-installation command which would run the tests from an unpacked sdist against an installation which is expected to already be on the python path. [1] http://lists.debian.org/debian-devel/2011/01/msg00813.html -- Brian Sutherland -- To UNSUBSCRIBE, email to debian-python-requ...@lists.debian.org with a subject of "unsubscribe". Trouble? Contact listmas...@lists.debian.org Archive: http://lists.debian.org/20110615091142.GF83513@Boo.local
Re: Current state of packaging Python software for Debian
W dniu 15.06.2011 03:28, Ben Finney pisze: If we are talking from a perspective of upstream developers that also maintain their packages then I would *love* to see setup.py sdist-test and would use it each day. ;-) How would a putative ‘sdist_test’ differ from ‘test’? Why is this an argument for a new command, and not an argument to improve what is done by ‘test’ anyway? Pure test runs the test on the check-out of the code and is usually invoked by the upstream developer. It can (and often does) run on a superset of files that are distributed with sdist). In contrast, the theoretical sdist_test would first create a release tarball with sdist, unpack it to some temporary directory and run `setup.py test` there. Such a command would be useful for checking that project manifest file contains everything desired to make the program run correctly (I often miss something and only realise it's missing when I start packaging). IMHO it should be a separate command to ensure that 1) stuff that works today keeps working 2) there is no performance penalty for running setup.py test that would be required by setup.py sdist_test. Best regards ZK -- To UNSUBSCRIBE, email to debian-python-requ...@lists.debian.org with a subject of "unsubscribe". Trouble? Contact listmas...@lists.debian.org Archive: http://lists.debian.org/4df86ad1.7020...@canonical.com
Re: Current state of packaging Python software for Debian
OoO En cette nuit striée d'éclairs du mercredi 15 juin 2011, vers 02:46, Jakub Wilk disait : >> On the debian side I always have to copy-paste the same-looking code >> over and over again (symlink jquery, > If you use dh_link in your debian/rules (most likely you do), you need > only a single line in debian/.links to do that. Don't you need to remove jquery.js from your package as well? -- Vincent Bernat ☯ http://www.luffy.cx Use free-form input when possible. - The Elements of Programming Style (Kernighan & Plauger) pgpH48B6xGw7Q.pgp Description: PGP signature
Re: Current state of packaging Python software for Debian
Yaroslav Halchenko writes: > On Wed, 15 Jun 2011, Zygmunt Krynicki wrote: > > Well joke aside you cannot fix the tarball that people release on > > pypi with half of the test code and data missing just because their > > MANIFEST.in was flaky. > > ;-) And that is when my evil "package from VCS" comes into play. For > some projects I package directly from a clone of upstream GIT (or even > gitsvn'ed upstream's SVN). And then I deal with actual sources, not > with possibly stripped or crippled source distribution, so I can fix > any issue like the one you brought up. What I do, and would recommend other packagers to do, is to have that situation prompt a discussion with upstream about fixing the broken tarball release. I see it as part of an OS packager's job to be an advocate on behalf of other OS packagers and users, informing/educating upstream about release practice improvements that can make their work more useable by the community. > Sure thing such approach is not universally-advisable: has its own > cons as well. Indeed. If an OS packager needs to start packaging from arbitrary VCS revisions instead of upstream-released tarballs with official version strings, I think that's a strong sign that the project's release process is (socially) broken and needs to be fixed upstream. > > If we are talking from a perspective of upstream developers that > > also maintain their packages then I would *love* to see setup.py > > sdist-test and would use it each day. > > ;-) How would a putative ‘sdist_test’ differ from ‘test’? Why is this an argument for a new command, and not an argument to improve what is done by ‘test’ anyway? -- \ “I do not believe in forgiveness as it is preached by the | `\church. We do not need the forgiveness of God, but of each | _o__)other and of ourselves.” —Robert G. Ingersoll | Ben Finney -- To UNSUBSCRIBE, email to debian-python-requ...@lists.debian.org with a subject of "unsubscribe". Trouble? Contact listmas...@lists.debian.org Archive: http://lists.debian.org/87oc1zsx1a@benfinney.id.au
Re: Current state of packaging Python software for Debian
W dniu 15.06.2011 02:46, Jakub Wilk pisze: * Zygmunt Krynicki , 2011-06-15, 01:03: In a setup.py world: $ python setup.py build_sphinx This is all fine and pretty (thanks to python). Only if you are happy that your extension modules are built in-place. :/ I did not use extension modules yes so I probably did not run into the problem. Out of curiosity (and future-proofing my packages), what is the good way of building those extension modules and why? In recent versions of debhelper, dh_compress doesn't touch these files anymore. Cool! build-depend on sphinx *sigh* Why sigh? and recommend jquery on -doc package. Why recommend? If you have a separate -doc package (hint: not every package have one), there's no good reason to leave jquery.js symlink dangling. AFAIR because it's really required to read the documentation. It is only used for the search feature. I don't remember the details but I found this pattern to be used by several existing packages. Always the same boring and useless text in -doc-base files. Care to propose an algorithm for generating them? It would help if there was an active user community that consumes those files somehow. I don't see much value to them TBH. We could simply use a placeholder text like "foobar-doc documents the python package python-foobar". Georg (the upstream Sphinx maintainer) makes a good point, which is that he really can't be expected to test Sphinx with any version of jquery than the one he ships. I missed this part earlier. Is it because jquery in sphinx is patched or is it because there is no good backwards compatibility between various jquery releases? Thanks ZK -- To UNSUBSCRIBE, email to debian-python-requ...@lists.debian.org with a subject of "unsubscribe". Trouble? Contact listmas...@lists.debian.org Archive: http://lists.debian.org/4df80520.7080...@canonical.com
Re: Current state of packaging Python software for Debian
Zygmunt Krynicki writes: > W dniu 15.06.2011 02:13, Ben Finney pisze: > >Zygmunt Krynicki writes: > >> 2) We keep stripping jquery and replacing it with a symlink to > >> libjs-jquery but we make the process less cumbersome and manual. Ah, I missed that you were implicitly referring only to Sphinx's use of jQuery. > I was referring to the seemingly identical code I see in my packages > that: > > 1) Build the documentation with sphinx > 2) Puts it into a separate package > 3) Replaces a copy of jquery with a symlink to libjs-jquery. > 4) Makes sure that .css and .js files are not compressed > > I'm only asking to make 1-4 a recommended practice for Debian packages > of python packages/modules that use sphinx for documentation. If we > can make 1-4 implicit (somehow) this would simplify a lot of things. That seems like a manageable improvement, thanks for retaining a good focus. -- \ “If history and science have taught us anything, it is that | `\ passion and desire are not the same as truth.” —E. O. Wilson, | _o__) _Consilience_, 1998 | Ben Finney -- To UNSUBSCRIBE, email to debian-python-requ...@lists.debian.org with a subject of "unsubscribe". Trouble? Contact listmas...@lists.debian.org Archive: http://lists.debian.org/87sjrbsxhy@benfinney.id.au
Re: Current state of packaging Python software for Debian
On Wed, 15 Jun 2011, Zygmunt Krynicki wrote: > >oops -- I bluntly believed that we are taking care about both > >aspects in Debian ;-) > Well joke aside you cannot fix the tarball that people release on > pypi with half of the test code and data missing just because their > MANIFEST.in was flaky. ;-) And that is when my evil "package from VCS" comes into play. For some projects I package directly from a clone of upstream GIT (or even gitsvn'ed upstream's SVN). And then I deal with actual sources, not with possibly stripped or crippled source distribution, so I can fix any issue like the one you brought up. Sure thing such approach is not universally-advisable: has its own cons as well. > In that sense we are not doing "release management" as there is no > equivalent of dist-check without being able to fix .orig.tar.gz. so... in that sense I do then ;-) moreover for some upstream projects upstream releases get synchronized with Debian -- quite usually I first check if everything is cool on my side and give 'Ok' (or commit few of such "release" fixes). And such approach is not conditioned on "package from upstream VCS" per se > If we are talking from a perspective of upstream developers that > also maintain their packages then I would *love* to see setup.py > sdist-test and would use it each day. ;-) -- =--= Keep in touch www.onerussian.com Yaroslav Halchenko www.ohloh.net/accounts/yarikoptic -- To UNSUBSCRIBE, email to debian-python-requ...@lists.debian.org with a subject of "unsubscribe". Trouble? Contact listmas...@lists.debian.org Archive: http://lists.debian.org/20110615011355.gs17...@onerussian.com
Re: Current state of packaging Python software for Debian
W dniu 15.06.2011 02:28, Yaroslav Halchenko pisze: On Wed, 15 Jun 2011, Zygmunt Krynicki wrote: That's different. IHMO you ask for make dist-check AFAIR (my automake-foo is getting old). Testing installed stuff is often harder and not supported as we don't want to include tests in the package (for build-deps vs install-deps). Source layout is also different from installed layout. Some data files required for testing are also not present in the production environment. valid points BUT so far the test batteries usually included with the python modules I package relied on reasonably small amount of data files, so upstream usually doesn't mind have them installed alongside with the tests installed alongside with the modules themselves. Benefits: any user can run tests later on (usually by "import m1; m1.test()") to verify that everything on his system is still functioning as promised (e.g. there were no upgrades of the 3rd party modules, breaking the functionality of the module in question; or effects of user-specific environment/settings/etc which could not be foreseeing during package building) I agree in the value but I don't want to assume that it is a default practice or requirement. I package/hack on the webapp (django) side of tests and after being spoiled with the goodness of python-testtools and python-testscenarios I ventured to create equivalents (compatible wrappers really) for django. This means that my packages build depend on 5 additional packages just for testing. In addition to that (due to the way django works) each django application I make (a python package really) has a helper django project (another python package) that allows it to invoke the test. In practice it means that: 1) Running tests is non-trivial. It works when you do it via setup.py test but it was not easy to get to that point. It's not something you can reproduce easily after installation without shipping the setup file somehow and (unfortunately) setup.py often uses find_packages() to discover where the code is. I did not check this but I suspect it might break in subtle ways as it was never intended to be used post-install. 2) Test code itself is large and has significant dependencies. Separating that from production deployments is important. Perhaps some python packages/modules are easy and have little or no extra data files but we cannot assume that's the case for all modules. I'm not trying to say "it's a bad idea", I want to figure out how to do it properly. Testing that python setup.py sdist output can still run tests is valuable but falls under release management - not packaging. oops -- I bluntly believed that we are taking care about both aspects in Debian ;-) Well joke aside you cannot fix the tarball that people release on pypi with half of the test code and data missing just because their MANIFEST.in was flaky. In that sense we are not doing "release management" as there is no equivalent of dist-check without being able to fix .orig.tar.gz. If we are talking from a perspective of upstream developers that also maintain their packages then I would *love* to see setup.py sdist-test and would use it each day. Thanks ZK -- To UNSUBSCRIBE, email to debian-python-requ...@lists.debian.org with a subject of "unsubscribe". Trouble? Contact listmas...@lists.debian.org Archive: http://lists.debian.org/4df7ff48.3000...@canonical.com
Re: Current state of packaging Python software for Debian
W dniu 15.06.2011 02:13, Ben Finney pisze: $ python setup.py test I would also like this to become the de-facto standard. Can we somehow make it happen? (debian policy, python something?)3.3). AFAICT it *is* the de facto standard. Perhaps you mean that you want it to be the de jure standard? What change are you wanting on this? You are right, I did me an "official" not "in practice", sorry about that. I guess I'm just asking, can we just do it? (flip a switch to make it official somehow). If not can we just start running `setup.py test` implicitly. It does nothing harmful when there is no test suite defined. I would say it's also incumbent on the jQuery developers to stop recommending that every application bundle the library leading to rampant proliferation of different versions and custom patches. But that's not something that will be fixed within the scope of this discussion. It's hard not to bundle unless you are only targeting Debian. I don't see anything that the upstream can recommend to make that better for us. 2) We keep stripping jquery and replacing it with a symlink to libjs-jquery but we make the process less cumbersome and manual. It would be good to work with the jQuery developers in some capacity on this. I don't see how jQuery upstream can help us with this. I was referring to the seemingly identical code I see in my packages that: 1) Build the documentation with sphinx 2) Puts it into a separate package 3) Replaces a copy of jquery with a symlink to libjs-jquery. 4) Makes sure that .css and .js files are not compressed I'm only asking to make 1-4 a recommended practice for Debian packages of python packages/modules that use sphinx for documentation. If we can make 1-4 implicit (somehow) this would simplify a lot of things. Thanks ZK -- To UNSUBSCRIBE, email to debian-python-requ...@lists.debian.org with a subject of "unsubscribe". Trouble? Contact listmas...@lists.debian.org Archive: http://lists.debian.org/4df7fd0e.7010...@canonical.com
Re: Current state of packaging Python software for Debian
* Zygmunt Krynicki , 2011-06-15, 01:03: In a setup.py world: $ python setup.py build_sphinx This is all fine and pretty (thanks to python). Only if you are happy that your extension modules are built in-place. :/ On the debian side I always have to copy-paste the same-looking code over and over again (symlink jquery, If you use dh_link in your debian/rules (most likely you do), you need only a single line in debian/.links to do that. don't compress .js and .css, In recent versions of debhelper, dh_compress doesn't touch these files anymore. build-depend on sphinx *sigh* and recommend jquery on -doc package. Why recommend? If you have a separate -doc package (hint: not every package have one), there's no good reason to leave jquery.js symlink dangling. Always the same boring and useless text in -doc-base files. Care to propose an algorithm for generating them? Georg (the upstream Sphinx maintainer) makes a good point, which is that he really can't be expected to test Sphinx with any version of jquery than the one he ships. Operating systems (Debian/Ubuntu) are the integrators, and as Jakub points out in that issue, if Debian deviates from upstream by replacing Sphinx's version of jquery, it's incumbent on Debian to ensure it works properly. As promised in <20110520200922.ga7...@jwilk.net>, this problem will be fixed for wheezy. Just give me some more time. :) I agree, still, in the end there are only two possible choices: 1) We stop replacing the bundled jquery and recommend this as best practice. Speaking with my Sphinx maintainer hat on, this is not going to happen. -- Jakub Wilk -- To UNSUBSCRIBE, email to debian-python-requ...@lists.debian.org with a subject of "unsubscribe". Trouble? Contact listmas...@lists.debian.org Archive: http://lists.debian.org/20110615004618.ga2...@jwilk.net
Re: Current state of packaging Python software for Debian
On Wed, 15 Jun 2011, Zygmunt Krynicki wrote: > That's different. IHMO you ask for make dist-check AFAIR (my > automake-foo is getting old). Testing installed stuff is often > harder and not supported as we don't want to include tests in the > package (for build-deps vs install-deps). Source layout is also > different from installed layout. Some data files required for > testing are also not present in the production environment. valid points BUT so far the test batteries usually included with the python modules I package relied on reasonably small amount of data files, so upstream usually doesn't mind have them installed alongside with the tests installed alongside with the modules themselves. Benefits: any user can run tests later on (usually by "import m1; m1.test()") to verify that everything on his system is still functioning as promised (e.g. there were no upgrades of the 3rd party modules, breaking the functionality of the module in question; or effects of user-specific environment/settings/etc which could not be foreseeing during package building) > Testing that python setup.py sdist output can still run tests is > valuable but falls under release management - not packaging. oops -- I bluntly believed that we are taking care about both aspects in Debian ;-) -- =--= Keep in touch www.onerussian.com Yaroslav Halchenko www.ohloh.net/accounts/yarikoptic -- To UNSUBSCRIBE, email to debian-python-requ...@lists.debian.org with a subject of "unsubscribe". Trouble? Contact listmas...@lists.debian.org Archive: http://lists.debian.org/20110615002849.gr17...@onerussian.com
Re: Current state of packaging Python software for Debian
Zygmunt Krynicki writes: > W dniu 15.06.2011 00:04, Barry Warsaw pisze: > In Python< 3.3, using > > setuptools/distribute/distutils2, this should be the standard > > interface: > > > > $ python setup.py test > > I would also like this to become the de-facto standard. Can we somehow > make it happen? (debian policy, python something?)3.3). AFAICT it *is* the de facto standard. Perhaps you mean that you want it to be the de jure standard? What change are you wanting on this? > > Georg (the upstream Sphinx maintainer) makes a good point, which is > > that he really can't be expected to test Sphinx with any version of > > jquery than the one he ships. Operating systems (Debian/Ubuntu) are > > the integrators, and as Jakub points out in that issue, if Debian > > deviates from upstream by replacing Sphinx's version of jquery, it's > > incumbent on Debian to ensure it works properly. I would say it's also incumbent on the jQuery developers to stop recommending that every application bundle the library leading to rampant proliferation of different versions and custom patches. But that's not something that will be fixed within the scope of this discussion. > I agree, still, in the end there are only two possible choices: > > 1) We stop replacing the bundled jquery and recommend this as best > practice. This is highly unlikely to satisfy the policies of the Debian security team. > 2) We keep stripping jquery and replacing it with a symlink to > libjs-jquery but we make the process less cumbersome and manual. It would be good to work with the jQuery developers in some capacity on this. -- \ “I've always wanted to be somebody, but I see now that I should | `\ have been more specific.” —Jane Wagner, via Lily Tomlin | _o__) | Ben Finney -- To UNSUBSCRIBE, email to debian-python-requ...@lists.debian.org with a subject of "unsubscribe". Trouble? Contact listmas...@lists.debian.org Archive: http://lists.debian.org/871uywt0ih@benfinney.id.au
Re: Current state of packaging Python software for Debian
W dniu 15.06.2011 01:06, Yaroslav Halchenko pisze: ideally I would like to see the helper which would allow (or even by default would do) to invoke test batteries against just installed (i.e. python setup.py install) version of the modules: in my experience it helped to avoid various issues of incomplete inclusion of submodules/data files. That's different. IHMO you ask for make dist-check AFAIR (my automake-foo is getting old). Testing installed stuff is often harder and not supported as we don't want to include tests in the package (for build-deps vs install-deps). Source layout is also different from installed layout. Some data files required for testing are also not present in the production environment. Testing that python setup.py sdist output can still run tests is valuable but falls under release management - not packaging. Best regards ZK -- To UNSUBSCRIBE, email to debian-python-requ...@lists.debian.org with a subject of "unsubscribe". Trouble? Contact listmas...@lists.debian.org Archive: http://lists.debian.org/4df7f2d9@canonical.com
Re: Current state of packaging Python software for Debian
W dniu 15.06.2011 00:04, Barry Warsaw pisze: On Jun 14, 2011, at 05:53 PM, Zygmunt Krynicki wrote: Can please we have standardized hooks to build sphinx documentation and run setup.py test tests? I'd like to see the packaging folks address this. Eric is subscribed to this list and can probably speak to packaging's take on it, but my preferences would be that $ pysetup test I have no practical knowledge of python3 (are we there yet?) so I cannot comment but... In Python< 3.3, using setuptools/distribute/distutils2, this should be the standard interface: $ python setup.py test I would also like this to become the de-facto standard. Can we somehow make it happen? (debian policy, python something?)3.3). Below I cut most of the discussion where we agree on sphinx documentation and python. In a setup.py world: $ python setup.py build_sphinx [cut] This is all fine and pretty (thanks to python). On the debian side I always have to copy-paste the same-looking code over and over again (symlink jquery, don't compress .js and .css, build-depend on sphinx and recommend jquery on -doc package. Always the same boring and useless text in -doc-base files. All begs for automation. Georg (the upstream Sphinx maintainer) makes a good point, which is that he really can't be expected to test Sphinx with any version of jquery than the one he ships. Operating systems (Debian/Ubuntu) are the integrators, and as Jakub points out in that issue, if Debian deviates from upstream by replacing Sphinx's version of jquery, it's incumbent on Debian to ensure it works properly. I agree, still, in the end there are only two possible choices: 1) We stop replacing the bundled jquery and recommend this as best practice. 2) We keep stripping jquery and replacing it with a symlink to libjs-jquery but we make the process less cumbersome and manual. Best regards ZK -- To UNSUBSCRIBE, email to debian-python-requ...@lists.debian.org with a subject of "unsubscribe". Trouble? Contact listmas...@lists.debian.org Archive: http://lists.debian.org/4df7e8a4.5090...@canonical.com
Re: Current state of packaging Python software for Debian
ideally I would like to see the helper which would allow (or even by default would do) to invoke test batteries against just installed (i.e. python setup.py install) version of the modules: in my experience it helped to avoid various issues of incomplete inclusion of submodules/data files. And sure thing the helper should run it against all supported by the package/system versions of python. On Tue, 14 Jun 2011, Barry Warsaw wrote: > 2to3 hackery that isn't quite working. In Python < 3.3, using > setuptools/distribute/distutils2, this should be the standard interface: > $ python setup.py test > and in fact that works quite well, even with 2to3 integration. You need to > add: > test_suite='path.to.testdir' > in your setup() call, and possibly a `convert_2to3_doctests` key as well (this > is the part that does not work for me in Python 3.3). -- .-. =-- /v\ = Keep in touch// \\ (yoh@|www.)onerussian.com Yaroslav Halchenko /( )\ ICQ#: 60653192 Linux User^^-^^[17] -- To UNSUBSCRIBE, email to debian-python-requ...@lists.debian.org with a subject of "unsubscribe". Trouble? Contact listmas...@lists.debian.org Archive: http://lists.debian.org/20110614230637.gh29...@novo.onerussian.com
Re: Current state of packaging Python software for Debian
On Jun 14, 2011, at 12:33 PM, Yaroslav Halchenko wrote: >ah -- wishlist! Well, just be careful. :) We have to clearly define what's the responsibility of upstream Python, what falls under third-party add-ons (e.g. Sphinx), and what is the integrator/OS-vender's responsibility. >On Tue, 14 Jun 2011, Zygmunt Krynicki wrote: >> Can please we have standardized hooks to build sphinx documentation >> and run setup.py test tests? Can those hooks do the right thing with >> generated documentation (dealing all the boring .doc-base files, >> replacing jquery with symlinks, ensuring proper requirements are >> used). > >and providing necessary tuning for matplotlib backends to assure >offscreen renderer (as I do e.g. in [1]) > >and not building documentation at all for binary-arch builds > >and if we go wild: take care about moving .so and _d.so >extensions into corresponding binary packages of architecture 'any', if >there is a corresponding python-MODULE-lib binary package defined. >Otherwise it looks as ugly as [2] > >;-) > >[1] https://github.com/yarikoptic/nitime/blob/HEAD/debian/rules >[2] https://github.com/yarikoptic/nipy/blob/debian/debian/rules#L46 > signature.asc Description: PGP signature
Re: Current state of packaging Python software for Debian
On Jun 14, 2011, at 05:53 PM, Zygmunt Krynicki wrote: >Can please we have standardized hooks to build sphinx documentation and run >setup.py test tests? I'd like to see the packaging folks address this. Eric is subscribed to this list and can probably speak to packaging's take on it, but my preferences would be that $ pysetup test would run all the tests in Python 3.3 and beyond (and in a distutils2 world). I don't think this is supported (yet?), and my limited testing required some 2to3 hackery that isn't quite working. In Python < 3.3, using setuptools/distribute/distutils2, this should be the standard interface: $ python setup.py test and in fact that works quite well, even with 2to3 integration. You need to add: test_suite='path.to.testdir' in your setup() call, and possibly a `convert_2to3_doctests` key as well (this is the part that does not work for me in Python 3.3). >Can those hooks do the right thing with generated documentation (dealing all >the boring .doc-base files, replacing jquery with symlinks, ensuring proper >requirements are used). Documentation and Sphinx integration is trickier because while Sphinx is the de-facto standard for Python documentation, it is a separate project from core Python, and I know of no initiatives to pull it into the core (nor, IMO should there be). In a setup.py world: $ python setup.py build_sphinx is the standard for creating the HTML from rST files, and again, this works well for me in Python 2. I might quibble with the command name, since I always get it wrong (sphinx_build ;). Of course, you need Sphinx installed, a proper conf.py file, and this bit in your setup.cfg: [build_sphinx] source_dir: path/to/root.rst This upstream Sphinx bug addresses jquery compatiblity: http://tinyurl.com/3rg3r7b Georg (the upstream Sphinx maintainer) makes a good point, which is that he really can't be expected to test Sphinx with any version of jquery than the one he ships. Operating systems (Debian/Ubuntu) are the integrators, and as Jakub points out in that issue, if Debian deviates from upstream by replacing Sphinx's version of jquery, it's incumbent on Debian to ensure it works properly. Cheers, -Barry signature.asc Description: PGP signature
Re: Current state of packaging Python software for Debian
ah -- wishlist! On Tue, 14 Jun 2011, Zygmunt Krynicki wrote: > Can please we have standardized hooks to build sphinx documentation > and run setup.py test tests? Can those hooks do the right thing with > generated documentation (dealing all the boring .doc-base files, > replacing jquery with symlinks, ensuring proper requirements are > used). and providing necessary tuning for matplotlib backends to assure offscreen renderer (as I do e.g. in [1]) and not building documentation at all for binary-arch builds and if we go wild: take care about moving .so and _d.so extensions into corresponding binary packages of architecture 'any', if there is a corresponding python-MODULE-lib binary package defined. Otherwise it looks as ugly as [2] ;-) [1] https://github.com/yarikoptic/nitime/blob/HEAD/debian/rules [2] https://github.com/yarikoptic/nipy/blob/debian/debian/rules#L46 -- =--= Keep in touch www.onerussian.com Yaroslav Halchenko www.ohloh.net/accounts/yarikoptic -- To UNSUBSCRIBE, email to debian-python-requ...@lists.debian.org with a subject of "unsubscribe". Trouble? Contact listmas...@lists.debian.org Archive: http://lists.debian.org/20110614163317.gh17...@onerussian.com
Re: Current state of packaging Python software for Debian
W dniu 14.06.2011 16:45, Barry Warsaw pisze: I want to be a conduit between Debian and upstream, so please, I encourage everyone who has concrete issues that can help the Debian story, to raise them here. Python is a big ship so it takes time to steer, but as I hope I've shown with PEP 3147 and 3149, we *can* steer it into a more Debian-friendly direction. Can please we have standardized hooks to build sphinx documentation and run setup.py test tests? Can those hooks do the right thing with generated documentation (dealing all the boring .doc-base files, replacing jquery with symlinks, ensuring proper requirements are used). Thanks ZK -- To UNSUBSCRIBE, email to debian-python-requ...@lists.debian.org with a subject of "unsubscribe". Trouble? Contact listmas...@lists.debian.org Archive: http://lists.debian.org/4df783e6.8080...@canonical.com
Re: Current state of packaging Python software for Debian (was: list of package for python_support -> dh_python2 ?)
On Jun 14, 2011, at 03:44 PM, Josselin Mouette wrote: >Le mardi 14 juin 2011 à 07:39 -0400, Barry Warsaw a écrit : >> Blog references, email threads, or other links to existing artifacts would be >> very helpful. Has anybody ever written a "What's Wrong With Python and How >> It >> Hurts Debian" article? > >I have something like that among the things I’d like to write, but it >would be very long so I haven’t found the time yet. I totally understand how difficult it is to get around to things like that. My own clone army is still not functional either. But I for one would find such an article valuable, so do let us know if you ever get around to writing it. I want to be a conduit between Debian and upstream, so please, I encourage everyone who has concrete issues that can help the Debian story, to raise them here. Python is a big ship so it takes time to steer, but as I hope I've shown with PEP 3147 and 3149, we *can* steer it into a more Debian-friendly direction. Cheers, -Barry signature.asc Description: PGP signature
Re: Current state of packaging Python software for Debian (was: list of package for python_support -> dh_python2 ?)
On Tue, Jun 14, 2011 at 03:44:33PM +0200, Josselin Mouette wrote: > Le mardi 14 juin 2011 à 07:39 -0400, Barry Warsaw a écrit : > > Blog references, email threads, or other links to existing artifacts would > > be > > very helpful. Has anybody ever written a "What's Wrong With Python and How > > It > > Hurts Debian" article? > > I have something like that among the things I’d like to write, but it > would be very long so I haven’t found the time yet. Maybe you could find the time to write the table of contents ? That would make it easier for other to try to fill the gaps. -- Nicolas Chauvat logilab.fr - services en informatique scientifique et gestion de connaissances -- To UNSUBSCRIBE, email to debian-python-requ...@lists.debian.org with a subject of "unsubscribe". Trouble? Contact listmas...@lists.debian.org Archive: http://lists.debian.org/20110614142709.ga8...@volans.logilab.fr
Re: Current state of packaging Python software for Debian (was: list of package for python_support -> dh_python2 ?)
Le mardi 14 juin 2011 à 07:39 -0400, Barry Warsaw a écrit : > Blog references, email threads, or other links to existing artifacts would be > very helpful. Has anybody ever written a "What's Wrong With Python and How It > Hurts Debian" article? I have something like that among the things I’d like to write, but it would be very long so I haven’t found the time yet. -- .''`. Josselin Mouette : :' : `. `' `- -- To UNSUBSCRIBE, email to debian-python-requ...@lists.debian.org with a subject of "unsubscribe". Trouble? Contact listmas...@lists.debian.org Archive: http://lists.debian.org/1308059073.18675.342.camel@pi0307572
Re: Current state of packaging Python software for Debian (was: list of package for python_support -> dh_python2 ?)
On Jun 14, 2011, at 07:00 PM, Ben Finney wrote: >I think (as someone with no special authority on Python nor Debian) that >the people who know most detail about what's painful for packaging >Python software for Debian are burned out on the topic. They therefore >don't want to spend the effort to summarise the problems, nor to verify >whether they're still problems in current versions. If that's true, it's sad, but understandable. It happens in many communities. The shame of it is that the people who get most burned out are exactly the people with the most valuable experiences and best institutional knowledge on what's wrong and how to fix it. >If that's not true I'd love for your question to be answered in detail >and documented publicly, of course. > >But if it is true, and the people who know the answer aren't motivated >to work on answering your question, I think someone (perhaps you) will >need to go over a lot of already-trod ground to discover what's still >painful and why, in order to get those details documented. Blog references, email threads, or other links to existing artifacts would be very helpful. Has anybody ever written a "What's Wrong With Python and How It Hurts Debian" article? -Barry signature.asc Description: PGP signature
Current state of packaging Python software for Debian (was: list of package for python_support -> dh_python2 ?)
Barry Warsaw writes: > On Jun 10, 2011, at 10:02 AM, Paul Wise wrote: > > >Personally I've always wondered why a Debian-specific helper should > >be needed instead of python upstream behaving the way we wanted. > > Can you get more specific about this? Obviously, there's little we can > do about Python 2, and I did spend Real Work Time on getting better > support specifically for Debian into Python 3.2. I'm willing to do > more for Python 3.3 but I'd like to better understand exactly what > folks here think needs to be done. I think (as someone with no special authority on Python nor Debian) that the people who know most detail about what's painful for packaging Python software for Debian are burned out on the topic. They therefore don't want to spend the effort to summarise the problems, nor to verify whether they're still problems in current versions. If that's not true I'd love for your question to be answered in detail and documented publicly, of course. But if it is true, and the people who know the answer aren't motivated to work on answering your question, I think someone (perhaps you) will need to go over a lot of already-trod ground to discover what's still painful and why, in order to get those details documented. -- \“The whole area of [treating source code as intellectual | `\property] is almost assuring a customer that you are not going | _o__) to do any innovation in the future.” —Gary Barnett | Ben Finney pgpJEhwQtVDxn.pgp Description: PGP signature
