Re: [Distutils] command hooks...
On Tue, May 29, 2012 at 7:41 PM, PJ Eby p...@telecommunity.com wrote: I might be confused; I haven't been following the goings-on of late with distutils2. At one point, I thought the plan was not to bless or include dependency-managing installers with the stdlib, or something like that. i.e., I thought the plan wasn't to support or bless full-service tools like buildout, easy_install, or pip, or anything comparable to them. Right, yeah, the plans in this area were fluid for awhile, but the eventual conclusion was that the stdlib should have a command-line utility capable of installing packages with dependencies. That exists in default branch now; it's called pysetup. It doesn't have nearly all the features of easy_install, buildout, or pip, but it can install packages from an index with deps. In any case, it still doesn't change the part where it's a good idea to ship a static setup.cfg, with hooks only needing to run on the sdist-building machine, unless they are actually part of the build process. There are use cases for calculated data to be in the initial setup.cfg, where the calculation machinery doesn't need to be on the target (like generating the file list or version from revision control info). So, a setup_requires (or maybe better named build_requires) would still be helpful, but probably shouldn't be used for setup.cfg stability. That's not a bad idea for certain kinds of metadata--version/vcs info for example. I like the idea of including a generated static setup.cfg in a source dist as a solution to that kind of problem. But that doesn't eliminate the need for setup_hooks (or even more complicated objects like custom commands) in an sdist. For example, the majority of projects I work on require Numpy to build one or two extension modules. They require hooks to check that the numpy package is importable, and then to use numpy's API to get the paths to the numpy headers and and them to the include_dirs for each extension module that requires them. That's not the only one though--one could have a whole suite of setup_hooks common to a bunch of projects. Custom Compiler classes are a possibility now too. One could ship a copy of those dependencies with each project, or have some kind of bootstrap script. But to be able to automatically download and add build dependencies to the path (a la setup_requires) would be much nicer. And packaging will have pysetup, so it should be doable. (Having the same capability for test dependencies and doc dependencies would be nice too, but not nearly as important). Erik ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] command hooks...
On Wed, May 30, 2012 at 11:54 AM, Erik Bray erik.m.b...@gmail.com wrote: On Tue, May 29, 2012 at 7:41 PM, PJ Eby p...@telecommunity.com wrote: I might be confused; I haven't been following the goings-on of late with distutils2. At one point, I thought the plan was not to bless or include dependency-managing installers with the stdlib, or something like that. i.e., I thought the plan wasn't to support or bless full-service tools like buildout, easy_install, or pip, or anything comparable to them. Right, yeah, the plans in this area were fluid for awhile, but the eventual conclusion was that the stdlib should have a command-line utility capable of installing packages with dependencies. That exists in default branch now; it's called pysetup. It doesn't have nearly all the features of easy_install, buildout, or pip, but it can install packages from an index with deps. In any case, it still doesn't change the part where it's a good idea to ship a static setup.cfg, with hooks only needing to run on the sdist-building machine, unless they are actually part of the build process. There are use cases for calculated data to be in the initial setup.cfg, where the calculation machinery doesn't need to be on the target (like generating the file list or version from revision control info). So, a setup_requires (or maybe better named build_requires) would still be helpful, but probably shouldn't be used for setup.cfg stability. That's not a bad idea for certain kinds of metadata--version/vcs info for example. I like the idea of including a generated static setup.cfg in a source dist as a solution to that kind of problem. But that doesn't eliminate the need for setup_hooks (or even more complicated objects like custom commands) in an sdist. For example, the majority of projects I work on require Numpy to build one or two extension modules. They require hooks to check that the numpy package is importable, and then to use numpy's API to get the paths to the numpy headers and and them to the include_dirs for each extension module that requires them. That's not the only one though--one could have a whole suite of setup_hooks common to a bunch of projects. Custom Compiler classes are a possibility now too. One could ship a copy of those dependencies with each project, or have some kind of bootstrap script. But to be able to automatically download and add build dependencies to the path (a la setup_requires) would be much nicer. And packaging will have pysetup, so it should be doable. (Having the same capability for test dependencies and doc dependencies would be nice too, but not nearly as important). Certainly. I was just saying that the generated-metadata cases need handling, too, and that people should also be informed that they don't need (and shouldn't use) setup_requires for simple metadata generation, and that it might be a good idea to call the feature build_requires, and perhaps distinguish between setup hooks (for developers to have nice things) and build hooks (for stuff that absolutely has to run on the install machine). ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] command hooks...
On Wed, May 30, 2012 at 12:19 PM, PJ Eby p...@telecommunity.com wrote: On Wed, May 30, 2012 at 11:54 AM, Erik Bray erik.m.b...@gmail.com wrote: On Tue, May 29, 2012 at 7:41 PM, PJ Eby p...@telecommunity.com wrote: I might be confused; I haven't been following the goings-on of late with distutils2. At one point, I thought the plan was not to bless or include dependency-managing installers with the stdlib, or something like that. i.e., I thought the plan wasn't to support or bless full-service tools like buildout, easy_install, or pip, or anything comparable to them. Right, yeah, the plans in this area were fluid for awhile, but the eventual conclusion was that the stdlib should have a command-line utility capable of installing packages with dependencies. That exists in default branch now; it's called pysetup. It doesn't have nearly all the features of easy_install, buildout, or pip, but it can install packages from an index with deps. In any case, it still doesn't change the part where it's a good idea to ship a static setup.cfg, with hooks only needing to run on the sdist-building machine, unless they are actually part of the build process. There are use cases for calculated data to be in the initial setup.cfg, where the calculation machinery doesn't need to be on the target (like generating the file list or version from revision control info). So, a setup_requires (or maybe better named build_requires) would still be helpful, but probably shouldn't be used for setup.cfg stability. That's not a bad idea for certain kinds of metadata--version/vcs info for example. I like the idea of including a generated static setup.cfg in a source dist as a solution to that kind of problem. But that doesn't eliminate the need for setup_hooks (or even more complicated objects like custom commands) in an sdist. For example, the majority of projects I work on require Numpy to build one or two extension modules. They require hooks to check that the numpy package is importable, and then to use numpy's API to get the paths to the numpy headers and and them to the include_dirs for each extension module that requires them. That's not the only one though--one could have a whole suite of setup_hooks common to a bunch of projects. Custom Compiler classes are a possibility now too. One could ship a copy of those dependencies with each project, or have some kind of bootstrap script. But to be able to automatically download and add build dependencies to the path (a la setup_requires) would be much nicer. And packaging will have pysetup, so it should be doable. (Having the same capability for test dependencies and doc dependencies would be nice too, but not nearly as important). Certainly. I was just saying that the generated-metadata cases need handling, too, and that people should also be informed that they don't need (and shouldn't use) setup_requires for simple metadata generation, and that it might be a good idea to call the feature build_requires, and perhaps distinguish between setup hooks (for developers to have nice things) and build hooks (for stuff that absolutely has to run on the install machine). I think we're on the same page then :) packaging/d2 also supports pre/post-command hooks which might, in most cases, be more appropriate for the build hooks that absolutely have to run case. In fact, I had completely forgotten this, but my aforementioned Numpy extension module hook is a pre-build_ext hook. That makes it absolutely clear that this is something we have to do before we can build an extension module, and that it doesn't have any purpose outside that context and shouldn't be executed ever time I run `pysetup something`. Erik ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] command hooks...
On Tue, May 29, 2012 at 10:20 AM, Erik Bray erik.m.b...@gmail.com wrote: This particular solution works for me. But the point is that it can be done pretty easily. However, the lack of a setup_requires-like feature still makes things pretty impossible short of shipping a copy of all the required setup hooks with the projects that use them. Certainly doable, but far from ideal. Right, and I don't think distutils2 can really add setup_requires without blessing a package manager. That means the alternative is shipping an altered setup.cfg with sdist builds, or using tools that generate setup.cfg in the first place, such that setup.cfg isn't the project's canonical form. Probably the simplest way to do it would be to just ship setup.cfg in the sdist -- as modified by the setup hooks, since this keeps the developer from having to use a different tool to generate the setup.cfg; it can all be done by setup hooks. In that case, the setup hooks themselves would need to be idempotent, so they don't mess things up if they run a second time in an sdist-ed version of the project. Another alternative would be to use a package manager bootstrap script in the project directory (ala ez_setup.py), which could then read and process a setup_requires from setup.cfg. But that's a much heavier-weight process; it would be much preferable if people could make all their setup_requires stuff run off an original copy only, and not be needed in an sdist copy. So for example, if you need Pyrex for generating C code, it should run pre-sdist, not post-sdist. (OTOH, it's possible that there are some distributions you need for building the package, for platform-specific build steps.) ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] command hooks...
On 05/29/2012 08:19 AM, PJ Eby wrote: On Tue, May 29, 2012 at 10:20 AM, Erik Bray erik.m.b...@gmail.com mailto:erik.m.b...@gmail.com wrote: This particular solution works for me. But the point is that it can be done pretty easily. However, the lack of a setup_requires-like feature still makes things pretty impossible short of shipping a copy of all the required setup hooks with the projects that use them. Certainly doable, but far from ideal. Right, and I don't think distutils2 can really add setup_requires without blessing a package manager. I'm confused by this statement. distutils2 _includes_ a package manager (pysetup); it has no need to bless an external one. What am I missing? Carl signature.asc Description: OpenPGP digital signature ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] command hooks...
On Tue, May 29, 2012 at 1:24 PM, Carl Meyer c...@oddbird.net wrote: On 05/29/2012 08:19 AM, PJ Eby wrote: On Tue, May 29, 2012 at 10:20 AM, Erik Bray erik.m.b...@gmail.com mailto:erik.m.b...@gmail.com wrote: This particular solution works for me. But the point is that it can be done pretty easily. However, the lack of a setup_requires-like feature still makes things pretty impossible short of shipping a copy of all the required setup hooks with the projects that use them. Certainly doable, but far from ideal. Right, and I don't think distutils2 can really add setup_requires without blessing a package manager. I'm confused by this statement. distutils2 _includes_ a package manager (pysetup); it has no need to bless an external one. What am I missing? I might be confused; I haven't been following the goings-on of late with distutils2. At one point, I thought the plan was not to bless or include dependency-managing installers with the stdlib, or something like that. i.e., I thought the plan wasn't to support or bless full-service tools like buildout, easy_install, or pip, or anything comparable to them. ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] command hooks...
On 05/29/2012 02:01 PM, PJ Eby wrote: On Tue, May 29, 2012 at 1:24 PM, Carl Meyer c...@oddbird.net wrote: On 05/29/2012 08:19 AM, PJ Eby wrote: Right, and I don't think distutils2 can really add setup_requires without blessing a package manager. I'm confused by this statement. distutils2 _includes_ a package manager (pysetup); it has no need to bless an external one. What am I missing? I might be confused; I haven't been following the goings-on of late with distutils2. At one point, I thought the plan was not to bless or include dependency-managing installers with the stdlib, or something like that. i.e., I thought the plan wasn't to support or bless full-service tools like buildout, easy_install, or pip, or anything comparable to them. Right, yeah, the plans in this area were fluid for awhile, but the eventual conclusion was that the stdlib should have a command-line utility capable of installing packages with dependencies. That exists in default branch now; it's called pysetup. It doesn't have nearly all the features of easy_install, buildout, or pip, but it can install packages from an index with deps. Carl signature.asc Description: OpenPGP digital signature ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] command hooks...
On Tue, May 29, 2012 at 5:17 PM, Carl Meyer c...@oddbird.net wrote: On 05/29/2012 02:01 PM, PJ Eby wrote: On Tue, May 29, 2012 at 1:24 PM, Carl Meyer c...@oddbird.net wrote: On 05/29/2012 08:19 AM, PJ Eby wrote: Right, and I don't think distutils2 can really add setup_requires without blessing a package manager. I'm confused by this statement. distutils2 _includes_ a package manager (pysetup); it has no need to bless an external one. What am I missing? I might be confused; I haven't been following the goings-on of late with distutils2. At one point, I thought the plan was not to bless or include dependency-managing installers with the stdlib, or something like that. i.e., I thought the plan wasn't to support or bless full-service tools like buildout, easy_install, or pip, or anything comparable to them. Right, yeah, the plans in this area were fluid for awhile, but the eventual conclusion was that the stdlib should have a command-line utility capable of installing packages with dependencies. That exists in default branch now; it's called pysetup. It doesn't have nearly all the features of easy_install, buildout, or pip, but it can install packages from an index with deps. In any case, it still doesn't change the part where it's a good idea to ship a static setup.cfg, with hooks only needing to run on the sdist-building machine, unless they are actually part of the build process. There are use cases for calculated data to be in the initial setup.cfg, where the calculation machinery doesn't need to be on the target (like generating the file list or version from revision control info). So, a setup_requires (or maybe better named build_requires) would still be helpful, but probably shouldn't be used for setup.cfg stability. One use case I ran across rather late in the game with setuptools was the part where sdist distributions need to be capable of rebuilding an sdist, for at least the bdist_rpm case, if not others. That had me tearing my hair, since I was dependent on what I could do on top of existing distutils. But for distutils2, shipping an updated setup.cfg (possibly *minus* certain setup_hooks) would solve the problem nicely. ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] command hooks...
On Fri, May 25, 2012 at 4:23 PM, Erik Bray erik.m.b...@gmail.com wrote:
On Fri, May 25, 2012 at 2:24 PM, Daniel Holth dho...@gmail.com wrote:
This discussion is confusing because we are either talking about a
general plugin discovery mechanism for Python, or a plugin mechanism
that packaging itself uses.
After the patch, entry_points.txt (which you are encouraged to write
yourself) is copied by means of a new {dist-info} category using the
resources = mechanism in packaging. This is a straightforward way to
copy anything into the .dist-info directory. Apart from copying the
file, 'packaging' has no role.
setup.cfg:
[files]
packages = mypackage
resources =
entry_points.txt={dist-info}
That's all there is to it.
Neat! I didn't realize there was a {dist-info} resource category.
That, plus the register_finder() approach mentioned in the other
thread could be a way to go for interoperability with packages using
setuptools. It would still require a bit of hackery though... I
suspect there would be resistance to adding support for this directly
in packaging. But a setup_hook could set this up.
Ideally, one should just be able to add sections to setup.cfg itself to
describe the entry points. Better yet: just include setup.cfg in the
dist-info directory, and then the replacement for entry points can look for
arbitrary sections in setup.cfg files. (The downside: slower first-use
performance, since every setup.cfg would have to be read at least once, but
on the upside, all subsequent entry point lookups would be faster.)
Is setup.cfg already in dist-info? I guess if it isn't, you could just add
it, using the same mechanism above. Hm (goes to look at packaging
docs)... Ouch. I'm seeing a bigger problem, which is that without either
the ability to do setup_requires or to ship an sdist with a hook-altered
setup.cfg, it doesn't look like you can actually implement all of
setuptools' build functionality with just packaging. (e.g. Setuptools can
ship an sdist whose contents and version were determined using revision
control info, but which does not then require the revision control tool
when installing from the sdist, as the sdist contains a pre-built manifest,
and a pre-built setup.cfg with hardcoded version numbers copied from the
original revision control info.)
I really hope I'm not going to have to write setuptools2 to work around
these limitations. ;-) (Preferably, if I do write a setuptools2, I hope
it can all be done using proper setup hooks on top of packaging/distutils2,
without monkeypatching anything.)
___
Distutils-SIG maillist - Distutils-SIG@python.org
http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] command hooks...
On Thu, May 24, 2012 at 8:27 PM, PJ Eby p...@telecommunity.com wrote: On Thu, May 24, 2012 at 6:56 PM, Daniel Holth dho...@gmail.com wrote: For something like Pyramid the main piece that's missing is support for an entry_points like system. As Tarek and other have pointed out the past, such a system could still be supported through a third-party plugin system that works via setup hooks and custom metadata. I think there was even a prototype for something like that at one point... 'packaging' will be powerful enough to copy entry_points.txt into the .dist-info directory as a resource. Python bug #11880. The existing entry points code has a plugin mechanism for itself that will make it easy to iterate over entry_points.txt in dist-info as well as egg-info directories. Just write that plugin and you should be good to go. Though you might have to install the plugin in an egg-info directory at first... What plugin mechanism are you talking about here, specifically? For that matter, which existing entry points code are you talking about, too? ;-) Yeah, I'm a little confused by this too. Sure you could include an entry_points.txt in the dist-info. Though it's not clear to me that there's an entirely straightforward way to do this yet. I think some ideas have been tossed about for how to make it easier to include custom files in dist-info, but that there were no conclusions on that. Or maybe it already is easy and I'm just missing something. Then there would need to be some interface like pkg_resources.iter_entry_points() available to all software that relies on entry_points. This could be provided by the same plugin that adds entry_points.txt to dist-info. That plugin would have to be an installation requirement for any software that relies on it (as is setuptools for any package that currently uses entry_points, so I don't think that's such a hardship...) Erik ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] command hooks...
This discussion is confusing because we are either talking about a
general plugin discovery mechanism for Python, or a plugin mechanism
that packaging itself uses.
After the patch, entry_points.txt (which you are encouraged to write
yourself) is copied by means of a new {dist-info} category using the
resources = mechanism in packaging. This is a straightforward way to
copy anything into the .dist-info directory. Apart from copying the
file, 'packaging' has no role.
setup.cfg:
[files]
packages = mypackage
resources =
entry_points.txt={dist-info}
That's all there is to it.
___
Distutils-SIG maillist - Distutils-SIG@python.org
http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] command hooks...
On Fri, May 25, 2012 at 2:24 PM, Daniel Holth dho...@gmail.com wrote:
This discussion is confusing because we are either talking about a
general plugin discovery mechanism for Python, or a plugin mechanism
that packaging itself uses.
After the patch, entry_points.txt (which you are encouraged to write
yourself) is copied by means of a new {dist-info} category using the
resources = mechanism in packaging. This is a straightforward way to
copy anything into the .dist-info directory. Apart from copying the
file, 'packaging' has no role.
setup.cfg:
[files]
packages = mypackage
resources =
entry_points.txt={dist-info}
That's all there is to it.
Neat! I didn't realize there was a {dist-info} resource category.
That, plus the register_finder() approach mentioned in the other
thread could be a way to go for interoperability with packages using
setuptools. It would still require a bit of hackery though... I
suspect there would be resistance to adding support for this directly
in packaging. But a setup_hook could set this up.
Erik
___
Distutils-SIG maillist - Distutils-SIG@python.org
http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] command hooks...
On Wed, May 23, 2012 at 11:59 PM, Chris McDonough chr...@plope.com wrote: It sounds like a reasonable task for me to try out. In the meantime we've had a bit of a family emergency which will take some time to overcome and I won't be able to dedicate much energy to this. As a result, I have to declare bankruptcy here, so I'll have to live with whatever I get. Sorry to hear about that. I'm hoping though that someone else will step up here and do an evaluation and try to get things like Pyramid and popular Zope packages installed in a way that makes sense for straddling Python 2 and Python 3. I suspect if no one does this, it's going to be rough going. - C For something like Pyramid I don't actually think it would be too difficult to get it *mostly* working with packaging/distutils2 in its present state. The word is mostly. And that's to say nothing of all its dependencies, though they don't all necessarily need to work with the same installer. For something like Pyramid the main piece that's missing is support for an entry_points like system. As Tarek and other have pointed out the past, such a system could still be supported through a third-party plugin system that works via setup hooks and custom metadata. I think there was even a prototype for something like that at one point... But that raises another issue: If packaging is going to be like a framework that additional features like plugin systems or console script generators can be tacked on to, there needs to be a way to specify build dependencies in setup.cfg, and to have those dependencies automatically downloaded and added to sys.path during setup, a la setup_requires in setuptools. I know there have been discussions and proposals in the past to add something like this to packaging, and to amend PEP 345 to include it. Metadata for test dependencies and docs dependencies have also been discussed in the past. But I think a way of adding build/setup dependencies is critical if packaging is going to be useful as a framework. I have a few packages that use d2to1, which supports setup_requires through setuptools. Thanks to that, I'm able to make those packages have an additional setup_requirement of a package that contains several pre-canned setup_hooks that I use in all my projects. I would point out specific examples, but they're a bit obscure... Anyways, I'm working right now to finish a software release. But then I want to spend some time working on this issue of adding build requirements support to packaging and to PEP 345. I might also work on a plugin system that can work in packaging as a replacement for entry_points, if no one else does. Without these features I don't see packaging being very successful as an installation framework, IMO. Erik ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] command hooks...
For something like Pyramid the main piece that's missing is support for an entry_points like system. As Tarek and other have pointed out the past, such a system could still be supported through a third-party plugin system that works via setup hooks and custom metadata. I think there was even a prototype for something like that at one point... 'packaging' will be powerful enough to copy entry_points.txt into the .dist-info directory as a resource. Python bug #11880. The existing entry points code has a plugin mechanism for itself that will make it easy to iterate over entry_points.txt in dist-info as well as egg-info directories. Just write that plugin and you should be good to go. Though you might have to install the plugin in an egg-info directory at first... ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] command hooks...
On Thu, May 24, 2012 at 6:56 PM, Daniel Holth dho...@gmail.com wrote: For something like Pyramid the main piece that's missing is support for an entry_points like system. As Tarek and other have pointed out the past, such a system could still be supported through a third-party plugin system that works via setup hooks and custom metadata. I think there was even a prototype for something like that at one point... 'packaging' will be powerful enough to copy entry_points.txt into the .dist-info directory as a resource. Python bug #11880. The existing entry points code has a plugin mechanism for itself that will make it easy to iterate over entry_points.txt in dist-info as well as egg-info directories. Just write that plugin and you should be good to go. Though you might have to install the plugin in an egg-info directory at first... What plugin mechanism are you talking about here, specifically? For that matter, which existing entry points code are you talking about, too? ;-) ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] command hooks...
On Mon, May 21, 2012 at 3:42 PM, Tarek Ziadé ta...@ziade.org wrote: If you are a packager for a project, you can describe in details your data files, and add more metadata that are understood by PyPI. If you are a debian packager, you will be able to define where the data files of a python project should be installed without having to patch some python code. This right here is the killer feature for me. By my limited observations, most people solve the data files problem either by dumping their data files inside their python modules (which is an ugly abuse of the filesystem), or are simply using the cumbersome autotools in order to record the installation prefix for their data files. It blows my mind that it is standard practice for many GNOME apps written in Python to use a C compilation preprocessor in order to set a python variable so their python scripts can find their data files. Currently I am hacking distutils in order to accomplish this, like so: https://github.com/robru/gottengeography/blob/master/setup.py But this is ugly because it modifies the file in-place, so I always have to be careful not to accidentally commit the munged file into my git repo. It's absolutely critical that any replacement for distutils have built-in functionality for installed python code being able to query at run-time the location that data files were placed at install time. -- http://exolucere.ca ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] command hooks...
On 5/23/12 8:16 AM, Robert Park wrote: On Mon, May 21, 2012 at 3:42 PM, Tarek Ziadéta...@ziade.org wrote: If you are a packager for a project, you can describe in details your data files, and add more metadata that are understood by PyPI. If you are a debian packager, you will be able to define where the data files of a python project should be installed without having to patch some python code. This right here is the killer feature for me. By my limited observations, most people solve the data files problem either by dumping their data files inside their python modules (which is an ugly abuse of the filesystem), or are simply using the cumbersome autotools in order to record the installation prefix for their data files. It blows my mind that it is standard practice for many GNOME apps written in Python to use a C compilation preprocessor in order to set a python variable so their python scripts can find their data files. Currently I am hacking distutils in order to accomplish this, like so: https://github.com/robru/gottengeography/blob/master/setup.py But this is ugly because it modifies the file in-place, so I always have to be careful not to accidentally commit the munged file into my git repo. It's absolutely critical that any replacement for distutils have built-in functionality for installed python code being able to query at run-time the location that data files were placed at install time. Please read this section and let us know what you think: http://docs.python.org/dev/packaging/setupcfg.html#resources This works in conjunction with the new sysconfig module, which can be configured system-wide by the linux distribution, or locally per projet Then you can use an API to get the file from your code. Gosh the documentation is a mess ... we need to fix this - it has bits from the previous version that should be removed :( Cheers Tarek ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
[Distutils] command hooks...
On 23 May 2012 07:48, Tarek Ziadé ta...@ziade.org wrote: On 5/23/12 8:16 AM, Robert Park wrote: On Mon, May 21, 2012 at 3:42 PM, Tarek Ziadéta...@ziade.org wrote: If you are a packager for a project, you can describe in details your data files, and add more metadata that are understood by PyPI. If you are a debian packager, you will be able to define where the data files of a python project should be installed without having to patch some python code. This right here is the killer feature for me. By my limited observations, most people solve the data files problem either by dumping their data files inside their python modules (which is an ugly abuse of the filesystem), or are simply using the cumbersome autotools in order to record the installation prefix for their data files. It blows my mind that it is standard practice for many GNOME apps written in Python to use a C compilation preprocessor in order to set a python variable so their python scripts can find their data files. Currently I am hacking distutils in order to accomplish this, like so: https://github.com/robru/**gottengeography/blob/master/**setup.pyhttps://github.com/robru/gottengeography/blob/master/setup.py But this is ugly because it modifies the file in-place, so I always have to be careful not to accidentally commit the munged file into my git repo. It's absolutely critical that any replacement for distutils have built-in functionality for installed python code being able to query at run-time the location that data files were placed at install time. Please read this section and let us know what you think: http://docs.python.org/dev/**packaging/setupcfg.html#**resourceshttp://docs.python.org/dev/packaging/setupcfg.html#resources How do gettext translation files fit into this scheme? Should they go under appdata? This works in conjunction with the new sysconfig module, which can be configured system-wide by the linux distribution, or locally per projet Then you can use an API to get the file from your code. Gosh the documentation is a mess ... we need to fix this - it has bits from the previous version that should be removed :( Cheers Tarek __**_ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/**mailman/listinfo/distutils-sighttp://mail.python.org/mailman/listinfo/distutils-sig ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] command hooks...
On Wed, May 23, 2012 at 1:48 AM, Tarek Ziadé ta...@ziade.org wrote:
On 5/23/12 8:16 AM, Robert Park wrote:
It's absolutely critical that any replacement for distutils have
built-in functionality for installed python code being able to query
at run-time the location that data files were placed at install time.
Please read this section and let us know what you think:
http://docs.python.org/dev/packaging/setupcfg.html#resources
Yeah, the [files] definition in setup.cfg looks fine, the space syntax
in the source definition I would consider highly important in order to
avoid installing things into {datadir}/doc/foo/doc/ as given in the
example.
This works in conjunction with the new sysconfig module, which can be
configured system-wide by the linux distribution, or locally per projet
Then you can use an API to get the file from your code.
I didn't see any mention of this aspect in the documentation you
linked. What does the API look like?
--
http://exolucere.ca
___
Distutils-SIG maillist - Distutils-SIG@python.org
http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] command hooks...
On 05/21/2012 04:59 PM, Tarek Ziadé wrote: On 5/16/12 6:26 PM, Chris McDonough wrote: On 05/16/2012 02:55 AM, Tarek Ziadé wrote: On 5/16/12 3:58 AM, Chris McDonough wrote: Adding two more (packaging and distutils2) which are similarly semi-documented and which don't even solve the problems that the previous ones do would serve no purpose, and baking them into Python itself will mean they can't evolve in important ways. Oh, I think I need to answer to this too since you said you wanted to help. Packaging is not intended to be similar to setuptools in its features. For instance we won't provide console scripts or entry points. The first one because 'script' is the same feature (except there's an indirection and I said before we could mimic this) The second one because we should build this kind of feature outside the stdlib (this is not something most people use, according to the survey I did back a few years ago, it's mostly zope/plone/repoze land) I suspect many people don't really know they're using entry points when they are. PasteDeploy configuration files rely heavily on entry points. Those are used in various ways by Pylons, Pyramid, Zope, and Turbogears. (PasteDeploy itself been downloaded almost a million times from PyPI.) yes. on a side note: The PyPI stats numbers are biased for many libraries because they are artificially increased by all the buildouts and Jenkins and Travis-CI calls out there. IOW: if a server downloads 100 times a day PasteDeploy because someone did not set a cache, does it make it 100 times more popular ? Same goes for any library that's a core part of the non-monolithic frameworks out there. I'm fine with needing to depend on an external package to scan for entrypoint-like-things registered as the result of the installation of a distribution. But there will need to be a way to register entrypoint-like things (along the lines of arbitrary egg-info metadata) as the result of distribution installation using pysetup. Maybe that already exists. we want to make the dist-info (see PEP 376) structure a directory where you can drop anything, so you can have arbitrary metadata. Right now, entry_points.txt is copied over in that directory by pysetup. the new setup.cfg file allows you to define extra metadata as well, and what's awesome: we want to publish that static file to PyPI so tools can browse it ! = http://docs.python.org/dev/packaging/setupcfg.html#extensibility IOW: packaging should only be the common basis and provide a basic installer - not a full fledge tool you can use to replace the most advanced setuptools features. And we want it pluggable enough so people can build pluggable features on the top of it, like Eric explained earlier Does that make sense ? I'd like to say it does, but I'm afraid it does not. I can see shipping both machinery and a full-fledged installer that handles all the use cases of existing installers. I can also see shipping just machinery and no installer at all. But I can't really see shipping both machinery and an installer that we know doesn't service use cases that existing installers already do. At least I can't see doing that in order to service an external shipping deadline. So what about this suggestion: let's make this goal : pysetup should be able to install Pyramid (without [extras] sorry, we can live without it) - with a few adaptations in a branch if needed. if we can make it work before 3.3rc1, great. If we fail we remove the installer part of packaging and move it to an external project That's just a suggestion, I'll defer the decision to Eric because while I can help around, I don't have the time, neither the motivation to do packaging work these days. But we have to hurry up - and check with Guido if he's ok with those decisions. It sounds like a reasonable task for me to try out. In the meantime we've had a bit of a family emergency which will take some time to overcome and I won't be able to dedicate much energy to this. As a result, I have to declare bankruptcy here, so I'll have to live with whatever I get. I'm hoping though that someone else will step up here and do an evaluation and try to get things like Pyramid and popular Zope packages installed in a way that makes sense for straddling Python 2 and Python 3. I suspect if no one does this, it's going to be rough going. - C ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] command hooks...
On Mon, May 21, 2012 at 4:42 PM, Tarek Ziadé ta...@ziade.org wrote: My understanding is that the legacy tools already work, for some vague definition of work. Your task is to convince me that packaging is better -- which I think will be hard to do if packaging is not better. It really depends what you put behind the word better. And no, My task is not to convince you that packaging is better. My tasks were : 1 - try to define standards which every packaging tool can implement, for the sake of interoperability, have a consensus on them, have them accepted 2 - allow more static definitions of metadata 3 - add in the standard library a/ a reference implementation for all the standards b/ a minimal installer. 3-b is not ready for prime time obviously - but this should not take too long. I don't usually do this, and I think most people on this list are already aware of it, but this seems like a decent place to insert a shameless plug for d2to1 [1]. d2to1 allows Python developers to take advantage of many of the features and advantages of packaging today--in particular static metadata and hook functions--while still supporting existing installers like easy_install and pip. It's a total hack, and only meant to aid with transition, but it actually works quite well in practice. Unfortunately I haven't found the time to work on it in a little while so I need to find that time. In particular it would be useful for it to add resources support. Erik [1] http://pypi.python.org/pypi/d2to1 ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] command hooks...
On 5/16/12 7:35 PM, Mark Sienkiewicz wrote: Can you see packaging as a set of utility at this point, rather than a tool that's going to replace instantly all the legacy tools ? If it is not going to replace the legacy tools, then why should I use it? Who are you ? If you are a user that just installs software, you might want to use it because you can *uninstall* things. This is now less relevant now that pip has this feature, but easy_install does not. And with the PEP we wrote, any tool should be able to install/uninstall any project. And well, have this feature bundled into Python makes sense. If you are a packager for a project, you can describe in details your data files, and add more metadata that are understood by PyPI. If you are a debian packager, you will be able to define where the data files of a python project should be installed without having to patch some python code. I have much more but it's a bit hard to summarize this in a mail - I have a packaging fatigue anyways. My understanding is that the legacy tools already work, for some vague definition of work. Your task is to convince me that packaging is better -- which I think will be hard to do if packaging is not better. It really depends what you put behind the word better. And no, My task is not to convince you that packaging is better. My tasks were : 1 - try to define standards which every packaging tool can implement, for the sake of interoperability, have a consensus on them, have them accepted 2 - allow more static definitions of metadata 3 - add in the standard library a/ a reference implementation for all the standards b/ a minimal installer. 3-b is not ready for prime time obviously - but this should not take too long. Cheers Tarek Mark S. ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] command hooks...
On 5/16/12 3:58 AM, Chris McDonough wrote: On 05/15/2012 04:39 PM, Éric Araujo wrote: Hi again, Le 01/05/2012 14:28, Paul Moore a écrit : On 1 May 2012 17:40, Chris McDonoughchr...@plope.com wrote: Is there a PEP for the packaging package? Is there any sort of unfinished business I can help with? AFAIK, there's no specific PEP for packaging (there are a number of related PEPs, but nothing specific like a roadmap). Yep. distutils2/packaging implement PEP 345 (Metadata 1.2), 376 (dist-info directory a.k.a. installation database) and 386 (version numbers), and also the older Metadata PEPs like distutils, but there was no PEP to discuss inclusion of d2 in Python 3.3: it just happened when a core developer (Martin von Löwis) indicated he was opposed to work on features (to add support for PEP 384 — Stable ABI for example) outside of the Python repository. I think that no PEP was asked by anyone because distutils2 is forked from code already in the stdlib and it implements accepted PEPs. There are small and big features added or in progress, many of them inspired by setuptools, that don’t have a PEP though. As I said on my other reply there is no friendly list of issues or roadmap, only unsorted bugs and what’s in my mind. OK. Is there a way for me to take a look at the unsorted bugs? I'm sure Éric can give you much better pointers on what would be useful, but one issue I've tried to raise a few times, and more recently Jim Fulton raised here (http://mail.python.org/pipermail/distutils-sig/2012-March/018323.html) is that of binary distribution support in packaging2. I've never had the time to shepherd a proposal through beyond the initial debate stage, and I know it's not getting high on Éric's list of priorities, but it would be good to see some movement on this. Indeed, in private email with Paul I agreed on the importance of a binary distribution format and did a pre-publication review of his PEP, but we did not finish our discussion nor incorporated the alternate proposal that was discussed at PyCon and on the mailing list (which makes it hard to see a clear picture — PEPs are good :) It seems unlikely that this hard topic can be solved for Python 3.3 / distutils2 1.0; what can be done however is to make sure that the extensibility hooks in d2 are well tested and documented so that when a bdist PEP reaches agreement and is implemented, a simple pysetup call and two lines of config will be all it takes to be able to use the new command. I know that the situation is far from ideal, and far from our goals for 3.3, but anyway d2 was never intended as a full replacement for setuptools and pip (more on that in an upcoming reply to another of your messages when you listed the setuptools features used by Pyramid). I think packaging in Python 3.3 will be a first version put in the stdlib to gather feedback and reports, not a finished stable product. I really don't want to add stop energy here, and I'm more than willing to row to get something going, but I'm afraid if that's the diagnosis, it means I'll personally have to oppose the inclusion of packaging in 3.3. We currently already have at least 3 competing solutions (setuptools, distribute, and distutils itself), and people are baffled about which to use and how to use it. Adding two more (packaging and distutils2) which are similarly semi-documented and which don't even solve the problems that the previous ones do would serve no purpose, and baking them into Python itself will mean they can't evolve in important ways. I'd suggest we just put the brakes on and slate something better for 3.4. Does that make sense, or does that make people sad? I know the idea of having packaging in the stdlib is something some people disliked, from day 1. And if I recall correctly, you did not like this either back then. I am -1 on for two reasons: 1/ your argument about people being baffled about which tool to use will be worse if we work outside the stdlib. Having packaging in the stdlib and distutils frozen here, leads the path: hey, the next tool in the stdlib is packaging 2/ packaging is completely mature for some parts, like the PEP 386 implementation. And having the packaging.version module in the stdlib is a great step forward for standardization The pep 376 implementation is also providing great APIs to push all tools to inter-operate. If we develop this tool outside, I am afraid we will just add more confusion in the mix. Can you see packaging as a set of utility at this point, rather than a tool that's going to replace instantly all the legacy tools ? I am adding Guido in cc for his opinion on this Cheers Tarek - C ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig ___ Distutils-SIG maillist - Distutils-SIG@python.org
Re: [Distutils] command hooks...
On 5/16/12 5:01 AM, Alex Clark wrote: Hi, On 5/15/12 9:58 PM, Chris McDonough wrote: On 05/15/2012 04:39 PM, Éric Araujo wrote: Hi again, Le 01/05/2012 14:28, Paul Moore a écrit : On 1 May 2012 17:40, Chris McDonoughchr...@plope.com wrote: Is there a PEP for the packaging package? Is there any sort of unfinished business I can help with? AFAIK, there's no specific PEP for packaging (there are a number of related PEPs, but nothing specific like a roadmap). Yep. distutils2/packaging implement PEP 345 (Metadata 1.2), 376 (dist-info directory a.k.a. installation database) and 386 (version numbers), and also the older Metadata PEPs like distutils, but there was no PEP to discuss inclusion of d2 in Python 3.3: it just happened when a core developer (Martin von Löwis) indicated he was opposed to work on features (to add support for PEP 384 — Stable ABI for example) outside of the Python repository. I think that no PEP was asked by anyone because distutils2 is forked from code already in the stdlib and it implements accepted PEPs. There are small and big features added or in progress, many of them inspired by setuptools, that don’t have a PEP though. As I said on my other reply there is no friendly list of issues or roadmap, only unsorted bugs and what’s in my mind. OK. Is there a way for me to take a look at the unsorted bugs? I'm sure Éric can give you much better pointers on what would be useful, but one issue I've tried to raise a few times, and more recently Jim Fulton raised here (http://mail.python.org/pipermail/distutils-sig/2012-March/018323.html) is that of binary distribution support in packaging2. I've never had the time to shepherd a proposal through beyond the initial debate stage, and I know it's not getting high on Éric's list of priorities, but it would be good to see some movement on this. Indeed, in private email with Paul I agreed on the importance of a binary distribution format and did a pre-publication review of his PEP, but we did not finish our discussion nor incorporated the alternate proposal that was discussed at PyCon and on the mailing list (which makes it hard to see a clear picture — PEPs are good :) It seems unlikely that this hard topic can be solved for Python 3.3 / distutils2 1.0; what can be done however is to make sure that the extensibility hooks in d2 are well tested and documented so that when a bdist PEP reaches agreement and is implemented, a simple pysetup call and two lines of config will be all it takes to be able to use the new command. I know that the situation is far from ideal, and far from our goals for 3.3, but anyway d2 was never intended as a full replacement for setuptools and pip (more on that in an upcoming reply to another of your messages when you listed the setuptools features used by Pyramid). I think packaging in Python 3.3 will be a first version put in the stdlib to gather feedback and reports, not a finished stable product. I really don't want to add stop energy here, and I'm more than willing to row to get something going, but I'm afraid if that's the diagnosis, it means I'll personally have to oppose the inclusion of packaging in 3.3. We currently already have at least 3 competing solutions (setuptools, distribute, and distutils itself), and people are baffled about which to use and how to use it. Adding two more (packaging and distutils2) which are similarly semi-documented and which don't even solve the problems that the previous ones do would serve no purpose, and baking them into Python itself will mean they can't evolve in important ways. I'd suggest we just put the brakes on and slate something better for 3.4. Does that make sense, or does that make people sad? +1. I think that makes sense. I'd be willing to work on something if I felt like I understood what was going on, which I don't. For example, distribute's goal seems very clear: provide a well-maintained and frequently-released setuptools. It would make sense (to me) to include distribute in a future release of Python, to close the loop of the non-std-lib-packaging-lib-built-on-top-distutils-but-never-included-in-the-stdlib-situation. This will never happen because Distribute's goal was 1/ to allow us to fix some bugs in setuptools and 2/ set some bridges for packaging adoption. But the final goal is to let it die where it is. But IIUC, it's too convoluted to attempt such a thing, and distutils2/packaging was meant to provide us the freedom to break backwards compat, but to what end? I don't think I fully understand the over-arching, real-world practical goals are for packaging in Python 3.x (other than to provide the pysetup command, and ini-style configuration over setup.py.) packaging provides two things: - reference implementations for the PEPs we've written - that can be used by any tool - a replacement for distutils-the-tool, with improvements and backward compatible helpers, like a way to browse packages installed by other
Re: [Distutils] command hooks...
On 5/16/12 3:58 AM, Chris McDonough wrote: Adding two more (packaging and distutils2) which are similarly semi-documented and which don't even solve the problems that the previous ones do would serve no purpose, and baking them into Python itself will mean they can't evolve in important ways. Oh, I think I need to answer to this too since you said you wanted to help. Packaging is not intended to be similar to setuptools in its features. For instance we won't provide console scripts or entry points. The first one because 'script' is the same feature (except there's an indirection and I said before we could mimic this) The second one because we should build this kind of feature outside the stdlib (this is not something most people use, according to the survey I did back a few years ago, it's mostly zope/plone/repoze land) I'd suggest you list what you can't do with packaging today and we work through that list to point which features are missing and should be developed *outside* the standard lib, and which ones are in packaging or should be IOW: packaging should only be the common basis and provide a basic installer - not a full fledge tool you can use to replace the most advanced setuptools features. And we want it pluggable enough so people can build pluggable features on the top of it, like Eric explained earlier Does that make sense ? Cheers Tarek ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] command hooks...
I think the biggest thing missing wrt to command hooks is the ability to have the tool automatically generate a OS specific wrapper (.exe for windows, extension less with a shebang for *NIX). While I think it would be useful for this feature to live in std lib, I don't think it's near as required as getting the core of packaging into stdlib. Attempting to create packaging as outside of stdlib brings up images of http://xkcd.com/927/, and to me the only thing prevent packaging from just being another standard is the Blessing that comes with being in stdlib. On Wednesday, May 16, 2012 at 2:55 AM, Tarek Ziadé wrote: On 5/16/12 3:58 AM, Chris McDonough wrote: Adding two more (packaging and distutils2) which are similarly semi-documented and which don't even solve the problems that the previous ones do would serve no purpose, and baking them into Python itself will mean they can't evolve in important ways. Oh, I think I need to answer to this too since you said you wanted to help. Packaging is not intended to be similar to setuptools in its features. For instance we won't provide console scripts or entry points. The first one because 'script' is the same feature (except there's an indirection and I said before we could mimic this) The second one because we should build this kind of feature outside the stdlib (this is not something most people use, according to the survey I did back a few years ago, it's mostly zope/plone/repoze land) I'd suggest you list what you can't do with packaging today and we work through that list to point which features are missing and should be developed *outside* the standard lib, and which ones are in packaging or should be IOW: packaging should only be the common basis and provide a basic installer - not a full fledge tool you can use to replace the most advanced setuptools features. And we want it pluggable enough so people can build pluggable features on the top of it, like Eric explained earlier Does that make sense ? Cheers Tarek ___ Distutils-SIG maillist - Distutils-SIG@python.org (mailto:Distutils-SIG@python.org) http://mail.python.org/mailman/listinfo/distutils-sig ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] command hooks...
On 16 May 2012 07:55, Tarek Ziadé ta...@ziade.org wrote: I'd suggest you list what you can't do with packaging today and we work through that list to point which features are missing and should be developed *outside* the standard lib, and which ones are in packaging or should be This would be a very good step - but rather than simply getting responses in the mailing list, can I suggest that we need some sort of central location where the features still outstanding for packaging can be tracked. Call it a roadmap if you like. Maybe it should be a PEP - simply because I can't think of a better place to put it, but I'm open to suggestions (I don't think the bug tracker is the right place, fwiw). At the moment, the biggest issue I see is that there are lots of discussions about what people believe is missing, but nothing clearly documenting what's intended to be there (and what is not - for example, your comment about entry points). As a starter, my key missing requirement is support for binary distributions - whether this is a new universal format, or whether it is reusing the bdist_wininst/bdist_msi formats, I don't really care, but it needs to be formalised with a migration path, backward compatibility support considered, etc. IOW: packaging should only be the common basis and provide a basic installer - not a full fledge tool you can use to replace the most advanced setuptools features. And we want it pluggable enough so people can build pluggable features on the top of it, like Eric explained earlier Does that make sense ? I would assume as a first guess that it should replace all of distutils, though? Ideally there should be no reason for people to use distutils for anything once packaging is available - am I right? Paul. ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] command hooks...
On 5/16/12 9:40 AM, Paul Moore wrote: On 16 May 2012 07:55, Tarek Ziadéta...@ziade.org wrote: I'd suggest you list what you can't do with packaging today and we work through that list to point which features are missing and should be developed *outside* the standard lib, and which ones are in packaging or should be This would be a very good step - but rather than simply getting responses in the mailing list, can I suggest that we need some sort of central location where the features still outstanding for packaging can be tracked. Call it a roadmap if you like. Maybe it should be a PEP - simply because I can't think of a better place to put it, but I'm open to suggestions (I don't think the bug tracker is the right place, fwiw). A wiki page sounds good, as long as each point turns into a bug at some point At the moment, the biggest issue I see is that there are lots of discussions about what people believe is missing, but nothing clearly documenting what's intended to be there (and what is not - for example, your comment about entry points). As a starter, my key missing requirement is support for binary distributions - whether this is a new universal format, or whether it is reusing the bdist_wininst/bdist_msi formats, I don't really care, but it needs to be formalised with a migration path, backward compatibility support considered, etc. Do you feel like you could start the windows story section on that wiki page ? IOW: packaging should only be the common basis and provide a basic installer - not a full fledge tool you can use to replace the most advanced setuptools features. And we want it pluggable enough so people can build pluggable features on the top of it, like Eric explained earlier Does that make sense ? I would assume as a first guess that it should replace all of distutils, though? Ideally there should be no reason for people to use distutils for anything once packaging is available - am I right? It does replace all distutils since it's a fork of it. The parts that is missing on purpose is bdist_rpm because we want anything RPM-related to be dealt outside the stdlib : there's a whole gap between how rhel/fedora guys do their packaging and what we offer with bdist_rpm, which seems to be perfect for... red hat 4 ? FWIW I have started the pypi2rpm project to group tools for RPM-izing python projects. Windows binaries were kept because it seems easy and feasible to maintain those from the stdlib -- but they still smell like old stuff because we don't have a Mr Windows in the distutils project. It was mostly Martin doing the legit work to make it work, and a couple of other people (forgive me if I forget someone) So... the windows binary story will improve if we get a champion for this. Cheers Tarek ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] command hooks...
On 16 May 2012 08:49, Tarek Ziadé ta...@ziade.org wrote: On 5/16/12 9:40 AM, Paul Moore wrote: This would be a very good step - but rather than simply getting responses in the mailing list, can I suggest that we need some sort of central location where the features still outstanding for packaging can be tracked. Call it a roadmap if you like. Maybe it should be a PEP - simply because I can't think of a better place to put it, but I'm open to suggestions (I don't think the bug tracker is the right place, fwiw). A wiki page sounds good, as long as each point turns into a bug at some point That sounds reasonable, I guess. As long as it's easy for people to find. As a starter, my key missing requirement is support for binary distributions - whether this is a new universal format, or whether it is reusing the bdist_wininst/bdist_msi formats, I don't really care, but it needs to be formalised with a migration path, backward compatibility support considered, etc. Do you feel like you could start the windows story section on that wiki page ? I'm happy to summarise this on there. Let me know a link to the wiki page. But 2 points: 1. Binary distributions is *not* a Windows issue. I originally pitched it as such, because that's where I care, but other people chimed in with an interest from a non-Windows POV. I can't comment on those requirements, though. And part of the problem is getting someone to make a unified statement. I drafted a PEP based around this, but Éric wanted to have a think about it, and it stalled. We also had other suggestions from Jim Fulton and Vinay Sanjip on various lists. So someone needs to take a lead on unifying all this. 2. The main Windows issue is that there are still a lot of bugs. Whenever I test stuff, I usually find myself hitting one or two bugs, often existing ones (I don't often get to report new bugs :-)) and then I'm stuck, unless I fix those bugs, which I often can't (or don't have the time to). So my efforts stall. I'll have another try at using packaging in the 3.3a2 release and see what the picture looks like now. (Note - my key use case at the moment is pysetup install stuff from pypi, I'm not writing my own packages, so it's the backward compatibility with existing distutils-based packages that I'll be testing, and which is key for me). Windows binaries were kept because it seems easy and feasible to maintain those from the stdlib -- but they still smell like old stuff because we don't have a Mr Windows in the distutils project. It was mostly Martin doing the legit work to make it work, and a couple of other people (forgive me if I forget someone) So... the windows binary story will improve if we get a champion for this. Agreed. Unfortunately, I don't have the time to be such a champion. But nevertheless, the current story with distutils/setuptools/distribute on Windows is perfectly fine, so unless packaging can get to that standard, I can't honestly suggest to people that they should switch over. It's a high bar to reach (particularly with such limited Windows expertise available) but necessary, IMO. Paul. ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] command hooks...
Taking off Guido, I don't think he needs to be in the whole thread On 5/16/12 10:30 AM, Paul Moore wrote: I'm happy to summarise this on there. Let me know a link to the wiki page. I created http://wiki.python.org/moin/Packaging/Roadmap But 2 points: 1. Binary distributions is *not* a Windows issue. I originally pitched it as such, because that's where I care, but other people chimed in with an interest from a non-Windows POV. I can't comment on those requirements, though. And part of the problem is getting someone to make a unified statement. I drafted a PEP based around this, but Éric wanted to have a think about it, and it stalled. We also had other suggestions from Jim Fulton and Vinay Sanjip on various lists. So someone needs to take a lead on unifying all this. Yeah definitely agreed. Although it impacts windows the most because it's easier on other platform to deal with source archives. 2. The main Windows issue is that there are still a lot of bugs. Whenever I test stuff, I usually find myself hitting one or two bugs, often existing ones (I don't often get to report new bugs :-)) and then I'm stuck, unless I fix those bugs, which I often can't (or don't have the time to). So my efforts stall. Yeah. And those bugs where in Distutils. What I mean is that Packaging has not improved much in this corner of distutils, but has not made it worse, and the tools are still accessible under the new version. I'll have another try at using packaging in the 3.3a2 release and see what the picture looks like now. (Note - my key use case at the moment is pysetup install stuff from pypi, I'm not writing my own packages, so it's the backward compatibility with existing distutils-based packages that I'll be testing, and which is key for me). Ok thanks Windows binaries were kept because it seems easy and feasible to maintain those from the stdlib -- but they still smell like old stuff because we don't have a Mr Windows in the distutils project. It was mostly Martin doing the legit work to make it work, and a couple of other people (forgive me if I forget someone) So... the windows binary story will improve if we get a champion for this. Agreed. Unfortunately, I don't have the time to be such a champion. But nevertheless, the current story with distutils/setuptools/distribute on Windows is perfectly fine, so unless packaging can get to that standard, I can't honestly suggest to people that they should switch over. It's a high bar to reach (particularly with such limited Windows expertise available) but necessary, IMO. Definitely ! I am afraid the scope is so wide that it's impossible to address all problems for v1 Another part were we lack a champion: compilers Paul. ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] command hooks...
On Wed, May 16, 2012 at 2:55 AM, Tarek Ziadé ta...@ziade.org wrote: On 5/16/12 3:58 AM, Chris McDonough wrote: Adding two more (packaging and distutils2) which are similarly semi-documented and which don't even solve the problems that the previous ones do would serve no purpose, and baking them into Python itself will mean they can't evolve in important ways. Oh, I think I need to answer to this too since you said you wanted to help. Packaging is not intended to be similar to setuptools in its features. For instance we won't provide console scripts or entry points. The first one because 'script' is the same feature (except there's an indirection and I said before we could mimic this) I don't know what this means. Will we have something functionally-equivalent to console-scripts? Or will we have something more similar to the old distutils scripts functionality. If the later, then I doubt that packaging will work well with buildout. Jim ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] command hooks...
On 5/16/12 3:19 PM, Jim Fulton wrote: On Wed, May 16, 2012 at 2:55 AM, Tarek Ziadéta...@ziade.org wrote: On 5/16/12 3:58 AM, Chris McDonough wrote: Adding two more (packaging and distutils2) which are similarly semi-documented and which don't even solve the problems that the previous ones do would serve no purpose, and baking them into Python itself will mean they can't evolve in important ways. Oh, I think I need to answer to this too since you said you wanted to help. Packaging is not intended to be similar to setuptools in its features. For instance we won't provide console scripts or entry points. The first one because 'script' is the same feature (except there's an indirection and I said before we could mimic this) I don't know what this means. Will we have something functionally-equivalent to console-scripts? Or will we have something more similar to the old distutils scripts functionality. If the later, then I doubt that packaging will work well with buildout. Please explain us exactly what you mean with this feature. What we want to have in packaging is the old distutils srcipt feature, with an extra option: a way to point a callable instead of a module, and then a wrapper like setuptools does. If you need something else, please explain what it is. (there's no need to declare an entry point and iterate over all projects, just to find back a callable to run your script, right? you can just... import the callable) Jim ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] command hooks...
On Wed, May 16, 2012 at 9:26 AM, Tarek Ziadé ta...@ziade.org wrote: On 5/16/12 3:19 PM, Jim Fulton wrote: On Wed, May 16, 2012 at 2:55 AM, Tarek Ziadéta...@ziade.org wrote: On 5/16/12 3:58 AM, Chris McDonough wrote: Adding two more (packaging and distutils2) which are similarly semi-documented and which don't even solve the problems that the previous ones do would serve no purpose, and baking them into Python itself will mean they can't evolve in important ways. Oh, I think I need to answer to this too since you said you wanted to help. Packaging is not intended to be similar to setuptools in its features. For instance we won't provide console scripts or entry points. The first one because 'script' is the same feature (except there's an indirection and I said before we could mimic this) I don't know what this means. Will we have something functionally-equivalent to console-scripts? Or will we have something more similar to the old distutils scripts functionality. If the later, then I doubt that packaging will work well with buildout. Please explain us exactly what you mean with this feature. What we want to have in packaging is the old distutils srcipt feature, with an extra option: a way to point a callable instead of a module, and then a wrapper like setuptools does. That's exactly what buildout needs. Jim -- Jim Fulton http://www.linkedin.com/in/jimfulton Jerky is better than bacon! http://www.dublinstore.com/ ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] command hooks...
On 05/16/2012 02:35 AM, Tarek Ziadé wrote: I know the idea of having packaging in the stdlib is something some people disliked, from day 1. And if I recall correctly, you did not like this either back then. I've always been pro-stdlib-contains-a-package-installer-and-machinery, and I still am. I am -1 on for two reasons: 1/ your argument about people being baffled about which tool to use will be worse if we work outside the stdlib. Having packaging in the stdlib and distutils frozen here, leads the path: hey, the next tool in the stdlib is packaging It's fine to develop it inside the stdlib. It's harder to think about releasing it in 3.3 though without good docs. Do you think we have enough time to document it and define its scope and limitations and future plans in written form before 3.3beta1 ships? 2/ packaging is completely mature for some parts, like the PEP 386 implementation. And having the packaging.version module in the stdlib is a great step forward for standardization The pep 376 implementation is also providing great APIs to push all tools to inter-operate. That's good to know. If we develop this tool outside, I am afraid we will just add more confusion in the mix. Can you see packaging as a set of utility at this point, rather than a tool that's going to replace instantly all the legacy tools ? I can. - C ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] command hooks...
On 05/16/2012 02:55 AM, Tarek Ziadé wrote: On 5/16/12 3:58 AM, Chris McDonough wrote: Adding two more (packaging and distutils2) which are similarly semi-documented and which don't even solve the problems that the previous ones do would serve no purpose, and baking them into Python itself will mean they can't evolve in important ways. Oh, I think I need to answer to this too since you said you wanted to help. Packaging is not intended to be similar to setuptools in its features. For instance we won't provide console scripts or entry points. The first one because 'script' is the same feature (except there's an indirection and I said before we could mimic this) The second one because we should build this kind of feature outside the stdlib (this is not something most people use, according to the survey I did back a few years ago, it's mostly zope/plone/repoze land) I suspect many people don't really know they're using entry points when they are. PasteDeploy configuration files rely heavily on entry points. Those are used in various ways by Pylons, Pyramid, Zope, and Turbogears. (PasteDeploy itself been downloaded almost a million times from PyPI.) I'm fine with needing to depend on an external package to scan for entrypoint-like-things registered as the result of the installation of a distribution. But there will need to be a way to register entrypoint-like things (along the lines of arbitrary egg-info metadata) as the result of distribution installation using pysetup. Maybe that already exists. I'd suggest you list what you can't do with packaging today and we work through that list to point which features are missing and should be developed *outside* the standard lib, and which ones are in packaging or should be I can try. IOW: packaging should only be the common basis and provide a basic installer - not a full fledge tool you can use to replace the most advanced setuptools features. And we want it pluggable enough so people can build pluggable features on the top of it, like Eric explained earlier Does that make sense ? I'd like to say it does, but I'm afraid it does not. I can see shipping both machinery and a full-fledged installer that handles all the use cases of existing installers. I can also see shipping just machinery and no installer at all. But I can't really see shipping both machinery and an installer that we know doesn't service use cases that existing installers already do. At least I can't see doing that in order to service an external shipping deadline. - C ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] command hooks...
On 05/16/2012 02:35 AM, Tarek Ziadé wrote: 1/ your argument about people being baffled about which tool to use will be worse if we work outside the stdlib. Having packaging in the stdlib and distutils frozen here, leads the path: hey, the next tool in the stdlib is packaging It all depends on the documentation. If you have complete documentation that explains how to do everything I need to do, then I can use the new tool -- someday, when I have time to read that documentation, figure out how to use it, and convert my software. If you do not have complete documentation or if your documentation does not describe some feature that I need, then the message to me is quite clear: Don't use this -- it doesn't do what you need. If you want people to adopt this new tool someday, then you should avoid training everybody to think that it is incomplete and that it doesn't work. Don't ask everybody to look at it before it is ready. ( If this were a commercial effort, you might want to release incomplete software so that you can build market share against your competitors. But in this case, there are no alternative suppliers who might release something any day now. There are only the legacy tools. ) Can you see packaging as a set of utility at this point, rather than a tool that's going to replace instantly all the legacy tools ? If it is not going to replace the legacy tools, then why should I use it? My understanding is that the legacy tools already work, for some vague definition of work. Your task is to convince me that packaging is better -- which I think will be hard to do if packaging is not better. Mark S. ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] command hooks...
Hi again, Le 01/05/2012 14:28, Paul Moore a écrit : On 1 May 2012 17:40, Chris McDonoughchr...@plope.com wrote: Is there a PEP for the packaging package? Is there any sort of unfinished business I can help with? AFAIK, there's no specific PEP for packaging (there are a number of related PEPs, but nothing specific like a roadmap). Yep. distutils2/packaging implement PEP 345 (Metadata 1.2), 376 (dist-info directory a.k.a. installation database) and 386 (version numbers), and also the older Metadata PEPs like distutils, but there was no PEP to discuss inclusion of d2 in Python 3.3: it just happened when a core developer (Martin von Löwis) indicated he was opposed to work on features (to add support for PEP 384 — Stable ABI for example) outside of the Python repository. I think that no PEP was asked by anyone because distutils2 is forked from code already in the stdlib and it implements accepted PEPs. There are small and big features added or in progress, many of them inspired by setuptools, that don’t have a PEP though. As I said on my other reply there is no friendly list of issues or roadmap, only unsorted bugs and what’s in my mind. I'm sure Éric can give you much better pointers on what would be useful, but one issue I've tried to raise a few times, and more recently Jim Fulton raised here (http://mail.python.org/pipermail/distutils-sig/2012-March/018323.html) is that of binary distribution support in packaging2. I've never had the time to shepherd a proposal through beyond the initial debate stage, and I know it's not getting high on Éric's list of priorities, but it would be good to see some movement on this. Indeed, in private email with Paul I agreed on the importance of a binary distribution format and did a pre-publication review of his PEP, but we did not finish our discussion nor incorporated the alternate proposal that was discussed at PyCon and on the mailing list (which makes it hard to see a clear picture — PEPs are good :) It seems unlikely that this hard topic can be solved for Python 3.3 / distutils2 1.0; what can be done however is to make sure that the extensibility hooks in d2 are well tested and documented so that when a bdist PEP reaches agreement and is implemented, a simple pysetup call and two lines of config will be all it takes to be able to use the new command. I know that the situation is far from ideal, and far from our goals for 3.3, but anyway d2 was never intended as a full replacement for setuptools and pip (more on that in an upcoming reply to another of your messages when you listed the setuptools features used by Pyramid). I think packaging in Python 3.3 will be a first version put in the stdlib to gather feedback and reports, not a finished stable product. Regards ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] command hooks...
On 05/15/2012 04:39 PM, Éric Araujo wrote: Hi again, Le 01/05/2012 14:28, Paul Moore a écrit : On 1 May 2012 17:40, Chris McDonoughchr...@plope.com wrote: Is there a PEP for the packaging package? Is there any sort of unfinished business I can help with? AFAIK, there's no specific PEP for packaging (there are a number of related PEPs, but nothing specific like a roadmap). Yep. distutils2/packaging implement PEP 345 (Metadata 1.2), 376 (dist-info directory a.k.a. installation database) and 386 (version numbers), and also the older Metadata PEPs like distutils, but there was no PEP to discuss inclusion of d2 in Python 3.3: it just happened when a core developer (Martin von Löwis) indicated he was opposed to work on features (to add support for PEP 384 — Stable ABI for example) outside of the Python repository. I think that no PEP was asked by anyone because distutils2 is forked from code already in the stdlib and it implements accepted PEPs. There are small and big features added or in progress, many of them inspired by setuptools, that don’t have a PEP though. As I said on my other reply there is no friendly list of issues or roadmap, only unsorted bugs and what’s in my mind. OK. Is there a way for me to take a look at the unsorted bugs? I'm sure Éric can give you much better pointers on what would be useful, but one issue I've tried to raise a few times, and more recently Jim Fulton raised here (http://mail.python.org/pipermail/distutils-sig/2012-March/018323.html) is that of binary distribution support in packaging2. I've never had the time to shepherd a proposal through beyond the initial debate stage, and I know it's not getting high on Éric's list of priorities, but it would be good to see some movement on this. Indeed, in private email with Paul I agreed on the importance of a binary distribution format and did a pre-publication review of his PEP, but we did not finish our discussion nor incorporated the alternate proposal that was discussed at PyCon and on the mailing list (which makes it hard to see a clear picture — PEPs are good :) It seems unlikely that this hard topic can be solved for Python 3.3 / distutils2 1.0; what can be done however is to make sure that the extensibility hooks in d2 are well tested and documented so that when a bdist PEP reaches agreement and is implemented, a simple pysetup call and two lines of config will be all it takes to be able to use the new command. I know that the situation is far from ideal, and far from our goals for 3.3, but anyway d2 was never intended as a full replacement for setuptools and pip (more on that in an upcoming reply to another of your messages when you listed the setuptools features used by Pyramid). I think packaging in Python 3.3 will be a first version put in the stdlib to gather feedback and reports, not a finished stable product. I really don't want to add stop energy here, and I'm more than willing to row to get something going, but I'm afraid if that's the diagnosis, it means I'll personally have to oppose the inclusion of packaging in 3.3. We currently already have at least 3 competing solutions (setuptools, distribute, and distutils itself), and people are baffled about which to use and how to use it. Adding two more (packaging and distutils2) which are similarly semi-documented and which don't even solve the problems that the previous ones do would serve no purpose, and baking them into Python itself will mean they can't evolve in important ways. I'd suggest we just put the brakes on and slate something better for 3.4. Does that make sense, or does that make people sad? - C ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] command hooks...
Hi, On 5/15/12 9:58 PM, Chris McDonough wrote: On 05/15/2012 04:39 PM, Éric Araujo wrote: Hi again, Le 01/05/2012 14:28, Paul Moore a écrit : On 1 May 2012 17:40, Chris McDonoughchr...@plope.com wrote: Is there a PEP for the packaging package? Is there any sort of unfinished business I can help with? AFAIK, there's no specific PEP for packaging (there are a number of related PEPs, but nothing specific like a roadmap). Yep. distutils2/packaging implement PEP 345 (Metadata 1.2), 376 (dist-info directory a.k.a. installation database) and 386 (version numbers), and also the older Metadata PEPs like distutils, but there was no PEP to discuss inclusion of d2 in Python 3.3: it just happened when a core developer (Martin von Löwis) indicated he was opposed to work on features (to add support for PEP 384 — Stable ABI for example) outside of the Python repository. I think that no PEP was asked by anyone because distutils2 is forked from code already in the stdlib and it implements accepted PEPs. There are small and big features added or in progress, many of them inspired by setuptools, that don’t have a PEP though. As I said on my other reply there is no friendly list of issues or roadmap, only unsorted bugs and what’s in my mind. OK. Is there a way for me to take a look at the unsorted bugs? I'm sure Éric can give you much better pointers on what would be useful, but one issue I've tried to raise a few times, and more recently Jim Fulton raised here (http://mail.python.org/pipermail/distutils-sig/2012-March/018323.html) is that of binary distribution support in packaging2. I've never had the time to shepherd a proposal through beyond the initial debate stage, and I know it's not getting high on Éric's list of priorities, but it would be good to see some movement on this. Indeed, in private email with Paul I agreed on the importance of a binary distribution format and did a pre-publication review of his PEP, but we did not finish our discussion nor incorporated the alternate proposal that was discussed at PyCon and on the mailing list (which makes it hard to see a clear picture — PEPs are good :) It seems unlikely that this hard topic can be solved for Python 3.3 / distutils2 1.0; what can be done however is to make sure that the extensibility hooks in d2 are well tested and documented so that when a bdist PEP reaches agreement and is implemented, a simple pysetup call and two lines of config will be all it takes to be able to use the new command. I know that the situation is far from ideal, and far from our goals for 3.3, but anyway d2 was never intended as a full replacement for setuptools and pip (more on that in an upcoming reply to another of your messages when you listed the setuptools features used by Pyramid). I think packaging in Python 3.3 will be a first version put in the stdlib to gather feedback and reports, not a finished stable product. I really don't want to add stop energy here, and I'm more than willing to row to get something going, but I'm afraid if that's the diagnosis, it means I'll personally have to oppose the inclusion of packaging in 3.3. We currently already have at least 3 competing solutions (setuptools, distribute, and distutils itself), and people are baffled about which to use and how to use it. Adding two more (packaging and distutils2) which are similarly semi-documented and which don't even solve the problems that the previous ones do would serve no purpose, and baking them into Python itself will mean they can't evolve in important ways. I'd suggest we just put the brakes on and slate something better for 3.4. Does that make sense, or does that make people sad? +1. I think that makes sense. I'd be willing to work on something if I felt like I understood what was going on, which I don't. For example, distribute's goal seems very clear: provide a well-maintained and frequently-released setuptools. It would make sense (to me) to include distribute in a future release of Python, to close the loop of the non-std-lib-packaging-lib-built-on-top-distutils-but-never-included-in-the-stdlib-situation. But IIUC, it's too convoluted to attempt such a thing, and distutils2/packaging was meant to provide us the freedom to break backwards compat, but to what end? I don't think I fully understand the over-arching, real-world practical goals are for packaging in Python 3.x (other than to provide the pysetup command, and ini-style configuration over setup.py.) Alex - C ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig -- Alex Clark · http://pythonpackages.com ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] command hooks
Hi Rob, I looked all around the bug tracker to try to add a comment and a thank you for providing it, but I could not even find a login button or if I were even logged in... You can log in with the small form in the sidebar on the left. When you are logged in, a bug page appears as an HTML form instead of just text. My issues that I was having regarding the command hooks is now fixed and working perfectly for me! I am so excited and grateful for this patch... One of the developers had made some incredible changes, but I was not able to even run them as I was plagued by this bug until now... Do you know when it will be committed into the DistUtils2-1.0 code base? I don’t know what you are talking about :) You had a problem, found a patch (for distutils2?) and it fixed it? Cheers ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
[Distutils] command hooks
Greetings: I looked all around the bug tracker to try to add a comment and a thank you for providing it, but I could not even find a login button or if I were even logged in... My issues that I was having regarding the command hooks is now fixed and working perfectly for me! I am so excited and grateful for this patch... One of the developers had made some incredible changes, but I was not able to even run them as I was plagued by this bug until now... Do you know when it will be committed into the DistUtils2-1.0 code base? I am running Fedora Core rawhide/Fc18.x86_64, and it works wonderful for me... -- Sincerely yours, Rob G. Healey ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] command hooks...
On 04/21/2012 01:51 AM, Éric Araujo wrote:
Hi again,
Is Distutils2 kind of your baby? Simply, you are the only one working
on it!
It’s more like my foster kid. At first there was distutils, in the
standard library since 1999. After a few years it stopped being
actively developed, and people started to rely on undocumented behavior
and internal things. Four years ago Tarek Ziadé became the new
maintainer for distutils and attacked the big pile of bugs and feature
requests. He hit a barrier when his changes broke other people’s code,
so it was decided two years ago to stop adding new features or clean up
code in distutils and instead start anew in a project named distutils2,
where backward compatibility would not stall all improvement efforts.
That’s when I started contributing, as a Google Summer of Code student
and then a volunteer. Tarek focused on distutils2 and passed on the
maintenance of distutils to me, and later I also became the maintainer
of distutils2, i.e. I reply to most bugs and make the releases. Tarek
is still the lead of the project and weighs in for important changes.
Georg Brandl was trying to herd cats on outstanding PEPs today:
http://mail.python.org/pipermail/python-dev/2012-May/119157.html
Is there a PEP for the packaging package? Is there any sort of
unfinished business I can help with?
- C
I am curious as to what you mean by this statement, when I get
some spare round tuits?
That’s a wordplay from the expression “I’ll get around to it” → “I’ll
get a round tuit”.
From the bug tracker page, this has been around for over a year!
Bugs live from a few minutes to a few years; with more than 300 bugs on
my list, I can’t get to everything at the same time. For the last
months I’ve also been busy with paperwork and a new job. But be sure
the bug is not forgotten; it’s one of the things that will be fixed
before distutils2 1.0 and Python 3.3.0.
I applied the diff file that was attached to the page to the current
Python-3.3.0a2, and it doesn't make a bit of difference on the
outcome...
If you read the full bug report you’ll see that this patch was already
applied and only fixes setup hooks; I said that I was working on a patch
for commands hooks but I did not upload it as it was not finished.
I also added the path to sys.path, like this:
sys.path.append(os.getcwd()) and it doesn't make a difference!
Probably because you did that in your code, but that makes no difference
because it happens after pysetup is started and tries to import your
module. I don’t know if I’ll have time to fix this bug tomorrow, as
I’ll have mentoring to do, so no promises here.
After the upcoming Distutils2 sprint, when will the work be patched
into Distutils2? When will we see the benefit of all the work that
will be done???
The changes from the sprint will go into the main repository
immediately, and after a short while they will get ported into packaging
in the cpython repository. I want to release distutils2 1.0a5 soon
after the sprint.
1) http://dl.dropbox.com/u/47247655/create_setup_cfg.py
[...]
1) Will create the setup.cfg file if one is not there...?
As you’re using open('setup.cfg', 'w'), the file will be recreated each
time you run the script. I’m not sure you need that script; why not
just create the config file once and keep it updated when you have changes?
Regards
___
Distutils-SIG maillist - Distutils-SIG@python.org
http://mail.python.org/mailman/listinfo/distutils-sig
___
Distutils-SIG maillist - Distutils-SIG@python.org
http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] command hooks...
On 1 May 2012 17:40, Chris McDonough chr...@plope.com wrote: Georg Brandl was trying to herd cats on outstanding PEPs today: http://mail.python.org/pipermail/python-dev/2012-May/119157.html Is there a PEP for the packaging package? Is there any sort of unfinished business I can help with? AFAIK, there's no specific PEP for packaging (there are a number of related PEPs, but nothing specific like a roadmap). I'm sure Éric can give you much better pointers on what would be useful, but one issue I've tried to raise a few times, and more recently Jim Fulton raised here (http://mail.python.org/pipermail/distutils-sig/2012-March/018323.html) is that of binary distribution support in packaging2. I've never had the time to shepherd a proposal through beyond the initial debate stage, and I know it's not getting high on Éric's list of priorities, but it would be good to see some movement on this. I can offer opinions, Windows testing, and assistance, but I really don't have the time to make this happen. But I honestly believe that unless packaging has binary distribution support, it won't get critical mass, on Windows if nowhere else. Just a thought - if it's where your interests lie, that would be great. Paul. ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] command hooks...
On 05/01/2012 02:28 PM, Paul Moore wrote: On 1 May 2012 17:40, Chris McDonoughchr...@plope.com wrote: Georg Brandl was trying to herd cats on outstanding PEPs today: http://mail.python.org/pipermail/python-dev/2012-May/119157.html Is there a PEP for the packaging package? Is there any sort of unfinished business I can help with? AFAIK, there's no specific PEP for packaging (there are a number of related PEPs, but nothing specific like a roadmap). I'm sure Éric can give you much better pointers on what would be useful, but one issue I've tried to raise a few times, and more recently Jim Fulton raised here (http://mail.python.org/pipermail/distutils-sig/2012-March/018323.html) is that of binary distribution support in packaging2. I've never had the time to shepherd a proposal through beyond the initial debate stage, and I know it's not getting high on Éric's list of priorities, but it would be good to see some movement on this. I can offer opinions, Windows testing, and assistance, but I really don't have the time to make this happen. But I honestly believe that unless packaging has binary distribution support, it won't get critical mass, on Windows if nowhere else. Yeah, it's kinda required. We currently tell people to avoid pip because it doesn't support installation of binary distributions. It's just too hard to document the corner cases of using both pip and easy_install at the same time, or telling Windows people to use easy_install and UNIX people to use pip, or some combination thereof. I'm not sure if other features we (the Pylons project) make heavy use of like extras, console scripts, entry points, and testing hooks are currently supported, but if they aren't I'd be willing to help add them as necessary. We wouldn't really be able to use it without this stuff, in any case. Just a thought - if it's where your interests lie, that would be great. Paul. In a perfect world, I'd not be volunteering, but it looks like I have to if I want to be able to get to bitch. ;-) - C ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] command hooks...
Dear Eric:
On Fri, Apr 20, 2012 at 10:51 PM, Éric Araujo mer...@netwok.org wrote:
Hi again,
Is Distutils2 kind of your baby? Simply, you are the only one working on it!
It’s more like my foster kid. At first there was distutils, in the
standard library since 1999. After a few years it stopped being actively
developed, and people started to rely on undocumented behavior and internal
things. Four years ago Tarek Ziadé became the new maintainer for distutils
and attacked the big pile of bugs and feature requests. He hit a barrier
when his changes broke other people’s code, so it was decided two years ago
to stop adding new features or clean up code in distutils and instead start
anew in a project named distutils2, where backward compatibility would not
stall all improvement efforts. That’s when I started contributing, as a
Google Summer of Code student
and then a volunteer. Tarek focused on distutils2 and passed on the
maintenance of distutils to me, and later I also became the maintainer of
distutils2, i.e. I reply to most bugs and make the releases. Tarek is still
the lead of the project and weighs in for important changes.
I understand now! My role at gramps is something like that too in one
manner of updating the two plugins in the webreport directory,
NarrativeWeb.py and WebCal.py...
I am full owner and maintainer of my plugin, EditExifImageMetadata.py,
which allows editing and creation of metadata fields for images...
Now anyone else can also modify this plugin and also the plugins
listed above, but it mainly falls on my shoulders...
I am curious as to what you mean by this statement, when I get some spare
round tuits?
That’s a wordplay from the expression “I’ll get around to it” → “I’ll get a
round tuit”.
I like it...
From the bug tracker page, this has been around for over a year! Bugs live
from a few minutes to a few years; with more than 300 bugs on my list, I
can’t get to everything at the same time. For the last months I’ve also
been busy with paperwork and a new job. But be sure the bug is not
forgotten; it’s one of the things that will be fixed before distutils2 1.0
and Python 3.3.0.
After the sprint and you release Distutils2-1.0a5, would you also
update the version_info too? If you import distutils2, and then
distutils2.version_info, you get a3 not a4?
I applied the diff file that was attached to the page to the current
Python-3.3.0a2, and it doesn't make a bit of difference on the outcome...
Obviously, I didn't get that far in reading the bug tracker! I will
re-download Python-3.3.0a2 and re-install it without the extra diff
file...
If you read the full bug report you’ll see that this patch was already
applied and only fixes setup hooks; I said that I was working on a patch for
commands hooks but I did not upload it as it was not finished.
I am more than willing to do testing of the patch for you if you would
like? Is there a way for me to get svn or git access to the
developmental version of Distutils2?
Since I am legally blind and have kidney failure, I spend a whole lot
of time at home on my computer! It is also one extra thing that I do
at gramps is alpha and beta testing...
I also added the path to sys.path, like this:
sys.path.append(os.getcwd()) and it doesn't make a difference!
What would I do in Linux to add my directory to the permanent $PYTHONPATH???
Probably because you did that in your code, but that makes no difference
because it happens after pysetup is started and tries to import your module.
I don’t know if I’ll have time to fix this bug tomorrow, as I’ll have
mentoring to do, so no promises here.
Is there access to the bug tracker for simply Distutils2 only? I
would like to see it and start getting familiar with the Distutils2
code???
After the upcoming Distutils2 sprint, when will the work be patched into
Distutils2? When will we see the benefit of all the work that will be
done???
The changes from the sprint will go into the main repository
immediately, and after a short while they will get ported into packaging in
the cpython repository. I want to release distutils2 1.0a5 soon after the
sprint.
I will certainly be looking forward to it! Once again, I have a lot
of time on my hands, so I am willing to do alpha/ beta testing...
1) http://dl.dropbox.com/u/47247655/create_setup_cfg.py
[...]
1) Will create the setup.cfg file if one is not there...?
As you’re using open('setup.cfg', 'w'), the file will be recreated each time
you run the script. I’m not sure you need that script; why not just create
the config file once and keep it updated when you have changes?
You are correct about, why not just update the setup.cfg file instead
of creating one each time I make changes!
Sincerely yours,
Rob G. Healey
Regards
___
Distutils-SIG maillist - Distutils-SIG@python.org
Re: [Distutils] command hooks...
Hi, After the sprint and you release Distutils2-1.0a5, would you also update the version_info too? If you import distutils2, and then distutils2.version_info, you get a3 not a4? I forgot to update distutils2.__version__ when I cut my first release :) Now I will remember it. I will re-download Python-3.3.0a2 and re-install it without the extra diff file... Yeah, editing a real installation to test things is fragile. Better use a Mercurial clone that is not installed and that lets you revert changes. Help is found at http://docs.python.org/devguide . I am more than willing to do testing of the patch for you if you would like? Is there a way for me to get svn or git access to the developmental version of Distutils2? hg clone http://hg.python.org/distutils2 I haven’t finished my patch because I ended up fixing too many things at once: hooks need to add the project directory to sys.path, hooks errors should result in warnings but not stop all operation, and maybe another bug. I need to split that into smaller changes. Do you have a user account on bugs.python.org? Now that I know that you’re willing to test things or give your opinion as a user (thanks!), if you create an account it will makes it easy to add you to a bug discussion. What would I do in Linux to add my directory to the permanent $PYTHONPATH??? Many ways to do that, but it is not a correct solution here. When you run pysetup, it should put the project directory in sys.path without the user having to do anything (and with my patch it does). Is there access to the bug tracker for simply Distutils2 only? I would like to see it and start getting familiar with the Distutils2 code??? The bug tracker has categories, called components, and there is one for d2: http://is.gd/mFZfqo . Some of the bugs are tagged “easy” to mean that they are good introductory bugs for people wanting to work on d2. I also intend to write more wiki pages or blog posts to explain how the codebase is laid out and internally works to reduce the bus factor of d2. Best regards ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
[Distutils] command hooks...
Greetings All:
I would like to know if anyone could help me with this problem?
Here is where you can find the files that are involved in this issue:
1) http://dl.dropbox.com/u/47247655/setup.cfg
2) http://dl.dropbox.com/u/47247655/setup_custom.py
These files are held in the public folder of my DropBox account...
Any help that you can give to me would be greatly appreciated!
BTW, these two files are within the same directory within the same project!
Here is the error that I get:
[root@SwimmingAntelope gep-026]# pysetup run install_dist
running install_dist
setup_custom
Traceback (most recent call last):
File /usr/lib/python2.7/site-packages/distutils2/run.py, line 651, in main
return dispatcher()
File /usr/lib/python2.7/site-packages/distutils2/run.py, line 639,
in __call__
return func(self, self.args)
File /usr/lib/python2.7/site-packages/distutils2/run.py, line 91, in wrapper
return f(*args, **kwargs)
File /usr/lib/python2.7/site-packages/distutils2/run.py, line 285, in _run
dist.run_command(cmd, dispatcher.command_options[cmd])
File /usr/lib/python2.7/site-packages/distutils2/dist.py, line
709, in run_command
cmd_obj.run()
File /usr/lib/python2.7/site-packages/distutils2/command/install_dist.py,
line 509, in run
self.run_command('build')
File /usr/lib/python2.7/site-packages/distutils2/command/cmd.py,
line 330, in run_command
self.distribution.run_command(command)
File /usr/lib/python2.7/site-packages/distutils2/dist.py, line
707, in run_command
self.run_command_hooks(cmd_obj, 'pre_hook')
File /usr/lib/python2.7/site-packages/distutils2/dist.py, line
732, in run_command_hooks
raise PackagingModuleError(e)
PackagingModuleError: setup_custom
--
Sincerely yours,
Rob G. Healey
___
Distutils-SIG maillist - Distutils-SIG@python.org
http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] command hooks...
Good evening, BTW, these two files are within the same directory within the same project! Here is the error that I get: [...] PackagingModuleError: setup_custom This looks like http://bugs.python.org/issue11637 The project directory needs to be added to sys.path when running command hooks. I have a patch in progress and will commit it when I get some spare round tuits. Thank you for testing and sending feedback! Regards ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] command hooks...
Dear Eric: Is Distutils2 kind of your baby? Simply, you are the only one working on it! I am curious as to what you mean by this statement, when I get some spare round tuits? From the bug tracker page, this has been around for over a year! I applied the diff file that was attached to the page to the current Python-3.3.0a2, and it doesn't make a bit of difference on the outcome... I also added the path to sys.path, like this: sys.path.append(os.getcwd()) and it doesn't make a difference! After the upcoming Distutils2 sprint, when will the work be patched into Distutils2? When will we see the benefit of all the work that will be done??? 1) http://dl.dropbox.com/u/47247655/create_setup_cfg.py 2) http://dl.dropbox.com/u/47247655/setup_custom.py 3) http://dl.dropbox.com/u/47247655/setup.cfg 1) Will create the setup.cfg file if one is not there...? The rest, you know about already... I have updated 2 and 3 and uploaded the new ones... Sincerely yours, Rob G. Healey On Fri, Apr 20, 2012 at 5:32 PM, Éric Araujo mer...@netwok.org wrote: Good evening, BTW, these two files are within the same directory within the same project! Here is the error that I get: [...] PackagingModuleError: setup_custom This looks like http://bugs.python.org/issue11637 The project directory needs to be added to sys.path when running command hooks. I have a patch in progress and will commit it when I get some spare round tuits. Thank you for testing and sending feedback! Regards ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig -- Sincerely yours, Rob G. Healey ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] command hooks...
Hi again,
Is Distutils2 kind of your baby? Simply, you are the only one working
on it!
It’s more like my foster kid. At first there was distutils, in the
standard library since 1999. After a few years it stopped being
actively developed, and people started to rely on undocumented behavior
and internal things. Four years ago Tarek Ziadé became the new
maintainer for distutils and attacked the big pile of bugs and feature
requests. He hit a barrier when his changes broke other people’s code,
so it was decided two years ago to stop adding new features or clean up
code in distutils and instead start anew in a project named distutils2,
where backward compatibility would not stall all improvement efforts.
That’s when I started contributing, as a Google Summer of Code student
and then a volunteer. Tarek focused on distutils2 and passed on the
maintenance of distutils to me, and later I also became the maintainer
of distutils2, i.e. I reply to most bugs and make the releases. Tarek
is still the lead of the project and weighs in for important changes.
I am curious as to what you mean by this statement, when I get
some spare round tuits?
That’s a wordplay from the expression “I’ll get around to it” → “I’ll
get a round tuit”.
From the bug tracker page, this has been around for over a year!
Bugs live from a few minutes to a few years; with more than 300 bugs on
my list, I can’t get to everything at the same time. For the last
months I’ve also been busy with paperwork and a new job. But be sure
the bug is not forgotten; it’s one of the things that will be fixed
before distutils2 1.0 and Python 3.3.0.
I applied the diff file that was attached to the page to the current
Python-3.3.0a2, and it doesn't make a bit of difference on the
outcome...
If you read the full bug report you’ll see that this patch was already
applied and only fixes setup hooks; I said that I was working on a patch
for commands hooks but I did not upload it as it was not finished.
I also added the path to sys.path, like this:
sys.path.append(os.getcwd()) and it doesn't make a difference!
Probably because you did that in your code, but that makes no difference
because it happens after pysetup is started and tries to import your
module. I don’t know if I’ll have time to fix this bug tomorrow, as
I’ll have mentoring to do, so no promises here.
After the upcoming Distutils2 sprint, when will the work be patched
into Distutils2? When will we see the benefit of all the work that
will be done???
The changes from the sprint will go into the main repository
immediately, and after a short while they will get ported into packaging
in the cpython repository. I want to release distutils2 1.0a5 soon
after the sprint.
1) http://dl.dropbox.com/u/47247655/create_setup_cfg.py
[...]
1) Will create the setup.cfg file if one is not there...?
As you’re using open('setup.cfg', 'w'), the file will be recreated each
time you run the script. I’m not sure you need that script; why not
just create the config file once and keep it updated when you have changes?
Regards
___
Distutils-SIG maillist - Distutils-SIG@python.org
http://mail.python.org/mailman/listinfo/distutils-sig
