Re: [Distutils] Builders vs Installers
On 28 March 2013 02:05, Nick Coghlan wrote: > On Thu, Mar 28, 2013 at 11:43 AM, Donald Stufft wrote: >> I don't think you can, nor should you be able to, explicitly depend on >> something that is a VCS checkout. > > I find it more useful to think of the issue as whether or not you > allow publication of source tarballs to satisfy a dependency, or > *require* publication of a fully populated sdist. If you allow raw > source tarballs, then you effectively allow VCS checkouts as well. I > prefer requiring an explicit publication step, but we also need to > acknowledge that the installer ecosystem we're trying to replace > allows them, and some people are relying on that feature. To give a real-life example of this issue, on Windows IPython depends on PyReadline. But the released version (1.7.x) of PyReadline is Python 2 only. So if you are using IPython on Python 3, you have to also depend on PyReadline from git. Now IPython doesn't declare a dependency on the VCS version (it just depends on "pyreadline"). And pyreadline is sufficiently stagnant that it hasn't declared anything much. But as an *end user* I have to make sure I force pip to install pyreadline from VCS if I want a working system. Paul. ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
[Distutils] Self-contained boostrap scripts [was: Re: A new, experimental packaging tool: distil]
On Tue, Mar 26, 2013 at 11:08 AM, Paul Moore wrote: > On 26 March 2013 09:49, Philippe Ombredanne wrote: >> Would anyone know of a better way to package things in a single >> python-executable bootstrapping script file without obfuscating the >> source contents in compressed/encoded/obfuscated byte arrays? > > Packaging as a zip file is a good way - but on Windows the file needs > to be named xxx.py (which is surprising, to say the least :-)) for the > relevant file association to be triggered (and on Unix, a #! line > needs to be prepended). Paul: I was not talking about this type of zips, but rather the same used in virtualenv, i.e. a string in a .py file that contains an encoded zip. That string is then decoded and unzipped at runtime as in here: https://github.com/pypa/virtualenv/blob/develop/virtualenv.py#L1933 This is not a zip, not an egg, not a wheel but some egg-in-py, zip-in-py or wheel-in-py and is similar to a shar shell archive. My point was that on the one hand, I like the fact that everything is self contained in one single .py file that you can execute right away. On the other hand, I find it somewhat discomforting as an emerging best way to package and distribute self-contained bootstrap scripts. Yet I cannot think of a better way atm: for instance splitting things in non-encoded non-binary plain strings would be quite weird too. Virtualenv does it, distil is doing it now, pip tried some of it here https://github.com/pypa/pip/blob/develop/contrib/get-pip.py In contrast, buildout, distribute and setuptools bootstrap scripts do not embed their dependencies and either try to get them satisfied locally or attempt to download the requirements. Having some support to do self-contained bootstrap scripts (as in requiring no network access and embedding all their dependencies) using this shar style could be something to consider normalizing? -- Philippe Ombredanne +1 650 799 0949 | pombreda...@nexb.com DejaCode Enterprise at http://www.dejacode.com nexB Inc. at http://www.nexb.com ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Self-contained boostrap scripts [was: Re: A new, experimental packaging tool: distil]
PEP XXX Yet Another Python .ZIP File Extension One reason Python ZIP applications have not been more widely adopted is because they have no Windows file association and would have to be confusingly named .py to be invoked by the interpreter. Henceforth, files with the extension .pyz shall be known as Python ZIP files. They consist of a ZIP format archive containing at minimum __main__.py, concatenated to two lines #!python or #!pythonw (or the full path to the interpreter), and an explanation # This is a ZIP format archive executable by the Python interpreter. The launcher will be updated to understand this format and Python will register this filename association when it is installed. On Thu, Mar 28, 2013 at 7:40 AM, Philippe Ombredanne wrote: > On Tue, Mar 26, 2013 at 11:08 AM, Paul Moore wrote: >> On 26 March 2013 09:49, Philippe Ombredanne wrote: >>> Would anyone know of a better way to package things in a single >>> python-executable bootstrapping script file without obfuscating the >>> source contents in compressed/encoded/obfuscated byte arrays? >> >> Packaging as a zip file is a good way - but on Windows the file needs >> to be named xxx.py (which is surprising, to say the least :-)) for the >> relevant file association to be triggered (and on Unix, a #! line >> needs to be prepended). > Paul: > I was not talking about this type of zips, but rather the same used in > virtualenv, i.e. a string in a .py file that contains an encoded zip. > That string is then decoded and unzipped at runtime as in here: > https://github.com/pypa/virtualenv/blob/develop/virtualenv.py#L1933 > > This is not a zip, not an egg, not a wheel but some egg-in-py, > zip-in-py or wheel-in-py and is similar to a shar shell archive. > > My point was that on the one hand, I like the fact that everything is > self contained in one single .py file that you can execute right away. > On the other hand, I find it somewhat discomforting as an emerging > best way to package and distribute self-contained bootstrap scripts. > Yet I cannot think of a better way atm: for instance splitting things > in non-encoded non-binary plain strings would be quite weird too. > > Virtualenv does it, distil is doing it now, pip tried some of it here > https://github.com/pypa/pip/blob/develop/contrib/get-pip.py > In contrast, buildout, distribute and setuptools bootstrap scripts do > not embed their dependencies and either try to get them satisfied > locally or attempt to download the requirements. > Having some support to do self-contained bootstrap scripts (as in > requiring no network access and embedding all their dependencies) > using this shar style could be something to consider normalizing? > > -- > Philippe Ombredanne > > +1 650 799 0949 | pombreda...@nexb.com > DejaCode Enterprise at http://www.dejacode.com > nexB Inc. at http://www.nexb.com > ___ > Distutils-SIG maillist - Distutils-SIG@python.org > http://mail.python.org/mailman/listinfo/distutils-sig ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Self-contained boostrap scripts [was: Re: A new, experimental packaging tool: distil]
On 28 March 2013 11:40, Philippe Ombredanne wrote: > This is not a zip, not an egg, not a wheel but some egg-in-py, > zip-in-py or wheel-in-py and is similar to a shar shell archive. > > My point was that on the one hand, I like the fact that everything is > self contained in one single .py file that you can execute right away. > On the other hand, I find it somewhat discomforting as an emerging > best way to package and distribute self-contained bootstrap scripts. > Yet I cannot think of a better way atm: for instance splitting things > in non-encoded non-binary plain strings would be quite weird too. Yes, my point was that Vinay's usage could be covered by distributing distil as a zip file. All it is doing is decoding it's blob of data (which is an encoded zip file) and then adding the resulting zip to sys.path. The virtualenv situation is different, as there we are trying to ensure that we remain single-file while embedding things that are *not* modules to add to sys.path. And we don't want to download our dependencies because we need to be able to run with no internet connection. But you are right, the embedded script approach is not ideal. I hope that "embedded binary blobs" does not become a common approach. I'd much rather that "runnable zip files" became the norm. It's certainly possible now, but I don't think it's well enough known (and there are administrative issues like the file extension question on Windows that make it more awkward than it should be). Hence my comments, trying to raise awareness a bit. Thanks for the feedback, and in particular the reminder that virtualenv could do with looking at this... I've added a virtualenv issue to remind me to think some more about it. Paul ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Self-contained boostrap scripts [was: Re: A new, experimental packaging tool: distil]
If it was distributed as such virtualenv could read blobs out of its own zip file, use the get_data() API to read non-module, or add subdirectories inside its zip file to sys.path On Thu, Mar 28, 2013 at 8:32 AM, Paul Moore wrote: > On 28 March 2013 11:40, Philippe Ombredanne wrote: >> This is not a zip, not an egg, not a wheel but some egg-in-py, >> zip-in-py or wheel-in-py and is similar to a shar shell archive. >> >> My point was that on the one hand, I like the fact that everything is >> self contained in one single .py file that you can execute right away. >> On the other hand, I find it somewhat discomforting as an emerging >> best way to package and distribute self-contained bootstrap scripts. >> Yet I cannot think of a better way atm: for instance splitting things >> in non-encoded non-binary plain strings would be quite weird too. > > Yes, my point was that Vinay's usage could be covered by distributing > distil as a zip file. All it is doing is decoding it's blob of data > (which is an encoded zip file) and then adding the resulting zip to > sys.path. > > The virtualenv situation is different, as there we are trying to > ensure that we remain single-file while embedding things that are > *not* modules to add to sys.path. And we don't want to download our > dependencies because we need to be able to run with no internet > connection. But you are right, the embedded script approach is not > ideal. > > I hope that "embedded binary blobs" does not become a common approach. > I'd much rather that "runnable zip files" became the norm. It's > certainly possible now, but I don't think it's well enough known (and > there are administrative issues like the file extension question on > Windows that make it more awkward than it should be). Hence my > comments, trying to raise awareness a bit. > > Thanks for the feedback, and in particular the reminder that > virtualenv could do with looking at this... I've added a virtualenv > issue to remind me to think some more about it. > Paul > ___ > Distutils-SIG maillist - Distutils-SIG@python.org > http://mail.python.org/mailman/listinfo/distutils-sig ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Self-contained boostrap scripts [was: Re: A new, experimental packaging tool: distil]
On 28 March 2013 12:26, Daniel Holth wrote: > The launcher will be updated to understand this format and Python will > register this filename association when it is installed. The launcher should need no changes. The Python msi installer would need a change to register the new extension, though. And *creating* such zips is mildly annoying on Windows, due to a general lack of tool support for manipulating binary files in text editors. Oh, and wouldn't "#!/usr/bin/env python(w)" be a better header? That would work on Unix, and the launcher recognises that format. But +1 on the idea in general. Paul ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Self-contained boostrap scripts [was: Re: A new, experimental packaging tool: distil]
On Thu, Mar 28, 2013 at 8:43 AM, Paul Moore wrote: > On 28 March 2013 12:26, Daniel Holth wrote: >> The launcher will be updated to understand this format and Python will >> register this filename association when it is installed. > > The launcher should need no changes. The Python msi installer would > need a change to register the new extension, though. > > And *creating* such zips is mildly annoying on Windows, due to a > general lack of tool support for manipulating binary files in text > editors. > > Oh, and wouldn't "#!/usr/bin/env python(w)" be a better header? That > would work on Unix, and the launcher recognises that format. > > But +1 on the idea in general. > Paul There is no 'cat header zip > newzip'? ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Self-contained boostrap scripts [was: Re: A new, experimental packaging tool: distil]
On 28 March 2013 12:45, Daniel Holth wrote: > On Thu, Mar 28, 2013 at 8:43 AM, Paul Moore wrote: >> On 28 March 2013 12:26, Daniel Holth wrote: >>> The launcher will be updated to understand this format and Python will >>> register this filename association when it is installed. >> >> The launcher should need no changes. The Python msi installer would >> need a change to register the new extension, though. >> >> And *creating* such zips is mildly annoying on Windows, due to a >> general lack of tool support for manipulating binary files in text >> editors. >> >> Oh, and wouldn't "#!/usr/bin/env python(w)" be a better header? That >> would work on Unix, and the launcher recognises that format. >> >> But +1 on the idea in general. >> Paul > > There is no 'cat header zip > newzip'? There are multiple options. And text file vs binary file issues to cover. CMD.EXE: copy /b header+zip newzip Powershell: get-content header,zip -enc Byte | set-content newzip -enc Byte Powershell: cmd /c copy /b header+zip newzip (because the previous version is so ugly...) Or write a Python script, which is what I did. Yes, I know :-( Paul ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Self-contained boostrap scripts [was: Re: A new, experimental packaging tool: distil]
> From: Paul Moore > > Yes, my point was that Vinay's usage could be covered by distributing > distil as a zip file. All it is doing is decoding it's blob of data > (which is an encoded zip file) and then adding the resulting zip to > sys.path. [snip] > I hope that "embedded binary blobs" does not become a common approach. > I'd much rather that "runnable zip files" became the norm. I don't know if it's that important to distinguish between the two. I found the approach I'm using with distil to be a tad more flexible in my case. A runnable zip has the advantage that it's harder to tinker with, but with the way distil.py is at the moment, you can tweak e.g. its logging just by changing distil.py. (At some point soon it will have an optional configuration file to control some aspects of its behaviour, but that's by the by.) It also does a bit of processing to process -e and -p and relaunches with a new Python interpreter if needed - developing this logic was quicker because I didn't have to add my changes to the .zip each time I tweaked something. Regards, Vinay Sajip ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Self-contained boostrap scripts [was: Re: A new, experimental packaging tool: distil]
On Thu, Mar 28, 2013 at 9:11 AM, Vinay Sajip wrote: >> From: Paul Moore > >> >> Yes, my point was that Vinay's usage could be covered by distributing >> distil as a zip file. All it is doing is decoding it's blob of data >> (which is an encoded zip file) and then adding the resulting zip to >> sys.path. > [snip] >> I hope that "embedded binary blobs" does not become a common approach. >> I'd much rather that "runnable zip files" became the norm. > > I don't know if it's that important to distinguish between the two. I found > the approach I'm using with distil to be a tad more flexible in my case. A > runnable zip has the advantage that it's harder to tinker with, but with the > way distil.py is at the moment, you can tweak e.g. its logging just by > changing distil.py. (At some point soon it will have an optional > configuration file to control some aspects of its behaviour, but that's by > the by.) It also does a bit of processing to process -e and -p and relaunches > with a new Python interpreter if needed - developing this logic was quicker > because I didn't have to add my changes to the .zip each time I tweaked > something. > > Regards, > > Vinay Sajip > > ___ > Distutils-SIG maillist - Distutils-SIG@python.org > http://mail.python.org/mailman/listinfo/distutils-sig Also if you are looking for tweakability you can run a directory with the same contents of the .zip exactly the same as if it was a zip. ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Self-contained boostrap scripts [was: Re: A new, experimental packaging tool: distil]
On 28 March 2013 13:11, Vinay Sajip wrote: > I don't know if it's that important to distinguish between the two. I found > the approach I'm using with distil to be a tad more flexible in my case. A > runnable zip has the advantage that it's harder to tinker with, but with the > way distil.py is at the moment, you can tweak e.g. its logging just by > changing distil.py. (At some point soon it will have an optional > configuration file to control some aspects of its behaviour, but that's by > the by.) It also does a bit of processing to process -e and -p and relaunches > with a new Python interpreter if needed - developing this logic was quicker > because I didn't have to add my changes to the .zip each time I tweaked > something. Good point. Paul ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Self-contained boostrap scripts [was: Re: A new, experimental packaging tool: distil]
> From: Philippe Ombredanne > On the other hand, I find it somewhat discomforting as an emerging > best way to package and distribute self-contained bootstrap scripts. But what is the root cause of that discomfort? The distil approach is slightly more discoverable than a pure zip would be, but for the security conscious all the code is there and available for inspection (unlike installing a distribution directly from PyPI, which will pull you-know-not-what from the network). > Virtualenv does it, distil is doing it now, pip tried some of it here > https://github.com/pypa/pip/blob/develop/contrib/get-pip.py > In contrast, buildout, distribute and setuptools bootstrap scripts do > not embed their dependencies and either try to get them satisfied > locally or attempt to download the requirements. And all this time, they would have been vulnerable to a MITM attack on PyPI because PyPI didn't support verifiable SSL connections until recently. It's good to be cautious, but Bruce Schneier has plenty of stories about caution directed in the wrong directions. > Having some support to do self-contained bootstrap scripts (as in > requiring no network access and embedding all their dependencies) > using this shar style could be something to consider normalizing? It seems like a decision for individual developers or developer teams to make on a case-by-case basis - it doesn't seem like something that needs to be "officially" encouraged or discouraged. Regards, Vinay Sajip ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Self-contained boostrap scripts [was: Re: A new, experimental packaging tool: distil]
> From: Daniel Holth > Also if you are looking for tweakability you can run a directory with > the same contents of the .zip exactly the same as if it was a zip. Sure, but my smoke testing involved copying the tweaked distil.py to a network share, then running that file from other Windows, Linux and OS X machines - of course I could have copied whole directory trees, but doing it the way I've done works well enough for me :-) Regards, Vinay Sajip ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Self-contained boostrap scripts [was: Re: A new, experimental packaging tool: distil]
On Thu, Mar 28, 2013 at 2:33 PM, Vinay Sajip wrote: >> From: Philippe Ombredanne >> On the other hand, I find it somewhat discomforting as an emerging >> best way to package and distribute self-contained bootstrap scripts. >> Virtualenv does it, distil is doing it now, pip tried some of it here >> https://github.com/pypa/pip/blob/develop/contrib/get-pip.py >> In contrast, buildout, distribute and setuptools bootstrap scripts do >> not embed their dependencies and either try to get them satisfied >> locally or attempt to download the requirements. > > And all this time, they would have been vulnerable to a MITM attack > on PyPI because PyPI didn't support verifiable SSL connections > until recently. It's good to be cautious, but Bruce Schneier has > plenty of stories about caution directed in the wrong directions. I am not so worried about security... I brought the point here because this is the packaging and distribution list, and I see this as an emerging pattern for the packaging and distribution of bootstrap scripts and this is something that has not been discussed much before. Conceptually I find these no different from setup.py scripts, and these have been mostly normalized (or at the minimum have a conventional name and a conventional if not specified interface.) Yet today, for the all important core package and environment management tools, we have bootstrap scripts each with different interfaces and different approaches to self containment or no containment. I feel this is worth discussing as bootstrapping is where everything begins :) -- Philippe Ombredanne +1 650 799 0949 | pombreda...@nexb.com DejaCode Enterprise at http://www.dejacode.com nexB Inc. at http://www.nexb.com ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Self-contained boostrap scripts [was: Re: A new, experimental packaging tool: distil]
Not really trying to tell Vinay to rewrite his script, but IMHO if you
expect it unzip is a lot easier than
file.write(module.random_attribute.decode('base64')). The runnable zip
feature is awesome, not well enough known, and totally worth promoting
over the shar pattern; with some minimal tooling you'd be good to go.
On Thu, Mar 28, 2013 at 10:44 AM, Philippe Ombredanne
wrote:
> On Thu, Mar 28, 2013 at 2:33 PM, Vinay Sajip wrote:
>>> From: Philippe Ombredanne
>>> On the other hand, I find it somewhat discomforting as an emerging
>>> best way to package and distribute self-contained bootstrap scripts.
>
>>> Virtualenv does it, distil is doing it now, pip tried some of it here
>>> https://github.com/pypa/pip/blob/develop/contrib/get-pip.py
>>> In contrast, buildout, distribute and setuptools bootstrap scripts do
>>> not embed their dependencies and either try to get them satisfied
>>> locally or attempt to download the requirements.
>>
>> And all this time, they would have been vulnerable to a MITM attack
>> on PyPI because PyPI didn't support verifiable SSL connections
>> until recently. It's good to be cautious, but Bruce Schneier has
>> plenty of stories about caution directed in the wrong directions.
>
> I am not so worried about security... I brought the point here because
> this is the packaging and distribution list, and I see this as an
> emerging pattern for the packaging and distribution of bootstrap
> scripts and this is something that has not been discussed much before.
>
> Conceptually I find these no different from setup.py scripts, and
> these have been mostly normalized (or at the minimum have a
> conventional name and a conventional if not specified interface.)
>
> Yet today, for the all important core package and environment
> management tools, we have bootstrap scripts each with different
> interfaces and different approaches to self containment or no
> containment.
>
> I feel this is worth discussing as bootstrapping is where everything begins :)
>
> --
> Philippe Ombredanne
>
> +1 650 799 0949 | pombreda...@nexb.com
> DejaCode Enterprise at http://www.dejacode.com
> nexB Inc. at http://www.nexb.com
> ___
> Distutils-SIG maillist - Distutils-SIG@python.org
> http://mail.python.org/mailman/listinfo/distutils-sig
___
Distutils-SIG maillist - Distutils-SIG@python.org
http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Self-contained boostrap scripts [was: Re: A new, experimental packaging tool: distil]
Philippe Ombredanne nexb.com> writes: > Conceptually I find these no different from setup.py scripts, and > these have been mostly normalized (or at the minimum have a > conventional name and a conventional if not specified interface.) Except that you programmatically interface (to distutils or setuptools) with setup.py, which is not the case with virtualenv or distil. > I feel this is worth discussing as bootstrapping is where everything begins :) Oh, certainly it's worthy of discussion - I wasn't meaning to imply otherwise. Regards, Vinay Sajip ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Self-contained boostrap scripts [was: Re: A new, experimental packaging tool: distil]
Daniel Holth gmail.com> writes:
> file.write(module.random_attribute.decode('base64')). The runnable zip
> feature is awesome, not well enough known, and totally worth promoting
> over the shar pattern; with some minimal tooling you'd be good to go.
Well, maybe the promoting would be better done by actually shipping something
this way that shows how well it works, rather than just talking about it. I
don't see that the user experience is any better with runnable zips, though I'm
not saying it's any worse. After that, it just comes down to individual
developer
taste, and there's no accounting for that :-)
Regards,
Vinay Sajip
___
Distutils-SIG maillist - Distutils-SIG@python.org
http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Importable wheels using distlib/distil
Jim Fulton zope.com> writes: > >> It would be far better IMO to just unzip the wheel and put that in > >> your path. (I'm hoping that wheels used this way are a suitable > >> replacement for eggs.) > > > > Well that's tantamount to installing the wheel, > > Not really. If you just unzip the wheel and add it to your path, > you can stop using it by just removing from your path. If you > install the wheel, it's contents will be poured into site-packages > (and other places).It's much heavier than just adding the > wheel (zipped or unzipped) to your path. > [snip] > by adding (unzipped) eggs to sys.path. Various plugin > systems (including buildout itself with extensions and recipes) > do this dynamically at run time. It's very useful. Thanks for the feedback. How about if I change mount()/unmount() to: def mount(self, append=False, destdir=None): """ Unzip the wheel's contents to the specified directory, or to a temporary directory if destdir is None. Add this directory to sys.path, either appending or prepending according to whether append is True or False. Before doing this, check that the wheel is compatible with the Python making the call to mount(). If successful, this makes the contents of the wheel's root directory - both Python packages and C extensions - importable via normal Python import mechanisms. """ def unmount(self): """ Remove the directory that was used for mounting from sys.path, thus making the wheel's code no longer importable. Return this directory. Note that the caller is responsible for deleting this directory and its contents, which might not be possible - e.g. in Windows, if a shared library has been imported and is linked to the running Python process, there will be an open handle to the shared library which will prevent its deletion. """ Regards, Vinay Sajip ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Importable wheels using distlib/distil
On 28 March 2013 16:02, Vinay Sajip wrote: > Return this directory. Note that the caller is responsible for > deleting this directory and its contents, which might not be > possible - e.g. in Windows, if a shared library has been > imported and is linked to the running Python process, there will be > an open handle to the shared library which will prevent its deletion. > """ That's the big issue I have with *any* approach like this. It's entirely possible that the directory cannot be deleted, and as a result the user ends up with the problem of managing clutter caused by this mechanism. Even if the directory is in %TEMP% the user still has the issue of clearing up. Consider a buildslave that continually runs tests - temp directory clutter is a definite issue in a situation like that. And of course, if an application user chooses to use this mechanism, I don't have an option to opt out unless we start getting into complex "if the package is installed use it, otherwise mount our internal wheel" logic. I'd like to hold off on this feature until there are actual requests for the functionality. It's not easy to argue against the idea purely on a "it might go wrong" basis without actual use cases to look at and see if/how they would handle the problem situations. Paul. ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Importable wheels using distlib/distil
On Thu, Mar 28, 2013 at 12:02 PM, Vinay Sajip wrote: > Jim Fulton zope.com> writes: > > >> >> It would be far better IMO to just unzip the wheel and put that in >> >> your path. (I'm hoping that wheels used this way are a suitable >> >> replacement for eggs.) >> > >> > Well that's tantamount to installing the wheel, >> >> Not really. If you just unzip the wheel and add it to your path, >> you can stop using it by just removing from your path. If you >> install the wheel, it's contents will be poured into site-packages >> (and other places).It's much heavier than just adding the >> wheel (zipped or unzipped) to your path. >> > [snip] >> by adding (unzipped) eggs to sys.path. Various plugin >> systems (including buildout itself with extensions and recipes) >> do this dynamically at run time. It's very useful. > > Thanks for the feedback. Thanks for trying to provide a useful feature. I hope my comments aren't too much of a downer. > How about if I change mount()/unmount() to: > > def mount(self, append=False, destdir=None): > """ > Unzip the wheel's contents to the specified directory, or to > a temporary directory if destdir is None. Add this directory to > sys.path, either appending or prepending according to whether > append is True or False. > > Before doing this, check that the wheel is compatible with the > Python making the call to mount(). > > If successful, this makes the contents of the wheel's root directory > - both Python packages and C extensions - importable via normal Python > import mechanisms. > """ > > def unmount(self): > """ > Remove the directory that was used for mounting from sys.path, > thus making the wheel's code no longer importable. > > Return this directory. Note that the caller is responsible for > deleting this directory and its contents, which might not be > possible - e.g. in Windows, if a shared library has been > imported and is linked to the running Python process, there will be > an open handle to the shared library which will prevent its deletion. > """ I'm not sure which users or use cases you're trying to serve here, so I'm not sure what to think of this. For buildout users, buildout would download and extract the wheel the first time it's used and keep it in a cache and then add it to a path at script generation time. For buildout's own uses (extensions and recipes) it would simply add the extracted wheel's location to sys.path at run time (downloading and extracting it first if necessary). So the win for buildout and it's users is to be able to have extracted (but not "installed" wheels) around to be mixed and matched either for script generation or run-time use. If I wasn't using buildout, I kinda doubt I'd want to use something like this rather than just installing wheels with pip. Jim P.S. I'm happy to see all the work you've done on distlib. I'm sorry to say I haven't had time to dig into it yet. I assume that buildout 3 will be based on it at some point. -- Jim Fulton http://www.linkedin.com/in/jimfulton ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Importable wheels using distlib/distil
Am 28.03.2013 17:42, schrieb Paul Moore: On 28 March 2013 16:02, Vinay Sajip wrote: Return this directory. Note that the caller is responsible for deleting this directory and its contents, which might not be possible - e.g. in Windows, if a shared library has been imported and is linked to the running Python process, there will be an open handle to the shared library which will prevent its deletion. """ That's the big issue I have with *any* approach like this. It's entirely possible that the directory cannot be deleted, and as a result the user ends up with the problem of managing clutter caused by this mechanism. Even if the directory is in %TEMP% the user still has the issue of clearing up. Consider a buildslave that continually runs tests - temp directory clutter is a definite issue in a situation like that. I made an experiment some time ago: It is possible to delete shared libs containing extension modules imported by Python if the Python process (after Py_Finalize()) calls FreeLibrary(hmod) in a loop for every extension until FreeLibrary returns zero; then the shared lib file can be deleted. Thomas ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Importable wheels using distlib/distil
Paul Moore gmail.com> writes: > That's the big issue I have with *any* approach like this. It's > entirely possible that the directory cannot be deleted, and as a > result the user ends up with the problem of managing clutter caused by > this mechanism. Even if the directory is in %TEMP% the user still has > the issue of clearing up. Consider a buildslave that continually runs > tests - temp directory clutter is a definite issue in a situation like > that. > > And of course, if an application user chooses to use this mechanism, I > don't have an option to opt out unless we start getting into complex > "if the package is installed use it, otherwise mount our internal > wheel" logic. Well, if you use the feature because it has its uses, you have to work around any costs that it has. At least the problem isn't being ducked. Plus, given that the wheel format is open, it's not a lot of work for an application developer to do a zipfile.extractall() followed by a sys.path.append(), whether distlib's Wheel has a mount() or not. Having mount() might be facilitating a useful feature in a (slightly more) controlled fashion. > I'd like to hold off on this feature until there are actual requests > for the functionality. It's not easy to argue against the idea purely > on a "it might go wrong" basis without actual use cases to look at and > see if/how they would handle the problem situations. Didn't Jim Fulton say in a post in this thread that it was a useful feature? I'm presuming he based this on real-world experience, but perhaps he would care to clarify. In terms of "actual requests" - there haven't been any actual requests for anything, other than suggestions to improve features that I've unilaterally introduced. This is another instance of the same thing, it seems to me. This is a feature that eggs have but nothing else does, so it seems reasonable to see if we can have alternatives. AFAICT, your worries would apply to eggs too, it's nothing to do with wheels or distlib in particular ... Regards, Vinay Sajip ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Importable wheels using distlib/distil
Jim Fulton zope.com> writes: > So the win for buildout and it's users is to be able to have extracted > (but not "installed" wheels) around to be mixed and matched either for > script generation or run-time use. > > If I wasn't using buildout, I kinda doubt I'd want to use something > like this rather than just installing wheels with pip. Ok, thanks for the clarification. Regards, Vinay Sajip ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Self-contained boostrap scripts [was: Re: A new, experimental packaging tool: distil]
Daniel Holth gmail.com> writes:
> file.write(module.random_attribute.decode('base64')). The runnable zip
> feature is awesome, not well enough known, and totally worth promoting
> over the shar pattern; with some minimal tooling you'd be good to go.
Runnable zips sound great - I certainly haven't come across them before (or if
I have, I didn't see the potential at the time).
That said, from a Windows perspective, shebangs and mixed text/binary files
worry me. The better approach on Windows would be to take a new extension
(.pyz? .pyp[ackage]?) and associate that with the launcher. (File extensions on
Windows are the moral equivalent of shebang lines.)
Changing .zip in any way will upset anyone who has a utility for opening ZIP
files (i.e. everyone) and there's no way to launch files differently based on
content without changing that association. And, I'm almost certain that most if
not all existing ZIP tools on Windows will fail to open files with a shebang,
since they've never had to deal with them.
I also think that a runnable zip may be a better package installation option
than MSIs, but that's another issue :)
Cheers,
Steve
___
Distutils-SIG maillist - Distutils-SIG@python.org
http://mail.python.org/mailman/listinfo/distutils-sig
[Distutils] Merge catalog-sig and distutils-sig
Is there much point in keeping catalog-sig and distutils-sig separate? It seems to me that most of the same people are on both lists, and the topics almost always have consequences to both sides of the coin. So much so that it's often hard to pick *which* of the two (or both) lists you post too. Further confused by the fact that distutils is hopefully someday going to go away :) Not sure if there's some official process for requesting it or not, but I think we should merge the two lists and just make packaging-sig to umbrella the entire packaging topics. - Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA signature.asc Description: Message signed with OpenPGP using GPGMail ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] [Catalog-sig] Merge catalog-sig and distutils-sig
On Thu, Mar 28, 2013 at 2:22 PM, Donald Stufft wrote: > Is there much point in keeping catalog-sig and distutils-sig separate? Not IMO. > It seems to me that most of the same people are on both lists, and the topics > almost always have consequences to both sides of the coin. So much so that > it's often hard to pick *which* of the two (or both) lists you post too. > Further confused by the fact that distutils is hopefully someday going to go > away :) > > Not sure if there's some official process for requesting it or not, but I > think we should merge the two lists and just make packaging-sig to umbrella > the entire packaging topics. +1 Jim -- Jim Fulton http://www.linkedin.com/in/jimfulton ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] [Catalog-sig] Merge catalog-sig and distutils-sig
On Thu, Mar 28, 2013 at 14:22 -0400, Donald Stufft wrote: > Is there much point in keeping catalog-sig and distutils-sig separate? > > It seems to me that most of the same people are on both lists, and the topics > almost always have consequences to both sides of the coin. So much so that > it's often hard to pick *which* of the two (or both) lists you post too. > Further confused by the fact that distutils is hopefully someday going to go > away :) +1 > Not sure if there's some official process for requesting it or not, but I > think we should merge the two lists and just make packaging-sig to umbrella > the entire packaging topics. > > - > Donald Stufft > PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA > > ___ > Catalog-SIG mailing list > catalog-...@python.org > http://mail.python.org/mailman/listinfo/catalog-sig ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] [Catalog-sig] Merge catalog-sig and distutils-sig
On Thu, Mar 28, 2013 at 2:22 PM, Donald Stufft wrote: > Is there much point in keeping catalog-sig and distutils-sig separate? No. The last time this was brought up, there were objections, but I don't remember what they were. I'll let people who think there's a point worry about that. > Not sure if there's some official process for requesting it or not, but > I think we should merge the two lists and just make packaging-sig to > umbrella the entire packaging topics. There is the meta-sig, but the description is out-dated: http://mail.python.org/mailman/listinfo/meta-sig and the last message in the archives is dated 2011, and sparked no discussion: http://mail.python.org/pipermail/meta-sig/2011-June.txt +1 on merging the lists. -Fred -- Fred L. Drake, Jr. "A storm broke loose in my mind." --Albert Einstein ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] [Catalog-sig] Merge catalog-sig and distutils-sig
+1 ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] [Catalog-sig] Merge catalog-sig and distutils-sig
On Thu, Mar 28, 2013 at 3:14 PM, Fred Drake wrote: > On Thu, Mar 28, 2013 at 2:22 PM, Donald Stufft wrote: >> Is there much point in keeping catalog-sig and distutils-sig separate? > > No. > > The last time this was brought up, there were objections, but I don't > remember what they were. I'll let people who think there's a point > worry about that. > >> Not sure if there's some official process for requesting it or not, but >> I think we should merge the two lists and just make packaging-sig to >> umbrella the entire packaging topics. > > There is the meta-sig, but the description is out-dated: > > http://mail.python.org/mailman/listinfo/meta-sig > > and the last message in the archives is dated 2011, and sparked no > discussion: > > http://mail.python.org/pipermail/meta-sig/2011-June.txt > > +1 on merging the lists. Can we do it by just dropping catalog-sig and keeping distutils-sig? I'm afraid we might lose some important distutils-sig population if the process involves renaming the list, resubscribing, etc. I also *really* don't want to invalidate archive links to the distutils-sig archive. All in all, +1 on not having two lists, but I'm really worried about "breaking" distutils-sig. We're still going to be talking about "distribution utilities", after all. ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] [Catalog-sig] Merge catalog-sig and distutils-sig
On Mar 28, 2013, at 3:39 PM, PJ Eby wrote: > On Thu, Mar 28, 2013 at 3:14 PM, Fred Drake wrote: >> On Thu, Mar 28, 2013 at 2:22 PM, Donald Stufft wrote: >>> Is there much point in keeping catalog-sig and distutils-sig separate? >> >> No. >> >> The last time this was brought up, there were objections, but I don't >> remember what they were. I'll let people who think there's a point >> worry about that. >> >>> Not sure if there's some official process for requesting it or not, but >>> I think we should merge the two lists and just make packaging-sig to >>> umbrella the entire packaging topics. >> >> There is the meta-sig, but the description is out-dated: >> >>http://mail.python.org/mailman/listinfo/meta-sig >> >> and the last message in the archives is dated 2011, and sparked no >> discussion: >> >>http://mail.python.org/pipermail/meta-sig/2011-June.txt >> >> +1 on merging the lists. > > Can we do it by just dropping catalog-sig and keeping distutils-sig? > I'm afraid we might lose some important distutils-sig population if > the process involves renaming the list, resubscribing, etc. I also > *really* don't want to invalidate archive links to the distutils-sig > archive. > > All in all, +1 on not having two lists, but I'm really worried about > "breaking" distutils-sig. We're still going to be talking about > "distribution utilities", after all. Worst case I'm sure subscribers can be transferred and the existing archive kept intact. - Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA signature.asc Description: Message signed with OpenPGP using GPGMail ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] [Catalog-sig] Merge catalog-sig and distutils-sig
On Mar 28, 2013, at 3:39 PM, PJ Eby wrote: > On Thu, Mar 28, 2013 at 3:14 PM, Fred Drake wrote: >> On Thu, Mar 28, 2013 at 2:22 PM, Donald Stufft wrote: >>> Is there much point in keeping catalog-sig and distutils-sig separate? >> >> No. >> >> The last time this was brought up, there were objections, but I don't >> remember what they were. I'll let people who think there's a point >> worry about that. >> >>> Not sure if there's some official process for requesting it or not, but >>> I think we should merge the two lists and just make packaging-sig to >>> umbrella the entire packaging topics. >> >> There is the meta-sig, but the description is out-dated: >> >>http://mail.python.org/mailman/listinfo/meta-sig >> >> and the last message in the archives is dated 2011, and sparked no >> discussion: >> >>http://mail.python.org/pipermail/meta-sig/2011-June.txt >> >> +1 on merging the lists. > > Can we do it by just dropping catalog-sig and keeping distutils-sig? > I'm afraid we might lose some important distutils-sig population if > the process involves renaming the list, resubscribing, etc. I also > *really* don't want to invalidate archive links to the distutils-sig > archive. > > All in all, +1 on not having two lists, but I'm really worried about > "breaking" distutils-sig. We're still going to be talking about > "distribution utilities", after all. Don't care how it's done. I don't know Mailman enough to know what is possible or how easy things are. I thought packaging-sig sounded nice but if you can't rename + redirect or merge or something in mailman I'm down for whatever. - Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA signature.asc Description: Message signed with OpenPGP using GPGMail ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Self-contained boostrap scripts [was: Re: A new, experimental packaging tool: distil]
On Thu, Mar 28, 2013 at 1:54 PM, Steve Dower wrote: > And, I'm almost certain that most if not all existing ZIP tools on Windows > will fail to open files with a shebang, since they've never had to deal with > them. Actually, the opposite is true, at least for 3rd-party (non-Microsoft) archiving tools: they work even when there's a whole .exe file stuck on the front. ;-) Some of them require you to rename from .exe to .zip first, but some actually detect that an .exe is a stub in front of a zip file and give you extraction options in an Explorer right-click. So, no worries on the prepended data front, even if the extension is .zip. What you probably can't safely do is *modify* a .zip with prepended data... and there I'm just guessing, because I've never actually tried. ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Self-contained boostrap scripts [was: Re: A new, experimental packaging tool: distil]
On Thu, Mar 28, 2013 at 3:49 PM, PJ Eby wrote: > On Thu, Mar 28, 2013 at 1:54 PM, Steve Dower > wrote: >> And, I'm almost certain that most if not all existing ZIP tools on Windows >> will fail to open files with a shebang, since they've never had to deal with >> them. > > Actually, the opposite is true, at least for 3rd-party (non-Microsoft) > archiving tools: they work even when there's a whole .exe file stuck > on the front. ;-) > > Some of them require you to rename from .exe to .zip first, but some > actually detect that an .exe is a stub in front of a zip file and give > you extraction options in an Explorer right-click. > > So, no worries on the prepended data front, even if the extension is > .zip. What you probably can't safely do is *modify* a .zip with > prepended data... and there I'm just guessing, because I've never > actually tried. It will all work. ZIP is cool! Every offset is relative from the end of the file. The wheel distribution even has a ZipFile subclass that lets you pop() files off the end by truncating the file and rewriting the index. This will work on any ordinary zip file that is just the concatenation of the compressed files in zip directory order, without data or extra space between the compressed files. ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] [Catalog-sig] Merge catalog-sig and distutils-sig
On Thu, Mar 28, 2013 at 15:42 -0400, Donald Stufft wrote: > On Mar 28, 2013, at 3:39 PM, PJ Eby wrote: > > > On Thu, Mar 28, 2013 at 3:14 PM, Fred Drake wrote: > >> On Thu, Mar 28, 2013 at 2:22 PM, Donald Stufft wrote: > >>> Is there much point in keeping catalog-sig and distutils-sig separate? > >> > >> No. > >> > >> The last time this was brought up, there were objections, but I don't > >> remember what they were. I'll let people who think there's a point > >> worry about that. > >> > >>> Not sure if there's some official process for requesting it or not, but > >>> I think we should merge the two lists and just make packaging-sig to > >>> umbrella the entire packaging topics. > >> > >> There is the meta-sig, but the description is out-dated: > >> > >>http://mail.python.org/mailman/listinfo/meta-sig > >> > >> and the last message in the archives is dated 2011, and sparked no > >> discussion: > >> > >>http://mail.python.org/pipermail/meta-sig/2011-June.txt > >> > >> +1 on merging the lists. > > > > Can we do it by just dropping catalog-sig and keeping distutils-sig? > > I'm afraid we might lose some important distutils-sig population if > > the process involves renaming the list, resubscribing, etc. I also > > *really* don't want to invalidate archive links to the distutils-sig > > archive. > > > > All in all, +1 on not having two lists, but I'm really worried about > > "breaking" distutils-sig. We're still going to be talking about > > "distribution utilities", after all. > > Don't care how it's done. I don't know Mailman enough to know what is > possible or how easy things are. I thought packaging-sig sounded nice but if > you can't rename + redirect or merge or something in mailman I'm down for > whatever. I've moved lists even from external sites to python.org and renamed them (latest was pytest-dev). That part works nicely and people can continue to use the old ML address. Merging two lists however makes it harder to get redirects for the old archives. But why not just keep distutils-sig and catalog-sig archives, but have all their mail arrive at a new packaging-sig and begin a new archive for the latter? holger > - > Donald Stufft > PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA > > ___ > Catalog-SIG mailing list > catalog-...@python.org > http://mail.python.org/mailman/listinfo/catalog-sig ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] [Catalog-sig] Merge catalog-sig and distutils-sig
That should work. Sounds like a plan. On Thu, Mar 28, 2013 at 4:04 PM, holger krekel wrote: > On Thu, Mar 28, 2013 at 15:42 -0400, Donald Stufft wrote: >> On Mar 28, 2013, at 3:39 PM, PJ Eby wrote: >> >> > On Thu, Mar 28, 2013 at 3:14 PM, Fred Drake wrote: >> >> On Thu, Mar 28, 2013 at 2:22 PM, Donald Stufft wrote: >> >>> Is there much point in keeping catalog-sig and distutils-sig separate? >> >> >> >> No. >> >> >> >> The last time this was brought up, there were objections, but I don't >> >> remember what they were. I'll let people who think there's a point >> >> worry about that. >> >> >> >>> Not sure if there's some official process for requesting it or not, but >> >>> I think we should merge the two lists and just make packaging-sig to >> >>> umbrella the entire packaging topics. >> >> >> >> There is the meta-sig, but the description is out-dated: >> >> >> >>http://mail.python.org/mailman/listinfo/meta-sig >> >> >> >> and the last message in the archives is dated 2011, and sparked no >> >> discussion: >> >> >> >>http://mail.python.org/pipermail/meta-sig/2011-June.txt >> >> >> >> +1 on merging the lists. >> > >> > Can we do it by just dropping catalog-sig and keeping distutils-sig? >> > I'm afraid we might lose some important distutils-sig population if >> > the process involves renaming the list, resubscribing, etc. I also >> > *really* don't want to invalidate archive links to the distutils-sig >> > archive. >> > >> > All in all, +1 on not having two lists, but I'm really worried about >> > "breaking" distutils-sig. We're still going to be talking about >> > "distribution utilities", after all. >> >> Don't care how it's done. I don't know Mailman enough to know what is >> possible or how easy things are. I thought packaging-sig sounded nice but if >> you can't rename + redirect or merge or something in mailman I'm down for >> whatever. > > I've moved lists even from external sites to python.org and renamed them > (latest was pytest-dev). That part works nicely and people can continue > to use the old ML address. Merging two lists however makes it harder > to get redirects for the old archives. But why not just keep distutils-sig > and catalog-sig archives, but have all their mail arrive at > a new packaging-sig and begin a new archive for the latter? > > holger > > >> - >> Donald Stufft >> PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA >> > > > >> ___ >> Catalog-SIG mailing list >> catalog-...@python.org >> http://mail.python.org/mailman/listinfo/catalog-sig > > ___ > Catalog-SIG mailing list > catalog-...@python.org > http://mail.python.org/mailman/listinfo/catalog-sig ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] [Catalog-sig] Merge catalog-sig and distutils-sig
On Thu, Mar 28, 2013 at 3:43 PM, Donald Stufft wrote: > On Mar 28, 2013, at 3:39 PM, PJ Eby wrote: >> Can we do it by just dropping catalog-sig and keeping distutils-sig? >> I'm afraid we might lose some important distutils-sig population if >> the process involves renaming the list, resubscribing, etc. I also >> *really* don't want to invalidate archive links to the distutils-sig >> archive. >> >> All in all, +1 on not having two lists, but I'm really worried about >> "breaking" distutils-sig. We're still going to be talking about >> "distribution utilities", after all. > > Worst case I'm sure subscribers can be transferred and the existing archive > kept intact. That's a great way to have a bunch of people complaining that they never subscribed to packaging-sig, not to mention the part where it breaks everyone's mail filters. I really don't see any gains for renaming the list. It's not like we can go and scrub the entire internet of references to distutils-sig. ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] [Catalog-sig] Merge catalog-sig and distutils-sig
On Mar 28, 2013, at 4:04 PM, holger krekel wrote: > On Thu, Mar 28, 2013 at 15:42 -0400, Donald Stufft wrote: >> On Mar 28, 2013, at 3:39 PM, PJ Eby wrote: >> >>> On Thu, Mar 28, 2013 at 3:14 PM, Fred Drake wrote: On Thu, Mar 28, 2013 at 2:22 PM, Donald Stufft wrote: > Is there much point in keeping catalog-sig and distutils-sig separate? No. The last time this was brought up, there were objections, but I don't remember what they were. I'll let people who think there's a point worry about that. > Not sure if there's some official process for requesting it or not, but > I think we should merge the two lists and just make packaging-sig to > umbrella the entire packaging topics. There is the meta-sig, but the description is out-dated: http://mail.python.org/mailman/listinfo/meta-sig and the last message in the archives is dated 2011, and sparked no discussion: http://mail.python.org/pipermail/meta-sig/2011-June.txt +1 on merging the lists. >>> >>> Can we do it by just dropping catalog-sig and keeping distutils-sig? >>> I'm afraid we might lose some important distutils-sig population if >>> the process involves renaming the list, resubscribing, etc. I also >>> *really* don't want to invalidate archive links to the distutils-sig >>> archive. >>> >>> All in all, +1 on not having two lists, but I'm really worried about >>> "breaking" distutils-sig. We're still going to be talking about >>> "distribution utilities", after all. >> >> Don't care how it's done. I don't know Mailman enough to know what is >> possible or how easy things are. I thought packaging-sig sounded nice but if >> you can't rename + redirect or merge or something in mailman I'm down for >> whatever. > > I've moved lists even from external sites to python.org and renamed them > (latest was pytest-dev). That part works nicely and people can continue > to use the old ML address. Merging two lists however makes it harder > to get redirects for the old archives. But why not just keep distutils-sig > and catalog-sig archives, but have all their mail arrive at > a new packaging-sig and begin a new archive for the latter? > > holger > > >> - >> Donald Stufft >> PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA >> > > > >> ___ >> Catalog-SIG mailing list >> catalog-...@python.org >> http://mail.python.org/mailman/listinfo/catalog-sig > sounds good to me. - Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA signature.asc Description: Message signed with OpenPGP using GPGMail ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] [Catalog-sig] Merge catalog-sig and distutils-sig
As a mostly-lurker on both who would love to cut down on the number of lists I have to follow: a hearty +1! Jacob On Thu, Mar 28, 2013 at 1:22 PM, Donald Stufft wrote: > Is there much point in keeping catalog-sig and distutils-sig separate? > > It seems to me that most of the same people are on both lists, and the topics > almost always have consequences to both sides of the coin. So much so that > it's often hard to pick *which* of the two (or both) lists you post too. > Further confused by the fact that distutils is hopefully someday going to go > away :) > > Not sure if there's some official process for requesting it or not, but I > think we should merge the two lists and just make packaging-sig to umbrella > the entire packaging topics. > > - > Donald Stufft > PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA > > > ___ > Catalog-SIG mailing list > catalog-...@python.org > http://mail.python.org/mailman/listinfo/catalog-sig > ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] [Catalog-sig] Merge catalog-sig and distutils-sig
C'mon, folks, we're arguing about a name. That's about as close to literal bikeshedding as we could get. How about we just let whoever has the keys make the change in whatever way's easiest and most logical for them? Jacob ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Merge catalog-sig and distutils-sig
On Thu, Mar 28, 2013 at 7:22 PM, Donald Stufft wrote: > Is there much point in keeping catalog-sig and distutils-sig separate? > > It seems to me that most of the same people are on both lists, and the topics > almost always have consequences to both sides of the coin. So much so that > it's often hard to pick *which* of the two (or both) lists you post too. > Further confused by the fact that distutils is hopefully someday going to go > away :) > > Not sure if there's some official process for requesting it or not, but I > think we should merge the two lists and just make packaging-sig to umbrella > the entire packaging topics. +1 -- Philippe Ombredanne +1 650 799 0949 | pombreda...@nexb.com DejaCode Enterprise at http://www.dejacode.com nexB Inc. at http://www.nexb.com ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] [Catalog-sig] Merge catalog-sig and distutils-sig
I think I'm the only one on the list who probably would have objected but I'm on both now so whatever :-) Richard On 29 March 2013 07:32, PJ Eby wrote: > On Thu, Mar 28, 2013 at 3:43 PM, Donald Stufft wrote: >> On Mar 28, 2013, at 3:39 PM, PJ Eby wrote: >>> Can we do it by just dropping catalog-sig and keeping distutils-sig? >>> I'm afraid we might lose some important distutils-sig population if >>> the process involves renaming the list, resubscribing, etc. I also >>> *really* don't want to invalidate archive links to the distutils-sig >>> archive. >>> >>> All in all, +1 on not having two lists, but I'm really worried about >>> "breaking" distutils-sig. We're still going to be talking about >>> "distribution utilities", after all. >> >> Worst case I'm sure subscribers can be transferred and the existing archive >> kept intact. > > That's a great way to have a bunch of people complaining that they > never subscribed to packaging-sig, not to mention the part where it > breaks everyone's mail filters. > > I really don't see any gains for renaming the list. It's not like we > can go and scrub the entire internet of references to distutils-sig. > ___ > Distutils-SIG maillist - Distutils-SIG@python.org > http://mail.python.org/mailman/listinfo/distutils-sig ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] [Catalog-sig] Merge catalog-sig and distutils-sig
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 On 03/28/2013 04:32 PM, PJ Eby wrote: > On Thu, Mar 28, 2013 at 3:43 PM, Donald Stufft > wrote: >> On Mar 28, 2013, at 3:39 PM, PJ Eby wrote: >>> Can we do it by just dropping catalog-sig and keeping >>> distutils-sig? I'm afraid we might lose some important >>> distutils-sig population if the process involves renaming the >>> list, resubscribing, etc. I also *really* don't want to >>> invalidate archive links to the distutils-sig archive. >>> >>> All in all, +1 on not having two lists, but I'm really worried >>> about "breaking" distutils-sig. We're still going to be talking >>> about "distribution utilities", after all. >> >> Worst case I'm sure subscribers can be transferred and the existing >> archive kept intact. > > That's a great way to have a bunch of people complaining that they > never subscribed to packaging-sig, not to mention the part where it > breaks everyone's mail filters. > > I really don't see any gains for renaming the list. It's not like we > can go and scrub the entire internet of references to distutils-sig. Not to mention breaking the gmane.org gateway, and those of us who sip the firehose there instead of trying to swallow it via e-mail. Tres. - -- === Tres Seaver +1 540-429-0999 tsea...@palladion.com Palladion Software "Excellence by Design"http://palladion.com -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.11 (GNU/Linux) Comment: Using GnuPG with undefined - http://www.enigmail.net/ iEYEARECAAYFAlFUuS4ACgkQ+gerLs4ltQ4zXACguC0D2F3EEE7GT4DGXRa08hy7 FdYAoM56YpHef9J0ScKOdY2OHv/48LOv =3UtH -END PGP SIGNATURE- ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] [Catalog-sig] Merge catalog-sig and distutils-sig
On Mar 28, 2013, at 5:42 PM, Tres Seaver wrote: > Signed PGP part > On 03/28/2013 04:32 PM, PJ Eby wrote: > > On Thu, Mar 28, 2013 at 3:43 PM, Donald Stufft > > wrote: > >> On Mar 28, 2013, at 3:39 PM, PJ Eby wrote: > >>> Can we do it by just dropping catalog-sig and keeping > >>> distutils-sig? I'm afraid we might lose some important > >>> distutils-sig population if the process involves renaming the > >>> list, resubscribing, etc. I also *really* don't want to > >>> invalidate archive links to the distutils-sig archive. > >>> > >>> All in all, +1 on not having two lists, but I'm really worried > >>> about "breaking" distutils-sig. We're still going to be talking > >>> about "distribution utilities", after all. > >> > >> Worst case I'm sure subscribers can be transferred and the existing > >> archive kept intact. > > > > That's a great way to have a bunch of people complaining that they > > never subscribed to packaging-sig, not to mention the part where it > > breaks everyone's mail filters. > > > > I really don't see any gains for renaming the list. It's not like we > > can go and scrub the entire internet of references to distutils-sig. > > Not to mention breaking the gmane.org gateway, and those of us who sip > the firehose there instead of trying to swallow it via e-mail. > > > Tres. > - -- > === > Tres Seaver +1 540-429-0999 tsea...@palladion.com > Palladion Software "Excellence by Design"http://palladion.com > > > ___ > Distutils-SIG maillist - Distutils-SIG@python.org > http://mail.python.org/mailman/listinfo/distutils-sig This problem is inherent no matter what name is picked. GMane will need updated and some messages will need sent to tell people about the new name. No matter what at least one name isn't going to be used anymore. It's not that big of a deal. - Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA signature.asc Description: Message signed with OpenPGP using GPGMail ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Self-contained boostrap scripts [was: Re: A new, experimental packaging tool: distil]
Daniel Holth wrote: > On Thu, Mar 28, 2013 at 3:49 PM, PJ Eby wrote: >> On Thu, Mar 28, 2013 at 1:54 PM, Steve Dower >> wrote: >>> And, I'm almost certain that most if not all existing ZIP tools on >>> Windows will fail to open files with a shebang, since they've >>> never had to deal with them. >> >> Actually, the opposite is true, at least for 3rd-party (non-Microsoft) >> archiving tools: they work even when there's a whole .exe file stuck >> on the front. ;-) >> >> Some of them require you to rename from .exe to .zip first, but some >> actually detect that an .exe is a stub in front of a zip file and give >> you extraction options in an Explorer right-click. >> >> So, no worries on the prepended data front, even if the extension is >> .zip. What you probably can't safely do is *modify* a .zip with >> prepended data... and there I'm just guessing, because I've never >> actually tried. > > It will all work. ZIP is cool! Every offset is relative from the end of the > file. > > The wheel distribution even has a ZipFile subclass that lets you pop() files > off the end by truncating the file and rewriting the index. This will work > on any ordinary zip file that is just the concatenation of the compressed > files in zip directory order, without data or extra space between the > compressed files. Ah of course, I totally forgot that it works from the end of the file. I'd still rather they got a new extension though, since nobody is going to teach WinZip what "#! python" means. That's probably an issue for the handful of Windows devs on python-dev rather than here, though. Cheers, Steve ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Self-contained boostrap scripts [was: Re: A new, experimental packaging tool: distil]
On Thu, Mar 28, 2013 at 6:19 PM, Steve Dower wrote: > Daniel Holth wrote: >> On Thu, Mar 28, 2013 at 3:49 PM, PJ Eby wrote: >>> On Thu, Mar 28, 2013 at 1:54 PM, Steve Dower >>> wrote: And, I'm almost certain that most if not all existing ZIP tools on Windows will fail to open files with a shebang, since they've never had to deal with them. >>> >>> Actually, the opposite is true, at least for 3rd-party (non-Microsoft) >>> archiving tools: they work even when there's a whole .exe file stuck >>> on the front. ;-) >>> >>> Some of them require you to rename from .exe to .zip first, but some >>> actually detect that an .exe is a stub in front of a zip file and give >>> you extraction options in an Explorer right-click. >>> >>> So, no worries on the prepended data front, even if the extension is >>> .zip. What you probably can't safely do is *modify* a .zip with >>> prepended data... and there I'm just guessing, because I've never >>> actually tried. >> >> It will all work. ZIP is cool! Every offset is relative from the end of the >> file. >> >> The wheel distribution even has a ZipFile subclass that lets you pop() files >> off the end by truncating the file and rewriting the index. This will work >> on any ordinary zip file that is just the concatenation of the compressed >> files in zip directory order, without data or extra space between the >> compressed files. > > Ah of course, I totally forgot that it works from the end of the file. > > I'd still rather they got a new extension though, since nobody is going to > teach WinZip what "#! python" means. That's probably an issue for the handful > of Windows devs on python-dev rather than here, though. > > > Cheers, > Steve WinZip will ignore anything in the front of the file since the zip directory doesn't reference it. The #! shebang is for Unix, would point to the correct Python, and the +x flag would make it executable. The mini PEP is for the .pyz registration and for publicity. ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] [Catalog-sig] Merge catalog-sig and distutils-sig
On Thu, Mar 28, 2013 at 5:15 PM, Jacob Kaplan-Moss wrote: > C'mon, folks, we're arguing about a name. That's about as close to > literal bikeshedding as we could get. I'm not arguing about the *name*. I just don't see the point in making everybody subscribe to a new list and change their mail filters (and update every book and webpage out there that mentions the distutils-sig), because a few people want to *change* the name -- a change that AFAICT doesn't actually provide any tangible benefit to anybody whatsoever. > How about we just let whoever has the keys make the change in whatever way's > easiest and most logical for them? Because it's not up to just the person with the keys. Neither SIG is a mere mailing list, it's a Python special interest group, and SIGs have their own formation and termination processes. In particular, if you're going to start a new SIG, one of the requirements to be met is "in particular, no other SIG nor the general Python newsgroup is already more suitable" (per the Python SIG Creation Guidelines). It's hard to argue that distutils-sig isn't already more suitable than whatever is being proposed to take its place. ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] [Catalog-sig] Merge catalog-sig and distutils-sig
On Mar 28, 2013, at 7:28 PM, PJ Eby wrote: > On Thu, Mar 28, 2013 at 5:15 PM, Jacob Kaplan-Moss wrote: >> C'mon, folks, we're arguing about a name. That's about as close to >> literal bikeshedding as we could get. > > I'm not arguing about the *name*. I just don't see the point in > making everybody subscribe to a new list and change their mail filters > (and update every book and webpage out there that mentions the > distutils-sig), because a few people want to *change* the name -- a > change that AFAICT doesn't actually provide any tangible benefit to > anybody whatsoever. > > >> How about we just let whoever has the keys make the change in whatever way's >> easiest and most logical for them? > > Because it's not up to just the person with the keys. Neither SIG is > a mere mailing list, it's a Python special interest group, and SIGs > have their own formation and termination processes. > > In particular, if you're going to start a new SIG, one of the > requirements to be met is "in particular, no other SIG nor the general > Python newsgroup is already more suitable" (per the Python SIG > Creation Guidelines). It's hard to argue that distutils-sig isn't > already more suitable than whatever is being proposed to take its > place. A requirement for a SIG is also that it has a clear goal and a start and end date. distutils-sig's goal is the distutils module. And the "end date" requirements seems to be completely ignored anymore so arguing strict adherence to the rules seems to be a wash. I suggested packaging-sig because discussion jumps back and forth between distutils-sig and catalog-sig and neither name nor stated goal really reflected what the sig was actually about which was packaging in python in general. I also suggested packaging because it matched the other current sigs which are generic topics and not about a single module. But whatever, I hate the pointless duplication and just want to kill the overlap. - Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA signature.asc Description: Message signed with OpenPGP using GPGMail ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] [Catalog-sig] Merge catalog-sig and distutils-sig
> But whatever, I hate the pointless duplication and just want to kill the overlap. Agree, +1 to merging into one list. On Thu, Mar 28, 2013 at 4:45 PM, Donald Stufft wrote: > > On Mar 28, 2013, at 7:28 PM, PJ Eby wrote: > > > On Thu, Mar 28, 2013 at 5:15 PM, Jacob Kaplan-Moss > wrote: > >> C'mon, folks, we're arguing about a name. That's about as close to > >> literal bikeshedding as we could get. > > > > I'm not arguing about the *name*. I just don't see the point in > > making everybody subscribe to a new list and change their mail filters > > (and update every book and webpage out there that mentions the > > distutils-sig), because a few people want to *change* the name -- a > > change that AFAICT doesn't actually provide any tangible benefit to > > anybody whatsoever. > > > > > >> How about we just let whoever has the keys make the change in whatever > way's easiest and most logical for them? > > > > Because it's not up to just the person with the keys. Neither SIG is > > a mere mailing list, it's a Python special interest group, and SIGs > > have their own formation and termination processes. > > > > In particular, if you're going to start a new SIG, one of the > > requirements to be met is "in particular, no other SIG nor the general > > Python newsgroup is already more suitable" (per the Python SIG > > Creation Guidelines). It's hard to argue that distutils-sig isn't > > already more suitable than whatever is being proposed to take its > > place. > > A requirement for a SIG is also that it has a clear goal and a start and > end date. distutils-sig's goal is the distutils module. And the "end date" > requirements seems to be completely ignored anymore so arguing strict > adherence to the rules seems to be a wash. > > I suggested packaging-sig because discussion jumps back and forth between > distutils-sig and catalog-sig and neither name nor stated goal really > reflected what the sig was actually about which was packaging in python in > general. I also suggested packaging because it matched the other current > sigs which are generic topics and not about a single module. But whatever, > I hate the pointless duplication and just want to kill the overlap. > > > - > Donald Stufft > PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 > DCFA > > > ___ > Distutils-SIG maillist - Distutils-SIG@python.org > http://mail.python.org/mailman/listinfo/distutils-sig > > ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Merge catalog-sig and distutils-sig
On Mar 28, 2013, at 02:22 PM, Donald Stufft wrote: >Is there much point in keeping catalog-sig and distutils-sig separate? Without yet reading the whole thread, I'll just mention that it's probably easier to just retire one or the other mailing lists and divert all discussion to the other one. Of course, the archives for the retired list would be retained for historical purposes. In fact, sigs are *supposed* to be periodically reviewed for renewal or retirement, though I think practically speaking we haven't done that in a very long time. If there's consensus on what you want to do, please contact postmaster@ and let them know. Let's say you just want to retire catalog-sig: we can set up forwards to distutils-sig and let the former be an "acceptable alias" to the latter so postings will be accepted when addressed to either. Of course, folks on the defunct list should manually subscribe to the good list (i.e. opt-in). -Barry signature.asc Description: PGP signature ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] [Catalog-sig] Merge catalog-sig and distutils-sig
On Mar 28, 2013, at 03:42 PM, Donald Stufft wrote: >Don't care how it's done. I don't know Mailman enough to know what is >possible or how easy things are. I thought packaging-sig sounded nice but if >you can't rename + redirect or merge or something in mailman I'm down for >whatever. Renaming can be done, but it's a bit of a pain. Of course, we can keep the archives for any retired list, so urls don't need to break. OTOH, it's definitely easier just to keep distutils-sig and retire catalog-sig. -Barry signature.asc Description: PGP signature ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
