Re: [Distutils] PEP 427
No non system packages were involved in the crash. On Feb 5, 2014 12:04 PM, "Éric Araujo" wrote: > Le 30/01/2014 12:44, Daniel Holth a écrit : > >> We do actually need a better way to turn libraries on and off >> individually, for example, if I install the wrong set of Python >> libraries on Ubuntu the plugin system for somesuch package crashes; >> > > Really? I thought the Debuntu Python maintainer changed site-packages to > dist-packages to avoid system programs being influenced by local installs. > > Regards > > having everything installed importable all the time is a problem. >> > ___ > Distutils-SIG maillist - Distutils-SIG@python.org > https://mail.python.org/mailman/listinfo/distutils-sig > ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] PEP 427
Le 30/01/2014 12:44, Daniel Holth a écrit : We do actually need a better way to turn libraries on and off individually, for example, if I install the wrong set of Python libraries on Ubuntu the plugin system for somesuch package crashes; Really? I thought the Debuntu Python maintainer changed site-packages to dist-packages to avoid system programs being influenced by local installs. Regards having everything installed importable all the time is a problem. ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] PEP 427
On 31 Jan 2014 02:48, "Donald Stufft" wrote: > > > > How am i supposed to manage that using pip and virtual envs in production? > > The same way you'd use them in development? Hell I believe you can even do: > > $ virtualenv my_virtualenv > $ my_virtualenv/bin/pip install path/to/wheelhouse/* > > The point is it's just installed libraries, asking this question doesn't really make much > sense, it's like asking how do you use apt-get or yum in production. Most Java apps don't typically use those either, they just drop jar files in a directory on CLASSPATH :P (Hence why I only *tolerate* that approach for wheels for advanced use cases that can't easily be handled any other way, while strongly discouraging it in general) Cheers, Nick. > > > > > Thanks, > > Eugene > > > - > Donald Stufft > PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA > > > ___ > Distutils-SIG maillist - Distutils-SIG@python.org > https://mail.python.org/mailman/listinfo/distutils-sig > ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] PEP 427
On Thu, Jan 30, 2014 at 12:23 PM, Evgeny Sazhin wrote: > On Thu, Jan 30, 2014 at 11:48 AM, Donald Stufft wrote: >> >> On Jan 30, 2014, at 11:44 AM, Evgeny Sazhin wrote: >> >>> On Thu, Jan 30, 2014 at 9:38 AM, Donald Stufft wrote: On Jan 29, 2014, at 11:29 PM, Evgeny Sazhin wrote: > > On Jan 29, 2014, at 11:17 PM, Donald Stufft wrote: > >> >> On Jan 29, 2014, at 11:16 PM, Evgeny Sazhin wrote: >> >>> >>> On Jan 29, 2014, at 10:49 PM, Donald Stufft wrote: >>> On Jan 29, 2014, at 10:47 PM, Evgeny Sazhin wrote: >> >> Wheel is a package format. Packages are for transmitting and >> installing bits. If you want to make some kind of self-unpacking >> executable please do it with something built for it. makeself is an >> excellent choice for these. >> > > I didn't say anything about self-unpacking executable. Egg already > knows to do what is needed, so i was correct in expecting the wheel > to do the same. Plus the notion of packages for transmitting and > installing should not exclude the running and importing. Otherwise it > is useless, at least for my purposes. As discussed before - jar does > that just fine and it is a package format. > > Funny thing - wheel allows to do the same! Why would i want to use > anything else then??? Because Python is not Java and Wheels are not Jars. You'll find very few packages actually support being run from a zipped file and the failure modes are not always obvious. >>> >>> I understand that not a lot of currently existing project are using >>> this capability - but I'm 100% positive that if the running from wheel >>> would be properly supported on error handling level and officially >>> declared at least for the pure python - most of the people would be >>> happy to have that! If we think about that, why would i want to use >>> anything else other than wheel and pip if this pair gives the >>> possibility to >> >> Pip does not install zipped wheels, and while it's not entirely up to me >> I would be opposed to it getting the ability to do so. > > I might be poorly wording things - but i never said I want pip to install > the zipped wheel. It seems that you're missing the point a bit. > I'm totally fine with the way pip handles things. > > again briefly My idea is to use the following: > > central location - flat folder with wheels, accessible to read for > everybody in network. > > for development : pip and virtual env. project has the virtual env > created, dependencies are deployed and available for development and > debugging in a standard manner. When done the project is packaged into > wheel that is getting deployed to central location. > > To *run* the program: i would create a script that bases on the pip > ability to resolve dependencies and basing on the requirements.txt from > inside my wheel it would generate PYTHONPATH to prepend the starting call > like that: > PYTHONPATH=1.whl:2.whl; python 3.whl > > where 3.whl is program with __main__.py and 1.whl and 2.whl are > dependencies needed. This works as of now! Just use pip and virtualenv in production. It's bad form to install things differently in development than in production. It *will* lead to production only bugs and in the case of zip imports it'll lead to hard to diagnose errors and bugs that you'll never be able to reproduce in development. >>> >>> >>> I'm happy to concede the run from zip thing if somebody could explain >>> how should i use pip and virtualenv in production? >>> Currently it seems to be very clean and clear. I can have one folder >>> where i can dump multiple versions of the same project in wheel format >>> and having requirements.txt in each of them i can construct PYTHONPATH >>> and run things. Simple! >>> >>> How am i supposed to manage that using pip and virtual envs in production? >> >> The same way you'd use them in development? Hell I believe you can even do: >> >> $ virtualenv my_virtualenv >> $ my_virtualenv/bin/pip install path/to/wheelhouse/* >> >> The point is it's just installed libraries, asking this question doesn't >> really make much >> sense, it's like asking how do you use apt-get or yum in production. >> >>> > > Could you please reread this > https://mail.python.org/pipermail/distutils-sig/2014-January/023636.html > I'm explaining there what is the task at hand and why I'm favoring the > jar-like workflow for deployment. It's clear that you favor the jar-like workflow. You will have discover for yourself why it works or does not work for you. There is more mature existing jar-like support for eggs (which will continue to w
Re: [Distutils] PEP 427
On Thu, Jan 30, 2014 at 11:48 AM, Donald Stufft wrote: > > On Jan 30, 2014, at 11:44 AM, Evgeny Sazhin wrote: > >> On Thu, Jan 30, 2014 at 9:38 AM, Donald Stufft wrote: >>> >>> On Jan 29, 2014, at 11:29 PM, Evgeny Sazhin wrote: >>> On Jan 29, 2014, at 11:17 PM, Donald Stufft wrote: > > On Jan 29, 2014, at 11:16 PM, Evgeny Sazhin wrote: > >> >> On Jan 29, 2014, at 10:49 PM, Donald Stufft wrote: >> >>> >>> On Jan 29, 2014, at 10:47 PM, Evgeny Sazhin wrote: >>> > > Wheel is a package format. Packages are for transmitting and > installing bits. If you want to make some kind of self-unpacking > executable please do it with something built for it. makeself is an > excellent choice for these. > I didn't say anything about self-unpacking executable. Egg already knows to do what is needed, so i was correct in expecting the wheel to do the same. Plus the notion of packages for transmitting and installing should not exclude the running and importing. Otherwise it is useless, at least for my purposes. As discussed before - jar does that just fine and it is a package format. Funny thing - wheel allows to do the same! Why would i want to use anything else then??? >>> >>> Because Python is not Java and Wheels are not Jars. You'll find very >>> few packages actually support being run from a zipped file and the >>> failure modes are not always obvious. >> >> I understand that not a lot of currently existing project are using this >> capability - but I'm 100% positive that if the running from wheel would >> be properly supported on error handling level and officially declared at >> least for the pure python - most of the people would be happy to have >> that! If we think about that, why would i want to use anything else >> other than wheel and pip if this pair gives the possibility to > > Pip does not install zipped wheels, and while it's not entirely up to me > I would be opposed to it getting the ability to do so. I might be poorly wording things - but i never said I want pip to install the zipped wheel. It seems that you're missing the point a bit. I'm totally fine with the way pip handles things. again briefly My idea is to use the following: central location - flat folder with wheels, accessible to read for everybody in network. for development : pip and virtual env. project has the virtual env created, dependencies are deployed and available for development and debugging in a standard manner. When done the project is packaged into wheel that is getting deployed to central location. To *run* the program: i would create a script that bases on the pip ability to resolve dependencies and basing on the requirements.txt from inside my wheel it would generate PYTHONPATH to prepend the starting call like that: PYTHONPATH=1.whl:2.whl; python 3.whl where 3.whl is program with __main__.py and 1.whl and 2.whl are dependencies needed. This works as of now! >>> >>> Just use pip and virtualenv in production. It's bad form to install things >>> differently in development than in production. It *will* lead to production >>> only bugs and in the case of zip imports it'll lead to hard to diagnose >>> errors and bugs that you'll never be able to reproduce in development. >> >> >> I'm happy to concede the run from zip thing if somebody could explain >> how should i use pip and virtualenv in production? >> Currently it seems to be very clean and clear. I can have one folder >> where i can dump multiple versions of the same project in wheel format >> and having requirements.txt in each of them i can construct PYTHONPATH >> and run things. Simple! >> >> How am i supposed to manage that using pip and virtual envs in production? > > The same way you'd use them in development? Hell I believe you can even do: > > $ virtualenv my_virtualenv > $ my_virtualenv/bin/pip install path/to/wheelhouse/* > > The point is it's just installed libraries, asking this question doesn't > really make much > sense, it's like asking how do you use apt-get or yum in production. > >> Could you please reread this https://mail.python.org/pipermail/distutils-sig/2014-January/023636.html I'm explaining there what is the task at hand and why I'm favoring the jar-like workflow for deployment. Thanks, Eugnee ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] PEP 427
On Jan 30, 2014, at 11:44 AM, Evgeny Sazhin wrote: > On Thu, Jan 30, 2014 at 9:38 AM, Donald Stufft wrote: >> >> On Jan 29, 2014, at 11:29 PM, Evgeny Sazhin wrote: >> >>> >>> On Jan 29, 2014, at 11:17 PM, Donald Stufft wrote: >>> On Jan 29, 2014, at 11:16 PM, Evgeny Sazhin wrote: > > On Jan 29, 2014, at 10:49 PM, Donald Stufft wrote: > >> >> On Jan 29, 2014, at 10:47 PM, Evgeny Sazhin wrote: >> Wheel is a package format. Packages are for transmitting and installing bits. If you want to make some kind of self-unpacking executable please do it with something built for it. makeself is an excellent choice for these. >>> >>> I didn't say anything about self-unpacking executable. Egg already >>> knows to do what is needed, so i was correct in expecting the wheel to >>> do the same. Plus the notion of packages for transmitting and >>> installing should not exclude the running and importing. Otherwise it >>> is useless, at least for my purposes. As discussed before - jar does >>> that just fine and it is a package format. >>> >>> Funny thing - wheel allows to do the same! Why would i want to use >>> anything else then??? >> >> Because Python is not Java and Wheels are not Jars. You'll find very few >> packages actually support being run from a zipped file and the failure >> modes are not always obvious. > > I understand that not a lot of currently existing project are using this > capability - but I'm 100% positive that if the running from wheel would > be properly supported on error handling level and officially declared at > least for the pure python - most of the people would be happy to have > that! If we think about that, why would i want to use anything else other > than wheel and pip if this pair gives the possibility to Pip does not install zipped wheels, and while it's not entirely up to me I would be opposed to it getting the ability to do so. >>> >>> I might be poorly wording things - but i never said I want pip to install >>> the zipped wheel. It seems that you're missing the point a bit. >>> I'm totally fine with the way pip handles things. >>> >>> again briefly My idea is to use the following: >>> >>> central location - flat folder with wheels, accessible to read for >>> everybody in network. >>> >>> for development : pip and virtual env. project has the virtual env created, >>> dependencies are deployed and available for development and debugging in a >>> standard manner. When done the project is packaged into wheel that is >>> getting deployed to central location. >>> >>> To *run* the program: i would create a script that bases on the pip ability >>> to resolve dependencies and basing on the requirements.txt from inside my >>> wheel it would generate PYTHONPATH to prepend the starting call like that: >>> PYTHONPATH=1.whl:2.whl; python 3.whl >>> >>> where 3.whl is program with __main__.py and 1.whl and 2.whl are >>> dependencies needed. This works as of now! >> >> Just use pip and virtualenv in production. It's bad form to install things >> differently in development than in production. It *will* lead to production >> only bugs and in the case of zip imports it'll lead to hard to diagnose >> errors and bugs that you'll never be able to reproduce in development. > > > I'm happy to concede the run from zip thing if somebody could explain > how should i use pip and virtualenv in production? > Currently it seems to be very clean and clear. I can have one folder > where i can dump multiple versions of the same project in wheel format > and having requirements.txt in each of them i can construct PYTHONPATH > and run things. Simple! > > How am i supposed to manage that using pip and virtual envs in production? The same way you’d use them in development? Hell I believe you can even do: $ virtualenv my_virtualenv $ my_virtualenv/bin/pip install path/to/wheelhouse/* The point is it’s just installed libraries, asking this question doesn’t really make much sense, it’s like asking how do you use apt-get or yum in production. > > Thanks, > Eugene - Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA signature.asc Description: Message signed with OpenPGP using GPGMail ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] PEP 427
On Thu, Jan 30, 2014 at 9:38 AM, Donald Stufft wrote: > > On Jan 29, 2014, at 11:29 PM, Evgeny Sazhin wrote: > >> >> On Jan 29, 2014, at 11:17 PM, Donald Stufft wrote: >> >>> >>> On Jan 29, 2014, at 11:16 PM, Evgeny Sazhin wrote: >>> On Jan 29, 2014, at 10:49 PM, Donald Stufft wrote: > > On Jan 29, 2014, at 10:47 PM, Evgeny Sazhin wrote: > >>> >>> Wheel is a package format. Packages are for transmitting and installing >>> bits. If you want to make some kind of self-unpacking executable please >>> do it with something built for it. makeself is an excellent choice for >>> these. >>> >> >> I didn't say anything about self-unpacking executable. Egg already knows >> to do what is needed, so i was correct in expecting the wheel to do the >> same. Plus the notion of packages for transmitting and installing should >> not exclude the running and importing. Otherwise it is useless, at least >> for my purposes. As discussed before - jar does that just fine and it is >> a package format. >> >> Funny thing - wheel allows to do the same! Why would i want to use >> anything else then??? > > Because Python is not Java and Wheels are not Jars. You'll find very few > packages actually support being run from a zipped file and the failure > modes are not always obvious. I understand that not a lot of currently existing project are using this capability - but I'm 100% positive that if the running from wheel would be properly supported on error handling level and officially declared at least for the pure python - most of the people would be happy to have that! If we think about that, why would i want to use anything else other than wheel and pip if this pair gives the possibility to >>> >>> Pip does not install zipped wheels, and while it's not entirely up to me I >>> would be opposed to it getting the ability to do so. >> >> I might be poorly wording things - but i never said I want pip to install >> the zipped wheel. It seems that you're missing the point a bit. >> I'm totally fine with the way pip handles things. >> >> again briefly My idea is to use the following: >> >> central location - flat folder with wheels, accessible to read for everybody >> in network. >> >> for development : pip and virtual env. project has the virtual env created, >> dependencies are deployed and available for development and debugging in a >> standard manner. When done the project is packaged into wheel that is >> getting deployed to central location. >> >> To *run* the program: i would create a script that bases on the pip ability >> to resolve dependencies and basing on the requirements.txt from inside my >> wheel it would generate PYTHONPATH to prepend the starting call like that: >> PYTHONPATH=1.whl:2.whl; python 3.whl >> >> where 3.whl is program with __main__.py and 1.whl and 2.whl are dependencies >> needed. This works as of now! > > Just use pip and virtualenv in production. It's bad form to install things > differently in development than in production. It *will* lead to production > only bugs and in the case of zip imports it'll lead to hard to diagnose > errors and bugs that you'll never be able to reproduce in development. I'm happy to concede the run from zip thing if somebody could explain how should i use pip and virtualenv in production? Currently it seems to be very clean and clear. I can have one folder where i can dump multiple versions of the same project in wheel format and having requirements.txt in each of them i can construct PYTHONPATH and run things. Simple! How am i supposed to manage that using pip and virtual envs in production? Thanks, Eugene ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] PEP 427
On Jan 29, 2014, at 11:29 PM, Evgeny Sazhin wrote: > > On Jan 29, 2014, at 11:17 PM, Donald Stufft wrote: > >> >> On Jan 29, 2014, at 11:16 PM, Evgeny Sazhin wrote: >> >>> >>> On Jan 29, 2014, at 10:49 PM, Donald Stufft wrote: >>> On Jan 29, 2014, at 10:47 PM, Evgeny Sazhin wrote: >> >> Wheel is a package format. Packages are for transmitting and installing >> bits. If you want to make some kind of self-unpacking executable please >> do it with something built for it. makeself is an excellent choice for >> these. >> > > I didn’t say anything about self-unpacking executable. Egg already knows > to do what is needed, so i was correct in expecting the wheel to do the > same. Plus the notion of packages for transmitting and installing should > not exclude the running and importing. Otherwise it is useless, at least > for my purposes. As discussed before - jar does that just fine and it is > a package format. > > Funny thing - wheel allows to do the same! Why would i want to use > anything else then??? Because Python is not Java and Wheels are not Jars. You’ll find very few packages actually support being run from a zipped file and the failure modes are not always obvious. >>> >>> I understand that not a lot of currently existing project are using this >>> capability - but I’m 100% positive that if the running from wheel would be >>> properly supported on error handling level and officially declared at least >>> for the pure python - most of the people would be happy to have that! If we >>> think about that, why would i want to use anything else other than wheel >>> and pip if this pair gives the possibility to >> >> Pip does not install zipped wheels, and while it’s not entirely up to me I >> would be opposed to it getting the ability to do so. > > I might be poorly wording things - but i never said I want pip to install the > zipped wheel. It seems that you’re missing the point a bit. > I’m totally fine with the way pip handles things. > > again briefly My idea is to use the following: > > central location - flat folder with wheels, accessible to read for everybody > in network. > > for development : pip and virtual env. project has the virtual env created, > dependencies are deployed and available for development and debugging in a > standard manner. When done the project is packaged into wheel that is getting > deployed to central location. > > To *run* the program: i would create a script that bases on the pip ability > to resolve dependencies and basing on the requirements.txt from inside my > wheel it would generate PYTHONPATH to prepend the starting call like that: > PYTHONPATH=1.whl:2.whl; python 3.whl > > where 3.whl is program with __main__.py and 1.whl and 2.whl are dependencies > needed. This works as of now! Just use pip and virtualenv in production. It’s bad form to install things differently in development than in production. It *will* lead to production only bugs and in the case of zip imports it’ll lead to hard to diagnose errors and bugs that you’ll never be able to reproduce in development. Even though the spec apparently allows it, using Wheels like this is an anti pattern. > > > > >> >>> 1. use minimum of “standard” tools, >>> >>> 2. *manage dependencies* in a unified and standardized way >>> >>> 3. use only *one format* and *one artifact* per project for everything >>> related to the code development, shipment, deployment and execution. >>> >>> What would be a motivation to learn yet another tool like buildout or >>> anything, pollute infrastructure with more libraries, manage several >>> different ways of working with the code, overcomplicate the production area >>> by requiring stuff to be unzipped and whatnot? >>> >>> I hope nobody would want to strip any of those nice features from the >>> wheel, but improve the running from it!;) >> >> I don’t understand what any of the rest of this have to do with wether you >> add zipped Wheels to your sys.path or not. > > see above > > >> >>> >>> Thanks, >>> Eugene >>> > > Thanks, > Eugene > ___ > Distutils-SIG maillist - Distutils-SIG@python.org > https://mail.python.org/mailman/listinfo/distutils-sig - Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA >> >> >> - >> Donald Stufft >> PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA - Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA signature.asc Description: Message signed with OpenPGP using GPGMail ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/dis
Re: [Distutils] PEP 427
On Jan 29, 2014, at 11:17 PM, Donald Stufft wrote: > > On Jan 29, 2014, at 11:16 PM, Evgeny Sazhin wrote: > >> >> On Jan 29, 2014, at 10:49 PM, Donald Stufft wrote: >> >>> >>> On Jan 29, 2014, at 10:47 PM, Evgeny Sazhin wrote: >>> > > Wheel is a package format. Packages are for transmitting and installing > bits. If you want to make some kind of self-unpacking executable please > do it with something built for it. makeself is an excellent choice for > these. > I didn’t say anything about self-unpacking executable. Egg already knows to do what is needed, so i was correct in expecting the wheel to do the same. Plus the notion of packages for transmitting and installing should not exclude the running and importing. Otherwise it is useless, at least for my purposes. As discussed before - jar does that just fine and it is a package format. Funny thing - wheel allows to do the same! Why would i want to use anything else then??? >>> >>> Because Python is not Java and Wheels are not Jars. You’ll find very few >>> packages actually support being run from a zipped file and the failure >>> modes are not always obvious. >> >> I understand that not a lot of currently existing project are using this >> capability - but I’m 100% positive that if the running from wheel would be >> properly supported on error handling level and officially declared at least >> for the pure python - most of the people would be happy to have that! If we >> think about that, why would i want to use anything else other than wheel and >> pip if this pair gives the possibility to > > Pip does not install zipped wheels, and while it’s not entirely up to me I > would be opposed to it getting the ability to do so. I might be poorly wording things - but i never said I want pip to install the zipped wheel. It seems that you’re missing the point a bit. I’m totally fine with the way pip handles things. again briefly My idea is to use the following: central location - flat folder with wheels, accessible to read for everybody in network. for development : pip and virtual env. project has the virtual env created, dependencies are deployed and available for development and debugging in a standard manner. When done the project is packaged into wheel that is getting deployed to central location. To *run* the program: i would create a script that bases on the pip ability to resolve dependencies and basing on the requirements.txt from inside my wheel it would generate PYTHONPATH to prepend the starting call like that: PYTHONPATH=1.whl:2.whl; python 3.whl where 3.whl is program with __main__.py and 1.whl and 2.whl are dependencies needed. This works as of now! > >> 1. use minimum of “standard” tools, >> >> 2. *manage dependencies* in a unified and standardized way >> >> 3. use only *one format* and *one artifact* per project for everything >> related to the code development, shipment, deployment and execution. >> >> What would be a motivation to learn yet another tool like buildout or >> anything, pollute infrastructure with more libraries, manage several >> different ways of working with the code, overcomplicate the production area >> by requiring stuff to be unzipped and whatnot? >> >> I hope nobody would want to strip any of those nice features from the wheel, >> but improve the running from it!;) > > I don’t understand what any of the rest of this have to do with wether you > add zipped Wheels to your sys.path or not. see above > >> >> Thanks, >> Eugene >> >>> Thanks, Eugene ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig >>> >>> >>> - >>> Donald Stufft >>> PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA > > > - > Donald Stufft > PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] PEP 427
On Jan 29, 2014, at 11:16 PM, Evgeny Sazhin wrote: > > On Jan 29, 2014, at 10:49 PM, Donald Stufft wrote: > >> >> On Jan 29, 2014, at 10:47 PM, Evgeny Sazhin wrote: >> Wheel is a package format. Packages are for transmitting and installing bits. If you want to make some kind of self-unpacking executable please do it with something built for it. makeself is an excellent choice for these. >>> >>> I didn’t say anything about self-unpacking executable. Egg already knows to >>> do what is needed, so i was correct in expecting the wheel to do the same. >>> Plus the notion of packages for transmitting and installing should not >>> exclude the running and importing. Otherwise it is useless, at least for my >>> purposes. As discussed before - jar does that just fine and it is a package >>> format. >>> >>> Funny thing - wheel allows to do the same! Why would i want to use anything >>> else then??? >> >> Because Python is not Java and Wheels are not Jars. You’ll find very few >> packages actually support being run from a zipped file and the failure modes >> are not always obvious. > > I understand that not a lot of currently existing project are using this > capability - but I’m 100% positive that if the running from wheel would be > properly supported on error handling level and officially declared at least > for the pure python - most of the people would be happy to have that! If we > think about that, why would i want to use anything else other than wheel and > pip if this pair gives the possibility to Pip does not install zipped wheels, and while it’s not entirely up to me I would be opposed to it getting the ability to do so. > 1. use minimum of “standard” tools, > > 2. *manage dependencies* in a unified and standardized way > > 3. use only *one format* and *one artifact* per project for everything > related to the code development, shipment, deployment and execution. > > What would be a motivation to learn yet another tool like buildout or > anything, pollute infrastructure with more libraries, manage several > different ways of working with the code, overcomplicate the production area > by requiring stuff to be unzipped and whatnot? > > I hope nobody would want to strip any of those nice features from the wheel, > but improve the running from it!;) I don’t understand what any of the rest of this have to do with wether you add zipped Wheels to your sys.path or not. > > Thanks, > Eugene > >> >>> >>> Thanks, >>> Eugene >>> ___ >>> Distutils-SIG maillist - Distutils-SIG@python.org >>> https://mail.python.org/mailman/listinfo/distutils-sig >> >> >> - >> Donald Stufft >> PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA - Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA signature.asc Description: Message signed with OpenPGP using GPGMail ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] PEP 427
On Jan 29, 2014, at 10:49 PM, Donald Stufft wrote: > > On Jan 29, 2014, at 10:47 PM, Evgeny Sazhin wrote: > >>> >>> Wheel is a package format. Packages are for transmitting and installing >>> bits. If you want to make some kind of self-unpacking executable please do >>> it with something built for it. makeself is an excellent choice for these. >>> >> >> I didn’t say anything about self-unpacking executable. Egg already knows to >> do what is needed, so i was correct in expecting the wheel to do the same. >> Plus the notion of packages for transmitting and installing should not >> exclude the running and importing. Otherwise it is useless, at least for my >> purposes. As discussed before - jar does that just fine and it is a package >> format. >> >> Funny thing - wheel allows to do the same! Why would i want to use anything >> else then??? > > Because Python is not Java and Wheels are not Jars. You’ll find very few > packages actually support being run from a zipped file and the failure modes > are not always obvious. I understand that not a lot of currently existing project are using this capability - but I’m 100% positive that if the running from wheel would be properly supported on error handling level and officially declared at least for the pure python - most of the people would be happy to have that! If we think about that, why would i want to use anything else other than wheel and pip if this pair gives the possibility to 1. use minimum of “standard” tools, 2. *manage dependencies* in a unified and standardized way 3. use only *one format* and *one artifact* per project for everything related to the code development, shipment, deployment and execution. What would be a motivation to learn yet another tool like buildout or anything, pollute infrastructure with more libraries, manage several different ways of working with the code, overcomplicate the production area by requiring stuff to be unzipped and whatnot? I hope nobody would want to strip any of those nice features from the wheel, but improve the running from it!;) Thanks, Eugene > >> >> Thanks, >> Eugene >> ___ >> Distutils-SIG maillist - Distutils-SIG@python.org >> https://mail.python.org/mailman/listinfo/distutils-sig > > > - > Donald Stufft > PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] PEP 427
On Jan 29, 2014, at 10:47 PM, Evgeny Sazhin wrote: >> >> Wheel is a package format. Packages are for transmitting and installing >> bits. If you want to make some kind of self-unpacking executable please do >> it with something built for it. makeself is an excellent choice for these. >> > > I didn’t say anything about self-unpacking executable. Egg already knows to > do what is needed, so i was correct in expecting the wheel to do the same. > Plus the notion of packages for transmitting and installing should not > exclude the running and importing. Otherwise it is useless, at least for my > purposes. As discussed before - jar does that just fine and it is a package > format. > > Funny thing - wheel allows to do the same! Why would i want to use anything > else then??? Because Python is not Java and Wheels are not Jars. You’ll find very few packages actually support being run from a zipped file and the failure modes are not always obvious. > > Thanks, > Eugene > ___ > Distutils-SIG maillist - Distutils-SIG@python.org > https://mail.python.org/mailman/listinfo/distutils-sig - Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA signature.asc Description: Message signed with OpenPGP using GPGMail ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] PEP 427
> > Wheel is a package format. Packages are for transmitting and installing bits. > If you want to make some kind of self-unpacking executable please do it with > something built for it. makeself is an excellent choice for these. > I didn’t say anything about self-unpacking executable. Egg already knows to do what is needed, so i was correct in expecting the wheel to do the same. Plus the notion of packages for transmitting and installing should not exclude the running and importing. Otherwise it is useless, at least for my purposes. As discussed before - jar does that just fine and it is a package format. Funny thing - wheel allows to do the same! Why would i want to use anything else then??? Thanks, Eugene ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] PEP 427
On Jan 29, 2014, at 6:34 PM, Evgeny Sazhin wrote: > Guys, > > I'm still unable to see my emails reaching the list for unknown reason... > Any ideas what can be the problem? > > FWIW I have tested it by adding the same __main__.py i used for the > egg variant of the distribution > to the wheel root and specifying > $ PYTHONPATH=projectA.whl; python executable_projectB.whl > > I was able to run the program! > So it seems like the stuff does behave like a jar for pure python modules! > > Don't know who to thank for that - because nobody seems to have > claimed this functionality;) > Well Nick said it supposed to be that way, So please don't break it!!;) Wether this was “supposed” to be there is a matter that’s up for debate. If you’re just using these for runtime you don’t need the Wheel format at all. Just regular old pyzips will do just fine. http://docs.python.org/2/library/zipfile#pyzipfile-objects > > It also means that I have the full picture in place without the > necessity to use the buildout or anything else other then the pip and > wheel from a flat folder structure! I'd appreciate if you could let me > know about any problems you could foresee (we are talking pure python > only) > > Thanks! > Eugene > > > On Wed, Jan 29, 2014 at 5:36 PM, Evgeny Sazhin wrote: >> I'm sorry for possible dup, but for whatever reason i don't see this >> email reaching the list, so i'm resending. >> >> >> On Wed, Jan 29, 2014 at 12:50 PM, Evgeny Sazhin wrote: >>> >>> >>> >>> >>> On Wed, Jan 29, 2014 at 9:11 AM, Vinay Sajip >>> wrote: > Does it mean that it actually makes sense to look into that > direction and make wheel usage closer to jar? There is a parallel discussion going on, with the title "Using Wheel with zipimport", which is relevant to this question, and other questions you raised (e.g. about supporting C extensions/pure-Python modules. >> >> >> >> I read all of it and got a bit lost in between the distil API and PEP >> process discussion;) >> >>> >>> > I have no knowledge about c extensions scope, but i feel > like it might be of less importance then pure python > packaging issues? Am I wrong? A lot of Python users depend on C extensions - and while it is a subset of all Python users, it is a large (and important) subset. Example: any usage of Python in numerical analysis or scientific applications involves use of C extensions. Regards, Vinay Sajip >>> >>> >>> >> >> I can see that it might be quite beneficial to have virtualenv and pip >> installing wheels locally for development needs, so here is what i was >> able to come up with so far: >> >> I have one folder on NFS where all python developed stuff should be >> *deployed* - pyhtonlib. It is impossible to use pip or virtualenv >> there - so i'm bound to artifacts. The only way something can appear >> there is by using the "release" program that knows how to put >> artifacts in specified locations. Currently most of the stuff there is >> the .py modules and few eggs (some are executable). But this stuff is >> not allowing for sane dependency management, neither code reuse. I >> actually don't like the idea of specifying dependencies in the code >> via sys.path. I think the resolved sys.path based on requirements.txt >> is much better solution. >> >> So i'm looking for a solution that would allow to use the same >> artifact for everything (like jar) so it can guarantee that the same >> subset of code that was tested, goes to production and used in dev. >> Currently I'm leaning towards using pip's capability to work with flat >> folders via --find-links, so i can deploy wheels to the pythonlib and >> then reuse them in the development environment. >> >> But in this setup how do i make my program executable from pythonlib >> location? I think I should I create some smart runner script that >> would be able to use the pip's dependency resolution, create the >> necessary sys.path basing on the wheel requirements.txt and then my >> program wheel should have an entry point like __main__.py >> >> As Nick pointed out the wheel is a superset of the egg - so I assume >> wheels can be executable, correct? How do i achieve that? > ___ > Distutils-SIG maillist - Distutils-SIG@python.org > https://mail.python.org/mailman/listinfo/distutils-sig - Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA signature.asc Description: Message signed with OpenPGP using GPGMail ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] PEP 427
On Jan 29, 2014, at 9:50 AM, Evgeny Sazhin wrote: > > > > On Wed, Jan 29, 2014 at 9:11 AM, Vinay Sajip wrote: > > Does it mean that it actually makes sense to look into that > > direction and make wheel usage closer to jar? > > There is a parallel discussion going on, with the title "Using Wheel with > zipimport", > which is relevant to this question, and other questions you raised (e.g. about > supporting C extensions/pure-Python modules. > > > I read all of it and got a bit lost in between the distil API and PEP process > discussion;) > > > > I have no knowledge about c extensions scope, but i feel > > like it might be of less importance then pure python > > packaging issues? Am I wrong? > > A lot of Python users depend on C extensions - and while it is a subset of > all Python > users, it is a large (and important) subset. Example: any usage of Python in > numerical analysis or scientific applications involves use of C extensions. > > Regards, > > Vinay Sajip > > > I can see that it might be quite beneficial to have virtualenv and pip > installing wheels locally for development needs, so here is what i was able > to come up with so far: > > I have one folder on NFS where all python developed stuff should be > *deployed* - pyhtonlib. It is impossible to use pip or virtualenv there - so > i'm bound to artifacts. The only way something can appear there is by using > the "release" program that knows how to put artifacts in specified locations. > Currently most of the stuff there is the .py modules and few eggs (some are > executable). But this stuff is not allowing for sane dependency management, > neither code reuse. I actually don't like the idea of specifying dependencies > in the code via sys.path. I think the resolved sys.path based on > requirements.txt is much better solution. > > So i'm looking for a solution that would allow to use the same artifact for > everything (like jar) so it can guarantee that the same subset of code that > was tested, goes to production and used in dev. Currently I'm leaning towards > using pip's capability to work with flat folders via --find-links, so i can > deploy wheels to the pythonlib and then reuse them in the development > environment. > > But in this setup how do i make my program executable from pythonlib > location? I think I should I create some smart runner script that would be > able to use the pip's dependency resolution, create the necessary sys.path > basing on the wheel requirements.txt and then my program wheel should have an > entry point like __main__.py > > As Nick pointed out the wheel is a superset of the egg - so I assume wheels > can be executable, correct? How do i achieve that? Wheel is a package format. Packages are for transmitting and installing bits. If you want to make some kind of self-unpacking executable please do it with something built for it. makeself is an excellent choice for these. --Noah signature.asc Description: Message signed with OpenPGP using GPGMail ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] PEP 427
I'm sorry for possible dup, but for whatever reason i don't see this email reaching the list, so i'm resending. On Wed, Jan 29, 2014 at 12:50 PM, Evgeny Sazhin wrote: > > > > > On Wed, Jan 29, 2014 at 9:11 AM, Vinay Sajip wrote: >> >> > Does it mean that it actually makes sense to look into that >> > direction and make wheel usage closer to jar? >> >> There is a parallel discussion going on, with the title "Using Wheel with >> zipimport", >> which is relevant to this question, and other questions you raised (e.g. >> about >> supporting C extensions/pure-Python modules. I read all of it and got a bit lost in between the distil API and PEP process discussion;) > > >> >> > I have no knowledge about c extensions scope, but i feel >> > like it might be of less importance then pure python >> > packaging issues? Am I wrong? >> >> A lot of Python users depend on C extensions - and while it is a subset of >> all Python >> users, it is a large (and important) subset. Example: any usage of Python in >> numerical analysis or scientific applications involves use of C extensions. >> >> Regards, >> >> Vinay Sajip > > > I can see that it might be quite beneficial to have virtualenv and pip installing wheels locally for development needs, so here is what i was able to come up with so far: I have one folder on NFS where all python developed stuff should be *deployed* - pyhtonlib. It is impossible to use pip or virtualenv there - so i'm bound to artifacts. The only way something can appear there is by using the "release" program that knows how to put artifacts in specified locations. Currently most of the stuff there is the .py modules and few eggs (some are executable). But this stuff is not allowing for sane dependency management, neither code reuse. I actually don't like the idea of specifying dependencies in the code via sys.path. I think the resolved sys.path based on requirements.txt is much better solution. So i'm looking for a solution that would allow to use the same artifact for everything (like jar) so it can guarantee that the same subset of code that was tested, goes to production and used in dev. Currently I'm leaning towards using pip's capability to work with flat folders via --find-links, so i can deploy wheels to the pythonlib and then reuse them in the development environment. But in this setup how do i make my program executable from pythonlib location? I think I should I create some smart runner script that would be able to use the pip's dependency resolution, create the necessary sys.path basing on the wheel requirements.txt and then my program wheel should have an entry point like __main__.py As Nick pointed out the wheel is a superset of the egg - so I assume wheels can be executable, correct? How do i achieve that? Thanks a lot! Eugene ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] PEP 427
On Wed, Jan 29, 2014 at 9:11 AM, Vinay Sajip wrote: > > Does it mean that it actually makes sense to look into that > > direction and make wheel usage closer to jar? > > There is a parallel discussion going on, with the title "Using Wheel with > zipimport", > which is relevant to this question, and other questions you raised (e.g. > about > supporting C extensions/pure-Python modules. > > I read all of it and got a bit lost in between the distil API and PEP process discussion;) > > I have no knowledge about c extensions scope, but i feel > > like it might be of less importance then pure python > > packaging issues? Am I wrong? > > A lot of Python users depend on C extensions - and while it is a subset of > all Python > users, it is a large (and important) subset. Example: any usage of Python > in > numerical analysis or scientific applications involves use of C extensions. > > Regards, > > Vinay Sajip > I can see that it might be quite beneficial to have virtualenv and pip installing wheels locally for development needs, so here is what i was able to come up with so far: I have one folder on NFS where all python developed stuff should be *deployed* - pyhtonlib. It is impossible to use pip or virtualenv there - so i'm bound to artifacts. The only way something can appear there is by using the "release" program that knows how to put artifacts in specified locations. Currently most of the stuff there is the .py modules and few eggs (some are executable). But this stuff is not allowing for sane dependency management, neither code reuse. I actually don't like the idea of specifying dependencies in the code via sys.path. I think the resolved sys.path based on requirements.txt is much better solution. So i'm looking for a solution that would allow to use the same artifact for everything (like jar) so it can guarantee that the same subset of code that was tested, goes to production and used in dev. Currently I'm leaning towards using pip's capability to work with flat folders via --find-links, so i can deploy wheels to the pythonlib and then reuse them in the development environment. But in this setup how do i make my program executable from pythonlib location? I think I should I create some smart runner script that would be able to use the pip's dependency resolution, create the necessary sys.path basing on the wheel requirements.txt and then my program wheel should have an entry point like __main__.py As Nick pointed out the wheel is a superset of the egg - so I assume wheels can be executable, correct? How do i achieve that? Thanks a lot! Eugene ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] PEP 427
Guys, I'm still unable to see my emails reaching the list for unknown reason... Any ideas what can be the problem? FWIW I have tested it by adding the same __main__.py i used for the egg variant of the distribution to the wheel root and specifying $ PYTHONPATH=projectA.whl; python executable_projectB.whl I was able to run the program! So it seems like the stuff does behave like a jar for pure python modules! Don't know who to thank for that - because nobody seems to have claimed this functionality;) Well Nick said it supposed to be that way, So please don't break it!!;) It also means that I have the full picture in place without the necessity to use the buildout or anything else other then the pip and wheel from a flat folder structure! I'd appreciate if you could let me know about any problems you could foresee (we are talking pure python only) Thanks! Eugene On Wed, Jan 29, 2014 at 5:36 PM, Evgeny Sazhin wrote: > I'm sorry for possible dup, but for whatever reason i don't see this > email reaching the list, so i'm resending. > > > On Wed, Jan 29, 2014 at 12:50 PM, Evgeny Sazhin wrote: >> >> >> >> >> On Wed, Jan 29, 2014 at 9:11 AM, Vinay Sajip wrote: >>> >>> > Does it mean that it actually makes sense to look into that >>> > direction and make wheel usage closer to jar? >>> >>> There is a parallel discussion going on, with the title "Using Wheel with >>> zipimport", >>> which is relevant to this question, and other questions you raised (e.g. >>> about >>> supporting C extensions/pure-Python modules. > > > > I read all of it and got a bit lost in between the distil API and PEP > process discussion;) > >> >> >>> >>> > I have no knowledge about c extensions scope, but i feel >>> > like it might be of less importance then pure python >>> > packaging issues? Am I wrong? >>> >>> A lot of Python users depend on C extensions - and while it is a subset of >>> all Python >>> users, it is a large (and important) subset. Example: any usage of Python in >>> numerical analysis or scientific applications involves use of C extensions. >>> >>> Regards, >>> >>> Vinay Sajip >> >> >> > > I can see that it might be quite beneficial to have virtualenv and pip > installing wheels locally for development needs, so here is what i was > able to come up with so far: > > I have one folder on NFS where all python developed stuff should be > *deployed* - pyhtonlib. It is impossible to use pip or virtualenv > there - so i'm bound to artifacts. The only way something can appear > there is by using the "release" program that knows how to put > artifacts in specified locations. Currently most of the stuff there is > the .py modules and few eggs (some are executable). But this stuff is > not allowing for sane dependency management, neither code reuse. I > actually don't like the idea of specifying dependencies in the code > via sys.path. I think the resolved sys.path based on requirements.txt > is much better solution. > > So i'm looking for a solution that would allow to use the same > artifact for everything (like jar) so it can guarantee that the same > subset of code that was tested, goes to production and used in dev. > Currently I'm leaning towards using pip's capability to work with flat > folders via --find-links, so i can deploy wheels to the pythonlib and > then reuse them in the development environment. > > But in this setup how do i make my program executable from pythonlib > location? I think I should I create some smart runner script that > would be able to use the pip's dependency resolution, create the > necessary sys.path basing on the wheel requirements.txt and then my > program wheel should have an entry point like __main__.py > > As Nick pointed out the wheel is a superset of the egg - so I assume > wheels can be executable, correct? How do i achieve that? ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] PEP 427
> Does it mean that it actually makes sense to look into that > direction and make wheel usage closer to jar? There is a parallel discussion going on, with the title "Using Wheel with zipimport", which is relevant to this question, and other questions you raised (e.g. about supporting C extensions/pure-Python modules. > I have no knowledge about c extensions scope, but i feel > like it might be of less importance then pure python > packaging issues? Am I wrong? A lot of Python users depend on C extensions - and while it is a subset of all Python users, it is a large (and important) subset. Example: any usage of Python in numerical analysis or scientific applications involves use of C extensions. Regards, Vinay Sajip ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] PEP 427
On 29 January 2014 13:41, Evgeny Sazhin wrote: > > I have no knowledge about c extensions scope, but i feel like it might be of > less importance then pure python packaging issues? Am I wrong? Most interesting Python projects will end up with C dependencies through things like NumPy, SQL Alchemy, GUI toolkits, image libraries, etc. You can do a lot with pure Python and the standard library, but interfacing with C is the norm rather than the exception the way it is in Java. > I'd think wheel should do everything in its power to provide pain free > workflow for pure python, and if there is a need to add some C extension then > solve that problem separately? Is the possibility to add C extension into the > wheel so critical, that it should prohibit useful feature for pure python > modules? The Python culture and the Java culture is very different in that regard - the JVM allows Java developers to almost completely ignore the C runtime on a system. CPython, by contrast, integrates with that C runtime directly, and this is then reflected in the ecosystem that has built up around it. So while there is some support for running without unpacking things to disk, we generally prefer approaches that work seamlessly with C extensions as well. Cheers, Nick. -- Nick Coghlan | ncogh...@gmail.com | Brisbane, Australia ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] PEP 427
On Tue, Jan 28, 2014 at 10:41 PM, Evgeny Sazhin wrote: > > On Jan 28, 2014, at 9:54 PM, Daniel Holth wrote: > >> On Tue, Jan 28, 2014 at 8:04 PM, Evgeny Sazhin wrote: >>> Hi, >>> >>> I saw that people from this list are responsible for Wheel related PEP. >>> I'm comparatively new to the python packaging and need some help >>> understanding the recommended way of dealing with python packages. >>> >>> I'm trying to create a development infrastructure that would allow for >>> simple and unified ways of sharing, *deploying* and *reusing* the code >>> within private entity. I can see that pip with virtual environments and >>> requirements.txt is very similar to dependency management provided by maven >>> or apache ivy. But there seems to be a disconnect between the egg carrying >>> the possibility to be importable and executable, but in the same time >>> considered to be deprecated format which is not fully supported by pip and >>> virtualenv and wheel not having those basic questions answered... >>> >>> So, i would like to ask few questions about it : >>> >>> 1. I'm coming from java world, so it is bit hard to understand why new >>> packaging format is not importable? What is the reason behind it? Wouldn't >>> it be way easier to deal with the dependencies and imports? >>> 2. Why unzipping and installing the wheel is the way to go? There is no >>> need to unzip java jar to import it into your code, neither to run your >>> code with it? Why wheel can't be used the same exact way? >>> >>> >>> I would appreciate any insight about the development infrastructure the >>> wheel designers have in mind. >>> >>> Thanks in advance, >>> Eugene >> > > >> The main reason the packaging format is not explicitly importable is >> simply because Python hasn't had "jar-like" deployment for as long or >> as consistently as Java. > > Thanks a lot for taking time to respond! > > Does it mean that it actually makes sense to look into that direction and > make wheel usage closer to jar? > If wheel is promoted as the ultimate solution to the packaging problems (at > least that's my impression ) aren't the abilities to import packages and > classes from it as well as make it runnable a kind of top priority and > something that is right on the surface? > >> So while Java code universally uses a "get >> resource" API to get stuff on the classpath, a a lot of Python code >> will try to open a file. C extensions can't be loaded from inside zip >> files. > > I have no knowledge about c extensions scope, but i feel like it might be of > less importance then pure python packaging issues? Am I wrong? > I'd think wheel should do everything in its power to provide pain free > workflow for pure python, and if there is a need to add some C extension then > solve that problem separately? Is the possibility to add C extension into the > wheel so critical, that it should prohibit useful feature for pure python > modules? > >> And it's a lot harder to edit .py files once they are zipped >> up, unlike Java where only the compiled and non-editable .class files >> are zipped. > > Why would i want to edit the zipped .py file from a wheel? I want to import a > package or a class from it, reuse a function. Nothing more. > > >> >> ZIP import is a great feature but wheels aren't really designed for >> it. It's more reliable that way. > > More reliable in what regard? > > My impression is that the whole thing about pip, virtualenv and wheels is > great - it takes into consideration a lot of end user problems, aka getting > proper module and installing it to RUN applications. But for whatever reason > it misses some important problems on the DEVELOPMENT side. > > I'm trying to find a sane way for developer to work, so if wheel is not an > answer - what is? Deprecated Egg? Or may be then egg is not going to go > anywhere and pip has to learn to deal with them as well as it does with > wheels? It's true that we have not put a lot of effort into trying to support jar-style development in this system. If you want that then you will have to do additional work. However a lot of people do find developing on the filesystem in virtualenvs to be quite sane. All the source code for the whole application remains easily editable for debugging. (By the way, these days the debugger does know how to load .py source from within zip files.) You should also give buildout a try. It installs each dependency into its own directory as an unzipped egg and does not use virtualenv. Egg will be around for a while longer, it's just not written down in a PEP. Wheel solves a coupling problem between the build system and the installer, allowing those two steps to be performed in separate processes or even on separate machines. It does not solve all problems. ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] PEP 427
On Jan 28, 2014, at 9:54 PM, Daniel Holth wrote: > On Tue, Jan 28, 2014 at 8:04 PM, Evgeny Sazhin wrote: >> Hi, >> >> I saw that people from this list are responsible for Wheel related PEP. >> I'm comparatively new to the python packaging and need some help >> understanding the recommended way of dealing with python packages. >> >> I'm trying to create a development infrastructure that would allow for >> simple and unified ways of sharing, *deploying* and *reusing* the code >> within private entity. I can see that pip with virtual environments and >> requirements.txt is very similar to dependency management provided by maven >> or apache ivy. But there seems to be a disconnect between the egg carrying >> the possibility to be importable and executable, but in the same time >> considered to be deprecated format which is not fully supported by pip and >> virtualenv and wheel not having those basic questions answered... >> >> So, i would like to ask few questions about it : >> >> 1. I'm coming from java world, so it is bit hard to understand why new >> packaging format is not importable? What is the reason behind it? Wouldn't >> it be way easier to deal with the dependencies and imports? >> 2. Why unzipping and installing the wheel is the way to go? There is no need >> to unzip java jar to import it into your code, neither to run your code with >> it? Why wheel can't be used the same exact way? >> >> >> I would appreciate any insight about the development infrastructure the >> wheel designers have in mind. >> >> Thanks in advance, >> Eugene > > The main reason the packaging format is not explicitly importable is > simply because Python hasn't had "jar-like" deployment for as long or > as consistently as Java. Thanks a lot for taking time to respond! Does it mean that it actually makes sense to look into that direction and make wheel usage closer to jar? If wheel is promoted as the ultimate solution to the packaging problems (at least that’s my impression ) aren’t the abilities to import packages and classes from it as well as make it runnable a kind of top priority and something that is right on the surface? > So while Java code universally uses a "get > resource" API to get stuff on the classpath, a a lot of Python code > will try to open a file. C extensions can't be loaded from inside zip > files. I have no knowledge about c extensions scope, but i feel like it might be of less importance then pure python packaging issues? Am I wrong? I’d think wheel should do everything in its power to provide pain free workflow for pure python, and if there is a need to add some C extension then solve that problem separately? Is the possibility to add C extension into the wheel so critical, that it should prohibit useful feature for pure python modules? > And it's a lot harder to edit .py files once they are zipped > up, unlike Java where only the compiled and non-editable .class files > are zipped. Why would i want to edit the zipped .py file from a wheel? I want to import a package or a class from it, reuse a function. Nothing more. > > ZIP import is a great feature but wheels aren't really designed for > it. It's more reliable that way. More reliable in what regard? My impression is that the whole thing about pip, virtualenv and wheels is great - it takes into consideration a lot of end user problems, aka getting proper module and installing it to RUN applications. But for whatever reason it misses some important problems on the DEVELOPMENT side. I’m trying to find a sane way for developer to work, so if wheel is not an answer - what is? Deprecated Egg? Or may be then egg is not going to go anywhere and pip has to learn to deal with them as well as it does with wheels? Thanks, Eugene ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] PEP 427
On Tue, Jan 28, 2014 at 8:04 PM, Evgeny Sazhin wrote: > Hi, > > I saw that people from this list are responsible for Wheel related PEP. > I'm comparatively new to the python packaging and need some help > understanding the recommended way of dealing with python packages. > > I'm trying to create a development infrastructure that would allow for simple > and unified ways of sharing, *deploying* and *reusing* the code within > private entity. I can see that pip with virtual environments and > requirements.txt is very similar to dependency management provided by maven > or apache ivy. But there seems to be a disconnect between the egg carrying > the possibility to be importable and executable, but in the same time > considered to be deprecated format which is not fully supported by pip and > virtualenv and wheel not having those basic questions answered... > > So, i would like to ask few questions about it : > > 1. I'm coming from java world, so it is bit hard to understand why new > packaging format is not importable? What is the reason behind it? Wouldn't it > be way easier to deal with the dependencies and imports? > 2. Why unzipping and installing the wheel is the way to go? There is no need > to unzip java jar to import it into your code, neither to run your code with > it? Why wheel can't be used the same exact way? > > > I would appreciate any insight about the development infrastructure the wheel > designers have in mind. > > Thanks in advance, > Eugene The main reason the packaging format is not explicitly importable is simply because Python hasn't had "jar-like" deployment for as long or as consistently as Java. So while Java code universally uses a "get resource" API to get stuff on the classpath, a a lot of Python code will try to open a file. C extensions can't be loaded from inside zip files. And it's a lot harder to edit .py files once they are zipped up, unlike Java where only the compiled and non-editable .class files are zipped. ZIP import is a great feature but wheels aren't really designed for it. It's more reliable that way. ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
[Distutils] PEP 427
Hi, I saw that people from this list are responsible for Wheel related PEP. I’m comparatively new to the python packaging and need some help understanding the recommended way of dealing with python packages. I’m trying to create a development infrastructure that would allow for simple and unified ways of sharing, *deploying* and *reusing* the code within private entity. I can see that pip with virtual environments and requirements.txt is very similar to dependency management provided by maven or apache ivy. But there seems to be a disconnect between the egg carrying the possibility to be importable and executable, but in the same time considered to be deprecated format which is not fully supported by pip and virtualenv and wheel not having those basic questions answered... So, i would like to ask few questions about it : 1. I’m coming from java world, so it is bit hard to understand why new packaging format is not importable? What is the reason behind it? Wouldn’t it be way easier to deal with the dependencies and imports? 2. Why unzipping and installing the wheel is the way to go? There is no need to unzip java jar to import it into your code, neither to run your code with it? Why wheel can’t be used the same exact way? I would appreciate any insight about the development infrastructure the wheel designers have in mind. Thanks in advance, Eugene ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] PEP 427 (wheel) clarifications
On 11 Mar 2013 04:38, "Daniel Holth" wrote:
>
> A few more clarifications to PEP 427 "wheel".
>
> +The ``b'#!pythonw'`` convention is allowed. ``b'#!pythonw'``
indicates
> +a GUI script instead of a console script.
> +
>
> ...
>
> { "hash": "sha256=ADD-r2urObZHcxBW3Cr-vDCu5RJwT4CaRTHiFmbcIYY" }
>
> +(The hash value is the same format used in RECORD.)
> +
>
> +What's the deal with "purelib" vs. "platlib"?
> +Wheel preserves the historic "purelib" vs. "platlib" distinction
> +even though both map to the same install location in any system the
> +author could find.
> +
> +For example, a wheel with "Root-Is-Purelib: false" with all its files
> +in ``{name}-{version}.data/purelib`` is equivalent to a wheel with
> +"Root-Is-Purelib: true" with those same files in the root, and it
> +is legal to have files in both the "purelib" and "platlib"
categories.
> +
> +In practice a wheel should have only one of "purelib" or "platlib"
> +depending on whether it is pure Python or not and those files should
> +be at the root.
This isn't entirely accurate - purelib vs platlib is there mainly for
distro maintainers converting to distro specific layouts, and anyone that
moves purelib on to a common network share.
Agreed that wheels are easier to deal with when they only use one or the
other, though.
Cheers,
Nick.
> ___
> Distutils-SIG maillist - Distutils-SIG@python.org
> http://mail.python.org/mailman/listinfo/distutils-sig
___
Distutils-SIG maillist - Distutils-SIG@python.org
http://mail.python.org/mailman/listinfo/distutils-sig
[Distutils] PEP 427 (wheel) clarifications
A few more clarifications to PEP 427 "wheel".
+The ``b'#!pythonw'`` convention is allowed. ``b'#!pythonw'`` indicates
+a GUI script instead of a console script.
+
...
{ "hash": "sha256=ADD-r2urObZHcxBW3Cr-vDCu5RJwT4CaRTHiFmbcIYY" }
+(The hash value is the same format used in RECORD.)
+
+What's the deal with "purelib" vs. "platlib"?
+Wheel preserves the historic "purelib" vs. "platlib" distinction
+even though both map to the same install location in any system the
+author could find.
+
+For example, a wheel with "Root-Is-Purelib: false" with all its files
+in ``{name}-{version}.data/purelib`` is equivalent to a wheel with
+"Root-Is-Purelib: true" with those same files in the root, and it
+is legal to have files in both the "purelib" and "platlib" categories.
+
+In practice a wheel should have only one of "purelib" or "platlib"
+depending on whether it is pure Python or not and those files should
+be at the root.
___
Distutils-SIG maillist - Distutils-SIG@python.org
http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] PEP 427 (wheel format) accepted
On Sat, Feb 16, 2013 at 9:47 PM, Nick Coghlan wrote: > Daniel submitted PEPs 425 and 427 to python-dev for acceptance, and I > have formally accepted 427 as BDFL delegate > (http://www.python.org/dev/peps/pep-0427/). > > There are still a couple of tweaks to be made to the explanatory > sections of PEP 425 before I accept it, but those pending adjustments > will have no effect on the wheel format itself. PEP 425 is now accepted as well. Cheers, Nick. -- Nick Coghlan | ncogh...@gmail.com | Brisbane, Australia ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] PEP 427 (wheel format) accepted
On Sat, Feb 16, 2013 at 3:47 AM, Nick Coghlan wrote: > Daniel submitted PEPs 425 and 427 to python-dev for acceptance, and I > have formally accepted 427 as BDFL delegate > (http://www.python.org/dev/peps/pep-0427/). This is awesome! Congratulations to Daniel -- Philippe Ombredanne ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] PEP 427 (wheel format) accepted
On 16 February 2013 13:58, Sebastien Douche wrote: > On Sat, Feb 16, 2013 at 12:47 PM, Nick Coghlan wrote: >> Daniel submitted PEPs 425 and 427 to python-dev for acceptance, and I >> have formally accepted 427 as BDFL delegate >> (http://www.python.org/dev/peps/pep-0427/). > > It's a red letter day. It has taken 10 years to have a standardized > binary format. > > Great work Daniel. +1 Paul ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] PEP 427 (wheel format) accepted
On Sat, Feb 16, 2013 at 12:47 PM, Nick Coghlan wrote: > Daniel submitted PEPs 425 and 427 to python-dev for acceptance, and I > have formally accepted 427 as BDFL delegate > (http://www.python.org/dev/peps/pep-0427/). It's a red letter day. It has taken 10 years to have a standardized binary format. Great work Daniel. -- Sebastien Douche Twitter: @sdouche / G+: +sdouche ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
[Distutils] PEP 427 (wheel format) accepted
Daniel submitted PEPs 425 and 427 to python-dev for acceptance, and I have formally accepted 427 as BDFL delegate (http://www.python.org/dev/peps/pep-0427/). There are still a couple of tweaks to be made to the explanatory sections of PEP 425 before I accept it, but those pending adjustments will have no effect on the wheel format itself. Regards, Nick. -- Nick Coghlan | ncogh...@gmail.com | Brisbane, Australia ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
[Distutils] PEP-427: Wheel
Here's the wheel pep itself again. I think the prose is precise enough but
it could be more clear. My goal is to get these PEPs accepted so the pip
maintainers will be willing to merge the waiting "pip can install wheel"
branch. It's not perfect, it's extensible.
PEP: 427
Title: The Wheel Binary Package Format 0.1
Version: $Revision$
Last-Modified: $Date$
Author: Daniel Holth
BDFL-Delegate: Nick Coghlan
Discussions-To:
Status: Draft
Type: Standards Track
Content-Type: text/x-rst
Created: 20-Sep-2012
Post-History:
Abstract
This PEP describes a built-package format for Python called "wheel".
A wheel is a ZIP-format archive with a specially formatted file name
and the ``.whl`` extension. It contains a single distribution nearly
as it would be installed according to PEP 376 with a particular
installation scheme. A wheel file may be installed by simply
unpacking into site-packages with the standard 'unzip' tool, while
preserving enough information to spread its contents out onto their
final paths at any later time.
Note
This draft PEP describes version 0.1 of the "wheel" format. When the PEP
is accepted, the version will be changed to 1.0. (The major version
is used to indicate potentially backwards-incompatible changes to the
format.)
Rationale
=
Python needs a package format that is easier to install than sdist.
Python's sdist packages are defined by and require the distutils and
setuptools build systems, running arbitrary code to build-and-install,
and re-compile, code just so it can be installed into a new
virtualenv. This system of conflating build-install is slow, hard to
maintain, and hinders innovation in both build systems and installers.
Wheel attempts to remedy these problems by providing a simpler
interface between the build system and the installer. The wheel
binary package format frees installers from having to know about the
build system, saves time by amortizing compile time over many
installations, and removes the need to install a build system in the
target environment.
Details
===
Installing a wheel 'distribution-1.0.py32.none.any.whl'
---
Wheel installation notionally consists of two phases:
- Unpack.
a. Parse ``distribution-1.0.dist-info/WHEEL``.
b. Check that installer is compatible with Wheel-Version. Warn if
minor version is greater, abort if major version is greater.
c. If Root-Is-Purelib == 'true', unpack archive into purelib
(site-packages).
d. Else unpack archive into platlib (site-packages).
- Spread.
a. Unpacked archive includes ``distribution-1.0.dist-info/`` and (if
there is data) ``distribution-1.0.data/``.
b. Move each subtree of ``distribution-1.0.data/`` onto its
destination path. Each subdirectory of ``distribution-1.0.data/``
is a key into a dict of destination directories, such as
``distribution-1.0.data/(purelib|platlib|headers|scripts|data)``.
The initially supported paths are taken from
``distutils.command.install``.
c. If applicable, update scripts starting with ``#!python`` to point
to the correct interpreter.
d. Update ``distribution-1.0.dist.info/RECORD`` with the installed
paths.
e. Remove empty ``distribution-1.0.data`` directory.
f. Compile any installed .py to .pyc. (Uninstallers should be smart
enough to remove .pyc even if it is not mentioned in RECORD.)
Recommended installer features
''
Rewrite ``#!python``.
In wheel, scripts are packaged in
``{distribution}-{version}.data/scripts/``. If the first line of
a file in ``scripts/`` starts with exactly ``b'#!python'``, rewrite to
point to the correct interpreter. Unix installers may need to add
the +x bit to these files if the archive was created on Windows.
Generate script wrappers.
In wheel, scripts packaged on Unix systems will certainly not have
accompanying .exe wrappers. Windows installers may want to add them
during install.
File Format
---
File name convention
The wheel filename is ``{distribution}-{version}(-{build
tag})?-{python tag}-{abi tag}-{platform tag}.whl``.
distribution
Distribution name, e.g. 'django', 'pyramid'.
version
PEP-386 compliant version, e.g. 1.0.
build tag
Optional build number. Must start with a digit. A tie breaker if
two wheels have the same version. Sort as None if unspecified,
else sort the initial digits as a number, and the remainder
lexicographically.
language implementation and version tag
E.g. 'py27', 'py2', 'py3'.
abi tag
E.g. 'cp33m', 'abi3', 'none'.
platform tag
E.g. 'linux_x86_64', 'any'.
For example, ``distribution-1.0-1-py27-none-any.whl`` is the first
build of a package called 'distribution', and is compatible with
Python 2.7 (any Python 2.7 implementation), with no ABI (pure Python),
on any CPU architecture.
The last three components of the fil
