Re: [Distutils] [Python-Dev] distlib updated with resources API
David Cournapeau gmail.com> writes: > I noticed that you put the classifiers list as a string (same for > platform). I think it is expected to be a list, no ? That's an oversight; there are doubtless others, too. > Maybe slightly more controversial, I think the manifest should be > "evaluated". The current system of inclusion + exclusion is too > baroque to my taste, and makes it near-impossible to make reproducible > builds. Would you care to give a little more detail about what you mean by "evaluate"? I've kept the manifest as it is for backward compatibility (i.e. so that my sanity checking of sdist follows the logic as is used by distutils/distribute). Regards, Vinay Sajip ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] [Python-Dev] distlib updated with resources API
On Mon, Oct 1, 2012 at 5:06 PM, Vinay Sajip wrote: > David Cournapeau gmail.com> writes: > >> The code that produces yaml files. The point is precisely that it >> would be easy for me to consume this and produce the internal package >> representation in bento, which would then allow to configure, build >> and install packages using the bento format. > > Well, it's not pretty, but here's a Gist of the method which produces YAML > metadata from what's passed to setup(): > > https://gist.github.com/3812561 Thanks. I don't think it is fair to expect pretty code there in any case. I noticed that you put the classifiers list as a string (same for platform). I think it is expected to be a list, no ? Maybe slightly more controversial, I think the manifest should be "evaluated". The current system of inclusion + exclusion is too baroque to my taste, and makes it near-impossible to make reproducible builds. David ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] pkg_resources API use according to Nullege search engine
On Oct 01, 2012, at 03:43 PM, Vinay Sajip wrote: >Barry Warsaw python.org> writes: >> Why not provide this by distlib? > >Only because it doesn't add much value, and (if you do all that pkg_resources >does) might mean that you have to maintain a cache. Well, it's just something that client code would have to do anyway, which leads to multiple wheel invention and bugs. >As a matter of interest, what are the APIs you're using which need filenames >but can't use bytes or streams? Two that come to mind. * Testing command line arguments that take a file name (e.g. test that --config works). * Generating file: urls to generalized downloaders for local resources. Cheers, -Barry ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] [Python-Dev] distlib updated with resources API
David Cournapeau gmail.com> writes: > The code that produces yaml files. The point is precisely that it > would be easy for me to consume this and produce the internal package > representation in bento, which would then allow to configure, build > and install packages using the bento format. Well, it's not pretty, but here's a Gist of the method which produces YAML metadata from what's passed to setup(): https://gist.github.com/3812561 > I am curious: do you attempt to parse the setup.py to get those > environment markers ? I personally gave up on this and just run the > setup.py to get whatever values are available in that precise Lord, no, life's too short :-) Regards, Vinay Sajip ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] pkg_resources API use according to Nullege search engine
Barry Warsaw python.org> writes: > >a third-party API which expects a filename) can handle that themselves, > >e.g. by saving the bytes to a temporary location and passing that to whatever > >needs a filename. > > Why not provide this by distlib? Only because it doesn't add much value, and (if you do all that pkg_resources does) might mean that you have to maintain a cache. As a matter of interest, what are the APIs you're using which need filenames but can't use bytes or streams? Regards, Vinay Sajip ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] pkg_resources API use according to Nullege search engine
> > Of course it's not really a mystery; a tremendous number of Apis only accept > a file name. That's why it's there. And only something well connected to > the importer system knows how and whether the extraction is needed. > Why is that the case? If the bytes are available, then whoever needs can write them to a temp file, pass that name, and clean up afterwards. Is managing a cache of arbitrary files extracted from zips is a mainstream use case which distlib needs to support? Regards, Vinay Sajip ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] [Python-Dev] distlib updated with resources API
Yes, and I've added its package.yaml to the original Gist at https://gist.github.com/3803556 Regards, Vinay Sajip > > From: Daniel Holth >To: Vinay Sajip >Sent: Monday, 1 October 2012, 15:27 >Subject: Re: [Distutils] [Python-Dev] distlib updated with resources API > >Have you read the IPython setup.py? >https://github.com/ipython/ipython/blob/master/setup.py > > > ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Differences in PEP386 and setuptools
On Thu, Sep 27, 2012 at 7:15 PM, Toshio Kuratomi wrote: > So to be more complete, the proposal is to do this:: > > 0.1 > < 0.1.post1 > < 1.0.dev1# The .dev that's changing position > < 1.0a1.dev1 > < 1.0a1 > < 1.0a1.post1 > < 1.0b1.dev1 > < 1.0b1 > < 1.0b1.post1 > < 1.0c1.dev1 > < 1.0c1 > < 1.0c1.post1 > # The current PEP386 position of 1.0.dev1 > < 1.0 > < 1.0.post1 > This looks perfect to me, and very unambiguous. Allowing the .dev and .post modifiers at every release stage I think clears up the issue here better than any other proposal. +1. (This is also semantically the same as semver--just not syntactically the same. I like semver but I think we're stuck with the existing syntax which is fine.) Erik ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] pkg_resources API use according to Nullege search engine
On Mon, Oct 1, 2012 at 10:46 AM, Barry Warsaw wrote: > On Sep 30, 2012, at 10:00 PM, Daniel Holth wrote: > >>resource_filename is very popular. I would have thought >>resource_stream would be more popular. Unless your package is zipped >>resource_filename is trivial to implement. > > I've used declare_namespace() once or twice. > > I use resource_filename(), resource_string(), resource_stream(), and > resource_listdir() quite a bit. declare_namespace() is special because we could probably rewrite it at install time (in Python 3.3, by deleting __init__.py) or by calling pkgutil. ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] pkg_resources API use according to Nullege search engine
On Oct 01, 2012, at 07:35 AM, Vinay Sajip wrote: >Yes, I find that odd, too. pkg_resources seems to extract files from zip into >a cache folder, then returns filenames from the location in the cache; it >seems a lot of trouble to go to just to be able to deliver a filename. Darn handy though when interacting with some APIs which require filenames. >In the Resource that I implemented in distlib, I have the "path" attribute of >a resource which is analogous, though it's not directly usable as a file path >for resources in a zip. However, since the resource is available as bytes or >a stream, those applications which really need a filename (perhaps to pass to >a third-party API which expects a filename) can handle that themselves, >e.g. by saving the bytes to a temporary location and passing that to whatever >needs a filename. Why not provide this by distlib? -Barry ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] pkg_resources API use according to Nullege search engine
On Sep 30, 2012, at 10:00 PM, Daniel Holth wrote: >resource_filename is very popular. I would have thought >resource_stream would be more popular. Unless your package is zipped >resource_filename is trivial to implement. I've used declare_namespace() once or twice. I use resource_filename(), resource_string(), resource_stream(), and resource_listdir() quite a bit. Cheers, -Barry ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] [Python-Dev] distlib updated with resources API
On Mon, Oct 1, 2012 at 3:15 PM, Vinay Sajip wrote: > David Cournapeau gmail.com> writes: > >> Note that in Cabal at least, those conditionals work not just for >> requirements, but for pretty much any section that is not metadata (so >> in the python world, you could condition on the package you want to >> install, etc...). > > Right, but the concept of environment markers is simple enough that we should > be able to extend to other areas. Requirements are just the most obvious > application. I think we just need to see how it looks like in real cases to get a good idea about tradeoffs syntax-wise. > >> That's great. May I ask you to put this code somewhere ? This would >> allow me to try using this inside bento. It would allow using whatever >> format you are ending up with directly with existing projects (bento >> is is already used by some complex packages such as numpy or scipy). >> It would be a good way to ensure the format provides enough semantics. > > Which code do you mean? Although I have written some code to produce metadata > in > YAML format, I have not yet got anything that consumes it to a reasonable > point > (apart from sdist generation, which should not be environment-specific). The code that produces yaml files. The point is precisely that it would be easy for me to consume this and produce the internal package representation in bento, which would then allow to configure, build and install packages using the bento format. > I will publish what I have as soon as it has reached a reasonable state of > usefulness. Precisely because it isn't declarative, existing > environment-specific > code in setup.py files in PyPI archives is not easy to convert to environment > markers automatically :-( I am curious: do you attempt to parse the setup.py to get those environment markers ? I personally gave up on this and just run the setup.py to get whatever values are available in that precise environment. Given that versions are not normalized, I am afraid trying to do better is bound to fail, but maybe I am not smart enough to do it :) David ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] [Python-Dev] distlib updated with resources API
David Cournapeau gmail.com> writes: > Note that in Cabal at least, those conditionals work not just for > requirements, but for pretty much any section that is not metadata (so > in the python world, you could condition on the package you want to > install, etc...). Right, but the concept of environment markers is simple enough that we should be able to extend to other areas. Requirements are just the most obvious application. > That's great. May I ask you to put this code somewhere ? This would > allow me to try using this inside bento. It would allow using whatever > format you are ending up with directly with existing projects (bento > is is already used by some complex packages such as numpy or scipy). > It would be a good way to ensure the format provides enough semantics. Which code do you mean? Although I have written some code to produce metadata in YAML format, I have not yet got anything that consumes it to a reasonable point (apart from sdist generation, which should not be environment-specific). I will publish what I have as soon as it has reached a reasonable state of usefulness. Precisely because it isn't declarative, existing environment-specific code in setup.py files in PyPI archives is not easy to convert to environment markers automatically :-( Regards, Vinay Sajip ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] [Python-Dev] distlib updated with resources API
On Mon, Oct 1, 2012 at 8:14 AM, Vinay Sajip wrote: > David Cournapeau gmail.com> writes: > >> I am not suggesting something very complicated (we don't want to >> re-create a language), but more something like cabal (see conditionals >> in > http://www.haskell.org/cabal/users-guide/developing-packages.html#package-descriptions), >> or even RPM (http://www.rpm.org/max-rpm/s1-rpm-inside-conditionals.html). >> This has the advantage of using something that has already been >> battle-tested for several years. > > Both the Cabal and RPM conditionals seem fairly narrow in scope (e.g. os/ > arch/impl/flag for Cabal), and therefore it seems that environment markers > should be able to do the job. Although of course environment markers aren't > battle tested, it seems worthwhile deploying them on the battlefield to see > how > they perform. Note that in Cabal at least, those conditionals work not just for requirements, but for pretty much any section that is not metadata (so in the python world, you could condition on the package you want to install, etc...). > >> But again, syntax is a really minor point, and what matters the most >> are the features. I was not able to express what I wanted with yaml >> for bento, but would be more than happy to scrap the custom format if >> someone manages to. > > If I can continue working on the YAML format that I've mentioned, I hope I can > report some progress in this area in due course. That's great. May I ask you to put this code somewhere ? This would allow me to try using this inside bento. It would allow using whatever format you are ending up with directly with existing projects (bento is is already used by some complex packages such as numpy or scipy). It would be a good way to ensure the format provides enough semantics. David ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] pkg_resources API use according to Nullege search engine
Daniel Holth gmail.com> writes: > resource_filename is very popular. I would have thought > resource_stream would be more popular. Unless your package is zipped > resource_filename is trivial to implement. Yes, I find that odd, too. pkg_resources seems to extract files from zip into a cache folder, then returns filenames from the location in the cache; it seems a lot of trouble to go to just to be able to deliver a filename. In the Resource that I implemented in distlib, I have the "path" attribute of a resource which is analogous, though it's not directly usable as a file path for resources in a zip. However, since the resource is available as bytes or a stream, those applications which really need a filename (perhaps to pass to a third-party API which expects a filename) can handle that themselves, e.g. by saving the bytes to a temporary location and passing that to whatever needs a filename. Regards, Vinay Sajip ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] [Python-Dev] distlib updated with resources API
David Cournapeau gmail.com> writes: > I am not suggesting something very complicated (we don't want to > re-create a language), but more something like cabal (see conditionals > in http://www.haskell.org/cabal/users-guide/developing-packages.html#package-descriptions), > or even RPM (http://www.rpm.org/max-rpm/s1-rpm-inside-conditionals.html). > This has the advantage of using something that has already been > battle-tested for several years. Both the Cabal and RPM conditionals seem fairly narrow in scope (e.g. os/ arch/impl/flag for Cabal), and therefore it seems that environment markers should be able to do the job. Although of course environment markers aren't battle tested, it seems worthwhile deploying them on the battlefield to see how they perform. > But again, syntax is a really minor point, and what matters the most > are the features. I was not able to express what I wanted with yaml > for bento, but would be more than happy to scrap the custom format if > someone manages to. If I can continue working on the YAML format that I've mentioned, I hope I can report some progress in this area in due course. Regards, Vinay Sajip ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
