Re: [Distutils] buildout & extra-path
On Tue, Nov 28, 2017 at 4:33 AM, Alessandro Dentella
wrote:
> Hi,
>
> I realized that extra-path is not always honored in buildout. I have a
> Django application that uses django-recipe with extra-path.
>
> I also want a pure ipython program that share the same sys.path. but
> if I write:
>
> [ipython]
> recipe = zc.recipe.egg
> eggs = ${buildout:eggs}
> extra-paths = ${buildout:directory}/apps
>
> I get a script that doesn't have the extra-path.
> I explicitly have to add:
>
> interpreter = ipython-dj
>
> is this the desired behaviour?
>
The extra-paths egg recipe option should work, regardless of whether the
interpreter option is used.
Please create an issue:
https://github.com/buildout/buildout/issues/new
and include a full configuration file that demonstrates the problem so we
can try to reproduce.
Jim
--
Jim Fulton
http://jimfulton.info
___
Distutils-SIG maillist - Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] PEP 517: Bootstrapping setuptools
Oh, gawd. From now on, someone will have to say buildout 3 times before I appear. On Tue, Aug 22, 2017 at 2:06 PM, xoviat wrote: > So are we going for a "primarily isolation" approach then where builds are > only isolated on the first level, but subsequent levels share the same > build environment? > I'm a huge fan of isolation, but isolation seems to me to be orthogonal to this discussion. I'm done with this thread. :) Jim > > 2017-08-22 10:23 GMT-05:00 Jim Fulton : > >> I didn't mention (nor do I recall anyone mentioning) venvs. >> >> Jim >> >> >> On Tue, Aug 22, 2017 at 11:15 AM, Matt Joyce >> wrote: >> >>> venvs within venvs... terrifying concept. >>> >>> On Tue, Aug 22, 2017 at 11:02 AM, Jim Fulton wrote: >>> >>>> >>>> >>>> On Tue, Aug 22, 2017 at 9:23 AM, Daniel Holth wrote: >>>> >>>>> Isn't this a special case of needing . on sys.path when building a >>>>> package? Then the same version of setuptools that is being packaged builds >>>>> itself. >>>>> >>>> No. The issue for us it wasn't setuptools itself, but it's dependencies >>>> and those dependencies conflicted with dependencies of of packages we were >>>> installing *and* those packages importing these dependences (indirectly) in >>>> their setup scripts. Setup scripts that import the thing they're about to >>>> install, generally to get the version :(, is something I'd love to see go >>>> away. >>>> >>>> Jim >>>> >>>> -- >>>> Jim Fulton >>>> http://jimfulton.info >>>> >>>> ___ >>>> Distutils-SIG maillist - Distutils-SIG@python.org >>>> https://mail.python.org/mailman/listinfo/distutils-sig >>>> >>>> >>> >> >> >> -- >> Jim Fulton >> http://jimfulton.info >> >> ___ >> Distutils-SIG maillist - Distutils-SIG@python.org >> https://mail.python.org/mailman/listinfo/distutils-sig >> >> > -- Jim Fulton http://jimfulton.info ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] PEP 517: Bootstrapping setuptools
I didn't mention (nor do I recall anyone mentioning) venvs. Jim On Tue, Aug 22, 2017 at 11:15 AM, Matt Joyce wrote: > venvs within venvs... terrifying concept. > > On Tue, Aug 22, 2017 at 11:02 AM, Jim Fulton wrote: > >> >> >> On Tue, Aug 22, 2017 at 9:23 AM, Daniel Holth wrote: >> >>> Isn't this a special case of needing . on sys.path when building a >>> package? Then the same version of setuptools that is being packaged builds >>> itself. >>> >> No. The issue for us it wasn't setuptools itself, but it's dependencies >> and those dependencies conflicted with dependencies of of packages we were >> installing *and* those packages importing these dependences (indirectly) in >> their setup scripts. Setup scripts that import the thing they're about to >> install, generally to get the version :(, is something I'd love to see go >> away. >> >> Jim >> >> -- >> Jim Fulton >> http://jimfulton.info >> >> ___ >> Distutils-SIG maillist - Distutils-SIG@python.org >> https://mail.python.org/mailman/listinfo/distutils-sig >> >> > -- Jim Fulton http://jimfulton.info ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] PEP 517: Bootstrapping setuptools
On Tue, Aug 22, 2017 at 9:23 AM, Daniel Holth wrote: > Isn't this a special case of needing . on sys.path when building a > package? Then the same version of setuptools that is being packaged builds > itself. > No. The issue for us it wasn't setuptools itself, but it's dependencies and those dependencies conflicted with dependencies of of packages we were installing *and* those packages importing these dependences (indirectly) in their setup scripts. Setup scripts that import the thing they're about to install, generally to get the version :(, is something I'd love to see go away. Jim -- Jim Fulton http://jimfulton.info ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] PEP 517: Bootstrapping setuptools
On Mon, Aug 21, 2017 at 5:39 PM, xoviat wrote: > This statement comes from something that Donald said: > > > The unvendoring means that setuptools and the project code are now > competing over who gets to define what an acceptable version is for these > libraries to be installed with. > > As if this isn't going to be true for any other build system, and > setuptools won't at all be in any way special in a PEP 517 world. So you > are always going to have complaints about "well build system X required a > newer version but my project requires an older version" when the real > problem is that people are requiring an older version/ projects do not have > a stable API. > This boils down to implementation details. pip dealt with this issue and buildout stumbled because of differences in the way they managed their paths. Buildout could have coped (eventually). Jim > 2017-08-21 16:27 GMT-05:00 Jim Fulton : > >> >> >> On Mon, Aug 21, 2017 at 5:17 PM, xoviat wrote: >> >>> Of course, to be frank, the principle failure with this plan is that >>> third-party tools (buildout, os packagers) will not be compliant with PEP >>> 517 even after it is adopted, and will then complain about having to update >>> their build systems. >>> >> >> I don't understand this statement. If buildout needs to be compliant, it >> will be (eventually). I can imagine a future where everything is >> distributed with wheels and buildout wouldn't need to build anything. >> >> Jim >> >> >>> >>> 2017-08-21 16:05 GMT-05:00 xoviat : >>> >>>> Previously, the attempt to move setuptools off of vendored dependencies >>>> failed because it was not done correctly: install_requires was set to the >>>> vendored dependencies but not setup_requires, which would have been >>>> required to correctly specify dependencies. However, setup_requires would >>>> have introduced circular dependencies that are impossible to correctly >>>> resolve, so that would have simply shifted an impossible problem onto pip. >>>> >>>> The principle issue then, is that setuptools requires setuptools to >>>> build itself. And although, this problem was previously not worth solving >>>> because of 'setup.py install', it is now not a difficult problem to solve >>>> if we rethink slightly what should be included in the setuptools >>>> respository: >>>> - Rather than re-generating egg_info on each setup.py invocation, we >>>> can cache dist-info in the setuptools respository. >>>> - We can implement a simple entry_points script that updates dist-info >>>> based on the contents of setup.py, which is only runnable of course after >>>> setuptools is installed >>>> - We can implement the reference build backend for setuptools: simply >>>> copy the files and dist-info into a new compliant zip archive >>>> >>>> Viola! Setuptools is no longer required to build setuptools, and the >>>> installation process is fully compliant with Python PEPs. >>>> >>> >>> >>> ___ >>> Distutils-SIG maillist - Distutils-SIG@python.org >>> https://mail.python.org/mailman/listinfo/distutils-sig >>> >>> >> >> >> -- >> Jim Fulton >> http://jimfulton.info >> > > -- Jim Fulton http://jimfulton.info ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] PEP 517: Bootstrapping setuptools
On Mon, Aug 21, 2017 at 5:17 PM, xoviat wrote: > Of course, to be frank, the principle failure with this plan is that > third-party tools (buildout, os packagers) will not be compliant with PEP > 517 even after it is adopted, and will then complain about having to update > their build systems. > I don't understand this statement. If buildout needs to be compliant, it will be (eventually). I can imagine a future where everything is distributed with wheels and buildout wouldn't need to build anything. Jim > > 2017-08-21 16:05 GMT-05:00 xoviat : > >> Previously, the attempt to move setuptools off of vendored dependencies >> failed because it was not done correctly: install_requires was set to the >> vendored dependencies but not setup_requires, which would have been >> required to correctly specify dependencies. However, setup_requires would >> have introduced circular dependencies that are impossible to correctly >> resolve, so that would have simply shifted an impossible problem onto pip. >> >> The principle issue then, is that setuptools requires setuptools to build >> itself. And although, this problem was previously not worth solving because >> of 'setup.py install', it is now not a difficult problem to solve if we >> rethink slightly what should be included in the setuptools respository: >> - Rather than re-generating egg_info on each setup.py invocation, we can >> cache dist-info in the setuptools respository. >> - We can implement a simple entry_points script that updates dist-info >> based on the contents of setup.py, which is only runnable of course after >> setuptools is installed >> - We can implement the reference build backend for setuptools: simply >> copy the files and dist-info into a new compliant zip archive >> >> Viola! Setuptools is no longer required to build setuptools, and the >> installation process is fully compliant with Python PEPs. >> > > > ___ > Distutils-SIG maillist - Distutils-SIG@python.org > https://mail.python.org/mailman/listinfo/distutils-sig > > -- Jim Fulton http://jimfulton.info ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Pain installing pinned versions of setuptools requirements.
On Sun, Apr 2, 2017 at 1:29 PM, Donald Stufft wrote: > > On Apr 2, 2017, at 1:24 PM, Jim Fulton wrote: > > > > Can you post this on https://github.com/pypa/setuptools/issues/980? That’s > where most of the discussion from the fall out of setuptools devendoring > has concentrated. > Yup. Jim -- Jim Fulton http://jimfulton.info ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
[Distutils] Pain installing pinned versions of setuptools requirements.
Today, I ran into trouble working with an old project that had six pinned to version 1.1.0. The install failed because buildout tried to install it as 1.10.0 and failed because 1.10.0 was already installed. The problem arose because six's setup.py imports setuptools and then imports six to get __version__. When Buildout runs a setup script, it puts it's own path ahead of the distribution, so the setup script would get whatever version buildout was running. IMO, this is a six bug, but wait, there's more. I tried installing a pinned version with pip, using ``pip install -U six==1.9.0``. This worked. I then tried with version 1.1.0, and this failed, because setuptools wouldn't work with 1.1.0. Pip puts the distribution ahead of it's own path when running a setup script. setuptools requires six >= 1.6, so pip can't be used to install pinned versions (in requirements.txt) earlier than 1.6. Six is a wildly popular package and has been around for a long time. Earlier pins are likely. I raise this here in the broader context of managing clashes between setuptools requirements and requirements of libraries (and applications using them) it's installing. I think Buildout's approach of putting it's path first is better, although it was more painful in this instance. I look forward to a time when we don't run scripts at install time (or are at least wildly less likely to). Buildout is growing wheel support. It should have provided a work around, but: - I happened to be trying to install a 1.1 pin and the earliest six wheel is for 1.. - I tried installing six 1.8. Buildout's wheel extension depended on pip, which depends on setuptools and six. When buildout tries to load the extension, it tries to get the extension's dependencies, which includes six while honoring the version pin, which means it has to install six before it has wheel support. Obviously, this is Buildout's problem, but it illustrates the complexity that arises when packaging dependencies overlap dependencies of packages being managed. IDK what the answer is. I'm just (re-)raising the issue and providing a data point. I suspect that packaging tools should manage their own dependencies independently. That's what was happening until recently IIUC for the pypa tools through vendoring. I didn't like vendoring, but I'm starting to see the wisdom of it. :) Jim -- Jim Fulton http://jimfulton.info ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
[Distutils] Announcing new Buildout documentation
The old horrible doctest-based buildout documentation has finally been replaced: http://docs.buildout.org Jim -- Jim Fulton http://jimfulton.info ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] GSoC 2017 - Plan of Action for dependency resolver
On Tue, Feb 28, 2017 at 10:14 AM, Pradyun Gedam wrote: ... > 4. (if time permits) Move any dependency resolution code out into a > separate library. > >This would make it possible for other projects (like buildout or a > future pip replacement) to reuse the dependency resolver. > Thank you! ... I do intend to reuse some of the work done by Robert Collins in PR #2716 on > pip's GitHub repository. > Are you aware of the proof of concept in distlib? https://distil.readthedocs.io/en/0.1.0/overview.html#actual-improvements Jim -- Jim Fulton http://jimfulton.info ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Issue with (latest?) buildout and package name containing underscore
On Fri, Feb 24, 2017 at 6:59 AM, Paul Moore wrote: > On 24 February 2017 at 11:54, Lele Gaifax wrote: > > Paul Moore writes: > > > >> On 24 February 2017 at 11:37, Lele Gaifax wrote: > >>> Can anyone shed some light on the problem? By what logic buildout used > a > >>> different name for that particular package? > >> > >> While I don't know anything about buildout, pyramid-tm is the > >> normalised version of pyramid_tm - see > >> https://www.python.org/dev/peps/pep-0503/#normalized-names > > > > Oh, I see, thank you. Does that mean that the right thing I should do is > > always using such normalized names in my requirements.txt/versions.cfg? > > I *think* it shouldn't matter. The problem will likely be with older > tools not normalising. So using normalised names throughout might help > such tools. > Thanks Paul. Yes, this is a buildout bug: https://github.com/buildout/buildout/issues/317 This case shed the light on the bug for me. Thanks. Jim -- Jim Fulton http://jimfulton.info ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] distlib and wheel metadata
On Wed, Feb 15, 2017 at 11:55 AM, Freddy Rietdijk wrote: > > Sort of repeating my earlier question, but how often does this happen > in reality? > > From a quick check in our repo we have patched about 1% of our packages to > remove the constraints. We have close to 2000 Python packages. We don't > necessarily patch all the constraints, only when they collide with the > version we would like the package to use so the actual percentage is likely > higher. > > Larger applications that have many dependencies that are fixed have been > kept out of Nixpkgs for now. Their fixed dependencies means we likely need > multiple versions of packages. While Nix can handle that, it means more > maintenance. We have a tool that can take e.g. a requirements.txt file and > generate expressions, but it won't help you much with bug-fix releases when > maintainers don't update their pinned requirements. > I suppose this isn't a problem for Java applications, which use jar files and per-application class paths. Jim -- Jim Fulton http://jimfulton.info ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] distlib and wheel metadata
On Tue, Feb 14, 2017 at 2:40 PM, Vinay Sajip wrote: > > Nope. Honestly, though, I wish there was *one* *library* that defined > the standard, > > which was the case for setuptools for a while (yeah, I know, the warts, > really, I know) > > because I really don't think there's a desire to innovate or a reason > for competition > > at this level. In the case of wheel, perhaps it makes sense for that > implementation to > > be authoritative. > > The problem, to me, is not whether it is authoritative - it's more that > it's ad hoc, just like > setuptools in some areas. For example, the decision to use "metadata.json" > rather than > "pydist.json" is arbitrary, and could change in the future, and anyone who > relies on how things > work now will have to play catch-up when that happens. Unless they depend on a public API provided by the wheel package. Of course, you could argue that the name of a file could be part of the API. In many ways, depending and building on a working implementation is better that drafting a standard from scratch. Packaging has moved forward largely by people who built things pragmatically that worked and solved every-day problems: setuptools/easy_install, buildout, pip, wheel... > That's sometimes just too much work for > volunteer activity - dig into what the problem is, put through a fix (for > now), rinse and > repeat - all the while, little or no value is really added. > > In theory this is an "infrastructure" area where a single blessed > implementation might be OK, > I think so. > but these de facto tools don't do everything one wants, so > interoperability remains important. > Or collaboration to improve the tool. That *should* have worked for setuptools, but sadly didn't, for various reasons. > There's no reason why we shouldn't look to innovate even in this area - > there's some talk of a > GSoC project now to look at dependency resolution Yay! (I saw that.) > for pip G. Why can't this be in a library? (Hopefully it will be.) - something that I had sort-of working > in the distil tool long ago (as a proof of concept) [1]. Almost is a hard sell. If this was usable as a library, I'd be interested in trying to integrate it with buildout. If it worked, many buildout users would be greatful. Perhaps the GSoC project could use it as a reference or starting point. We've gotten so used to how pip and setuptools work, and because they are "good enough", there is a real > failure of imagination > to see how things might be done better. > I think there is a failure of energy. Packaging should largely be boring and most people don't want to work on it. I certainly don't, even though I have. But you picked a good example. There are major differences (I almost said competition) between pip and buildout. They provide two different models (traditional Python system installs vs Java-like component/path installs) that address different use cases. IMO, these systems should complement each other and build on common foundations. Maybe there are more cases for innovation at lower levels than I'm aware of. Jim -- Jim Fulton http://jimfulton.info ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] distlib and wheel metadata
On Tue, Feb 14, 2017 at 1:10 PM, Vinay Sajip wrote: > > humpty in term uses uses distlib which seems to mishandle wheel > > metadata. (For example, it chokes if there's extra distribution meta and > > makes it impossible for buildout to install python-dateutil from a > wheel.) > > I looked into the "mishandling". It's that the other tools don't adhere to > [the current state of] PEP 426 as closely as distlib does. For example, > wheel writes JSON metadata to metadata.json in the .dist-info directory, > whereas PEP 426 calls for that data to be in pydist.json. The non-JSON > metadata in the wheel (the METADATA file) does not strictly adhere to any > of the metadata PEPs 241, 314, 345 or 426 (it has a mixture of incompatible > fields). > > I can change distlib to look for metadata.json, and relax the rules to be > more liberal regarding which fields to accept, but adhering to the PEP > isn't mishandling things, as I see it. > Fair enough. Notice that I said "seems to". :-] I suppose whether to be strict or not depends on use case. In my case, I was just trying to install a wheel as an egg, so permissive is definately what *I* want. Other use cases might want to be more strict. > > Work on distlib has slowed right down since around the time when PEP 426 > was deferred indefinitely, and there seems to be little interest in > progressing via metadata or other standardisation - we have to go by what > the de facto tools (setuptools, wheel) choose to do. It's not an ideal > situation, and incompatibilities can crop up, as you've seen. > Nope. Honestly, though, I wish there was *one* *library* that defined the standard, which was the case for setuptools for a while (yeah, I know, the warts, really, I know) because I really don't think there's a desire to innovate or a reason for competition at this level. In the case of wheel, perhaps it makes sense for that implementation to be authoritative. Thanks. Jim -- Jim Fulton http://jimfulton.info ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
[Distutils] Announcing experimental wheel support in Buildout
I've just released zc.buildout 2.8.0 and the buildout.wheel extension. If you have zc.buildout 2.8.0 or later, and you include: extensions = buildout.wheel In the buildout section of your buildout configuration, then buildout should be able to install distributions as wheels. This allowed me to install numpy using buildout, which wasn't possible before. This is a someone experimental version, which uses humpty to convert wheels to eggs. humpty in term uses uses distlib which seems to mishandle wheel metadata. (For example, it chokes if there's extra distribution meta and makes it impossible for buildout to install python-dateutil from a wheel.) Jim -- Jim Fulton http://jimfulton.info ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] How to specify dependencies in Python
On Tue, Jan 17, 2017 at 11:34 AM, Thomas Güttler < guettl...@thomas-guettler.de> wrote: > > > Am 16.01.2017 um 18:06 schrieb Jim Fulton: > >> >> >> On Mon, Jan 16, 2017 at 11:03 AM, Thomas Güttler < >> guettl...@thomas-guettler.de <mailto:guettl...@thomas-guettler.de>> >> wrote: >> >> >> >> Am 13.01.2017 um 16:25 schrieb Jim Fulton: >> >> >> >> On Fri, Jan 13, 2017 at 7:23 AM, Thomas Güttler < >> guettl...@thomas-guettler.de >> <mailto:guettl...@thomas-guettler.de> > guettl...@thomas-guettler.de >> <mailto:guettl...@thomas-guettler.de>>> wrote: >> >> What is an application for you? >> >> >> Another way to think about this, FWIW, is to distinguish between >> the "whole system" (for which "Application" is >> often a >> useful shorthand), as opposed to components (aka libraries). >> >> It's important for components to be flexible, so they're >> typically very flexible about versions of their >> dependencies. >> >> For whole systems, OTOH, it's important that once a configuration >> is tested, that the same configuration is used in >> production, so systems typically pin all of their dependencies, >> ideally extending to their environments (which is a >> reason why container technology is attractive). >> >> >> Yes, install_requires in setup.py should define flexible >> dependencies, but requirements.txt should define fixed >> dependencies via fixed version. >> >> Do you agree with my sentence from above? >> >> >> Are you speaking of a component/library or whole system? >> > > I am speaking of both. And: I think requirements.txt is optional. Then I disagree with your statement. :) I should stop, but I'll take one more stab at this. It matters whether you're talking about components(/libraries) or whole systems (/applications). For components: Consumers of a component need to be able to to determine the component's dependencies. The component uses install_required (and extras_require) for this. The version specifications in these dependencies should be as flexible as possible, to allow reuse in as many whole systems as possible. Developers of a component will use tools like pip and buildout to automate their development activities. For pip, that will usually entail one or more requirements.txt files. For buildout, that will entail one or more (for different development activities) buildout configs and a single versions.cfg. For whole systems: Many whole systems only assemble components. For these systems, there is no setup.py file (no python project). If a whole system includes a Python project (that isn't distributed separately), it's a matter of taste how much information is included in setup.py. Personally, I would treat the Python project like a component project and include its direct dependencies and minimal version constraints. Typically a whole system managed with pip will use a requirements.txt file (or possibly multiple) and a system developed with buildout will have a buildout config and a versions config. A whole system could fix all of its dependent versions in install_requires in a setup script, but that would be cumbersome. By using requirements.txt or a buildout versions config, a developer can avail themselves of automation to help maintain the files. Jim -- Jim Fulton http://jimfulton.info ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] How to specify dependencies in Python
On Mon, Jan 16, 2017 at 11:03 AM, Thomas Güttler < guettl...@thomas-guettler.de> wrote: > > > Am 13.01.2017 um 16:25 schrieb Jim Fulton: > >> >> >> On Fri, Jan 13, 2017 at 7:23 AM, Thomas Güttler < >> guettl...@thomas-guettler.de <mailto:guettl...@thomas-guettler.de>> >> wrote: >> >> What is an application for you? >> >> >> Another way to think about this, FWIW, is to distinguish between the >> "whole system" (for which "Application" is often a >> useful shorthand), as opposed to components (aka libraries). >> >> It's important for components to be flexible, so they're typically very >> flexible about versions of their dependencies. >> >> For whole systems, OTOH, it's important that once a configuration is >> tested, that the same configuration is used in >> production, so systems typically pin all of their dependencies, ideally >> extending to their environments (which is a >> reason why container technology is attractive). >> > > Yes, install_requires in setup.py should define flexible dependencies, but > requirements.txt should define fixed dependencies via fixed version. > > Do you agree with my sentence from above? Are you speaking of a component/library or whole system? Jim -- Jim Fulton http://jimfulton.info ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] How to specify dependencies in Python
On Fri, Jan 13, 2017 at 7:23 AM, Thomas Güttler < guettl...@thomas-guettler.de> wrote: > What is an application for you? Another way to think about this, FWIW, is to distinguish between the "whole system" (for which "Application" is often a useful shorthand), as opposed to components (aka libraries). It's important for components to be flexible, so they're typically very flexible about versions of their dependencies. For whole systems, OTOH, it's important that once a configuration is tested, that the same configuration is used in production, so systems typically pin all of their dependencies, ideally extending to their environments (which is a reason why container technology is attractive). Jim -- Jim Fulton http://jimfulton.info ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Buoldout2 e setup.py: install_requires ignored
Gaaa, I should have stayed mum. Missed referring to the develop egg too, as Leonardo pointed out. Thanks. Jim On Tue, Sep 13, 2016 at 5:18 PM, Fred Drake wrote: > On Tue, Sep 13, 2016 at 4:44 PM, Jim Fulton wrote: >> You're missing: >> >>develop . >> >> to have buildout create and include a develop egg using the current >> directory with it's dependencies. > > Ahem. > > develop = . > > > > -Fred > > -- > Fred L. Drake, Jr. > "A storm broke loose in my mind." --Albert Einstein -- Jim Fulton http://jimfulton.info ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Buoldout2 e setup.py: install_requires ignored
On Tue, Sep 13, 2016 at 4:07 PM, Alessandro Dentella wrote: > Hi, > > > I just migrated from buildout 1.7 to buildout 2.5. In this migration I > stopped using a recipe that created a virtualenv as a part of > buildout and I now use an external (basic) virtualenv to calll > bootrap, so I can't compare the two configuration in a strict way. > > My problem is that now it seems that install_requires declared in > setup.py are simply ignored, while all packages declared in the eggs > directory are used/installed along with all the packages they depends > on. > > I can't find what's wrong with my configuration, I report below a > minimal setup that shows the problem. > > thanks in advance for any hints > > sandro > *:-) > > sandro@bluff:/tmp/new$ cat buildout.cfg > [buildout] > parts = django > eggs = django >ipython >django_extensions > project-name = mytest You're missing: develop . to have buildout create and include a develop egg using the current directory with it's dependencies. Jim -- Jim Fulton http://jimfulton.info ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Why I like eggs (or similar mechanisms) and my thoughts on future of buildout
On Sun, Aug 21, 2016 at 12:29 PM, Donald Stufft wrote: > > On Aug 21, 2016, at 12:19 PM, Jim Fulton wrote: > > I'm open to shelling out, but pessimistic that it will turn out well. I > started with that approach initially with easy_install and it fell apart > quickly. But when we get into it... who knows? > > > > Shelling out is currently the only exposed “API” that pip has, we’re not > opposed to adding extra APIs though. Our current approach has been to wait > and see for people to come out with specific use cases they have for an API > and then work together to figure out what API we can create that satisfies > that. Thus far we’ve accomplished this by creating new libraries that > aren’t pip and moving functionality out of pip (and setuptools) and into > those libraries, and then making pip/setuptools consume those. This has > generally worked pretty well I think, as it’s easier to be careful not to > accidentally expose some terrible internal details of pip as public API > when it’s a new, carefully designed thing, and we can make working with > those libraries better than it is to simply expose some part of pip. We > generally pair this along with defining things in PEPs so that these new > libraries don’t become the new distutils/setuptools/pip (e.g., > implementation defined standards) which should ideally allow anyone to > create a from scratch implementation and have it interopt just fine. > Sounds reasonable. (I'd seen similar statements before, which I'd alluded to in another message.) Thanks. Jim -- Jim Fulton http://jimfulton.info ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Why I like eggs (or similar mechanisms) and my thoughts on future of buildout
On Sat, Aug 20, 2016 at 6:31 PM, Wes Turner wrote: ... > I may be optimistic, but should adding buildout support for wheel be more > complicated than shelling out to pip with the correct, uhm, --prefix etc? > I'm open to shelling out, but pessimistic that it will turn out well. I started with that approach initially with easy_install and it fell apart quickly. But when we get into it... who knows? Jim -- Jim Fulton http://jimfulton.info ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Why I like eggs (or similar mechanisms) and my thoughts on future of buildout
On Sat, Aug 20, 2016 at 11:07 PM, Nick Coghlan wrote: > On 21 August 2016 at 05:46, Jim Fulton wrote: > > On Sat, Aug 20, 2016 at 3:02 PM, Nick Coghlan > wrote: > >> > I have the impression that uninstalling things can be > >> > problematic, but maybe that's been fixed. > >> > >> Uninstallation is fine, as we *do* have a full file manifest after a > >> component has been installed. > > > > So, if package A and B install the same file, X, and then A is > uninstalled, > > is X still there? If then B is uninstalled, does X go away? Does this > > machinery depend on whether X has the the same contents in A and B? > > If A and B install to the same path, B will overwrite A's version, and > if either is uninstalled, they'll delete whichever version is > currently there. Since "the same path" in this case usually refers to > either a script or a Python module, the unhandled conflict is really > at install time - we'd want something closer to the RPM/deb style "A > file already exists at that destination, so we're not installing the > requested package at all", rather than a Windows-style reference > counting system. > > As Daniel notes, this kind of check is actually already possible when > installing from a wheel file today, it just requires someone with the > roundtuits to add it to pip (perhaps via a new vendorable package for > working with Python installation metadata and detecting conflicts > between what's already installed and a wheel being considered for > installation). > The most common source of conflicts, I expect, are namespace package __init__.py files. You wouldn't want to reject installing a package with a conflicting __init__.py file in a namespace package. I realize that PEP420 and the demise of Python 2 should make this case go away, someday, although PEP420 doesn't prohibit __init__.py files in namespace packages. OK, I'm done beating this horse. It was dead a while ago. :) Jim -- Jim Fulton http://jimfulton.info ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Why I like eggs (or similar mechanisms) and my thoughts on future of buildout
On Sat, Aug 20, 2016 at 3:02 PM, Nick Coghlan wrote: ... > I have the impression that uninstalling things can be > > problematic, but maybe that's been fixed. > > Uninstallation is fine, as we *do* have a full file manifest after a > component has been installed. > So, if package A and B install the same file, X, and then A is uninstalled, is X still there? If then B is uninstalled, does X go away? Does this machinery depend on whether X has the the same contents in A and B? ... Jim -- Jim Fulton http://jimfulton.info ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
[Distutils] Why I like eggs (or similar mechanisms) and my thoughts on future of buildout
n the long run, buildout and pip installed things differently, especially given security considerations. In the long run, I suspect it would be best for some buildout developers to offer pip PRs that abstract functionality into libraries that buildout could call (and that pip called internally), although it sounds like this may already be happening without our help. Jim -- Jim Fulton http://jimfulton.info ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Deprecating little used file types/extensions on PyPI?
On Tue, Aug 16, 2016 at 11:15 AM, Leonardo Rochael Almeida < leoroch...@gmail.com> wrote: ... > A wheel2egg might be nice, but unless it's integrated into setuptools > proper, it would mean adding another dependency to buildout and the > developers would rather depend on a single library for talking to PyPI > I'm not sure this is as important as it once was. Depending on setuptools and it's bootstrapping mechanism easier, but setuptools' bootstrapping functionality is going away. We've discussed giving up on allowing buildout to bootstrap itself without modifying the Python install, although I still rather like that feature. Jim -- Jim Fulton http://jimfulton.info ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Outdated packages on pypi
Well said. IMO, package names shouldn't be reused. Also, IMO, we have a namespace problem, for which there's a common solution that we avoid (domain based names, which can also be reused, but ...). OTOH, here's an idea. What if in addition to the project name, we also assigned a unique id. When a package was added to a consuming project, we'd store both the packages name, and it's project id. When looking up a package, we'd supply both the project name and id. If a name was reused, the new project with the same name would have a new project id and wouldn't be confused with the old one. We could even still server the old project's released without advertising them. This way, if we did decide to reuse a name, we could do so pretty safely. Jim On Wed, Jul 13, 2016 at 12:54 AM, Donald Stufft wrote: > >> On Jul 12, 2016, at 4:45 PM, Glyph Lefkowitz wrote: >> >> My feeling is that there should be a "dead man's switch" sort of mechanism >> for this. Require manual intervention from at least one package owner at >> least once a year. I believe if you dig around in the archives there's been >> quite a bit of discussion around messaging to package owners and that sort >> of thing - and the main sticking point is that someone needs to volunteer to >> do the work on Warehouse. Are you that person? :) > > > I suspect any change like this will require some sort of PEP or something > similar to it. It’s something that I think is going to hard to get just right > (if it’s something we want to do at all). > > Software can be “finished” without needing more releases, and sometimes > projects stop getting updates until the maintainer has more time (or a new > maintainer comes along). An example is setuptools which had no releases > between Oct 2009 and Jun 2013. Another nice example is ``wincertstore`` which > has had two releases one in 2013 and one in 2014 and is one of the most > downloaded projects on PyPI. It doesn’t need any updates because it’s just a > wrapper around Windows APIs via ctypes. > > Another thing we need to be careful about is what do we do once said dead > man’s switch triggers? We can’t just release the package to allow anyone to > register it, that’s just pointing a security shaped footgun at the foot of > every person using that project? It doesn’t make sense to block new uploads > for that project since there’s no point to disallowing new uploads. Flagging > it to allow someone to “take over” (possibly with some sort of review) has > some of the security shaped footguns as well as a problem with deciding who > to trust with a name or not. > > — > Donald Stufft > > > > ___ > Distutils-SIG maillist - Distutils-SIG@python.org > https://mail.python.org/mailman/listinfo/distutils-sig -- Jim Fulton http://jimfulton.info ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Outdated packages on pypi
On Tue, Jul 12, 2016 at 7:55 AM, Dima Tisnek wrote: > Hi all, > > Is anyone working on pruning old packages from pypi? > > I found something last updated in 2014, which, looking at the source > appears half-done. > Github link doesn't work any longer, no description, etc. > > I managed to find author's email address out of band, and he responded > that he can't remember the password, yada yada. > > I wonder if some basic automation is possible here -- check if url's > are reachable and if existing package satisfies basic requirements, > failing that mark it as "possibly out of date" I'm curious why you view this as a problem that needs to be solved? - Do you want to take over the name yourself? - Are you afraid someone will stumble on this package and use it? Something else? Jim -- Jim Fulton http://jimfulton.info ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Docker, development, buildout, virtualenv, local/global install
On Thu, Jun 16, 2016 at 8:47 PM, Nick Coghlan wrote: > On 16 June 2016 at 05:01, Jim Fulton wrote: >> I'm a fan of docker, but it seems to me that build workflow is a an >> unsolved problem if you need build tools that you don't want be >> included at run time. > > For OpenShift's source-to-image (which combines builder images with a > git repo to create your deployment image [1]), they tackled that > problem by setting up a multi-step build pipeline where the first > builder image created the binary artifact (they use a WAR file in > their example, but it can be an arbitrary tarball since you're not > doing anything with it except handing it to the next stage of the > pipeline), and then a second builder image that provides the required > runtime environment and also unpacks the tarball into the right place. > > For pure Python (Ruby, JavaScript, etc) projects you'll often be able > to get away without the separate builder image, but I suspect that > kind of multi-step model would be a better fit for the way buildout > works. Or pip. I don't think this is really about buildout or pip. I agree that a multi-step process is needed if builds are required. I suggested that in my original response. When I said this was an unsolved problem, I didn't mean that there weren't solutions, but that there didn't seem to be widely accepted workflows for this within the docker community (or at least docs). The common assumption is that images are defined by a single build file, to the point that images are often "published" as git repositories with build files, rather than through a docker repository. Anyway, getting back to Reinout's original question, I don't think docker build should be used for development. Build should assemble (again, using buildout, pip, or whatever) existing resources without actually building anything. (Of course, docker can be used for development, and docker build could and probably should be used to create build environments.) Jim -- Jim Fulton http://jimfulton.info ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Docker, development, buildout, virtualenv, local/global install
On Wed, Jun 15, 2016 at 5:39 PM, Reinout van Rees wrote: > Op 15-06-16 om 13:53 schreef Jim Fulton: >> >> >> 1. Creating production docker images works best when all you're doing >> is installing a bunch of binary artifacts (e.g. .debs, eggs, wheels). > > > That's where pip and its "everything is in requirements.txt anyway" has an > advantage. The buildouts I use always have most/all dependencies mentioned > in the setup.py with version pins in the buildout config. So I need the > setup.py and so I need my full source code as the setup.py won't work > otherwise. No. If you had a built version of your project, the requirement information in the setup.py would still be available and used by buildout. How you specify requirements is beside the point. Whether you use buildout or pip, you need built versions of your non-pure-oython dependencies available and used. > > I *do* like to specify all "install_requires" items in my setup.py, so this > is something I need to think about. I don't see how this is a problem. If your project is pure python, you don't need build tools to build it, so building it as part of building a docker image isn't a problem. If it isn't pure Python, then you'll need to have a built version available, and the meta-data in that built version will provide the necessary requirements information. I'm not trying here to plug buildout vs pip. I'm just saying that the pip vs builtout (or both) decision is irrelevant to building a docker image or docker workflow. > >> 3. IMO, you're better off separating development and image-building >> workflows, with development feeding image building. Unfortunately, >> AFAIK, there aren't standard docker workflows for this, although I >> haven't been paying close attention to this lately. > > > docker-compose helps a little bit here in that you can "compose" your docker > into several scenarios. It *does* mean you have to watch carefully what > you're doing, as one of the goals of docker, as far as I see it, is to have > a large measure of "production/development parity" as per the "12 factor > app" manifesto (http://12factor.net/) I think this only helps to the extent that it encourages you to decompose applications into lots of separately deployed bits. I'm a fan of docker, but it seems to me that build workflow is a an unsolved problem if you need build tools that you don't want be included at run time. Jim -- Jim Fulton http://jimfulton.info ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Docker, development, buildout, virtualenv, local/global install
On Wed, Jun 15, 2016 at 7:57 AM, Donald Stufft wrote: > >> On Jun 15, 2016, at 7:53 AM, Jim Fulton wrote: >> >> If you actually build programs as part of image building, then your >> image contains build tools, leading to image bloat and potentially >> security problems as the development tools provide a greater attack >> surface. > > This isn’t strictly true, the layering in Docker works on a per RUN command > basis, so if you compose a single command that installs the build tools, > builds the thing, installs the thing, and uninstalls the build tools (and > cleans up any cache), then that’s roughly equivalent to installing a single > binary (except of course, in the time it takes). OK, fair enough. People would typically start from an image that had the build tools installed already. But as you point out, you could have a single step that installed the build tools, built and then uninstalled the build tools. You'd avoid the bloat, but have extremely long build times. Jim -- Jim Fulton http://jimfulton.info ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Docker, development, buildout, virtualenv, local/global install
On Wed, Jun 15, 2016 at 5:07 AM, Reinout van Rees wrote: > Hi, > > Buzzword bingo in the subject... > > Situation: I'm experimenting with docker, mostly in combination with > buildout. But it also applies to pip/virtualenv. > > I build a docker container with a Dockerfile: install some .deb packages, > add the current directory as /code/, run buildout (or pip), ready. Works > fine. > > Now local development: it is normal to mount the current directory as > /code/, so that now is overlayed over the originally-added-to-the-docker > /code/. > > > > This means that anything done inside /code/ is effectively discarded in > development. So a "bin/buildout" run has to be done again, because the bin/, > parts/, eggs/ etc directories are gone. > > Same problem with a virtualenv. *Not* though when you run pip directly and > let it install packages globally! Those are installed outside of /code in > /usr/local/somewhere. > > > > A comment and a question: > > - Comment: "everybody" uses virtualenv, but be aware that it is apparently > normal *not* to use virtualenv when building dockers. > > - Question: buildout, like virtualenv+pip, installs everything in the > current directory. Would an option to install it globally instead make > sense? I don't know if it is possible. So, a number of comments. I'm not going to address your points directly. 1. Creating production docker images works best when all you're doing is installing a bunch of binary artifacts (e.g. .debs, eggs, wheels). If you actually build programs as part of image building, then your image contains build tools, leading to image bloat and potentially security problems as the development tools provide a greater attack surface. You can uninstall the development tools after building, but that does nothing to reduce the bloat -- it just increases it, and increases buid time. 2. #1 is a shame, as it dilutes the simplicity of using docker. :( 3. IMO, you're better off separating development and image-building workflows, with development feeding image building. Unfortunately, AFAIK, there aren't standard docker workflows for this, although I haven't been paying close attention to this lately. There are standard docker images for building system packages (debs, rpms), but, for myself, part of the benefit of docker is not having to build system packages anymore. (Building system packages, especially applications, for use in docker could be a lot easier than normal as you could be a lot sloppier about the ways that packages are built, thus simpler packaging files...) 4. I've tried to come up with good buildout-based workflows for building docker images in the past. One approach was to build in a dev environment and then, when building the production images, using eggs built in development, or built on a CI server. At the time my attempts were stymied by the fact that setuptools prefers source distributions over eggs so if it saw the download cache, it would try to build from source. Looking back, I'm not sure why this wasn't surmountable -- probably just lack of priority and attention at the time. 5. As far as setting up a build environment goes, just mounting a host volume is simple and works well and even works with docker-machine. It's a little icky to have linux build artifacts in my Mac directories, but it's a minor annoyance. (I've also created images with ssh servers that I could log into and do development in. Emacs makes this easy for me and provides all the development capabilities I normally use, so this setup works well for me, but probably doesn't work for non-emacs users. :) ) Jim -- Jim Fulton http://jimfulton.info ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Basic Markdown Readme Support
On Tue, May 3, 2016 at 9:18 AM, Fred Drake wrote: > My perspective, for what it's worth, is that while I find Markdown a > horrible pain, But wait, it's worse. Unlike ReStructuredText, there's no Markdown standard. In my last job, I had to use a suite of tools (from a single company that I won't name but is easy to guess :) ) for which no 2 tools used the same dialect of Markdown. :( ... > So while I'm all for encouraging developers to prefer > reStructuredText, I'm in favor of supporting Markdown as a > long_description format. The format for a README file just doesn't > seem such a big deal that alienating potential community members is > worth it. Which begs the question, which dialect of Markdown are you suggesting we support. :) Jim -- Jim Fulton http://jimfulton.info ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Heads up: https://bootstrap.pypa.io/ez_setup.py is broken
On Mon, Apr 25, 2016 at 2:45 PM, Jim Fulton wrote: > Due to recent changes in PyPI URLs. This breaks buildout's > bootstrap.py as well as other tools. I created an PR against the setuptools repo that should fix this. Jim -- Jim Fulton http://jimfulton.info ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
[Distutils] Heads up: https://bootstrap.pypa.io/ez_setup.py is broken
Due to recent changes in PyPI URLs. This breaks buildout's bootstrap.py as well as other tools. Jim -- Jim Fulton http://jimfulton.info ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] The mypy package
I suggest measuring activity by downloads, not releases. Sometimes maintained packages are boring enough not to need releases, while many projects depend on them. Jim On Mon, Apr 18, 2016 at 11:18 AM, Chris Barker - NOAA Federal wrote: >> Though I do wonder how effective that would be in this case. For all we >> know, in the case of mypy, the maintainer is simply ignoring someone else >> who is trying to take the name they registered. (I get emails all the time >> for people trying to get me to sign over my domain names; > > Domain names are a different system -- you need to maintain your registration. > > PyPi names, on the other hand, are all too easy to setup, and then > completely ignore, maybe even forget you used it. > > We really should have SOME way to determine if a PyPi name has been > abandoned. Or even be proactive--PyPi names must be maintained in SOME > way, perhaps: > > Push a change or update at least once a year (or some other interval). > > Or > > Respond to some sort of "do you still want this" email. At least once a year. > > If neither of these occurs, then we could have a deprecation period. > > Details aside, as PyPi continues to grow, we really need a way to > clear out the abandoned stuff -- the barrier to entry for creating a > new name on PyPi is just too low. > > This is all too late for MyPy, but it has certainly come up before, > and will again, more and more. > > -CHB > ___ > Distutils-SIG maillist - Distutils-SIG@python.org > https://mail.python.org/mailman/listinfo/distutils-sig -- Jim Fulton http://jimfulton.info ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
[Distutils] Table of contents formatting in PyPI pages generated from long descriptions
I like to include tables of contents in my distribution long descriptions. Normally. ReST formats these with links, so that when someone clicks on and entry in the TOC, they jump to that position in the document. Recently (last month?) PyPI stopped displaying TOCs with links. Was this intentional? Jim -- Jim Fulton http://jimfulton.info ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] New buildout release needed
On Tue, Jun 30, 2015 at 10:45 PM, Reinout van Rees wrote:
> Hi,
>
> I've done quite a lot of work on buildout in the last two or three weeks.
> Merging pull requests and also submitting a couple of my own.
Thanks!
>
> If you look at the list of pull requests
> (https://github.com/buildout/buildout/pulls) you'll see a bunch that need
> further work ("doc or tests needed") and a handful of "plus or minus" pull
> requests, dealing with plus-minus stuff regarding sections. I don't use that
> much and I haven't looked at those yet. Perhaps someone else wants to?
>
> I've also set up travis-ci.org caching so that the builds on travis run two
> to three times faster.
>
> And I've fixed several issues dealing with non-ascii filenames. Apparently,
> if you install pyramid (for instance), buildout will fail to run. Apparently
> it is enough to install something like mr.bob to break buildout totally. It
> is now fixed in https://github.com/buildout/buildout/pull/250
>
>
> There are two things that need doing now that I cannot do:
>
>
> - Review https://github.com/buildout/buildout/pull/248 with a number of
> bootstrap fixes. The fixes themselves aren't controversial, I think (adding
> a version, for instance). The one thing I want feedback on is that buildout
> now deletes the contents of the develop-eggs/ directory when it bootstraps.
> This helps a lot with faulty left-over egg-links in corner cases. The old
> osc.recipe.sysegg recipe for instance used to reliably wreak my buildouts
> (syseggrecipe is the replacement that does the right thing). Zapping the
> develop-eggs contents on bootstrap helps solve several problems. In my case,
> I really want this change to go in as it might mean the difference between
> my company using buildout or not... I'm getting tired of saying "oh, remove
> the develop-eggs directory contents".
>
> - Make a new zc.buildout, zc.recipe.egg and bootstrap.py release. See issue
> https://github.com/buildout/buildout/issues/249 .
You can make a bootstrap release by merging to the bootstrap_release branch.
I can make releases this weekend, or I can empower you. Which would you prefer?
Jim
--
Jim Fulton
http://jimfulton.info
___
Distutils-SIG maillist - Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] PyPI is a sick sick hoarder
On Fri, May 15, 2015 at 2:57 PM, Robert Collins wrote: > So, I am working on pip issue 988: pip doesn't resolve packages at all. > > This is O(packages^alternatives_per_package): if you are resolving 10 > packages with 10 versions each, there are approximately 10^10 or 10G > combinations. 10 packages with 100 versions each - 10^100. > > So - its going to depend pretty heavily on some good heuristics in > whatever final algorithm makes its way in, but the problem is > exacerbated by PyPI's nature. > > Most Linux (all that i'm aware of) distributions have at most 5 > versions of a package to consider at any time - installed(might be > None), current release, current release security updates, new release > being upgraded to, new release being upgraded to's security updates. > And their common worst case is actually 2 versions: installed==current > release and one new release present. They map alternatives out into > separate packages (e.g. when an older soname is deliberately kept > across an ABI incompatibility, you end up with 2 packages, not 2 > versions of one package). To when comparing pip's challenge to apt's: > apt has ~20-30K packages, with altnernatives ~= 2, or > pip has ~60K packages, with alternatives ~= 5.7 (I asked dstufft) > > Scaling the number of packages is relatively easy; scaling the number > of alternatives is harder. Even 300 packages (the dependency tree for > openstack) is ~2.4T combinations to probe. > > I wonder if it makes sense to give some back-pressure to people, or at > the very least encourage them to remove distributions that: > - they don't support anymore > - have security holes > > If folk consider PyPI a sort of historical archive then perhaps we > could have a feature to select 'supported' versions by the author, and > allow a query parameter to ask for all the versions. You could simply limit the number of versions from PyPI you consider. Jim -- Jim Fulton http://www.linkedin.com/in/jimfulton ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] PyPI and Uploading Documentation
On Fri, May 15, 2015 at 9:48 AM, Donald Stufft wrote: > Hey! > > First, for anyone who isn't aware we recently migrated PyPI and TestPyPI so > that instead of storing files and documentation locally (really in a glusterfs > cluster) it will store them inside of S3. This will reduce maintenance > overhead > of running PyPI by two servers since we'll no longer need to run our own > glusterfs cluster as well as improve the reliaiblity and scalability of the > PyPI service as a whole since we've had nothing but problems from glusterfs in > this regard. > > One of the things that this brought to light was that the documentation > upload ability in PyPI is something that is not often used* however it > represents something which is one of our slowest routes. It's not a well > supported feature and I feel that it's going outside of the core competancy > for > PyPI itself and instead PyPI should be focused on the files themselves. In > addition since the time this was added to PyPI a number of free services or > cheap services have came about that allow people to sanely upload raw document > without a reliance on any particular documentation system and we've also had > the rise of ReadTheDocs for when someone is using Sphinx as their > documentation > system. > > I think that it's time to retire this aspect of PyPI which has never been well > supported and instead focus on just the things that are core to PyPI. I don't > have a fully concrete proposal for doing this, but I wanted to reach out here > and figure out if anyone had any ideas. The rough idea I have currently is to > simply disable new documentation uploads and add two new small features. One > will allow users to delete their existing documentation from PyPI and the > other > would allow them to register a redirect which would take them from the current > location to wherever they move their documentation too. In order to prevent > breaking documentation for projects which are defunct or not actively > maintained we would maintain the archived documentation (sans what anyone has > deleted) indefinetely. > > Ideally I hope people start to use ReadTheDocs instead of PyPI itself. I think > that ReadTheDocs is a great service with heavy ties to the Python community. > They will do a better job at hosting documentation than PyPI ever could since > that is their core goal. In addition there is a dialog between ReadTheDocs and > PyPI where there is an opportunity to add integration between the two sites as > well as features to ReadTheDocs that it currently lacks that people feel are a > requirement before we move PyPI's documentation to read-only. > > Thoughts? +1 > * Out of ~60k projects only ~2.8k have ever uploaded documentation. It's not > easy to tell if all of them are still using it as their primary source of > documentation though or if it's old documentation that they just can't > delete. I know I have documentation for at least one project hosted this way. I don't remember how I set that up. :) I assume there will be some way to notify owners of effected documentation. Jim -- Jim Fulton http://www.linkedin.com/in/jimfulton ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] buildout/setuptools slow as it scans the whole project dir
On Wed, Apr 15, 2015 at 10:01 AM, Reinout van Rees wrote: > Hi, > > In some of my projects, buildout takes a looong time to complete. > Sometimes the same problem occurs on the server or on another developer's > laptop, sometimes not. The difference is something like "15 seconds if > there's no problem" and "5 minutes if the problem occurs". ... I wonder if the culprit is _dir_hash in zc.buildout.buildlout. Buildout reinstalls a part if it has changed. It considers a part to have changed if it's arguments have changed or it's recipe has changed. If a recipe is a develop egg, then it computes a hash for the recipe based on it's package contents. A quick thing to try might be to hack _dir_hash to be a noop (e.g. add ``return 42`` at the top and see if it makes the delay go away.) Jim -- Jim Fulton http://www.linkedin.com/in/jimfulton ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Detecting rst errors when uploading to pypi
On Sat, Dec 27, 2014 at 11:26 PM, John Anderson wrote: > Hey, I'm trying to get my README to render properly on pypi but I haven't > found a way to detect the errors pypi is experiencing. > > The package I'm trying to upload is here: > https://pypi.python.org/pypi/pyramid_celery > > I've checked the README.rst with restview, rst2html, and collective.showdocs > and they all render the RST just fine. > > It seems like I'm not the only one having this issue: > https://bitbucket.org/pypa/pypi/issue/161/rest-formatting-fails-and-there-is-no-way > > So what is the recommended route for debugging the problem? I would really > like to fix it but without know the problem I would just to make random > guesses. I run: setup.py --long-description and pipe the output into a ReST processor to check for errors. (I use rst2, https://pypi.python.org/pypi/zc.rst2, which is a thin but convenient wrapper around docutils for converting ReST to various formats.) Jim -- Jim Fulton http://www.linkedin.com/in/jimfulton ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Can zc.buildout use same comparison operator for versions as setup.py
On Thu, Dec 25, 2014 at 10:21 PM, William Zhang wrote: > Hello > Can zc.buildout use same comparison operator for versions as setup.py? No. They do technically allow limited ranges, but the primary purpose is to set/pin/nail specific versions. Fred did a good job of explaining this. ... Why do you ask? > Is there any way for "show-picked-versions" print version be used by > setup.py install_requires? If you run buildout in verbose mode (-v), it will will give an indication of why it's trying to get a particular requirement. Note that requirements can come from a variety of places, including setup.py, the versions section, and requirements stated in egg and similar parts. Jim -- Jim Fulton http://www.linkedin.com/in/jimfulton ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] setuptools-8.4.zip?
On Sat, Dec 27, 2014 at 1:47 PM, Alex Gaynor wrote: > Jim Fulton zope.com> writes: > >> >> Did someone forget to upload setuptools-8.4.zip? >> >> Jim >> > > > I don't really understand how it came about, but the sever was on the wrong > branch or something. Benjamin Peterson whacked it with a stick and now it > should be good. Thanks. Jim -- Jim Fulton http://www.linkedin.com/in/jimfulton ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
[Distutils] setuptools-8.4.zip?
Bootstrapping buildout is failing because buildout uses https://bootstrap.pypa.io/ez_setup.py which tried to download: https://pypi.python.org/packages/source/s/setuptools/setuptools-8.4.zip which doesn't exist. Did someone forget to upload setuptools-8.4.zip? Jim -- Jim Fulton http://www.linkedin.com/in/jimfulton ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] setuptools 8 changes are great, but ...
On Tue, Dec 16, 2014 at 11:48 AM, Donald Stufft wrote: ... > Yea, I’m incredibly lucky to have found someone willing to pay me for my > hobby. Rackspace pays me to work on Python packaging Thanks Rackspace! Jim -- Jim Fulton http://www.linkedin.com/in/jimfulton ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
[Distutils] Buildout 2.3.1 released
This: - Fixes the picked-version bug - Suppresses spurious (and not spurious) legacy version warnings. (When that setuptools/packaging problem is fixed. I'll release 2.3.2 that undoes this.) If we're still in significant pain tomorrow, then I'll release a 2.4 that requires setuptools <8. This isn't a long-term, because old setuptools won't handle new-style (better) versions. Jim -- Jim Fulton http://www.linkedin.com/in/jimfulton ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] setuptools 8 changes are great, but ...
On Mon, Dec 15, 2014 at 6:48 PM, Donald Stufft wrote: > >> On Dec 15, 2014, at 6:46 PM, Maurits van Rees >> wrote: >> >> Maurits van Rees schreef op 15-12-14 23:50: >>> TMP2/setuptools-8.0.4-py2.7.egg/pkg_resources.py:2425: RuntimeWarning: >>> 'zc.buildout-2.3.0 ()' is being parsed as a legacy, non PEP 440, >>> version. You may find odd behavior and sort order. In particular it will >>> be sorted as less than 0.0. It is recommend to migrate to PEP 440 >>> compatible versions. >> >> Actually, I have configured zc.buildout to use a download-cache directory >> where it saves downloaded zips/tarballs/eggs from PyPI. Buildout then adds >> this directory as a find-link and setuptools calls package_index.scan_url on >> it. This prints the warnings, because it sees for example a file >> '/Users/mauritsvanrees/cached-downloads/dist/Acquisition-2.13.8.zip' >> This gets parsed as a Distribution with project_name 'Acquisition-2.13.8' >> and an empty version. >> >> Ah, a demo with plain setuptools 0.8.4 is easy to setup: >> >> $ mkdir bar >> $ touch bar/project-1.0.zip >> $ . bin/activate >> (venv) $ python >> Python 2.7.8 (default, Jul 28 2014, 10:41:45) >> [GCC 4.2.1 Compatible Apple LLVM 5.1 (clang-503.0.40)] on darwin >> Type "help", "copyright", "credits" or "license" for more information. >> >>> from setuptools import package_index >> >>> pi = package_index.PackageIndex() >> >>> pi.add_find_links(['bar']) >> /Users/mauritsvanrees/tmp/venv/lib/python2.7/site-packages/pkg_resources.py:2425: >> RuntimeWarning: 'project-1.0 ()' is being parsed as a legacy, non PEP 440, >> version. You may find odd behavior and sort order. In particular it will be >> sorted as less than 0.0. It is recommend to migrate to PEP 440 compatible >> versions. >> RuntimeWarning, >> >>> > > Ah, this is probably an issue with how I detected a legacy version and how > setuptools parses a filename. I can probably get rid of the warning spam. That's awesome, because with large download caches, there can be a lot of spam, as in 2800 lines of spam for me. :) I guess this is a good a time as any to clean up my cache. Jim -- Jim Fulton http://www.linkedin.com/in/jimfulton ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] change in setuptools 8.0
On Mon, Dec 15, 2014 at 7:05 PM, Maurits van Rees wrote: > Jim Fulton schreef op 15-12-14 19:15: >> >> Buildout doesn't try to merge dependency requirements from different >> packages. >> It just installs packages and their dependencies and builds up a working >> set. >> >> It does merge constraints from the versions section (if any) with >> whatever requirement >> it's working on at any point in time. >> >> I'm guessing that the requirement: zc.buildout==2.2.5,>=1.5.0 >> is coming from a setup spec or from some buildout section. It was a >> valid spec once >> but isn't any more. > > > Let's try a buildout with Plone. This one goes fine (except for possible > warnings about legacy versions): > > $ curl -O https://bootstrap.pypa.io/bootstrap-buildout.py > $ cat buildout.cfg > [buildout] > extends = http://dist.plone.org/release/4.3-latest/versions.cfg > versions = versions > parts = instance > > [versions] > # Override version pins that are in the file of the extends line. > zc.buildout = 2.3.0 > setuptools = 8.0.4 > > [instance] > recipe = plone.recipe.zope2instance > user = admin:admin > http-address = 8080 > eggs = > Plone > > $ python bootstrap-buildout.py > $ bin/buildout > > All is well. > > Now edit buildout.cfg and add this line in the [buildout] section: > > allow-picked-versions = false > > $ bin/buildout > ... > While: > Installing. > Getting section instance. > Initializing section instance. > Installing recipe plone.recipe.zope2instance. > Getting distribution for 'ZODB3==3.10.5,>=3.9'. > Error: Picked: ZODB3 = 3.10.5 > > Somehow a problem appears because the buildout config pins ZODB3 to 3.10.5, > and buildout installs plone.recipe.zope2instance which requires 'ZODB3 >= > 3.9'. (I wonder if this behaviour is specific for recipes, but I guess > not.) > > Now revert the change, and instead add this line in the [buildout] section: > > show-picked-versions = true > > $ bin/buildout > ... > Versions had to be automatically picked. > The following part definition lists the versions picked: > [versions] > AccessControl = 3.0.8 > Markdown = 2.0.3 > Products.ATContentTypes = 2.1.14 > Products.OFSP = 2.13.2 > Products.PlonePAS = 4.1.4 > ZODB3 = 3.10.5 > Zope2 = 2.13.22 > archetypes.schemaextender = 2.1.4 > collective.z3cform.datetimewidget = 1.2.6 > diazo = 1.0.6 > lxml = 2.3.6 > mechanize = 0.2.5 > plone.app.contentmenu = 2.0.10 > plone.app.imaging = 1.0.11 > plone.app.jquery = 1.7.2 > plone.app.linkintegrity = 1.5.5 > plone.app.registry = 1.2.3 > plone.app.z3cform = 0.7.7 > plone.autoform = 1.6.1 > plone.behavior = 1.0.2 > plone.browserlayer = 2.1.3 > plone.dexterity = 2.2.4 > plone.keyring = 2.0.1 > plone.namedfile = 2.0.7 > plone.registry = 1.0.2 > plone.resource = 1.0.3 > plone.scale = 1.3.4 > plone.schemaeditor = 1.3.8 > plone.supermodel = 1.2.6 > plone.z3cform = 0.8.0 > python-gettext = 1.2 > python-openid = 2.2.5 > repoze.xmliter = 0.5 > transaction = 1.1.1 > z3c.form = 3.2.1 > zope.app.locales = 3.6.2 > zope.tales = 3.5.3 > > All these versions are definitely pinned in the file we extend (or in files > that this file extends): > http://dist.plone.org/release/4.3-latest/versions.cfg > > So: strange behaviour. I'm not sure how this relates to what you reported before, but this is a known bug that I think I introduced on Sunday. I'm about to work on a fix. Jim -- Jim Fulton http://www.linkedin.com/in/jimfulton ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] setuptools 8 changes are great, but ...
On Mon, Dec 15, 2014 at 1:45 PM, Donald Stufft wrote: > >> On Dec 15, 2014, at 1:27 PM, Jim Fulton wrote: >> >> I think the changes in version management in setuptools 8 are a great >> step forward, >> but I think the transition is going to hurt a lot. >> >> For buildout, I'm thinking of of releasing 2.3.1 that reverts the >> changes in 2.3 and >> adds a requirement for setuptools <8 to give more time to respond to >> these changes. > > Is there something I’m not aware that is broken currently? I thought the > transition was going pretty smoothly overall considering that a core piece > of code inside of setuptools was touched. The problem is that version numbers and requirements aren't interpreted the same way. I think that's going to take some time to sort out. In the mean time, I'd like to try to stop the pain. I think you've been very responsive and that's much appreciated. Jim -- Jim Fulton http://www.linkedin.com/in/jimfulton ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
[Distutils] setuptools 8 changes are great, but ...
I think the changes in version management in setuptools 8 are a great step forward, but I think the transition is going to hurt a lot. For buildout, I'm thinking of of releasing 2.3.1 that reverts the changes in 2.3 and adds a requirement for setuptools <8 to give more time to respond to these changes. Jim -- Jim Fulton http://www.linkedin.com/in/jimfulton ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] change in setuptools 8.0
sistent with the requirement,
>> 'five.localsitemanager>2.0dev'.
>> While:
>> Installing zeoclient.
>> Error: Bad constraint 2.0.5 five.localsitemanager>2.0dev
>>
>> The bad constraint '>2.0dev' is in the five.grok package. I guess it should
>> have been '>2.0.dev0' (or by now simply '>=2.0'). I'll pick it up for that
>> package.
>>
>
> I don’t know much (or anything really) about buildout or what it does with
> that.
Buildout doesn't try to merge dependency requirements from different packages.
It just installs packages and their dependencies and builds up a working set.
It does merge constraints from the versions section (if any) with
whatever requirement
it's working on at any point in time.
I'm guessing that the requirement: zc.buildout==2.2.5,>=1.5.0
is coming from a setup spec or from some buildout section. It was a
valid spec once
but isn't any more.
Buildout 2 before 2.3 did a lot of strange gymnastics to try to merge
version constraints
with requirement specs given the strange old rules.
Now, with the new sane rules, it just adds the constraint to the
whatever existing spec is
in the requirement.
There error "Bad constraint 2.0.5 five.localsitemanager>2.0dev" looks
to me like a bug. :)
When buildout gets a single-version constraint, like 2.0.5, it tests
whether that version
is "in" (__contains__) a requirement.
>>> import pkg_resources
>>> r = pkg_resources.Requirement.parse("five.localsitemanager>2.0dev")
>>> '2.0.5' in r
False
>>> r = pkg_resources.Requirement.parse("five.localsitemanager>2.0.dev0")
>>> '2.0.5' in r
True
I have a feeling the definition of "dev" has changed too.
Jim
--
Jim Fulton
http://www.linkedin.com/in/jimfulton
___
Distutils-SIG maillist - Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig
[Distutils] zc.buildout 2.3.0 released
This fixes some compatibility issues with (and requires) setuptools 8. The bootstrap file has been fixed and also moved to a more secure location: https://bootstrap.pypa.io/bootstrap-buildout.py Jim -- Jim Fulton http://www.linkedin.com/in/jimfulton ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Are there any plans to move to pip/wheels in buildout?
On Wed, Dec 3, 2014 at 11:14 AM, Paul Moore wrote: > On 3 December 2014 at 16:05, Daniel Holth wrote: >> Wheel also has its own installer. It hasn't received as much attention >> since pip gained support. > > That's actually a very good point - the wheel project has its own > "install a wheel" API, not just the command line tool. If that's all > buildout needs, it may be a simple way of adding wheel support. If > buildout needs "the other stuff" that pip/setuptools provides, around > discovery and downloading from PyPI, version/requirements checking, > script wrappers, etc, then that's a bigger issue of course. Yup. Buildout uses setuptools to find distributions. I suppose the wheel installer library could facilitate adding wheel support to setuptools. Maybe someone on the buildout team should look at that. My guess is that adding discovery of wheels in indexes and such probably isn't that hard. Jim -- Jim Fulton http://www.linkedin.com/in/jimfulton ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Are there any plans to move to pip/wheels in buildout?
On Wed, Dec 3, 2014 at 10:12 AM, Paul Moore wrote: > On 3 December 2014 at 14:47, Jim Fulton wrote: >>> Am I missing something about binary support with wheels? >> >> I don't think so. > > Agreed. > > It seems to me that the key issue currently is that for binary > distribution, pip only supports wheels, whereas buildout only supports > eggs. (And .exe) > There are other tools which also tend to only support one or the > other (e.g., easy_install -> egg, distil -> wheel) but I think they > are the minority. I don't know relative popularity between pip and > buildout - I suspect it depends on the community involved. > > For a project, decisions on which binary formats to provide probably > need to be based on the expected numbers of users of each tool. > > The wheel and egg formats are similar enough to allow auto-conversion > (wheel convert does egg->wheel, I don't know if there's a tool to do > wheel->egg, but it wouldn't be hard to write) but expecting users to > maintain local converted repositories is a burden in itself, so the > question remains what binaries to upload to PyPI. For now, it would be humane to upload both wheel and egg. :) We're happy to move buildout to using wheel. It will make me very sad if we have to do that by invoking pip as a subprocess, but I'll cope if I have to. I think the next step is for me to enumerate buildout's packaging api requirements and desires. I'll try to get that done this weekend. (I'll need to spend some time refreshing my memory on some things.) Jim -- Jim Fulton http://www.linkedin.com/in/jimfulton ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Are there any plans to move to pip/wheels in buildout?
On Wed, Dec 3, 2014 at 9:32 AM, Randy Syring wrote: > > On 12/03/2014 09:01 AM, Jim Fulton wrote: > > Many users, like myself, download source releases exclusively, and > to us, wheels are irrelevent. However, users on Windows or who use > systems without build tools for some reason, will experience > breakage as soon as people stop creating traditional binary releases > in favor of wheels. > > > Does that have to be the case? Yes, users of buildout on systems without build tools will experience breakage if packagers stop distributing any of the older binary formats. This will remain true until either setuptools grows wheel support, or until buildout uses something else (other than setuptools). > As one of the maintainers of pymssql, we are > in the process of figuring out what type of packages we want to ship as > maintainers. Wheel's can be binary releases and, in fact, is what we are > going to build for Windows support going forward. That may certainly > frustrate some automated processes that expect the old package types we used > to ship, but we absolutely want to support binary releases with wheels > specifically to support users that don't have build tools. I understand. I assume you'll continue to supply source distributions. I don't want to dictate, but the situation is what it is. If you've followed this thread, I hope you can appreciate the quandary we're in. > Am I missing something about binary support with wheels? I don't think so. Jim -- Jim Fulton http://www.linkedin.com/in/jimfulton ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Are there any plans to move to pip/wheels in buildout?
On Wed, Dec 3, 2014 at 9:12 AM, Donald Stufft wrote: > >> On Dec 3, 2014, at 9:01 AM, Jim Fulton wrote: >> >> - Implementing and maintaining packaging code isn't really all that >> interesting to me and, I'd hazard to guess, to anyone else on the >> buildout team. Buildout has more packaging code than I ever wanted >> to write. It does largely because it needed policies and features >> not provided by easy_install. (This is not meant to disparage >> easy_install, which was a ground-breaking tool.) > > If you can enumerate what buildout needs or wants from a packaging tool that > would be useful to keep in mind going forward. Great idea. I'll do that. Jim -- Jim Fulton http://www.linkedin.com/in/jimfulton ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Are there any plans to move to pip/wheels in buildout?
On Tue, Dec 2, 2014 at 8:09 AM, Piotr Dobrogost wrote: > On Tue, Dec 2, 2014 at 1:53 PM, Reinout van Rees wrote: >> >> The alternative is, like Marius said in his reply, to use virtualenv + pip + >> makefiles or so. And you'd have to add something that replaces the recipes. >> For that, I really like buildout :-) > > It seems like building/installing python packages should be moved from > buildout's core into plain recipe so that different tools for this > task could be used. That's how buildout works today. There's nothing stopping someone from building a recipe today that uses pip + virtualenv. You might lose some features with this, like caching of built packages but that may not be important to you. It looks like pip has no way to cache builds as buildout does, possibly because it's allergic to eggs. But it does support download caches. Yay! And it should be possible to "cache" builds in wheels, though I think this would still be slower than what buildout does with eggs. But you'd probably gain some things, like a much shorter path to search at run time. Buildout still wants to download packages for it's own use. I suppose buildout could run in it's own virtualenv with recipes installed by pip. It remains that the API to pip would be via subprocess calls or through a hideous distutils command interface. > Even if this is possible I guess it's really major > change (and paradigm shift to some extent). There would be be no change needed to buildout to provide a recipe that built using virtualenv and pip. At some point, somethings going to have to change: - As you noted, Python packaging is changing, and eventually the changes will break buildout. Many users, like myself, download source releases exclusively, and to us, wheels are irrelevent. However, users on Windows or who use systems without build tools for some reason, will experience breakage as soon as people stop creating traditional binary releases in favor of wheels. - Implementing and maintaining packaging code isn't really all that interesting to me and, I'd hazard to guess, to anyone else on the buildout team. Buildout has more packaging code than I ever wanted to write. It does largely because it needed policies and features not provided by easy_install. (This is not meant to disparage easy_install, which was a ground-breaking tool.) Buildout needs to build on something else. For the time being, it makes sense to continue to use setuptools, as this requires very little effort. Eventually, this will be something else and the transition is bound to be bumpy. In the long run, I think keeping buildout alive with a manageble effort is worth transition pain. I think buildout offers a lot of benefit for a lot of projects, and I'd hate to see people use something worse. Jim P.S. I think it's a fallacy to assume that lack of activity or commits (buildout has had both recently) necessarily indicates that a project is dead. Lack of activity often just means that something doesn't require change. I prefer to spread my work over many small projects and it's not uncommon for projects I use every day to go years without change. I realize that activity is an easy metric, especially when looking at projects with which you aren't familiar. I often use it myself, but it's important to keep it's limitations in mind. -- Jim Fulton http://www.linkedin.com/in/jimfulton ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Are there any plans to move to pip/wheels in buildout?
On Tue, Dec 2, 2014 at 7:53 AM, Reinout van Rees wrote: > On 01-12-14 14:55, Piotr Dobrogost wrote: >> >> Are there any plans to move from easy_install/eggs to pip/wheels in >> buildout? > > > I asked that too, once. I think Jim Fulton said that there was a mismatch > between pip's and buildout's goal. If I remember correctly the problem is > this: > > - Pip installs everything into (normally) a virtualenv. So your have ONE > project with ONE single set of packages. > > - Buildout allows different combinations of packages per part. So your > sphinx documentation buildout doesn't have to have your webserver packages > in its sys.path. > > Jim, is that a fair summary? (Of course, this only looks at the python > package part of what buildout does, not at the whole recipe mechanism). Yes, but ... > Something I don't understand here is that there *is* a single version list. Right, although explicit requirements in a part can override the versions list. In practice, most projects use the same versions for everything. In the past, this has been an important requirement for us (ZC). We've even had projects that used different versions of Python for different parts, although ultimately, this "requirement" didn't justify the cost. Something to keep in mind (I know you know this) is that buildout is used to assemble applications with many parts that may be separate applications and may not all be written in Python. > And so it feels like it doesn't really matter that much if you just lump all > the packages together (like pip does). In practice, for most projects this is true. > I could imagine a buildout that just uses pip every time it has to install a > python package. Feeding pip the version list, of course. And then puts the > buildout.cfg mechanism on top of it. I only use versions lists for applications. I almost never pin versions for libraries. Another important goal of buildout is that, given a buildout configuration, two buildouts run at the same time get equivalent results regardless of their ages and histories. For example, I create a buildout on Monday and it gets version 42 of package A. On a different machine, I checkout and update the buildout to add package B. It gets version 9 of package B and version 43 of package A. If I go back to the other machine, and rerun the buildout, I'll get versions 43 and 9 for package A and B. This was impossible to achieve with easy_install, which is what drove me down to setuptools. I don't know how difficult this would be with pip. Given that pip had largely the same goals as easy_install, I'm not optimistic. It's worth looking into. There's also the fact that pip doesn't provide an API other than the command line or the "command" interface which is essentially the same things plus ... distutils. I'm not opposed to trying to create a version of buildout on top of pip+virtualenv, but I don't think that's a very good architecture and I don't really expect it to turn out well. If no standard API emerges, we may have no choice. Maybe we should bet on distlib, but with pip not using it... > The alternative is, like Marius said in his reply, to use virtualenv + pip + > makefiles or so. That sounds like a terrible idea to me. On multiple levels. > And you'd have to add something that replaces the recipes. Well, people use chef and puppet and similar tools to configure docker containers. Then there are Java-based tools, like ant, maven, gradle, sbt... The recipe pattern is well established. > For that, I really like buildout :-) Me too. As long as I'm using Python, I'll maintain buildout if only for my own use. I think recipe-based automation systems make a lot of sense, and I prefer to write recipes, like most other things, in Python. Jim -- Jim Fulton http://www.linkedin.com/in/jimfulton ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Are there any plans to move to pip/wheels in buildout?
On Tue, Dec 2, 2014 at 8:10 AM, Leonardo Rochael Almeida wrote: > > On 2 December 2014 at 10:53, Reinout van Rees wrote: >> >> [...] >> >> >> Something I don't understand here is that there *is* a single version >> list. And so it feels like it doesn't really matter that much if you just >> lump all the packages together (like pip does). > > > FWIW, In the past I did mention that it would be nice if there was a way to > specify a different `[versions]` section per part that would layered on top > of the original `[versions]`. The APIs used could easily accommodate that, > last I checked. I'm still waiting for the round tuits to implement it in a > fork for a pull-request. This works now. If you specify specific versions in a part, they override the versions list. > >> >> I could imagine a buildout that just uses pip every time it has to install >> a python package. Feeding pip the version list, of course. And then puts the >> buildout.cfg mechanism on top of it. >> >> [...] > > > I could imagine a buildout that uses distlib to install wheels in a > per-distribution directory (i.e. allowing multiple versions of the same > wheel to coexist). Yes. Effectively eggs. > These directories would be then added to sys.path in the > script wrappers like buildout usually does. Yup. This is certainly an option. My concern is that no one seems to be using distlib. Jim -- Jim Fulton http://www.linkedin.com/in/jimfulton ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
[Distutils] Standard packaging API? (was Re: Are there any plans to move to pip/wheels in buildout?)
On Mon, Dec 1, 2014 at 8:55 AM, Piotr Dobrogost wrote: > Are there any plans to move from easy_install/eggs to pip/wheels in buildout? Buildout doesn't really use easy_install. It uses setuptools. Originally, I tried to use easy_install directly (and do in some special cases where I shouldn't), but I needed a real API, which is why I moved down to setuptools. My hope is that some new API will emerge to replace setuptools. > I have an impression that buildout project has stagnated I prefer to say it's stable. :) A great sign is that other folks on the project have driven recent work. > which is > unfortunate taking into consideration how much python packaging has > changed recently. Buildout as it, is is entirely dependent on setuptools to add wheel support, at least until a new API emerges. AFAIK, pip doesn't provide an API for use by other tools. I'd be very happy to find out I'm wrong. I hope there's room for more than one command-line tool for working with packages in the ecosystem. It would be crazy for each tool to implement the low-level packaging machinery separately. We need a library to replace setuptools that pip uses and that other tools can use. Jim -- Jim Fulton http://www.linkedin.com/in/jimfulton ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Immutable Files on PyPI
On Sun, Sep 28, 2014 at 3:31 PM, Donald Stufft wrote: > Hello All! > > I'd like to discuss the idea of moving PyPI to having immutable files. ... +1 Jim -- Jim Fulton http://www.linkedin.com/in/jimfulton ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Buildout setuid
On Wed, Sep 17, 2014 at 1:47 PM, Lennart Regebro wrote: > While writing a blog post about software configuration management I looked > into buildout, and using it as an SCM tool. And it has one big restriction: > > You can't run certain parts as root. > > I think adding that would actually not be too hard. Are there any principal > arguments against it? I looked at making an extension, but I would need a > hook that is run before and after each step in that case. > > I was thinking that you could define which parts should run as root in one > of two ways: > > 1. A parameter in the part config > 2. Having a global configuration with a list of parts. This for the case > when the parts recipe itself has a parameter that clashes with the parameter > in 1. > > I'm leaning towards having a setuid parameter, so you can set to other id's > than 0. > > Technically it would be done by setuid to root for the configured parts, and > then back after it has run. You would have to run buildout as a whole with > sudo for this to work. It would use the login name as the "normal" setuid, > unless configured explicitly with a global setuid parameter. > > Thoughts? We deploy to production with buildout and have never needed this. Our approach is to have separate buildouts for building software (RPMs currently) and for deploying to production machines. The deployment buildouts are run as root (typically from another process that runs from root, https://bitbucket.org/zc/zkdeployment). These 2 buildouts are run at very different times and situations. A better approach IMO is to deploy with Docker. With Docker, all of your "deployment" is done when you build an image, still as root. Unfortunately, our current scheme is working well enough and we have enough other priorities that I fear I won't find time to dockerfy our processes soon. Jim -- Jim Fulton http://www.linkedin.com/in/jimfulton ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Download error on (…) hostname doesn't match either of '*.c.ssl.fastly.net', (…) when running buildout behind proxy
On Mon, Sep 8, 2014 at 11:31 AM, Piotr Dobrogost wrote: > On Mon, Sep 8, 2014 at 3:46 PM, Jim Fulton wrote: >> >> This is why I *always* use a clean python built from source. >> I recommend people do the same, or use a virtualenv. > > Well, I read http://www.buildout.org/en/latest/docs/tutorial.html but > nowhere there was virtualenv mentioned. I guess this might be due to > the fact virtualenv didn't even exist in 2007 or was not popular? :) I > somehow had an impression that one of the main features of buildout > has always been isolation from system Python thus I did not even > thought about running it inside virtualenv. However, looking now for > "virtualenv" at https://pypi.python.org/pypi/zc.buildout/2.2.1 brought > the following bullet from version 2.0.0: > > "Buildout no-longer tries to provide full or partial isolation from > system Python installations. If you want isolation, use buildout with > virtualenv, or use a clean build of Python to begin with." > > which is quite clear. > > Btw, what's the reason documentation at > http://www.buildout.org/en/latest/docs/index.html is for ancient > version 1.2.1? Whimper. Sadly, that's the most recent version independent of the doctests. Jim -- Jim Fulton http://www.linkedin.com/in/jimfulton ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Download error on (…) hostname doesn't match either of '*.c.ssl.fastly.net', (…) when running buildout behind proxy
On Mon, Sep 8, 2014 at 7:44 AM, Piotr Dobrogost wrote: > Jim, thanks for taking time to reply. > > On Sun, Sep 7, 2014 at 4:09 PM, Jim Fulton wrote: >> >> Wow. That's really old and not really supported any more. >> >> It also uses very old and unsupported versions of setuptools. >> > (...) >> >> Buildout uses setuptools, which is what easy_install uses. (Buildout >> originally >> used easy_install more or less directly and still does in some narrow cases.) >> >> Please upgrade to buildout 2. > > Ok, I tried with current bootstrap.py from > http://downloads.buildout.org/2/bootstrap.py and got the same error. > This time when running bootstrap itself (not buildout): > > pdobrogost@host:~/projects/ > projectx/projectx_buildout$ python bootstrap.py > Downloading > https://pypi.python.org/packages/source/s/setuptools/setuptools-5.7.zip > Extracting in /tmp/tmpj6ZiDN > Now working in /tmp/tmpj6ZiDN/setuptools-5.7 > Building a Setuptools egg in /tmp/tmpF9r1g6 > /tmp/tmpF9r1g6/setuptools-5.7-py2.7.egg > Download error on https://pypi.python.org/simple/zc.buildout/: > hostname 'proxy.site.local' doesn't match either of > '*.c.ssl.fastly.net', 'c.ssl.fastly.net', '*.target.com', '*.vhx.tv', > '*.snappytv.com', '*.atlassian.net', 'secure.lessthan3.com', > '*.atlassian.com', 'a.sellpoint.net', 'cdn.upthere.com', > '*.tissuu.com', '*.issuu.com', '*.kekofan.com', '*.python.org', > '*.theverge.com', '*.sbnation.com', '*.polygon.com', > '*.twobrightlights.com', '*.2brightlights.info', '*.vox.com', > 'staging-cdn.upthere.com', '*.zeebox.com', '*.beamly.com', > '*.aticpan.org', 'stream.svc.7digital.net', > 'stream-test.svc.7digital.net', '*.articulate.com', 's.t.st', > 'vid.thestreet.com', '*.planet-labs.com', '*.url2png.com', 'turn.com', > 'www.turn.com', 'rivergathering.org', > 'social.icfglobal2014-europe.org', '*.innogamescdn.com', > '*.pathable.com', '*.staging.pathable.com', '*.kickstarter.com', > 'sparkingchange.org', 'www.swedavia.se', 'www.swedavia.com', > 'js-agent.newrelic.com', '*.fastly-streams.com', 'cdn.brandisty.com', > 'fastly.hightailcdn.com', '*.fl.yelpcdn.com', '*.feedmagnet.com', > 'api.contentbody.com', '*.acquia.com', '*.swarmapp.com', '*.pypa.io', > 'pypa.io', 'static.qbranch.se', '*.krxd.net', '*.room.co', > '*.metrological.com', 'room.co', 'cdn.evbuc.com', 'cdn.adagility.com', > '*.bandpage.com', '*.ibmserviceengage.com', '*.quirky.com', > '*.veez.co', '*.x.io', '*.otoycdn.net', '*.scribd.com', > 'www.dwin1.com', 'api.imgur-ysports.com', 'i.imgur-ysports.com', > '*.fxcm.co.jp', 'listora.com', '*.listora.com', 'blendle.nl', > '*.blendle.nl', '*.modeanalytics.com', 'modeanalytics.com', > 'krux.com', '*.krux.com', '*.udemy.com', '*.1stdibs.com', > 'api.keep.com', 'www.piriform.com', '*.ustream.tv', 'www.zimbio.com', > 'm.zimbio.com', 'www.stylebistro.com', 'm.stylebistro.com', > 'm.lonny.com', 'www.lonny.com', 'assets.trabiancdn.com', > '*.socialchorus.com', '*.heritagestatic.com', '*.theoutbound.com', > 'img.rakuten.com', 'images.rakuten.com', 'img1.r10.io', 'ast1.r10.io', > 'scribd.com' -- Some packages may not be found! > Couldn't find index page for 'zc.buildout' (maybe misspelled?) > Download error on https://pypi.python.org/simple/: hostname > 'proxy.site.local' doesn't match either of '*.c.ssl.fastly.net', > 'c.ssl.fastly.net', '*.target.com', '*.vhx.tv', '*.snappytv.com', > '*.atlassian.net', 'secure.lessthan3.com', '*.atlassian.com', > 'a.sellpoint.net', 'cdn.upthere.com', '*.tissuu.com', '*.issuu.com', > '*.kekofan.com', '*.python.org', '*.theverge.com', '*.sbnation.com', &
Re: [Distutils] Download error on (…) hostname doesn't match either of '*.c.ssl.fastly.net', (…) when running buildout behind proxy
On Fri, Sep 5, 2014 at 8:38 AM, Piotr Dobrogost wrote: > Hi! > > According to http://www.buildout.org/en/latest/community.html this > mailing list is the right place to ask about buildout Yup. > so here is my > question. > > I'm having hard time finding out why buildout doesn't work with my proxy. > Here is the error I get: > > pdobrogost@host:~/projects/projectx/projectx_buildout$ ./bin/buildout > -vNc buildout-devel.cfg custom:cvsuser=pdobrogost > Installing 'mr.developer'. > We have no distributions for mr.developer that satisfies 'mr.developer'. > Download error on http://pypi.python.org/simple/mr.developer/: > hostname 'proxy.site.local' doesn't match either of > '*.c.ssl.fastly.net', 'c.ssl.fastly. (...) > > I'm using Buildout 1.4.3 Wow. That's really old and not really supported any more. It also uses very old and unsupported versions of setuptools. > with Python 2.7.7 on Debian 6.0.10 > Details can be found at http://stackoverflow.com/q/25682165/95735 > > Btw, does buildout use easy_install to download/build/install eggs or > something else? If it uses easy_install is there any way to pass > options to easy_install used? Buildout uses setuptools, which is what easy_install uses. (Buildout originally used easy_install more or less directly and still does in some narrow cases.) Please upgrade to buildout 2. Jim -- Jim Fulton http://www.linkedin.com/in/jimfulton ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] zc.buildout & Docker container images
On Thu, Jul 3, 2014 at 11:36 AM, Michael Merickel wrote: > I don't normally like shameless plugs but I haven't seen many people use > docker in the following way yet and I happen to be using it with > zc.buildout. > > I have built a small tool called marina[1] that I'm using to build binary > slugs that can be installed in production. It uses throwaway (clean-room) > containers to compile the application (using buildout or whatever else), and > then dumps out an archive (or tags a new docker image) of the built assets > which can be deployed. Marina is not using dockerfile's explicitly but > rather you make a parent image (with buildpacks or whatever else) however > you like. The image is then executed with your scripts (such as buildout) > and the assets are archived. The approach keeps ssh keys and credentials out > of the images. A separate data-only container is also mounted to allow > caching of assets (eggs folder?) between runs which speeds up the build > process dramatically. I'm using it on OS X to build binary tarballs for > Ubuntu. The tarballs themselves can then be deployed as a docker image or > just directly on a VM. It's all very alpha but I'm using it successfully > with ansible to distribute tarballs to VMs in production. > > [1] https://github.com/mmerickel/marina Not sure I follow all the above, but it sounds interesting. I look forward to studying it. Jim -- Jim Fulton http://www.linkedin.com/in/jimfulton ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] zc.buildout & Docker container images
On Thu, Jul 3, 2014 at 6:24 AM, Reinout van Rees wrote: > On 30-06-14 17:56, Nick Coghlan wrote: >> >> Yeah, it's the "you still need a way to define what goes into the image" >> part that intrigues me with respect to combining tools like zc.buildout >> with Docker. > > > Buildout, to me, solves all there is to solve regarding python packages and > a bit of configuration. Including calling bower to go grab the necessary > css/js :-) > > That css/js is quite an important part of "what goes into the image". Bower > with it's dependency mechanism solves that (and it can be called from > buildout). > > A third important one is system packages: "what do I apt-get install". If you're building system packages, then you can use their dependency system. If you're using docker images, you can put this in a base layer. > **Question/idea**: what about some mechanism to get this apt-get information > out of a python package? If a site or package absolutely requires gdal or > redis or memcache, it feels natural to me to have that knowledge somewhere > in the python package. > > Does anyone do something like this? I was thinking along the lines of a > simple 'debian_dependencies.txt' that I could use as input for > ansible/fabric/fpm/whatever. Depending exactly on what "this" is, Fred Drake's package grinder, for building RPMs, might come close: https://bitbucket.org/zc/packagegrinder I'm really hoping that Docker will eventually allow us to stop building system packages though. Jim -- Jim Fulton http://www.linkedin.com/in/jimfulton ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] zc.buildout & Docker container images
On Sun, Jun 29, 2014 at 12:40 AM, Nick Coghlan wrote: > Hey all (but especially the zc.buildout users), > > With zc.buildout being used to define repeatable application > deployments, and Docker images acting as a way to snapshot entire > execution environments for ease of deployment on any environment than > can run a container (which is basically any modern Linux at this > point, or will be once CentOS 7 is available), has any experimented > with using zc.buildout from a Dockerfile to define an image? Yes. > If yes, is it written up anywhere? No. :) A challenge for us has been that our apps typically register themselves with ZooKeeper and, until recently, this has been really awkward with Docker. Now that you can share host networking with containers, I look forward to making more use of Docker in the future. For now, we've packaged a number of legacy applications, including svn.zope.org, as docker containers. > If not, is anyone intrigued enough by the idea to considering trying > it out and writing up the results as a blog post somewhere? Absolutely, although I'm not sure when. In some ways, docker potentially reduces the need for tools like buildout and virtualenv to provide isolation. With a docker image, you don't share the system Python with other apps, so it's not as problematic to use it. You also don't need tools to automate deployment of production configurations when an application is deployed, as this is mostly done when building an image. The isolation provided by docker containers also allows configuration to be simpler. There's still benefit of having a system like buildout with Python recipes to automate assembly (often including non-python bits like JavaScript libraries) and configuration. Jim -- Jim Fulton http://www.linkedin.com/in/jimfulton ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Packaging buildout-based project for release
On Wed, May 28, 2014 at 1:58 PM, Travis Jensen wrote: > Hi, all, > > I'm hoping for some advice. I've got a Django web app that I've used > buildout to build. It includes effectively three source trees (a Celery task > tree, a set of re-usable Django database models, and the actual Django web > application project. Yes, these could all be different repos, and if they > should be, I will make them such, it just makes early development much > easier. :) Each of them has their own setup.py since they each have their > own deployment story (celery tasks on worker systems, shared models just > about anywhere, and the web app on the web tier). buildout's > collective.recipe.template and collective.recipe.cmd are also being used to > install node modules (lessc, coffeescript, and yuglify). > > So, one option for deploying is "git clone myproj.git && cd myproj && > buildout". I'm not a fan for a variety of reasons: it is *slow*; it is not > necessarily deterministic (even with version pinning on our own simple > index, somebody could replace a file); and it requires additional software > to be installed on production systems that I'd rather not be installed. An > ideal world would solve all of them. but I could live with the (unlikely) > non-determinism. An option you should consider is doing this in a Docker container and deploying with Docker. (We don't do this now due to some Docker networking limitations that appear to have gone away lately.) > I'm really happy with the automation of the development environment that > buildout gives. The question is whether I can take that and turn it into a > deployable system or if I should be looking somewhere else. An ideal world > would be three RPMs (and a "meta"-RPM that installed all three) with all the > Python and Node dependencies built in. I wrote zc.sourcerelease to help solve this problem. Here's a script that we use to build RPMs from buildouts in git: https://gist.github.com/jimfulton/6629791 Some notes: - The script is run from a git checkout. - The buildout-source-release script needs to be provided by the buildout. - You need to provide a RPM spec file. It can be pretty generic. Here's an example project that uses it: https://bitbucket.org/zc/zrs-rpm/src Note the buildout.cfg and spec file. Also note that my philosophy is that RPMs should contain software, not configuration. We configure processes and such separately from building RPMs. We use buildout to build run-time configurations. :) Of course, Docker, potentially, makes this a *lot* simpler. You can do most or all of your configuration at build time. Jim -- Jim Fulton http://www.linkedin.com/in/jimfulton ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
[Distutils] [buildout] New Buildout 2 bootstrap script
I've posted a new buildout 2 and 2.1 bootstrap script: http://downloads.buildout.org/2/bootstrap.py That has the new location of the ez_setup.py script. Jim -- Jim Fulton http://www.linkedin.com/in/jimfulton ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] [buildout] HTTP Error 503 downloading https://bitbucket.org/pypa/setuptools/raw/0.7.2/ez_setup.py
ez_setup.py has moved to https://bootstrap.pypa.io/ez_setup.py
I just updated the buildout 2 (and 2.1) bootstrap file to use the new location.
Sorry, I should have done that sooner.
Jim
On Tue, May 6, 2014 at 5:09 PM, Chris Withers wrote:
> Hi All,
>
> Seeing lots of Jenkins jobs intermittently failing with this:
>
> Traceback (most recent call last):
> File "bootstrap.py", line 79, in
>
> exec(urlopen('https://bitbucket.org/pypa/setuptools/raw/0.7.2/ez_setup.py'
> File "/usr/local/lib/python2.7/urllib2.py", line 126, in urlopen
> return _opener.open(url, data, timeout)
> File "/usr/local/lib/python2.7/urllib2.py", line 400, in open
> response = meth(req, response)
> File "/usr/local/lib/python2.7/urllib2.py", line 513, in http_response
> 'http', request, response, code, msg, hdrs)
> File "/usr/local/lib/python2.7/urllib2.py", line 438, in error
> return self._call_chain(*args)
> File "/usr/local/lib/python2.7/urllib2.py", line 372, in _call_chain
> result = func(*args)
> File "/usr/local/lib/python2.7/urllib2.py", line 521, in
> http_error_default
> raise HTTPError(req.get_full_url(), code, msg, hdrs, fp)
> urllib2.HTTPError: HTTP Error 503: SERVICE UNAVAILABLE
>
> Anyone else seen similar?
>
> Chris
>
> --
> Simplistix - Content Management, Batch Processing & Python Consulting
>- http://www.simplistix.co.uk
> ___
> Distutils-SIG maillist - Distutils-SIG@python.org
> https://mail.python.org/mailman/listinfo/distutils-sig
--
Jim Fulton
http://www.linkedin.com/in/jimfulton
___
Distutils-SIG maillist - Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] buildout prepends eggs to sys.path, performance issues
On Tue, Apr 29, 2014 at 7:33 AM, Jan Van Hees wrote: > Hi list, > > > > A bit of context: we’re running buildout 2.0.1, building zope and plone > applications, zope 2.13.10, combined with plone.recipe.zope2instance. This > question only impacts the performance of the startup of an instance, the the > performance of a running instance. But starting an instance is something > developpers do quite often on a days development. > > > > While debugging some instance startup performance issues, I came across the > following. > > The buildout Scripts prepends all the eggs to the system path, before the > python path. > > > > In our setup this causes quite some delays, because imports from standard > python modules, also try to find that module in every eggs directory, before > it can find in in the default python location (because the eggs are > prepended). ... You make a good performance argument. > As far as I’ve always understood, the default procedure working with paths > should be to append, unless you have a good reason. The good reason in this > case, that I see, could be that you want to prepend certain packages and > that way make sure you use your version instead of what’s present in > site-packages. Yes, exactly. > Now my expectation would be that that is a fairly limited set > of packages that need to be prepended. But buildout can't know what they are. > If there are many, options like > virtualenv exist to avoid taking site-packages at all (that’s what we do > btw). Buildout doesn't require virtualenv. I've always prepended appliocation paths for predicability. I'm not interesated in changing this behavior of buildout, at least not as a default. I'll give this some thought, but some possible ideas: - Provide an option to install the eggs for a part in a single directory. collective.recipe.omlette does this, although it doesn't seem to be used for run-time package lookup afaict from the docs. You might try using this. - Provide an option to append, rather than prepend eggs. Jim -- Jim Fulton http://www.linkedin.com/in/jimfulton ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Enterprise Python - Multicomponent Projects
On Wed, Apr 16, 2014 at 12:18 PM, Tin Tvrtković wrote: > Hello packaging community, > > I'm investigating ways of setting up Python projects at my workplace. We're > predominantly a Java shop, but we might be dipping our toes in Python waters > (finally!) due to a fortuitous project and my multi-year insistence, so I'm > contemplating how to set up our Python build system to minimize workflow > differences for other developers (well, and myself). > > I've actually written uš a lengthy description of Maven and why we use it > but I'll spare you for now. :) To keep the story short, I'm interested in > options for setting up a multi-module Python project. By 'multi-module' I > don't mean a single setuptools-based project with several .py files inside, > but a way of triggering a complex build process with a single command that > would build all sub-modules (essentially sub-projects) and produce a number > of end artifacts - just like Maven. Imagine a repository containing 30 > separate Django apps, packaged independently, 10 utility libraries, 10 > Django projects combining those app, and 10 RPM building projects for > packaging it all up for deployment. > > As far as I know, just using setuptools isn't adequate for a workflow like > this - setuptools deals with the build process (testing, packaging, etc) of > a single project only. Solutions that come to mind are: a hierarchy of > Makefiles, shell scripts, or maybe Twitter's Pants, which sort of looks like > Maven for Python but would probably need contributions to do what we want, > and looks predisposed to building PEX files which, while very interesting, > I'm not looking to do right now. None of these solutions are really ideal, > especially if I want to support development on Windows (which I absolutely > want). > > I've even thought about actually using Maven, but that's just a Pandora's > box of problems waiting to happen. > > I'd appreciate insight on this from anyone who's thought about (and maybe > solved) problems like this. I'm also willing to engage and contribute to > improving the situation, especially if there's low hanging fruit to be > picked. How do other companies handle large Python repositories with a lot > of subcomponents? Checkout Buildout, http://www.buildout.org It addresses a lot of the same problems. Of course, for Python, compiling is (mostly) not all that important. For us (Zope Corporation) buildout is more about assembly/deployment, both for development and production, that building. Jim -- Jim Fulton http://www.linkedin.com/in/jimfulton ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Is there any sense to put setuptools as a requirement?
On Thu, Feb 27, 2014 at 6:49 AM, Piotr Dobrogost
wrote:
> Hi!
>
> I've seen people putting 'setuptools' in 'install_requires' in
> setup.py starting with import from setuptools like this:
> from setuptools import setup, find_packages
>
> Does it make any sense?
> In what circumstances should 'setuptools' be listed in
> 'install_requires' or 'setup_requires'?
It makes sense when you use setuptools to implement namespace packages.
So, for example in zope packages, the __init__.py in the zope namespace
package typically has:
import pkg_resources
pkg_resources.declare_namespace(__name__)
Jim
--
Jim Fulton
http://www.linkedin.com/in/jimfulton
___
Distutils-SIG maillist - Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] buildout bootstrap question
On Tue, Feb 18, 2014 at 6:41 AM, Eric V. Smith wrote: > On 2/17/2014 6:13 AM, Jim Fulton wrote: >> On Sun, Feb 16, 2014 at 10:46 PM, Eric V. Smith wrote: >>> In older versions of bootstrap.py, the --setup-source option would let >>> me select an alternate source of ez_setup.py. I just downloaded a new >>> version today from http://downloads.buildout.org/2/bootstrap.py, and I >>> notice this option is gone. bootstrap always will download ez_setup from >>> http://downloads.buildout.org/2/bootstrap.py (assuming setuptools isn't >>> installed, which in my case it is not). >>> >>> I run buildout on machines that have access to my own internal http >>> servers, but do not have internet access. The requirement for internet >>> access breaks this setup. >>> >>> Is there some reason --setup-source was removed? >> >> I didn't know people needed it and was trying to reduce the complexity >> of the bootstrap source. We can add it back, > > While working up a patch for this, Thanks! > I realize that I'll also need to add > back --download-base to pass to ez_setup.use_setuptools. Yup. Feel free to propose something simpler. > Is the best way to do this to open an issue on > https://github.com/buildout/buildout, and create a fork with my proposed > changes? Yes. > I'm not sure of the normal buildout workflow. It's pretty standard open source workflow based on pull requests. For a non-trivial change, you'd need to sign a Zope contributor agreement, but I'd be happy to consider this trivial. :) Jim -- Jim Fulton http://www.linkedin.com/in/jimfulton ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] buildout bootstrap question
On Sun, Feb 16, 2014 at 10:46 PM, Eric V. Smith wrote: > In older versions of bootstrap.py, the --setup-source option would let > me select an alternate source of ez_setup.py. I just downloaded a new > version today from http://downloads.buildout.org/2/bootstrap.py, and I > notice this option is gone. bootstrap always will download ez_setup from > http://downloads.buildout.org/2/bootstrap.py (assuming setuptools isn't > installed, which in my case it is not). > > I run buildout on machines that have access to my own internal http > servers, but do not have internet access. The requirement for internet > access breaks this setup. > > Is there some reason --setup-source was removed? I didn't know people needed it and was trying to reduce the complexity of the bootstrap source. We can add it back, Jim -- Jim Fulton http://www.linkedin.com/in/jimfulton ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Buildout index URL override
On Sun, Jan 5, 2014 at 6:31 AM, Christian Ullrich wrote: > Hello all, > > does buildout ever take its default index URL from the distutils > configuration, i.e. [py]distutils.cfg? > No. It has it's own configuration for the index url ... which sadly isn't documented. Weird. You can control the index URL using the buildout index option. Jim -- Jim Fulton http://www.linkedin.com/in/jimfulton ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Isolation, virtualenvs and buildout
On Mon, Nov 11, 2013 at 10:26 AM, Achim Domma wrote: > Hi, > > I'm used to virtualenvs and I think I understood the general idea of build > out. I need to build a pice of software out of several independent Python > packages and I think buildout is the tool to use. But I don't get how > buildout and virtualenv are related to each other. Asking Google, I found > some hints that buildout is supposed to provide the same kind of isolation as > virtualenvs - but only in the "old" 1.x version. The "new"(?) 2.0 version > will change that. So I'm totally confused and ask for help. Buildout1 tried to provide isolation, but it was incomplete and too hard to maintain. virtualenv works crazy hard to provide isolation. (/me applauds virtualenv.) Many people run buildout inside virtualenvs. Personally, I always keep a unadulterated Python around that I run buildout with. Buildouts main contribution is that it doesn't install things into site packages and won't mess up your clean Python or your virtualenv. You could use virtualenv to make yourself a clean Python and run any number of buildouts with that virtualenv. > My use case is simple: I have some python modules on our private pypi server > and would like to deploy a pice of software composed out of them (including > all dependencies). That's similar to our use cases. > Any starting guidance would be very appreciated! FWIW, here's a script I use to make RPMs out of buildouts: https://gist.github.com/jimfulton/6629791 It builds on zc.sourcerelease, which builds self-contained source tar balls. We (ZC) deploy *software* using RPM and use other buildout based tools to deploy application configuration. I suspect in the future we'll to docker or maybe something simpler than RPM, which doesn't really match our use cases very well and tends to cause us no end of pain. Jim -- Jim Fulton http://www.linkedin.com/in/jimfulton ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Deprecate and Block requires/provides
On Thu, Oct 17, 2013 at 11:26 AM, Donald Stufft wrote: > > On Oct 17, 2013, at 11:23 AM, Jim Fulton wrote: > >> On Thu, Oct 17, 2013 at 11:20 AM, Michael Foord wrote: >>> >>> >>> >>> On 17 October 2013 11:56, Donald Stufft wrote: >>>> >>>> Arguably it's already broken. I've had to explain to a number of people >>>> that it won't cause their dependencies to install. I think its way more >>>> user >>>> friendly to tell them up front then to confuse them when it doesn't work or >>>> when it appears to work but they get an error from a - >>>> >>> >>> You're proposing replacing "arguably broken" (by some definition) >> >> Is there any reason to think that it ever actually worked in any way? > > Depends what you mean by "worked", it does nothing except tell users > what they need available to ``import``. It doesn't tell them how to get that > or what provides it. As far as this piece of metadata is concerned requires > "xml" can be satisfied by ``touch xml.py``. > > So it "works" in that it does what it was originally intended to do, it's > just that > what it was originally intended to do is backwards and useless. I'm 92% sure it was intended to support automated dependency management, but then setuptools went in a different (better) direction. Jim -- Jim Fulton http://www.linkedin.com/in/jimfulton ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Deprecate and Block requires/provides
On Thu, Oct 17, 2013 at 11:20 AM, Michael Foord wrote: > > > > On 17 October 2013 11:56, Donald Stufft wrote: >> >> Arguably it's already broken. I've had to explain to a number of people >> that it won't cause their dependencies to install. I think its way more user >> friendly to tell them up front then to confuse them when it doesn't work or >> when it appears to work but they get an error from a - >> > > You're proposing replacing "arguably broken" (by some definition) Is there any reason to think that it ever actually worked in any way? Jim -- Jim Fulton http://www.linkedin.com/in/jimfulton ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Deprecate and Block requires/provides
On Wed, Oct 16, 2013 at 11:01 PM, Donald Stufft wrote: > So what I would like to do is "remove" these fields. This would consist > of modifying PyPI to return an error code if they are included and hiding > the existing data in the UX. It might at a future time also consist of > removing > the data from the DB all together. > > What do people think? IIUC, you're proposing to error on uploads of distributions with these fields set. This wouldn't effect distributions already uploaded and wouldn't cause (new) breakage for consumers of those distributions. The only breakage would be for authors uploading the buggy distributions. These are the people who could actually address the breakage and who would benefit from the breakage by finding out that they have a bug in their distributions. This seems an appropriate form of breakage to me, so +1. Jim -- Jim Fulton http://www.linkedin.com/in/jimfulton ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] zc.buildout bootstrap.py v1 died / python-distribute.org down
On Tue, Oct 15, 2013 at 10:27 AM, Adam GROSZER wrote: > On 10/15/2013 03:38 PM, Jim Fulton wrote: >> >> On Tue, Oct 15, 2013 at 3:43 AM, Adam GROSZER >> wrote: >>> >>> On 10/15/2013 12:14 AM, Alex Clark wrote: >>>> >>>> >>>> Adam GROSZER gmail.com> writes: >>>> >>>> >>>>> python-distribute.org seems to be down, that breaks zc.buildout v1 >>>>> bootstrap.py when used with "-d" to use distribute. >>>> >>>> >>>> >>>> >>>> Distribute is dead, long live Setuptools (don't use -d anymore) >>> >>> >>> >>> Well easier said than done. >>> This is windows, 64bit >> >> >> Does setuptools not work with 64-bit windows? > > > 0.6c11 does not work There are quite a few 0.6c12 releases that were never uploaded to PyPI. Perhaps one of those works. I'd be happy to provide the eggs I have if someone wants to try them out. Maybe PJE knows of they work on 64-bit windows. > > >> >>> On top of that http://downloads.buildout.org/1/bootstrap.py references >>> that >>> URL. >> >> >> Is there a different URL that should be used? Or do you suggest dropping >> distribute support from the buildout 1 bootstrap script? >> >> Jim >> > > This breaks current projects. > We don't have time right now to update to buildout 2.x. > A workaround is to patch the v1 bootstrap with: > > distribute_source = > 'https://bitbucket.org/pypa/setuptools/raw/f657df1f1ed46596d236376649c99a470662b4ba/distribute_setup.py' > > Maybe that should get into http://downloads.buildout.org/1/bootstrap.py OK, I'll work on that. Jim -- Jim Fulton http://www.linkedin.com/in/jimfulton ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] zc.buildout bootstrap.py v1 died / python-distribute.org down
On Tue, Oct 15, 2013 at 3:43 AM, Adam GROSZER wrote: > On 10/15/2013 12:14 AM, Alex Clark wrote: >> >> Adam GROSZER gmail.com> writes: >> >> >>> python-distribute.org seems to be down, that breaks zc.buildout v1 >>> bootstrap.py when used with "-d" to use distribute. >> >> >> >> Distribute is dead, long live Setuptools (don't use -d anymore) > > > Well easier said than done. > This is windows, 64bit Does setuptools not work with 64-bit windows? > On top of that http://downloads.buildout.org/1/bootstrap.py references that > URL. Is there a different URL that should be used? Or do you suggest dropping distribute support from the buildout 1 bootstrap script? Jim -- Jim Fulton http://www.linkedin.com/in/jimfulton ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Deviation between distutils and setuptools over package_data
On Fri, Sep 20, 2013 at 6:41 AM, Marius Gedminas wrote: > On Fri, Sep 20, 2013 at 04:43:23PM +1000, Nick Coghlan wrote: >> On 20 September 2013 00:02, Benjamin Root wrote: >> > I will point out that even with setuptools 1.1.6, sdist isn't picking up >> > all >> > the files in version control, as I have a few other files under version >> > control in my package that I didn't list for package_data. So, I still >> > think >> > there is an issue with crawling an SVN 1.7 repository. >> >> That part I have no idea about :) > > I would recommend not relying on setuptools version control magic. > Instead write a MANIFEST.in. It appears that MANIFEST.in is ignored if setuptools recognizes your VCS. Although there's so much magic in this particular facet of setuptools that I never have much confidence in my understanding. Did setuptools recently learn about git? For a while, an advantage of git was that setuptools ignored it. I would love an option to ignore VCS and let me specify what I want *explicitly*. Jim -- Jim Fulton http://www.linkedin.com/in/jimfulton ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] "Please use a mix of different-case letters and numbers in your password"
On Wed, Sep 4, 2013 at 9:08 AM, Antoine Pitrou wrote: > Jim Fulton zope.com> writes: >> >> Some password managers (including both that I've used) let you access your >> passwords via the web, so they aren't stored locally. > > Will they work with setup.py too (e.g. the register command)? They will work with anything that requires typing, as you can copy and paste (or even view and transcribe) a password from a web interface. Jim ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] "Please use a mix of different-case letters and numbers in your password"
On Wed, Sep 4, 2013 at 8:11 AM, Antoine Pitrou wrote: > Jim Fulton zope.com> writes: >> >> People (at least technical people) should use password managers. > > I will gladly use a password manager on my personal computer, just *not* > on a computer which other people may access. In these cases it is important > to be able to choose a rememberable enough password, precisely because I > don't want the computer to store it. Some password managers (including both that I've used) let you access your passwords via the web, so they aren't stored locally. Jim -- Jim Fulton http://www.linkedin.com/in/jimfulton ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] "Please use a mix of different-case letters and numbers in your password"
On Wed, Sep 4, 2013 at 6:33 AM, Antoine Pitrou wrote: > Donald Stufft stufft.io> writes: >> >> On Sep 4, 2013, at 4:27 AM, Antoine Pitrou python.org> wrote: >> >> > >> > Hi, >> > >> > On PyPI: >> > "Please use a mix of different-case letters and numbers in your password" >> > >> > Ok... has anyone decided to play BOFH on this one? >> > >> > Displaying recommendations is fine (and, why not, some kind of entropy >> > meter), enforcing stupid rules like that is not. >> > >> > Regards >> > >> > Antoine, trying to access his PyPI account... >> > >> > >> > ___ >> > Distutils-SIG maillist - Distutils-SIG python.org >> > https://mail.python.org/mailman/listinfo/distutils-sig >> >> Use a better password, > > Ok, let me try to explain this, despite the fact that I would have > preferred not to lose time with this: > > Users don't want their security concerns to be dictated by a service > provider. Programmatically refusing passwords which are deemed "too > weak" is the kind of policy that I thought had disappeared since the 1990s > (yes, it's been tried before, like other stupid requirements such as > having to change passwords every month). > > Mandating that users choose hard-to-remember passwords only leads to them > writing down those passwords on post-it stickers (or send themselves > clear-text reminder e-mais, etc.). It's counter-productive in addition > to being an annoyance when trying to do real work. > > I think it would be beneficial if you changed your attitude a bit here. > Caring about security is good. Mandating that other people follow > *your* security principles when dealing with *their* data is obnoxious > (and here the accent is really on "mandating"; it's fine to give advice). People (at least technical people) should use password managers. What annoys me is when a 40-character random password is rejected because it doesn't contain a number (or a capitalized character letter or whatever), when the same system would accept a 7-character password. (It's easy enough to add the missing bits to the password, which makes it merely annoying, but It also makes me think the system is sorta stupid.) Jim -- Jim Fulton http://www.linkedin.com/in/jimfulton ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Multi-version import support for wheel files
On Sun, Aug 25, 2013 at 5:21 PM, Nick Coghlan wrote: > > On 26 Aug 2013 07:00, "Donald Stufft" wrote: >> >> >> On Aug 25, 2013, at 4:51 PM, Nick Coghlan wrote: >> >>> Anyway, I like Paul's suggestion of defining a specific runtime format >>> for this, even if it's just "wheel layout plus a RECORD file". I'm currently >>> thinking of using the ".dist" suffix, matching the existing egg vs egg-info >>> naming convention. >> >> >> It seems to me the easiest thing to do is just continue using eggs for >> this feature for now especially if the proposal is just standardizing what >> eggs do and doesn't offer any benefits besides standardization. That gets >> you all the benefits sans standardization and doesn't spend time putting a >> PEP through (and all the back and forth that entails) for something that >> already works when we can spend the time on stuff that still needs actual >> design work. > > Egg based multi-version installs still suffer from the problem of lacking a > RECORD file so you need an external tool to manage them properly. Well, I'd argue that eggs are effectively also records. You can find out what's installed by simply looking at the names in whatever directory you put eggs. The harder part, of course, is deciding when an egg is no longer needed. I assume the RECORD file doesn't address that either. Note that with multi-version support, uninstalling things is an optimization, not a necessity. The only harm a never-uninstalled egg does is take up space and maybe make tools that scan for what's installed take more time. Jim -- Jim Fulton http://www.linkedin.com/in/jimfulton ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Multi-version import support for wheel files
On Sun, Aug 25, 2013 at 3:06 AM, Nick Coghlan wrote: > On 25 August 2013 16:41, Donald Stufft wrote: >> >> On Aug 25, 2013, at 1:57 AM, Nick Coghlan wrote: >> >>> [snip] >> >> I'll look at this closer, my off the cuff response isn't good but before I >> commit to a side I want to dig into how it actually works currently. > > The clumsiness of the __main__.__requires__ workaround aside, the main > advantage this offers is that it *should* result in a relatively > straightforward addition to pkg_resources to make it work with wheel > files as well as eggs. That's important, because anyone that is > currently doing side-by-side multi-versioning in Python is using the > pkg_resources API to do it, since that's the only option currently > available. No. It isn't. Buildout doesn't use pks_resources to do it. (Buildout used pkg_resources at build time to manage package meta data, but I think that's orthogonal to what you're talking about.) I'd also hazard to guess that most of the folks with multi-version installs are using buildout to do it, as buildout does have a fair number of users. Jim -- Jim Fulton http://www.linkedin.com/in/jimfulton ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Multi-version import support for wheel files
On Sun, Aug 25, 2013 at 1:57 AM, Nick Coghlan wrote: > I'm currently working on the docs for the __main__.__requires__ > feature of pkg_resources, and have been generally poking around inside > pkg_resources before I started on that. It gave me an idea for a > question that has come up a few times: how should we support parallel > installation of multiple versions of the same distribution in the same > Python installation, *without* requiring complete isolation in the > form of virtual environments. > > The current solution (at least in Fedora), is to use the multi-version > support in pkg_resources by installing unpacked egg files. This is also the approach used by buildout. (It's also the approach (except for the unpacked part) used in modern Java-ecosystem-based deployments, FWIW. Collect jar files, typically in a cache, and set application-specific classpaths to point to the right ones.) ... > CherryPy 2 is *not* the default, and is instead installed as an > unpacked "CherryPy-2.3.0-py2.7.egg" directory. You can force this > directory to be added to sys.path by doing the following in your > __main__ module: > > __requires__ = ["CherryPy < 3"] > import pkg_resources I'd never see this. Interesting. > While I'm not a fan (to put it mildly) of non-trivial side effects > when importing a module, Me neither. > this approach to multi-version imports *does* > work well (and, as noted, I'm currently working on improving the docs > for it), and I think the approach to the filesystem layout in > particular makes sense - the alternative versions are installed to the > usual location, but pushed down a level in a subdirectory or zip > archive. Note that buildout takes a different approach that I think is worth keeping in mind. In buildout, of course, there's a specific "build" step that assembles an application from its parts. In the case of Python distributions, this means creating an application specific Python path as a list of installed eggs. This works well. It's explicit and pretty non-invasive. No import magic, .pth files or funny site.py files*. Buildout really wants to have self-contained distribution installations, whether they be eggs or wheels or whatever, to function properly. Jim * There was a well-intention but unfortunate deviation in later buildout 1 versions. This was, somewhat ironically in pursuit of better integration with system Python installations. -- Jim Fulton http://www.linkedin.com/in/jimfulton ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] a plea for backward-compatibility / smooth transitions
On Mon, Jul 29, 2013 at 2:15 PM, Donald Stufft wrote: > >> On Jul 29, 2013, at 1:18 PM, Paul Moore wrote: >> >> But even I am getting a little frustrated by the constant claims that "what >> we have now is insecure and broken, and must be fixed ASAP". The reality is >> that everything's more or less OK - there's a risk, certainly, and it could >> be severe, but many, many people are routinely using PyPI all the time >> without issues. And telling them that they are wrong to do so, or that they >> are being extremely naive over security, isn't helping. > This shows a fundamental misunderstanding of how security issues present > themselves. Of course things just work for people because security issues > are not like regular bugs. They don't negatively affect you until someone > attempts to use them to attack you. Keep your front door unlocked on your > house and your valuables will remain inside _until_ someone decides to try > and rob you. If you wait until people are affected by a security > vulnerability then the horse has already fled the pasture and you're just > attempting to close the gate after the fact. > > I'm pushing hard on doing what we can to secure the infrastructure because > this shit matters. Everything is more or less OK, only because no one has > decided that people installing from PyPI are not a valuable enough target to > go after. Prior to this push that was basically the only thing prevent > someone from attacking people, that they had never decided to bother too. We > are better, it's somewhat harder now, but in many areas that's still the > only thing keeping people safe. Well said. Security is a pain, but I'm really glad and appreciate that you and others are paying attention to it. Jim -- Jim Fulton http://www.linkedin.com/in/jimfulton ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
[Distutils] Buildout 2.2.0 released
zc.buildout 2.2.0 has been released to PyPI. The main feature of this release is support for setuptools 0.7 and later. See: https://pypi.python.org/pypi/zc.buildout/2.2.0#id3 for a full list of changes. To get this release, you need to download and run a new bootstrap.py file from: http://downloads.buildout.org/2/bootstrap.py Jim -- Jim Fulton http://www.linkedin.com/in/jimfulton ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] "python" option in buildout 2.1.1
On Wed, Jul 3, 2013 at 4:03 PM, Joe Steeve wrote: > From the buildout 2.1.1 docs on pypi: > > python >The name of a section containing information about the That's a documentation bug. :( I'll fix that. > Is the above still relevant? No. The support for multiple Python interpreters in a single buildout was dropped from buildout 2. Jim -- Jim Fulton http://www.linkedin.com/in/jimfulton ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
[Distutils] Important! buildout 2.1.1 released, prepare for distribute 0.7 and buildout 2.2
Distribute/setuptools 0.7 bring some big changes. These changes can't be handled by buildout's normal automatic upgrade mechanism. For this reason, it's important to do one of the following if you're using buildout 2: - Pin your distribute version to <0.7dev or to some other 0.6 version. - Upgrade to buildout 2.1.1, which was just released. This will prevent automatic upgrade to distribute 0.7 or buildout 2.2. - Be prepared to re-bootstrap your buildouts with the the new buildout 2.2 bootstrap script, which will be released soon after the release of distribute 0.7. Jim -- Jim Fulton http://www.linkedin.com/in/jimfulton ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Heads up: I plan to release buildout 2.2.0 this weekend.
On Thu, Jun 20, 2013 at 11:06 AM, Sebastien Douche wrote: > On Thu, Jun 20, 2013 at 4:37 PM, Jim Fulton wrote: >> Testing has gone well, but I still anticipate a little excitement. > > Yeah. Maybe a good idea to wait Setuptools 0.8? Why? It will pick up setuptools.8 when it becomes available. Jim -- Jim Fulton http://www.linkedin.com/in/jimfulton ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
[Distutils] Heads up: I plan to release buildout 2.2.0 this weekend.
Testing has gone well, but I still anticipate a little excitement. Jim -- Jim Fulton http://www.linkedin.com/in/jimfulton ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
[Distutils] Buildout 2.2.0b1 is available fort testing
Changes: - Switch dependency from ``distribute 0.6.x`` to ``setuptools 0.7.x``. - Handle both addition and subtraction of elements (+= and -=) on the same key in the same section. Forward-ported from buildout 1.6. - Suppress the useless ``Link to ***BLOCKED*** by --allow-hosts`` error message being emitted by distribute / setuptools. - Extend distutils script generation to support module docstrings and __future__ imports. - Refactored picked versions logic to make it easier to use for plugins. - Use ``get_win_launcher`` API to find Windows launcher (falling back to ``resource_string`` for ``cli.exe``). - Remove ``data_files`` from ``setup.py``: it was installing ``README.txt`` in current directory during installation (merged from 1.x branch). To try it out, copy the new bootstrap file from: http://downloads.buildout.org/2.2.0b1/bootstrap.py Then run the bootstrap, making sure to include the -t option: python bootstrap.py -t The -t option is necessary to use the non-final buildout release. We'd like to release 2.2 final soon, so your testing is really helpful. Special thanks to Tres Seaver for updating buildout to work with setuptools 0.7 and for reviewing and merging a number of outstanding pull requests. Of course, thanks to the folks to contributed the many other changes in this release. Jim -- Jim Fulton http://www.linkedin.com/in/jimfulton ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] buildout 1 still not managing to pin buildout to 1.x
On Tue, Jun 11, 2013 at 7:48 PM, Fred Drake wrote: > On Tue, Jun 11, 2013 at 7:22 PM, Tres Seaver wrote: >> When in doubt, remove .installed.cfg and re-run buildout after any such >> change. > > So the real question is: Why doesn't bootstrap.py just do this? Because it's not just an optimization and removing .installed.cfg prevents cleanup that might be desirable. Removing it as part of bootstrap seems too magic for me. I'd rather: - Error if there's a .installed.cfg - adding a --force option that tries to uninstall and then removed .instaled.cfg. > I've stumbled on this many times as well; I don't see an advantage to > this not being handled by the bootstrap, because it will surprise us > eventually. It rarely surprises me. > Some of us (include me, apparently) are more easily surprised than others. No kidding. ;) Jim -- Jim Fulton http://www.linkedin.com/in/jimfulton ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] buildout 1 still not managing to pin buildout to 1.x
On Tue, Jun 11, 2013 at 7:22 PM, Tres Seaver wrote: > -BEGIN PGP SIGNED MESSAGE- > Hash: SHA1 > > On 06/11/2013 05:27 AM, Chris Withers wrote: >> On 10/06/2013 18:57, Tres Seaver wrote: >>> Works for me: >>> >>> $ mkdir /tmp/withers $ cd /tmp/withers $ echo "[buildout] parts ="> >>> buildout.cfg $ wget http://downloads.buildout.org/1/bootstrap.py >>> ... $ /opt/Python-2.5.x/bin/python bootstrap.py >> >> Try with 2.7, and also try going from 1 to 2 and back again by >> switching bootstraps. >> >> There's likely something funky going on (.installed.cfg, develop-eggs >> or some other goat sacrifice in bin/) but it definitely happened. > > That same recipe works with 2.7. In general, the presence of > .installed.cfg should be thought of as problematic for any big surgery > (like changing bootstraps, or major versions of buildout), because it > hampers repeatability in order to make the build run faster. No. It is not an optimization. It's there to know what needs to be uninstalled. It's often OK to just remove it. but by doing so you're forgoing cleanup. It would be better to uninstall everything with the old buildout: bin/buildout buildout:parts= Before removing the installed file. Jim -- Jim Fulton http://www.linkedin.com/in/jimfulton ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
