Re: [Distutils] Beyond wheels 1.0: helping downstream, FHS and more
On Tue, Apr 14, 2015 at 4:19 PM, Paul Moore wrote: > On 14 April 2015 at 22:02, Chris Barker wrote: > > Personally, I'm not a fan of auto-installing, so I'd hope for > something more like pip would fail to install if a required extension > were missing. The user would then install the extension and redo the > install. But that may be a minority opinion - it's a bit like > setup_requires in principle, and people seem to prefer that to be > auto-installed. > > (lurker surfaces) I'm with Paul on this one. It seems to me that auto-installing the extension would destroy most of the advantages of distributing the extensions separately. I _might_ not hate it if pip prompted the user and _then_ installed, but then again, I might. (lurker sinks back into the depths) -- Kevin Horn ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] version for VCS/local builds between alpha and release
On 14 April 2015 at 23:31, Joost Molenaar wrote: > On 14 April 2015 at 10:04, Robert Collins wrote: > >> The basic scenario here is developers and CD deployers building >> versions from VCS of arbitrary commits. So we need to be able to >> deliver strictly increasing version numbers, automatically, without >> interfering with actual publishing of pre-release and release versions >> to PyPI. > > I think the advice in PEP440 about using dev tags[1] is a little misguided, > because dev tags count towards a known version in the future, while DVCS tags > (at least in Git) count the number of commits since a known version in the > past. In this respect, 'git describe' most closely resembles the post release > tags in PEP440, so that's what I've chosen to use in my build scripts, in > spite > of the recommendations in PEP440. > > > [1] https://www.python.org/dev/peps/pep-0440/#dvcs-based-version-labels Fair enough - what we're doing is using semver to predict the next version based on the git history - for instance the pseudo header Sem-Ver: api-break will cause the next version to be a major version up. -Rob -- Robert Collins Distinguished Technologist HP Converged Cloud ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] pip/warehouse feature idea: "help needed"
Nick Coghlan writes: > Yep, Guido's keynote was the genesis of the thread. I can't find it online, can you give a URL so we can see the talk? > Past suggestions for social features have related to providing users > with a standard way to reach maintainers and each other, and I'd > prefer to leave maintainers in full control of that aspect of the > maintainer experience. I'm not alone in feeling that way, hence why > such features tend not to be viewed especially positively. Thanks for this detailed response differentiating this proposal from previous ones, it's exactly what I was asking for. -- \ “For mad scientists who keep brains in jars, here's a tip: why | `\ not add a slice of lemon to each jar, for freshness?” —Jack | _o__) Handey | Ben Finney ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] pip/warehouse feature idea: "help needed"
On 14 April 2015 at 11:19, Trishank Karthik Kuppusamy wrote: > On 14 April 2015 at 11:16, Brett Cannon wrote: >> I agree. Even something as simple as a boolean that triggers a banner >> saying "this project is looking for a new maintainer" would be useful both >> from the perspective of project owners who want to move on or from the >> perspective of users who can't tell if a project is maintained based on how >> long it has been since a project uploaded a new version (which is why I >> think someone suggested sending an annual email asking for a human action to >> say "alive and kicking" to help determine if a project is completely >> abandoned). > > Yeah, I think Guido said something to this effect in his keynote. Yep, Guido's keynote was the genesis of the thread. For folks that haven't seen it, the specific points of concern raised were: * seeking a new maintainer from amongst their users * seeking help with enabling Python 3 support Past suggestions for social features have related to providing users with a standard way to reach maintainers and each other, and I'd prefer to leave maintainers in full control of that aspect of the maintainer experience. I'm not alone in feeling that way, hence why such features tend not to be viewed especially positively. The one thing that *only* PyPI can provide is the combination of a publication channel for maintainers to reach their user base without either side needing to share contact information they aren't already sharing, together with the creation of the clear understanding that providing sustaining engineering for a piece of software represents a significant time commitment that users benefiting from an open source maintainer's generosity should respect. This thread regarding maintainers being able to more clearly communicate maintenance status to users also relates to my blog post ( http://www.curiousefficiency.org/posts/2015/04/stop-supporting-python26.html) regarding the fact that folks that: a) don't personally need to ensure software they maintain works on old versions of Python; and b) aren't getting paid to ensure it works on old versions of Python; c) shouldn't feel obliged to provide such support for free Supporting legacy platforms is generally tedious work that isn't inherently interesting or rewarding. Folks that want such legacy platform support should thus be expecting to have to pay for it, and demanding it for free is unreasonable. The perception that open source software is provided by magic internet pixies that don't need to eat (or at the very least to be thanked for the time their generosity has saved us) is unfortunately widespread and pernicious [1], and PyPI is in a position to help shift that situation to one where open source maintainers at least have the opportunity to clearly explain the sustaining engineering model backing their software while deflecting any criticism for the mere existence of such explanations onto the PyPI maintainers rather than having to cope with any negative feedback themselves. Regards, Nick. [1] As far as *how* that mistaken perception that failing to adequately compensate open source developers is OK became so widespread, my own theory is that it stems from the fact that open source was popularised first in the context of Linux, which relies heavily on the corporate patronage model where companies like Red Hat make money from customers that often aren't interested in technology for its own sake, while making the underlying software freely available as a basis for shared collaboration and experimentation. I personally like that model [2], but plenty of folks have entirely reasonable concerns about it and hence need to support their open source work in other ways. My view is that appropriately addressing that complex situation involves actively challenging the common assumption that adequately compensating the project developers for their work is somebody else's problem, and thus that it makes sense to eventually build the ability to issue that challenge directly into the software distribution infrastructure. It's not at the top of the priority list right now, but Guido's keynote made me realise it should be on the list somewhere. [2] http://community.redhat.com/blog/2015/02/the-quid-pro-quo-of-open-infrastructure/ -- Nick Coghlan | ncogh...@gmail.com | Brisbane, Australia ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] pip/warehouse feature idea: "help needed"
> On Apr 14, 2015, at 6:46 PM, Ben Finney wrote: > > Trishank Karthik Kuppusamy writes: > >> Yeah, I think Guido said something to this effect in his keynote. > > Apparently I'm missing that context, then. The original post didn't help > me understand why this proposal is significantly different from past > “add a bunch of social to PyPI” rejected in the past. > > Can someone help by distinguishing this from past proposals of that > kind? I think one of the distinguishing characteristics is that we’ve (as a community) changed and we’re more willing to evolve PyPI to do more and be a more central part of the life of a project than previously. --- Donald Stufft PGP: 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA signature.asc Description: Message signed with OpenPGP using GPGMail ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] pip/warehouse feature idea: "help needed"
Trishank Karthik Kuppusamy writes: > Yeah, I think Guido said something to this effect in his keynote. Apparently I'm missing that context, then. The original post didn't help me understand why this proposal is significantly different from past “add a bunch of social to PyPI” rejected in the past. Can someone help by distinguishing this from past proposals of that kind? -- \ “I am as agnostic about God as I am about fairies and the | `\ Flying Spaghetti Monster.” —Richard Dawkins, 2006-10-13 | _o__) | Ben Finney ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Beyond wheel 1.0: more fine-grained installation scheme
On 14 Apr 2015 09:28, "Daniel Holth" wrote: > > That's exactly what I would like to do. Then > distribution-1.0.data/sysconfdir/file in a wheel would install into > /etc/file in the default scheme, but would probably really wind up in > $VIRTUAL_ENV/etc/... for most of us web developers. > > IIRC extra package-1.0-data/* directories in wheel are undefined. I > would have no problem putting fine-grained install schemes in 2.0 and > putting some of the other "wheel 2.0" features into wheel 3. > Incrementing the major version number would cause older pip to reject > the newer wheels, incrementing the minor version would produce a > warning. +1, although I expect bdist_wheel would likely need support on the generation side to use the lowest viable version of the wheel spec for a given package. Cheers, Nick. ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Beyond wheels 1.0: helping downstream, FHS and more
On 14 April 2015 at 22:02, Chris Barker wrote: > - pip auto-installs it (if not already there) when the user goes to install > the wheel. Personally, I'm not a fan of auto-installing, so I'd hope for something more like pip would fail to install if a required extension were missing. The user would then install the extension and redo the install. But that may be a minority opinion - it's a bit like setup_requires in principle, and people seem to prefer that to be auto-installed. Paul Paul ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Beyond wheels 1.0: helping downstream, FHS and more
On Tue, Apr 14, 2015 at 9:46 AM, Paul Moore wrote: > > Could an "extension" be -- "run this arbitrary Python script" ? > > The main point (as I see it) of an "extension" is that it's > distributed independently of the packages that use it. So you get to > decide to use an extension (and by inference audit it if you want) > *before* it gets run as part of an installation. OK, I think this is getting clearer to me now -- an Extension is (I suppose arbitrary) block of python code, but what goes into the wheel is not the code, but rather a declarative configuration for the extension. then at install-time, the actual code that runs is separate from the wheel, which gives the end user greater control, plus these nifty features > Extensions get peer > review by the community, and bad ones get weeded out, > the independent review and quality control > > there's also portability. And code quality. And > consistency. > And I'll add that this would promote code re-use and DRY. I'd be much happier installing a project that used a well-known "start > menu manager extension" So where would that code live? and how would it be managed? I'm thinking: - In package on PyPi like anything else - a specification in install_requires - pip auto-installs it (if not already there) when the user goes to install the wheel. Is that the idea? Of course, if the project I want to install makes using the extension > mandatory for the install to work, I still don't have a real choice - > I accept the extension or I can't use the code I want - well, you can't easily auto-install it anyway -- you could still do a source install, presumably. but there's an > extra level of transparency involved. And hopefully most extensions > will be optional, in practice. > There's a bit to think about in the API/UI here. If an installation_extension is used by a package, and it's specified in install_requires, then it's going to get auto-magically installed an used with a regular old "pip install". If we are worried about code review and users being in control of what extensions they use, then how to we make it obvious that a given extension is in use, but optional, and how to turn it off if you want? -CHB -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R(206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception chris.bar...@noaa.gov ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Beyond wheel 1.0: more fine-grained installation scheme
Just implement it. You could also try editing wheel's own proof of concept installer. https://bitbucket.org/pypa/wheel/src/tip/wheel/install.py?at=default#cl-246 On Tue, Apr 14, 2015 at 3:20 PM, David Cournapeau wrote: > > > On Tue, Apr 14, 2015 at 9:27 AM, Daniel Holth wrote: >> >> That's exactly what I would like to do. Then >> distribution-1.0.data/sysconfdir/file in a wheel would install into >> /etc/file in the default scheme, but would probably really wind up in >> $VIRTUAL_ENV/etc/... for most of us web developers. > > > $prefix would be set to sys.prefix and $eprefix to sys.exec_prefix to handle > automatically. > > Should I work on a PEP for wheel 2.0, or a pip implementation first ? > >> >> IIRC extra package-1.0-data/* directories in wheel are undefined. > > > pip will actually fail to install any wheel with an undefined directory > package-1.0-data/* (unfortunately this is not detected before installing, so > it ends up with a half installed package). > > David > >> >> I >> would have no problem putting fine-grained install schemes in 2.0 and >> putting some of the other "wheel 2.0" features into wheel 3. >> Incrementing the major version number would cause older pip to reject >> the newer wheels, incrementing the minor version would produce a >> warning. > > > >> >> >> >> On Tue, Apr 14, 2015 at 8:56 AM, David Cournapeau >> wrote: >> > Hi, >> > >> > I am splitting up the previous thread into one thread / proposal to >> > focus >> > the discussion. >> > >> > Assuming the basis of this proposal does not sound too horrible, I would >> > make a proof of concept in a pip branch, so that we can flush out the >> > details and then write an actual spec (I guess an updated wheel format >> > would >> > need a new PEP ?). >> > >> > The goal of this thread is to flush out a more fine-grained installation >> > scheme, so that wheels can install files anywhere they want (at least >> > within >> > sys.prefix/sys.exec_prefix). I see two issues: >> > >> > 1. defining what the scheme should be >> > 2. how should it be implemented in wheel: there are different trade-offs >> > depending on whether we want this feature to be part of wheel format 1.* >> > or >> > 2.0. >> > >> > First, my understanding of the current situation: >> > >> > * per the wheel PEP 427, anything in the wheel top directory and not in >> > distribution-1.0.data is installed in site-package >> > * every top directory in distribution-1.0.data/ needs to be mapped to >> > the >> > scheme as defined in distutils install command. >> > * pip rejects any directory in distribution-1.0.data/ which is not in >> > the >> > scheme from 2. >> > >> > My suggestion for a better scheme would be to use an extended version of >> > the >> > various default directories defined by autotools. The extension would >> > handle >> > windows-specifics. More concretely: >> > >> > # Suggested variables >> > >> > The goal of supporting those variables is to take something that is >> > flexible >> > enough to support almost any installation scheme, without putting >> > additional >> > burden on the developer. People who do not want/need the flexibility >> > will >> > not need to do anything more than what they do today. >> > >> > The variables I would suggest are every variable defined in >> > >> > https://github.com/cournape/Bento/blob/master/bento/core/platforms/sysconfig.py#L10, >> > except for destdir which is not relevant here. >> > >> > On unix, the defaults follow autotools, and on windows, I mapped almost >> > everything relative to sys.exec_prefix, except for the >> > bindir/sbindir/libexecdir which map to "$prefix\Scripts". >> > >> > The $sitedir variable would need to be updated to use the value from >> > distutils instead of the hardcoded value I put in that file as well. >> > >> > # How to handle the augmented scheme >> > >> > Right now, if I wanted to install something in say $prefix/share/doc, I >> > would need to put it in distribution-1.0.data/data/share/doc, but this >> > prevents use from handling different platforms differently. >> > >> > OTOH, this is the only way I can see to make the new scheme backward >> > compatible with pip versions who would not understand the new scheme. I >> > don't have a good sense of what we should do there, the core pip team >> > may >> > have a better sense. >> > >> > For now, I would be happy to just make a proof of concept not caring >> > about >> > backward compatibility in a pip branch. Does that sound like a workable >> > basis to flush out an actual proposal ? >> > >> > thanks, >> > David >> > >> > >> > ___ >> > Distutils-SIG maillist - Distutils-SIG@python.org >> > https://mail.python.org/mailman/listinfo/distutils-sig >> > > > ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Beyond wheel 1.0: more fine-grained installation scheme
On Tue, Apr 14, 2015 at 9:27 AM, Daniel Holth wrote: > That's exactly what I would like to do. Then > distribution-1.0.data/sysconfdir/file in a wheel would install into > /etc/file in the default scheme, but would probably really wind up in > $VIRTUAL_ENV/etc/... for most of us web developers. > $prefix would be set to sys.prefix and $eprefix to sys.exec_prefix to handle automatically. Should I work on a PEP for wheel 2.0, or a pip implementation first ? > IIRC extra package-1.0-data/* directories in wheel are undefined. pip will actually fail to install any wheel with an undefined directory package-1.0-data/* (unfortunately this is not detected before installing, so it ends up with a half installed package). David > I > would have no problem putting fine-grained install schemes in 2.0 and > putting some of the other "wheel 2.0" features into wheel 3. > Incrementing the major version number would cause older pip to reject > the newer wheels, incrementing the minor version would produce a > warning. > > > > On Tue, Apr 14, 2015 at 8:56 AM, David Cournapeau > wrote: > > Hi, > > > > I am splitting up the previous thread into one thread / proposal to focus > > the discussion. > > > > Assuming the basis of this proposal does not sound too horrible, I would > > make a proof of concept in a pip branch, so that we can flush out the > > details and then write an actual spec (I guess an updated wheel format > would > > need a new PEP ?). > > > > The goal of this thread is to flush out a more fine-grained installation > > scheme, so that wheels can install files anywhere they want (at least > within > > sys.prefix/sys.exec_prefix). I see two issues: > > > > 1. defining what the scheme should be > > 2. how should it be implemented in wheel: there are different trade-offs > > depending on whether we want this feature to be part of wheel format 1.* > or > > 2.0. > > > > First, my understanding of the current situation: > > > > * per the wheel PEP 427, anything in the wheel top directory and not in > > distribution-1.0.data is installed in site-package > > * every top directory in distribution-1.0.data/ needs to be mapped to the > > scheme as defined in distutils install command. > > * pip rejects any directory in distribution-1.0.data/ which is not in the > > scheme from 2. > > > > My suggestion for a better scheme would be to use an extended version of > the > > various default directories defined by autotools. The extension would > handle > > windows-specifics. More concretely: > > > > # Suggested variables > > > > The goal of supporting those variables is to take something that is > flexible > > enough to support almost any installation scheme, without putting > additional > > burden on the developer. People who do not want/need the flexibility will > > not need to do anything more than what they do today. > > > > The variables I would suggest are every variable defined in > > > https://github.com/cournape/Bento/blob/master/bento/core/platforms/sysconfig.py#L10 > , > > except for destdir which is not relevant here. > > > > On unix, the defaults follow autotools, and on windows, I mapped almost > > everything relative to sys.exec_prefix, except for the > > bindir/sbindir/libexecdir which map to "$prefix\Scripts". > > > > The $sitedir variable would need to be updated to use the value from > > distutils instead of the hardcoded value I put in that file as well. > > > > # How to handle the augmented scheme > > > > Right now, if I wanted to install something in say $prefix/share/doc, I > > would need to put it in distribution-1.0.data/data/share/doc, but this > > prevents use from handling different platforms differently. > > > > OTOH, this is the only way I can see to make the new scheme backward > > compatible with pip versions who would not understand the new scheme. I > > don't have a good sense of what we should do there, the core pip team may > > have a better sense. > > > > For now, I would be happy to just make a proof of concept not caring > about > > backward compatibility in a pip branch. Does that sound like a workable > > basis to flush out an actual proposal ? > > > > thanks, > > David > > > > > > ___ > > Distutils-SIG maillist - Distutils-SIG@python.org > > https://mail.python.org/mailman/listinfo/distutils-sig > > > ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] pip/warehouse feature idea: "help needed"
On 14 April 2015 at 11:16, Brett Cannon wrote: > > I agree. Even something as simple as a boolean that triggers a banner > saying "this project is looking for a new maintainer" would be useful both > from the perspective of project owners who want to move on or from the > perspective of users who can't tell if a project is maintained based on how > long it has been since a project uploaded a new version (which is why I > think someone suggested sending an annual email asking for a human action > to say "alive and kicking" to help determine if a project is completely > abandoned). > > Yeah, I think Guido said something to this effect in his keynote. ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Beyond wheels 1.0: helping downstream, FHS and more
On 14 April 2015 at 17:10, Chris Barker wrote: >> If there’s a plugin that understands this extension >> > installed, let it do something before you actually move the files into >> > place”. This let’s Wheels themselves still be declarative and moves the >> > responsibility of implementing these bits into their own PyPI projects >> > that can be versioned and independently upgraded and such. We’d probably >> > need some method of marking an extension as “critical” (e.g. bail out >> > and >> > don’t install this Wheel if you don’t have something that knows how to >> > handle >> > it) and then non critical extensions just get ignored if we don’t know >> > how to handle it. > > Could an "extension" be -- "run this arbitrary Python script" ? The main point (as I see it) of an "extension" is that it's distributed independently of the packages that use it. So you get to decide to use an extension (and by inference audit it if you want) *before* it gets run as part of an installation. Extensions get peer review by the community, and bad ones get weeded out, in a way that just having a chunk of code in your setup.py or the postinstall section of your wheel doesn't. > We've got a full featured scripting language (with batteries included!) -- > isn't that all the extension you need? Up to a point yes. It's the independent review and quality control aspects that matter to me. > Or is this about security? We don't want to let a package do virtually > anything on install? Security is one aspect, and one that a lot of people will pick up on immediately. But there's also portability. And code quality. And consistency. I'd be much happier installing a project that used a well-known "start menu manager extension" than one that just used custom code. I'd be willing to assume that the author of the extension had thought about Unix/Windows compatibility, how to handle use in a virtualenv, handling user preferences (such as the end user *not wanting* shortcuts), etc etc. And I could look at the extension project's issue tracker to see how happy I was with the state of the project. Of course, if the project I want to install makes using the extension mandatory for the install to work, I still don't have a real choice - I accept the extension or I can't use the code I want - but there's an extra level of transparency involved. And hopefully most extensions will be optional, in practice. Paul ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Beyond wheels 1.0: helping downstream, FHS and more
> > If there’s a plugin that understands this extension > > installed, let it do something before you actually move the files into > > place”. This let’s Wheels themselves still be declarative and moves the > > responsibility of implementing these bits into their own PyPI projects > > that can be versioned and independently upgraded and such. We’d probably > > need some method of marking an extension as “critical” (e.g. bail out and > > don’t install this Wheel if you don’t have something that knows how to > handle > > it) and then non critical extensions just get ignored if we don’t know > > how to handle it. > Could an "extension" be -- "run this arbitrary Python script" ? We've got a full featured scripting language (with batteries included!) -- isn't that all the extension you need? Or is this about security? We don't want to let a package do virtually anything on install? -CHB > > Right, this is the intent of the "Required extension handling" > feature: > https://www.python.org/dev/peps/pep-0426/#required-extension-handling > > If a package flags an extension as "installer_must_handle", then > attempts to install that package are supposed to fail if the installer > doesn't recognise the extension. Otherwise, installers are free to > ignore extensions they don't understand. > > So meta-installers like canopy could add their own extensions to their > generated wheel files, flag those extensions as required, and other > installers would correctly reject those wheels as unsupported. > > Cheers, > Nick. > > -- > Nick Coghlan | ncogh...@gmail.com | Brisbane, Australia > ___ > Distutils-SIG maillist - Distutils-SIG@python.org > https://mail.python.org/mailman/listinfo/distutils-sig > -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R(206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception chris.bar...@noaa.gov ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Beyond wheels 1.0: helping downstream, FHS and more
On Tue, Apr 14, 2015 at 8:41 AM, Nick Coghlan wrote: > The main two language independent solutions I've identified for this > general "user level package management" problem in the Fedora > Environments & Stacks context > ( > https://fedoraproject.org/wiki/Env_and_Stacks/Projects/UserLevelPackageManagement > ) > are conda (http://conda.pydata.org/) and Nix > (https://nixos.org/nix/about.html), cool -- I hadn't seem nix before. > From a Python upstream perspective, Nix falls a long way behind conda > due to the fact that Nix currently relies on Cygwin for Windows > support - The other thing that's nice about conda is that while it was designed for the general case, it has a lot of python-specific features. Being a Python guy -- I llke that ;-) -- it may not work nearly as well for Ruby or what have you -- I wouldn't know. > The point where I draw the line is supporting *dynamic* > linking between modules - I'm confused -- you don't want a system to be able to install ONE version of a lib that various python packages can all link to? That's really the key use-case for me > that's the capability I view as defining the > boundary between "enabling an add-on ecosystem for a programming > language runtime" and "providing a comprehensive software development > platform" :) > Well, with it's target audience being scientific programmers, conda IS trying to give you a "comprehensive software development platform" We're talking about Python here -- it's a development tool. It turns out that for scientific development, pure python is simply not enough -- hence the need for conda and friends. I guess this is what it comes down to -- I'm all for adding a few features to wheel -- it would be nice to be abel to pip install most of what I, and people like me, need. But maybe it's not possible -- you can solve the shared lib problem, and the scripts problem, and maybe the menu entires problem, but eventually, you end up with "I want to use numba" -- and then you need LLVM, etc. -- and pretty soon you are building a tool that provides a "comprehensive software development platform". ;-) -CHB -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R(206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception chris.bar...@noaa.gov ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Beyond wheels 1.0: helping downstream, FHS and more
On 13 April 2015 at 22:29, Donald Stufft wrote: > So a possible way for this to work is in a PEP 426 world, simply define > a twisted.plugins extension that says, in a declarative way, “hey when > you install this Wheel, if there’s a plugin that understands this extension > installed, let it do something before you actually move the files into > place”. This let’s Wheels themselves still be declarative and moves the > responsibility of implementing these bits into their own PyPI projects > that can be versioned and independently upgraded and such. We’d probably > need some method of marking an extension as “critical” (e.g. bail out and > don’t install this Wheel if you don’t have something that knows how to handle > it) and then non critical extensions just get ignored if we don’t know > how to handle it. Right, this is the intent of the "Required extension handling" feature: https://www.python.org/dev/peps/pep-0426/#required-extension-handling If a package flags an extension as "installer_must_handle", then attempts to install that package are supposed to fail if the installer doesn't recognise the extension. Otherwise, installers are free to ignore extensions they don't understand. So meta-installers like canopy could add their own extensions to their generated wheel files, flag those extensions as required, and other installers would correctly reject those wheels as unsupported. Cheers, Nick. -- Nick Coghlan | ncogh...@gmail.com | Brisbane, Australia ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Beyond wheels 1.0: helping downstream, FHS and more
On 13 April 2015 at 12:56, Chris Barker wrote: > Which brings me back to the question: should the python tools (i.e. wheel) > be extended to support more use-cases, specifically non-python dependencies? > Or do we just figure that that's a problem better solved by projects with a > larger scope (i.e. rpm, deb, conda, canopy). > > I'm on the fence here. I mostly care about Python, and I think we're pretty > darn close with allowing wheel to support the non-python dependencies, which > would allow us all to "simply pip install" pretty much anything -- that > would be cool. But maybe it's a bit of a slippery slope, and if we go there, > we'll end up re-writing conda. The main two language independent solutions I've identified for this general "user level package management" problem in the Fedora Environments & Stacks context (https://fedoraproject.org/wiki/Env_and_Stacks/Projects/UserLevelPackageManagement) are conda (http://conda.pydata.org/) and Nix (https://nixos.org/nix/about.html), backed up by Pulp for plugin-based format independent repository management (http://www.pulpproject.org/). >From a Python upstream perspective, Nix falls a long way behind conda due to the fact that Nix currently relies on Cygwin for Windows support - it's interesting to me for Fedora because Nix ticks a lot of boxes from a system administrator perspective that conda doesn't (in particular, system administrators can more easily track what users have installed, and ensure that packages are updated appropriately in the face of security updates in dependencies). I definitely see value in Python upstream formats being able to bundle additional files like config files, desktop integration files, service definition files, statically linked extensions modules, etc, in a way that not only supports direct installation onto end user machines, but also conversion into platform specific formats (whether that platform is an operating system, or a cross-platform platform like nix, canopy or conda). The point where I draw the line is supporting *dynamic* linking between modules - that's the capability I view as defining the boundary between "enabling an addon ecosystem for a programming language runtime" and "providing a comprehensive software development platform" :) Cheers, Nick. -- Nick Coghlan | ncogh...@gmail.com | Brisbane, Australia ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Make PEP 426 less boring
On 13 April 2015 at 02:24, Thomas Güttler wrote:
> Hi,
>
> somehow I feel bored if I read PEP 426.
> https://www.python.org/dev/peps/pep-0426/
If anyone didn't find the complexities of real world software
distribution tedious, frustrating and often mindnumbingly dull, I'd
assume they weren't paying attention :)
> One concrete improvement would be to remove this paragraph:
>
> {{{
> The design draws on the Python community's 15 years of experience with
> distutils based software distribution, and incorporates ideas and concepts
> from other distribution systems, including Python's setuptools, pip and
> other projects, Ruby's gems, Perl's CPAN, Node.js's npm, PHP's composer and
> Linux packaging systems such as RPM and APT.
> }}}
>
> Because something like this was already saied some lines before
>
> {{{
> Metadata 2.0 represents a major upgrade to the Python packaging ecosystem,
> and attempts to incorporate experience gained over the 15 years(!) since
> distutils was first added to the standard library. Some of that is just
> incorporating existing practices from setuptools/pip/etc, some of it is
> copying from other distribution systems (like Linux distros or other
> development language communities) and some of it is attempting to solve
> problems which haven't yet been well solved by anyone (like supporting clean
> conversion of Python source packages to distro policy compliant source
> packages for at least Debian and Fedora, and perhaps other platform specific
> distribution systems).
> }}}
>
> **And** I would move the historic background (the second of the above
> quotes) at the end.
>
> Meta: are you interested in feedback like this?
Not so much - I haven't done a serious editing pass myself, as I'm
still mostly interested in engaging folks that are already invested in
the packaging tools ecosystem, rather than making it readily
accessible to newcomers. Once we're happy we know what good looks
like, then it will be part of the role of packaging.python.org to
provide the "just the facts" introduction that lowers the barriers to
entry, leaving the raw spec to the folks that are inclined to spend
our time reading RFCs and other specs :)
Cheers,
Nick.
--
Nick Coghlan | ncogh...@gmail.com | Brisbane, Australia
___
Distutils-SIG maillist - Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] pip/warehouse feature idea: "help needed"
On Mon, Apr 13, 2015 at 10:13 PM Donald Stufft wrote: > > > On Apr 13, 2015, at 8:57 PM, Ben Finney > wrote: > > > > Nick Coghlan writes: > > > >> On 11 Apr 2015 12:22, "Alexander Walters" > wrote: > >>> Is the package index really the best place to put this? This is a > >>> very social-networking feature for the authoritative repository of > >>> just about all the third party module, and it feels like either it > >>> could corrupt the 'sanctity' of the repository (in the absolute > >>> worst case) > >> > >> If you're concerned that this feature might weaken the comforting > >> illusion that PyPI published software is contributed and maintained by > >> faceless automatons rather than living, breathing human beings, then > >> yes, encouraging folks to think more about where the software they use > >> is coming from would be a large part of the point of adding such a > >> feature. > > > > I can't speak for Alexander, but I'm also −1 to have this *on PyPI*. > > > > I'm all for such features existing. What is at issue is whether PyPI is > > the place to put them. > > > > We have been gradually improving the function of PyPI as an > > authoritative *index* of packages; that's possible because it is a > > repository of uncontroversial facts, not opinions (i.e. “what is the > > packaging metadata of this distribution”, “where is its documentation”, > > “where is its VCS”, etc.). > > > >>> I am not saying the PSF shouldn't do this, but is pypi REALLY the > >>> best part of python.org to put it? > >> > >> I personally believe so, yes - sustaining software over the long term is > >> expensive in people's time, but it's often something we take for > granted. > >> The specific example Guido brought up in his keynote was the challenge > of > >> communicating a project's openness to Python 3 porting assistance. > > > > The people doing the work of maintaining PyPI have said many times in > > recent years that there just isn't enough person-power to add a whole > > bunch of features that have been requested. Why would we think > > moderating a social-networking rating, opinion, discussion, or other > > non-factual database is something reasonable to ask of the PyPI > > maintainers? > > > > Conversely, if we are under the impression that adding ratings, > > feedback, reviews, discussion, and other features to PyPI is *not* going > > to be a massive increase in workload for the maintainers, I think that's > > a foolish delusion which will be quite costly to the reputation PyPI has > > recently gained through hard effort to clarify its role. > > > > By all means, set up a well-maintained social ecosystem around Python > > packages. But not on PyPI itself: The Python Package Index is feasible > > in part because it has a clear and simple job, though, and that's not > > it. > > > > -- > > \“If you can't hear me sometimes, it's because I'm in | > > `\ parentheses.” —Steven Wright | > > _o__) | > > Ben Finney > > > > ___ > > Distutils-SIG maillist - Distutils-SIG@python.org > > https://mail.python.org/mailman/listinfo/distutils-sig > > > I don’t see any problem with the general idea of adding features to PyPI to > enable package maintainers to find more help maintaining specific parts of > their projects. I do have a problem with expecting the PyPI administrators > to fill out or otherwise populate this information. Saying “Here’s a place > you can donate to me” is still a fact, it’s just a more social fact than > what we currently enable. > > I’m kind of down on the idea of linking to CVs or linkedin as part of the > project metadata because that’s not project specific and is really more > maintainer specific. I think that particular feature would be better suited > to some sort of global “Python profile” that could then be linked to from > PyPI instead of trying to bake it into PyPI itself. > > However things like “Looking for New Maintainers / Orphan a Project”, > or some call to actions on “here are some issues that need fixed” or other > things doesn’t seem unreasonable to me. Particularly the ability to orphan > a project or look for new maintainers seems like a useful thing to me that > really can’t live anywhere other than PyPI reasonably. > I agree. Even something as simple as a boolean that triggers a banner saying "this project is looking for a new maintainer" would be useful both from the perspective of project owners who want to move on or from the perspective of users who can't tell if a project is maintained based on how long it has been since a project uploaded a new version (which is why I think someone suggested sending an annual email asking for a human action to say "alive and kicking" to help determine if a project is completely abandoned). -Brett ___ Distutils-SIG maillist - Distutils-SIG@pyt
Re: [Distutils] Beyond wheel 1.0: more fine-grained installation scheme
That's exactly what I would like to do. Then distribution-1.0.data/sysconfdir/file in a wheel would install into /etc/file in the default scheme, but would probably really wind up in $VIRTUAL_ENV/etc/... for most of us web developers. IIRC extra package-1.0-data/* directories in wheel are undefined. I would have no problem putting fine-grained install schemes in 2.0 and putting some of the other "wheel 2.0" features into wheel 3. Incrementing the major version number would cause older pip to reject the newer wheels, incrementing the minor version would produce a warning. On Tue, Apr 14, 2015 at 8:56 AM, David Cournapeau wrote: > Hi, > > I am splitting up the previous thread into one thread / proposal to focus > the discussion. > > Assuming the basis of this proposal does not sound too horrible, I would > make a proof of concept in a pip branch, so that we can flush out the > details and then write an actual spec (I guess an updated wheel format would > need a new PEP ?). > > The goal of this thread is to flush out a more fine-grained installation > scheme, so that wheels can install files anywhere they want (at least within > sys.prefix/sys.exec_prefix). I see two issues: > > 1. defining what the scheme should be > 2. how should it be implemented in wheel: there are different trade-offs > depending on whether we want this feature to be part of wheel format 1.* or > 2.0. > > First, my understanding of the current situation: > > * per the wheel PEP 427, anything in the wheel top directory and not in > distribution-1.0.data is installed in site-package > * every top directory in distribution-1.0.data/ needs to be mapped to the > scheme as defined in distutils install command. > * pip rejects any directory in distribution-1.0.data/ which is not in the > scheme from 2. > > My suggestion for a better scheme would be to use an extended version of the > various default directories defined by autotools. The extension would handle > windows-specifics. More concretely: > > # Suggested variables > > The goal of supporting those variables is to take something that is flexible > enough to support almost any installation scheme, without putting additional > burden on the developer. People who do not want/need the flexibility will > not need to do anything more than what they do today. > > The variables I would suggest are every variable defined in > https://github.com/cournape/Bento/blob/master/bento/core/platforms/sysconfig.py#L10, > except for destdir which is not relevant here. > > On unix, the defaults follow autotools, and on windows, I mapped almost > everything relative to sys.exec_prefix, except for the > bindir/sbindir/libexecdir which map to "$prefix\Scripts". > > The $sitedir variable would need to be updated to use the value from > distutils instead of the hardcoded value I put in that file as well. > > # How to handle the augmented scheme > > Right now, if I wanted to install something in say $prefix/share/doc, I > would need to put it in distribution-1.0.data/data/share/doc, but this > prevents use from handling different platforms differently. > > OTOH, this is the only way I can see to make the new scheme backward > compatible with pip versions who would not understand the new scheme. I > don't have a good sense of what we should do there, the core pip team may > have a better sense. > > For now, I would be happy to just make a proof of concept not caring about > backward compatibility in a pip branch. Does that sound like a workable > basis to flush out an actual proposal ? > > thanks, > David > > > ___ > Distutils-SIG maillist - Distutils-SIG@python.org > https://mail.python.org/mailman/listinfo/distutils-sig > ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
[Distutils] Beyond wheel 1.0: more fine-grained installation scheme
Hi, I am splitting up the previous thread into one thread / proposal to focus the discussion. Assuming the basis of this proposal does not sound too horrible, I would make a proof of concept in a pip branch, so that we can flush out the details and then write an actual spec (I guess an updated wheel format would need a new PEP ?). The goal of this thread is to flush out a more fine-grained installation scheme, so that wheels can install files anywhere they want (at least within sys.prefix/sys.exec_prefix). I see two issues: 1. defining what the scheme should be 2. how should it be implemented in wheel: there are different trade-offs depending on whether we want this feature to be part of wheel format 1.* or 2.0. First, my understanding of the current situation: * per the wheel PEP 427, anything in the wheel top directory and not in distribution-1.0.data is installed in site-package * every top directory in distribution-1.0.data/ needs to be mapped to the scheme as defined in distutils install command. * pip rejects any directory in distribution-1.0.data/ which is not in the scheme from 2. My suggestion for a better scheme would be to use an extended version of the various default directories defined by autotools. The extension would handle windows-specifics. More concretely: # Suggested variables The goal of supporting those variables is to take something that is flexible enough to support almost any installation scheme, without putting additional burden on the developer. People who do not want/need the flexibility will not need to do anything more than what they do today. The variables I would suggest are every variable defined in https://github.com/cournape/Bento/blob/master/bento/core/platforms/sysconfig.py#L10, except for destdir which is not relevant here. On unix, the defaults follow autotools, and on windows, I mapped almost everything relative to sys.exec_prefix, except for the bindir/sbindir/libexecdir which map to "$prefix\Scripts". The $sitedir variable would need to be updated to use the value from distutils instead of the hardcoded value I put in that file as well. # How to handle the augmented scheme Right now, if I wanted to install something in say $prefix/share/doc, I would need to put it in distribution-1.0.data/data/share/doc, but this prevents use from handling different platforms differently. OTOH, this is the only way I can see to make the new scheme backward compatible with pip versions who would not understand the new scheme. I don't have a good sense of what we should do there, the core pip team may have a better sense. For now, I would be happy to just make a proof of concept not caring about backward compatibility in a pip branch. Does that sound like a workable basis to flush out an actual proposal ? thanks, David ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] version for VCS/local builds between alpha and release
On 14 April 2015 at 10:04, Robert Collins wrote: > The basic scenario here is developers and CD deployers building > versions from VCS of arbitrary commits. So we need to be able to > deliver strictly increasing version numbers, automatically, without > interfering with actual publishing of pre-release and release versions > to PyPI. I think the advice in PEP440 about using dev tags[1] is a little misguided, because dev tags count towards a known version in the future, while DVCS tags (at least in Git) count the number of commits since a known version in the past. In this respect, 'git describe' most closely resembles the post release tags in PEP440, so that's what I've chosen to use in my build scripts, in spite of the recommendations in PEP440. [1] https://www.python.org/dev/peps/pep-0440/#dvcs-based-version-labels -- Joost Molenaar ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
[Distutils] version for VCS/local builds between alpha and release
Tl;dr: I woud like to change PEP-440 to remove the stigmata around dev versions that come between pre-releases and releases. The basic scenario here is developers and CD deployers building versions from VCS of arbitrary commits. So we need to be able to deliver strictly increasing version numbers, automatically, without interfering with actual publishing of pre-release and release versions to PyPI. Today, this is fairly sane as a sort order and mechanism to deliver this: >>> v = ['1.0', '1.2.3.a1.dev1', '1.2.3.a1', '1.2.3.a2.dev1', '1.2.3.b1', >>> '1.2.3.b2.dev1', '1.2.3.rc1', '1.2.3.rc2.dev1', '1.2.3', '1.2.3.dev1'] >>> sorted(v, key=packaging.version.parse) ['1.0', '1.2.3.dev1', '1.2.3.a1.dev1', '1.2.3.a1', '1.2.3.a2.dev1', '1.2.3.b1', '1.2.3.b2.dev1', '1.2.3.rc1', '1.2.3.rc2.dev1', '1.2.3'] But it doesn't feel good using a 'strongly recommended against' order. I think the recommendation is overly strong: If we touch up the language we can make the spec clearer, and a simple example like above will speak a thousand words :). -Rob -- Robert Collins Distinguished Technologist HP Converged Cloud ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
