[Distutils] Clarifying the meaning of the sysconfig "data" directory
In the binary dependency management threads, Daniel pointed out that
in all currently defined sysconfig schemes, the "data" directory will
end up pointing to the target installation directory. The "data" name
in the scheme doesn't actually mean "this is a data file", it means
"this has no defined semantic meaning at the sysconfig level, so just
put it exactly where the relative path says to put it".
That means that you can ship relatively arbitrary software in a wheel
file by dumping it in "{distribution}-{version}.data/data/", and
then installing it to the appropriate target location (e.g. if you use
FHS paths inside the data directory, then you would just need to
install the wheel as "pip install --root /" to get things into the
desired location).
However, you'd be forgiven for not realising that from the sysconfig
docs - I certainly didn't realise it until Daniel pointed it out. From
the sysconfig docs, I had expected "data" to mean "application data
files" not "arbitrary, potentially executable, file with no assigned
semantics".
So, I think we should fix the docs to reflect the way the schemes are
defined in practice. I'd like to clarify them to say that "data" will
always point to the installation target directory for the software
being installed (on POSIX systems, typically /, /usr, /usr/local,
~/.local or a virtual environment directory), and thus the "data" path
is intended for any file which doesn't fit one of the predefined
categories - they're arbitrary data files from the *packaging
system's* point of view, but not necessarily from the *application's*
point of view.
Cheers,
Nick.
--
Nick Coghlan | ncogh...@gmail.com | Brisbane, Australia
___
Distutils-SIG maillist - Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] More control over uploading an sdist and egg to PyPI.
On 7 December 2013 11:50, Kannan Goundan wrote: > We use setuptools for our library. I'm trying to figure out a way to upload > an sdist and an egg to PyPI without having to put my PyPI credentials in a > file. Please let me know if this is not the right mailing list for this > question. > > Our setup is that the entire team has access to the source code repository, > but only the release managers have access to the credentials needed to make > releases to PyPI (they're in a "fake" home directory). To do a release, you > run: > >cd project-dir >HOME="/release-manager-secrets/fake-pypi-home-dir" \ > python setup.py bdist_egg sdist upload > > This works ok, but we would prefer to not have our PyPI credentials on the > filesystem (unencrypted) at all. I'd like to run "setup.py bdist_egg sdist" > first and then run a separate command that uploaded the egg and sdist. This > command would ask for my credentials on stdin. > > How can I write such a program? Is there a PyPI HTTP API I can use? Does > setuptools have a public interface that I can call? Hi Kanaan, I believe twine (https://pypi.python.org/pypi/twine/) should work as the independent upload utility that you're looking for. Cheers, Nick. -- Nick Coghlan | ncogh...@gmail.com | Brisbane, Australia ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Install a script to /sbin instead of /bin
On Thursday, December 05, 2013 06:37:39 PM Michael Jansen wrote: > On Tuesday, December 03, 2013 12:33:22 PM Michael Jansen wrote: > > > Changes to distutils itself are fairly pointless, since the earliest > > > possible date for publication of any such changes is now as part of > > > Python 3.5 in 2015. The sheer impracticality of that approach when > > > plenty of people are still running Python 2.6 is why we're going > > > through the somewhat painful exercise of decoupling the preferred > > > build system from the standard library :) > > > > > > So, at this point, the appropriate place to make such a change is in > > > setuptools: https://bitbucket.org/pypa/setuptools > > > > > > That will allow it to be made in a way that supports Python 2.6+, > > > whereas a distutils change won't really help anyone. > > > > A point well made :) . Will do that. > > I made a proof of concept implementation (not ready to be merged) here No one interested in that? Got no answer from the maintainers too. Mike ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Clarifying the meaning of the sysconfig "data" directory
On 7 December 2013 08:19, Nick Coghlan wrote:
> That means that you can ship relatively arbitrary software in a wheel
> file by dumping it in "{distribution}-{version}.data/data/", and
> then installing it to the appropriate target location (e.g. if you use
> FHS paths inside the data directory, then you would just need to
> install the wheel as "pip install --root /" to get things into the
> desired location).
>
> However, you'd be forgiven for not realising that from the sysconfig
> docs - I certainly didn't realise it until Daniel pointed it out. From
> the sysconfig docs, I had expected "data" to mean "application data
> files" not "arbitrary, potentially executable, file with no assigned
> semantics".
The problem with what you're proposing (which isn't a change, as you
say, so much as a subtle reinterpretation) is that if you include
something in a wheel under /etc or /usr/include, you *are*
assigning semantics to it - and the semantics you are assigning is
POSIX. That's not a disaster, of course, but can make for a less than
ideal layout on Windows. For example, if packages started putting
config data under "etc" that would not be normal Windows practice.
It's not *bad* per se, just disconcerting (and TBH, when I've seen
this done, it gives the impression of a lazily-ported Unix package).
Also, this paves the way for people starting to put data files under
"/usr/share" or whatever the FHS location is. That is definitely
*not* the Python convention, which is to put the data inside the
package and locate it via package_data(). It also makes the package
fail when used zipped or via tools like py2exe.
So while I have no issue with this practice as a means of dealing with
things like the RPM build issue for purely POSIX packages, I do think
we should be careful about raising its visibility too much.
Personally, I see it as a last resort "escape hatch" for data that
simply cannot be placed in any of the standard Python locations.
Paul
___
Distutils-SIG maillist - Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Clarifying the meaning of the sysconfig "data" directory
On Saturday, December 07, 2013 06:19:42 PM Nick Coghlan wrote:
> In the binary dependency management threads, Daniel pointed out that
> in all currently defined sysconfig schemes, the "data" directory will
> end up pointing to the target installation directory. The "data" name
> in the scheme doesn't actually mean "this is a data file", it means
> "this has no defined semantic meaning at the sysconfig level, so just
> put it exactly where the relative path says to put it".
>
> That means that you can ship relatively arbitrary software in a wheel
> file by dumping it in "{distribution}-{version}.data/data/", and
> then installing it to the appropriate target location (e.g. if you use
> FHS paths inside the data directory, then you would just need to
> install the wheel as "pip install --root /" to get things into the
> desired location).
No need for --root. What people use (those that have to get work done instead
of discussing
semantics and stuff :) ) is this:
data_files = [
('sbin/', [ ... ]),
('/etc', []),
('/var/lib', [...]),
# I have seen this too
('../etc', [])
That naturally (sometimes) breaks installation into prefixes != /usr.
They do it because python packaging tools leave them alone on anything this is
not script and
package . it does not even have a way to install into the correct
/usr/share/... (and windows
equivalent).
And i have seen people setting the install_data dir in setup.cfg to something
like
share/ if they only needed to install something there. To shorten
their paths in
setup.py.
Setuptools needs fhs support. Done in a platform compatible way.
Mike
--
Michael Jansen
http://michael-jansen.biz
___
Distutils-SIG maillist - Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Install a script to /sbin instead of /bin
On 7 December 2013 10:12, Michael Jansen wrote: >> I made a proof of concept implementation (not ready to be merged) here > > No one interested in that? Got no answer from the maintainers too. Personally, I'm against the idea on principle, because the bin/sbin distinction is not a cross-platform concept (Windows does not have it). For the core tools, I'd prefer *not* to include tools that encourage platform-specific behaviour (I've seen too many projects in the past which would work perfectly on Windows except for something like this). As a 3rd-party setuptools extension providing utilities that make writing POSIX-specific packages easier, I see no problem with it, though. Paul ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Clarifying the meaning of the sysconfig "data" directory
On Saturday, December 07, 2013 10:26:15 AM Paul Moore wrote:
> On 7 December 2013 08:19, Nick Coghlan wrote:
> > That means that you can ship relatively arbitrary software in a wheel
> > file by dumping it in "{distribution}-{version}.data/data/", and
> > then installing it to the appropriate target location (e.g. if you use
> > FHS paths inside the data directory, then you would just need to
> > install the wheel as "pip install --root /" to get things into the
> > desired location).
> >
> > However, you'd be forgiven for not realising that from the sysconfig
> > docs - I certainly didn't realise it until Daniel pointed it out. From
> > the sysconfig docs, I had expected "data" to mean "application data
> > files" not "arbitrary, potentially executable, file with no assigned
> > semantics".
>
> The problem with what you're proposing (which isn't a change, as you
> say, so much as a subtle reinterpretation) is that if you include
> something in a wheel under /etc or /usr/include, you *are*
> assigning semantics to it - and the semantics you are assigning is
> POSIX. That's not a disaster, of course, but can make for a less than
> ideal layout on Windows. For example, if packages started putting
> config data under "etc" that would not be normal Windows practice.
> It's not *bad* per se, just disconcerting (and TBH, when I've seen
> this done, it gives the impression of a lazily-ported Unix package).
>
> Also, this paves the way for people starting to put data files under
> "/usr/share" or whatever the FHS location is. That is definitely
> *not* the Python convention, which is to put the data inside the
> package and locate it via package_data(). It also makes the package
> fail when used zipped or via tools like py2exe.
I guess we agree here. But people already do that because they have no other
choice. Nick only
wants to document the way it works better. Many people already figured this out
and use it
anyway.
And its very hard to use data_files in a cross platform way. If you want that
you are out of luck.
You can't use it easily. You would have to check the platform and adapt pathes
yourself.
See yourself.
http://code.ohloh.net/search?s=setup%20data_files&p=4&pp=0&mp=1&ml=1&me=1&md=1&ff=1&;
filterChecked=true
Mike
___
Distutils-SIG maillist - Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Install a script to /sbin instead of /bin
On Saturday, December 07, 2013 10:56:16 AM Paul Moore wrote: > On 7 December 2013 10:12, Michael Jansen wrote: > >> I made a proof of concept implementation (not ready to be merged) here > > > > No one interested in that? Got no answer from the maintainers too. > > Personally, I'm against the idea on principle, because the bin/sbin > distinction is not a cross-platform concept (Windows does not have > it). For the core tools, I'd prefer *not* to include tools that > encourage platform-specific behaviour (I've seen too many projects in > the past which would work perfectly on Windows except for something > like this). > > As a 3rd-party setuptools extension providing utilities that make > writing POSIX-specific packages easier, I see no problem with it, > though. My next step would be to implement support for fhs in the way that you can specify it like that '$bindir/bin_script', '$sbindir/sbin_script', ['$sbindir', ['sbin/sbin_script']], ['$bindir', ['bin/bin_script2']], ['$whateverwwwisnamedinfhs/myprefix', ['bin/mymod.wsgi']] And then setting those variables in cross platform compatible ways. Mike -- Michael Jansen http://michael-jansen.biz ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Clarifying the meaning of the sysconfig "data" directory
On 7 December 2013 11:10, Michael Jansen wrote: > I guess we agree here. But people already do that because they have no other > choice. Nick only wants to document the way it works better. Many people > already figured this out and use it anyway. I believe it's an "attractive nuisance" and should not be used except where necessary. While I'm not a fan of "security by obscurity", the current documentation limits the damage because only people who have a real need will investigate it and find out that it works. I'd rather we had a better solution that helped people do things in a cross-platform way, but in the absence of that having something that works, but isn't promoted, is a reasonable status quo. I have yet to see a use of that isn't specifically for Unix/FHS conformance. Those uses are fine, but they are *not* portable to Windows (often they work to an extent, but there are issues - e.g., they are unnatural, and they don't work in zipfile or py2exe formats). To get a standard solution, we need a proper discussion between Windows users and the various camps from the POSIX users. That's been tried before, and it's a great way of wasting many months ;-) (It also needs more Windows users than are generally available - I'm opinionated, but my use cases are atypical :-)) I don't disagree with clarifying the docs, but let's keep the wording platform-neutral, and let's not sell it as the ultimate solution, just "better than nothing, for the moment". Paul ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Install a script to /sbin instead of /bin
On 7 December 2013 11:14, Michael Jansen wrote: > And then setting those variables in cross platform compatible ways. Well, OK, but there may not *be* a "cross platform compatible" answer - we could set sbin to the same as bin on Windows, but www is probably inside the Apache installation (or somewhere arbitrary if the user has configured things to non-default) on Windows, so there's no way Python can know this. The whole point here is that FHS gives you a way of naming things consistently, on a platform where absolue paths are the norm. On Windows, *relative* paths are the norm (often relative to the application installation directory by default) and applications' files are isolated from each other much more strongly. FHS doesn't really make sense in such an environment. But as I say, I don't have a problem with what you're doing as a POSIX solution. I just don't think we want to get into the huge cross-platform filesystem layout issue here. Just accept that it's not Windows-compatible and move on :-) Paul ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Install a script to /sbin instead of /bin
On Saturday, December 07, 2013 12:18:39 PM Paul Moore wrote: > On 7 December 2013 11:14, Michael Jansen wrote: > But as I say, I don't have a problem with what you're doing as a POSIX > solution. I just don't think we want to get into the huge > cross-platform filesystem layout issue here. Just accept that it's not > Windows-compatible and move on :-) So you consider anything that has parts that need to be installed into/under a web server as not windows compatible? Anything that wants to install pdf/html documentation? Manual Pages? Anything that needs configuration files which usually end up under /etc Anything that requires a spool file (/var/spool normally)? I am really just curious. How is this stuff solved under windows? Mike -- Michael Jansen http://michael-jansen.biz ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Install a script to /sbin instead of /bin
I really don't have all the answers here, but... On 7 December 2013 12:30, Michael Jansen wrote: > So you consider anything that has parts that need to be installed into/under > a web server as not windows compatible? Not at all. If you're talking about documents, what's to "install"? Just ask the user to drop them under the docroot. If they are something like an Apache DLL, just ask the user in the installer where Apache is located. Or something like that. There's just no answer that assumes everyone's installation is the same and you don't have to get information from the user. > Anything that wants to install pdf/html documentation? Manual Pages? Why install them. Just put them somewhere for the user to read. Add a start menu item if you want them accessible. > Anything that needs configuration files which usually end up under /etc Well, that's a leading question. There are no such configuration files on Windows because there's no /etc. But I know what you mean and the answer is put them next to the executable. Or in the registry if you believe in that. And generally give the user a configuration app rather than expect them to manually edit a file. > Anything that requires a spool file (/var/spool normally)? I can't think of anything like that, you'd have to give an example. I'm not sure the concept exists on Windows... Of course, all of the above are *applications*, not Python modules, so I wouldn't expect you to distribute them with setuptools/PyPI anyway. But that's another aspect of the issue. Paul ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Install a script to /sbin instead of /bin
On Saturday, December 07, 2013 01:16:27 PM Paul Moore wrote: > I really don't have all the answers here, but... > > On 7 December 2013 12:30, Michael Jansen wrote: > > So you consider anything that has parts that need to be installed > > into/under a web server as not windows compatible? > > Not at all. If you're talking about documents, what's to "install"? > Just ask the user to drop them under the docroot. If they are > something like an Apache DLL, just ask the user in the installer where > Apache is located. Or something like that. There's just no answer that > assumes everyone's installation is the same and you don't have to get > information from the user. If we add something like $wwwroot setup.py could notice this is used and then ask the user where to put it. pip/wheel too. Currently it all gets lumped somewhere under data_files and no one has any chance to automate this. On linux the default could be to check whatever is appropriate for the distribution (/srv/www or /var/www) and if those are not there bail out with the message you have to specify the correct path on the command line. That could be the default on windows. > > > Anything that wants to install pdf/html documentation? Manual Pages? > > Why install them. Just put them somewhere for the user to read. Add a > start menu item if you want them accessible. For me installing means putting them somewhere for the user to read in an automated, reliable and reproducable fashion from sources. So building them (with sphinx or whatever) and copying somewhere for the user to read is installing. > > Anything that needs configuration files which usually end up under /etc > > Well, that's a leading question. There are no such configuration files > on Windows because there's no /etc. But I know what you mean and the > answer is put them next to the executable. Or in the registry if you > believe in that. And generally give the user a configuration app > rather than expect them to manually edit a file. But when enhancing distutils with something like $etcdir we could support that easily with minimal work for the developer. On linux install into /etc, On windows install wherever it is ok to put those stuff (without asking the user) and then provide the script author with an easy way to find out where $etcdir is on the platform so he can find his stuff. Your point of view (i don't want to think about it. Solve it posix only) means people have to develop solutions for windows separately and on top of that each and every time manually (and most won't do the work) or the other way around. A windows package will not work on linux. > > Anything that requires a spool file (/var/spool normally)? > > I can't think of anything like that, you'd have to give an example. > I'm not sure the concept exists on Windows... Thats just a file where one process pumps data in that another process reads out and handles at his leisure. > > Of course, all of the above are *applications*, not Python modules, so > I wouldn't expect you to distribute them with setuptools/PyPI anyway. > But that's another aspect of the issue. Yes it is. And i agree with you that distributing something as cobblerd with pip, pypi is not really a good idea. But setuptools is the ONLY build system that works with python code so i HAVE to use it to build and install cobblerd from sources. So it should gain support to support applications too. Not only libraries and simple scripts. Anyway. You are at least willing to discuss with me. So thanks. I guess i give up on improving setuptools and instead just implement enough to fix cobblerd. After that i guess i will take with me the hope that the next time i have contact with python packaging the situation improved but i had that hope ten years ago too :( . Mike -- Michael Jansen http://michael-jansen.biz ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Install a script to /sbin instead of /bin
On 7 December 2013 13:45, Michael Jansen wrote: > Anyway. You are at least willing to discuss with me. So thanks. No problem! I hope I wasn't too discouraging - I get the impression that I was :-( My problem is that I don't know the answers on Windows, I only really know enough to know when something feels wrong. In this area, Unix and Windows really are very different, and it shows. Very few people are willing or able to put the effort into developing cross-platform solutions, and those that do are too busy doing it to worry about establishing general principles, etc. > I guess i give up on improving setuptools and instead just implement enough > to fix cobblerd. After that i guess i will take with me the hope that the > next time i have contact with python packaging the situation improved but i > had that hope ten years ago too :( . I can understand your situation, but please don't give up! The packaging community definitely needs people willing to help work on these issues. But it's much less about coding solutions, and more about negotiating compromises between people with very different viewpoints. To go back to your original issue (sbin vs bin) - from my point of view as a Windows developer, packages just export commands. Why would I choose to put them in the sbin list rather than the bin list? If that's not clear, then I'll possibly develop packages that don't work "right" on Unix. Let's thrash out the answer to that question first, and *then* worry about implementing it. Is the relevant point that "bin" is on PATH but sbin (typically) isn't? If both are on PATH, I don't see why you bother having two directories. So maybe the real idea here is to split entry points into "commands that should be on PATH" and "commands that should be provided, but not on PATH by default". Or is it? My understanding of sbin on Unix is that it's somehow "system utilities" - but that's just a classification with no practical impact. (What makes formatting a disk a "system utility"? I used to do it all the time when floppy disks were common.) Or maybe sbin is only on PATH when you're root. How does that translate to typical Windows systems, where there is usually only one user, and he has access to everything? Maybe in that context, sbin should be for utilities that require elevation? Lots of questions, but no obvious answers... As I said, and I'll say again, you don't *have* to answer these questions. But not doing so is not "I don't want to think about it, just solve POSIX". Rather it's "let's not make everything wait for a perfect solution when POSIX people need something now". But equally, let's not stop working on the cross-platform solution because the POSIX people have something and no longer care - that's *why* the problem never gets solved properly. One other thing I should say here - you mention your project is "cobblerd". The "d" on the end makes me thing it's a daemon. That's inherently POSIX only - on Windows this would be a service (which is a whole other beast). Actually looking at Cobbler on PyPI it says "Cobbler is a Linux boot server..." so Windows compatibility really isn't crucial in this particular instance, I guess :-) Paul ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Clarifying the meaning of the sysconfig "data" directory
As part of the docs clarification, I was planning to point out that such layouts are almost always platform (and even distro) specific. That's still a valid use case for sdist and wheel, though, so it makes sense to me to document it properly. My other related question is whether it's possible to force bdist_wheel to include the platform tag as part of the build process, since such wheels *should* be flagged as platform specific. Cheers, Nick. ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Install a script to /sbin instead of /bin
FWIW, tackling FHS compliance is one of the things I have in mind as being suitable for a metadata 2.0 extension. Because Mac OS X and Windows just bundle (almost) everything into the application directory, FHS concepts simply don't map properly, so the current best FHS compliant solution is to use the "data" scheme path, FHS subdirectories and flag the project as Linux specific in the docs (or just not publish it on PyPI). Alternatively, cross platform Python code sometimes just ignores FHS compliance and bundles everything in with the module code (using pkgutil.get_data to retrieve it at runtime). Getting such packages ready for distro inclusion then ends up being quite a pain :P Cheers, Nick. ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Install a script to /sbin instead of /bin
On Saturday, December 07, 2013 03:15:15 PM Paul Moore wrote: > On 7 December 2013 13:45, Michael Jansen wrote: > I can understand your situation, but please don't give up! The > packaging community definitely needs people willing to help work on > these issues. But it's much less about coding solutions, and more > about negotiating compromises between people with very different > viewpoints. And thats where i disagree. I have some experience in open source contributions (KDE, cobbler and my own). 1. Usable code always wins. Always. Ideas and Agreements solve nothing. 2. This is a already solved problem. Both autotools and even more so cmake make it possible to develop cross platform. Why python should be differently is something i won't understand. If you want to wait for a compromise and an agreement nothing will ever be implemented. > To go back to your original issue (sbin vs bin) - from my point of > view as a Windows developer, packages just export commands. Why would > I choose to put them in the sbin list rather than the bin list? If > that's not clear, then I'll possibly develop packages that don't work > "right" on Unix. Let's thrash out the answer to that question first, > and *then* worry about implementing it. Is the relevant point that > "bin" is on PATH but sbin (typically) isn't? If both are on PATH, I > don't see why you bother having two directories. So maybe the real > idea here is to split entry points into "commands that should be on > PATH" and "commands that should be provided, but not on PATH by > default". Or is it? My understanding of sbin on Unix is that it's > somehow "system utilities" - but that's just a classification with no > practical impact. (What makes formatting a disk a "system utility"? I > used to do it all the time when floppy disks were common.) Or maybe > sbin is only on PATH when you're root. How does that translate to > typical Windows systems, where there is usually only one user, and he > has access to everything? Maybe in that context, sbin should be for > utilities that require elevation? Lots of questions, but no obvious > answers... A windows developer should not care about the difference. Both end up in the same directory. A unix developer will either not care too (and be reminded by the distro) or has read http://www.pathname.com/fhs/pub/fhs-2.3.html#SBINSYSTEMBINARIES So the not caring unix/windows developer will one day get a bug report/patch from a unix user or distro packager if some script belongs to sbin. > As I said, and I'll say again, you don't *have* to answer these > questions. But not doing so is not "I don't want to think about it, > just solve POSIX". Rather it's "let's not make everything wait for a > perfect solution when POSIX people need something now". But equally, > let's not stop working on the cross-platform solution because the > POSIX people have something and no longer care - that's *why* the > problem never gets solved properly. Thats why i prefer to improve step by step. The sbin/bin distinction can be solved in a platform neutral way so why not do it. > One other thing I should say here - you mention your project is > "cobblerd". The "d" on the end makes me thing it's a daemon. That's > inherently POSIX only - on Windows this would be a service (which is a > whole other beast). Actually looking at Cobbler on PyPI it says > "Cobbler is a Linux boot server..." so Windows compatibility really > isn't crucial in this particular instance, I guess :-) It most likely will never run on windows but i have seen weirder things (productive clearcase running on windows xp *shudder*). But there is nothing that prevents it from ever running on windows. So if i solved the multi distro support there is nothing that prevents someone to make it running under windows. Why shouldn't you boot and provision your unix virtual machines with a server running on windows? But i guess its unlikely. And its not my project. I only contribute since about three weeks. My homepage says something about "drive-by coding" and thats what i do. I like the project and noticed it currently only runs on redhat and installed into /usr. There are people using it on sles (its a rather enterprisey thing) which use patches. So i solved all obstacles to get it running out of the box (with correct paths) on opensuse already and thought i could do another attempt to get those improvements that make sense into setuptools. But i think that won't happen. And it was not you. I like to discuss and i always applaud people that help other brainstorm. Anyway. I will solve my problems on top of setuptools for cobblerd only for now. Then i will think about what to do with the results. I need and partially have solution for: - configure some files (patch @@var@@) (implemented) - eg. patch VIRTUALENV into a apache config file for wsgi virtualenv support - patch some distro / platform specific paths in config files. - instal
Re: [Distutils] Install a script to /sbin instead of /bin
On 7 December 2013 16:56, Michael Jansen wrote: > A windows developer should not care about the difference. Both end up in the > same directory. [...] will one day get a bug report/patch from a unix user OK, cool. That makes sense. Paul ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
[Distutils] Setuptools 2.0 Released
The PyPA is pleased to announce the Setuptools 2.0 release. This backward-incompatible release drops support for Python 2.4 and Python 2.5, but is otherwise compatible with 1.x releases. The 1.x series will continue to be supported for bug and security fixes for the foreseeable future. This means that any systems with older versions of Python should install "setuptools<2dev" or use the legacy bootstrap script, available at https://bitbucket.org/pypa/setuptools/raw/bootstrap-py24/ez_setup.py for Python 2.4 and Python 2.5 users. That link is also published in the README as install instructions. Please report any emergent issues at https://bitbucket.org/pypa/setuptools/issues. Jason PyPA ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Clarifying the meaning of the sysconfig "data" directory
> As part of the docs clarification, I was planning to point out that such > layouts are almost always platform (and even distro) specific. That's still > a valid use case for sdist and wheel, though, so it makes sense to me to > document it properly. > > My other related question is whether it's possible to force bdist_wheel to > include the platform tag as part of the build process, since such wheels > *should* be flagged as platform specific. > that would be nice at least for absolute paths (like "/etc"), or relative paths that go up the folder hierarchy into who knows where. for non-upward relative paths, it's less clear whether it should be marked platform specific ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Clarifying the meaning of the sysconfig "data" directory
actually, bdist_wheel doesn't handle absolute paths in the "data_files"
keyword like standard setuptools installs do, which honor it as absolute
(which seems to match the examples in the distutils docs)
when using absolute paths, the data ends up in the packaged wheel at the
top level (not in the {distribution}-{version}.data folder).
And get's installed relative to site-packages (along with the project
packages)
so, bdist_wheel is re-interpreting distutil's "data_files" differently.
maybe better for wheel to fail to build projects with absolute data_files,
than to just reinterpret it like that.
On Sat, Dec 7, 2013 at 12:48 PM, Marcus Smith wrote:
>
> As part of the docs clarification, I was planning to point out that such
>> layouts are almost always platform (and even distro) specific. That's still
>> a valid use case for sdist and wheel, though, so it makes sense to me to
>> document it properly.
>>
>> My other related question is whether it's possible to force bdist_wheel
>> to include the platform tag as part of the build process, since such wheels
>> *should* be flagged as platform specific.
>>
> that would be nice at least for absolute paths (like "/etc"), or relative
> paths that go up the folder hierarchy into who knows where.
> for non-upward relative paths, it's less clear whether it should be marked
> platform specific
>
>
>
___
Distutils-SIG maillist - Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Clarifying the meaning of the sysconfig "data" directory
https://bitbucket.org/dholth/wheel/issue/92/bdist_wheel-makes-absolute-data_files
On Sat, Dec 7, 2013 at 1:32 PM, Marcus Smith wrote:
> actually, bdist_wheel doesn't handle absolute paths in the "data_files"
> keyword like standard setuptools installs do, which honor it as absolute
> (which seems to match the examples in the distutils docs)
> when using absolute paths, the data ends up in the packaged wheel at the
> top level (not in the {distribution}-{version}.data folder).
> And get's installed relative to site-packages (along with the project
> packages)
>
> so, bdist_wheel is re-interpreting distutil's "data_files" differently.
> maybe better for wheel to fail to build projects with absolute data_files,
> than to just reinterpret it like that.
>
>
>
>
>
>
> On Sat, Dec 7, 2013 at 12:48 PM, Marcus Smith wrote:
>
>>
>> As part of the docs clarification, I was planning to point out that such
>>> layouts are almost always platform (and even distro) specific. That's still
>>> a valid use case for sdist and wheel, though, so it makes sense to me to
>>> document it properly.
>>>
>>> My other related question is whether it's possible to force bdist_wheel
>>> to include the platform tag as part of the build process, since such wheels
>>> *should* be flagged as platform specific.
>>>
>> that would be nice at least for absolute paths (like "/etc"), or relative
>> paths that go up the folder hierarchy into who knows where.
>> for non-upward relative paths, it's less clear whether it should be
>> marked platform specific
>>
>>
>>
>
___
Distutils-SIG maillist - Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Clarifying the meaning of the sysconfig "data" directory
On 7 December 2013 21:32, Marcus Smith wrote:
> actually, bdist_wheel doesn't handle absolute paths in the "data_files"
> keyword like standard setuptools installs do, which honor it as absolute
> (which seems to match the examples in the distutils docs)
> when using absolute paths, the data ends up in the packaged wheel at the top
> level (not in the {distribution}-{version}.data folder).
> And get's installed relative to site-packages (along with the project
> packages)
>
> so, bdist_wheel is re-interpreting distutil's "data_files" differently.
> maybe better for wheel to fail to build projects with absolute data_files,
> than to just reinterpret it like that.
+1. It seems to me (from a Windows user's perspective) that projects
that use absolute paths (or relative paths that go above the
installation root) are fundamentally different from ones that don't.
For a start they feel more like "applications" than "Python packages".
Maybe what we want is a sysconfig path "root" that explicitly encodes
the filesystem root. Then we can require that *all* paths are relative
to some sysconfig path, and must not use ".." to go "up" from there.
Projects that use the "root" path can be clearly noted as not
wheel-compatible (and quite likely not cross-platform).
By making it an explicit path, we can make the policy explicit and
provide better error messages and metadata.
Paul
PS Please note that as a Windows user, I don't have any practical
experience of projects with requirements for absolue paths - they are
alien to the Windows world. So my views should be taken in that light
(i.e, as being of limited value :-)).
___
Distutils-SIG maillist - Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Install a script to /sbin instead of /bin
Just a note here: the wxWidgets (and thus wxPython, natch) project has a wxStandardPaths object: http://docs.wxwidgets.org/trunk/classwx_standard_paths.html It provides a cross platform way to get, well, the standard paths an application might need: GetAppDocumentsDir () GetConfigDir () GetDataDir () GetDocumentsDir () GetExecutablePath () GetInstallPrefix () GetLocalDataDir () GetLocalizedResourcesDir () GetPluginsDir () GetResourcesDir () GetTempDir () GetUserConfigDir () GetUserDataDir () GetUserLocalDataDir () . These all "do the right thing" on the supported platforms -- something may be in the application install dir on Windows, int he app bundle on the MAc, and it /etc on Linux, for instance. granted, wx is an application framework, so it needs this stuff, but while pip is about installing python packages, there really is no clear line between at package with a script or two and an application, and even simple scripts may need a few of these things. I don't see why we couldn't include a set like this, and have platform-specific mappings. -Chris -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R(206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception chris.bar...@noaa.gov ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] More control over uploading an sdist and egg to PyPI.
Nick Coghlan gmail.com> writes: > Hi Kanaan, > > I believe twine (https://pypi.python.org/pypi/twine/) should work as > the independent upload utility that you're looking for. > > Cheers, > Nick. That's perfect, thanks. (Also thanks to Daniel Holth who replied off-list.) Another great thing about Twine is that it lets you do the GPG signing yourself. This is important for us since our release signing key isn't in each user's keyring but it is in a separate keyring file. ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Clarifying the meaning of the sysconfig "data" directory
On 8 December 2013 08:22, Paul Moore wrote:
> On 7 December 2013 21:32, Marcus Smith wrote:
>> actually, bdist_wheel doesn't handle absolute paths in the "data_files"
>> keyword like standard setuptools installs do, which honor it as absolute
>> (which seems to match the examples in the distutils docs)
>> when using absolute paths, the data ends up in the packaged wheel at the top
>> level (not in the {distribution}-{version}.data folder).
>> And get's installed relative to site-packages (along with the project
>> packages)
>>
>> so, bdist_wheel is re-interpreting distutil's "data_files" differently.
>> maybe better for wheel to fail to build projects with absolute data_files,
>> than to just reinterpret it like that.
>
> +1. It seems to me (from a Windows user's perspective) that projects
> that use absolute paths (or relative paths that go above the
> installation root) are fundamentally different from ones that don't.
> For a start they feel more like "applications" than "Python packages".
This started with my suggestion to create a separate directory in in
the wheel format for bundling arbitrary other files:
https://bitbucket.org/pypa/pypi-metadata-formats/issue/13/add-a-new-subdirectory-to-allow-wheels-to
Daniel initially suggested that such a thing would be redundant with
the existing "data" directory in sysconfig, but given the subsequent
discussions, I no longer think that is true.
A new scheme directory would be a simpler solution than changing the
wheel format, though.
> Maybe what we want is a sysconfig path "root" that explicitly encodes
> the filesystem root.
s/filesystem root/installation root/
I'm assuming that's what you meant, since you'd know better than I do
that "filesystem root" isn't a particularly useful concept on Windows
:)
I guess "drive root" would be another way to interpret it, but that
doesn't seem useful either.
> Then we can require that *all* paths are relative
> to some sysconfig path, and must not use ".." to go "up" from there.
> Projects that use the "root" path can be clearly noted as not
> wheel-compatible (and quite likely not cross-platform).
Well, they'd be wheel compatible - the wheels would just need to be
marked as platform specific.
> By making it an explicit path, we can make the policy explicit and
> provide better error messages and metadata.
Agreed, it sounds like "data" is currently being used to mean two
different things, and we need to define a new scheme directory to
separate out the FHS compliance use case.
> Paul
>
> PS Please note that as a Windows user, I don't have any practical
> experience of projects with requirements for absolue paths - they are
> alien to the Windows world. So my views should be taken in that light
> (i.e, as being of limited value :-)).
Yeah, absolute paths are mostly about playing nice with FHS, which is
mostly about playing nice with standardised system administration
tools. For a server:
- /var/* is the mutable data you need to back up
- /etc/* is the config to be controlled by your config management system
- most everything else should be controlled by the system package manager
Distro packagers and system administrators get grumpy when upstream
software doesn't play by those rules, because it makes things harder
to audit, harder to back up, and just generally harder to administer.
Perhaps rather than trying to be generic with a name like "root", we
could just define an explicit "fhs" path?
Cheers,
Nick.
--
Nick Coghlan | ncogh...@gmail.com | Brisbane, Australia
___
Distutils-SIG maillist - Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig
