Re: Tool support for private modules
Josselin Mouette [EMAIL PROTECTED] writes: Le jeudi 11 octobre 2007 à 10:50 +1000, Ben Finney a écrit : The main reason I use distutils is to assist those people using operating systems that *don't* have good package dependency management, which seems to be the primary target market for setuptools. This is a laudable goal, but not when done at the expense of proper support of operating systems which have one. Indeed. The rest of the message, which you chose not to address this time, asks for help avoiding exactly that trap. Care to answer some of the specific questions in that message and help Python packagers improve their practices? -- \ Remember men, we're fighting for this woman's honour; which is | `\probably more than she ever did. -- Groucho Marx | _o__) | Ben Finney -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of unsubscribe. Trouble? Contact [EMAIL PROTECTED]
Re: Tool support for private modules
Bernd, please follow the Debian mailing list code of conduct URL:http://www.debian.org/MailingLists#codeofconduct, in particular by *not* sending personal copies of messages that are also sent to the list. Also, please preserve attribution lines so we can keep track of who wrote what quoted material. Bernd Zeimetz [EMAIL PROTECTED] writes: As an example, here's a Python package I'm trying to get packaged for Debian. [...] URL:http://cheeseshop.python.org/pypi/gracie/ The first thing I'd do here is to patch the ez_ stuff away, together with setuptools. I presume you mean the 'ez_setup.py' module (there's no other 'ez_*' files in that package). ez_... is known to break things badly (like trying to download things on buildds and other really broken things). Setuptools is broken by design imho (see a thread some time ago about this), but it may work. But as I run into trouble with it too often (like missing __init__.py files in random directories), I replace it by distutils. The main reason I use distutils is to assist those people using operating systems that *don't* have good package dependency management, which seems to be the primary target market for setuptools. I also want my package listed properly at the Python Cheeseshop; this is much easier using setuptools than distutils. Since we're not building eggs there's not need for setuptools at all (afaik distutils is able to build eggs now, too - at least the egg info files, which is enough for us). Better to patch those things before getting FTBFS reports form the buildds. Okay. So you're suggesting that I should continue to maintain the setuptools functionality upstream, but patch it out in the Debian package of the same software? I'm also unclear on what you mean by replace it by distutils. What does this mean for a package that already uses setuptools, and will continue to do so upstream? Although I didn't test it, there should be no special thign to do with your setup.py while building a package, python setup.py build/install --root= should do the job (with and without ez_... and setuptools). I'll address this in a separate thread, as it seems I'm not explaining the problem very well. -- \ I bought some powdered water, but I don't know what to add. | `\ -- Steven Wright | _o__) | Ben Finney -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of unsubscribe. Trouble? Contact [EMAIL PROTECTED]
Re: Tool support for private modules
Ben Finney [EMAIL PROTECTED] writes: The main reason I use distutils is to assist those people using operating systems that *don't* have good package dependency management, which seems to be the primary target market for setuptools. This should, of course, read The main reason I use setuptools … -- \ Pinky, are you pondering what I'm pondering? I think so, | `\Brain, but this time *you* put the trousers on the chimp. -- | _o__)_Pinky and The Brain_ | Ben Finney -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of unsubscribe. Trouble? Contact [EMAIL PROTECTED]
Re: Tool support for private modules
Le lundi 01 octobre 2007 à 23:56 +1000, Ben Finney a écrit : Hmm. I am hoping that modify the programs [to add an absolute path to the search path] is not a necessary part of this. If upstream hasn't thought of it, it is. You only need to add one line in the program, before the module is imported. How can upstream anticipate the arbitrary path I pass to './setup.py install --home=/foo/bar' if the path /foo/bar is up to me as a Debian packager? I don't see how any Python program can be written to allow for that without modification every time the /foo/bar changes. Distutils may be extended to do such a thing, but most of the software I've seen doing it is using automake or changing the files with custom hackery. -- .''`. Josselin Mouette/\./\ : :' : [EMAIL PROTECTED] `. `'[EMAIL PROTECTED] `- Debian GNU/Linux -- The power of freedom signature.asc Description: Ceci est une partie de message numériquement signée
Re: Tool support for private modules
(Please preserve attribution lines on quoted material. I don't know who wrote what in the following.) Josselin Mouette [EMAIL PROTECTED] writes: Le lundi 01 octobre 2007 à 23:56 +1000, Ben Finney a écrit : Hmm. I am hoping that modify the programs [to add an absolute path to the search path] is not a necessary part of this. If upstream hasn't thought of it, it is. You only need to add one line in the program, before the module is imported. [...] I don't see how any Python program can be written to allow for that without modification every time the /foo/bar changes. Distutils may be extended to do such a thing, but most of the software I've seen doing it is using automake or changing the files with custom hackery. I'm confused, then. How does this fit with the implication (in the above quoted teset) that upstream should have thought of [changing the location of the modules]? If downstream hackery is required, I don't see what upstream is expected to have done. -- \The most merciful thing in the world... is the inability of | `\ the human mind to correlate all its contents. -- Howard | _o__)Philips Lovecraft | Ben Finney -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of unsubscribe. Trouble? Contact [EMAIL PROTECTED]
Re: Tool support for private modules
Le lundi 01 octobre 2007 à 14:42 +1000, Ben Finney a écrit : 3.1.1 Programs Shipping Private Modules A program using /usr/bin/python as interpreter can come up with private Python modules. These modules should be installed in /usr/share/module, or /usr/lib/module if the modules are architecture-dependent (e.g. extensions). /usr/lib/site-python is deprecated and should no longer be used for this purpose. I think the policy is a bit confusing here. The purpose of /usr/lib/site-python has never been to ship private modules, but rather to ship public modules supposed to work with any python version. This is now handled by python-support or python-central instead. But the Python distutils and setuptools will install the modules to /usr/lib/site-python/. Hrm, they shouldn't. With a default setup, public modules are shipped to /usr/lib/python2.X/site-packages. My 'debian/rules' already has a 'dh_pycentral' line. That would work if the files were shipped in /usr/lib/python2.X/site-packages. How can I best conform to the above policy? The dumb way, I imagine, would be to manually install the foles to those specific locations. I'm wondering what tools (e.g. debhelper) support something better, and how to use them to conform to the policy. If the modules are indeed private, it looks like you need to change the path by hand, and to add it with sys.path.append(/the/path) at the beginning of the binary. If, as the location suggests, they are public, you should look at the setup.py file to see if there's something unusual that would install the modules in this unusual location. -- .''`. Josselin Mouette/\./\ : :' : [EMAIL PROTECTED] `. `'[EMAIL PROTECTED] `- Debian GNU/Linux -- The power of freedom signature.asc Description: Ceci est une partie de message numériquement signée
Re: Tool support for private modules
Josselin Mouette [EMAIL PROTECTED] writes: Le lundi 01 octobre 2007 à 14:42 +1000, Ben Finney a écrit : But the Python distutils and setuptools will install the modules to /usr/lib/site-python/. Hrm, they shouldn't. With a default setup, public modules are shipped to /usr/lib/python2.X/site-packages. Yes, my apologies, that's what the tools do. I've no idea why I typed that unrelated path above. My 'debian/rules' already has a 'dh_pycentral' line. That would work if the files were shipped in /usr/lib/python2.X/site-packages. That's where the distutils and setuptools place them by default, yes. I don't know what magic is required to put them elsewhere; that may be part of the answer, if someone can instruct me on how to do it. The trouble is, these are modules that clearly fall under the private modules for the program description in the policy document. I fully agree with the policy that modules intended for internal use by a discrete set of programs should not be installed to the public site-packages directory. How can I use the tools available — distutils, setuptools, debhelper — to install these package-specific modules to a package-specific location, such that all the programs in the package will be able to find them? If, as the location suggests, they are public The location does indeed suggest that, but AFAICT that's only because both distutils and setuptools makes no distinction between modules intended for general use on the system and modules only intended for use by specific programs. If the modules are indeed private, it looks like you need to change the path by hand, and to add it with sys.path.append(/the/path) at the beginning of the binary. Hmm. I am hoping that modify the programs is not a necessary part of this. -- \ He who laughs last, thinks slowest. -- Anonymous | `\ | _o__) | Ben Finney -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of unsubscribe. Trouble? Contact [EMAIL PROTECTED]
Re: Tool support for private modules
That's where the distutils and setuptools place them by default, yes. I don't know what magic is required to put them elsewhere; that may be part of the answer, if someone can instruct me on how to do it. You shoul dupload your work somewhere, teaching you is almost impossible if one can't see what's wrong. -- Bernd Zeimetz [EMAIL PROTECTED] http://bzed.de/ -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of unsubscribe. Trouble? Contact [EMAIL PROTECTED]
Re: Tool support for private modules
Bernd Zeimetz [EMAIL PROTECTED] writes: You shoul dupload your work somewhere, teaching you is almost impossible if one can't see what's wrong. I'm not presenting something as wrong. I'm asking for guidance on how to do things right. If the policy recommends that packages be set up a particular way (put package-specific modules in '/usr/share/package/'), but the standard tools behave differently (put all modules by default in '/usr/lib/pythonX.Y/site-packages/'), then there's a step that needs to be taken to get from the default behaviour to the behaviour recommended by policy. I'm asking how to take that step, in a way that isn't brute-force manually hack each package to conform to policy. -- \ Dvorak users of the world flgkd! —Kirsten Chevalier, | `\rec.humor.oracle.d | _o__) | Ben Finney -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of unsubscribe. Trouble? Contact [EMAIL PROTECTED]
Re: Tool support for private modules
If the policy recommends that packages be set up a particular way (put package-specific modules in '/usr/share/package/'), but the standard tools behave differently (put all modules by default in '/usr/lib/pythonX.Y/site-packages/'), then there's a step that needs to be taken to get from the default behaviour to the behaviour recommended by policy. The standard tools Do it right. So there must be a bug somewhere in your packaging or upstream's source. Unfortunately this bug is not shown in my fishbowl. You can read trough http://wiki.debian.org/DebianPython/NewPolicy and check if your packaging contains all the pieces mentioned there. If you think it does, show your code. -- Bernd Zeimetz [EMAIL PROTECTED] http://bzed.de/ -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of unsubscribe. Trouble? Contact [EMAIL PROTECTED]
Re: Tool support for private modules
Le lundi 01 octobre 2007 à 18:37 +1000, Ben Finney a écrit : That would work if the files were shipped in /usr/lib/python2.X/site-packages. That's where the distutils and setuptools place them by default, yes. I don't know what magic is required to put them elsewhere; that may be part of the answer, if someone can instruct me on how to do it. The trouble is, these are modules that clearly fall under the private modules for the program description in the policy document. I fully agree with the policy that modules intended for internal use by a discrete set of programs should not be installed to the public site-packages directory. As long as they are using their own namespace without a too generic name, it's not *that* bad. It is not as if no other package was doing such things. How can I use the tools available — distutils, setuptools, debhelper — to install these package-specific modules to a package-specific location, such that all the programs in the package will be able to find them? The easiest way, if the modules are relocatable (99% of them are) is to simply move them after installation. Otherwise, you can pass specific arguments to setup.py. That would be, python setup.py install --home=/usr/share/$package --install-purelib=. More information: http://www.python.org/doc/2.4/inst/search-path.html If, as the location suggests, they are public The location does indeed suggest that, but AFAICT that's only because both distutils and setuptools makes no distinction between modules intended for general use on the system and modules only intended for use by specific programs. Well, in fact they do, but upstream developers generally simply ship public modules. If the modules are indeed private, it looks like you need to change the path by hand, and to add it with sys.path.append(/the/path) at the beginning of the binary. Hmm. I am hoping that modify the programs is not a necessary part of this. If upstream hasn't thought of it, it is. You only need to add one line in the program, before the module is imported. Cheers, -- .''`. Josselin Mouette/\./\ : :' : [EMAIL PROTECTED] `. `'[EMAIL PROTECTED] `- Debian GNU/Linux -- The power of freedom signature.asc Description: Ceci est une partie de message numériquement signée
Re: Tool support for private modules
Josselin Mouette [EMAIL PROTECTED] writes: Le lundi 01 octobre 2007 à 18:37 +1000, Ben Finney a écrit : How can I use the tools available — distutils, setuptools, debhelper — to install these package-specific modules to a package-specific location, such that all the programs in the package will be able to find them? The easiest way, if the modules are relocatable (99% of them are) is to simply move them after installation. Otherwise, you can pass specific arguments to setup.py. That would be, python setup.py install --home=/usr/share/$package --install-purelib=. More information: http://www.python.org/doc/2.4/inst/search-path.html Thanks, this is a useful pointer. I wasn't aware such fine-grained control was available over locations with distutils. (I've certainly never seen any Python author using that control.) Hmm. I am hoping that modify the programs [to add an absolute path to the search path] is not a necessary part of this. If upstream hasn't thought of it, it is. You only need to add one line in the program, before the module is imported. How can upstream anticipate the arbitrary path I pass to './setup.py install --home=/foo/bar' if the path /foo/bar is up to me as a Debian packager? I don't see how any Python program can be written to allow for that without modification every time the /foo/bar changes. -- \ Everything you read in newspapers is absolutely true, except | `\for that rare story of which you happen to have first-hand | _o__) knowledge. -- Erwin Knoll | Ben Finney -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of unsubscribe. Trouble? Contact [EMAIL PROTECTED]
Re: Tool support for private modules
Ben Finney [EMAIL PROTECTED] writes: How can I best conform to the [Debian policy for Python modules specific to a single package]? As an example, here's a Python package I'm trying to get packaged for Debian. (I am the upstream author of this one, but I'm interested in a solution that *doesn't* involve significant changes to the upstream code.) URL:http://cheeseshop.python.org/pypi/gracie/ How should I modify 'setup.py' to allow binaries and modules for this package to be installed properly, and have the programs continue to find the modules? How should I construct the command line for 'setup.py' when I create the 'install-python%' target in 'debian/rules' for this package? -- \ Eccles: I just saw the Earth through the clouds! Lew: Did | `\ it look round? Eccles: Yes, but I don't think it saw me. | _o__) -- The Goon Show, _Wings Over Dagenham_ | Ben Finney -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of unsubscribe. Trouble? Contact [EMAIL PROTECTED]
Re: Tool support for private modules
As an example, here's a Python package I'm trying to get packaged for
Debian. (I am the upstream author of this one, but I'm interested in a
solution that *doesn't* involve significant changes to the upstream
code.)
URL:http://cheeseshop.python.org/pypi/gracie/
The first thing I'd do here is to patch the ez_ stuff away, together
with setuptools. ez_... is known to break things badly (like trying to
download things on buildds and other really broken things). Setuptools
is broken by design imho (see a thread some time ago about this), but it
may work. But as I run into trouble with it too often (like missing
__init__.py files in random directories), I replace it by distutils.
Since we're not building eggs there's not need for setuptools at all
(afaik distutils is able to build eggs now, too - at least the egg info
files, which is enough for us).
Better to patch those things before getting FTBFS reports form the buildds.
Although I didn't test it, there should be no special thign to do with
your setup.py while building a package, python setup.py build/install
--root= should do the job (with and without ez_... and setuptools).
http://svn.debian.org/wsvn/python-modules/packages/python-contract/trunk/debian/rules?op=filerev=0sc=0
is a pretty simple debian/rules file, which also tries to build the
package for all python versions and runs the tests to ensure nothing is
buggy.
If you like it really simple, cdbs is something you can use:
http://svn.debian.org/wsvn/python-modules/packages/ll-core/trunk/debian/rules?op=filerev=0sc=0
Aa you can see on the cdbs example, sometimes upstream does not install
all files in the way one would like it to have in debian, often out of
personal taste - that's easy to fix though. Although at least I
appreciate if upstras isntalls everything at the proper place, at least
distutils supports installing completely non-py related files into the
right directories.
Depending on what you like you have to call dh_py{central,support} to
move your installed files in their directories, of course you need to
add the neccessary informations for them at the apropriate places (like
which python versions you want to support).
How should I construct the command line for 'setup.py' when I create
the 'install-python%' target in 'debian/rules' for this package?
Have a look at the first example.
Both will work for arch: all and arch: any packages. For arch: all
packages you don;t need to build/install with all python versions, just
using the default one is enough, everythign else will be handled by
python-support/central while installing the package. It makes sense to
build them all, though, especially when you want to run tests.
Hope that helps,
Bernd
--
Bernd Zeimetz
[EMAIL PROTECTED] http://bzed.de/
--
To UNSUBSCRIBE, email to [EMAIL PROTECTED]
with a subject of unsubscribe. Trouble? Contact [EMAIL PROTECTED]
