Re: __file__ is a disease
Le samedi 30 janvier 2010 à 19:20 -0500, Barry Warsaw a écrit : I think it's not so much that pkg_resources is complex, but that it has a lot of extra API that most people won't need. Hopefully the distutils-sig will distill the essential pieces of that and push it into the stdlib. I was trying to see if you objected to the specific explicit use of __file__ or the concept of co-locating data files with the source code, since that violates the FHS, but isn't really changed by use of pkg_resources. Co-locating data with an API to access it brings Python on par with what Java can do. That’s not really a good reference, I’m afraid, but it still means progress. The advantage of using an API is that it can abstract away changes in policy or practice. Indeed but it would require to stop developing for a moment and try designing. I have stopped following the distutils developments when people proposed to introduce yet another layer of symlink farms to work around the current design issues of pkg_resources; I have stopped hoping to see something correct come of upstream since then. Basically, I’d say it tries to address the problem at the wrong level: build/installation issues should be addressed by the build system, not by the code. Except that this is where build/installation and code requirements intersect. The code at run time needs certain data files. It doesn't really care where they are, but it has to guarantee that it can find them. The installation system may care deeply where such files go so it has to communicate this to the code at run-time. Using an abstraction like pkg_resources helps bridge that gap. It’s a hammer to crush a mosquito. The only thing you need is a few file paths, and there has been a cross-language way to do that for decades: string replacement. When a solution turns out to be more complex and less flexible than string replacement, there’s a big problem with it. Maybe. IME though, many folks who are very comfortable contributing Python code to a project run away in abject fear when asked to update autotools files. IMO, they're a necessary evil for complex C or C++ projects, but way too heavy for simple Python libraries or applications. If you have some time, please have a look at e.g. the hamster-applet upstream packaging, and tell me what’s complicated with it. I’m afraid the main problem Python developers find with it is that it’s not written in Python. So it’s more a NIH problem - the same Java has. There are huge projects that use the autotools without needing to jump through hoops. A simple look at NumPy and its 1+ lines of specific Python code just to be able to build the C pieces should be enough to convince anyone to use a better tool. Maybe so, but that seems like an extreme example for the Python universe of stuff. Most libraries don't need anything nearly as complicated. Indeed, but there are some projects just as complicated out there, and apart from (an arguably high number of) upstreams who like to make their life more complicated, they don’t need 1 lines of build scripts. I’m not implying the autotools don’t have flaws: the insanity of libtool bugs, the arcanes of m4 and the gazillions of broken m4 scripts that lie around… But these flaws are about features that don’t even exist in Python-specific tools. For most Python libraries and applications, they're YAGNI anyway. If there are specific problems with Python tools for the majority of straightforward Python libraries, let's fix those tools. But I think they mostly work, and where they don't, the distutils-sig is paving the way of the future. How is the following YAGNI? AC_CONFIG_FILES(foo.py) -- .''`. Josselin Mouette : :' : `. `' “I recommend you to learn English in hope that you in `- future understand things” -- Jörg Schilling signature.asc Description: Ceci est une partie de message numériquement signée
Re: __file__ is a disease
On Jan 30, 2010, at 4:56 PM, Josselin Mouette wrote: Maybe you don’t understand it is a plague, because you are not trying to package the things you write with __file__ for a distribution. The location of module files on the system should be a hidden implementation detail. Because of __file__, it is not and this implementation detail has to be exposed by packaging tools, restricting what they can do in ways you don’t imagine, making it impossible to just abstract them behind the “module” concept - which is the only one that should matter for a programmer. Do you feel the same way about pkg_resources? Going back to the topic, please try using autoconf, waf or even cmake to distribute your modules. These tools were designed to abstract things like filesystem locations and to generate everything needed at installation time. Python-specific tools like setuptools are not able to do that, not unless you bundle specific scripts with your packages. I've never used waf or cmake, but tools like autoconf and scons are not trivial systems to use, modify or debug. I don't think that they make life easier for a robust open source project. -Barry -- To UNSUBSCRIBE, email to debian-python-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org
Re: __file__ is a disease
Josselin Mouette j...@debian.org writes: Going back to the topic, please try using autoconf, waf or even cmake to distribute your modules. These tools were designed to abstract things like filesystem locations and to generate everything needed at installation time. The focus of autoconf and CMake, though, is on managing the complexity of building packages that require compilation and linking, and much of that resulting complexity is wasted for installing a pure Python package. I wasn't aware of Waf URL:http://code.google.com/p/waf/; it looks more suitable, thank you. Python-specific tools like setuptools are not able to do that, not unless you bundle specific scripts with your packages. Yes, I agree the Python-specific build systems (Distutils, Setuptools, maybe Distribute) are not yet able to perform the necessary module abstractions. After many years of stagnation there has, in 2009, been some encouraging progress in maintenance and discussions. We'll see whether those tools improve this year. Meanwhile, the OP might like to look at the list of Python tools at URL:http://wiki.python.org/moin/ConfigurationAndBuildTools for this purpose. -- \ “What I have to do is see, at any rate, that I do not lend | `\ myself to the wrong which I condemn.” —Henry Thoreau, _Civil | _o__)Disobedience_ | Ben Finney -- To UNSUBSCRIBE, email to debian-python-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org
Re: __file__ is a disease
Il giorno sab, 30/01/2010 alle 22.56 +0100, Josselin Mouette ha scritto: Le samedi 30 janvier 2010 à 21:40 +0100, Pietro Battiston a écrit : In my ignorance, what I've exposed is the only way I know to get things working as I want, so I'll be happy to get in touch with better designs... for me, so far, __file__ may very well have been a hack, but certainly not a plague. Maybe you don’t understand it is a plague, because you are not trying to package the things you write with __file__ for a distribution. Not only I did try (and apparently succeeded) packaging some things I wrote for a distribution (called Debian, some of you may have heard about it), but I developed my style of packaging - which may certainly have its flaws - explicitly thinking to something that would - work uninstalled, - work installed, _and_ - reduce the packaging work to the minimum The location of module files on the system should be a hidden implementation detail. Because of __file__, it is not and this implementation detail has to be exposed by packaging tools, restricting what they can do in ways you don’t imagine, making it impossible to just abstract them behind the “module” concept - which is the only one that should matter for a programmer. I use __file__ exactly in detecting the case in which there is _no_ packaging tool involved. I perfectly know that this hack - like every hack - has its drawbacks: if the binary happens to be in the same folder of a folder called stuff which contains a file called $appname.svg, _then_ it will erroneously think it's running uninstalled. I consider this a minor problem, but feel free to point out anything I'm missing. Python is the only widespread high-level language to do that. I’ve never seen a Perl, Java or C# module look for its installation path before deciding what to do. In these languages, modules are abstract objects and you can do whatever you want with them on the filesystem without changing any line of code. Believe me, I love the Python language, but the interpreter is plagued with such issues. Actually, this is the only case in which I use __file__, and apparently you are much more competent than me about Python. So I have no reason to not believe you, in general. Going back to the topic, please try using autoconf, waf or even cmake to distribute your modules. These tools were designed to abstract things like filesystem locations and to generate everything needed at installation time. Python-specific tools like setuptools are not able to do that, not unless you bundle specific scripts with your packages. I find setuptools much more elegant, I like its ease of maintenance, I experienced it is easily extendable, I trust that if it has flaws they will get fixed... but most importantly for our discussion, if I did use some of those systems that you suggest (and that I frankly thought were just obsolete stuff still hanging around and nobody would adopt voluntarily for a Python app today), I think I would still use __file__ to know when I'm running uninstalled. Pietro -- To UNSUBSCRIBE, email to debian-python-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org
Re: __file__ is a disease
/me strongly discourages using __file__ and waf regarding __file__ - things might change soon (see this[1] thread) [1] http://mail.python.org/pipermail/distutils-sig/2009-November/014424.html -- Piotr Ożarowski Debian GNU/Linux Developer www.ozarowski.pl www.griffith.cc www.debian.org GPG Fingerprint: 1D2F A898 58DA AF62 1786 2DF7 AEF6 F1A2 A745 7645 signature.asc Description: Digital signature
Re: __file__ is a disease
Le samedi 30 janvier 2010 à 17:22 -0500, Barry Warsaw a écrit : Do you feel the same way about pkg_resources? pkg_resources allows to do things correctly, but I find it too complex; it’s no wonder why people still hack stuff around __file__: it’s much simpler. Basically, I’d say it tries to address the problem at the wrong level: build/installation issues should be addressed by the build system, not by the code. I've never used waf or cmake, but tools like autoconf and scons are not trivial systems to use, modify or debug. I don't think that they make life easier for a robust open source project. I beg to disagree. For a simple project, the autotools use files just as light and simple as setuptools, and if you look at a few sample Python projects using them, you will see the amount and the complexity of build scripts is similar. And as soon as you are not dealing only with pure Python code, the difference becomes dramatic. For example the absence of pkg-config support in setuptools makes it require specific code for each and every dependency - except those that provide dependency information in the Python-specific egg-info format, ha, ha. There are huge projects that use the autotools without needing to jump through hoops. A simple look at NumPy and its 1+ lines of specific Python code just to be able to build the C pieces should be enough to convince anyone to use a better tool. I’m not implying the autotools don’t have flaws: the insanity of libtool bugs, the arcanes of m4 and the gazillions of broken m4 scripts that lie around… But these flaws are about features that don’t even exist in Python-specific tools. Cheers, -- .''`. Josselin Mouette : :' : `. `' “I recommend you to learn English in hope that you in `- future understand things” -- Jörg Schilling signature.asc Description: Ceci est une partie de message numériquement signée
Re: __file__ is a disease
On Jan 31, 2010, at 01:09 AM, Josselin Mouette wrote: pkg_resources allows to do things correctly, but I find it too complex; it’s no wonder why people still hack stuff around __file__: it’s much simpler. I think it's not so much that pkg_resources is complex, but that it has a lot of extra API that most people won't need. Hopefully the distutils-sig will distill the essential pieces of that and push it into the stdlib. I was trying to see if you objected to the specific explicit use of __file__ or the concept of co-locating data files with the source code, since that violates the FHS, but isn't really changed by use of pkg_resources. The advantage of using an API is that it can abstract away changes in policy or practice. Basically, I’d say it tries to address the problem at the wrong level: build/installation issues should be addressed by the build system, not by the code. Except that this is where build/installation and code requirements intersect. The code at run time needs certain data files. It doesn't really care where they are, but it has to guarantee that it can find them. The installation system may care deeply where such files go so it has to communicate this to the code at run-time. Using an abstraction like pkg_resources helps bridge that gap. I've never used waf or cmake, but tools like autoconf and scons are not trivial systems to use, modify or debug. I don't think that they make life easier for a robust open source project. I beg to disagree. For a simple project, the autotools use files just as light and simple as setuptools, and if you look at a few sample Python projects using them, you will see the amount and the complexity of build scripts is similar. Maybe. IME though, many folks who are very comfortable contributing Python code to a project run away in abject fear when asked to update autotools files. IMO, they're a necessary evil for complex C or C++ projects, but way too heavy for simple Python libraries or applications. And as soon as you are not dealing only with pure Python code, the difference becomes dramatic. For example the absence of pkg-config support in setuptools makes it require specific code for each and every dependency - except those that provide dependency information in the Python-specific egg-info format, ha, ha. I guess when you're talking about more complicated projects, with mixes of various types of build artifacts, then the build system will necessarily have to be more complex. There are huge projects that use the autotools without needing to jump through hoops. A simple look at NumPy and its 1+ lines of specific Python code just to be able to build the C pieces should be enough to convince anyone to use a better tool. Maybe so, but that seems like an extreme example for the Python universe of stuff. Most libraries don't need anything nearly as complicated. I’m not implying the autotools don’t have flaws: the insanity of libtool bugs, the arcanes of m4 and the gazillions of broken m4 scripts that lie around… But these flaws are about features that don’t even exist in Python-specific tools. For most Python libraries and applications, they're YAGNI anyway. If there are specific problems with Python tools for the majority of straightforward Python libraries, let's fix those tools. But I think they mostly work, and where they don't, the distutils-sig is paving the way of the future. -Barry signature.asc Description: PGP signature
