On 06/21/2012 09:05 PM, Tarek Ziadé wrote:
On 6/21/12 4:26 PM, Dag Sverre Seljebotn wrote:
project should give me so I can compile its extensions ?" I think this
has nothing to do with the tools/implementations.
If you sit down and ask your self: "what are the information a python
I'm not sure if I understand. A project can't "give the information
needed to build it". The build system is an integrated piece of the
code and package itself. Making the build of library X work on some
ugly HPC setup Y is part of the development of X.
To my mind a solution looks something like (and Bento is close to this):
Step 1) "Some standard" to do configuration of a package (--prefix and
other what-goes-where options, what libraries to link with, what
compilers to use...)
Step 2) Launch the package's custom build system (may be Unix shell
script or makefile in some cases (sometimes portability is not a
goal), may be a waf build)
Step 3) "Some standard" to be able to cleanly
install/uninstall/upgrade the product of step 2)
An attempt to do Step 2) in a major way in the packaging framework
itself, and have the package just "declare" its C extensions, would
not work. It's fine to have a way in the packaging framework that
works for trivial cases, but it's impossible to create something that
works for every case.
I think we should, as you proposed, list a few projects w/ compilation
needs -- from the simplest to the more complex, then see how a standard
*description* could be used by any tool
It's not clear to me what you mean by description. Package metadata,
install information or description of what/how to build?
I hope you don't mean the latter, that would be insane...it would
effectively amount to creating a build tool that's both more elegant and
more powerful than any option that's currently already out there.
Assuming you mean the former, that's what David did to create Bento.
Reading and understanding Bento and the design decisions going into it
would be a better use of time than redoing a discussion, and would at
least be a very good starting point.
But anyway, some project types from simple to advanced:
- Simple library using Cython + NumPy C API
- Wrappers around HPC codes like mpi4py, petsc4py
- NumPy
- SciPy (uses Fortran compilers too)
- Library using code generation, Cython, NumPy C API, Fortran 90 code,
some performance tuning with CPU characteristics (instruction set, cache
size, optimal loop structure) decided compile-time
And if we're able to write down in a PEP this, e.g. the information a
compiler is looking for to do its job, then any tool out there waf,
scons, cmake, jam, ant, etc, can do the job, no ?
Anyway: I really don't want to start a flame-war here. So let's accept
up front that we likely won't agree here; I just wanted to clarify my
position.
After 4 years I still don't understand what "we won't agree" means in
this context. *NO ONE* ever ever came and told me : here's what I want a
Python project to describe for its extensions.
That's unfortunate. To be honest, it's probably partly because it's
easier to say what won't work than come with a constructive
suggestion. A lot of people (me included) just use
waf/cmake/autotools, and forget about making the code installable
through PyPI or any of the standard Python tools. Just because that
works *now* for us, but we don't have any good ideas for how to make
this into something that works on a wider scale.
I think David is one of the few who has really dug into the matter and
tried to find something that can both do builds and work through
standard install mechanisms. I can't answer for why you haven't been
able to understand one another.
It may also be an issue with how much one can constructively do on
mailing lists. Perhaps the only route forward is to to bring people
together in person and walk distutils2 people through some hairy
scientific HPC builds (and vice versa).
Like versions scheme, I think it's fine if you guys have a more complex
system to build software. But there should be a way to share a common
standard for complation, even if people that uses distutils2 or xxx, are
just doing the dumbest things, like simple C libs compilation.
Just "we won't agree" or "distutils sucks" :)
Gosh I hope we will overcome this lock one day, and move forward :D
Well, me too.
The other thing is, the folks in distutils2 and myself, have zero
knowledge about compilers. That's why we got very frustrated not to see
people with that knowledge come and help us in this area.
Here's the flip side: If you have zero knowledge about compilers, it's
going to be almost impossible to have a meaningful discussion about a
compilation PEP. It's very hard to discuss standards unless everybody
involved have the necessary prerequisite knowledge. You don't go
discussing details of the Linux kernel without some solid C experience
either.
The necessary prerequisites in this case is not merely "knowledge of
compilers". To avoid repeating mistakes of the past, the prerequisites
for a meaningful discussion is years of hard-worn experience building
software in various languages, on different platforms, using different
build tools.
Look, these problems are really hard to deal with. Myself I have
experience with building 2-3 languages using 2-3 build tools on 2
platforms, and I consider myself a complete novice and usually decide to
trust David's instincts over trying to make up an opinion of my own --
simply because I know he's got a lot more experience than I have.
Theoretically it is possible to separate and isolate concerns so that
one set of people discuss build integration and another set of people
discuss installation. Problem is that all the problems tangle -- in
particular when the starting point is distutils!
That's why *sometimes*, not always, design by committee is the wrong
approach, and one-man-shows is what brings technology forwards.
So, I reiterate my proposal, and it could also be expressed like this:
1/ David writes a PEP where he describes how Bento interact with a
project -- metadata, description files, etc
2/ Someone from distutils2 completes the PEP by describing how setup.cfg
works wrt Extensions
3/ we see if we can have a common standard even if it's a subset of
bento capabilities
bento isn't a build tool, it's a packaging tool, competing directly with
distutils2. It can deal with simple distutils-like builds using a
bundled build tool, and currently has integration with waf for
complicated builds; integration with other build systems will presumably
be added later as people need it (the main point is that bento is
designed for it).
Dag
_______________________________________________
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
