Re: [Distutils] Pre-pre-PEP: Requirements for the Python BUILDS Specification

[Toshio Kuratomi]

Ian Bicking wrote:
> Paul Moore wrote:
>> My feeling, by the way, is that "system packagers" are the more
>> relevant group on Linux/Unix (where most users install Python modules
>> via system packages, or else they are developers)
> 
> I think this is part of why I don't understand the system packager
> perspective.  Developers shouldn't use system packages, it just doesn't
> make any sense to have that intermediation.  Users don't use Python
> modules, they use applications.  Users only care that their applications
> work, that they can install applications without unnecessary conflicts,
> that the applications don't break based on unintentional environment
> changes (e.g., the value of PYTHONPATH).
> 
> Packagers seem to care a great deal about having applications share
> libraries on the packaging level, but this is for their own accounting,
> there's no reason for users to care (except for the too-small-to-matter
> issue of disk space).  Also, packagers seem to jump the gun on this
> library sharing, as they are concerned about libraries when one (or
> often zero!) applications depend on the library.  Some widely used
> libraries seem reasonable, but for every widely used library there are a
> dozen or more niche libraries.

There is a great deal of incentive for everything to use system
libraries.  take formencode-1.0 vs formencode-1.0.1 for instance.  If
every TurboGears application and whatnot that used formencode were using
their own copy of formencode-1.0 and then the discovery that
formencode-1.0 wasn't doing chained validators correctly came out, the
end users have to go about finding every copy of formencode and updating
it.  If there is only a single formencode library on the system and all
the TurboGears apps are using it, that's a single package that needs to
be updated.

>  Users also don't care about /usr/share
> or /usr/lib -- the only thing *I* ever care about is /usr/share/doc,
> /usr/bin, /etc, and maybe a man page.
> 

They do when user == system administrator.  Ask someone who manages
Linux systems at a large University if they care about the FHS and the
answer will be yes.  The various distinctions in the FHS between
writable vs nonwritable, architecture dependent vs architecture
independent, etc allow system administrators to maintain a large number
of systems with less pain.

-Toshio



signature.asc
Description: OpenPGP digital signature
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
http://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Pre-pre-PEP: Requirements for the Python BUILDS Specification

[Gael Varoquaux]

On Fri, Oct 10, 2008 at 02:34:30PM -0400, Ian Bicking wrote:
> Developers shouldn't use system packages, it just doesn't 
> make any sense to have that intermediation. 

I don't agree. I work as a developer/scientist in a lab. 

I develop algorithms that sometimes will never be used on another box
than mine (that's when they are bad). We happen to be standardised on a
Linux distro that is a bit old, and with little packages. I can tell you
it really annoys me to have to install manually a package each time I want
to use it. The pure-Python ones, with no other dependencies are not hard,
easy-install does the trick, but these actually represent a small
fraction of the important ones (wxpython, numpy, pytables, matplotlib,
elementtree, pyobjects, cython, ets, pyvtk). 

So I lose a lot of time installing packages on my box. But the real
nightmare begins when I want to share the algorithms and applications I
developed. Across our institute we have NFS-shared drives on which we
have to compile all the dependencies, for all three platforms we support.
And that's only for distributing software in the institute, when we want
to distribute outside we end up shipping big binary blobs with
everything, from our own version of Python, to numpy, wxPython, ATLAS
(that's bad). If only we could rely on package managers to provide us
with base packages (and API-stability of these packages to be sure they
work).

> Users don't use Python modules, they use applications.  Users only care
> that their applications work, that they can install applications
> without unnecessary conflicts, that the applications don't break based
> on unintentional environment changes (e.g., the value of PYTHONPATH).

That is true, as long as you ship "the world", ie everything you'll ever
need for your extensible application, including the c libraries that go
behind. This means putting a huge burden on your development team, and if
you think about out, this is exactly the work of a distribution. So you
are duplicating this effort.

I addition, one of the great things about Python, is that it is easy to
read and to code. Advanced users will want to write plugins, extend there
programs, and programs should be written for this: 

"Hey, I have this cool application that does brain segmentation, and
visualization of the tissues, and I have this cool other app that has an
algorithm to identify tumorous tissues, I want to plug them together, and
because the APIs are well-written, it should be trivial...

Darn, the two apps have confined Pythons, and they can't import from
each-others. No big deal, I'll unbundle the whole thing...

Darn, one has been build with UCS-2, the other one with UCS-4, and
there C libraries are incompatible...

So now the easy task has become a long effort of building the two
apps, and I can't easily share my work."

I can hardly believe this happens only in the scientific computing world.
I have fought (and I still fighting) to deploy a Django-based
application. It didn't get along with the server's apache version.

Gaël

___
Distutils-SIG maillist  -  Distutils-SIG@python.org
http://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Pre-pre-PEP: Requirements for the Python BUILDS Specification

[David Cournapeau]

Ian Bicking wrote:
> I think this is part of why I don't understand the system packager
> perspective.  Developers shouldn't use system packages, it just
> doesn't make any sense to have that intermediation.

I can't see why you would think that. Of course it makes sense to have
that intermediation, for packages which are not directly relevant to
you. I am more than happy to use packages myself for almost everything
but the things I am directly working on (and maybe the layer just below).

> Users don't use Python modules, they use applications.

What is an application ? I mean, people who use numpy/scipy, they use a
module. People who are scons users, they use a software which is just
one module + some scripts.

The application / library difference is blurry in python.

>   Users only care that their applications work, that they can install
> applications without unnecessary conflicts, that the applications
> don't break based on unintentional environment changes (e.g., the
> value of PYTHONPATH).

Yes, of course. But this is obviously linked to deployment issues. Then,
the argument is that when deploying things, multiple / concurrent
modules is a headache. Api versioning and stability is not a linux
packagers crazy idea
>
> Packagers seem to care a great deal about having applications share
> libraries on the packaging level, but this is for their own
> accounting, there's no reason for users to care (except for the
> too-small-to-matter issue of disk space).

Did you read carefully what was written by the packagers ? They almost
never mention disk space (which does matter on some systems BTW; not
everybody uses a desktop or a server). They mention security, they
mention maintanability. Both those are made much more complicated when
you enable multiple / concurrent versions.

>   Also, packagers seem to jump the gun on this library sharing, as
> they are concerned about libraries when one (or often zero!)
> applications depend on the library.  Some widely used libraries seem
> reasonable, but for every widely used library there are a dozen or
> more niche libraries.

Those niche libraries are not always packaged. If a library is used by
only one package, it is quite likely that it is included in the package
itself.

cheers,

David
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
http://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Pre-pre-PEP: Requirements for the Python BUILDS Specification

[Floris Bruynooghe]

On Fri, Oct 10, 2008 at 10:56:02PM +0300, Marius Gedminas wrote:
> On Fri, Oct 10, 2008 at 02:34:30PM -0400, Ian Bicking wrote:
> > Paul Moore wrote:
> > >My feeling, by the way, is that "system packagers" are the more
> > >relevant group on Linux/Unix (where most users install Python modules
> > >via system packages, or else they are developers)
> > 
> > I think this is part of why I don't understand the system packager 
> > perspective.  Developers shouldn't use system packages, it just doesn't 
> > make any sense to have that intermediation.
> 
> This only applies to packages you're developing, not those you're using
> to develop your app/library/whatever.
> 
> Why I, as a developer, prefer system packages:
> 
>  * easier installation
>  * easier upgrades
>  * I don't have to keep track of security updates myself

I'd like to add to this: making myself develop against system packages
will make sure my work will be easy to package up and hence deploy.


Regards
Floris

-- 
Debian GNU/Linux -- The Power of Freedom
www.debian.org | www.gnu.org | www.kernel.org
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
http://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Pre-pre-PEP: Requirements for the Python BUILDS Specification

[Marius Gedminas]

On Fri, Oct 10, 2008 at 02:34:30PM -0400, Ian Bicking wrote:
> Paul Moore wrote:
> >My feeling, by the way, is that "system packagers" are the more
> >relevant group on Linux/Unix (where most users install Python modules
> >via system packages, or else they are developers)
> 
> I think this is part of why I don't understand the system packager 
> perspective.  Developers shouldn't use system packages, it just doesn't 
> make any sense to have that intermediation.

This only applies to packages you're developing, not those you're using
to develop your app/library/whatever.

Why I, as a developer, prefer system packages:

 * easier installation
 * easier upgrades
 * I don't have to keep track of security updates myself

In fact those are the same reasons why I prefer system packages as a
user.

There are downsides (e.g. you usually don't get the latest version, and
sometimes distributions apply broken patches).

Marius Gedminas
-- 
You'll find creativity working hand in hand with engineering. It will feel
strange and you might feel like things are out of control. Relax - they are.
-- Richard Gabriel on software
   http://www.dreamsongs.net/LessonsFromNothing.html


signature.asc
Description: Digital signature
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
http://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Pre-pre-PEP: Requirements for the Python BUILDS Specification

[Paul Moore]

2008/10/10 Ian Bicking <[EMAIL PROTECTED]>:
> Paul Moore wrote:
>>
>> My feeling, by the way, is that "system packagers" are the more
>> relevant group on Linux/Unix (where most users install Python modules
>> via system packages, or else they are developers)
>
> I think this is part of why I don't understand the system packager
> perspective.  Developers shouldn't use system packages, it just doesn't make
> any sense to have that intermediation.  Users don't use Python modules, they
> use applications.  Users only care that their applications work, that they
> can install applications without unnecessary conflicts, that the
> applications don't break based on unintentional environment changes (e.g.,
> the value of PYTHONPATH).

Interesting viewpoint (from my Windows developer/user POV :-))

>From my point of view, as a Windows developer, I want to use "system
packages" (bdist_wininst installers) because then I get the benefits
of having a listing of what I have installed, being able to quickly,
easily and cleanly install and uninstall packages to "try them out"
etc. I have the same environment for my one-off scripts as for my
development environment, which again I feel is good as I'm using a
consistent, familiar, toolset.

As a user, I want my applications packaged with py2exe, and
independent of the "system Python" - which for me is the development
environment, and therefore not stable enough to rely on for apps! And
as a developer, py2exe cleanly picks out those bits of my development
environment which are needed in the delivered application, so I don't
have to worry about installing "too much" before I package my
application up.

This probably reflects the fact that per-user, or any other sort of
private, installations of Python, are the exception rather than the
norm on Windows, whereas on Unix, building a custom Python to do your
development with (either from scratch, or with things like virtualenv)
is easy and sensible.

I guess the more we see how "the other half" lives (without the
flamewars :-)), the better we'll be able to compromise or provide
options to help everyone.

(Skipped the comments on system packaging, as you've confirmed what I
thought, which is that it's very much a Unix phenomenon, and I'm not
qualified to have an opinion...)

Paul.
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
http://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Pre-pre-PEP: Requirements for the Python BUILDS Specification

[Ian Bicking]

Paul Moore wrote:

My feeling, by the way, is that "system packagers" are the more
relevant group on Linux/Unix (where most users install Python modules
via system packages, or else they are developers)


I think this is part of why I don't understand the system packager 
perspective.  Developers shouldn't use system packages, it just doesn't 
make any sense to have that intermediation.  Users don't use Python 
modules, they use applications.  Users only care that their applications 
work, that they can install applications without unnecessary conflicts, 
that the applications don't break based on unintentional environment 
changes (e.g., the value of PYTHONPATH).


Packagers seem to care a great deal about having applications share 
libraries on the packaging level, but this is for their own accounting, 
there's no reason for users to care (except for the too-small-to-matter 
issue of disk space).  Also, packagers seem to jump the gun on this 
library sharing, as they are concerned about libraries when one (or 
often zero!) applications depend on the library.  Some widely used 
libraries seem reasonable, but for every widely used library there are a 
dozen or more niche libraries.  Users also don't care about /usr/share 
or /usr/lib -- the only thing *I* ever care about is /usr/share/doc, 
/usr/bin, /etc, and maybe a man page.



--
Ian Bicking : [EMAIL PROTECTED] : http://blog.ianbicking.org
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
http://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Pre-pre-PEP: Requirements for the Python BUILDS Specification

[Josselin Mouette]

Le mardi 07 octobre 2008 à 19:25 -0400, Phillip J. Eby a écrit :
> At 12:43 AM 10/8/2008 +0200, Josselin Mouette wrote:
> >What I am afraid of is that, by adding just another layer, you will
> >introduce new problems while fixing existing ones. Currently we already
> >have a hard time maintaining a symlink farm, and adding a second symlink
> >farm on top of it is not going to make things more understandable
> >(currently, package maintainers already have a hard time understanding
> >how to deal with Python modules) nor more reliable.
> 
> I guess I'm confused a bit here, since the idea is not to have 
> maintainers need to understand anything but how to be able to tell 
> upstream, "hey, you didn't configure file X correctly".  At least, 
> that's what I'd hope would be all that's needed.  :-)

First, it is wrong to expect maintainers to not understand what they do.
It is a guaranteed recipe for failure.

Second, it is not realistic to assume that all the maintainer will do is
forward bugs to the upstream developers, and that upstream developers
will fix it in the minute – especially for packaging issues, which
developers generally disregard. The maintainer often needs to patch the
sources or the installation scripts so that the software installs
properly, and to do so he needs to understand what he does.

> >The first kind of issues I can think of is that dpkg handles symlinks to
> >directories very badly, so you’re likely to run into issues that have to
> >be dealt with by hand by maintainers. I haven’t thought about it much,
> >but I have the feeling that other things will break.
> 
> Actually, until I saw the Twisted plugins directory use case, I 
> hadn't actually seen any use cases for symlinks to directories that 
> couldn't be handled by using a directory of symlinks instead.  Are 
> there any other Python packages you know of that would need a symlink 
> to a directory?

Yes, you can arrange for not meeting this specific issue. What I’m
saying is not that such issues don’t have workarounds, I’m saying that
we will discover new ones, again and again, as long as the only thing we
do to fix each issue is add another carpet to hide the dust.

> The idea of the BUILDS "install locations" (IL) manifest is just that 
> it lists the contents of the project distribution, and describes the 
> logical locations that things go in.  So, part of the BUILDS spec 
> would be what categories exist for the above.

If the only thing it does is allowing to move files and symlink them, I
frankly doubt the benefits will outperform the cost of maintaining this
new symlink farm.

> >What some Python projects using autoconf are doing, and I’d like to
> see
> >in BUILDS, is the generation of a config.py file at installation time.
> >It would be useful to specify install-time stuff like installation
> >directories, optional features and installation parameters.
> 
> It looks like a great idea, and I'm all for recommending people use 
> it, or something like it.  I'm also fine with BUILDS having support 
> for some sort of "install-time editing" like this.  I just don't want 
> it to be a requirement that everybody change their ways.  Not 
> everybody even uses pkg_resources' API for accessing data files yet, after 
> all.

You don’t have to make it a requirement. If you agree with the idea of
such a feature, a good approach might be to:
  * always generate this file;
  * provide the possibility to easily install it;
  * let developers gradually start using it directly in the code.

> >Another thing from the autotools that I’d like to see is seamless
> >integration with pkg-config, for modules with C extensions. Bonus points
> >for being able to use not only compilation flags and installation paths,
> >but also macros you need to use inside the code (they could go in
> >config.py).
> 
> Sorry, what is pkg-config?

pkg-config is becoming the de-facto standard for providing information
about installed development libraries. With it you can easily:
  * obtain compile-time parameters for dependent programs (CFLAGS
and LIBS);
  * require a range of versions for a number of libraries you depend
on;
  * obtain any parameter that the developer wanted you to know (for
example, the name of a dbus service you need to start, the path
to a helper utility, or the installation path for plugins);
  * with the automake integration, enable programs or features based
on the availability of dependencies.

For a Python example: you want to know where to install your definition
files for GNOME Python bindings? Just use:
pkg-config --variable=codegendir pygtk-2.0

Cheers,
-- 
 .''`.
: :' :  We are debian.org. Lower your prices, surrender your code.
`. `'   We will add your hardware and software distinctiveness to
  `-our own. Resistance is futile.


signature.asc
Description: Ceci est une partie de message	numériquement signée
__

Re: [Distutils] Pre-pre-PEP: Requirements for the Python BUILDS Specification

[David Cournapeau]

Phillip J. Eby wrote:
>
> Sorry, what is pkg-config?

http://pkg-config.freedesktop.org/wiki/

David
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
http://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Pre-pre-PEP: Requirements for the Python BUILDS Specification

[Phillip J. Eby]

At 12:43 AM 10/8/2008 +0200, Josselin Mouette wrote:

Le mardi 07 octobre 2008 à 16:51 -0400, Phillip J. Eby a écrit :
> >If you write a tool to do that, why not make it simply move files
> >properly and let the code locate them, instead of adding yet another
> >layer on top of the existing stuff? The tool will not be more
> >complicated this way.
>
> I'm not sure I follow you.  What I'm proposing is that:
>
> 1. A BUILDS "manifest" for a project should list the files included
> in the project by what "kind" of file they are, as well as by
> inter-file relationships (e.g., "these 3 files are part of Python
> package 'foo.bar' within this project, and they are static data used
> at runtime but not read or written").
>
> 2. BUILDS-compatible install tools will be free to install those
> files wherever they like (or create .debs, .rpms, etc. that do), and
> use symlinks so that the installed project can find them in the
> places that the developer wanted to find them.
>
> Is this what you're asking for?  Because it's what I've been trying
> to say we should do.

What I am afraid of is that, by adding just another layer, you will
introduce new problems while fixing existing ones. Currently we already
have a hard time maintaining a symlink farm, and adding a second symlink
farm on top of it is not going to make things more understandable
(currently, package maintainers already have a hard time understanding
how to deal with Python modules) nor more reliable.


I guess I'm confused a bit here, since the idea is not to have 
maintainers need to understand anything but how to be able to tell 
upstream, "hey, you didn't configure file X correctly".  At least, 
that's what I'd hope would be all that's needed.  :-)




The first kind of issues I can think of is that dpkg handles symlinks to
directories very badly, so you’re likely to run into issues that have to
be dealt with by hand by maintainers. I haven’t thought about it much,
but I have the feeling that other things will break.


Actually, until I saw the Twisted plugins directory use case, I 
hadn't actually seen any use cases for symlinks to directories that 
couldn't be handled by using a directory of symlinks instead.  Are 
there any other Python packages you know of that would need a symlink 
to a directory?




In the end, if you are designing a new packaging specification and a new
tool,


The BUILDS manifest is neither packaging nor a tool; it's just 
information about a bunch of files.  ;-)  We don't want to design *a* 
tool, we want to make it possible for tools to be 
written.  Currently, there are a whole bunch of installation and 
maintenance tools out there built on top of setuptools, but they 
aren't necessarily very compatible with each other, and building 
tools that approach things very differently from setuptools is hard.


The idea of a specification is to allow installation tools to be 
built that apply different policies on different platforms, not to 
build The One True Tool[tm].


In other words, the goal isn't to "do the right thing", but to allow 
various useful things to be done.




> For example, if Debian needs metadata that's not in the core BUILDS
> metadata, it should be possible within BUILDS to include it, so that
> it can be contributed upstream.  Likewise, it should be possible for
> a BUILDS install tool to use a third-party supplement to get that
> data, so that when you're initially porting a project, you could
> write a file with the additional data while doing the initial port,
> then submit that file upstream.

If there is a need for Debian-specific metadata, it will go in the
debian/ directory. It looks interesting to integrate it upstream only if
it is relevant for other distributors.


If I have to explicitly mark my locale files for Debian to be happy, 
then that *is* Debian-specific metadata from my perspective as a 
cross-platform developer.


That is, I didn't say anything about metadata that was unique to 
Debian, just metadata needed by a Debian BUILDS installer to be 
happy, as opposed to a generic BUILDS installer that doesn't care 
about the FHS.  (For example, an Enthought installer for Windows 
that's also installing its own Python interpreter might want to 
consume BUILDS but it sure as heck wouldn't be following FHS 
standards; it'd have its own.)




There is no use in being compatible with the autotools, the point is
about stealing good ideas.

The first thing that comes to mind, and that we already discussed at
large, is the ability to specify at install time where data,
documentation, manual pages and so on have to go.


The idea of the BUILDS "install locations" (IL) manifest is just that 
it lists the contents of the project distribution, and describes the 
logical locations that things go in.  So, part of the BUILDS spec 
would be what categories exist for the above.


It would then be the responsibility of people building 
BUILDS-consuming install tools to define the actual locations.  I 

Re: [Distutils] Pre-pre-PEP: Requirements for the Python BUILDS Specification

[Josselin Mouette]

Le mardi 07 octobre 2008 à 16:51 -0400, Phillip J. Eby a écrit :
> >If you write a tool to do that, why not make it simply move files
> >properly and let the code locate them, instead of adding yet another
> >layer on top of the existing stuff? The tool will not be more
> >complicated this way.
> 
> I'm not sure I follow you.  What I'm proposing is that:
> 
> 1. A BUILDS "manifest" for a project should list the files included 
> in the project by what "kind" of file they are, as well as by 
> inter-file relationships (e.g., "these 3 files are part of Python 
> package 'foo.bar' within this project, and they are static data used 
> at runtime but not read or written").
> 
> 2. BUILDS-compatible install tools will be free to install those 
> files wherever they like (or create .debs, .rpms, etc. that do), and 
> use symlinks so that the installed project can find them in the 
> places that the developer wanted to find them.
> 
> Is this what you're asking for?  Because it's what I've been trying 
> to say we should do.

What I am afraid of is that, by adding just another layer, you will
introduce new problems while fixing existing ones. Currently we already
have a hard time maintaining a symlink farm, and adding a second symlink
farm on top of it is not going to make things more understandable
(currently, package maintainers already have a hard time understanding
how to deal with Python modules) nor more reliable.

The first kind of issues I can think of is that dpkg handles symlinks to
directories very badly, so you’re likely to run into issues that have to
be dealt with by hand by maintainers. I haven’t thought about it much,
but I have the feeling that other things will break.

In the end, if you are designing a new packaging specification and a new
tool, I think you’d better take this as an occasion to do the right
thing instead of adding this new layer.

> For example, if Debian needs metadata that's not in the core BUILDS 
> metadata, it should be possible within BUILDS to include it, so that 
> it can be contributed upstream.  Likewise, it should be possible for 
> a BUILDS install tool to use a third-party supplement to get that 
> data, so that when you're initially porting a project, you could 
> write a file with the additional data while doing the initial port, 
> then submit that file upstream.

If there is a need for Debian-specific metadata, it will go in the
debian/ directory. It looks interesting to integrate it upstream only if
it is relevant for other distributors.

> Less flippantly, our goal here is to have something that has a better 
> cost/benefit ratio for adoption by a Python developer who has not 
> seen any pressing need for autoconf in their life to date.  And if 
> somebody wants to make a BUILDS tool to generate autoconf sources 
> from a project manifest, then great!

No, no. Please don’t :)
But if you can express in the project manifest the same flexibility as
with autoconf (or even the tenth of it, we don’t need *that* much
flexibility), that should be enough.

> >BTW, I would consider it a good approach to try bringing BUILDS on par
> >with the autotools capabilities. These tools have serious drawbacks, but
> >they were written with the ability to work for distributors in mind.
> 
> Sure.  I like stealing designs and requirements definition from 
> mature projects, even if you might argue that I chose poor role 
> models for setuptools.  (Java and OSGi.)
> 
> However, since autotools compatibility isn't a goal in itself, I 
> think I'll leave it to those people who have interest/experience to 
> propose what specific autotools features they'd like to see in 
> BUILDS, and the use cases they have for them.  If I were to just skim 
> the docs trying to figure that out, I might think that something cool 
> in it is actually useful, or vice versa.  ;-)  So, it's better here I 
> think to hear from experienced users such as yourself.

There is no use in being compatible with the autotools, the point is
about stealing good ideas.

The first thing that comes to mind, and that we already discussed at
large, is the ability to specify at install time where data,
documentation, manual pages and so on have to go.


Another thing that I consider a good design (when expunged of the
useless stuff in it) is the config.h file. It is a file that is
generated at configure time and which contains macros you may need
inside your code: package version, installation directories, enabled
features, etc.

What some Python projects using autoconf are doing, and I’d like to see
in BUILDS, is the generation of a config.py file at installation time.
It would be useful to specify install-time stuff like installation
directories, optional features and installation parameters.

For example it could contain:
version = "1.2.3"
builddate = "Tue 7 Oct 2008, 23:12 +0100"
templates_dir = "/usr/share/foobar/templates"
has_libamazing = True # _foobar.so is built with libamazing supp

Re: [Distutils] Pre-pre-PEP: Requirements for the Python BUILDS Specification

[Paul Moore]

2008/10/7 Phillip J. Eby <[EMAIL PROTECTED]>:
> At 10:04 AM 10/7/2008 +0100, Paul Moore wrote:
>>
>> 2008/10/7 Phillip J. Eby <[EMAIL PROTECTED]>:
>> > You can see that this is also what I did in the design of easy_install
>> > and
>> > setuptools, except that in that effort I only considered developers and
>> > users, not system packagers.
>>
>> I'd argue (you may differ) that the most significant area where you
>> missed the mark on user benefits with easy_install and setuptools is
>> the lack of easy *uninstall* and easy *list* options.
>
> Well, I'd certainly agree that those features are desirable.  But I didn't
> "miss the mark", in that those features were not part of my mark.  ;-)  The
> goal was to get widespread adoption by developers, and thus the primary
> target audience was developers.  Any features that attracted users were
> there only insofar as the benefit to users would specifically drive adoption
> by developers.

OK, I was assuming that when you said "I only considered developers
and users", you were implying that you were aiming to provide for
users benefits (and hence, the user benefit of package management -
uninstall and list - was important). As you say, though, your key aim
was to drive adoption, so making *installation* easy for users is
important, but making *uninstall* easy isn't, as it's not something
related to adoption - rather the opposite :-)

Of course, as a user, my perspectives are different :-) And as a user
who is increasingly resistant to setuptools-enabled packages precisely
because they don't address user-level issues, I suspect that catering
for user level requirements (in addition to system packagers', and the
already covered developers) is more relevant this time round.

My feeling, by the way, is that "system packagers" are the more
relevant group on Linux/Unix (where most users install Python modules
via system packages, or else they are developers) whereas on Windows,
users are the more relevant group (as there isn't really a significant
system packager role in the first place). Whether this, coupled with
the notoriously difficult task of getting Windows and Linux/Unix users
to understand each other's perspectives, is an important fact, only
time will tell.

Thanks for taking the time to clarify for me.

Paul.
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
http://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Pre-pre-PEP: Requirements for the Python BUILDS Specification

[Phillip J. Eby]

At 08:58 PM 10/7/2008 +0200, Josselin Mouette wrote:

Le mardi 07 octobre 2008 à 14:40 -0400, Phillip J. Eby a écrit :
> At 09:57 AM 10/7/2008 +0200, Josselin Mouette wrote:
> >Symlinks are a real pain to handle. We can use them transparently
> >for .pyc files, but if we want to relocate data files to some other
> >directories, currently it has to be done by hand, and this is why most
> >maintainers don’t do it.
.
>
> Which is why the idea for the BUILDS spec to include a way for
> automated tools to do it, so that you won't have to do it manually.

If you write a tool to do that, why not make it simply move files
properly and let the code locate them, instead of adding yet another
layer on top of the existing stuff? The tool will not be more
complicated this way.


I'm not sure I follow you.  What I'm proposing is that:

1. A BUILDS "manifest" for a project should list the files included 
in the project by what "kind" of file they are, as well as by 
inter-file relationships (e.g., "these 3 files are part of Python 
package 'foo.bar' within this project, and they are static data used 
at runtime but not read or written").


2. BUILDS-compatible install tools will be free to install those 
files wherever they like (or create .debs, .rpms, etc. that do), and 
use symlinks so that the installed project can find them in the 
places that the developer wanted to find them.


Is this what you're asking for?  Because it's what I've been trying 
to say we should do.



> Or are you saying that that isn't an improvement over the current 
situation?


It is. I didn’t understand you wanted to automate the symlinks creation.
It indeed means less burden on the maintainers, but it would be a shame
to keep the same mess in binary packages, while having a tool to do that
would allow to make things cleaner.


I'm not sure I understand you here, either.  I'm saying that a 
BUILDS-compatible install tools for Debian and RPM systems should 
create system packages with the stuff wherever the OS people want it 
to go, *and* symlinks so the Python code can find it.  And that this 
should be an automated process, to the extent possible/practical.


For example, if Debian needs metadata that's not in the core BUILDS 
metadata, it should be possible within BUILDS to include it, so that 
it can be contributed upstream.  Likewise, it should be possible for 
a BUILDS install tool to use a third-party supplement to get that 
data, so that when you're initially porting a project, you could 
write a file with the additional data while doing the initial port, 
then submit that file upstream.




> Yes, it's true that I'm saying that developers should not be
> *required* to add the extra data to their packages, but that they
> should be *able* to, and if it is trivial to add the extra data, most
> should accept patches or respond to requests to do so.
>
> Right now, it's not even *possible* for them to do so, however.

It is possible - if they use autoconf instead.


Sure, but who's going to convince them to do *that*?  ;-)

Less flippantly, our goal here is to have something that has a better 
cost/benefit ratio for adoption by a Python developer who has not 
seen any pressing need for autoconf in their life to date.  And if 
somebody wants to make a BUILDS tool to generate autoconf sources 
from a project manifest, then great!




BTW, I would consider it a good approach to try bringing BUILDS on par
with the autotools capabilities. These tools have serious drawbacks, but
they were written with the ability to work for distributors in mind.


Sure.  I like stealing designs and requirements definition from 
mature projects, even if you might argue that I chose poor role 
models for setuptools.  (Java and OSGi.)


However, since autotools compatibility isn't a goal in itself, I 
think I'll leave it to those people who have interest/experience to 
propose what specific autotools features they'd like to see in 
BUILDS, and the use cases they have for them.  If I were to just skim 
the docs trying to figure that out, I might think that something cool 
in it is actually useful, or vice versa.  ;-)  So, it's better here I 
think to hear from experienced users such as yourself.


___
Distutils-SIG maillist  -  Distutils-SIG@python.org
http://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Pre-pre-PEP: Requirements for the Python BUILDS Specification

[Josselin Mouette]

Le mardi 07 octobre 2008 à 14:40 -0400, Phillip J. Eby a écrit :
> At 09:57 AM 10/7/2008 +0200, Josselin Mouette wrote:
> >Symlinks are a real pain to handle. We can use them transparently
> >for .pyc files, but if we want to relocate data files to some other
> >directories, currently it has to be done by hand, and this is why most
> >maintainers don’t do it.
> 
> Which is why the idea for the BUILDS spec to include a way for 
> automated tools to do it, so that you won't have to do it manually.

If you write a tool to do that, why not make it simply move files
properly and let the code locate them, instead of adding yet another
layer on top of the existing stuff? The tool will not be more
complicated this way.

> Or are you saying that that isn't an improvement over the current situation?

It is. I didn’t understand you wanted to automate the symlinks creation.
It indeed means less burden on the maintainers, but it would be a shame
to keep the same mess in binary packages, while having a tool to do that
would allow to make things cleaner.

> Yes, it's true that I'm saying that developers should not be 
> *required* to add the extra data to their packages, but that they 
> should be *able* to, and if it is trivial to add the extra data, most 
> should accept patches or respond to requests to do so.
> 
> Right now, it's not even *possible* for them to do so, however.

It is possible - if they use autoconf instead.

BTW, I would consider it a good approach to try bringing BUILDS on par
with the autotools capabilities. These tools have serious drawbacks, but
they were written with the ability to work for distributors in mind.

> At the same time, the people I'd most like to see on the PEP team 
> from the developer-user side would definitely include folks from 
> Twisted, Numpy, Enthought, and Zope, as they are the folks who have 
> most stressed the distutils to the limits and beyond.  Just getting 
> you and them in the "same room" so to speak seems to already be 
> producing some benefits.

Looks like a sane approach.

Cheers,
-- 
 .''`.
: :' :  We are debian.org. Lower your prices, surrender your code.
`. `'   We will add your hardware and software distinctiveness to
  `-our own. Resistance is futile.


signature.asc
Description: Ceci est une partie de message	numériquement signée
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
http://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Pre-pre-PEP: Requirements for the Python BUILDS Specification

[Phillip J. Eby]

At 09:57 AM 10/7/2008 +0200, Josselin Mouette wrote:

Le lundi 06 octobre 2008 à 21:33 -0400, Phillip J. Eby a écrit :
> >Does this mean actively avoiding an API that would allow developers to
> >access certain types of data files (I'm thinking of the discussion
> >about locale data and not putting anything else but .py/.pyc/.pyo
> >files in packages) or merely making sure the existing way (of shipping
> >data files in packages and finding them by os.path and __file__) keeps
> >working?
>
> I would actively avoid it for a "BUILDS 1.0" spec, because on any
> platform where the people building installation tools care about
> relocating these files, symlinks are available, so both sides can be
> happy without needing a new API.

I fail to see what this would bring over the current situation, then.

> That is, unless I have misunderstood Josselin and Toshio, I
> understand symlinks to currently be an acceptable compromise.  (For
> example, Debian uses them to relocate .pyc files currently.)

Symlinks are a real pain to handle. We can use them transparently
for .pyc files, but if we want to relocate data files to some other
directories, currently it has to be done by hand, and this is why most
maintainers don’t do it.


Which is why the idea for the BUILDS spec to include a way for 
automated tools to do it, so that you won't have to do it manually.


Or are you saying that that isn't an improvement over the current situation?

Yes, it's true that I'm saying that developers should not be 
*required* to add the extra data to their packages, but that they 
should be *able* to, and if it is trivial to add the extra data, most 
should accept patches or respond to requests to do so.


Right now, it's not even *possible* for them to do so, however.

Also, I wasn't aware of those peculiarities regarding Twisted and 
GStreamer -- neither one uses setuptools.  But certainly they should 
make interesting test cases for any proposed standard, wrt to finding 
ways to satisfy both parties.


Please understand that I'm not saying we must be 100% backward 
compatible with 100% of the packages; we just need to be compatible 
enough with enough of the packages for network effects to drive the 
adoption of the rest.


At the same time, the people I'd most like to see on the PEP team 
from the developer-user side would definitely include folks from 
Twisted, Numpy, Enthought, and Zope, as they are the folks who have 
most stressed the distutils to the limits and beyond.  Just getting 
you and them in the "same room" so to speak seems to already be 
producing some benefits.


___
Distutils-SIG maillist  -  Distutils-SIG@python.org
http://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Pre-pre-PEP: Requirements for the Python BUILDS Specification

[Phillip J. Eby]

At 10:04 AM 10/7/2008 +0100, Paul Moore wrote:

2008/10/7 Phillip J. Eby <[EMAIL PROTECTED]>:
> In the case of BUILDS, I propose to do the same: define a standard whose
> cost/benefit ratios are ideally balanced for each participant.  This does
> not, by the way, mean that everybody ends up with the same cost/benefit
> ratio; it simply means that the cost/benefit ratios are best for those
> people whose participation is most required for the standard to be widely
> adopted.
>
> You can see that this is also what I did in the design of easy_install and
> setuptools, except that in that effort I only considered developers and
> users, not system packagers.

I'd argue (you may differ) that the most significant area where you
missed the mark on user benefits with easy_install and setuptools is
the lack of easy *uninstall* and easy *list* options.


Well, I'd certainly agree that those features are desirable.  But I 
didn't "miss the mark", in that those features were not part of my 
mark.  ;-)  The goal was to get widespread adoption by developers, 
and thus the primary target audience was developers.  Any features 
that attracted users were there only insofar as the benefit to users 
would specifically drive adoption by developers.


And that means that features that only matter *after* you have things 
installed (i.e., easy uninstall and list) were of relatively little 
importance in the initial feature set.  It was only necessary to 
provide the *possibility* of uninstall and list features, and then 
allow others to scratch the resulting itches.


Now, for BUILDS, the situation is different.  Just as setuptools' 
competitor was distutils, BUILDS' competitor is setuptools.  So, 
BUILDS has to offer developers a switching benefit.  And, unlike the 
situation before, now developers are much more likely to have a lot 
of packages they'd like to list or uninstall, so such features have 
considerably more value now, than they did then, which makes them 
worth putting some effort into.


Also, since that switching benefit is largely going to come from 
having better management tools, the management tools need to be easy 
to build (so that people will build them).


However, since system packagers are experiencing pain right now wrt 
Python packages, there is at least some benefit to their 
participating in the process.  Back when setuptools was developed, 
people from the distros weren't hanging out here in the 
distutils-sig...  or at least, they never answered any of my calls 
for interested parties.




 Most of the
issues I hear from users about setuptools (filtered by my prejudices,
admittedly) is that there's no management options (which brings in the
system packagers, and their concerns).

Can I suggest that this be included somehow in the new spec, so that
metadata is available to make wtiting uninstallers and listers as easy
as writing installers?


Since the single most important part of BUILDS (and the part that 
should be done first) is an installation manifest for packaging tools 
to know what files are of what kind and where, it should be easy for 
uninstallers to work.


Regarding listers, there are already a number of them available in 
the Cheeseshop, and if you're working with Python 2.5+ (and not using 
an older distro that deletes .egg-info files), they should also work 
with distutils-installed packages.


The idea behind the installation manifest and metadata (the "IL" and 
"DS" of BUILDS) is that it should be possible for anyone to write 
their own management tools for specific scenarios, and not have to 
rely on a bunch of implementation-specific and not-very-well-documented stuff.


(It should also be possible, given the "IL" part, to make a distutils 
and/or setuptools command that generates this data from a 
sufficiently well-behaved setup.py.)


___
Distutils-SIG maillist  -  Distutils-SIG@python.org
http://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Pre-pre-PEP: Requirements for the Python BUILDS Specification

[Josselin Mouette]

Le mardi 07 octobre 2008 à 10:02 -0400, Jean-Paul Calderone a écrit :
> The expectations of the Nevow developers was that a file included in Nevow,
> nevow_widget.py, would be installed to
> 
>   /usr/lib/python2.5/site-packages/twisted/plugins/nevow_widget.py

This expectation is wrong. You’re shipping it as a Python module, the
only thing you can expect is to be able to import it.

This wouldn’t happen if you were shipping plugins in a specific,
private, plugins directory.

> The ideal fix, from my perspective as a Twisted developer, is to install
> the nevow_widget.py file into
> 
>   /usr/lib/python2.5/site-packages/twisted/plugins/

As explained on the debian-python list (see
http://lists.debian.org/debian-python/2008/05/msg00032.html), there are
strong reasons for not putting managed modules in the same directory as
modules included in the Python distribution. So you can’t expect modules
to be in a specific directory (but this shouldn’t be a problem since
these are not modules, hmmm?).

(I have some ideas for keeping all files at the same place and moving
only .pyc files to /var, but it requires patching the interpreter and
the Debian Python maintainer disagrees, so we’ll have to stick to the
current situation for now.)

> along with all the other Twisted plugin files (perhaps using a symlink, I
> can't think of any reason why that would make a difference, particularly
> since all the other .py files in that area are symlinks).
> 
> In other words, Debian took advantage of a feature which was intended to be
> used only as a convenience for development and was never intended to assist
> installation.

The fact that we use a .pth to specify the different module hierarchy is
an implementation detail but is indeed abusing this feature; we should
be shipping a modified setup.py instead.

However this has nothing to do with the inability of Twisted to cope
with multiple module paths. Let’s take another example that is not
Debian specific: what if I want to install a plugin to /usr/local? The
fact that you are abusing the python modules directory forces me to
install it to /usr/local/python2.5/site-packages/twisted/plugins instead
of e.g. /usr/local/share/twisted/plugins, where a normal application or
library would be looking for its plugins. And there, I have to deal with
namespaces and other python module specificities that should be
irrelevant for plugins.

In the beginning, Debian only expected the Python modules directory to
contain, well, Python modules. It seems even that is too much to expect,
since module developers are taking advantage of a set of features
(module file location) that were initially meant for introspection.

> Have I succeeded in explaining why the Twisted plugin system isn't making
> any unusual requirements of the installation system nor using the Python
> module system in an unusual way?

Not really. You have succeeded in explaining what is the exact nature of
the unusual requirements you make, though. Thanks for that, as it is
much clearer than the original explanations I received.

> Can I expect to see any improvements to the Debian packaging of Twisted and
> projects which supply plugins to Twisted? :)

Until you fix your plugin system, we are bound to treat Twisted packages
as a special case.

Cheers,
-- 
 .''`.
: :' :  We are debian.org. Lower your prices, surrender your code.
`. `'   We will add your hardware and software distinctiveness to
  `-our own. Resistance is futile.


signature.asc
Description: Ceci est une partie de message	numériquement signée
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
http://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Pre-pre-PEP: Requirements for the Python BUILDS Specification

[Jean-Paul Calderone]

On Tue, 07 Oct 2008 15:19:57 +0200, Josselin Mouette <[EMAIL PROTECTED]> wrote:

Le mardi 07 octobre 2008 à 08:58 -0400, Jean-Paul Calderone a écrit :

I don't think the details of the plugin system are relevant to the topic
under discussion here.  The installation requirements are not unusual for
the most part - that a directory full of .py files be copied to the install
location, just as any package would be.  The one unusual thing is that
non-Twisted package X might want to copy files into twisted/plugins/ (if it
is providing plugins).


This is not unusual, but other frameworks that do the same don’t break
when you add namespace packages. In fact, the very fact that you can’t
add namespace packages shows that you are abusing the python modules
directory: you are shipping a twisted/plugins/foo/foo.py file which is
not meant to be available as a twisted.plugins.foo.foo module.


I think you meant "shipping a twisted/plugins/foo.py file which is not
meant to be available as a twisted.plugins.foo module", since as far as
I know, no one is putting packages (as opposed to modules) into the
twisted.plugins package (and I don't think it works, either).


The ability to install several incompatible versions of Twisted on the
system is the reason I was given by glyph; if there is another reason,
I’d be happy to hear the explanation, but the result is the same: we had
to code a special case for Twisted.



Above correction aside, your understanding of the system is still flawed
(I am fully aware that this is likely due to our failure to document the
system sufficiently, so please don't interpret that as an attack).  I
think that you also misunderstood the explanation that Glyph provided.

In your earlier message, you wrote this:


The reason is to allow plugins for several incompatible Twisted versions
to be installed in the *same* directory.


However, no such feature is offered by the Twisted plugin system. :)

The issue which arose when Debian packaged Twisted and (for example) a
new version of Nevow which provides a plugin to Twisted is that Nevow's
plugin was installed to a location which was not the location intended
or expected by the Twisted developers or the Nevow developers.  The
expectation of the Twisted developers was that the installed directory
structure for Twisted files would look like this:

 /usr/lib/python2.5/site-packages/twisted/
 /usr/lib/python2.5/site-packages/twisted/__init__.py
 /usr/lib/python2.5/site-packages/twisted/plugins/
 /usr/lib/python2.5/site-packages/twisted/plugins/__init__.py

(obviously this is only part of what a Twisted installation would look like)

The expectations of the Nevow developers was that a file included in Nevow,
nevow_widget.py, would be installed to

 /usr/lib/python2.5/site-packages/twisted/plugins/nevow_widget.py

On Debian, difficulties were encountered because, while Twisted was
installed more or less to the expected location, the file in Nevow
was instead installed to

 /var/lib/python-support/python2.5/twisted/plugins/nevow_widget.py

and the infrastructure for installing the file to this location also
creates

 /var/lib/python-support/python2.5/twisted/
 /var/lib/python-support/python2.5/twisted/__init__.py
 /var/lib/python-support/python2.5/twisted/plugins/
 /var/lib/python-support/python2.5/twisted/plugins/__init__.py

This is problematic because there are now two Twisted packages in the
module search path.  Python's module system can't do anything with the
second one (which one is second depends on the order of sys.path).  If
the version created for nevow_widget.py came first, the Twisted package
itself would basically be completely broken.  Fortunately, it doesn't,
so that doesn't happen.  However, since nevow_widget.py is in the second
version, the only way it can be imported is if some extra help is supplied
to the import system.

In previous releases of Twisted, this help was given by twisted/plugins/
__init__.py.  It included an extremely primitive implementation of the
concept of namespace packages (_extremely_ primitive).  It would find all
directories named "twisted/plugins" which were in the module search path
and add them to its own __path__.  The purpose of this feature wasn't to
support installations, however.  The purpose of this feature was as a
developer convenience.  It allows developers to edit plugins in a non-
installed location (for example, their home directory) and then use them
without copying them to the installed Twisted's twisted/plugins/ directory.

In a more recent release of Twisted, this feature was tweaked slightly.
The reason for the change is that there was a significant bug in the old
implementation of namespace packages.  If a developer was using their own
copy of Twisted (for example, an SVN checkout in their home directory)
instead of the installed version, the namespace package implementation
would still load plugins from the *installed* version of Twisted.  In the
best possible case, this gave you two copies

Re: [Distutils] Pre-pre-PEP: Requirements for the Python BUILDS Specification

[Josselin Mouette]

Le mardi 07 octobre 2008 à 08:58 -0400, Jean-Paul Calderone a écrit :
> I don't think the details of the plugin system are relevant to the topic
> under discussion here.  The installation requirements are not unusual for
> the most part - that a directory full of .py files be copied to the install
> location, just as any package would be.  The one unusual thing is that
> non-Twisted package X might want to copy files into twisted/plugins/ (if it
> is providing plugins).

This is not unusual, but other frameworks that do the same don’t break
when you add namespace packages. In fact, the very fact that you can’t
add namespace packages shows that you are abusing the python modules
directory: you are shipping a twisted/plugins/foo/foo.py file which is
not meant to be available as a twisted.plugins.foo.foo module.

The ability to install several incompatible versions of Twisted on the
system is the reason I was given by glyph; if there is another reason,
I’d be happy to hear the explanation, but the result is the same: we had
to code a special case for Twisted.

Cheers,
-- 
 .''`.
: :' :  We are debian.org. Lower your prices, surrender your code.
`. `'   We will add your hardware and software distinctiveness to
  `-our own. Resistance is futile.


signature.asc
Description: Ceci est une partie de message	numériquement signée
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
http://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Pre-pre-PEP: Requirements for the Python BUILDS Specification

[Jean-Paul Calderone]

On Tue, 7 Oct 2008 13:12:03 +0100, Floris Bruynooghe <[EMAIL PROTECTED]> wrote:

On Tue, Oct 07, 2008 at 08:01:28AM -0400, Jean-Paul Calderone wrote:

On Tue, 07 Oct 2008 09:57:56 +0200, Josselin Mouette <[EMAIL PROTECTED]> wrote:


Case 2: twisted. Plugins, consisting of a few .py files, are shipped in
a plugins/ subdirectory, and its content is dynamically added to
sys.path after some sanity checks (which fail when you add namespace
packages). The reason is to allow plugins for several incompatible
Twisted versions to be installed in the *same* directory. When there are
solutions as simple as versioning the plugin directory, I can’t
understand how people can think of (and implement) such broken
solutions.


The facts you have presented above are not correct.


It would be helpful if you could correct them in that case.



I don't think the details of the plugin system are relevant to the topic
under discussion here.  The installation requirements are not unusual for
the most part - that a directory full of .py files be copied to the install
location, just as any package would be.  The one unusual thing is that
non-Twisted package X might want to copy files into twisted/plugins/ (if it
is providing plugins).

For details, see the Twisted plugin documentation:

 http://twistedmatrix.com/projects/core/documentation/howto/plugin.html

Or this discussion in the Debian issue tracker:

 http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=474630

I would be happy to answer any questions not addressed in the documentation,
but perhaps off-list if they are not directly relevant to distribution
topics.

Jean-Paul
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
http://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Pre-pre-PEP: Requirements for the Python BUILDS Specification

[Floris Bruynooghe]

On Tue, Oct 07, 2008 at 08:01:28AM -0400, Jean-Paul Calderone wrote:
> On Tue, 07 Oct 2008 09:57:56 +0200, Josselin Mouette <[EMAIL PROTECTED]> 
> wrote:
>>
>> Case 2: twisted. Plugins, consisting of a few .py files, are shipped in
>> a plugins/ subdirectory, and its content is dynamically added to
>> sys.path after some sanity checks (which fail when you add namespace
>> packages). The reason is to allow plugins for several incompatible
>> Twisted versions to be installed in the *same* directory. When there are
>> solutions as simple as versioning the plugin directory, I can’t
>> understand how people can think of (and implement) such broken
>> solutions.
>
> The facts you have presented above are not correct.

It would be helpful if you could correct them in that case.


Cheers
Floris

-- 
Debian GNU/Linux -- The Power of Freedom
www.debian.org | www.gnu.org | www.kernel.org
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
http://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Pre-pre-PEP: Requirements for the Python BUILDS Specification

[Jean-Paul Calderone]

On Tue, 07 Oct 2008 09:57:56 +0200, Josselin Mouette <[EMAIL PROTECTED]> wrote:


Case 2: twisted. Plugins, consisting of a few .py files, are shipped in
a plugins/ subdirectory, and its content is dynamically added to
sys.path after some sanity checks (which fail when you add namespace
packages). The reason is to allow plugins for several incompatible
Twisted versions to be installed in the *same* directory. When there are
solutions as simple as versioning the plugin directory, I can’t
understand how people can think of (and implement) such broken
solutions.


Hi Josselin,

The facts you have presented above are not correct.

Jean-Paul
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
http://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Pre-pre-PEP: Requirements for the Python BUILDS Specification

[Paul Moore]

2008/10/7 Phillip J. Eby <[EMAIL PROTECTED]>:
> In the case of BUILDS, I propose to do the same: define a standard whose
> cost/benefit ratios are ideally balanced for each participant.  This does
> not, by the way, mean that everybody ends up with the same cost/benefit
> ratio; it simply means that the cost/benefit ratios are best for those
> people whose participation is most required for the standard to be widely
> adopted.
>
> You can see that this is also what I did in the design of easy_install and
> setuptools, except that in that effort I only considered developers and
> users, not system packagers.

I'd argue (you may differ) that the most significant area where you
missed the mark on user benefits with easy_install and setuptools is
the lack of easy *uninstall* and easy *list* options. Most of the
issues I hear from users about setuptools (filtered by my prejudices,
admittedly) is that there's no management options (which brings in the
system packagers, and their concerns).

Can I suggest that this be included somehow in the new spec, so that
metadata is available to make wtiting uninstallers and listers as easy
as writing installers?

Paul.
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
http://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Pre-pre-PEP: Requirements for the Python BUILDS Specification

[Josselin Mouette]

Le lundi 06 octobre 2008 à 21:33 -0400, Phillip J. Eby a écrit :
> >Does this mean actively avoiding an API that would allow developers to
> >access certain types of data files (I'm thinking of the discussion
> >about locale data and not putting anything else but .py/.pyc/.pyo
> >files in packages) or merely making sure the existing way (of shipping
> >data files in packages and finding them by os.path and __file__) keeps
> >working?
> 
> I would actively avoid it for a "BUILDS 1.0" spec, because on any 
> platform where the people building installation tools care about 
> relocating these files, symlinks are available, so both sides can be 
> happy without needing a new API.

I fail to see what this would bring over the current situation, then.

> That is, unless I have misunderstood Josselin and Toshio, I 
> understand symlinks to currently be an acceptable compromise.  (For 
> example, Debian uses them to relocate .pyc files currently.)

Symlinks are a real pain to handle. We can use them transparently
for .pyc files, but if we want to relocate data files to some other
directories, currently it has to be done by hand, and this is why most
maintainers don’t do it.

Furthermore, you seem to be unaware of the amount of abuse that was
produced by the mere idea of using __file__ to locate data. It is not
only about loading a few image files, let me show you some cases.

Case 1: pastescript. This module ships templates of python modules, that
are named with the .py extension but are not usable as is from the
interpreter, and shipped in the modules directory. How in the world can
our tools know that they shouldn’t be byte-compiled and that they should
be relocated to somewhere in /usr/share/$package?

Case 2: twisted. Plugins, consisting of a few .py files, are shipped in
a plugins/ subdirectory, and its content is dynamically added to
sys.path after some sanity checks (which fail when you add namespace
packages). The reason is to allow plugins for several incompatible
Twisted versions to be installed in the *same* directory. When there are
solutions as simple as versioning the plugin directory, I can’t
understand how people can think of (and implement) such broken
solutions.

Case 3: GStreamer. The modules installation directory is detected at
compile time and then hardcoded in the modules themselves so that some
subdirectories can be added to sys.path upon loading. In the end, the
modules are not relocatable.

I keep wondering what Python module developers will invent next to
complicate packaging even more. Frankly, we’d lose much less hair if
such things were simply impossible. At least, it should be documented
that such practice is wrong, so that we can actually consider it buggy
and make our tools simpler.

Cheers,
-- 
 .''`.
: :' :  We are debian.org. Lower your prices, surrender your code.
`. `'   We will add your hardware and software distinctiveness to
  `-our own. Resistance is futile.


signature.asc
Description: Ceci est une partie de message	numériquement signée
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
http://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Pre-pre-PEP: Requirements for the Python BUILDS Specification

[Phillip J. Eby]

At 10:25 PM 10/6/2008 +0100, Floris Bruynooghe wrote:

On Fri, Oct 03, 2008 at 12:35:11PM -0400, Phillip J. Eby wrote:
> I'm thinking about putting together a pre-PEP for a "Build Utilities,
> Installation Locations, & Distribution Standards" (BUILDS)
> specification.

Hehe, clever name!

> The basic idea for the first PEP is to:
[...]
> 2. Comment on some lessons learned from the WSGI definition process and
> WSGI in the field, and how they can be applied to the BUILDS process

That will be interesting to read.  Not being a web developer I have no
idea how WSGI came to be and what exactly it does.


WSGI came about because I was frustrated with the non-progress of the 
Web-SIG in creating standard request and response objects; it was 
quickly obvious to me that it would never go anywhere, due to 
socio/psych/economic considerations.  I proposed it in order to break 
the logjam by having a standard that more or less allowed everybody 
to do whatever they wanted and still be able to co-operate...  by 
arranging things so that the costs and benefits of adoption were 
balanced for all parties.


In the case of BUILDS, I propose to do the same: define a standard 
whose cost/benefit ratios are ideally balanced for each 
participant.  This does not, by the way, mean that everybody ends up 
with the same cost/benefit ratio; it simply means that the 
cost/benefit ratios are best for those people whose participation is 
most required for the standard to be widely adopted.


You can see that this is also what I did in the design of 
easy_install and setuptools, except that in that effort I only 
considered developers and users, not system packagers.  I propose to 
rectify that with BUILDS, although developers are still the most 
important audience a new standard must satisfy, in the sense that 
they require the best cost/benefit ratio for widespread adoption to 
occur.  (And this is of course in system packagers' best interest, 
since widespread adoption of the standard should make their lives 
easier, as long as the standard provides an improvement over the 
status quo for them.)


There are of course some other differences between this effort and 
those of WSGI and setuptools, but that's the 30-second summary.  :)




[...]
> 4. Lay out *non-goals* for the BUILDS project,
[...]
> or doing anything that requires
> them to change the runtime contents of their projects (as opposed to
> merely porting their setup.py, setup.cfg, etc.)

Does this mean actively avoiding an API that would allow developers to
access certain types of data files (I'm thinking of the discussion
about locale data and not putting anything else but .py/.pyc/.pyo
files in packages) or merely making sure the existing way (of shipping
data files in packages and finding them by os.path and __file__) keeps
working?


I would actively avoid it for a "BUILDS 1.0" spec, because on any 
platform where the people building installation tools care about 
relocating these files, symlinks are available, so both sides can be 
happy without needing a new API.


That is, unless I have misunderstood Josselin and Toshio, I 
understand symlinks to currently be an acceptable compromise.  (For 
example, Debian uses them to relocate .pyc files currently.)




> 5. Define rigorous terminology to be used for discussion of requirements
> and design, including such terms as "project", "release", "distribution",
> "system package", "installed distribution", etc.  (This is incredibly
> important, because the discussions we're having now are already having
> Tower-of-Babel confusions.)

Yay, please make sure the terminology finally allows us to talk about
packages in the .deb and .rpm meaning too.


The term I use for such packages is "system packages", btw.

___
Distutils-SIG maillist  -  Distutils-SIG@python.org
http://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Pre-pre-PEP: Requirements for the Python BUILDS Specification

[Floris Bruynooghe]

On Fri, Oct 03, 2008 at 12:35:11PM -0400, Phillip J. Eby wrote:
> I'm thinking about putting together a pre-PEP for a "Build Utilities,  
> Installation Locations, & Distribution Standards" (BUILDS)  
> specification.

Hehe, clever name!

> The basic idea for the first PEP is to:
[...]
> 2. Comment on some lessons learned from the WSGI definition process and 
> WSGI in the field, and how they can be applied to the BUILDS process

That will be interesting to read.  Not being a web developer I have no
idea how WSGI came to be and what exactly it does.

[...]
> 4. Lay out *non-goals* for the BUILDS project,
[...]
> or doing anything that requires  
> them to change the runtime contents of their projects (as opposed to  
> merely porting their setup.py, setup.cfg, etc.)

Does this mean actively avoiding an API that would allow developers to
access certain types of data files (I'm thinking of the discussion
about locale data and not putting anything else but .py/.pyc/.pyo
files in packages) or merely making sure the existing way (of shipping
data files in packages and finding them by os.path and __file__) keeps
working?


> 5. Define rigorous terminology to be used for discussion of requirements 
> and design, including such terms as "project", "release", "distribution", 
> "system package", "installed distribution", etc.  (This is incredibly 
> important, because the discussions we're having now are already having 
> Tower-of-Babel confusions.)

Yay, please make sure the terminology finally allows us to talk about
packages in the .deb and .rpm meaning too.  The current distutils
terminology doesn't and it really doesn't help.


> I also think we're going to 
> want a working group for this, or maybe multiple working groups, and it 
> might be best not to use the general distutils-SIG for discussion past 
> the first PEP, to allow people to filter threads better.

Hmm, this all seems quite on topic for distutils-sig.


Regards
Floris

-- 
Debian GNU/Linux -- The Power of Freedom
www.debian.org | www.gnu.org | www.kernel.org
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
http://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Pre-pre-PEP: Requirements for the Python BUILDS Specification

[Phillip J. Eby]

At 07:00 PM 10/3/2008 +0200, Tarek Ziade wrote:

Phillip,

You said it was too early, 2 days ago when I sent a similar (less 
detailed) mail where I expressed the need to start writing things 
down in Python Wiki .


I understood you to be specifying implementation designs, not 
creating a charter for requirements gathering.  And it is definitely 
still too early for that.




But this work has already started, by Thoshio, me, and some others. mail.

http://wiki.python.org/moin/Distribute/Functionality 
for instance.
or here for the overview, 
http://wiki.python.org/moin/Distribute


we are also hanging in #distutils

Can't you join ? and help on it ?


What is it you think I'm doing with the email you replied to?  ;-)

___
Distutils-SIG maillist  -  Distutils-SIG@python.org
http://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Pre-pre-PEP: Requirements for the Python BUILDS Specification

[Tarek Ziade]

Phillip,

You said it was too early, 2 days ago when I sent a similar (less detailed)
mail where I expressed the need to start writing things down in Python Wiki
.

But this work has already started, by Thoshio, me, and some others. mail.

http://wiki.python.org/moin/Distribute/Functionality for instance.
or here for the overview, http://wiki.python.org/moin/Distribute

we are also hanging in #distutils

Can't you join ? and help on it ?

We where waiting for that in fact..

Also , I am -1 on having another Mailing List for that. Distutils is the
place to be imho. Since main talks are about that.

And the working group is there : people that are talking for now a few
weeks, a bit before you joined the discussions.

Tarek

2008/10/3 Phillip J. Eby <[EMAIL PROTECTED]>

> I'm thinking about putting together a pre-PEP for a "Build Utilities,
> Installation Locations, & Distribution Standards" (BUILDS) specification.
>  But first, I want to throw out a few ideas to test the waters, and to give
> a general idea of what the first PEP would cover.
>
> The basic idea for the first PEP is to:
>
> 1. Give an overview of the current situation (problems w/distutils and
> setuptools, mainly, but also some of the successes)
>
> 2. Comment on some lessons learned from the WSGI definition process and
> WSGI in the field, and how they can be applied to the BUILDS process
>
> 3. Lay out high-level goals for the BUILDS project, such as:
>
>   * distributing responsibility/authority for build tools development
>   * adding extensibility to installation processes,
>   * providing a 100% open playing field for build & install tools,
>   * interoperability with the existing "egg" infrastructure, and
>   * interoperability (but not 100% backward-compatibility) with distutils
>   * allowing an incremental transition to the new standard
>
> 4. Lay out *non-goals* for the BUILDS project, such as trying to get
> developers to become system packagers or doing anything that requires them
> to change the runtime contents of their projects (as opposed to merely
> porting their setup.py, setup.cfg, etc.), defining and implementing the
> "perfect" system, etc.
>
> 5. Define rigorous terminology to be used for discussion of requirements
> and design, including such terms as "project", "release", "distribution",
> "system package", "installed distribution", etc.  (This is incredibly
> important, because the discussions we're having now are already having
> Tower-of-Babel confusions.)
>
> 6. Sketch an overall design concept (build libraries in the stdlib
> establishing a Python API for invoking build tools, that in turn build an
> installation manifest to be used by installation tools), but without
> specifying actual APIs, manifest format, or a full enumeration of release
> metadata.
>
> 7. Present a vision/plan for how migration can occur from current tools.
>
> 8. Set the scope for the PEPs that should follow, on the installation
> manifest format, build architecture, build tool API, compiler/config
> infrastructure, etc.
>
> Whew.  As you can see, just defining the problem adequately is a big job,
> and it may take a while to get even this first PEP right.  So, I'd like some
> feedback, if anybody has some ideas about what else needs to be added to
> this.
>
> I also don't want to end up writing all of the PEPs, or else it may be a
> year or so before they all get done.  ;-)  I also think we're going to want
> a working group for this, or maybe multiple working groups, and it might be
> best not to use the general distutils-SIG for discussion past the first PEP,
> to allow people to filter threads better.
>
> Anyway.  Thoughts, comments, volunteers, anyone?




>
> ___
> Distutils-SIG maillist  -  Distutils-SIG@python.org
> http://mail.python.org/mailman/listinfo/distutils-sig
>



-- 
Tarek Ziadé - Directeur Technique
INGENIWEB (TM) - SAS 5 Euros - RC B 438 725 632
Bureaux de la Colline - 1 rue Royale - Bâtiment D - 9ème étage
92210 Saint Cloud - France
Phone : 01.78.15.24.00 / Fax : 01 46 02 44 04
http://www.ingeniweb.com - une société du groupe Alter Way
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
http://mail.python.org/mailman/listinfo/distutils-sig