Re: [Python-Dev] [Distutils] how to easily consume just the parts of eggs that are good for you

[Stanley A. Klein]

On Wed, 2008-04-09 at 18:17 -0500, Dave Peterson wrote:
>   I think I can sum up any further points by simply asking: "Should it
> be safe to assume I can distribute my application via eggs /
> easy_install just because it is written in Python?"


I think that based on this discussion the bottom line answer to this
question is "No".


Stan Klein







On Wed, 2008-04-09 at 18:17 -0500, Dave Peterson wrote:

  I think I can sum up any further points by simply asking: "Should it be safe to assume I can distribute my application via eggs / easy_install just because it is written in Python?"



I think that based on this discussion the bottom line answer to this question is "No".


Stan Klein


___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] [Distutils] how to easily consume just the parts of eggs that are good for you

[Gael Varoquaux]

On Wed, Apr 09, 2008 at 11:52:08PM +0100, Paul Moore wrote:
> And I would say that Windows doesn't have a problem. Are any Windows
> users proposing building a package management system for Windows
> (Python-specific or otherwise)? It's a genuine question - is this
> something that Windows users are after, or is it just Linux users
> trying to show Windows users what they are missing?

Well, users don't phrase this that way, because they don't know what
package management (or rather automatic dependency tracking) is, but yes,
they are some usecases. It is nowadays really tedious to deploy Python
applications making uses of many packages on Python.

The scientific community is a domain in which this problem is crucial, as
we are trying to ship desktop applications to non-computer-savy people,
with many dependencies outside the standard library.

Enthought is working on shipping a Python distribution with some sort of
package management for this purpose ( see
http://code.enthought.com/enstaller/ ), and finding it is not an easy
problem.

Cheers,

Gael
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] [Distutils] how to easily consume just the parts of eggs that are good for you

[Gael Varoquaux]

On Wed, Apr 09, 2008 at 11:46:19PM +0100, Paul Moore wrote:
> I find this whole discussion hugely confusing, because a lot of people
> are stating opinions about environments which it seems they don't use,
> or know much about. I don't know how to avoid this, but it does make
> it highly unlikely that any practical progress will get made.

I find that something that doesn't help at all the discussion move
forward is that everybody has different usecases in mind, on different
platforms, and is not interested in other people's usecases.

Hopefuly I am wrong,

Cheers,

Gaël
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] [Distutils] how to easily consume just the parts of eggs that are good for you

[Stanley A. Klein]

On Wed, April 9, 2008 3:40 pm, Phillip J. Eby wrote:
> At 11:52 AM 4/9/2008 -0400, Stanley A. Klein wrote:
>>However, are you implying that the installation information for Python
>> egg
>>packages accesses and coordinates with the rpm database?
>
> Yes, when the information isn't stripped out.  Try a more recent Fedora.
>
>
>>IMHO, the main system without a package manager is Windows.
>
> You're ignoring shared environments and development
> environments.  (Not to mention Mac OS.)
>

I don't understand what you mean by "shared environments and development
 environments".  I also don't know much about Mac OS, except that its
underlying Darwin system is a version of BSD (that I assume would follow
the Unix FHS).


>
>>   A reasonable
>>way to deal with Windows would be to create a package manager for it that
>>could be used by Python and anyone else who wanted to use it.
>
> Let us know when you've finished it, along with the one for Mac OS.  :)

I have enough trouble with what I'm already doing.  :-)

>
> Of course this still won't do anything for shared environments and
> development environments.
>
>
>>You are talking here about bdist_rpm and not about a tool that would take
>>a Python package distributed as an egg file and convert the egg to an rpm
>>or a deb.  Unfortunately, some Python packagers are beginning to limit
>>their focus only to egg distribution.  That creates a problem for users
>>who have native operating system package management.
>
> That is indeed a problem -- but it's a social one, not a technical
> one.  It's trivial for the publisher of an egg to change their
> command line from "setup.py bdist_egg upload" to "setup.py sdist
> bdist_egg upload", as soon as their users (politely) request that they do
> so.
>

I agree that we are dealing with a combination of technical and social
issues here.  However, I think it takes a lot more understanding for a
publisher to get everything straight.

>
>> > Applying LSB and FHS to the innards of Python packages makes as much
>> > sense as applying them to the contents of Java .jar files -- i.e.,
>> > none.  If it's unchanging data that's part of a program or library,
>> > then it's a program or library, just like static data declared in a C
>> > program or library.  Whether the file extension is .py, .so, or even
>> > .png is irrelevant.
>>
>>The FHS defines places to put specific kinds of files, such as command
>>scripts (/bin, /usr/bin, /sbin, or /usr/sbin), documentation
>>(/usr/share/doc/package-name), and configuration files (/etc).  There are
>>several kinds of files identified and places defined to put them.
>>Distribution by eggs has a tendency to scoop up all of those files and
>> put
>>them in /usr/lib/python/site-packages, regardless of where they belong.
>
> Eggs don't include documentation or configuration files, and they
> install scripts in script directories, so I don't get what you're
> talking about here.  For any other data that a package accesses at
> runtime, my earlier comments apply.
>

But rpms and debs do include these files, plus manual pages, localization
files and a lot of other ancillary stuff.

IIRC, you once mentioned that you have a CENTOS system.  Do an "rpm -qa
|sort|less" to get an alphabetized list of your installed packages, and
then an "rpm -qil" on some of the packages, and you will see the range of
different kinds of files in there.

>
>>Having eggs support conformance to FHS would mean recognizing and tagging
>>the relevant files.  A tool for converting eggs to rpms or debs would
>>essentially reformat the egg to rpm or deb and put files where they
>>belong.
>
> No, because such files as you describe don't exist.  If you think
> they do, then either you have misunderstood the nature of the files
> in question, or the developer has incorrectly placed non-runtime
> files in their installation tree.
>

Most of the Python tarballs I have downloaded have all kinds of files in
their installation trees.  This is a major pain in the you-know-what for
someone trying to use bdist_rpm and get proper, FHS-compliant rpms.  If
eggs are supposed to be strictly runtime files, I think very few
developers actually understand that.  Better yet, how do you define what
should be included in an installation?  It sounds like the egg concept
doesn't include several kinds of files that rpm and deb would include in
an installation.  I think that may be an important issue here.


Stan Klein


___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] [Distutils] how to easily consume just the parts of eggs that are good for you

[Stanley A. Klein]

On Wed, April 9, 2008 3:19 pm, Gael Varoquaux wrote:
> On Wed, Apr 09, 2008 at 02:26:31PM -0400, Stanley A. Klein wrote:
>> The rpm and deb package managers (and their yum and other higher level
>> dependency managers) do a lot of things:
>
>> 1.  They install packages and maintain databases of what packages were
>> installed
>> 2.  They manage dependencies
>> 3.  They support clean uninstalling of packages
>> 4.  They can query packages, both installed (via their databases) and
>> not
>> yet installed (e.g., as rpm or deb files), to determine attributes, such
>> as files they install, dependencies, and other information defined at
>> packaging time.
>> 5.  They build packages and (in some cases) can rebuild packages.
>> 6.  They can verify packages for integrity and security purposes.
>> 7.  They can download package files and maintain archives of installed
>> package files for use as local repositories.
>
> You are collapsing three different functionalities in one:
>
>   * Dealing with repositories and downloading: yum/apt
>
>  * Installing + uninstalling packages, and dealing with system
>consistency (thus checking the dependencies are available): rpm/dpkg
>
>   * Building
>
> For me it is important that the 3 are separated:
>
>   * I may want to download the dependencies of a package to burn to a CD
> for a computer that does not have internet access.
>
>   * I may want to send a tarball to a build server that does the building,
> but no install (so as not to corrupt my working system).
>
> Cheers,
>
> Gaël
>

Gael -

The functionalities are combined in programs but are not necessarily
required to be used all at the same time.

I'm not that familiar with apt, but yum also installs, including
downloading both a package and its dependencies.  Yum also has a query
capability (yum list, yum info).  I think synaptic does the same thing yum
does, and adds a GUI and search capabilities similar to yum info as well.

The build capabilities of rpm were moved to rpmbuild, but the building
remains part of the rpm system.  IIRC, bdist_rpm actually calls rpmbuild
as part of its processing.

Also, IIRC, rpmbuild can build from a tarball if it contains an rpm spec. 
It does not install in the same process.  That is a separate step.  You
would not corrupt your working system by building an rpm from a tarball on
it.

BTW, I would not want to do dependencies with rpm if yum is available. 
Doing dependencies with rpm is very difficult and it is easy to wind up in
"dependency hell".  Yum will find the dependencies and install them as
long as they are in repositories that are registered in the yum
configuration.

I looked at "man yum" and couldn't find an option to download dependencies
to the local repository without installing.  However, if you did install a
package and its dependencies, and if you have selected the option of
retaining the cache and not cleaning it after installation, the rpms
(e.g., for updates) are in /var/cache/yum/updates/packages/.  They can be
copied from there to a CD for a system without internet connectivity. 
Also, both Fedora and Ubuntu have software for building installable live
CD's, although I don't know how they get their package files.


Stan Klein


___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] [Distutils] how to easily consume just the parts of eggs that are good for you

[Gael Varoquaux]

On Wed, Apr 09, 2008 at 02:26:31PM -0400, Stanley A. Klein wrote:
> The rpm and deb package managers (and their yum and other higher level
> dependency managers) do a lot of things:

> 1.  They install packages and maintain databases of what packages were
> installed
> 2.  They manage dependencies
> 3.  They support clean uninstalling of packages
> 4.  They can query packages, both installed (via their databases) and not
> yet installed (e.g., as rpm or deb files), to determine attributes, such
> as files they install, dependencies, and other information defined at
> packaging time.
> 5.  They build packages and (in some cases) can rebuild packages.
> 6.  They can verify packages for integrity and security purposes.
> 7.  They can download package files and maintain archives of installed
> package files for use as local repositories.

You are collapsing three different functionalities in one:

  * Dealing with repositories and downloading: yum/apt

 * Installing + uninstalling packages, and dealing with system
   consistency (thus checking the dependencies are available): rpm/dpkg

  * Building

For me it is important that the 3 are separated:

  * I may want to download the dependencies of a package to burn to a CD
for a computer that does not have internet access.

  * I may want to send a tarball to a build server that does the building,
but no install (so as not to corrupt my working system).

Cheers,

Gaël
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] [Distutils] how to easily consume just the parts of eggs that are good for you

[Stanley A. Klein]

All my development is done on Linux.  I use Windows very minimally (such
as for tax preparation) and unless forced to do so for specific
circumstances (such as submittal to grants.gov) do not expose Windows to
the Internet.

In the future there may possibly arise a need for us to port some
Linux-developed Python code to Windows, but we will have to cross that
bridge when we get there.

I think you raise an interesting issue:  What is a package manager?  I
have minimal experience installing packages on Windows over the last 5-10
years, but in my experience a Windows package comes as an executable that,
when run, installs itself.  Unless a third-party program monitors the
installation, uninstalling is a nasty chore, as is finding out what files
were installed or where they went.

The rpm and deb package managers (and their yum and other higher level
dependency managers) do a lot of things:

1.  They install packages and maintain databases of what packages were
installed
2.  They manage dependencies
3.  They support clean uninstalling of packages
4.  They can query packages, both installed (via their databases) and not
yet installed (e.g., as rpm or deb files), to determine attributes, such
as files they install, dependencies, and other information defined at
packaging time.
5.  They build packages and (in some cases) can rebuild packages.
6.  They can verify packages for integrity and security purposes.
7.  They can download package files and maintain archives of installed
package files for use as local repositories.

There may be other functions, but the above is a top-of-the-head list.

I can say that I'm not terribly happy with Python packaging that is only
minimally compatible with rpm.  I haven't used Ubuntu all that much. I do
like Ubuntu's packaging and package management, and I do know that there
are  programs, such as alias, that can translate from rpm to deb formats.

I don't think I ever said that Windows is broken in the area of package
management.  My own experience is that the files of Windows programs tend
to be put in a directory devoted to the program, rather than put in
directories with other files having similar purposes.  At one time, the
default location in Windows for word processing files was even in a
sub-directory of the word processing program.  That changed to having a
form of user home directory, but it didn't change much for the program
files themselves.  Unix/Linux puts the files in specific areas of the file
system having functional commonality.  One could almost say that the
Windows default approach to structuring its filesystem avoids or minimizes
the need for package management.

I repeat that this issue mainly arises because Windows doesn't have the
same kind of filesystem structure (and therefore the need for package
management) that other systems have.  I don't know what Windows add/remove
programs function does, but all it might do is to run the executable to
install packages and record the installation (as was previously done by
third party programs) to facilitate clean removal.  Unless you can perform
more of the other functions I listed above, I doubt I would call
add/remove a package manager.


Stan Klein



On Wed, April 9, 2008 1:23 pm, Paul Moore wrote:
> On 09/04/2008, Stanley A. Klein <[EMAIL PROTECTED]> wrote:
>> IMHO, the main system without a package manager is Windows.  A
>> reasonable
>>  way to deal with Windows would be to create a package manager for it
>> that
>>  could be used by Python and anyone else who wanted to use it.  The
>> package
>>  manager could establish a file hierarchy similar to the Unix FHS and
>>  install files appropriately, except for what is needed to satisfy the
>>  Windows OS.  That would probably go a long way to addressing the issues
>>  being discussed here.  This is primarily a Windows problem, not a
>> Python
>>  problem.
>
> Windows does have a package manager - the add/remove programs
> application. It's extremely limited, and doesn't make any attempt at
> doing dependency resolution, certainly - but that's a separate issue.
>
> I don't know if you use Windows (as in, develop programs using Python
> on Windows). If you do, then I'd be interested in your views on
> bdist_wininst and bdist_msi installers, and how they fit into the
> setuptools/egg environment, particularly with regard to the package
> manager you are proposing. If you don't use Windows, then I don't see
> how you can usefully comment.
>
> Personally, as I've said before, I don't have a problem with a
> Python-only package manager, as long as it replaces or integrates
> bdist_wininst and bdist_msi. Having two package managers is far worse
> than having none - and claiming that add/remove programs "isn't a
> package manager" is just ignoring reality (if it isn't, then why do
> bdist_wininst and bdist_msi exist?).
>
> Are the Linux users happy with having a Python package manager that
> ignores RPM/apt? Why should Windows users be any happier?
>
> Sorry - I'm f

Re: [Python-Dev] [Distutils] how to easily consume just the parts of eggs that are good for you

[Stanley A. Klein]

On Wed, April 9, 2008 12:41 am, Phillip J. Eby wrote:
> At 10:49 PM 4/8/2008 -0400, Stanley A. Klein wrote:
>>On Tue, April 8, 2008 9:37 pm, Ben Finney
>><[EMAIL PROTECTED]> wrote:
>> > Date: Wed, 09 Apr 2008 11:37:07 +1000
>> > From: Ben Finney <[EMAIL PROTECTED]>
>> > Subject: Re: [Distutils] how to easily consume just the parts of eggs
>> >   thatare good for you
>> > To: [EMAIL PROTECTED]
>> >
>> >
>> > zooko <[EMAIL PROTECTED]> writes:
>> >> eyes and said "So they are planning to reinvent apt!".
>> >
>> > That's pretty much my reaction, too.
>>
>>I have the same reaction.
>
> I'm curious.  Have any of you actually read PEP 262 in any detail?  I
> have seen precious little discussion so far that doesn't appear to be
> based on significant misunderstandings of either the purpose of
> reviving the PEP, or the mechanics of its proposed implementation.

I haven't read the PEP at all.  I generally don't read PEP's.

>
>
>>I have tried in the past to use easy_install, but have run into problems
>>because there is no communication between easy_install and the rpm
>>database, resulting in failure of easy_install to recognize that
>>dependencies have already been installed using rpms.
>
> This problem doesn't exist with Python 2.5, unless you're using a
> platform that willfully strips out the installation information that
> Python 2.5 provides for these packages.
>

IIRC, I have had the problem with Python 2.5 on Fedora 7.  Until recently,
Fedora packagers did strip out the egg information included with Python
packages they packaged.  I left those files in when packaging myself using
bdist_rpm.

However, are you implying that the installation information for Python egg
packages accesses and coordinates with the rpm database?  I found myself
having to go into the setup.py for the relevant package(s) and delete any
statements regarding dependencies.  Otherwise, IIRC, the packaging
couldn't proceed because the Python packaging tool couldn't find the
dependencies that had already been installed as rpms.  After installation,
Python managed to find the relevant files, but the packaging tool
couldn't.

>
>>A database focused only on Python packages is highly inappropriate for
>>Linux systems, violates the Linux standards, and creates problems because
>>eggs are not coordinated with the operating system package manager.
>
> The revamp of PEP 262 is aimed at removing .egg files and directories
> from the process, by allowing system packagers to tell Python what
> files belong to them and should not be messed with.  And conversely,
> allowing systems and installation targets *without* package managers
> to safely manage their Python installations.

IMHO, the main system without a package manager is Windows.  A reasonable
way to deal with Windows would be to create a package manager for it that
could be used by Python and anyone else who wanted to use it.  The package
manager could establish a file hierarchy similar to the Unix FHS and
install files appropriately, except for what is needed to satisfy the
Windows OS.  That would probably go a long way to addressing the issues
being discussed here.  This is primarily a Windows problem, not a Python
problem.


>
>
>>   The
>>way to achieve a database for Python would be to provide tools for
>>conversion of eggs to rpms and debs,
>
> Such tools already exist, although the conversion takes place from
> source distributions rather than egg distributions.
>

You are talking here about bdist_rpm and not about a tool that would take
a Python package distributed as an egg file and convert the egg to an rpm
or a deb.  Unfortunately, some Python packagers are beginning to limit
their focus only to egg distribution.  That creates a problem for users
who have native operating system package management.

>
>>to have eggs support conformance to
>>the LSB and FHS,
>
> Applying LSB and FHS to the innards of Python packages makes as much
> sense as applying them to the contents of Java .jar files -- i.e.,
> none.  If it's unchanging data that's part of a program or library,
> then it's a program or library, just like static data declared in a C
> program or library.  Whether the file extension is .py, .so, or even
> .png is irrelevant.

The FHS defines places to put specific kinds of files, such as command
scripts (/bin, /usr/bin, /sbin, or /usr/sbin), documentation
(/usr/share/doc/package-name), and configuration files (/etc).  There are
several kinds of files identified and places defined to put them. 
Distribution by eggs has a tendency to scoop up all of those files and put
them in /usr/lib/python/site-packages, regardless of where they belong. 
Having eggs support conformance to FHS would mean recognizing and tagging
the relevant files.  A tool for converting eggs to rpms or debs would
essentially reformat the egg to rpm or deb and put files where they
belong.


Stan Klein



___
Python-Dev mailing list
Python-Dev@python.org
htt

Re: [Python-Dev] [Distutils] how to easily consume just the parts of eggs that are good for you

[Gael Varoquaux]

On Wed, Apr 09, 2008 at 12:41:32AM -0400, Phillip J. Eby wrote:
> >The way to achieve a database for Python would be to provide tools for
> >conversion of eggs to rpms and debs,

> Such tools already exist, although the conversion takes place from 
> source distributions rather than egg distributions.

What is the status of the deb backend? The only one I know is unofficial
maintained by Andrew Straw, but my information my be lagging behind.

By the way, if these tools work well, they are priceless!

Cheers,

Gaël
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] [Distutils] how to easily consume just the parts of eggs that are good for you

[Gael Varoquaux]

On Wed, Apr 09, 2008 at 11:37:07AM +1000, Ben Finney wrote:
> zooko <[EMAIL PROTECTED]> writes:

> > I am skeptical that prorgammers are going to be willing to use a new
> > database format. They already have a database -- their filesystem --
> > and they already have the tools to control it -- mv, rm, and
> > PYTHONPATH. Many of them already hate the existence the
> > "easy_instlal.pth" database file, and I don't see why a new database
> > file would be any different.

> Moreover, many of us already have a database of *all* packages on the
> system, not just Python-language ones: the package database of our
> operating system. Adding another, parallel, database which needs
> separate maintenance, and only applies to Python packages, is not a
> step forward in such a situation.

90 % (at least) of the world does not have such database. I, and probably
you, have such a very nice database. I works well, and we can choose to
forget the problems our users are facing. It does not solve them though.

In addition, packaging is system-specific. I recently had to learn some
Debian packaging, because I wanted my Ubuntu and Debian users to be able
to use my projects seamlessly. What about RPMs for RHEL, Fedora,
Mandriva? ... and coronary packages? and MSIs? ... When do I find time to
do development if I have to learn all this packaging.

It would be fantastic to have an abstraction on all these packaging
systems, including, as you point out, their database. I do agree that
reusing the system packaging's database is great, and would be the best
option for system-wide install. However one of the very neat features of
setuptools and eggs is that you don't need administrator access to
install the packages, and that is great in a shared environment, like a
computation cluster. The system's database is thus unfortunately not a
complete solution to the problem.

My 2 cents,

Gaël
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] [Distutils] how to easily consume just the parts of eggs that are good for you

[Lloyd Kvam]

On Tue, 2008-04-08 at 10:01 -0700, zooko wrote:
> They both agreed that it made perfect sense.  I told one of them  
> about the alternate proposal to define a new database file to
> contain  
> a list of installed packages, and he sighed and rolled his eyes and  
> said "So they are planning to reinvent apt!".

When I wear my sysadmin hat, eggs become a nuisance.  They are not
listed in the system packages; if zipped they won't work when the apache
user tries to import them; easy_install can produce unexpected upgrades.
The system package manager (apt or yum) is much preferred.

As a developer, eggs are great.  If a python module is not already
available from my system packagers, easy_install will find it, get it,
and install it.  I waste almost no time with system administration
issues while developing.

Fortunately, distutils includes tools like bdist_rpm so that python
modules can be packaged for easy processing by the system package
manager.  So once I need to switch back to a sysadmin role, I can use
the system tools to install and track packages.

-- 
Lloyd Kvam
Venix Corp
DLSLUG/GNHLUG library
http://www.librarything.com/catalog/dlslug
http://www.librarything.com/profile/dlslug
http://www.librarything.com/rsshtml/recent/dlslug

___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] [Distutils] how to easily consume just the parts of eggs that are good for you

[Joachim König]

Phillip J. Eby wrote:
> It would be, if .eggs were a packaging format, rather than a binary 
> distribution/runtime format.
>
> Remember "eggs are to Python as jars are to Java" -- a Java .jar 
> doesn't contain documentation either, unless it's needed at 
> runtime.  Same for configuration files.
>   
But there's generally no need to easily have a look into a .class file 
with a tool the user
is used to whereas for python, we're often interested in knowing the 
details. And having
a zip file in my way to the source has left me frustrated often enough.

If you want to be consequent and consistent leave out the .py files from 
eggs, a jar file
normally doesn't contain .java sources files either.

> They're not system packages, in other words.  The assumption that 
> they are is another marketing failure, due to conflation of "package 
> == distribution of python code" and "package == thing you manage with 
> a system packager".  People see, "I put my package in an .egg" and 
> think it's the latter definition, when it's barely even the former.  :)
>   
I agree that they are not system packages, but I would have prefered to 
install multiple versions
of a package to separate "site-packages" directories, something that is 
really well understood by
most unsofisticated python programmers. The selection of the version 
could then be made at
runtime by a PYTHONPATH setting and not by fiddling with .pth files 
(something that could
be autmated by a tool and persisted in batch files).

___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] [Distutils] how to easily consume just the parts of eggs that are good for you

[Steven Bethard]

On Wed, Apr 9, 2008 at 4:48 PM, Paul Moore <[EMAIL PROTECTED]> wrote:
> On 09/04/2008, Phillip J. Eby <[EMAIL PROTECTED]> wrote:
>  > It would be, if .eggs were a packaging format, rather than a binary
>  >  distribution/runtime format.
>  >
>  >  Remember "eggs are to Python as jars are to Java" -- a Java .jar
>  >  doesn't contain documentation either, unless it's needed at
>  >  runtime.  Same for configuration files.
>
>  And yet, Java doesn't have an equivalent of easy_install for jar
>  files, to my knowledge. If Python had eggs but no easy_install, maybe
>  this whole discussion wouldn't be taking place.
>
>  (I know I personally like the idea of egg-as-jar-file, but *hate* the
>  idea of egg-as-dependency-handling-tool-and-everything-else).

So then do you really need anything more than the 2.6 support for
executing directories (and zipfiles) with a __main__ file?

$ python_d.exe --version
Python 2.6a1+

$ more foo\__main__.py
print 'here I am!'

$ python_d.exe foo
here I am!
[8698 refs]

$ python_d.exe foo.zip
here I am!
[8563 refs]

STeVe
-- 
I'm not *in*-sane. Indeed, I am so far *out* of sane that you appear a
tiny blip on the distant coast of sanity.
 --- Bucky Katt, Get Fuzzy
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] [Distutils] how to easily consume just the parts of eggs that are good for you

[Alexander Michael]

On Wed, Apr 9, 2008 at 3:40 PM, Phillip J. Eby <[EMAIL PROTECTED]> wrote:
>  That is indeed a problem -- but it's a social one, not a technical
>  one.  It's trivial for the publisher of an egg to change their
>  command line from "setup.py bdist_egg upload" to "setup.py sdist
>  bdist_egg upload", as soon as their users (politely) request that they do so.

I know you say it in
,
but perhaps you should be more explicit about the best practice being
to distribute an sdist as well as one or more eggs and highlight this
with some standout text like:

"""
When uploading your project to PyPI, always upload your sdist in
addition to any eggs unless you have a good idiosyncratic reason not
to. The easy_install tool can also download and install sdist's on all
platforms (universally for pure-python packages and provided compilers
are available for distributions containing python extensions), so
uploading them gives your project the widest possible audience.
"""

Perhaps this will help some the social challenges.

Regards,
Alex
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] [Distutils] how to easily consume just the parts of eggs that are good for you

[zooko]

On Apr 9, 2008, at 4:12 PM, Phillip J. Eby wrote:

>> http://allmydata.org/trac/setuptools/ticket/5 # binary eggs should
>> come with .py files by default, rather than .pyc files
>
> Filling your tracker with already-rejected proposals isn't likely  
> to encourage me to look at it, especially when I've personally  
> rejected them to you in IRC.  That goes for your ticket #4 as well.

Part of my motivation in maintaining this tracker is to take issue  
discussions from IRC, and from mailing lists, and make them more  
permanent and structured.  This part is useful even for rejected  
proposals, as an historical record that other people interested in  
those issues can consult.

I will mark those two tickets as "rejected by PJE".  Could you please  
repeat (so that I don't misrepresent you due to my faulty memory of  
our IRC discussion from more than a year ago) your reason for  
rejecting these two:

http://allmydata.org/trac/setuptools/ticket/4 (when considering  
whether to zip, err on the side of safety rather than performance)

http://allmydata.org/trac/setuptools/ticket/5 (binary eggs should  
come with .py files by default, rather than .pyc files)

You are of course welcome to log in to that trac and update those  
tickets yourself, but if you prefer not to then I will paste your  
reasons into those tickets.

Regards,

Zooko
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] [Distutils] how to easily consume just the parts of eggs that are good for you

[Phillip J. Eby]

At 12:51 AM 4/10/2008 +0200, Gael Varoquaux wrote:
>On Wed, Apr 09, 2008 at 11:46:19PM +0100, Paul Moore wrote:
> > I find this whole discussion hugely confusing, because a lot of people
> > are stating opinions about environments which it seems they don't use,
> > or know much about. I don't know how to avoid this, but it does make
> > it highly unlikely that any practical progress will get made.
>
>I find that something that doesn't help at all the discussion move
>forward is that everybody has different usecases in mind, on different
>platforms, and is not interested in other people's usecases.
>
>Hopefuly I am wrong,

You're not wrong at all.  I have to deal with *all* the platforms and 
use cases, which makes it quite frustrating when people who haven't 
even read the requirements are making proposals to "solve" things in 
ways that break for everyone except their own niche platform+usecase 
combination.

Guido, can I borrow the time machine and go back and NOT try to 
improve on the distutils?  Or is there already too much collateral 
damage to the timeline? ;-)

___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] [Distutils] how to easily consume just the parts of eggs that are good for you

[Dave Peterson]

Phillip J. Eby wrote:



Applying LSB and FHS to the innards of Python packages makes as much
sense as applying them to the contents of Java .jar files -- i.e.,
none.  If it's unchanging data that's part of a program or library,
then it's a program or library, just like static data declared in a C
program or library.  Whether the file extension is .py, .so, or even
.png is irrelevant.
  

The FHS defines places to put specific kinds of files, such as command
scripts (/bin, /usr/bin, /sbin, or /usr/sbin), documentation
(/usr/share/doc/package-name), and configuration files (/etc).  There are
several kinds of files identified and places defined to put them.
Distribution by eggs has a tendency to scoop up all of those files and put
them in /usr/lib/python/site-packages, regardless of where they belong.



Eggs don't include documentation or configuration files, and they 
install scripts in script directories, so I don't get what you're 
talking about here.  For any other data that a package accesses at 
runtime, my earlier comments apply.
  


We've talked a bit about this before, and IIRC, at that time you 
(Phillip) were willing to consider patches to setuptools that allowed 
for the inclusion of documentation in eggs such that it was placed into 
an LSB/FHS appropriate directory (or some standard dir for non-LSB 
systems) during the install process.   I'm assuming that something 
similar for config files wouldn't be a problem either?  Or is this whole 
idea out the window given the way the discussion has trended and the 
reiteration above that eggs are meant to be similar in principal to jars?  



Not that I have a patch yet, but we've been working on it in our "spare 
time" over here at Enthought.  I'm now wondering if we're wasting our 
time. :-)



I think the biggest use-case confusion in the current discussion is 
whether we're talking about applications or libraries?  If we're talking 
about libraries, then clearly distribution of only executables is 
sufficient because anything else should be handled by the application 
distribution when that library is used in an app.  Whereas if we're 
talking about applications, you probably *do* want to include 
documentation, configuration info, etc. in your distribution.   I think 
I can sum up any further points by simply asking: "Should it be safe to 
assume I can distribute my application via eggs / easy_install just 
because it is written in Python?"



-- Dave

___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] [Distutils] how to easily consume just the parts of eggs that are good for you

[Phillip J. Eby]

At 11:48 PM 4/9/2008 +0100, Paul Moore wrote:
>On 09/04/2008, Phillip J. Eby <[EMAIL PROTECTED]> wrote:
> > It would be, if .eggs were a packaging format, rather than a binary
> >  distribution/runtime format.
> >
> >  Remember "eggs are to Python as jars are to Java" -- a Java .jar
> >  doesn't contain documentation either, unless it's needed at
> >  runtime.  Same for configuration files.
>
>And yet, Java doesn't have an equivalent of easy_install for jar
>files, to my knowledge.

Actually, OSGi and Eclipse plugins and "feature sites" come quite 
close, and setuptools rips off many of its features from them.  OSGi 
is basically a standard for additional .jar metadata to encompass 
dependencies and other info.

___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] [Distutils] how to easily consume just the parts of eggs that are good for you

[Phillip J. Eby]

At 03:20 PM 4/9/2008 -0700, zooko wrote:
>I've opened a ticket on my setuptools trac about this proposal:
>
>http://allmydata.org/trac/setuptools/ticket/5 # binary eggs should
>come with .py files by default, rather than .pyc files

Filling your tracker with already-rejected proposals isn't likely to 
encourage me to look at it, especially when I've personally rejected 
them to you in IRC.  That goes for your ticket #4 as well.

___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] [Distutils] how to easily consume just the parts of eggs that are good for you

[Paul Moore]

On 09/04/2008, Ben Finney <[EMAIL PROTECTED]> wrote:
> "Stanley A. Klein" <[EMAIL PROTECTED]> writes:
>  > A reasonable way to deal with Windows would be to create a package
>  > manager for it that could be used by Python and anyone else who
> > wanted to use it. [...] This is primarily a Windows problem, not a
>  > Python problem.
>
>  I'd rephrase this as: If you *must* re-invent package management for
>  those legacy systems without it, please *don't* make it specific to
>  Python.

And I would say that Windows doesn't have a problem. Are any Windows
users proposing building a package management system for Windows
(Python-specific or otherwise)? It's a genuine question - is this
something that Windows users are after, or is it just Linux users
trying to show Windows users what they are missing?

(BTW, it's unreasonable to call Windows "legacy" - whatever that word
means, Windows ain't it (yet...))

Paul.
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] [Distutils] how to easily consume just the parts of eggs that are good for you

[Paul Moore]

On 09/04/2008, Phillip J. Eby <[EMAIL PROTECTED]> wrote:
> It would be, if .eggs were a packaging format, rather than a binary
>  distribution/runtime format.
>
>  Remember "eggs are to Python as jars are to Java" -- a Java .jar
>  doesn't contain documentation either, unless it's needed at
>  runtime.  Same for configuration files.

And yet, Java doesn't have an equivalent of easy_install for jar
files, to my knowledge. If Python had eggs but no easy_install, maybe
this whole discussion wouldn't be taking place.

(I know I personally like the idea of egg-as-jar-file, but *hate* the
idea of egg-as-dependency-handling-tool-and-everything-else).

Paul.
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] [Distutils] how to easily consume just the parts of eggs that are good for you

[Paul Moore]

On 09/04/2008, Stanley A. Klein <[EMAIL PROTECTED]> wrote:
>  I think you raise an interesting issue:  What is a package manager?

My (very simplistic) answer is that it's whatever someone uses to
manage packages. What level of functionality it has is irrelevant, as
long as it suits an individual's way of working.

>  I don't think I ever said that Windows is broken in the area of package
>  management.

I hope I didn't say you had - but it is an often-raised point, and it
does grate on me (as by implication, it says that what I do isn't
"real" package management). It's just another flavour of the old
Windows vs Linux OS wars, which I don't want to participate in :-)

>  My own experience is that the files of Windows programs tend
>  to be put in a directory devoted to the program, rather than put in
>  directories with other files having similar purposes.  At one time, the
>  default location in Windows for word processing files was even in a
>  sub-directory of the word processing program.  That changed to having a
>  form of user home directory, but it didn't change much for the program
>  files themselves.  Unix/Linux puts the files in specific areas of the file
>  system having functional commonality.  One could almost say that the
>  Windows default approach to structuring its filesystem avoids or minimizes
>  the need for package management.

Agreed. The minimal package manager on Windows is arguably reasonable
precisely because the standard layout doesn't require much more.

On the other hand, Microsoft has a bad habit of changing their
"standards", and in particular Vista changes a lot of things. But
that's very off-topic, so I'll avoid going into detail here.

>  I repeat that this issue mainly arises because Windows doesn't have the
>  same kind of filesystem structure (and therefore the need for package
>  management) that other systems have.  I don't know what Windows add/remove
>  programs function does, but all it might do is to run the executable to
>  install packages and record the installation (as was previously done by
>  third party programs) to facilitate clean removal.

Precisely. I could argue that the Linix filesystem structure is over
complex, and causes the need for complex package managers, precisely
because it's impossible to manually keep track of what file is owned
by what package. But this way lies OS wars, so I won't make a major
point of this, but just ask that people consider it as a possibility.

(I believe that Mac OS X goes for an even simpler structure -
applications store *everything* in the one directory, so that
install/uninstall is simply a directory copy/remove. I won't comment
on what that proves...)

>  Unless you can perform
>  more of the other functions I listed above, I doubt I would call
>  add/remove a package manager.

OK. I would, though, as I don't really want much more.

Later, you said:

> I just don't do Windows.  :-)

And you've been pretty clear that you're looking for information
rather than trying to explain how you think Windows should work. But
many people aren't so balanced in their views, and that makes it hard
to have a discussion - I'd suggest that people without Windows
experience leave the Windows side to the Windows people, and accept
that when they say "XXX won't work for us", that the statement is
probably true.

I find this whole discussion hugely confusing, because a lot of people
are stating opinions about environments which it seems they don't use,
or know much about. I don't know how to avoid this, but it does make
it highly unlikely that any practical progress will get made.

Paul.
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] [Distutils] how to easily consume just the parts of eggs that are good for you

[zooko]

On Apr 9, 2008, at 12:40 PM, Phillip J. Eby wrote:
>>
>> You are talking here about bdist_rpm and not about a tool that  
>> would take
>> a Python package distributed as an egg file and convert the egg to  
>> an rpm
>> or a deb.  Unfortunately, some Python packagers are beginning to  
>> limit
>> their focus only to egg distribution.  That creates a problem for  
>> users
>> who have native operating system package management.
>
> That is indeed a problem -- but it's a social one, not a technical
> one.  It's trivial for the publisher of an egg to change their
> command line from "setup.py bdist_egg upload" to "setup.py sdist
> bdist_egg upload", as soon as their users (politely) request that  
> they do so.

In general, it would be good if eggs came with .py files by default  
instead of .pyc files.

I've opened a ticket on my setuptools trac about this proposal:

http://allmydata.org/trac/setuptools/ticket/5 # binary eggs should  
come with .py files by default, rather than .pyc files

Regards,

Zooko
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] [Distutils] how to easily consume just the parts of eggs that are good for you

[zooko]

On Apr 9, 2008, at 6:00 AM, Phillip J. Eby wrote:
>>
>> By the way, if these tools work well, they are priceless!
>
> I haven't had need to use any of them, so I don't really know.

They are easydeb [1] and stddeb [2].  The former appears to be  
incomplete and unmaintained.  The latter appears to be usable, but  
somewhat incomplete -- substantial manual labor is required to use it  
successfully, as documented by my programming partner Brian Warner in  
this ticket: [3].

Regards,

Zooko

[1] http://easy-deb.sourceforge.net/
[2] http://stdeb.python-hosting.com/
[3] http://allmydata.org/trac/tahoe/ticket/251
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] [Distutils] how to easily consume just the parts of eggs that are good for you

[Phillip J. Eby]

At 04:43 PM 4/9/2008 -0400, Stanley A. Klein wrote:
>I don't understand what you mean by "shared environments and development
>  environments".

I mean that in a shared or development environment, a system packager 
isn't useful, since it expects things to live in *one* place, and 
usually to have only one *version*, as well.


>I agree that we are dealing with a combination of technical and social
>issues here.  However, I think it takes a lot more understanding for a
>publisher to get everything straight.

If they provide you with the source distribution, you can make any 
sort of package you want.


> > Eggs don't include documentation or configuration files, and they
> > install scripts in script directories, so I don't get what you're
> > talking about here.  For any other data that a package accesses at
> > runtime, my earlier comments apply.
> >
>
>But rpms and debs do include these files, plus manual pages, localization
>files and a lot of other ancillary stuff.

...just one of the many reasons that eggs are not a replacement for 
rpms and debs.  :)


>Most of the Python tarballs I have downloaded have all kinds of files in
>their installation trees.  This is a major pain in the you-know-what for
>someone trying to use bdist_rpm and get proper, FHS-compliant rpms.  If
>eggs are supposed to be strictly runtime files, I think very few
>developers actually understand that.  Better yet, how do you define what
>should be included in an installation?  It sounds like the egg concept
>doesn't include several kinds of files that rpm and deb would include in
>an installation.  I think that may be an important issue here.

It would be, if .eggs were a packaging format, rather than a binary 
distribution/runtime format.

Remember "eggs are to Python as jars are to Java" -- a Java .jar 
doesn't contain documentation either, unless it's needed at 
runtime.  Same for configuration files.

They're not system packages, in other words.  The assumption that 
they are is another marketing failure, due to conflation of "package 
== distribution of python code" and "package == thing you manage with 
a system packager".  People see, "I put my package in an .egg" and 
think it's the latter definition, when it's barely even the former.  :)

___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] [Distutils] how to easily consume just the parts of eggs that are good for you

[Phillip J. Eby]

At 10:30 AM 4/9/2008 -0700, zooko wrote:
>PEP 262 sounds like a non-starter to me because
>
>1.  It appears to be backwards-incompatible with setuptools/ 
>easy_install/eggs, thus losing all the recently gained cooperation
>that I mentioned in the previous paragraph, and

No.  It provides a forward-compatibility path for the distutils, so 
that easy_install doesn't have to install things in .egg format in 
the future.  There's no compatibility breakage at all.


>2.  It defines a new database file,

No, it defines *files*.  One file per installed distribution, 
containing (among other things) an installation manifest.


>where I would prefer either:
>a.  Doing away with database files entirely and relying on the
>filesystem alone to hold that information, or

...which is what PEP 262 *does*.

Unfortunately, PEP 262's title is bad for marketing, as you've 
effectively pointed out.  It would be better titled something 
"package installation manifests" or "package contents files", or 
something of that sort.

>b.  Continuing to use the current ".pth" database file format,
>possibly improved by having native support for .pth files in the
>Python import machinery.

These mechanisms are orthogonal to this issue.


>3.  Because of #2, it triggers programmers to exclaim "They are
>planning to reinvent apt!", thus making it unlikely that the new
>proposal will recapture the cooperation that setuptools has already
>(slowly) gained.

Yeah, we need a new name.  Everybody is going off of "database of 
installed packages" and thinking "apt", because they aren't paying 
any closer attention.  However, given that we are discussing this on 
Python-Dev and distutils-sig, I do think it's reasonable to expect 
(if perhaps not reasonable to receive) that people discussing the PEP 
have *read* the freaking PEP first, prior to trashing it or offering 
alternatives.

And it's not like I'm personally offended or anything -- I didn't 
even write the PEP in question.  But what's the point of having PEPs 
if people read nothing but their titles?  We could just delete 
everything but PEP 0.  :)


>Perhaps PEP 262 and my proposal are not actually alternatives, but
>are complementary.

As I've already pointed out, your proposal does not address multiple 
installed versions of a package, and I see no sane way to modify it to do so.


>What I want is for the already implemented, tested, and deployed 
>code- re-use features of setuptools/easy_install to be more widely
>accepted.  This is best and most easily achieved by fixing the two
>most frequent objections to setuptools/easy_install: 1.  That you
>can't conveniently install into an arbitrary directory, and 2. that
>it subverts the meaning of your PYTHONPATH.

As I've already stated, the only way for these problems to be fixed 
is for easy_install to not install files in .egg form -- which also 
solves the general objection to using .eggs in the first place.

And the only way to do that, is to have a way to keep track of what 
files are installed.  Rather than have easy_install come up with its 
own way of doing that, I would prefer to share a standard with the 
distutils.  Hence, the PEP discussion.

For earlier versions of Python, it will still be possible to install 
and uninstall with setuptools using this approach.  You just won't be 
able to uninstall pure distutils-based packages, unless you installed 
them using easy_install.

Meanwhile, it has occurred to me that the easiest way of handling 
compatibility is not to require that other packaging tools mark their 
files for non-removability, but simply not allow easy_install to 
remove or overwrite anything that *isn't* claimed by a manifest.  In 
that way, easy_install would be immediately usable in the new mode, 
without any updates to Python or to system packaging tools.

___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] [Distutils] how to easily consume just the parts of eggs that are good for you

[Phillip J. Eby]

At 11:52 AM 4/9/2008 -0400, Stanley A. Klein wrote:
>However, are you implying that the installation information for Python egg
>packages accesses and coordinates with the rpm database?

Yes, when the information isn't stripped out.  Try a more recent Fedora.


>IMHO, the main system without a package manager is Windows.

You're ignoring shared environments and development 
environments.  (Not to mention Mac OS.)


>   A reasonable
>way to deal with Windows would be to create a package manager for it that
>could be used by Python and anyone else who wanted to use it.

Let us know when you've finished it, along with the one for Mac OS.  :)

Of course this still won't do anything for shared environments and 
development environments.


>You are talking here about bdist_rpm and not about a tool that would take
>a Python package distributed as an egg file and convert the egg to an rpm
>or a deb.  Unfortunately, some Python packagers are beginning to limit
>their focus only to egg distribution.  That creates a problem for users
>who have native operating system package management.

That is indeed a problem -- but it's a social one, not a technical 
one.  It's trivial for the publisher of an egg to change their 
command line from "setup.py bdist_egg upload" to "setup.py sdist 
bdist_egg upload", as soon as their users (politely) request that they do so.


> > Applying LSB and FHS to the innards of Python packages makes as much
> > sense as applying them to the contents of Java .jar files -- i.e.,
> > none.  If it's unchanging data that's part of a program or library,
> > then it's a program or library, just like static data declared in a C
> > program or library.  Whether the file extension is .py, .so, or even
> > .png is irrelevant.
>
>The FHS defines places to put specific kinds of files, such as command
>scripts (/bin, /usr/bin, /sbin, or /usr/sbin), documentation
>(/usr/share/doc/package-name), and configuration files (/etc).  There are
>several kinds of files identified and places defined to put them.
>Distribution by eggs has a tendency to scoop up all of those files and put
>them in /usr/lib/python/site-packages, regardless of where they belong.

Eggs don't include documentation or configuration files, and they 
install scripts in script directories, so I don't get what you're 
talking about here.  For any other data that a package accesses at 
runtime, my earlier comments apply.


>Having eggs support conformance to FHS would mean recognizing and tagging
>the relevant files.  A tool for converting eggs to rpms or debs would
>essentially reformat the egg to rpm or deb and put files where they
>belong.

No, because such files as you describe don't exist.  If you think 
they do, then either you have misunderstood the nature of the files 
in question, or the developer has incorrectly placed non-runtime 
files in their installation tree.

___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] [Distutils] how to easily consume just the parts of eggs that are good for you

[zooko]

On Apr 8, 2008, at 9:41 PM, Phillip J. Eby wrote:
>
> I'm curious.  Have any of you actually read PEP 262 in any detail?

I read it, though not in fine detail.

I didn't write that you are planning to reinvent apt.  I wrote that  
when programmers hear about this PEP they exclaim "They are planning  
to reinvent apt!".

That is a matter of perception and marketing -- the value that I want  
to gain from Python packages is the value of a critical mass of good  
programmers using compatible tools for code re-use.  If a lot of  
programmers hate an idea, then it doesn't matter what the details are  
-- it isn't going to provide this value to me.

I think part of our disagreement is that we are talking about two  
overlapping use cases: programmer and sysadmin (and "end user" which  
is much like sysadmin).  I am not, at this time, interested in the  
sysadmin use case.  As I've mentioned, my sysadmin needs are  
currently well satisfied by apt (and sometimes by GNU stow), and my  
fellow sysadmins with whom I work are absolutely not going to relax  
their "apt-only policy" for the forseeable future, so I cannot use  
such a tool unless it is named "apt" and written by Debian/Ubuntu.

On the other hand I am very interested in the programmer use case,  
because setuptools/easy_install already works pretty well for that,  
and we are already very close to achieving a critical mass of good  
programmers.  Recently several more packages that my project [1]  
relies on have become easy_installable -- Twisted, pywin32 (thanks to  
you, PJE), and foolscap -- and several more have had bugfixes which  
make them work better with easy_install/setuptools -- Nevow and  
zope.interface -- and there are some patches in the queue to make  
another one compatible with setuptools -- pyflakes [2, 3].

So setuptools/easy_install is already (slowly) winning.  I want to  
accelerate that process by reducing the degree to which it is  
incompatible, inconvenient, or objectionable to other programmers.

PEP 262 sounds like a non-starter to me because

1.  It appears to be backwards-incompatible with setuptools/ 
easy_install/eggs, thus losing all the recently gained cooperation  
that I mentioned in the previous paragraph, and

2.  It defines a new database file, where I would prefer either:
a.  Doing away with database files entirely and relying on the  
filesystem alone to hold that information, or
b.  Continuing to use the current ".pth" database file format,  
possibly improved by having native support for .pth files in the  
Python import machinery.

3.  Because of #2, it triggers programmers to exclaim "They are  
planning to reinvent apt!", thus making it unlikely that the new  
proposal will recapture the cooperation that setuptools has already  
(slowly) gained.

I'm sorry, PJE -- I know it must be frustrating to you to have your  
proposal criticized on social rather than technical grounds -- but  
social benefits are what I am seeking right now.

Perhaps PEP 262 and my proposal are not actually alternatives, but  
are complementary.  I do not object to Python maintaining a database  
of installed packages for itself (although I cannot *rely* upon such  
behavior, not least because I will be maintaining backwards  
compatibility with Python 2.4 for at least the next several years,  
and with Python 2.5 for at least the next several years after that).  
What I want is for the already implemented, tested, and deployed code- 
re-use features of setuptools/easy_install to be more widely  
accepted.  This is best and most easily achieved by fixing the two  
most frequent objections to setuptools/easy_install: 1.  That you  
can't conveniently install into an arbitrary directory, and 2. that  
it subverts the meaning of your PYTHONPATH.

Regards,

Zooko

[1] http://allmydata.org/source/tahoe/trunk/docs/install.html
[2] http://divmod.org/trac/ticket/2535
[3] http://divmod.org/trac/ticket/2048
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] [Distutils] how to easily consume just the parts of eggs that are good for you

[Paul Moore]

On 09/04/2008, Stanley A. Klein <[EMAIL PROTECTED]> wrote:
> IMHO, the main system without a package manager is Windows.  A reasonable
>  way to deal with Windows would be to create a package manager for it that
>  could be used by Python and anyone else who wanted to use it.  The package
>  manager could establish a file hierarchy similar to the Unix FHS and
>  install files appropriately, except for what is needed to satisfy the
>  Windows OS.  That would probably go a long way to addressing the issues
>  being discussed here.  This is primarily a Windows problem, not a Python
>  problem.

Windows does have a package manager - the add/remove programs
application. It's extremely limited, and doesn't make any attempt at
doing dependency resolution, certainly - but that's a separate issue.

I don't know if you use Windows (as in, develop programs using Python
on Windows). If you do, then I'd be interested in your views on
bdist_wininst and bdist_msi installers, and how they fit into the
setuptools/egg environment, particularly with regard to the package
manager you are proposing. If you don't use Windows, then I don't see
how you can usefully comment.

Personally, as I've said before, I don't have a problem with a
Python-only package manager, as long as it replaces or integrates
bdist_wininst and bdist_msi. Having two package managers is far worse
than having none - and claiming that add/remove programs "isn't a
package manager" is just ignoring reality (if it isn't, then why do
bdist_wininst and bdist_msi exist?).

Are the Linux users happy with having a Python package manager that
ignores RPM/apt? Why should Windows users be any happier?

Sorry - I'm feeling a little grumpy. I've read one too many "Windows
is so broken that people who use it obviously don't care about doing
things right" postings this week :-(

Paul.
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] [Distutils] how to easily consume just the parts of eggs that are good for you

[Phillip J. Eby]

At 10:00 AM 4/9/2008 +0200, Gael Varoquaux wrote:
>On Wed, Apr 09, 2008 at 12:41:32AM -0400, Phillip J. Eby wrote:
> > >The way to achieve a database for Python would be to provide tools for
> > >conversion of eggs to rpms and debs,
>
> > Such tools already exist, although the conversion takes place from
> > source distributions rather than egg distributions.
>
>What is the status of the deb backend? The only one I know is unofficial
>maintained by Andrew Straw, but my information my be lagging behind.

I was under the impression that there were 2 .deb tools, neither one 
"official" in any sense, any more than 'bdist_rpm' is really 
"official" for RPM-based systems.


>By the way, if these tools work well, they are priceless!

I haven't had need to use any of them, so I don't really know.

___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] [Distutils] how to easily consume just the parts of eggs that are good for you

[Phillip J. Eby]

At 10:49 PM 4/8/2008 -0400, Stanley A. Klein wrote:
>On Tue, April 8, 2008 9:37 pm, Ben Finney
><[EMAIL PROTECTED]> wrote:
> > Date: Wed, 09 Apr 2008 11:37:07 +1000
> > From: Ben Finney <[EMAIL PROTECTED]>
> > Subject: Re: [Distutils] how to easily consume just the parts of eggs
> >   thatare good for you
> > To: [EMAIL PROTECTED]
> >
> >
> > zooko <[EMAIL PROTECTED]> writes:
> >> eyes and said "So they are planning to reinvent apt!".
> >
> > That's pretty much my reaction, too.
>
>I have the same reaction.

I'm curious.  Have any of you actually read PEP 262 in any detail?  I 
have seen precious little discussion so far that doesn't appear to be 
based on significant misunderstandings of either the purpose of 
reviving the PEP, or the mechanics of its proposed implementation.


>I have tried in the past to use easy_install, but have run into problems
>because there is no communication between easy_install and the rpm
>database, resulting in failure of easy_install to recognize that
>dependencies have already been installed using rpms.

This problem doesn't exist with Python 2.5, unless you're using a 
platform that willfully strips out the installation information that 
Python 2.5 provides for these packages.


>A database focused only on Python packages is highly inappropriate for
>Linux systems, violates the Linux standards, and creates problems because
>eggs are not coordinated with the operating system package manager.

The revamp of PEP 262 is aimed at removing .egg files and directories 
from the process, by allowing system packagers to tell Python what 
files belong to them and should not be messed with.  And conversely, 
allowing systems and installation targets *without* package managers 
to safely manage their Python installations.


>   The
>way to achieve a database for Python would be to provide tools for
>conversion of eggs to rpms and debs,

Such tools already exist, although the conversion takes place from 
source distributions rather than egg distributions.


>to have eggs support conformance to
>the LSB and FHS,

Applying LSB and FHS to the innards of Python packages makes as much 
sense as applying them to the contents of Java .jar files -- i.e., 
none.  If it's unchanging data that's part of a program or library, 
then it's a program or library, just like static data declared in a C 
program or library.  Whether the file extension is .py, .so, or even 
.png is irrelevant.

___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] [Distutils] how to easily consume just the parts of eggs that are good for you

[zooko]

On Apr 8, 2008, at 11:27 AM, Lloyd Kvam wrote:
>
> When I wear my sysadmin hat, eggs become a nuisance.
...
> As a developer, eggs are great.
...
> Fortunately, distutils includes tools like bdist_rpm so that python
> modules can be packaged for easy processing by the system package
> manager.  So once I need to switch back to a sysadmin role, I can use
> the system tools to install and track packages.

This is the same experience I have.  I rely on setuptools and eggs  
extensively in developing our software, and I use setuptools and eggs  
as the primary method of giving our source code to other programmers.

But no software is ever installed on our production servers unless  
that software is in .deb form in an apt-gettable repository, and this  
requirement is unlikely to change for the forseeable future.

Regards,

Zooko

___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com