[Distutils] Get install prefix for module at runtime
Hi all,
i am pretty new to distutils (since yesterday) and am struggling with
accessing installed data_files. I want to ship a library that benefits
from some data_files from another package.
My wishes:
* distribution should go in 'PREFIX/lib/foo' (site-packages?)
* datafiles should go in 'PREFIX/share/foo'
* Access to data files from modules
My question is: "How to reliably access data files from a module in
foo?"
A thorough explanation follows. If you can answer the aforementioned
question just think tl;dr and skip it ;-)
Installation Schemes
There seem to be three different installation schemes regarding setting
of this PREFIX. The following examples assume installation of package
'foo' which contains module 'bar' and data file 'baz.data'.
Default install (setup.py install)
-
* Packages installation path:
'sys.prefix/lib/pythonX.Y/site-packages/'
* Data files installation path:
'sys.prefix/share/'
That means that in the aforementioned example the following file system
structure will be created upon installation:
Package:/usr/lib/python2.5/site-packages/foo/bar.py
Data: /usr/share/foo/baz.data
Home scheme (setup.py install --home=HOME)
---
* Packages installation path:
'HOME/lib/python'
* Data files installation path:
'HOME/share/'
Resulting structure:
Package:HOME/lib/foo/bar.py
DataHOME/share/foo/baz.data
Prefix scheme (setup.py --prefix=PREFIX)
* Package installation path:
'PREFIX/lib/pythonX.Y/site-packages'
* Data installation path:
'PREFIX/share/'
Resulting structure (for PREFIX==/usr/local)
Package:/usr/local/lib/python2.5/site-packages/foo/bar.py
Data: /usr/share/foo/baz.data
Finding Data files
==
My problem is finding a reliable way to access the data files from
within bar.py.
Approach 1 - using module.__file__
--
bar_path_elements = bar.__file__.split(os.path.sep)
prefix = os.path.join(
os.path.sep,
*itertools.takewhile(lambda el: el != 'lib'))
data_file_path = os.path.join(prefix, 'share', 'foo')
* Will this work reliably?
* What if the user specifies a different data directory with
'--install-data' ?
Approach 2 - writing a build.py file at installation time
-
The last question made me think, that Approach 1 works for the three
standard installation schemes, but fails miserably if the user decides
to do anything fancy. As i do not comprehend all the options you can give
to 'setup.py install' or know about them i am not sure if there are not
many more problems hidden in there.
The solution seems to be to create a build.py file that includes this
information at installation time and access the data there. My problem
is just: HOW?
I tried with the following setup.py script, but am unsure how to access
the correct options:
--- snip ---
# -*- coding: UTF-8 -*-
#!/usr/bin/env python
from __future__ import with_statement
from distutils.core import setup
import sys
import getopt
def create_build_file():
try:
opts, args = getopt.getopt(sys.argv[1:], "", [])
What to do here
except getopt.GetoptError, err:
pass
with open('lib/mwdb/build.py', 'w') as build_fp:
#build_fp.write('# -*- coding: UTF-8 -*-\n')
#build_fp.write("LIB_PREFIX = '%s'\n"%(options.lib_prefix))
#build_fp.write("DATA_PREFIX = '%s'\n"%(options.data_prefix))
#build_fp.write("EXEC_PREFIX = '%s'\n"%(options.exec_prefix))
try:
if 'install' == sys.argv[1]:
create_build_file()
except IndexError:
pass
setup(name='foo',
version='0.1',
packages=['foo']
package_dir = { '':'lib' },
...
)
--- snip ---
The problem i am facing now is: How to parse the options? Do I really have
to recreate the complete distutils option parsing logic - and probably
fail? All that without optparse! How can i get the aforementioned
options? Is the file creation way the correct way to handle this?
I really don't know what to do now and am also wondering why nobody had
that problem before! Are all other libraries shipped with external data
buggy for some installation schemes? How is this normally solved?
with kind regards and thanks for reading all this
Wolodja Wentland
signature.asc
Description: Digital signature
___
Distutils-SIG maillist - Distutils-SIG@python.org
http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Get install prefix for module at runtime
On Tue, Sep 15, 2009 at 18:37 +0200, Tarek Ziadé wrote: > > At 05:38 PM 9/15/2009 +0200, Wolodja Wentland wrote: >>> How is this normally solved? > Note #2: Beware that this also assumes that you use Setuptools instead > of Distutils for your project. I don't actually plan to use setuptools but distutils, although i will definitely consider if there is no other way. > I'll answer for a Distutils solution as soon as I have some time > (hopefully tonight) That would be great! with kind regards Wolodja Wentland signature.asc Description: Digital signature ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
[Distutils] Greylisting?
Hi all, i've just seen the following in a mail header: --- snip --- X-Greylist: delayed 2135 seconds by postgrey-1.31 at albatross; Tue, 15 Sep 2009 18:14:38 CEST --- snip --- Why do my mail got delayed for more than 35 min? Is there anything I, as a subscriber to this list, can do about it? with kind regards Wolodja Wentland signature.asc Description: Digital signature ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Get install prefix for module at runtime
On Tue, Sep 15, 2009 at 17:38 +0200, Wolodja Wentland wrote:
> My question is: "How to reliably access data files from a module in
> foo?"
> Approach 2 - writing a build.py file at installation time
> -
>
> The last question made me think, that Approach 1 works for the three
> standard installation schemes, but fails miserably if the user decides
> to do anything fancy.
I implemented this approach like this:
--- snip ---
# -*- coding: UTF-8 -*-
#!/usr/bin/env python
from __future__ import with_statement
import distutils.core as core
from distutils.command.install import install as _install
import sys
import getopt
import os.path
class install(_install):
"""install command
This specific install command will create a file 'build_config.py' which
contains information on """
def run(self):
with open('lib/foo/build_config.py', 'w') as build_fp:
build_fp.write('# -*- coding: UTF-8 -*-\n\n')
build_fp.write("DATA_DIR = '%s'\n"%(
os.path.join(self.install_data, 'share')))
build_fp.write("LIB_DIR = '%s'\n"%(self.install_lib))
build_fp.write("SCRIPT_DIR = '%s'\n"%(self.install_scripts))
_install.run(self)
core.setup(name='foo',
version='0.1',
description='Foo is Bar and Baz',
author='Wolodja Wentland',
...
)
--- snip ---
Is this a good approach? Is there anything i can do better?
with kind regards
Wolodja Wentland
signature.asc
Description: Digital signature
___
Distutils-SIG maillist - Distutils-SIG@python.org
http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Get install prefix for module at runtime
On Wed, Sep 16, 2009 at 17:40 +0200, Wolodja Wentland wrote:
> On Tue, Sep 15, 2009 at 17:38 +0200, Wolodja Wentland wrote:
> > My question is: "How to reliably access data files from a module in
> > foo?"
>
> > Approach 2 - writing a build.py file at installation time
> > -
> >
> > The last question made me think, that Approach 1 works for the three
> > standard installation schemes, but fails miserably if the user decides
> > to do anything fancy.
>
> I implemented this approach like this:
>
> --- snip ---
> # -*- coding: UTF-8 -*-
> #!/usr/bin/env python
>
> from __future__ import with_statement
>
> import distutils.core as core
> from distutils.command.install import install as _install
> import sys
> import getopt
> import os.path
>
> class install(_install):
> """install command
>
> This specific install command will create a file 'build_config.py' which
> contains information on """
> def run(self):
> with open('lib/foo/build_config.py', 'w') as build_fp:
> build_fp.write('# -*- coding: UTF-8 -*-\n\n')
> build_fp.write("DATA_DIR = '%s'\n"%(
> os.path.join(self.install_data, 'share')))
> build_fp.write("LIB_DIR = '%s'\n"%(self.install_lib))
> build_fp.write("SCRIPT_DIR = '%s'\n"%(self.install_scripts))
> _install.run(self)
>
> core.setup(name='foo',
>version='0.1',
>description='Foo is Bar and Baz',
>author='Wolodja Wentland',
>...
> )
> --- snip ---
I get the following error if i try to install the library:
--- snip ---
reating new virtualenv environment in /home/bar/.virtualenvs/test
New python executable in /home/bar/.virtualenvs/test/bin/python
Please make sure you remove any previous custom paths from your
/home/bar/.pydistutils.cfg file.
Installing setuptools...done.
Unpacking ./dist/foo-0.1.tar.bz2
Running setup.py egg_info for package from
file:///home/bar/src/foo/dist/foo-0.1.tar.bz2
Installing collected packages: foo
Running setup.py install for foo
usage: -c [global_opts] cmd1 [cmd1_opts] [cmd2 [cmd2_opts] ...]
or: -c --help [cmd1 cmd2 ...]
or: -c --help-commands
or: -c cmd --help
error: option --single-version-externally-managed not recognized
Complete output from command /home/bar/.virtualenvs/test/bin/python -c
"import setuptools; __file__='/tmp/pip-CITX4k-build/setup.py';
execfile('/tmp/pip-CITX4k-build/setup.py')" install
--single-version-externally-managed --record
/tmp/pip-8lWuFa-record/install-record.txt --install-headers /lib/include:
usage: -c [global_opts] cmd1 [cmd1_opts] [cmd2 [cmd2_opts] ...]
or: -c --help [cmd1 cmd2 ...]
or: -c --help-commands
or: -c cmd --help
error: option --single-version-externally-managed not recognized
Command /home/bar/.virtualenvs/test/bin/python -c "import setuptools;
__file__='/tmp/pip-CITX4k-build/setup.py';
execfile('/tmp/pip-CITX4k-build/setup.py')" install
--single-version-externally-managed --record
/tmp/pip-8lWuFa-record/install-record.txt --install-headers /lib/include failed
with error code 1
Storing complete log in ./pip-log.txt
Complete output from command /home/bar/.virtualenvs/test/bin/python
/usr/lib/python2.5/site-packages/pip-0.4-py2.5.egg/pip.py install -E test
dist/foo-0.1.tar.bz2 /home/bar/.virtualenvs/test ___VENV_RESTART___:
Traceback (most recent call last):
File "/usr/bin/pip", line 5, in
pkg_resources.run_script('pip==0.4', 'pip')
File "/usr/lib/python2.5/site-packages/pkg_resources.py", line 448, in
run_script
self.require(requires)[0].run_script(script_name, ns)
File "/usr/lib/python2.5/site-packages/pkg_resources.py", line 1166, in
run_script
execfile(script_filename, namespace, namespace)
File
"/usr/lib/python2.5/site-packages/pip-0.4-py2.5.egg/EGG-INFO/scripts/pip", line
3, in
pip.main()
File "/usr/lib/python2.5/site-packages/pip-0.4-py2.5.egg/pip.py", line 926,
in main
return command.main(initial_args, args[1:], options)
File "/usr/lib/python2.5/site-packages/pip-0.4-py2.5.egg/pip.py", line 258,
in main
restart_in_venv(options.venv, site_packages, complete_args)
File "/usr/lib/python2.5/site-packages/pip-0.4-py2.5.egg/pip.py", line 1009,
in restart_in_venv
call_subprocess([python, file] + args +
Re: [Distutils] Get install prefix for module at runtime
On Thu, Sep 17, 2009 at 18:51 +0200, Tarek Ziadé wrote:
> you might be able to alter it on-the-fly by overriding the build_py command
> instead of the install command
That worked perfectly! Thanks again for the help and pointers you gave
me. I really appreciate that.
The solution i came up with is:
--- snip ---
class build_py(_build_py):
"""build_py command
This specific build_py command will modify module 'foo.build_config' so
that it
contains information on installation prefixes afterwards.
"""
def build_module (self, module, module_file, package):
if type(package) is StringType:
_package = string.split(package, '.')
elif type(package) not in (ListType, TupleType):
raise TypeError, \
"'package' must be a string (dot-separated), list, or tuple"
if ( module == 'build_info' and len(_package) == 1 and
package[0] == 'foo'):
iobj = self.distribution.command_obj['install']
with open(module_file, 'w') as module_fp:
module_fp.write('# -*- coding: UTF-8 -*-\n\n')
module_fp.write("DATA_DIR = '%s'\n"%(
os.path.join(iobj.install_data, 'share')))
module_fp.write("LIB_DIR = '%s'\n"%(iobj.install_lib))
module_fp.write("SCRIPT_DIR = '%s'\n"%(iobj.install_scripts))
_build_py.build_module(self, module, module_file, package)
--- snip ---
I might change the 'detect my module' logic a little because i rely on
python short circuit evaluation a bit too much.
I have some final questions:
1. Is the distutils API i am using here likely to change?
2. Is there a better way to access install_{lib,scripts,data} than going
through the install command object available from distribution?
3. Could you include something like this in distutils? My idea on how to
handle this would be to define an additional argument 'foo_bar' for
core.setup() which will take a user defined dotted module name in
which information like this will be saved.
So if a user defines:
core.setup( ...
foo_bar : 'foo.build_info',
...
)
A file lib_prefix/foo/build_info.py will be injected into the
library or an already one would be altered according to user defined
string templates.
Something like this would end module.__file__ hacks once and for all
and is IMHO a much cleaner way to advertise this information to
libraries/application than trying to take care of this externally.
so long
Wolodja Wentland
signature.asc
Description: Digital signature
___
Distutils-SIG maillist - Distutils-SIG@python.org
http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Get install prefix for module at runtime
On Thu, Sep 17, 2009 at 21:05 +0200, Wolodja Wentland wrote:
> On Thu, Sep 17, 2009 at 18:51 +0200, Tarek Ziadé wrote:
> > you might be able to alter it on-the-fly by overriding the build_py command
> > instead of the install command
>
> That worked perfectly! Thanks again for the help and pointers you gave
> me. I really appreciate that.
>
> The solution i came up with is:
>
> --- snip ---
> class build_py(_build_py):
> """build_py command
>
> This specific build_py command will modify module 'foo.build_config' so
> that it
> contains information on installation prefixes afterwards.
> """
> def build_module (self, module, module_file, package):
> if type(package) is StringType:
> _package = string.split(package, '.')
> elif type(package) not in (ListType, TupleType):
> raise TypeError, \
> "'package' must be a string (dot-separated), list, or tuple"
>
> if ( module == 'build_info' and len(_package) == 1 and
> package[0] == 'foo'):
> iobj = self.distribution.command_obj['install']
>
> with open(module_file, 'w') as module_fp:
> module_fp.write('# -*- coding: UTF-8 -*-\n\n')
> module_fp.write("DATA_DIR = '%s'\n"%(
> os.path.join(iobj.install_data, 'share')))
> module_fp.write("LIB_DIR = '%s'\n"%(iobj.install_lib))
> module_fp.write("SCRIPT_DIR = '%s'\n"%(iobj.install_scripts))
>
> _build_py.build_module(self, module, module_file, package)
> --- snip ---
>
> I might change the 'detect my module' logic a little because i rely on
> python short circuit evaluation a bit too much.
>
> I have some final questions:
>
> 1. Is the distutils API i am using here likely to change?
>
> 2. Is there a better way to access install_{lib,scripts,data} than going
>through the install command object available from distribution?
>
> 3. Could you include something like this in distutils? My idea on how to
>handle this would be to define an additional argument 'foo_bar' for
>core.setup() which will take a user defined dotted module name in
>which information like this will be saved.
>
>So if a user defines:
>
> core.setup( ...
> foo_bar : 'foo.build_info',
> ...
> )
>
> A file lib_prefix/foo/build_info.py will be injected into the
> library or an already one would be altered according to user defined
> string templates.
>
> Something like this would end module.__file__ hacks once and for all
> and is IMHO a much cleaner way to advertise this information to
> libraries/application than trying to take care of this externally.
Bump.
Is this approach a feasible one, or am I better served by using the
Distribute API, which I am not familiar at this point?
What about the distutils internal I am using here?
TIA
Wolodja
signature.asc
Description: Digital signature
___
Distutils-SIG maillist - Distutils-SIG@python.org
http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] distutils: packaging a generated configuration file
On Mon, Oct 05, 2009 at 20:02 +0100, Mark Dickinson wrote: [...] > specified with 'package_data' or 'data_files'. The catch is that > parts of the configuration file are generated at setup time: that is, > the setup script gathers various pieces of system information (e.g., > library locations) and uses those to transform a hard-coded > 'config.in' file into the 'config' file that I want to install. [...] How do you get this data? (just curious) > What's the best way to manage generation and installation of the > config file? I've got as far as subclassing the build command and > making __run__ create the config file from the config.in file, putting > it into the build_temp directory; I can't figure out how to tell > distutils to install it in the right place. What do you consider right place? Where do you expect your config file to be installed and how do you find it from within your code? > It seems like I want a sort of build_data/install_data pair of > commands, that allows creation and installation of *generated* data > files. I have been in a similar situation some time ago and had one simple, but unsupported, need when packaging data files. I wanted to access them from within my code after they have been installed. I detailed the problems in [1]. The solution I came up with [2] was to subclass the build_py command in distutils, include a 'build_info.py' at the package root and modify that file within the subclassed build_py command. I am still not sure if that is a good approach and haven't got any feedback on it. If the config file you want to install is *not* a python module and you expect it to be installed in DATA_PREFIX/config you might have some success with subclassing the install_data command, modify the config file before it is copied and include the file with package_data or data_files. You could also try to modify the data_files attribute of that problem if you can't specify all generated data files beforehand. But that might just cause havoc and major breakage... Note that installing data files within the python package violates the FHS, so I would recommend using data_files, if you are interested to have your software included in any *nix or BSD distribution. You will still have a problem to find it later on though. I hope that helps and I am *very* interested in any other way to solve this. good luck Wolodja Wentland [1] http://mail.python.org/pipermail/distutils-sig/2009-September/013238.html [2] http://mail.python.org/pipermail/distutils-sig/2009-September/013284.html signature.asc Description: Digital signature ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] A Python Packaging Tutorial
On Sat, Nov 07, 2009 at 14:44 +0100, Tarek Ziadé wrote: > As Ian suggested, and since others thinks its a good idea, I suggest > that we start writing a "python packaging guide" using > reStructuredText and DVCS, explaining how to handle packaging today. > (with *all* existing tools) I think this is a great idea, but what is meant by "today" ? Do you intend to describe packaging with the 0.6 or 0.7 branch of Distribute? > When it's done, we could add it in python's Doc/ with the help of Georg. This is indeed of uttermost importance as the natural way to start packaging today is to read the "Distribute Python Modules" document in the Python documentation. One problem I was wasting my time on was trying to get the "Requires" and "Provides" arguments to *distutils* setup right, just to learn that these are not used at all and all the effort was wasted. Things like this should not happen. I know the reasons for that now, but it was confusing nonetheless. > I think Pauli provided a good starting list of what is not obvious for > a new comer. As a new comer I might want to add some points: 1. Distribute 0.6 / 0.7 * Should I use Distribute 0.6 features in my software right now? I have the feeling that using Distribute 0.6 and learning the setuptools API is not a wise decision right now, given all the changes that are happening. * Will there be an upgrade path? 2. Data Files It looks like this *important* part of packaging software is neglected in the ongoing discussion and development. What are the plans regarding packaging data files? pkgutil.get_data() is absolutely useless for retrieving data files IMHO. The alternative (subclassing distutil commands) should *not* be the *standard* way to handle this. kind regards and thanks for all the work! Wolodja signature.asc Description: Digital signature ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] People want CPAN
On Wed, Nov 11, 2009 at 14:16 +0100, Tarek Ziadé wrote: > On Wed, Nov 11, 2009 at 1:31 PM, David Cournapeau wrote: > But, at the end, since an option is either global, either specific to > a command, I guess a simple API in the command class should be enough > to avoid this hack: >get_option(command_name, option_name) +1 -- .''`. Wolodja Wentland : :' : `. `'` 4096R/CAF14EFC `- 081C B7CD FF04 2BA9 94EA 36B2 8B7F 7D30 CAF1 4EFC signature.asc Description: Digital signature ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
[Distutils] Install time prefixes and data files
l reject
a package if it does not conform to a standard or at least warns the user
about it
* retrieval of certain types of data easily at run time
Distribution.get_data_file('$configuration', 'my.cfg') -> file object
* ...
thanks!
--
.''`. Wolodja Wentland
: :' :
`. `'` 4096R/CAF14EFC
`- 081C B7CD FF04 2BA9 94EA 36B2 8B7F 7D30 CAF1 4EFC
signature.asc
Description: Digital signature
___
Distutils-SIG maillist - Distutils-SIG@python.org
http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Install time prefixes and data files
On Wed, Nov 11, 2009 at 16:24 +0100, Wolodja Wentland wrote: > The installation prefixes could for example be saved within a suitable > file in .egg-info/ which would mandate changes to PEP 376. I could think > of a PREFIX file and the following methods/attribute for e.g. the > Distribution class I guess the pkgutil module is a much better place for these functions. (Sorry, am not that familiar with all the PEPs yet) > * get_data_file(type, name) -> get data file of given type and name (see This could for example be easily implemented once Distribution.get_installed_files(local=False) is done, if the type of the files are noted somewhere. -- .''`. Wolodja Wentland : :' : `. `'` 4096R/CAF14EFC `- 081C B7CD FF04 2BA9 94EA 36B2 8B7F 7D30 CAF1 4EFC signature.asc Description: Digital signature ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Fwd: People want CPAN :-)
On Wed, Nov 11, 2009 at 16:22 +, Antonio Cavallo wrote: > 1. I don't need a replacement for rpm/yast/yum/synaptic: they provide > file tracking and package resolution/dependency already and > pepole are using them (and it has been for a very long time) +1 I think that the ongoing discussion about Python packaging should focus on the needs of the users. I made a proposal that might ease basic packaging tasks for *nix package maintainers and think it would be a great time to contact the following groups of users and encourage them to provide some feedback on their needs: * Python developers - How can distributions be queried? - How is access to shipped data handled? - What are common patterns/workarounds they use in their setup.py that should be automated/made easier? * Distribution package maintainers - Should a distutils packaged provide more Metadata about the files that are shipped? - What would help in the packaging process? * Python users - Should PEP 370 compliant installation become the default? (--user the default) - What is missing from virtualenv? (like easy creation of a virtualenv and automatically installing distribution according to a pip reqirements file) - Would users like PyPi to be stricter in what it accepts (i.e. automatic testing and rejection of distributions of bad quality)? I think that some of these questions are already answered in the PEPs that are under consideration right now. I would *strongly* suggest to get them into shape before any discussion on CPAN and that is started and binds a lot of resources. I like the aproach taken by Vinay Sajip with PEP 391. He (and others?) worked on them until they were deemed to be of sufficient quality. Afterwards he asked python-dev for input, incorporated these changes and then asked the whole Python community on python-list. I happily provided input and got them incorporated into the PEP. THAT was a great user experience and a big advantage of open (source) communities. I would therefore suggest to: 1. Get the PEPs into shape 2. Contact the people and spread the word that *now* is the time to provide input that influences the way Python packaging will be done for some time to come. 3. Collect input on some Wiki/document/whatever and discuss the issues and solutions. 4. Change the PEPs to incorporate the solutions. 5. Goto 2 if necessary 6. Design the distutils/distribute/virtualenv/pip API to reflect this 7. Implement the changes. I will ask on some distribution specific MLs (like debian-python) what they might need, once we singled out some ideas that look promising. -- .''`. Wolodja Wentland : :' : `. `'` 4096R/CAF14EFC `- 081C B7CD FF04 2BA9 94EA 36B2 8B7F 7D30 CAF1 4EFC signature.asc Description: Digital signature ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Install time prefixes and data files
On Fri, Nov 13, 2009 at 18:22 +0900, David Cournapeau wrote:
> On Thu, Nov 12, 2009 at 12:24 AM, Wolodja Wentland
> wrote:
> > The FHS differentiates between various classes of files and defines
> > proper location for them. We could define platform dependent
> > standard infixes for the following data file classes for distribution
> > foo:
> >
> > * configuration etc
> > * shared data usr/share/foo/
> > * readme usr/share/foo/README README, TODO, ... could
> > be automatically
> > discovered
> > * examples usr/share/foo/examples
> > * documentation usr/share/foo/doc
> > * man files usr/share/man
> > * variable var/lib/foo
> > * ...
> One could start from all the autoconf-defined variables such as
> prefix, eprefix, sbindir, bindir, etc... One would need in addition
> some variables for python files (.py, pyc and pyo). Then, each scheme
> would just be different mapping for those variables. For example,
> --user option just means changing prefix to $HOME/.local on linux,
> gobolinux (which as a similar file organization as mac os x) could
> easily package as they do for autoconf packages.
+1
> For data files, the major problems currently in distutils are:
> - that you cannot 'tag' the different kind of data files. Tarek
> suggestion of using ('$variable_name', [list of files]) is one way of
> solving this. One could also imagine having an argument to setup per
> type (man_files=(list of files), etc...).
^
I thought of this as well yesterday and actually like it a lot better.
> - One should think about defining its own data 'type' as well. For
> example, let's say application MegaBar can be extended through python
> plugins. You may want to add your plugin to MegaBar plugin directory:
> ($megabar_dir, [list of files for megabar plugins]). megabar_dir value
> should be customizable from the build process.
Could you elaborate on this idea please? Do you mean that megabars
plugins are shipped in a different distribution than megabar itself?
I would like this, because it enables developers to split distributions
into different small distributions that might have different release
cycles, only enhance the program/library in question, ...
--
.''`. Wolodja Wentland
: :' :
`. `'` 4096R/CAF14EFC
`- 081C B7CD FF04 2BA9 94EA 36B2 8B7F 7D30 CAF1 4EFC
signature.asc
Description: Digital signature
___
Distutils-SIG maillist - Distutils-SIG@python.org
http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Install time prefixes and data files
On Fri, Nov 13, 2009 at 21:36 +0900, David Cournapeau wrote:
> On Fri, Nov 13, 2009 at 7:38 PM, Wolodja Wentland
> wrote:
> > Could you elaborate on this idea please? Do you mean that megabars
> > plugins are shipped in a different distribution than megabar itself?
>
> Yes. Imagine that I create a library, which includes a firefox plugin
> in python. I need to differentiate between what will end up in firefox
> and the rest.
I can certainly see the need for this and would be +1 on this idea.
One way to do this could be:
In megabar:
setup(
...
custom_prefixes=[
('$plugins', '$purelib/plugins'),
('$foo', '$base_prefix/foo'),
...
],
...
data_files=[
('$plugins', 'plugins/standard_plugin.py'),
...
]
)
And in megabar-plugins:
setup(
...
data_files=[
(pkgutil.get_prefix('megabar', '$plugin'),
'plugins/additional_plugin.py'),
...
]
)
You could use a standard prefix if megabar does not define a $plugin
custom prefix within megabar-plugins, which might even be the default if
introduction of custom_prefixes is not accepted/ok.
I think we need to:
* compile a list of standard prefixes and their default values on all
platforms. I still like the idea to be able to modify these *only* by
setting environment variables, so it would look something like this:
stdprefix1 = os.environ.get('PYDIST_STDPREFIX1',
distutils.sysconfig.get_std_prefix('stdprefix1'))
Create this list on a wiki?
* Define an API that clarifies how to query this information:
- within setup.py
- at built time
- at runtime
- ... ??
Suggestions? Corrections? Better ideas? I am just brainstorming...
--
.''`. Wolodja Wentland
: :' :
`. `'` 4096R/CAF14EFC
`- 081C B7CD FF04 2BA9 94EA 36B2 8B7F 7D30 CAF1 4EFC
signature.asc
Description: Digital signature
___
Distutils-SIG maillist - Distutils-SIG@python.org
http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Install time prefixes and data files
On Fri, Nov 13, 2009 at 13:04 +, Floris Bruynooghe wrote: > +1 :-) > But don't expect all package maintainers to suddenly classify their > datafiles properly. Last time this got discussed there was > significant opposition to this, maintainers did not see the point and > didn't want *any* change to their code at all unless it made *their* > lives easier. True, but we will see about that. > Personally I think the API you proposed is reasonable and would love > to see something like that in distutils so at least distros could > patch upstream (and hopefully get it accepted). Simplifying the work of distro maintainers easier is one of the main incentives I had in mind with this proposal. > The only issue I can see it that I'd like the API to retrieve the > data-files to work both when running from a development repository as > from an installed distribution. Making the API methods of the > Distribution class would force you to have an installed Distribution > on you PYTHONPATH I think (as would relying on a file in the egg-info > directory). Any ideas on how to implement this? -- .''`. Wolodja Wentland : :' : `. `'` 4096R/CAF14EFC `- 081C B7CD FF04 2BA9 94EA 36B2 8B7F 7D30 CAF1 4EFC signature.asc Description: Digital signature ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Install time prefixes and data files
On Fri, Nov 13, 2009 at 21:36 +0900, David Cournapeau wrote: > Yes. Imagine that I create a library, which includes a firefox plugin > in python. I need to differentiate between what will end up in firefox > and the rest. One thing I find noteworthy/horrifying about your *Firefox* plugin example is that this would mean that pip would have to install files in a directory completely unrelated to Python and one that is under the control of the package manager of the system. I guess a lot of distribution maintainers would curse the distutils developers if this becomes common practice for a lot of python packages. But this is more related to the fact that Python provides a second package manager that is *distinct* from the system one. It might be a good idea to display a big WARNING whenever somebody tries to install distributions (-> python, who came up with this terminology) within a path that is usually handled by the systems package manager. This would mean that the *default* install should be PEP 370 compliant or inside a virtualenv and that installing into system paths should be enabled explicitly. BTW - How are uninstalls of distributions (python) handled on which other distributions depend. Will all rdepends also be uninstalled? -- .''`. Wolodja Wentland : :' : `. `'` 4096R/CAF14EFC `- 081C B7CD FF04 2BA9 94EA 36B2 8B7F 7D30 CAF1 4EFC signature.asc Description: Digital signature ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
[Distutils] Wiki page for ideas and proposals
Hi all, I have created a Wiki page that gives an overview on the PEPs that are currently drafted and some of the proposals that were mentioned here on the list. You can find it at: http://wiki.python.org/moin/Distutils/DiscussionOverview I would be happy if you could add your own proposals/ideas or comment on those that are already present. -- .''`. Wolodja Wentland : :' : `. `'` 4096R/CAF14EFC `- 081C B7CD FF04 2BA9 94EA 36B2 8B7F 7D30 CAF1 4EFC signature.asc Description: Digital signature ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] PEP 345 update + RFC on "Requires-External" and "Requires-Python"
On Tue, Nov 17, 2009 at 11:49 +, Chris Withers wrote: > True. Oh well. Requires-Python it is then... Can anyone add that proposal to the wiki so it won't be forgotten? Furthermore it can be seen by people that try to get an overview of the changes that might be integrated. http://wiki.python.org/moin/Distutils/DiscussionOverview TIA! -- .''`. Wolodja Wentland : :' : `. `'` 4096R/CAF14EFC `- 081C B7CD FF04 2BA9 94EA 36B2 8B7F 7D30 CAF1 4EFC signature.asc Description: Digital signature ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] PEP 345 update + RFC on "Requires-External" and "Requires-Python"
On Tue, Nov 17, 2009 at 14:20 +0100, Tarek Ziadé wrote: > On Tue, Nov 17, 2009 at 2:20 PM, Wolodja Wentland > It's already in PEP 345 Even better :-D -- .''`. Wolodja Wentland : :' : `. `'` 4096R/CAF14EFC `- 081C B7CD FF04 2BA9 94EA 36B2 8B7F 7D30 CAF1 4EFC signature.asc Description: Digital signature ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] PEP 376 - RECORD file / Data files + pip feedback ?
On Sat, Feb 06, 2010 at 11:16 +0100, Tarek Ziadé wrote: Hi all, sorry for joining in so late. I am pretty busy with my thesis and examinations, which is why I can't work on this right now. I am still subscribed to python-distutils and read the mails regularly, so there is no need to CC me. > IOW some paths in the PREFIXES file (or the RECORD file as long as we > are able to find-and-replace paths in there) *need* to be updated at > installation time no matter what we decide to do here. Exactly! > So, it seems to me that we need to keep unexpanded paths [1] in the > RECORD file and let the install command expand them using installation > option . But we have to avoid this expansion when doing binary > releases. I still think that a PREFIX file, that contains *all* prefixes which are configurable during installation right now, should make it to PEP 376 and to Python 2.7/3.2 . These prefixes could be kept unexpanded in the RECORD file and it would therefore only be necessary to change them within the PREFIX file if one wants to relocate an installation. (Is this really a common use case?) The PEP lists $PREFIX and $EXEC_PREFIX, but the set of prefixes that are configurable during installation comprises: Prefix Default value Command line option --- PREFIX sys.prefix --install-purelib EXEC_PREFIX sys.exec_prefix --install-platlib SCRIPTS_PREFIX $PREFIX/bin --install-scripts DATA_PREFIX $PREFIX/share --install-data I think that it'll help Linux distribution developers *and* Python developers a lot if these prefixes where available during runtime or could be obtained by parsing the PREFIX file. This would mean the end of __file__ magic to find data files and Python packagers could adapt their tools to copy, for example, a configuration files from $DATA_PREFIX/foo.cfg to a location that complies with the FHS (i.e. /etc/foo.cfg). > It seems feasible to me in both cases. And Wolojda could work later on > adding more paths definitions and how to configure them through > distutils options. Will do :-) Yours sincerely -- .''`. Wolodja Wentland : :' : `. `'` 4096R/CAF14EFC `- 081C B7CD FF04 2BA9 94EA 36B2 8B7F 7D30 CAF1 4EFC signature.asc Description: Digital signature ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
