Re: Sharing Python installation between architectures
In article mailman.2687.1384556852.18130.python-l...@python.org, Paul Smith p...@mad-scientist.net wrote: One thing I always liked about Perl was the way you can create a single installation directory which can be shared between archictures. Say what you will about the language: the Porters have an enormous amount of experience and expertise producing portable and flexible interpreter installations. By this I mean, basically, multiple architectures (Linux, Solaris, MacOSX, even Windows) sharing the same $prefix/lib/python2.7 directory. The large majority of the contents there are completely portable across architectures (aren't they?) so why should I have to duplicate many megabytes worth of files? The solution is of course to replace all duplicates by hard links. A tool for this is useful in a lot of other circumstances too. In a re-installation of the whole or parts, the hard links will be removed, and the actual files are only removed if they aren't needed for any of the installations, so this is transparent for reinstallation. After a lot of reinstallation you want to run the tool again. This is of course only possible on real file systems (probably not on FAT), but your files reside on a server, so chances are they are on a real file system. (The above is partly in jest. It is a real solution to storage problems, but storage problems are unheard of in these days of Tera byte disks. It doesn't help with the clutter, which was probably the main motivation.) Symbolic links are not as transparent, but they may work very well too. Have the common part set apart and replace everything else by symbolic links. There is always one more way to skin a cat. Groetjes Albert -- Albert van der Horst, UTRECHT,THE NETHERLANDS Economic growth -- being exponential -- ultimately falters. albert@spearc.xs4all.nl =n http://home.hccnet.nl/a.w.m.van.der.horst -- https://mail.python.org/mailman/listinfo/python-list
Re: Sharing Python installation between architectures
On Sat, 2013-11-16 at 19:28 -0500, Paul Smith wrote: On Fri, 2013-11-15 at 18:00 -0500, Paul Smith wrote: By this I mean, basically, multiple architectures (Linux, Solaris, MacOSX, even Windows) sharing the same $prefix/lib/python2.7 directory. The large majority of the contents there are completely portable across architectures (aren't they?) so why should I have to duplicate many megabytes worth of files? OK, after some investigation and reading the code in Modules/getpath.c to determine exactly how sys.prefix and sys.exec_prefix are computed (it would be nice if this algorithm was documented somewhere... maybe it is but I couldn't find it) I have a solution for this that appears to be working fairly well. Ouch. I spoke a bit too soon. The standard Python installation and interpreter works fine with the split --prefix and --exec-prefix configuration, even with relocation of the installation directory. However, that configuration doesn't work for embedded Python (for example, if you embed the Python interpreter in GDB by linking libpython2.7.a) if you relocate it. In that configuration when the interpreter looks for the lib/python directory at runtime it appears to use only the prefix path Python was originally compiled with (I'm using strace on Linux to see what paths are being probed at startup). It doesn't use the current installation path via argv[0], which means it's not relocatable at all. I can, of course, set PYTHONHOME to point to the right place. Unfortunately, if you set PYTHONHOME then it's used for both $PREFIX and $EXECPREFIX without any path probing whatsoever, so PYTHONHOME is unusable with an installation where you've used different values for --prefix and --exec-prefix during configure. We'd need a new environment variable, like PYTHONEXECHOME or something, which would be tested first when looking for the exec_path (only). If that didn't exist, it could try PYTHONHOME as before. I'm willing to do this and file a bug with a patch if there's any interest in pursuing it further. Or should this be discussed on python-dev? -- https://mail.python.org/mailman/listinfo/python-list
Re: Sharing Python installation between architectures
On Mon, Nov 18, 2013 at 2:46 AM, Paul Smith p...@mad-scientist.net wrote: However, that configuration doesn't work for embedded Python (for example, if you embed the Python interpreter in GDB by linking libpython2.7.a) if you relocate it. ... I'm willing to do this and file a bug with a patch if there's any interest in pursuing it further. Or should this be discussed on python-dev? I don't have any experience with your actual issue, so though I've been reading your posts, I haven't anything to add to the thread. But one small side point: If you're going to propose patches that materially change functionality, they won't be applied to 2.7, which is now closed for new features (and there won't be a 2.8). So the first thing I'd recommend doing is trying the same things with 3.3, or possibly an alpha of 3.4 (or the beta, if you can wait one week for its launch). If it's exactly the same, then you could propose changes to the 3.x branch (probably too late for 3.4 now, but 3.5), and possibly they could be backported to 2.7 if it's considered a bugfix. Or you might find that, by the magic of Guido's time machine, the problem's already been solved in 3.x! Maybe you can't actually migrate your codebase fully, but at very least, try this sort of thing in both versions - at worst, you spend a bit of time spinning up a duplicate and then say Same thing happens in 3.3.2. ChrisA -- https://mail.python.org/mailman/listinfo/python-list
Re: Sharing Python installation between architectures
On Sun, 2013-11-17 at 10:46 -0500, Paul Smith wrote: Unfortunately, if you set PYTHONHOME then it's used for both $PREFIX and $EXECPREFIX without any path probing whatsoever, so PYTHONHOME is unusable with an installation where you've used different values for --prefix and --exec-prefix during configure. Gak. Never mind. That's what you get when you're trying to hack relocatable installations at 3am: even after some sleep you think you know what's going on. PYTHONHOME accepts a colon-separated list for prefix:exec-prefix. This is even clearly documented, even in the error output from the interpreter when it can't find its installation! Making this change in my wrapper script for GDB gets everything working. Cheers! -- https://mail.python.org/mailman/listinfo/python-list
Re: Sharing Python installation between architectures
On Fri, 2013-11-15 at 18:00 -0500, Paul Smith wrote: By this I mean, basically, multiple architectures (Linux, Solaris, MacOSX, even Windows) sharing the same $prefix/lib/python2.7 directory. The large majority of the contents there are completely portable across architectures (aren't they?) so why should I have to duplicate many megabytes worth of files? OK, after some investigation and reading the code in Modules/getpath.c to determine exactly how sys.prefix and sys.exec_prefix are computed (it would be nice if this algorithm was documented somewhere... maybe it is but I couldn't find it) I have a solution for this that appears to be working fairly well. In order to get this to work you need to use the following arguments when you run configure to build your Python: configure --prefix=$PREFIX --exec-prefix=$PREFIX/$ARCH where $ARCH can be pretty much whatever you want, but should be unique for each different architecture of course. The $PREFIX should be the same for all architectures. The magic here is ensuring that the --exec-prefix directory is a SUBDIRECTORY of --prefix. If that's not true, nothing works! The resulting python interpreter will live in $PREFIX/$ARCH/bin: you have to leave it there! If you move it nothing works. Although you can get rid of the bin and move it up into $PREFIX/$ARCH if you want; that's OK. What I do is have a little shell-script wrapper installed somewhere else that runs 'exec $EXECPREFIX/bin/python $@' You can also correctly install extra packages with setup.py, even if those packages have their own shared objects (like pycrypto or whatever). I should say, I've not thought about Windows yet. I don't know if this will work out for Windows. However, Windows is such a different beast anyway I think (at least in my environment) it will be OK to treat it separately and require Windows people to download/install their own Python. There are few nitty things that I needed to handle: 1. The _sysconfigdata.py file is put into $PREFIX not $EXECPREFIX, which is wrong since that file is very much architecture-specific. As a post-processing step I moved it from $PREFIX/... into $EXECPREFIX/.../lib-dynload. It's not quite correct since it's not a dynamic object, but lib-dynload is the only standard path on sys.paths from $EXECPREFIX. It works OK anyway. 2. All the scripts, even the ones in $PREFIX/bin, have hardcoded #! paths which go to a specific python in $EXECPREFIX/bin which is wrong (they can't be shared that way). I use a simple sed -i to replace them all with #!/usr/bin/env python instead. 3. There are some scripts that get dumped into $EXECPREFIX/bin rather than into $PREFIX/bin: 2to3, idle, pydoc, smtpd.py. I think this is simply a bug in the installation and those should all go into $PREFIX/bin. Another weird thing is that the normal installation (this has nothing to do with the above; it happens even if you don't set --exec-prefix) contains TWO copies of libpython2.7.a; one in $EXECPREFIX/lib and one in $EXECPREFIX/lib/python2.7/config. These are over 14M each so it's not inconsequential to have two. I'm deleting the one in lib/python2.7/config and things still seem to work OK. The pkgconfig python definition references the one in $EXECPREFIX/lib. -- https://mail.python.org/mailman/listinfo/python-list
Sharing Python installation between architectures
One thing I always liked about Perl was the way you can create a single installation directory which can be shared between archictures. Say what you will about the language: the Porters have an enormous amount of experience and expertise producing portable and flexible interpreter installations. By this I mean, basically, multiple architectures (Linux, Solaris, MacOSX, even Windows) sharing the same $prefix/lib/python2.7 directory. The large majority of the contents there are completely portable across architectures (aren't they?) so why should I have to duplicate many megabytes worth of files? The only parts of the install which are not shareable (as far as I can tell) are the .so dynamic objects (and the python executable itself obviously). If the default sys.path included platform-specific directories as well as the generic lib-dynload, it would be possible. I do see that there are plat-* directories available in the default path. Is it possible to make use of these (say, by renaming each architecture's lib-dynload to the appropriate plat-* name)? If that works, the remaining issue is the site-packages directory. There is no ability (that I can see) to separate out the shareable vs. non-sharable aspects of the add-on site-packages. Any comments or suggestions? Am I overestimating the amount of sharing that's possible? Thanks! -- https://mail.python.org/mailman/listinfo/python-list
