Re: bytecode non-backcompatibility
Maurice LING <[EMAIL PROTECTED]> writes: > So if C extension API (or whatever that is really called) is stable, > the system admin can just copy all of /sw/lib/python2.3/site-packages > into /sw/lib/python2.4/site-packages and it should work. From what > you've said, it seems that this isn't possible. Correct. This isn't possible. It's not clear it's *desirable*, either. For one thing, code can fail to port across versions, so things will break at random (instead of all at once, I admit). For another, I dislike the idea of a module sitting unupdate for long periods of time (decades, say). Updating Python provides a good excuse to update all the python modules, etc. Now, it would be nice if Python provided an easy way to do the update. The FreeBSD packaging system happens to provide a way to automate this process, but it's still not exactly easy. > So my alternative > solution is that PyPI have a mechanism to maintain what had been > installed in the site-package directory and to download the libraries > and install into the new site-package directory... PyPI is the wrong place to maintain a record of installed software. Distutils is the software that does the installing (ok, on most modules, anyway), and is the software that should record what packages are installed. Neither distutils nor PyPI has a "download" functionality. For that matter, PyPI doesn't have a hard requirement on listing where a module comes from. I'd say distutils is the software that should provide it, so I can say: python setup.py install --depends and get all the dependencies. But that's not at all clear. PyPI will have to cooperate by providing URLs to packages, instead of to pages. http://www.mired.org/home/mwm/ Independent WWW/Perforce/FreeBSD/Unix consultant, email for more information. -- http://mail.python.org/mailman/listinfo/python-list
Re: bytecode non-backcompatibility
Maurice LING <[EMAIL PROTECTED]> writes: >>>Python can then have a built-in mechanism to read the description file >>>and download the source codes and do the standard "sudo python >>>setup.py install" to install the library into site-package. >> I don't like this - it would make Python depend on sudo being >> available. I'd rather it not do that, and let each systems >> administrator issue the command according to *their* security policy. > > If you are installing packages into your home directory, then sudo is > not needed. But if you are installing it for everybody's use, then it > is necessary. Fink runs using superuser privileges. No, sudo isn't necessary. It isn't provided by default for all Unix installations, so Python would have to add a dependency on it, which would be a bad thing. sudo is sufficient. Other means are also sufficient. It would be wrong for Python to assume a specific Unix security model (i.e. - "sudo") for installations. http://www.mired.org/home/mwm/ Independent WWW/Perforce/FreeBSD/Unix consultant, email for more information. -- http://mail.python.org/mailman/listinfo/python-list
bytecode non-backcompatibility
Martin v. Löwis wrote:
Maurice LING wrote:
I've emailed to catelog-sig mailing list and is still waiting to hear
something. Currently, I have no idea of the structure of PyPI. I hope I
can hear from them soon and generate some starting points...
Posting questions is not the only way to find answers. The source code
of PyPI is available: sf.net/projects/pypi.
Regards,
Martin
I've browsed the source codes of PyPI in sf.net, nothing pops up as very
useful to me... I reckoned that sitting in here or hoping for replies
from catelog-sig isn't going to help much.
On that, I've prototyped a very simple proof-of-concept of what I have
in mind, regarding a Fink-like tool for Python's 3rd party libraries
management. Please bear in mind that this is a proof-of-concept
prototype, really bare bones. It can be found in 'centipyde' module in
sf.net/projects/ib-dwb. In case the CVS hadn't updated by the time
someone reads this, the directory layout is this:
../centipyde
../centipyde/centipyde.py
../centipyde/pgkinfo
../centipyde/pgkinfo/ply15.info
ply15.info contains the following text (pretty much modelled against Fink):
package=ply15
maintainer=.
dependencies=.
downloadurl=http://systems.cs.uchicago.edu/ply/ply-1.5.tar.gz
prebuildscript=tar zxvf ply-1.5.tar.gz
sourcedir=ply-1.5
buildscript=python setup.py build
installscript=sudo python setup.py install
centipyde.py is the following:
=
"""
Author: Maurice H.T. Ling <[EMAIL PROTECTED]>
Copyright (c) 2005 Maurice H.T. Ling
Date created : 28th April 2005
"""
PKGINFOPATH = 'pkginfo'
INSTALL_LOG = 'install.log'
import os
import string
import sys
def install_package(package_name):
f = open(os.getcwd() + os.sep + PKGINFOPATH + os.sep + package_name
+ '.info', 'r')
install_info = {}
for line in f.readlines():
line = string.split(line, '=')
if line[1][-1] == os.linesep:
install_info[line[0]] = string.strip(line[1][:-1])
else: install_info[line[0]] = string.strip(line[1])
f.close()
print "Package Installation Information: " + str(install_info)
os.system('curl -O ' + str(install_info['downloadurl']))
preinstall = []
preinstall = string.split(install_info['prebuildscript'], ';')
for cmd in preinstall: os.system(cmd)
cwd = os.getcwd()
print cwd
os.chdir(os.path.join(os.getcwd(), install_info['sourcedir']))
print os.getcwd()
buildscript = []
buildscript = string.split(install_info['buildscript'], ';')
for cmd in buildscript: os.system(cmd)
installscript = []
installscript = string.split(install_info['installscript'], ';')
for cmd in installscript: os.system(cmd)
if sys.argv[1] == 'install':
install_package(sys.argv[2])
=
When I run "python centipyde.py install ply15", PLY1.5 gets downloaded
from David Beazley's website, uncompressed and installed into the
site-package.
All comments and code donations are welcomed.
Cheers
Maurice
--
http://mail.python.org/mailman/listinfo/python-list
Re: bytecode non-backcompatibility
Maurice LING wrote: > I've emailed to catelog-sig mailing list and is still waiting to hear > something. Currently, I have no idea of the structure of PyPI. I hope I > can hear from them soon and generate some starting points... Posting questions is not the only way to find answers. The source code of PyPI is available: sf.net/projects/pypi. Regards, Martin -- http://mail.python.org/mailman/listinfo/python-list
Re: bytecode non-backcompatibility
Maurice LING wrote: >> I doubt anyone disputes that upgrades are more hassle than we would >> like. My main point was that freezing CPython's technology is not the >> solution. Any other upgrade helper that you can contribute will be >> welcome. > > So there is no way of releasing a close-source application written in > Python? I fail to see the relationship between the statement and your question. Again, whether or not the C API is frozen has nothing to do with whether or not the bytecode format is frozen. So any implication from not freezing the C API to not being able to ship byte-code only is invalid. As for only shipping byte code: sure, that is certainly possible, and people do that all the time. Regards, Martin -- http://mail.python.org/mailman/listinfo/python-list
Re: bytecode non-backcompatibility
Martin v. Löwis wrote: [automatically install a set of packages] What do you think? That would certainly be possible. Contributions are welcome. Regards, Martin I've emailed to catelog-sig mailing list and is still waiting to hear something. Currently, I have no idea of the structure of PyPI. I hope I can hear from them soon and generate some starting points... Maurice -- http://mail.python.org/mailman/listinfo/python-list
Re: bytecode non-backcompatibility
Terry Reedy wrote: I doubt anyone disputes that upgrades are more hassle than we would like. My main point was that freezing CPython's technology is not the solution. Any other upgrade helper that you can contribute will be welcome. Terry J. Reedy So there is no way of releasing a close-source application written in Python? Forget about reverse engineering or decompiling, things like EU Council Directive, Berne Convention , and legislation of individual countries are to be used and argued for handling what decompilation is allowed or not allowed. It does make a difference between looking at the codes when it is given and performing an act of decompilation to look at the codes. As mentioned in one of my postings, it may be analogous to the difference between you dropping some cash on the ground and I take it, and I performing the act of pickpocketing and take your money. I am not advocating close source in any sense and I believe the choice of open or close source distribution is individual's and corporate's rights. Cheers Maurice -- http://mail.python.org/mailman/listinfo/python-list
Re: bytecode non-backcompatibility
Maurice LING wrote: > So if C extension API (or whatever that is really called) is stable, the > system admin can just copy all of /sw/lib/python2.3/site-packages into > /sw/lib/python2.4/site-packages and it should work. It would be counter-productive to make it stable, as this would render certain desirable changes impossible. Backwards compatibility on C API (source) level is a goal, binary compatibility within a 2.x release is a goal, binary compatibility across 2.x releases is not. [automatically install a set of packages] > What do you think? That would certainly be possible. Contributions are welcome. Regards, Martin -- http://mail.python.org/mailman/listinfo/python-list
Re: bytecode non-backcompatibility
"Maurice LING" <[EMAIL PROTECTED]> wrote in message news:[EMAIL PROTECTED] > So my alternative solution is that PyPI have a mechanism to maintain what > had been installed in the site-package directory and to download the > libraries and install into the new site-package directory... > What do you think? I doubt anyone disputes that upgrades are more hassle than we would like. My main point was that freezing CPython's technology is not the solution. Any other upgrade helper that you can contribute will be welcome. Terry J. Reedy -- http://mail.python.org/mailman/listinfo/python-list
Re: bytecode non-backcompatibility
Martin v. Löwis wrote: Maurice LING wrote: technicalities are wrong but situation remains unchanged. For C modules, it is very likely that new versions of Python will continue to break the ABI, by changing the layout of structures. The most straight-forward way to deal with it as a sysadmin or user is to install multiple versions of Python on a single machine. If Fink considers python2.4 as a replacement for python2.3, then this is a flaw in Fink. In Debian, there is a python package, which currently depends on python2.3. Sometime in the future, it will depend on python2.4. Users which update will then get python2.4, however, python2.3 will remain installed and usable, with all the extension modules that were installed for it. Regards, Martin Fink does not consider python2.4 to be a replacement for python2.3. In fact, you can install python2.2, 2.3 and 2.4 in the same machine with Fink. It will maintain 3 sets of libraries as /sw/lib/python2.2, /sw/lib/python2.3 and /sw/lib/python2.4. The chore is that when say Fink installs python2.4, all the libraries in /sw/lib/python2.3/site-packages have to be re-installed into /sw/lib/python2.4/site-packages, one by one. There is no simple way of doing that... which makes any system admin needing to re-install 50 3rd party libraries into /sw/lib/python2.4/site-packages a big task, as well as satisfying the necessary dependencies. So if C extension API (or whatever that is really called) is stable, the system admin can just copy all of /sw/lib/python2.3/site-packages into /sw/lib/python2.4/site-packages and it should work. From what you've said, it seems that this isn't possible. So my alternative solution is that PyPI have a mechanism to maintain what had been installed in the site-package directory and to download the libraries and install into the new site-package directory... What do you think? Cheers maurice -- http://mail.python.org/mailman/listinfo/python-list
Re: bytecode non-backcompatibility
"Maurice LING" <[EMAIL PROTECTED]> wrote in message news:[EMAIL PROTECTED] > From a technical perspective, I can accept that .pyc files are private > and temporary. To a certain extend, it does helps development cycle. > Every time I amend my source codes, I just run it without having to > consider or needing to re-compile the source files. >From a user perspective, source code is the run-time program. > The idea of having to release the program or library as source files does > ring alarms in many executives in corporate world. I understand that people use Python while resisting its purpose and design. But I also understand that it is *their* responsibilty to hide their code, which may possibly mean not using Python, or which may mean developing proprietary methods to translate to something designed to *not* be readable. For all we know, some of the developers have been paid to do exactly that -- and not talk about it. Python is *designed* for human readability. That is one of its big features! The same also seems somewhat true for CPython's bytecodes, especially when disassembled with the dis module that comes with the interpreter. You even get all the object names included in the code object. Terry J. Reedy -- http://mail.python.org/mailman/listinfo/python-list
Re: bytecode non-backcompatibility
Maurice LING wrote: > technicalities are wrong but situation remains unchanged. For C modules, it is very likely that new versions of Python will continue to break the ABI, by changing the layout of structures. The most straight-forward way to deal with it as a sysadmin or user is to install multiple versions of Python on a single machine. If Fink considers python2.4 as a replacement for python2.3, then this is a flaw in Fink. In Debian, there is a python package, which currently depends on python2.3. Sometime in the future, it will depend on python2.4. Users which update will then get python2.4, however, python2.3 will remain installed and usable, with all the extension modules that were installed for it. Regards, Martin -- http://mail.python.org/mailman/listinfo/python-list
Re: bytecode non-backcompatibility
Maurice LING wrote: > Now I understand that Python bytecodes are only dealing with pure python > source codes. However, the same question lies, how can it (set of > bytecodes) be made stable, like Java bytecodes, which are pretty stable? > Perhaps a better question will be, what techniques Java VM designers > use that enables Java class files (and JAR files) to be stable and > usable across versions that is lacking in Python? The Java class file format is not any more stable than the Python .pyc format - I think every major release of the JVM has changed details in the class file format. The difference is that the newer JVMs accept old class files, by means of separate loader code for each class file version that was ever released (and very possibly also support for class file versions that were never released). The same approach would be possible in Python, but nobody has contributed code to do so. It is unlikely that future Python versions will provide such compatibility with the current byte code format, either, unless somebody steps forward and volunteers to maintain that compatibility. Maintaining this backwards compatibiltiy is a tedious and boring task, given the many much easier alternatives, so that volunteers are unlikely to jump in. Sun manages to provide the compatibility by paying somebody to actually maintain it. Regards, Martin -- http://mail.python.org/mailman/listinfo/python-list
Re: bytecode non-backcompatibility
Mike Meyer wrote: Maurice LING <[EMAIL PROTECTED]> writes: The idea of having to release the program or library as source files does ring alarms in many executives in corporate world. Less freezing the modules (which renders it platform dependent) or using Jython (which is not possible when C extensions are involved), there is no middle grounds in CPython for distribution without source codes. You're right - and byte codes *as they exist now* aren't an acceptable middle ground, either. The problem isn't that the interpreter might change (that can be defeated by distributing the interpreter with the bytecode files), but that the byte code can be decompiled to get your Python source back. See http://www.crazy-compilers.com/decompyle/ > for an example. I know that bytecodes can be decompiled back to python source. Similarly Java class files can be decompiled back to Java source files quite easily. IANAL, however, I do think that reverse engineering can be seen as a purposeful act in the eyes of law. I remembered that EU Council Directive 9/250/ECC does explicits the allowable scope of decompilation (for interoperability, preserving expression of ideas, achieving performance of objectives). For companies, I think it is touchy to ask permission to release the source codes even though they may be alright with releasing the same Java JAR file into the community. We do know that regardless of whatever methods of compilation there may be, if some genius is willing to spend sufficient time to crack a program. Even if he has to read the binary codes, it will still break. But that is a wilfull act rather than just looking at the source codes when it is distributed with it. It's just like picking up your money from the ground if you had dropped it as compared to pickpocketing you as compared to hold you at knife point and robbing you.. Same end result but it is viewed differently in the eyes of law and morals. Selling byte codes to someone who's worried about shipping source is selling them snake oil. I think it's unprofessional, at best. -- http://mail.python.org/mailman/listinfo/python-list
Re: bytecode non-backcompatibility
What can be done in PYAN is to encourage all 3rd party library developers to centralize their libraries in it, which I think all will gladly respond. All that is needed to be deposited in PYAN is a description file, like the simplest setup.py file. All that is needed in this description file is 1. where to download the source codes (e.g. CVS)? 2. version number? 3. standard stuffs (optional) like authors, descriptions, copyright? To copy all of what CPAN does, you need to know what other modules it depends on, so you can automatically download those as well. Yes, dependencies are necessary to be declared. The maintainers of each package should list the dependencies as well to make it work. Python can then have a built-in mechanism to read the description file and download the source codes and do the standard "sudo python setup.py install" to install the library into site-package. I don't like this - it would make Python depend on sudo being available. I'd rather it not do that, and let each systems administrator issue the command according to *their* security policy. If you are installing packages into your home directory, then sudo is not needed. But if you are installing it for everybody's use, then it is necessary. Fink runs using superuser privileges. I think something like this can be set up for Python quite easily. I recall some time back, there's a long discussion about setting up Python's version of CPAN but I can't recall the contents of the discussions. It's not clear it's easy. It's *very* clear it won't happen until someone steps up to do it. The latter is a problem. I've posted my method in catelog-sig list. And is willing to offer some time on it... if that helps... But to get it working, I think it will need 5-10x more time than I can put into it... Cheers maurice -- http://mail.python.org/mailman/listinfo/python-list
Re: bytecode non-backcompatibility
Maurice LING <[EMAIL PROTECTED]> writes: > The idea of having to release the program or library as source files > does ring alarms in many executives in corporate world. Less freezing > the modules (which renders it platform dependent) or using Jython > (which is not possible when C extensions are involved), there is no > middle grounds in CPython for distribution without source codes. You're right - and byte codes *as they exist now* aren't an acceptable middle ground, either. The problem isn't that the interpreter might change (that can be defeated by distributing the interpreter with the bytecode files), but that the byte code can be decompiled to get your Python source back. See http://www.crazy-compilers.com/decompyle/ > for an example. Selling byte codes to someone who's worried about shipping source is selling them snake oil. I think it's unprofessional, at best. http://www.mired.org/home/mwm/ Independent WWW/Perforce/FreeBSD/Unix consultant, email for more information. -- http://mail.python.org/mailman/listinfo/python-list
Re: bytecode non-backcompatibility
Maurice LING <[EMAIL PROTECTED]> writes: >>>5. Unstable bytecodes makes updating to a newer version of Python very >>>tedious and risk breaking old scripts, if they uses C modules. >> Unstable bytecodes have nothing to do with these problems. The >> current CPython installation process puts the python command and >> the libraries for different versions in different directories. This >> allows you to have multiple versions installed so you can keep old >> scripts working with the old version until you've had time to test >> them. It also makes life much easier on the developers, as they can >> have a development version installed on the machine at the same time >> as they have a production version without breaking the old scripts. It >> also means you have to reinstall all your modules on the new >> installation - which is what makes the update process tedious for me. >> Now, this could be mitigated by having Python libraries installed in >> the same location for all versions. You could fix all the bytecode >> files by running compileall.py as a script. Sometimes, a module won't >> work properly on a new version of Python, and will have to be >> updated. You'll have to find those by trial and error and fix them as >> you find them. You'll also have to recompile all the C modules, which >> will break running scripts on the old interpreter. >> Under the currewnt system, old scripts that are run by the new >> interpreter will break if all the modules they need aren't installed >> yet. Once they're installed, the scripts should work. >> If you have scripts that use C modules that aren't installed in the >> standard Python search path, they may well break on the new >> interpreter until you recompile them. Having both versions of the >> interpreter available makes this tolerable. >> The cure I proposed seems worse than disease. If you've got a better >> solution, I'm sure we'd be interested in hearing it. >> > > Perhaps this is another call for Python version of CPAN (CPyAN or > PYAN). It can be modelled after Fink or Darwinports. > > If I remembered correctly, Fink uses apt-get and curl. > > What can be done in PYAN is to encourage all 3rd party library > developers to centralize their libraries in it, which I think all will > gladly respond. All that is needed to be deposited in PYAN is a > description file, like the simplest setup.py file. All that is needed > in this description file is > 1. where to download the source codes (e.g. CVS)? > 2. version number? > 3. standard stuffs (optional) like authors, descriptions, copyright? To copy all of what CPAN does, you need to know what other modules it depends on, so you can automatically download those as well. > Python can then have a built-in mechanism to read the description file > and download the source codes and do the standard "sudo python > setup.py install" to install the library into site-package. I don't like this - it would make Python depend on sudo being available. I'd rather it not do that, and let each systems administrator issue the command according to *their* security policy. > I think something like this can be set up for Python quite easily. I > recall some time back, there's a long discussion about setting up > Python's version of CPAN but I can't recall the contents of the > discussions. It's not clear it's easy. It's *very* clear it won't happen until someone steps up to do it. The latter is a problem. >> The dependencies problem is actually pretty easy to solve. In fact, >> in >> many Python environments, it's already solved. On my system, if I >> install Eric3 using the provided installation package, the >> dependencies will be picked up automatically. Not very helpfull if >> you're not on such a system, I know. The long-term solution is for >> PyPI to grow to include this functionality. > > I must say that I do not quite understand your system. Please enlighten me. FreeBSD has a packaging system (like fink on the Mac) called either "Ports" (if you build from source) or "Packages" (if you install pre-compiled binaries). I use Ports, so I'm going to talk about them. But Packages are built from Ports, and all the information in the port winds up in the package, so you get the same functionality with a slightly different command set. Basically, each port has a list of all the other ports it depends on. When you install a port, the system will check to see if each of the dependent ports is installed - usually by checking if a file it should install exists - and if it doesn't, will install the dependent port. The dependencies of that port are recursively checked. So if I were to install the Eric3 port, it would note that I need the qscintilla, py-sip and py-qt (ports of python modules are all named "py-"). py-sip and py-qt are already installed, so it would skip thos. qscintilla isn't, so that would get installed. qscintilla doesn't have any dependencies, so that would stop there. After qscintilla is installed, eric3 would be insta
Re: bytecode non-backcompatibility
The same *can* be said for some decade-old .py files. Certainly, most everything written for 2.0 onward still works. The same will also be true for .pyc files as long as you run them with their corresponding binary and as long as the binary stills run. However, .pyc files are private, temporary caches created by a CPython interpreter for its own use and tied to its current implementation technology. They are not intended to be a public distribution form and indeed cannot be a means to run Python programs on other interpreters. Guido has stated that he want the freedom to change or even replace parts of the internal implementation as he sees fit. (2.5 will probably get a new compiler, with the difference mostly transparent to users.) One of the principles of OOP is separation of interface from implementation. User of a class should only use the public interface and not depend on private implementation. One reason is so implementation can be changed even radically as long as interface is kept constant. The Python language is interface. CPython bytecodes are implementation. From a technical perspective, I can accept that .pyc files are private and temporary. To a certain extend, it does helps development cycle. Every time I amend my source codes, I just run it without having to consider or needing to re-compile the source files. The idea of having to release the program or library as source files does ring alarms in many executives in corporate world. Less freezing the modules (which renders it platform dependent) or using Jython (which is not possible when C extensions are involved), there is no middle grounds in CPython for distribution without source codes. Every now and then, there will be new threads in this list about people asking the means and possibilities of releasing a module/program without having to release the source code (there's another new thread on it today, "Can .py be compiled?")... What I do have resources (time and energy) for is to work with the maintainers of PyPI to implement the package maintenance system I've described.. Good. I agree that we need more along that line. I've posted my idea on catelog-sig mailing list. Hope it get picked up... Cheers Maurice -- http://mail.python.org/mailman/listinfo/python-list
Re: bytecode non-backcompatibility
"Maurice LING" <[EMAIL PROTECTED]> wrote in message news:[EMAIL PROTECTED] > So there are currently 7 implementations or variations of the Python > language and you are suggesting I make another one? Perhaps you missed the winkie ;-) > The Pacman (binary executable) that I played on a 286 a decade ago still > works well on my brother's pentium III system, so from the point of a > user, it is backward compatible. How well does it work on a P4 WinXP system? If it does, you're lucky. I have games that will not run in spite of the compatibility settings being set. And so I will keep a Win95 machine and a Win98 machine for as long as they run. >The same can't be said for .pyc files The same *can* be said for some decade-old .py files. Certainly, most everything written for 2.0 onward still works. The same will also be true for .pyc files as long as you run them with their corresponding binary and as long as the binary stills run. However, .pyc files are private, temporary caches created by a CPython interpreter for its own use and tied to its current implementation technology. They are not intended to be a public distribution form and indeed cannot be a means to run Python programs on other interpreters. Guido has stated that he want the freedom to change or even replace parts of the internal implementation as he sees fit. (2.5 will probably get a new compiler, with the difference mostly transparent to users.) One of the principles of OOP is separation of interface from implementation. User of a class should only use the public interface and not depend on private implementation. One reason is so implementation can be changed even radically as long as interface is kept constant. The Python language is interface. CPython bytecodes are implementation. > What I do have resources (time and energy) for is to work with the > maintainers of PyPI to implement the package maintenance system I've > described.. Good. I agree that we need more along that line. Terry J. Reedy -- http://mail.python.org/mailman/listinfo/python-list
Re: bytecode non-backcompatibility
One difference between Java and Python is this: Java bytecodes are, as I understand it, part of the Java language definition. CPython bytecodes are intentionally not part of the language at all. Except maybe fore PyPy, other implementations do not use them. Jython translates Python source to Java bytecodes. Pyrex translates augmented Python source to C, to be compiled to native machine code. Ironman translates, I presume, to .NET common language. PyParrot (don't know if it has an official name) translates to Parrot bytecodes. Viper translated to OCamel. If you want an implementation with frozen bytecodes, you are free to make one ;-) Terry J. Reedy So there are currently 7 implementations or variations of the Python language and you are suggesting I make another one? If I am to do that, I will use CPython 2.4.1 and call it a implementation that maintains the current set of bytecodes (newer versions can have more bytecodes and deprecated bytecodes are still left there for backward compatibility) and C API interface. Till now I still do not know what is so exceedingly prohibitive to do that? Why should I create such a direct fork? Personally I do not have the resources to maintain this fork. The Pacman (binary executable) that I played on a 286 a decade ago still works well on my brother's pentium III system, so from the point of a user, it is backward compatible. The same can't be said for .pyc files or .so files (I may be wrong with .so files here). What I do have resources (time and energy) for is to work with the maintainers of PyPI to implement the package maintenance system I've described.. maurice -- http://mail.python.org/mailman/listinfo/python-list
Re: bytecode non-backcompatibility
Terry Reedy wrote: "Maurice LING" <[EMAIL PROTECTED]> wrote in message news:[EMAIL PROTECTED] Now I understand that Python bytecodes are only dealing with pure python source codes. Then stop blaming (machine-independent) CPython 'bytecodes' for any problems you have with compiled-to-machine-language C extensions, which have nothing to do with bytecodes. Bytecodes also have nothing directly to do with source-code compatibility. technicalities are wrong but situation remains unchanged. maurice -- http://mail.python.org/mailman/listinfo/python-list
Re: bytecode non-backcompatibility
"Maurice LING" <[EMAIL PROTECTED]> wrote in message news:[EMAIL PROTECTED] > I think backward compatibility is good enough. A Java JAR file compiled > under for JVM 1.2 can still run in JVM 1.4. I don't think anyone will or > can expect forward compatibility, that will require godly powers... One difference between Java and Python is this: Java bytecodes are, as I understand it, part of the Java language definition. CPython bytecodes are intentionally not part of the language at all. Except maybe fore PyPy, other implementations do not use them. Jython translates Python source to Java bytecodes. Pyrex translates augmented Python source to C, to be compiled to native machine code. Ironman translates, I presume, to .NET common language. PyParrot (don't know if it has an official name) translates to Parrot bytecodes. Viper translated to OCamel. If you want an implementation with frozen bytecodes, you are free to make one ;-) Terry J. Reedy -- http://mail.python.org/mailman/listinfo/python-list
Re: bytecode non-backcompatibility
"Maurice LING" <[EMAIL PROTECTED]> wrote in message news:[EMAIL PROTECTED] > Now I understand that Python bytecodes are only dealing with pure python > source codes. Then stop blaming (machine-independent) CPython 'bytecodes' for any problems you have with compiled-to-machine-language C extensions, which have nothing to do with bytecodes. Bytecodes also have nothing directly to do with source-code compatibility. -- http://mail.python.org/mailman/listinfo/python-list
Re: bytecode non-backcompatibility
2. The current compilation scheme (compiling to bytecode as and when it is needed) works well for scripting purposes but is less desirable in commercial settings. Less distribution happens when it is used purely for scripting purposes, such as system maintenance or tuning. The solution with the current situation depends on what you mean by "commercial settings". Commerically, I will mean distributing the program without the codes. The best way under the current method is to freeze the application. In Java, you can just distribute the JAR files or CLASS files. I can imagine there will be some level of touchiness when source codes are distributed... 3. Using Python in commercial settings will usually require distribution of resulting software and it is may or may not be desirable to distribute source codes as well. Unless the application is frozen, distributing source code is a must. People have recommended that you distribute the Python interpreter with commercial applications. This makes installing them much easier for the end user. It also means the byte code and the interpreter will always match. 4. One advantage that Java platform has is that it does not require the release of source files and still promotes platform-independence. Java is only "platform-independent" to the degree that the jvm and libraries you need for your application are already available. A quick google finds applications that require jvm versions - so I suspect that you can't always run Java code built for a new version of the jvm on older jvms. So you have to make sure the target platform has a recent enough implementation of the jvm installed. You may have problems if you try using a different groups jvm as well. I haven't looked into that in a number of years. So you can't just ship java bytecodes to someone expecting they'll be able to run it out of the box. They may well need to update their java environment, or install a second one (I recall one time having four Java products that took in total three different jvms to run. Bleah.) This isn't really different from Python. In both cases, the end user has to have a suitable version of the bytecode interpreter and libraries installed. Java is a little better in that they provide backwards compatability. I think backward compatibility is good enough. A Java JAR file compiled under for JVM 1.2 can still run in JVM 1.4. I don't think anyone will or can expect forward compatibility, that will require godly powers... According to Steve Holden's reply, Python C API is only backward compatible in minor versions but not major versions. The issue is, why not major versions as well unless there's something exceedingly prohibitive? 5. Unstable bytecodes makes updating to a newer version of Python very tedious and risk breaking old scripts, if they uses C modules. Unstable bytecodes have nothing to do with these problems. The current CPython installation process puts the python command and the libraries for different versions in different directories. This allows you to have multiple versions installed so you can keep old scripts working with the old version until you've had time to test them. It also makes life much easier on the developers, as they can have a development version installed on the machine at the same time as they have a production version without breaking the old scripts. It also means you have to reinstall all your modules on the new installation - which is what makes the update process tedious for me. Now, this could be mitigated by having Python libraries installed in the same location for all versions. You could fix all the bytecode files by running compileall.py as a script. Sometimes, a module won't work properly on a new version of Python, and will have to be updated. You'll have to find those by trial and error and fix them as you find them. You'll also have to recompile all the C modules, which will break running scripts on the old interpreter. Under the currewnt system, old scripts that are run by the new interpreter will break if all the modules they need aren't installed yet. Once they're installed, the scripts should work. If you have scripts that use C modules that aren't installed in the standard Python search path, they may well break on the new interpreter until you recompile them. Having both versions of the interpreter available makes this tolerable. The cure I proposed seems worse than disease. If you've got a better solution, I'm sure we'd be interested in hearing it. Perhaps this is another call for Python version of CPAN (CPyAN or PYAN). It can be modelled after Fink or Darwinports. If I remembered correctly, Fink uses apt-get and curl. What can be done in PYAN is to encourage all 3rd party library developers to centralize their libraries in it, which I think all will gladly respond. All that is needed to be deposited in PYAN is a description file, like the simplest setup.py file. All that is needed in this description file
Re: bytecode non-backcompatibility
Maurice LING <[EMAIL PROTECTED]> writes: >> All you have to do is convince the developers to declare the set of >> bytecodes fixed, and they'd be stable. I don't think that will be >> easy, as it removes one of the methods used to improve the performance >> of Python. Since rebuilding the bytecode files is trivial (just invoke >> compileall.py in the library as a script), this isn't really a >> problem. The only thing that breaks are modules that muck about inside >> the bytecode. I don't know how much bytecode improvements have sped >> things up, but I do know that if I ever used a module that mucked >> around with byte codes, I wasn't aware of it - so this is a tradeoff >> I'm more than happy to make. >> > > Thanks Mike, > > My arguments to the developers will be: > > 1. Python had gone from a purely scripting language to a general > purpose programming language. I think that's irrelevant. It just means that there are more applications around that have to be dealt with when you update. > 2. The current compilation scheme (compiling to bytecode as and when > it is needed) works well for scripting purposes but is less desirable > in commercial settings. Less distribution happens when it is used > purely for scripting purposes, such as system maintenance or tuning. The solution with the current situation depends on what you mean by "commercial settings". > 3. Using Python in commercial settings will usually require > distribution of resulting software and it is may or may not be > desirable to distribute source codes as well. Unless the application > is frozen, distributing source code is a must. People have recommended that you distribute the Python interpreter with commercial applications. This makes installing them much easier for the end user. It also means the byte code and the interpreter will always match. > 4. One advantage that Java platform has is that it does not require > the release of source files and still promotes platform-independence. Java is only "platform-independent" to the degree that the jvm and libraries you need for your application are already available. A quick google finds applications that require jvm versions - so I suspect that you can't always run Java code built for a new version of the jvm on older jvms. So you have to make sure the target platform has a recent enough implementation of the jvm installed. You may have problems if you try using a different groups jvm as well. I haven't looked into that in a number of years. So you can't just ship java bytecodes to someone expecting they'll be able to run it out of the box. They may well need to update their java environment, or install a second one (I recall one time having four Java products that took in total three different jvms to run. Bleah.) This isn't really different from Python. In both cases, the end user has to have a suitable version of the bytecode interpreter and libraries installed. Java is a little better in that they provide backwards compatability. > 5. Unstable bytecodes makes updating to a newer version of Python very > tedious and risk breaking old scripts, if they uses C modules. Unstable bytecodes have nothing to do with these problems. The current CPython installation process puts the python command and the libraries for different versions in different directories. This allows you to have multiple versions installed so you can keep old scripts working with the old version until you've had time to test them. It also makes life much easier on the developers, as they can have a development version installed on the machine at the same time as they have a production version without breaking the old scripts. It also means you have to reinstall all your modules on the new installation - which is what makes the update process tedious for me. Now, this could be mitigated by having Python libraries installed in the same location for all versions. You could fix all the bytecode files by running compileall.py as a script. Sometimes, a module won't work properly on a new version of Python, and will have to be updated. You'll have to find those by trial and error and fix them as you find them. You'll also have to recompile all the C modules, which will break running scripts on the old interpreter. Under the currewnt system, old scripts that are run by the new interpreter will break if all the modules they need aren't installed yet. Once they're installed, the scripts should work. If you have scripts that use C modules that aren't installed in the standard Python search path, they may well break on the new interpreter until you recompile them. Having both versions of the interpreter available makes this tolerable. The cure I proposed seems worse than disease. If you've got a better solution, I'm sure we'd be interested in hearing it. > 6. Unstable bytecodes also makes research work into Python, such as, > just-in-time compilation and just-in-time specialization unfavourable > as they may only be a
Re: bytecode non-backcompatibility
Maurice LING wrote: All you have to do is convince the developers to declare the set of bytecodes fixed, and they'd be stable. I don't think that will be easy, as it removes one of the methods used to improve the performance of Python. Since rebuilding the bytecode files is trivial (just invoke compileall.py in the library as a script), this isn't really a problem. The only thing that breaks are modules that muck about inside the bytecode. I don't know how much bytecode improvements have sped things up, but I do know that if I ever used a module that mucked around with byte codes, I wasn't aware of it - so this is a tradeoff I'm more than happy to make. Thanks Mike, [... arguments about bytecode stability ...] What other point are there? I may be chopped by saying this but by having a stable set of bytecodes, we may lose a means of optimization. But we may gain more from filling the need for commerical distribution of applications writing in Python and ease of upgrading... At current stage, every time a new version of Python is installed in my server or system, I have to test and ensure the needed libraries are there... which may be a horror in corporate settings. Couple that with messy dependencies of the libraries to be installed. I remembered the time I was trying to install Eric 3, the dependencies makes me want to give up... I have to install Qt, then pyQt, then something else, then Eric3. Imagine you need 10 of those libraries... which may happen... I am not yet masochistic enough to take pleasures in this... I also hope some python developers are reading this thread as well.. Call this a desperate plea from some of us.. Cheers Maurice There's enough attention to backward compatibility that the chance of code breakage is quite small for pure-Python modules. Given that case, all we really need for them is a suitable way of moving them forward to an updated distribution (though, of course, the module authors may have brought out new versions that use more advanced language features, there is no real compulsion to migrate to those if you seek stability). I don't believe that the major problem is the changes to the bytecodes - extension modules (those written in C and/or C++) don't generally use the bytecodes anyway. The real problem is the API to the interpreter, which again the developers retain the right to change between minor (but not micro) versions. regards Steve -- Steve Holden+1 703 861 4237 +1 800 494 3119 Holden Web LLC http://www.holdenweb.com/ Python Web Programming http://pydish.holdenweb.com/ -- http://mail.python.org/mailman/listinfo/python-list
Re: bytecode non-backcompatibility
All you have to do is convince the developers to declare the set of bytecodes fixed, and they'd be stable. I don't think that will be easy, as it removes one of the methods used to improve the performance of Python. Since rebuilding the bytecode files is trivial (just invoke compileall.py in the library as a script), this isn't really a problem. The only thing that breaks are modules that muck about inside the bytecode. I don't know how much bytecode improvements have sped things up, but I do know that if I ever used a module that mucked around with byte codes, I wasn't aware of it - so this is a tradeoff I'm more than happy to make. Thanks Mike, My arguments to the developers will be: 1. Python had gone from a purely scripting language to a general purpose programming language. 2. The current compilation scheme (compiling to bytecode as and when it is needed) works well for scripting purposes but is less desirable in commercial settings. Less distribution happens when it is used purely for scripting purposes, such as system maintenance or tuning. 3. Using Python in commercial settings will usually require distribution of resulting software and it is may or may not be desirable to distribute source codes as well. Unless the application is frozen, distributing source code is a must. 4. One advantage that Java platform has is that it does not require the release of source files and still promotes platform-independence. 5. Unstable bytecodes makes updating to a newer version of Python very tedious and risk breaking old scripts, if they uses C modules. 6. Unstable bytecodes also makes research work into Python, such as, just-in-time compilation and just-in-time specialization unfavourable as they may only be applicable to a specific version of Python. There is much less chance of getting a project grant than if the same project is applied for Java (stable bytecodes). What other point are there? I may be chopped by saying this but by having a stable set of bytecodes, we may lose a means of optimization. But we may gain more from filling the need for commerical distribution of applications writing in Python and ease of upgrading... At current stage, every time a new version of Python is installed in my server or system, I have to test and ensure the needed libraries are there... which may be a horror in corporate settings. Couple that with messy dependencies of the libraries to be installed. I remembered the time I was trying to install Eric 3, the dependencies makes me want to give up... I have to install Qt, then pyQt, then something else, then Eric3. Imagine you need 10 of those libraries... which may happen... I am not yet masochistic enough to take pleasures in this... I also hope some python developers are reading this thread as well.. Call this a desperate plea from some of us.. Cheers Maurice -- http://mail.python.org/mailman/listinfo/python-list
Re: bytecode non-backcompatibility
Maurice LING <[EMAIL PROTECTED]> writes: > I find this part of the story a nuisance, C components in 3rd party > modules... What are the C components compiled into? What are actually > "so" files? Shared object files. They're executable code that can be linked into a program at runtime. Like DLL's on Windows. > I am using Fink to maintain my Python installation and Fink maintains > the 3rd party libraries in /sw/lib/python2.3/site-packages. Imagine > the trouble if I do a "fink selfupdate" and "fink update-all" and > finds that my Python is replaced by a newer version, say Python > 2.4. Then all my 3rd party libraries in > /sw/lib/python2.3/site-packages are un-usable... And I will have to > re-download and re-install all the libraries again. Yeah, it's pain. It would be nice if setup.py kept a list of installed packages that you could extract before updating, so you would know what needed to be updated. Last time I updated, I used the systems package handler. I asked it for the names of all the Python packages that were installed and saved that to a file. Then I installed the new version of Python. Then I edited the package list to turn it into a series of package update commands, and ran that file. > Now I understand that Python bytecodes are only dealing with pure > python source codes. However, the same question lies, how can it (set > of bytecodes) be made stable, like Java bytecodes, which are pretty > stable? Perhaps a better question will be, what techniques Java VM > designers use that enables Java class files (and JAR files) to be > stable and usable across versions that is lacking in Python? All you have to do is convince the developers to declare the set of bytecodes fixed, and they'd be stable. I don't think that will be easy, as it removes one of the methods used to improve the performance of Python. Since rebuilding the bytecode files is trivial (just invoke compileall.py in the library as a script), this isn't really a problem. The only thing that breaks are modules that muck about inside the bytecode. I don't know how much bytecode improvements have sped things up, but I do know that if I ever used a module that mucked around with byte codes, I wasn't aware of it - so this is a tradeoff I'm more than happy to make. http://www.mired.org/home/mwm/ Independent WWW/Perforce/FreeBSD/Unix consultant, email for more information. -- http://mail.python.org/mailman/listinfo/python-list
Re: bytecode non-backcompatibility
It is a nuisance. It's more of a nuisance when third part modules with 'c' components are compiled for the new version of python (a windows specific problem). I did find that the last upgrade forced me to evaluate which extensions I actually used. The answer was 'a few less than I thought'. It became a good opportunity to spring clean my 'site-packages' folder. I find this part of the story a nuisance, C components in 3rd party modules... What are the C components compiled into? What are actually "so" files? I am using Fink to maintain my Python installation and Fink maintains the 3rd party libraries in /sw/lib/python2.3/site-packages. Imagine the trouble if I do a "fink selfupdate" and "fink update-all" and finds that my Python is replaced by a newer version, say Python 2.4. Then all my 3rd party libraries in /sw/lib/python2.3/site-packages are un-usable... And I will have to re-download and re-install all the libraries again. Yes, I do agree that it is a nice time to clean out "site-packages" but imagine the work of a system admin who has to re-install 50 packages... I do believe that we can use some help here, if possible... Perhaps someone can enlighten us on the technicalities of "so" files and/or C bindings in Python... Now I understand that Python bytecodes are only dealing with pure python source codes. However, the same question lies, how can it (set of bytecodes) be made stable, like Java bytecodes, which are pretty stable? Perhaps a better question will be, what techniques Java VM designers use that enables Java class files (and JAR files) to be stable and usable across versions that is lacking in Python? Thanks. Cheers Maurice -- http://mail.python.org/mailman/listinfo/python-list
Re: bytecode non-backcompatibility
I think the OP is confusing three different CPython implementation features -- bytecodes, marshal format, and C API, all of which change pretty slowly -- and a MS Windows/C (dis)feature. CPython bytecodes only concern code written in Python but are mostly not a concern because recompilation is automatic when needed. The exception is code that directly mucks around with bytecodes, especially the intentially semi-undocumented numerical codes. Code objects also include marshaled object values. The code object for 'haha = 987654321' must contain representations for the constants 'haha' and '987654321' as well as the bytecode for the assignment. So even if bytecodes remain the same, marshal can change (as it did for 2.4, I believe, and as it will for floats in 2.5 to fix a bug) and trigger auto recompiles. C extensions interact with the interpreter via function calls that constitute the C API. As much as possible, the developers consciously avoid changes that break old code or even old binaries. Guido has claimed that for Linux, there are extension binaries compiled for 2.0 that still work with 2.4. However, for MS Windows/C, there is a 'feature' with respect to DLLs that requires recompilation of extensions to work with a new version of the Python DLL, even if the C API is unchanged (or so people who understand this have said). So the OPs complaint about having to get and install new extension binaries for each Python version might well be directed to MS. This does not, of course, negate the idea that it would be nice if the update process were somehow make easier. Indeed, the more-batteries included distributions from ActiveState and Enthought specifically aim at this. Terry J. Reedy -- http://mail.python.org/mailman/listinfo/python-list
Re: bytecode non-backcompatibility
Maurice LING wrote: > Hi, > > I've been using Python for about 2 years now, for my honours project and > now my postgrad project. I must say that I am loving it more and more > now. From my knowledge, Python bytecodes are not back-compatible. I must > say that my technical background isn't strong enough but is there any > good reason for not being back-compatible in bytecodes? > It *shouldn't* be a problem for pure python modules. The interpreter will recognise that the bytecode has the wrong 'magic number' and recompile. Seeing as a new major version of python will probably be installed in a new directory at the very least you will have to copy the modules across. If they have install files (or a setup.py) wouldn't it be better to use that *anyway* ? > My problem is not about pure python modules or libraries but the problem > is with 3rd party libraries with C bindings (not python pure). It means > that with every upgrade of python, I have to reinstall all my 3rd party > libraries which can be quite a bit of work... > It is a nuisance. It's more of a nuisance when third part modules with 'c' components are compiled for the new version of python (a windows specific problem). I did find that the last upgrade forced me to evaluate which extensions I actually used. The answer was 'a few less than I thought'. It became a good opportunity to spring clean my 'site-packages' folder. Best Regards, Fuzzy http://www.voidspace.org.uk/python > I do hope this problem will be sorted out some day. > > Cheers > Maurice -- http://mail.python.org/mailman/listinfo/python-list
Re: bytecode non-backcompatibility
Maurice LING wrote: Hi, I've been using Python for about 2 years now, for my honours project and now my postgrad project. I must say that I am loving it more and more now. From my knowledge, Python bytecodes are not back-compatible. I must say that my technical background isn't strong enough but is there any good reason for not being back-compatible in bytecodes? > My problem is not about pure python modules or libraries but the problem is with 3rd party libraries with C bindings (not python pure). Then this has nothing to do with bytecode incompatibility. Only pure-Python modules get compiled to bytecode. You mean binary compatibility of the C API. If you're going to have significant improvements in the core interpreter, you have to break binary compatibility from time to time. It means that with every upgrade of python, I have to reinstall all my 3rd party libraries which can be quite a bit of work... I do hope this problem will be sorted out some day. This problem is mitigated somewhat by the fact that the devs keep binary compatibility within a major revision series (e.g. 2.3.0, 2.3.1, 2.3.2, ...). -- Robert Kern [EMAIL PROTECTED] "In the fields of hell where the grass grows high Are the graves of dreams allowed to die." -- Richard Harter -- http://mail.python.org/mailman/listinfo/python-list
bytecode non-backcompatibility
Hi, I've been using Python for about 2 years now, for my honours project and now my postgrad project. I must say that I am loving it more and more now. From my knowledge, Python bytecodes are not back-compatible. I must say that my technical background isn't strong enough but is there any good reason for not being back-compatible in bytecodes? My problem is not about pure python modules or libraries but the problem is with 3rd party libraries with C bindings (not python pure). It means that with every upgrade of python, I have to reinstall all my 3rd party libraries which can be quite a bit of work... I do hope this problem will be sorted out some day. Cheers Maurice -- http://mail.python.org/mailman/listinfo/python-list
