Re: [opensuse-packaging] How to deal with Python .pyc and .pyo files?
jan matejek wrote: > Michal Marek napsal(a): >> So... could python stop reporting errors in such cases and just silently >> ignore the precompiled files? For mere mortals it's hard to understand >> that they can use the same package with ELF binaries for years, but have >> to rebuild python packages each time the python bytecode format changes. > > well sorry if python bytecode format changes more often than ELF standard :P > There was no major (read 'incompatible') change since 2.0 release, i > just went through the changelogs. Then this is one more reason not to install python stuff into a versioned directory. I thought the whole point of using .../python-$major.$minor is to avoid problems with incompatible bytecode files. Michal - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
Re: [opensuse-packaging] How to deal with Python .pyc and .pyo files?
Michal Marek napsal(a): > Jan Matejek wrote: >> Second worst case would be if you upgraded to a new major version of >> python, i.e. from 1.5 to 2.0. In that case python would report errors >> about bad bytecode, and i'm really not sure if it would recompile >> automatically. But i think it would. > > So... could python stop reporting errors in such cases and just silently > ignore the precompiled files? For mere mortals it's hard to understand > that they can use the same package with ELF binaries for years, but have > to rebuild python packages each time the python bytecode format changes. well sorry if python bytecode format changes more often than ELF standard :P There was no major (read 'incompatible') change since 2.0 release, i just went through the changelogs. So if you see any such errors somewhere, please report a bug and i'll make them go away. And, there *will* be a big breakage when python3k comes. But that is going to be an entirely new level of fun - py3k will not be backwards compatible and most apps will need to be modified. Let's not concern ourselves with that, for now... > "Why scripts and bytecode files have to be in lib64" comes as next ;) Because there is no better place for them, that's why ;e) Either we have to keep them in lib[64], and then they're arch-dependent because of that. Or, we go the Ubuntu way and store all .py in /usr/share/python or something, but that's a lot of trouble in itself (half-binary-half-python packages need to be installed in an interesting way, which can be a problem if you don't use distutils), and to add insult to injury, it violates FHS. Or somebody comes up with a clever way to keep stuff in lib and make python in lib64 recognize it and not break on binary files it finds there. (which i've been trying to do, but with little success so far) regards m. > > Michal > - > To unsubscribe, e-mail: [EMAIL PROTECTED] > For additional commands, e-mail: [EMAIL PROTECTED] > - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
Re: [opensuse-packaging] How to deal with Python .pyc and .pyo files?
Johannes Meixner napsal(a): > > By the way: > > Isn't there a general problem with "python files strewn > all over the system"? Well, yes, that's why i wrote it's bad packaging. But it's a minor problem. Python will work with it (unless the misplaced thing is a python module, in which case python won't be able to find and import it without some hassle) and ordinary user will not notice. This might change when Python is accepted into LSB, because there will probably be some restriction on .py files placement. > How does Python deal with .py files from whatever personal > Python projects in user's home directories? > The user has write access so that he gets .pyc or .pyo files > stored where his .py files are. > What happens if the system admin upgrades to a new major > version of Python? I'm not entirely sure about this, but i think that python will complain and recompile the bytecode files in question. Sadly, i don't have an old enough python anywhere nearby, so i can't test the assumption now. > I think the same problem (perhaps even in a worse shape) > exists for whatever third-party Python stuff which gets > installed somewhere under /opt/ or /usr/local/. > Think about proprietary third-party software which cannot > be changed and which exists for more than one SLES version. Oldest python we support is version 2.2.1 in SLES8, and all new versions since that are at least backwards compatible. So if anyone has a third-party software for python 1.5, too bad for them. But i think that third party vendors would provide python2 compatible packages by now. > I don't know if the Python license allows to have proprietary > Python stuff but I think you understand the basic idea behind. > http://www.python.org/doc/1.5.1p1/tut/node43.html > seems to indicate that at least semi-proprietary Python stuff > can exist because it reads: > --- > It is possible to have a file called "spam.pyc" (or "spam.pyo" > when -O is used) without a module "spam.py" in the same module. > This can be used to distribute a library of Python code in a > form that is moderately hard to reverse engineer. > --- > There could be even "binary-only" third-party Python stuff > where no .py files exist. > What happens if a system upgrade (e.g. SLES N -> SLES N+1) > upgrades Python to a new major version? The software will no longer work. Users would need to either keep the older python around, or ask the vendor to recompile the package. But the bytecode files will not be deleted. >> But even more important is that regular user doesn't have write access >> to wherever the .py files are stored. You would need to run each python >> program as root first, otherwise python would need to compile the >> sources every time the program is run. > > This has another consequence: > It doesn't help not to provide .pyc or .pyo files in the RPM > to avoid problems with outdated .pyc or .pyo files if the > Python major version changes because when root runs whatever > .py files, the matching .pyc or .pyo files are created in any case. Right. > > >>> I am no Python expert at all and I would be happy if a Python expert >>> could provide some background information. >> is this background sufficient? ;e) > > Yes, it is perfect and I am happy. glad to read that. regards jan matejek - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
Re: [opensuse-packaging] How to deal with Python .pyc and .pyo files?
Hello Jan, On Nov 15 20:09 Jan Matejek wrote (shortened): > Johannes Meixner napsal(a): > > > > Are byte-compiled Python .pyc and .pyo files the same for any Python > > and/or is any Python sufficiently smart to know when .pyc and/or .pyo > > files are outdated (even if the matching .py files are unchanged)? > > They are (mostly) the same, AND there is some kind of version check in > the file. Don't worry about this, python would notice any problem by > itself ;e) > > Second worst case would be if you upgraded to a new major version of > python, i.e. from 1.5 to 2.0. In that case python would report errors > about bad bytecode, and i'm really not sure if it would recompile > automatically. But i think it would. > > Worst case would be if you downgraded, i don't think python 1.5 could > cope... But on the other hand, i don't think this problem is worth > considering. > > That's why python has versioned site-packages. > In your other post you noticed that python files are strewn all over the > system. In most cases that is bad packaging. Generally speaking, all > python files should be located in /usr/lib[64]/pythonx.y/site-packages > (of course there are exceptions, but let's not go into that). Many thanks for the explanation! Now I understand the stuff and now I understand why and what I must fix in particular in my hplip package which installs its .py files (but no .pyc or .pyo file at all) under /usr/share/hplip/* Currently I just run HP's "make install" (HPLIP is made by HP) and I need to find a solution together with HP to get it fixed. By the way: Isn't there a general problem with "python files strewn all over the system"? How does Python deal with .py files from whatever personal Python projects in user's home directories? The user has write access so that he gets .pyc or .pyo files stored where his .py files are. What happens if the system admin upgrades to a new major version of Python? I think the same problem (perhaps even in a worse shape) exists for whatever third-party Python stuff which gets installed somewhere under /opt/ or /usr/local/. Think about proprietary third-party software which cannot be changed and which exists for more than one SLES version. I don't know if the Python license allows to have proprietary Python stuff but I think you understand the basic idea behind. http://www.python.org/doc/1.5.1p1/tut/node43.html seems to indicate that at least semi-proprietary Python stuff can exist because it reads: --- It is possible to have a file called "spam.pyc" (or "spam.pyo" when -O is used) without a module "spam.py" in the same module. This can be used to distribute a library of Python code in a form that is moderately hard to reverse engineer. --- There could be even "binary-only" third-party Python stuff where no .py files exist. What happens if a system upgrade (e.g. SLES N -> SLES N+1) upgrades Python to a new major version? > But even more important is that regular user doesn't have write access > to wherever the .py files are stored. You would need to run each python > program as root first, otherwise python would need to compile the > sources every time the program is run. This has another consequence: It doesn't help not to provide .pyc or .pyo files in the RPM to avoid problems with outdated .pyc or .pyo files if the Python major version changes because when root runs whatever .py files, the matching .pyc or .pyo files are created in any case. > > I am no Python expert at all and I would be happy if a Python expert > > could provide some background information. > > is this background sufficient? ;e) Yes, it is perfect and I am happy. Kind Regards Johannes Meixner -- SUSE LINUX Products GmbH, Maxfeldstrasse 5, 90409 Nuernberg, Germany AG Nuernberg, HRB 16746, GF: Markus Rex - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
Re: [opensuse-packaging] How to deal with Python .pyc and .pyo files?
Jan Matejek wrote: > Second worst case would be if you upgraded to a new major version of > python, i.e. from 1.5 to 2.0. In that case python would report errors > about bad bytecode, and i'm really not sure if it would recompile > automatically. But i think it would. So... could python stop reporting errors in such cases and just silently ignore the precompiled files? For mere mortals it's hard to understand that they can use the same package with ELF binaries for years, but have to rebuild python packages each time the python bytecode format changes. "Why scripts and bytecode files have to be in lib64" comes as next ;) Michal - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
Re: [opensuse-packaging] How to deal with Python .pyc and .pyo files?
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Johannes Meixner napsal(a): > Hello, > > On Nov 14 16:00 Marcus Rueckert wrote (shortened): > >>> On Wed, 2007-11-14 at 15:08 +0100, Johannes Meixner wrote: Hello, what is the recommended way for packaging Python stuff? Either have byte-compiled Python .pyc and .pyo files packaged into the RPM or let this be done by the currently actually installed Python system during run-time on the end-user's computer and not by whatever (perhaps different) Python system during package build-time? >> as we build with the same python version as the user will run it later. >> you can safely package the .pyc files. > > I understand that it is the same Python version directly after > a new system installation. > > But what happens when the user later change/update his Pyhon? > > Think about the "worst case" when the user replaces our Pyhon > with whatever self-compiled Pyhon. > > Are byte-compiled Python .pyc and .pyo files the same for any Python > and/or is any Python sufficiently smart to know when .pyc and/or .pyo > files are outdated (even if the matching .py files are unchanged)? They are (mostly) the same, AND there is some kind of version check in the file. Don't worry about this, python would notice any problem by itself ;e) Second worst case would be if you upgraded to a new major version of python, i.e. from 1.5 to 2.0. In that case python would report errors about bad bytecode, and i'm really not sure if it would recompile automatically. But i think it would. Worst case would be if you downgraded, i don't think python 1.5 could cope... But on the other hand, i don't think this problem is worth considering. That's why python has versioned site-packages. In your other post you noticed that python files are strewn all over the system. In most cases that is bad packaging. Generally speaking, all python files should be located in /usr/lib[64]/pythonx.y/site-packages (of course there are exceptions, but let's not go into that). >> i would even recommend packaging >> them so they get removed if you uninstall the package. > > Removal when uninstalling could also be done by whatever > other magic (e.g. via %ghost or a %preun scriptlet in RPM). Bad magic, bad bad magic. We have enough trouble with scriptlets as they are now. > I wonder why in this case small RPMs seem not to count. > > For example the installed .pyc and .pyo files in python > on a openSUSE 10.3 i386 system are more than 7MB and > still about 1.6MB after "bzip2" so that those files > should make the python RPM about 1.6MB bigger. > > At least according to > http://www.python.org/doc/1.5.1p1/tut/node43.html > I wonder if there is a noticeable advantage to provide > .pyc and .pyo files in the RPM because it reads: > "A program doesn't run any faster when it is read from > a ".pyc" or ".pyo" file than when it is read from a ".py" file; > the only thing that's faster about ".pyc" or ".pyo" files is > the speed with which they are loaded." True, and that's why we provide them. Think about it this way: python programs need to be compiled too. We provide the "binaries" form so that users' computers don't need to do the compilation themselves. But even more important is that regular user doesn't have write access to wherever the .py files are stored. You would need to run each python program as root first, otherwise python would need to compile the sources every time the program is run. > I am no Python expert at all and I would be happy if a Python expert > could provide some background information. is this background sufficient? ;e) > > > Kind Regards > Johannes Meixner regards jan matejek -BEGIN PGP SIGNATURE- Version: GnuPG v2.0.4-svn0 (GNU/Linux) Comment: Using GnuPG with SUSE - http://enigmail.mozdev.org iD8DBQFHPJmGjBrWA+AvBr8RAu4nAJ44givXOzCG9LY1ge9DWoSouwTFHQCgjlf2 WA9ogJSDXliAvm2NxmAsy2E= =zapY -END PGP SIGNATURE- - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
Re: [opensuse-packaging] How to deal with Python .pyc and .pyo files?
Hello, On Nov 15 14:25 Dirk Mueller wrote (shortened): > On Thursday 15 November 2007, Johannes Meixner wrote: > > > Think about the "worst case" when the user replaces our Pyhon > > with whatever self-compiled Pyhon. > > > > Are byte-compiled Python .pyc and .pyo files the same for any Python > > and/or is any Python sufficiently smart to know when .pyc and/or .pyo > > files are outdated (even if the matching .py files are unchanged)? > > they`re installed in a versioned directory. Does this mean that .pyc and .pyo files can be different for different Python versions? Python .pyc and .pyo files are not necessarily installed in a versioned directory like /usr/lib/python2.5/. They are installed where the matching .py file is. On a openSUSE 10.3 i386 default system I find .py files in /usr/share/emacs/22.1/etc/ /usr/share/hplip/* /usr/share/texmf/doc/generic/enctex/ /usr/lib/gimp/2.0/* /usr/lib/ooo-2.0/* /usr/bin/ What happens if Python version N creates /somewhere/file.pyc and later there is an update to Python version M? Is Python version M sufficiently smart to know that /somewhere/file.pyc is probably outdated? > > I wonder why in this case small RPMs seem not to count. > > So far printing with more than 100MB of data is the bigger factor > compared to a couple of mb we could save by not packaging pyc files. I do not understand this logic. I would understand if you said that packaging pyc files doesn't matter regarding RPM package size but I wonder why you suddenly start talking about printing? What do you want to tell reagrding printing, Python, and RPM package size? Kind Regards Johannes Meixner -- SUSE LINUX Products GmbH, Maxfeldstrasse 5, 90409 Nuernberg, Germany AG Nuernberg, HRB 16746, GF: Markus Rex - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
Re: [opensuse-packaging] How to deal with Python .pyc and .pyo files?
On Wednesday 14 November 2007, Stephan Kulow wrote: > will look for .pyo and python looks for .pyc. So you need to > package both, but can hardlink them if they are the same (fdupes is your > friend) Oh, we did test meanwhile that this works? ;) judging from the strace it looks like python correctly unlinks the file before trying to re-write it, so it should be okay. -- RPMLINT information under http://en.opensuse.org/Packaging/RpmLint - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
Re: [opensuse-packaging] How to deal with Python .pyc and .pyo files?
On Thursday 15 November 2007, Johannes Meixner wrote: > Think about the "worst case" when the user replaces our Pyhon > with whatever self-compiled Pyhon. > > Are byte-compiled Python .pyc and .pyo files the same for any Python > and/or is any Python sufficiently smart to know when .pyc and/or .pyo > files are outdated (even if the matching .py files are unchanged)? they`re installed in a versioned directory. python does not do any checking other than timestamp comparison, so it will never read an outdated bytecompiled version. Thats about it. > I wonder why in this case small RPMs seem not to count. So far printing with more than 100MB of data is the bigger factor compared to a couple of mb we could save by not packaging pyc files. > I am no Python expert at all and I would be happy if a Python expert > could provide some background information. if you %fdupe the pyc/pyo files, the overhead is lower. and the parsing overhead is significant for smaller short lived python scripts. Greetings, Dirk -- RPMLINT information under http://en.opensuse.org/Packaging/RpmLint - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
Re: [opensuse-packaging] How to deal with Python .pyc and .pyo files?
Hello, On Nov 14 16:00 Marcus Rueckert wrote (shortened): > > On Wed, 2007-11-14 at 15:08 +0100, Johannes Meixner wrote: > > > Hello, > > > > > > what is the recommended way for packaging Python stuff? > > > > > > Either > > > have byte-compiled Python .pyc and .pyo files packaged into the RPM > > > or > > > let this be done by the currently actually installed Python system > > > during run-time on the end-user's computer and not by whatever > > > (perhaps different) Python system during package build-time? > > as we build with the same python version as the user will run it later. > you can safely package the .pyc files. I understand that it is the same Python version directly after a new system installation. But what happens when the user later change/update his Pyhon? Think about the "worst case" when the user replaces our Pyhon with whatever self-compiled Pyhon. Are byte-compiled Python .pyc and .pyo files the same for any Python and/or is any Python sufficiently smart to know when .pyc and/or .pyo files are outdated (even if the matching .py files are unchanged)? > i would even recommend packaging > them so they get removed if you uninstall the package. Removal when uninstalling could also be done by whatever other magic (e.g. via %ghost or a %preun scriptlet in RPM). I wonder why in this case small RPMs seem not to count. For example the installed .pyc and .pyo files in python on a openSUSE 10.3 i386 system are more than 7MB and still about 1.6MB after "bzip2" so that those files should make the python RPM about 1.6MB bigger. At least according to http://www.python.org/doc/1.5.1p1/tut/node43.html I wonder if there is a noticeable advantage to provide .pyc and .pyo files in the RPM because it reads: "A program doesn't run any faster when it is read from a ".pyc" or ".pyo" file than when it is read from a ".py" file; the only thing that's faster about ".pyc" or ".pyo" files is the speed with which they are loaded." I am no Python expert at all and I would be happy if a Python expert could provide some background information. Kind Regards Johannes Meixner -- SUSE LINUX Products GmbH, Maxfeldstrasse 5, 90409 Nuernberg, Germany AG Nuernberg, HRB 16746, GF: Markus Rex - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
Re: [opensuse-packaging] How to deal with Python .pyc and .pyo files?
Am Mittwoch 14 November 2007 schrieb JP Rosevear: > On Wed, 2007-11-14 at 15:08 +0100, Johannes Meixner wrote: > > Hello, > > > > what is the recommended way for packaging Python stuff? > > > > Either > > have byte-compiled Python .pyc and .pyo files packaged into the RPM > > or > > let this be done by the currently actually installed Python system > > during run-time on the end-user's computer and not by whatever > > (perhaps different) Python system during package build-time? > > This is kind of what I was trying to figure out myself for > python-gtkglext. The python automake macros seem to create both at > install time, but the files are identical. > The problem is: they don't have to be identical. And python -O will look for .pyo and python looks for .pyc. So you need to package both, but can hardlink them if they are the same (fdupes is your friend) Greetings, Stephan - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
Re: [opensuse-packaging] How to deal with Python .pyc and .pyo files?
On 2007-11-14 09:33:03 -0500, JP Rosevear wrote: > On Wed, 2007-11-14 at 15:08 +0100, Johannes Meixner wrote: > > Hello, > > > > what is the recommended way for packaging Python stuff? > > > > Either > > have byte-compiled Python .pyc and .pyo files packaged into the RPM > > or > > let this be done by the currently actually installed Python system > > during run-time on the end-user's computer and not by whatever > > (perhaps different) Python system during package build-time? > > This is kind of what I was trying to figure out myself for > python-gtkglext. The python automake macros seem to create both at > install time, but the files are identical. as we build with the same python version as the user will run it later. you can safely package the .pyc files. i would even recommend packaging them so they get removed if you uninstall the package. hope this helps darix -- openSUSE - SUSE Linux is my linux openSUSE is good for you www.opensuse.org - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
Re: [opensuse-packaging] How to deal with Python .pyc and .pyo files?
On Wed, 2007-11-14 at 15:08 +0100, Johannes Meixner wrote: > Hello, > > what is the recommended way for packaging Python stuff? > > Either > have byte-compiled Python .pyc and .pyo files packaged into the RPM > or > let this be done by the currently actually installed Python system > during run-time on the end-user's computer and not by whatever > (perhaps different) Python system during package build-time? This is kind of what I was trying to figure out myself for python-gtkglext. The python automake macros seem to create both at install time, but the files are identical. -JP -- JP Rosevear <[EMAIL PROTECTED]> Novell, Inc. - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
[opensuse-packaging] How to deal with Python .pyc and .pyo files?
Hello, what is the recommended way for packaging Python stuff? Either have byte-compiled Python .pyc and .pyo files packaged into the RPM or let this be done by the currently actually installed Python system during run-time on the end-user's computer and not by whatever (perhaps different) Python system during package build-time? Kind Regards Johannes Meixner -- SUSE LINUX Products GmbH, Maxfeldstrasse 5, 90409 Nuernberg, Germany AG Nuernberg, HRB 16746, GF: Markus Rex - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
