Re: [Python-Dev] Mysterious Python pyc file corruption problems
On Thu, 16 May 2013 11:42:30 -0400 Barry Warsaw ba...@python.org wrote: On May 16, 2013, at 08:33 AM, Nick Coghlan wrote: Personally, I would be suspicious of developmental web services doing auto-reloading while an installer is recompiling the world. I don't have enough context to be sure how plausible that is as a possible explanation, though. It's possible that some Python written system service is getting invoked during the bytecompile-on-installation phase. But now that we've found cases of this problem with Python 3.3, I'm less inclined to follow that line of reasoning since it implements atomic rename. Please try to reproduce it by adding e.g. some sleep() calls in the middle of the writing routine. Regards Antoine. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Mysterious Python pyc file corruption problems
Hi all, How about using the shared-or-exclusive advisory file locks (with flock() or fcntl())? It may only work on Posix though. A bientôt, Armin. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Mysterious Python pyc file corruption problems
On May 16, 2013, at 02:19 PM, Guido van Rossum wrote: Now consider the following scenario. It involves *three* processes. - Two unrelated processes both start and want to import the same module. - They both see the .pyc file is missing/corrupt and decide to write it. - The first process finishing writing the file, writing the correct header. - Now a third process wants to import the module, sees the valid header, and starts reading the file. - However, while this is going on, the second process gets ready to write the file. - The second process truncates the file, writes the dummy header, and then stalls. - At this point the third process (which thought it was reading a valid file) sees an unexpected EOF because the file has been truncated. Now, this would explain the EOFError, but not necessarily the ValueError with unknown type code. However, it looks like marshal doesn't always check for EOF immediately (sometimes it calls getc() without checking the result, and sometimes it doesn't check the error state after calling r_string()), so I think all the errors are actually explainable from this scenario. Thanks for this, it's a very interesting scenario. I think this isn't a complete explanation of what's going on though. I've spoken with our defect analyst and looked at a bunch of the bug reports, and as far as we can tell, the corruptions are permanent. Users generally have to take manual action to delete the .pyc files and re-create them. One thing I hadn't realized until now is that until Python 3.4, py_compile.py doesn't write the pyc files atomically, and in fact this is the mechanism we're using to create the pyc files at package installation time. That could explain why we're still seeing these issues even in Python 3.3. I've also uncovered a bug from 2010 reported in Debian[1] about pyc file corruptions that happened when the byte-compilation driver program exited before its workers[2] could complete. We're definitely seeing issues post-landing of this fix, so I need to do some more analysis to see if that fix was enough. If it wasn't, and we're not doing atomic renames, than that could explain the permanent corruptions. Cheers, -Barry [1] http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=590224 [2] the workers each call calling `$PYTHON -m py_compile - py-filenames` signature.asc Description: PGP signature ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Mysterious Python pyc file corruption problems
-BEGIN PGP SIGNED MESSAGE- Hash: SHA256 On May 16, 2013, at 11:48 PM, Tres Seaver wrote: I can confirm at least that I have seen this problem within the last two weeks on Ubuntu boxes unrelated to the thw Debian / Ubuntu build infrastruction. Hi Tres. If you see this happen, *please* get in touch with me, preferably before you fix it ;). I'd like to do some additional analysis on a broken system in semi-realtime. Cheers, - -Barry -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.12 (GNU/Linux) iQIcBAEBCAAGBQJRllovAAoJEBJutWOnSwa/+kgP/iyqwiKJvSszzPITSSU8lOps Ll75gWXyyrLyQ/NBp4neo+d7hsf/GqLT3wn6dc7dcq23khqo4c6tYlmdVaZ5QJGk qu9BjSPdKLdDQFt3k3MyHpjJKwOa6Fn/JmCyZnoPd2zST/RZUsL6kCBOosG6FGTU QJUeEM/GIv5wQ60tfdVmI3zBtYYkjfrCX6uHcw2xFt1JHAKKgH/rNh72t2q3YKKG yqVAbyxQyjxEkS7IQmacPPSy7sEbT5GD8xR9F/P6w5h05DaTqbau9haF2kyxhPvI wEP0UQZWsZ/QmMNaS9w/WcWx31ASCNR4NIYqdWYd5KHQqs7Y1Vf6sFW1hNhfPFR9 0C4wJhublK1ewKYf5AjHeLOEVddN1xNGqNZr/7FouOqNlDYwRObde8J0FONWzHkl GV6qqxvKnq4FI4Y+EtUaRS52j3NkMhD22bpkTf2EJij1LV80AsHeB01o/ZBUjTqu jmgJ2QLS34h3gGIU9+OgE9rtzTACdI783SQ5827hiCVaE62IhTgrkLGBNSv1WFxp Onc+E2YteEwHtYMsBK1ck8YQkbgC/XYpSAVZGoW8xbXy5/+33fQRUZPj9zXNBW2F Jzgo+mGqPC5SIKkaiZY22GYERC6bvwCsoV9o+DpwmCUhwT/2ATb7IAp+8Vembjit JqMBWyvdEg4uzjnDuAYZ =93Og -END PGP SIGNATURE- ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Mysterious Python pyc file corruption problems
On May 17, 2013, at 12:10 AM, Thomas Wouters wrote: The 'unknown type codes' can also be explained if the two processes writing to the .pyc files are *different Python versions*. As you may recall, at Google we used to use modified Python interpreters that used '.pyc-2.2', '.pyc-2.4', etc, for the pyc extension. That was because otherwise different Python versions would keep overwriting the .pyc files of shared Python modules, and at Google scale it caused all manner of problems... I guess Ubuntu is approaching Google scale ;-) I'd like to think so. :) But I don't think this is part of the equation. For Python 2 on Debian/Ubuntu, we use an elaborate symlink farm to keep all pyc files in Python-version-specific directories. The stdlib files are already segregated, but the symlink farm takes care of package installs. Note that the symlinks are to the .py files, not the pyc files. Fortunately we almost don't care about this anymore. We dropped 2.6 in Ubuntu a while ago and we'll very likely drop 2.6 in Debian Jessie. We don't care about any Python 3s earlier than 3.2, and getting rid of the symlink farm was the primary motivator for PEP 3147. Cheers, -Barry signature.asc Description: PGP signature ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Mysterious Python pyc file corruption problems
On May 16, 2013, at 05:30 PM, Brett Cannon wrote: Just so people know, this is how we used to do it. In importlib we write the entire file to a temp file and then to an atomic rename. Yep, and I suspect that our fix, even if we don't completely identify the root cause, will be to change py_compile.py to do atomic renames. Whether that would be an appropriate fix for 3.2, 3.3 and 2.7 is a different discussion. -Barry signature.asc Description: PGP signature ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Mysterious Python pyc file corruption problems
On May 16, 2013, at 04:52 PM, Terry Jan Reedy wrote: If the corruption only happens on Ubuntu, that would constitute 'rhyme' ;-). I realize that asking for reports on other systems is part of the reason you posted, but I don't remember seeing any others yet. Right. :) It's harder to dig out similar problems in Debian[1] but it's pretty clear that there have been *some* similar reports in Debian. Ubuntu and Debian share almost all their Python infrastructure. It would definitely be interesting to whether Fedora/RedHat or any other Linux distros have seen similar problems. I don't know how Fedora/RH does package installation. In Debian/Ubuntu, we do not ship pyc files, but instead they are generated in post-installation scripts, which boil down to calls to `$PYTHON -m py_compile - filenames`. Do failures only occur during compileall process? (or whatever substitute you use). No, they are all post-installation failures in unrelated packages that try to import pure-Python modules. AFAICT, the post-installation byte-compilation scripts are not erroring. Doing a post-compilation verification step might be interesting, but I bet backporting atomic renames to py_compile.py will fix the problem, or at least band-aid over it. ;) -Barry ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Mysterious Python pyc file corruption problems
On Fri, 2013-05-17 at 12:42 -0400, Barry Warsaw wrote: On May 16, 2013, at 04:52 PM, Terry Jan Reedy wrote: If the corruption only happens on Ubuntu, that would constitute 'rhyme' ;-). I realize that asking for reports on other systems is part of the reason you posted, but I don't remember seeing any others yet. Right. :) It's harder to dig out similar problems in Debian[1] but it's pretty clear that there have been *some* similar reports in Debian. Ubuntu and Debian share almost all their Python infrastructure. It would definitely be interesting to whether Fedora/RedHat or any other Linux distros have seen similar problems. FWIW I don't recall seeing such problems on Fedora/RH, though that could be due to... I don't know how Fedora/RH does package installation. In Debian/Ubuntu, we do not ship pyc files, but instead they are generated in post-installation scripts, which boil down to calls to `$PYTHON -m py_compile - filenames`. Fedora/RH pregenerate the .pyc files during rpm creation, and they exist as part of the rpm payload. Dave ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Mysterious Python pyc file corruption problems
On May 17, 2013, at 01:19 PM, David Malcolm wrote: Fedora/RH pregenerate the .pyc files during rpm creation, and they exist as part of the rpm payload. Good to know, thanks. Do you use `$PYTHON -m py_compile` to generate the pyc files at build time? -Barry signature.asc Description: PGP signature ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Mysterious Python pyc file corruption problems
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 On 05/17/2013 12:26 PM, Barry Warsaw wrote: On May 16, 2013, at 11:48 PM, Tres Seaver wrote: I can confirm at least that I have seen this problem within the last two weeks on Ubuntu boxes unrelated to the thw Debian / Ubuntu build infrastruction. Hi Tres. If you see this happen, *please* get in touch with me, preferably before you fix it ;). I'd like to do some additional analysis on a broken system in semi-realtime. Wilco (although I don't know for sure what provoked it: my memory is that it was while running 'tox' or 'detox' for ZODB). Tres. - -- === Tres Seaver +1 540-429-0999 tsea...@palladion.com Palladion Software Excellence by Designhttp://palladion.com -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.11 (GNU/Linux) Comment: Using GnuPG with undefined - http://www.enigmail.net/ iEYEARECAAYFAlGWfCAACgkQ+gerLs4ltQ5YcQCguzlxAP8InrLEgdGx7JiK0as4 z9MAnR53bubpntt+272Y0BNYlEO8YcdI =LSAR -END PGP SIGNATURE- ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Mysterious Python pyc file corruption problems
On 5/17/2013 12:42 PM, Barry Warsaw wrote: On May 16, 2013, at 04:52 PM, Terry Jan Reedy wrote: Do failures only occur during compileall process? (or whatever substitute you use). No, they are all post-installation failures in unrelated packages that try to import pure-Python modules. What I mean is, is the corruption (not the detection of corruption) only happening during mass compilation of the stdlib? When user imports a single non-stdlib file he has written the first time, does that ever get corrupted. AFAICT, the post-installation byte-compilation scripts are not erroring. I THINK that you are answering my question by saying that corruption only happens during installation mass compilation. Doing a post-compilation verification step might be interesting, but I bet backporting atomic renames to py_compile.py will fix the problem, or at least band-aid over it. ;) I intended to suggest that py_compile be changed to do that. Then Brett said it already had for 3.4. I see no reason why not to backport, but maybe someone else will. The main design use of marshal is to produce .pyc files that consist of a prefix and marshalled codeobject. Perhaps marshal.dump(s) should be extended to take a third prefix='' parameter that would be prepended to the result as produced today. .dump first does .dumps, though inline. I assume that .dumps constructs a string by starting with [], appending pieces, and joining. At least, any composite object dump would. The change would amount to starting with [prefix] instead of []. Then py_compile would amount to pyc = pyc-prefix marshal.dump(codeobject, file, pyc) Terry ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Mysterious Python pyc file corruption problems
On May 17, 2013, at 03:02 PM, Terry Jan Reedy wrote: What I mean is, is the corruption (not the detection of corruption) only happening during mass compilation of the stdlib? When user imports a single non-stdlib file he has written the first time, does that ever get corrupted. It's not limited to the stdlib, but yes, as far as we can tell, it happens during package installation on an end-user system, when the trigger script mass byte-compiles a package's Python source files. I intended to suggest that py_compile be changed to do that. Then Brett said it already had for 3.4. I see no reason why not to backport, but maybe someone else will. I tend to agree. The main design use of marshal is to produce .pyc files that consist of a prefix and marshalled codeobject. Perhaps marshal.dump(s) should be extended to take a third prefix='' parameter that would be prepended to the result as produced today. .dump first does .dumps, though inline. I assume that .dumps constructs a string by starting with [], appending pieces, and joining. At least, any composite object dump would. The change would amount to starting with [prefix] instead of []. Then py_compile would amount to pyc = pyc-prefix marshal.dump(codeobject, file, pyc) That wouldn't be a backportable change, and right now I'm trying to stay focused on fixing this specific problem. ;) -Barry ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Mysterious Python pyc file corruption problems
On Fri, 2013-05-17 at 14:23 -0400, Barry Warsaw wrote: On May 17, 2013, at 01:19 PM, David Malcolm wrote: Fedora/RH pregenerate the .pyc files during rpm creation, and they exist as part of the rpm payload. Good to know, thanks. Do you use `$PYTHON -m py_compile` to generate the pyc files at build time? We use compileall.compiledir() most of the time, but occasionally use py_compile.compile() Specifically, for python 2, the core rpm-build package has a script: /usr/lib/rpm/brp-python-bytecompile run automatically in a postprocessing phase after the upstream source has installed to a DESTDIR, and this invokes compileall.compiledir() on all .py files in the package payload, with various logic to segment the different parts of the filesystem to be bytecompiled by the appropriate python binary (since we have duplicate .py files for different python runtimes). This is all done sequentially, so I'd be surprised if different pythons splatted on each other's .pyc files at this time. In addition, python3-devel contains a: /etc/rpm/macros.pybytecompile which defines a py_byte_compile() macro, which can be used for overriding these rules (IIRC), and this does use pycompile.compile() Hope this is helpful Dave ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Mysterious Python pyc file corruption problems
Am 15.05.2013 22:58, schrieb Barry Warsaw: I am looking into a particularly vexing Python problem on Ubuntu that manifests in several different ways. I think the problem is the same one described in http://bugs.python.org/issue13146 and I sent a message on the subject to the ubuntu-devel list: https://lists.ubuntu.com/archives/ubuntu-devel/2013-May/037129.html please consider that Ubuntu does have some other upgrade issues, when files in the dpkg database (/var/lib/dpkg/info/*) are corrupted, or just are files having null bytes. So these and the pyc issues share in common that these files are written after a package is unpacked. I'm not saying that the problem might be with the pyc writing, but we do see other file corruption as well. Matthias ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Mysterious Python pyc file corruption problems
On Sat, May 18, 2013 at 3:19 AM, David Malcolm dmalc...@redhat.com wrote: On Fri, 2013-05-17 at 12:42 -0400, Barry Warsaw wrote: On May 16, 2013, at 04:52 PM, Terry Jan Reedy wrote: If the corruption only happens on Ubuntu, that would constitute 'rhyme' ;-). I realize that asking for reports on other systems is part of the reason you posted, but I don't remember seeing any others yet. Right. :) It's harder to dig out similar problems in Debian[1] but it's pretty clear that there have been *some* similar reports in Debian. Ubuntu and Debian share almost all their Python infrastructure. It would definitely be interesting to whether Fedora/RedHat or any other Linux distros have seen similar problems. FWIW I don't recall seeing such problems on Fedora/RH, though that could be due to... I don't know how Fedora/RH does package installation. In Debian/Ubuntu, we do not ship pyc files, but instead they are generated in post-installation scripts, which boil down to calls to `$PYTHON -m py_compile - filenames`. Fedora/RH pregenerate the .pyc files during rpm creation, and they exist as part of the rpm payload. So in effect, we'd always be doing an atomic rename (and file conflicts between RPMs wouldn't be allowed in the first place). Combined with Brett's info that even 3.3 doesn't use atomic renames for pre-compilation (only for implicit compilation), I consider that strong evidence in favour of Guido's theory that Debian are getting write conflicts somewhere. Cheers, Nick. -- Nick Coghlan | ncogh...@gmail.com | Brisbane, Australia ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Mysterious Python pyc file corruption problems
-BEGIN PGP SIGNED MESSAGE- Hash: SHA256 On May 15, 2013, at 06:06 PM, Tres Seaver wrote: On 05/15/2013 04:58 PM, Barry Warsaw wrote: This leads me to hypothesize that the bug is due to an as yet unidentified race condition during installation of Python source code on Ubuntu, which is normally when we automatically byte compile the source to .pyc files. Any chance you are using 'detox' or the equivalent to run tests on mutliple interpreters in parallel? The only bad marshall data errors I have seen lately seemed to be provoked by that kind of practice. Nope. PyPI's detox isn't even available in Ubuntu currently. (The detox package in Ubuntu is something else.) Tests should only be run at package build time, not installation time, and the byte compiling of source files at installation time *should* be single threaded and single process. We've since found a few cases where Python 3.3 pyc files are probably corrupted, so that shoots down my theory about a race condition on reading/writing pyc files, since 3.3 implements atomic-rename and *should* be immune to that kind of thing. It's still a mystery though. - -Barry -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.12 (GNU/Linux) iQIcBAEBCAAGBQJRlP3SAAoJEBJutWOnSwa/Xy8QAI6Ul/s0nF+rac8nGy6fieLB FHHEfmjIgj6MkyUUw/zcbR48ELiOPkkV+GM4HJnY/H2ZG9vZqwuQYWFI0cgIBmxj EmHfKK7OTdaaHZeNTt83RDwGRLUS/gYXQ7JVikGyFnSbftfmUoN/y/ndlX5DX1hT ecDHVtXCH/ti/kcOWe2OlMABONZQPW0qYB7/0PiCCmaOxulqUsz20Ofy8SfWmSPd Rbig5i8fSnI98dkLVUzyy1tbUkdRkLBro/hawu1V9y7qVkoYx1Jz6p8XkQLp3jES m22m+6CLrnD39HxvJGGNkIaYmu5xTW+rK/Li8OrfOKx6QVIZ+XQRJFkiXnKmiezk sMYv/psySWJ/BSImsQOSt/sLHJWAGh8fkMIBpx9tI3BWMvyMkI0Hs9l7JyQn0moo oSTNb9AbgRSrkh0rVv4fhOhd1Ir3LXYTGwwYE5+o7tB/Pp0AKi2tX/XTBctDpy86 xqNHOaCV0hRA2Y+/C2QAAA7LRruP0yv10DfkciVUHR7UzbXgViICEEUizGmnkni7 utGg9EDk5VcSeg2ySxhX9Uj3E2M/ijOuYpXUJ2Gwd4UNUT5XGK1+6i2JTO2pEQOM HqhsGqk4WJsfEBTIrAt4NSxZyEuQ2nRV3MIsNaVCDp1FDySZWt3Cckq8hkZ/6vOM 7ncE6aG1cJgq4WKErvCk =BMFW -END PGP SIGNATURE- ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Mysterious Python pyc file corruption problems
On May 16, 2013, at 08:33 AM, Nick Coghlan wrote: Personally, I would be suspicious of developmental web services doing auto-reloading while an installer is recompiling the world. I don't have enough context to be sure how plausible that is as a possible explanation, though. It's possible that some Python written system service is getting invoked during the bytecompile-on-installation phase. But now that we've found cases of this problem with Python 3.3, I'm less inclined to follow that line of reasoning since it implements atomic rename. Unless there's a flaw in importlib we haven't identified yet. -Barry ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Mysterious Python pyc file corruption problems
-BEGIN PGP SIGNED MESSAGE- Hash: SHA256 Am 16.05.2013 17:40, schrieb Barry Warsaw: We've since found a few cases where Python 3.3 pyc files are probably corrupted, so that shoots down my theory about a race condition on reading/writing pyc files, since 3.3 implements atomic-rename and *should* be immune to that kind of thing. It's still a mystery though. Are you able to reproduce the issue? Perhaps you could use inotify to track down file activity. It shouldn't affect timing much and you can track if more than one process it writing to the same file. -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.11 (GNU/Linux) Comment: Using GnuPG with undefined - http://www.enigmail.net/ iQIcBAEBCAAGBQJRlQe4AAoJEMeIxMHUVQ1F7K8P/1QoH8/iJP60dHQHfU12AYFY nUu1ztRmwTSx0eYEumwsRF5iUuNse7kFzYM0u02lEmZyuk34hLtBBcnNGA0wJ4mZ SdiXL1ZA2levg9Qlr8cPQqgnlm9aXnIazQKbUJ+/MOGBdTPBemMunMyMSsNg5ENT gLEVb/lufNssAoo+M0QKq9EjE2xSQEsFjUDM575KHbq006EzdHp7on2xQ20pJzLc iq/qWAFh+kjS42Udk9luvAKy3iGJcGXnG9AY0hkLBh8tQYhISWplsBo5wiigZLyv PZ0tbh5h3bsi80FjDlSfVPFOzlt34xI6tPRONUj/XWLPfvBCGzwqjYGc5Z6+CkAF pPAamF0ntwq76mNBl9EABAY1q85SgEU+toft8KQdxm+SHuKINJc95R8x6ypsnYIQ Ol+L5nUy+zV3vCJe9TM2U2cUB/UWHLM0qGTSYowLqTXtv+1Y+J55g63kOLkfCrnF znVXMU5FMotlh6i1rK/uwBttJ+NdjOTL0+eVbVqm39bBA6PU7UgANNNSIWVPCbfu HwucdVwkY932TxiVpWZBSPVLQmjNHIOlVj8uFIkhBnEeWSYkpIu+wV4f+Gc048AP 7EYYMMHTdlodNdKRlr+ksczhJvO67STjKH0a+vB2fro/wjuxwcqD7A0qG0PfvURg 7vofW0fs9lKap0wk9DsT =v709 -END PGP SIGNATURE- ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Mysterious Python pyc file corruption problems
-BEGIN PGP SIGNED MESSAGE- Hash: SHA256 On May 16, 2013, at 06:22 PM, Christian Heimes wrote: Are you able to reproduce the issue? Perhaps you could use inotify to track down file activity. It shouldn't affect timing much and you can track if more than one process it writing to the same file. Sadly, no. I've never seen it on any of my own desktops or servers, and none of the bug reporters have been able to provide a reproducible test case. I tried some smaller test cases to see if I could trigger pyc writing race conditions, but failed to reproduce it there either. - -Barry -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.12 (GNU/Linux) iQIcBAEBCAAGBQJRlQuDAAoJEBJutWOnSwa/03AP/04BxVqqm0nrGIdtuKDej3MW 4qhXxbYExgpcFpclesFw479TatGCh3hBwDoosrYdk5Lrf8Bwa9FGUNbRJozdAGmE wAB1vq30mGm+2QtBuVPoXu3xrNWGGmUUtI0yzBSwnvxlfuIzsLkibZPMIMfdVCi+ f3/LSldowWx0DJp8V5TUq4GIhOfe3yccgIxMU55YbDj8cplzFJuBuBtO4DOGsoFI IPlFLwGPG503Nju37zzdkoq3Xkw4Og+vXtXsCv/rhAWIqnZgKYNF/CLv0dolZWFy GhjM5bfQtUWwxH6Ng1Wl2kcuCVmF1/vD2vTUCsgpA4qQc0nYrTy/q1OPho72x40o DvvaVHueDqH7N1xm64KL75sFxu6QDIniBbgV7gklU1z6P6ZVADwoilon8HC9FnJN w5I0sYLTnIHxUIrM0h0wi517gQTZHTSF0bQxKqynNV+PrZBprvB9lEkYCpy5tV0s LEqf+oUwXvGIOZ6Nmv2MyjQb0xajxHmzz+RO1qQ3R4tbiQjwGoqc43CrlxhVduJh 1VGM6b7ysZ2iwyJG+q0aVi9YSaStzzUvMPO2F+HTmE+r3MvgdTcKQQzLDuRF6LfV 74eWwtHBpiJuvdBG37uDQj5bU/oLWiYyfM52vASgHB4zoKOx0EUxAd1Wf5nyxc1E Bo0G3kYwbFaNvSnwcJZw =a4x0 -END PGP SIGNATURE- ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Mysterious Python pyc file corruption problems
On 05/16/2013 09:38 AM, Barry Warsaw wrote: -BEGIN PGP SIGNED MESSAGE- Hash: SHA256 On May 16, 2013, at 06:22 PM, Christian Heimes wrote: Are you able to reproduce the issue? Perhaps you could use inotify to track down file activity. It shouldn't affect timing much and you can track if more than one process it writing to the same file. Sadly, no. I've never seen it on any of my own desktops or servers, and none of the bug reporters have been able to provide a reproducible test case. I tried some smaller test cases to see if I could trigger pyc writing race conditions, but failed to reproduce it there either. Is it happening on the same machines? If so, perhaps a daemon to monitor those files and then scream and shout when one changes. Might help track down what's going on at the time. (Yeah, that does sound like saying 'inotify' but with more words...) -- ~Ethan~ ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Mysterious Python pyc file corruption problems
On May 16, 2013, at 09:44 AM, Ethan Furman wrote: Is it happening on the same machines? If so, perhaps a daemon to monitor those files and then scream and shout when one changes. Might help track down what's going on at the time. (Yeah, that does sound like saying 'inotify' but with more words...) No, it's all different kinds of machines, at different times, on different files. So far, there's no rhyme or reason to the corruptions that I can tell. We're trying to instrument things to collect more data when these failures do occur. -Barry ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Mysterious Python pyc file corruption problems
On Thu, May 16, 2013 at 11:04 AM, Barry Warsaw ba...@python.org wrote: On May 16, 2013, at 09:44 AM, Ethan Furman wrote: Is it happening on the same machines? If so, perhaps a daemon to monitor those files and then scream and shout when one changes. Might help track down what's going on at the time. (Yeah, that does sound like saying 'inotify' but with more words...) No, it's all different kinds of machines, at different times, on different files. So far, there's no rhyme or reason to the corruptions that I can tell. We're trying to instrument things to collect more data when these failures do occur. Even on machines with ECC ram and reliable storage, not owned by l33t gam0rzs weenies who overclock things? -gps ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Mysterious Python pyc file corruption problems
On 5/16/2013 2:04 PM, Barry Warsaw wrote: No, it's all different kinds of machines, at different times, on different files. So far, there's no rhyme or reason to the corruptions that I can tell. If the corruption only happens on Ubuntu, that would constitute 'rhyme' ;-). I realize that asking for reports on other systems is part of the reason you posted, but I don't remember seeing any others yet. We're trying to instrument things to collect more data when these failures do occur. Do failures only occur during compileall process? (or whatever substitute you use). At the end of py_compile.complile, after the with block that opens, writes, flushes, and closes, you could add with open(cfile, 'rb') as fc: read header and unmarshal rest This would be a high-level write and verify. Verify would be a bit faster if marshal.dump were replaced by marshal.dumps + write to keep alive the string version of the code object. Then the codeobject comparison in the verify step would be replaced by string comparison. You could also read and verify (by unmarshal) after the compile-all process (faster than importing). Terry ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Mysterious Python pyc file corruption problems
This reminds me of the following bug, which can happen when two processes are both writing the .pyc file and a third is reading it. First some background. When writing a .pyc file, we use the following strategy: - open the file for writing - write a dummy header (four null bytes) - write the .py file's mtime - write the marshalled code object - replace the dummy heaer with the correct magic word Even py_compile.py (used by compileall.py) uses this strategy. When reading a .pyc file, we ignore it when the magic word isn't there (or when the mtime doesn't match that of the .py file exactly), and then we will write it back like described above. Now consider the following scenario. It involves *three* processes. - Two unrelated processes both start and want to import the same module. - They both see the .pyc file is missing/corrupt and decide to write it. - The first process finishing writing the file, writing the correct header. - Now a third process wants to import the module, sees the valid header, and starts reading the file. - However, while this is going on, the second process gets ready to write the file. - The second process truncates the file, writes the dummy header, and then stalls. - At this point the third process (which thought it was reading a valid file) sees an unexpected EOF because the file has been truncated. Now, this would explain the EOFError, but not necessarily the ValueError with unknown type code. However, it looks like marshal doesn't always check for EOF immediately (sometimes it calls getc() without checking the result, and sometimes it doesn't check the error state after calling r_string()), so I think all the errors are actually explainable from this scenario. -- --Guido van Rossum (python.org/~guido) ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Mysterious Python pyc file corruption problems
On Thu, May 16, 2013 at 5:19 PM, Guido van Rossum gu...@python.org wrote: This reminds me of the following bug, which can happen when two processes are both writing the .pyc file and a third is reading it. First some background. When writing a .pyc file, we use the following strategy: - open the file for writing - write a dummy header (four null bytes) - write the .py file's mtime - write the marshalled code object - replace the dummy heaer with the correct magic word Just so people know, this is how we used to do it. In importlib we write the entire file to a temp file and then to an atomic rename. Even py_compile.py (used by compileall.py) uses this strategy. py_compile as of Python 3.4 now just uses importlib directly, so it matches its semantics. -Brett When reading a .pyc file, we ignore it when the magic word isn't there (or when the mtime doesn't match that of the .py file exactly), and then we will write it back like described above. Now consider the following scenario. It involves *three* processes. - Two unrelated processes both start and want to import the same module. - They both see the .pyc file is missing/corrupt and decide to write it. - The first process finishing writing the file, writing the correct header. - Now a third process wants to import the module, sees the valid header, and starts reading the file. - However, while this is going on, the second process gets ready to write the file. - The second process truncates the file, writes the dummy header, and then stalls. - At this point the third process (which thought it was reading a valid file) sees an unexpected EOF because the file has been truncated. Now, this would explain the EOFError, but not necessarily the ValueError with unknown type code. However, it looks like marshal doesn't always check for EOF immediately (sometimes it calls getc() without checking the result, and sometimes it doesn't check the error state after calling r_string()), so I think all the errors are actually explainable from this scenario. -- --Guido van Rossum (python.org/~guido) ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/brett%40python.org ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Mysterious Python pyc file corruption problems
I still suspect this might explain most of what Barry saw, if not all. — Sent from Mailbox On Thu, May 16, 2013 at 2:36 PM, Brett Cannon br...@python.org wrote: On Thu, May 16, 2013 at 5:19 PM, Guido van Rossum gu...@python.org wrote: This reminds me of the following bug, which can happen when two processes are both writing the .pyc file and a third is reading it. First some background. When writing a .pyc file, we use the following strategy: - open the file for writing - write a dummy header (four null bytes) - write the .py file's mtime - write the marshalled code object - replace the dummy heaer with the correct magic word Just so people know, this is how we used to do it. In importlib we write the entire file to a temp file and then to an atomic rename. Even py_compile.py (used by compileall.py) uses this strategy. py_compile as of Python 3.4 now just uses importlib directly, so it matches its semantics. -Brett When reading a .pyc file, we ignore it when the magic word isn't there (or when the mtime doesn't match that of the .py file exactly), and then we will write it back like described above. Now consider the following scenario. It involves *three* processes. - Two unrelated processes both start and want to import the same module. - They both see the .pyc file is missing/corrupt and decide to write it. - The first process finishing writing the file, writing the correct header. - Now a third process wants to import the module, sees the valid header, and starts reading the file. - However, while this is going on, the second process gets ready to write the file. - The second process truncates the file, writes the dummy header, and then stalls. - At this point the third process (which thought it was reading a valid file) sees an unexpected EOF because the file has been truncated. Now, this would explain the EOFError, but not necessarily the ValueError with unknown type code. However, it looks like marshal doesn't always check for EOF immediately (sometimes it calls getc() without checking the result, and sometimes it doesn't check the error state after calling r_string()), so I think all the errors are actually explainable from this scenario. -- --Guido van Rossum (python.org/~guido) ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/brett%40python.org___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Mysterious Python pyc file corruption problems
On Thu, May 16, 2013 at 5:40 PM, Guido van Rossum gvanros...@gmail.com wrote: I still suspect this might explain most of what Barry saw, if not all. Quite possible, especially since he is seeing more issues on 3.2 than 3.3. Just wanted to fill people in on how 3.3 onwards does things is all. -Brett — Sent from Mailbox On Thu, May 16, 2013 at 2:36 PM, Brett Cannon br...@python.org wrote: On Thu, May 16, 2013 at 5:19 PM, Guido van Rossum gu...@python.org wrote: This reminds me of the following bug, which can happen when two processes are both writing the .pyc file and a third is reading it. First some background. When writing a .pyc file, we use the following strategy: - open the file for writing - write a dummy header (four null bytes) - write the .py file's mtime - write the marshalled code object - replace the dummy heaer with the correct magic word Just so people know, this is how we used to do it. In importlib we write the entire file to a temp file and then to an atomic rename. Even py_compile.py (used by compileall.py) uses this strategy. py_compile as of Python 3.4 now just uses importlib directly, so it matches its semantics. -Brett When reading a .pyc file, we ignore it when the magic word isn't there (or when the mtime doesn't match that of the .py file exactly), and then we will write it back like described above. Now consider the following scenario. It involves *three* processes. - Two unrelated processes both start and want to import the same module. - They both see the .pyc file is missing/corrupt and decide to write it. - The first process finishing writing the file, writing the correct header. - Now a third process wants to import the module, sees the valid header, and starts reading the file. - However, while this is going on, the second process gets ready to write the file. - The second process truncates the file, writes the dummy header, and then stalls. - At this point the third process (which thought it was reading a valid file) sees an unexpected EOF because the file has been truncated. Now, this would explain the EOFError, but not necessarily the ValueError with unknown type code. However, it looks like marshal doesn't always check for EOF immediately (sometimes it calls getc() without checking the result, and sometimes it doesn't check the error state after calling r_string()), so I think all the errors are actually explainable from this scenario. -- --Guido van Rossum (python.org/~guido) ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/brett%40python.org ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Mysterious Python pyc file corruption problems
On 5/16/2013 5:30 PM, Brett Cannon wrote: On Thu, May 16, 2013 at 5:19 PM, Guido van Rossum gu...@python.org wrote: This reminds me of the following bug, which can happen when two processes are both writing the .pyc file and a third is reading it. First some background. When writing a .pyc file, we use the following strategy: - open the file for writing - write a dummy header (four null bytes) - write the .py file's mtime - write the marshalled code object - replace the dummy heaer with the correct magic word Just so people know, this is how we used to do it. In importlib we write the entire file to a temp file and then to an atomic rename. Even py_compile.py (used by compileall.py) uses this strategy. py_compile as of Python 3.4 now just uses importlib directly, so it matches its semantics. But in 3.3, it still is as Guido describes, even though importlib is improved. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Mysterious Python pyc file corruption problems
On Thu, May 16, 2013 at 11:19 PM, Guido van Rossum gu...@python.org wrote: This reminds me of the following bug, which can happen when two processes are both writing the .pyc file and a third is reading it. First some background. When writing a .pyc file, we use the following strategy: - open the file for writing - write a dummy header (four null bytes) - write the .py file's mtime - write the marshalled code object - replace the dummy heaer with the correct magic word Even py_compile.py (used by compileall.py) uses this strategy. When reading a .pyc file, we ignore it when the magic word isn't there (or when the mtime doesn't match that of the .py file exactly), and then we will write it back like described above. Now consider the following scenario. It involves *three* processes. - Two unrelated processes both start and want to import the same module. - They both see the .pyc file is missing/corrupt and decide to write it. - The first process finishing writing the file, writing the correct header. - Now a third process wants to import the module, sees the valid header, and starts reading the file. - However, while this is going on, the second process gets ready to write the file. - The second process truncates the file, writes the dummy header, and then stalls. - At this point the third process (which thought it was reading a valid file) sees an unexpected EOF because the file has been truncated. Now, this would explain the EOFError, but not necessarily the ValueError with unknown type code. The 'unknown type codes' can also be explained if the two processes writing to the .pyc files are *different Python versions*. As you may recall, at Google we used to use modified Python interpreters that used '.pyc-2.2', '.pyc-2.4', etc, for the pyc extension. That was because otherwise different Python versions would keep overwriting the .pyc files of shared Python modules, and at Google scale it caused all manner of problems... I guess Ubuntu is approaching Google scale ;-) (The decision to rename to an awkward extension broke a lot of third-party tools; it was made before I -- or you, for that matter -- joined Google... Now we just turn on -B by default :) However, it looks like marshal doesn't always check for EOF immediately (sometimes it calls getc() without checking the result, and sometimes it doesn't check the error state after calling r_string()), so I think all the errors are actually explainable from this scenario. -- --Guido van Rossum (python.org/~guido) ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/thomas%40python.org -- Thomas Wouters tho...@python.org Hi! I'm an email virus! Think twice before sending your email to help me spread! ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Mysterious Python pyc file corruption problems
Guido van Rossum wrote: This reminds me of the following bug, which can happen when two processes are both writing the .pyc file and a third is reading it. ... I think all the errors are actually explainable from this scenario. The second writer will still carry on to write a valid .pyc file, though, won't it? So this wouldn't result in a permanently broken .pyc file being left behind, which is what the original problem description seemed say was happening. -- Greg ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Mysterious Python pyc file corruption problems
On Thu, May 16, 2013 at 3:27 PM, Greg Ewing greg.ew...@canterbury.ac.nz wrote: Guido van Rossum wrote: This reminds me of the following bug, which can happen when two processes are both writing the .pyc file and a third is reading it. ... I think all the errors are actually explainable from this scenario. The second writer will still carry on to write a valid .pyc file, though, won't it? So this wouldn't result in a permanently broken .pyc file being left behind, which is what the original problem description seemed say was happening. From the evidence that is not completely clear to me. Thomas Wouters' scenario with two different Python versions writing the same .pyc file could cause that; I don't know if Barry has ruled that possibility out yet. -- --Guido van Rossum (python.org/~guido) ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Mysterious Python pyc file corruption problems
On 17 May 2013 08:37, Guido van Rossum gu...@python.org wrote: On Thu, May 16, 2013 at 3:27 PM, Greg Ewing greg.ew...@canterbury.ac.nz wrote: Guido van Rossum wrote: This reminds me of the following bug, which can happen when two processes are both writing the .pyc file and a third is reading it. ... I think all the errors are actually explainable from this scenario. The second writer will still carry on to write a valid .pyc file, though, won't it? So this wouldn't result in a permanently broken .pyc file being left behind, which is what the original problem description seemed say was happening. From the evidence that is not completely clear to me. Thomas Wouters' scenario with two different Python versions writing the same .pyc file could cause that; I don't know if Barry has ruled that possibility out yet. 3.2 uses __pycache__, so it should only potentially conflict within the same version. I haven't heard any rumblings about anything like this in Fedora or RHEL, so my suspicions still lean towards a Debian or Ubuntu specific background service somehow managing to interfere. However, I'll ask explicitly on the Fedora Python list to see if anyone has encountered anything similar. Cheers, Nick. -- --Guido van Rossum (python.org/~guido) ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/ncoghlan%40gmail.com ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Mysterious Python pyc file corruption problems
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 On 05/16/2013 06:59 PM, Nick Coghlan wrote: 3.2 uses __pycache__, so it should only potentially conflict within the same version. I haven't heard any rumblings about anything like this in Fedora or RHEL, so my suspicions still lean towards a Debian or Ubuntu specific background service somehow managing to interfere. However, I'll ask explicitly on the Fedora Python list to see if anyone has encountered anything similar. I can confirm at least that I have seen this problem within the last two weeks on Ubuntu boxes unrelated to the thw Debian / Ubuntu build infrastruction. Tres. - -- === Tres Seaver +1 540-429-0999 tsea...@palladion.com Palladion Software Excellence by Designhttp://palladion.com -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.11 (GNU/Linux) Comment: Using GnuPG with undefined - http://www.enigmail.net/ iEYEARECAAYFAlGVqIgACgkQ+gerLs4ltQ6ksACePs7jO1TynGm3kNodpV4lPA2b VbgAoNNHMmQhJQhOvxuHMO/LFyv+Umho =KNdc -END PGP SIGNATURE- ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] Mysterious Python pyc file corruption problems
I am looking into a particularly vexing Python problem on Ubuntu that manifests in several different ways. I think the problem is the same one described in http://bugs.python.org/issue13146 and I sent a message on the subject to the ubuntu-devel list: https://lists.ubuntu.com/archives/ubuntu-devel/2013-May/037129.html I don't know what's causing the problem and have no way to reproduce it, but all the clues point to corrupt pyc files in Pythons 3.3. The common way this manifests is a traceback on an import statement. The actual error can be a ValueError: bad marshal data (unknown type code) such as in http://pad.lv/1010077 or an EOFError: EOF read where not expected as in http://pad.lv/1060842. We have many more instances of both of these. Since both error messages come from marshal.c when trying to read the pyc for a module being imported, I suspect that something is causing the pyc files to get partially overwritten or corrupted. The workaround is always to essentially blow away the .pyc file and re-create it. (Various different techniques can be used, but they all boil down to the same thing.) Another commonality is that this bug -- so far -- has not been observed in any Python 3.3 code, only 3.2 and earlier, including 2.7 and 2.6. This strengthens my hypothesis, since importlib in Python 3.3 included an atomic rename of the .pyc file whereas older Pythons only do an exclusive open on the pyc files, but do *not* do an atomic rename AFAICT. This leads me to hypothesize that the bug is due to an as yet unidentified race condition during installation of Python source code on Ubuntu, which is normally when we automatically byte compile the source to .pyc files. This can happen at package installation/upgrade time, or during a fresh install. In each of these cases there *should* be only one process attempting to write the .pyc, but my guess is that for some reason, multiple processes are trying to do this, triggering a truncation or other bogus content of .pyc files. Even in Python 3.3, it should not be possible to corrupt a .pyc when only a single process is involved, due to the import lock and/or GIL. The exclusive open of the .pyc file is clearly not enough of a protection in a multiprocess situation, since the bug has already been identified in Python on buildbots during test_multiprocessing. See http://bugs.python.org/issue13146 I think the list of errors we've seen is too extensive to chalk up to a hardware bug, and I think the systems involved are modern enough to not be subject to file system data loss. There could be a missing fsync somewhere though that might be involved. I think it's doubtful that buggy remote file systems (e.g. NFSv2) are involved. I could be wrong about any of that. I have not succeeded in writing a standalone reproducer using Python 2.7. So, the mystery is: what process on Ubuntu is exploiting holes in the exclusive open and causing this problem? Issue 13146 is closed because the fix was applied to Python 3.3 (see above), but it was not backported to earlier versions. I think it would not be that difficult to backport it, and I would be willing to do so for Python 2.7 and 3.2. We might include 2.6 in that, but only in Ubuntu since I can't see how this bug could be exploited as a security vulnerability. Thoughts? -Barry signature.asc Description: PGP signature ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Mysterious Python pyc file corruption problems
On Wed, May 15, 2013 at 4:58 PM, Barry Warsaw ba...@python.org wrote: I am looking into a particularly vexing Python problem on Ubuntu that manifests in several different ways. I think the problem is the same one described in http://bugs.python.org/issue13146 and I sent a message on the subject to the ubuntu-devel list: https://lists.ubuntu.com/archives/ubuntu-devel/2013-May/037129.html I don't know what's causing the problem and have no way to reproduce it, but all the clues point to corrupt pyc files in Pythons 3.3. The common way this manifests is a traceback on an import statement. The actual error can be a ValueError: bad marshal data (unknown type code) such as in http://pad.lv/1010077 or an EOFError: EOF read where not expected as in http://pad.lv/1060842. We have many more instances of both of these. Since both error messages come from marshal.c when trying to read the pyc for a module being imported, I suspect that something is causing the pyc files to get partially overwritten or corrupted. The workaround is always to essentially blow away the .pyc file and re-create it. (Various different techniques can be used, but they all boil down to the same thing.) Another commonality is that this bug -- so far -- has not been observed in any Python 3.3 code, only 3.2 and earlier, including 2.7 and 2.6. This strengthens my hypothesis, since importlib in Python 3.3 included an atomic rename of the .pyc file whereas older Pythons only do an exclusive open on the pyc files, but do *not* do an atomic rename AFAICT. Just an FYI, the renaming has caught at least one person off-guard: http://bugs.python.org/issue17222, so you might have to be careful about considering a backport. -Brett ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Mysterious Python pyc file corruption problems
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 On 05/15/2013 04:58 PM, Barry Warsaw wrote: This leads me to hypothesize that the bug is due to an as yet unidentified race condition during installation of Python source code on Ubuntu, which is normally when we automatically byte compile the source to .pyc files. Any chance you are using 'detox' or the equivalent to run tests on mutliple interpreters in parallel? The only bad marshall data errors I have seen lately seemed to be provoked by that kind of practice. Tres. - -- === Tres Seaver +1 540-429-0999 tsea...@palladion.com Palladion Software Excellence by Designhttp://palladion.com -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.11 (GNU/Linux) Comment: Using GnuPG with undefined - http://www.enigmail.net/ iEYEARECAAYFAlGUBvkACgkQ+gerLs4ltQ7nCwCcCfcAEGEN26qjQ9sGPaFRx1o4 DhwAoIlNwVU2lcJQ/hs5vQ1PXYT1uUwl =0s+X -END PGP SIGNATURE- ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Mysterious Python pyc file corruption problems
On 16 May 2013 08:11, Tres Seaver tsea...@palladion.com wrote: -BEGIN PGP SIGNED MESSAGE- Hash: SHA1 On 05/15/2013 04:58 PM, Barry Warsaw wrote: This leads me to hypothesize that the bug is due to an as yet unidentified race condition during installation of Python source code on Ubuntu, which is normally when we automatically byte compile the source to .pyc files. Any chance you are using 'detox' or the equivalent to run tests on mutliple interpreters in parallel? The only bad marshall data errors I have seen lately seemed to be provoked by that kind of practice. 3.2 shouldn't have a problem with that if the interpreters are different versions. Personally, I would be suspicious of developmental web services doing auto-reloading while an installer is recompiling the world. I don't have enough context to be sure how plausible that is as a possible explanation, though. Cheers, Nick. Tres. - -- === Tres Seaver +1 540-429-0999 tsea...@palladion.com Palladion Software Excellence by Designhttp://palladion.com -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.11 (GNU/Linux) Comment: Using GnuPG with undefined - http://www.enigmail.net/ iEYEARECAAYFAlGUBvkACgkQ+gerLs4ltQ7nCwCcCfcAEGEN26qjQ9sGPaFRx1o4 DhwAoIlNwVU2lcJQ/hs5vQ1PXYT1uUwl =0s+X -END PGP SIGNATURE- ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/ncoghlan%40gmail.com ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
