[issue42369] Reading ZipFile not thread-safe

Change by Serhiy Storchaka : -- resolution: -> fixed stage: patch review -> resolved status: open -> closed versions: +Python 3.10, Python 3.11, Python 3.9 -Python 3.7, Python 3.8 ___ Python tracker

[issue42369] Reading ZipFile not thread-safe

miss-islington added the comment: New changeset 4aa8b802513340d12a6ffea3d5e2228ac6c7d5b8 by Miss Islington (bot) in branch '3.9': bpo-42369: Fix thread safety of zipfile._SharedFile.tell (GH-26974) https://github.com/python/cpython/commit/4aa8b802513340d12a6ffea3d5e2228ac6c7d5b8 --

[issue42369] Reading ZipFile not thread-safe

miss-islington added the comment: New changeset 4352ca234e979ad1c7158981addf899b119cd448 by Miss Islington (bot) in branch '3.10': bpo-42369: Fix thread safety of zipfile._SharedFile.tell (GH-26974) https://github.com/python/cpython/commit/4352ca234e979ad1c7158981addf899b119cd448

[issue42369] Reading ZipFile not thread-safe

Change by miss-islington : -- pull_requests: +30097 pull_request: https://github.com/python/cpython/pull/32009 ___ Python tracker ___

[issue42369] Reading ZipFile not thread-safe

Change by miss-islington : -- nosy: +miss-islington nosy_count: 9.0 -> 10.0 pull_requests: +30096 pull_request: https://github.com/python/cpython/pull/32008 ___ Python tracker

[issue42369] Reading ZipFile not thread-safe

Serhiy Storchaka added the comment: New changeset e730ae7effe4f13b24f1b5fb1fca005709c86acb by Kevin Mehall in branch 'main': bpo-42369: Fix thread safety of zipfile._SharedFile.tell (GH-26974) https://github.com/python/cpython/commit/e730ae7effe4f13b24f1b5fb1fca005709c86acb --

[issue42369] Reading ZipFile not thread-safe

Matthew Davis added the comment: In addition to fixing any unexpected behavior, can we update the documentation [1] to state what the expected behavior is in terms of thread safety? [1] https://docs.python.org/3/library/zipfile.html -- nosy: +mdavis-xyz

[issue42369] Reading ZipFile not thread-safe

Thomas added the comment: @khaledk I finally got some time off, so here you go https://github.com/1/ParallelZipFile I can not offer any support for a more correct implementation of the zip specification due to time constraints, but maybe the code is useful for you anyway. --

[issue42369] Reading ZipFile not thread-safe

Change by Spencer Brown : -- nosy: +Spencer Brown ___ Python tracker ___ ___ Python-bugs-list mailing list Unsubscribe:

[issue42369] Reading ZipFile not thread-safe

Khaled K added the comment: Hi Thomas, I'm facing the same issue. Would you care to opensource your implementation. -- nosy: +khaledkoutini ___ Python tracker ___

[issue42369] Reading ZipFile not thread-safe

Thomas added the comment: The monkey patch works for me! Thank you very much! (I have only tested reading, not writing). However, the lock contention of Python's ZipFile is so bad that using multiple threads actually makes the code run _slower_ than single threaded code when reading a zip

[issue42369] Reading ZipFile not thread-safe

Kevin Mehall added the comment: I think I found the root cause of this problem and proposed a fix in https://github.com/python/cpython/pull/26974 To monkey-patch this fix on existing versions of Python, I'm using: class PatchedSharedFile(zipfile._SharedFile): def __init__(self, *args):

[issue42369] Reading ZipFile not thread-safe

Change by Kevin Mehall : -- keywords: +patch nosy: +kevinmehall nosy_count: 5.0 -> 6.0 pull_requests: +25538 stage: -> patch review pull_request: https://github.com/python/cpython/pull/26974 ___ Python tracker

[issue42369] Reading ZipFile not thread-safe

Change by Jen Garcia : -- nosy: +cuibonobo ___ Python tracker ___ ___ Python-bugs-list mailing list Unsubscribe:

[issue42369] Reading ZipFile not thread-safe

Change by Serhiy Storchaka : -- nosy: +serhiy.storchaka ___ Python tracker ___ ___ Python-bugs-list mailing list Unsubscribe:

[issue42369] Reading ZipFile not thread-safe

Thomas added the comment: I have simplified the test case a bit more: import multiprocessing.pool, zipfile # Create a ZipFile with two files and same content with zipfile.ZipFile("test.zip", "w", zipfile.ZIP_STORED) as z: z.writestr("file1", b"0"*1) z.writestr("file2",

[issue42369] Reading ZipFile not thread-safe

Thomas added the comment: Scratch what I said in the previous message. I thought that the lock was created in _SharedFile and did not notice that it was passed as a parameter. -- ___ Python tracker

[issue42369] Reading ZipFile not thread-safe

Thomas added the comment: I have not observed any segfaults yet. Only zipfile.BadZipFile exceptions so far. The exact file at which it crashes is fairly random. It even crashes if all threads try to read the same file multiple times. I think the root cause of the problem is that the reads

[issue42369] Reading ZipFile not thread-safe

Eric V. Smith added the comment: I'm changing from "crash" to "behavior". We use "crash" for a segfault or equivalent. I realize that most people are unlikely to know this, but we consider "crash" to be more alarming, so I want to make sure it's correct. Also: when this happens, is it

[issue42369] Reading ZipFile not thread-safe

Change by Ma Lin : -- nosy: +malin ___ Python tracker ___ ___ Python-bugs-list mailing list Unsubscribe:

[issue42369] Reading ZipFile not thread-safe

Change by Thomas : -- components: +Library (Lib) type: -> crash ___ Python tracker ___ ___ Python-bugs-list mailing list

[issue42369] Reading ZipFile not thread-safe

New submission from Thomas : According to https://docs.python.org/3.5/whatsnew/changelog.html#id108 bpo-14099, reading multiple ZipExtFiles should be thread-safe, but it is not. I created a small example where two threads try to read files from the same ZipFile simultaneously, which crashes