[issue30693] tarfile add uses random order

Change by Roundup Robot : -- nosy: +python-dev nosy_count: 8.0 -> 9.0 pull_requests: +29831 pull_request: https://github.com/python/cpython/pull/31713 ___ Python tracker ___

[issue30693] tarfile add uses random order

Change by Serhiy Storchaka : -- resolution: -> fixed stage: patch review -> resolved status: open -> closed ___ Python tracker

[issue30693] tarfile add uses random order

Serhiy Storchaka added the comment: New changeset 2c6f6682768f401c297c584ef106d48c78697f67 by Serhiy Storchaka (Miss Islington (bot)) in branch '3.7': bpo-30693: Fix tarfile test cleanup on MSWindows (GH-5557) (GH-5567)

[issue30693] tarfile add uses random order

Change by miss-islington : -- pull_requests: +5388 ___ Python tracker ___

[issue30693] tarfile add uses random order

Serhiy Storchaka added the comment: New changeset 4ad703b7ca463d1183539277dde90ffb1c808487 by Serhiy Storchaka (Bernhard M. Wiedemann) in branch 'master': bpo-30693: Fix tarfile test cleanup on MSWindows (#5557)

[issue30693] tarfile add uses random order

Bernhard M. Wiedemann added the comment: Serhiy, can you test https://github.com/python/cpython/pull/5557 -- ___ Python tracker

[issue30693] tarfile add uses random order

Change by Bernhard M. Wiedemann : -- keywords: +patch pull_requests: +5379 stage: needs patch -> patch review ___ Python tracker

[issue30693] tarfile add uses random order

Serhiy Storchaka added the comment: Tests are failing on Windows. == ERROR: test_ordered_recursion (test.test_tarfile.Bz2WriteTest)

[issue30693] tarfile add uses random order

Ned Deily added the comment: New changeset 57750be4ad3fa2cfd3473b5be1f1e1a5d0fa9f50 by Ned Deily (Bernhard M. Wiedemann) in branch '3.7': bpo-30693: zip+tarfile: sort directory listing (#2263) https://github.com/python/cpython/commit/57750be4ad3fa2cfd3473b5be1f1e1a5d0fa9f50

[issue30693] tarfile add uses random order

Bernhard M. Wiedemann added the comment: @Serhiy IMHO, just because we fix one problem, we do not have to fix all other problems at the same time. You can still open a pull-request for the others, but I know too little about those to test them. And having commits pending

[issue30693] tarfile add uses random order

Serhiy Storchaka added the comment: I requested additional changes in msg310337. -- ___ Python tracker ___

[issue30693] tarfile add uses random order

STINNER Victor added the comment: > We missed beta freeze deadline. :/ I merged the PR. We will have to create a cherry-pick request once the 3.7 branch will be created. If Ned rejects it, we have to change the version number of documentation.

[issue30693] tarfile add uses random order

STINNER Victor added the comment: New changeset 84521047e413d7d1150aaa1c333580b683b3f4b1 by Victor Stinner (Bernhard M. Wiedemann) in branch 'master': bpo-30693: zip+tarfile: sort directory listing (#2263)

[issue30693] tarfile add uses random order

Christian Heimes added the comment: We missed beta freeze deadline. :/ Ned, can we get this change into beta 2? It's low risk change to make the tarballs and other archives have a stable sort order. We even considered to backport the change to 3.6 and 2.7. --

[issue30693] tarfile add uses random order

STINNER Victor added the comment: I now agree to leave Python 2.7 and 3.6 unchanged. -- ___ Python tracker ___

[issue30693] tarfile add uses random order

Serhiy Storchaka added the comment: If make this change you need to make similar changes in other places that recursively add files to archives: shutil, zipapp, distutils, and maybe more. -- versions: -Python 2.7, Python 3.6

[issue30693] tarfile add uses random order

Ned Deily added the comment: This doesn't seem appropriate to me for backporting to existing releases (3.6. and 2.7). AFAIK, the current file-system-order behavior has never been identified as a bug. Unless there is a stronger case for changing the existing 3.6.x behavior,

[issue30693] tarfile add uses random order

Christian Heimes added the comment: The patch changes behavior. It's fine for 3.7 but not for 3.6/2.7. Somebody may depend on filesystem order. -- versions: -Python 3.5 ___ Python tracker

[issue30693] tarfile add uses random order

STINNER Victor added the comment: The only warranty in that TarFile.getmembers(), TarFile.getnames() and ZipFile.infolist() returns members/names "in the same order as the members in the archive". Currently, there is no warranty when packing, only on unpack.

[issue30693] tarfile add uses random order

R. David Murray added the comment: Ah, I was just going to ask about that. I guess I'm -0 on the backport as well. The other reproducible build stuff is only going to land in 3.7. However, this is in a more general category than the pyc stuff, so I can see the

[issue30693] tarfile add uses random order

STINNER Victor added the comment: Since we currently don't warranty *anything* about ordering, I like the idea of *fixing* Python 2.7 and 3.6 as well. +1 for fix it in 2.7, 3.6 and master. -- nosy: +vstinner ___ Python

[issue30693] tarfile add uses random order

Christian Heimes added the comment: PS: I'm -0 to backport the change to 3.6 and 2.7. 3.5 is in security fix mode and therefore completely out of scope. -- versions: -Python 3.5 ___ Python tracker

[issue30693] tarfile add uses random order

Christian Heimes added the comment: +1 from me In my opinion it's both a good idea to not sort the result of glob.glob() and make the order of tar and zip module content ordered. The glob module is low level and it makes sense to expose the file system sort order. On the

[issue30693] tarfile add uses random order

R. David Murray added the comment: Given the reproducible builds angle, I'd say this was worth doing. -- nosy: +r.david.murray ___ Python tracker

[issue30693] tarfile add uses random order

Bernhard M. Wiedemann added the comment: note: recent GNU tar versions (1.28?) added an option --sort=name and the overhead of sorting (e.g. I measured 4ms for 1 files) is negligible compared to the other processing done on the files here. --

[issue30693] tarfile add uses random order

Serhiy Storchaka added the comment: The patch for similar issue with the glob module was rejected recently since it is easy to sort the result of glob.glob() (see issue30461). This issue looks similar, but there are differences. On one side, the command line tar utility doesn't have the

[issue30693] tarfile add uses random order

Changes by Bernhard M. Wiedemann : -- pull_requests: +2313 ___ Python tracker ___ ___

[issue30693] tarfile add uses random order

New submission from Bernhard M. Wiedemann: Filesystems do not give any guarantees about ordering of files returned in directory listings, thus tarfile.add adds files in random order, when using os.listdir in recursion. See also https://reproducible-builds.org/docs/stable-inputs/ on that