[issue5411] Add xz support to shutil
Akira Li added the comment: sphinx generates warning for the current docs introduced by this issue: WARNING: Explicit markup ends without a blank line; unexpected unindent. I've uploaded a documentation patch that fixes it. -- nosy: +akira Added file: http://bugs.python.org/file36342/docs-fix-sphinx-warning.patch ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue5411 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue5411] Add xz support to shutil
Berker Peksag added the comment: Fixed. Thanks for the patch, Akira. http://hg.python.org/cpython/rev/7fcfb634ccca -- nosy: +berker.peksag ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue5411 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue5411] Add xz support to shutil
Changes by Serhiy Storchaka storch...@gmail.com: -- assignee: - serhiy.storchaka dependencies: -Misc fixes and cleanups in archiving code in shutil and test_shutil ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue5411 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue5411] Add xz support to shutil
Roundup Robot added the comment: New changeset e57db221b6c4 by Serhiy Storchaka in branch 'default': Issue #5411: Added support for the xztar format in the shutil module. http://hg.python.org/cpython/rev/e57db221b6c4 -- nosy: +python-dev ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue5411 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue5411] Add xz support to shutil
Roundup Robot added the comment: New changeset a47996c10579 by Serhiy Storchaka in branch 'default': Issue #5411: Fixed version number. http://hg.python.org/cpython/rev/a47996c10579 -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue5411 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue5411] Add xz support to shutil
Serhiy Storchaka added the comment: Oh, my fault. Thank you Antoine! -- resolution: - fixed stage: patch review - resolved status: open - closed ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue5411 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue5411] Add xz support to shutil
Antoine Pitrou added the comment: Thank you Serhiy! You need to fix the version number in the versionchanged. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue5411 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue5411] Add xz support to shutil
Antoine Pitrou added the comment: I think we should hurry on to be in time for 3.5. Your patch looks ok to me. Just update the version number in the docs ;) -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue5411 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue5411] Add xz support to shutil
Serhiy Storchaka added the comment: Éric? -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue5411 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue5411] Add xz support to shutil
Éric Araujo added the comment: I’m afraid I changed computers once or twice since I worked on that, so I can’t readily find my clone and get the latest patch. I know where the hard drives are but I can’t say when I will have time to search them. -- assignee: eric.araujo - ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue5411 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue5411] Add xz support to shutil
Serhiy Storchaka added the comment: Now 3.4 is released, but shutil still doesn't support the xz compression. I think we should hurry on to be in time for 3.5. -- versions: +Python 3.5 -Python 3.4 ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue5411 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue5411] Add xz support to shutil
Changes by Serhiy Storchaka storch...@gmail.com: Removed file: http://bugs.python.org/file27723/shutil-lzma_2.patch ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue5411 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue5411] Add xz support to shutil
Serhiy Storchaka added the comment: Serhiy's patch needs a versionchanged or versionadded tag in the Docs. Done. -- Added file: http://bugs.python.org/file32163/shutil-lzma_3.patch ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue5411 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue5411] Add xz support to shutil
Changes by Arfrever Frehtes Taifersar Arahesis arfrever@gmail.com: -- nosy: +Arfrever ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue5411 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue5411] Add xz support to shutil
Serhiy Storchaka added the comment: 3.4 beta 1 will be soon. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue5411 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue5411] Add xz support to shutil
Antoine Pitrou added the comment: Serhiy's patch needs a versionchanged or versionadded tag in the Docs. -- nosy: +pitrou ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue5411 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue5411] Add xz support to shutil
Éric Araujo added the comment: I will upload my patch and compare it with Serhiy’s. Now that 3.3.0 is released, there is no hurry. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue5411 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue5411] Add xz support to shutil
Hynek Schlawack added the comment: Éric, what’s your take on this approach (not code)? We have time enough till 3.4 but it seems this doesn't really move forward. Any thoughts how to get this moving? Unfortunately I'm not invested enough in this to make a educated decision. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue5411 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue5411] Add xz support to shutil
Serhiy Storchaka added the comment: I have not seen this issue and was created a new issue16313 with almost same patch as Eric's one (only I have changed the documentation too). Here's the patch. I wonder that it was not committed yet to 3.3. I think it would be better first to add xz support and the code cleanup to do then (if it takes so much time). -- nosy: +serhiy.storchaka versions: +Python 3.4 -Python 3.3 Added file: http://bugs.python.org/file27722/shutil-lzma.patch ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue5411 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue5411] Add xz support to shutil
Changes by Serhiy Storchaka storch...@gmail.com: -- stage: needs patch - patch review ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue5411 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue5411] Add xz support to shutil
Changes by Serhiy Storchaka storch...@gmail.com: Removed file: http://bugs.python.org/file27722/shutil-lzma.patch ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue5411 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue5411] Add xz support to shutil
Serhiy Storchaka added the comment: Oh, I see the '.bz2' bug. Patch updated. -- Added file: http://bugs.python.org/file27723/shutil-lzma_2.patch ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue5411 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue5411] Add xz support to shutil
Nadeem Vawda nadeem.va...@gmail.com added the comment:
So, do you agree that “not automated but not ugly” is better than “automated
with ugly klutches”?
Definitely. If we have to add special cases that are almost as long as
the original code, the automation seems pointless.
Note that there is a way to get fully automated support for tar formats:
tarfile could expose, in addition to the list compression_formats, another
structure with the descriptions (e.g. “gzip’ed tar file”) and file extensions
(e.g. ['.gz', '.tgz'] —no, it’s not '.tar.gz', which is unfortunate, and
could cause Lars to reject that idea). I’m just putting this here for
reference, but my preference is still for the second idea I talk about in my
precedent message.
As much as it would be nice to have all of the information centralized in
one place, I don't think this is practical - some of the data that would
be stored in this structure really is specific to shutil, so I agree that
your second option is better.
I think we can restructure the code a bit to reduce the work involved in
adding a new format, though. Maybe something like this:
_ARCHIVE_FORMATS = {}
_UNPACK_FORMATS = {}
for name, tarname, progname, extensions, desc in [
(xz, xztar, xz, [.xz], xz'ed tar file),
(bz2, bztar, bzip2, [.bz2], bzip2'ed tar file),
(gz, gztar, gzip, [.gz, .tgz], gzip'ed tar file)]:
if name in tarfile.compression_formats:
_ARCHIVE_FORMATS[tarname] = _make_tarball, [(compress,
progname)], desc
_UNPACK_FORMATS[tarname] = extensions, _unpack_tarfile, [], desc
By the way, is there any reason why you've used .gz instead of
.tar.gz as the extension for gztar in your proposed change? The
current version of shutil.py uses .tar.gz, but has .bz2 for bztar.
Also, I notice that _make_tarball() will need to be updated to add the
info for xz to the tar_compression and compress_ext dictionaries. It
seems a bit ugly to duplicate this information when it's already present
in the format dicts, so maybe it would be better to pass the file
extension in directly instead? I assume that _make_tarball() is not part
of shutil's public API.
If we did this, there would be no need for a separate progname field
in the initialization code I suggested above:
for name, tarname, extensions, desc in [
(xz, xztar, [.xz], xz'ed tar file),
(bz2, bztar, [.bz2], bzip2'ed tar file),
(gz, gztar, [.gz, .tgz], gzip'ed tar file)]:
if name in tarfile.compression_formats:
_ARCHIVE_FORMATS[tarname] = _make_tarball, [(compress, name)],
desc
_UNPACK_FORMATS[tarname] = extensions, _unpack_tarfile, [], desc
--
___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue5411
___
___
Python-bugs-list mailing list
Unsubscribe:
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue5411] Add xz support to shutil
Nadeem Vawda nadeem.va...@gmail.com added the comment: For the xztar format, you should also perhaps recognize the .txz extension - I've seen this used by FreeBSD. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue5411 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue5411] Add xz support to shutil
Éric Araujo mer...@netwok.org added the comment: I think we can restructure the code a bit to reduce the work involved in adding a new format, though. Maybe something like this: [snip] Thanks, I’ve applied your obvious refactor in my clone :) For clarity, I will split my patch into many patches for different reports (bug fixes and doc improvements for 3.2, then code cleanup in 3.3, then adding xz). By the way, is there any reason why you've used .gz instead of .tar.gz as the extension for gztar in your proposed change? The current version of shutil.py uses .tar.gz, but has .bz2 for bztar. Yes, I confused the two formats. I reported the '.bz2' bug (should be '.tar.bz2') and will fix it. Also, I notice that _make_tarball() will need to be updated to add the info for xz to the tar_compression and compress_ext dictionaries. I ripped them off. I could have pasted my full patch to save you time, sorry! For the xztar format, you should also perhaps recognize the .txz extension Done. Google finds a number of matches. I won’t add '.tbz' though, as I don’t know if it’s bzip or bzip2, and as it does not seem used much. -- dependencies: -tarfile should expose supported formats ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue5411 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue5411] Add xz support to shutil
Changes by Antoine Pitrou pit...@free.fr: -- nosy: +hynek ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue5411 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue5411] Add xz support to shutil
Éric Araujo mer...@netwok.org added the comment: s/cleanup in did in the code/cleanup I did in the code/ -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue5411 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue5411] Add xz support to shutil
Éric Araujo mer...@netwok.org added the comment:
I have a working updated shutil module, tests pass and the documentation is
improved. I will make sure to make different commits for improving the tests,
cleaning up some things, adding tarfile.compression_formats and removing
duplication in shutil (see dep bugs), to be sure not to introduce regressions.
Before I finish and post the patch, I’d like feedback on a choice I made. I
don’t think it’s possible to have shutil automatically support all the formats
that tarfile does, because of the spelling issue I mentioned. Here’s what the
code would look like (let me know if I should post it elsewhere or as a file to
let you get syntax coloring):
_ARCHIVE_FORMATS = {}
_UNPACK_FORMATS = {}
for fmt in tarfile.compression_formats:
code = fmt + 'tar'
ext = '.' + fmt
desc = %s'ed tar file % fmt
_ARCHIVE_FORMATS[code] = (_make_tarball, [('compress', fmt)], desc)
_UNPACK_FORMATS[code] = ([ext], _unpack_tarfile, [], desc)
# kludge to add alternate extension
if 'gztar' in _ARCHIVE_FORMATS:
_UNPACK_FORMATS['gztar'][0].append('.tgz')
# XXX desc should say gzip'ed tar file, not gz'ed
# rectify naming incompatibility
if 'bz2tar' in _ARCHIVE_FORMATS:
# XXX alternative: make 'bztar' alias for 'bz2tar', but would complicate
# manipulating the registries
del _ARCHIVE_FORMATS['bz2tar']
del _UNPACK_FORMATS['bz2tar']
desc = bzip2'ed tar file
_ARCHIVE_FORMATS['bztar'] = (_make_tarball, [('compress', 'bz2')], desc)
_UNPACK_FORMATS['bztar'] = (['.bz2'], _unpack_tarfile, [], desc)
# now add uncompressed tar and zip file
I really don’t like that code. Given that tarfile is not extensible at run
time, it is not a big deal to have to update shutil whenever we add a
compression format to tarfile. Therefore, I backtracked on my “automatic
support” idea but kept a lot of cleanup in did in the code. Here’s what the
code looks like:
_ARCHIVE_FORMATS = {}
_UNPACK_FORMATS = {}
if 'xz' in tarfile.compression_formats:
desc = xz'ed tar file
# XXX '.xz' is not great, '.tar.xz' would be better
_ARCHIVE_FORMATS['xztar'] = (_make_tarball, [('compress', 'xz')], desc)
_UNPACK_FORMATS['xztar'] = (['.xz'], _unpack_tarfile, [], desc)
if 'bz2' in tarfile.compression_formats:
desc = bzip2'ed tar file
_ARCHIVE_FORMATS['bztar'] = (_make_tarball, [('compress', 'bz2')], desc)
_UNPACK_FORMATS['bztar'] = (['.bz2'], _unpack_tarfile, [], desc)
if 'gz' in tarfile.compression_formats:
desc = gzip'ed tar file
_ARCHIVE_FORMATS['gztar'] = (_make_tarball, [('compress', 'gz')], desc)
_UNPACK_FORMATS['gztar'] = (['.gz', '.tgz'], _unpack_tarfile, [], desc)
So, do you agree that “not automated but not ugly” is better than “automated
with ugly klutches”?
--
title: shutil should support all formats supported by tarfile automatically -
Add xz support to shutil
___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue5411
___
___
Python-bugs-list mailing list
Unsubscribe:
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue5411] Add xz support to shutil
Éric Araujo mer...@netwok.org added the comment: Note that there is a way to get fully automated support for tar formats: tarfile could expose, in addition to the list compression_formats, another structure with the descriptions (e.g. “gzip’ed tar file”) and file extensions (e.g. ['.gz', '.tgz'] —no, it’s not '.tar.gz', which is unfortunate, and could cause Lars to reject that idea). I’m just putting this here for reference, but my preference is still for the second idea I talk about in my precedent message. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue5411 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
