[issue6715] xz compressor support
Peter added the comment: Apologies again for the noise, but I've just made the first public release of the lzma backport at http://pypi.python.org/pypi/backports.lzma/ with the repository as mentioned before at https://github.com/peterjc/backports.lzma I have tested this on Python 2.6, 2.7 and 3.0 through 3.3 under Linux and Mac OS. I've not tried building this on Windows yet, but it should be possible and I will be reviewing Amaury's notes on this thread. My thanks to Nadeem and Per for looking over this with helpful feedback, and agreeing to the use of the 3-clause BSD license for the backport. If anyone has any further questions about the backport, please get in touch via GitHub or email me. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue6715 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue6715] xz compressor support
Peter added the comment: Apologies for noise, but since a backport was discussed, I'm mentioning this here. I've started implementing a backport, currently working and tested on Mac OS X and Linux, back to Python 3.0 - supporting Python 2 would be nice but probably significantly more work (assistance welcome - please get in touch on github): https://github.com/peterjc/backports.lzma Assuming Nadeem has no objections, I intend to publish this on PyPI (I have tried to email directly but perhaps I'm using an old email address or he has been busy, another reason for commenting here). Thanks! -- nosy: +maubp ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue6715 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue6715] xz compressor support
Nadeem Vawda nadeem.va...@gmail.com added the comment: I had intended to wait until the code had been tested on all of the buildbots before closing it. However, it's taking a while to get xz-utils installed on the last few bots, so it doesn't make sense to keep the issue open for this. -- resolution: - fixed stage: patch review - committed/rejected status: open - closed ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue6715 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue6715] xz compressor support
STINNER Victor victor.stin...@gmail.com added the comment: Why is this issue still open? -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue6715 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue6715] xz compressor support
Roundup Robot devn...@psf.upfronthosting.co.za added the comment: New changeset 7c22281e967c by Éric Araujo in branch '2.7': Improve interlinking of archiving/compression modules docs. http://hg.python.org/cpython/rev/7c22281e967c -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue6715 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue6715] xz compressor support
Éric Araujo mer...@netwok.org added the comment: BTW, any plans on a PyPI backport for fun and profit? -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue6715 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue6715] xz compressor support
Nadeem Vawda nadeem.va...@gmail.com added the comment: BTW, any plans on a PyPI backport for fun and profit? At present, no. I'll look into it later in the year, but at the moment I don't have the time to work on it - I suspect the parts written in C will require substantial changes to work under 2.x. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue6715 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue6715] xz compressor support
Roundup Robot devn...@psf.upfronthosting.co.za added the comment: New changeset 11bd2d32b4e8 by Éric Araujo in branch '3.2': Improve interlinking of archiving/compression modules docs. http://hg.python.org/cpython/rev/11bd2d32b4e8 New changeset 1cb9b8126534 by Éric Araujo in branch 'default': Merge edits from 3.2 (#13716, #1040439, #2945, #13770, #6715) http://hg.python.org/cpython/rev/1cb9b8126534 -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue6715 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue6715] xz compressor support
Antoine Pitrou pit...@free.fr added the comment: Is there any reason why this issue is still open? -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue6715 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue6715] xz compressor support
Nadeem Vawda nadeem.va...@gmail.com added the comment: Yes, almost half of the buildbots still don't have the xz-utils headers installed, and thus are not building/testing the lzma module. I've kept the issue open as a reminder to myself to follow up on this. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue6715 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue6715] xz compressor support
Éric Araujo mer...@netwok.org added the comment: I’ll commit my doc patch to all branches later. I checked the 4 red 3.3 builbots and they can’t build _lzma. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue6715 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue6715] xz compressor support
Nadeem Vawda nadeem.va...@gmail.com added the comment: I’ll commit my doc patch to all branches later. OK, great. I checked the 4 red 3.3 builbots and they can’t build _lzma. For the record, the 3.x buildbots that currently aren't able to build the _lzma module are: * x86 Ubuntu Shared * sparc solaris10 gcc * AMD64 Snow Leopard 2 * x86 debian parallel * ARM Ubuntu * x86 Tiger * AMD64 OpenIndiana * x86 OpenIndiana * x86 FreeBSD 7.2 * x86 FreeBSD 6.4 Additionally, the AMD64 debian bigmem, PPC Tiger and PPC Leopard builders have been offline/broken since before changeset 74d182cf0187 was committed. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue6715 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue6715] xz compressor support
Antoine Pitrou pit...@free.fr added the comment: Le jeudi 08 décembre 2011 à 17:40 +, Nadeem Vawda a écrit : Nadeem Vawda nadeem.va...@gmail.com added the comment: I’ll commit my doc patch to all branches later. OK, great. I checked the 4 red 3.3 builbots and they can’t build _lzma. For the record, the 3.x buildbots that currently aren't able to build the _lzma module are: Ok, I've e-mailed the owners. Will someone write an entry in the what's new file for 3.3? (Doc/whatsnew/3.3.rst) Regards Antoine. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue6715 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue6715] xz compressor support
Roundup Robot devn...@psf.upfronthosting.co.za added the comment: New changeset 66df5ace0eee by Nadeem Vawda in branch 'default': What's New in Python 3.3: Add entry for lzma module (issue #6715). http://hg.python.org/cpython/rev/66df5ace0eee -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue6715 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue6715] xz compressor support
Nadeem Vawda nadeem.va...@gmail.com added the comment: Ok, I've e-mailed the owners. Thanks. I was just thinking I should send a reminder, in case they missed the note in my announcement on python-dev. Will someone write an entry in the what's new file for 3.3? Done -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue6715 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue6715] xz compressor support
Ned Deily n...@acm.org added the comment: I've opened Issue13507 to track adding liblzma to the OS X installer builds. -- nosy: +ned.deily ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue6715 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue6715] xz compressor support
Per Øyvind Karlsen peroyv...@mandriva.org added the comment: Ah, I thought that he had reused most of the original C code in _lzmamodule.c not replaced by python code, but I see that not being the case now (only slight fragments;). Oh well, I thought that I'd still earned a note with some slight credit at least, but I guess I won't go postal or anything in lack of either.. :p -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue6715 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue6715] xz compressor support
Amaury Forgeot d'Arc amaur...@gmail.com added the comment: Oh well, I thought that I'd still earned a note with some slight credit at least I completely agree. Sometimes people get credit for simple bug fixes (count me among them) so the author of the first working implementation deserves some recognition IMO. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue6715 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue6715] xz compressor support
Éric Araujo mer...@netwok.org added the comment: Nadeem: Instead of duplicating the list of archiving/compression modules in each doc, what about only linking to the shutil doc for archives and the archiving.rst file? (I can make the patch, just wanted feedback first) -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue6715 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue6715] xz compressor support
Nadeem Vawda nadeem.va...@gmail.com added the comment: Not meaning to sound petty, but wouldn't it be common etiquette to retain some original copyright notice from original code intact..? It seemed to me that Nadeem had rewritten everything from scratch. Is there any code of yours in the current module? That is correct. Based on my experience with the bz2 module, rewriting from scratch seemed to make more sense. Oh well, I thought that I'd still earned a note with some slight credit at least I completely agree. Sometimes people get credit for simple bug fixes (count me among them) so the author of the first working implementation deserves some recognition IMO. Of course. How does this look? diff --git a/Misc/ACKS b/Misc/ACKS --- a/Misc/ACKS +++ b/Misc/ACKS @@ -502,6 +502,7 @@ Peter van Kampen Rafe Kaplan Jacob Kaplan-Moss +Per Øyvind Karlsen Lou Kates Hiroaki Kawai Sebastien Keim diff --git a/Misc/NEWS b/Misc/NEWS --- a/Misc/NEWS +++ b/Misc/NEWS @@ -400,6 +400,7 @@ --- - Issue #6715: Add a module 'lzma' for compression using the LZMA algorithm. + Thanks to Per Øyvind Karlsen for the initial implementation. - Issue #13487: Make inspect.getmodule robust against changes done to sys.modules while it is iterating over it. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue6715 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue6715] xz compressor support
Nadeem Vawda nadeem.va...@gmail.com added the comment: Instead of duplicating the list of archiving/compression modules in each doc, what about only linking to the shutil doc for archives and the archiving.rst file? Sure, go ahead. I actually hadn't realized that each section of the library docs had a sub-index page like that until now... -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue6715 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue6715] xz compressor support
Éric Araujo mer...@netwok.org added the comment: --- a/Misc/NEWS +++ b/Misc/NEWS @@ -400,6 +400,7 @@ --- - Issue #6715: Add a module 'lzma' for compression using the LZMA algorithm. + Thanks to Per Øyvind Karlsen for the initial implementation. The entry in Misc/ACKS, Doc/whatsnew/3.3.rst and the commit message should be enough. Lately I’ve noticed some attributions in NEWS, but it’s usually not done (as redundant). -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue6715 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue6715] xz compressor support
Amaury Forgeot d'Arc amaur...@gmail.com added the comment: what about a mention in lzmamodule.c? -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue6715 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue6715] xz compressor support
Éric Araujo mer...@netwok.org added the comment: I actually hadn't realized that each section of the library docs had a sub-index page like that That’s multiple-entry navigation :) You can jump to a module page, or use the index, or use the search, or browse and see the sub-indexes. As the docs for zlib, gzip, bz2, lzma, zipfile and tarfile are in the archiving subsection, there’s already a link to the subsection index, so I just removed the “See also zlib, etc.” lines (except for the link from zlib to gzip). I added a link from archiving.rst to shutil and more links and reST targets in shutil. -- Added file: http://bugs.python.org/file23820/archiving-modules-links.diff ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue6715 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue6715] xz compressor support
Antoine Pitrou pit...@free.fr added the comment: The entry in Misc/ACKS, Doc/whatsnew/3.3.rst and the commit message should be enough. Lately I’ve noticed some attributions in NEWS, but it’s usually not done (as redundant). I generally add attributions in NEWS since that's where most people learn about changes (for changes which aren't in whatsnew). -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue6715 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue6715] xz compressor support
Roundup Robot devn...@psf.upfronthosting.co.za added the comment: New changeset 6cde416ef03b by Nadeem Vawda in branch 'default': Credit Per Øyvind Karlsen for the initial implementation of the lzma module (issue #6715). http://hg.python.org/cpython/rev/6cde416ef03b -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue6715 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue6715] xz compressor support
Nadeem Vawda nadeem.va...@gmail.com added the comment: As the docs for zlib, gzip, bz2, lzma, zipfile and tarfile are in the archiving subsection, there’s already a link to the subsection index, so I just removed the “See also zlib, etc.” lines (except for the link from zlib to gzip). I added a link from archiving.rst to shutil and more links and reST targets in shutil. Looks good to me. The entry in Misc/ACKS, Doc/whatsnew/3.3.rst and the commit message should be enough. Lately I’ve noticed some attributions in NEWS, but it’s usually not done (as redundant). According to Doc/whatsnew/3.3.rst: The maintainer will go through Misc/NEWS periodically and add changes; it's therefore more important to add your changes to Misc/NEWS than to this file. what about a mention in lzmamodule.c? Done and committed. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue6715 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue6715] xz compressor support
Roundup Robot devn...@psf.upfronthosting.co.za added the comment: New changeset 74d182cf0187 by Nadeem Vawda in branch 'default': Issue #6715: Add module for compression using the LZMA algorithm. http://hg.python.org/cpython/rev/74d182cf0187 -- nosy: +python-dev ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue6715 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue6715] xz compressor support
Nadeem Vawda nadeem.va...@gmail.com added the comment: The patch has been committed. I'm leaving this issue open for a while longer, until more of the buildbots are building and testing the module. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue6715 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue6715] xz compressor support
Per Øyvind Karlsen peroyv...@mandriva.org added the comment: Not meaning to sound petty, but wouldn't it be common etiquette to retain some original copyright notice from original code intact..? -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue6715 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue6715] xz compressor support
Antoine Pitrou pit...@free.fr added the comment: Not meaning to sound petty, but wouldn't it be common etiquette to retain some original copyright notice from original code intact..? It seemed to me that Nadeem had rewritten everything from scratch. Is there any code of yours in the current module? -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue6715 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue6715] xz compressor support
Nadeem Vawda nadeem.va...@gmail.com added the comment: Victor: Thanks for the review; I've replied to your comments and updated the patch. Let me know what you think of the changes. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue6715 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue6715] xz compressor support
STINNER Victor victor.stin...@haypocalc.com added the comment: The last patch (9276fc685c05.diff) looks good to me. Go ahead for the commit! -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue6715 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue6715] xz compressor support
Changes by Nadeem Vawda nadeem.va...@gmail.com: Added file: http://bugs.python.org/file23732/9276fc685c05.diff ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue6715 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue6715] xz compressor support
Changes by Nadeem Vawda nadeem.va...@gmail.com: Removed file: http://bugs.python.org/file23466/371a133b770a.diff ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue6715 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue6715] xz compressor support
Antoine Pitrou pit...@free.fr added the comment: What is the status on this, Nadeem? It would be lovely to get the feature in the stdlib. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue6715 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue6715] xz compressor support
Amaury Forgeot d'Arc amaur...@gmail.com added the comment: I just added the missing files liblzma.a to the externals repository. If someone can quickly check that it works on win32, I don't have anything else to add to this change. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue6715 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue6715] xz compressor support
Nadeem Vawda nadeem.va...@gmail.com added the comment: Everything is ready to go, as far as I'm concerned. Once I've tested the Windows build using the binary from the externals repository, I'll be ready to commit it to the main repository. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue6715 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue6715] xz compressor support
Nadeem Vawda nadeem.va...@gmail.com added the comment: I've tested the Windows build, and it works fine. If there are no objections, I'll commit on Saturday. Btw, what is the preferred channel for communicating with buildbot owners? I'll want to send out an email asking that all the *nix bots have liblzma-dev installed, so that they can actually run the tests... -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue6715 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue6715] xz compressor support
Antoine Pitrou pit...@free.fr added the comment: I've tested under 64-bit Windows and it worked fine. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue6715 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue6715] xz compressor support
STINNER Victor victor.stin...@haypocalc.com added the comment: I wrote a short review on http://bugs.python.org/review/6715/show -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue6715 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue6715] xz compressor support
Antoine Pitrou pit...@free.fr added the comment: Apart from this, is there anything else that needs to be done before this change can be committed to the main repo? Things look fine to me, apart from a couple of minor comments I've made on the review tool. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue6715 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue6715] xz compressor support
Changes by Nadeem Vawda nadeem.va...@gmail.com: Removed file: http://bugs.python.org/file23393/bdf0afbbbd80.diff ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue6715 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue6715] xz compressor support
Changes by Nadeem Vawda nadeem.va...@gmail.com: Added file: http://bugs.python.org/file23466/371a133b770a.diff ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue6715 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue6715] xz compressor support
Nadeem Vawda nadeem.va...@gmail.com added the comment: After disabling a bunch of services and skipping the tests that are failing on the Windows buildbots, it seems that the failures I was getting last week have disappeared. It looks like they were due to the Windows search indexer preventing temp directories from being deleted... _ I have imported xz-5.0.3 into the externals repository now. I tried to test this out, but it looks like liblzma.a is missing from the SVN repo. There should be files with that name in the bin_i486 and bin_x86-64 directories. Apart from this, is there anything else that needs to be done before this change can be committed to the main repo? -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue6715 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue6715] xz compressor support
Martin v. Löwis mar...@v.loewis.de added the comment: I have imported xz-5.0.3 into the externals repository now. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue6715 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue6715] xz compressor support
Martin v. Löwis mar...@v.loewis.de added the comment: Modules/_lzmamodule.c:364: Py_BEGIN_ALLOW_THREADS It seems that the Windows version at least is not thread-safe. If so, you would need an LZMA lock when releasing the GIL. Does the class need to be thread-safe, though? As a matter of principle, Python code must not be able to crash the interpreter or corrupt memory. There are known bugs in this area, but if it's known in advance that an issue exists, we should avoid it. ISTM that there isn't any sensible use case for having two threads feeding data through the same compressor concurrently. Right. So having a per-compressor mutex lock would be entirely reasonable. I could also accept a per-module lock. I could even accept the GIL, and if no other code is forth-coming, I would prefer to keep holding the GIL during comprssion over risking crashes. (If we *do* want thread-safety, then it doesn't matter whether the underlying lib is internally thread-safe or not. We would still need to guard against the possibility of the _lzmamodule.c code in one thread modifying the lzma_stream's input or output pointer while lzma_code is operating on the stream's data in another thread.) I haven't reviewed the module in this respect. If you say that it wouldn't be thread-safe even if LZMA was compiled as thread-safe, then this definitely must be fixed. To elaborate on the policy: giving bogus data in cases of race conditions is ok; crashing the interpreter or corrupting memory is not. If bogus data is given, it would be useful if the bogosity can be specified (e.g. when multiple threads read from the same POSIX file concurrently, they also get bogus data, but in a manner where each input byte is given to exactly one thread). -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue6715 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue6715] xz compressor support
Martin v. Löwis mar...@v.loewis.de added the comment: Correct. I copied the algorithm from _io.FileIO, under the assumption that there was a reason for not using a simpler O(n log n) doubling strategy. Do you know of any reason for this? Or is it safe to ignore it? I don't know, but I'd say it's safe to ignore it. To elaborate: ISTM that it's actually a bug in FileIO. I can imagine where it's coming from (i.e. somebody feeling that overhead shouldn't grow unbounded), but I think that's ill-advised - *if* somebody really produces multi-gigabyte data (and some people eventually will), they still deserve good performance, and they will be able to afford the memory overhead (else they couldn't store the actual output, either). Generally we use a less-than-doubling strategy, to conserve memory (see e.g. bytearray.c). Most definitely. In case it isn't clear (but it probably is here): any constant factor 1.0 will provide amortized linear complexity. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue6715 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue6715] xz compressor support
Nadeem Vawda nadeem.va...@gmail.com added the comment: To elaborate on the policy: giving bogus data in cases of race conditions is ok; crashing the interpreter or corrupting memory is not. If bogus data is given, it would be useful if the bogosity can be specified (e.g. when multiple threads read from the same POSIX file concurrently, they also get bogus data, but in a manner where each input byte is given to exactly one thread). OK, that makes sense. My next patch will include per-compressor locks for LZMACompressor and LZMADecompressor. To elaborate: ISTM that it's actually a bug in FileIO. I've filed issue 13159 proposing that FileIO be fixed. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue6715 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue6715] xz compressor support
Changes by Nadeem Vawda nadeem.va...@gmail.com: Added file: http://bugs.python.org/file23393/bdf0afbbbd80.diff ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue6715 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue6715] xz compressor support
Changes by Nadeem Vawda nadeem.va...@gmail.com: Removed file: http://bugs.python.org/file23168/591277fe6b4a.diff ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue6715 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue6715] xz compressor support
Nadeem Vawda nadeem.va...@gmail.com added the comment: Patch update time! Noteworthy changes: * Windows build support, thanks to Amaury * Thread safety for LZMACompressor and LZMADecompressor, by means of per-instance locks * O(n) buffer growth strategy I've tried running the tests on Windows, but I've been getting failures that don't look like they're related to the new code. I think they are due to issue 7443. I'll try and tweak my Windows setup a bit over the weekend to see if I can get things working smoothly. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue6715 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue6715] xz compressor support
Nadeem Vawda nadeem.va...@gmail.com added the comment: Awesome stuff! I'll post an updated patch during the course of the day. Martin: I've been having problems with Rietveld lately, so I'm posting my replies to your comments here instead. Modules/_lzmamodule.c:115: return _PyBytes_Resize(buf, size + BIGCHUNK); This has quadratic performance. Correct. I copied the algorithm from _io.FileIO, under the assumption that there was a reason for not using a simpler O(n log n) doubling strategy. Do you know of any reason for this? Or is it safe to ignore it? Modules/_lzmamodule.c:364: Py_BEGIN_ALLOW_THREADS It seems that the Windows version at least is not thread-safe. If so, you would need an LZMA lock when releasing the GIL. Does the class need to be thread-safe, though? ISTM that there isn't any sensible use case for having two threads feeding data through the same compressor concurrently. (If we *do* want thread-safety, then it doesn't matter whether the underlying lib is internally thread-safe or not. We would still need to guard against the possibility of the _lzmamodule.c code in one thread modifying the lzma_stream's input or output pointer while lzma_code is operating on the stream's data in another thread.) -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue6715 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue6715] xz compressor support
Antoine Pitrou pit...@free.fr added the comment: Modules/_lzmamodule.c:115: return _PyBytes_Resize(buf, size + BIGCHUNK); This has quadratic performance. Correct. I copied the algorithm from _io.FileIO, under the assumption that there was a reason for not using a simpler O(n log n) doubling strategy. Do you know of any reason for this? Or is it safe to ignore it? I don't know, but I'd say it's safe to ignore it. Maybe open a separate bug about that. Generally we use a less-than-doubling strategy, to conserve memory (see e.g. bytearray.c). (note: it is O(n), not O(n log n)) Modules/_lzmamodule.c:364: Py_BEGIN_ALLOW_THREADS It seems that the Windows version at least is not thread-safe. If so, you would need an LZMA lock when releasing the GIL. Does the class need to be thread-safe, though? ISTM that there isn't any sensible use case for having two threads feeding data through the same compressor concurrently. We certainly want to avoid crashes so, if that's what one risks from using lzma from several threads, we should protect against it. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue6715 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue6715] xz compressor support
Amaury Forgeot d'Arc amaur...@gmail.com added the comment: *Very* good news for lzma on windows: The precompiled static library liblzma.a works very well with MSVC (tested with VS2008 on Windows XP, 32bit). This was a surprise for me... Here is a patch for the win32 build files, to be applied after Nadeem's. I did not update the svn external repository, for my local copy I simply extracted xz-5.0.3-windows.zip. Tests pass in release and debug builds, and depends.exe shows no special requirements for _lzma.pyd. I could not test win64. Nadeem, in test.support precisionbigmemtest was recently renamed to bigmemtest, could you update your patch? -- Added file: http://bugs.python.org/file23371/lzma-win32.diff ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue6715 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue6715] xz compressor support
Amaury Forgeot d'Arc amaur...@gmail.com added the comment: On http://tukaani.org/xz, I downloaded the file named xz-5.0.3-windows.zip. It contains precompiled dlls for both platforms: bin_i486/liblzma.dll and bin_x86_64/liblzma.dll Unfortunately, there is no import library for VS. It should not be too difficult to make one, though: the provided headers are C89, so it's enough to write stubs for the functions used by the extension module. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue6715 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue6715] xz compressor support
Nadeem Vawda nadeem.va...@gmail.com added the comment: Hmm... according to http://git.tukaani.org/?p=xz.git;a=blob;f=windows/README-Windows.txt;hb=HEAD#l80, the MinGW-compiled static libs *can* be used with MSVC. Not sure how reliable the information is, but it's worth a try at least. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue6715 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue6715] xz compressor support
Amaury Forgeot d'Arc amaur...@gmail.com added the comment: Ah indeed, the zip archive contains a doc/liblzma.def which can be used to build a liblzma.lib -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue6715 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue6715] xz compressor support
Amaury Forgeot d'Arc amaur...@gmail.com added the comment: Hey, today I learnt something about mingw! Rename liblzma.a to e.g. liblzma_static.lib and tell MSVC to link against it. Apparently mingw can generate COFF libraries. This may simplify things *a lot*. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue6715 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue6715] xz compressor support
Nadeem Vawda nadeem.va...@gmail.com added the comment: Wow, this discussion has gotten quite busy while I've been travelling... Martin, could you explain what the problems are with bundling a precompiled DLL for Windows? I am willing to do the work of getting liblzma to compile with VS if necessary, but I don't know how receptive the upstream maintainer will be to the changes. If I can explain how lack of VS support is a problem for us, the request should carry more weight. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue6715 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue6715] xz compressor support
Martin v. Löwis mar...@v.loewis.de added the comment: Am 06.10.11 13:01, schrieb Nadeem Vawda: Nadeem Vawdanadeem.va...@gmail.com added the comment: Wow, this discussion has gotten quite busy while I've been travelling... Martin, could you explain what the problems are with bundling a precompiled DLL for Windows? Off-hand, it's only minor issues: e.g. when running Python out of its build directory, the DLL must be in the same directory. Now, since the output directory differs depending on build option, getting the DLL there might be tricky. Things I wonder about and couldn't quickly answer from the web: where exactly is the DLL that we would use? is there a AMD64 version of it? Does it come with import libraries usable by VS? -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue6715 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue6715] xz compressor support
Jesús Cea Avión j...@jcea.es added the comment: I agree with Martin here. We should *NOT* have first and second class OS support, if we can avoid it. That said, I wonder what happens in Windows with the BZ2 module, for instance :-?. Do we include the BZ2 sourcecode to compile it under windows?. I know, for instance, that Windows 2.* builds include Berkeley DB. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue6715 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue6715] xz compressor support
Antoine Pitrou pit...@free.fr added the comment: I agree with Martin here. We should *NOT* have first and second class OS support, if we can avoid it. The key word being if we can avoid it. Jesus, if you are a Windows expert, your contribution is welcome. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue6715 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue6715] xz compressor support
Amaury Forgeot d'Arc amaur...@gmail.com added the comment: For bz2, Tools/buildbot/external-common.bat has code to download bz2 source, and PCbuild/_bz2.vcproj include and compile these files together with _bz2.pyd. The _ssl module does a similar thing, except that libeay32.lib and libssleay32.lib are built in a separate step. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue6715 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue6715] xz compressor support
Martin v. Löwis mar...@v.loewis.de added the comment: That said, I wonder what happens in Windows with the BZ2 module, for instance :-?. External code currently lives at http://svn.python.org/projects/external/. The build process gets it from there, and we may have local modifications to libraries where necessary. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue6715 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue6715] xz compressor support
STINNER Victor victor.stin...@haypocalc.com added the comment: I agree with Martin here. We should *NOT* have first and second class OS support, if we can avoid it. Ok but who will do the job? If nobody is motivated to fix compiler issues, it would be a pity to not add the module for that. -- nosy: +haypo ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue6715 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue6715] xz compressor support
Jesús Cea Avión j...@jcea.es added the comment: Antoine, I am a Linux/Solaris/Illumos guy. I only use Windows (virtualized) to sync my iPhone with iTunes :) -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue6715 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue6715] xz compressor support
Antoine Pitrou pit...@free.fr added the comment: Based on Amaury's report, I would suggest going forward integrating the xz module for configure-based systems, and letting someone else handle Windows integration later if a solution is found. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue6715 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue6715] xz compressor support
Changes by Éric Araujo mer...@netwok.org: Removed file: http://bugs.python.org/file23300/unnamed ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue6715 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue6715] xz compressor support
Martin v. Löwis mar...@v.loewis.de added the comment: - liblzma can't be compiled by Visual Studio: too many C99 isms, mostly variables declared in the middle of a block. It's doable for sure, but it's a lot of work. I'd be in favor of doing so, and then feeding patches upstream. Hopefully, eventually, the code would compile out of the box on VS. - liblzma is normally compiled with mingw, but we have to be sure that is uses the correct MSCRT C runtime, and what about debug builds? In principle, it's not necessary to use the same CRT, as long as we aren't passing CRT objects across DLL boundaries (memory blocks managed by malloc/free would be candidates). I haven't reviewed the module to find out whether the liblzma interface involves CRT objects. - The way recommended by XZ is to use a precompiled liblzma.dll; Then it should be easy to build an extension module, but its would be the first time that we distribute an extension module which needs a non-system DLL. Is it enough to copy it next to _lzma.pyd? Is there some work to do in the installer? It wouldn't actually be the first time. We also ship Tcl DLLs. But it's a pain, so it would be much better if the sources were actually referenced in the VS project - so we would not need a library at all. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue6715 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue6715] xz compressor support
Martin v. Löwis mar...@v.loewis.de added the comment: Based on Amaury's report, I would suggest going forward integrating the xz module for configure-based systems, and letting someone else handle Windows integration later if a solution is found. -1000. I feel quite strongly that this should not be added unless there is also support in the Windows build process for it. I'll see what I can do, but it may take some time - until then, I urge to stall this issue, i.e. not proceed with checking it in. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue6715 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue6715] xz compressor support
Antoine Pitrou pit...@free.fr added the comment: Based on Amaury's report, I would suggest going forward integrating the xz module for configure-based systems, and letting someone else handle Windows integration later if a solution is found. -1000. I feel quite strongly that this should not be added unless there is also support in the Windows build process for it. Why? -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue6715 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue6715] xz compressor support
Martin v. Löwis mar...@v.loewis.de added the comment: Am 04.10.11 19:08, schrieb Antoine Pitrou: Antoine Pitroupit...@free.fr added the comment: Based on Amaury's report, I would suggest going forward integrating the xz module for configure-based systems, and letting someone else handle Windows integration later if a solution is found. -1000. I feel quite strongly that this should not be added unless there is also support in the Windows build process for it. Why? This module is only useful in the standard library if it is available on all systems. If it is not available on all systems, it may just as well be available from PyPI only. Now, as it needs to be available on Windows, I want to see the actual Windows support, else we would have to block the release until Windows support is available, possibly then reverting the module because no Windows support is forthcoming. So it's easier not to commit in the first place. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue6715 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue6715] xz compressor support
Antoine Pitrou pit...@free.fr added the comment: This module is only useful in the standard library if it is available on all systems. Not really. xz is becoming a defacto standard under Linux (and perhaps other free Unices) while I guess it is marginal under Windows. We have other system-specific functionality, and nobody sees it as a bad thing. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue6715 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue6715] xz compressor support
Martin v. Löwis mar...@v.loewis.de added the comment: Not really. xz is becoming a defacto standard under Linux (and perhaps other free Unices) while I guess it is marginal under Windows. We have other system-specific functionality, and nobody sees it as a bad thing. That's because all system-specific functionality that we have really depends on system features which just can't be available elsewhere. For all functionality that in principle works on all systems, it also actually works on all systems for Python. In cases where stuff was only available on Linux even though it could work on other systems, people *did* see it as a bad thing. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue6715 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue6715] xz compressor support
Antoine Pitrou pit...@free.fr added the comment: That's because all system-specific functionality that we have really depends on system features which just can't be available elsewhere. For all functionality that in principle works on all systems, it also actually works on all systems for Python. In cases where stuff was only available on Linux even though it could work on other systems, people *did* see it as a bad thing. Agreed, but it's a probably with the external library. That's like saying we are responsible if libffi fails building with the AIX compiler. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue6715 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue6715] xz compressor support
Antoine Pitrou pit...@free.fr added the comment: That's because all system-specific functionality that we have really depends on system features which just can't be available elsewhere. For all functionality that in principle works on all systems, it also actually works on all systems for Python. In cases where stuff was only available on Linux even though it could work on other systems, people *did* see it as a bad thing. Agreed, but it's a probably with the external library. That's like saying we are responsible if libffi fails building with the AIX compiler. s/probably/problem/ ;) -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue6715 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue6715] xz compressor support
Nadeem Vawda nadeem.va...@gmail.com added the comment: Thanks for investigating the Windows situation. - liblzma can't be compiled by Visual Studio: too many C99 isms, mostly variables declared in the middle of a block. It's doable for sure, but it's a lot of work. I don't think that creating our own MSVC-friendly fork of liblzma is really an option. Over and above the work of porting it in the first place (and all the opportunities for bugs to creep in along the way), we'd also have to worry about keeping up to date with upstream changes. I believe we currently do something similar with libffi (for ctypes), and the impression I've gotten is that it's caused a lot of trouble. - The way recommended by XZ is to use a precompiled liblzma.dll; Then it should be easy to build an extension module, but its would be the first time that we distribute an extension module which needs a non-system DLL. Is it enough to copy it next to _lzma.pyd? Is there some work to do in the installer? I would guess that this is sufficient, but my knowledge of how Windows DLLs work is minimal. Could someone with more platform knowledge weigh in on whether this would work (and if there are any problems it might cause)? -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue6715 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue6715] xz compressor support
Dan Stromberg strom...@gmail.com added the comment: On Sun, Oct 2, 2011 at 3:49 PM, Nadeem Vawda rep...@bugs.python.org wrote: Nadeem Vawda nadeem.va...@gmail.com added the comment: Thanks for investigating the Windows situation. - liblzma can't be compiled by Visual Studio: too many C99 isms, mostly variables declared in the middle of a block. It's doable for sure, but it's a lot of work. I don't think that creating our own MSVC-friendly fork of liblzma is really an option. Over and above the work of porting it in the first place (and all the opportunities for bugs to creep in along the way), we'd also have to worry about keeping up to date with upstream changes. I believe we currently do something similar with libffi (for ctypes), and the impression I've gotten is that it's caused a lot of trouble. It's much better to contribute patches upstream. - The way recommended by XZ is to use a precompiled liblzma.dll; Then it should be easy to build an extension module, but its would be the first time that we distribute an extension module which needs a non-system DLL. Is it enough to copy it next to _lzma.pyd? Is there some work to do in the installer? I would guess that this is sufficient, but my knowledge of how Windows DLLs work is minimal. Could someone with more platform knowledge weigh in on whether this would work (and if there are any problems it might cause)? I've not done much with windows dll's, but I've heard they're pretty similar to AIX shared libraries which I've done some work with. AIX shared libraries don't deal with versioning well - if you have two version of the same library on a system, you have to pop them into two different loader domains, or suffer unresolved externals at runtime. Then your applications are in of the two loader domains, and if they're in the wrong one, you again suffer unresolved externals at runtime. -- Added file: http://bugs.python.org/file23300/unnamed ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue6715 ___brdiv class=gmail_quoteOn Sun, Oct 2, 2011 at 3:49 PM, Nadeem Vawda span dir=ltrlt;a href=mailto:rep...@bugs.python.org;rep...@bugs.python.org/agt;/span wrote:brblockquote class=gmail_quote style=margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex; div class=imbr Nadeem Vawda lt;a href=mailto:nadeem.va...@gmail.com;nadeem.va...@gmail.com/agt; added the comment:br br /divThanks for investigating the Windows situation.br div class=imbr gt; - liblzma can#39;t be compiled by Visual Studio: too many C99 isms, mostlybr gt; variables declared in the middle of a block.  It#39;s doable for sure, but it#39;s abr gt; lot of work.br br /divI don#39;t think that creating our own MSVC-friendly fork of liblzma is really anbr option. Over and above the work of porting it in the first place (and all thebr opportunities for bugs to creep in along the way), we#39;d also have to worry aboutbr keeping up to date with upstream changes. I believe we currently do somethingbr similar with libffi (for ctypes), and the impression I#39;ve gotten is that it#39;sbr caused a lot of trouble./blockquotedivbr/divdivIt#39;s much better to contribute patches upstream./divdiv /divblockquote class=gmail_quote style=margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex; div class=im gt; - The way recommended by XZ is to use a precompiled liblzma.dll; Then itbr gt; should be easy to build an extension module, but its would be the first timebr gt; that we distribute an extension module which needs a non-system DLL.  Is itbr gt; enough to copy it next to _lzma.pyd?  Is there some work to do in thebr gt; installer?br br /divI would guess that this is sufficient, but my knowledge of how Windows DLLs workbr is minimal. Could someone with more platform knowledge weigh in on whether thisbr would work (and if there are any problems it might cause)?/blockquotedivbr/divdivI#39;ve not done much with windows dll#39;s, but I#39;ve heard they#39;re pretty similar to AIX shared libraries which I#39;ve done some work with.  AIX shared libraries don#39;t deal with versioning well - if you have two version of the same library on a system, you have to pop them into two different loader domains, or suffer unresolved externals at runtime.  Then your applications are in of the two loader domains, and if they#39;re in the wrong one, you again suffer unresolved externals at runtime./div div /div/div-- brDan Strombergbr ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue6715] xz compressor support
Martin v. Löwis mar...@v.loewis.de added the comment: Nadeem, I started reviewing this path, but now stopped since the patch I reviewed isn't available anymore. Please let us know when the patch has a state where you don't plan to make more changes. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue6715 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue6715] xz compressor support
Nadeem Vawda nadeem.va...@gmail.com added the comment: Ah, sorry about that. It probably wasn't necessary to regenerate the patch after I addressed the comments in each review. The patch should be stable now; I don't plan to make any further changes for a while. (At some point I'll want to add a few more tests, and get the module to compile on Windows. But that can wait.) -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue6715 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue6715] xz compressor support
Amaury Forgeot d'Arc amaur...@gmail.com added the comment: I'm a bit worried by the Windows version: - liblzma can't be compiled by Visual Studio: too many C99 isms, mostly variables declared in the middle of a block. It's doable for sure, but it's a lot of work. - liblzma is normally compiled with mingw, but we have to be sure that is uses the correct MSCRT C runtime, and what about debug builds? - All extension modules use static libraries: zlib, expat, sqlite. But a gcc static library can't be used by Visual Studio. - The way recommended by XZ is to use a precompiled liblzma.dll; Then it should be easy to build an extension module, but its would be the first time that we distribute an extension module which needs a non-system DLL. Is it enough to copy it next to _lzma.pyd? Is there some work to do in the installer? Too many buts :) -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue6715 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue6715] xz compressor support
Changes by Nadeem Vawda nadeem.va...@gmail.com: Added file: http://bugs.python.org/file23158/95016f363e6c.diff ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue6715 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue6715] xz compressor support
Changes by Nadeem Vawda nadeem.va...@gmail.com: Added file: http://bugs.python.org/file23159/3017ee1d548d.diff ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue6715 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue6715] xz compressor support
Changes by Nadeem Vawda nadeem.va...@gmail.com: Removed file: http://bugs.python.org/file23158/95016f363e6c.diff ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue6715 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue6715] xz compressor support
Changes by Nadeem Vawda nadeem.va...@gmail.com: Removed file: http://bugs.python.org/file23063/f3cf187208ea.diff ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue6715 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue6715] xz compressor support
Changes by Nadeem Vawda nadeem.va...@gmail.com: Removed file: http://bugs.python.org/file23129/fe2c9998f329.diff ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue6715 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue6715] xz compressor support
Nadeem Vawda nadeem.va...@gmail.com added the comment: I've attached another patch that includes proper ReST documentation, a few more tests, and some fixups following Ezio and Amaury's reviews. -- stage: needs patch - patch review ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue6715 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue6715] xz compressor support
Lars Gustäbel l...@gustaebel.de added the comment: Today I played around with lzma support for tarfile based on your last patch (see issue5689). There are a few minor issues that I just wanted to mention, as they break the tarfile testsuite: - LZMAFile does not expose a name attribute. BZ2File doesn't either (not in 3.x anyway), but GzipFile does. - LZMAFile does not allow a 'b' in the mode argument, unlike GzipFile and BZ2File. - The bz2 module exposes many error conditions as standard Python exceptions, e.g. IOError, EOFError. The lzma module uses LZMAError for all errors without distinction. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue6715 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue6715] xz compressor support
Changes by Nadeem Vawda nadeem.va...@gmail.com: Added file: http://bugs.python.org/file23168/591277fe6b4a.diff ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue6715 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue6715] xz compressor support
Changes by Nadeem Vawda nadeem.va...@gmail.com: Removed file: http://bugs.python.org/file23159/3017ee1d548d.diff ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue6715 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue6715] xz compressor support
Nadeem Vawda nadeem.va...@gmail.com added the comment: Thanks for getting the ball rolling with that. I'll want to try and familiarize myself with the code for tarfile module before I start changing stuff, but those points shouldn't be hard to fix, one way or the other. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue6715 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue6715] xz compressor support
Changes by John Reese jre...@leetcode.net: -- nosy: -jreese ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue6715 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue6715] xz compressor support
Changes by Nadeem Vawda nadeem.va...@gmail.com: Added file: http://bugs.python.org/file23129/fe2c9998f329.diff ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue6715 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue6715] xz compressor support
Nadeem Vawda nadeem.va...@gmail.com added the comment: I've attached another patch (fe2c9998f329.diff) with a more complete implementation of the lzma module. All that's left now is to write the documentation, and make sure that the module can build on Windows. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue6715 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue6715] xz compressor support
Nam Nguyen bits...@gmail.com added the comment: In uint32_converter, I'm not sure the if statement comparing val and UINT32_MAX makes sense. val was defined to be unsigned long, which is 32bit on 32-bit compiler. -- nosy: +Nam.Nguyen ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue6715 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue6715] xz compressor support
Nadeem Vawda nadeem.va...@gmail.com added the comment: In uint32_converter, I'm not sure the if statement comparing val and UINT32_MAX makes sense. val was defined to be unsigned long, which is 32bit on 32-bit compiler. Yes, on a 32-bit system, this comparison will always be false. However, on a 64-bit system, an unsigned long will (usually) be 64 bits wide. In this case, we do need to check that the value fits into a uint32_t. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue6715 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue6715] xz compressor support
Changes by Barry A. Warsaw ba...@python.org: -- nosy: +barry ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue6715 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue6715] xz compressor support
Changes by Nadeem Vawda nadeem.va...@gmail.com: -- hgrepos: +64 ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue6715 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue6715] xz compressor support
Changes by Nadeem Vawda nadeem.va...@gmail.com: Added file: http://bugs.python.org/file23063/f3cf187208ea.diff ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue6715 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
