[issue1625] bz2.BZ2File doesn't support multiple streams

Darko Veberic added the comment: It is interesting, this issue was fixed in boost 4 years ago: https://svn.boost.org/trac/boost/ticket/3853 and a comment from the pbzip2 author suggests that not supporting multiple streams is in violation of the bz2 file format spec. -- nosy: +Darko.Veb

[issue1625] bz2.BZ2File doesn't support multiple streams

Roundup Robot added the comment: New changeset ad20324229f4 by Nadeem Vawda in branch '2.7': Clarify note in BZ2File docs about lack of multi-stream support (issue #1625). http://hg.python.org/cpython/rev/ad20324229f4 New changeset e4c4595033ad by Nadeem Vawda in branch '3.2': Clarify note in B

[issue1625] bz2.BZ2File doesn't support multiple streams

Éric Araujo added the comment: Just a thought: maybe the doc note should mention that bz2file is a backport of 3.3’s improved class, so that people know that 1) it’s well-supported code 2) a future Python version will remove the need for the external dependency. -- __

[issue1625] bz2.BZ2File doesn't support multiple streams

Roundup Robot added the comment: New changeset e73d549b7458 by Nadeem Vawda in branch '2.7': Issue #1625: Document BZ2File's lack of support for multi-stream inputs. http://hg.python.org/cpython/rev/e73d549b7458 New changeset 190826ee0450 by Nadeem Vawda in branch '3.2': Issue #1625: Document B

[issue1625] bz2.BZ2File doesn't support multiple streams

Nadeem Vawda added the comment: > In fact, you can create those files from python, files that python can not > unpack later. Really? How so? BZ2File only started accepting the "a" mode in 3.3 (thanks to Nir's patch for this issue, actually). > If the refusal of backporting this to 3.2 and 2.7

[issue1625] bz2.BZ2File doesn't support multiple streams

Éric Araujo added the comment: As the bug/feature judgment is not easy to make, I think python-dev should be asked. -- ___ Python tracker ___ ___

[issue1625] bz2.BZ2File doesn't support multiple streams

Jesús Cea Avión added the comment: Éric, bz2 module in Python is documented as able to manage bz2 files. But that is not true, since it is unable to manage popular bz2 files. That looks like a bug to me. In fact, you can create those files from python, files that python can not unpack later.

[issue1625] bz2.BZ2File doesn't support multiple streams

Éric Araujo added the comment: > I think this support should be backported to Python 2.7 and 3.2. I think our policy is pretty clear: the module docs did not say multiple streams were supported, so when support for them was added it was clearly a new feature. > Current code can't decompress f

[issue1625] bz2.BZ2File doesn't support multiple streams

Nadeem Vawda added the comment: > An alternative solution I'd like to pursue is to backport 3.3's BZ2File > implementation to run on 2.7, and release it on PyPI. Well, that was easier than I expected. It didn't take much work to get it working under 2.6, 2.7 and 3.2. I've put up this "bz2file"

[issue1625] bz2.BZ2File doesn't support multiple streams

Nadeem Vawda added the comment: > I am just recompressing a 77GB file because of this :-(. Sorry to hear that :( > I would consider that a bug, not a feature request. Semantic issues aside, my concern here is that the patch for 2.7 is considerably larger than the one for 3.3, and the code it

[issue1625] bz2.BZ2File doesn't support multiple streams

Jesús Cea Avión added the comment: I think this support should be backported to Python 2.7 and 3.2. Current code can't decompress files generated by "pbzip2", fairly popular. I would consider that a bug, not a feature request. I am just recompressing a 77GB file because of this :-(.

[issue1625] bz2.BZ2File doesn't support multiple streams

Changes by STINNER Victor : -- nosy: +haypo ___ Python tracker ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.o

[issue1625] bz2.BZ2File doesn't support multiple streams

Nadeem Vawda added the comment: > This is all fine and well, but this is clearly a bug and not a feature. No, it is not at all clear that this is a bug. I agree that this is a desirable capability to have, but nowhere does the module claim to support multi-stream files. Nor is it an inherent f

[issue1625] bz2.BZ2File doesn't support multiple streams

Gökçen Eraslan added the comment: I mean "as a feature". -- ___ Python tracker ___ ___ Python-bugs-list mailing list Unsubscribe: htt

[issue1625] bz2.BZ2File doesn't support multiple streams

Gökçen Eraslan added the comment: +1. If we think this as a bug, python 2.x users will never be able to extract multiple-stream bz2 files. -- ___ Python tracker ___

[issue1625] bz2.BZ2File doesn't support multiple streams

Crispin Wellington added the comment: This is all fine and well, but this is clearly a bug and not a feature. Can we please see this bug fix go into 2.7 at some point? -- nosy: +Crispin.Wellington ___ Python tracker

[issue1625] bz2.BZ2File doesn't support multiple streams

Nadeem Vawda added the comment: Ozan: Thanks for taking the time to backport the patch. Unfortunately, as Éric said, 2.7 is in maintenance mode, so it no longer receives new features. -- ___ Python tracker

[issue1625] bz2.BZ2File doesn't support multiple streams

Éric Araujo added the comment: We don’t add news features in stable releases. Nadeem has closed this bug as fixed for 3.3 and it can’t go in 2.7, so I think we’re done here. -- ___ Python tracker

[issue1625] bz2.BZ2File doesn't support multiple streams

ozan caglayan added the comment: Attached patch is a revised version of bz2ms.patch against Python 2.7.2. The patch is tested using tarfile and bz2 modules. It also passes the included tests correctly. It also imports a missing class from BytesIO to fix the tests. It's up to you to take that

[issue1625] bz2.BZ2File doesn't support multiple streams

Changes by Gökçen Eraslan : -- nosy: +Gökçen.Eraslan ___ Python tracker ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail

[issue1625] bz2.BZ2File doesn't support multiple streams

ozan caglayan added the comment: With the current tip everything works correctly. I think it's because of the complete rewrite of the bz2 module with python and the refactoring of _bz2.so. -- ___ Python tracker __

[issue1625] bz2.BZ2File doesn't support multiple streams

ozan caglayan added the comment: I ported the bz2ms.patch to Python 2.7.2 and it works correctly within the bz2 module. But when you open a multistream (tar)bz2 with the tarfile module, even the tarfile uses the BZ2File() class, there exists unextracted missing files. I'll now try with the c

[issue1625] bz2.BZ2File doesn't support multiple streams

Changes by Antoine Pitrou : -- resolution: accepted -> fixed ___ Python tracker ___ ___ Python-bugs-list mailing list Unsubscribe: htt

[issue1625] bz2.BZ2File doesn't support multiple streams

Roundup Robot added the comment: New changeset 3e5200abf8eb by Nadeem Vawda in branch 'default': Issue #1625: Add stream ordering test to test_bz2. http://hg.python.org/cpython/rev/3e5200abf8eb -- ___ Python tracker

[issue1625] bz2.BZ2File doesn't support multiple streams

Nadeem Vawda added the comment: > If you’re logged into Roundup, you should automatically be logged into > our Rietveld instance. I thought this was the case, but it isn't working for me. I've filed a bug on the meta-tracker. -- ___ Python tracker

[issue1625] bz2.BZ2File doesn't support multiple streams

Roundup Robot added the comment: New changeset 48e837b2a327 by Nadeem Vawda in branch 'default': Miscellaneous cleanups to bz2 and test_bz2 following issue #1625. http://hg.python.org/cpython/rev/48e837b2a327 -- ___ Python tracker

[issue1625] bz2.BZ2File doesn't support multiple streams

Éric Araujo added the comment: If you’re logged into Roundup, you should automatically be logged into our Rietveld instance. You can file a bug on the meta-tracker (link in the left sidebar) if this does not work. -- ___ Python tracker

[issue1625] bz2.BZ2File doesn't support multiple streams

Nadeem Vawda added the comment: I seem to be unable to log in to rietveld, so I'll reply here. >> result += decomp.decompress(data) > Is this efficient? I understood that other Python implementations > had poorly performing str.__iadd__, and therefore that using a list > was the common idiom (

[issue1625] bz2.BZ2File doesn't support multiple streams

Éric Araujo added the comment: I made a few comments and asked two questions on the review page. (I should have said so here.) -- nosy: +eric.araujo ___ Python tracker ___

[issue1625] bz2.BZ2File doesn't support multiple streams

Nadeem Vawda added the comment: Committed. Once again, thanks for the patch! -- resolution: -> accepted stage: patch review -> committed/rejected status: open -> closed ___ Python tracker _

[issue1625] bz2.BZ2File doesn't support multiple streams

Roundup Robot added the comment: New changeset 8cebbc6473d9 by Nadeem Vawda in branch 'default': Issue #1625: BZ2File and bz2.decompress() now support multi-stream files. http://hg.python.org/cpython/rev/8cebbc6473d9 New changeset 0be55601f948 by Nadeem Vawda in branch 'default': Update bz2 doc

[issue1625] bz2.BZ2File doesn't support multiple streams

Changes by Nir Aides : Removed file: http://bugs.python.org/file22087/cpython-bz2-streams.patch ___ Python tracker ___ ___ Python-bugs-list mai

[issue1625] bz2.BZ2File doesn't support multiple streams

Nir Aides added the comment: Right! I updated the patch and added a test for the aligned stream/buffer case. -- Added file: http://bugs.python.org/file22114/cpython-bz2-streams.patch ___ Python tracker

[issue1625] bz2.BZ2File doesn't support multiple streams

Nadeem Vawda added the comment: > False alarm; go ahead with the review. I took a look too early in the > morning before caffeine kicked in. No worries. I know the feeling. The tests look fine. The bodies of testRead() and testReadMultiStream() appear to have been swapped, though. I'm guessing

[issue1625] bz2.BZ2File doesn't support multiple streams

Nir Aides added the comment: False alarm; go ahead with the review. I took a look too early in the morning before caffeine kicked in. Note Lib/test/test_bz2.py was directly upgraded from bz2ms.patch. A note on bz2 behavior: A BZ2Decompressor object is only good for one stream; after that eof

[issue1625] bz2.BZ2File doesn't support multiple streams

Nadeem Vawda added the comment: OK, I'll hold off on doing a detailed review until then. -- ___ Python tracker ___ ___ Python-bugs-lis

[issue1625] bz2.BZ2File doesn't support multiple streams

Nir Aides added the comment: Wait, the tests seem wrong. I'll post an update later today. -- ___ Python tracker ___ ___ Python-bugs-li

[issue1625] bz2.BZ2File doesn't support multiple streams

Nadeem Vawda added the comment: Thanks for the patch. I'll review it tomorrow. -- ___ Python tracker ___ ___ Python-bugs-list mailing

[issue1625] bz2.BZ2File doesn't support multiple streams

Nir Aides added the comment: Hi, I attach a patch to Python 3.3 Lib/bz2.py with updated tests: cpython-bz2-streams.patch -- keywords: +needs review stage: needs patch -> patch review Added file: http://bugs.python.org/file22087/cpython-bz2-streams.patch

[issue1625] bz2.BZ2File doesn't support multiple streams

Changes by Nir Aides : -- assignee: niemeyer -> nirai ___ Python tracker ___ ___ Python-bugs-list mailing list Unsubscribe: http://mai

[issue1625] bz2.BZ2File doesn't support multiple streams

Changes by Nir Aides : -- nosy: +nirai ___ Python tracker ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/ma

[issue1625] bz2.BZ2File doesn't support multiple streams

Antoine Pitrou added the comment: The patch here is totally out of date, following issue5863. -- nosy: +nvawda stage: patch review -> needs patch versions: +Python 3.3 -Python 3.2 ___ Python tracker ___

[issue1625] bz2.BZ2File doesn't support multiple streams

Oliver Deppert added the comment: Thanks for the update Like I mentioned before in my previous comment, I'm still searching for a solution/patch for python 2.x able to handle multiple streams of bz2. Does anybody know a work-around or have a solution porting the p3k-patch to the good ol

[issue1625] bz2.BZ2File doesn't support multiple streams

Antoine Pitrou added the comment: Here is an update of the patch against current py3k. I've added a copyright mention at the top of Modules/bz2module.c which I hope manages to capture the essence of msg93721. Martin, what do you think? -- nosy: +loewis stage: -> patch review versions

[issue1625] bz2.BZ2File doesn't support multiple streams

Changes by Dirkjan Ochtman : -- nosy: +djc ___ Python tracker ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.or

[issue1625] bz2.BZ2File doesn't support multiple streams

Oliver Deppert added the comment: Dear all, first of all, thank you for the patch making multiple file-streams in bz2 available in python. Yesterday, I've tried to adapt the recent patch for python 3k to the actual python 2.7. Most of the hunks could be easy adapted by editing just one ore t

[issue1625] bz2.BZ2File doesn't support multiple streams

David Bonner added the comment: Picking this back up again. There's actually no docs changes necessary...the docs never mentioned that the module didn't support multiple logical streams, and I didn't see any other mentions in the docs that seemed to need updating. I supposed I could add som

[issue1625] bz2.BZ2File doesn't support multiple streams

Antoine Pitrou added the comment: > That was mostly just out of paranoia, since the comments mentioned > multiple calls to close being legal. Looking at it again, that particular > case isn't an issue, since we don't hit that call when the mode is > MODE_CLOSED. The testsuite runs happily w

[issue1625] bz2.BZ2File doesn't support multiple streams

David Bonner added the comment: That was mostly just out of paranoia, since the comments mentioned multiple calls to close being legal. Looking at it again, that particular case isn't an issue, since we don't hit that call when the mode is MODE_CLOSED. The testsuite runs happily with those

[issue1625] bz2.BZ2File doesn't support multiple streams

Antoine Pitrou added the comment: I'm not comfortable with the following change (which appears twice in the patch): - BZ2_bzReadClose(&bzerror, self->fp); + if (self->fp) + BZ2_bzReadClose(&bzerror, self->fp);

[issue1625] bz2.BZ2File doesn't support multiple streams

David Bonner added the comment: Understandable. New patch attached. -- Added file: http://bugs.python.org/file15177/py3k_bz2.patch ___ Python tracker ___ ___

[issue1625] bz2.BZ2File doesn't support multiple streams

R. David Murray added the comment: A new patch will make it more likely that it will actually get applied :) Thanks for your work on this. -- ___ Python tracker ___

[issue1625] bz2.BZ2File doesn't support multiple streams

David Bonner added the comment: Hrm...yeah, I should probably be setting it to closed as soon as BZ2_bzReadClose() returns, and then back to open once BZ2_bzReadOpen succeeds. Wasn't intentional...thanks for the catch. You guys need a new patch with that change in it? I'll try and get a 2.7 p

[issue1625] bz2.BZ2File doesn't support multiple streams

Antoine Pitrou added the comment: As far as I can tell, the patch looks mostly good. I just wonder, in Util_HandleBZStreamEnd(), why you don't set self->mode to MODE_CLOSED if BZ2_bzReadOpen() fails. As a sidenote, the bz2 module implementation seems to have changed quite a bit between trunk an

[issue1625] bz2.BZ2File doesn't support multiple streams

David Bonner added the comment: I can remove the boilerplate from the code as long as I add the following to the submittal: VMware, Inc. is providing this bz2 module patch to you under the terms of the Apache License 2.0 with the understanding that you plan to re-license this under the terms an

[issue1625] bz2.BZ2File doesn't support multiple streams

R. David Murray added the comment: If the patch is substantial enough that legal boilerplate is even an issue, then I'm pretty sure a contributor agreement will be required for patch acceptance, at which point I think the boilerplate won't be needed. The Apache license is certainly acceptable.

[issue1625] bz2.BZ2File doesn't support multiple streams

David Bonner added the comment: Thanks for the reply. My company's legal dept. told me that we needed to put the boilerplate into the files as part of releasing it under the apache license. I used a tarball because they also recommended including a full copy of the license with the patch. I'm

[issue1625] bz2.BZ2File doesn't support multiple streams

Antoine Pitrou added the comment: Some notes about posting patches: - you should post the patch alone, not in an archive - generally you should post patches against the 2.7 trunk, we take care of merging them to py3k ourselves (but in this case the difference should be minimal anyway) - I'm not

[issue1625] bz2.BZ2File doesn't support multiple streams

David Bonner added the comment: sorry, the previous patch was from an old version. attaching the correct version now. apologies for the noise. -- Added file: http://bugs.python.org/file15001/bz2_patch.tar.bz2 ___ Python tracker

[issue1625] bz2.BZ2File doesn't support multiple streams

Changes by David Bonner : Removed file: http://bugs.python.org/file15000/bz2_patch.tar.bz2 ___ Python tracker ___ ___ Python-bugs-list mailing

[issue1625] bz2.BZ2File doesn't support multiple streams

David Bonner added the comment: I've got a patch that fixes this. It allows BZ2File to read multi-stream files as generated by pbzip2, allows BZ2File to open files in append mode, and also updates bz2.decompress to allow it to handle multi-stream chunks of data. We originally wrote it against

[issue1625] bz2.BZ2File doesn't support multiple streams

Changes by Sean Reifschneider <[EMAIL PROTECTED]>: -- assignee: -> niemeyer nosy: +niemeyer priority: -> normal __ Tracker <[EMAIL PROTECTED]> __

[issue1625] bz2.BZ2File doesn't support multiple streams

Thomas Herve added the comment: The gzip module supports reopening an existing file to add another stream. I think the bz2 module should not the same. __ Tracker <[EMAIL PROTECTED]> __

[issue1625] bz2.BZ2File doesn't support multiple streams

A.M. Kuchling added the comment: Like gzip, you can concatenate two bzip2 files: bzip2 -c /etc/passwd >/tmp/pass.bz2 bzip2 -c /etc/passwd >>/tmp/pass.bz2 bunzip2 will output both parts, generating two copies of the file. So nothing needs to be done on compression, but uncompression needs to l

[issue1625] bz2.BZ2File doesn't support multiple streams

Thomas Lee added the comment: If you're referring to an 'append' mode for bz2file objects, it may be a limitation of the underlying library: my version of bzlib.h only provides BZ2_bzWriteOpen and BZ2_bzReadOpen - it's not immediately clear how you would open a BZ2File in append mode looking at t

[issue1625] bz2.BZ2File doesn't support multiple streams

New submission from Thomas Herve: The BZ2File class only supports one stream per file. It possible to have multiple streams concatenated in one file, it the resulting data should be the concatenation of all the streams. It's what the bunzip2 program produces, for example. It's also supported by t