[issue17804] streaming struct unpacking

Antoine Pitrou added the comment: Ok, I don't want the bikeshedding to last too long, so I committed the patch with docs. Thanks everyone! -- resolution: -> fixed stage: patch review -> committed/rejected status: open -> closed ___ Python tracker <

[issue17804] streaming struct unpacking

Roundup Robot added the comment: New changeset d232cff25bbd by Antoine Pitrou in branch 'default': Issue #17804: New function ``struct.iter_unpack`` allows for streaming struct unpacking. http://hg.python.org/cpython/rev/d232cff25bbd -- nosy: +python-dev ___

[issue17804] streaming struct unpacking

Terry J. Reedy added the comment: I think 'iter_unpack' is deceptive and wrong for the following reason. Up to now, 'ixyz' or 'iterxyz' or 'iter_xyz' has meant a version of 'xyz' that presents items one at a time rather than all at once in a collection object (usually in a list). Unpack return

[issue17804] streaming struct unpacking

Raymond Hettinger added the comment: iter_unpack() is closer to how we name other iter variants, so I think you ought to stick with that. The other names are too creative (and hence less guessable). -- nosy: +rhettinger ___ Python tracker

[issue17804] streaming struct unpacking

Antoine Pitrou added the comment: > Small bikeshed on the name: I think 'unpack_iter' would be more > consistent with what is already there, e.g. 'unpack' and 'unpack_from'. > In fact, when experimenting with this patch I found myself typing > 'unpack_iter' several times. I thought so, but "unpa

[issue17804] streaming struct unpacking

Meador Inge added the comment: This seems reasonable to me to. So +1. Small bikeshed on the name: I think 'unpack_iter' would be more consistent with what is already there, e.g. 'unpack' and 'unpack_from'. In fact, when experimenting with this patch I found myself typing 'unpack_iter' several t

[issue17804] streaming struct unpacking

Serhiy Storchaka added the comment: > I'm not against adding useful C tools to itertools, but you may have to convince Raymond ;) C implementation speeds up the benchmark only 1.5x. Granting the fact that this idiom is used in stdlib less than two dozens times (without tests and iobench), I do

[issue17804] streaming struct unpacking

Antoine Pitrou added the comment: > Yes, It's mainly because a grouper written on Python. When it will be > implemented in C, the difference will be less. This function will be > useful beside struct. I'm not against adding useful C tools to itertools, but you may have to convince Raymond ;) >

[issue17804] streaming struct unpacking

Serhiy Storchaka added the comment: > Well, according to a quick benchmark, iter_unpack() is 3x to 6x faster than > the grouper() + unpack() recipe. > (it's also a bit more user-friendly) Yes, It's mainly because a grouper written on Python. When it will be implemented in C, the difference wil

[issue17804] streaming struct unpacking

Mark Dickinson added the comment: This seems like an attractive idea. There's definitely a need for repeated unpacking with the same pattern, and I agree that putting the repetition into the pattern is suboptimal (not least from the point of view of caching structs). One thing that feels a bi

[issue17804] streaming struct unpacking

Changes by Phil Connell : -- nosy: +pconnell ___ Python tracker ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python

[issue17804] streaming struct unpacking

Antoine Pitrou added the comment: > - I'm no expert on the C API, but in s_iter_unpack do you not need to > check for failure of PyType_GenericAlloc before calling > PyObject_GetBuffer? Yes, good catch. > - I'm not a fan of separate iter_ functions (and there seemed to be a > general move away

[issue17804] streaming struct unpacking

Martin Morrison added the comment: On 20 Apr 2013, at 23:01, Martin Morrison wrote: > - I'm not a fan of separate iter_ functions (and there seemed to be a general > move away from them elsewhere in Python3; obviously here we have to maintain > backwards compat though). Perhaps a boolean keywo

[issue17804] streaming struct unpacking

Martin Morrison added the comment: I like the idea of this. Two comments: - I'm no expert on the C API, but in s_iter_unpack do you not need to check for failure of PyType_GenericAlloc before calling PyObject_GetBuffer? - I'm not a fan of separate iter_ functions (and there seemed to be a gene

[issue17804] streaming struct unpacking

Antoine Pitrou added the comment: For the record, here is the benchmark script. -- Added file: http://bugs.python.org/file29955/bench_unpack.py ___ Python tracker ___ ___

[issue17804] streaming struct unpacking

Antoine Pitrou added the comment: Well, according to a quick benchmark, iter_unpack() is 3x to 6x faster than the grouper() + unpack() recipe. (it's also a bit more user-friendly) -- ___ Python tracker ___

[issue17804] streaming struct unpacking

Serhiy Storchaka added the comment: Perhaps we need not iter_unpack(), but a grouper (some sort of)? def grouper(seq, size): for i in range(0, len(seq), size): yield seq[i: i + size] unpack = struct.Struct('!I').unpack for chunk in grouper(data, 4): word, = unpack(chunk) ...

[issue17804] streaming struct unpacking

Changes by Antoine Pitrou : -- nosy: +isoschiz ___ Python tracker ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.pyth

[issue17804] streaming struct unpacking

Antoine Pitrou added the comment: Here is a patch (still lacking docs). Comments welcome. -- keywords: +patch Added file: http://bugs.python.org/file29953/iter_unpack.patch ___ Python tracker __

[issue17804] streaming struct unpacking

Antoine Pitrou added the comment: (my initial intuition here was to use memoryview.cast() but it doesn't support non-native formats) -- nosy: +skrah ___ Python tracker ___ _

[issue17804] streaming struct unpacking

New submission from Antoine Pitrou: For certain applications, you want to unpack repeatedly the same pattern. This came in issue17618 (base85 decoding), where you want to unpack a stream of bytes as 32-bit big-endian unsigned ints. The solution adopted in issue17618 patch (struct.Struct("!{}I"