[issue19251] bitwise ops for bytes of equal length

Change by Matthias Gilch : -- nosy: +MGilch ___ Python tracker ___ ___

[issue19251] bitwise ops for bytes of equal length

Change by Guido van Rossum : -- nosy: -gvanrossum ___ Python tracker ___ ___

[issue19251] bitwise ops for bytes of equal length

Nick Coghlan added the comment: While it does match Christian's original suggestion, I'm not taking the "bytes" in the issue title as literally requiring that the feature be implemented as operator support on the bytes type (implementing it on memoryview or a new view

[issue19251] bitwise ops for bytes of equal length

Serhiy Storchaka added the comment: I don't understand what relation this issue has with Nick's use case. If you need to access separate fields in a binary packed structure, adding support of extended PEP 3118 like syntax in memoryview can help you. If you want to

[issue19251] bitwise ops for bytes of equal length

Terry J. Reedy added the comment: The last three posts have convinced me that 'efficient bit operations', not tied to the int type, are worth exploring, without immediate restriction to a particular API. I can see that micropython is a significant new use case. --

[issue19251] bitwise ops for bytes of equal length

Nick Coghlan added the comment: I think Antoine's right that another venue (such as python-ideas) might be a better venue for this discussion, but I'll still try to explain the potential analogy I see to bytes.upper()/.lower()/etc: those operations let you treat ASCII

[issue19251] bitwise ops for bytes of equal length

Antoine Pitrou added the comment: Please, let's have this API discussion outside of the bug tracker. This deserves a PEP. Also because I have an alternative API to suggest :-) -- ___ Python tracker

[issue19251] bitwise ops for bytes of equal length

Stefan Behnel added the comment: I'd second the proposal of considering the "array.array" type for this, instead of the bytes/bytearray types. Why? Because it is somewhat of a niche case after all, and the bytes type seems too exposed for it. Also, array.array supports

[issue19251] bitwise ops for bytes of equal length

Antoine Pitrou added the comment: Le 17/05/2018 à 18:20, Nick Coghlan a écrit : > > The question is whether we might be able to avoid some > bytes->Python-objects->bytes cycles if there were a few more > contiguous-binary-data-centric operations on bytes and/or memoryview

[issue19251] bitwise ops for bytes of equal length

Nick Coghlan added the comment: Thanks for the link Serhiy (I'd forgotten about the struct changes proposed in PEP 3118), but the existing struct formatting codes are fine for my purposes. The question is whether we might be able to avoid some bytes->Python-objects->bytes

[issue19251] bitwise ops for bytes of equal length

Serhiy Storchaka added the comment: Nick, for your tasks you may be interested in PEP 3118 which still is not completely implemented (issue3132). -- ___ Python tracker

[issue19251] bitwise ops for bytes of equal length

Nick Coghlan added the comment: This issue isn't at the stage where a PR would help - the core question is still "Should we add better native support for multi-byte bitwise operations?", not the specifics of what they API might look like or how we would implement it.

[issue19251] bitwise ops for bytes of equal length

Марк Коренберг added the comment: @ncoghlan Could you please create Pull-request on Github ? -- ___ Python tracker ___

[issue19251] bitwise ops for bytes of equal length

Nick Coghlan added the comment: I'm back in the embedded software world now, and hence working with the combination of: - low level serial formats (including fixed length CAN packets) - C firmware developers that are quite capable of writing supporting C-in-Python code

[issue19251] bitwise ops for bytes of equal length

Cameron Simpson added the comment: Amendment: I wrote above: "Just because a lot of things can be written/constructed as one liners doesn't mean they should be operators". Of course I meant to write "doesn't mean they should _not_ be operators". --

[issue19251] bitwise ops for bytes of equal length

Cameron Simpson added the comment: I'd like speak my support for bitwise ops on bytes and bytearray (agree, on equal lengths only). I've got 2 arguments here: - readability: a ^ b, a | b and so forth are clear and direct - all the various incantation presented must be _understood_, not to

[issue19251] bitwise ops for bytes of equal length

Changes by Cameron Simpson : -- nosy: +cameron ___ Python tracker ___ ___ Python-bugs-list

[issue19251] bitwise ops for bytes of equal length

STINNER Victor added the comment: >> If yes, maybe we should start with a module on PyPI. Is there a volunteer to >> try this option? > > bitsets – ordered subsets over a predefined domain This is an array of *bits*, not an array of bytes (or integers). > bitarray – efficient boolean array

[issue19251] bitwise ops for bytes of equal length

cowlicks added the comment: I'll look through the list @serhiy.storchaka posted and make sure this still seems sane to me. -- ___ Python tracker ___

[issue19251] bitwise ops for bytes of equal length

cowlicks added the comment: @gvanrossum in this previous comment https://bugs.python.org/issue19251?@ok_message=msg%20264184%20created%0Aissue%2019251%20message_count%2C%20messages%20edited%20ok&@template=item#msg257964 I pointed out code from the wild which would be more readable, and posted

[issue19251] bitwise ops for bytes of equal length

Serhiy Storchaka added the comment: > If yes, maybe we should start with a module on PyPI. Is there a volunteer to > try this option? bitsets – ordered subsets over a predefined domain bitarray – efficient boolean array implemented as C extension bitstring – pure-Python bit string based on

[issue19251] bitwise ops for bytes of equal length

Gregory P. Smith added the comment: I have wanted bytes/bytearray bit operations (be sure to include the in place operators for bytearray) when using micropython where it is normal to be operating on binary data. that said, i'd need someone from micropython to chime in as to if they can

[issue19251] bitwise ops for bytes of equal length

Guido van Rossum added the comment: Can you point to some examples of existing code that would become more readable and faster when this feature exists? Separately, how is it more secure? On Mon, Apr 25, 2016 at 9:17 AM, cowlicks wrote: > > cowlicks added the comment:

[issue19251] bitwise ops for bytes of equal length

STINNER Victor added the comment: > I like to add the bitwise protocol between byte objects of equal length Would it make sense to add such operators in a new module like the existing (but deprecated) audioop module? If yes, maybe we should start with a module on PyPI. Is there a volunteer to

[issue19251] bitwise ops for bytes of equal length

cowlicks added the comment: To reiterate, this issue would make more readable, secure, and speed up a lot of code. The concerns about this being a numpy-like vector operation are confusing to me. The current implementation is already vector-like, but lacks size checking. Isn't "int ^ int"

[issue19251] bitwise ops for bytes of equal length

Марк Коренберг added the comment: in order to increase perofrmance even more, use block operation on bytes. I.e. Xor by 8 bytes first (on 64-bit system) while size remainig is bigger or equal to 8, then by 4 bytes using same loop, and then xor remaining bytes by one byte. This will increase

[issue19251] bitwise ops for bytes of equal length

Changes by Gregory P. Smith : -- nosy: +gregory.p.smith ___ Python tracker ___ ___

[issue19251] bitwise ops for bytes of equal length

cowlicks added the comment: @Andrew Barnert > Maybe if you're coming to Python from... I'm not sure if your trying argue that my expectations are unusual? Python is my first programming language. To reiterate: I expected cpython to support bitwise operations on binary data. I don't think that

[issue19251] bitwise ops for bytes of equal length

cowlicks added the comment: I've attached a diff that adds ^, |, and & to bytes and bytearray object on the master branch (err the analogous hg thing). It also includes a test file which definitely is in the wrong place, but demonstrates what is working. Personally this came up while I was

[issue19251] bitwise ops for bytes of equal length

Andrew Barnert added the comment: On Jan 10, 2016, at 06:48, cowlicks wrote: > > > Personally this came up while I was playing with toy crypto problems. I > expected to already be part of the language, but it wasn't. I think this is a > natural expectation. Maybe if

[issue19251] bitwise ops for bytes of equal length

Christian Heimes added the comment: Serhiy’s bitarray idea has another benefit. Bit masking is only haft of the story. Since Python's int type has no length information, it not possible to handle shifting with overflow and rotation. A bit array can provide bit shifting ops like rshift,

[issue19251] bitwise ops for bytes of equal length

Andrew Barnert added the comment: There are a number of existing libraries on PyPI for bit arrays and bit sets. There are a lot more API choices than you'd think of in advance, and between them they explore most of the useful space. And I don't think any of them need to be in the stdlib.

[issue19251] bitwise ops for bytes of equal length

Martin Panter added the comment: FWIW a long time ago I wanted fast XORing of 512-byte “sectors” of Rar files. Initially I think I used array.array with the largest word size available, or numpy if available. Later when I learnt more Python I discovered the int.from_bytes() trick, and used

[issue19251] bitwise ops for bytes of equal length

Марк Коренберг added the comment: Just mention it here :) https://github.com/KeepSafe/aiohttp/issues/686 -- nosy: +mmarkk ___ Python tracker ___

[issue19251] bitwise ops for bytes of equal length

Guido van Rossum added the comment: I'm very skeptical of this. I expect it would cause quite a few surprises for people who aren't used to bitmask operations on integers, let alone on (byte) strings. -- nosy: +gvanrossum ___ Python tracker

[issue19251] bitwise ops for bytes of equal length

Andrew Barnert added the comment: For what it's worth, I looked at some old code (a clean re-implementation of a crypto algorithm in Python, used as a unit test for production code in C++) and found this: class BytesBits(bytes): # from a quick test, 1.12us in NumPy, 1.39us in C, 2.55us

[issue19251] bitwise ops for bytes of equal length

Changes by Josh Rosenberg shadowran...@gmail.com: -- nosy: +ShadowRanger ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue19251 ___ ___

[issue19251] bitwise ops for bytes of equal length

Changes by Martin Panter vadmium...@gmail.com: -- nosy: +vadmium ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue19251 ___ ___ Python-bugs-list

[issue19251] bitwise ops for bytes of equal length

STINNER Victor added the comment: You got me, Antoine! I'm working on a Python-only implementation of PBKDF2_HMAC. It involves XOR of two bytes in one place. If you want super-fast code, you should probably reimplement it in C. Python is not designed for performances... --

[issue19251] bitwise ops for bytes of equal length

Christian Heimes added the comment: I see that the feature idea is more controversial than I initially expected. It's probably best to have a long bike-shedding discussion on Python-ideas... :) Right now from_bytes/to_bytes trick is fast enough for my needs anyway. Therefore I'm deferring the

[issue19251] bitwise ops for bytes of equal length

Changes by STINNER Victor victor.stin...@gmail.com: -- nosy: +haypo ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue19251 ___ ___ Python-bugs-list

[issue19251] bitwise ops for bytes of equal length

Ramchandra Apte added the comment: On 21 October 2013 14:45, Christian Heimes rep...@bugs.python.org wrote: Christian Heimes added the comment: I see that the feature idea is more controversial than I initially expected. It's probably best to have a long bike-shedding discussion on

[issue19251] bitwise ops for bytes of equal length

Serhiy Storchaka added the comment: By the way I'd also be happy with a set of vector ops in the operator module, e.g. vector_xor(a, b). This is one direction. Other direction is adding the bitarray or bitset container and the bitview adapter. --

[issue19251] bitwise ops for bytes of equal length

Raymond Hettinger added the comment: Christian, we need multiple motivating use cases to warrant API expansion for fundamental types. I'm concerned about starting to move numpy vector-op functionality into the core of Python because it optimizes your one use crypto use case. -- nosy:

[issue19251] bitwise ops for bytes of equal length

Christian Heimes added the comment: bytes(x ^ y for x, y in zip(a, b)) is super-slow if you have to do XOR inside a hot loop for a couple of ten thousand times. int.from_bytes + int.to_bytes is about ten times faster. I expect bitwise ops of bytes to be even faster and more readable. $

[issue19251] bitwise ops for bytes of equal length

Terry J. Reedy added the comment: 'XOR of two bytes in one place' strikes me as a thin excuse for a new feature that abbreviates a simple, short, one-liner. To me, bytes(x ^ y for x, y in zip(a, b)) looks fine. a and b can be any iterables of ints. -- nosy: +terry.reedy

[issue19251] bitwise ops for bytes of equal length

Ramchandra Apte added the comment: On 19 October 2013 04:56, Terry J. Reedy rep...@bugs.python.org wrote: Terry J. Reedy added the comment: 'XOR of two bytes in one place' strikes me as a thin excuse for a new feature that abbreviates a simple, short, one-liner. To me, bytes(x ^ y for x,

[issue19251] bitwise ops for bytes of equal length

New submission from Christian Heimes: I like to propose a new feature for bytes and perhaps bytearray. None of the bytes types support bitwise operations like , | and ^. I like to add the bitwise protocol between byte objects of equal length: a, b = b123, babc bytes(x ^ y for x, y in zip(a,

[issue19251] bitwise ops for bytes of equal length

Antoine Pitrou added the comment: bytearray should certainly get the same functionality as bytes. -- nosy: +pitrou ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue19251 ___

[issue19251] bitwise ops for bytes of equal length

Georg Brandl added the comment: I assume you have a use case? -- nosy: +georg.brandl ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue19251 ___ ___

[issue19251] bitwise ops for bytes of equal length

Serhiy Storchaka added the comment: Use int's. (int.from_bytes(a, 'big') ^ int.from_bytes(b, 'big')).to_bytes(len(a), 'big') Adding and | operations to bytes will be confused because bytes is a sequence and this will conflict with set operations. -- nosy: +serhiy.storchaka

[issue19251] bitwise ops for bytes of equal length

Antoine Pitrou added the comment: The int-based spelling isn't very pretty though. And sequences are not sets :-) I suppose the use case has something to do with cryptography? -- ___ Python tracker rep...@bugs.python.org

[issue19251] bitwise ops for bytes of equal length

Serhiy Storchaka added the comment: When you work with int's instead of bytes it is spelled pretty enough. Sets are sequences. So having for example iteration and | operation in one function we can't be sure what it means. Please don't turn Python to APL or Perl. Perhaps separated mutable

[issue19251] bitwise ops for bytes of equal length

Antoine Pitrou added the comment: Sets are sequences. from collections import abc issubclass(abc.Set, abc.Sequence) False issubclass(abc.Sequence, abc.Set) False Perhaps separated mutable bitset (or bitlist?) type will be more useful. Then I would prefer a view, using e.g. the buffer

[issue19251] bitwise ops for bytes of equal length

Christian Heimes added the comment: You got me, Antoine! I'm working on a Python-only implementation of PBKDF2_HMAC. It involves XOR of two bytes in one place. Serhiy, I'm not going to turn Python into Perl. I merely like to provide a simple and well defined feature with existing syntax. I

[issue19251] bitwise ops for bytes of equal length

Ramchandra Apte added the comment: +1 -- nosy: +Ramchandra Apte ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue19251 ___ ___ Python-bugs-list