[issue27507] bytearray.extend lacks overflow check when increasing buffer

Xiang Zhang added the comment: Thanks for your work. -- ___ Python tracker ___ ___ Python-bugs-list mailing

[issue27507] bytearray.extend lacks overflow check when increasing buffer

Martin Panter added the comment: Committed without any macro for the time being -- resolution: not a bug -> fixed stage: patch review -> resolved status: open -> closed ___ Python tracker

[issue27507] bytearray.extend lacks overflow check when increasing buffer

Roundup Robot added the comment: New changeset 54cc0480904c by Martin Panter in branch '3.5': Issue #27507: Check for integer overflow in bytearray.extend() https://hg.python.org/cpython/rev/54cc0480904c New changeset 6e166b66aa44 by Martin Panter in branch '2.7': Issue #27507: Check for

[issue27507] bytearray.extend lacks overflow check when increasing buffer

Antti Haapala added the comment: Ah indeed, this is a bytearray and it is indeed possible to theoretically allocate PY_SSIZE_T_MAX bytes, if on an architecture that does segmented memory. As for if (addition > PY_SSIZE_T_MAX - len - 1) { it is very clear to *us* but it is not quite

[issue27507] bytearray.extend lacks overflow check when increasing buffer

Decorater added the comment: Only 1 way to find out, test it till it breaks. :P -- ___ Python tracker ___ ___

[issue27507] bytearray.extend lacks overflow check when increasing buffer

Martin Panter added the comment: Yeah I see your point. Anyway I think the current patch is fine. -- ___ Python tracker ___

[issue27507] bytearray.extend lacks overflow check when increasing buffer

Xiang Zhang added the comment: I can't totally agree the point. The code means every time we increase the buffer by half the current length. So when the length arrives 2/3 * PY_SSIZE_T_MAX it's going to overflow. There is a 1/3 * PY_SSIZE_T_MAX gap between the theoretical upper limit. I think

[issue27507] bytearray.extend lacks overflow check when increasing buffer

Martin Panter added the comment: Not particularly related, but the special fast case in Objects/listobject.c:811, listextend(), also seems to lack an overflow check. “An alternative would be to raise the error without trying to allocate Py_SSIZE_T_MAX first”: what I meant was removing the

[issue27507] bytearray.extend lacks overflow check when increasing buffer

Xiang Zhang added the comment: Thanks for the analysis Antti. I don't think if (len == PY_SSIZE_T_MAX) can be moved inside the *other* if. Their handlings are different. if (len == PY_SSIZE_T_MAX) is True, it should exit immediately. But in the *other* if, you can still have a try to allocate

[issue27507] bytearray.extend lacks overflow check when increasing buffer

Antti Haapala added the comment: if (len == PY_SSIZE_T_MAX) is necessary for the case that the iterable is already PY_SSIZE_T_MAX items. However it could be moved inside the *other* if because if (len == PY_SSIZE_T_MAX) should also fail the overflow check. However, I believe it is theoretical

[issue27507] bytearray.extend lacks overflow check when increasing buffer

Xiang Zhang added the comment: So would you like to merge it like this or Martin's alternative way? But honestly speaking I don't get Martin's point, "without trying to allocate Py_SSIZE_T_MAX first", where is this place? -- ___ Python tracker

[issue27507] bytearray.extend lacks overflow check when increasing buffer

Changes by Decorater : -- nosy: +Decorater ___ Python tracker ___ ___ Python-bugs-list

[issue27507] bytearray.extend lacks overflow check when increasing buffer

Serhiy Storchaka added the comment: Totally agreed with Martin. -- ___ Python tracker ___ ___

[issue27507] bytearray.extend lacks overflow check when increasing buffer

Xiang Zhang added the comment: Nice to see the real example. I don't think of that. -- ___ Python tracker ___

[issue27507] bytearray.extend lacks overflow check when increasing buffer

Martin Panter added the comment: It is possible to make an infinite iterable, e.g. iter(int, 1), so it is definitely worth checking for overflow. The patch looks okay to me. An alternative would be to raise the error without trying to allocate Py_SSIZE_T_MAX first, but I am okay with either

[issue27507] bytearray.extend lacks overflow check when increasing buffer

Xiang Zhang added the comment: Sorry, after checking again, this is still needed but maybe `if (len == PY_SSIZE_T_MAX)` part is not necessary since the iterable's length should not be larger than PY_SSIZE_T_MAX. But keep it to avoid theoretically bug is not a bad idea. -- status:

[issue27507] bytearray.extend lacks overflow check when increasing buffer

Xiang Zhang added the comment: Ohh, sorry for the disturb. I made a mistake. This is not needed and now close it. -- resolution: -> not a bug status: open -> closed ___ Python tracker

[issue27507] bytearray.extend lacks overflow check when increasing buffer

New submission from Xiang Zhang: As the title, bytearray.extend simply use `buf_size = len + (len >> 1) + 1;` to determine next *buf_size* when increasing buffer. But this can overflow in theory. So I suggest adding overflow check. -- components: Interpreter Core files: