[issue22939] integer overflow in iterator object

Serhiy Storchaka added the comment: Thank you for your contribution Clement. -- resolution: -> fixed stage: commit review -> resolved status: open -> closed ___ Python tracker _

[issue22939] integer overflow in iterator object

Roundup Robot added the comment: New changeset d6179accca20 by Serhiy Storchaka in branch '2.7': Fixed issue number for issue #22939. https://hg.python.org/cpython/rev/d6179accca20 New changeset 7fa2f4afcf5a by Serhiy Storchaka in branch '3.4': Fixed issue number for issue #22939. https://hg.pyt

[issue22939] integer overflow in iterator object

Serhiy Storchaka added the comment: If Raymond will not stop me, I'll commit the patch tomorrow. -- assignee: -> serhiy.storchaka ___ Python tracker ___

[issue22939] integer overflow in iterator object

Changes by Serhiy Storchaka : -- versions: +Python 2.7, Python 3.4 ___ Python tracker ___ ___ Python-bugs-list mailing list Unsubscrib

[issue22939] integer overflow in iterator object

Clement Rouault added the comment: After a few months, I am back to you on this issue. What should be the next step of the process ? -- status: pending -> open ___ Python tracker ___

[issue22939] integer overflow in iterator object

Serhiy Storchaka added the comment: > That way a virtual sequence with PY_SSIZE_T_MAX-1 items would still work > (instead of failing unexpectedly). Actually with PY_SSIZE_T_MAX+1 items (indices from 0 to PY_SSIZE_T_MAX inclusive). If Raymond insists I can write more complicated patch, but I d

[issue22939] integer overflow in iterator object

Ronald Oussoren added the comment: Is it necessary to raise when it_index is PY_SSIZE_T_MAX? An alternative is to set it_index to -1 when there would be overflow and raise an exception on the next call to next(). That way a virtual sequence with PY_SSIZE_T_MAX-1 items would still work (instea

[issue22939] integer overflow in iterator object

Clement Rouault added the comment: > > The call to PySequence_GetItem() may be expensive, and we have to drop > > the result if an OverflowError is raised after the call. > You do realize that this error will be very rare and therefore > inconsequential. The real question is: why would you cal

[issue22939] integer overflow in iterator object

Raymond Hettinger added the comment: > The call to PySequence_GetItem() may be expensive, and we have to drop > the result if an OverflowError is raised after the call. You do realize that this error will be very rare and therefore inconsequential. -- __

[issue22939] integer overflow in iterator object

STINNER Victor added the comment: I prefer to raise an OverflowError *before* calling PySequence_GetItem(). The call to PySequence_GetItem() may be expensive, and we have to drop the result if an OverflowError is raised after the call. At the end, the behaviour is the same: an OverflowError is ra

[issue22939] integer overflow in iterator object

Clement Rouault added the comment: > Also, PY_SSIZE_T_MAX is a valid value to pass to PySequence_GetItem(), so it > shouldn't be blocked unless necessary. I agree with you, that's why my first path was checking at the next call if it->it_index had overflowed. But then it relies on undefined be

[issue22939] integer overflow in iterator object

Serhiy Storchaka added the comment: This doesn't matter because next() will raise an exception if it_index == PY_SSIZE_T_MAX in any case. The code should be changed much more to allow yielding an item with index PY_SSIZE_T_MAX, use other (negative) signal value and change the behavior of __set

[issue22939] integer overflow in iterator object

Raymond Hettinger added the comment: I would think that the PY_SSIZE_T_MAX check belongs inside the: if (result != NULL) { it->it_index++; return result; } just before the increment which could cause the overflow. Also, PY_SSIZE_T_MAX is a valid value to pass to PySequ

[issue22939] integer overflow in iterator object

Serhiy Storchaka added the comment: Ah, __setstate__ already handles negative indices. Then the patch LGTM. -- stage: needs patch -> commit review ___ Python tracker ___

[issue22939] integer overflow in iterator object

Serhiy Storchaka added the comment: __setstate__ should check that an index is not negative. All values from 0 to PY_SSIZE_T_MAX are acceptable. -- ___ Python tracker ___ __

[issue22939] integer overflow in iterator object

Clement Rouault added the comment: Thanks for the comments. I corrected the patch and updated the test. I also added a test that check the behavior of setstate with negative indice. Just one question: > __setstate__ must implement the same check. Why should __setstate__ check for PY_SSIZE_T_

[issue22939] integer overflow in iterator object

Serhiy Storchaka added the comment: > check if the index is greater or equal than PY_SSIZET_MAX - 1. Just PY_SSIZET_MAX. Added other comments on Rietveld (look the email in the Spam folder). -- ___ Python tracker

[issue22939] integer overflow in iterator object

STINNER Victor added the comment: You should not rely on undefined behaviour: check if the index is greater or equal than PY_SSIZET_MAX - 1. __setstate__ must implement the same check. You must always raise an error, not hide it in a second call to next (raise StopIteration). Your unit test must

[issue22939] integer overflow in iterator object

Clement Rouault added the comment: Here is a first try for a patch. There are two points I am not sure about: 1) The message for the OverflowError: is that explicit enough ? 2) The behaviour of the iterator after the raise of OverflowError. With this patch every call to `next(it)` where `it`

[issue22939] integer overflow in iterator object

Serhiy Storchaka added the comment: > What is your use case? Something like range(). I agree that it is very unlike to encounter this problem in real work, and we can live with implementation-related limitation (for which OverflowError exists). --

[issue22939] integer overflow in iterator object

STINNER Victor added the comment: > I think OverflowError is good for maintained releases, but for 3.5 Clement's > idea with long index looks attractive to me. What is your use case? -- ___ Python tracker ___

[issue22939] integer overflow in iterator object

Serhiy Storchaka added the comment: I think OverflowError is good for maintained releases, but for 3.5 Clement's idea with long index looks attractive to me. In any case an exception should be raised for negative argument in __setstate__(). Let split this issue on two parts. First fix the bug

[issue22939] integer overflow in iterator object

STINNER Victor added the comment: > The `it_index` attribute used by the iterator is a `Py_ssize_t` but overflow > is never checked. Yes it is a bug. iter_iternext() must raises an OverflowError if it->it_index is equal to PY_SSIZE_T_MAX. -- nosy: +haypo _

[issue22939] integer overflow in iterator object

R. David Murray added the comment: For possibly relevant background information, see issue 21444 and the issues linked from it, and issue 14794. -- nosy: +r.david.murray ___ Python tracker

[issue22939] integer overflow in iterator object

Changes by Serhiy Storchaka : -- nosy: +rhettinger, serhiy.storchaka stage: -> needs patch ___ Python tracker ___ ___ Python-bugs-lis

[issue22939] integer overflow in iterator object

New submission from Clement Rouault: I found an interger overflow in the standard iterator object (Objects/iterobject.c) The `it_index` attribute used by the iterator is a `Py_ssize_t` but overflow is never checked. So after the index `PY_SSIZE_T_MAX`, the iterator object will ask for the ind