[issue23985] Crash when deleting slices from duplicated bytearray
Roundup Robot added the comment: New changeset 274c1b0a2494 by Serhiy Storchaka in branch '2.7': Issue #23985: Fixed integer overflow in iterator object. Original patch by https://hg.python.org/cpython/rev/274c1b0a2494 New changeset 5b86a1abc8c3 by Serhiy Storchaka in branch '3.4': Issue #23985: Fixed integer overflow in iterator object. Patch by https://hg.python.org/cpython/rev/5b86a1abc8c3 New changeset 9f2a1d9d7164 by Serhiy Storchaka in branch 'default': Issue #23985: Fixed integer overflow in iterator object. Patch by https://hg.python.org/cpython/rev/9f2a1d9d7164 -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue23985 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue23985] Crash when deleting slices from duplicated bytearray
Serhiy Storchaka added the comment: Sorry, it was related to issue22939. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue23985 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue23985] Crash when deleting slices from duplicated bytearray
Martin Panter added the comment: Posting a new patch which combines both fixes and adds some test cases. However the test needs Python to be built with “./configure --with-pydebug” to detect the buffer overrun; without this the test will probably silently pass. I removed the offending buffer space check, and let it always call PyBufferArray_Resize(). I also looked around the bytearray module for similar errors for other operations but I couldn’t find any. The other cases already tend to always call PyByteArray_Resize(). -- Added file: http://bugs.python.org/file39432/bytearray-fixes.v2.patch ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue23985 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue23985] Crash when deleting slices from duplicated bytearray
Roundup Robot added the comment: New changeset 98c1201d8eea by Antoine Pitrou in branch '3.4': Issue #23985: Fix a possible buffer overrun when deleting a slice from the front of a bytearray and then appending some other bytes data. https://hg.python.org/cpython/rev/98c1201d8eea New changeset 06fab9093973 by Antoine Pitrou in branch 'default': Issue #23985: Fix a possible buffer overrun when deleting a slice from the front of a bytearray and then appending some other bytes data. https://hg.python.org/cpython/rev/06fab9093973 -- nosy: +python-dev ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue23985 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue23985] Crash when deleting slices from duplicated bytearray
Antoine Pitrou added the comment: I've committed the patch. Thanks, Martin! -- resolution: - fixed stage: patch review - resolved status: open - closed ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue23985 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue23985] Crash when deleting slices from duplicated bytearray
Changes by Steve Dower steve.do...@microsoft.com: -- nosy: -steve.dower ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue23985 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue23985] Crash when deleting slices from duplicated bytearray
Martin Panter added the comment: Antoine, would you have a chance to review my patches? I assume you were responsible for adding the ob_start field. It would be nice to see this bug fixed in the next 3.4 and 3.5 releases. As well as the original poster’s problem, I suspect this bug may be the cause of some occasional strange behaviour I have seen in my own bytearray() FIFO type code. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue23985 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue23985] Crash when deleting slices from duplicated bytearray
Antoine Pitrou added the comment: Sorry. I'll take a look! -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue23985 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue23985] Crash when deleting slices from duplicated bytearray
Johan Dahlberg added the comment: Thank you all for working really fast on this issue! I'm happy to see that a fix is already being tried out. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue23985 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue23985] Crash when deleting slices from duplicated bytearray
Martin Panter added the comment: A test case for this that would trigger when memory debugging is enabled could look something like the following. Would it be appropriate to add it to the test suite? a = bytearray(10) size = sys.getsizeof(a) a.pop() # Defeat expanding buffer off-by-one quirk self.assertEqual(sys.getsizeof(a), size, Quirk not defeated) del a[:1] # Or a.pop(0) # Does not trigger bug # Or a[:1] = () # Triggers bug self.assertEqual(sys.getsizeof(a), size, Test assumes buffer not resized) a += bytes(2) # Add exactly the number of free bytes in buffer # Or a.extend(bytes(2)) # Unaffected # Or a.append(0); a.append(0) # Unaffected # Or a[8:] = bytes(2) # Unaffected del a # Trigger memory buffer to be freed, with verification -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue23985 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue23985] Crash when deleting slices from duplicated bytearray
Martin Panter added the comment: This bug might have been caused by the changes for Issue 19087, so I left a note there. It looks like that issue added the ob_start field to bytearray() objects, so that deleting from the start does not require memory copying. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue23985 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue23985] Crash when deleting slices from duplicated bytearray
New submission from Johan Dahlberg: Python 3.4.3 crashes after some time when running the attached program under Windows 7. The program appends a fixed bytes string to two independent bytearray buffers. If slices are removed from the beginnging of the two buffers and the two buffers are print:ed, the program will crash at some random occation. -- components: Windows files: bytearray_bug.py messages: 241316 nosy: johan, steve.dower, tim.golden, zach.ware priority: normal severity: normal status: open title: Crash when deleting slices from duplicated bytearray type: crash versions: Python 3.4 Added file: http://bugs.python.org/file39086/bytearray_bug.py ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue23985 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue23985] Crash when deleting slices from duplicated bytearray
Ned Deily added the comment: Reproduced with top of trunk default branch on OS X (so, not Windows-specific). With debug enabled: $ ./bin/python3.5 /tmp/b/bytearray_bug.py buf2: 13 5 bytearray(b'1234567890123') buf1: 13 3 bytearray(b'1234567890123') buf2: 21 2 bytearray(b'678901231234567890123') buf1: 23 5 bytearray(b'45678901231234567890123') buf2: 32 28 bytearray(b'89012312345678901231234567890123') buf1: 31 21 bytearray(b'9012312345678901231234567890123') buf2: 17 10 bytearray(b'01231234567890123') buf1: 23 9 bytearray(b'45678901231234567890123') buf2: 20 3 bytearray(b'78901231234567890123') buf1: 27 14 bytearray(b'312345678901231234567890123') buf2: 30 22 bytearray(b'012312345678901231234567890123') buf1: 26 17 bytearray(b'12345678901231234567890123') buf2: 21 1 bytearray(b'678901231234567890123') buf1: 22 12 bytearray(b'5678901231234567890123') buf2: 33 13 bytearray(b'789012312345678901231234567890123') buf1: 23 4 bytearray(b'45678901231234567890123') buf2: 33 23 bytearray(b'789012312345678901231234567890123') buf1: 32 4 bytearray(b'89012312345678901231234567890123') buf2: 23 19 bytearray(b'45678901231234567890123') buf1: 41 36 bytearray(b'23123456789012312345678901231234567890123') buf2: 17 3 bytearray(b'01231234567890123') buf1: 18 3 bytearray(b'901231234567890123') buf2: 27 23 bytearray(b'312345678901231234567890123') buf1: 28 22 bytearray(b'2312345678901231234567890123') buf2: 17 2 bytearray(b'01231234567890123') buf1: 19 5 bytearray(b'8901231234567890123') buf2: 28 10 bytearray(b'2312345678901231234567890123') buf1: 27 2 bytearray(b'312345678901231234567890123') buf2: 31 7 bytearray(b'9012312345678901231234567890123') buf1: 38 2 bytearray(b'23456789012312345678901231234567890123') buf2: 37 23 bytearray(b'3456789012312345678901231234567890123') buf1: 49 2 bytearray(b'4567890123123456789012312345678901231234567890123') buf2: 27 21 bytearray(b'312345678901231234567890123') buf1: 60 1 bytearray(b'678901231234567890123123456789012312345678901231234567890123') Debug memory block at address p=0x10bd238e0: API 'o' 61 bytes originally requested The 7 pad bytes at p-7 are FORBIDDENBYTE, as expected. The 8 pad bytes at tail=0x10bd2391d are not all FORBIDDENBYTE (0xfb): at tail+0: 0x33 *** OUCH at tail+1: 0x00 *** OUCH at tail+2: 0xfb at tail+3: 0xfb at tail+4: 0xfb at tail+5: 0xfb at tail+6: 0xfb at tail+7: 0xfb The block was made by call #44842 to debug malloc/realloc. Data at p: 34 35 36 37 38 39 30 31 ... 35 36 37 38 39 30 31 32 Fatal Python error: bad trailing pad byte Current thread 0x7fff70e18300 (most recent call first): File /tmp/b/bytearray_bug.py, line 25 in module Abort trap: 6 -- components: +Interpreter Core -Windows nosy: +ned.deily stage: - needs patch versions: +Python 3.5 ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue23985 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue23985] Crash when deleting slices from duplicated bytearray
Ned Deily added the comment: Above was in 64-bit mode (x86_64). Same run in 32-bit (i386): $ ./bin/python3.5-32 /tmp/b/bytearray_bug.py buf2: 13 2 bytearray(b'1234567890123') buf1: 13 11 bytearray(b'1234567890123') buf2: 24 16 bytearray(b'345678901231234567890123') buf1: 15 4 bytearray(b'231234567890123') buf2: 21 16 bytearray(b'678901231234567890123') buf1: 24 3 bytearray(b'345678901231234567890123') buf2: 18 1 bytearray(b'901231234567890123') buf1: 34 28 bytearray(b'6789012312345678901231234567890123') buf2: 30 21 bytearray(b'012312345678901231234567890123') buf1: 19 8 bytearray(b'8901231234567890123') buf2: 22 6 bytearray(b'5678901231234567890123') buf1: 24 11 bytearray(b'345678901231234567890123') buf2: 29 23 bytearray(b'12312345678901231234567890123') buf1: 26 15 bytearray(b'12345678901231234567890123') buf2: 19 15 bytearray(b'8901231234567890123') buf1: 24 4 bytearray(b'345678901231234567890123') buf2: 17 7 bytearray(b'01231234567890123') buf1: 33 21 bytearray(b'789012312345678901231234567890123') buf2: 23 13 bytearray(b'45678901231234567890123') buf1: 25 20 bytearray(b'2345678901231234567890123') buf2: 23 8 bytearray(b'45678901231234567890123') buf1: 18 7 bytearray(b'901231234567890123') buf2: 28 3 bytearray(b'2312345678901231234567890123') buf1: 24 8 bytearray(b'345678901231234567890123') buf2: 38 1 bytearray(b'23456789012312345678901231234567890123') buf1: 29 7 bytearray(b'12312345678901231234567890123') buf2: 50 38 bytearray(b'34567890123123456789012312345678901231234567890123') buf1: 35 7 bytearray(b'56789012312345678901231234567890123') buf2: 25 4 bytearray(b'2345678901231234567890123') buf1: 41 20 bytearray(b'23123456789012312345678901231234567890123') buf2: 34 24 bytearray(b'6789012312345678901231234567890123') buf1: 34 19 bytearray(b'6789012312345678901231234567890123') buf2: 23 1 bytearray(b'45678901231234567890123') buf1: 28 18 bytearray(b'2312345678901231234567890123') buf2: 35 20 bytearray(b'56789012312345678901231234567890123') buf1: 23 13 bytearray(b'45678901231234567890123') buf2: 28 16 bytearray(b'2312345678901231234567890123') buf1: 23 19 bytearray(b'45678901231234567890123') buf2: 25 16 bytearray(b'2345678901231234567890123') buf1: 17 4 bytearray(b'01231234567890123') buf2: 22 18 bytearray(b'5678901231234567890123') buf1: 26 18 bytearray(b'12345678901231234567890123') buf2: 17 14 bytearray(b'01231234567890123') buf1: 21 18 bytearray(b'678901231234567890123') buf2: 16 14 bytearray(b'1231234567890123') buf1: 16 11 bytearray(b'1231234567890123') buf2: 15 10 bytearray(b'231234567890123') buf1: 18 2 bytearray(b'901231234567890123') buf2: 18 2 bytearray(b'901231234567890123') buf1: 29 3 bytearray(b'12312345678901231234567890123') buf2: 29 11 bytearray(b'12312345678901231234567890123') buf1: 39 9 bytearray(b'123456789012312345678901231234567890123') buf2: 31 23 bytearray(b'9012312345678901231234567890123') buf1: 43 31 bytearray(b'0123123456789012312345678901231234567890123') Debug memory block at address p=0x7aec88: API 'o' 49 bytes originally requested The 3 pad bytes at p-3 are FORBIDDENBYTE, as expected. The 4 pad bytes at tail=0x7aecb9 are not all FORBIDDENBYTE (0xfb): at tail+0: 0x31 *** OUCH at tail+1: 0x32 *** OUCH at tail+2: 0x33 *** OUCH at tail+3: 0x00 *** OUCH The block was made by call #44809 to debug malloc/realloc. Data at p: 31 32 33 34 35 36 37 38 ... 33 34 35 36 37 38 39 30 Fatal Python error: bad trailing pad byte Current thread 0xa0ddc1d4 (most recent call first): File /tmp/b/bytearray_bug.py, line 25 in module Abort trap: 6 -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue23985 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue23985] Crash when deleting slices from duplicated bytearray
Wolfgang Maier added the comment: Also happening with Python 3.4.0 on Ubuntu 14.04 (after ~ half a minute and A LOT of output): [skipping lots of lines] buf2: 29 13 bytearray(b'12312345678901231234567890123') buf1: 25 9 bytearray(b'2345678901231234567890123') buf2: 29 2 bytearray(b'12312345678901231234567890123') buf1: 29 5 bytearray(b'12312345678901231234567890123') buf2: 40 8 bytearray(b'3123456789012312345678901231234567890123') buf1: 37 10 bytearray(b'3456789012312345678901231234567890123') buf2: 45 31 bytearray(b'890123123456789012312345678901231234567890123') buf1: 40 7 bytearray(b'3123456789012312345678901231234567890123') buf2: 27 3 bytearray(b'312345678901231234567890123') buf1: 46 9 bytearray(b'7890123123456789012312345678901231234567890123') buf2: 37 6 bytearray(b'3456789012312345678901231234567890123') buf1: 50 15 bytearray(b'34567890123123456789012312345678901231234567890123') buf2: 44 5 bytearray(b'90123123456789012312345678901231234567890123') buf1: 48 27 bytearray(b'567890123123456789012312345678901231234567890123') buf2: 52 3 bytearray(b'1234567890123123456789012312345678901231234567890123') buf1: 34 1 bytearray(b'6789012312345678901231234567890123') buf2: 62 16 bytearray(b'45678901231234567890123123456789012312345678901231234567890123') buf1: 46 13 bytearray(b'7890123123456789012312345678901231234567890123') *** Error in `python3': realloc(): invalid pointer: 0x7f2a4558 *** -- nosy: +wolma ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue23985 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue23985] Crash when deleting slices from duplicated bytearray
Changes by Serhiy Storchaka storch...@gmail.com: -- nosy: +pitrou, serhiy.storchaka priority: normal - high ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue23985 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue23985] Crash when deleting slices from duplicated bytearray
Alexei Romanov added the comment: No problem with python 2.7.6 on Ubuntu 14.04.2 LTS amd64. Attached script was working about 2 hours - no crash. -- nosy: +alexei.romanov ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue23985 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue23985] Crash when deleting slices from duplicated bytearray
Wolfgang Maier added the comment: Surprisingly, a much simpler version with just one bytearray seems to run stably (for several minutes at least), but when you wait a while then hit Ctrl-C, you are getting a Segmentation fault: Python 3.4.0 (default, Apr 11 2014, 13:05:11) [GCC 4.8.2] on linux Type help, copyright, credits or license for more information. import random buf1 = bytearray() data = b1234567890123 while True: ... buf1 += data ... l = len(buf1) ... n = random.randrange(1, l-1) ... del buf1[:n] ... ^CSegmentation fault (core dumped) The same code crashes spontaneously (without attempting a keyboard interrupt) when run in IDLE. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue23985 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue23985] Crash when deleting slices from duplicated bytearray
Terry J. Reedy added the comment: Win7, 64 bit, 3.5.0a3: crash in about 10 seconds. -- nosy: +terry.reedy ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue23985 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue23985] Crash when deleting slices from duplicated bytearray
Martin Panter added the comment: Here is a reduced version without using random numbers that reliably crashes for me on 64-bit Linux. Hopefully it also crashes for others, and they can investigate further. Crashes for me with the standard Arch Linux binary: Python 3.4.3 (default, Feb 26 2015, 23:01:07) [GCC 4.9.2 20150204 (prerelease)] on linux and my recently built v3.5 version: Python 3.5.0a3+ (default:0b3027a2abbc, Apr 11 2015, 23:27:07) [GCC 4.9.1] on linux It does not crash on 32 bit computer I tried though. -- nosy: +vadmium Added file: http://bugs.python.org/file39094/linux-x86-64.py ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue23985 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue23985] Crash when deleting slices from duplicated bytearray
Martin Panter added the comment: After cleaning my build and rebuilding with “./configure --with-pymalloc --with-pydebug”, I reduced my script to these four lines: b1 = bytearray(babcdefghij) # 10 bytes del b1[:1] del b1[:1] b1 += bklmnopq # 7 bytes Patch bytearray-fix.patch fixes the bug by taking account fact that ob_start is offset into the allocated memory buffer when checking if a reallocation is necessary. Explanation with the unpatched code and my four-line script above: 1. First line allocates 11 bytes of memory (10 for the byte string + 1 NUL terminator) 2. First “del” reduces the bytearray length to 9 bytes, but actually reallocates an expanded memory buffer of 16 bytes! (quirky but not the bug) 3. Second “del” reduces the bytearray length to 8 bytes, and increments an internal ob_start offset without any memory copying or reallocation. (Fine.) 4. Appending step needs to add 7 bytes to the 8-byte array. 7 + 8 + 1 is 16 bytes total required for bytearray and NUL terminator, but since ob_start is offset from the start of the allocated memory buffer, we overwrite past the end of the buffer. Memory debugging output and extra debug printfs of my own: Resizing to size 10 (current log. offset 0 alloc 0) = Major upsize, new alloc = 11 Assigning to linear slice - Shifting ob_start for negative growth -1 Resizing to size 9 (current log. offset 1 alloc 11) = Moderate upsize, new alloc = 16 - Done assigning to linear slice Assigning to linear slice - Shifting ob_start for negative growth -1 Resizing to size 8 (current log. offset 1 alloc 16) = Minor downsize - Done assigning to linear slice Debug memory block at address p=0x7f1af630a0d0: API 'o' 16 bytes originally requested The 7 pad bytes at p-7 are FORBIDDENBYTE, as expected. The 8 pad bytes at tail=0x7f1af630a0e0 are not all FORBIDDENBYTE (0xfb): at tail+0: 0x00 *** OUCH at tail+1: 0xfb at tail+2: 0xfb at tail+3: 0xfb at tail+4: 0xfb at tail+5: 0xfb at tail+6: 0xfb at tail+7: 0xfb The block was made by call #32897 to debug malloc/realloc. Data at p: 62 63 64 65 66 67 68 69 6a 6b 6c 6d 6e 6f 70 71 Fatal Python error: bad trailing pad byte -- keywords: +patch Added file: http://bugs.python.org/file39098/bytearray-fix.patch ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue23985 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue23985] Crash when deleting slices from duplicated bytearray
Martin Panter added the comment: Posting bytearray-resize.patch which stops “del” from expanding the allocated buffer. This one is not necessary to fix the reported bug, but avoids the separate quirk identified in step 2. -- stage: needs patch - patch review Added file: http://bugs.python.org/file39099/bytearray-resize.patch ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue23985 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
