[issue18843] Py_FatalError (msg=0x7f0e3b373232 "bad leading pad byte") at Python-2.7.5/Python/pythonrun.c:1689

Charles-François Natali added the comment: Thanks for the feedback! -- resolution: -> invalid stage: -> committed/rejected status: open -> closed ___ Python tracker ___ ___

[issue18843] Py_FatalError (msg=0x7f0e3b373232 "bad leading pad byte") at Python-2.7.5/Python/pythonrun.c:1689

Martin Mokrejs added the comment: Hi, I think I should report back what I found on the hardware side. While memory testing tools like memtest86+ and other did not find any error, the built in Dell ePSA test suite likely does compute a checksum of tested memory regions. It reported some addre

[issue18843] Py_FatalError (msg=0x7f0e3b373232 "bad leading pad byte") at Python-2.7.5/Python/pythonrun.c:1689

Changes by Charles-François Natali : -- status: open -> pending ___ Python tracker ___ ___ Python-bugs-list mailing list Unsubscribe:

[issue18843] Py_FatalError (msg=0x7f0e3b373232 "bad leading pad byte") at Python-2.7.5/Python/pythonrun.c:1689

Stephen J. Turnbull added the comment: Yeah, hope is a good thing. But I've spent the last 20 years debugging an X11 application based on a Lisp interpreter, I save hope for fireflies, my dog, and my daughter these days. :-) To the OP: I don't follow Gentoo closely, but I have acquaintances

[issue18843] Py_FatalError (msg=0x7f0e3b373232 "bad leading pad byte") at Python-2.7.5/Python/pythonrun.c:1689

Tim Peters added the comment: Thanks for that, Stephen! I don't know of anything else you could try that would be helpful. The OP doesn't appear able to reproduce his problems either, and last I heard he was off running `emerge` under DUMA: http://duma.sourceforge.net/ Why? Hope spring

[issue18843] Py_FatalError (msg=0x7f0e3b373232 "bad leading pad byte") at Python-2.7.5/Python/pythonrun.c:1689

Stephen J. Turnbull added the comment: OK, I backed off the aggressive CFLAGS/CXXFLAGS to " -ggdb -pipe", and ran "emerge =dev-lang/python-2.7.5-r1" *once* each with and without the 'EXTRA_ECONF="--with-pydebug"' flag. Compiled with GCC 4.7.3. No crash, same test results as described previous

[issue18843] Py_FatalError (msg=0x7f0e3b373232 "bad leading pad byte") at Python-2.7.5/Python/pythonrun.c:1689

Tim Peters added the comment: Someone may find the new stress.valgrind.stderr interesting, but - since I've never used valgrind - it doesn't mean much to me. I _expected_ you'd run the little stress program under a debug Python and without valgrind, since that's the only combination you've tri

[issue18843] Py_FatalError (msg=0x7f0e3b373232 "bad leading pad byte") at Python-2.7.5/Python/pythonrun.c:1689

Stephen J. Turnbull added the comment: I did "emerge =dev-lang/python-2.7.5-r1" *twice* with the environment configuration described in msg196520, then *once* with it disabled because one of the cases you described was when you tried to revert to a non-debug Python. (Besides, I am willing to

[issue18843] Py_FatalError (msg=0x7f0e3b373232 "bad leading pad byte") at Python-2.7.5/Python/pythonrun.c:1689

Antoine Pitrou added the comment: Hardware failures or glitches can involve any number of parameters. Instead of trying to reproduce the memory corruption with other Python programs, I would suggest you run the exact same workload (including Python version and build options) on another computer

[issue18843] Py_FatalError (msg=0x7f0e3b373232 "bad leading pad byte") at Python-2.7.5/Python/pythonrun.c:1689

Martin Mokrejs added the comment: So with your test program I did not yet hit the error. It used in the end 1.3GB of RAM, maybe in the good region. while this was python-2.7.5 configures as --without-pymalloc maybe the output from valgrind becomes more useful. -- Added file: http://bug

[issue18843] Py_FatalError (msg=0x7f0e3b373232 "bad leading pad byte") at Python-2.7.5/Python/pythonrun.c:1689

Tim Peters added the comment: OK, it sounds to me like you do not have a reproducible test case, of any kind. It that's true, this bug report isn't going anywhere :-( Python isn't a memory-testing program, so it would be insanely inefficient for it to (for example) read up every byte it write

[issue18843] Py_FatalError (msg=0x7f0e3b373232 "bad leading pad byte") at Python-2.7.5/Python/pythonrun.c:1689

Martin Mokrejs added the comment: I think it would be of tremendous help if python reported the physical memory address though do not know how much work is that for you. I already said that, I wonder why memtest86+ could not find an error, I even tried other tools for that. Could --with-pydebu

[issue18843] Py_FatalError (msg=0x7f0e3b373232 "bad leading pad byte") at Python-2.7.5/Python/pythonrun.c:1689

Tim Peters added the comment: Martin, would it be possible to borrow someone else's machine and try to reproduce this? If you can, that would greatly reduce the probability of this being a HW error. It would also leave us with an exact set of commands to share so others can try it on their b

[issue18843] Py_FatalError (msg=0x7f0e3b373232 "bad leading pad byte") at Python-2.7.5/Python/pythonrun.c:1689

Charles-François Natali added the comment: 2013/8/30 Martin Mokrejs : > Per comment from Charles-François, so you mean that this single-bit change > won't be caught by valgrind, right? Why does not memtest86+ detect that? > Could python when compiled with the --with-pydebug print also physical,

[issue18843] Py_FatalError (msg=0x7f0e3b373232 "bad leading pad byte") at Python-2.7.5/Python/pythonrun.c:1689

Stefan Krah added the comment: I uderstand that you are building Python using "emerge". I would try to get a release from python.org, do a normal build ... make distclean ./configure --prefix=/tmp --with-pydebug --with-valgrind make make install ... then install matplotlib using /tmp/bin/pyt

[issue18843] Py_FatalError (msg=0x7f0e3b373232 "bad leading pad byte") at Python-2.7.5/Python/pythonrun.c:1689

Martin Mokrejs added the comment: Hi Stephen, I discussed the USE=debug here (https://bugs.gentoo.org/show_bug.cgi?id=482348) and it is denied by portage maintainers because it is not only a debug addition but a whole API change. We have to live with: mkdir -p /etc/portage/env echo 'EXTRA_E

[issue18843] Py_FatalError (msg=0x7f0e3b373232 "bad leading pad byte") at Python-2.7.5/Python/pythonrun.c:1689

Charles-François Natali added the comment: Just two things: - running under valgrind with the suppression file would help pinpoint this - usually, stack/heap overflows or invalid pointer dereference affect a contiguous chunk of memory: here, there's a *single bit* flipped, not even a byte 0xfb

[issue18843] Py_FatalError (msg=0x7f0e3b373232 "bad leading pad byte") at Python-2.7.5/Python/pythonrun.c:1689

Tim Peters added the comment: Martin, can you please supply exact commands Stephen can use to try to reproduce the problem you saw running `emerge`? And try them yourself first, to make sure you can reproduce the problem too. Any detail may be important. For example, it's possible this is du

[issue18843] Py_FatalError (msg=0x7f0e3b373232 "bad leading pad byte") at Python-2.7.5/Python/pythonrun.c:1689

Tim Peters added the comment: Thanks for chiming in, Stephen! I can't answer your questions, but agree the original poster needs to tell you exactly what he did -- else we'd just be thrashing at random. There's been enough of that already ;-) -- _

[issue18843] Py_FatalError (msg=0x7f0e3b373232 "bad leading pad byte") at Python-2.7.5/Python/pythonrun.c:1689

Stephen J. Turnbull added the comment: I have a gentoo host, but I'd like to know how did the OP get a debug Python in the first place? The ebuild for python 2.7.5-r1 doesn't say anything about debug options. "How" would preferably include information about the C compiler used, etc. If ther

[issue18843] Py_FatalError (msg=0x7f0e3b373232 "bad leading pad byte") at Python-2.7.5/Python/pythonrun.c:1689

Tim Peters added the comment: I sent a msg to Python-Dev, asking for a Gentoo user to try to reproduce this. Thanks! -- ___ Python tracker ___ _

[issue18843] Py_FatalError (msg=0x7f0e3b373232 "bad leading pad byte") at Python-2.7.5/Python/pythonrun.c:1689

Martin Mokrejs added the comment: http://www.gentoo.org/proj/en/portage/index.xml http://dev.gentoo.org/~zmedico/portage/archives -- ___ Python tracker ___ __

[issue18843] Py_FatalError (msg=0x7f0e3b373232 "bad leading pad byte") at Python-2.7.5/Python/pythonrun.c:1689

Martin Mokrejs added the comment: That is why I asked if other process can interfere. So, they are isolated on Linux, good. ;-) The crash in #msg196481 is just the emerge written in python, at the start it is resolving some graph of package dependencies ... once it resolves order of packages a

[issue18843] Py_FatalError (msg=0x7f0e3b373232 "bad leading pad byte") at Python-2.7.5/Python/pythonrun.c:1689

Tim Peters added the comment: It would be a severely lame OS that allowed a process to overwrite another process's memory ;-) "Bad C or C++ code", in the process you're running, is still the best guess. A memory module that sometimes dropped the last bit _could_ be at fault, but I doubt it (

[issue18843] Py_FatalError (msg=0x7f0e3b373232 "bad leading pad byte") at Python-2.7.5/Python/pythonrun.c:1689

Martin Mokrejs added the comment: Thanks, will recompile without pymalloc. I don't understand why always is only 1 bit different. Could that be overwritten by another use process or do you believe it must be the python or some of the modules imported into it on runtime? -- __

[issue18843] Py_FatalError (msg=0x7f0e3b373232 "bad leading pad byte") at Python-2.7.5/Python/pythonrun.c:1689

Tim Peters added the comment: Yet Another Tool ;-) Python's "small object" allocator grabs memory in chunks of 256KB from the system, and carves up the space itself. Other memory tools (like Valgrind ...) only see that Python has grabbed 256KB chunks, so can't detect anything wrong _within_

[issue18843] Py_FatalError (msg=0x7f0e3b373232 "bad leading pad byte") at Python-2.7.5/Python/pythonrun.c:1689

Martin Mokrejs added the comment: Sorry, I should explain a bit more. Gentoo Linux uses a tool named emerge to handle packages. It is written in python. So, by that command I initiated re-compilation of python itself but it crashed quickly. At that moment emerge called the python available on

[issue18843] Py_FatalError (msg=0x7f0e3b373232 "bad leading pad byte") at Python-2.7.5/Python/pythonrun.c:1689

Tim Peters added the comment: Hmm. I don't quite know what you're doing: you said you're getting away from --with-pydebug, but these "bad leading pad byte" messages can't be generated unless Python is compiled with (at least) PYMALLOC_DEBUG defined. That said, my advice remains the same: wh

[issue18843] Py_FatalError (msg=0x7f0e3b373232 "bad leading pad byte") at Python-2.7.5/Python/pythonrun.c:1689

Martin Mokrejs added the comment: One more note. At that time I had running my application which at that time was parsing an XML file using xml.parsers.expat! That is being run in my pipeline before I render figures (the initially reported crash case). So, matplotlib/numpy is ruled out. -

[issue18843] Py_FatalError (msg=0x7f0e3b373232 "bad leading pad byte") at Python-2.7.5/Python/pythonrun.c:1689

Martin Mokrejs added the comment: I wanted to move away from the --with-pydebug to a normal python and I failed with: # emerge dev-lang/python:2.7 * IMPORTANT: 11 news items need reading for repository 'gentoo'. * Use eselect news to read news items. Calculating dependencies... done! Debug

[issue18843] Py_FatalError (msg=0x7f0e3b373232 "bad leading pad byte") at Python-2.7.5/Python/pythonrun.c:1689

Tim Peters added the comment: I opened issue 18859 about the lack of --with-valgrind info in Misc/README.valgrind. Thanks for noticing! -- ___ Python tracker ___ __

[issue18843] Py_FatalError (msg=0x7f0e3b373232 "bad leading pad byte") at Python-2.7.5/Python/pythonrun.c:1689

Martin Mokrejs added the comment: I was just checking whether configure picked up my --with-pymalloc and incidentally saw: --with-valgrind Enable Valgrind support maybe Misc/README.valgrind needs revision and should explain what that does as well? ;-) It should also explain what is

[issue18843] Py_FatalError (msg=0x7f0e3b373232 "bad leading pad byte") at Python-2.7.5/Python/pythonrun.c:1689

Tim Peters added the comment: I don't know why there isn't a configure switch for this - but then I've never used valgrind - LOL ;-) Other developers use valgrind on Python routinely, though. So it's unlikely you'll find a legitimate problem _in Python_ reported by valgrind. --

[issue18843] Py_FatalError (msg=0x7f0e3b373232 "bad leading pad byte") at Python-2.7.5/Python/pythonrun.c:1689

Martin Mokrejs added the comment: No, I did not know that. Thanks, I did now. * Uncomment Py_USING_MEMORY_DEBUGGER in Objects/obmalloc.c, then rebuild Python * Uncomment the lines in Misc/valgrind-python.supp that suppress the warnings for PyObject_Free and PyObject_Realloc Why i

[issue18843] Py_FatalError (msg=0x7f0e3b373232 "bad leading pad byte") at Python-2.7.5/Python/pythonrun.c:1689

Tim Peters added the comment: Did you read Misc/README.valgrind (in the Python tree)? The warnings you've seen so far are probably all nonsense, and README.valgrind explains how to avoid getting them. -- ___ Python tracker

[issue18843] Py_FatalError (msg=0x7f0e3b373232 "bad leading pad byte") at Python-2.7.5/Python/pythonrun.c:1689

Martin Mokrejs added the comment: I took a crack from another angle. I converted my application using cython and then used gcc. Finally, ran valgrind over the binary. It is not finished yet but already spotted plenty of hints. -- Added file: http://bugs.python.org/file31490/python2.7_

[issue18843] Py_FatalError (msg=0x7f0e3b373232 "bad leading pad byte") at Python-2.7.5/Python/pythonrun.c:1689

Tim Peters added the comment: Well, if you delete a giant list, and the list held the only references remaining to the objects the list contained, then the memory for those objects will be free'd, one object at a time. A debug build would then detect the memory corruption in those objects. B

[issue18843] Py_FatalError (msg=0x7f0e3b373232 "bad leading pad byte") at Python-2.7.5/Python/pythonrun.c:1689

Martin Mokrejs added the comment: Yes, I have rebuilt all python modules but even gdb exited on startup due to python ABI change. I am using Gentoo Linux (https://bugs.gentoo.org/show_bug.cgi?id=482348) and unless python-updater forgot to include some package in the listing of those needed to

[issue18843] Py_FatalError (msg=0x7f0e3b373232 "bad leading pad byte") at Python-2.7.5/Python/pythonrun.c:1689

Tim Peters added the comment: By the way, if memory serves, compiling with --with-pydebug changes the memory layout of Python objects, so a Python compiled this way _cannot_ be used successfully with extension modules that were compiled without the same options. Did you rebuild your extension

[issue18843] Py_FatalError (msg=0x7f0e3b373232 "bad leading pad byte") at Python-2.7.5/Python/pythonrun.c:1689

Tim Peters added the comment: Memory corruption can be difficult to track down. Best thing you can do is strive to find a test case as small and fast as possible that shows the same kind of error. By "rogue extension module" I just mean 3rd-party C code (like, for example, matplotlib). I do

[issue18843] Py_FatalError (msg=0x7f0e3b373232 "bad leading pad byte") at Python-2.7.5/Python/pythonrun.c:1689

Martin Mokrejs added the comment: Thank you for explanation what is going on. I called matplotlibs drawing function to include 49308 dots and corresponding legend items with their colors. That's all I can say. I am not a native English speaker so I don't know what 'rogue extension modules' mea

[issue18843] Py_FatalError (msg=0x7f0e3b373232 "bad leading pad byte") at Python-2.7.5/Python/pythonrun.c:1689

Changes by Antoine Pitrou : -- nosy: +haypo ___ Python tracker ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.

[issue18843] Py_FatalError (msg=0x7f0e3b373232 "bad leading pad byte") at Python-2.7.5/Python/pythonrun.c:1689

Tim Peters added the comment: Python's debug-mode memory allocators add some "magic values" before and after each allocated chunk of memory, and check them when the chunk is freed to make sure nobody overwrote them. In this case, someone did overwrite the byte at p-5, where p is the address o

[issue18843] Py_FatalError (msg=0x7f0e3b373232 "bad leading pad byte") at Python-2.7.5/Python/pythonrun.c:1689

Martin Mokrejs added the comment: Grr, forgot to look into a file where I recorded STDERR. Debug memory block at address p=0x449e6900: API 'o' 80 bytes originally requested The 7 pad bytes at p-7 are not all FORBIDDENBYTE (0xfb): at p-7: 0xfb at p-6: 0xfb at p-5:

[issue18843] Py_FatalError (msg=0x7f0e3b373232 "bad leading pad byte") at Python-2.7.5/Python/pythonrun.c:1689

Martin Mokrejs added the comment: Would you please guide me what gdb commands I should issue for you? Thank you. BTW, I ran memtest86+ few days ago, although this is non-ECC memory I think HW is fine. -- ___ Python tracker

[issue18843] Py_FatalError (msg=0x7f0e3b373232 "bad leading pad byte") at Python-2.7.5/Python/pythonrun.c:1689

Amaury Forgeot d'Arc added the comment: This is a memory corruption. Please look at the memory before the freed address (0x449e6900), maybe there is an indication of some buffer overflow? -- nosy: +amaury.forgeotdarc ___ Python tracker

[issue18843] Py_FatalError (msg=0x7f0e3b373232 "bad leading pad byte") at Python-2.7.5/Python/pythonrun.c:1689

Martin Mokrejs added the comment: Should have included from the head of gdb output: Program terminated with signal 6, Aborted. -- ___ Python tracker ___

[issue18843] Py_FatalError (msg=0x7f0e3b373232 "bad leading pad byte") at Python-2.7.5/Python/pythonrun.c:1689

New submission from Martin Mokrejs: Hi, it happened to me that using faulthandler and python compiled with --with-pydebug and C*FLAGS=-ggdb I got this stacktrace (will attach longer version as a file): (gdb) where #0 0x7f0e3af8aacb in raise () from /lib64/libpthread.so.0 #1 0x7f0e