On Fri, 18 Nov 2016 08:55:11 -0800 "Kevin O'Gorman" <kevinogorm...@gmail.com> wrote:
> All of the python code is a single thread. The closest I come > is a few times where I use subprocess.Popen to create what amounts to > a pipeline, and one place where I start a number of copies of a C > program in parallel, but each is a separate process with its own > input and output files. These C programs have been in use for a > number of months for earlier stages of this project, and I regard > them as quite reliable. None of them uses threads, and they are > mostly very simple filters. As you know, a process started with Popen cannot corrupt the Python process's memory. If you're not doing anything to defeat the GIL, a segfault inside the Python interpreter would be considered a bug. But is it happening in the interpreter, or in SQLite for that matter? ISTM that's what you need to know. To know that, you're going to need to run a debug version of the interpreter under gdb. When it faults, a backtrace will tell you where. That's not definititive proof; memory corruption is often detected far from where it was caused. But if the fault is at a consistent place in SQLite code, for example, you can use a hardware watchpoint to discover what's writing to it. I don't know what more to suggest. I would be surprised if you find a fault in Python, in the Python standard library, or in SQLite. I'm sure it won't be in anything on the other side of a popen call. Are there non-standard libraries or Python modules in use that you haven't mentioned? The most likely culprit in my mind is RAM. You're exercising new memory pretty hard, running a bunch of processes at it at full tilt. Any defect in the chips or DMA could explain what you're seeing. An easy test, not necessarily cheap, would be to replace the RAM (or, if possible, run with some removed). I have two war stories related to rotten I/O hardware, where the device appeared to work for all intents and purposes, but was actually a high-speed bit munger. Those were both over 20 years ago. It will be interesting to hear if that turns out to be your issue. HTH. --jkl _______________________________________________ sqlite-users mailing list sqlite-users@mailinglists.sqlite.org http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users
