On Fri, Nov 18, 2016 at 3:19 PM, James K. Lowden <jklow...@schemamania.org> wrote:
> On Fri, 18 Nov 2016 10:56:37 -0800 > Roger Binns <rog...@rogerbinns.com> wrote: > > > Popen calls fork (it seems like you are doing Unix/Mac, not Windows). > > fork() duplicates the process including all open file descriptors. > > One or more of those descriptors belong to open SQLite databases and > > ancillary files. > > Good catch, Roger. It's a liability, but I slightly disagree with your > characterization. > > > - Running any Python code (destructors can be called which then run in > > the parent and child) > > Yes, if those destructors affect shared resources. The OP did say the > processes on the other side of popen were C programs. > > > - Not having file descriptors closed so the child process trashes them > > (close_fds Popen argument is False in python 2.x but True in python > > 3.x). > > The child process can't "trash" the parent's descriptors. When the > child exits, the OS will close its descriptors, that's all. But, yes, > if the child process is making some assumption about open descriptors > it receives at startup, that could be lead to problems. Especially if > it scribbles on the SQLite database. > > > Also python 2.x subprocess module is broken in many ways. > > My foray into Unicode in Python convinced me once and for all that > Python 3 is the only way to go. But would you care to elaborate on the > problems with 2.x subprocess? > > > You can all put aside worries about Python 2. I only started using Python a couple of months ago, and went with 3. I have two different machines running this stuff. Only one is having the seg faults, but they are different enough that this does not convince me to blame hardware. System with problems: Running Xubuntu Linux 16.04.1, Python 3.5.2. CPU: core i7, MOBO: Asus Z97 Deluxe, 32GB ram. 4 cores, hyperthreaded to 8. Home-built. System without this problem: Running Ubuntu Linux 14.04.5, Python 3.4.3. This is a System76 Silverback. CPU: dual Intel Xeon, 256 GB RAM(!) 16 cores, hyperthreaded to 32. On both: Filesystem is an 11-TB software RAID (Linux LVM) on top of three 4-TB hard drives. ------------------ Python modules are all pretty standard (I think): stat argparse datetime fileinput glob os re sqlite3 subprocess sys time children on the other side of Popen are either standard Linux utilities like split/grep/sort or my own C code, which is mostly very simple with one exception, and that is a pretty well-tested implementation of minimax search through a game tree. But even in that one, the I/O is very simple: one input line yields one output line on one of two output streams. I never set close_fds=False, so the Python3 default protects the database. Questions, though: I don't bother to commit in the midst of anything, so you'll notice my journal gets pretty big. That's not a problem for my filesystem, but could it be a problem internally to Python? -- #define QUESTION ((bb) || (!bb)) /* Shakespeare */ _______________________________________________ sqlite-users mailing list sqlite-users@mailinglists.sqlite.org http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users
