I'm starting to see traces like the following in my UpLib (OS X 10.5.8, 32-bit Python 2.5, Java 6, JCC-2.6, PyLucene-2.9.3) that indicate an out-of-memory issue. I spawn a lot of short-lived threads in Python, and each of them is "attached" to Java, and "detached" after the "run" method returns. I've run test programs that do nothing but repeatedly start new threads that then invoke pylucene to index a document, and see no problems.
I'm trying to come up with a hypothesis for this. One of the things I'm wondering is if my Python memory space is approaching the limit, does PyLucene arrange for the Java garbage collector to invoke the Python garbage collector if it can't allocate memory? I keep a lot of objects via weak references in my Python memory space, and I may just be filling up VM so that Java can't allocate enough heap/stack space for a new thread. Note that the thread being unsuccessfully started isn't mine; it's being started by Java. Bill thr1730: Running document rippers raised the following exception: thr1730: Traceback (most recent call last): thr1730: File "/local/share/UpLib-1.7.9/code/uplib/newFolder.py", line 282, in _run_rippers thr1730: ripper.rip(folderpath, id) thr1730: File "/local/share/UpLib-1.7.9/code/uplib/createIndexEntry.py", line 187, in rip thr1730: index_folder(location, self.repository().index_path()) thr1730: File "/local/share/UpLib-1.7.9/code/uplib/createIndexEntry.py", line 82, in index_folder thr1730: c.index(folder, doc_id) thr1730: File "/local/share/UpLib-1.7.9/code/uplib/indexing.py", line 813, in index thr1730: self.reopen() thr1730: File "/local/share/UpLib-1.7.9/code/uplib/indexing.py", line 635, in reopen thr1730: self.current_writer.flush() thr1730: JavaError: java.lang.OutOfMemoryError: unable to create new native thread thr1730: Java stacktrace: thr1730: java.lang.OutOfMemoryError: unable to create new native thread thr1730: at java.lang.Thread.start0(Native Method) thr1730: at java.lang.Thread.start(Thread.java:592) thr1730: at org.apache.lucene.index.ConcurrentMergeScheduler.merge(ConcurrentMergeScheduler.java:221) thr1730: at org.apache.lucene.index.IndexWriter.maybeMerge(IndexWriter.java:3070) thr1730: at org.apache.lucene.index.IndexWriter.maybeMerge(IndexWriter.java:3065) thr1730: at org.apache.lucene.index.IndexWriter.maybeMerge(IndexWriter.java:3061) thr1730: at org.apache.lucene.index.IndexWriter.flush(IndexWriter.java:4256) thr1730: at org.apache.lucene.index.IndexWriter.flush(IndexWriter.java:4060)
