Hello, Andi :)
I am working on fixing that error
GC Warning: Out of Memory! Returning NIL!
I have following structure of program:
I am trying to create 10 running threads of LuceneWorkerThread from
main thread.
class LuceneWorkerThread(PythonThread):
def __init__(self, luceneSearcher, qIn, qOut, id):
PythonThread.__init__(self, name='LuceneWorker-%s' % id)
self.luceneSearcher = luceneSearcher
self.queueIn = qIn
self.queueOut = qOut
self.id = id
self._log = False
def run(self):
while True:
method, args = self.queueIn.get()
if method=='stop':
break
elif method=='doc':
result = self.doc(*args)
else:
result = getattr(self, method)(*args)
self.queueOut.put(result)
if __name__=='__main__':
freeWorkers = 10
luceneSearcher = LuceneSearcher()
for i in range(freeWorkers):
luceneWorker = LuceneWorkerThread(luceneSearcher, Queue(), Queue(),
i)
luceneWorker.setDaemon(1)
luceneWorker.start()
queueWorkers.put(luceneWorker)
luceneSearcher.path = path
luceneSearcher.sort = Sort()
luceneSearcher.sort.setSort([SortField("modified", SortField.STRING,
True)])
luceneSearcher.directory = FSDirectory.getDirectory('index', False)
luceneSearcher.searcher = IndexSearcher(luceneSearcher.directory)
LuceneSearcher is just an empty class which purpose is to store reference to
IndexSearcher object
as you can see. I pass reference of this object to threads
constructor.
My goal is to have many running PythonThread's objects which have
references to LuceneSearcher object.
When index is small everything is okay. When I try to run this code on
big index (20Gb) then I receive exception from IndexSearcher()
constructor: GC Warning: Out of Memory! Returning NIL!
When I set number of threads to 4 or less then it runs without
exception. Number of PythonThreads affect to this exception!
Any ideas?
Yura Smolsky.
_______________________________________________
pylucene-dev mailing list
[email protected]
http://lists.osafoundation.org/mailman/listinfo/pylucene-dev
