Ajay, Here is another thread I started on the same issue.
http://stackoverflow.com/questions/1362460/why-does-lucene-cause-oom-whe n-indexing-large-files Paul -----Original Message----- From: java-user-return-45254-paul.b.murdoch=saic....@lucene.apache.org [mailto:java-user-return-45254-paul.b.murdoch=saic....@lucene.apache.org ] On Behalf Of ajay_gupta Sent: Tuesday, March 02, 2010 8:28 AM To: java-user@lucene.apache.org Subject: Lucene Indexing out of memory Hi, It might be general question though but I couldn't find the answer yet. I have around 90k documents sizing around 350 MB. Each document contains a record which has some text content. For each word in this text I want to store context for that word and index it so I am reading each document and for each word in that document I am appending fixed number of surrounding words. To do that first I search in existing indices if this word already exist and if it is then I get the content and append the new context and update the document. In case no context exist I create a document with fields "word" and "context" and add these two fields with values as word value and context value. I tried this in RAM but after certain no of docs it gave out of memory error so I thought to use FSDirectory method but surprisingly after 70k documents it also gave OOM error. I have enough disk space but still I am getting this error.I am not sure even for disk based indexing why its giving this error. I thought disk based indexing will be slow but atleast it will be scalable. Could someone suggest what could be the issue ? Thanks Ajay -- View this message in context: http://old.nabble.com/Lucene-Indexing-out-of-memory-tp27755872p27755872. html Sent from the Lucene - Java Users mailing list archive at Nabble.com. --------------------------------------------------------------------- To unsubscribe, e-mail: java-user-unsubscr...@lucene.apache.org For additional commands, e-mail: java-user-h...@lucene.apache.org --------------------------------------------------------------------- To unsubscribe, e-mail: java-user-unsubscr...@lucene.apache.org For additional commands, e-mail: java-user-h...@lucene.apache.org
