But, I am facing the problem even with -Xms 256m and -Xmx 1024m. Yes, the file was not added to the index because java process was already using 1156m of memory, which is much higher than the max heap memory. But, even after waiting for a few minutes till the memory came below the max heap value, the index was not responsive (no documents were being added/deleted in the index). What could be the reason for this ?
Divya -----Original Message----- From: David Xiao [mailto:[EMAIL PROTECTED] Sent: Tuesday, April 24, 2007 6:01 PM To: java-user@lucene.apache.org Subject: RE: Adding large files to index Use java -Xms50m to start your program, that gives a 50M initial heap size. The OutofHeapMemory is because the default heap memory is not enough for your application. -----Original Message----- From: Divya Rajendranath [mailto:[EMAIL PROTECTED] Sent: Tuesday, April 24, 2007 7:01 PM To: java-user@lucene.apache.org Subject: Adding large files to index Hello All, Could any one help me find solution to the following problem ? I am facing problems while trying to add files of size 50MB to my application. The application has on-demand indexing of documents in place.whenever we add a file to our application, we first put the file details/metadata to our database, then create an encrytpted copy of the file in our own filesystem, then extract the content and add it to Lucene index. We set Java heap min = 256 M and max = 1024 mb. Extraction & adding to index is done in a background thread. Soon, after we extract the document's content, add it to index and close the index writer, we are getting OutOfMemory Error (memory used is about 1160mb). Because, the main thread is unaware of the exception, it allows users to continue using the application. I observed that the write.lock file continues to remain in /var/tmp directory for about 10 mins, it is not getting cleaned up soon. On searching, the 50 MB document do not come up in search. But, when I continue using my application by deleting some other documents, after adding the 50 MB file, I find that lucene is not getting updated with the recent changes. Once, this error occurs lucene is not getting modified/updated even if I add a new 10KB file nor delete any file. Here, is the stack trace === 2007-04-24 13:16:47,454 INFO [docman.index.Index] Adding 20 to physical index 2007-04-24 13:16:47,454 INFO [docman.index.Index] Index::addToIndex() took 27207 msec. 2007-04-24 13:16:47,454 INFO [docman.index.Index] going to close index writer 2007-04-24 13:17:00,945 INFO [STDOUT] Exception in thread "Thread-12" 2007-04-24 13:17:00,945 INFO [STDOUT] java.lang.OutOfMemoryError: Java heap space 2007-04-24 13:17:00,945 INFO [STDOUT] at org.apache.lucene.store.IndexInput.readString(IndexInput.java:91) 2007-04-24 13:17:00,945 INFO [STDOUT] at org.apache.lucene.index.FieldsReader.doc(FieldsReader.java:130) 2007-04-24 13:17:00,945 INFO [STDOUT] at org.apache.lucene.index.SegmentReader.document(SegmentReader.java:284) 2007-04-24 13:17:00,945 INFO [STDOUT] at org.apache.lucene.index.SegmentMerger.mergeFields(SegmentMerger.java:186 ) 2007-04-24 13:17:00,945 INFO [STDOUT] at org.apache.lucene.index.SegmentMerger.merge(SegmentMerger.java:88) 2007-04-24 13:17:00,945 INFO [STDOUT] at org.apache.lucene.index.IndexWriter.mergeSegments(IndexWriter.java:709) 2007-04-24 13:17:00,945 INFO [STDOUT] at org.apache.lucene.index.IndexWriter.mergeSegments(IndexWriter.java:686) 2007-04-24 13:17:00,945 INFO [STDOUT] at org.apache.lucene.index.IndexWriter.flushRamSegments(IndexWriter.java:65 6) 2007-04-24 13:17:00,945 INFO [STDOUT] at org.apache.lucene.index.IndexWriter.close(IndexWriter.java:402) 2007-04-24 13:17:00,945 INFO [STDOUT] at docman.index.Index.closeIndex( Index.java:594) 2007-04-24 13:17:00,946 INFO [STDOUT] at docman.index.Index.addToIndex( Index.java:401) 2007-04-24 13:17:00,946 INFO [STDOUT] at docman.index.Index.index(Index.java:437) 2007-04-24 13:17:00,946 INFO [STDOUT] at docman.index.FastIndex$IndexWorker.run(FastIndex.java:174) === [EMAIL PROTECTED]:/u/user/test/application> ls -al index total 450208 drwxrwsr-x 2 user group 4096 Apr 24 15:40 . drwxrwsrwx 10 user group 4096 Apr 23 18:36 .. -rw-rw-r-- 1 user group 80289688 Apr 23 18:45 _j.cfs -rw-rw-r-- 1 user group 114021843 Apr 23 19:34 _x.cfs -rw-rw-r-- 1 user group 35937776 Apr 24 15:40 _z.fdt -rw-rw-r-- 1 user group 16 Apr 24 15:40 _z.fdx -rw-rw-r-- 1 user group 503 Apr 24 15:40 _z.fnm -rw-rw-r-- 1 user group 4 Apr 23 19:34 deletable -rw-rw-r-- 1 user group 34 Apr 23 19:34 segments [EMAIL PROTECTED]:/u/user/test/application There are about 30 files in the application of which 6-7 are of 50 MB, and the remaining are hardly a few KB. Is the OutFMemory error corrupting the index ? Why is it that the index is not responsive to delete operation after the error ? I waited for 10mins for the lock file to get removed on its own, then tried deleting/adding new 10KB files (after memory usage came back within java max heap size), still the index is not responsive. Please let me know the cause and the solution to the problem. Divya. --------------------------------------------------------------------- To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED] --------------------------------------------------------------------- To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]