Finally we are seeing great improvement once we switch to 64-bit java and MMapDirectory. Our Test run (multiple requests) used to take 26 minutes on 32-bit and is now improved to 10 minutes on 64-bit java.
We load stored documents from lucene and pass the documents to a third party libray (closed source) for further processing (Note: we can not utilize inverted index for this processing). The third party library is memory intensive and it needs around 10GB of heap memory for our load. We have a cap of 16GB system memory available and our index is of 8GB in size note: sorry for the repost. There was an issue with formatting earlier and might have created confusion.
