Hi Team,
I am a new user of Lucene 4.8.1. I encountered a Lucene indexing
performance issue which slow down my application greatly. I tried several
ways from google searchs but still couldn't resolve it. Any suggestions
from your experts might help me a lot.
One of my application uses the lucene index for fast data searching. When I
start my application, I will index all the necessary data from database
which will be 88 MB index data after indexing is done. In this case,
indexing only takes less than 4 minutes.
I have another shell script task running every night, which send a JMX call
to my application to re-indexing all the data. The re-indexing method will
clear my current indexing directory data, reading data from database and
recreating the index from the ground. Everything works fine at the
beginning, indexing only takes a little more than 3 mins. But after my
application running for a while(one day or two), the re-indexing speed
slows down greatly which now takes more than 22 mins.
Here is the procedure of my Lucene indexing and re-indexing:
1. If index data exists inside index directory, remove all the index
data.
2. Create IndexWriter with 200MB RAMBUFFERSIZE, (6.6) MaxMergesAndThreads
3. Process DB result set
- When I loop the result set, I reuse the same Document instance.
- At the end of each loop, I call indexWriter.addDocument(doc)
4. IndexWriter.commit()
5. IndexWriter.close();
I did a profiling when it was slow and found out that
indexWriter.addDocument method took most of the time. Then, i put some
logging code as below:
long start = System.currentTimeMillis();
indexWriter.addDocument(doc);
totalAddDocTime += (System.currentTimeMillis() - start);
After several tests, when the indexing is slow down, the total time took by
indexWriter.addDocument(doc) is about 20 mins.
During indexing, i also observed the cpu usage sometimes above 100.
6G memory assigned to my application. When indexing, other processing
modules are all suspended waiting for indexing finish and I don't see any
memory leak in my application.
Can you give me some suggestions about my issue?
Thank you,
Jason
