It sounds like you have a fairly busy system, perhaps 100% load on the
process is not that strange, at least not during short periods of time.
A simpler solution would be to nice the process a little bit in order to
give your background jobs some more time to think.
Running a profiler is still the best advice I can think of. It should
clearly show you what is going on when you run out of CPU.
--
karl
1 aug 2007 kl. 04.29 skrev Chew Yee Chuang:
Hi,
Thanks for the link provided, actually I've go through those
article when I
developing the index and search function for my application. I
haven’t try
profiler yet, but I monitor the CPU usage and notice that whatever
index or
search performing, the CPU usage raise to 100%. Below I will try to
elaborate more on what my application is doing and how I index and
search.
There are many concurrent process running, first, the application
will write
records that received into a text file with tab separated each
different
field. Application will point to a new file every 10mins and start
writing
to it. So every file will contains only 10mins record, approximate
600,000
records per file. Then, the indexing process will check whether
there is a
text file to be index, if it is, the thread will wake up and start
perform
indexing.
The indexing process will first add documents to RAMDir, Then
later, add
RAMDir into FSDir by calling addIndexNoOptimize() when there is
100,000
documents(32 fields per doc) in RAMDir. There is only 1 IndexWriter
(FSDir)
was created but a few IndexWriter(RAMDir) was created during the whole
process. Below are some configuration for IndexWriters that I
mentioned:-
IndexWriter (RAMDir)
- SimpleAnalyzer
- setMaxBufferedDocs(10000)
- Filed.Store.YES
- Field.Index.NO_NORMS
IndexWriter (FSDir)
- SimpleAnalyzer
- setMergeFactor(20)
- addIndexesNoOptimize()
For the searching, because there are many queries(20,000) run
continuously
to generate the aggregate table for reporting purpose. All this
queries is
run in nested loop, and there is only 1 Searcher created, I try
searcher and
filter as well, filter give me better result, but both also utilize
lots of
CPU resources.
Hope this info will help and sorry for my bad English.
Thanks
eChuang, Chew
-----Original Message-----
From: karl wettin [mailto:[EMAIL PROTECTED]
Sent: Tuesday, July 31, 2007 5:54 PM
To: java-user@lucene.apache.org
Subject: Re: High CPU usage duing index and search
31 jul 2007 kl. 05.25 skrev Chew Yee Chuang:
But just notice that when Lucene performing search or index,
the CPU usage on my machine raise to 100%, because of this issue,
some of my
others backend process will slow down eventually. Just want to know
does
anyone face this problem before ? and is it any idea on how to
overcome this
problem ?
Did you run a profiler to see what it is that consume all the
resources?
It is very hard to guess based on the information you supplied. Start
here:
http://wiki.apache.org/lucene-java/BasicsOfPerformance
http://wiki.apache.org/lucene-java/ImproveIndexingSpeed
http://wiki.apache.org/lucene-java/ImproveSearchingSpeed
--
karl
---------------------------------------------------------------------
To unsubscribe, e-mail: [EMAIL PROTECTED]
For additional commands, e-mail: [EMAIL PROTECTED]
No virus found in this incoming message.
Checked by AVG Free Edition.
Version: 7.5.476 / Virus Database: 269.11.0/927 - Release Date:
7/30/2007
5:02 PM
No virus found in this outgoing message.
Checked by AVG Free Edition.
Version: 7.5.476 / Virus Database: 269.11.0/929 - Release Date:
7/31/2007
5:26 PM
---------------------------------------------------------------------
To unsubscribe, e-mail: [EMAIL PROTECTED]
For additional commands, e-mail: [EMAIL PROTECTED]
---------------------------------------------------------------------
To unsubscribe, e-mail: [EMAIL PROTECTED]
For additional commands, e-mail: [EMAIL PROTECTED]
