Where are you getting your numbers from? That is, where are your
timers? Are you timing the rs.next() loop, or the individual calls
to Lucene? What do the getXXXXX methods look like? How big are your
queries? How big is your index?
Essentially, we need more info to really help you. From what I can
tell, you are generating 3 different Lucene queries for each record
in the database. Frankly, I surprised your slowdown is only linear.
On Jul 24, 2007, at 4:31 PM, Askar Zaidi wrote:
I have 512MB RAM allocated to JVM Heap. If I double my system RAM
from 768MB
to say 2GB or so, and give JVM 1.5GB Heap space, will I get quicker
results
?
Can I expect results which take 1 minute to be returned in 30
seconds with
more RAM ? Should I also get a more powerful CPU ? A real server class
machine ?
I have also done some of the optimizations that are mentioned on
the Lucene
website.
thanks,
AZ
On 7/24/07, Askar Zaidi <[EMAIL PROTECTED]> wrote:
Hey Guys,
I just finished up using Lucene in my application. I have data in a
database , so while indexing I extract this data from the database
and pump
it into the index. Specifically , I have the following data in the
index:
<itemID> <tags> <title> <summary> <contents>
where itemID is just a number (primary key in the DB)
tags : text
titie: text
summary: text
contents: Huge text (text extracted from files: pdfs, docs etc).
Now while running a search query I realized that the response time
increases in a linear fashion as the number of <itemID> increase
in the DB.
If I have 50 items, its 8 seconds
100 items, its 17 seconds.
300+ items, its 60 seconds and maybe more.
In a perfect world, I'd like to search on 300+ items within 10-15
seconds.
Can anyone give me tips to fine tune lucene ?
Heres a code snippet:
sql query = "SELECT itemID from items where creator = 'askar' ;
--execute query--
while(rs.next()){
score = doTagSearch(askar,text,itemID);
scoreTitle = doTitleSearch(askar,text,itemID);
scoreSummary = doSummarySearch(askar,text,itemID);
----
}
So this code asks Lucene to search for the "text" in the itemID
passed.
itemID is already indexed. The while loop will run 300 times if
there are
300 items....that gets slow...what can I do here ??
thanks for the replies,
AZ
--------------------------
Grant Ingersoll
Center for Natural Language Processing
http://www.cnlp.org/tech/lucene.asp
Read the Lucene Java FAQ at http://wiki.apache.org/lucene-java/LuceneFAQ
---------------------------------------------------------------------
To unsubscribe, e-mail: [EMAIL PROTECTED]
For additional commands, e-mail: [EMAIL PROTECTED]
