Thanks for the reply.
I am timing the entire search process with a stop watch, a bit ghetto style.
My getXXX methods are:
Document doc = hits.doc(i);
String str = doc.get("item");
So you can see that I am retrieving the entire document in a search query.
Ideally , I'd like to just retrieve the Field object that I want to run the
search on. I know this will give me a boost as one of my Fields is really
huge.
My query is selecting the entire user data-set in the database. I'd like to
do some SQL based search in the query too so that I pick only those items
where the phrase matches.
Index contains about 650MB of data. Index file size is 14478869 bytes.
thanks,
AZ
On 7/24/07, Grant Ingersoll <[EMAIL PROTECTED]> wrote:
>
> Where are you getting your numbers from? That is, where are your
> timers? Are you timing the rs.next() loop, or the individual calls
> to Lucene? What do the getXXXXX methods look like? How big are your
> queries? How big is your index?
>
> Essentially, we need more info to really help you. From what I can
> tell, you are generating 3 different Lucene queries for each record
> in the database. Frankly, I surprised your slowdown is only linear.
>
> On Jul 24, 2007, at 4:31 PM, Askar Zaidi wrote:
>
> > I have 512MB RAM allocated to JVM Heap. If I double my system RAM
> > from 768MB
> > to say 2GB or so, and give JVM 1.5GB Heap space, will I get quicker
> > results
> > ?
> >
> > Can I expect results which take 1 minute to be returned in 30
> > seconds with
> > more RAM ? Should I also get a more powerful CPU ? A real server class
> > machine ?
> >
> > I have also done some of the optimizations that are mentioned on
> > the Lucene
> > website.
> >
> > thanks,
> > AZ
> >
> > On 7/24/07, Askar Zaidi <[EMAIL PROTECTED]> wrote:
> >>
> >> Hey Guys,
> >>
> >> I just finished up using Lucene in my application. I have data in a
> >> database , so while indexing I extract this data from the database
> >> and pump
> >> it into the index. Specifically , I have the following data in the
> >> index:
> >>
> >> <itemID> <tags> <title> <summary> <contents>
> >>
> >> where itemID is just a number (primary key in the DB)
> >> tags : text
> >> titie: text
> >> summary: text
> >> contents: Huge text (text extracted from files: pdfs, docs etc).
> >>
> >> Now while running a search query I realized that the response time
> >> increases in a linear fashion as the number of <itemID> increase
> >> in the DB.
> >>
> >> If I have 50 items, its 8 seconds
> >> 100 items, its 17 seconds.
> >> 300+ items, its 60 seconds and maybe more.
> >>
> >> In a perfect world, I'd like to search on 300+ items within 10-15
> >> seconds.
> >> Can anyone give me tips to fine tune lucene ?
> >>
> >> Heres a code snippet:
> >>
> >> sql query = "SELECT itemID from items where creator = 'askar' ;
> >>
> >> --execute query--
> >>
> >> while(rs.next()){
> >>
> >> score = doTagSearch(askar,text,itemID);
> >> scoreTitle = doTitleSearch(askar,text,itemID);
> >> scoreSummary = doSummarySearch(askar,text,itemID);
> >>
> >> ----
> >>
> >> }
> >>
> >> So this code asks Lucene to search for the "text" in the itemID
> >> passed.
> >> itemID is already indexed. The while loop will run 300 times if
> >> there are
> >> 300 items....that gets slow...what can I do here ??
> >>
> >> thanks for the replies,
> >>
> >> AZ
> >>
>
> --------------------------
> Grant Ingersoll
> Center for Natural Language Processing
> http://www.cnlp.org/tech/lucene.asp
>
> Read the Lucene Java FAQ at http://wiki.apache.org/lucene-java/LuceneFAQ
>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: [EMAIL PROTECTED]
> For additional commands, e-mail: [EMAIL PROTECTED]
>
>
