Re: Text storing design and performance question

2007-01-10 Thread Erik Hatcher
You don't have to store a field to highlight text. If you've got it in your database, retrieve it from there and pass that string to the highlighter instead. Erik On Jan 10, 2007, at 10:45 AM, moraleslos wrote: I'm running into a little dilemma with Lucene highlighting and ind

Re: Text storing design and performance question

2007-01-10 Thread moraleslos
Hi Erik, Would that slow performance a bit? For example, say I receive 50,000 hits from a search. From your explanation, I have to retrieve the DB id from each hit, perform a query to the DB using the id to retrieve the full contents for each hit, run highlighter on each content, and then retur

Re: Text storing design and performance question

2007-01-10 Thread Mark Miller
Usually a user cannot easily browse 50,000 on a single display, and so you would only highlight the docs as they became visible to the user. This is generally a small amount...often one at a time. - Mark moraleslos wrote: Hi Erik, Would that slow performance a bit? For example, say I receiv

Re: Text storing design and performance question

2007-01-10 Thread moraleslos
Hi Mark, Looks like I've got to implement some sort of pagination for my clients. Problem is everything is stateless so looks like there's some work I need to do on my end. Thanks. -los markrmiller wrote: > > Usually a user cannot easily browse 50,000 on a single display, and so > you wou

Re: Text storing design and performance question

2007-01-10 Thread Mark Miller
Being stateless should not be much of an issue. As Erick mentioned, the highlighter just expects you to pass it the query again and the text to be highlighted. So when you show the pagination you just need to keep around what query generated the current page...then shove each piece of relevant tex

RE: Text storing design and performance question

2007-01-10 Thread Renaud Waldura
just storing term vectors would keep the index lean and allow for fast highlighting? --Renaud -Original Message- From: Mark Miller [mailto:[EMAIL PROTECTED] Sent: Wednesday, January 10, 2007 9:54 AM To: java-user@lucene.apache.org Subject: Re: Text storing design and performance que

Re: Text storing design and performance question

2007-01-10 Thread moraleslos
So I guess what you're stating is that Lucene is really fast on searches that doing repeated searches for pagination wouldn't make much of a difference (even though its redundant)... I'll give this a shot. Thanks Mark. -los markrmiller wrote: > > Being stateless should not be much of an issu

RE: Text storing design and performance question

2007-01-10 Thread moraleslos
the index lean and > allow for fast highlighting? > > --Renaud > > > > -Original Message- > From: Mark Miller [mailto:[EMAIL PROTECTED] > Sent: Wednesday, January 10, 2007 9:54 AM > To: java-user@lucene.apache.org > Subject: Re: Text storing design and performanc

RE: Text storing design and performance question

2007-01-10 Thread Renaud Waldura
: java-user@lucene.apache.org Subject: RE: Text storing design and performance question Maybe keeping the data in the DB would make it quicker? Seems like the I/O performance would cause most of the performance issues you're seeing. -los Renaud Waldura-5 wrote: > > We used to store

Re: Text storing design and performance question

2007-01-10 Thread Jason Pump
dex lean and allow for fast highlighting? --Renaud -Original Message- From: Mark Miller [mailto:[EMAIL PROTECTED] Sent: Wednesday, January 10, 2007 9:54 AM To: java-user@lucene.apache.org Subject: Re: Text storing design and performance question Being stateless should not be much of

RE: Text storing design and performance question

2007-01-11 Thread Renaud Waldura
0, 2007 1:49 PM To: java-user@lucene.apache.org Subject: Re: Text storing design and performance question Renaud, one optimization you can do on this is to try the first 10kb, see if it finds text worth highlighting, if not, with a slight overlap try the next 9.9kb - 19.9kb or just 9.9kb -> end if

Re: Text storing design and performance question

2007-01-11 Thread Jason Pump
e match? -Original Message- From: Jason Pump [mailto:[EMAIL PROTECTED] Sent: Wednesday, January 10, 2007 1:49 PM To: java-user@lucene.apache.org Subject: Re: Text storing design and performance question Renaud, one optimization you can do on this is to try the first 10kb, see if it

Re: Text storing design and performance question

2007-01-11 Thread Chris Hostetter
In general, if you are having performance issues with highlighting, the first thing to do is double check what the bottleneck is: is it accessing the text to by highlighted, or is it running the highlighter? you suggested earlier in the thread that the problem was with accessing the text... : >>