Search failing for matched text in large field

2011-03-23 Thread Paul
I'm using solr 1.4.1. I have a document that has a pretty big field. If I search for a phrase that occurs near the start of that field, it works fine. If I search for a phrase that appears even a little ways into the field, it doesn't find it. Is there some limit to how far into a field solr will

Re: Search failing for matched text in large field

2011-03-23 Thread Sascha Szott
Hi Paul, did you increase the value of the maxFieldLength parameter in your solrconfig.xml? -Sascha On 23.03.2011 17:05, Paul wrote: I'm using solr 1.4.1. I have a document that has a pretty big field. If I search for a phrase that occurs near the start of that field, it works fine. If I

Re: Search failing for matched text in large field

2011-03-23 Thread Paul
Ah, no, I'll try that now. What is the disadvantage of setting that to a really large number? I do want the search to work for every word I give to solr. Otherwise I wouldn't have indexed it to begin with. On Wed, Mar 23, 2011 at 11:15 AM, Sascha Szott sz...@zib.de wrote: Hi Paul, did you

Re: Search failing for matched text in large field

2011-03-23 Thread Jonathan Rochkind
How large? But rather than think about if there's something in the searching that's not working, the first step might be to make sure that everything in the _indexing_ is working -- that your field is actually being indexed as you intend. I forget the best way to view what's in your index

Re: Search failing for matched text in large field

2011-03-23 Thread Paul
I increased maxFieldLength and reindexed a small number of documents. That worked -- I got the correct results. In 3 minutes! I assume that if I reindex all my documents that all searches will become even slower. Is there any way to get all the results in a way that is quick enough that my user

Re: Search failing for matched text in large field

2011-03-23 Thread Jonathan Rochkind
Hmm, there's no reason it should take anywhere close 3 minutes to get a result from a simple search, even with very large documents/term lists. Especially if you're really JUST doing a simple search, you aren't using facetting or statistics component or highlighting etc at this point. (If you

Re: Search failing for matched text in large field

2011-03-23 Thread Sascha Szott
On 23.03.2011 18:52, Paul wrote: I increased maxFieldLength and reindexed a small number of documents. That worked -- I got the correct results. In 3 minutes! Did you mark the field in question as stored = false? -Sascha I assume that if I reindex all my documents that all searches will

Re: Search failing for matched text in large field

2011-03-23 Thread Paul
I looked into the search that I'm doing a little closer and it seems like the highlighting is slowing it down. If I do the query without requesting highlighting it is fast. (BTW, I also have faceting and pagination in my query. Faceting doesn't seem to change the response time much, adding rows=

Re: Search failing for matched text in large field

2011-03-23 Thread Jonathan Rochkind
Yeah, you aren't going to be able to do highlighting on a very very large field without terrible performance. I believe it's just the nature of the algorithm used by the highlighting component. I don't know of any workaround. Other than inventing a new algorithm for highlighting and writing

Re: Search failing for matched text in large field

2011-03-23 Thread Markus Jelsma
Enable TermVectors for fields that you're going tot highlight. If it is disabled Solr will reanalyze the field, killing performance. I looked into the search that I'm doing a little closer and it seems like the highlighting is slowing it down. If I do the query without requesting highlighting