Thanks for the pointer, Lance! Is there an example of this somewhere?
-Peter On Jul 31, 2010, at 3:13 PM, Lance Norskog wrote: > Ah! You're not just highlighting, you're snippetizing. This makes it easier. > > Highlighting does not stream- it pulls the entire stored contents into > one string and then pulls out the snippet. If you want this to be > fast, you have to split up the text into small pieces and only > snippetize from the most relevant text. So, separate documents with a > common group id for the document it came from. You might have to do 2 > queries to achieve what you want, but the second query for the same > query will be blindingly fast. Often <1ms. > > Good luck! > > Lance > > On Sat, Jul 31, 2010 at 1:12 PM, Peter Spam <ps...@mac.com> wrote: >> However, I do need to search the entire document, or else the highlighting >> will sometimes be blank :-( >> Thanks! >> >> - Peter >> >> ps. sorry for the many responses - I'm rushing around trying to get this >> working. >> >> On Jul 31, 2010, at 1:11 PM, Peter Spam wrote: >> >>> Correction - it went from 17 seconds to 10 seconds - I was changing the >>> hl.regex.maxAnalyzedChars the first time. >>> Thanks! >>> >>> -Peter >>> >>> On Jul 31, 2010, at 1:06 PM, Peter Spam wrote: >>> >>>> On Jul 30, 2010, at 1:16 PM, Peter Karich wrote: >>>> >>>>> did you already try other values for hl.maxAnalyzedChars=2147483647 >>>> >>>> Yes, I tried dropping it down to 21, but it didn't have much of an impact >>>> (one search I just tried went from 17 seconds to 15.8 seconds, and this is >>>> an 8-core Mac Pro with 6GB RAM - 4GB for java). >>>> >>>>> ? Also regular expression highlighting is more expensive, I think. >>>>> What does the 'fuzzy' variable mean? If you use this to query via >>>>> "~someTerm" instead "someTerm" >>>>> then you should try the trunk of solr which is a lot faster for fuzzy or >>>>> other wildcard search. >>>> >>>> "fuzzy" could be set to "*" but isn't right now. >>>> >>>> Thanks for the tips, Peter - this has been very frustrating! >>>> >>>> >>>> - Peter >>>> >>>>> Regards, >>>>> Peter. >>>>> >>>>>> Data set: About 4,000 log files (will eventually grow to millions). >>>>>> Average log file is 850k. Largest log file (so far) is about 70MB. >>>>>> >>>>>> Problem: When I search for common terms, the query time goes from under >>>>>> 2-3 seconds to about 60 seconds. TermVectors etc are enabled. When I >>>>>> disable highlighting, performance improves a lot, but is still slow for >>>>>> some queries (7 seconds). Thanks in advance for any ideas! >>>>>> >>>>>> >>>>>> -Peter >>>>>> >>>>>> >>>>>> ------------------------------------------------------------------------------------------------------------------------------------- >>>>>> >>>>>> 4GB RAM server >>>>>> % java -Xms2048M -Xmx3072M -jar start.jar >>>>>> >>>>>> ------------------------------------------------------------------------------------------------------------------------------------- >>>>>> >>>>>> schema.xml changes: >>>>>> >>>>>> <fieldType name="text_pl" class="solr.TextField"> >>>>>> <analyzer> >>>>>> <tokenizer class="solr.WhitespaceTokenizerFactory"/> >>>>>> <filter class="solr.LowerCaseFilterFactory"/> >>>>>> <filter class="solr.WordDelimiterFilterFactory" generateWordParts="0" >>>>>> generateNumberParts="0" catenateWords="0" catenateNumbers="0" >>>>>> catenateAll="0" splitOnCaseChange="0"/> >>>>>> </analyzer> >>>>>> </fieldType> >>>>>> >>>>>> ... >>>>>> >>>>>> <field name="body" type="text_pl" indexed="true" stored="true" >>>>>> multiValued="false" termVectors="true" termPositions="true" >>>>>> termOffsets="true" /> >>>>>> <field name="timestamp" type="date" indexed="true" stored="true" >>>>>> default="NOW" multiValued="false"/> >>>>>> <field name="version" type="string" indexed="true" stored="true" >>>>>> multiValued="false"/> >>>>>> <field name="device" type="string" indexed="true" stored="true" >>>>>> multiValued="false"/> >>>>>> <field name="filename" type="string" indexed="true" stored="true" >>>>>> multiValued="false"/> >>>>>> <field name="filesize" type="long" indexed="true" stored="true" >>>>>> multiValued="false"/> >>>>>> <field name="pversion" type="int" indexed="true" stored="true" >>>>>> multiValued="false"/> >>>>>> <field name="first2md5" type="string" indexed="false" stored="true" >>>>>> multiValued="false"/> >>>>>> <field name="ckey" type="string" indexed="true" stored="true" >>>>>> multiValued="false"/> >>>>>> >>>>>> ... >>>>>> >>>>>> <dynamicField name="*" type="ignored" multiValued="true" /> >>>>>> <defaultSearchField>body</defaultSearchField> >>>>>> <solrQueryParser defaultOperator="AND"/> >>>>>> >>>>>> ------------------------------------------------------------------------------------------------------------------------------------- >>>>>> >>>>>> solrconfig.xml changes: >>>>>> >>>>>> <maxFieldLength>2147483647</maxFieldLength> >>>>>> <ramBufferSizeMB>128</ramBufferSizeMB> >>>>>> >>>>>> ------------------------------------------------------------------------------------------------------------------------------------- >>>>>> >>>>>> The query: >>>>>> >>>>>> rowStr = "&rows=10" >>>>>> facet = >>>>>> "&facet=true&facet.limit=10&facet.field=device&facet.field=ckey&facet.field=version" >>>>>> fields = "&fl=id,score,filename,version,device,first2md5,filesize,ckey" >>>>>> termvectors = "&tv=true&qt=tvrh&tv.all=true" >>>>>> hl = "&hl=true&hl.fl=body&hl.snippets=1&hl.fragsize=400" >>>>>> regexv = "(?m)^.*\n.*\n.*$" >>>>>> hl_regex = "&hl.regex.pattern=" + CGI::escape(regexv) + >>>>>> "&hl.regex.slop=1&hl.fragmenter=regex&hl.regex.maxAnalyzedChars=2147483647&hl.maxAnalyzedChars=2147483647" >>>>>> justq = '&q=' + CGI::escape('body:' + fuzzy + p['q'].to_s.gsub(/\\/, >>>>>> '').gsub(/([:~!<>="])/,'\\\\\1') + fuzzy + minLogSizeStr) >>>>>> >>>>>> thequery = '/solr/select?timeAllowed=5000&wt=ruby' + (p['fq'].empty? ? >>>>>> '' : ('&fq='+p['fq'].to_s) ) + justq + rowStr + facet + fields + >>>>>> termvectors + hl + hl_regex >>>>>> >>>>>> baseurl = '/cgi-bin/search.rb?q=' + CGI::escape(p['q'].to_s) + '&rows=' >>>>>> + p['rows'].to_s + '&minLogSize=' + p['minLogSize'].to_s >>>>>> >>>>>> >>>>>> >>>>> >>>>> >>>>> -- >>>>> http://karussell.wordpress.com/ >>>>> >>>> >>> >> >> > > > > -- > Lance Norskog > goks...@gmail.com