Hi,

Thanks a lot for your detailed problem description. It definitely is an error. 
Would you be so kind to register it as a bug ticket, including your 
descriptions from this email? 
http://wiki.apache.org/solr/HowToContribute#JIRA_tips_.28our_issue.2BAC8-bug_tracker.29.
 Also please attach to the issue your polish hunspell dictionaries. Then we'll 
try to reproduce the error.

I wonder if this performance decrease is also seen for English dictionaries?

--
Jan Høydahl, search solution architect
Cominvent AS - www.cominvent.com
Solr Training - www.solrtraining.com

On 13. mars 2012, at 16:42, Agnieszka Kukałowicz wrote:

> Hi,
> 
> I did some more tests for Hunspell in solr 3.4, 4.0:
> 
> Solr 3.4, full import 489017 documents:
> 
> StempelPolishStemFilterFactory -  2908 seconds, 168 docs/sec
> HunspellStemFilterFactory - 3922 seconds, 125 docs/sec
> 
> Solr 4.0, full import 489017 documents:
> 
> StempelPolishStemFilterFactory - 3016 seconds, 162 docs/sec
> HunspellStemFilterFactory - 44580 seconds (more than 12 hours), 11 docs/sec
> 
> Server specification and Java settings are the same as before.
> 
> Cheers
> Agnieszka
> 
> 
>> -----Original Message-----
>> From: Agnieszka Kukałowicz [mailto:agnieszka.kukalow...@usable.pl]
>> Sent: Tuesday, March 13, 2012 10:39 AM
>> To: 'solr-user@lucene.apache.org'
>> Subject: RE: solr 3.5 and indexing performance
>> 
>> Hi,
>> 
>> Yes, I confirmed that without Hunspell indexing has normal speed.
>> I did tests in solr 4.0 with Hunspell and PolishStemmer.
>> With StempelPolishStemFilterFactory the speed is normal.
>> 
>> My schema is quit easy. For Hunspell I have one text field I copy 14
>> text fields to:
>> 
>> "<field name="text" type="text_pl_hunspell" indexed="true"
>> stored="false" multiValued="true"/>"
>> 
>> 
>> <copyField source="field1" dest="text"/>  <copyField source="field2"
>> dest="text"/>  <copyField source="field3" dest="text"/>  <copyField
>> source="field4" dest="text"/>  <copyField source="field5" dest="text"/>
>> <copyField source="field6" dest="text"/>  <copyField source="field7"
>> dest="text"/>  <copyField source="field8" dest="text"/>  <copyField
>> source="field9" dest="text"/>  <copyField source="field10" dest="text"/>
>> <copyField source="field11" dest="text"/>  <copyField source="field12"
>> dest="text"/>  <copyField source="field13" dest="text"/>  <copyField
>> source="field14" dest="text"/>
>> 
>> The "text_pl_hunspell" configuration:
>> 
>> <fieldType name="text_pl_hunspell" class="solr.TextField"
>> positionIncrementGap="100">
>>      <analyzer type="index">
>>        <tokenizer class="solr.StandardTokenizerFactory"/>
>>        <filter class="solr.StopFilterFactory"
>>                ignoreCase="true"
>>                words="dict/stopwords_pl.txt"
>>                enablePositionIncrements="true"
>>                />
>>        <filter class="solr.LowerCaseFilterFactory"/>
>>        <filter class="solr.HunspellStemFilterFactory"
>> dictionary="dict/pl_PL.dic" affix="dict/pl_PL.aff" ignoreCase="true"
>>        <!--filter class="solr.KeywordMarkerFilterFactory"
>> protected="protwords_pl.txt"/-->
>>      </analyzer>
>>      <analyzer type="query">
>>        <tokenizer class="solr.StandardTokenizerFactory"/>
>>        <filter class="solr.SynonymFilterFactory"
>> synonyms="dict/synonyms_pl.txt" ignoreCase="true" expand="true"/>
>>        <filter class="solr.StopFilterFactory"
>>                ignoreCase="true"
>>                words="dict/stopwords_pl.txt"
>>                enablePositionIncrements="true"
>>                />
>>        <filter class="solr.LowerCaseFilterFactory"/>
>>        <filter class="solr.HunspellStemFilterFactory"
>> dictionary="dict/pl_PL.dic" affix="dict/pl_PL.aff" ignoreCase="true"
>>        <filter class="solr.KeywordMarkerFilterFactory"
>> protected="dict/protwords_pl.txt"/>
>>      </analyzer>
>>    </fieldType>
>> 
>> I use Polish dictionary (files stopwords_pl.txt, protwords_pl.txt,
>> synonyms_pl.txt are empy)- pl_PL.dic, pl_PL.aff. These are the same
>> files I used in 3.4 version.
>> 
>> For Polish Stemmer the diffrence is only in definion text field:
>> 
>> "<field name="text" type="text_pl" indexed="true" stored="false"
>> multiValued="true"/>"
>> 
>>    <fieldType name="text_pl" class="solr.TextField"
>> positionIncrementGap="100">
>>      <analyzer type="index">
>>        <tokenizer class="solr.StandardTokenizerFactory"/>
>>        <filter class="solr.StopFilterFactory"
>>                ignoreCase="true"
>>                words="dict/stopwords_pl.txt"
>>                enablePositionIncrements="true"
>>                />
>>        <filter class="solr.LowerCaseFilterFactory"/>
>>        <filter class="solr.StempelPolishStemFilterFactory"/>
>>        <filter class="solr.KeywordMarkerFilterFactory"
>> protected="dict/protwords_pl.txt"/>
>>      </analyzer>
>>      <analyzer type="query">
>>        <tokenizer class="solr.StandardTokenizerFactory"/>
>>        <filter class="solr.SynonymFilterFactory"
>> synonyms="dict/synonyms_pl.txt" ignoreCase="true" expand="true"/>
>>        <filter class="solr.StopFilterFactory"
>>                ignoreCase="true"
>>                words="dict/stopwords_pl.txt"
>>                enablePositionIncrements="true"
>>                />
>>        <filter class="solr.LowerCaseFilterFactory"/>
>>        <filter class="solr.StempelPolishStemFilterFactory"/>
>>        <filter class="solr.KeywordMarkerFilterFactory"
>> protected="dict/protwords_pl.txt"/>
>>      </analyzer>
>>    </fieldType>
>> 
>> One document has 23 fields:
>> - 14 text fields copy to one text field (above) that is only indexed
>> - 8 other indexed fields (2 strings, 2 tdates, 3 tint, 1 tfloat) The
>> size of one document is 3-4 kB.
>> So, I think this is not very complicated schema.
>> 
>> My environment is:
>> - Linux, RedHat 6.2, kernel 2.6.32
>> - 2 physical CPU Xeon 5606 (4 cores each)
>> - 32 GB RAM
>> - 2 SSD disks in RAID 0
>> - java version:
>> 
>> java -version
>> java version "1.6.0_26"
>> Java(TM) SE Runtime Environment (build 1.6.0_26-b03) Java HotSpot(TM)
>> 64-Bit Server VM (build 20.1-b02, mixed mode)
>> 
>> - java is running with -server -Xms4096M -Xmx4096M (I tried a lot of
>> other settings and always I have the same effect)
>> - solr has default configuration except Rambuffersize (128MB)
>> - solr 4.0 from nightly builds (2012-02-21 build).
>> 
>> If you need more information, please let me know.
>> I also will try to use profile to see what happens.
>> 
>> Agnieszka
>> 
>> 
>>> -----Original Message-----
>>> From: Jan Høydahl [mailto:jan....@cominvent.com]
>>> Sent: Tuesday, March 13, 2012 9:47 AM
>>> To: solr-user@lucene.apache.org
>>> Subject: Re: solr 3.5 and indexing performance
>>> 
>>> Hi,
>>> 
>>> Have you confirmed that disabling Hunspell in solrconfig gets you back
>>> to normal speed?
>>> What Hunspell configuration and dictionaries do you have?
>>> Can you share more about your environment and documents?
>>> Do you have a chance to run a profiler on your Solr instance? Try i.e.
>>> VisualVM and run the profiler to see what part of the code takes up
>>> the time
>>> http://docs.oracle.com/javase/6/docs/technotes/tools/share/jvisualvm.h
>>> t
>>> ml
>>> 
>>> --
>>> Jan Høydahl, search solution architect Cominvent AS -
>>> www.cominvent.com Solr Training - www.solrtraining.com
>>> 
>>> On 12. mars 2012, at 16:42, Agnieszka Kukałowicz wrote:
>>> 
>>>> Hi guys,
>>>> 
>>>> I have hit the same problem with Hunspell.
>>>> Doing a few tests for 500 000 documents, I've got:
>>>> 
>>>> Hunspell from http://code.google.com/p/lucene-hunspell/ with 3.4
>>>> version -
>>>> 125 documents per second
>>>> Build Hunspell from 4.0 trunk - 11 documents per second.
>>>> 
>>>> All the tests were made on 8 core CPU with 32 GB RAM and index on
>>>> SSD disks.
>>>> For Solr 3.5 I've tried to change JVM heap size, rambuffersize,
>>>> mergefactor but the speed of indexing was about 10 -20 documents per
>>>> second.
>>>> 
>>>> Is it possible that there is some performance bug with Solr 4.0?
>>>> According to previous post the problem exists in 3.5 version.
>>>> 
>>>> Best regards
>>>> Agnieszka Kukałowicz
>>>> 
>>>> 
>>>>> -----Original Message-----
>>>>> From: mizayah [mailto:miza...@gmail.com]
>>>>> Sent: Thursday, February 23, 2012 10:19 AM
>>>>> To: solr-user@lucene.apache.org
>>>>> Subject: Re: solr 3.5 and indexing performance
>>>>> 
>>>>> Ok i found it.
>>>>> 
>>>>> Its becouse of Hunspell which now is in solr. Somehow when im using
>>>>> it by myself in 3.4 it is a lot of faster then one from 3.5.
>>>>> 
>>>>> Dont know about differences, but is there any way i use my old
>>> Google
>>>>> Hunspell jar?
>>>>> 
>>>>> --
>>>>> View this message in context:
>>>>> http://lucene.472066.n3.nabble.com/solr-
>>>>> 3-5-and-indexing-performance-tp3766653p3769139.html
>>>>> Sent from the Solr - User mailing list archive at Nabble.com.

Reply via email to