[ https://issues.apache.org/jira/browse/LUCENE-3233?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Robert Muir updated LUCENE-3233: -------------------------------- Attachment: LUCENE-3233.patch here is a patch with a little microbenchmark... so we have some tuning to do. the benchmark analyzes a short string a million times, that doesn't match any synonyms (actually hte solr default) ||impl||ms|| |SynonymsFilter|1692| |FST with array arcs|2794| |FST with no array arcs|8823| so, disabling the array arcs is a pretty crucial hit here. but we could do other options to speed up this common case, e.g. with daciuk we could build a charrunautomaton of the K-prefixes of the synonyms, this would be really fast to reject these terms that don't match any syns. or we could explicitly put our bytesref output in a byte[], and use long pointers as outputs. or we could speed up FST! But i think its interesting to see how important this parameter is. > HuperDuperSynonymsFilterâ„¢ > ------------------------- > > Key: LUCENE-3233 > URL: https://issues.apache.org/jira/browse/LUCENE-3233 > Project: Lucene - Java > Issue Type: Improvement > Reporter: Robert Muir > Attachments: LUCENE-3223.patch, LUCENE-3233.patch, LUCENE-3233.patch, > LUCENE-3233.patch, LUCENE-3233.patch, LUCENE-3233.patch, LUCENE-3233.patch, > LUCENE-3233.patch, LUCENE-3233.patch, LUCENE-3233.patch, synonyms.zip > > > The current synonymsfilter uses a lot of ram and cpu, especially at build > time. > I think yesterday I heard about "huge synonyms files" three times. > So, I think we should use an FST-based structure, sharing the inputs and > outputs. > And we should be more efficient with the tokenStream api, e.g. using > save/restoreState instead of cloneAttributes() -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira --------------------------------------------------------------------- To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org