Yes the branch_3x works for me as well. The addition of the OffsetAttribute probably corrected this issue. I will either switch to WhitespaceAnalyzer, patch my distribution or wait for 3.6 to resolve this.
Thanks. On Thu, Dec 15, 2011 at 4:17 PM, Brandon Fish <brandon.j.f...@gmail.com>wrote: > Hi Steve, > > I was using branch 3.5. I will try this on tip of branch_3x too. > > Thanks. > > > On Thu, Dec 15, 2011 at 4:14 PM, Steven A Rowe <sar...@syr.edu> wrote: > >> Hi Brandon, >> >> When I add the following to SpellingQueryConverterTest.java on the tip of >> branch_3x (will be released as Solr 3.6), the test succeeds: >> >> @Test >> public void testStandardAnalyzerWithHyphen() { >> SpellingQueryConverter converter = new SpellingQueryConverter(); >> converter.init(new NamedList()); >> converter.setAnalyzer(new StandardAnalyzer(Version.LUCENE_35)); >> String original = "another-test"; >> Collection<Token> tokens = converter.convert(original); >> assertTrue("tokens is null and it shouldn't be", tokens != null); >> assertEquals("tokens Size: " + tokens.size() + " is not 2", 2, >> tokens.size()); >> assertTrue("Token offsets do not match", isOffsetCorrect(original, >> tokens)); >> } >> >> What version of Solr/Lucene are you using? >> >> Steve >> >> > -----Original Message----- >> > From: Brandon Fish [mailto:brandon.j.f...@gmail.com] >> > Sent: Thursday, December 15, 2011 3:08 PM >> > To: solr-user@lucene.apache.org >> > Subject: Is there an issue with hypens in SpellChecker with >> > StandardTokenizer? >> > >> > I am getting an error using the SpellChecker component with the query >> > "another-test" >> > java.lang.StringIndexOutOfBoundsException: String index out of range: -7 >> > >> > This appears to be related to this >> > issue<https://issues.apache.org/jira/browse/SOLR-1630> which >> > has been marked as fixed. My configuration and test case that follows >> > appear to reproduce the error I am seeing. Both "another" and "test" get >> > changed into tokens with start and end offsets of 0 and 12. >> > <analyzer> >> > <tokenizer class="solr.StandardTokenizerFactory"/> >> > <filter class="solr.StopFilterFactory" ignoreCase="true" >> > words="stopwords.txt"/> >> > <filter class="solr.LowerCaseFilterFactory"/> >> > </analyzer> >> > >> > &spellcheck=true&spellcheck.collate=true >> > >> > Is this an issue with my configuration/test or is there an issue with >> the >> > SpellingQueryConverter? Is there a recommended work around such as the >> > WhitespaceTokenizer as mention in the issue comments? >> > >> > Thank you for your help. >> > >> > package org.apache.solr.spelling; >> > import static org.junit.Assert.assertTrue; >> > import java.util.Collection; >> > import org.apache.lucene.analysis.Token; >> > import org.apache.lucene.analysis.standard.StandardAnalyzer; >> > import org.apache.lucene.util.Version; >> > import org.apache.solr.common.util.NamedList; >> > import org.junit.Test; >> > public class SimpleQueryConverterTest { >> > @Test >> > public void testSimpleQueryConversion() { >> > SpellingQueryConverter converter = new SpellingQueryConverter(); >> > converter.init(new NamedList()); >> > converter.setAnalyzer(new StandardAnalyzer(Version.LUCENE_35)); >> > String original = "another-test"; >> > Collection<Token> tokens = converter.convert(original); >> > assertTrue("Token offsets do not match", >> > isOffsetCorrect(original, tokens)); >> > } >> > private boolean isOffsetCorrect(String s, Collection<Token> tokens) { >> > for (Token token : tokens) { >> > int start = token.startOffset(); >> > int end = token.endOffset(); >> > if (!s.substring(start, end).equals(token.toString())) >> > return false; >> > } >> > return true; >> > } >> > } >> > >