[ https://issues.apache.org/jira/browse/SOLR-1706?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12797366#action_12797366 ]
Robert Muir commented on SOLR-1706: ----------------------------------- by the way, i do not have a patch here. i am putting the finishing touches on converting this tokenstream to the new tokenstream API so one alternative is to fix it under SOLR-1657. the problem is i am autogenerating many test cases for all 512 combos of the 9 boolean options across various strings and seeing things like this. so, at the least i would like agreement that its buggy behavior.. if someone knows how to fix the existing code that would be even better, it would make testing easier on me. > wrong tokens output from WordDelimiterFilter when english possessives are in > the text > ------------------------------------------------------------------------------------- > > Key: SOLR-1706 > URL: https://issues.apache.org/jira/browse/SOLR-1706 > Project: Solr > Issue Type: Bug > Components: Schema and Analysis > Affects Versions: 1.4 > Reporter: Robert Muir > > the WordDelimiterFilter english possessive stemming "'s" removal (on by > default) unfortunately causes strange behavior: > below you can see that when I have requested to only output numeric > concatenations (not words), these english possessive stems are still > sometimes output, ignoring the options i have provided, and even then, in a > very inconsistent way. > {code} > assertWdf("Super-Duper-XL500-42-AutoCoder's", 0,0,0,1,0,0,0,0,1, null, > new String[] { "42", "AutoCoder" }, > new int[] { 18, 21 }, > new int[] { 20, 30 }, > new int[] { 1, 1 }); > assertWdf("Super-Duper-XL500-42-AutoCoder's-56", 0,0,0,1,0,0,0,0,1, null, > new String[] { "42", "AutoCoder", "56" }, > new int[] { 18, 21, 33 }, > new int[] { 20, 30, 35 }, > new int[] { 1, 1, 1 }); > assertWdf("Super-Duper-XL500-AB-AutoCoder's", 0,0,0,1,0,0,0,0,1, null, > new String[] { }, > new int[] { }, > new int[] { }, > new int[] { }); > assertWdf("Super-Duper-XL500-42-AutoCoder's-BC", 0,0,0,1,0,0,0,0,1, null, > new String[] { "42" }, > new int[] { 18 }, > new int[] { 20 }, > new int[] { 1 }); > {code} > where assertWdf is > {code} > void assertWdf(String text, int generateWordParts, int generateNumberParts, > int catenateWords, int catenateNumbers, int catenateAll, > int splitOnCaseChange, int preserveOriginal, int splitOnNumerics, > int stemEnglishPossessive, CharArraySet protWords, String expected[], > int startOffsets[], int endOffsets[], String types[], int posIncs[]) > throws IOException { > TokenStream ts = new WhitespaceTokenizer(new StringReader(text)); > WordDelimiterFilter wdf = new WordDelimiterFilter(ts, generateWordParts, > generateNumberParts, catenateWords, catenateNumbers, catenateAll, > splitOnCaseChange, preserveOriginal, splitOnNumerics, > stemEnglishPossessive, protWords); > assertTokenStreamContents(wdf, expected, startOffsets, endOffsets, types, > posIncs); > } > {code} -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.