[
https://issues.apache.org/jira/browse/LUCENE-1545?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12916033#action_12916033
]
Steven Rowe commented on LUCENE-1545:
-------------------------------------
I updated AnalyzerTest.java:
{code:java}
import java.io.FileOutputStream;
import java.io.OutputStreamWriter;
import java.io.StringReader;
import org.apache.lucene.analysis.TokenStream;
import org.apache.lucene.analysis.standard.StandardAnalyzer;
import org.apache.lucene.util.Version;
public class AnalyzerTest {
public static void test() throws Exception {
StandardAnalyzer a = new StandardAnalyzer(Version.LUCENE_31);
TokenStream ts = a.tokenStream("", new StringReader("moͤchte m mo\u0364chte
"));
OutputStreamWriter writer = new OutputStreamWriter(new
FileOutputStream("output.txt"), "UTF-8");
while (ts.incrementToken()) {
writer.append(ts.toString()).append(System.getProperty("line.separator"));
}
writer.flush();
writer.close();
}
public static void main(String[] argv) throws Exception {
test();
}
}
{code}
Here's what goes into output.txt when I compile {{AnalyzerTest.java}} with
{{javac -encoding UTF-8 -cp
lucene/dev/branches/branch_3x/lucene/build/lucene-core-3.1-SNAPSHOT.jar"
AnalyzerTest}}:
{code:java}
(moͤchte,startOffset=0,endOffset=7,positionIncrement=1,type=<ALPHANUM>)
(m,startOffset=8,endOffset=9,positionIncrement=1,type=<ALPHANUM>)
(moͤchte,startOffset=10,endOffset=17,positionIncrement=1,type=<ALPHANUM>)
{code}
With LUCENE-2167 committed on the 3.X branch and on trunk, I think this issue
is resolved. Please reopen if you see different behavior.
> Standard analyzer does not correctly tokenize combining character U+0364
> COMBINING LATIN SMALL LETTRE E
> -------------------------------------------------------------------------------------------------------
>
> Key: LUCENE-1545
> URL: https://issues.apache.org/jira/browse/LUCENE-1545
> Project: Lucene - Java
> Issue Type: Bug
> Components: contrib/analyzers
> Affects Versions: 2.4
> Environment: Linux x86_64, Sun Java 1.6
> Reporter: Andreas Hauser
> Priority: Minor
> Fix For: 3.1, 4.0
>
> Attachments: AnalyzerTest.java
>
>
> Standard analyzer does not correctly tokenize combining character U+0364
> COMBINING LATIN SMALL LETTRE E.
> The word "moͤchte" is incorrectly tokenized into "mo" "chte", the combining
> character is lost.
> Expected result is only on token "moͤchte".
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
