[ https://issues.apache.org/jira/browse/LUCENE-966?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12517003 ]
Michael McCandless commented on LUCENE-966: ------------------------------------------- Oddly, the patch for TestStandardAnalyzer failed to apply for me (but the rest did), so I manually merged those changes in. Oh, I see: it was the "Korean words" test -- somehow the characters got mapped to ?'s in your patch. This is why the patch didn't apply, I think? Maybe you used a diffing tool that wasn't happy with unicode or something? I also see the quality test failing in contrib benchmark. I fear something about the new StandardAnalyzer is in fact causing this test to fail (it passes on a clean checkout). That test uses StandardAnalyzer. KO I re-tested the old vs new StandardAnalyzer on Wikipedia and I still found some differences, I think only on these very large URL-like tokens. Here's one: OLD (money.cnn.com,1382,1395,type=<HOST>) (magazines,1396,1405,type=<ALPHANUM>) (fortune,1406,1413,type=<ALPHANUM>) (fortune,1414,1421,type=<ALPHANUM>) (archive/2007/03/19/8402357,1422,1448,type=<NUM>) (index.htm,1449,1458,type=<HOST>) NEW (/money.cnn.com/magazines/fortune/fortune_archive/2007/03/19/8402357/index.htm,1381,1458,type=<NUM>) I like the NEW behavior better but I fear we should try to match the old one? Here's another one: OLD (mid-20th,2436,2444,type=<NUM>) NEW (mid,2436,2439,type=<ALPHANUM>) (-20th,2439,2444,type=<NUM>) I like the old behavior better here. Another one: OLD (safari-0-sheikh,12011,12026,type=<NUM>) (zayed,12027,12032,type=<ALPHANUM>) (grand,12033,12038,type=<ALPHANUM>) (mosque.jpg,12039,12049,type=<HOST>) NEW (safari,12011,12017,type=<ALPHANUM>) (0-sheikh-zayed-grand-mosque.jpg,12018,12049,type=<NUM>) Another one: OLD (semitica-01.png,616,631,type=<NUM>) NEW (-semitica-01.png,615,631,type=<NUM>) > A faster JFlex-based replacement for StandardAnalyzer > ----------------------------------------------------- > > Key: LUCENE-966 > URL: https://issues.apache.org/jira/browse/LUCENE-966 > Project: Lucene - Java > Issue Type: Improvement > Components: Analysis > Reporter: Stanislaw Osinski > Fix For: 2.3 > > Attachments: AnalyzerBenchmark.java, jflex-analyzer-patch.txt, > jflex-analyzer-r560135-patch.txt, jflex-analyzer-r561292-patch.txt, > jflex-analyzer-r561693-compatibility.txt > > > JFlex (http://www.jflex.de/) can be used to generate a faster (up to several > times) replacement for StandardAnalyzer. Will add a patch and a simple > benchmark code in a while. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online. --------------------------------------------------------------------- To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]