[jira] Commented: (LUCENE-2911) synchronize grammar/token types across StandardTokenizer, UAX29EmailURLTokenizer, ICUTokenizer, add CJK types.
[ https://issues.apache.org/jira/browse/LUCENE-2911?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12992014#comment-12992014 ] Steven Rowe commented on LUCENE-2911: - The generated top-level domain macro file has a bunch of new entries when I run this, but these are not included in your patch, and I think we should keep this list up-to-date. The patch is missing HangulSupp macro generation in modules/icu/src/tools/.../GenerateJFlexSupplementaryMacros.java, but since the Hangul macro is not used in the jflex grammar, this doesn't cause a problem. It would be nice to remove the hard-coded ranges for the intersection of Hangul & ALetter, but when I tried to use JFlex negation and union to produce the equivalent, memory usage exploded and I couldn't get JFlex to generate, so I guess we'll have to wait on native JFlex supplementary character support before we can change it. > synchronize grammar/token types across StandardTokenizer, > UAX29EmailURLTokenizer, ICUTokenizer, add CJK types. > -- > > Key: LUCENE-2911 > URL: https://issues.apache.org/jira/browse/LUCENE-2911 > Project: Lucene - Java > Issue Type: Sub-task > Components: Analysis >Reporter: Robert Muir >Assignee: Robert Muir > Fix For: 3.1 > > Attachments: LUCENE-2911.patch > > > I'd like to do LUCENE-2906 (better cjk support for these tokenizers) for a > future target such as 3.2 > But, in 3.1 I would like to do a little cleanup first, and synchronize all > these token types, etc. -- This message is automatically generated by JIRA. - For more information on JIRA, see: http://www.atlassian.com/software/jira - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] Commented: (LUCENE-2911) synchronize grammar/token types across StandardTokenizer, UAX29EmailURLTokenizer, ICUTokenizer, add CJK types.
[ https://issues.apache.org/jira/browse/LUCENE-2911?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12992066#comment-12992066 ] Robert Muir commented on LUCENE-2911: - {quote} The generated top-level domain macro file has a bunch of new entries when I run this, but these are not included in your patch, and I think we should keep this list up-to-date. {quote} Yeah, i would re-run it before committing? in general i didn't "re-generate" so you wouldnt see a lot of generated differences in the patch. {quote} The patch is missing HangulSupp macro generation in modules/icu/src/tools/.../GenerateJFlexSupplementaryMacros.java, but since the Hangul macro is not used in the jflex grammar, this doesn't cause a problem. {quote} Oh i did actually mean to include this, sorry I forgot... its a one liner though, I can include this easily. > synchronize grammar/token types across StandardTokenizer, > UAX29EmailURLTokenizer, ICUTokenizer, add CJK types. > -- > > Key: LUCENE-2911 > URL: https://issues.apache.org/jira/browse/LUCENE-2911 > Project: Lucene - Java > Issue Type: Sub-task > Components: Analysis >Reporter: Robert Muir >Assignee: Robert Muir > Fix For: 3.1 > > Attachments: LUCENE-2911.patch > > > I'd like to do LUCENE-2906 (better cjk support for these tokenizers) for a > future target such as 3.2 > But, in 3.1 I would like to do a little cleanup first, and synchronize all > these token types, etc. -- This message is automatically generated by JIRA. - For more information on JIRA, see: http://www.atlassian.com/software/jira - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] Commented: (LUCENE-2911) synchronize grammar/token types across StandardTokenizer, UAX29EmailURLTokenizer, ICUTokenizer, add CJK types.
[ https://issues.apache.org/jira/browse/LUCENE-2911?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12992531#comment-12992531 ] Steven Rowe commented on LUCENE-2911: - bq. I think this one is ready to commit. +1 I applied the patch, jflex generates properly, tests pass > synchronize grammar/token types across StandardTokenizer, > UAX29EmailURLTokenizer, ICUTokenizer, add CJK types. > -- > > Key: LUCENE-2911 > URL: https://issues.apache.org/jira/browse/LUCENE-2911 > Project: Lucene - Java > Issue Type: Sub-task > Components: Analysis >Reporter: Robert Muir >Assignee: Robert Muir > Fix For: 3.1 > > Attachments: LUCENE-2911.patch, LUCENE-2911.patch > > > I'd like to do LUCENE-2906 (better cjk support for these tokenizers) for a > future target such as 3.2 > But, in 3.1 I would like to do a little cleanup first, and synchronize all > these token types, etc. -- This message is automatically generated by JIRA. - For more information on JIRA, see: http://www.atlassian.com/software/jira - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] Commented: (LUCENE-2911) synchronize grammar/token types across StandardTokenizer, UAX29EmailURLTokenizer, ICUTokenizer, add CJK types.
[ https://issues.apache.org/jira/browse/LUCENE-2911?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12992601#comment-12992601 ] Robert Muir commented on LUCENE-2911: - Committed revision 1068979. Now backporting... > synchronize grammar/token types across StandardTokenizer, > UAX29EmailURLTokenizer, ICUTokenizer, add CJK types. > -- > > Key: LUCENE-2911 > URL: https://issues.apache.org/jira/browse/LUCENE-2911 > Project: Lucene - Java > Issue Type: Sub-task > Components: Analysis >Reporter: Robert Muir >Assignee: Robert Muir > Fix For: 3.1 > > Attachments: LUCENE-2911.patch, LUCENE-2911.patch > > > I'd like to do LUCENE-2906 (better cjk support for these tokenizers) for a > future target such as 3.2 > But, in 3.1 I would like to do a little cleanup first, and synchronize all > these token types, etc. -- This message is automatically generated by JIRA. - For more information on JIRA, see: http://www.atlassian.com/software/jira - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org