[ https://issues.apache.org/jira/browse/LUCENE-6913?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15430895#comment-15430895 ]
Robert Muir commented on LUCENE-6913: ------------------------------------- I started looking into this but I think our find-replace hacks in our ant jflex task are incompatible with the code generated from jflex's master branch. I tried messing around with them but couldn't make tests pass, so I'm stuck on that first. There is already an issue at jflex (https://github.com/jflex-de/jflex/issues/196), I added a comment with the idea I am looking at. > Standard/Classic/UAX tokenizers could be more ram efficient > ----------------------------------------------------------- > > Key: LUCENE-6913 > URL: https://issues.apache.org/jira/browse/LUCENE-6913 > Project: Lucene - Core > Issue Type: Improvement > Reporter: Robert Muir > Attachments: LUCENE-6913.not.a.patch > > > These tokenizers map codepoints to character classes with the following > datastructure (loaded in clinit): > {noformat} > private static char [] zzUnpackCMap(String packed) { > char [] map = new char[0x110000]; > {noformat} > This requires 2MB RAM for each tokenizer class (in trunk 6MB if all 3 classes > are loaded, in branch_5x 10MB since there are 2 additional backwards compat > classes). > On the other hand, none of our tokenizers actually use a huge number of > character classes, so {{char}} is overkill: e.g. this map can safely be a > byte [] and we can save half the memory. Perhaps it could make these > tokenizers faster too. -- This message was sent by Atlassian JIRA (v6.3.4#6332) --------------------------------------------------------------------- To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org