[jira] [Commented] (LUCENE-8464) Implement ConstantScoreScorer#setMinCompetitiveScore
[ https://issues.apache.org/jira/browse/LUCENE-8464?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16678314#comment-16678314 ] Christophe Bismuth commented on LUCENE-8464: Hi, I'd like to work on this one. > Implement ConstantScoreScorer#setMinCompetitiveScore > > > Key: LUCENE-8464 > URL: https://issues.apache.org/jira/browse/LUCENE-8464 > Project: Lucene - Core > Issue Type: Improvement >Reporter: Adrien Grand >Priority: Minor > Labels: newdev > > We should make it so the iterator returns NO_MORE_DOCS after > setMinCompetitiveScore is called with a value that is greater than the > constant score. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (LUCENE-8464) Implement ConstantScoreScorer#setMinCompetitiveScore
[ https://issues.apache.org/jira/browse/LUCENE-8464?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16680002#comment-16680002 ] Christophe Bismuth commented on LUCENE-8464: Hi, I've opened PR [#495|https://github.com/apache/lucene-solr/pull/495] on GitHub to implement this, could you please tell me if this implementation could fit? Thank you. > Implement ConstantScoreScorer#setMinCompetitiveScore > > > Key: LUCENE-8464 > URL: https://issues.apache.org/jira/browse/LUCENE-8464 > Project: Lucene - Core > Issue Type: Improvement >Reporter: Adrien Grand >Priority: Minor > Labels: newdev > Time Spent: 10m > Remaining Estimate: 0h > > We should make it so the iterator returns NO_MORE_DOCS after > setMinCompetitiveScore is called with a value that is greater than the > constant score. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (LUCENE-8463) Early-terminate queries sorted by SortField.DOC
[ https://issues.apache.org/jira/browse/LUCENE-8463?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16681232#comment-16681232 ] Christophe Bismuth commented on LUCENE-8463: Hi, I'd like to work on this one. > Early-terminate queries sorted by SortField.DOC > --- > > Key: LUCENE-8463 > URL: https://issues.apache.org/jira/browse/LUCENE-8463 > Project: Lucene - Core > Issue Type: Improvement >Reporter: Adrien Grand >Priority: Minor > Labels: newdev > > Currently TopFieldCollector only early-terminates when the search sort is a > prefix of the index sort, but it could also early-terminate when sorting by > doc id. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (LUCENE-8463) Early-terminate queries sorted by SortField.DOC
[ https://issues.apache.org/jira/browse/LUCENE-8463?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16683584#comment-16683584 ] Christophe Bismuth commented on LUCENE-8463: Hi, I've opened PR [#496|https://github.com/apache/lucene-solr/pull/496] to implement this improvement. > Early-terminate queries sorted by SortField.DOC > --- > > Key: LUCENE-8463 > URL: https://issues.apache.org/jira/browse/LUCENE-8463 > Project: Lucene - Core > Issue Type: Improvement >Reporter: Adrien Grand >Priority: Minor > Labels: newdev > Time Spent: 10m > Remaining Estimate: 0h > > Currently TopFieldCollector only early-terminates when the search sort is a > prefix of the index sort, but it could also early-terminate when sorting by > doc id. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (LUCENE-8026) ExitableDirectoryReader does not instrument points
[ https://issues.apache.org/jira/browse/LUCENE-8026?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16683763#comment-16683763 ] Christophe Bismuth commented on LUCENE-8026: Hi, I've opened PR [#497|https://github.com/apache/lucene-solr/pull/497] to fix this bug. > ExitableDirectoryReader does not instrument points > -- > > Key: LUCENE-8026 > URL: https://issues.apache.org/jira/browse/LUCENE-8026 > Project: Lucene - Core > Issue Type: Bug >Reporter: Adrien Grand >Priority: Trivial > Labels: newdev > Time Spent: 10m > Remaining Estimate: 0h > > This means it cannot interrupt range or geo queries. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (LUCENE-8552) optimize getMergedFieldInfos for one-segment FieldInfos
[ https://issues.apache.org/jira/browse/LUCENE-8552?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16683779#comment-16683779 ] Christophe Bismuth commented on LUCENE-8552: Hi, I'd like to work on this one. > optimize getMergedFieldInfos for one-segment FieldInfos > --- > > Key: LUCENE-8552 > URL: https://issues.apache.org/jira/browse/LUCENE-8552 > Project: Lucene - Core > Issue Type: New Feature >Reporter: David Smiley >Priority: Minor > > FieldInfos.getMergedFieldInfos could trivially return the FieldInfos of the > first and only LeafReader if there is only one LeafReader. > Also... if there is more than one LeafReader, and if FieldInfos & FieldInfo > implemented equals() & hashCode() (including a cached hashCode), maybe we > could also call equals() iterating through the FieldInfos to see if we should > bother adding it to the FieldInfos.Builder? Admittedly this is speculative; > may not be worth the bother. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (LUCENE-8552) optimize getMergedFieldInfos for one-segment FieldInfos
[ https://issues.apache.org/jira/browse/LUCENE-8552?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16683801#comment-16683801 ] Christophe Bismuth commented on LUCENE-8552: Is the underlying idea to limit the number of {{FieldInfos}} instances added to the {{FieldInfos.Builder}} for performances purpose? > optimize getMergedFieldInfos for one-segment FieldInfos > --- > > Key: LUCENE-8552 > URL: https://issues.apache.org/jira/browse/LUCENE-8552 > Project: Lucene - Core > Issue Type: New Feature >Reporter: David Smiley >Priority: Minor > > FieldInfos.getMergedFieldInfos could trivially return the FieldInfos of the > first and only LeafReader if there is only one LeafReader. > Also... if there is more than one LeafReader, and if FieldInfos & FieldInfo > implemented equals() & hashCode() (including a cached hashCode), maybe we > could also call equals() iterating through the FieldInfos to see if we should > bother adding it to the FieldInfos.Builder? Admittedly this is speculative; > may not be worth the bother. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (LUCENE-8294) KeywordTokenizer hangs with user misconfigured inputs
[ https://issues.apache.org/jira/browse/LUCENE-8294?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16683907#comment-16683907 ] Christophe Bismuth commented on LUCENE-8294: Issue can be closed as fixed in [906679adc80f0fad1e5c311b03023c7bd95633d7|https://github.com/apache/lucene-solr/commit/906679adc80f0fad1e5c311b03023c7bd95633d7]. > KeywordTokenizer hangs with user misconfigured inputs > - > > Key: LUCENE-8294 > URL: https://issues.apache.org/jira/browse/LUCENE-8294 > Project: Lucene - Core > Issue Type: Bug >Affects Versions: 2.1 >Reporter: John Doe >Priority: Minor > > When a user configures the bufferSize to be 0, the while loop in > KeywordTokenizer.next() function hangs endlessly. Here is the code snippet. > {code:java} > public KeywordTokenizer(Reader input, int bufferSize) { > super(input); > this.buffer = new char[bufferSize];//bufferSize is misconfigured with 0 > this.done = false; > } > public Token next() throws IOException { > if (!done) { > done = true; > StringBuffer buffer = new StringBuffer(); > int length; > while (true) { > length = input.read(this.buffer); //length is always 0 when the > buffer.size == 0 > if (length == -1) break; > buffer.append(this.buffer, 0, length); > } > String text = buffer.toString(); > return new Token(text, 0, text.length()); > } > return null; > } > {code} -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (LUCENE-8464) Implement ConstantScoreScorer#setMinCompetitiveScore
[ https://issues.apache.org/jira/browse/LUCENE-8464?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16719272#comment-16719272 ] Christophe Bismuth commented on LUCENE-8464: Thanks a lot [~romseygeek], you made my day :D [~jim.ferenczi] has made some really great mentoring with me on this one (y) I hope to find some other great issues to work on! > Implement ConstantScoreScorer#setMinCompetitiveScore > > > Key: LUCENE-8464 > URL: https://issues.apache.org/jira/browse/LUCENE-8464 > Project: Lucene - Core > Issue Type: Improvement >Reporter: Adrien Grand >Priority: Minor > Labels: newdev > Fix For: master (8.0) > > Time Spent: 8h > Remaining Estimate: 0h > > We should make it so the iterator returns NO_MORE_DOCS after > setMinCompetitiveScore is called with a value that is greater than the > constant score. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (LUCENE-8552) optimize getMergedFieldInfos for one-segment FieldInfos
[ https://issues.apache.org/jira/browse/LUCENE-8552?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16686470#comment-16686470 ] Christophe Bismuth commented on LUCENE-8552: Thanks a lot [~dsmiley]! I'll come back to you as soon as I have a patch. > optimize getMergedFieldInfos for one-segment FieldInfos > --- > > Key: LUCENE-8552 > URL: https://issues.apache.org/jira/browse/LUCENE-8552 > Project: Lucene - Core > Issue Type: New Feature >Reporter: David Smiley >Priority: Minor > > FieldInfos.getMergedFieldInfos could trivially return the FieldInfos of the > first and only LeafReader if there is only one LeafReader. > Also... if there is more than one LeafReader, and if FieldInfos & FieldInfo > implemented equals() & hashCode() (including a cached hashCode), maybe we > could also call equals() iterating through the FieldInfos to see if we should > bother adding it to the FieldInfos.Builder? Admittedly this is speculative; > may not be worth the bother. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Comment Edited] (LUCENE-8552) optimize getMergedFieldInfos for one-segment FieldInfos
[ https://issues.apache.org/jira/browse/LUCENE-8552?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16683801#comment-16683801 ] Christophe Bismuth edited comment on LUCENE-8552 at 11/15/18 8:42 AM: -- Is the underlying idea to limit the number of {{FieldInfo}} instances added to the {{FieldInfos.Builder}} for performances purpose? was (Author: cbismuth): Is the underlying idea to limit the number of {{FieldInfos}} instances added to the {{FieldInfos.Builder}} for performances purpose? > optimize getMergedFieldInfos for one-segment FieldInfos > --- > > Key: LUCENE-8552 > URL: https://issues.apache.org/jira/browse/LUCENE-8552 > Project: Lucene - Core > Issue Type: New Feature >Reporter: David Smiley >Priority: Minor > > FieldInfos.getMergedFieldInfos could trivially return the FieldInfos of the > first and only LeafReader if there is only one LeafReader. > Also... if there is more than one LeafReader, and if FieldInfos & FieldInfo > implemented equals() & hashCode() (including a cached hashCode), maybe we > could also call equals() iterating through the FieldInfos to see if we should > bother adding it to the FieldInfos.Builder? Admittedly this is speculative; > may not be worth the bother. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Comment Edited] (LUCENE-8552) optimize getMergedFieldInfos for one-segment FieldInfos
[ https://issues.apache.org/jira/browse/LUCENE-8552?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16687694#comment-16687694 ] Christophe Bismuth edited comment on LUCENE-8552 at 11/15/18 9:21 AM: -- Hi [~dsmiley], I've opened PR [#8552|https://github.com/apache/lucene-solr/pull/499] on GitHub to implement this feature. was (Author: cbismuth): Hi [~dsmiley], I've opened PR [#8552|https://github.com/apache/lucene-solr/pull/499] on GitHub to implement this improvement. > optimize getMergedFieldInfos for one-segment FieldInfos > --- > > Key: LUCENE-8552 > URL: https://issues.apache.org/jira/browse/LUCENE-8552 > Project: Lucene - Core > Issue Type: New Feature >Reporter: David Smiley >Priority: Minor > Time Spent: 10m > Remaining Estimate: 0h > > FieldInfos.getMergedFieldInfos could trivially return the FieldInfos of the > first and only LeafReader if there is only one LeafReader. > Also... if there is more than one LeafReader, and if FieldInfos & FieldInfo > implemented equals() & hashCode() (including a cached hashCode), maybe we > could also call equals() iterating through the FieldInfos to see if we should > bother adding it to the FieldInfos.Builder? Admittedly this is speculative; > may not be worth the bother. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (LUCENE-8552) optimize getMergedFieldInfos for one-segment FieldInfos
[ https://issues.apache.org/jira/browse/LUCENE-8552?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16687694#comment-16687694 ] Christophe Bismuth commented on LUCENE-8552: Hi [~dsmiley], I've opened PR [#8552|https://github.com/apache/lucene-solr/pull/499] on GitHub to implement this improvement. > optimize getMergedFieldInfos for one-segment FieldInfos > --- > > Key: LUCENE-8552 > URL: https://issues.apache.org/jira/browse/LUCENE-8552 > Project: Lucene - Core > Issue Type: New Feature >Reporter: David Smiley >Priority: Minor > Time Spent: 10m > Remaining Estimate: 0h > > FieldInfos.getMergedFieldInfos could trivially return the FieldInfos of the > first and only LeafReader if there is only one LeafReader. > Also... if there is more than one LeafReader, and if FieldInfos & FieldInfo > implemented equals() & hashCode() (including a cached hashCode), maybe we > could also call equals() iterating through the FieldInfos to see if we should > bother adding it to the FieldInfos.Builder? Admittedly this is speculative; > may not be worth the bother. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (LUCENE-8551) Purge unused FieldInfo on segment merge
[ https://issues.apache.org/jira/browse/LUCENE-8551?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16687703#comment-16687703 ] Christophe Bismuth commented on LUCENE-8551: Sounds challenging, I'd like to work in it! > Purge unused FieldInfo on segment merge > --- > > Key: LUCENE-8551 > URL: https://issues.apache.org/jira/browse/LUCENE-8551 > Project: Lucene - Core > Issue Type: Improvement > Components: core/index >Reporter: David Smiley >Priority: Major > > If a field is effectively unused (no norms, terms index, term vectors, > docValues, stored value, points index), it will nonetheless hang around in > FieldInfos indefinitely. It would be nice to be able to recognize an unused > FieldInfo and allow it to disappear after a merge (or two). > SegmentMerger merges FieldInfo (from each segment) as nearly the first thing > it does. After that, the different index parts, before it's known what's > "used" or not. After writing, we theoretically know which fields are used or > not, though we're not doing any bookkeeping to track it. Maybe we should > track the fields used during writing so we write a filtered merged fieldInfo > at the end instead of unfiltered up front? Or perhaps upon reading a > segment, we make it cheap/easy for each index type (e.g. terms index, stored > fields, ...) to know which fields have data for the corresponding type. > Then, on a subsequent merge, we know up front to filter the FieldInfos. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (LUCENE-8551) Purge unused FieldInfo on segment merge
[ https://issues.apache.org/jira/browse/LUCENE-8551?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16687705#comment-16687705 ] Christophe Bismuth commented on LUCENE-8551: I'll first implement unused {{FieldInfo}} tracking and let you know. > Purge unused FieldInfo on segment merge > --- > > Key: LUCENE-8551 > URL: https://issues.apache.org/jira/browse/LUCENE-8551 > Project: Lucene - Core > Issue Type: Improvement > Components: core/index >Reporter: David Smiley >Priority: Major > > If a field is effectively unused (no norms, terms index, term vectors, > docValues, stored value, points index), it will nonetheless hang around in > FieldInfos indefinitely. It would be nice to be able to recognize an unused > FieldInfo and allow it to disappear after a merge (or two). > SegmentMerger merges FieldInfo (from each segment) as nearly the first thing > it does. After that, the different index parts, before it's known what's > "used" or not. After writing, we theoretically know which fields are used or > not, though we're not doing any bookkeeping to track it. Maybe we should > track the fields used during writing so we write a filtered merged fieldInfo > at the end instead of unfiltered up front? Or perhaps upon reading a > segment, we make it cheap/easy for each index type (e.g. terms index, stored > fields, ...) to know which fields have data for the corresponding type. > Then, on a subsequent merge, we know up front to filter the FieldInfos. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Comment Edited] (LUCENE-8551) Purge unused FieldInfo on segment merge
[ https://issues.apache.org/jira/browse/LUCENE-8551?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16687703#comment-16687703 ] Christophe Bismuth edited comment on LUCENE-8551 at 11/15/18 2:40 PM: -- Sounds challenging, I'd like to work on it! was (Author: cbismuth): Sounds challenging, I'd like to work in it! > Purge unused FieldInfo on segment merge > --- > > Key: LUCENE-8551 > URL: https://issues.apache.org/jira/browse/LUCENE-8551 > Project: Lucene - Core > Issue Type: Improvement > Components: core/index >Reporter: David Smiley >Priority: Major > > If a field is effectively unused (no norms, terms index, term vectors, > docValues, stored value, points index), it will nonetheless hang around in > FieldInfos indefinitely. It would be nice to be able to recognize an unused > FieldInfo and allow it to disappear after a merge (or two). > SegmentMerger merges FieldInfo (from each segment) as nearly the first thing > it does. After that, the different index parts, before it's known what's > "used" or not. After writing, we theoretically know which fields are used or > not, though we're not doing any bookkeeping to track it. Maybe we should > track the fields used during writing so we write a filtered merged fieldInfo > at the end instead of unfiltered up front? Or perhaps upon reading a > segment, we make it cheap/easy for each index type (e.g. terms index, stored > fields, ...) to know which fields have data for the corresponding type. > Then, on a subsequent merge, we know up front to filter the FieldInfos. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (LUCENE-8551) Purge unused FieldInfo on segment merge
[ https://issues.apache.org/jira/browse/LUCENE-8551?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16688170#comment-16688170 ] Christophe Bismuth commented on LUCENE-8551: The overhead makes me think a dedicated optimize/purge API would be wiser. But, I don't know NRT internals enough to have a valuable opinion on the second point. > Purge unused FieldInfo on segment merge > --- > > Key: LUCENE-8551 > URL: https://issues.apache.org/jira/browse/LUCENE-8551 > Project: Lucene - Core > Issue Type: Improvement > Components: core/index >Reporter: David Smiley >Priority: Major > > If a field is effectively unused (no norms, terms index, term vectors, > docValues, stored value, points index), it will nonetheless hang around in > FieldInfos indefinitely. It would be nice to be able to recognize an unused > FieldInfo and allow it to disappear after a merge (or two). > SegmentMerger merges FieldInfo (from each segment) as nearly the first thing > it does. After that, the different index parts, before it's known what's > "used" or not. After writing, we theoretically know which fields are used or > not, though we're not doing any bookkeeping to track it. Maybe we should > track the fields used during writing so we write a filtered merged fieldInfo > at the end instead of unfiltered up front? Or perhaps upon reading a > segment, we make it cheap/easy for each index type (e.g. terms index, stored > fields, ...) to know which fields have data for the corresponding type. > Then, on a subsequent merge, we know up front to filter the FieldInfos. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (LUCENE-8548) Reevaluate scripts boundary break in Nori's tokenizer
[ https://issues.apache.org/jira/browse/LUCENE-8548?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16688242#comment-16688242 ] Christophe Bismuth commented on LUCENE-8548: Hi [~jim.ferenczi], have you started to work on patch or maybe I could help? I'm not an Unicode guru but I can read the docs and learn. Feel free to let me know. > Reevaluate scripts boundary break in Nori's tokenizer > - > > Key: LUCENE-8548 > URL: https://issues.apache.org/jira/browse/LUCENE-8548 > Project: Lucene - Core > Issue Type: Improvement >Reporter: Jim Ferenczi >Priority: Minor > > This was first reported in https://issues.apache.org/jira/browse/LUCENE-8526: > {noformat} > Tokens are split on different character POS types (which seem to not quite > line up with Unicode character blocks), which leads to weird results for > non-CJK tokens: > εἰμί is tokenized as three tokens: ε/SL(Foreign language) + ἰ/SY(Other > symbol) + μί/SL(Foreign language) > ka̠k̚t͡ɕ͈a̠k̚ is tokenized as ka/SL(Foreign language) + ̠/SY(Other symbol) + > k/SL(Foreign language) + ̚/SY(Other symbol) + t/SL(Foreign language) + > ͡ɕ͈/SY(Other symbol) + a/SL(Foreign language) + ̠/SY(Other symbol) + > k/SL(Foreign language) + ̚/SY(Other symbol) > Ба̀лтичко̄ is tokenized as ба/SL(Foreign language) + ̀/SY(Other symbol) + > лтичко/SL(Foreign language) + ̄/SY(Other symbol) > don't is tokenized as don + t; same for don't (with a curly apostrophe). > אוֹג׳וּ is tokenized as אוֹג/SY(Other symbol) + וּ/SY(Other symbol) > Мoscow (with a Cyrillic М and the rest in Latin) is tokenized as м + oscow > While it is still possible to find these words using Nori, there are many > more chances for false positives when the tokens are split up like this. In > particular, individual numbers and combining diacritics are indexed > separately (e.g., in the Cyrillic example above), which can lead to a > performance hit on large corpora like Wiktionary or Wikipedia. > Work around: use a character filter to get rid of combining diacritics before > Nori processes the text. This doesn't solve the Greek, Hebrew, or English > cases, though. > Suggested fix: Characters in related Unicode blocks—like "Greek" and "Greek > Extended", or "Latin" and "IPA Extensions"—should not trigger token splits. > Combining diacritics should not trigger token splits. Non-CJK text should be > tokenized on spaces and punctuation, not by character type shifts. > Apostrophe-like characters should not trigger token splits (though I could > see someone disagreeing on this one).{noformat} > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (LUCENE-8464) Implement ConstantScoreScorer#setMinCompetitiveScore
[ https://issues.apache.org/jira/browse/LUCENE-8464?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16689497#comment-16689497 ] Christophe Bismuth commented on LUCENE-8464: Thank you [~jim.ferenczi], it was a great experience (y) > Implement ConstantScoreScorer#setMinCompetitiveScore > > > Key: LUCENE-8464 > URL: https://issues.apache.org/jira/browse/LUCENE-8464 > Project: Lucene - Core > Issue Type: Improvement >Reporter: Adrien Grand >Priority: Minor > Labels: newdev > Fix For: master (8.0) > > Time Spent: 7h 50m > Remaining Estimate: 0h > > We should make it so the iterator returns NO_MORE_DOCS after > setMinCompetitiveScore is called with a value that is greater than the > constant score. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (LUCENE-8463) Early-terminate queries sorted by SortField.DOC
[ https://issues.apache.org/jira/browse/LUCENE-8463?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16689987#comment-16689987 ] Christophe Bismuth commented on LUCENE-8463: Thanks a lot [~jim.ferenczi] :D > Early-terminate queries sorted by SortField.DOC > --- > > Key: LUCENE-8463 > URL: https://issues.apache.org/jira/browse/LUCENE-8463 > Project: Lucene - Core > Issue Type: Improvement >Reporter: Adrien Grand >Priority: Minor > Labels: newdev > Fix For: master (8.0), 7.7 > > Time Spent: 2h > Remaining Estimate: 0h > > Currently TopFieldCollector only early-terminates when the search sort is a > prefix of the index sort, but it could also early-terminate when sorting by > doc id. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (LUCENE-8548) Reevaluate scripts boundary break in Nori's tokenizer
[ https://issues.apache.org/jira/browse/LUCENE-8548?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16690004#comment-16690004 ] Christophe Bismuth commented on LUCENE-8548: Yes, I'm interested in this issue (y) I'll start to work on it and let you know. > Reevaluate scripts boundary break in Nori's tokenizer > - > > Key: LUCENE-8548 > URL: https://issues.apache.org/jira/browse/LUCENE-8548 > Project: Lucene - Core > Issue Type: Improvement >Reporter: Jim Ferenczi >Priority: Minor > > This was first reported in https://issues.apache.org/jira/browse/LUCENE-8526: > {noformat} > Tokens are split on different character POS types (which seem to not quite > line up with Unicode character blocks), which leads to weird results for > non-CJK tokens: > εἰμί is tokenized as three tokens: ε/SL(Foreign language) + ἰ/SY(Other > symbol) + μί/SL(Foreign language) > ka̠k̚t͡ɕ͈a̠k̚ is tokenized as ka/SL(Foreign language) + ̠/SY(Other symbol) + > k/SL(Foreign language) + ̚/SY(Other symbol) + t/SL(Foreign language) + > ͡ɕ͈/SY(Other symbol) + a/SL(Foreign language) + ̠/SY(Other symbol) + > k/SL(Foreign language) + ̚/SY(Other symbol) > Ба̀лтичко̄ is tokenized as ба/SL(Foreign language) + ̀/SY(Other symbol) + > лтичко/SL(Foreign language) + ̄/SY(Other symbol) > don't is tokenized as don + t; same for don't (with a curly apostrophe). > אוֹג׳וּ is tokenized as אוֹג/SY(Other symbol) + וּ/SY(Other symbol) > Мoscow (with a Cyrillic М and the rest in Latin) is tokenized as м + oscow > While it is still possible to find these words using Nori, there are many > more chances for false positives when the tokens are split up like this. In > particular, individual numbers and combining diacritics are indexed > separately (e.g., in the Cyrillic example above), which can lead to a > performance hit on large corpora like Wiktionary or Wikipedia. > Work around: use a character filter to get rid of combining diacritics before > Nori processes the text. This doesn't solve the Greek, Hebrew, or English > cases, though. > Suggested fix: Characters in related Unicode blocks—like "Greek" and "Greek > Extended", or "Latin" and "IPA Extensions"—should not trigger token splits. > Combining diacritics should not trigger token splits. Non-CJK text should be > tokenized on spaces and punctuation, not by character type shifts. > Apostrophe-like characters should not trigger token splits (though I could > see someone disagreeing on this one).{noformat} > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (LUCENE-8552) optimize getMergedFieldInfos for one-segment FieldInfos
[ https://issues.apache.org/jira/browse/LUCENE-8552?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16690466#comment-16690466 ] Christophe Bismuth commented on LUCENE-8552: Thank you for your help [~dsmiley] (y) > optimize getMergedFieldInfos for one-segment FieldInfos > --- > > Key: LUCENE-8552 > URL: https://issues.apache.org/jira/browse/LUCENE-8552 > Project: Lucene - Core > Issue Type: New Feature >Reporter: David Smiley >Assignee: David Smiley >Priority: Minor > Fix For: 7.7 > > Time Spent: 10m > Remaining Estimate: 0h > > FieldInfos.getMergedFieldInfos could trivially return the FieldInfos of the > first and only LeafReader if there is only one LeafReader. > Also... if there is more than one LeafReader, and if FieldInfos & FieldInfo > implemented equals() & hashCode() (including a cached hashCode), maybe we > could also call equals() iterating through the FieldInfos to see if we should > bother adding it to the FieldInfos.Builder? Admittedly this is speculative; > may not be worth the bother. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (LUCENE-8548) Reevaluate scripts boundary break in Nori's tokenizer
[ https://issues.apache.org/jira/browse/LUCENE-8548?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16693478#comment-16693478 ] Christophe Bismuth commented on LUCENE-8548: I'll use the test failure below as a starting point. {code:java} // LUCENE-8548 - file TestKoreanAnalyzer.java public void testCyrillicWord() throws IOException { final Analyzer analyzer = new KoreanAnalyzer(TestKoreanTokenizer.readDict(), KoreanTokenizer.DEFAULT_DECOMPOUND, KoreanPartOfSpeechStopFilter.DEFAULT_STOP_TAGS, false); assertAnalyzesTo(analyzer, "мoscow", new String[]{"мoscow"}, new int[]{0}, new int[]{6}, new int[]{1} ); } {code} > Reevaluate scripts boundary break in Nori's tokenizer > - > > Key: LUCENE-8548 > URL: https://issues.apache.org/jira/browse/LUCENE-8548 > Project: Lucene - Core > Issue Type: Improvement >Reporter: Jim Ferenczi >Priority: Minor > > This was first reported in https://issues.apache.org/jira/browse/LUCENE-8526: > {noformat} > Tokens are split on different character POS types (which seem to not quite > line up with Unicode character blocks), which leads to weird results for > non-CJK tokens: > εἰμί is tokenized as three tokens: ε/SL(Foreign language) + ἰ/SY(Other > symbol) + μί/SL(Foreign language) > ka̠k̚t͡ɕ͈a̠k̚ is tokenized as ka/SL(Foreign language) + ̠/SY(Other symbol) + > k/SL(Foreign language) + ̚/SY(Other symbol) + t/SL(Foreign language) + > ͡ɕ͈/SY(Other symbol) + a/SL(Foreign language) + ̠/SY(Other symbol) + > k/SL(Foreign language) + ̚/SY(Other symbol) > Ба̀лтичко̄ is tokenized as ба/SL(Foreign language) + ̀/SY(Other symbol) + > лтичко/SL(Foreign language) + ̄/SY(Other symbol) > don't is tokenized as don + t; same for don't (with a curly apostrophe). > אוֹג׳וּ is tokenized as אוֹג/SY(Other symbol) + וּ/SY(Other symbol) > Мoscow (with a Cyrillic М and the rest in Latin) is tokenized as м + oscow > While it is still possible to find these words using Nori, there are many > more chances for false positives when the tokens are split up like this. In > particular, individual numbers and combining diacritics are indexed > separately (e.g., in the Cyrillic example above), which can lead to a > performance hit on large corpora like Wiktionary or Wikipedia. > Work around: use a character filter to get rid of combining diacritics before > Nori processes the text. This doesn't solve the Greek, Hebrew, or English > cases, though. > Suggested fix: Characters in related Unicode blocks—like "Greek" and "Greek > Extended", or "Latin" and "IPA Extensions"—should not trigger token splits. > Combining diacritics should not trigger token splits. Non-CJK text should be > tokenized on spaces and punctuation, not by character type shifts. > Apostrophe-like characters should not trigger token splits (though I could > see someone disagreeing on this one).{noformat} > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Updated] (LUCENE-8548) Reevaluate scripts boundary break in Nori's tokenizer
[ https://issues.apache.org/jira/browse/LUCENE-8548?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Christophe Bismuth updated LUCENE-8548: --- Attachment: testCyrillicWord.dot.png > Reevaluate scripts boundary break in Nori's tokenizer > - > > Key: LUCENE-8548 > URL: https://issues.apache.org/jira/browse/LUCENE-8548 > Project: Lucene - Core > Issue Type: Improvement >Reporter: Jim Ferenczi >Priority: Minor > Attachments: testCyrillicWord.dot.png > > > This was first reported in https://issues.apache.org/jira/browse/LUCENE-8526: > {noformat} > Tokens are split on different character POS types (which seem to not quite > line up with Unicode character blocks), which leads to weird results for > non-CJK tokens: > εἰμί is tokenized as three tokens: ε/SL(Foreign language) + ἰ/SY(Other > symbol) + μί/SL(Foreign language) > ka̠k̚t͡ɕ͈a̠k̚ is tokenized as ka/SL(Foreign language) + ̠/SY(Other symbol) + > k/SL(Foreign language) + ̚/SY(Other symbol) + t/SL(Foreign language) + > ͡ɕ͈/SY(Other symbol) + a/SL(Foreign language) + ̠/SY(Other symbol) + > k/SL(Foreign language) + ̚/SY(Other symbol) > Ба̀лтичко̄ is tokenized as ба/SL(Foreign language) + ̀/SY(Other symbol) + > лтичко/SL(Foreign language) + ̄/SY(Other symbol) > don't is tokenized as don + t; same for don't (with a curly apostrophe). > אוֹג׳וּ is tokenized as אוֹג/SY(Other symbol) + וּ/SY(Other symbol) > Мoscow (with a Cyrillic М and the rest in Latin) is tokenized as м + oscow > While it is still possible to find these words using Nori, there are many > more chances for false positives when the tokens are split up like this. In > particular, individual numbers and combining diacritics are indexed > separately (e.g., in the Cyrillic example above), which can lead to a > performance hit on large corpora like Wiktionary or Wikipedia. > Work around: use a character filter to get rid of combining diacritics before > Nori processes the text. This doesn't solve the Greek, Hebrew, or English > cases, though. > Suggested fix: Characters in related Unicode blocks—like "Greek" and "Greek > Extended", or "Latin" and "IPA Extensions"—should not trigger token splits. > Combining diacritics should not trigger token splits. Non-CJK text should be > tokenized on spaces and punctuation, not by character type shifts. > Apostrophe-like characters should not trigger token splits (though I could > see someone disagreeing on this one).{noformat} > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (LUCENE-8548) Reevaluate scripts boundary break in Nori's tokenizer
[ https://issues.apache.org/jira/browse/LUCENE-8548?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16694827#comment-16694827 ] Christophe Bismuth commented on LUCENE-8548: I'm hacking around in the {{KoreanTokenizer}} class, but I'll need some mentoring to keep going on. Here is what I've done so far: * Implement a Cyrillic test failure (see previous comment) * Locate the {{KoreanAnalyzer}} and {{KoreanTokenizer}} classes * Locate the {{ICUTokenizer}} and its {{CompositeBreakIterator}} class attribute (following UAX #29: Unicode Text Segmentation) * Try to make Ant {{nori}} module depend on {{icu}} module to try to reuse some {{ICUTokenizer}} logic parts (but I failed to tweak Ant scripts) * Enable verbose output (see output below) * Enable Graphiz ouput (see attached picture) * Debug step by step the {{org.apache.lucene.analysis.ko.KoreanTokenizer#parse}} method * Add a breakpoint in the {{DictionaryToken}} constructor to try to understand how and when tokens are built (I also played with {{outputUnknownUnigrams}} parameters) I would need some code or documentation pointers when you have time. !testCyrillicWord.dot.png! Tokenizer verbose output below. {noformat} PARSE extend @ pos=0 char=м hex=43c 1 arcs in UNKNOWN word len=1 1 wordIDs fromIDX=0: cost=138 (prevCost=0 wordCost=795 bgCost=138 spacePenalty=0) leftID=1793 leftPOS=SL) ** + cost=933 wordID=36 leftID=1793 leastIDX=0 toPos=1 toPos.idx=0 backtrace: endPos=1 pos=1; 1 characters; last=0 cost=933 add token=DictionaryToken("м" pos=0 length=1 posLen=1 type=UNKNOWN wordId=0 leftID=1798) freeBefore pos=1 TEST-TestKoreanAnalyzer.testCyrillicWord-seed#[9AA9487A32EFEB]:incToken: return token=DictionaryToken("м" pos=0 length=1 posLen=1 type=UNKNOWN wordId=0 leftID=1798) PARSE extend @ pos=1 char=o hex=6f 1 arcs in UNKNOWN word len=6 1 wordIDs fromIDX=0: cost=-1030 (prevCost=0 wordCost=795 bgCost=-1030 spacePenalty=0) leftID=1793 leftPOS=SL) ** + cost=-235 wordID=30 leftID=1793 leastIDX=0 toPos=7 toPos.idx=0 no arcs in; skip pos=2 no arcs in; skip pos=3 no arcs in; skip pos=4 no arcs in; skip pos=5 no arcs in; skip pos=6 end: 1 nodes backtrace: endPos=7 pos=7; 6 characters; last=1 cost=-235 add token=DictionaryToken("w" pos=6 length=1 posLen=1 type=UNKNOWN wordId=0 leftID=1798) add token=DictionaryToken("o" pos=5 length=1 posLen=1 type=UNKNOWN wordId=0 leftID=1798) add token=DictionaryToken("c" pos=4 length=1 posLen=1 type=UNKNOWN wordId=0 leftID=1798) add token=DictionaryToken("s" pos=3 length=1 posLen=1 type=UNKNOWN wordId=0 leftID=1798) add token=DictionaryToken("s" pos=2 length=1 posLen=1 type=UNKNOWN wordId=0 leftID=1798) add token=DictionaryToken("o" pos=1 length=1 posLen=1 type=UNKNOWN wordId=0 leftID=1798) freeBefore pos=7 {noformat} > Reevaluate scripts boundary break in Nori's tokenizer > - > > Key: LUCENE-8548 > URL: https://issues.apache.org/jira/browse/LUCENE-8548 > Project: Lucene - Core > Issue Type: Improvement >Reporter: Jim Ferenczi >Priority: Minor > Attachments: testCyrillicWord.dot.png > > > This was first reported in https://issues.apache.org/jira/browse/LUCENE-8526: > {noformat} > Tokens are split on different character POS types (which seem to not quite > line up with Unicode character blocks), which leads to weird results for > non-CJK tokens: > εἰμί is tokenized as three tokens: ε/SL(Foreign language) + ἰ/SY(Other > symbol) + μί/SL(Foreign language) > ka̠k̚t͡ɕ͈a̠k̚ is tokenized as ka/SL(Foreign language) + ̠/SY(Other symbol) + > k/SL(Foreign language) + ̚/SY(Other symbol) + t/SL(Foreign language) + > ͡ɕ͈/SY(Other symbol) + a/SL(Foreign language) + ̠/SY(Other symbol) + > k/SL(Foreign language) + ̚/SY(Other symbol) > Ба̀лтичко̄ is tokenized as ба/SL(Foreign language) + ̀/SY(Other symbol) + > лтичко/SL(Foreign language) + ̄/SY(Other symbol) > don't is tokenized as don + t; same for don't (with a curly apostrophe). > אוֹג׳וּ is tokenized as אוֹג/SY(Other symbol) + וּ/SY(Other symbol) > Мoscow (with a Cyrillic М and the rest in Latin) is tokenized as м + oscow > While it is still possible to find these words using Nori, there are many > more chances for false positives when the tokens are split up like this. In > particular, individual numbers and combining diacritics are indexed > separately (e.g., in the Cyrillic example above), which can lead to a > performance hit on large corpora like Wiktionary or Wikipedia. > Work around: use a character filter to get rid of combining diacritics before > Nori processes the text. This doesn't solve the Greek, Hebrew, or English > cases, though. > Suggested fix: Characters in related Unicode blocks—like "
[jira] [Comment Edited] (LUCENE-8548) Reevaluate scripts boundary break in Nori's tokenizer
[ https://issues.apache.org/jira/browse/LUCENE-8548?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16694827#comment-16694827 ] Christophe Bismuth edited comment on LUCENE-8548 at 11/21/18 3:11 PM: -- I'm hacking around in the {{KoreanTokenizer}} class, but I'll need some mentoring to keep going on. Here is what I've done so far: * Implement a Cyrillic test failure (see previous comment) * Locate the {{KoreanAnalyzer}} and {{KoreanTokenizer}} classes * Locate the {{ICUTokenizer}} and its {{CompositeBreakIterator}} class attribute (following UAX #29: Unicode Text Segmentation) * Try to make Ant {{nori}} module depend on {{icu}} module to try to reuse some {{ICUTokenizer}} logic parts (but I failed to tweak Ant scripts) * Enable verbose output (see output below) * Enable Graphiz ouput (see attached picture) * Debug step by step the {{org.apache.lucene.analysis.ko.KoreanTokenizer#parse}} method * Add a breakpoint in the {{DictionaryToken}} constructor to try to understand how and when tokens are built (I also played with {{outputUnknownUnigrams}} parameter) I would need some code or documentation pointers when you have time. !testCyrillicWord.dot.png! Tokenizer verbose output below. {noformat} PARSE extend @ pos=0 char=м hex=43c 1 arcs in UNKNOWN word len=1 1 wordIDs fromIDX=0: cost=138 (prevCost=0 wordCost=795 bgCost=138 spacePenalty=0) leftID=1793 leftPOS=SL) ** + cost=933 wordID=36 leftID=1793 leastIDX=0 toPos=1 toPos.idx=0 backtrace: endPos=1 pos=1; 1 characters; last=0 cost=933 add token=DictionaryToken("м" pos=0 length=1 posLen=1 type=UNKNOWN wordId=0 leftID=1798) freeBefore pos=1 TEST-TestKoreanAnalyzer.testCyrillicWord-seed#[9AA9487A32EFEB]:incToken: return token=DictionaryToken("м" pos=0 length=1 posLen=1 type=UNKNOWN wordId=0 leftID=1798) PARSE extend @ pos=1 char=o hex=6f 1 arcs in UNKNOWN word len=6 1 wordIDs fromIDX=0: cost=-1030 (prevCost=0 wordCost=795 bgCost=-1030 spacePenalty=0) leftID=1793 leftPOS=SL) ** + cost=-235 wordID=30 leftID=1793 leastIDX=0 toPos=7 toPos.idx=0 no arcs in; skip pos=2 no arcs in; skip pos=3 no arcs in; skip pos=4 no arcs in; skip pos=5 no arcs in; skip pos=6 end: 1 nodes backtrace: endPos=7 pos=7; 6 characters; last=1 cost=-235 add token=DictionaryToken("w" pos=6 length=1 posLen=1 type=UNKNOWN wordId=0 leftID=1798) add token=DictionaryToken("o" pos=5 length=1 posLen=1 type=UNKNOWN wordId=0 leftID=1798) add token=DictionaryToken("c" pos=4 length=1 posLen=1 type=UNKNOWN wordId=0 leftID=1798) add token=DictionaryToken("s" pos=3 length=1 posLen=1 type=UNKNOWN wordId=0 leftID=1798) add token=DictionaryToken("s" pos=2 length=1 posLen=1 type=UNKNOWN wordId=0 leftID=1798) add token=DictionaryToken("o" pos=1 length=1 posLen=1 type=UNKNOWN wordId=0 leftID=1798) freeBefore pos=7 {noformat} was (Author: cbismuth): I'm hacking around in the {{KoreanTokenizer}} class, but I'll need some mentoring to keep going on. Here is what I've done so far: * Implement a Cyrillic test failure (see previous comment) * Locate the {{KoreanAnalyzer}} and {{KoreanTokenizer}} classes * Locate the {{ICUTokenizer}} and its {{CompositeBreakIterator}} class attribute (following UAX #29: Unicode Text Segmentation) * Try to make Ant {{nori}} module depend on {{icu}} module to try to reuse some {{ICUTokenizer}} logic parts (but I failed to tweak Ant scripts) * Enable verbose output (see output below) * Enable Graphiz ouput (see attached picture) * Debug step by step the {{org.apache.lucene.analysis.ko.KoreanTokenizer#parse}} method * Add a breakpoint in the {{DictionaryToken}} constructor to try to understand how and when tokens are built (I also played with {{outputUnknownUnigrams}} parameters) I would need some code or documentation pointers when you have time. !testCyrillicWord.dot.png! Tokenizer verbose output below. {noformat} PARSE extend @ pos=0 char=м hex=43c 1 arcs in UNKNOWN word len=1 1 wordIDs fromIDX=0: cost=138 (prevCost=0 wordCost=795 bgCost=138 spacePenalty=0) leftID=1793 leftPOS=SL) ** + cost=933 wordID=36 leftID=1793 leastIDX=0 toPos=1 toPos.idx=0 backtrace: endPos=1 pos=1; 1 characters; last=0 cost=933 add token=DictionaryToken("м" pos=0 length=1 posLen=1 type=UNKNOWN wordId=0 leftID=1798) freeBefore pos=1 TEST-TestKoreanAnalyzer.testCyrillicWord-seed#[9AA9487A32EFEB]:incToken: return token=DictionaryToken("м" pos=0 length=1 posLen=1 type=UNKNOWN wordId=0 leftID=1798) PARSE extend @ pos=1 char=o hex=6f 1 arcs in UNKNOWN word len=6 1 wordIDs fromIDX=0: cost=-1030 (prevCost=0 wordCost=795 bgCost=-1030 spacePenalty=0) leftID=1793 leftPOS=SL) ** + cost=-235 wordID=30 leftID=1793 leastIDX=0 toPos=7 toPos.idx=0 no arcs in;
[jira] [Comment Edited] (LUCENE-8548) Reevaluate scripts boundary break in Nori's tokenizer
[ https://issues.apache.org/jira/browse/LUCENE-8548?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16694827#comment-16694827 ] Christophe Bismuth edited comment on LUCENE-8548 at 11/21/18 3:12 PM: -- I'm hacking around in the {{KoreanTokenizer}} class, but I'll need some mentoring to keep going on. Here is what I've done so far: * Implement a Cyrillic test failure (see previous comment) * Locate the {{KoreanAnalyzer}} and {{KoreanTokenizer}} classes * Locate the {{ICUTokenizer}} and its {{CompositeBreakIterator}} class attribute (following UAX #29: Unicode Text Segmentation) * Try to make Ant {{nori}} module depend on {{icu}} module to try to reuse some {{ICUTokenizer}} logic parts (but I failed to tweak Ant scripts) * Enable verbose output (see output below) * Enable Graphiz ouput (see attached picture) * Debug step by step the {{org.apache.lucene.analysis.ko.KoreanTokenizer#parse}} method * Add a breakpoint in the {{DictionaryToken}} constructor to try to understand how and when tokens are built (I also played with {{outputUnknownUnigrams}} parameter) I would need some code or documentation pointers when you have time. !testCyrillicWord.dot.png! Tokenizer verbose output: {noformat} PARSE extend @ pos=0 char=м hex=43c 1 arcs in UNKNOWN word len=1 1 wordIDs fromIDX=0: cost=138 (prevCost=0 wordCost=795 bgCost=138 spacePenalty=0) leftID=1793 leftPOS=SL) ** + cost=933 wordID=36 leftID=1793 leastIDX=0 toPos=1 toPos.idx=0 backtrace: endPos=1 pos=1; 1 characters; last=0 cost=933 add token=DictionaryToken("м" pos=0 length=1 posLen=1 type=UNKNOWN wordId=0 leftID=1798) freeBefore pos=1 TEST-TestKoreanAnalyzer.testCyrillicWord-seed#[9AA9487A32EFEB]:incToken: return token=DictionaryToken("м" pos=0 length=1 posLen=1 type=UNKNOWN wordId=0 leftID=1798) PARSE extend @ pos=1 char=o hex=6f 1 arcs in UNKNOWN word len=6 1 wordIDs fromIDX=0: cost=-1030 (prevCost=0 wordCost=795 bgCost=-1030 spacePenalty=0) leftID=1793 leftPOS=SL) ** + cost=-235 wordID=30 leftID=1793 leastIDX=0 toPos=7 toPos.idx=0 no arcs in; skip pos=2 no arcs in; skip pos=3 no arcs in; skip pos=4 no arcs in; skip pos=5 no arcs in; skip pos=6 end: 1 nodes backtrace: endPos=7 pos=7; 6 characters; last=1 cost=-235 add token=DictionaryToken("w" pos=6 length=1 posLen=1 type=UNKNOWN wordId=0 leftID=1798) add token=DictionaryToken("o" pos=5 length=1 posLen=1 type=UNKNOWN wordId=0 leftID=1798) add token=DictionaryToken("c" pos=4 length=1 posLen=1 type=UNKNOWN wordId=0 leftID=1798) add token=DictionaryToken("s" pos=3 length=1 posLen=1 type=UNKNOWN wordId=0 leftID=1798) add token=DictionaryToken("s" pos=2 length=1 posLen=1 type=UNKNOWN wordId=0 leftID=1798) add token=DictionaryToken("o" pos=1 length=1 posLen=1 type=UNKNOWN wordId=0 leftID=1798) freeBefore pos=7 {noformat} was (Author: cbismuth): I'm hacking around in the {{KoreanTokenizer}} class, but I'll need some mentoring to keep going on. Here is what I've done so far: * Implement a Cyrillic test failure (see previous comment) * Locate the {{KoreanAnalyzer}} and {{KoreanTokenizer}} classes * Locate the {{ICUTokenizer}} and its {{CompositeBreakIterator}} class attribute (following UAX #29: Unicode Text Segmentation) * Try to make Ant {{nori}} module depend on {{icu}} module to try to reuse some {{ICUTokenizer}} logic parts (but I failed to tweak Ant scripts) * Enable verbose output (see output below) * Enable Graphiz ouput (see attached picture) * Debug step by step the {{org.apache.lucene.analysis.ko.KoreanTokenizer#parse}} method * Add a breakpoint in the {{DictionaryToken}} constructor to try to understand how and when tokens are built (I also played with {{outputUnknownUnigrams}} parameter) I would need some code or documentation pointers when you have time. !testCyrillicWord.dot.png! Tokenizer verbose output below: {noformat} PARSE extend @ pos=0 char=м hex=43c 1 arcs in UNKNOWN word len=1 1 wordIDs fromIDX=0: cost=138 (prevCost=0 wordCost=795 bgCost=138 spacePenalty=0) leftID=1793 leftPOS=SL) ** + cost=933 wordID=36 leftID=1793 leastIDX=0 toPos=1 toPos.idx=0 backtrace: endPos=1 pos=1; 1 characters; last=0 cost=933 add token=DictionaryToken("м" pos=0 length=1 posLen=1 type=UNKNOWN wordId=0 leftID=1798) freeBefore pos=1 TEST-TestKoreanAnalyzer.testCyrillicWord-seed#[9AA9487A32EFEB]:incToken: return token=DictionaryToken("м" pos=0 length=1 posLen=1 type=UNKNOWN wordId=0 leftID=1798) PARSE extend @ pos=1 char=o hex=6f 1 arcs in UNKNOWN word len=6 1 wordIDs fromIDX=0: cost=-1030 (prevCost=0 wordCost=795 bgCost=-1030 spacePenalty=0) leftID=1793 leftPOS=SL) ** + cost=-235 wordID=30 leftID=1793 leastIDX=0 toPos=7 toPos.idx=0 no arcs in; skip
[jira] [Comment Edited] (LUCENE-8548) Reevaluate scripts boundary break in Nori's tokenizer
[ https://issues.apache.org/jira/browse/LUCENE-8548?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16694827#comment-16694827 ] Christophe Bismuth edited comment on LUCENE-8548 at 11/21/18 3:12 PM: -- I'm hacking around in the {{KoreanTokenizer}} class, but I'll need some mentoring to keep going on. Here is what I've done so far: * Implement a Cyrillic test failure (see previous comment) * Locate the {{KoreanAnalyzer}} and {{KoreanTokenizer}} classes * Locate the {{ICUTokenizer}} and its {{CompositeBreakIterator}} class attribute (following UAX #29: Unicode Text Segmentation) * Try to make Ant {{nori}} module depend on {{icu}} module to try to reuse some {{ICUTokenizer}} logic parts (but I failed to tweak Ant scripts) * Enable verbose output (see output below) * Enable Graphiz ouput (see attached picture) * Debug step by step the {{org.apache.lucene.analysis.ko.KoreanTokenizer#parse}} method * Add a breakpoint in the {{DictionaryToken}} constructor to try to understand how and when tokens are built (I also played with {{outputUnknownUnigrams}} parameter) I would need some code or documentation pointers when you have time. !testCyrillicWord.dot.png! Tokenizer verbose output below: {noformat} PARSE extend @ pos=0 char=м hex=43c 1 arcs in UNKNOWN word len=1 1 wordIDs fromIDX=0: cost=138 (prevCost=0 wordCost=795 bgCost=138 spacePenalty=0) leftID=1793 leftPOS=SL) ** + cost=933 wordID=36 leftID=1793 leastIDX=0 toPos=1 toPos.idx=0 backtrace: endPos=1 pos=1; 1 characters; last=0 cost=933 add token=DictionaryToken("м" pos=0 length=1 posLen=1 type=UNKNOWN wordId=0 leftID=1798) freeBefore pos=1 TEST-TestKoreanAnalyzer.testCyrillicWord-seed#[9AA9487A32EFEB]:incToken: return token=DictionaryToken("м" pos=0 length=1 posLen=1 type=UNKNOWN wordId=0 leftID=1798) PARSE extend @ pos=1 char=o hex=6f 1 arcs in UNKNOWN word len=6 1 wordIDs fromIDX=0: cost=-1030 (prevCost=0 wordCost=795 bgCost=-1030 spacePenalty=0) leftID=1793 leftPOS=SL) ** + cost=-235 wordID=30 leftID=1793 leastIDX=0 toPos=7 toPos.idx=0 no arcs in; skip pos=2 no arcs in; skip pos=3 no arcs in; skip pos=4 no arcs in; skip pos=5 no arcs in; skip pos=6 end: 1 nodes backtrace: endPos=7 pos=7; 6 characters; last=1 cost=-235 add token=DictionaryToken("w" pos=6 length=1 posLen=1 type=UNKNOWN wordId=0 leftID=1798) add token=DictionaryToken("o" pos=5 length=1 posLen=1 type=UNKNOWN wordId=0 leftID=1798) add token=DictionaryToken("c" pos=4 length=1 posLen=1 type=UNKNOWN wordId=0 leftID=1798) add token=DictionaryToken("s" pos=3 length=1 posLen=1 type=UNKNOWN wordId=0 leftID=1798) add token=DictionaryToken("s" pos=2 length=1 posLen=1 type=UNKNOWN wordId=0 leftID=1798) add token=DictionaryToken("o" pos=1 length=1 posLen=1 type=UNKNOWN wordId=0 leftID=1798) freeBefore pos=7 {noformat} was (Author: cbismuth): I'm hacking around in the {{KoreanTokenizer}} class, but I'll need some mentoring to keep going on. Here is what I've done so far: * Implement a Cyrillic test failure (see previous comment) * Locate the {{KoreanAnalyzer}} and {{KoreanTokenizer}} classes * Locate the {{ICUTokenizer}} and its {{CompositeBreakIterator}} class attribute (following UAX #29: Unicode Text Segmentation) * Try to make Ant {{nori}} module depend on {{icu}} module to try to reuse some {{ICUTokenizer}} logic parts (but I failed to tweak Ant scripts) * Enable verbose output (see output below) * Enable Graphiz ouput (see attached picture) * Debug step by step the {{org.apache.lucene.analysis.ko.KoreanTokenizer#parse}} method * Add a breakpoint in the {{DictionaryToken}} constructor to try to understand how and when tokens are built (I also played with {{outputUnknownUnigrams}} parameter) I would need some code or documentation pointers when you have time. !testCyrillicWord.dot.png! Tokenizer verbose output below. {noformat} PARSE extend @ pos=0 char=м hex=43c 1 arcs in UNKNOWN word len=1 1 wordIDs fromIDX=0: cost=138 (prevCost=0 wordCost=795 bgCost=138 spacePenalty=0) leftID=1793 leftPOS=SL) ** + cost=933 wordID=36 leftID=1793 leastIDX=0 toPos=1 toPos.idx=0 backtrace: endPos=1 pos=1; 1 characters; last=0 cost=933 add token=DictionaryToken("м" pos=0 length=1 posLen=1 type=UNKNOWN wordId=0 leftID=1798) freeBefore pos=1 TEST-TestKoreanAnalyzer.testCyrillicWord-seed#[9AA9487A32EFEB]:incToken: return token=DictionaryToken("м" pos=0 length=1 posLen=1 type=UNKNOWN wordId=0 leftID=1798) PARSE extend @ pos=1 char=o hex=6f 1 arcs in UNKNOWN word len=6 1 wordIDs fromIDX=0: cost=-1030 (prevCost=0 wordCost=795 bgCost=-1030 spacePenalty=0) leftID=1793 leftPOS=SL) ** + cost=-235 wordID=30 leftID=1793 leastIDX=0 toPos=7 toPos.idx=0 no arcs in;
[jira] [Commented] (LUCENE-8548) Reevaluate scripts boundary break in Nori's tokenizer
[ https://issues.apache.org/jira/browse/LUCENE-8548?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16696052#comment-16696052 ] Christophe Bismuth commented on LUCENE-8548: Great! Thank you [~rcmuir], I'll dig into this (y) > Reevaluate scripts boundary break in Nori's tokenizer > - > > Key: LUCENE-8548 > URL: https://issues.apache.org/jira/browse/LUCENE-8548 > Project: Lucene - Core > Issue Type: Improvement >Reporter: Jim Ferenczi >Priority: Minor > Attachments: testCyrillicWord.dot.png > > > This was first reported in https://issues.apache.org/jira/browse/LUCENE-8526: > {noformat} > Tokens are split on different character POS types (which seem to not quite > line up with Unicode character blocks), which leads to weird results for > non-CJK tokens: > εἰμί is tokenized as three tokens: ε/SL(Foreign language) + ἰ/SY(Other > symbol) + μί/SL(Foreign language) > ka̠k̚t͡ɕ͈a̠k̚ is tokenized as ka/SL(Foreign language) + ̠/SY(Other symbol) + > k/SL(Foreign language) + ̚/SY(Other symbol) + t/SL(Foreign language) + > ͡ɕ͈/SY(Other symbol) + a/SL(Foreign language) + ̠/SY(Other symbol) + > k/SL(Foreign language) + ̚/SY(Other symbol) > Ба̀лтичко̄ is tokenized as ба/SL(Foreign language) + ̀/SY(Other symbol) + > лтичко/SL(Foreign language) + ̄/SY(Other symbol) > don't is tokenized as don + t; same for don't (with a curly apostrophe). > אוֹג׳וּ is tokenized as אוֹג/SY(Other symbol) + וּ/SY(Other symbol) > Мoscow (with a Cyrillic М and the rest in Latin) is tokenized as м + oscow > While it is still possible to find these words using Nori, there are many > more chances for false positives when the tokens are split up like this. In > particular, individual numbers and combining diacritics are indexed > separately (e.g., in the Cyrillic example above), which can lead to a > performance hit on large corpora like Wiktionary or Wikipedia. > Work around: use a character filter to get rid of combining diacritics before > Nori processes the text. This doesn't solve the Greek, Hebrew, or English > cases, though. > Suggested fix: Characters in related Unicode blocks—like "Greek" and "Greek > Extended", or "Latin" and "IPA Extensions"—should not trigger token splits. > Combining diacritics should not trigger token splits. Non-CJK text should be > tokenized on spaces and punctuation, not by character type shifts. > Apostrophe-like characters should not trigger token splits (though I could > see someone disagreeing on this one).{noformat} > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (LUCENE-8548) Reevaluate scripts boundary break in Nori's tokenizer
[ https://issues.apache.org/jira/browse/LUCENE-8548?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16696093#comment-16696093 ] Christophe Bismuth commented on LUCENE-8548: That is really nice, thank you [~jim.ferenczi] and [~rcmuir], I should be able to start patch, thanks again! > Reevaluate scripts boundary break in Nori's tokenizer > - > > Key: LUCENE-8548 > URL: https://issues.apache.org/jira/browse/LUCENE-8548 > Project: Lucene - Core > Issue Type: Improvement >Reporter: Jim Ferenczi >Priority: Minor > Attachments: testCyrillicWord.dot.png > > > This was first reported in https://issues.apache.org/jira/browse/LUCENE-8526: > {noformat} > Tokens are split on different character POS types (which seem to not quite > line up with Unicode character blocks), which leads to weird results for > non-CJK tokens: > εἰμί is tokenized as three tokens: ε/SL(Foreign language) + ἰ/SY(Other > symbol) + μί/SL(Foreign language) > ka̠k̚t͡ɕ͈a̠k̚ is tokenized as ka/SL(Foreign language) + ̠/SY(Other symbol) + > k/SL(Foreign language) + ̚/SY(Other symbol) + t/SL(Foreign language) + > ͡ɕ͈/SY(Other symbol) + a/SL(Foreign language) + ̠/SY(Other symbol) + > k/SL(Foreign language) + ̚/SY(Other symbol) > Ба̀лтичко̄ is tokenized as ба/SL(Foreign language) + ̀/SY(Other symbol) + > лтичко/SL(Foreign language) + ̄/SY(Other symbol) > don't is tokenized as don + t; same for don't (with a curly apostrophe). > אוֹג׳וּ is tokenized as אוֹג/SY(Other symbol) + וּ/SY(Other symbol) > Мoscow (with a Cyrillic М and the rest in Latin) is tokenized as м + oscow > While it is still possible to find these words using Nori, there are many > more chances for false positives when the tokens are split up like this. In > particular, individual numbers and combining diacritics are indexed > separately (e.g., in the Cyrillic example above), which can lead to a > performance hit on large corpora like Wiktionary or Wikipedia. > Work around: use a character filter to get rid of combining diacritics before > Nori processes the text. This doesn't solve the Greek, Hebrew, or English > cases, though. > Suggested fix: Characters in related Unicode blocks—like "Greek" and "Greek > Extended", or "Latin" and "IPA Extensions"—should not trigger token splits. > Combining diacritics should not trigger token splits. Non-CJK text should be > tokenized on spaces and punctuation, not by character type shifts. > Apostrophe-like characters should not trigger token splits (though I could > see someone disagreeing on this one).{noformat} > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Updated] (LUCENE-8548) Reevaluate scripts boundary break in Nori's tokenizer
[ https://issues.apache.org/jira/browse/LUCENE-8548?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Christophe Bismuth updated LUCENE-8548: --- Attachment: screenshot-1.png > Reevaluate scripts boundary break in Nori's tokenizer > - > > Key: LUCENE-8548 > URL: https://issues.apache.org/jira/browse/LUCENE-8548 > Project: Lucene - Core > Issue Type: Improvement >Reporter: Jim Ferenczi >Priority: Minor > Attachments: screenshot-1.png, testCyrillicWord.dot.png > > Time Spent: 10m > Remaining Estimate: 0h > > This was first reported in https://issues.apache.org/jira/browse/LUCENE-8526: > {noformat} > Tokens are split on different character POS types (which seem to not quite > line up with Unicode character blocks), which leads to weird results for > non-CJK tokens: > εἰμί is tokenized as three tokens: ε/SL(Foreign language) + ἰ/SY(Other > symbol) + μί/SL(Foreign language) > ka̠k̚t͡ɕ͈a̠k̚ is tokenized as ka/SL(Foreign language) + ̠/SY(Other symbol) + > k/SL(Foreign language) + ̚/SY(Other symbol) + t/SL(Foreign language) + > ͡ɕ͈/SY(Other symbol) + a/SL(Foreign language) + ̠/SY(Other symbol) + > k/SL(Foreign language) + ̚/SY(Other symbol) > Ба̀лтичко̄ is tokenized as ба/SL(Foreign language) + ̀/SY(Other symbol) + > лтичко/SL(Foreign language) + ̄/SY(Other symbol) > don't is tokenized as don + t; same for don't (with a curly apostrophe). > אוֹג׳וּ is tokenized as אוֹג/SY(Other symbol) + וּ/SY(Other symbol) > Мoscow (with a Cyrillic М and the rest in Latin) is tokenized as м + oscow > While it is still possible to find these words using Nori, there are many > more chances for false positives when the tokens are split up like this. In > particular, individual numbers and combining diacritics are indexed > separately (e.g., in the Cyrillic example above), which can lead to a > performance hit on large corpora like Wiktionary or Wikipedia. > Work around: use a character filter to get rid of combining diacritics before > Nori processes the text. This doesn't solve the Greek, Hebrew, or English > cases, though. > Suggested fix: Characters in related Unicode blocks—like "Greek" and "Greek > Extended", or "Latin" and "IPA Extensions"—should not trigger token splits. > Combining diacritics should not trigger token splits. Non-CJK text should be > tokenized on spaces and punctuation, not by character type shifts. > Apostrophe-like characters should not trigger token splits (though I could > see someone disagreeing on this one).{noformat} > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (LUCENE-8548) Reevaluate scripts boundary break in Nori's tokenizer
[ https://issues.apache.org/jira/browse/LUCENE-8548?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16697336#comment-16697336 ] Christophe Bismuth commented on LUCENE-8548: I've made some progress and opened PR [#505|https://github.com/apache/lucene-solr/pull/505] to share them with you. Feel free to stop me as I don't want to make you loose your time. Here is what has been done so far: * Break on script boundaries with built-in JDK API, * Track character classes in a growing byte array, * I feel a tiny bit lost when it comes to extract costs: should I call {{unkDictionary.lookupWordIds(characterId, wordIdRef)}} for each tracked character class? * {{мoscow}} word is correctly parsed in the Graphviz output below ... * ... but test failed on this [line|https://github.com/apache/lucene-solr/blob/1d85cd783863f75cea133fb9c452302214165a4d/lucene/test-framework/src/java/org/apache/lucene/analysis/BaseTokenStreamTestCase.java#L199] and I still have to understand why. !screenshot-1.png! > Reevaluate scripts boundary break in Nori's tokenizer > - > > Key: LUCENE-8548 > URL: https://issues.apache.org/jira/browse/LUCENE-8548 > Project: Lucene - Core > Issue Type: Improvement >Reporter: Jim Ferenczi >Priority: Minor > Attachments: screenshot-1.png, testCyrillicWord.dot.png > > Time Spent: 10m > Remaining Estimate: 0h > > This was first reported in https://issues.apache.org/jira/browse/LUCENE-8526: > {noformat} > Tokens are split on different character POS types (which seem to not quite > line up with Unicode character blocks), which leads to weird results for > non-CJK tokens: > εἰμί is tokenized as three tokens: ε/SL(Foreign language) + ἰ/SY(Other > symbol) + μί/SL(Foreign language) > ka̠k̚t͡ɕ͈a̠k̚ is tokenized as ka/SL(Foreign language) + ̠/SY(Other symbol) + > k/SL(Foreign language) + ̚/SY(Other symbol) + t/SL(Foreign language) + > ͡ɕ͈/SY(Other symbol) + a/SL(Foreign language) + ̠/SY(Other symbol) + > k/SL(Foreign language) + ̚/SY(Other symbol) > Ба̀лтичко̄ is tokenized as ба/SL(Foreign language) + ̀/SY(Other symbol) + > лтичко/SL(Foreign language) + ̄/SY(Other symbol) > don't is tokenized as don + t; same for don't (with a curly apostrophe). > אוֹג׳וּ is tokenized as אוֹג/SY(Other symbol) + וּ/SY(Other symbol) > Мoscow (with a Cyrillic М and the rest in Latin) is tokenized as м + oscow > While it is still possible to find these words using Nori, there are many > more chances for false positives when the tokens are split up like this. In > particular, individual numbers and combining diacritics are indexed > separately (e.g., in the Cyrillic example above), which can lead to a > performance hit on large corpora like Wiktionary or Wikipedia. > Work around: use a character filter to get rid of combining diacritics before > Nori processes the text. This doesn't solve the Greek, Hebrew, or English > cases, though. > Suggested fix: Characters in related Unicode blocks—like "Greek" and "Greek > Extended", or "Latin" and "IPA Extensions"—should not trigger token splits. > Combining diacritics should not trigger token splits. Non-CJK text should be > tokenized on spaces and punctuation, not by character type shifts. > Apostrophe-like characters should not trigger token splits (though I could > see someone disagreeing on this one).{noformat} > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (LUCENE-8548) Reevaluate scripts boundary break in Nori's tokenizer
[ https://issues.apache.org/jira/browse/LUCENE-8548?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16701958#comment-16701958 ] Christophe Bismuth commented on LUCENE-8548: Thanks a lot for sharing this [~jim.ferenczi], and no worries at all as the first iteration was an interesting journey! I think taking time to read about Viterbi would help me some more, let's add it to my pretty own todo list :D I diffed your patch with {{master}} and debugged new tests step-by-step, and I think I understand the big picture. Among others, I totally missed the {{if (isCommonOrInherited(scriptCode) && isCommonOrInherited(sc) == false)}} condition which is essential. I still have one more question, could you please explain what information is contained in the {{wordIdRef}} variable and what the {{unkDictionary.lookupWordIds(characterId, wordIdRef)}} statement does? The debugger tells me {{wordIdRef.length}} is always equal to 36 or 42 and even though 42 is a really great number, I'm a tiny lost in there ... > Reevaluate scripts boundary break in Nori's tokenizer > - > > Key: LUCENE-8548 > URL: https://issues.apache.org/jira/browse/LUCENE-8548 > Project: Lucene - Core > Issue Type: Improvement >Reporter: Jim Ferenczi >Priority: Minor > Attachments: LUCENE-8548.patch, screenshot-1.png, > testCyrillicWord.dot.png > > Time Spent: 10m > Remaining Estimate: 0h > > This was first reported in https://issues.apache.org/jira/browse/LUCENE-8526: > {noformat} > Tokens are split on different character POS types (which seem to not quite > line up with Unicode character blocks), which leads to weird results for > non-CJK tokens: > εἰμί is tokenized as three tokens: ε/SL(Foreign language) + ἰ/SY(Other > symbol) + μί/SL(Foreign language) > ka̠k̚t͡ɕ͈a̠k̚ is tokenized as ka/SL(Foreign language) + ̠/SY(Other symbol) + > k/SL(Foreign language) + ̚/SY(Other symbol) + t/SL(Foreign language) + > ͡ɕ͈/SY(Other symbol) + a/SL(Foreign language) + ̠/SY(Other symbol) + > k/SL(Foreign language) + ̚/SY(Other symbol) > Ба̀лтичко̄ is tokenized as ба/SL(Foreign language) + ̀/SY(Other symbol) + > лтичко/SL(Foreign language) + ̄/SY(Other symbol) > don't is tokenized as don + t; same for don't (with a curly apostrophe). > אוֹג׳וּ is tokenized as אוֹג/SY(Other symbol) + וּ/SY(Other symbol) > Мoscow (with a Cyrillic М and the rest in Latin) is tokenized as м + oscow > While it is still possible to find these words using Nori, there are many > more chances for false positives when the tokens are split up like this. In > particular, individual numbers and combining diacritics are indexed > separately (e.g., in the Cyrillic example above), which can lead to a > performance hit on large corpora like Wiktionary or Wikipedia. > Work around: use a character filter to get rid of combining diacritics before > Nori processes the text. This doesn't solve the Greek, Hebrew, or English > cases, though. > Suggested fix: Characters in related Unicode blocks—like "Greek" and "Greek > Extended", or "Latin" and "IPA Extensions"—should not trigger token splits. > Combining diacritics should not trigger token splits. Non-CJK text should be > tokenized on spaces and punctuation, not by character type shifts. > Apostrophe-like characters should not trigger token splits (though I could > see someone disagreeing on this one).{noformat} > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (LUCENE-8548) Reevaluate scripts boundary break in Nori's tokenizer
[ https://issues.apache.org/jira/browse/LUCENE-8548?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16707497#comment-16707497 ] Christophe Bismuth commented on LUCENE-8548: That's great, thanks [~jim.ferenczi] for all the details (y) > Reevaluate scripts boundary break in Nori's tokenizer > - > > Key: LUCENE-8548 > URL: https://issues.apache.org/jira/browse/LUCENE-8548 > Project: Lucene - Core > Issue Type: Improvement >Reporter: Jim Ferenczi >Priority: Minor > Fix For: master (8.0), 7.7 > > Attachments: LUCENE-8548.patch, screenshot-1.png, > testCyrillicWord.dot.png > > Time Spent: 10m > Remaining Estimate: 0h > > This was first reported in https://issues.apache.org/jira/browse/LUCENE-8526: > {noformat} > Tokens are split on different character POS types (which seem to not quite > line up with Unicode character blocks), which leads to weird results for > non-CJK tokens: > εἰμί is tokenized as three tokens: ε/SL(Foreign language) + ἰ/SY(Other > symbol) + μί/SL(Foreign language) > ka̠k̚t͡ɕ͈a̠k̚ is tokenized as ka/SL(Foreign language) + ̠/SY(Other symbol) + > k/SL(Foreign language) + ̚/SY(Other symbol) + t/SL(Foreign language) + > ͡ɕ͈/SY(Other symbol) + a/SL(Foreign language) + ̠/SY(Other symbol) + > k/SL(Foreign language) + ̚/SY(Other symbol) > Ба̀лтичко̄ is tokenized as ба/SL(Foreign language) + ̀/SY(Other symbol) + > лтичко/SL(Foreign language) + ̄/SY(Other symbol) > don't is tokenized as don + t; same for don't (with a curly apostrophe). > אוֹג׳וּ is tokenized as אוֹג/SY(Other symbol) + וּ/SY(Other symbol) > Мoscow (with a Cyrillic М and the rest in Latin) is tokenized as м + oscow > While it is still possible to find these words using Nori, there are many > more chances for false positives when the tokens are split up like this. In > particular, individual numbers and combining diacritics are indexed > separately (e.g., in the Cyrillic example above), which can lead to a > performance hit on large corpora like Wiktionary or Wikipedia. > Work around: use a character filter to get rid of combining diacritics before > Nori processes the text. This doesn't solve the Greek, Hebrew, or English > cases, though. > Suggested fix: Characters in related Unicode blocks—like "Greek" and "Greek > Extended", or "Latin" and "IPA Extensions"—should not trigger token splits. > Combining diacritics should not trigger token splits. Non-CJK text should be > tokenized on spaces and punctuation, not by character type shifts. > Apostrophe-like characters should not trigger token splits (though I could > see someone disagreeing on this one).{noformat} > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org