Taewoo Kim has uploaded a new change for review. https://asterix-gerrit.ics.uci.edu/1448
Change subject: Add a corner case handling for NGramUTF8StringBinaryTokenizer ...................................................................... Add a corner case handling for NGramUTF8StringBinaryTokenizer - For a corner case where the length of given string is less than the given gram length, it returns 0 as the total number of grams. Change-Id: I5965856b4da018276b37460bed7fb1fc60d8c2f3 --- M hyracks-fullstack/hyracks/hyracks-storage-am-lsm-invertedindex/src/main/java/org/apache/hyracks/storage/am/lsm/invertedindex/tokenizers/NGramUTF8StringBinaryTokenizer.java 1 file changed, 5 insertions(+), 1 deletion(-) git pull ssh://asterix-gerrit.ics.uci.edu:29418/asterixdb refs/changes/48/1448/1 diff --git a/hyracks-fullstack/hyracks/hyracks-storage-am-lsm-invertedindex/src/main/java/org/apache/hyracks/storage/am/lsm/invertedindex/tokenizers/NGramUTF8StringBinaryTokenizer.java b/hyracks-fullstack/hyracks/hyracks-storage-am-lsm-invertedindex/src/main/java/org/apache/hyracks/storage/am/lsm/invertedindex/tokenizers/NGramUTF8StringBinaryTokenizer.java index 4c486c5..7b3af14 100644 --- a/hyracks-fullstack/hyracks/hyracks-storage-am-lsm-invertedindex/src/main/java/org/apache/hyracks/storage/am/lsm/invertedindex/tokenizers/NGramUTF8StringBinaryTokenizer.java +++ b/hyracks-fullstack/hyracks/hyracks-storage-am-lsm-invertedindex/src/main/java/org/apache/hyracks/storage/am/lsm/invertedindex/tokenizers/NGramUTF8StringBinaryTokenizer.java @@ -110,7 +110,11 @@ if (usePrePost) { totalGrams = numChars + gramLength - 1; } else { - totalGrams = numChars - gramLength + 1; + if (length >= gramLength) { + totalGrams = numChars - gramLength + 1; + } else { + totalGrams = 0; + } } } -- To view, visit https://asterix-gerrit.ics.uci.edu/1448 To unsubscribe, visit https://asterix-gerrit.ics.uci.edu/settings Gerrit-MessageType: newchange Gerrit-Change-Id: I5965856b4da018276b37460bed7fb1fc60d8c2f3 Gerrit-PatchSet: 1 Gerrit-Project: asterixdb Gerrit-Branch: master Gerrit-Owner: Taewoo Kim <wangs...@yahoo.com>