The GitHub Actions job "Update Dependency Graph" on texera.git/main has failed. Run started by GitHub user github-merge-queue[bot] (triggered by github-merge-queue[bot]).
Head commit for run: 110a54c151dc692a0720235f3951c41e60ec363a / Kary Zheng <[email protected]> fix(KeywordSearch): keep punctuation handling consistent in case-sensitive mode (#6801) ### What changes were proposed in this PR? Fixes an unintended side effect of the KeywordSearch "case sensitive" option. `KeywordSearchOpExec` selects an analyzer based on `isCaseSensitive`: - **off (default)** → `StandardAnalyzer`, whose `StandardTokenizer` splits on Unicode word boundaries and strips punctuation, so `"perfect."` tokenizes to `perfect`. - **on** → `CaseSensitiveAnalyzer`, which used a bare `WhitespaceTokenizer`. That splits **only** on whitespace, so punctuation stays glued to the token: `"...absolutely perfect."` tokenizes to `[absolutely, perfect.]`. The intent of `CaseSensitiveAnalyzer` was only to *skip lowercasing*. But swapping the tokenizer also dropped Unicode word-boundary tokenization, so enabling "case sensitive" silently changed **punctuation handling** too — a side effect the option name does not imply. A user who turns on case sensitivity to match `Trump` exactly finds that `perfect` no longer matches `perfect.`, `USA` no longer matches `USA.`, etc. The fix makes `CaseSensitiveAnalyzer` mirror `StandardAnalyzer`'s pipeline **minus** the `LowerCaseFilter`: it now uses `StandardTokenizer` (Unicode word-boundary tokenization, punctuation stripped) and omits lowercasing. Case sensitivity becomes the *only* behavioral difference from the default analyzer. ```diff -import org.apache.lucene.analysis.core.WhitespaceTokenizer +import org.apache.lucene.analysis.standard.StandardTokenizer ... - val tokenizer = new WhitespaceTokenizer() + val tokenizer = new StandardTokenizer() val stream: TokenStream = new StopFilter(tokenizer, CharArraySet.EMPTY_SET) ``` ### Any related issues, documentation, discussions? Closes #6761 Note: the standalone/Python translation of KeywordSearch (`KeywordSearchOpDesc.generateStandaloneCode`) does not implement case sensitivity at all — that is a separate gap and out of scope here. ### How was this PR tested? Added a regression test to `KeywordSearchOpExecSpec` asserting that in case-sensitive mode, keyword `perfect` matches the row `everything was absolutely perfect.` (a word adjacent to trailing punctuation). This test fails on `main` (the row is not matched) and passes with this change. Ran the full suite locally: ``` sbt "WorkflowOperator/testOnly org.apache.texera.amber.operator.keywordSearch.KeywordSearchOpExecSpec" ... Tests: succeeded 18, failed 0, canceled 0, ignored 0, pending 0 All tests passed. ``` The existing case-sensitive tests (`twitter`/`Twitter`) still pass, confirming case sensitivity itself is unchanged. ### Was this PR authored or co-authored using generative AI tooling? Generated-by: Claude Code (Claude Opus 4.8) --------- Co-authored-by: Claude Opus 4.8 (1M context) <[email protected]> Report URL: https://github.com/apache/texera/actions/runs/29979572847 With regards, GitHub Actions via GitBox
