kz930 opened a new pull request, #6801:
URL: https://github.com/apache/texera/pull/6801
### What changes were proposed in this PR?
Fixes an unintended side effect of the KeywordSearch "case sensitive" option.
`KeywordSearchOpExec` selects an analyzer based on `isCaseSensitive`:
- **off (default)** → `StandardAnalyzer`, whose `StandardTokenizer` splits
on Unicode word boundaries and strips punctuation, so `"perfect."` tokenizes to
`perfect`.
- **on** → `CaseSensitiveAnalyzer`, which used a bare `WhitespaceTokenizer`.
That splits **only** on whitespace, so punctuation stays glued to the token:
`"...absolutely perfect."` tokenizes to `[absolutely, perfect.]`.
The intent of `CaseSensitiveAnalyzer` was only to *skip lowercasing*. But
swapping the tokenizer also dropped Unicode word-boundary tokenization, so
enabling "case sensitive" silently changed **punctuation handling** too — a
side effect the option name does not imply. A user who turns on case
sensitivity to match `Trump` exactly finds that `perfect` no longer matches
`perfect.`, `USA` no longer matches `USA.`, etc.
The fix makes `CaseSensitiveAnalyzer` mirror `StandardAnalyzer`'s pipeline
**minus** the `LowerCaseFilter`: it now uses `StandardTokenizer` (Unicode
word-boundary tokenization, punctuation stripped) and omits lowercasing. Case
sensitivity becomes the *only* behavioral difference from the default analyzer.
```diff
-import org.apache.lucene.analysis.core.WhitespaceTokenizer
+import org.apache.lucene.analysis.standard.StandardTokenizer
...
- val tokenizer = new WhitespaceTokenizer()
+ val tokenizer = new StandardTokenizer()
val stream: TokenStream = new StopFilter(tokenizer,
CharArraySet.EMPTY_SET)
```
### Any related issues, documentation, discussions?
Closes #6761
Note: the standalone/Python translation of KeywordSearch
(`KeywordSearchOpDesc.generateStandaloneCode`) does not implement case
sensitivity at all — that is a separate gap and out of scope here.
### How was this PR tested?
Added a regression test to `KeywordSearchOpExecSpec` asserting that in
case-sensitive mode, keyword `perfect` matches the row `everything was
absolutely perfect.` (a word adjacent to trailing punctuation). This test fails
on `main` (the row is not matched) and passes with this change.
Ran the full suite locally:
```
sbt "WorkflowOperator/testOnly
org.apache.texera.amber.operator.keywordSearch.KeywordSearchOpExecSpec"
...
Tests: succeeded 18, failed 0, canceled 0, ignored 0, pending 0
All tests passed.
```
The existing case-sensitive tests (`twitter`/`Twitter`) still pass,
confirming case sensitivity itself is unchanged.
### Was this PR authored or co-authored using generative AI tooling?
Generated-by: Claude Code (Claude Opus 4.8)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
