kz930 opened a new issue, #8989: URL: https://github.com/apache/texera/issues/8989
### What happened? The exported Python for Regular Expression and Substring Search disagrees with the engine on non-ASCII text. Regular Expression: Java's Pattern ignores case only for ASCII letters unless UNICODE_CASE is set, and its \w, \d and \s match only ASCII. Python's re is Unicode by default. So with Case Insensitive on, "é" matches "É" in the export and not in the engine, and with it off, \w+ matches "é" and \d matches "٣" in the export only. Substring Search: with Case Sensitive off, the engine compares toLowerCase of both sides, and pandas' str.contains(case=False) compares upper(). So "ss" matches "ß", "ς" matches "σ" and "ff" matches "FF" in the export only, and "i̇" matches "İ" in the engine only. ### How to reproduce? 1. Feed a table with a string column holding "É" into Regular Expression with regex "é" and Case Insensitive on. The engine keeps no row. 2. Export the workflow as Python and run it on the same table. It keeps the row. 3. Likewise, Substring Search with substring "ss", Case Sensitive off, on a row holding "ß": the engine keeps no row, the export keeps it. ### Version/Branch 1.4.0-incubating-SNAPSHOT (main) ### Commit Hash (Optional) 6a29c59c6 ### What browsers are you seeing the problem on? _No response_ ### Relevant log output _No response_ -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
