stuartcarnie opened a new pull request, #5152: URL: https://github.com/apache/arrow-datafusion/pull/5152
# Which issue does this PR close? <!-- We generally require a GitHub issue to be filed for all bug fixes and enhancements and this helps us generate change logs for our releases. You can link an issue to this PR using the GitHub syntax. For example `Closes #123` indicates that this PR will close issue #123. --> Closes #. # Rationale for this change This change teaches DataFusion how to coerce Dictionary(_, string-like-type) to a string-like type for regex conditional expressions, so the comparison may succeed. # What changes are included in this PR? Updated the type coercion rules when encountering one of the regular expression operators. # Are these changes tested? Yes, new tests were added to validate the changes. # Are there any user-facing changes? Queries that previously would fail with an error that the Dictionary type is incompatible with a regular expression condition will now succeed. IOx uses `Dictionary(Int32, Utf8)` columns for tags. In some cases, regular expression queries succeed, such as: ```sql SELECT usage_idle FROM cpu WHERE cpu ~ '9' ``` As the optimiser changes the filter plan from a regex conditional to a `LIKE '%9%'`, and the `LIKE` operator has additional code to coerce dictionary types: https://github.com/apache/arrow-datafusion/blob/031534d94efb305eb26a7c16fd7e06ae3bcd88bb/datafusion/expr/src/type_coercion/binary.rs#L522 however, other cases unexpectedly fail: ```sql SELECT usage_idle FROM cpu WHERE cpu ~ '9$' ``` as the optimiser does not rewrite this to a `LIKE` expression[^1] [^1]: Incidentally, the optimiser could also rewrite this to `LIKE '%9'` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
