alexeykudinkin opened a new pull request #4996: URL: https://github.com/apache/hudi/pull/4996
## *Tips* - *Thank you very much for contributing to Apache Hudi.* - *Please review https://hudi.apache.org/contribute/how-to-contribute before opening a pull request.* ## What is the purpose of the pull request Supporting Composite Expressions (including standard Spark functions and UDFs) as Filter Expressions in Data Skipping flow. For example, if previously we were only supporting rather simple expressions over Data Table attributes like `WHERE columnA = 42`, now we will be supporting much broader scope of expressions (strictly defined below) allowing for example Data Skipping to properly digest queries like `WHERE date_format(columnA, 'MM/dd/yyyy') = '01/01/2022'` referencing Spark's standard function `date_format`. Formally, now supported are any expressions such that it 1. References exactly 1 attribute (column, meaning that expressions like `A * B = 0` are not supported) 2. Does not contain sub-queries Also, now as "value expression" we support any expression such that it 1. Does not reference any other attribute (`A = B` filters are not supported) 2. Does not contain sub-queries ## Brief change log - Expanded scope for value expressions - Expanded scope for attribute expressions - Fixed `resolveExpr` util to properly resolve Spark functions - Grouped together logically equivalent expressions - Added tests ## Verify this pull request This pull request is already covered by existing tests, such as *(please describe tests)*. This change added tests ## Committer checklist - [ ] Has a corresponding JIRA in PR title & commit - [ ] Commit message is descriptive of the change - [ ] CI is green - [ ] Necessary doc changes done or have another open PR - [ ] For large changes, please consider breaking it into sub-tasks under an umbrella JIRA. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
