LuciferYang opened a new issue, #13546: URL: https://github.com/apache/gluten/issues/13546
Spark keeps a comparison such as `100 > id` as written, with the literal on the left. Vanilla Spark flips it when it translates data source filters, but Gluten passes the Catalyst filter to the native scan as is, and `SparkExprToSubfieldFilterParser::leafCallToSubfieldFilter` only looks for a column on the left side. The predicate is still evaluated as a remaining filter, so results are correct, but the scan cannot turn it into a subfield filter or use it for row group pruning. On a sorted Parquet file with 10 row groups, `id < 100` skips 9 row groups while `100 > id` reads all 10. This issue was written with the assistance of AI (Claude Opus). -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
