rui-mo commented on issue #8897: URL: https://github.com/apache/incubator-gluten/issues/8897#issuecomment-2699049143
Hi @ayushi-agarwal, let me explain the context here. The reason for this plan change as you noticed is that before https://github.com/apache/incubator-gluten/pull/6754 Gluten decides which filters can be pushed down, which cannot, so the TableScan node is created with that information and is like below. > TableScan[0][table: hive_table, range filters: [(ss_list_price, DoubleRange: (-inf, 0.000000) no nulls)]] However, after #6754 Gluten depends on 'extractFiltersFromRemainingFilter' to extract the push-down filters internally in Velox, so in Gluten the TableScan node is created with remaining filter only, and the plan is like below. > TableScan[0][table: hive_table, remaining filter: (and(isnotnull("ss_list_price"),lessthan("ss_list_price",0)))] This plan does not imply that all filters are not pushed down. If we try to print the 'remainingFilter' and 'filters_' at [HiveDataSource](https://github.com/facebookincubator/velox/blob/55257c24487ae0002a0e8ea5ee25972fc955699f/velox/connectors/hive/HiveDataSource.cpp#L135-L140), we will find the 'remainingFilter' is nullptr and 'filters_' contains the `DoubleRange: (-inf, 0.000000) no nulls`, which means the remaining filter is converted to push-down filters completely. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
