alkis commented on code in PR #45202:
URL: https://github.com/apache/spark/pull/45202#discussion_r1499234784
##########
sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/parquet/ParquetFilters.scala:
##########
@@ -609,8 +609,8 @@ class ParquetFilters(
// Parquet's type in the given file should be matched to the value's type
// in the pushed filter in order to push down the filter to Parquet.
- private def valueCanMakeFilterOn(name: String, value: Any): Boolean = {
- value == null || (nameToParquetField(name).fieldType match {
+ private def valueCanMakeFilterOn(name: String, value: Any, allowNull:
Boolean): Boolean = {
+ (value == null && allowNull) || (nameToParquetField(name).fieldType match {
Review Comment:
I think doing the check at callsite is better since it puts the null check
next to the operator.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
