Github user mshtelma commented on a diff in the pull request:
https://github.com/apache/spark/pull/21147#discussion_r184376159
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/plans/logical/statsEstimation/FilterEstimation.scala
---
@@ -392,13 +392,13 @@ case class FilterEstimation(plan: Filter) extends
Logging {
val dataType = attr.dataType
var newNdv = ndv
- if (ndv.toDouble == 0 || colStat.min.isEmpty || colStat.max.isEmpty) {
- return Some(0.0)
- }
-
// use [min, max] to filter the original hSet
dataType match {
case _: NumericType | BooleanType | DateType | TimestampType =>
+ if (ndv.toDouble == 0 || colStat.min.isEmpty ||
colStat.max.isEmpty) {
--- End diff --
min/max can be None if the column contains only null values. This is
exactly the case for my query.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
