Github user VinceShieh commented on a diff in the pull request: https://github.com/apache/spark/pull/14858#discussion_r77465278 --- Diff: mllib/src/main/scala/org/apache/spark/ml/feature/QuantileDiscretizer.scala --- @@ -114,10 +115,10 @@ final class QuantileDiscretizer @Since("1.6.0") (@Since("1.6.0") override val ui splits(0) = Double.NegativeInfinity splits(splits.length - 1) = Double.PositiveInfinity - val distinctSplits = splits.distinct + val distinctSplits = splits.filter(!_.isNaN).distinct --- End diff -- @srowen then maybe we should, as we discussed earlier on JIRA, align with R, by having a NaN checker in approxQuantile, that is, having a NaN filter inside of approxQuantile, rather than ahead of calling approxQuantile. We can also have a same flag for user to choose to either remove NaN values or throw an error when there is NaN in data, although, this API change will introduce collateral impact on several existing function calls.
--- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- --------------------------------------------------------------------- To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org