Github user ymazari commented on a diff in the pull request: https://github.com/apache/spark/pull/20367#discussion_r164260027 --- Diff: mllib/src/main/scala/org/apache/spark/ml/feature/CountVectorizer.scala --- @@ -160,6 +187,11 @@ class CountVectorizer @Since("1.5.0") (@Since("1.5.0") override val uid: String) } else { $(minDF) * input.cache().count() } + val maxDf = if ($(maxDF) >= 1.0) { + $(maxDF) + } else { + $(maxDF) * input.cache().count() + } --- End diff -- Good points. - I added a check that maxDF >= minDF - Changed the code so that counting (and caching) is done only once - I refactored the code so that "filter()" is only invoked if minDF or maxDF is set - Added an un-persisting the input after the counting is done.
--- --------------------------------------------------------------------- To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org