Github user mengxr commented on a diff in the pull request:
https://github.com/apache/spark/pull/21501#discussion_r194244754
--- Diff:
mllib/src/main/scala/org/apache/spark/ml/feature/StopWordsRemover.scala ---
@@ -84,7 +86,28 @@ class StopWordsRemover @Since("1.5.0") (@Since("1.5.0")
override val uid: String
@Since("1.5.0")
def getCaseSensitive: Boolean = $(caseSensitive)
- setDefault(stopWords ->
StopWordsRemover.loadDefaultStopWords("english"), caseSensitive -> false)
+ /**
+ * Locale of the input for case insensitive matching. Ignored when
[[caseSensitive]]
+ * is true.
+ * Default: Locale.getDefault.toString
--- End diff --
Locale is usually configured at system level. Setting the default locale to
`en` is a breaking change to users who set system locale to `tr` and have the
following code:
~~~
val remover = new StopWordsRemover()
.setStopWords(loadDefaultStopWords("turkish")
~~~
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
