spark git commit: [SPARK-13590][ML][DOC] Document spark.ml LiR, LoR and AFTSurvivalRegression behavior difference
Repository: spark
Updated Branches:
refs/heads/master 890baaca5 -> 6ecedf39b
[SPARK-13590][ML][DOC] Document spark.ml LiR, LoR and AFTSurvivalRegression
behavior difference
## What changes were proposed in this pull request?
When fitting ```LinearRegressionModel```(by "l-bfgs" solver) and
```LogisticRegressionModel``` w/o intercept on dataset with constant nonzero
column, spark.ml produce same model as R glmnet but different from LIBSVM.
When fitting ```AFTSurvivalRegressionModel``` w/o intercept on dataset with
constant nonzero column, spark.ml produce different model compared with R
survival::survreg.
We should output a warning message and clarify in document for this condition.
## How was this patch tested?
Document change, no unit test.
cc mengxr
Author: Yanbo Liang
Closes #12731 from yanboliang/spark-13590.
Project: http://git-wip-us.apache.org/repos/asf/spark/repo
Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/6ecedf39
Tree: http://git-wip-us.apache.org/repos/asf/spark/tree/6ecedf39
Diff: http://git-wip-us.apache.org/repos/asf/spark/diff/6ecedf39
Branch: refs/heads/master
Commit: 6ecedf39b44c9acd58cdddf1a31cf11e8e24428c
Parents: 890baac
Author: Yanbo Liang
Authored: Tue Jun 7 15:25:36 2016 -0700
Committer: Yanbo Liang
Committed: Tue Jun 7 15:25:36 2016 -0700
--
docs/ml-classification-regression.md| 6 ++
.../apache/spark/ml/classification/LogisticRegression.scala | 7 +++
.../apache/spark/ml/regression/AFTSurvivalRegression.scala | 9 -
.../org/apache/spark/ml/regression/LinearRegression.scala | 7 +++
4 files changed, 28 insertions(+), 1 deletion(-)
--
http://git-wip-us.apache.org/repos/asf/spark/blob/6ecedf39/docs/ml-classification-regression.md
--
diff --git a/docs/ml-classification-regression.md
b/docs/ml-classification-regression.md
index ff8dec6..88457d4 100644
--- a/docs/ml-classification-regression.md
+++ b/docs/ml-classification-regression.md
@@ -62,6 +62,8 @@ For more background and more details about the
implementation, refer to the docu
> The current implementation of logistic regression in `spark.ml` only
supports binary classes. Support for multiclass regression will be added in the
future.
+ > When fitting LogisticRegressionModel without intercept on dataset with
constant nonzero column, Spark MLlib outputs zero coefficients for constant
nonzero columns. This behavior is the same as R glmnet but different from
LIBSVM.
+
**Example**
The following example shows how to train a logistic regression model
@@ -351,6 +353,8 @@ Refer to the [Python API
docs](api/python/pyspark.ml.html#pyspark.ml.classificat
The interface for working with linear regression models and model
summaries is similar to the logistic regression case.
+ > When fitting LinearRegressionModel without intercept on dataset with
constant nonzero column by "l-bfgs" solver, Spark MLlib outputs zero
coefficients for constant nonzero columns. This behavior is the same as R
glmnet but different from LIBSVM.
+
**Example**
The following
@@ -666,6 +670,8 @@ The optimization algorithm underlying the implementation is
L-BFGS.
The implementation matches the result from R's survival function
[survreg](https://stat.ethz.ch/R-manual/R-devel/library/survival/html/survreg.html)
+ > When fitting AFTSurvivalRegressionModel without intercept on dataset with
constant nonzero column, Spark MLlib outputs zero coefficients for constant
nonzero columns. This behavior is different from R survival::survreg.
+
**Example**
http://git-wip-us.apache.org/repos/asf/spark/blob/6ecedf39/mllib/src/main/scala/org/apache/spark/ml/classification/LogisticRegression.scala
--
diff --git
a/mllib/src/main/scala/org/apache/spark/ml/classification/LogisticRegression.scala
b/mllib/src/main/scala/org/apache/spark/ml/classification/LogisticRegression.scala
index 1ea4d90..51ede15 100644
---
a/mllib/src/main/scala/org/apache/spark/ml/classification/LogisticRegression.scala
+++
b/mllib/src/main/scala/org/apache/spark/ml/classification/LogisticRegression.scala
@@ -333,6 +333,13 @@ class LogisticRegression @Since("1.2.0") (
val featuresMean = summarizer.mean.toArray
val featuresStd = summarizer.variance.toArray.map(math.sqrt)
+if (!$(fitIntercept) && (0 until numFeatures).exists { i =>
+ featuresStd(i) == 0.0 && featuresMean(i) != 0.0 }) {
+ logWarning("Fitting LogisticRegressionModel without intercept on
dataset with " +
+"constant nonzero column, Spark MLlib outputs zero coefficients
for constant " +
+"nonzero columns. This behavior is the same as R glmnet but
different from LIBSVM.")
+}
+
spark git commit: [SPARK-13590][ML][DOC] Document spark.ml LiR, LoR and AFTSurvivalRegression behavior difference
Repository: spark
Updated Branches:
refs/heads/branch-2.0 9e16f23e7 -> e21a9ddef
[SPARK-13590][ML][DOC] Document spark.ml LiR, LoR and AFTSurvivalRegression
behavior difference
## What changes were proposed in this pull request?
When fitting ```LinearRegressionModel```(by "l-bfgs" solver) and
```LogisticRegressionModel``` w/o intercept on dataset with constant nonzero
column, spark.ml produce same model as R glmnet but different from LIBSVM.
When fitting ```AFTSurvivalRegressionModel``` w/o intercept on dataset with
constant nonzero column, spark.ml produce different model compared with R
survival::survreg.
We should output a warning message and clarify in document for this condition.
## How was this patch tested?
Document change, no unit test.
cc mengxr
Author: Yanbo Liang
Closes #12731 from yanboliang/spark-13590.
(cherry picked from commit 6ecedf39b44c9acd58cdddf1a31cf11e8e24428c)
Signed-off-by: Yanbo Liang
Project: http://git-wip-us.apache.org/repos/asf/spark/repo
Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/e21a9dde
Tree: http://git-wip-us.apache.org/repos/asf/spark/tree/e21a9dde
Diff: http://git-wip-us.apache.org/repos/asf/spark/diff/e21a9dde
Branch: refs/heads/branch-2.0
Commit: e21a9ddefed074de84d3b3bb0f347d64b82696c6
Parents: 9e16f23
Author: Yanbo Liang
Authored: Tue Jun 7 15:25:36 2016 -0700
Committer: Yanbo Liang
Committed: Tue Jun 7 15:26:14 2016 -0700
--
docs/ml-classification-regression.md| 6 ++
.../apache/spark/ml/classification/LogisticRegression.scala | 7 +++
.../apache/spark/ml/regression/AFTSurvivalRegression.scala | 9 -
.../org/apache/spark/ml/regression/LinearRegression.scala | 7 +++
4 files changed, 28 insertions(+), 1 deletion(-)
--
http://git-wip-us.apache.org/repos/asf/spark/blob/e21a9dde/docs/ml-classification-regression.md
--
diff --git a/docs/ml-classification-regression.md
b/docs/ml-classification-regression.md
index ff8dec6..88457d4 100644
--- a/docs/ml-classification-regression.md
+++ b/docs/ml-classification-regression.md
@@ -62,6 +62,8 @@ For more background and more details about the
implementation, refer to the docu
> The current implementation of logistic regression in `spark.ml` only
supports binary classes. Support for multiclass regression will be added in the
future.
+ > When fitting LogisticRegressionModel without intercept on dataset with
constant nonzero column, Spark MLlib outputs zero coefficients for constant
nonzero columns. This behavior is the same as R glmnet but different from
LIBSVM.
+
**Example**
The following example shows how to train a logistic regression model
@@ -351,6 +353,8 @@ Refer to the [Python API
docs](api/python/pyspark.ml.html#pyspark.ml.classificat
The interface for working with linear regression models and model
summaries is similar to the logistic regression case.
+ > When fitting LinearRegressionModel without intercept on dataset with
constant nonzero column by "l-bfgs" solver, Spark MLlib outputs zero
coefficients for constant nonzero columns. This behavior is the same as R
glmnet but different from LIBSVM.
+
**Example**
The following
@@ -666,6 +670,8 @@ The optimization algorithm underlying the implementation is
L-BFGS.
The implementation matches the result from R's survival function
[survreg](https://stat.ethz.ch/R-manual/R-devel/library/survival/html/survreg.html)
+ > When fitting AFTSurvivalRegressionModel without intercept on dataset with
constant nonzero column, Spark MLlib outputs zero coefficients for constant
nonzero columns. This behavior is different from R survival::survreg.
+
**Example**
http://git-wip-us.apache.org/repos/asf/spark/blob/e21a9dde/mllib/src/main/scala/org/apache/spark/ml/classification/LogisticRegression.scala
--
diff --git
a/mllib/src/main/scala/org/apache/spark/ml/classification/LogisticRegression.scala
b/mllib/src/main/scala/org/apache/spark/ml/classification/LogisticRegression.scala
index 1ea4d90..51ede15 100644
---
a/mllib/src/main/scala/org/apache/spark/ml/classification/LogisticRegression.scala
+++
b/mllib/src/main/scala/org/apache/spark/ml/classification/LogisticRegression.scala
@@ -333,6 +333,13 @@ class LogisticRegression @Since("1.2.0") (
val featuresMean = summarizer.mean.toArray
val featuresStd = summarizer.variance.toArray.map(math.sqrt)
+if (!$(fitIntercept) && (0 until numFeatures).exists { i =>
+ featuresStd(i) == 0.0 && featuresMean(i) != 0.0 }) {
+ logWarning("Fitting LogisticRegressionModel without intercept on
dataset with " +
+"constant nonzero column, Spark MLlib outputs zero coefficients
for constant " +
+
spark git commit: [SPARK-15738][PYSPARK][ML] Adding Pyspark ml RFormula __str__ method similar to Scala API
Repository: spark
Updated Branches:
refs/heads/master 254bc8c34 -> 7d7a0a5e0
[SPARK-15738][PYSPARK][ML] Adding Pyspark ml RFormula __str__ method similar to
Scala API
## What changes were proposed in this pull request?
Adding __str__ to RFormula and model that will show the set formula param and
resolved formula. This is currently present in the Scala API, found missing in
PySpark during Spark 2.0 coverage review.
## How was this patch tested?
run pyspark-ml tests locally
Author: Bryan Cutler
Closes #13481 from BryanCutler/pyspark-ml-rformula_str-SPARK-15738.
Project: http://git-wip-us.apache.org/repos/asf/spark/repo
Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/7d7a0a5e
Tree: http://git-wip-us.apache.org/repos/asf/spark/tree/7d7a0a5e
Diff: http://git-wip-us.apache.org/repos/asf/spark/diff/7d7a0a5e
Branch: refs/heads/master
Commit: 7d7a0a5e0749909e97d90188707cc9220a1bb73a
Parents: 254bc8c
Author: Bryan Cutler
Authored: Fri Jun 10 11:27:30 2016 -0700
Committer: Yanbo Liang
Committed: Fri Jun 10 11:27:30 2016 -0700
--
.../scala/org/apache/spark/ml/feature/RFormula.scala | 2 +-
.../org/apache/spark/ml/feature/RFormulaParser.scala | 14 +-
python/pyspark/ml/feature.py | 12
3 files changed, 26 insertions(+), 2 deletions(-)
--
http://git-wip-us.apache.org/repos/asf/spark/blob/7d7a0a5e/mllib/src/main/scala/org/apache/spark/ml/feature/RFormula.scala
--
diff --git a/mllib/src/main/scala/org/apache/spark/ml/feature/RFormula.scala
b/mllib/src/main/scala/org/apache/spark/ml/feature/RFormula.scala
index 2916b6d..a7ca0fe 100644
--- a/mllib/src/main/scala/org/apache/spark/ml/feature/RFormula.scala
+++ b/mllib/src/main/scala/org/apache/spark/ml/feature/RFormula.scala
@@ -182,7 +182,7 @@ class RFormula(override val uid: String)
override def copy(extra: ParamMap): RFormula = defaultCopy(extra)
- override def toString: String = s"RFormula(${get(formula)}) (uid=$uid)"
+ override def toString: String = s"RFormula(${get(formula).getOrElse("")})
(uid=$uid)"
}
@Since("2.0.0")
http://git-wip-us.apache.org/repos/asf/spark/blob/7d7a0a5e/mllib/src/main/scala/org/apache/spark/ml/feature/RFormulaParser.scala
--
diff --git
a/mllib/src/main/scala/org/apache/spark/ml/feature/RFormulaParser.scala
b/mllib/src/main/scala/org/apache/spark/ml/feature/RFormulaParser.scala
index 19aecff..2dd565a 100644
--- a/mllib/src/main/scala/org/apache/spark/ml/feature/RFormulaParser.scala
+++ b/mllib/src/main/scala/org/apache/spark/ml/feature/RFormulaParser.scala
@@ -126,7 +126,19 @@ private[ml] case class ParsedRFormula(label: ColumnRef,
terms: Seq[Term]) {
* @param hasIntercept whether the formula specifies fitting with an intercept.
*/
private[ml] case class ResolvedRFormula(
- label: String, terms: Seq[Seq[String]], hasIntercept: Boolean)
+ label: String, terms: Seq[Seq[String]], hasIntercept: Boolean) {
+
+ override def toString: String = {
+val ts = terms.map {
+ case t if t.length > 1 =>
+s"${t.mkString("{", ",", "}")}"
+ case t =>
+t.mkString
+}
+val termStr = ts.mkString("[", ",", "]")
+s"ResolvedRFormula(label=$label, terms=$termStr,
hasIntercept=$hasIntercept)"
+ }
+}
/**
* R formula terms. See the R formula docs here for more information:
http://git-wip-us.apache.org/repos/asf/spark/blob/7d7a0a5e/python/pyspark/ml/feature.py
--
diff --git a/python/pyspark/ml/feature.py b/python/pyspark/ml/feature.py
index bfb2fb7..ca77ac3 100755
--- a/python/pyspark/ml/feature.py
+++ b/python/pyspark/ml/feature.py
@@ -2528,6 +2528,8 @@ class RFormula(JavaEstimator, HasFeaturesCol,
HasLabelCol, JavaMLReadable, JavaM
True
>>> loadedRF.getLabelCol() == rf.getLabelCol()
True
+>>> str(loadedRF)
+'RFormula(y ~ x + s) (uid=...)'
>>> modelPath = temp_path + "/rFormulaModel"
>>> model.save(modelPath)
>>> loadedModel = RFormulaModel.load(modelPath)
@@ -2542,6 +2544,8 @@ class RFormula(JavaEstimator, HasFeaturesCol,
HasLabelCol, JavaMLReadable, JavaM
|0.0|0.0| a|[0.0,1.0]| 0.0|
+---+---+---+-+-+
...
+>>> str(loadedModel)
+'RFormulaModel(ResolvedRFormula(label=y, terms=[x,s], hasIntercept=true))
(uid=...)'
.. versionadded:: 1.5.0
"""
@@ -2586,6 +2590,10 @@ class RFormula(JavaEstimator, HasFeaturesCol,
HasLabelCol, JavaMLReadable, JavaM
def _create_model(self, java_model):
return RFormulaModel(java_model)
+def __str__(self):
+formulaStr = self.getFormula() if self.isDefined(self.formula) else ""
+return "RFormula(%s) (uid=%s)" % (formulaStr, self.uid)
spark git commit: [SPARK-15738][PYSPARK][ML] Adding Pyspark ml RFormula __str__ method similar to Scala API
Repository: spark
Updated Branches:
refs/heads/branch-2.0 8b6742a37 -> 80b8711b3
[SPARK-15738][PYSPARK][ML] Adding Pyspark ml RFormula __str__ method similar to
Scala API
## What changes were proposed in this pull request?
Adding __str__ to RFormula and model that will show the set formula param and
resolved formula. This is currently present in the Scala API, found missing in
PySpark during Spark 2.0 coverage review.
## How was this patch tested?
run pyspark-ml tests locally
Author: Bryan Cutler
Closes #13481 from BryanCutler/pyspark-ml-rformula_str-SPARK-15738.
(cherry picked from commit 7d7a0a5e0749909e97d90188707cc9220a1bb73a)
Signed-off-by: Yanbo Liang
Project: http://git-wip-us.apache.org/repos/asf/spark/repo
Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/80b8711b
Tree: http://git-wip-us.apache.org/repos/asf/spark/tree/80b8711b
Diff: http://git-wip-us.apache.org/repos/asf/spark/diff/80b8711b
Branch: refs/heads/branch-2.0
Commit: 80b8711b342c5a569fe89d7ffbdd552653b9b6ec
Parents: 8b6742a
Author: Bryan Cutler
Authored: Fri Jun 10 11:27:30 2016 -0700
Committer: Yanbo Liang
Committed: Fri Jun 10 14:01:55 2016 -0700
--
.../scala/org/apache/spark/ml/feature/RFormula.scala | 2 +-
.../org/apache/spark/ml/feature/RFormulaParser.scala | 14 +-
python/pyspark/ml/feature.py | 12
3 files changed, 26 insertions(+), 2 deletions(-)
--
http://git-wip-us.apache.org/repos/asf/spark/blob/80b8711b/mllib/src/main/scala/org/apache/spark/ml/feature/RFormula.scala
--
diff --git a/mllib/src/main/scala/org/apache/spark/ml/feature/RFormula.scala
b/mllib/src/main/scala/org/apache/spark/ml/feature/RFormula.scala
index 2916b6d..a7ca0fe 100644
--- a/mllib/src/main/scala/org/apache/spark/ml/feature/RFormula.scala
+++ b/mllib/src/main/scala/org/apache/spark/ml/feature/RFormula.scala
@@ -182,7 +182,7 @@ class RFormula(override val uid: String)
override def copy(extra: ParamMap): RFormula = defaultCopy(extra)
- override def toString: String = s"RFormula(${get(formula)}) (uid=$uid)"
+ override def toString: String = s"RFormula(${get(formula).getOrElse("")})
(uid=$uid)"
}
@Since("2.0.0")
http://git-wip-us.apache.org/repos/asf/spark/blob/80b8711b/mllib/src/main/scala/org/apache/spark/ml/feature/RFormulaParser.scala
--
diff --git
a/mllib/src/main/scala/org/apache/spark/ml/feature/RFormulaParser.scala
b/mllib/src/main/scala/org/apache/spark/ml/feature/RFormulaParser.scala
index 19aecff..2dd565a 100644
--- a/mllib/src/main/scala/org/apache/spark/ml/feature/RFormulaParser.scala
+++ b/mllib/src/main/scala/org/apache/spark/ml/feature/RFormulaParser.scala
@@ -126,7 +126,19 @@ private[ml] case class ParsedRFormula(label: ColumnRef,
terms: Seq[Term]) {
* @param hasIntercept whether the formula specifies fitting with an intercept.
*/
private[ml] case class ResolvedRFormula(
- label: String, terms: Seq[Seq[String]], hasIntercept: Boolean)
+ label: String, terms: Seq[Seq[String]], hasIntercept: Boolean) {
+
+ override def toString: String = {
+val ts = terms.map {
+ case t if t.length > 1 =>
+s"${t.mkString("{", ",", "}")}"
+ case t =>
+t.mkString
+}
+val termStr = ts.mkString("[", ",", "]")
+s"ResolvedRFormula(label=$label, terms=$termStr,
hasIntercept=$hasIntercept)"
+ }
+}
/**
* R formula terms. See the R formula docs here for more information:
http://git-wip-us.apache.org/repos/asf/spark/blob/80b8711b/python/pyspark/ml/feature.py
--
diff --git a/python/pyspark/ml/feature.py b/python/pyspark/ml/feature.py
index bfb2fb7..ca77ac3 100755
--- a/python/pyspark/ml/feature.py
+++ b/python/pyspark/ml/feature.py
@@ -2528,6 +2528,8 @@ class RFormula(JavaEstimator, HasFeaturesCol,
HasLabelCol, JavaMLReadable, JavaM
True
>>> loadedRF.getLabelCol() == rf.getLabelCol()
True
+>>> str(loadedRF)
+'RFormula(y ~ x + s) (uid=...)'
>>> modelPath = temp_path + "/rFormulaModel"
>>> model.save(modelPath)
>>> loadedModel = RFormulaModel.load(modelPath)
@@ -2542,6 +2544,8 @@ class RFormula(JavaEstimator, HasFeaturesCol,
HasLabelCol, JavaMLReadable, JavaM
|0.0|0.0| a|[0.0,1.0]| 0.0|
+---+---+---+-+-+
...
+>>> str(loadedModel)
+'RFormulaModel(ResolvedRFormula(label=y, terms=[x,s], hasIntercept=true))
(uid=...)'
.. versionadded:: 1.5.0
"""
@@ -2586,6 +2590,10 @@ class RFormula(JavaEstimator, HasFeaturesCol,
HasLabelCol, JavaMLReadable, JavaM
def _create_model(self, java_model):
return RFormulaModel(java_model)
+def __str__(self):
+formulaStr = self.getFormula()
spark git commit: [SPARK-15945][MLLIB] Conversion between old/new vector columns in a DataFrame (Scala/Java)
Repository: spark
Updated Branches:
refs/heads/branch-2.0 b75542603 -> f277cdf78
[SPARK-15945][MLLIB] Conversion between old/new vector columns in a DataFrame
(Scala/Java)
## What changes were proposed in this pull request?
This PR provides conversion utils between old/new vector columns in a
DataFrame. So users can use it to migrate their datasets and pipelines
manually. The methods are implemented under `MLUtils` and called
`convertVectorColumnsToML` and `convertVectorColumnsFromML`. Both take a
DataFrame and a list of vector columns to be converted. It is a no-op on vector
columns that are already converted. A warning message is logged if actual
conversion happens.
This is the first sub-task under SPARK-15944 to make it easier to migrate
existing pipelines to Spark 2.0.
## How was this patch tested?
Unit tests in Scala and Java.
cc: yanboliang
Author: Xiangrui Meng
Closes #13662 from mengxr/SPARK-15945.
(cherry picked from commit 63e0aebe22ba41c636ecaddd8647721d7690a1ec)
Signed-off-by: Yanbo Liang
Project: http://git-wip-us.apache.org/repos/asf/spark/repo
Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/f277cdf7
Tree: http://git-wip-us.apache.org/repos/asf/spark/tree/f277cdf7
Diff: http://git-wip-us.apache.org/repos/asf/spark/diff/f277cdf7
Branch: refs/heads/branch-2.0
Commit: f277cdf787de4402cd6cdba5e15e38bb71d8c5c7
Parents: b755426
Author: Xiangrui Meng
Authored: Tue Jun 14 18:57:45 2016 -0700
Committer: Yanbo Liang
Committed: Tue Jun 14 18:58:02 2016 -0700
--
.../org/apache/spark/mllib/util/MLUtils.scala | 117 ++-
.../spark/mllib/util/JavaMLUtilsSuite.java | 49
.../apache/spark/mllib/util/MLUtilsSuite.scala | 60 +-
3 files changed, 218 insertions(+), 8 deletions(-)
--
http://git-wip-us.apache.org/repos/asf/spark/blob/f277cdf7/mllib/src/main/scala/org/apache/spark/mllib/util/MLUtils.scala
--
diff --git a/mllib/src/main/scala/org/apache/spark/mllib/util/MLUtils.scala
b/mllib/src/main/scala/org/apache/spark/mllib/util/MLUtils.scala
index f0346e6..7d5bdff 100644
--- a/mllib/src/main/scala/org/apache/spark/mllib/util/MLUtils.scala
+++ b/mllib/src/main/scala/org/apache/spark/mllib/util/MLUtils.scala
@@ -17,14 +17,19 @@
package org.apache.spark.mllib.util
+import scala.annotation.varargs
import scala.reflect.ClassTag
import org.apache.spark.SparkContext
import org.apache.spark.annotation.Since
-import org.apache.spark.mllib.linalg.{DenseVector, SparseVector, Vector,
Vectors}
+import org.apache.spark.internal.Logging
+import org.apache.spark.ml.linalg.{VectorUDT => MLVectorUDT}
+import org.apache.spark.mllib.linalg._
import org.apache.spark.mllib.linalg.BLAS.dot
import org.apache.spark.mllib.regression.LabeledPoint
import org.apache.spark.rdd.{PartitionwiseSampledRDD, RDD}
+import org.apache.spark.sql.{DataFrame, Dataset}
+import org.apache.spark.sql.functions.{col, udf}
import org.apache.spark.storage.StorageLevel
import org.apache.spark.util.random.BernoulliCellSampler
@@ -32,7 +37,7 @@ import org.apache.spark.util.random.BernoulliCellSampler
* Helper methods to load, save and pre-process data used in ML Lib.
*/
@Since("0.8.0")
-object MLUtils {
+object MLUtils extends Logging {
private[mllib] lazy val EPSILON = {
var eps = 1.0
@@ -50,7 +55,6 @@ object MLUtils {
* where the indices are one-based and in ascending order.
* This method parses each line into a
[[org.apache.spark.mllib.regression.LabeledPoint]],
* where the feature indices are converted to zero-based.
- *
* @param sc Spark context
* @param path file or directory path in any Hadoop-supported file system URI
* @param numFeatures number of features, which will be determined from the
input data if a
@@ -145,7 +149,6 @@ object MLUtils {
* Save labeled data in LIBSVM format.
* @param data an RDD of LabeledPoint to be saved
* @param dir directory to save the data
- *
* @see [[org.apache.spark.mllib.util.MLUtils#loadLibSVMFile]]
*/
@Since("1.0.0")
@@ -254,6 +257,110 @@ object MLUtils {
}
/**
+ * Converts vector columns in an input Dataset from the
[[org.apache.spark.mllib.linalg.Vector]]
+ * type to the new [[org.apache.spark.ml.linalg.Vector]] type under the
`spark.ml` package.
+ * @param dataset input dataset
+ * @param cols a list of vector columns to be converted. New vector columns
will be ignored. If
+ * unspecified, all old vector columns will be converted except
nested ones.
+ * @return the input [[DataFrame]] with old vector columns converted to the
new vector type
+ */
+ @Since("2.0.0")
+ @varargs
+ def convertVectorColumnsToML(dataset: Dataset[_], cols: String*): DataFrame
= {
+val schema = dataset.schema
+val colSet =
spark git commit: [SPARK-15945][MLLIB] Conversion between old/new vector columns in a DataFrame (Scala/Java)
Repository: spark
Updated Branches:
refs/heads/master 42a28caf1 -> 63e0aebe2
[SPARK-15945][MLLIB] Conversion between old/new vector columns in a DataFrame
(Scala/Java)
## What changes were proposed in this pull request?
This PR provides conversion utils between old/new vector columns in a
DataFrame. So users can use it to migrate their datasets and pipelines
manually. The methods are implemented under `MLUtils` and called
`convertVectorColumnsToML` and `convertVectorColumnsFromML`. Both take a
DataFrame and a list of vector columns to be converted. It is a no-op on vector
columns that are already converted. A warning message is logged if actual
conversion happens.
This is the first sub-task under SPARK-15944 to make it easier to migrate
existing pipelines to Spark 2.0.
## How was this patch tested?
Unit tests in Scala and Java.
cc: yanboliang
Author: Xiangrui Meng
Closes #13662 from mengxr/SPARK-15945.
Project: http://git-wip-us.apache.org/repos/asf/spark/repo
Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/63e0aebe
Tree: http://git-wip-us.apache.org/repos/asf/spark/tree/63e0aebe
Diff: http://git-wip-us.apache.org/repos/asf/spark/diff/63e0aebe
Branch: refs/heads/master
Commit: 63e0aebe22ba41c636ecaddd8647721d7690a1ec
Parents: 42a28ca
Author: Xiangrui Meng
Authored: Tue Jun 14 18:57:45 2016 -0700
Committer: Yanbo Liang
Committed: Tue Jun 14 18:57:45 2016 -0700
--
.../org/apache/spark/mllib/util/MLUtils.scala | 117 ++-
.../spark/mllib/util/JavaMLUtilsSuite.java | 49
.../apache/spark/mllib/util/MLUtilsSuite.scala | 60 +-
3 files changed, 218 insertions(+), 8 deletions(-)
--
http://git-wip-us.apache.org/repos/asf/spark/blob/63e0aebe/mllib/src/main/scala/org/apache/spark/mllib/util/MLUtils.scala
--
diff --git a/mllib/src/main/scala/org/apache/spark/mllib/util/MLUtils.scala
b/mllib/src/main/scala/org/apache/spark/mllib/util/MLUtils.scala
index f0346e6..7d5bdff 100644
--- a/mllib/src/main/scala/org/apache/spark/mllib/util/MLUtils.scala
+++ b/mllib/src/main/scala/org/apache/spark/mllib/util/MLUtils.scala
@@ -17,14 +17,19 @@
package org.apache.spark.mllib.util
+import scala.annotation.varargs
import scala.reflect.ClassTag
import org.apache.spark.SparkContext
import org.apache.spark.annotation.Since
-import org.apache.spark.mllib.linalg.{DenseVector, SparseVector, Vector,
Vectors}
+import org.apache.spark.internal.Logging
+import org.apache.spark.ml.linalg.{VectorUDT => MLVectorUDT}
+import org.apache.spark.mllib.linalg._
import org.apache.spark.mllib.linalg.BLAS.dot
import org.apache.spark.mllib.regression.LabeledPoint
import org.apache.spark.rdd.{PartitionwiseSampledRDD, RDD}
+import org.apache.spark.sql.{DataFrame, Dataset}
+import org.apache.spark.sql.functions.{col, udf}
import org.apache.spark.storage.StorageLevel
import org.apache.spark.util.random.BernoulliCellSampler
@@ -32,7 +37,7 @@ import org.apache.spark.util.random.BernoulliCellSampler
* Helper methods to load, save and pre-process data used in ML Lib.
*/
@Since("0.8.0")
-object MLUtils {
+object MLUtils extends Logging {
private[mllib] lazy val EPSILON = {
var eps = 1.0
@@ -50,7 +55,6 @@ object MLUtils {
* where the indices are one-based and in ascending order.
* This method parses each line into a
[[org.apache.spark.mllib.regression.LabeledPoint]],
* where the feature indices are converted to zero-based.
- *
* @param sc Spark context
* @param path file or directory path in any Hadoop-supported file system URI
* @param numFeatures number of features, which will be determined from the
input data if a
@@ -145,7 +149,6 @@ object MLUtils {
* Save labeled data in LIBSVM format.
* @param data an RDD of LabeledPoint to be saved
* @param dir directory to save the data
- *
* @see [[org.apache.spark.mllib.util.MLUtils#loadLibSVMFile]]
*/
@Since("1.0.0")
@@ -254,6 +257,110 @@ object MLUtils {
}
/**
+ * Converts vector columns in an input Dataset from the
[[org.apache.spark.mllib.linalg.Vector]]
+ * type to the new [[org.apache.spark.ml.linalg.Vector]] type under the
`spark.ml` package.
+ * @param dataset input dataset
+ * @param cols a list of vector columns to be converted. New vector columns
will be ignored. If
+ * unspecified, all old vector columns will be converted except
nested ones.
+ * @return the input [[DataFrame]] with old vector columns converted to the
new vector type
+ */
+ @Since("2.0.0")
+ @varargs
+ def convertVectorColumnsToML(dataset: Dataset[_], cols: String*): DataFrame
= {
+val schema = dataset.schema
+val colSet = if (cols.nonEmpty) {
+ cols.flatMap { c =>
+val dataType = schema(c).dataType
+if (d
spark git commit: [SPARK-15608][ML][EXAMPLES][DOC] add examples and documents of ml.isotonic regression
Repository: spark Updated Branches: refs/heads/branch-2.0 b3678eb7e -> 68e7a25cc [SPARK-15608][ML][EXAMPLES][DOC] add examples and documents of ml.isotonic regression ## What changes were proposed in this pull request? add ml doc for ml isotonic regression add scala example for ml isotonic regression add java example for ml isotonic regression add python example for ml isotonic regression modify scala example for mllib isotonic regression modify java example for mllib isotonic regression modify python example for mllib isotonic regression add data/mllib/sample_isotonic_regression_libsvm_data.txt delete data/mllib/sample_isotonic_regression_data.txt ## How was this patch tested? N/A Author: WeichenXu Closes #13381 from WeichenXu123/add_isotonic_regression_doc. (cherry picked from commit 9040d83bc2cdce06dab0e1bdee4f796da9a9a55c) Signed-off-by: Yanbo Liang Project: http://git-wip-us.apache.org/repos/asf/spark/repo Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/68e7a25c Tree: http://git-wip-us.apache.org/repos/asf/spark/tree/68e7a25c Diff: http://git-wip-us.apache.org/repos/asf/spark/diff/68e7a25c Branch: refs/heads/branch-2.0 Commit: 68e7a25cc06cbfe357be8d224c117abaa7ba94f4 Parents: b3678eb Author: WeichenXu Authored: Thu Jun 16 17:35:40 2016 -0700 Committer: Yanbo Liang Committed: Thu Jun 16 17:35:51 2016 -0700 -- data/mllib/sample_isotonic_regression_data.txt | 100 --- .../sample_isotonic_regression_libsvm_data.txt | 100 +++ docs/ml-classification-regression.md| 70 + .../ml/JavaIsotonicRegressionExample.java | 62 .../mllib/JavaIsotonicRegressionExample.java| 19 ++-- .../python/ml/isotonic_regression_example.py| 54 ++ .../python/mllib/isotonic_regression_example.py | 11 +- .../examples/ml/IsotonicRegressionExample.scala | 62 .../mllib/IsotonicRegressionExample.scala | 9 +- 9 files changed, 373 insertions(+), 114 deletions(-) -- http://git-wip-us.apache.org/repos/asf/spark/blob/68e7a25c/data/mllib/sample_isotonic_regression_data.txt -- diff --git a/data/mllib/sample_isotonic_regression_data.txt b/data/mllib/sample_isotonic_regression_data.txt deleted file mode 100644 index d257b50..000 --- a/data/mllib/sample_isotonic_regression_data.txt +++ /dev/null @@ -1,100 +0,0 @@ -0.24579296,0.01 -0.28505864,0.02 -0.31208567,0.03 -0.35900051,0.04 -0.35747068,0.05 -0.16675166,0.06 -0.17491076,0.07 -0.04181540,0.08 -0.04793473,0.09 -0.03926568,0.10 -0.12952575,0.11 -0.,0.12 -0.01376849,0.13 -0.13105558,0.14 -0.08873024,0.15 -0.12595614,0.16 -0.15247323,0.17 -0.25956145,0.18 -0.20040796,0.19 -0.19581846,0.20 -0.15757267,0.21 -0.13717491,0.22 -0.19020908,0.23 -0.19581846,0.24 -0.20091790,0.25 -0.16879143,0.26 -0.18510964,0.27 -0.20040796,0.28 -0.29576747,0.29 -0.43396226,0.30 -0.53391127,0.31 -0.52116267,0.32 -0.48546660,0.33 -0.49209587,0.34 -0.54156043,0.35 -0.59765426,0.36 -0.56144824,0.37 -0.58592555,0.38 -0.52983172,0.39 -0.50178480,0.40 -0.52626211,0.41 -0.58286588,0.42 -0.64660887,0.43 -0.68077511,0.44 -0.74298827,0.45 -0.64864865,0.46 -0.67261601,0.47 -0.65782764,0.48 -0.69811321,0.49 -0.63029067,0.50 -0.61601224,0.51 -0.63233044,0.52 -0.65323814,0.53 -0.65323814,0.54 -0.67363590,0.55 -0.67006629,0.56 -0.51555329,0.57 -0.50892402,0.58 -0.33299337,0.59 -0.36206017,0.60 -0.43090260,0.61 -0.45996940,0.62 -0.56348802,0.63 -0.54920959,0.64 -0.48393677,0.65 -0.48495665,0.66 -0.46965834,0.67 -0.45181030,0.68 -0.45843957,0.69 -0.47118817,0.70 -0.51555329,0.71 -0.58031617,0.72 -0.55481897,0.73 -0.56297807,0.74 -0.56603774,0.75 -0.57929628,0.76 -0.64762876,0.77 -0.66241713,0.78 -0.69301377,0.79 -0.65119837,0.80 -0.68332483,0.81 -0.66598674,0.82 -0.73890872,0.83 -0.73992861,0.84 -0.84242733,0.85 -0.91330954,0.86 -0.88016318,0.87 -0.90719021,0.88 -0.93115757,0.89 -0.93115757,0.90 -0.91942886,0.91 -0.92911780,0.92 -0.95665477,0.93 -0.95002550,0.94 -0.96940337,0.95 -1.,0.96 -0.89801122,0.97 -0.90311066,0.98 -0.90362060,0.99 -0.83477817,1.0 \ No newline at end of file http://git-wip-us.apache.org/repos/asf/spark/blob/68e7a25c/data/mllib/sample_isotonic_regression_libsvm_data.txt -- diff --git a/data/mllib/sample_isotonic_regression_libsvm_data.txt b/data/mllib/sample_isotonic_regression_libsvm_data.txt new file mode 100644 index 000..f39fe02 --- /dev/null +++ b/data/mllib/sample_isotonic_regression_libsvm_data.txt @@ -0,0 +1,100 @@ +0.24579296 1:0.01 +0.28505864 1:0.02 +0.31208567 1:0.03 +0.35900051 1:0.04 +0.35747068 1:0.05 +0.16675166 1:0.06 +0.17491076 1:0.07 +0.04181540 1:0.08 +0.04793473 1:0.09 +0.03926568 1:0.10 +0.12952575 1:0.11 +0. 1:0.12 +0.01376849 1:0.13
spark git commit: [SPARK-15608][ML][EXAMPLES][DOC] add examples and documents of ml.isotonic regression
Repository: spark Updated Branches: refs/heads/master d9c6628c4 -> 9040d83bc [SPARK-15608][ML][EXAMPLES][DOC] add examples and documents of ml.isotonic regression ## What changes were proposed in this pull request? add ml doc for ml isotonic regression add scala example for ml isotonic regression add java example for ml isotonic regression add python example for ml isotonic regression modify scala example for mllib isotonic regression modify java example for mllib isotonic regression modify python example for mllib isotonic regression add data/mllib/sample_isotonic_regression_libsvm_data.txt delete data/mllib/sample_isotonic_regression_data.txt ## How was this patch tested? N/A Author: WeichenXu Closes #13381 from WeichenXu123/add_isotonic_regression_doc. Project: http://git-wip-us.apache.org/repos/asf/spark/repo Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/9040d83b Tree: http://git-wip-us.apache.org/repos/asf/spark/tree/9040d83b Diff: http://git-wip-us.apache.org/repos/asf/spark/diff/9040d83b Branch: refs/heads/master Commit: 9040d83bc2cdce06dab0e1bdee4f796da9a9a55c Parents: d9c6628 Author: WeichenXu Authored: Thu Jun 16 17:35:40 2016 -0700 Committer: Yanbo Liang Committed: Thu Jun 16 17:35:40 2016 -0700 -- data/mllib/sample_isotonic_regression_data.txt | 100 --- .../sample_isotonic_regression_libsvm_data.txt | 100 +++ docs/ml-classification-regression.md| 70 + .../ml/JavaIsotonicRegressionExample.java | 62 .../mllib/JavaIsotonicRegressionExample.java| 19 ++-- .../python/ml/isotonic_regression_example.py| 54 ++ .../python/mllib/isotonic_regression_example.py | 11 +- .../examples/ml/IsotonicRegressionExample.scala | 62 .../mllib/IsotonicRegressionExample.scala | 9 +- 9 files changed, 373 insertions(+), 114 deletions(-) -- http://git-wip-us.apache.org/repos/asf/spark/blob/9040d83b/data/mllib/sample_isotonic_regression_data.txt -- diff --git a/data/mllib/sample_isotonic_regression_data.txt b/data/mllib/sample_isotonic_regression_data.txt deleted file mode 100644 index d257b50..000 --- a/data/mllib/sample_isotonic_regression_data.txt +++ /dev/null @@ -1,100 +0,0 @@ -0.24579296,0.01 -0.28505864,0.02 -0.31208567,0.03 -0.35900051,0.04 -0.35747068,0.05 -0.16675166,0.06 -0.17491076,0.07 -0.04181540,0.08 -0.04793473,0.09 -0.03926568,0.10 -0.12952575,0.11 -0.,0.12 -0.01376849,0.13 -0.13105558,0.14 -0.08873024,0.15 -0.12595614,0.16 -0.15247323,0.17 -0.25956145,0.18 -0.20040796,0.19 -0.19581846,0.20 -0.15757267,0.21 -0.13717491,0.22 -0.19020908,0.23 -0.19581846,0.24 -0.20091790,0.25 -0.16879143,0.26 -0.18510964,0.27 -0.20040796,0.28 -0.29576747,0.29 -0.43396226,0.30 -0.53391127,0.31 -0.52116267,0.32 -0.48546660,0.33 -0.49209587,0.34 -0.54156043,0.35 -0.59765426,0.36 -0.56144824,0.37 -0.58592555,0.38 -0.52983172,0.39 -0.50178480,0.40 -0.52626211,0.41 -0.58286588,0.42 -0.64660887,0.43 -0.68077511,0.44 -0.74298827,0.45 -0.64864865,0.46 -0.67261601,0.47 -0.65782764,0.48 -0.69811321,0.49 -0.63029067,0.50 -0.61601224,0.51 -0.63233044,0.52 -0.65323814,0.53 -0.65323814,0.54 -0.67363590,0.55 -0.67006629,0.56 -0.51555329,0.57 -0.50892402,0.58 -0.33299337,0.59 -0.36206017,0.60 -0.43090260,0.61 -0.45996940,0.62 -0.56348802,0.63 -0.54920959,0.64 -0.48393677,0.65 -0.48495665,0.66 -0.46965834,0.67 -0.45181030,0.68 -0.45843957,0.69 -0.47118817,0.70 -0.51555329,0.71 -0.58031617,0.72 -0.55481897,0.73 -0.56297807,0.74 -0.56603774,0.75 -0.57929628,0.76 -0.64762876,0.77 -0.66241713,0.78 -0.69301377,0.79 -0.65119837,0.80 -0.68332483,0.81 -0.66598674,0.82 -0.73890872,0.83 -0.73992861,0.84 -0.84242733,0.85 -0.91330954,0.86 -0.88016318,0.87 -0.90719021,0.88 -0.93115757,0.89 -0.93115757,0.90 -0.91942886,0.91 -0.92911780,0.92 -0.95665477,0.93 -0.95002550,0.94 -0.96940337,0.95 -1.,0.96 -0.89801122,0.97 -0.90311066,0.98 -0.90362060,0.99 -0.83477817,1.0 \ No newline at end of file http://git-wip-us.apache.org/repos/asf/spark/blob/9040d83b/data/mllib/sample_isotonic_regression_libsvm_data.txt -- diff --git a/data/mllib/sample_isotonic_regression_libsvm_data.txt b/data/mllib/sample_isotonic_regression_libsvm_data.txt new file mode 100644 index 000..f39fe02 --- /dev/null +++ b/data/mllib/sample_isotonic_regression_libsvm_data.txt @@ -0,0 +1,100 @@ +0.24579296 1:0.01 +0.28505864 1:0.02 +0.31208567 1:0.03 +0.35900051 1:0.04 +0.35747068 1:0.05 +0.16675166 1:0.06 +0.17491076 1:0.07 +0.04181540 1:0.08 +0.04793473 1:0.09 +0.03926568 1:0.10 +0.12952575 1:0.11 +0. 1:0.12 +0.01376849 1:0.13 +0.13105558 1:0.14 +0.08873024 1:0.15 +0.12595614 1:0.16 +0.15247323 1:0.17 +0.25956145 1:0.18 +0.20040796
spark git commit: [SPARK-15946][MLLIB] Conversion between old/new vector columns in a DataFrame (Python)
Repository: spark
Updated Branches:
refs/heads/master af2a4b082 -> edb23f9e4
[SPARK-15946][MLLIB] Conversion between old/new vector columns in a DataFrame
(Python)
## What changes were proposed in this pull request?
This PR implements python wrappers for #13662 to convert old/new vector columns
in a DataFrame.
## How was this patch tested?
doctest in Python
cc: yanboliang
Author: Xiangrui Meng
Closes #13731 from mengxr/SPARK-15946.
Project: http://git-wip-us.apache.org/repos/asf/spark/repo
Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/edb23f9e
Tree: http://git-wip-us.apache.org/repos/asf/spark/tree/edb23f9e
Diff: http://git-wip-us.apache.org/repos/asf/spark/diff/edb23f9e
Branch: refs/heads/master
Commit: edb23f9e47eecfe60992dde0e037ec1985c77e1d
Parents: af2a4b0
Author: Xiangrui Meng
Authored: Fri Jun 17 21:22:29 2016 -0700
Committer: Yanbo Liang
Committed: Fri Jun 17 21:22:29 2016 -0700
--
.../spark/mllib/api/python/PythonMLLibAPI.scala | 14
python/pyspark/mllib/util.py| 82
2 files changed, 96 insertions(+)
--
http://git-wip-us.apache.org/repos/asf/spark/blob/edb23f9e/mllib/src/main/scala/org/apache/spark/mllib/api/python/PythonMLLibAPI.scala
--
diff --git
a/mllib/src/main/scala/org/apache/spark/mllib/api/python/PythonMLLibAPI.scala
b/mllib/src/main/scala/org/apache/spark/mllib/api/python/PythonMLLibAPI.scala
index 7df6160..f2c70ba 100644
---
a/mllib/src/main/scala/org/apache/spark/mllib/api/python/PythonMLLibAPI.scala
+++
b/mllib/src/main/scala/org/apache/spark/mllib/api/python/PythonMLLibAPI.scala
@@ -1201,6 +1201,20 @@ private[python] class PythonMLLibAPI extends
Serializable {
val spark = SparkSession.builder().sparkContext(sc).getOrCreate()
spark.createDataFrame(blockMatrix.blocks)
}
+
+ /**
+ * Python-friendly version of [[MLUtils.convertVectorColumnsToML()]].
+ */
+ def convertVectorColumnsToML(dataset: DataFrame, cols: JArrayList[String]):
DataFrame = {
+MLUtils.convertVectorColumnsToML(dataset, cols.asScala: _*)
+ }
+
+ /**
+ * Python-friendly version of [[MLUtils.convertVectorColumnsFromML()]]
+ */
+ def convertVectorColumnsFromML(dataset: DataFrame, cols:
JArrayList[String]): DataFrame = {
+MLUtils.convertVectorColumnsFromML(dataset, cols.asScala: _*)
+ }
}
/**
http://git-wip-us.apache.org/repos/asf/spark/blob/edb23f9e/python/pyspark/mllib/util.py
--
diff --git a/python/pyspark/mllib/util.py b/python/pyspark/mllib/util.py
index a316ee1..a7e6bcc 100644
--- a/python/pyspark/mllib/util.py
+++ b/python/pyspark/mllib/util.py
@@ -26,6 +26,7 @@ if sys.version > '3':
from pyspark import SparkContext, since
from pyspark.mllib.common import callMLlibFunc, inherit_doc
from pyspark.mllib.linalg import Vectors, SparseVector, _convert_to_vector
+from pyspark.sql import DataFrame
class MLUtils(object):
@@ -200,6 +201,86 @@ class MLUtils(object):
"""
return callMLlibFunc("loadVectors", sc, path)
+@staticmethod
+@since("2.0.0")
+def convertVectorColumnsToML(dataset, *cols):
+"""
+Converts vector columns in an input DataFrame from the
+:py:class:`pyspark.mllib.linalg.Vector` type to the new
+:py:class:`pyspark.ml.linalg.Vector` type under the `spark.ml`
+package.
+
+:param dataset:
+ input dataset
+:param cols:
+ a list of vector columns to be converted.
+ New vector columns will be ignored. If unspecified, all old
+ vector columns will be converted excepted nested ones.
+:return:
+ the input dataset with old vector columns converted to the
+ new vector type
+
+>>> import pyspark
+>>> from pyspark.mllib.linalg import Vectors
+>>> from pyspark.mllib.util import MLUtils
+>>> df = spark.createDataFrame(
+... [(0, Vectors.sparse(2, [1], [1.0]), Vectors.dense(2.0, 3.0))],
+... ["id", "x", "y"])
+>>> r1 = MLUtils.convertVectorColumnsToML(df).first()
+>>> isinstance(r1.x, pyspark.ml.linalg.SparseVector)
+True
+>>> isinstance(r1.y, pyspark.ml.linalg.DenseVector)
+True
+>>> r2 = MLUtils.convertVectorColumnsToML(df, "x").first()
+>>> isinstance(r2.x, pyspark.ml.linalg.SparseVector)
+True
+>>> isinstance(r2.y, pyspark.mllib.linalg.DenseVector)
+True
+"""
+if not isinstance(dataset, DataFrame):
+raise TypeError("Input dataset must be a DataFrame but got
{}.".format(type(dataset)))
+return callMLlibFunc("convertVectorColumnsToML", dataset, list(cols))
+
+@staticmethod
+@since("2.0.0")
+def con
spark git commit: [SPARK-15946][MLLIB] Conversion between old/new vector columns in a DataFrame (Python)
Repository: spark
Updated Branches:
refs/heads/branch-2.0 f0de45cb1 -> 0a8fd2eb8
[SPARK-15946][MLLIB] Conversion between old/new vector columns in a DataFrame
(Python)
## What changes were proposed in this pull request?
This PR implements python wrappers for #13662 to convert old/new vector columns
in a DataFrame.
## How was this patch tested?
doctest in Python
cc: yanboliang
Author: Xiangrui Meng
Closes #13731 from mengxr/SPARK-15946.
(cherry picked from commit edb23f9e47eecfe60992dde0e037ec1985c77e1d)
Signed-off-by: Yanbo Liang
Project: http://git-wip-us.apache.org/repos/asf/spark/repo
Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/0a8fd2eb
Tree: http://git-wip-us.apache.org/repos/asf/spark/tree/0a8fd2eb
Diff: http://git-wip-us.apache.org/repos/asf/spark/diff/0a8fd2eb
Branch: refs/heads/branch-2.0
Commit: 0a8fd2eb8966afaff3030adef5fc6fd73171607c
Parents: f0de45c
Author: Xiangrui Meng
Authored: Fri Jun 17 21:22:29 2016 -0700
Committer: Yanbo Liang
Committed: Fri Jun 17 21:22:41 2016 -0700
--
.../spark/mllib/api/python/PythonMLLibAPI.scala | 14
python/pyspark/mllib/util.py| 82
2 files changed, 96 insertions(+)
--
http://git-wip-us.apache.org/repos/asf/spark/blob/0a8fd2eb/mllib/src/main/scala/org/apache/spark/mllib/api/python/PythonMLLibAPI.scala
--
diff --git
a/mllib/src/main/scala/org/apache/spark/mllib/api/python/PythonMLLibAPI.scala
b/mllib/src/main/scala/org/apache/spark/mllib/api/python/PythonMLLibAPI.scala
index 7df6160..f2c70ba 100644
---
a/mllib/src/main/scala/org/apache/spark/mllib/api/python/PythonMLLibAPI.scala
+++
b/mllib/src/main/scala/org/apache/spark/mllib/api/python/PythonMLLibAPI.scala
@@ -1201,6 +1201,20 @@ private[python] class PythonMLLibAPI extends
Serializable {
val spark = SparkSession.builder().sparkContext(sc).getOrCreate()
spark.createDataFrame(blockMatrix.blocks)
}
+
+ /**
+ * Python-friendly version of [[MLUtils.convertVectorColumnsToML()]].
+ */
+ def convertVectorColumnsToML(dataset: DataFrame, cols: JArrayList[String]):
DataFrame = {
+MLUtils.convertVectorColumnsToML(dataset, cols.asScala: _*)
+ }
+
+ /**
+ * Python-friendly version of [[MLUtils.convertVectorColumnsFromML()]]
+ */
+ def convertVectorColumnsFromML(dataset: DataFrame, cols:
JArrayList[String]): DataFrame = {
+MLUtils.convertVectorColumnsFromML(dataset, cols.asScala: _*)
+ }
}
/**
http://git-wip-us.apache.org/repos/asf/spark/blob/0a8fd2eb/python/pyspark/mllib/util.py
--
diff --git a/python/pyspark/mllib/util.py b/python/pyspark/mllib/util.py
index a316ee1..a7e6bcc 100644
--- a/python/pyspark/mllib/util.py
+++ b/python/pyspark/mllib/util.py
@@ -26,6 +26,7 @@ if sys.version > '3':
from pyspark import SparkContext, since
from pyspark.mllib.common import callMLlibFunc, inherit_doc
from pyspark.mllib.linalg import Vectors, SparseVector, _convert_to_vector
+from pyspark.sql import DataFrame
class MLUtils(object):
@@ -200,6 +201,86 @@ class MLUtils(object):
"""
return callMLlibFunc("loadVectors", sc, path)
+@staticmethod
+@since("2.0.0")
+def convertVectorColumnsToML(dataset, *cols):
+"""
+Converts vector columns in an input DataFrame from the
+:py:class:`pyspark.mllib.linalg.Vector` type to the new
+:py:class:`pyspark.ml.linalg.Vector` type under the `spark.ml`
+package.
+
+:param dataset:
+ input dataset
+:param cols:
+ a list of vector columns to be converted.
+ New vector columns will be ignored. If unspecified, all old
+ vector columns will be converted excepted nested ones.
+:return:
+ the input dataset with old vector columns converted to the
+ new vector type
+
+>>> import pyspark
+>>> from pyspark.mllib.linalg import Vectors
+>>> from pyspark.mllib.util import MLUtils
+>>> df = spark.createDataFrame(
+... [(0, Vectors.sparse(2, [1], [1.0]), Vectors.dense(2.0, 3.0))],
+... ["id", "x", "y"])
+>>> r1 = MLUtils.convertVectorColumnsToML(df).first()
+>>> isinstance(r1.x, pyspark.ml.linalg.SparseVector)
+True
+>>> isinstance(r1.y, pyspark.ml.linalg.DenseVector)
+True
+>>> r2 = MLUtils.convertVectorColumnsToML(df, "x").first()
+>>> isinstance(r2.x, pyspark.ml.linalg.SparseVector)
+True
+>>> isinstance(r2.y, pyspark.mllib.linalg.DenseVector)
+True
+"""
+if not isinstance(dataset, DataFrame):
+raise TypeError("Input dataset must be a DataFrame but got
{}.".format(type(dataset)))
+return callMLlibFu
spark git commit: [SPARK-18401][SPARKR][ML] SparkR random forest should support output original label.
Repository: spark
Updated Branches:
refs/heads/master a3356343c -> 5ddf69470
[SPARK-18401][SPARKR][ML] SparkR random forest should support output original
label.
## What changes were proposed in this pull request?
SparkR ```spark.randomForest``` classification prediction should output
original label rather than the indexed label. This issue is very similar with
[SPARK-18291](https://issues.apache.org/jira/browse/SPARK-18291).
## How was this patch tested?
Add unit tests.
Author: Yanbo Liang
Closes #15842 from yanboliang/spark-18401.
Project: http://git-wip-us.apache.org/repos/asf/spark/repo
Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/5ddf6947
Tree: http://git-wip-us.apache.org/repos/asf/spark/tree/5ddf6947
Diff: http://git-wip-us.apache.org/repos/asf/spark/diff/5ddf6947
Branch: refs/heads/master
Commit: 5ddf69470b93c0b8a28bb4ac905e7670d9c50a95
Parents: a335634
Author: Yanbo Liang
Authored: Thu Nov 10 17:13:10 2016 -0800
Committer: Yanbo Liang
Committed: Thu Nov 10 17:13:10 2016 -0800
--
R/pkg/inst/tests/testthat/test_mllib.R | 24 +
.../r/RandomForestClassificationWrapper.scala | 28 +---
2 files changed, 48 insertions(+), 4 deletions(-)
--
http://git-wip-us.apache.org/repos/asf/spark/blob/5ddf6947/R/pkg/inst/tests/testthat/test_mllib.R
--
diff --git a/R/pkg/inst/tests/testthat/test_mllib.R
b/R/pkg/inst/tests/testthat/test_mllib.R
index 33e9d0d..b76f75d 100644
--- a/R/pkg/inst/tests/testthat/test_mllib.R
+++ b/R/pkg/inst/tests/testthat/test_mllib.R
@@ -935,6 +935,10 @@ test_that("spark.randomForest Classification", {
expect_equal(stats$numTrees, 20)
expect_error(capture.output(stats), NA)
expect_true(length(capture.output(stats)) > 6)
+ # Test string prediction values
+ predictions <- collect(predict(model, data))$prediction
+ expect_equal(length(grep("setosa", predictions)), 50)
+ expect_equal(length(grep("versicolor", predictions)), 50)
modelPath <- tempfile(pattern = "spark-randomForestClassification", fileext
= ".tmp")
write.ml(model, modelPath)
@@ -947,6 +951,26 @@ test_that("spark.randomForest Classification", {
expect_equal(stats$numClasses, stats2$numClasses)
unlink(modelPath)
+
+ # Test numeric response variable
+ labelToIndex <- function(species) {
+switch(as.character(species),
+ setosa = 0.0,
+ versicolor = 1.0,
+ virginica = 2.0
+)
+ }
+ iris$NumericSpecies <- lapply(iris$Species, labelToIndex)
+ data <- suppressWarnings(createDataFrame(iris[-5]))
+ model <- spark.randomForest(data, NumericSpecies ~ Petal_Length +
Petal_Width, "classification",
+ maxDepth = 5, maxBins = 16)
+ stats <- summary(model)
+ expect_equal(stats$numFeatures, 2)
+ expect_equal(stats$numTrees, 20)
+ # Test numeric prediction values
+ predictions <- collect(predict(model, data))$prediction
+ expect_equal(length(grep("1.0", predictions)), 50)
+ expect_equal(length(grep("2.0", predictions)), 50)
})
test_that("spark.gbt", {
http://git-wip-us.apache.org/repos/asf/spark/blob/5ddf6947/mllib/src/main/scala/org/apache/spark/ml/r/RandomForestClassificationWrapper.scala
--
diff --git
a/mllib/src/main/scala/org/apache/spark/ml/r/RandomForestClassificationWrapper.scala
b/mllib/src/main/scala/org/apache/spark/ml/r/RandomForestClassificationWrapper.scala
index 6947ba7..31f846d 100644
---
a/mllib/src/main/scala/org/apache/spark/ml/r/RandomForestClassificationWrapper.scala
+++
b/mllib/src/main/scala/org/apache/spark/ml/r/RandomForestClassificationWrapper.scala
@@ -23,9 +23,9 @@ import org.json4s.JsonDSL._
import org.json4s.jackson.JsonMethods._
import org.apache.spark.ml.{Pipeline, PipelineModel}
-import org.apache.spark.ml.attribute.AttributeGroup
+import org.apache.spark.ml.attribute.{Attribute, AttributeGroup,
NominalAttribute}
import org.apache.spark.ml.classification.{RandomForestClassificationModel,
RandomForestClassifier}
-import org.apache.spark.ml.feature.RFormula
+import org.apache.spark.ml.feature.{IndexToString, RFormula}
import org.apache.spark.ml.linalg.Vector
import org.apache.spark.ml.util._
import org.apache.spark.sql.{DataFrame, Dataset}
@@ -35,6 +35,8 @@ private[r] class RandomForestClassifierWrapper private (
val formula: String,
val features: Array[String]) extends MLWritable {
+ import RandomForestClassifierWrapper._
+
private val rfcModel: RandomForestClassificationModel =
pipeline.stages(1).asInstanceOf[RandomForestClassificationModel]
@@ -46,7 +48,9 @@ private[r] class RandomForestClassifierWrapper private (
def summary: String = rfcModel.toDebugString
def transform(dataset: Dataset[_]): DataFrame = {
-pipeline.transform(d
spark git commit: [SPARK-18401][SPARKR][ML] SparkR random forest should support output original label.
Repository: spark
Updated Branches:
refs/heads/branch-2.1 064d4315f -> 51dca6143
[SPARK-18401][SPARKR][ML] SparkR random forest should support output original
label.
## What changes were proposed in this pull request?
SparkR ```spark.randomForest``` classification prediction should output
original label rather than the indexed label. This issue is very similar with
[SPARK-18291](https://issues.apache.org/jira/browse/SPARK-18291).
## How was this patch tested?
Add unit tests.
Author: Yanbo Liang
Closes #15842 from yanboliang/spark-18401.
(cherry picked from commit 5ddf69470b93c0b8a28bb4ac905e7670d9c50a95)
Signed-off-by: Yanbo Liang
Project: http://git-wip-us.apache.org/repos/asf/spark/repo
Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/51dca614
Tree: http://git-wip-us.apache.org/repos/asf/spark/tree/51dca614
Diff: http://git-wip-us.apache.org/repos/asf/spark/diff/51dca614
Branch: refs/heads/branch-2.1
Commit: 51dca6143670ec1c1cb090047c3941becaf41fa9
Parents: 064d431
Author: Yanbo Liang
Authored: Thu Nov 10 17:13:10 2016 -0800
Committer: Yanbo Liang
Committed: Thu Nov 10 17:13:26 2016 -0800
--
R/pkg/inst/tests/testthat/test_mllib.R | 24 +
.../r/RandomForestClassificationWrapper.scala | 28 +---
2 files changed, 48 insertions(+), 4 deletions(-)
--
http://git-wip-us.apache.org/repos/asf/spark/blob/51dca614/R/pkg/inst/tests/testthat/test_mllib.R
--
diff --git a/R/pkg/inst/tests/testthat/test_mllib.R
b/R/pkg/inst/tests/testthat/test_mllib.R
index 1e456ef..33e85b7 100644
--- a/R/pkg/inst/tests/testthat/test_mllib.R
+++ b/R/pkg/inst/tests/testthat/test_mllib.R
@@ -935,6 +935,10 @@ test_that("spark.randomForest Classification", {
expect_equal(stats$numTrees, 20)
expect_error(capture.output(stats), NA)
expect_true(length(capture.output(stats)) > 6)
+ # Test string prediction values
+ predictions <- collect(predict(model, data))$prediction
+ expect_equal(length(grep("setosa", predictions)), 50)
+ expect_equal(length(grep("versicolor", predictions)), 50)
modelPath <- tempfile(pattern = "spark-randomForestClassification", fileext
= ".tmp")
write.ml(model, modelPath)
@@ -947,6 +951,26 @@ test_that("spark.randomForest Classification", {
expect_equal(stats$numClasses, stats2$numClasses)
unlink(modelPath)
+
+ # Test numeric response variable
+ labelToIndex <- function(species) {
+switch(as.character(species),
+ setosa = 0.0,
+ versicolor = 1.0,
+ virginica = 2.0
+)
+ }
+ iris$NumericSpecies <- lapply(iris$Species, labelToIndex)
+ data <- suppressWarnings(createDataFrame(iris[-5]))
+ model <- spark.randomForest(data, NumericSpecies ~ Petal_Length +
Petal_Width, "classification",
+ maxDepth = 5, maxBins = 16)
+ stats <- summary(model)
+ expect_equal(stats$numFeatures, 2)
+ expect_equal(stats$numTrees, 20)
+ # Test numeric prediction values
+ predictions <- collect(predict(model, data))$prediction
+ expect_equal(length(grep("1.0", predictions)), 50)
+ expect_equal(length(grep("2.0", predictions)), 50)
})
test_that("spark.gbt", {
http://git-wip-us.apache.org/repos/asf/spark/blob/51dca614/mllib/src/main/scala/org/apache/spark/ml/r/RandomForestClassificationWrapper.scala
--
diff --git
a/mllib/src/main/scala/org/apache/spark/ml/r/RandomForestClassificationWrapper.scala
b/mllib/src/main/scala/org/apache/spark/ml/r/RandomForestClassificationWrapper.scala
index 6947ba7..31f846d 100644
---
a/mllib/src/main/scala/org/apache/spark/ml/r/RandomForestClassificationWrapper.scala
+++
b/mllib/src/main/scala/org/apache/spark/ml/r/RandomForestClassificationWrapper.scala
@@ -23,9 +23,9 @@ import org.json4s.JsonDSL._
import org.json4s.jackson.JsonMethods._
import org.apache.spark.ml.{Pipeline, PipelineModel}
-import org.apache.spark.ml.attribute.AttributeGroup
+import org.apache.spark.ml.attribute.{Attribute, AttributeGroup,
NominalAttribute}
import org.apache.spark.ml.classification.{RandomForestClassificationModel,
RandomForestClassifier}
-import org.apache.spark.ml.feature.RFormula
+import org.apache.spark.ml.feature.{IndexToString, RFormula}
import org.apache.spark.ml.linalg.Vector
import org.apache.spark.ml.util._
import org.apache.spark.sql.{DataFrame, Dataset}
@@ -35,6 +35,8 @@ private[r] class RandomForestClassifierWrapper private (
val formula: String,
val features: Array[String]) extends MLWritable {
+ import RandomForestClassifierWrapper._
+
private val rfcModel: RandomForestClassificationModel =
pipeline.stages(1).asInstanceOf[RandomForestClassificationModel]
@@ -46,7 +48,9 @@ private[r] class RandomForestClassifierWrapper private (
def summary: String
spark git commit: [SPARK-14077][ML][FOLLOW-UP] Minor refactor and cleanup for NaiveBayes
Repository: spark
Updated Branches:
refs/heads/master bc41d997e -> 22cb3a060
[SPARK-14077][ML][FOLLOW-UP] Minor refactor and cleanup for NaiveBayes
## What changes were proposed in this pull request?
* Refactor out ```trainWithLabelCheck``` and make ```mllib.NaiveBayes``` call
into it.
* Avoid capturing the outer object for ```modelType```.
* Move ```requireNonnegativeValues``` and ```requireZeroOneBernoulliValues```
to companion object.
## How was this patch tested?
Existing tests.
Author: Yanbo Liang
Closes #15826 from yanboliang/spark-14077-2.
Project: http://git-wip-us.apache.org/repos/asf/spark/repo
Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/22cb3a06
Tree: http://git-wip-us.apache.org/repos/asf/spark/tree/22cb3a06
Diff: http://git-wip-us.apache.org/repos/asf/spark/diff/22cb3a06
Branch: refs/heads/master
Commit: 22cb3a060a440205281b71686637679645454ca6
Parents: bc41d99
Author: Yanbo Liang
Authored: Sat Nov 12 06:13:22 2016 -0800
Committer: Yanbo Liang
Committed: Sat Nov 12 06:13:22 2016 -0800
--
.../spark/ml/classification/NaiveBayes.scala| 72 ++--
.../spark/mllib/classification/NaiveBayes.scala | 6 +-
2 files changed, 39 insertions(+), 39 deletions(-)
--
http://git-wip-us.apache.org/repos/asf/spark/blob/22cb3a06/mllib/src/main/scala/org/apache/spark/ml/classification/NaiveBayes.scala
--
diff --git
a/mllib/src/main/scala/org/apache/spark/ml/classification/NaiveBayes.scala
b/mllib/src/main/scala/org/apache/spark/ml/classification/NaiveBayes.scala
index b03a07a..f1a7676 100644
--- a/mllib/src/main/scala/org/apache/spark/ml/classification/NaiveBayes.scala
+++ b/mllib/src/main/scala/org/apache/spark/ml/classification/NaiveBayes.scala
@@ -76,7 +76,7 @@ class NaiveBayes @Since("1.5.0") (
extends ProbabilisticClassifier[Vector, NaiveBayes, NaiveBayesModel]
with NaiveBayesParams with DefaultParamsWritable {
- import NaiveBayes.{Bernoulli, Multinomial}
+ import NaiveBayes._
@Since("1.5.0")
def this() = this(Identifiable.randomUID("nb"))
@@ -110,21 +110,20 @@ class NaiveBayes @Since("1.5.0") (
@Since("2.1.0")
def setWeightCol(value: String): this.type = set(weightCol, value)
+ override protected def train(dataset: Dataset[_]): NaiveBayesModel = {
+trainWithLabelCheck(dataset, positiveLabel = true)
+ }
+
/**
* ml assumes input labels in range [0, numClasses). But this implementation
* is also called by mllib NaiveBayes which allows other kinds of input
labels
- * such as {-1, +1}. Here we use this parameter to switch between different
processing logic.
- * It should be removed when we remove mllib NaiveBayes.
+ * such as {-1, +1}. `positiveLabel` is used to determine whether the label
+ * should be checked and it should be removed when we remove mllib
NaiveBayes.
*/
- private[spark] var isML: Boolean = true
-
- private[spark] def setIsML(isML: Boolean): this.type = {
-this.isML = isML
-this
- }
-
- override protected def train(dataset: Dataset[_]): NaiveBayesModel = {
-if (isML) {
+ private[spark] def trainWithLabelCheck(
+ dataset: Dataset[_],
+ positiveLabel: Boolean): NaiveBayesModel = {
+if (positiveLabel) {
val numClasses = getNumClasses(dataset)
if (isDefined(thresholds)) {
require($(thresholds).length == numClasses,
this.getClass.getSimpleName +
@@ -133,28 +132,9 @@ class NaiveBayes @Since("1.5.0") (
}
}
-val requireNonnegativeValues: Vector => Unit = (v: Vector) => {
- val values = v match {
-case sv: SparseVector => sv.values
-case dv: DenseVector => dv.values
- }
-
- require(values.forall(_ >= 0.0),
-s"Naive Bayes requires nonnegative feature values but found $v.")
-}
-
-val requireZeroOneBernoulliValues: Vector => Unit = (v: Vector) => {
- val values = v match {
-case sv: SparseVector => sv.values
-case dv: DenseVector => dv.values
- }
-
- require(values.forall(v => v == 0.0 || v == 1.0),
-s"Bernoulli naive Bayes requires 0 or 1 feature values but found $v.")
-}
-
+val modelTypeValue = $(modelType)
val requireValues: Vector => Unit = {
- $(modelType) match {
+ modelTypeValue match {
case Multinomial =>
requireNonnegativeValues
case Bernoulli =>
@@ -226,13 +206,33 @@ class NaiveBayes @Since("1.5.0") (
@Since("1.6.0")
object NaiveBayes extends DefaultParamsReadable[NaiveBayes] {
/** String name for multinomial model type. */
- private[spark] val Multinomial: String = "multinomial"
+ private[classification] val Multinomial: String = "multinomial"
/** String name for Bernoulli model type. */
- private[spark] val Bernoulli: String = "bernoulli"
+ private[classification
spark git commit: [SPARK-14077][ML][FOLLOW-UP] Minor refactor and cleanup for NaiveBayes
Repository: spark
Updated Branches:
refs/heads/branch-2.1 893355143 -> b2ba83d10
[SPARK-14077][ML][FOLLOW-UP] Minor refactor and cleanup for NaiveBayes
## What changes were proposed in this pull request?
* Refactor out ```trainWithLabelCheck``` and make ```mllib.NaiveBayes``` call
into it.
* Avoid capturing the outer object for ```modelType```.
* Move ```requireNonnegativeValues``` and ```requireZeroOneBernoulliValues```
to companion object.
## How was this patch tested?
Existing tests.
Author: Yanbo Liang
Closes #15826 from yanboliang/spark-14077-2.
(cherry picked from commit 22cb3a060a440205281b71686637679645454ca6)
Signed-off-by: Yanbo Liang
Project: http://git-wip-us.apache.org/repos/asf/spark/repo
Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/b2ba83d1
Tree: http://git-wip-us.apache.org/repos/asf/spark/tree/b2ba83d1
Diff: http://git-wip-us.apache.org/repos/asf/spark/diff/b2ba83d1
Branch: refs/heads/branch-2.1
Commit: b2ba83d10ac06614c0126f4b0d913f6979051682
Parents: 8933551
Author: Yanbo Liang
Authored: Sat Nov 12 06:13:22 2016 -0800
Committer: Yanbo Liang
Committed: Sat Nov 12 06:18:45 2016 -0800
--
.../spark/ml/classification/NaiveBayes.scala| 72 ++--
.../spark/mllib/classification/NaiveBayes.scala | 6 +-
2 files changed, 39 insertions(+), 39 deletions(-)
--
http://git-wip-us.apache.org/repos/asf/spark/blob/b2ba83d1/mllib/src/main/scala/org/apache/spark/ml/classification/NaiveBayes.scala
--
diff --git
a/mllib/src/main/scala/org/apache/spark/ml/classification/NaiveBayes.scala
b/mllib/src/main/scala/org/apache/spark/ml/classification/NaiveBayes.scala
index b03a07a..f1a7676 100644
--- a/mllib/src/main/scala/org/apache/spark/ml/classification/NaiveBayes.scala
+++ b/mllib/src/main/scala/org/apache/spark/ml/classification/NaiveBayes.scala
@@ -76,7 +76,7 @@ class NaiveBayes @Since("1.5.0") (
extends ProbabilisticClassifier[Vector, NaiveBayes, NaiveBayesModel]
with NaiveBayesParams with DefaultParamsWritable {
- import NaiveBayes.{Bernoulli, Multinomial}
+ import NaiveBayes._
@Since("1.5.0")
def this() = this(Identifiable.randomUID("nb"))
@@ -110,21 +110,20 @@ class NaiveBayes @Since("1.5.0") (
@Since("2.1.0")
def setWeightCol(value: String): this.type = set(weightCol, value)
+ override protected def train(dataset: Dataset[_]): NaiveBayesModel = {
+trainWithLabelCheck(dataset, positiveLabel = true)
+ }
+
/**
* ml assumes input labels in range [0, numClasses). But this implementation
* is also called by mllib NaiveBayes which allows other kinds of input
labels
- * such as {-1, +1}. Here we use this parameter to switch between different
processing logic.
- * It should be removed when we remove mllib NaiveBayes.
+ * such as {-1, +1}. `positiveLabel` is used to determine whether the label
+ * should be checked and it should be removed when we remove mllib
NaiveBayes.
*/
- private[spark] var isML: Boolean = true
-
- private[spark] def setIsML(isML: Boolean): this.type = {
-this.isML = isML
-this
- }
-
- override protected def train(dataset: Dataset[_]): NaiveBayesModel = {
-if (isML) {
+ private[spark] def trainWithLabelCheck(
+ dataset: Dataset[_],
+ positiveLabel: Boolean): NaiveBayesModel = {
+if (positiveLabel) {
val numClasses = getNumClasses(dataset)
if (isDefined(thresholds)) {
require($(thresholds).length == numClasses,
this.getClass.getSimpleName +
@@ -133,28 +132,9 @@ class NaiveBayes @Since("1.5.0") (
}
}
-val requireNonnegativeValues: Vector => Unit = (v: Vector) => {
- val values = v match {
-case sv: SparseVector => sv.values
-case dv: DenseVector => dv.values
- }
-
- require(values.forall(_ >= 0.0),
-s"Naive Bayes requires nonnegative feature values but found $v.")
-}
-
-val requireZeroOneBernoulliValues: Vector => Unit = (v: Vector) => {
- val values = v match {
-case sv: SparseVector => sv.values
-case dv: DenseVector => dv.values
- }
-
- require(values.forall(v => v == 0.0 || v == 1.0),
-s"Bernoulli naive Bayes requires 0 or 1 feature values but found $v.")
-}
-
+val modelTypeValue = $(modelType)
val requireValues: Vector => Unit = {
- $(modelType) match {
+ modelTypeValue match {
case Multinomial =>
requireNonnegativeValues
case Bernoulli =>
@@ -226,13 +206,33 @@ class NaiveBayes @Since("1.5.0") (
@Since("1.6.0")
object NaiveBayes extends DefaultParamsReadable[NaiveBayes] {
/** String name for multinomial model type. */
- private[spark] val Multinomial: String = "multinomial"
+ private[classification] val Multinomial: String = "multinomial"
/** String name fo
spark git commit: [SPARK-18412][SPARKR][ML] Fix exception for some SparkR ML algorithms training on libsvm data
Repository: spark
Updated Branches:
refs/heads/master b91a51bb2 -> 07be232ea
[SPARK-18412][SPARKR][ML] Fix exception for some SparkR ML algorithms training
on libsvm data
## What changes were proposed in this pull request?
* Fix the following exceptions which throws when
```spark.randomForest```(classification), ```spark.gbt```(classification),
```spark.naiveBayes``` and ```spark.glm```(binomial family) were fitted on
libsvm data.
```
java.lang.IllegalArgumentException: requirement failed: If label column already
exists, forceIndexLabel can not be set with true.
```
See [SPARK-18412](https://issues.apache.org/jira/browse/SPARK-18412) for more
detail about how to reproduce this bug.
* Refactor out ```getFeaturesAndLabels``` to RWrapperUtils, since lots of ML
algorithm wrappers use this function.
* Drop some unwanted columns when making prediction.
## How was this patch tested?
Add unit test.
Author: Yanbo Liang
Closes #15851 from yanboliang/spark-18412.
Project: http://git-wip-us.apache.org/repos/asf/spark/repo
Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/07be232e
Tree: http://git-wip-us.apache.org/repos/asf/spark/tree/07be232e
Diff: http://git-wip-us.apache.org/repos/asf/spark/diff/07be232e
Branch: refs/heads/master
Commit: 07be232ea12dfc8dc3701ca948814be7dbebf4ee
Parents: b91a51b
Author: Yanbo Liang
Authored: Sun Nov 13 20:25:12 2016 -0800
Committer: Yanbo Liang
Committed: Sun Nov 13 20:25:12 2016 -0800
--
R/pkg/inst/tests/testthat/test_mllib.R | 18 --
.../spark/ml/r/GBTClassificationWrapper.scala | 18 --
.../r/GeneralizedLinearRegressionWrapper.scala | 5 ++-
.../apache/spark/ml/r/NaiveBayesWrapper.scala | 14 +++-
.../org/apache/spark/ml/r/RWrapperUtils.scala | 36 +---
.../r/RandomForestClassificationWrapper.scala | 18 --
6 files changed, 68 insertions(+), 41 deletions(-)
--
http://git-wip-us.apache.org/repos/asf/spark/blob/07be232e/R/pkg/inst/tests/testthat/test_mllib.R
--
diff --git a/R/pkg/inst/tests/testthat/test_mllib.R
b/R/pkg/inst/tests/testthat/test_mllib.R
index b76f75d..07df4b6 100644
--- a/R/pkg/inst/tests/testthat/test_mllib.R
+++ b/R/pkg/inst/tests/testthat/test_mllib.R
@@ -881,7 +881,8 @@ test_that("spark.kstest", {
expect_match(capture.output(stats)[1], "Kolmogorov-Smirnov test summary:")
})
-test_that("spark.randomForest Regression", {
+test_that("spark.randomForest", {
+ # regression
data <- suppressWarnings(createDataFrame(longley))
model <- spark.randomForest(data, Employed ~ ., "regression", maxDepth = 5,
maxBins = 16,
numTrees = 1)
@@ -923,9 +924,8 @@ test_that("spark.randomForest Regression", {
expect_equal(stats$treeWeights, stats2$treeWeights)
unlink(modelPath)
-})
-test_that("spark.randomForest Classification", {
+ # classification
data <- suppressWarnings(createDataFrame(iris))
model <- spark.randomForest(data, Species ~ Petal_Length + Petal_Width,
"classification",
maxDepth = 5, maxBins = 16)
@@ -971,6 +971,12 @@ test_that("spark.randomForest Classification", {
predictions <- collect(predict(model, data))$prediction
expect_equal(length(grep("1.0", predictions)), 50)
expect_equal(length(grep("2.0", predictions)), 50)
+
+ # spark.randomForest classification can work on libsvm data
+ data <-
read.df(absoluteSparkPath("data/mllib/sample_multiclass_classification_data.txt"),
+source = "libsvm")
+ model <- spark.randomForest(data, label ~ features, "classification")
+ expect_equal(summary(model)$numFeatures, 4)
})
test_that("spark.gbt", {
@@ -1039,6 +1045,12 @@ test_that("spark.gbt", {
expect_equal(iris2$NumericSpecies, as.double(collect(predict(m,
df))$prediction))
expect_equal(s$numFeatures, 5)
expect_equal(s$numTrees, 20)
+
+ # spark.gbt classification can work on libsvm data
+ data <-
read.df(absoluteSparkPath("data/mllib/sample_binary_classification_data.txt"),
+source = "libsvm")
+ model <- spark.gbt(data, label ~ features, "classification")
+ expect_equal(summary(model)$numFeatures, 692)
})
sparkR.session.stop()
http://git-wip-us.apache.org/repos/asf/spark/blob/07be232e/mllib/src/main/scala/org/apache/spark/ml/r/GBTClassificationWrapper.scala
--
diff --git
a/mllib/src/main/scala/org/apache/spark/ml/r/GBTClassificationWrapper.scala
b/mllib/src/main/scala/org/apache/spark/ml/r/GBTClassificationWrapper.scala
index 8946025..aacb41e 100644
--- a/mllib/src/main/scala/org/apache/spark/ml/r/GBTClassificationWrapper.scala
+++ b/mllib/src/main/scala/org/apache/spark/ml/r/GBTClassificationWrapper.scala
@@ -23,10 +23,10 @@ import org.json4s.JsonDSL._
spark git commit: [SPARK-18412][SPARKR][ML] Fix exception for some SparkR ML algorithms training on libsvm data
Repository: spark
Updated Branches:
refs/heads/branch-2.1 0c69224ed -> 8fc6455c0
[SPARK-18412][SPARKR][ML] Fix exception for some SparkR ML algorithms training
on libsvm data
## What changes were proposed in this pull request?
* Fix the following exceptions which throws when
```spark.randomForest```(classification), ```spark.gbt```(classification),
```spark.naiveBayes``` and ```spark.glm```(binomial family) were fitted on
libsvm data.
```
java.lang.IllegalArgumentException: requirement failed: If label column already
exists, forceIndexLabel can not be set with true.
```
See [SPARK-18412](https://issues.apache.org/jira/browse/SPARK-18412) for more
detail about how to reproduce this bug.
* Refactor out ```getFeaturesAndLabels``` to RWrapperUtils, since lots of ML
algorithm wrappers use this function.
* Drop some unwanted columns when making prediction.
## How was this patch tested?
Add unit test.
Author: Yanbo Liang
Closes #15851 from yanboliang/spark-18412.
(cherry picked from commit 07be232ea12dfc8dc3701ca948814be7dbebf4ee)
Signed-off-by: Yanbo Liang
Project: http://git-wip-us.apache.org/repos/asf/spark/repo
Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/8fc6455c
Tree: http://git-wip-us.apache.org/repos/asf/spark/tree/8fc6455c
Diff: http://git-wip-us.apache.org/repos/asf/spark/diff/8fc6455c
Branch: refs/heads/branch-2.1
Commit: 8fc6455c0b77f81be79908bb65e6264bf61c90e7
Parents: 0c69224
Author: Yanbo Liang
Authored: Sun Nov 13 20:25:12 2016 -0800
Committer: Yanbo Liang
Committed: Sun Nov 13 20:25:30 2016 -0800
--
R/pkg/inst/tests/testthat/test_mllib.R | 18 --
.../spark/ml/r/GBTClassificationWrapper.scala | 18 --
.../r/GeneralizedLinearRegressionWrapper.scala | 5 ++-
.../apache/spark/ml/r/NaiveBayesWrapper.scala | 14 +++-
.../org/apache/spark/ml/r/RWrapperUtils.scala | 36 +---
.../r/RandomForestClassificationWrapper.scala | 18 --
6 files changed, 68 insertions(+), 41 deletions(-)
--
http://git-wip-us.apache.org/repos/asf/spark/blob/8fc6455c/R/pkg/inst/tests/testthat/test_mllib.R
--
diff --git a/R/pkg/inst/tests/testthat/test_mllib.R
b/R/pkg/inst/tests/testthat/test_mllib.R
index 33e85b7..4831ce2 100644
--- a/R/pkg/inst/tests/testthat/test_mllib.R
+++ b/R/pkg/inst/tests/testthat/test_mllib.R
@@ -881,7 +881,8 @@ test_that("spark.kstest", {
expect_match(capture.output(stats)[1], "Kolmogorov-Smirnov test summary:")
})
-test_that("spark.randomForest Regression", {
+test_that("spark.randomForest", {
+ # regression
data <- suppressWarnings(createDataFrame(longley))
model <- spark.randomForest(data, Employed ~ ., "regression", maxDepth = 5,
maxBins = 16,
numTrees = 1)
@@ -923,9 +924,8 @@ test_that("spark.randomForest Regression", {
expect_equal(stats$treeWeights, stats2$treeWeights)
unlink(modelPath)
-})
-test_that("spark.randomForest Classification", {
+ # classification
data <- suppressWarnings(createDataFrame(iris))
model <- spark.randomForest(data, Species ~ Petal_Length + Petal_Width,
"classification",
maxDepth = 5, maxBins = 16)
@@ -971,6 +971,12 @@ test_that("spark.randomForest Classification", {
predictions <- collect(predict(model, data))$prediction
expect_equal(length(grep("1.0", predictions)), 50)
expect_equal(length(grep("2.0", predictions)), 50)
+
+ # spark.randomForest classification can work on libsvm data
+ data <-
read.df(absoluteSparkPath("data/mllib/sample_multiclass_classification_data.txt"),
+source = "libsvm")
+ model <- spark.randomForest(data, label ~ features, "classification")
+ expect_equal(summary(model)$numFeatures, 4)
})
test_that("spark.gbt", {
@@ -1039,6 +1045,12 @@ test_that("spark.gbt", {
expect_equal(iris2$NumericSpecies, as.double(collect(predict(m,
df))$prediction))
expect_equal(s$numFeatures, 5)
expect_equal(s$numTrees, 20)
+
+ # spark.gbt classification can work on libsvm data
+ data <-
read.df(absoluteSparkPath("data/mllib/sample_binary_classification_data.txt"),
+source = "libsvm")
+ model <- spark.gbt(data, label ~ features, "classification")
+ expect_equal(summary(model)$numFeatures, 692)
})
sparkR.session.stop()
http://git-wip-us.apache.org/repos/asf/spark/blob/8fc6455c/mllib/src/main/scala/org/apache/spark/ml/r/GBTClassificationWrapper.scala
--
diff --git
a/mllib/src/main/scala/org/apache/spark/ml/r/GBTClassificationWrapper.scala
b/mllib/src/main/scala/org/apache/spark/ml/r/GBTClassificationWrapper.scala
index 8946025..aacb41e 100644
--- a/mllib/src/main/scala/org/apache/spark/ml/r/GBTClassificationWrapper.scala
+++ b/mllib/src/main/sc
spark git commit: [SPARK-18438][SPARKR][ML] spark.mlp should support RFormula.
Repository: spark
Updated Branches:
refs/heads/master 4ac9759f8 -> 95eb06bd7
[SPARK-18438][SPARKR][ML] spark.mlp should support RFormula.
## What changes were proposed in this pull request?
```spark.mlp``` should support ```RFormula``` like other ML algorithm wrappers.
BTW, I did some cleanup and improvement for ```spark.mlp```.
## How was this patch tested?
Unit tests.
Author: Yanbo Liang
Closes #15883 from yanboliang/spark-18438.
Project: http://git-wip-us.apache.org/repos/asf/spark/repo
Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/95eb06bd
Tree: http://git-wip-us.apache.org/repos/asf/spark/tree/95eb06bd
Diff: http://git-wip-us.apache.org/repos/asf/spark/diff/95eb06bd
Branch: refs/heads/master
Commit: 95eb06bd7d0f7110ef62c8d1cb6337c72b10d99f
Parents: 4ac9759
Author: Yanbo Liang
Authored: Wed Nov 16 01:04:18 2016 -0800
Committer: Yanbo Liang
Committed: Wed Nov 16 01:04:18 2016 -0800
--
R/pkg/R/generics.R | 2 +-
R/pkg/R/mllib.R | 30 ++
R/pkg/inst/tests/testthat/test_mllib.R | 63 +---
.../MultilayerPerceptronClassifierWrapper.scala | 61 ++-
4 files changed, 96 insertions(+), 60 deletions(-)
--
http://git-wip-us.apache.org/repos/asf/spark/blob/95eb06bd/R/pkg/R/generics.R
--
diff --git a/R/pkg/R/generics.R b/R/pkg/R/generics.R
index 7653ca7..499c7b2 100644
--- a/R/pkg/R/generics.R
+++ b/R/pkg/R/generics.R
@@ -1373,7 +1373,7 @@ setGeneric("spark.logit", function(data, formula, ...) {
standardGeneric("spark.
#' @rdname spark.mlp
#' @export
-setGeneric("spark.mlp", function(data, ...) { standardGeneric("spark.mlp") })
+setGeneric("spark.mlp", function(data, formula, ...) {
standardGeneric("spark.mlp") })
#' @rdname spark.naiveBayes
#' @export
http://git-wip-us.apache.org/repos/asf/spark/blob/95eb06bd/R/pkg/R/mllib.R
--
diff --git a/R/pkg/R/mllib.R b/R/pkg/R/mllib.R
index 1065b4b..265e64e 100644
--- a/R/pkg/R/mllib.R
+++ b/R/pkg/R/mllib.R
@@ -525,7 +525,7 @@ setMethod("write.ml", signature(object = "LDAModel", path =
"character"),
#' @note spark.isoreg since 2.1.0
setMethod("spark.isoreg", signature(data = "SparkDataFrame", formula =
"formula"),
function(data, formula, isotonic = TRUE, featureIndex = 0, weightCol
= NULL) {
-formula <- paste0(deparse(formula), collapse = "")
+formula <- paste(deparse(formula), collapse = "")
if (is.null(weightCol)) {
weightCol <- ""
@@ -775,7 +775,7 @@ setMethod("spark.logit", signature(data = "SparkDataFrame",
formula = "formula")
tol = 1E-6, fitIntercept = TRUE, family = "auto",
standardization = TRUE,
thresholds = 0.5, weightCol = NULL, aggregationDepth = 2,
probabilityCol = "probability") {
-formula <- paste0(deparse(formula), collapse = "")
+formula <- paste(deparse(formula), collapse = "")
if (is.null(weightCol)) {
weightCol <- ""
@@ -858,6 +858,8 @@ setMethod("summary", signature(object =
"LogisticRegressionModel"),
#' Multilayer Perceptron}
#'
#' @param data a \code{SparkDataFrame} of observations and labels for model
fitting.
+#' @param formula a symbolic description of the model to be fitted. Currently
only a few formula
+#'operators are supported, including '~', '.', ':', '+', and
'-'.
#' @param blockSize blockSize parameter.
#' @param layers integer vector containing the number of nodes for each layer
#' @param solver solver parameter, supported options: "gd" (minibatch gradient
descent) or "l-bfgs".
@@ -870,7 +872,7 @@ setMethod("summary", signature(object =
"LogisticRegressionModel"),
#' @param ... additional arguments passed to the method.
#' @return \code{spark.mlp} returns a fitted Multilayer Perceptron
Classification Model.
#' @rdname spark.mlp
-#' @aliases spark.mlp,SparkDataFrame-method
+#' @aliases spark.mlp,SparkDataFrame,formula-method
#' @name spark.mlp
#' @seealso \link{read.ml}
#' @export
@@ -879,7 +881,7 @@ setMethod("summary", signature(object =
"LogisticRegressionModel"),
#' df <- read.df("data/mllib/sample_multiclass_classification_data.txt",
source = "libsvm")
#'
#' # fit a Multilayer Perceptron Classification Model
-#' model <- spark.mlp(df, blockSize = 128, layers = c(4, 3), solver = "l-bfgs",
+#' model <- spark.mlp(df, label ~ features, blockSize = 128, layers = c(4, 3),
solver = "l-bfgs",
#'maxIter = 100, tol = 0.5, stepSize = 1, seed = 1,
#'initialWeights = c(0, 0, 0, 0, 0, 5, 5, 5, 5, 5, 9, 9,
9, 9, 9))
#'
@@ -896,9 +898,10 @@ setMethod("summary", signat
spark git commit: [SPARK-18438][SPARKR][ML] spark.mlp should support RFormula.
Repository: spark
Updated Branches:
refs/heads/branch-2.1 436ae201f -> 7b57e480d
[SPARK-18438][SPARKR][ML] spark.mlp should support RFormula.
## What changes were proposed in this pull request?
```spark.mlp``` should support ```RFormula``` like other ML algorithm wrappers.
BTW, I did some cleanup and improvement for ```spark.mlp```.
## How was this patch tested?
Unit tests.
Author: Yanbo Liang
Closes #15883 from yanboliang/spark-18438.
(cherry picked from commit 95eb06bd7d0f7110ef62c8d1cb6337c72b10d99f)
Signed-off-by: Yanbo Liang
Project: http://git-wip-us.apache.org/repos/asf/spark/repo
Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/7b57e480
Tree: http://git-wip-us.apache.org/repos/asf/spark/tree/7b57e480
Diff: http://git-wip-us.apache.org/repos/asf/spark/diff/7b57e480
Branch: refs/heads/branch-2.1
Commit: 7b57e480d2f2c0695eb4036199cd0db52c6f2008
Parents: 436ae20
Author: Yanbo Liang
Authored: Wed Nov 16 01:04:18 2016 -0800
Committer: Yanbo Liang
Committed: Wed Nov 16 01:05:23 2016 -0800
--
R/pkg/R/generics.R | 2 +-
R/pkg/R/mllib.R | 30 ++
R/pkg/inst/tests/testthat/test_mllib.R | 63 +---
.../MultilayerPerceptronClassifierWrapper.scala | 61 ++-
4 files changed, 96 insertions(+), 60 deletions(-)
--
http://git-wip-us.apache.org/repos/asf/spark/blob/7b57e480/R/pkg/R/generics.R
--
diff --git a/R/pkg/R/generics.R b/R/pkg/R/generics.R
index 7653ca7..499c7b2 100644
--- a/R/pkg/R/generics.R
+++ b/R/pkg/R/generics.R
@@ -1373,7 +1373,7 @@ setGeneric("spark.logit", function(data, formula, ...) {
standardGeneric("spark.
#' @rdname spark.mlp
#' @export
-setGeneric("spark.mlp", function(data, ...) { standardGeneric("spark.mlp") })
+setGeneric("spark.mlp", function(data, formula, ...) {
standardGeneric("spark.mlp") })
#' @rdname spark.naiveBayes
#' @export
http://git-wip-us.apache.org/repos/asf/spark/blob/7b57e480/R/pkg/R/mllib.R
--
diff --git a/R/pkg/R/mllib.R b/R/pkg/R/mllib.R
index 1065b4b..265e64e 100644
--- a/R/pkg/R/mllib.R
+++ b/R/pkg/R/mllib.R
@@ -525,7 +525,7 @@ setMethod("write.ml", signature(object = "LDAModel", path =
"character"),
#' @note spark.isoreg since 2.1.0
setMethod("spark.isoreg", signature(data = "SparkDataFrame", formula =
"formula"),
function(data, formula, isotonic = TRUE, featureIndex = 0, weightCol
= NULL) {
-formula <- paste0(deparse(formula), collapse = "")
+formula <- paste(deparse(formula), collapse = "")
if (is.null(weightCol)) {
weightCol <- ""
@@ -775,7 +775,7 @@ setMethod("spark.logit", signature(data = "SparkDataFrame",
formula = "formula")
tol = 1E-6, fitIntercept = TRUE, family = "auto",
standardization = TRUE,
thresholds = 0.5, weightCol = NULL, aggregationDepth = 2,
probabilityCol = "probability") {
-formula <- paste0(deparse(formula), collapse = "")
+formula <- paste(deparse(formula), collapse = "")
if (is.null(weightCol)) {
weightCol <- ""
@@ -858,6 +858,8 @@ setMethod("summary", signature(object =
"LogisticRegressionModel"),
#' Multilayer Perceptron}
#'
#' @param data a \code{SparkDataFrame} of observations and labels for model
fitting.
+#' @param formula a symbolic description of the model to be fitted. Currently
only a few formula
+#'operators are supported, including '~', '.', ':', '+', and
'-'.
#' @param blockSize blockSize parameter.
#' @param layers integer vector containing the number of nodes for each layer
#' @param solver solver parameter, supported options: "gd" (minibatch gradient
descent) or "l-bfgs".
@@ -870,7 +872,7 @@ setMethod("summary", signature(object =
"LogisticRegressionModel"),
#' @param ... additional arguments passed to the method.
#' @return \code{spark.mlp} returns a fitted Multilayer Perceptron
Classification Model.
#' @rdname spark.mlp
-#' @aliases spark.mlp,SparkDataFrame-method
+#' @aliases spark.mlp,SparkDataFrame,formula-method
#' @name spark.mlp
#' @seealso \link{read.ml}
#' @export
@@ -879,7 +881,7 @@ setMethod("summary", signature(object =
"LogisticRegressionModel"),
#' df <- read.df("data/mllib/sample_multiclass_classification_data.txt",
source = "libsvm")
#'
#' # fit a Multilayer Perceptron Classification Model
-#' model <- spark.mlp(df, blockSize = 128, layers = c(4, 3), solver = "l-bfgs",
+#' model <- spark.mlp(df, label ~ features, blockSize = 128, layers = c(4, 3),
solver = "l-bfgs",
#'maxIter = 100, tol = 0.5, stepSize = 1, seed = 1,
#'initialWeigh
spark git commit: [SPARK-18434][ML] Add missing ParamValidations for ML algos
Repository: spark Updated Branches: refs/heads/master 241e04bc0 -> c68f1a38a [SPARK-18434][ML] Add missing ParamValidations for ML algos ## What changes were proposed in this pull request? Add missing ParamValidations for ML algos ## How was this patch tested? existing tests Author: Zheng RuiFeng Closes #15881 from zhengruifeng/arg_checking. Project: http://git-wip-us.apache.org/repos/asf/spark/repo Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/c68f1a38 Tree: http://git-wip-us.apache.org/repos/asf/spark/tree/c68f1a38 Diff: http://git-wip-us.apache.org/repos/asf/spark/diff/c68f1a38 Branch: refs/heads/master Commit: c68f1a38af67957ee28889667193da8f64bb4342 Parents: 241e04b Author: Zheng RuiFeng Authored: Wed Nov 16 02:46:27 2016 -0800 Committer: Yanbo Liang Committed: Wed Nov 16 02:46:27 2016 -0800 -- .../main/scala/org/apache/spark/ml/feature/IDF.scala | 3 ++- .../main/scala/org/apache/spark/ml/feature/PCA.scala | 3 ++- .../scala/org/apache/spark/ml/feature/Word2Vec.scala | 13 - .../spark/ml/regression/IsotonicRegression.scala | 3 ++- .../apache/spark/ml/regression/LinearRegression.scala | 6 +- .../scala/org/apache/spark/ml/tree/treeParams.scala| 4 +++- 6 files changed, 22 insertions(+), 10 deletions(-) -- http://git-wip-us.apache.org/repos/asf/spark/blob/c68f1a38/mllib/src/main/scala/org/apache/spark/ml/feature/IDF.scala -- diff --git a/mllib/src/main/scala/org/apache/spark/ml/feature/IDF.scala b/mllib/src/main/scala/org/apache/spark/ml/feature/IDF.scala index 6386dd8..46a0730 100644 --- a/mllib/src/main/scala/org/apache/spark/ml/feature/IDF.scala +++ b/mllib/src/main/scala/org/apache/spark/ml/feature/IDF.scala @@ -44,7 +44,8 @@ private[feature] trait IDFBase extends Params with HasInputCol with HasOutputCol * @group param */ final val minDocFreq = new IntParam( -this, "minDocFreq", "minimum number of documents in which a term should appear for filtering") +this, "minDocFreq", "minimum number of documents in which a term should appear for filtering" + + " (>= 0)", ParamValidators.gtEq(0)) setDefault(minDocFreq -> 0) http://git-wip-us.apache.org/repos/asf/spark/blob/c68f1a38/mllib/src/main/scala/org/apache/spark/ml/feature/PCA.scala -- diff --git a/mllib/src/main/scala/org/apache/spark/ml/feature/PCA.scala b/mllib/src/main/scala/org/apache/spark/ml/feature/PCA.scala index 6b91348..444006f 100644 --- a/mllib/src/main/scala/org/apache/spark/ml/feature/PCA.scala +++ b/mllib/src/main/scala/org/apache/spark/ml/feature/PCA.scala @@ -44,7 +44,8 @@ private[feature] trait PCAParams extends Params with HasInputCol with HasOutputC * The number of principal components. * @group param */ - final val k: IntParam = new IntParam(this, "k", "the number of principal components") + final val k: IntParam = new IntParam(this, "k", "the number of principal components (> 0)", +ParamValidators.gt(0)) /** @group getParam */ def getK: Int = $(k) http://git-wip-us.apache.org/repos/asf/spark/blob/c68f1a38/mllib/src/main/scala/org/apache/spark/ml/feature/Word2Vec.scala -- diff --git a/mllib/src/main/scala/org/apache/spark/ml/feature/Word2Vec.scala b/mllib/src/main/scala/org/apache/spark/ml/feature/Word2Vec.scala index d53f3df..3ed08c9 100644 --- a/mllib/src/main/scala/org/apache/spark/ml/feature/Word2Vec.scala +++ b/mllib/src/main/scala/org/apache/spark/ml/feature/Word2Vec.scala @@ -43,7 +43,8 @@ private[feature] trait Word2VecBase extends Params * @group param */ final val vectorSize = new IntParam( -this, "vectorSize", "the dimension of codes after transforming from words") +this, "vectorSize", "the dimension of codes after transforming from words (> 0)", +ParamValidators.gt(0)) setDefault(vectorSize -> 100) /** @group getParam */ @@ -55,7 +56,8 @@ private[feature] trait Word2VecBase extends Params * @group expertParam */ final val windowSize = new IntParam( -this, "windowSize", "the window size (context words from [-window, window])") +this, "windowSize", "the window size (context words from [-window, window]) (> 0)", +ParamValidators.gt(0)) setDefault(windowSize -> 5) /** @group expertGetParam */ @@ -67,7 +69,8 @@ private[feature] trait Word2VecBase extends Params * @group param */ final val numPartitions = new IntParam( -this, "numPartitions", "number of partitions for sentences of words") +this, "numPartitions", "number of partitions for sentences of words (> 0)", +ParamValidators.gt(0)) setDefault(numPartitions -> 1) /** @group getParam */ @@ -80,7 +83,7 @@ pri
spark git commit: [SPARK-18434][ML] Add missing ParamValidations for ML algos
Repository: spark Updated Branches: refs/heads/branch-2.1 820847008 -> 6b6eb4e52 [SPARK-18434][ML] Add missing ParamValidations for ML algos ## What changes were proposed in this pull request? Add missing ParamValidations for ML algos ## How was this patch tested? existing tests Author: Zheng RuiFeng Closes #15881 from zhengruifeng/arg_checking. (cherry picked from commit c68f1a38af67957ee28889667193da8f64bb4342) Signed-off-by: Yanbo Liang Project: http://git-wip-us.apache.org/repos/asf/spark/repo Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/6b6eb4e5 Tree: http://git-wip-us.apache.org/repos/asf/spark/tree/6b6eb4e5 Diff: http://git-wip-us.apache.org/repos/asf/spark/diff/6b6eb4e5 Branch: refs/heads/branch-2.1 Commit: 6b6eb4e520d07a27aa68d3450f3c7613b233d928 Parents: 8208470 Author: Zheng RuiFeng Authored: Wed Nov 16 02:46:27 2016 -0800 Committer: Yanbo Liang Committed: Wed Nov 16 02:46:54 2016 -0800 -- .../main/scala/org/apache/spark/ml/feature/IDF.scala | 3 ++- .../main/scala/org/apache/spark/ml/feature/PCA.scala | 3 ++- .../scala/org/apache/spark/ml/feature/Word2Vec.scala | 13 - .../spark/ml/regression/IsotonicRegression.scala | 3 ++- .../apache/spark/ml/regression/LinearRegression.scala | 6 +- .../scala/org/apache/spark/ml/tree/treeParams.scala| 4 +++- 6 files changed, 22 insertions(+), 10 deletions(-) -- http://git-wip-us.apache.org/repos/asf/spark/blob/6b6eb4e5/mllib/src/main/scala/org/apache/spark/ml/feature/IDF.scala -- diff --git a/mllib/src/main/scala/org/apache/spark/ml/feature/IDF.scala b/mllib/src/main/scala/org/apache/spark/ml/feature/IDF.scala index 6386dd8..46a0730 100644 --- a/mllib/src/main/scala/org/apache/spark/ml/feature/IDF.scala +++ b/mllib/src/main/scala/org/apache/spark/ml/feature/IDF.scala @@ -44,7 +44,8 @@ private[feature] trait IDFBase extends Params with HasInputCol with HasOutputCol * @group param */ final val minDocFreq = new IntParam( -this, "minDocFreq", "minimum number of documents in which a term should appear for filtering") +this, "minDocFreq", "minimum number of documents in which a term should appear for filtering" + + " (>= 0)", ParamValidators.gtEq(0)) setDefault(minDocFreq -> 0) http://git-wip-us.apache.org/repos/asf/spark/blob/6b6eb4e5/mllib/src/main/scala/org/apache/spark/ml/feature/PCA.scala -- diff --git a/mllib/src/main/scala/org/apache/spark/ml/feature/PCA.scala b/mllib/src/main/scala/org/apache/spark/ml/feature/PCA.scala index 6b91348..444006f 100644 --- a/mllib/src/main/scala/org/apache/spark/ml/feature/PCA.scala +++ b/mllib/src/main/scala/org/apache/spark/ml/feature/PCA.scala @@ -44,7 +44,8 @@ private[feature] trait PCAParams extends Params with HasInputCol with HasOutputC * The number of principal components. * @group param */ - final val k: IntParam = new IntParam(this, "k", "the number of principal components") + final val k: IntParam = new IntParam(this, "k", "the number of principal components (> 0)", +ParamValidators.gt(0)) /** @group getParam */ def getK: Int = $(k) http://git-wip-us.apache.org/repos/asf/spark/blob/6b6eb4e5/mllib/src/main/scala/org/apache/spark/ml/feature/Word2Vec.scala -- diff --git a/mllib/src/main/scala/org/apache/spark/ml/feature/Word2Vec.scala b/mllib/src/main/scala/org/apache/spark/ml/feature/Word2Vec.scala index d53f3df..3ed08c9 100644 --- a/mllib/src/main/scala/org/apache/spark/ml/feature/Word2Vec.scala +++ b/mllib/src/main/scala/org/apache/spark/ml/feature/Word2Vec.scala @@ -43,7 +43,8 @@ private[feature] trait Word2VecBase extends Params * @group param */ final val vectorSize = new IntParam( -this, "vectorSize", "the dimension of codes after transforming from words") +this, "vectorSize", "the dimension of codes after transforming from words (> 0)", +ParamValidators.gt(0)) setDefault(vectorSize -> 100) /** @group getParam */ @@ -55,7 +56,8 @@ private[feature] trait Word2VecBase extends Params * @group expertParam */ final val windowSize = new IntParam( -this, "windowSize", "the window size (context words from [-window, window])") +this, "windowSize", "the window size (context words from [-window, window]) (> 0)", +ParamValidators.gt(0)) setDefault(windowSize -> 5) /** @group expertGetParam */ @@ -67,7 +69,8 @@ private[feature] trait Word2VecBase extends Params * @group param */ final val numPartitions = new IntParam( -this, "numPartitions", "number of partitions for sentences of words") +this, "numPartitions", "number of partitions for sentences of words (> 0)", +
spark git commit: [SPARK-18282][ML][PYSPARK] Add python clustering summaries for GMM and BKM
Repository: spark
Updated Branches:
refs/heads/master 658547974 -> e811fbf9e
[SPARK-18282][ML][PYSPARK] Add python clustering summaries for GMM and BKM
## What changes were proposed in this pull request?
Add model summary APIs for `GaussianMixtureModel` and `BisectingKMeansModel` in
pyspark.
## How was this patch tested?
Unit tests.
Author: sethah
Closes #15777 from sethah/pyspark_cluster_summaries.
Project: http://git-wip-us.apache.org/repos/asf/spark/repo
Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/e811fbf9
Tree: http://git-wip-us.apache.org/repos/asf/spark/tree/e811fbf9
Diff: http://git-wip-us.apache.org/repos/asf/spark/diff/e811fbf9
Branch: refs/heads/master
Commit: e811fbf9ed131bccbc46f3c5701c4ff317222fd9
Parents: 6585479
Author: sethah
Authored: Mon Nov 21 05:36:49 2016 -0800
Committer: Yanbo Liang
Committed: Mon Nov 21 05:36:49 2016 -0800
--
.../ml/classification/LogisticRegression.scala | 11 +-
.../spark/ml/clustering/BisectingKMeans.scala | 9 +-
.../spark/ml/clustering/GaussianMixture.scala | 9 +-
.../org/apache/spark/ml/clustering/KMeans.scala | 9 +-
.../GeneralizedLinearRegression.scala | 11 +-
.../spark/ml/regression/LinearRegression.scala | 14 +-
.../LogisticRegressionSuite.scala | 2 +
.../ml/clustering/BisectingKMeansSuite.scala| 3 +
.../ml/clustering/GaussianMixtureSuite.scala| 3 +
.../spark/ml/clustering/KMeansSuite.scala | 3 +
.../GeneralizedLinearRegressionSuite.scala | 2 +
.../ml/regression/LinearRegressionSuite.scala | 2 +
python/pyspark/ml/classification.py | 15 +-
python/pyspark/ml/clustering.py | 162 ++-
python/pyspark/ml/regression.py | 16 +-
python/pyspark/ml/tests.py | 32
16 files changed, 256 insertions(+), 47 deletions(-)
--
http://git-wip-us.apache.org/repos/asf/spark/blob/e811fbf9/mllib/src/main/scala/org/apache/spark/ml/classification/LogisticRegression.scala
--
diff --git
a/mllib/src/main/scala/org/apache/spark/ml/classification/LogisticRegression.scala
b/mllib/src/main/scala/org/apache/spark/ml/classification/LogisticRegression.scala
index f58efd3..d07b4ad 100644
---
a/mllib/src/main/scala/org/apache/spark/ml/classification/LogisticRegression.scala
+++
b/mllib/src/main/scala/org/apache/spark/ml/classification/LogisticRegression.scala
@@ -648,7 +648,7 @@ class LogisticRegression @Since("1.2.0") (
$(labelCol),
$(featuresCol),
objectiveHistory)
- model.setSummary(logRegSummary)
+ model.setSummary(Some(logRegSummary))
} else {
model
}
@@ -790,9 +790,9 @@ class LogisticRegressionModel private[spark] (
}
}
- private[classification] def setSummary(
- summary: LogisticRegressionTrainingSummary): this.type = {
-this.trainingSummary = Some(summary)
+ private[classification]
+ def setSummary(summary: Option[LogisticRegressionTrainingSummary]):
this.type = {
+this.trainingSummary = summary
this
}
@@ -887,8 +887,7 @@ class LogisticRegressionModel private[spark] (
override def copy(extra: ParamMap): LogisticRegressionModel = {
val newModel = copyValues(new LogisticRegressionModel(uid,
coefficientMatrix, interceptVector,
numClasses, isMultinomial), extra)
-if (trainingSummary.isDefined) newModel.setSummary(trainingSummary.get)
-newModel.setParent(parent)
+newModel.setSummary(trainingSummary).setParent(parent)
}
override protected def raw2prediction(rawPrediction: Vector): Double = {
http://git-wip-us.apache.org/repos/asf/spark/blob/e811fbf9/mllib/src/main/scala/org/apache/spark/ml/clustering/BisectingKMeans.scala
--
diff --git
a/mllib/src/main/scala/org/apache/spark/ml/clustering/BisectingKMeans.scala
b/mllib/src/main/scala/org/apache/spark/ml/clustering/BisectingKMeans.scala
index f8a606d..e6ca3aed 100644
--- a/mllib/src/main/scala/org/apache/spark/ml/clustering/BisectingKMeans.scala
+++ b/mllib/src/main/scala/org/apache/spark/ml/clustering/BisectingKMeans.scala
@@ -95,8 +95,7 @@ class BisectingKMeansModel private[ml] (
@Since("2.0.0")
override def copy(extra: ParamMap): BisectingKMeansModel = {
val copied = copyValues(new BisectingKMeansModel(uid, parentModel), extra)
-if (trainingSummary.isDefined) copied.setSummary(trainingSummary.get)
-copied.setParent(this.parent)
+copied.setSummary(trainingSummary).setParent(this.parent)
}
@Since("2.0.0")
@@ -132,8 +131,8 @@ class BisectingKMeansModel private[ml] (
private var trainingSummary: Option[BisectingKMeansSummary] = None
- private[clustering] def setSummary(summary: BisectingKMeansSummary):
spark git commit: [SPARK-18282][ML][PYSPARK] Add python clustering summaries for GMM and BKM
Repository: spark
Updated Branches:
refs/heads/branch-2.1 fb4e6359d -> 31002e4a7
[SPARK-18282][ML][PYSPARK] Add python clustering summaries for GMM and BKM
## What changes were proposed in this pull request?
Add model summary APIs for `GaussianMixtureModel` and `BisectingKMeansModel` in
pyspark.
## How was this patch tested?
Unit tests.
Author: sethah
Closes #15777 from sethah/pyspark_cluster_summaries.
(cherry picked from commit e811fbf9ed131bccbc46f3c5701c4ff317222fd9)
Signed-off-by: Yanbo Liang
Project: http://git-wip-us.apache.org/repos/asf/spark/repo
Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/31002e4a
Tree: http://git-wip-us.apache.org/repos/asf/spark/tree/31002e4a
Diff: http://git-wip-us.apache.org/repos/asf/spark/diff/31002e4a
Branch: refs/heads/branch-2.1
Commit: 31002e4a77ca56492f41bf35e7c8f263d767d3aa
Parents: fb4e635
Author: sethah
Authored: Mon Nov 21 05:36:49 2016 -0800
Committer: Yanbo Liang
Committed: Mon Nov 21 05:37:34 2016 -0800
--
.../ml/classification/LogisticRegression.scala | 11 +-
.../spark/ml/clustering/BisectingKMeans.scala | 9 +-
.../spark/ml/clustering/GaussianMixture.scala | 9 +-
.../org/apache/spark/ml/clustering/KMeans.scala | 9 +-
.../GeneralizedLinearRegression.scala | 11 +-
.../spark/ml/regression/LinearRegression.scala | 14 +-
.../LogisticRegressionSuite.scala | 2 +
.../ml/clustering/BisectingKMeansSuite.scala| 3 +
.../ml/clustering/GaussianMixtureSuite.scala| 3 +
.../spark/ml/clustering/KMeansSuite.scala | 3 +
.../GeneralizedLinearRegressionSuite.scala | 2 +
.../ml/regression/LinearRegressionSuite.scala | 2 +
python/pyspark/ml/classification.py | 15 +-
python/pyspark/ml/clustering.py | 162 ++-
python/pyspark/ml/regression.py | 16 +-
python/pyspark/ml/tests.py | 32
16 files changed, 256 insertions(+), 47 deletions(-)
--
http://git-wip-us.apache.org/repos/asf/spark/blob/31002e4a/mllib/src/main/scala/org/apache/spark/ml/classification/LogisticRegression.scala
--
diff --git
a/mllib/src/main/scala/org/apache/spark/ml/classification/LogisticRegression.scala
b/mllib/src/main/scala/org/apache/spark/ml/classification/LogisticRegression.scala
index f58efd3..d07b4ad 100644
---
a/mllib/src/main/scala/org/apache/spark/ml/classification/LogisticRegression.scala
+++
b/mllib/src/main/scala/org/apache/spark/ml/classification/LogisticRegression.scala
@@ -648,7 +648,7 @@ class LogisticRegression @Since("1.2.0") (
$(labelCol),
$(featuresCol),
objectiveHistory)
- model.setSummary(logRegSummary)
+ model.setSummary(Some(logRegSummary))
} else {
model
}
@@ -790,9 +790,9 @@ class LogisticRegressionModel private[spark] (
}
}
- private[classification] def setSummary(
- summary: LogisticRegressionTrainingSummary): this.type = {
-this.trainingSummary = Some(summary)
+ private[classification]
+ def setSummary(summary: Option[LogisticRegressionTrainingSummary]):
this.type = {
+this.trainingSummary = summary
this
}
@@ -887,8 +887,7 @@ class LogisticRegressionModel private[spark] (
override def copy(extra: ParamMap): LogisticRegressionModel = {
val newModel = copyValues(new LogisticRegressionModel(uid,
coefficientMatrix, interceptVector,
numClasses, isMultinomial), extra)
-if (trainingSummary.isDefined) newModel.setSummary(trainingSummary.get)
-newModel.setParent(parent)
+newModel.setSummary(trainingSummary).setParent(parent)
}
override protected def raw2prediction(rawPrediction: Vector): Double = {
http://git-wip-us.apache.org/repos/asf/spark/blob/31002e4a/mllib/src/main/scala/org/apache/spark/ml/clustering/BisectingKMeans.scala
--
diff --git
a/mllib/src/main/scala/org/apache/spark/ml/clustering/BisectingKMeans.scala
b/mllib/src/main/scala/org/apache/spark/ml/clustering/BisectingKMeans.scala
index f8a606d..e6ca3aed 100644
--- a/mllib/src/main/scala/org/apache/spark/ml/clustering/BisectingKMeans.scala
+++ b/mllib/src/main/scala/org/apache/spark/ml/clustering/BisectingKMeans.scala
@@ -95,8 +95,7 @@ class BisectingKMeansModel private[ml] (
@Since("2.0.0")
override def copy(extra: ParamMap): BisectingKMeansModel = {
val copied = copyValues(new BisectingKMeansModel(uid, parentModel), extra)
-if (trainingSummary.isDefined) copied.setSummary(trainingSummary.get)
-copied.setParent(this.parent)
+copied.setSummary(trainingSummary).setParent(this.parent)
}
@Since("2.0.0")
@@ -132,8 +131,8 @@ class BisectingKMeansModel private[ml] (
private var trainingSummary: Option
spark git commit: [SPARK-18444][SPARKR] SparkR running in yarn-cluster mode should not download Spark package.
Repository: spark
Updated Branches:
refs/heads/master ebeb0830a -> acb971577
[SPARK-18444][SPARKR] SparkR running in yarn-cluster mode should not download
Spark package.
## What changes were proposed in this pull request?
When running SparkR job in yarn-cluster mode, it will download Spark package
from apache website which is not necessary.
```
./bin/spark-submit --master yarn-cluster ./examples/src/main/r/dataframe.R
```
The following is output:
```
Attaching package: âSparkRâ
The following objects are masked from âpackage:statsâ:
cov, filter, lag, na.omit, predict, sd, var, window
The following objects are masked from âpackage:baseâ:
as.data.frame, colnames, colnames<-, drop, endsWith, intersect,
rank, rbind, sample, startsWith, subset, summary, transform, union
Spark not found in SPARK_HOME:
Spark not found in the cache directory. Installation will start.
MirrorUrl not provided.
Looking for preferred site from apache website...
..
```
There's no ```SPARK_HOME``` in yarn-cluster mode since the R process is in a
remote host of the yarn cluster rather than in the client host. The JVM comes
up first and the R process then connects to it. So in such cases we should
never have to download Spark as Spark is already running.
## How was this patch tested?
Offline test.
Author: Yanbo Liang
Closes #15888 from yanboliang/spark-18444.
Project: http://git-wip-us.apache.org/repos/asf/spark/repo
Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/acb97157
Tree: http://git-wip-us.apache.org/repos/asf/spark/tree/acb97157
Diff: http://git-wip-us.apache.org/repos/asf/spark/diff/acb97157
Branch: refs/heads/master
Commit: acb97157796231fef74aba985825b05b607b9279
Parents: ebeb083
Author: Yanbo Liang
Authored: Tue Nov 22 00:05:30 2016 -0800
Committer: Yanbo Liang
Committed: Tue Nov 22 00:05:30 2016 -0800
--
R/pkg/R/sparkR.R| 20
R/pkg/R/utils.R | 4 +++
R/pkg/inst/tests/testthat/test_sparkR.R | 46
3 files changed, 64 insertions(+), 6 deletions(-)
--
http://git-wip-us.apache.org/repos/asf/spark/blob/acb97157/R/pkg/R/sparkR.R
--
diff --git a/R/pkg/R/sparkR.R b/R/pkg/R/sparkR.R
index 6b4a2f2..a7152b4 100644
--- a/R/pkg/R/sparkR.R
+++ b/R/pkg/R/sparkR.R
@@ -373,8 +373,13 @@ sparkR.session <- function(
overrideEnvs(sparkConfigMap, paramMap)
}
+ deployMode <- ""
+ if (exists("spark.submit.deployMode", envir = sparkConfigMap)) {
+deployMode <- sparkConfigMap[["spark.submit.deployMode"]]
+ }
+
if (!exists(".sparkRjsc", envir = .sparkREnv)) {
-retHome <- sparkCheckInstall(sparkHome, master)
+retHome <- sparkCheckInstall(sparkHome, master, deployMode)
if (!is.null(retHome)) sparkHome <- retHome
sparkExecutorEnvMap <- new.env()
sparkR.sparkContext(master, appName, sparkHome, sparkConfigMap,
sparkExecutorEnvMap,
@@ -550,24 +555,27 @@ processSparkPackages <- function(packages) {
#
# @param sparkHome directory to find Spark package.
# @param master the Spark master URL, used to check local or remote mode.
+# @param deployMode whether to deploy your driver on the worker nodes (cluster)
+#or locally as an external client (client).
# @return NULL if no need to update sparkHome, and new sparkHome otherwise.
-sparkCheckInstall <- function(sparkHome, master) {
+sparkCheckInstall <- function(sparkHome, master, deployMode) {
if (!isSparkRShell()) {
if (!is.na(file.info(sparkHome)$isdir)) {
msg <- paste0("Spark package found in SPARK_HOME: ", sparkHome)
message(msg)
NULL
} else {
- if (!nzchar(master) || isMasterLocal(master)) {
-msg <- paste0("Spark not found in SPARK_HOME: ",
- sparkHome)
+ if (isMasterLocal(master)) {
+msg <- paste0("Spark not found in SPARK_HOME: ", sparkHome)
message(msg)
packageLocalDir <- install.spark()
packageLocalDir
- } else {
+ } else if (isClientMode(master) || deployMode == "client") {
msg <- paste0("Spark not found in SPARK_HOME: ",
sparkHome, "\n", installInstruction("remote"))
stop(msg)
+ } else {
+NULL
}
}
} else {
http://git-wip-us.apache.org/repos/asf/spark/blob/acb97157/R/pkg/R/utils.R
--
diff --git a/R/pkg/R/utils.R b/R/pkg/R/utils.R
index 2000454..098c0e3 100644
--- a/R/pkg/R/utils.R
+++ b/R/pkg/R/utils.R
@@ -777,6 +777,10 @@ isMasterLocal <- function(master) {
grepl("^local(\\[([0-9]+|\\*)\\])?$", master, perl = TRUE)
}
+isClientMode <- function(master) {
+ grepl("([a-z]+)-client$", master, perl = TRUE)
+}
+
isSparkRShell <- function() {
g
spark git commit: [SPARK-18444][SPARKR] SparkR running in yarn-cluster mode should not download Spark package.
Repository: spark
Updated Branches:
refs/heads/branch-2.1 aaa2a173a -> c70214075
[SPARK-18444][SPARKR] SparkR running in yarn-cluster mode should not download
Spark package.
## What changes were proposed in this pull request?
When running SparkR job in yarn-cluster mode, it will download Spark package
from apache website which is not necessary.
```
./bin/spark-submit --master yarn-cluster ./examples/src/main/r/dataframe.R
```
The following is output:
```
Attaching package: âSparkRâ
The following objects are masked from âpackage:statsâ:
cov, filter, lag, na.omit, predict, sd, var, window
The following objects are masked from âpackage:baseâ:
as.data.frame, colnames, colnames<-, drop, endsWith, intersect,
rank, rbind, sample, startsWith, subset, summary, transform, union
Spark not found in SPARK_HOME:
Spark not found in the cache directory. Installation will start.
MirrorUrl not provided.
Looking for preferred site from apache website...
..
```
There's no ```SPARK_HOME``` in yarn-cluster mode since the R process is in a
remote host of the yarn cluster rather than in the client host. The JVM comes
up first and the R process then connects to it. So in such cases we should
never have to download Spark as Spark is already running.
## How was this patch tested?
Offline test.
Author: Yanbo Liang
Closes #15888 from yanboliang/spark-18444.
(cherry picked from commit acb97157796231fef74aba985825b05b607b9279)
Signed-off-by: Yanbo Liang
Project: http://git-wip-us.apache.org/repos/asf/spark/repo
Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/c7021407
Tree: http://git-wip-us.apache.org/repos/asf/spark/tree/c7021407
Diff: http://git-wip-us.apache.org/repos/asf/spark/diff/c7021407
Branch: refs/heads/branch-2.1
Commit: c7021407597480bddf226ffa6d1d3f682408dfeb
Parents: aaa2a17
Author: Yanbo Liang
Authored: Tue Nov 22 00:05:30 2016 -0800
Committer: Yanbo Liang
Committed: Tue Nov 22 00:05:54 2016 -0800
--
R/pkg/R/sparkR.R| 20
R/pkg/R/utils.R | 4 +++
R/pkg/inst/tests/testthat/test_sparkR.R | 46
3 files changed, 64 insertions(+), 6 deletions(-)
--
http://git-wip-us.apache.org/repos/asf/spark/blob/c7021407/R/pkg/R/sparkR.R
--
diff --git a/R/pkg/R/sparkR.R b/R/pkg/R/sparkR.R
index 6b4a2f2..a7152b4 100644
--- a/R/pkg/R/sparkR.R
+++ b/R/pkg/R/sparkR.R
@@ -373,8 +373,13 @@ sparkR.session <- function(
overrideEnvs(sparkConfigMap, paramMap)
}
+ deployMode <- ""
+ if (exists("spark.submit.deployMode", envir = sparkConfigMap)) {
+deployMode <- sparkConfigMap[["spark.submit.deployMode"]]
+ }
+
if (!exists(".sparkRjsc", envir = .sparkREnv)) {
-retHome <- sparkCheckInstall(sparkHome, master)
+retHome <- sparkCheckInstall(sparkHome, master, deployMode)
if (!is.null(retHome)) sparkHome <- retHome
sparkExecutorEnvMap <- new.env()
sparkR.sparkContext(master, appName, sparkHome, sparkConfigMap,
sparkExecutorEnvMap,
@@ -550,24 +555,27 @@ processSparkPackages <- function(packages) {
#
# @param sparkHome directory to find Spark package.
# @param master the Spark master URL, used to check local or remote mode.
+# @param deployMode whether to deploy your driver on the worker nodes (cluster)
+#or locally as an external client (client).
# @return NULL if no need to update sparkHome, and new sparkHome otherwise.
-sparkCheckInstall <- function(sparkHome, master) {
+sparkCheckInstall <- function(sparkHome, master, deployMode) {
if (!isSparkRShell()) {
if (!is.na(file.info(sparkHome)$isdir)) {
msg <- paste0("Spark package found in SPARK_HOME: ", sparkHome)
message(msg)
NULL
} else {
- if (!nzchar(master) || isMasterLocal(master)) {
-msg <- paste0("Spark not found in SPARK_HOME: ",
- sparkHome)
+ if (isMasterLocal(master)) {
+msg <- paste0("Spark not found in SPARK_HOME: ", sparkHome)
message(msg)
packageLocalDir <- install.spark()
packageLocalDir
- } else {
+ } else if (isClientMode(master) || deployMode == "client") {
msg <- paste0("Spark not found in SPARK_HOME: ",
sparkHome, "\n", installInstruction("remote"))
stop(msg)
+ } else {
+NULL
}
}
} else {
http://git-wip-us.apache.org/repos/asf/spark/blob/c7021407/R/pkg/R/utils.R
--
diff --git a/R/pkg/R/utils.R b/R/pkg/R/utils.R
index 2000454..098c0e3 100644
--- a/R/pkg/R/utils.R
+++ b/R/pkg/R/utils.R
@@ -777,6 +777,10 @@ isMasterLocal <- function(master) {
grepl("^local(\\[([0-9]+|\\*)\\])?$", master, perl = TRUE)
}
+isClientMode <- fun
spark git commit: [SPARK-18444][SPARKR] SparkR running in yarn-cluster mode should not download Spark package.
Repository: spark
Updated Branches:
refs/heads/branch-2.0 9dad3a7b0 -> a37238b06
[SPARK-18444][SPARKR] SparkR running in yarn-cluster mode should not download
Spark package.
## What changes were proposed in this pull request?
When running SparkR job in yarn-cluster mode, it will download Spark package
from apache website which is not necessary.
```
./bin/spark-submit --master yarn-cluster ./examples/src/main/r/dataframe.R
```
The following is output:
```
Attaching package: âSparkRâ
The following objects are masked from âpackage:statsâ:
cov, filter, lag, na.omit, predict, sd, var, window
The following objects are masked from âpackage:baseâ:
as.data.frame, colnames, colnames<-, drop, endsWith, intersect,
rank, rbind, sample, startsWith, subset, summary, transform, union
Spark not found in SPARK_HOME:
Spark not found in the cache directory. Installation will start.
MirrorUrl not provided.
Looking for preferred site from apache website...
..
```
There's no ```SPARK_HOME``` in yarn-cluster mode since the R process is in a
remote host of the yarn cluster rather than in the client host. The JVM comes
up first and the R process then connects to it. So in such cases we should
never have to download Spark as Spark is already running.
## How was this patch tested?
Offline test.
Author: Yanbo Liang
Closes #15888 from yanboliang/spark-18444.
(cherry picked from commit acb97157796231fef74aba985825b05b607b9279)
Signed-off-by: Yanbo Liang
Project: http://git-wip-us.apache.org/repos/asf/spark/repo
Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/a37238b0
Tree: http://git-wip-us.apache.org/repos/asf/spark/tree/a37238b0
Diff: http://git-wip-us.apache.org/repos/asf/spark/diff/a37238b0
Branch: refs/heads/branch-2.0
Commit: a37238b06f525a1e870750650cf1a4f2885ea265
Parents: 9dad3a7
Author: Yanbo Liang
Authored: Tue Nov 22 00:05:30 2016 -0800
Committer: Yanbo Liang
Committed: Tue Nov 22 00:08:51 2016 -0800
--
R/pkg/R/sparkR.R| 20
R/pkg/R/utils.R | 4 +++
R/pkg/inst/tests/testthat/test_sparkR.R | 46
3 files changed, 64 insertions(+), 6 deletions(-)
--
http://git-wip-us.apache.org/repos/asf/spark/blob/a37238b0/R/pkg/R/sparkR.R
--
diff --git a/R/pkg/R/sparkR.R b/R/pkg/R/sparkR.R
index cc6d591..6476693 100644
--- a/R/pkg/R/sparkR.R
+++ b/R/pkg/R/sparkR.R
@@ -369,8 +369,13 @@ sparkR.session <- function(
overrideEnvs(sparkConfigMap, paramMap)
}
+ deployMode <- ""
+ if (exists("spark.submit.deployMode", envir = sparkConfigMap)) {
+deployMode <- sparkConfigMap[["spark.submit.deployMode"]]
+ }
+
if (!exists(".sparkRjsc", envir = .sparkREnv)) {
-retHome <- sparkCheckInstall(sparkHome, master)
+retHome <- sparkCheckInstall(sparkHome, master, deployMode)
if (!is.null(retHome)) sparkHome <- retHome
sparkExecutorEnvMap <- new.env()
sparkR.sparkContext(master, appName, sparkHome, sparkConfigMap,
sparkExecutorEnvMap,
@@ -546,24 +551,27 @@ processSparkPackages <- function(packages) {
#
# @param sparkHome directory to find Spark package.
# @param master the Spark master URL, used to check local or remote mode.
+# @param deployMode whether to deploy your driver on the worker nodes (cluster)
+#or locally as an external client (client).
# @return NULL if no need to update sparkHome, and new sparkHome otherwise.
-sparkCheckInstall <- function(sparkHome, master) {
+sparkCheckInstall <- function(sparkHome, master, deployMode) {
if (!isSparkRShell()) {
if (!is.na(file.info(sparkHome)$isdir)) {
msg <- paste0("Spark package found in SPARK_HOME: ", sparkHome)
message(msg)
NULL
} else {
- if (!nzchar(master) || isMasterLocal(master)) {
-msg <- paste0("Spark not found in SPARK_HOME: ",
- sparkHome)
+ if (isMasterLocal(master)) {
+msg <- paste0("Spark not found in SPARK_HOME: ", sparkHome)
message(msg)
packageLocalDir <- install.spark()
packageLocalDir
- } else {
+ } else if (isClientMode(master) || deployMode == "client") {
msg <- paste0("Spark not found in SPARK_HOME: ",
sparkHome, "\n", installInstruction("remote"))
stop(msg)
+ } else {
+NULL
}
}
} else {
http://git-wip-us.apache.org/repos/asf/spark/blob/a37238b0/R/pkg/R/utils.R
--
diff --git a/R/pkg/R/utils.R b/R/pkg/R/utils.R
index 248c575..581a9a4 100644
--- a/R/pkg/R/utils.R
+++ b/R/pkg/R/utils.R
@@ -694,6 +694,10 @@ isMasterLocal <- function(master) {
grepl("^local(\\[([0-9]+|\\*)\\])?$", master, perl = TRUE)
}
+isClientMode <- fun
spark git commit: [SPARK-18501][ML][SPARKR] Fix spark.glm errors when fitting on collinear data
Repository: spark
Updated Branches:
refs/heads/master d0212eb0f -> 982b82e32
[SPARK-18501][ML][SPARKR] Fix spark.glm errors when fitting on collinear data
## What changes were proposed in this pull request?
* Fix SparkR ```spark.glm``` errors when fitting on collinear data, since
```standard error of coefficients, t value and p value``` are not available in
this condition.
* Scala/Python GLM summary should throw exception if users get ```standard
error of coefficients, t value and p value``` but the underlying WLS was solved
by local "l-bfgs".
## How was this patch tested?
Add unit tests.
Author: Yanbo Liang
Closes #15930 from yanboliang/spark-18501.
Project: http://git-wip-us.apache.org/repos/asf/spark/repo
Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/982b82e3
Tree: http://git-wip-us.apache.org/repos/asf/spark/tree/982b82e3
Diff: http://git-wip-us.apache.org/repos/asf/spark/diff/982b82e3
Branch: refs/heads/master
Commit: 982b82e32e0fc7d30c5d557944a79eb3e6d2da59
Parents: d0212eb
Author: Yanbo Liang
Authored: Tue Nov 22 19:17:48 2016 -0800
Committer: Yanbo Liang
Committed: Tue Nov 22 19:17:48 2016 -0800
--
R/pkg/R/mllib.R | 21 ++--
R/pkg/inst/tests/testthat/test_mllib.R | 9
.../r/GeneralizedLinearRegressionWrapper.scala | 54 +++-
.../GeneralizedLinearRegression.scala | 46 ++---
.../GeneralizedLinearRegressionSuite.scala | 21
5 files changed, 115 insertions(+), 36 deletions(-)
--
http://git-wip-us.apache.org/repos/asf/spark/blob/982b82e3/R/pkg/R/mllib.R
--
diff --git a/R/pkg/R/mllib.R b/R/pkg/R/mllib.R
index 265e64e..02bc645 100644
--- a/R/pkg/R/mllib.R
+++ b/R/pkg/R/mllib.R
@@ -278,8 +278,10 @@ setMethod("glm", signature(formula = "formula", family =
"ANY", data = "SparkDat
#' @param object a fitted generalized linear model.
#' @return \code{summary} returns a summary object of the fitted model, a list
of components
-#' including at least the coefficients, null/residual deviance,
null/residual degrees
-#' of freedom, AIC and number of iterations IRLS takes.
+#' including at least the coefficients matrix (which includes
coefficients, standard error
+#' of coefficients, t value and p value), null/residual deviance,
null/residual degrees of
+#' freedom, AIC and number of iterations IRLS takes. If there are
collinear columns
+#' in you data, the coefficients matrix only provides coefficients.
#'
#' @rdname spark.glm
#' @export
@@ -303,9 +305,18 @@ setMethod("summary", signature(object =
"GeneralizedLinearRegressionModel"),
} else {
dataFrame(callJMethod(jobj, "rDevianceResiduals"))
}
-coefficients <- matrix(coefficients, ncol = 4)
-colnames(coefficients) <- c("Estimate", "Std. Error", "t value",
"Pr(>|t|)")
-rownames(coefficients) <- unlist(features)
+# If the underlying WeightedLeastSquares using "normal" solver, we
can provide
+# coefficients, standard error of coefficients, t value and p
value. Otherwise,
+# it will be fitted by local "l-bfgs", we can only provide
coefficients.
+if (length(features) == length(coefficients)) {
+ coefficients <- matrix(coefficients, ncol = 1)
+ colnames(coefficients) <- c("Estimate")
+ rownames(coefficients) <- unlist(features)
+} else {
+ coefficients <- matrix(coefficients, ncol = 4)
+ colnames(coefficients) <- c("Estimate", "Std. Error", "t value",
"Pr(>|t|)")
+ rownames(coefficients) <- unlist(features)
+}
ans <- list(deviance.resid = deviance.resid, coefficients =
coefficients,
dispersion = dispersion, null.deviance = null.deviance,
deviance = deviance, df.null = df.null, df.residual =
df.residual,
http://git-wip-us.apache.org/repos/asf/spark/blob/982b82e3/R/pkg/inst/tests/testthat/test_mllib.R
--
diff --git a/R/pkg/inst/tests/testthat/test_mllib.R
b/R/pkg/inst/tests/testthat/test_mllib.R
index 2a97a51..467e00c 100644
--- a/R/pkg/inst/tests/testthat/test_mllib.R
+++ b/R/pkg/inst/tests/testthat/test_mllib.R
@@ -169,6 +169,15 @@ test_that("spark.glm summary", {
df <- suppressWarnings(createDataFrame(data))
regStats <- summary(spark.glm(df, b ~ a1 + a2, regParam = 1.0))
expect_equal(regStats$aic, 14.00976, tolerance = 1e-4) # 14.00976 is from
summary() result
+
+ # Test spark.glm works on collinear data
+ A <- matrix(c(1, 2, 3, 4, 2, 4, 6, 8), 4, 2)
+ b <- c(1, 2, 3, 4)
+ data <- as.data.frame(cbi
spark git commit: [SPARK-18501][ML][SPARKR] Fix spark.glm errors when fitting on collinear data
Repository: spark
Updated Branches:
refs/heads/branch-2.1 3be2d1e0b -> fc5fee83e
[SPARK-18501][ML][SPARKR] Fix spark.glm errors when fitting on collinear data
## What changes were proposed in this pull request?
* Fix SparkR ```spark.glm``` errors when fitting on collinear data, since
```standard error of coefficients, t value and p value``` are not available in
this condition.
* Scala/Python GLM summary should throw exception if users get ```standard
error of coefficients, t value and p value``` but the underlying WLS was solved
by local "l-bfgs".
## How was this patch tested?
Add unit tests.
Author: Yanbo Liang
Closes #15930 from yanboliang/spark-18501.
(cherry picked from commit 982b82e32e0fc7d30c5d557944a79eb3e6d2da59)
Signed-off-by: Yanbo Liang
Project: http://git-wip-us.apache.org/repos/asf/spark/repo
Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/fc5fee83
Tree: http://git-wip-us.apache.org/repos/asf/spark/tree/fc5fee83
Diff: http://git-wip-us.apache.org/repos/asf/spark/diff/fc5fee83
Branch: refs/heads/branch-2.1
Commit: fc5fee83e363bc6df22459a9b1ba2ba11bfdfa20
Parents: 3be2d1e
Author: Yanbo Liang
Authored: Tue Nov 22 19:17:48 2016 -0800
Committer: Yanbo Liang
Committed: Tue Nov 22 19:18:30 2016 -0800
--
R/pkg/R/mllib.R | 21 ++--
R/pkg/inst/tests/testthat/test_mllib.R | 9
.../r/GeneralizedLinearRegressionWrapper.scala | 54 +++-
.../GeneralizedLinearRegression.scala | 46 ++---
.../GeneralizedLinearRegressionSuite.scala | 21
5 files changed, 115 insertions(+), 36 deletions(-)
--
http://git-wip-us.apache.org/repos/asf/spark/blob/fc5fee83/R/pkg/R/mllib.R
--
diff --git a/R/pkg/R/mllib.R b/R/pkg/R/mllib.R
index 265e64e..02bc645 100644
--- a/R/pkg/R/mllib.R
+++ b/R/pkg/R/mllib.R
@@ -278,8 +278,10 @@ setMethod("glm", signature(formula = "formula", family =
"ANY", data = "SparkDat
#' @param object a fitted generalized linear model.
#' @return \code{summary} returns a summary object of the fitted model, a list
of components
-#' including at least the coefficients, null/residual deviance,
null/residual degrees
-#' of freedom, AIC and number of iterations IRLS takes.
+#' including at least the coefficients matrix (which includes
coefficients, standard error
+#' of coefficients, t value and p value), null/residual deviance,
null/residual degrees of
+#' freedom, AIC and number of iterations IRLS takes. If there are
collinear columns
+#' in you data, the coefficients matrix only provides coefficients.
#'
#' @rdname spark.glm
#' @export
@@ -303,9 +305,18 @@ setMethod("summary", signature(object =
"GeneralizedLinearRegressionModel"),
} else {
dataFrame(callJMethod(jobj, "rDevianceResiduals"))
}
-coefficients <- matrix(coefficients, ncol = 4)
-colnames(coefficients) <- c("Estimate", "Std. Error", "t value",
"Pr(>|t|)")
-rownames(coefficients) <- unlist(features)
+# If the underlying WeightedLeastSquares using "normal" solver, we
can provide
+# coefficients, standard error of coefficients, t value and p
value. Otherwise,
+# it will be fitted by local "l-bfgs", we can only provide
coefficients.
+if (length(features) == length(coefficients)) {
+ coefficients <- matrix(coefficients, ncol = 1)
+ colnames(coefficients) <- c("Estimate")
+ rownames(coefficients) <- unlist(features)
+} else {
+ coefficients <- matrix(coefficients, ncol = 4)
+ colnames(coefficients) <- c("Estimate", "Std. Error", "t value",
"Pr(>|t|)")
+ rownames(coefficients) <- unlist(features)
+}
ans <- list(deviance.resid = deviance.resid, coefficients =
coefficients,
dispersion = dispersion, null.deviance = null.deviance,
deviance = deviance, df.null = df.null, df.residual =
df.residual,
http://git-wip-us.apache.org/repos/asf/spark/blob/fc5fee83/R/pkg/inst/tests/testthat/test_mllib.R
--
diff --git a/R/pkg/inst/tests/testthat/test_mllib.R
b/R/pkg/inst/tests/testthat/test_mllib.R
index 70a033d..b05be47 100644
--- a/R/pkg/inst/tests/testthat/test_mllib.R
+++ b/R/pkg/inst/tests/testthat/test_mllib.R
@@ -169,6 +169,15 @@ test_that("spark.glm summary", {
df <- suppressWarnings(createDataFrame(data))
regStats <- summary(spark.glm(df, b ~ a1 + a2, regParam = 1.0))
expect_equal(regStats$aic, 14.00976, tolerance = 1e-4) # 14.00976 is from
summary() result
+
+ # Test spark.glm works on colline
spark git commit: [SPARK-18520][ML] Add missing setXXXCol methods for BisectingKMeansModel and GaussianMixtureModel
Repository: spark
Updated Branches:
refs/heads/master 223fa218e -> 2dfabec38
[SPARK-18520][ML] Add missing setXXXCol methods for BisectingKMeansModel and
GaussianMixtureModel
## What changes were proposed in this pull request?
add `setFeaturesCol` and `setPredictionCol` for BiKModel and GMModel
add `setProbabilityCol` for GMModel
## How was this patch tested?
existing tests
Author: Zheng RuiFeng
Closes #15957 from zhengruifeng/bikm_set.
Project: http://git-wip-us.apache.org/repos/asf/spark/repo
Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/2dfabec3
Tree: http://git-wip-us.apache.org/repos/asf/spark/tree/2dfabec3
Diff: http://git-wip-us.apache.org/repos/asf/spark/diff/2dfabec3
Branch: refs/heads/master
Commit: 2dfabec38c24174e7f747c27c7144f7738483ec1
Parents: 223fa21
Author: Zheng RuiFeng
Authored: Thu Nov 24 05:46:05 2016 -0800
Committer: Yanbo Liang
Committed: Thu Nov 24 05:46:05 2016 -0800
--
.../apache/spark/ml/clustering/BisectingKMeans.scala| 8
.../apache/spark/ml/clustering/GaussianMixture.scala| 12
2 files changed, 20 insertions(+)
--
http://git-wip-us.apache.org/repos/asf/spark/blob/2dfabec3/mllib/src/main/scala/org/apache/spark/ml/clustering/BisectingKMeans.scala
--
diff --git
a/mllib/src/main/scala/org/apache/spark/ml/clustering/BisectingKMeans.scala
b/mllib/src/main/scala/org/apache/spark/ml/clustering/BisectingKMeans.scala
index e6ca3aed..cf11ba3 100644
--- a/mllib/src/main/scala/org/apache/spark/ml/clustering/BisectingKMeans.scala
+++ b/mllib/src/main/scala/org/apache/spark/ml/clustering/BisectingKMeans.scala
@@ -98,6 +98,14 @@ class BisectingKMeansModel private[ml] (
copied.setSummary(trainingSummary).setParent(this.parent)
}
+ /** @group setParam */
+ @Since("2.1.0")
+ def setFeaturesCol(value: String): this.type = set(featuresCol, value)
+
+ /** @group setParam */
+ @Since("2.1.0")
+ def setPredictionCol(value: String): this.type = set(predictionCol, value)
+
@Since("2.0.0")
override def transform(dataset: Dataset[_]): DataFrame = {
transformSchema(dataset.schema, logging = true)
http://git-wip-us.apache.org/repos/asf/spark/blob/2dfabec3/mllib/src/main/scala/org/apache/spark/ml/clustering/GaussianMixture.scala
--
diff --git
a/mllib/src/main/scala/org/apache/spark/ml/clustering/GaussianMixture.scala
b/mllib/src/main/scala/org/apache/spark/ml/clustering/GaussianMixture.scala
index 92d0b7d..19998ca 100644
--- a/mllib/src/main/scala/org/apache/spark/ml/clustering/GaussianMixture.scala
+++ b/mllib/src/main/scala/org/apache/spark/ml/clustering/GaussianMixture.scala
@@ -87,6 +87,18 @@ class GaussianMixtureModel private[ml] (
@Since("2.0.0") val gaussians: Array[MultivariateGaussian])
extends Model[GaussianMixtureModel] with GaussianMixtureParams with
MLWritable {
+ /** @group setParam */
+ @Since("2.1.0")
+ def setFeaturesCol(value: String): this.type = set(featuresCol, value)
+
+ /** @group setParam */
+ @Since("2.1.0")
+ def setPredictionCol(value: String): this.type = set(predictionCol, value)
+
+ /** @group setParam */
+ @Since("2.1.0")
+ def setProbabilityCol(value: String): this.type = set(probabilityCol, value)
+
@Since("2.0.0")
override def copy(extra: ParamMap): GaussianMixtureModel = {
val copied = copyValues(new GaussianMixtureModel(uid, weights, gaussians),
extra)
-
To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org
For additional commands, e-mail: commits-h...@spark.apache.org
spark git commit: [SPARK-18520][ML] Add missing setXXXCol methods for BisectingKMeansModel and GaussianMixtureModel
Repository: spark
Updated Branches:
refs/heads/branch-2.1 27d81d000 -> 04ec74f12
[SPARK-18520][ML] Add missing setXXXCol methods for BisectingKMeansModel and
GaussianMixtureModel
## What changes were proposed in this pull request?
add `setFeaturesCol` and `setPredictionCol` for BiKModel and GMModel
add `setProbabilityCol` for GMModel
## How was this patch tested?
existing tests
Author: Zheng RuiFeng
Closes #15957 from zhengruifeng/bikm_set.
(cherry picked from commit 2dfabec38c24174e7f747c27c7144f7738483ec1)
Signed-off-by: Yanbo Liang
Project: http://git-wip-us.apache.org/repos/asf/spark/repo
Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/04ec74f1
Tree: http://git-wip-us.apache.org/repos/asf/spark/tree/04ec74f1
Diff: http://git-wip-us.apache.org/repos/asf/spark/diff/04ec74f1
Branch: refs/heads/branch-2.1
Commit: 04ec74f1274a164b2f72b31e2c147e042bf41bd9
Parents: 27d81d0
Author: Zheng RuiFeng
Authored: Thu Nov 24 05:46:05 2016 -0800
Committer: Yanbo Liang
Committed: Thu Nov 24 05:47:02 2016 -0800
--
.../apache/spark/ml/clustering/BisectingKMeans.scala| 8
.../apache/spark/ml/clustering/GaussianMixture.scala| 12
2 files changed, 20 insertions(+)
--
http://git-wip-us.apache.org/repos/asf/spark/blob/04ec74f1/mllib/src/main/scala/org/apache/spark/ml/clustering/BisectingKMeans.scala
--
diff --git
a/mllib/src/main/scala/org/apache/spark/ml/clustering/BisectingKMeans.scala
b/mllib/src/main/scala/org/apache/spark/ml/clustering/BisectingKMeans.scala
index e6ca3aed..cf11ba3 100644
--- a/mllib/src/main/scala/org/apache/spark/ml/clustering/BisectingKMeans.scala
+++ b/mllib/src/main/scala/org/apache/spark/ml/clustering/BisectingKMeans.scala
@@ -98,6 +98,14 @@ class BisectingKMeansModel private[ml] (
copied.setSummary(trainingSummary).setParent(this.parent)
}
+ /** @group setParam */
+ @Since("2.1.0")
+ def setFeaturesCol(value: String): this.type = set(featuresCol, value)
+
+ /** @group setParam */
+ @Since("2.1.0")
+ def setPredictionCol(value: String): this.type = set(predictionCol, value)
+
@Since("2.0.0")
override def transform(dataset: Dataset[_]): DataFrame = {
transformSchema(dataset.schema, logging = true)
http://git-wip-us.apache.org/repos/asf/spark/blob/04ec74f1/mllib/src/main/scala/org/apache/spark/ml/clustering/GaussianMixture.scala
--
diff --git
a/mllib/src/main/scala/org/apache/spark/ml/clustering/GaussianMixture.scala
b/mllib/src/main/scala/org/apache/spark/ml/clustering/GaussianMixture.scala
index 92d0b7d..19998ca 100644
--- a/mllib/src/main/scala/org/apache/spark/ml/clustering/GaussianMixture.scala
+++ b/mllib/src/main/scala/org/apache/spark/ml/clustering/GaussianMixture.scala
@@ -87,6 +87,18 @@ class GaussianMixtureModel private[ml] (
@Since("2.0.0") val gaussians: Array[MultivariateGaussian])
extends Model[GaussianMixtureModel] with GaussianMixtureParams with
MLWritable {
+ /** @group setParam */
+ @Since("2.1.0")
+ def setFeaturesCol(value: String): this.type = set(featuresCol, value)
+
+ /** @group setParam */
+ @Since("2.1.0")
+ def setPredictionCol(value: String): this.type = set(predictionCol, value)
+
+ /** @group setParam */
+ @Since("2.1.0")
+ def setProbabilityCol(value: String): this.type = set(probabilityCol, value)
+
@Since("2.0.0")
override def copy(extra: ParamMap): GaussianMixtureModel = {
val copied = copyValues(new GaussianMixtureModel(uid, weights, gaussians),
extra)
-
To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org
For additional commands, e-mail: commits-h...@spark.apache.org
spark git commit: [SPARK-18481][ML] ML 2.1 QA: Remove deprecated methods for ML
Repository: spark
Updated Branches:
refs/heads/master a88329d45 -> c4a7eef0c
[SPARK-18481][ML] ML 2.1 QA: Remove deprecated methods for ML
## What changes were proposed in this pull request?
Remove deprecated methods for ML.
## How was this patch tested?
Existing tests.
Author: Yanbo Liang
Closes #15913 from yanboliang/spark-18481.
Project: http://git-wip-us.apache.org/repos/asf/spark/repo
Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/c4a7eef0
Tree: http://git-wip-us.apache.org/repos/asf/spark/tree/c4a7eef0
Diff: http://git-wip-us.apache.org/repos/asf/spark/diff/c4a7eef0
Branch: refs/heads/master
Commit: c4a7eef0ce2d305c5c90a0a9a73b5a32eccfba95
Parents: a88329d
Author: Yanbo Liang
Authored: Sat Nov 26 05:28:41 2016 -0800
Committer: Yanbo Liang
Committed: Sat Nov 26 05:28:41 2016 -0800
--
.../scala/org/apache/spark/ml/Pipeline.scala| 4 +
.../spark/ml/classification/GBTClassifier.scala | 6 ++
.../ml/classification/LogisticRegression.scala | 8 +-
.../classification/RandomForestClassifier.scala | 11 +--
.../apache/spark/ml/feature/ChiSqSelector.scala | 7 --
.../org/apache/spark/ml/param/params.scala | 15
.../spark/ml/regression/GBTRegressor.scala | 6 ++
.../spark/ml/regression/LinearRegression.scala | 3 -
.../ml/regression/RandomForestRegressor.scala | 10 +--
.../org/apache/spark/ml/tree/treeModels.scala | 5 --
.../org/apache/spark/ml/tree/treeParams.scala | 90 +---
.../org/apache/spark/ml/util/ReadWrite.scala| 2 +-
.../ml/classification/GBTClassifierSuite.scala | 8 ++
.../LogisticRegressionSuite.scala | 6 ++
project/MimaExcludes.scala | 30 +++
python/pyspark/ml/util.py | 40 -
16 files changed, 144 insertions(+), 107 deletions(-)
--
http://git-wip-us.apache.org/repos/asf/spark/blob/c4a7eef0/mllib/src/main/scala/org/apache/spark/ml/Pipeline.scala
--
diff --git a/mllib/src/main/scala/org/apache/spark/ml/Pipeline.scala
b/mllib/src/main/scala/org/apache/spark/ml/Pipeline.scala
index f406f8c..38176b9 100644
--- a/mllib/src/main/scala/org/apache/spark/ml/Pipeline.scala
+++ b/mllib/src/main/scala/org/apache/spark/ml/Pipeline.scala
@@ -46,6 +46,10 @@ abstract class PipelineStage extends Params with Logging {
*
* Check transform validity and derive the output schema from the input
schema.
*
+ * We check validity for interactions between parameters during
`transformSchema` and
+ * raise an exception if any parameter value is invalid. Parameter value
checks which
+ * do not depend on other parameters are handled by `Param.validate()`.
+ *
* Typical implementation should first conduct verification on schema change
and parameter
* validity, including complex parameter interaction checks.
*/
http://git-wip-us.apache.org/repos/asf/spark/blob/c4a7eef0/mllib/src/main/scala/org/apache/spark/ml/classification/GBTClassifier.scala
--
diff --git
a/mllib/src/main/scala/org/apache/spark/ml/classification/GBTClassifier.scala
b/mllib/src/main/scala/org/apache/spark/ml/classification/GBTClassifier.scala
index 52f93f5..ca52231 100644
---
a/mllib/src/main/scala/org/apache/spark/ml/classification/GBTClassifier.scala
+++
b/mllib/src/main/scala/org/apache/spark/ml/classification/GBTClassifier.scala
@@ -203,6 +203,12 @@ class GBTClassificationModel private[ml](
@Since("1.4.0")
override def trees: Array[DecisionTreeRegressionModel] = _trees
+ /**
+ * Number of trees in ensemble
+ */
+ @Since("2.0.0")
+ val getNumTrees: Int = trees.length
+
@Since("1.4.0")
override def treeWeights: Array[Double] = _treeWeights
http://git-wip-us.apache.org/repos/asf/spark/blob/c4a7eef0/mllib/src/main/scala/org/apache/spark/ml/classification/LogisticRegression.scala
--
diff --git
a/mllib/src/main/scala/org/apache/spark/ml/classification/LogisticRegression.scala
b/mllib/src/main/scala/org/apache/spark/ml/classification/LogisticRegression.scala
index fe29926..41b84f4 100644
---
a/mllib/src/main/scala/org/apache/spark/ml/classification/LogisticRegression.scala
+++
b/mllib/src/main/scala/org/apache/spark/ml/classification/LogisticRegression.scala
@@ -40,7 +40,7 @@ import org.apache.spark.mllib.util.MLUtils
import org.apache.spark.rdd.RDD
import org.apache.spark.sql.{DataFrame, Dataset, Row}
import org.apache.spark.sql.functions.{col, lit}
-import org.apache.spark.sql.types.DoubleType
+import org.apache.spark.sql.types.{DataType, DoubleType, StructType}
import org.apache.spark.storage.StorageLevel
import org.apache.spark.util.VersionUtils
@@ -176,8 +176,12 @@ private[classification] tra
spark git commit: [SPARK-18481][ML] ML 2.1 QA: Remove deprecated methods for ML
Repository: spark
Updated Branches:
refs/heads/branch-2.1 da66b9742 -> 830ee1345
[SPARK-18481][ML] ML 2.1 QA: Remove deprecated methods for ML
## What changes were proposed in this pull request?
Remove deprecated methods for ML.
## How was this patch tested?
Existing tests.
Author: Yanbo Liang
Closes #15913 from yanboliang/spark-18481.
(cherry picked from commit c4a7eef0ce2d305c5c90a0a9a73b5a32eccfba95)
Signed-off-by: Yanbo Liang
Project: http://git-wip-us.apache.org/repos/asf/spark/repo
Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/830ee134
Tree: http://git-wip-us.apache.org/repos/asf/spark/tree/830ee134
Diff: http://git-wip-us.apache.org/repos/asf/spark/diff/830ee134
Branch: refs/heads/branch-2.1
Commit: 830ee1345b491bf10fd089d931ef22e28f98e615
Parents: da66b97
Author: Yanbo Liang
Authored: Sat Nov 26 05:28:41 2016 -0800
Committer: Yanbo Liang
Committed: Sat Nov 26 05:29:32 2016 -0800
--
.../scala/org/apache/spark/ml/Pipeline.scala| 4 +
.../spark/ml/classification/GBTClassifier.scala | 6 ++
.../ml/classification/LogisticRegression.scala | 8 +-
.../classification/RandomForestClassifier.scala | 11 +--
.../apache/spark/ml/feature/ChiSqSelector.scala | 7 --
.../org/apache/spark/ml/param/params.scala | 15
.../spark/ml/regression/GBTRegressor.scala | 6 ++
.../spark/ml/regression/LinearRegression.scala | 3 -
.../ml/regression/RandomForestRegressor.scala | 10 +--
.../org/apache/spark/ml/tree/treeModels.scala | 5 --
.../org/apache/spark/ml/tree/treeParams.scala | 90 +---
.../org/apache/spark/ml/util/ReadWrite.scala| 2 +-
.../ml/classification/GBTClassifierSuite.scala | 8 ++
.../LogisticRegressionSuite.scala | 6 ++
project/MimaExcludes.scala | 30 +++
python/pyspark/ml/util.py | 40 -
16 files changed, 144 insertions(+), 107 deletions(-)
--
http://git-wip-us.apache.org/repos/asf/spark/blob/830ee134/mllib/src/main/scala/org/apache/spark/ml/Pipeline.scala
--
diff --git a/mllib/src/main/scala/org/apache/spark/ml/Pipeline.scala
b/mllib/src/main/scala/org/apache/spark/ml/Pipeline.scala
index f406f8c..38176b9 100644
--- a/mllib/src/main/scala/org/apache/spark/ml/Pipeline.scala
+++ b/mllib/src/main/scala/org/apache/spark/ml/Pipeline.scala
@@ -46,6 +46,10 @@ abstract class PipelineStage extends Params with Logging {
*
* Check transform validity and derive the output schema from the input
schema.
*
+ * We check validity for interactions between parameters during
`transformSchema` and
+ * raise an exception if any parameter value is invalid. Parameter value
checks which
+ * do not depend on other parameters are handled by `Param.validate()`.
+ *
* Typical implementation should first conduct verification on schema change
and parameter
* validity, including complex parameter interaction checks.
*/
http://git-wip-us.apache.org/repos/asf/spark/blob/830ee134/mllib/src/main/scala/org/apache/spark/ml/classification/GBTClassifier.scala
--
diff --git
a/mllib/src/main/scala/org/apache/spark/ml/classification/GBTClassifier.scala
b/mllib/src/main/scala/org/apache/spark/ml/classification/GBTClassifier.scala
index 52f93f5..ca52231 100644
---
a/mllib/src/main/scala/org/apache/spark/ml/classification/GBTClassifier.scala
+++
b/mllib/src/main/scala/org/apache/spark/ml/classification/GBTClassifier.scala
@@ -203,6 +203,12 @@ class GBTClassificationModel private[ml](
@Since("1.4.0")
override def trees: Array[DecisionTreeRegressionModel] = _trees
+ /**
+ * Number of trees in ensemble
+ */
+ @Since("2.0.0")
+ val getNumTrees: Int = trees.length
+
@Since("1.4.0")
override def treeWeights: Array[Double] = _treeWeights
http://git-wip-us.apache.org/repos/asf/spark/blob/830ee134/mllib/src/main/scala/org/apache/spark/ml/classification/LogisticRegression.scala
--
diff --git
a/mllib/src/main/scala/org/apache/spark/ml/classification/LogisticRegression.scala
b/mllib/src/main/scala/org/apache/spark/ml/classification/LogisticRegression.scala
index fe29926..41b84f4 100644
---
a/mllib/src/main/scala/org/apache/spark/ml/classification/LogisticRegression.scala
+++
b/mllib/src/main/scala/org/apache/spark/ml/classification/LogisticRegression.scala
@@ -40,7 +40,7 @@ import org.apache.spark.mllib.util.MLUtils
import org.apache.spark.rdd.RDD
import org.apache.spark.sql.{DataFrame, Dataset, Row}
import org.apache.spark.sql.functions.{col, lit}
-import org.apache.spark.sql.types.DoubleType
+import org.apache.spark.sql.types.{DataType, DoubleType, StructType}
import org.apache.spark.storage.
spark git commit: [SPARK-15819][PYSPARK][ML] Add KMeanSummary in KMeans of PySpark
Repository: spark
Updated Branches:
refs/heads/master 489845f3a -> 4c82ca86d
[SPARK-15819][PYSPARK][ML] Add KMeanSummary in KMeans of PySpark
## What changes were proposed in this pull request?
Add python api for KMeansSummary
## How was this patch tested?
unit test added
Author: Jeff Zhang
Closes #13557 from zjffdu/SPARK-15819.
Project: http://git-wip-us.apache.org/repos/asf/spark/repo
Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/4c82ca86
Tree: http://git-wip-us.apache.org/repos/asf/spark/tree/4c82ca86
Diff: http://git-wip-us.apache.org/repos/asf/spark/diff/4c82ca86
Branch: refs/heads/master
Commit: 4c82ca86d979e5526a1583eef3c79c37dc68
Parents: 489845f
Author: Jeff Zhang
Authored: Tue Nov 29 20:51:27 2016 -0800
Committer: Yanbo Liang
Committed: Tue Nov 29 20:51:27 2016 -0800
--
python/pyspark/ml/clustering.py | 41
python/pyspark/ml/tests.py | 15 +
2 files changed, 56 insertions(+)
--
http://git-wip-us.apache.org/repos/asf/spark/blob/4c82ca86/python/pyspark/ml/clustering.py
--
diff --git a/python/pyspark/ml/clustering.py b/python/pyspark/ml/clustering.py
index 7f8d845..35d0aef 100644
--- a/python/pyspark/ml/clustering.py
+++ b/python/pyspark/ml/clustering.py
@@ -292,6 +292,17 @@ class GaussianMixtureSummary(ClusteringSummary):
return self._call_java("probability")
+class KMeansSummary(ClusteringSummary):
+"""
+.. note:: Experimental
+
+Summary of KMeans.
+
+.. versionadded:: 2.1.0
+"""
+pass
+
+
class KMeansModel(JavaModel, JavaMLWritable, JavaMLReadable):
"""
Model fitted by KMeans.
@@ -312,6 +323,27 @@ class KMeansModel(JavaModel, JavaMLWritable,
JavaMLReadable):
"""
return self._call_java("computeCost", dataset)
+@property
+@since("2.1.0")
+def hasSummary(self):
+"""
+Indicates whether a training summary exists for this model instance.
+"""
+return self._call_java("hasSummary")
+
+@property
+@since("2.1.0")
+def summary(self):
+"""
+Gets summary (e.g. cluster assignments, cluster sizes) of the model
trained on the
+training set. An exception is thrown if no summary exists.
+"""
+if self.hasSummary:
+return KMeansSummary(self._call_java("summary"))
+else:
+raise RuntimeError("No training summary available for this %s" %
+ self.__class__.__name__)
+
@inherit_doc
class KMeans(JavaEstimator, HasFeaturesCol, HasPredictionCol, HasMaxIter,
HasTol, HasSeed,
@@ -337,6 +369,13 @@ class KMeans(JavaEstimator, HasFeaturesCol,
HasPredictionCol, HasMaxIter, HasTol
True
>>> rows[2].prediction == rows[3].prediction
True
+>>> model.hasSummary
+True
+>>> summary = model.summary
+>>> summary.k
+2
+>>> summary.clusterSizes
+[2, 2]
>>> kmeans_path = temp_path + "/kmeans"
>>> kmeans.save(kmeans_path)
>>> kmeans2 = KMeans.load(kmeans_path)
@@ -345,6 +384,8 @@ class KMeans(JavaEstimator, HasFeaturesCol,
HasPredictionCol, HasMaxIter, HasTol
>>> model_path = temp_path + "/kmeans_model"
>>> model.save(model_path)
>>> model2 = KMeansModel.load(model_path)
+>>> model2.hasSummary
+False
>>> model.clusterCenters()[0] == model2.clusterCenters()[0]
array([ True, True], dtype=bool)
>>> model.clusterCenters()[1] == model2.clusterCenters()[1]
http://git-wip-us.apache.org/repos/asf/spark/blob/4c82ca86/python/pyspark/ml/tests.py
--
diff --git a/python/pyspark/ml/tests.py b/python/pyspark/ml/tests.py
index c0f0d40..a0c288a 100755
--- a/python/pyspark/ml/tests.py
+++ b/python/pyspark/ml/tests.py
@@ -1129,6 +1129,21 @@ class TrainingSummaryTest(SparkSessionTestCase):
self.assertEqual(len(s.clusterSizes), 2)
self.assertEqual(s.k, 2)
+def test_kmeans_summary(self):
+data = [(Vectors.dense([0.0, 0.0]),), (Vectors.dense([1.0, 1.0]),),
+(Vectors.dense([9.0, 8.0]),), (Vectors.dense([8.0, 9.0]),)]
+df = self.spark.createDataFrame(data, ["features"])
+kmeans = KMeans(k=2, seed=1)
+model = kmeans.fit(df)
+self.assertTrue(model.hasSummary)
+s = model.summary
+self.assertTrue(isinstance(s.predictions, DataFrame))
+self.assertEqual(s.featuresCol, "features")
+self.assertEqual(s.predictionCol, "prediction")
+self.assertTrue(isinstance(s.cluster, DataFrame))
+self.assertEqual(len(s.clusterSizes), 2)
+self.assertEqual(s.k, 2)
+
class OneVsRestTests(SparkSessionTestCase):
---
spark git commit: [SPARK-15819][PYSPARK][ML] Add KMeanSummary in KMeans of PySpark
Repository: spark
Updated Branches:
refs/heads/branch-2.1 55b1142bd -> b95aad7ca
[SPARK-15819][PYSPARK][ML] Add KMeanSummary in KMeans of PySpark
## What changes were proposed in this pull request?
Add python api for KMeansSummary
## How was this patch tested?
unit test added
Author: Jeff Zhang
Closes #13557 from zjffdu/SPARK-15819.
(cherry picked from commit 4c82ca86d979e5526a1583eef3c79c37dc68)
Signed-off-by: Yanbo Liang
Project: http://git-wip-us.apache.org/repos/asf/spark/repo
Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/b95aad7c
Tree: http://git-wip-us.apache.org/repos/asf/spark/tree/b95aad7c
Diff: http://git-wip-us.apache.org/repos/asf/spark/diff/b95aad7c
Branch: refs/heads/branch-2.1
Commit: b95aad7cad99a62851fe5e61692fda9bceb4b160
Parents: 55b1142
Author: Jeff Zhang
Authored: Tue Nov 29 20:51:27 2016 -0800
Committer: Yanbo Liang
Committed: Tue Nov 29 20:52:21 2016 -0800
--
python/pyspark/ml/clustering.py | 41
python/pyspark/ml/tests.py | 15 +
2 files changed, 56 insertions(+)
--
http://git-wip-us.apache.org/repos/asf/spark/blob/b95aad7c/python/pyspark/ml/clustering.py
--
diff --git a/python/pyspark/ml/clustering.py b/python/pyspark/ml/clustering.py
index 7f8d845..35d0aef 100644
--- a/python/pyspark/ml/clustering.py
+++ b/python/pyspark/ml/clustering.py
@@ -292,6 +292,17 @@ class GaussianMixtureSummary(ClusteringSummary):
return self._call_java("probability")
+class KMeansSummary(ClusteringSummary):
+"""
+.. note:: Experimental
+
+Summary of KMeans.
+
+.. versionadded:: 2.1.0
+"""
+pass
+
+
class KMeansModel(JavaModel, JavaMLWritable, JavaMLReadable):
"""
Model fitted by KMeans.
@@ -312,6 +323,27 @@ class KMeansModel(JavaModel, JavaMLWritable,
JavaMLReadable):
"""
return self._call_java("computeCost", dataset)
+@property
+@since("2.1.0")
+def hasSummary(self):
+"""
+Indicates whether a training summary exists for this model instance.
+"""
+return self._call_java("hasSummary")
+
+@property
+@since("2.1.0")
+def summary(self):
+"""
+Gets summary (e.g. cluster assignments, cluster sizes) of the model
trained on the
+training set. An exception is thrown if no summary exists.
+"""
+if self.hasSummary:
+return KMeansSummary(self._call_java("summary"))
+else:
+raise RuntimeError("No training summary available for this %s" %
+ self.__class__.__name__)
+
@inherit_doc
class KMeans(JavaEstimator, HasFeaturesCol, HasPredictionCol, HasMaxIter,
HasTol, HasSeed,
@@ -337,6 +369,13 @@ class KMeans(JavaEstimator, HasFeaturesCol,
HasPredictionCol, HasMaxIter, HasTol
True
>>> rows[2].prediction == rows[3].prediction
True
+>>> model.hasSummary
+True
+>>> summary = model.summary
+>>> summary.k
+2
+>>> summary.clusterSizes
+[2, 2]
>>> kmeans_path = temp_path + "/kmeans"
>>> kmeans.save(kmeans_path)
>>> kmeans2 = KMeans.load(kmeans_path)
@@ -345,6 +384,8 @@ class KMeans(JavaEstimator, HasFeaturesCol,
HasPredictionCol, HasMaxIter, HasTol
>>> model_path = temp_path + "/kmeans_model"
>>> model.save(model_path)
>>> model2 = KMeansModel.load(model_path)
+>>> model2.hasSummary
+False
>>> model.clusterCenters()[0] == model2.clusterCenters()[0]
array([ True, True], dtype=bool)
>>> model.clusterCenters()[1] == model2.clusterCenters()[1]
http://git-wip-us.apache.org/repos/asf/spark/blob/b95aad7c/python/pyspark/ml/tests.py
--
diff --git a/python/pyspark/ml/tests.py b/python/pyspark/ml/tests.py
index c0f0d40..a0c288a 100755
--- a/python/pyspark/ml/tests.py
+++ b/python/pyspark/ml/tests.py
@@ -1129,6 +1129,21 @@ class TrainingSummaryTest(SparkSessionTestCase):
self.assertEqual(len(s.clusterSizes), 2)
self.assertEqual(s.k, 2)
+def test_kmeans_summary(self):
+data = [(Vectors.dense([0.0, 0.0]),), (Vectors.dense([1.0, 1.0]),),
+(Vectors.dense([9.0, 8.0]),), (Vectors.dense([8.0, 9.0]),)]
+df = self.spark.createDataFrame(data, ["features"])
+kmeans = KMeans(k=2, seed=1)
+model = kmeans.fit(df)
+self.assertTrue(model.hasSummary)
+s = model.summary
+self.assertTrue(isinstance(s.predictions, DataFrame))
+self.assertEqual(s.featuresCol, "features")
+self.assertEqual(s.predictionCol, "prediction")
+self.assertTrue(isinstance(s.cluster, DataFrame))
+self.assertEqual(len(s.clusterSizes), 2)
+self.assertEqual(s.k, 2)
+
class
spark git commit: [SPARK-18476][SPARKR][ML] SparkR Logistic Regression should should support output original label.
Repository: spark
Updated Branches:
refs/heads/master 0a811210f -> 2eb6764fb
[SPARK-18476][SPARKR][ML] SparkR Logistic Regression should should support
output original label.
## What changes were proposed in this pull request?
Similar to SPARK-18401, as a classification algorithm, logistic regression
should support output original label instead of supporting index label.
In this PR, original label output is supported and test cases are modified and
added. Document is also modified.
## How was this patch tested?
Unit tests.
Author: wm...@hotmail.com
Closes #15910 from wangmiao1981/audit.
Project: http://git-wip-us.apache.org/repos/asf/spark/repo
Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/2eb6764f
Tree: http://git-wip-us.apache.org/repos/asf/spark/tree/2eb6764f
Diff: http://git-wip-us.apache.org/repos/asf/spark/diff/2eb6764f
Branch: refs/heads/master
Commit: 2eb6764fbb23553fc17772d8a4a1cad55ff7ba6e
Parents: 0a81121
Author: wm...@hotmail.com
Authored: Wed Nov 30 20:32:17 2016 -0800
Committer: Yanbo Liang
Committed: Wed Nov 30 20:32:17 2016 -0800
--
R/pkg/R/mllib.R | 19 +-
R/pkg/inst/tests/testthat/test_mllib.R | 26 +-
.../scala/org/apache/spark/SparkContext.scala | 2 +-
.../spark/ml/r/LogisticRegressionWrapper.scala | 37 ++--
4 files changed, 54 insertions(+), 30 deletions(-)
--
http://git-wip-us.apache.org/repos/asf/spark/blob/2eb6764f/R/pkg/R/mllib.R
--
diff --git a/R/pkg/R/mllib.R b/R/pkg/R/mllib.R
index 02bc645..eed8293 100644
--- a/R/pkg/R/mllib.R
+++ b/R/pkg/R/mllib.R
@@ -712,7 +712,6 @@ setMethod("predict", signature(object = "KMeansModel"),
#'of L1 and L2. Default is 0.0 which is an L2 penalty.
#' @param maxIter maximum iteration number.
#' @param tol convergence tolerance of iterations.
-#' @param fitIntercept whether to fit an intercept term.
#' @param family the name of family which is a description of the label
distribution to be used in the model.
#' Supported options:
#' \itemize{
@@ -747,11 +746,11 @@ setMethod("predict", signature(object = "KMeansModel"),
#' \dontrun{
#' sparkR.session()
#' # binary logistic regression
-#' label <- c(1.0, 1.0, 1.0, 0.0, 0.0)
-#' feature <- c(1.1419053, 0.9194079, -0.9498666, -1.1069903, 0.2809776)
-#' binary_data <- as.data.frame(cbind(label, feature))
+#' label <- c(0.0, 0.0, 0.0, 1.0, 1.0)
+#' features <- c(1.1419053, 0.9194079, -0.9498666, -1.1069903, 0.2809776)
+#' binary_data <- as.data.frame(cbind(label, features))
#' binary_df <- createDataFrame(binary_data)
-#' blr_model <- spark.logit(binary_df, label ~ feature, thresholds = 1.0)
+#' blr_model <- spark.logit(binary_df, label ~ features, thresholds = 1.0)
#' blr_predict <- collect(select(predict(blr_model, binary_df), "prediction"))
#'
#' # summary of binary logistic regression
@@ -783,7 +782,7 @@ setMethod("predict", signature(object = "KMeansModel"),
#' @note spark.logit since 2.1.0
setMethod("spark.logit", signature(data = "SparkDataFrame", formula =
"formula"),
function(data, formula, regParam = 0.0, elasticNetParam = 0.0,
maxIter = 100,
- tol = 1E-6, fitIntercept = TRUE, family = "auto",
standardization = TRUE,
+ tol = 1E-6, family = "auto", standardization = TRUE,
thresholds = 0.5, weightCol = NULL, aggregationDepth = 2,
probabilityCol = "probability") {
formula <- paste(deparse(formula), collapse = "")
@@ -795,10 +794,10 @@ setMethod("spark.logit", signature(data =
"SparkDataFrame", formula = "formula")
jobj <-
callJStatic("org.apache.spark.ml.r.LogisticRegressionWrapper", "fit",
data@sdf, formula, as.numeric(regParam),
as.numeric(elasticNetParam),
as.integer(maxIter),
-as.numeric(tol), as.logical(fitIntercept),
-as.character(family),
as.logical(standardization),
-as.array(thresholds), as.character(weightCol),
-as.integer(aggregationDepth),
as.character(probabilityCol))
+as.numeric(tol), as.character(family),
+as.logical(standardization),
as.array(thresholds),
+as.character(weightCol),
as.integer(aggregationDepth),
+as.character(probabilityCol))
new("LogisticRegressionModel", jobj = jobj)
})
http://git-wip-us.apache.org/repos/asf/spark/blob/2eb6764f/R/pkg/inst/tests/testthat/test_mllib.R
-
spark git commit: [SPARK-18476][SPARKR][ML] SparkR Logistic Regression should should support output original label.
Repository: spark
Updated Branches:
refs/heads/branch-2.1 7d4596734 -> e8d8e3509
[SPARK-18476][SPARKR][ML] SparkR Logistic Regression should should support
output original label.
## What changes were proposed in this pull request?
Similar to SPARK-18401, as a classification algorithm, logistic regression
should support output original label instead of supporting index label.
In this PR, original label output is supported and test cases are modified and
added. Document is also modified.
## How was this patch tested?
Unit tests.
Author: wm...@hotmail.com
Closes #15910 from wangmiao1981/audit.
(cherry picked from commit 2eb6764fbb23553fc17772d8a4a1cad55ff7ba6e)
Signed-off-by: Yanbo Liang
Project: http://git-wip-us.apache.org/repos/asf/spark/repo
Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/e8d8e350
Tree: http://git-wip-us.apache.org/repos/asf/spark/tree/e8d8e350
Diff: http://git-wip-us.apache.org/repos/asf/spark/diff/e8d8e350
Branch: refs/heads/branch-2.1
Commit: e8d8e350998e6e44a6dee7f78dbe2d1aa997c1d6
Parents: 7d45967
Author: wm...@hotmail.com
Authored: Wed Nov 30 20:32:17 2016 -0800
Committer: Yanbo Liang
Committed: Wed Nov 30 20:33:07 2016 -0800
--
R/pkg/R/mllib.R | 19 +-
R/pkg/inst/tests/testthat/test_mllib.R | 26 +-
.../scala/org/apache/spark/SparkContext.scala | 2 +-
.../spark/ml/r/LogisticRegressionWrapper.scala | 37 ++--
4 files changed, 54 insertions(+), 30 deletions(-)
--
http://git-wip-us.apache.org/repos/asf/spark/blob/e8d8e350/R/pkg/R/mllib.R
--
diff --git a/R/pkg/R/mllib.R b/R/pkg/R/mllib.R
index 02bc645..eed8293 100644
--- a/R/pkg/R/mllib.R
+++ b/R/pkg/R/mllib.R
@@ -712,7 +712,6 @@ setMethod("predict", signature(object = "KMeansModel"),
#'of L1 and L2. Default is 0.0 which is an L2 penalty.
#' @param maxIter maximum iteration number.
#' @param tol convergence tolerance of iterations.
-#' @param fitIntercept whether to fit an intercept term.
#' @param family the name of family which is a description of the label
distribution to be used in the model.
#' Supported options:
#' \itemize{
@@ -747,11 +746,11 @@ setMethod("predict", signature(object = "KMeansModel"),
#' \dontrun{
#' sparkR.session()
#' # binary logistic regression
-#' label <- c(1.0, 1.0, 1.0, 0.0, 0.0)
-#' feature <- c(1.1419053, 0.9194079, -0.9498666, -1.1069903, 0.2809776)
-#' binary_data <- as.data.frame(cbind(label, feature))
+#' label <- c(0.0, 0.0, 0.0, 1.0, 1.0)
+#' features <- c(1.1419053, 0.9194079, -0.9498666, -1.1069903, 0.2809776)
+#' binary_data <- as.data.frame(cbind(label, features))
#' binary_df <- createDataFrame(binary_data)
-#' blr_model <- spark.logit(binary_df, label ~ feature, thresholds = 1.0)
+#' blr_model <- spark.logit(binary_df, label ~ features, thresholds = 1.0)
#' blr_predict <- collect(select(predict(blr_model, binary_df), "prediction"))
#'
#' # summary of binary logistic regression
@@ -783,7 +782,7 @@ setMethod("predict", signature(object = "KMeansModel"),
#' @note spark.logit since 2.1.0
setMethod("spark.logit", signature(data = "SparkDataFrame", formula =
"formula"),
function(data, formula, regParam = 0.0, elasticNetParam = 0.0,
maxIter = 100,
- tol = 1E-6, fitIntercept = TRUE, family = "auto",
standardization = TRUE,
+ tol = 1E-6, family = "auto", standardization = TRUE,
thresholds = 0.5, weightCol = NULL, aggregationDepth = 2,
probabilityCol = "probability") {
formula <- paste(deparse(formula), collapse = "")
@@ -795,10 +794,10 @@ setMethod("spark.logit", signature(data =
"SparkDataFrame", formula = "formula")
jobj <-
callJStatic("org.apache.spark.ml.r.LogisticRegressionWrapper", "fit",
data@sdf, formula, as.numeric(regParam),
as.numeric(elasticNetParam),
as.integer(maxIter),
-as.numeric(tol), as.logical(fitIntercept),
-as.character(family),
as.logical(standardization),
-as.array(thresholds), as.character(weightCol),
-as.integer(aggregationDepth),
as.character(probabilityCol))
+as.numeric(tol), as.character(family),
+as.logical(standardization),
as.array(thresholds),
+as.character(weightCol),
as.integer(aggregationDepth),
+as.character(probabilityCol))
new("LogisticRegressionModel", jobj = jobj)
})
http://git-wip-us.apache.org/repos/asf/spark/
spark git commit: [SPARK-18625][ML] OneVsRestModel should support setFeaturesCol and setPredictionCol
Repository: spark
Updated Branches:
refs/heads/branch-2.1 c13c2939f -> 88e07efe8
[SPARK-18625][ML] OneVsRestModel should support setFeaturesCol and
setPredictionCol
## What changes were proposed in this pull request?
add `setFeaturesCol` and `setPredictionCol` for `OneVsRestModel`
## How was this patch tested?
added tests
Author: Zheng RuiFeng
Closes #16059 from zhengruifeng/ovrm_setCol.
(cherry picked from commit bdfe7f67468ecfd9927a1fec60d6605dd05ebe3f)
Signed-off-by: Yanbo Liang
Project: http://git-wip-us.apache.org/repos/asf/spark/repo
Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/88e07efe
Tree: http://git-wip-us.apache.org/repos/asf/spark/tree/88e07efe
Diff: http://git-wip-us.apache.org/repos/asf/spark/diff/88e07efe
Branch: refs/heads/branch-2.1
Commit: 88e07efe86512142eeada6a6f1f7fe858204c59b
Parents: c13c293
Author: Zheng RuiFeng
Authored: Mon Dec 5 00:32:58 2016 -0800
Committer: Yanbo Liang
Committed: Mon Dec 5 00:33:21 2016 -0800
--
.../apache/spark/ml/classification/OneVsRest.scala| 9 +
.../spark/ml/classification/OneVsRestSuite.scala | 14 +-
2 files changed, 22 insertions(+), 1 deletion(-)
--
http://git-wip-us.apache.org/repos/asf/spark/blob/88e07efe/mllib/src/main/scala/org/apache/spark/ml/classification/OneVsRest.scala
--
diff --git
a/mllib/src/main/scala/org/apache/spark/ml/classification/OneVsRest.scala
b/mllib/src/main/scala/org/apache/spark/ml/classification/OneVsRest.scala
index f4ab0a0..e58b30d 100644
--- a/mllib/src/main/scala/org/apache/spark/ml/classification/OneVsRest.scala
+++ b/mllib/src/main/scala/org/apache/spark/ml/classification/OneVsRest.scala
@@ -140,6 +140,14 @@ final class OneVsRestModel private[ml] (
this(uid, Metadata.empty, models.asScala.toArray)
}
+ /** @group setParam */
+ @Since("2.1.0")
+ def setFeaturesCol(value: String): this.type = set(featuresCol, value)
+
+ /** @group setParam */
+ @Since("2.1.0")
+ def setPredictionCol(value: String): this.type = set(predictionCol, value)
+
@Since("1.4.0")
override def transformSchema(schema: StructType): StructType = {
validateAndTransformSchema(schema, fitting = false,
getClassifier.featuresDataType)
@@ -175,6 +183,7 @@ final class OneVsRestModel private[ml] (
val updateUDF = udf { (predictions: Map[Int, Double], prediction:
Vector) =>
predictions + ((index, prediction(1)))
}
+model.setFeaturesCol($(featuresCol))
val transformedDataset = model.transform(df).select(columns: _*)
val updatedDataset = transformedDataset
.withColumn(tmpColName, updateUDF(col(accColName),
col(rawPredictionCol)))
http://git-wip-us.apache.org/repos/asf/spark/blob/88e07efe/mllib/src/test/scala/org/apache/spark/ml/classification/OneVsRestSuite.scala
--
diff --git
a/mllib/src/test/scala/org/apache/spark/ml/classification/OneVsRestSuite.scala
b/mllib/src/test/scala/org/apache/spark/ml/classification/OneVsRestSuite.scala
index 3f9bcec..aacb792 100644
---
a/mllib/src/test/scala/org/apache/spark/ml/classification/OneVsRestSuite.scala
+++
b/mllib/src/test/scala/org/apache/spark/ml/classification/OneVsRestSuite.scala
@@ -22,7 +22,7 @@ import org.apache.spark.ml.attribute.NominalAttribute
import org.apache.spark.ml.classification.LogisticRegressionSuite._
import org.apache.spark.ml.feature.LabeledPoint
import org.apache.spark.ml.feature.StringIndexer
-import org.apache.spark.ml.linalg.{DenseMatrix, Vectors}
+import org.apache.spark.ml.linalg.Vectors
import org.apache.spark.ml.param.{ParamMap, ParamsSuite}
import org.apache.spark.ml.util.{DefaultReadWriteTest, MetadataUtils,
MLTestingUtils}
import org.apache.spark.mllib.classification.LogisticRegressionWithLBFGS
@@ -33,6 +33,7 @@ import org.apache.spark.mllib.util.MLlibTestSparkContext
import org.apache.spark.mllib.util.TestingUtils._
import org.apache.spark.rdd.RDD
import org.apache.spark.sql.Dataset
+import org.apache.spark.sql.functions._
import org.apache.spark.sql.types.Metadata
class OneVsRestSuite extends SparkFunSuite with MLlibTestSparkContext with
DefaultReadWriteTest {
@@ -136,6 +137,17 @@ class OneVsRestSuite extends SparkFunSuite with
MLlibTestSparkContext with Defau
assert(outputFields.contains("p"))
}
+ test("SPARK-18625 : OneVsRestModel should support setFeaturesCol and
setPredictionCol") {
+val ova = new OneVsRest().setClassifier(new LogisticRegression)
+val ovaModel = ova.fit(dataset)
+val dataset2 = dataset.select(col("label").as("y"),
col("features").as("fea"))
+ovaModel.setFeaturesCol("fea")
+ovaModel.setPredictionCol("pred")
+val transformedDataset = ovaModel.transform(dataset2)
+val outputFields =
spark git commit: [SPARK-18625][ML] OneVsRestModel should support setFeaturesCol and setPredictionCol
Repository: spark
Updated Branches:
refs/heads/master e9730b707 -> bdfe7f674
[SPARK-18625][ML] OneVsRestModel should support setFeaturesCol and
setPredictionCol
## What changes were proposed in this pull request?
add `setFeaturesCol` and `setPredictionCol` for `OneVsRestModel`
## How was this patch tested?
added tests
Author: Zheng RuiFeng
Closes #16059 from zhengruifeng/ovrm_setCol.
Project: http://git-wip-us.apache.org/repos/asf/spark/repo
Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/bdfe7f67
Tree: http://git-wip-us.apache.org/repos/asf/spark/tree/bdfe7f67
Diff: http://git-wip-us.apache.org/repos/asf/spark/diff/bdfe7f67
Branch: refs/heads/master
Commit: bdfe7f67468ecfd9927a1fec60d6605dd05ebe3f
Parents: e9730b7
Author: Zheng RuiFeng
Authored: Mon Dec 5 00:32:58 2016 -0800
Committer: Yanbo Liang
Committed: Mon Dec 5 00:32:58 2016 -0800
--
.../apache/spark/ml/classification/OneVsRest.scala| 9 +
.../spark/ml/classification/OneVsRestSuite.scala | 14 +-
2 files changed, 22 insertions(+), 1 deletion(-)
--
http://git-wip-us.apache.org/repos/asf/spark/blob/bdfe7f67/mllib/src/main/scala/org/apache/spark/ml/classification/OneVsRest.scala
--
diff --git
a/mllib/src/main/scala/org/apache/spark/ml/classification/OneVsRest.scala
b/mllib/src/main/scala/org/apache/spark/ml/classification/OneVsRest.scala
index f4ab0a0..e58b30d 100644
--- a/mllib/src/main/scala/org/apache/spark/ml/classification/OneVsRest.scala
+++ b/mllib/src/main/scala/org/apache/spark/ml/classification/OneVsRest.scala
@@ -140,6 +140,14 @@ final class OneVsRestModel private[ml] (
this(uid, Metadata.empty, models.asScala.toArray)
}
+ /** @group setParam */
+ @Since("2.1.0")
+ def setFeaturesCol(value: String): this.type = set(featuresCol, value)
+
+ /** @group setParam */
+ @Since("2.1.0")
+ def setPredictionCol(value: String): this.type = set(predictionCol, value)
+
@Since("1.4.0")
override def transformSchema(schema: StructType): StructType = {
validateAndTransformSchema(schema, fitting = false,
getClassifier.featuresDataType)
@@ -175,6 +183,7 @@ final class OneVsRestModel private[ml] (
val updateUDF = udf { (predictions: Map[Int, Double], prediction:
Vector) =>
predictions + ((index, prediction(1)))
}
+model.setFeaturesCol($(featuresCol))
val transformedDataset = model.transform(df).select(columns: _*)
val updatedDataset = transformedDataset
.withColumn(tmpColName, updateUDF(col(accColName),
col(rawPredictionCol)))
http://git-wip-us.apache.org/repos/asf/spark/blob/bdfe7f67/mllib/src/test/scala/org/apache/spark/ml/classification/OneVsRestSuite.scala
--
diff --git
a/mllib/src/test/scala/org/apache/spark/ml/classification/OneVsRestSuite.scala
b/mllib/src/test/scala/org/apache/spark/ml/classification/OneVsRestSuite.scala
index 3f9bcec..aacb792 100644
---
a/mllib/src/test/scala/org/apache/spark/ml/classification/OneVsRestSuite.scala
+++
b/mllib/src/test/scala/org/apache/spark/ml/classification/OneVsRestSuite.scala
@@ -22,7 +22,7 @@ import org.apache.spark.ml.attribute.NominalAttribute
import org.apache.spark.ml.classification.LogisticRegressionSuite._
import org.apache.spark.ml.feature.LabeledPoint
import org.apache.spark.ml.feature.StringIndexer
-import org.apache.spark.ml.linalg.{DenseMatrix, Vectors}
+import org.apache.spark.ml.linalg.Vectors
import org.apache.spark.ml.param.{ParamMap, ParamsSuite}
import org.apache.spark.ml.util.{DefaultReadWriteTest, MetadataUtils,
MLTestingUtils}
import org.apache.spark.mllib.classification.LogisticRegressionWithLBFGS
@@ -33,6 +33,7 @@ import org.apache.spark.mllib.util.MLlibTestSparkContext
import org.apache.spark.mllib.util.TestingUtils._
import org.apache.spark.rdd.RDD
import org.apache.spark.sql.Dataset
+import org.apache.spark.sql.functions._
import org.apache.spark.sql.types.Metadata
class OneVsRestSuite extends SparkFunSuite with MLlibTestSparkContext with
DefaultReadWriteTest {
@@ -136,6 +137,17 @@ class OneVsRestSuite extends SparkFunSuite with
MLlibTestSparkContext with Defau
assert(outputFields.contains("p"))
}
+ test("SPARK-18625 : OneVsRestModel should support setFeaturesCol and
setPredictionCol") {
+val ova = new OneVsRest().setClassifier(new LogisticRegression)
+val ovaModel = ova.fit(dataset)
+val dataset2 = dataset.select(col("label").as("y"),
col("features").as("fea"))
+ovaModel.setFeaturesCol("fea")
+ovaModel.setPredictionCol("pred")
+val transformedDataset = ovaModel.transform(dataset2)
+val outputFields = transformedDataset.schema.fieldNames.toSet
+assert(outputFields === Set("y", "fea", "pred"))
+ }
+
spark git commit: [SPARK-18279][DOC][ML][SPARKR] Add R examples to ML programming guide.
Repository: spark
Updated Branches:
refs/heads/master bdfe7f674 -> eb8dd6813
[SPARK-18279][DOC][ML][SPARKR] Add R examples to ML programming guide.
## What changes were proposed in this pull request?
Add R examples to ML programming guide for the following algorithms as POC:
* spark.glm
* spark.survreg
* spark.naiveBayes
* spark.kmeans
The four algorithms were added to SparkR since 2.0.0, more docs for algorithms
added during 2.1 release cycle will be addressed in a separate follow-up PR.
## How was this patch tested?
This is the screenshots of generated ML programming guide for
```GeneralizedLinearRegression```:
Author: Yanbo Liang
Closes #16136 from yanboliang/spark-18279.
Project: http://git-wip-us.apache.org/repos/asf/spark/repo
Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/eb8dd681
Tree: http://git-wip-us.apache.org/repos/asf/spark/tree/eb8dd681
Diff: http://git-wip-us.apache.org/repos/asf/spark/diff/eb8dd681
Branch: refs/heads/master
Commit: eb8dd68132998aa00902dfeb935db1358781e1c1
Parents: bdfe7f6
Author: Yanbo Liang
Authored: Mon Dec 5 00:39:44 2016 -0800
Committer: Yanbo Liang
Committed: Mon Dec 5 00:39:44 2016 -0800
--
docs/ml-classification-regression.md | 22 ++
docs/ml-clustering.md| 8
2 files changed, 30 insertions(+)
--
http://git-wip-us.apache.org/repos/asf/spark/blob/eb8dd681/docs/ml-classification-regression.md
--
diff --git a/docs/ml-classification-regression.md
b/docs/ml-classification-regression.md
index 43cc79b..5759593 100644
--- a/docs/ml-classification-regression.md
+++ b/docs/ml-classification-regression.md
@@ -389,6 +389,14 @@ Refer to the [Python API
docs](api/python/pyspark.ml.html#pyspark.ml.classificat
{% include_example python/ml/naive_bayes_example.py %}
+
+
+
+Refer to the [R API docs](api/R/spark.naiveBayes.html) for more details.
+
+{% include_example naiveBayes r/ml.R %}
+
+
@@ -566,6 +574,13 @@ Refer to the [Python API
docs](api/python/pyspark.ml.html#pyspark.ml.regression.
{% include_example python/ml/generalized_linear_regression_example.py %}
+
+
+Refer to the [R API docs](api/R/spark.glm.html) for more details.
+
+{% include_example glm r/ml.R %}
+
+
@@ -755,6 +770,13 @@ Refer to the [Python API
docs](api/python/pyspark.ml.html#pyspark.ml.regression.
{% include_example python/ml/aft_survival_regression.py %}
+
+
+Refer to the [R API docs](api/R/spark.survreg.html) for more details.
+
+{% include_example survreg r/ml.R %}
+
+
http://git-wip-us.apache.org/repos/asf/spark/blob/eb8dd681/docs/ml-clustering.md
--
diff --git a/docs/ml-clustering.md b/docs/ml-clustering.md
index eedacb1..da23442 100644
--- a/docs/ml-clustering.md
+++ b/docs/ml-clustering.md
@@ -86,6 +86,14 @@ Refer to the [Python API
docs](api/python/pyspark.ml.html#pyspark.ml.clustering.
{% include_example python/ml/kmeans_example.py %}
+
+
+
+Refer to the [R API docs](api/R/spark.kmeans.html) for more details.
+
+{% include_example kmeans r/ml.R %}
+
+
## Latent Dirichlet allocation (LDA)
-
To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org
For additional commands, e-mail: commits-h...@spark.apache.org
spark git commit: [SPARK-18279][DOC][ML][SPARKR] Add R examples to ML programming guide.
Repository: spark
Updated Branches:
refs/heads/branch-2.1 88e07efe8 -> 1821cbead
[SPARK-18279][DOC][ML][SPARKR] Add R examples to ML programming guide.
## What changes were proposed in this pull request?
Add R examples to ML programming guide for the following algorithms as POC:
* spark.glm
* spark.survreg
* spark.naiveBayes
* spark.kmeans
The four algorithms were added to SparkR since 2.0.0, more docs for algorithms
added during 2.1 release cycle will be addressed in a separate follow-up PR.
## How was this patch tested?
This is the screenshots of generated ML programming guide for
```GeneralizedLinearRegression```:
Author: Yanbo Liang
Closes #16136 from yanboliang/spark-18279.
(cherry picked from commit eb8dd68132998aa00902dfeb935db1358781e1c1)
Signed-off-by: Yanbo Liang
Project: http://git-wip-us.apache.org/repos/asf/spark/repo
Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/1821cbea
Tree: http://git-wip-us.apache.org/repos/asf/spark/tree/1821cbea
Diff: http://git-wip-us.apache.org/repos/asf/spark/diff/1821cbea
Branch: refs/heads/branch-2.1
Commit: 1821cbead1875fbe1c16d7c50563aa0839e1f70f
Parents: 88e07ef
Author: Yanbo Liang
Authored: Mon Dec 5 00:39:44 2016 -0800
Committer: Yanbo Liang
Committed: Mon Dec 5 00:40:33 2016 -0800
--
docs/ml-classification-regression.md | 22 ++
docs/ml-clustering.md| 8
2 files changed, 30 insertions(+)
--
http://git-wip-us.apache.org/repos/asf/spark/blob/1821cbea/docs/ml-classification-regression.md
--
diff --git a/docs/ml-classification-regression.md
b/docs/ml-classification-regression.md
index c72c01f..5148ad0 100644
--- a/docs/ml-classification-regression.md
+++ b/docs/ml-classification-regression.md
@@ -389,6 +389,14 @@ Refer to the [Python API
docs](api/python/pyspark.ml.html#pyspark.ml.classificat
{% include_example python/ml/naive_bayes_example.py %}
+
+
+
+Refer to the [R API docs](api/R/spark.naiveBayes.html) for more details.
+
+{% include_example naiveBayes r/ml.R %}
+
+
@@ -566,6 +574,13 @@ Refer to the [Python API
docs](api/python/pyspark.ml.html#pyspark.ml.regression.
{% include_example python/ml/generalized_linear_regression_example.py %}
+
+
+Refer to the [R API docs](api/R/spark.glm.html) for more details.
+
+{% include_example glm r/ml.R %}
+
+
@@ -755,6 +770,13 @@ Refer to the [Python API
docs](api/python/pyspark.ml.html#pyspark.ml.regression.
{% include_example python/ml/aft_survival_regression.py %}
+
+
+Refer to the [R API docs](api/R/spark.survreg.html) for more details.
+
+{% include_example survreg r/ml.R %}
+
+
http://git-wip-us.apache.org/repos/asf/spark/blob/1821cbea/docs/ml-clustering.md
--
diff --git a/docs/ml-clustering.md b/docs/ml-clustering.md
index 8a0a61c..4731abc 100644
--- a/docs/ml-clustering.md
+++ b/docs/ml-clustering.md
@@ -86,6 +86,14 @@ Refer to the [Python API
docs](api/python/pyspark.ml.html#pyspark.ml.clustering.
{% include_example python/ml/kmeans_example.py %}
+
+
+
+Refer to the [R API docs](api/R/spark.kmeans.html) for more details.
+
+{% include_example kmeans r/ml.R %}
+
+
## Latent Dirichlet allocation (LDA)
-
To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org
For additional commands, e-mail: commits-h...@spark.apache.org
spark git commit: [SPARK-18686][SPARKR][ML] Several cleanup and improvements for spark.logit.
Repository: spark
Updated Branches:
refs/heads/master 5c6bcdbda -> 90b59d1bf
[SPARK-18686][SPARKR][ML] Several cleanup and improvements for spark.logit.
## What changes were proposed in this pull request?
Several cleanup and improvements for ```spark.logit```:
* ```summary``` should return coefficients matrix, and should output labels for
each class if the model is multinomial logistic regression model.
* ```summary``` should not return ```areaUnderROC, roc, pr, ...```, since most
of them are DataFrame which are less important for R users. Meanwhile, these
metrics ignore instance weights (setting all to 1.0) which will be changed in
later Spark version. In case it will introduce breaking changes, we do not
expose them currently.
* SparkR test improvement: comparing the training result with native R glmnet.
* Remove argument ```aggregationDepth``` from ```spark.logit```, since it's an
expert Param(related with Spark architecture and job execution) that would be
used rarely by R users.
## How was this patch tested?
Unit tests.
The ```summary``` output after this change:
multinomial logistic regression:
```
> df <- suppressWarnings(createDataFrame(iris))
> model <- spark.logit(df, Species ~ ., regParam = 0.5)
> summary(model)
$coefficients
versicolor virginica setosa
(Intercept) 1.514031-2.609108 1.095077
Sepal_Length 0.02511006 0.2649821 -0.2900921
Sepal_Width -0.5291215 -0.02016446 0.549286
Petal_Length 0.03647411 0.1544119 -0.190886
Petal_Width 0.000236092 0.4195804 -0.4198165
```
binomial logistic regression:
```
> df <- suppressWarnings(createDataFrame(iris))
> training <- df[df$Species %in% c("versicolor", "virginica"), ]
> model <- spark.logit(training, Species ~ ., regParam = 0.5)
> summary(model)
$coefficients
Estimate
(Intercept) -6.053815
Sepal_Length 0.2449379
Sepal_Width 0.1648321
Petal_Length 0.4730718
Petal_Width 1.031947
```
Author: Yanbo Liang
Closes #16117 from yanboliang/spark-18686.
Project: http://git-wip-us.apache.org/repos/asf/spark/repo
Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/90b59d1b
Tree: http://git-wip-us.apache.org/repos/asf/spark/tree/90b59d1b
Diff: http://git-wip-us.apache.org/repos/asf/spark/diff/90b59d1b
Branch: refs/heads/master
Commit: 90b59d1bf262b41c3a5f780697f504030f9d079c
Parents: 5c6bcdb
Author: Yanbo Liang
Authored: Wed Dec 7 00:31:11 2016 -0800
Committer: Yanbo Liang
Committed: Wed Dec 7 00:31:11 2016 -0800
--
R/pkg/R/mllib.R | 86 +++--
R/pkg/inst/tests/testthat/test_mllib.R | 183 +--
.../spark/ml/r/LogisticRegressionWrapper.scala | 81
3 files changed, 203 insertions(+), 147 deletions(-)
--
http://git-wip-us.apache.org/repos/asf/spark/blob/90b59d1b/R/pkg/R/mllib.R
--
diff --git a/R/pkg/R/mllib.R b/R/pkg/R/mllib.R
index eed8293..074e9cb 100644
--- a/R/pkg/R/mllib.R
+++ b/R/pkg/R/mllib.R
@@ -733,8 +733,6 @@ setMethod("predict", signature(object = "KMeansModel"),
#' excepting that at most one value may be 0. The class with
largest value p/t is predicted, where p
#' is the original probability of that class and t is the
class's threshold.
#' @param weightCol The weight column name.
-#' @param aggregationDepth depth for treeAggregate (>= 2). If the dimensions
of features or the number of partitions
-#' are large, this param could be adjusted to a larger
size.
#' @param probabilityCol column name for predicted class conditional
probabilities.
#' @param ... additional arguments passed to the method.
#' @return \code{spark.logit} returns a fitted logistic regression model
@@ -746,45 +744,35 @@ setMethod("predict", signature(object = "KMeansModel"),
#' \dontrun{
#' sparkR.session()
#' # binary logistic regression
-#' label <- c(0.0, 0.0, 0.0, 1.0, 1.0)
-#' features <- c(1.1419053, 0.9194079, -0.9498666, -1.1069903, 0.2809776)
-#' binary_data <- as.data.frame(cbind(label, features))
-#' binary_df <- createDataFrame(binary_data)
-#' blr_model <- spark.logit(binary_df, label ~ features, thresholds = 1.0)
-#' blr_predict <- collect(select(predict(blr_model, binary_df), "prediction"))
-#'
-#' # summary of binary logistic regression
-#' blr_summary <- summary(blr_model)
-#' blr_fmeasure <- collect(select(blr_summary$fMeasureByThreshold,
"threshold", "F-Measure"))
+#' df <- createDataFrame(iris)
+#' training <- df[df$Species %in% c("versicolor", "virginica"), ]
+#' model <- spark.logit(training, Species ~ ., regParam = 0.5)
+#' summary <- summary(model)
+#'
+#' # fitted values on training data
+#' fitted <- predict(model, training)
+#'
#' # save fitted model to input path
#' path <- "path/to/model"
-#' write.ml(blr_model, path)
+#' write.ml(m
spark git commit: [SPARK-18686][SPARKR][ML] Several cleanup and improvements for spark.logit.
Repository: spark
Updated Branches:
refs/heads/branch-2.1 3750c6e9b -> 340e9aea4
[SPARK-18686][SPARKR][ML] Several cleanup and improvements for spark.logit.
## What changes were proposed in this pull request?
Several cleanup and improvements for ```spark.logit```:
* ```summary``` should return coefficients matrix, and should output labels for
each class if the model is multinomial logistic regression model.
* ```summary``` should not return ```areaUnderROC, roc, pr, ...```, since most
of them are DataFrame which are less important for R users. Meanwhile, these
metrics ignore instance weights (setting all to 1.0) which will be changed in
later Spark version. In case it will introduce breaking changes, we do not
expose them currently.
* SparkR test improvement: comparing the training result with native R glmnet.
* Remove argument ```aggregationDepth``` from ```spark.logit```, since it's an
expert Param(related with Spark architecture and job execution) that would be
used rarely by R users.
## How was this patch tested?
Unit tests.
The ```summary``` output after this change:
multinomial logistic regression:
```
> df <- suppressWarnings(createDataFrame(iris))
> model <- spark.logit(df, Species ~ ., regParam = 0.5)
> summary(model)
$coefficients
versicolor virginica setosa
(Intercept) 1.514031-2.609108 1.095077
Sepal_Length 0.02511006 0.2649821 -0.2900921
Sepal_Width -0.5291215 -0.02016446 0.549286
Petal_Length 0.03647411 0.1544119 -0.190886
Petal_Width 0.000236092 0.4195804 -0.4198165
```
binomial logistic regression:
```
> df <- suppressWarnings(createDataFrame(iris))
> training <- df[df$Species %in% c("versicolor", "virginica"), ]
> model <- spark.logit(training, Species ~ ., regParam = 0.5)
> summary(model)
$coefficients
Estimate
(Intercept) -6.053815
Sepal_Length 0.2449379
Sepal_Width 0.1648321
Petal_Length 0.4730718
Petal_Width 1.031947
```
Author: Yanbo Liang
Closes #16117 from yanboliang/spark-18686.
(cherry picked from commit 90b59d1bf262b41c3a5f780697f504030f9d079c)
Signed-off-by: Yanbo Liang
Project: http://git-wip-us.apache.org/repos/asf/spark/repo
Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/340e9aea
Tree: http://git-wip-us.apache.org/repos/asf/spark/tree/340e9aea
Diff: http://git-wip-us.apache.org/repos/asf/spark/diff/340e9aea
Branch: refs/heads/branch-2.1
Commit: 340e9aea4853805c42b8739004d93efe8fe16ba4
Parents: 3750c6e
Author: Yanbo Liang
Authored: Wed Dec 7 00:31:11 2016 -0800
Committer: Yanbo Liang
Committed: Wed Dec 7 00:32:32 2016 -0800
--
R/pkg/R/mllib.R | 86 +++--
R/pkg/inst/tests/testthat/test_mllib.R | 183 +--
.../spark/ml/r/LogisticRegressionWrapper.scala | 81
3 files changed, 203 insertions(+), 147 deletions(-)
--
http://git-wip-us.apache.org/repos/asf/spark/blob/340e9aea/R/pkg/R/mllib.R
--
diff --git a/R/pkg/R/mllib.R b/R/pkg/R/mllib.R
index eed8293..074e9cb 100644
--- a/R/pkg/R/mllib.R
+++ b/R/pkg/R/mllib.R
@@ -733,8 +733,6 @@ setMethod("predict", signature(object = "KMeansModel"),
#' excepting that at most one value may be 0. The class with
largest value p/t is predicted, where p
#' is the original probability of that class and t is the
class's threshold.
#' @param weightCol The weight column name.
-#' @param aggregationDepth depth for treeAggregate (>= 2). If the dimensions
of features or the number of partitions
-#' are large, this param could be adjusted to a larger
size.
#' @param probabilityCol column name for predicted class conditional
probabilities.
#' @param ... additional arguments passed to the method.
#' @return \code{spark.logit} returns a fitted logistic regression model
@@ -746,45 +744,35 @@ setMethod("predict", signature(object = "KMeansModel"),
#' \dontrun{
#' sparkR.session()
#' # binary logistic regression
-#' label <- c(0.0, 0.0, 0.0, 1.0, 1.0)
-#' features <- c(1.1419053, 0.9194079, -0.9498666, -1.1069903, 0.2809776)
-#' binary_data <- as.data.frame(cbind(label, features))
-#' binary_df <- createDataFrame(binary_data)
-#' blr_model <- spark.logit(binary_df, label ~ features, thresholds = 1.0)
-#' blr_predict <- collect(select(predict(blr_model, binary_df), "prediction"))
-#'
-#' # summary of binary logistic regression
-#' blr_summary <- summary(blr_model)
-#' blr_fmeasure <- collect(select(blr_summary$fMeasureByThreshold,
"threshold", "F-Measure"))
+#' df <- createDataFrame(iris)
+#' training <- df[df$Species %in% c("versicolor", "virginica"), ]
+#' model <- spark.logit(training, Species ~ ., regParam = 0.5)
+#' summary <- summary(model)
+#'
+#' # fitted values on training data
+#' fitted <- predict(model, training)
+#'
#'
spark git commit: [SPARK-18705][ML][DOC] Update user guide to reflect one pass solver for L1 and elastic-net
Repository: spark
Updated Branches:
refs/heads/master 9ab725eab -> 82253617f
[SPARK-18705][ML][DOC] Update user guide to reflect one pass solver for L1 and
elastic-net
## What changes were proposed in this pull request?
WeightedLeastSquares now supports L1 and elastic net penalties and has an
additional solver option: QuasiNewton. The docs are updated to reflect this
change.
## How was this patch tested?
Docs only. Generated documentation to make sure Latex looks ok.
Author: sethah
Closes #16139 from sethah/SPARK-18705.
Project: http://git-wip-us.apache.org/repos/asf/spark/repo
Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/82253617
Tree: http://git-wip-us.apache.org/repos/asf/spark/tree/82253617
Diff: http://git-wip-us.apache.org/repos/asf/spark/diff/82253617
Branch: refs/heads/master
Commit: 82253617f5b3cdbd418c48f94e748651ee80077e
Parents: 9ab725e
Author: sethah
Authored: Wed Dec 7 19:41:32 2016 -0800
Committer: Yanbo Liang
Committed: Wed Dec 7 19:41:32 2016 -0800
--
docs/ml-advanced.md | 24
1 file changed, 16 insertions(+), 8 deletions(-)
--
http://git-wip-us.apache.org/repos/asf/spark/blob/82253617/docs/ml-advanced.md
--
diff --git a/docs/ml-advanced.md b/docs/ml-advanced.md
index 12a03d3..2747f2d 100644
--- a/docs/ml-advanced.md
+++ b/docs/ml-advanced.md
@@ -59,17 +59,25 @@ Given $n$ weighted observations $(w_i, a_i, b_i)$:
The number of features for each observation is $m$. We use the following
weighted least squares formulation:
`\[
-minimize_{x}\frac{1}{2} \sum_{i=1}^n \frac{w_i(a_i^T x -b_i)^2}{\sum_{k=1}^n
w_k} + \frac{1}{2}\frac{\lambda}{\delta}\sum_{j=1}^m(\sigma_{j} x_{j})^2
+\min_{\mathbf{x}}\frac{1}{2} \sum_{i=1}^n \frac{w_i(\mathbf{a}_i^T \mathbf{x}
-b_i)^2}{\sum_{k=1}^n w_k} + \frac{\lambda}{\delta}\left[\frac{1}{2}(1 -
\alpha)\sum_{j=1}^m(\sigma_j x_j)^2 + \alpha\sum_{j=1}^m |\sigma_j x_j|\right]
\]`
-where $\lambda$ is the regularization parameter, $\delta$ is the population
standard deviation of the label
+where $\lambda$ is the regularization parameter, $\alpha$ is the elastic-net
mixing parameter, $\delta$ is the population standard deviation of the label
and $\sigma_j$ is the population standard deviation of the j-th feature column.
-This objective function has an analytic solution and it requires only one pass
over the data to collect necessary statistics to solve.
-Unlike the original dataset which can only be stored in a distributed system,
-these statistics can be loaded into memory on a single machine if the number
of features is relatively small, and then we can solve the objective function
through Cholesky factorization on the driver.
+This objective function requires only one pass over the data to collect the
statistics necessary to solve it. For an
+$n \times m$ data matrix, these statistics require only $O(m^2)$ storage and
so can be stored on a single machine when $m$ (the number of features) is
+relatively small. We can then solve the normal equations on a single machine
using local methods like direct Cholesky factorization or iterative
optimization programs.
-WeightedLeastSquares only supports L2 regularization and provides options to
enable or disable regularization and standardization.
-In order to make the normal equation approach efficient, WeightedLeastSquares
requires that the number of features be no more than 4096. For larger problems,
use L-BFGS instead.
+Spark MLlib currently supports two types of solvers for the normal equations:
Cholesky factorization and Quasi-Newton methods (L-BFGS/OWL-QN). Cholesky
factorization
+depends on a positive definite covariance matrix (i.e. columns of the data
matrix must be linearly independent) and will fail if this condition is
violated. Quasi-Newton methods
+are still capable of providing a reasonable solution even when the covariance
matrix is not positive definite, so the normal equation solver can also fall
back to
+Quasi-Newton methods in this case. This fallback is currently always enabled
for the `LinearRegression` and `GeneralizedLinearRegression` estimators.
+
+`WeightedLeastSquares` supports L1, L2, and elastic-net regularization and
provides options to enable or disable regularization and standardization. In
the case where no
+L1 regularization is applied (i.e. $\alpha = 0$), there exists an analytical
solution and either Cholesky or Quasi-Newton solver may be used. When $\alpha >
0$ no analytical
+solution exists and we instead use the Quasi-Newton solver to find the
coefficients iteratively.
+
+In order to make the normal equation approach efficient,
`WeightedLeastSquares` requires that the number of features be no more than
4096. For larger problems, use L-BFGS instead.
## Iteratively reweighted least
spark git commit: [SPARK-18705][ML][DOC] Update user guide to reflect one pass solver for L1 and elastic-net
Repository: spark
Updated Branches:
refs/heads/branch-2.1 617ce3ba7 -> ab865cfd9
[SPARK-18705][ML][DOC] Update user guide to reflect one pass solver for L1 and
elastic-net
## What changes were proposed in this pull request?
WeightedLeastSquares now supports L1 and elastic net penalties and has an
additional solver option: QuasiNewton. The docs are updated to reflect this
change.
## How was this patch tested?
Docs only. Generated documentation to make sure Latex looks ok.
Author: sethah
Closes #16139 from sethah/SPARK-18705.
(cherry picked from commit 82253617f5b3cdbd418c48f94e748651ee80077e)
Signed-off-by: Yanbo Liang
Project: http://git-wip-us.apache.org/repos/asf/spark/repo
Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/ab865cfd
Tree: http://git-wip-us.apache.org/repos/asf/spark/tree/ab865cfd
Diff: http://git-wip-us.apache.org/repos/asf/spark/diff/ab865cfd
Branch: refs/heads/branch-2.1
Commit: ab865cfd9dc87154e7d4fc5d09168868c88db6b0
Parents: 617ce3b
Author: sethah
Authored: Wed Dec 7 19:41:32 2016 -0800
Committer: Yanbo Liang
Committed: Wed Dec 7 19:42:06 2016 -0800
--
docs/ml-advanced.md | 24
1 file changed, 16 insertions(+), 8 deletions(-)
--
http://git-wip-us.apache.org/repos/asf/spark/blob/ab865cfd/docs/ml-advanced.md
--
diff --git a/docs/ml-advanced.md b/docs/ml-advanced.md
index 12a03d3..2747f2d 100644
--- a/docs/ml-advanced.md
+++ b/docs/ml-advanced.md
@@ -59,17 +59,25 @@ Given $n$ weighted observations $(w_i, a_i, b_i)$:
The number of features for each observation is $m$. We use the following
weighted least squares formulation:
`\[
-minimize_{x}\frac{1}{2} \sum_{i=1}^n \frac{w_i(a_i^T x -b_i)^2}{\sum_{k=1}^n
w_k} + \frac{1}{2}\frac{\lambda}{\delta}\sum_{j=1}^m(\sigma_{j} x_{j})^2
+\min_{\mathbf{x}}\frac{1}{2} \sum_{i=1}^n \frac{w_i(\mathbf{a}_i^T \mathbf{x}
-b_i)^2}{\sum_{k=1}^n w_k} + \frac{\lambda}{\delta}\left[\frac{1}{2}(1 -
\alpha)\sum_{j=1}^m(\sigma_j x_j)^2 + \alpha\sum_{j=1}^m |\sigma_j x_j|\right]
\]`
-where $\lambda$ is the regularization parameter, $\delta$ is the population
standard deviation of the label
+where $\lambda$ is the regularization parameter, $\alpha$ is the elastic-net
mixing parameter, $\delta$ is the population standard deviation of the label
and $\sigma_j$ is the population standard deviation of the j-th feature column.
-This objective function has an analytic solution and it requires only one pass
over the data to collect necessary statistics to solve.
-Unlike the original dataset which can only be stored in a distributed system,
-these statistics can be loaded into memory on a single machine if the number
of features is relatively small, and then we can solve the objective function
through Cholesky factorization on the driver.
+This objective function requires only one pass over the data to collect the
statistics necessary to solve it. For an
+$n \times m$ data matrix, these statistics require only $O(m^2)$ storage and
so can be stored on a single machine when $m$ (the number of features) is
+relatively small. We can then solve the normal equations on a single machine
using local methods like direct Cholesky factorization or iterative
optimization programs.
-WeightedLeastSquares only supports L2 regularization and provides options to
enable or disable regularization and standardization.
-In order to make the normal equation approach efficient, WeightedLeastSquares
requires that the number of features be no more than 4096. For larger problems,
use L-BFGS instead.
+Spark MLlib currently supports two types of solvers for the normal equations:
Cholesky factorization and Quasi-Newton methods (L-BFGS/OWL-QN). Cholesky
factorization
+depends on a positive definite covariance matrix (i.e. columns of the data
matrix must be linearly independent) and will fail if this condition is
violated. Quasi-Newton methods
+are still capable of providing a reasonable solution even when the covariance
matrix is not positive definite, so the normal equation solver can also fall
back to
+Quasi-Newton methods in this case. This fallback is currently always enabled
for the `LinearRegression` and `GeneralizedLinearRegression` estimators.
+
+`WeightedLeastSquares` supports L1, L2, and elastic-net regularization and
provides options to enable or disable regularization and standardization. In
the case where no
+L1 regularization is applied (i.e. $\alpha = 0$), there exists an analytical
solution and either Cholesky or Quasi-Newton solver may be used. When $\alpha >
0$ no analytical
+solution exists and we instead use the Quasi-Newton solver to find the
coefficients iteratively.
+
+In order to make the normal equation approach efficient,
`WeightedLeastSquares` requires that the number of f
spark git commit: [SPARK-18326][SPARKR][ML] Review SparkR ML wrappers API for 2.1
Repository: spark
Updated Branches:
refs/heads/branch-2.1 ab865cfd9 -> 1c3f1da82
[SPARK-18326][SPARKR][ML] Review SparkR ML wrappers API for 2.1
## What changes were proposed in this pull request?
Reviewing SparkR ML wrappers API for 2.1 release, mainly two issues:
* Remove ```probabilityCol``` from the argument list of ```spark.logit``` and
```spark.randomForest```. Since it was used when making prediction and should
be an argument of ```predict```, and we will work on this at
[SPARK-18618](https://issues.apache.org/jira/browse/SPARK-18618) in the next
release cycle.
* Fix ```spark.als``` params to make it consistent with MLlib.
## How was this patch tested?
Existing tests.
Author: Yanbo Liang
Closes #16169 from yanboliang/spark-18326.
(cherry picked from commit 97255497d885f0f8ccfc808e868bc8aa5e4d1063)
Signed-off-by: Yanbo Liang
Project: http://git-wip-us.apache.org/repos/asf/spark/repo
Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/1c3f1da8
Tree: http://git-wip-us.apache.org/repos/asf/spark/tree/1c3f1da8
Diff: http://git-wip-us.apache.org/repos/asf/spark/diff/1c3f1da8
Branch: refs/heads/branch-2.1
Commit: 1c3f1da82356426b6b550fee67e66dc82eaf1c85
Parents: ab865cf
Author: Yanbo Liang
Authored: Wed Dec 7 20:23:28 2016 -0800
Committer: Yanbo Liang
Committed: Wed Dec 7 20:23:45 2016 -0800
--
R/pkg/R/mllib.R | 23 +---
R/pkg/inst/tests/testthat/test_mllib.R | 4 ++--
.../spark/ml/r/LogisticRegressionWrapper.scala | 4 +---
.../r/RandomForestClassificationWrapper.scala | 2 --
4 files changed, 13 insertions(+), 20 deletions(-)
--
http://git-wip-us.apache.org/repos/asf/spark/blob/1c3f1da8/R/pkg/R/mllib.R
--
diff --git a/R/pkg/R/mllib.R b/R/pkg/R/mllib.R
index 074e9cb..632e4ad 100644
--- a/R/pkg/R/mllib.R
+++ b/R/pkg/R/mllib.R
@@ -733,7 +733,6 @@ setMethod("predict", signature(object = "KMeansModel"),
#' excepting that at most one value may be 0. The class with
largest value p/t is predicted, where p
#' is the original probability of that class and t is the
class's threshold.
#' @param weightCol The weight column name.
-#' @param probabilityCol column name for predicted class conditional
probabilities.
#' @param ... additional arguments passed to the method.
#' @return \code{spark.logit} returns a fitted logistic regression model
#' @rdname spark.logit
@@ -772,7 +771,7 @@ setMethod("predict", signature(object = "KMeansModel"),
setMethod("spark.logit", signature(data = "SparkDataFrame", formula =
"formula"),
function(data, formula, regParam = 0.0, elasticNetParam = 0.0,
maxIter = 100,
tol = 1E-6, family = "auto", standardization = TRUE,
- thresholds = 0.5, weightCol = NULL, probabilityCol =
"probability") {
+ thresholds = 0.5, weightCol = NULL) {
formula <- paste(deparse(formula), collapse = "")
if (is.null(weightCol)) {
@@ -784,7 +783,7 @@ setMethod("spark.logit", signature(data = "SparkDataFrame",
formula = "formula")
as.numeric(elasticNetParam),
as.integer(maxIter),
as.numeric(tol), as.character(family),
as.logical(standardization),
as.array(thresholds),
-as.character(weightCol),
as.character(probabilityCol))
+as.character(weightCol))
new("LogisticRegressionModel", jobj = jobj)
})
@@ -1425,7 +1424,7 @@ setMethod("predict", signature(object =
"GaussianMixtureModel"),
#' @param userCol column name for user ids. Ids must be (or can be coerced
into) integers.
#' @param itemCol column name for item ids. Ids must be (or can be coerced
into) integers.
#' @param rank rank of the matrix factorization (> 0).
-#' @param reg regularization parameter (>= 0).
+#' @param regParam regularization parameter (>= 0).
#' @param maxIter maximum number of iterations (>= 0).
#' @param nonnegative logical value indicating whether to apply nonnegativity
constraints.
#' @param implicitPrefs logical value indicating whether to use implicit
preference.
@@ -1464,21 +1463,21 @@ setMethod("predict", signature(object =
"GaussianMixtureModel"),
#'
#' # set other arguments
#' modelS <- spark.als(df, "rating", "user", "item", rank = 20,
-#' reg = 0.1, nonnegative = TRUE)
+#' regParam = 0.1, nonnegative = TRUE)
#' statsS <- summary(modelS)
#' }
#' @note spark.als since 2.1.0
setMethod("spark.als", signature(data = "SparkDataFrame"),
function(data, ratingCol = "rating", userCol = "user", itemCol =
"item",
- rank = 10, reg = 0.1, maxI
spark git commit: [SPARK-18326][SPARKR][ML] Review SparkR ML wrappers API for 2.1
Repository: spark
Updated Branches:
refs/heads/master 82253617f -> 97255497d
[SPARK-18326][SPARKR][ML] Review SparkR ML wrappers API for 2.1
## What changes were proposed in this pull request?
Reviewing SparkR ML wrappers API for 2.1 release, mainly two issues:
* Remove ```probabilityCol``` from the argument list of ```spark.logit``` and
```spark.randomForest```. Since it was used when making prediction and should
be an argument of ```predict```, and we will work on this at
[SPARK-18618](https://issues.apache.org/jira/browse/SPARK-18618) in the next
release cycle.
* Fix ```spark.als``` params to make it consistent with MLlib.
## How was this patch tested?
Existing tests.
Author: Yanbo Liang
Closes #16169 from yanboliang/spark-18326.
Project: http://git-wip-us.apache.org/repos/asf/spark/repo
Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/97255497
Tree: http://git-wip-us.apache.org/repos/asf/spark/tree/97255497
Diff: http://git-wip-us.apache.org/repos/asf/spark/diff/97255497
Branch: refs/heads/master
Commit: 97255497d885f0f8ccfc808e868bc8aa5e4d1063
Parents: 8225361
Author: Yanbo Liang
Authored: Wed Dec 7 20:23:28 2016 -0800
Committer: Yanbo Liang
Committed: Wed Dec 7 20:23:28 2016 -0800
--
R/pkg/R/mllib.R | 23 +---
R/pkg/inst/tests/testthat/test_mllib.R | 4 ++--
.../spark/ml/r/LogisticRegressionWrapper.scala | 4 +---
.../r/RandomForestClassificationWrapper.scala | 2 --
4 files changed, 13 insertions(+), 20 deletions(-)
--
http://git-wip-us.apache.org/repos/asf/spark/blob/97255497/R/pkg/R/mllib.R
--
diff --git a/R/pkg/R/mllib.R b/R/pkg/R/mllib.R
index 074e9cb..632e4ad 100644
--- a/R/pkg/R/mllib.R
+++ b/R/pkg/R/mllib.R
@@ -733,7 +733,6 @@ setMethod("predict", signature(object = "KMeansModel"),
#' excepting that at most one value may be 0. The class with
largest value p/t is predicted, where p
#' is the original probability of that class and t is the
class's threshold.
#' @param weightCol The weight column name.
-#' @param probabilityCol column name for predicted class conditional
probabilities.
#' @param ... additional arguments passed to the method.
#' @return \code{spark.logit} returns a fitted logistic regression model
#' @rdname spark.logit
@@ -772,7 +771,7 @@ setMethod("predict", signature(object = "KMeansModel"),
setMethod("spark.logit", signature(data = "SparkDataFrame", formula =
"formula"),
function(data, formula, regParam = 0.0, elasticNetParam = 0.0,
maxIter = 100,
tol = 1E-6, family = "auto", standardization = TRUE,
- thresholds = 0.5, weightCol = NULL, probabilityCol =
"probability") {
+ thresholds = 0.5, weightCol = NULL) {
formula <- paste(deparse(formula), collapse = "")
if (is.null(weightCol)) {
@@ -784,7 +783,7 @@ setMethod("spark.logit", signature(data = "SparkDataFrame",
formula = "formula")
as.numeric(elasticNetParam),
as.integer(maxIter),
as.numeric(tol), as.character(family),
as.logical(standardization),
as.array(thresholds),
-as.character(weightCol),
as.character(probabilityCol))
+as.character(weightCol))
new("LogisticRegressionModel", jobj = jobj)
})
@@ -1425,7 +1424,7 @@ setMethod("predict", signature(object =
"GaussianMixtureModel"),
#' @param userCol column name for user ids. Ids must be (or can be coerced
into) integers.
#' @param itemCol column name for item ids. Ids must be (or can be coerced
into) integers.
#' @param rank rank of the matrix factorization (> 0).
-#' @param reg regularization parameter (>= 0).
+#' @param regParam regularization parameter (>= 0).
#' @param maxIter maximum number of iterations (>= 0).
#' @param nonnegative logical value indicating whether to apply nonnegativity
constraints.
#' @param implicitPrefs logical value indicating whether to use implicit
preference.
@@ -1464,21 +1463,21 @@ setMethod("predict", signature(object =
"GaussianMixtureModel"),
#'
#' # set other arguments
#' modelS <- spark.als(df, "rating", "user", "item", rank = 20,
-#' reg = 0.1, nonnegative = TRUE)
+#' regParam = 0.1, nonnegative = TRUE)
#' statsS <- summary(modelS)
#' }
#' @note spark.als since 2.1.0
setMethod("spark.als", signature(data = "SparkDataFrame"),
function(data, ratingCol = "rating", userCol = "user", itemCol =
"item",
- rank = 10, reg = 0.1, maxIter = 10, nonnegative = FALSE,
+ rank = 10, regParam = 0.1, maxIter = 10, nonnegative =
spark git commit: [SPARK-18325][SPARKR][ML] SparkR ML wrappers example code and user guide
Repository: spark
Updated Branches:
refs/heads/master b47b892e4 -> 9bf8f3cd4
[SPARK-18325][SPARKR][ML] SparkR ML wrappers example code and user guide
## What changes were proposed in this pull request?
* Add all R examples for ML wrappers which were added during 2.1 release cycle.
* Split the whole ```ml.R``` example file into individual example for each
algorithm, which will be convenient for users to rerun them.
* Add corresponding examples to ML user guide.
* Update ML section of SparkR user guide.
Note: MLlib Scala/Java/Python examples will be consistent, however, SparkR
examples may different from them, since R users may use the algorithms in a
different way, for example, using R ```formula``` to specify ```featuresCol```
and ```labelCol```.
## How was this patch tested?
Run all examples manually.
Author: Yanbo Liang
Closes #16148 from yanboliang/spark-18325.
Project: http://git-wip-us.apache.org/repos/asf/spark/repo
Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/9bf8f3cd
Tree: http://git-wip-us.apache.org/repos/asf/spark/tree/9bf8f3cd
Diff: http://git-wip-us.apache.org/repos/asf/spark/diff/9bf8f3cd
Branch: refs/heads/master
Commit: 9bf8f3cd4f62f921c32fb50b8abf49576a80874f
Parents: b47b892
Author: Yanbo Liang
Authored: Thu Dec 8 06:19:38 2016 -0800
Committer: Yanbo Liang
Committed: Thu Dec 8 06:19:38 2016 -0800
--
docs/ml-classification-regression.md | 67 +++-
docs/ml-clustering.md| 18 +++-
docs/ml-collaborative-filtering.md | 8 ++
docs/sparkr.md | 46
examples/src/main/r/ml.R | 148 --
examples/src/main/r/ml/als.R | 45
examples/src/main/r/ml/gaussianMixture.R | 42
examples/src/main/r/ml/gbt.R | 63 +++
examples/src/main/r/ml/glm.R | 57 ++
examples/src/main/r/ml/isoreg.R | 42
examples/src/main/r/ml/kmeans.R | 44
examples/src/main/r/ml/kstest.R | 39 +++
examples/src/main/r/ml/lda.R | 46
examples/src/main/r/ml/logit.R | 63 +++
examples/src/main/r/ml/ml.R | 65 +++
examples/src/main/r/ml/mlp.R | 48 +
examples/src/main/r/ml/naiveBayes.R | 41 +++
examples/src/main/r/ml/randomForest.R| 63 +++
examples/src/main/r/ml/survreg.R | 43
19 files changed, 810 insertions(+), 178 deletions(-)
--
http://git-wip-us.apache.org/repos/asf/spark/blob/9bf8f3cd/docs/ml-classification-regression.md
--
diff --git a/docs/ml-classification-regression.md
b/docs/ml-classification-regression.md
index bb9390f..782ee58 100644
--- a/docs/ml-classification-regression.md
+++ b/docs/ml-classification-regression.md
@@ -75,6 +75,13 @@ More details on parameters can be found in the [Python API
documentation](api/py
{% include_example python/ml/logistic_regression_with_elastic_net.py %}
+
+
+More details on parameters can be found in the [R API
documentation](api/R/spark.logit.html).
+
+{% include_example binomial r/ml/logit.R %}
+
+
The `spark.ml` implementation of logistic regression also supports
@@ -171,6 +178,13 @@ model with elastic net regularization.
{% include_example
python/ml/multiclass_logistic_regression_with_elastic_net.py %}
+
+
+More details on parameters can be found in the [R API
documentation](api/R/spark.logit.html).
+
+{% include_example multinomial r/ml/logit.R %}
+
+
@@ -242,6 +256,14 @@ Refer to the [Python API
docs](api/python/pyspark.ml.html#pyspark.ml.classificat
{% include_example python/ml/random_forest_classifier_example.py %}
+
+
+
+Refer to the [R API docs](api/R/spark.randomForest.html) for more details.
+
+{% include_example classification r/ml/randomForest.R %}
+
+
## Gradient-boosted tree classifier
@@ -275,6 +297,14 @@ Refer to the [Python API
docs](api/python/pyspark.ml.html#pyspark.ml.classificat
{% include_example python/ml/gradient_boosted_tree_classifier_example.py %}
+
+
+
+Refer to the [R API docs](api/R/spark.gbt.html) for more details.
+
+{% include_example classification r/ml/gbt.R %}
+
+
## Multilayer perceptron classifier
@@ -324,6 +354,13 @@ Refer to the [Python API
docs](api/python/pyspark.ml.html#pyspark.ml.classificat
{% include_example python/ml/multilayer_perceptron_classification.py %}
+
+
+Refer to the [R API docs](api/R/spark.mlp.html) for more details.
+
+{% include_example r/ml/mlp.R %}
+
+
@@ -400,7 +437,7 @@ Refer to the [Python API
docs](api/python/pyspark.ml.html#pyspark.ml.classificat
Refer to the [R API docs](api/R/spark.naiveBayes.html) for more details.
-{% include_example naiveBayes r/ml.R %}
+{%
spark git commit: [SPARK-18325][SPARKR][ML] SparkR ML wrappers example code and user guide
Repository: spark
Updated Branches:
refs/heads/branch-2.1 48aa6775d -> 9095c152e
[SPARK-18325][SPARKR][ML] SparkR ML wrappers example code and user guide
## What changes were proposed in this pull request?
* Add all R examples for ML wrappers which were added during 2.1 release cycle.
* Split the whole ```ml.R``` example file into individual example for each
algorithm, which will be convenient for users to rerun them.
* Add corresponding examples to ML user guide.
* Update ML section of SparkR user guide.
Note: MLlib Scala/Java/Python examples will be consistent, however, SparkR
examples may different from them, since R users may use the algorithms in a
different way, for example, using R ```formula``` to specify ```featuresCol```
and ```labelCol```.
## How was this patch tested?
Run all examples manually.
Author: Yanbo Liang
Closes #16148 from yanboliang/spark-18325.
(cherry picked from commit 9bf8f3cd4f62f921c32fb50b8abf49576a80874f)
Signed-off-by: Yanbo Liang
Project: http://git-wip-us.apache.org/repos/asf/spark/repo
Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/9095c152
Tree: http://git-wip-us.apache.org/repos/asf/spark/tree/9095c152
Diff: http://git-wip-us.apache.org/repos/asf/spark/diff/9095c152
Branch: refs/heads/branch-2.1
Commit: 9095c152e7fedf469dcc4887f5b6a1882cd74c28
Parents: 48aa677
Author: Yanbo Liang
Authored: Thu Dec 8 06:19:38 2016 -0800
Committer: Yanbo Liang
Committed: Thu Dec 8 06:20:28 2016 -0800
--
docs/ml-classification-regression.md | 67 +++-
docs/ml-clustering.md| 18 +++-
docs/ml-collaborative-filtering.md | 8 ++
docs/sparkr.md | 46
examples/src/main/r/ml.R | 148 --
examples/src/main/r/ml/als.R | 45
examples/src/main/r/ml/gaussianMixture.R | 42
examples/src/main/r/ml/gbt.R | 63 +++
examples/src/main/r/ml/glm.R | 57 ++
examples/src/main/r/ml/isoreg.R | 42
examples/src/main/r/ml/kmeans.R | 44
examples/src/main/r/ml/kstest.R | 39 +++
examples/src/main/r/ml/lda.R | 46
examples/src/main/r/ml/logit.R | 63 +++
examples/src/main/r/ml/ml.R | 65 +++
examples/src/main/r/ml/mlp.R | 48 +
examples/src/main/r/ml/naiveBayes.R | 41 +++
examples/src/main/r/ml/randomForest.R| 63 +++
examples/src/main/r/ml/survreg.R | 43
19 files changed, 810 insertions(+), 178 deletions(-)
--
http://git-wip-us.apache.org/repos/asf/spark/blob/9095c152/docs/ml-classification-regression.md
--
diff --git a/docs/ml-classification-regression.md
b/docs/ml-classification-regression.md
index 557a53c..2ffea64 100644
--- a/docs/ml-classification-regression.md
+++ b/docs/ml-classification-regression.md
@@ -75,6 +75,13 @@ More details on parameters can be found in the [Python API
documentation](api/py
{% include_example python/ml/logistic_regression_with_elastic_net.py %}
+
+
+More details on parameters can be found in the [R API
documentation](api/R/spark.logit.html).
+
+{% include_example binomial r/ml/logit.R %}
+
+
The `spark.ml` implementation of logistic regression also supports
@@ -171,6 +178,13 @@ model with elastic net regularization.
{% include_example
python/ml/multiclass_logistic_regression_with_elastic_net.py %}
+
+
+More details on parameters can be found in the [R API
documentation](api/R/spark.logit.html).
+
+{% include_example multinomial r/ml/logit.R %}
+
+
@@ -242,6 +256,14 @@ Refer to the [Python API
docs](api/python/pyspark.ml.html#pyspark.ml.classificat
{% include_example python/ml/random_forest_classifier_example.py %}
+
+
+
+Refer to the [R API docs](api/R/spark.randomForest.html) for more details.
+
+{% include_example classification r/ml/randomForest.R %}
+
+
## Gradient-boosted tree classifier
@@ -275,6 +297,14 @@ Refer to the [Python API
docs](api/python/pyspark.ml.html#pyspark.ml.classificat
{% include_example python/ml/gradient_boosted_tree_classifier_example.py %}
+
+
+
+Refer to the [R API docs](api/R/spark.gbt.html) for more details.
+
+{% include_example classification r/ml/gbt.R %}
+
+
## Multilayer perceptron classifier
@@ -324,6 +354,13 @@ Refer to the [Python API
docs](api/python/pyspark.ml.html#pyspark.ml.classificat
{% include_example python/ml/multilayer_perceptron_classification.py %}
+
+
+Refer to the [R API docs](api/R/spark.mlp.html) for more details.
+
+{% include_example r/ml/mlp.R %}
+
+
@@ -400,7 +437,7 @@ Refer to the [Python API
docs](api/python/pyspark.ml.html#pyspark.ml.classificat
Refer to the [
spark git commit: [MINOR][SPARKR] fix kstest example error and add unit test
Repository: spark
Updated Branches:
refs/heads/master e104e55c1 -> f2ddabfa0
[MINOR][SPARKR] fix kstest example error and add unit test
## What changes were proposed in this pull request?
While adding vignettes for kstest, I found some errors in the example:
1. There is a typo of kstest;
2. print.summary.KStest doesn't work with the example;
Fix the example errors;
Add a new unit test for print.summary.KStest;
## How was this patch tested?
Manual test;
Add new unit test;
Author: wm...@hotmail.com
Closes #16259 from wangmiao1981/ks.
Project: http://git-wip-us.apache.org/repos/asf/spark/repo
Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/f2ddabfa
Tree: http://git-wip-us.apache.org/repos/asf/spark/tree/f2ddabfa
Diff: http://git-wip-us.apache.org/repos/asf/spark/diff/f2ddabfa
Branch: refs/heads/master
Commit: f2ddabfa09fda26ff0391d026dd67545dab33e01
Parents: e104e55
Author: wm...@hotmail.com
Authored: Tue Dec 13 18:52:05 2016 -0800
Committer: Yanbo Liang
Committed: Tue Dec 13 18:52:05 2016 -0800
--
R/pkg/R/mllib.R| 4 ++--
R/pkg/inst/tests/testthat/test_mllib.R | 6 ++
2 files changed, 8 insertions(+), 2 deletions(-)
--
http://git-wip-us.apache.org/repos/asf/spark/blob/f2ddabfa/R/pkg/R/mllib.R
--
diff --git a/R/pkg/R/mllib.R b/R/pkg/R/mllib.R
index 5df843c..d736bbb 100644
--- a/R/pkg/R/mllib.R
+++ b/R/pkg/R/mllib.R
@@ -1595,14 +1595,14 @@ setMethod("write.ml", signature(object = "ALSModel",
path = "character"),
#' \dontrun{
#' data <- data.frame(test = c(0.1, 0.15, 0.2, 0.3, 0.25))
#' df <- createDataFrame(data)
-#' test <- spark.ktest(df, "test", "norm", c(0, 1))
+#' test <- spark.kstest(df, "test", "norm", c(0, 1))
#'
#' # get a summary of the test result
#' testSummary <- summary(test)
#' testSummary
#'
#' # print out the summary in an organized way
-#' print.summary.KSTest(test)
+#' print.summary.KSTest(testSummary)
#' }
#' @note spark.kstest since 2.1.0
setMethod("spark.kstest", signature(data = "SparkDataFrame"),
http://git-wip-us.apache.org/repos/asf/spark/blob/f2ddabfa/R/pkg/inst/tests/testthat/test_mllib.R
--
diff --git a/R/pkg/inst/tests/testthat/test_mllib.R
b/R/pkg/inst/tests/testthat/test_mllib.R
index 986af4a..0f0d831 100644
--- a/R/pkg/inst/tests/testthat/test_mllib.R
+++ b/R/pkg/inst/tests/testthat/test_mllib.R
@@ -986,6 +986,12 @@ test_that("spark.kstest", {
expect_equal(stats$p.value, rStats$p.value, tolerance = 1e-4)
expect_equal(stats$statistic, unname(rStats$statistic), tolerance = 1e-4)
expect_match(capture.output(stats)[1], "Kolmogorov-Smirnov test summary:")
+
+ # Test print.summary.KSTest
+ printStats <- capture.output(print.summary.KSTest(stats))
+ expect_match(printStats[1], "Kolmogorov-Smirnov test summary:")
+ expect_match(printStats[5],
+ "Low presumption against null hypothesis: Sample follows
theoretical distribution. ")
})
test_that("spark.randomForest", {
-
To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org
For additional commands, e-mail: commits-h...@spark.apache.org
spark git commit: [MINOR][SPARKR] fix kstest example error and add unit test
Repository: spark
Updated Branches:
refs/heads/branch-2.1 019d1fa3d -> 8ef005931
[MINOR][SPARKR] fix kstest example error and add unit test
## What changes were proposed in this pull request?
While adding vignettes for kstest, I found some errors in the example:
1. There is a typo of kstest;
2. print.summary.KStest doesn't work with the example;
Fix the example errors;
Add a new unit test for print.summary.KStest;
## How was this patch tested?
Manual test;
Add new unit test;
Author: wm...@hotmail.com
Closes #16259 from wangmiao1981/ks.
(cherry picked from commit f2ddabfa09fda26ff0391d026dd67545dab33e01)
Signed-off-by: Yanbo Liang
Project: http://git-wip-us.apache.org/repos/asf/spark/repo
Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/8ef00593
Tree: http://git-wip-us.apache.org/repos/asf/spark/tree/8ef00593
Diff: http://git-wip-us.apache.org/repos/asf/spark/diff/8ef00593
Branch: refs/heads/branch-2.1
Commit: 8ef005931a242d087f4879805571be0660aefaf9
Parents: 019d1fa
Author: wm...@hotmail.com
Authored: Tue Dec 13 18:52:05 2016 -0800
Committer: Yanbo Liang
Committed: Tue Dec 13 18:52:22 2016 -0800
--
R/pkg/R/mllib.R| 4 ++--
R/pkg/inst/tests/testthat/test_mllib.R | 6 ++
2 files changed, 8 insertions(+), 2 deletions(-)
--
http://git-wip-us.apache.org/repos/asf/spark/blob/8ef00593/R/pkg/R/mllib.R
--
diff --git a/R/pkg/R/mllib.R b/R/pkg/R/mllib.R
index 5df843c..d736bbb 100644
--- a/R/pkg/R/mllib.R
+++ b/R/pkg/R/mllib.R
@@ -1595,14 +1595,14 @@ setMethod("write.ml", signature(object = "ALSModel",
path = "character"),
#' \dontrun{
#' data <- data.frame(test = c(0.1, 0.15, 0.2, 0.3, 0.25))
#' df <- createDataFrame(data)
-#' test <- spark.ktest(df, "test", "norm", c(0, 1))
+#' test <- spark.kstest(df, "test", "norm", c(0, 1))
#'
#' # get a summary of the test result
#' testSummary <- summary(test)
#' testSummary
#'
#' # print out the summary in an organized way
-#' print.summary.KSTest(test)
+#' print.summary.KSTest(testSummary)
#' }
#' @note spark.kstest since 2.1.0
setMethod("spark.kstest", signature(data = "SparkDataFrame"),
http://git-wip-us.apache.org/repos/asf/spark/blob/8ef00593/R/pkg/inst/tests/testthat/test_mllib.R
--
diff --git a/R/pkg/inst/tests/testthat/test_mllib.R
b/R/pkg/inst/tests/testthat/test_mllib.R
index 46dffe3..40c0446 100644
--- a/R/pkg/inst/tests/testthat/test_mllib.R
+++ b/R/pkg/inst/tests/testthat/test_mllib.R
@@ -986,6 +986,12 @@ test_that("spark.kstest", {
expect_equal(stats$p.value, rStats$p.value, tolerance = 1e-4)
expect_equal(stats$statistic, unname(rStats$statistic), tolerance = 1e-4)
expect_match(capture.output(stats)[1], "Kolmogorov-Smirnov test summary:")
+
+ # Test print.summary.KSTest
+ printStats <- capture.output(print.summary.KSTest(stats))
+ expect_match(printStats[1], "Kolmogorov-Smirnov test summary:")
+ expect_match(printStats[5],
+ "Low presumption against null hypothesis: Sample follows
theoretical distribution. ")
})
test_that("spark.randomForest", {
-
To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org
For additional commands, e-mail: commits-h...@spark.apache.org
spark git commit: [SPARK-17645][MLLIB][ML] add feature selector method based on: False Discovery Rate (FDR) and Family wise error rate (FWE)
Repository: spark Updated Branches: refs/heads/master 2af8b5cff -> 79ff85363 [SPARK-17645][MLLIB][ML] add feature selector method based on: False Discovery Rate (FDR) and Family wise error rate (FWE) ## What changes were proposed in this pull request? Univariate feature selection works by selecting the best features based on univariate statistical tests. FDR and FWE are a popular univariate statistical test for feature selection. In 2005, the Benjamini and Hochberg paper on FDR was identified as one of the 25 most-cited statistical papers. The FDR uses the Benjamini-Hochberg procedure in this PR. https://en.wikipedia.org/wiki/False_discovery_rate. In statistics, FWE is the probability of making one or more false discoveries, or type I errors, among all the hypotheses when performing multiple hypotheses tests. https://en.wikipedia.org/wiki/Family-wise_error_rate We add FDR and FWE methods for ChiSqSelector in this PR, like it is implemented in scikit-learn. http://scikit-learn.org/stable/modules/feature_selection.html#univariate-feature-selection ## How was this patch tested? ut will be added soon (Please explain how this patch was tested. E.g. unit tests, integration tests, manual tests) (If this patch involves UI changes, please attach a screenshot; otherwise, remove this) Author: Peng Author: Peng, Meng Closes #15212 from mpjlu/fdr_fwe. Project: http://git-wip-us.apache.org/repos/asf/spark/repo Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/79ff8536 Tree: http://git-wip-us.apache.org/repos/asf/spark/tree/79ff8536 Diff: http://git-wip-us.apache.org/repos/asf/spark/diff/79ff8536 Branch: refs/heads/master Commit: 79ff8536315aef97ee940c52d71cd8de777c7ce6 Parents: 2af8b5c Author: Peng Authored: Wed Dec 28 00:49:36 2016 -0800 Committer: Yanbo Liang Committed: Wed Dec 28 00:49:36 2016 -0800 -- docs/ml-features.md | 6 +- docs/mllib-feature-extraction.md| 4 +- .../apache/spark/ml/feature/ChiSqSelector.scala | 48 +- .../spark/mllib/api/python/PythonMLLibAPI.scala | 4 + .../spark/mllib/feature/ChiSqSelector.scala | 62 ++-- .../spark/ml/feature/ChiSqSelectorSuite.scala | 6 + .../mllib/feature/ChiSqSelectorSuite.scala | 147 +++ python/pyspark/ml/feature.py| 74 +- python/pyspark/mllib/feature.py | 50 ++- 9 files changed, 337 insertions(+), 64 deletions(-) -- http://git-wip-us.apache.org/repos/asf/spark/blob/79ff8536/docs/ml-features.md -- diff --git a/docs/ml-features.md b/docs/ml-features.md index ca1ccc4..1d34497 100644 --- a/docs/ml-features.md +++ b/docs/ml-features.md @@ -1423,12 +1423,12 @@ for more details on the API. `ChiSqSelector` stands for Chi-Squared feature selection. It operates on labeled data with categorical features. ChiSqSelector uses the [Chi-Squared test of independence](https://en.wikipedia.org/wiki/Chi-squared_test) to decide which -features to choose. It supports three selection methods: `numTopFeatures`, `percentile`, `fpr`: - +features to choose. It supports five selection methods: `numTopFeatures`, `percentile`, `fpr`, `fdr`, `fwe`: * `numTopFeatures` chooses a fixed number of top features according to a chi-squared test. This is akin to yielding the features with the most predictive power. * `percentile` is similar to `numTopFeatures` but chooses a fraction of all features instead of a fixed number. * `fpr` chooses all features whose p-value is below a threshold, thus controlling the false positive rate of selection. - +* `fdr` uses the [Benjamini-Hochberg procedure](https://en.wikipedia.org/wiki/False_discovery_rate#Benjamini.E2.80.93Hochberg_procedure) to choose all features whose false discovery rate is below a threshold. +* `fwe` chooses all features whose p-values is below a threshold, thus controlling the family-wise error rate of selection. By default, the selection method is `numTopFeatures`, with the default number of top features set to 50. The user can choose a selection method using `setSelectorType`. http://git-wip-us.apache.org/repos/asf/spark/blob/79ff8536/docs/mllib-feature-extraction.md -- diff --git a/docs/mllib-feature-extraction.md b/docs/mllib-feature-extraction.md index 42568c3..acd2894 100644 --- a/docs/mllib-feature-extraction.md +++ b/docs/mllib-feature-extraction.md @@ -227,11 +227,13 @@ both speed and statistical learning behavior. [`ChiSqSelector`](api/scala/index.html#org.apache.spark.mllib.feature.ChiSqSelector) implements Chi-Squared feature selection. It operates on labeled data with categorical features. ChiSqSelector uses the [Chi-Squared test of independence](https://e
spark git commit: [MINOR][ML] Correct test cases of LoR raw2prediction & probability2prediction.
Repository: spark
Updated Branches:
refs/heads/master 79ff85363 -> 9cff67f34
[MINOR][ML] Correct test cases of LoR raw2prediction & probability2prediction.
## What changes were proposed in this pull request?
Correct test cases of ```LogisticRegression``` raw2prediction &
probability2prediction.
## How was this patch tested?
Changed unit tests.
Author: Yanbo Liang
Closes #16407 from yanboliang/raw-probability.
Project: http://git-wip-us.apache.org/repos/asf/spark/repo
Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/9cff67f3
Tree: http://git-wip-us.apache.org/repos/asf/spark/tree/9cff67f3
Diff: http://git-wip-us.apache.org/repos/asf/spark/diff/9cff67f3
Branch: refs/heads/master
Commit: 9cff67f3465bc6ffe1b5abee9501e3c17f8fd194
Parents: 79ff853
Author: Yanbo Liang
Authored: Wed Dec 28 01:24:18 2016 -0800
Committer: Yanbo Liang
Committed: Wed Dec 28 01:24:18 2016 -0800
--
.../LogisticRegressionSuite.scala | 20 ++--
1 file changed, 18 insertions(+), 2 deletions(-)
--
http://git-wip-us.apache.org/repos/asf/spark/blob/9cff67f3/mllib/src/test/scala/org/apache/spark/ml/classification/LogisticRegressionSuite.scala
--
diff --git
a/mllib/src/test/scala/org/apache/spark/ml/classification/LogisticRegressionSuite.scala
b/mllib/src/test/scala/org/apache/spark/ml/classification/LogisticRegressionSuite.scala
index 9c4c59a..f8bcbee 100644
---
a/mllib/src/test/scala/org/apache/spark/ml/classification/LogisticRegressionSuite.scala
+++
b/mllib/src/test/scala/org/apache/spark/ml/classification/LogisticRegressionSuite.scala
@@ -359,8 +359,16 @@ class LogisticRegressionSuite
assert(pred == predFromProb)
}
-// force it to use probability2prediction
+// force it to use raw2prediction
model.setProbabilityCol("")
+val resultsUsingRaw2Predict =
+
model.transform(smallMultinomialDataset).select("prediction").as[Double].collect()
+
resultsUsingRaw2Predict.zip(results.select("prediction").as[Double].collect()).foreach
{
+ case (pred1, pred2) => assert(pred1 === pred2)
+}
+
+// force it to use probability2prediction
+model.setRawPredictionCol("")
val resultsUsingProb2Predict =
model.transform(smallMultinomialDataset).select("prediction").as[Double].collect()
resultsUsingProb2Predict.zip(results.select("prediction").as[Double].collect()).foreach
{
@@ -405,8 +413,16 @@ class LogisticRegressionSuite
assert(pred == predFromProb)
}
-// force it to use probability2prediction
+// force it to use raw2prediction
model.setProbabilityCol("")
+val resultsUsingRaw2Predict =
+
model.transform(smallBinaryDataset).select("prediction").as[Double].collect()
+
resultsUsingRaw2Predict.zip(results.select("prediction").as[Double].collect()).foreach
{
+ case (pred1, pred2) => assert(pred1 === pred2)
+}
+
+// force it to use probability2prediction
+model.setRawPredictionCol("")
val resultsUsingProb2Predict =
model.transform(smallBinaryDataset).select("prediction").as[Double].collect()
resultsUsingProb2Predict.zip(results.select("prediction").as[Double].collect()).foreach
{
-
To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org
For additional commands, e-mail: commits-h...@spark.apache.org
spark git commit: [SPARK-17772][ML][TEST] Add test functions for ML sample weights
Repository: spark
Updated Branches:
refs/heads/master d7bce3bd3 -> 6a475ae46
[SPARK-17772][ML][TEST] Add test functions for ML sample weights
## What changes were proposed in this pull request?
More and more ML algos are accepting sample weights, and they have been tested
rather heterogeneously and with code duplication. This patch adds extensible
helper methods to `MLTestingUtils` that can be reused by various algorithms
accepting sample weights. Up to now, there seems to be a few tests that have
been implemented commonly:
* Check that oversampling is the same as giving the instances sample weights
proportional to the number of samples
* Check that outliers with tiny sample weights do not affect the algorithm's
performance
This patch adds an additional test:
* Check that algorithms are invariant to constant scaling of the sample
weights. i.e. uniform sample weights with `w_i = 1.0` is effectively the same
as uniform sample weights with `w_i = 1` or `w_i = 0.0001`
The instances of these tests occurred in LinearRegression, NaiveBayes, and
LogisticRegression. Those tests have been removed/modified to use the new
helper methods. These helper functions will be of use when
[SPARK-9478](https://issues.apache.org/jira/browse/SPARK-9478) is implemented.
## How was this patch tested?
This patch only involves modifying test suites.
## Other notes
Both IsotonicRegression and GeneralizedLinearRegression also extend
`HasWeightCol`. I did not modify these test suites because it will make this
patch easier to review, and because they did not duplicate the same tests as
the three suites that were modified. If we want to change them later, we can
create a JIRA for it now, but it's open for debate.
Author: sethah
Closes #15721 from sethah/SPARK-17772.
Project: http://git-wip-us.apache.org/repos/asf/spark/repo
Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/6a475ae4
Tree: http://git-wip-us.apache.org/repos/asf/spark/tree/6a475ae4
Diff: http://git-wip-us.apache.org/repos/asf/spark/diff/6a475ae4
Branch: refs/heads/master
Commit: 6a475ae466a7ce28d507244bf6db91be06ed81ef
Parents: d7bce3b
Author: sethah
Authored: Wed Dec 28 07:01:14 2016 -0800
Committer: Yanbo Liang
Committed: Wed Dec 28 07:01:14 2016 -0800
--
.../LogisticRegressionSuite.scala | 60 +++---
.../ml/classification/NaiveBayesSuite.scala | 81 +
.../ml/regression/LinearRegressionSuite.scala | 120 ++-
.../apache/spark/ml/util/MLTestingUtils.scala | 111 +++--
4 files changed, 154 insertions(+), 218 deletions(-)
--
http://git-wip-us.apache.org/repos/asf/spark/blob/6a475ae4/mllib/src/test/scala/org/apache/spark/ml/classification/LogisticRegressionSuite.scala
--
diff --git
a/mllib/src/test/scala/org/apache/spark/ml/classification/LogisticRegressionSuite.scala
b/mllib/src/test/scala/org/apache/spark/ml/classification/LogisticRegressionSuite.scala
index f8bcbee..1308210 100644
---
a/mllib/src/test/scala/org/apache/spark/ml/classification/LogisticRegressionSuite.scala
+++
b/mllib/src/test/scala/org/apache/spark/ml/classification/LogisticRegressionSuite.scala
@@ -1836,52 +1836,24 @@ class LogisticRegressionSuite
.forall(x => x(0) >= x(1)))
}
- test("binary logistic regression with weighted data") {
-val numClasses = 2
-val numPoints = 40
-val outlierData =
MLTestingUtils.genClassificationInstancesWithWeightedOutliers(spark,
- numClasses, numPoints)
-val testData = Array.tabulate[LabeledPoint](numClasses) { i =>
- LabeledPoint(i.toDouble, Vectors.dense(i.toDouble))
-}.toSeq.toDF()
-val lr = new
LogisticRegression().setFamily("binomial").setWeightCol("weight")
-val model = lr.fit(outlierData)
-val results = model.transform(testData).select("label",
"prediction").collect()
-
-// check that the predictions are the one to one mapping
-results.foreach { case Row(label: Double, pred: Double) =>
- assert(label === pred)
+ test("logistic regression with sample weights") {
+def modelEquals(m1: LogisticRegressionModel, m2: LogisticRegressionModel):
Unit = {
+ assert(m1.coefficientMatrix ~== m2.coefficientMatrix absTol 0.05)
+ assert(m1.interceptVector ~== m2.interceptVector absTol 0.05)
}
-val (overSampledData, weightedData) =
- MLTestingUtils.genEquivalentOversampledAndWeightedInstances(outlierData,
"label", "features",
-42L)
-val weightedModel = lr.fit(weightedData)
-val overSampledModel = lr.setWeightCol("").fit(overSampledData)
-assert(weightedModel.coefficientMatrix ~==
overSampledModel.coefficientMatrix relTol 0.01)
- }
-
- test("multinomial logistic regression with weighted data") {
-val numClasses = 5
-val numPoints =
spark git commit: [MINOR][ML][MLLIB] Remove work around for breeze sparse matrix.
Repository: spark
Updated Branches:
refs/heads/master cdeb97a8c -> 1b001b520
[MINOR][ML][MLLIB] Remove work around for breeze sparse matrix.
## What changes were proposed in this pull request?
Since we have updated breeze version to 0.12, we should remove work around for
bug of breeze sparse matrix in v0.11.
I checked all mllib code and found this is the only work around for breeze 0.11.
## How was this patch tested?
Existing tests.
Author: Yanbo Liang
Closes #14953 from yanboliang/matrices.
Project: http://git-wip-us.apache.org/repos/asf/spark/repo
Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/1b001b52
Tree: http://git-wip-us.apache.org/repos/asf/spark/tree/1b001b52
Diff: http://git-wip-us.apache.org/repos/asf/spark/diff/1b001b52
Branch: refs/heads/master
Commit: 1b001b5203444cc8d5c4887a30e03e8fb298d17d
Parents: cdeb97a
Author: Yanbo Liang
Authored: Sun Sep 4 05:38:47 2016 -0700
Committer: Yanbo Liang
Committed: Sun Sep 4 05:38:47 2016 -0700
--
.../main/scala/org/apache/spark/ml/linalg/Matrices.scala | 10 +-
.../scala/org/apache/spark/mllib/linalg/Matrices.scala| 10 +-
2 files changed, 2 insertions(+), 18 deletions(-)
--
http://git-wip-us.apache.org/repos/asf/spark/blob/1b001b52/mllib-local/src/main/scala/org/apache/spark/ml/linalg/Matrices.scala
--
diff --git
a/mllib-local/src/main/scala/org/apache/spark/ml/linalg/Matrices.scala
b/mllib-local/src/main/scala/org/apache/spark/ml/linalg/Matrices.scala
index 98080bb..207f662 100644
--- a/mllib-local/src/main/scala/org/apache/spark/ml/linalg/Matrices.scala
+++ b/mllib-local/src/main/scala/org/apache/spark/ml/linalg/Matrices.scala
@@ -846,16 +846,8 @@ object Matrices {
case dm: BDM[Double] =>
new DenseMatrix(dm.rows, dm.cols, dm.data, dm.isTranspose)
case sm: BSM[Double] =>
-// Spark-11507. work around breeze issue 479.
-val mat = if (sm.colPtrs.last != sm.data.length) {
- val matCopy = sm.copy
- matCopy.compact()
- matCopy
-} else {
- sm
-}
// There is no isTranspose flag for sparse matrices in Breeze
-new SparseMatrix(mat.rows, mat.cols, mat.colPtrs, mat.rowIndices,
mat.data)
+new SparseMatrix(sm.rows, sm.cols, sm.colPtrs, sm.rowIndices, sm.data)
case _ =>
throw new UnsupportedOperationException(
s"Do not support conversion from type ${breeze.getClass.getName}.")
http://git-wip-us.apache.org/repos/asf/spark/blob/1b001b52/mllib/src/main/scala/org/apache/spark/mllib/linalg/Matrices.scala
--
diff --git a/mllib/src/main/scala/org/apache/spark/mllib/linalg/Matrices.scala
b/mllib/src/main/scala/org/apache/spark/mllib/linalg/Matrices.scala
index ad882c9..8659cea 100644
--- a/mllib/src/main/scala/org/apache/spark/mllib/linalg/Matrices.scala
+++ b/mllib/src/main/scala/org/apache/spark/mllib/linalg/Matrices.scala
@@ -983,16 +983,8 @@ object Matrices {
case dm: BDM[Double] =>
new DenseMatrix(dm.rows, dm.cols, dm.data, dm.isTranspose)
case sm: BSM[Double] =>
-// Spark-11507. work around breeze issue 479.
-val mat = if (sm.colPtrs.last != sm.data.length) {
- val matCopy = sm.copy
- matCopy.compact()
- matCopy
-} else {
- sm
-}
// There is no isTranspose flag for sparse matrices in Breeze
-new SparseMatrix(mat.rows, mat.cols, mat.colPtrs, mat.rowIndices,
mat.data)
+new SparseMatrix(sm.rows, sm.cols, sm.colPtrs, sm.rowIndices, sm.data)
case _ =>
throw new UnsupportedOperationException(
s"Do not support conversion from type ${breeze.getClass.getName}.")
-
To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org
For additional commands, e-mail: commits-h...@spark.apache.org
spark git commit: [MINOR][ML] Correct weights doc of MultilayerPerceptronClassificationModel.
Repository: spark
Updated Branches:
refs/heads/master 6f13aa7df -> 39d538ddd
[MINOR][ML] Correct weights doc of MultilayerPerceptronClassificationModel.
## What changes were proposed in this pull request?
```weights``` of ```MultilayerPerceptronClassificationModel``` should be the
output weights of layers rather than initial weights, this PR correct it.
## How was this patch tested?
Doc change.
Author: Yanbo Liang
Closes #14967 from yanboliang/mlp-weights.
Project: http://git-wip-us.apache.org/repos/asf/spark/repo
Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/39d538dd
Tree: http://git-wip-us.apache.org/repos/asf/spark/tree/39d538dd
Diff: http://git-wip-us.apache.org/repos/asf/spark/diff/39d538dd
Branch: refs/heads/master
Commit: 39d538dddf7d44bf4603c966d0f7b2c92f1e951a
Parents: 6f13aa7
Author: Yanbo Liang
Authored: Tue Sep 6 03:30:37 2016 -0700
Committer: Yanbo Liang
Committed: Tue Sep 6 03:30:37 2016 -0700
--
.../spark/ml/classification/MultilayerPerceptronClassifier.scala | 2 +-
python/pyspark/ml/classification.py| 2 +-
2 files changed, 2 insertions(+), 2 deletions(-)
--
http://git-wip-us.apache.org/repos/asf/spark/blob/39d538dd/mllib/src/main/scala/org/apache/spark/ml/classification/MultilayerPerceptronClassifier.scala
--
diff --git
a/mllib/src/main/scala/org/apache/spark/ml/classification/MultilayerPerceptronClassifier.scala
b/mllib/src/main/scala/org/apache/spark/ml/classification/MultilayerPerceptronClassifier.scala
index 7264a99..88fe7cb 100644
---
a/mllib/src/main/scala/org/apache/spark/ml/classification/MultilayerPerceptronClassifier.scala
+++
b/mllib/src/main/scala/org/apache/spark/ml/classification/MultilayerPerceptronClassifier.scala
@@ -288,7 +288,7 @@ object MultilayerPerceptronClassifier
*
* @param uid uid
* @param layers array of layer sizes including input and output layers
- * @param weights vector of initial weights for the model that consists of the
weights of layers
+ * @param weights the weights of layers
* @return prediction model
*/
@Since("1.5.0")
http://git-wip-us.apache.org/repos/asf/spark/blob/39d538dd/python/pyspark/ml/classification.py
--
diff --git a/python/pyspark/ml/classification.py
b/python/pyspark/ml/classification.py
index d1522d7..b4c01fd 100644
--- a/python/pyspark/ml/classification.py
+++ b/python/pyspark/ml/classification.py
@@ -1276,7 +1276,7 @@ class MultilayerPerceptronClassificationModel(JavaModel,
JavaPredictionModel, Ja
@since("2.0.0")
def weights(self):
"""
-vector of initial weights for the model that consists of the weights
of layers.
+the weights of layers.
"""
return self._call_java("weights")
-
To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org
For additional commands, e-mail: commits-h...@spark.apache.org
spark git commit: [SPARK-17456][CORE] Utility for parsing Spark versions
Repository: spark
Updated Branches:
refs/heads/master 92ce8d484 -> 65b814bf5
[SPARK-17456][CORE] Utility for parsing Spark versions
## What changes were proposed in this pull request?
This patch adds methods for extracting major and minor versions as Int types in
Scala from a Spark version string.
Motivation: There are many hacks within Spark's codebase to identify and
compare Spark versions. We should add a simple utility to standardize these
code paths, especially since there have been mistakes made in the past. This
will let us add unit tests as well. Currently, I want this functionality to
check Spark versions to provide backwards compatibility for ML model
persistence.
## How was this patch tested?
Unit tests
Author: Joseph K. Bradley
Closes #15017 from jkbradley/version-parsing.
Project: http://git-wip-us.apache.org/repos/asf/spark/repo
Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/65b814bf
Tree: http://git-wip-us.apache.org/repos/asf/spark/tree/65b814bf
Diff: http://git-wip-us.apache.org/repos/asf/spark/diff/65b814bf
Branch: refs/heads/master
Commit: 65b814bf50e92e2e9b622d1602f18bacd217181c
Parents: 92ce8d4
Author: Joseph K. Bradley
Authored: Fri Sep 9 05:35:10 2016 -0700
Committer: Yanbo Liang
Committed: Fri Sep 9 05:35:10 2016 -0700
--
.../org/apache/spark/util/VersionUtils.scala| 52 ++
.../apache/spark/util/VersionUtilsSuite.scala | 76
2 files changed, 128 insertions(+)
--
http://git-wip-us.apache.org/repos/asf/spark/blob/65b814bf/core/src/main/scala/org/apache/spark/util/VersionUtils.scala
--
diff --git a/core/src/main/scala/org/apache/spark/util/VersionUtils.scala
b/core/src/main/scala/org/apache/spark/util/VersionUtils.scala
new file mode 100644
index 000..828153b8
--- /dev/null
+++ b/core/src/main/scala/org/apache/spark/util/VersionUtils.scala
@@ -0,0 +1,52 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.util
+
+/**
+ * Utilities for working with Spark version strings
+ */
+private[spark] object VersionUtils {
+
+ private val majorMinorRegex = """^(\d+)\.(\d+)(\..*)?$""".r
+
+ /**
+ * Given a Spark version string, return the major version number.
+ * E.g., for 2.0.1-SNAPSHOT, return 2.
+ */
+ def majorVersion(sparkVersion: String): Int =
majorMinorVersion(sparkVersion)._1
+
+ /**
+ * Given a Spark version string, return the minor version number.
+ * E.g., for 2.0.1-SNAPSHOT, return 0.
+ */
+ def minorVersion(sparkVersion: String): Int =
majorMinorVersion(sparkVersion)._2
+
+ /**
+ * Given a Spark version string, return the (major version number, minor
version number).
+ * E.g., for 2.0.1-SNAPSHOT, return (2, 0).
+ */
+ def majorMinorVersion(sparkVersion: String): (Int, Int) = {
+majorMinorRegex.findFirstMatchIn(sparkVersion) match {
+ case Some(m) =>
+(m.group(1).toInt, m.group(2).toInt)
+ case None =>
+throw new IllegalArgumentException(s"Spark tried to parse
'$sparkVersion' as a Spark" +
+ s" version string, but it could not find the major and minor version
numbers.")
+}
+ }
+}
http://git-wip-us.apache.org/repos/asf/spark/blob/65b814bf/core/src/test/scala/org/apache/spark/util/VersionUtilsSuite.scala
--
diff --git a/core/src/test/scala/org/apache/spark/util/VersionUtilsSuite.scala
b/core/src/test/scala/org/apache/spark/util/VersionUtilsSuite.scala
new file mode 100644
index 000..aaf79eb
--- /dev/null
+++ b/core/src/test/scala/org/apache/spark/util/VersionUtilsSuite.scala
@@ -0,0 +1,76 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You ma
spark git commit: [SPARK-17464][SPARKR][ML] SparkR spark.als argument reg should be 0.1 by default.
Repository: spark
Updated Branches:
refs/heads/master 65b814bf5 -> 2ed601217
[SPARK-17464][SPARKR][ML] SparkR spark.als argument reg should be 0.1 by
default.
## What changes were proposed in this pull request?
SparkR ```spark.als``` arguments ```reg``` should be 0.1 by default, which need
to be consistent with ML.
## How was this patch tested?
Existing tests.
Author: Yanbo Liang
Closes #15021 from yanboliang/spark-17464.
Project: http://git-wip-us.apache.org/repos/asf/spark/repo
Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/2ed60121
Tree: http://git-wip-us.apache.org/repos/asf/spark/tree/2ed60121
Diff: http://git-wip-us.apache.org/repos/asf/spark/diff/2ed60121
Branch: refs/heads/master
Commit: 2ed601217ffd8945829ac762fae35202f3e55686
Parents: 65b814b
Author: Yanbo Liang
Authored: Fri Sep 9 05:43:34 2016 -0700
Committer: Yanbo Liang
Committed: Fri Sep 9 05:43:34 2016 -0700
--
R/pkg/R/mllib.R | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
--
http://git-wip-us.apache.org/repos/asf/spark/blob/2ed60121/R/pkg/R/mllib.R
--
diff --git a/R/pkg/R/mllib.R b/R/pkg/R/mllib.R
index f321fd1..f8d1095 100644
--- a/R/pkg/R/mllib.R
+++ b/R/pkg/R/mllib.R
@@ -1241,7 +1241,7 @@ setMethod("predict", signature(object =
"GaussianMixtureModel"),
#' @note spark.als since 2.1.0
setMethod("spark.als", signature(data = "SparkDataFrame"),
function(data, ratingCol = "rating", userCol = "user", itemCol =
"item",
- rank = 10, reg = 1.0, maxIter = 10, nonnegative = FALSE,
+ rank = 10, reg = 0.1, maxIter = 10, nonnegative = FALSE,
implicitPrefs = FALSE, alpha = 1.0, numUserBlocks = 10,
numItemBlocks = 10,
checkpointInterval = 10, seed = 0) {
-
To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org
For additional commands, e-mail: commits-h...@spark.apache.org
spark git commit: [SPARK-15509][FOLLOW-UP][ML][SPARKR] R MLlib algorithms should support input columns "features" and "label"
Repository: spark Updated Branches: refs/heads/master 1fec3ce4e -> bcdd259c3 [SPARK-15509][FOLLOW-UP][ML][SPARKR] R MLlib algorithms should support input columns "features" and "label" ## What changes were proposed in this pull request? #13584 resolved the issue of features and label columns conflict with ```RFormula``` default ones when loading libsvm data, but it still left some issues should be resolved: 1, Itâs not necessary to check and rename label column. Since we have considerations on the design of ```RFormula```, it can handle the case of label column already exists(with restriction of the existing label column should be numeric/boolean type). So itâs not necessary to change the column name to avoid conflict. If the label column is not numeric/boolean type, ```RFormula``` will throw exception. 2, We should rename features column name to new one if there is conflict, but appending a random value is enough since it was used internally only. We done similar work when implementing ```SQLTransformer```. 3, We should set correct new features column for the estimators. Take ```GLM``` as example: ```GLM``` estimator should set features column with the changed one(rFormula.getFeaturesCol) rather than the default âfeaturesâ. Although itâs same when training model, but it involves problems when predicting. The following is the prediction result of GLM before this PR:  We should drop the internal used feature column name, otherwise, it will appear on the prediction DataFrame which will confused users. And this behavior is same as other scenarios which does not exist column name conflict. After this PR:  ## How was this patch tested? Existing unit tests. Author: Yanbo Liang Closes #14993 from yanboliang/spark-15509. Project: http://git-wip-us.apache.org/repos/asf/spark/repo Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/bcdd259c Tree: http://git-wip-us.apache.org/repos/asf/spark/tree/bcdd259c Diff: http://git-wip-us.apache.org/repos/asf/spark/diff/bcdd259c Branch: refs/heads/master Commit: bcdd259c371b1dcdb41baf227867d7e2ecb923c6 Parents: 1fec3ce Author: Yanbo Liang Authored: Sat Sep 10 00:27:10 2016 -0700 Committer: Yanbo Liang Committed: Sat Sep 10 00:27:10 2016 -0700 -- .../ml/r/AFTSurvivalRegressionWrapper.scala | 1 + .../spark/ml/r/GaussianMixtureWrapper.scala | 1 + .../r/GeneralizedLinearRegressionWrapper.scala | 1 + .../spark/ml/r/IsotonicRegressionWrapper.scala | 1 + .../org/apache/spark/ml/r/KMeansWrapper.scala | 1 + .../apache/spark/ml/r/NaiveBayesWrapper.scala | 1 + .../org/apache/spark/ml/r/RWrapperUtils.scala | 34 +++- .../apache/spark/ml/r/RWrapperUtilsSuite.scala | 16 +++-- 8 files changed, 14 insertions(+), 42 deletions(-) -- http://git-wip-us.apache.org/repos/asf/spark/blob/bcdd259c/mllib/src/main/scala/org/apache/spark/ml/r/AFTSurvivalRegressionWrapper.scala -- diff --git a/mllib/src/main/scala/org/apache/spark/ml/r/AFTSurvivalRegressionWrapper.scala b/mllib/src/main/scala/org/apache/spark/ml/r/AFTSurvivalRegressionWrapper.scala index 67d037e..bd965ac 100644 --- a/mllib/src/main/scala/org/apache/spark/ml/r/AFTSurvivalRegressionWrapper.scala +++ b/mllib/src/main/scala/org/apache/spark/ml/r/AFTSurvivalRegressionWrapper.scala @@ -99,6 +99,7 @@ private[r] object AFTSurvivalRegressionWrapper extends MLReadable[AFTSurvivalReg val aft = new AFTSurvivalRegression() .setCensorCol(censorCol) .setFitIntercept(rFormula.hasIntercept) + .setFeaturesCol(rFormula.getFeaturesCol) val pipeline = new Pipeline() .setStages(Array(rFormulaModel, aft)) http://git-wip-us.apache.org/repos/asf/spark/blob/bcdd259c/mllib/src/main/scala/org/apache/spark/ml/r/GaussianMixtureWrapper.scala -- diff --git a/mllib/src/main/scala/org/apache/spark/ml/r/GaussianMixtureWrapper.scala b/mllib/src/main/scala/org/apache/spark/ml/r/GaussianMixtureWrapper.scala index b654233..b708702 100644 --- a/mllib/src/main/scala/org/apache/spark/ml/r/GaussianMixtureWrapper.scala +++ b/mllib/src/main/scala/org/apache/spark/ml/r/GaussianMixtureWrapper.scala @@ -85,6 +85,7 @@ private[r] object GaussianMixtureWrapper extends MLReadable[GaussianMixtureWrapp .setK(k) .setMaxIter(maxIter) .setTol(tol) + .setFeaturesCol(rFormula.getFeaturesCol) val pipeline = new Pipeline() .setStages(Array(rFormulaModel, gm)) http://git-wip-us.apache.org/repos/asf/spark/blob/bcdd259c/mllib/src/m
spark git commit: [SPARK-17585][PYSPARK][CORE] PySpark SparkContext.addFile supports adding files recursively
Repository: spark
Updated Branches:
refs/heads/master 61876a427 -> d3b886976
[SPARK-17585][PYSPARK][CORE] PySpark SparkContext.addFile supports adding files
recursively
## What changes were proposed in this pull request?
Users would like to add a directory as dependency in some cases, they can use
```SparkContext.addFile``` with argument ```recursive=true``` to recursively
add all files under the directory by using Scala. But Python users can only add
file not directory, we should also make it supported.
## How was this patch tested?
Unit test.
Author: Yanbo Liang
Closes #15140 from yanboliang/spark-17585.
Project: http://git-wip-us.apache.org/repos/asf/spark/repo
Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/d3b88697
Tree: http://git-wip-us.apache.org/repos/asf/spark/tree/d3b88697
Diff: http://git-wip-us.apache.org/repos/asf/spark/diff/d3b88697
Branch: refs/heads/master
Commit: d3b88697638dcf32854fe21a6c53dfb3782773b9
Parents: 61876a4
Author: Yanbo Liang
Authored: Wed Sep 21 01:37:03 2016 -0700
Committer: Yanbo Liang
Committed: Wed Sep 21 01:37:03 2016 -0700
--
.../spark/api/java/JavaSparkContext.scala | 13 +
python/pyspark/context.py | 7 +--
python/pyspark/tests.py | 20 +++-
python/test_support/hello.txt | 1 -
python/test_support/hello/hello.txt | 1 +
.../test_support/hello/sub_hello/sub_hello.txt | 1 +
6 files changed, 35 insertions(+), 8 deletions(-)
--
http://git-wip-us.apache.org/repos/asf/spark/blob/d3b88697/core/src/main/scala/org/apache/spark/api/java/JavaSparkContext.scala
--
diff --git
a/core/src/main/scala/org/apache/spark/api/java/JavaSparkContext.scala
b/core/src/main/scala/org/apache/spark/api/java/JavaSparkContext.scala
index 131f36f..4e50c26 100644
--- a/core/src/main/scala/org/apache/spark/api/java/JavaSparkContext.scala
+++ b/core/src/main/scala/org/apache/spark/api/java/JavaSparkContext.scala
@@ -670,6 +670,19 @@ class JavaSparkContext(val sc: SparkContext)
}
/**
+ * Add a file to be downloaded with this Spark job on every node.
+ * The `path` passed can be either a local file, a file in HDFS (or other
Hadoop-supported
+ * filesystems), or an HTTP, HTTPS or FTP URI. To access the file in Spark
jobs,
+ * use `SparkFiles.get(fileName)` to find its download location.
+ *
+ * A directory can be given if the recursive option is set to true.
Currently directories are only
+ * supported for Hadoop-supported filesystems.
+ */
+ def addFile(path: String, recursive: Boolean): Unit = {
+sc.addFile(path, recursive)
+ }
+
+ /**
* Adds a JAR dependency for all tasks to be executed on this SparkContext
in the future.
* The `path` passed can be either a local file, a file in HDFS (or other
Hadoop-supported
* filesystems), or an HTTP, HTTPS or FTP URI.
http://git-wip-us.apache.org/repos/asf/spark/blob/d3b88697/python/pyspark/context.py
--
diff --git a/python/pyspark/context.py b/python/pyspark/context.py
index 5c32f8e..7a7f59c 100644
--- a/python/pyspark/context.py
+++ b/python/pyspark/context.py
@@ -767,7 +767,7 @@ class SparkContext(object):
SparkContext._next_accum_id += 1
return Accumulator(SparkContext._next_accum_id - 1, value, accum_param)
-def addFile(self, path):
+def addFile(self, path, recursive=False):
"""
Add a file to be downloaded with this Spark job on every node.
The C{path} passed can be either a local file, a file in HDFS
@@ -778,6 +778,9 @@ class SparkContext(object):
L{SparkFiles.get(fileName)} with the
filename to find its download location.
+A directory can be given if the recursive option is set to True.
+Currently directories are only supported for Hadoop-supported
filesystems.
+
>>> from pyspark import SparkFiles
>>> path = os.path.join(tempdir, "test.txt")
>>> with open(path, "w") as testFile:
@@ -790,7 +793,7 @@ class SparkContext(object):
>>> sc.parallelize([1, 2, 3, 4]).mapPartitions(func).collect()
[100, 200, 300, 400]
"""
-self._jsc.sc().addFile(path)
+self._jsc.sc().addFile(path, recursive)
def addPyFile(self, path):
"""
http://git-wip-us.apache.org/repos/asf/spark/blob/d3b88697/python/pyspark/tests.py
--
diff --git a/python/pyspark/tests.py b/python/pyspark/tests.py
index 0a029b6..b075691 100644
--- a/python/pyspark/tests.py
+++ b/python/pyspark/tests.py
@@ -409,13 +409,23 @@ class AddFileTests(PySparkTestCase):
self.assertEqual("Hello World!"
spark git commit: [SPARK-17577][SPARKR][CORE] SparkR support add files to Spark job and get by executors
Repository: spark
Updated Branches:
refs/heads/master 7cbe21644 -> c133907c5
[SPARK-17577][SPARKR][CORE] SparkR support add files to Spark job and get by
executors
## What changes were proposed in this pull request?
Scala/Python users can add files to Spark job by submit options ```--files```
or ```SparkContext.addFile()```. Meanwhile, users can get the added file by
```SparkFiles.get(filename)```.
We should also support this function for SparkR users, since they also have the
requirements for some shared dependency files. For example, SparkR users can
download third party R packages to driver firstly, add these files to the Spark
job as dependency by this API and then each executor can install these packages
by ```install.packages```.
## How was this patch tested?
Add unit test.
Author: Yanbo Liang
Closes #15131 from yanboliang/spark-17577.
Project: http://git-wip-us.apache.org/repos/asf/spark/repo
Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/c133907c
Tree: http://git-wip-us.apache.org/repos/asf/spark/tree/c133907c
Diff: http://git-wip-us.apache.org/repos/asf/spark/diff/c133907c
Branch: refs/heads/master
Commit: c133907c5d9a6e6411b896b5e0cff48b2beff09f
Parents: 7cbe216
Author: Yanbo Liang
Authored: Wed Sep 21 20:08:28 2016 -0700
Committer: Yanbo Liang
Committed: Wed Sep 21 20:08:28 2016 -0700
--
R/pkg/NAMESPACE | 3 ++
R/pkg/R/context.R | 48
R/pkg/inst/tests/testthat/test_context.R| 13 ++
.../scala/org/apache/spark/SparkContext.scala | 6 +--
4 files changed, 67 insertions(+), 3 deletions(-)
--
http://git-wip-us.apache.org/repos/asf/spark/blob/c133907c/R/pkg/NAMESPACE
--
diff --git a/R/pkg/NAMESPACE b/R/pkg/NAMESPACE
index a5e9cbd..267a38c 100644
--- a/R/pkg/NAMESPACE
+++ b/R/pkg/NAMESPACE
@@ -336,6 +336,9 @@ export("as.DataFrame",
"read.parquet",
"read.text",
"spark.lapply",
+ "spark.addFile",
+ "spark.getSparkFilesRootDirectory",
+ "spark.getSparkFiles",
"sql",
"str",
"tableToDF",
http://git-wip-us.apache.org/repos/asf/spark/blob/c133907c/R/pkg/R/context.R
--
diff --git a/R/pkg/R/context.R b/R/pkg/R/context.R
index 13ade49..4793578 100644
--- a/R/pkg/R/context.R
+++ b/R/pkg/R/context.R
@@ -225,6 +225,54 @@ setCheckpointDir <- function(sc, dirName) {
invisible(callJMethod(sc, "setCheckpointDir",
suppressWarnings(normalizePath(dirName
}
+#' Add a file or directory to be downloaded with this Spark job on every node.
+#'
+#' The path passed can be either a local file, a file in HDFS (or other
Hadoop-supported
+#' filesystems), or an HTTP, HTTPS or FTP URI. To access the file in Spark
jobs,
+#' use spark.getSparkFiles(fileName) to find its download location.
+#'
+#' @rdname spark.addFile
+#' @param path The path of the file to be added
+#' @export
+#' @examples
+#'\dontrun{
+#' spark.addFile("~/myfile")
+#'}
+#' @note spark.addFile since 2.1.0
+spark.addFile <- function(path) {
+ sc <- getSparkContext()
+ invisible(callJMethod(sc, "addFile", suppressWarnings(normalizePath(path
+}
+
+#' Get the root directory that contains files added through spark.addFile.
+#'
+#' @rdname spark.getSparkFilesRootDirectory
+#' @return the root directory that contains files added through spark.addFile
+#' @export
+#' @examples
+#'\dontrun{
+#' spark.getSparkFilesRootDirectory()
+#'}
+#' @note spark.getSparkFilesRootDirectory since 2.1.0
+spark.getSparkFilesRootDirectory <- function() {
+ callJStatic("org.apache.spark.SparkFiles", "getRootDirectory")
+}
+
+#' Get the absolute path of a file added through spark.addFile.
+#'
+#' @rdname spark.getSparkFiles
+#' @param fileName The name of the file added through spark.addFile
+#' @return the absolute path of a file added through spark.addFile.
+#' @export
+#' @examples
+#'\dontrun{
+#' spark.getSparkFiles("myfile")
+#'}
+#' @note spark.getSparkFiles since 2.1.0
+spark.getSparkFiles <- function(fileName) {
+ callJStatic("org.apache.spark.SparkFiles", "get", as.character(fileName))
+}
+
#' Run a function over a list of elements, distributing the computations with
Spark
#'
#' Run a function over a list of elements, distributing the computations with
Spark. Applies a
http://git-wip-us.apache.org/repos/asf/spark/blob/c133907c/R/pkg/inst/tests/testthat/test_context.R
--
diff --git a/R/pkg/inst/tests/testthat/test_context.R
b/R/pkg/inst/tests/testthat/test_context.R
index 1ab7f31..0495418 100644
--- a/R/pkg/inst/tests/testthat/test_context.R
+++ b/R/pkg/inst/tests/testthat/test_context.R
@@ -166,3 +166,16 @@ test_that("spark
spark git commit: [SPARK-17315][FOLLOW-UP][SPARKR][ML] Fix print of Kolmogorov-Smirnov test summary
Repository: spark
Updated Branches:
refs/heads/master c133907c5 -> 6902edab7
[SPARK-17315][FOLLOW-UP][SPARKR][ML] Fix print of Kolmogorov-Smirnov test
summary
## What changes were proposed in this pull request?
#14881 added Kolmogorov-Smirnov Test wrapper to SparkR. I found that
```print.summary.KSTest``` was implemented inappropriately and result in no
effect.
Running the following code for KSTest:
```Scala
data <- data.frame(test = c(0.1, 0.15, 0.2, 0.3, 0.25, -1, -0.5))
df <- createDataFrame(data)
testResult <- spark.kstest(df, "test", "norm")
summary(testResult)
```
Before this PR:
After this PR:
The new implementation is similar with
[```print.summary.GeneralizedLinearRegressionModel```](https://github.com/apache/spark/blob/master/R/pkg/R/mllib.R#L284)
of SparkR and
[```print.summary.glm```](https://svn.r-project.org/R/trunk/src/library/stats/R/glm.R)
of native R.
BTW, I removed the comparison of ```print.summary.KSTest``` in unit test, since
it's only wrappers of the summary output which has been checked. Another reason
is that these comparison will output summary information to the test console,
it will make the test output in a mess.
## How was this patch tested?
Existing test.
Author: Yanbo Liang
Closes #15139 from yanboliang/spark-17315.
Project: http://git-wip-us.apache.org/repos/asf/spark/repo
Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/6902edab
Tree: http://git-wip-us.apache.org/repos/asf/spark/tree/6902edab
Diff: http://git-wip-us.apache.org/repos/asf/spark/diff/6902edab
Branch: refs/heads/master
Commit: 6902edab7e80e96e3f57cf80f26cefb209d4d63c
Parents: c133907
Author: Yanbo Liang
Authored: Wed Sep 21 20:14:18 2016 -0700
Committer: Yanbo Liang
Committed: Wed Sep 21 20:14:18 2016 -0700
--
R/pkg/R/mllib.R| 16 +---
R/pkg/inst/tests/testthat/test_mllib.R | 16 ++--
2 files changed, 11 insertions(+), 21 deletions(-)
--
http://git-wip-us.apache.org/repos/asf/spark/blob/6902edab/R/pkg/R/mllib.R
--
diff --git a/R/pkg/R/mllib.R b/R/pkg/R/mllib.R
index 234b208..98db367 100644
--- a/R/pkg/R/mllib.R
+++ b/R/pkg/R/mllib.R
@@ -1398,20 +1398,22 @@ setMethod("summary", signature(object = "KSTest"),
distParams <- unlist(callJMethod(jobj, "distParams"))
degreesOfFreedom <- callJMethod(jobj, "degreesOfFreedom")
-list(p.value = pValue, statistic = statistic, nullHypothesis =
nullHypothesis,
- nullHypothesis.name = distName, nullHypothesis.parameters =
distParams,
- degreesOfFreedom = degreesOfFreedom)
+ans <- list(p.value = pValue, statistic = statistic,
nullHypothesis = nullHypothesis,
+nullHypothesis.name = distName,
nullHypothesis.parameters = distParams,
+degreesOfFreedom = degreesOfFreedom, jobj = jobj)
+class(ans) <- "summary.KSTest"
+ans
})
# Prints the summary of KSTest
#' @rdname spark.kstest
-#' @param x test result object of KSTest by \code{spark.kstest}.
+#' @param x summary object of KSTest returned by \code{summary}.
#' @export
#' @note print.summary.KSTest since 2.1.0
print.summary.KSTest <- function(x, ...) {
- jobj <- x@jobj
+ jobj <- x$jobj
summaryStr <- callJMethod(jobj, "summary")
- cat(summaryStr)
- invisible(summaryStr)
+ cat(summaryStr, "\n")
+ invisible(x)
}
http://git-wip-us.apache.org/repos/asf/spark/blob/6902edab/R/pkg/inst/tests/testthat/test_mllib.R
--
diff --git a/R/pkg/inst/tests/testthat/test_mllib.R
b/R/pkg/inst/tests/testthat/test_mllib.R
index 5b1404c..24c40a8 100644
--- a/R/pkg/inst/tests/testthat/test_mllib.R
+++ b/R/pkg/inst/tests/testthat/test_mllib.R
@@ -760,13 +760,7 @@ test_that("spark.kstest", {
expect_equal(stats$p.value, rStats$p.value, tolerance = 1e-4)
expect_equal(stats$statistic, unname(rStats$statistic), tolerance = 1e-4)
-
- printStr <- print.summary.KSTest(testResult)
- expect_match(printStr, paste0("Kolmogorov-Smirnov test summary:\\n",
-"degrees of freedom = 0 \\n",
-"statistic = 0.38208[0-9]* \\n",
-"pValue = 0.19849[0-9]* \\n",
-".*"), perl = TRUE)
+ expect_match(capture.output(stats)[1], "Kolmogorov-Smirnov test summary:")
testResult <- spark.kstest(df, "test", "norm", -0.5)
stats <- summary(testResult)
@@ -775,13 +769,7 @@ test_that("spark.kstes
spark git commit: [SPARK-17281][ML][MLLIB] Add treeAggregateDepth parameter for AFTSurvivalRegression
Repository: spark
Updated Branches:
refs/heads/master 646f38346 -> 72d9fba26
[SPARK-17281][ML][MLLIB] Add treeAggregateDepth parameter for
AFTSurvivalRegression
## What changes were proposed in this pull request?
Add treeAggregateDepth parameter for AFTSurvivalRegression to keep consistent
with LiR/LoR.
## How was this patch tested?
Existing tests.
Author: WeichenXu
Closes #14851 from WeichenXu123/add_treeAggregate_param_for_survival_regression.
Project: http://git-wip-us.apache.org/repos/asf/spark/repo
Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/72d9fba2
Tree: http://git-wip-us.apache.org/repos/asf/spark/tree/72d9fba2
Diff: http://git-wip-us.apache.org/repos/asf/spark/diff/72d9fba2
Branch: refs/heads/master
Commit: 72d9fba26c19aae73116fd0d00b566967934c6fc
Parents: 646f383
Author: WeichenXu
Authored: Thu Sep 22 04:35:54 2016 -0700
Committer: Yanbo Liang
Committed: Thu Sep 22 04:35:54 2016 -0700
--
.../ml/regression/AFTSurvivalRegression.scala | 24
python/pyspark/ml/regression.py | 11 +
2 files changed, 25 insertions(+), 10 deletions(-)
--
http://git-wip-us.apache.org/repos/asf/spark/blob/72d9fba2/mllib/src/main/scala/org/apache/spark/ml/regression/AFTSurvivalRegression.scala
--
diff --git
a/mllib/src/main/scala/org/apache/spark/ml/regression/AFTSurvivalRegression.scala
b/mllib/src/main/scala/org/apache/spark/ml/regression/AFTSurvivalRegression.scala
index 3179f48..9d5ba99 100644
---
a/mllib/src/main/scala/org/apache/spark/ml/regression/AFTSurvivalRegression.scala
+++
b/mllib/src/main/scala/org/apache/spark/ml/regression/AFTSurvivalRegression.scala
@@ -46,7 +46,7 @@ import org.apache.spark.storage.StorageLevel
*/
private[regression] trait AFTSurvivalRegressionParams extends Params
with HasFeaturesCol with HasLabelCol with HasPredictionCol with HasMaxIter
- with HasTol with HasFitIntercept with Logging {
+ with HasTol with HasFitIntercept with HasAggregationDepth with Logging {
/**
* Param for censor column name.
@@ -184,6 +184,17 @@ class AFTSurvivalRegression @Since("1.6.0")
(@Since("1.6.0") override val uid: S
setDefault(tol -> 1E-6)
/**
+ * Suggested depth for treeAggregate (>= 2).
+ * If the dimensions of features or the number of partitions are large,
+ * this param could be adjusted to a larger size.
+ * Default is 2.
+ * @group expertSetParam
+ */
+ @Since("2.1.0")
+ def setAggregationDepth(value: Int): this.type = set(aggregationDepth, value)
+ setDefault(aggregationDepth -> 2)
+
+ /**
* Extract [[featuresCol]], [[labelCol]] and [[censorCol]] from input
dataset,
* and put it in an RDD with strong types.
*/
@@ -207,7 +218,9 @@ class AFTSurvivalRegression @Since("1.6.0")
(@Since("1.6.0") override val uid: S
val combOp = (c1: MultivariateOnlineSummarizer, c2:
MultivariateOnlineSummarizer) => {
c1.merge(c2)
}
- instances.treeAggregate(new MultivariateOnlineSummarizer)(seqOp, combOp)
+ instances.treeAggregate(
+new MultivariateOnlineSummarizer
+ )(seqOp, combOp, $(aggregationDepth))
}
val featuresStd = featuresSummarizer.variance.toArray.map(math.sqrt)
@@ -222,7 +235,7 @@ class AFTSurvivalRegression @Since("1.6.0")
(@Since("1.6.0") override val uid: S
val bcFeaturesStd = instances.context.broadcast(featuresStd)
-val costFun = new AFTCostFun(instances, $(fitIntercept), bcFeaturesStd)
+val costFun = new AFTCostFun(instances, $(fitIntercept), bcFeaturesStd,
$(aggregationDepth))
val optimizer = new BreezeLBFGS[BDV[Double]]($(maxIter), 10, $(tol))
/*
@@ -591,7 +604,8 @@ private class AFTAggregator(
private class AFTCostFun(
data: RDD[AFTPoint],
fitIntercept: Boolean,
-bcFeaturesStd: Broadcast[Array[Double]]) extends DiffFunction[BDV[Double]]
{
+bcFeaturesStd: Broadcast[Array[Double]],
+aggregationDepth: Int) extends DiffFunction[BDV[Double]] {
override def calculate(parameters: BDV[Double]): (Double, BDV[Double]) = {
@@ -604,7 +618,7 @@ private class AFTCostFun(
},
combOp = (c1, c2) => (c1, c2) match {
case (aggregator1, aggregator2) => aggregator1.merge(aggregator2)
- })
+ }, depth = aggregationDepth)
bcParameters.destroy(blocking = false)
(aftAggregator.loss, aftAggregator.gradient)
http://git-wip-us.apache.org/repos/asf/spark/blob/72d9fba2/python/pyspark/ml/regression.py
--
diff --git a/python/pyspark/ml/regression.py b/python/pyspark/ml/regression.py
index 19afc72..55d3803 100644
--- a/python/pyspark/ml/regression.py
+++ b/python/pyspark/ml/regression.py
@@ -1088,7 +1088,8 @@ class GBTRegressionModel(TreeEnsembleModel,
JavaPredi
spark git commit: [MINOR][SPARKR] Add sparkr-vignettes.html to gitignore.
Repository: spark Updated Branches: refs/heads/master 248916f55 -> 7945daed1 [MINOR][SPARKR] Add sparkr-vignettes.html to gitignore. ## What changes were proposed in this pull request? Add ```sparkr-vignettes.html``` to ```.gitignore```. ## How was this patch tested? No need test. Author: Yanbo Liang Closes #15215 from yanboliang/ignore. Project: http://git-wip-us.apache.org/repos/asf/spark/repo Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/7945daed Tree: http://git-wip-us.apache.org/repos/asf/spark/tree/7945daed Diff: http://git-wip-us.apache.org/repos/asf/spark/diff/7945daed Branch: refs/heads/master Commit: 7945daed12542587d51ece8f07e5c828b40db14a Parents: 248916f Author: Yanbo Liang Authored: Sat Sep 24 01:03:11 2016 -0700 Committer: Yanbo Liang Committed: Sat Sep 24 01:03:11 2016 -0700 -- .gitignore | 1 + 1 file changed, 1 insertion(+) -- http://git-wip-us.apache.org/repos/asf/spark/blob/7945daed/.gitignore -- diff --git a/.gitignore b/.gitignore index cfa8ad0..39d17e1 100644 --- a/.gitignore +++ b/.gitignore @@ -24,6 +24,7 @@ R-unit-tests.log R/unit-tests.out R/cran-check.out +R/pkg/vignettes/sparkr-vignettes.html build/*.jar build/apache-maven* build/scala* - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
[2/2] spark git commit: [SPARK-16356][ML] Add testImplicits for ML unit tests and promote toDF()
[SPARK-16356][ML] Add testImplicits for ML unit tests and promote toDF()
## What changes were proposed in this pull request?
This was suggested in
https://github.com/apache/spark/commit/101663f1ae222a919fc40510aa4f2bad22d1be6f#commitcomment-17114968.
This PR adds `testImplicits` to `MLlibTestSparkContext` so that some implicits
such as `toDF()` can be sued across ml tests.
This PR also changes all the usages of `spark.createDataFrame( ... )` to
`toDF()` where applicable in ml tests in Scala.
## How was this patch tested?
Existing tests should work.
Author: hyukjinkwon
Closes #14035 from HyukjinKwon/minor-ml-test.
Project: http://git-wip-us.apache.org/repos/asf/spark/repo
Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/f234b7cd
Tree: http://git-wip-us.apache.org/repos/asf/spark/tree/f234b7cd
Diff: http://git-wip-us.apache.org/repos/asf/spark/diff/f234b7cd
Branch: refs/heads/master
Commit: f234b7cd795dd9baa3feff541c211b4daf39ccc6
Parents: 50b89d0
Author: hyukjinkwon
Authored: Mon Sep 26 04:19:39 2016 -0700
Committer: Yanbo Liang
Committed: Mon Sep 26 04:19:39 2016 -0700
--
.../org/apache/spark/ml/PipelineSuite.scala | 13 +-
.../ml/classification/ClassifierSuite.scala | 16 +--
.../DecisionTreeClassifierSuite.scala | 3 +-
.../ml/classification/GBTClassifierSuite.scala | 6 +-
.../LogisticRegressionSuite.scala | 43 +++
.../MultilayerPerceptronClassifierSuite.scala | 26 ++--
.../ml/classification/NaiveBayesSuite.scala | 20 +--
.../ml/classification/OneVsRestSuite.scala | 4 +-
.../RandomForestClassifierSuite.scala | 3 +-
.../apache/spark/ml/clustering/LDASuite.scala | 6 +-
.../BinaryClassificationEvaluatorSuite.scala| 14 ++-
.../evaluation/RegressionEvaluatorSuite.scala | 8 +-
.../spark/ml/feature/BinarizerSuite.scala | 16 +--
.../spark/ml/feature/BucketizerSuite.scala | 15 ++-
.../spark/ml/feature/ChiSqSelectorSuite.scala | 3 +-
.../spark/ml/feature/CountVectorizerSuite.scala | 30 ++---
.../org/apache/spark/ml/feature/DCTSuite.scala | 10 +-
.../spark/ml/feature/HashingTFSuite.scala | 10 +-
.../org/apache/spark/ml/feature/IDFSuite.scala | 6 +-
.../spark/ml/feature/InteractionSuite.scala | 53
.../spark/ml/feature/MaxAbsScalerSuite.scala| 5 +-
.../spark/ml/feature/MinMaxScalerSuite.scala| 13 +-
.../apache/spark/ml/feature/NGramSuite.scala| 35 +++---
.../spark/ml/feature/NormalizerSuite.scala | 4 +-
.../spark/ml/feature/OneHotEncoderSuite.scala | 10 +-
.../org/apache/spark/ml/feature/PCASuite.scala | 4 +-
.../ml/feature/PolynomialExpansionSuite.scala | 11 +-
.../apache/spark/ml/feature/RFormulaSuite.scala | 126 ---
.../spark/ml/feature/SQLTransformerSuite.scala | 8 +-
.../spark/ml/feature/StandardScalerSuite.scala | 12 +-
.../ml/feature/StopWordsRemoverSuite.scala | 29 +++--
.../spark/ml/feature/StringIndexerSuite.scala | 32 ++---
.../spark/ml/feature/TokenizerSuite.scala | 17 +--
.../spark/ml/feature/VectorAssemblerSuite.scala | 10 +-
.../spark/ml/feature/VectorIndexerSuite.scala | 15 ++-
.../regression/AFTSurvivalRegressionSuite.scala | 26 ++--
.../spark/ml/regression/GBTRegressorSuite.scala | 7 +-
.../GeneralizedLinearRegressionSuite.scala | 115 -
.../ml/regression/IsotonicRegressionSuite.scala | 14 +--
.../ml/regression/LinearRegressionSuite.scala | 62 +
.../tree/impl/GradientBoostedTreesSuite.scala | 6 +-
.../spark/ml/tuning/CrossValidatorSuite.scala | 12 +-
.../ml/tuning/TrainValidationSplitSuite.scala | 13 +-
.../apache/spark/mllib/util/MLUtilsSuite.scala | 18 +--
.../mllib/util/MLlibTestSparkContext.scala | 13 +-
45 files changed, 462 insertions(+), 460 deletions(-)
--
http://git-wip-us.apache.org/repos/asf/spark/blob/f234b7cd/mllib/src/test/scala/org/apache/spark/ml/PipelineSuite.scala
--
diff --git a/mllib/src/test/scala/org/apache/spark/ml/PipelineSuite.scala
b/mllib/src/test/scala/org/apache/spark/ml/PipelineSuite.scala
index 3b490cd..6413ca1 100644
--- a/mllib/src/test/scala/org/apache/spark/ml/PipelineSuite.scala
+++ b/mllib/src/test/scala/org/apache/spark/ml/PipelineSuite.scala
@@ -36,6 +36,8 @@ import org.apache.spark.sql.types.StructType
class PipelineSuite extends SparkFunSuite with MLlibTestSparkContext with
DefaultReadWriteTest {
+ import testImplicits._
+
abstract class MyModel extends Model[MyModel]
test("pipeline") {
@@ -183,12 +185,11 @@ class PipelineSuite extends SparkFunSuite with
MLlibTestSparkContext with Defaul
}
test("pipeline validateParams") {
-val df = spark.createDataFrame(
- Seq(
-(1, Vectors.dense(0.0, 1.0, 4.0),
[1/2] spark git commit: [SPARK-16356][ML] Add testImplicits for ML unit tests and promote toDF()
Repository: spark
Updated Branches:
refs/heads/master 50b89d05b -> f234b7cd7
http://git-wip-us.apache.org/repos/asf/spark/blob/f234b7cd/mllib/src/test/scala/org/apache/spark/ml/feature/StringIndexerSuite.scala
--
diff --git
a/mllib/src/test/scala/org/apache/spark/ml/feature/StringIndexerSuite.scala
b/mllib/src/test/scala/org/apache/spark/ml/feature/StringIndexerSuite.scala
index b478fea..a6bbb94 100644
--- a/mllib/src/test/scala/org/apache/spark/ml/feature/StringIndexerSuite.scala
+++ b/mllib/src/test/scala/org/apache/spark/ml/feature/StringIndexerSuite.scala
@@ -29,6 +29,8 @@ import org.apache.spark.sql.types.{DoubleType, StringType,
StructField, StructTy
class StringIndexerSuite
extends SparkFunSuite with MLlibTestSparkContext with DefaultReadWriteTest {
+ import testImplicits._
+
test("params") {
ParamsSuite.checkParams(new StringIndexer)
val model = new StringIndexerModel("indexer", Array("a", "b"))
@@ -38,8 +40,8 @@ class StringIndexerSuite
}
test("StringIndexer") {
-val data = sc.parallelize(Seq((0, "a"), (1, "b"), (2, "c"), (3, "a"), (4,
"a"), (5, "c")), 2)
-val df = spark.createDataFrame(data).toDF("id", "label")
+val data = Seq((0, "a"), (1, "b"), (2, "c"), (3, "a"), (4, "a"), (5, "c"))
+val df = data.toDF("id", "label")
val indexer = new StringIndexer()
.setInputCol("label")
.setOutputCol("labelIndex")
@@ -61,10 +63,10 @@ class StringIndexerSuite
}
test("StringIndexerUnseen") {
-val data = sc.parallelize(Seq((0, "a"), (1, "b"), (4, "b")), 2)
-val data2 = sc.parallelize(Seq((0, "a"), (1, "b"), (2, "c")), 2)
-val df = spark.createDataFrame(data).toDF("id", "label")
-val df2 = spark.createDataFrame(data2).toDF("id", "label")
+val data = Seq((0, "a"), (1, "b"), (4, "b"))
+val data2 = Seq((0, "a"), (1, "b"), (2, "c"))
+val df = data.toDF("id", "label")
+val df2 = data2.toDF("id", "label")
val indexer = new StringIndexer()
.setInputCol("label")
.setOutputCol("labelIndex")
@@ -92,8 +94,8 @@ class StringIndexerSuite
}
test("StringIndexer with a numeric input column") {
-val data = sc.parallelize(Seq((0, 100), (1, 200), (2, 300), (3, 100), (4,
100), (5, 300)), 2)
-val df = spark.createDataFrame(data).toDF("id", "label")
+val data = Seq((0, 100), (1, 200), (2, 300), (3, 100), (4, 100), (5, 300))
+val df = data.toDF("id", "label")
val indexer = new StringIndexer()
.setInputCol("label")
.setOutputCol("labelIndex")
@@ -119,7 +121,7 @@ class StringIndexerSuite
}
test("StringIndexerModel can't overwrite output column") {
-val df = spark.createDataFrame(Seq((1, 2), (3, 4))).toDF("input", "output")
+val df = Seq((1, 2), (3, 4)).toDF("input", "output")
intercept[IllegalArgumentException] {
new StringIndexer()
.setInputCol("input")
@@ -161,9 +163,7 @@ class StringIndexerSuite
test("IndexToString.transform") {
val labels = Array("a", "b", "c")
-val df0 = spark.createDataFrame(Seq(
- (0, "a"), (1, "b"), (2, "c"), (0, "a")
-)).toDF("index", "expected")
+val df0 = Seq((0, "a"), (1, "b"), (2, "c"), (0, "a")).toDF("index",
"expected")
val idxToStr0 = new IndexToString()
.setInputCol("index")
@@ -187,8 +187,8 @@ class StringIndexerSuite
}
test("StringIndexer, IndexToString are inverses") {
-val data = sc.parallelize(Seq((0, "a"), (1, "b"), (2, "c"), (3, "a"), (4,
"a"), (5, "c")), 2)
-val df = spark.createDataFrame(data).toDF("id", "label")
+val data = Seq((0, "a"), (1, "b"), (2, "c"), (3, "a"), (4, "a"), (5, "c"))
+val df = data.toDF("id", "label")
val indexer = new StringIndexer()
.setInputCol("label")
.setOutputCol("labelIndex")
@@ -220,8 +220,8 @@ class StringIndexerSuite
}
test("StringIndexer metadata") {
-val data = sc.parallelize(Seq((0, "a"), (1, "b"), (2, "c"), (3, "a"), (4,
"a"), (5, "c")), 2)
-val df = spark.createDataFrame(data).toDF("id", "label")
+val data = Seq((0, "a"), (1, "b"), (2, "c"), (3, "a"), (4, "a"), (5, "c"))
+val df = data.toDF("id", "label")
val indexer = new StringIndexer()
.setInputCol("label")
.setOutputCol("labelIndex")
http://git-wip-us.apache.org/repos/asf/spark/blob/f234b7cd/mllib/src/test/scala/org/apache/spark/ml/feature/TokenizerSuite.scala
--
diff --git
a/mllib/src/test/scala/org/apache/spark/ml/feature/TokenizerSuite.scala
b/mllib/src/test/scala/org/apache/spark/ml/feature/TokenizerSuite.scala
index f30bdc3..c895659 100644
--- a/mllib/src/test/scala/org/apache/spark/ml/feature/TokenizerSuite.scala
+++ b/mllib/src/test/scala/org/apache/spark/ml/feature/TokenizerSuite.scala
@@ -46,6 +46,7 @@ class RegexTokenizerSuite
extends SparkFunSuite with MLlibTestSparkContext with DefaultReadWriteTest {
import org.apa
spark git commit: [SPARK-17577][FOLLOW-UP][SPARKR] SparkR spark.addFile supports adding directory recursively
Repository: spark
Updated Branches:
refs/heads/master 00be16df6 -> 93c743f1a
[SPARK-17577][FOLLOW-UP][SPARKR] SparkR spark.addFile supports adding directory
recursively
## What changes were proposed in this pull request?
#15140 exposed ```JavaSparkContext.addFile(path: String, recursive: Boolean)```
to Python/R, then we can update SparkR ```spark.addFile``` to support adding
directory recursively.
## How was this patch tested?
Added unit test.
Author: Yanbo Liang
Closes #15216 from yanboliang/spark-17577-2.
Project: http://git-wip-us.apache.org/repos/asf/spark/repo
Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/93c743f1
Tree: http://git-wip-us.apache.org/repos/asf/spark/tree/93c743f1
Diff: http://git-wip-us.apache.org/repos/asf/spark/diff/93c743f1
Branch: refs/heads/master
Commit: 93c743f1aca433144611b11d4e1b169d66e0f57b
Parents: 00be16d
Author: Yanbo Liang
Authored: Mon Sep 26 16:47:57 2016 -0700
Committer: Yanbo Liang
Committed: Mon Sep 26 16:47:57 2016 -0700
--
R/pkg/R/context.R| 9 +++--
R/pkg/inst/tests/testthat/test_context.R | 22 ++
2 files changed, 29 insertions(+), 2 deletions(-)
--
http://git-wip-us.apache.org/repos/asf/spark/blob/93c743f1/R/pkg/R/context.R
--
diff --git a/R/pkg/R/context.R b/R/pkg/R/context.R
index 4793578..fe2f3e3 100644
--- a/R/pkg/R/context.R
+++ b/R/pkg/R/context.R
@@ -231,17 +231,22 @@ setCheckpointDir <- function(sc, dirName) {
#' filesystems), or an HTTP, HTTPS or FTP URI. To access the file in Spark
jobs,
#' use spark.getSparkFiles(fileName) to find its download location.
#'
+#' A directory can be given if the recursive option is set to true.
+#' Currently directories are only supported for Hadoop-supported filesystems.
+#' Refer Hadoop-supported filesystems at
\url{https://wiki.apache.org/hadoop/HCFS}.
+#'
#' @rdname spark.addFile
#' @param path The path of the file to be added
+#' @param recursive Whether to add files recursively from the path. Default is
FALSE.
#' @export
#' @examples
#'\dontrun{
#' spark.addFile("~/myfile")
#'}
#' @note spark.addFile since 2.1.0
-spark.addFile <- function(path) {
+spark.addFile <- function(path, recursive = FALSE) {
sc <- getSparkContext()
- invisible(callJMethod(sc, "addFile", suppressWarnings(normalizePath(path
+ invisible(callJMethod(sc, "addFile", suppressWarnings(normalizePath(path)),
recursive))
}
#' Get the root directory that contains files added through spark.addFile.
http://git-wip-us.apache.org/repos/asf/spark/blob/93c743f1/R/pkg/inst/tests/testthat/test_context.R
--
diff --git a/R/pkg/inst/tests/testthat/test_context.R
b/R/pkg/inst/tests/testthat/test_context.R
index 0495418..caca069 100644
--- a/R/pkg/inst/tests/testthat/test_context.R
+++ b/R/pkg/inst/tests/testthat/test_context.R
@@ -169,6 +169,7 @@ test_that("spark.lapply should perform simple transforms", {
test_that("add and get file to be downloaded with Spark job on every node", {
sparkR.sparkContext()
+ # Test add file.
path <- tempfile(pattern = "hello", fileext = ".txt")
filename <- basename(path)
words <- "Hello World!"
@@ -177,5 +178,26 @@ test_that("add and get file to be downloaded with Spark
job on every node", {
download_path <- spark.getSparkFiles(filename)
expect_equal(readLines(download_path), words)
unlink(path)
+
+ # Test add directory recursively.
+ path <- paste0(tempdir(), "/", "recursive_dir")
+ dir.create(path)
+ dir_name <- basename(path)
+ path1 <- paste0(path, "/", "hello.txt")
+ file.create(path1)
+ sub_path <- paste0(path, "/", "sub_hello")
+ dir.create(sub_path)
+ path2 <- paste0(sub_path, "/", "sub_hello.txt")
+ file.create(path2)
+ words <- "Hello World!"
+ sub_words <- "Sub Hello World!"
+ writeLines(words, path1)
+ writeLines(sub_words, path2)
+ spark.addFile(path, recursive = TRUE)
+ download_path1 <- spark.getSparkFiles(paste0(dir_name, "/", "hello.txt"))
+ expect_equal(readLines(download_path1), words)
+ download_path2 <- spark.getSparkFiles(paste0(dir_name, "/",
"sub_hello/sub_hello.txt"))
+ expect_equal(readLines(download_path2), sub_words)
+ unlink(path, recursive = TRUE)
sparkR.session.stop()
})
-
To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org
For additional commands, e-mail: commits-h...@spark.apache.org
spark git commit: [SPARK-17138][ML][MLIB] Add Python API for multinomial logistic regression
Repository: spark
Updated Branches:
refs/heads/master 85b0a1575 -> 7f16affa2
[SPARK-17138][ML][MLIB] Add Python API for multinomial logistic regression
## What changes were proposed in this pull request?
Add Python API for multinomial logistic regression.
- add `family` param in python api.
- expose `coefficientMatrix` and `interceptVector` for `LogisticRegressionModel`
- add python-side testcase for multinomial logistic regression
- update python doc.
## How was this patch tested?
existing and added doc tests.
Author: WeichenXu
Closes #14852 from WeichenXu123/add_MLOR_python.
Project: http://git-wip-us.apache.org/repos/asf/spark/repo
Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/7f16affa
Tree: http://git-wip-us.apache.org/repos/asf/spark/tree/7f16affa
Diff: http://git-wip-us.apache.org/repos/asf/spark/diff/7f16affa
Branch: refs/heads/master
Commit: 7f16affa262b059580ed2775a7b05a767aa72315
Parents: 85b0a15
Author: WeichenXu
Authored: Tue Sep 27 00:00:21 2016 -0700
Committer: Yanbo Liang
Committed: Tue Sep 27 00:00:21 2016 -0700
--
python/pyspark/ml/classification.py | 90 +---
1 file changed, 70 insertions(+), 20 deletions(-)
--
http://git-wip-us.apache.org/repos/asf/spark/blob/7f16affa/python/pyspark/ml/classification.py
--
diff --git a/python/pyspark/ml/classification.py
b/python/pyspark/ml/classification.py
index b4c01fd..505e7bf 100644
--- a/python/pyspark/ml/classification.py
+++ b/python/pyspark/ml/classification.py
@@ -67,21 +67,34 @@ class LogisticRegression(JavaEstimator, HasFeaturesCol,
HasLabelCol, HasPredicti
HasWeightCol, HasAggregationDepth, JavaMLWritable,
JavaMLReadable):
"""
Logistic regression.
-Currently, this class only supports binary classification.
+This class supports multinomial logistic (softmax) and binomial logistic
regression.
>>> from pyspark.sql import Row
>>> from pyspark.ml.linalg import Vectors
->>> df = sc.parallelize([
+>>> bdf = sc.parallelize([
... Row(label=1.0, weight=2.0, features=Vectors.dense(1.0)),
... Row(label=0.0, weight=2.0, features=Vectors.sparse(1, [],
[]))]).toDF()
->>> lr = LogisticRegression(maxIter=5, regParam=0.01, weightCol="weight")
->>> model = lr.fit(df)
->>> model.coefficients
+>>> blor = LogisticRegression(maxIter=5, regParam=0.01, weightCol="weight")
+>>> blorModel = blor.fit(bdf)
+>>> blorModel.coefficients
DenseVector([5.5...])
->>> model.intercept
+>>> blorModel.intercept
-2.68...
+>>> mdf = sc.parallelize([
+... Row(label=1.0, weight=2.0, features=Vectors.dense(1.0)),
+... Row(label=0.0, weight=2.0, features=Vectors.sparse(1, [], [])),
+... Row(label=2.0, weight=2.0, features=Vectors.dense(3.0))]).toDF()
+>>> mlor = LogisticRegression(maxIter=5, regParam=0.01, weightCol="weight",
+... family="multinomial")
+>>> mlorModel = mlor.fit(mdf)
+>>> print(mlorModel.coefficientMatrix)
+DenseMatrix([[-2.3...],
+ [ 0.2...],
+ [ 2.1... ]])
+>>> mlorModel.interceptVector
+DenseVector([2.0..., 0.8..., -2.8...])
>>> test0 = sc.parallelize([Row(features=Vectors.dense(-1.0))]).toDF()
->>> result = model.transform(test0).head()
+>>> result = blorModel.transform(test0).head()
>>> result.prediction
0.0
>>> result.probability
@@ -89,23 +102,23 @@ class LogisticRegression(JavaEstimator, HasFeaturesCol,
HasLabelCol, HasPredicti
>>> result.rawPrediction
DenseVector([8.22..., -8.22...])
>>> test1 = sc.parallelize([Row(features=Vectors.sparse(1, [0],
[1.0]))]).toDF()
->>> model.transform(test1).head().prediction
+>>> blorModel.transform(test1).head().prediction
1.0
->>> lr.setParams("vector")
+>>> blor.setParams("vector")
Traceback (most recent call last):
...
TypeError: Method setParams forces keyword arguments.
>>> lr_path = temp_path + "/lr"
->>> lr.save(lr_path)
+>>> blor.save(lr_path)
>>> lr2 = LogisticRegression.load(lr_path)
>>> lr2.getMaxIter()
5
>>> model_path = temp_path + "/lr_model"
->>> model.save(model_path)
+>>> blorModel.save(model_path)
>>> model2 = LogisticRegressionModel.load(model_path)
->>> model.coefficients[0] == model2.coefficients[0]
+>>> blorModel.coefficients[0] == model2.coefficients[0]
True
->>> model.intercept == model2.intercept
+>>> blorModel.intercept == model2.intercept
True
.. versionadded:: 1.3.0
@@ -117,24 +130,29 @@ class LogisticRegression(JavaEstimator, HasFeaturesCol,
HasLabelCol, HasPredicti
"e.g. if threshold is p, then thresholds must be equal
to [1-p, p].",
spark git commit: [SPARK-16356][FOLLOW-UP][ML] Enforce ML test of exception for local/distributed Dataset.
Repository: spark
Updated Branches:
refs/heads/master 37eb9184f -> a19a1bb59
[SPARK-16356][FOLLOW-UP][ML] Enforce ML test of exception for local/distributed
Dataset.
## What changes were proposed in this pull request?
#14035 added ```testImplicits``` to ML unit tests and promoted ```toDF()```,
but left one minor issue at ```VectorIndexerSuite```. If we create the
DataFrame by ```Seq(...).toDF()```, it will throw different error/exception
compared with ```sc.parallelize(Seq(...)).toDF()``` for one of the test cases.
After in-depth study, I found it was caused by different behavior of local and
distributed Dataset if the UDF failed at ```assert```. If the data is local
Dataset, it throws ```AssertionError``` directly; If the data is distributed
Dataset, it throws ```SparkException``` which is the wrapper of
```AssertionError```. I think we should enforce this test to cover both case.
## How was this patch tested?
Unit test.
Author: Yanbo Liang
Closes #15261 from yanboliang/spark-16356.
Project: http://git-wip-us.apache.org/repos/asf/spark/repo
Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/a19a1bb5
Tree: http://git-wip-us.apache.org/repos/asf/spark/tree/a19a1bb5
Diff: http://git-wip-us.apache.org/repos/asf/spark/diff/a19a1bb5
Branch: refs/heads/master
Commit: a19a1bb59411177caaf99581e89098826b7d0c7b
Parents: 37eb918
Author: Yanbo Liang
Authored: Thu Sep 29 00:54:26 2016 -0700
Committer: Yanbo Liang
Committed: Thu Sep 29 00:54:26 2016 -0700
--
.../apache/spark/ml/feature/VectorIndexerSuite.scala | 13 +
1 file changed, 9 insertions(+), 4 deletions(-)
--
http://git-wip-us.apache.org/repos/asf/spark/blob/a19a1bb5/mllib/src/test/scala/org/apache/spark/ml/feature/VectorIndexerSuite.scala
--
diff --git
a/mllib/src/test/scala/org/apache/spark/ml/feature/VectorIndexerSuite.scala
b/mllib/src/test/scala/org/apache/spark/ml/feature/VectorIndexerSuite.scala
index 4da1b13..b28ce2a 100644
--- a/mllib/src/test/scala/org/apache/spark/ml/feature/VectorIndexerSuite.scala
+++ b/mllib/src/test/scala/org/apache/spark/ml/feature/VectorIndexerSuite.scala
@@ -88,9 +88,7 @@ class VectorIndexerSuite extends SparkFunSuite with
MLlibTestSparkContext
densePoints1 = densePoints1Seq.map(FeatureData).toDF()
sparsePoints1 = sparsePoints1Seq.map(FeatureData).toDF()
-// TODO: If we directly use `toDF` without parallelize, the test in
-// "Throws error when given RDDs with different size vectors" is failed
for an unknown reason.
-densePoints2 = sc.parallelize(densePoints2Seq, 2).map(FeatureData).toDF()
+densePoints2 = densePoints2Seq.map(FeatureData).toDF()
sparsePoints2 = sparsePoints2Seq.map(FeatureData).toDF()
badPoints = badPointsSeq.map(FeatureData).toDF()
}
@@ -121,10 +119,17 @@ class VectorIndexerSuite extends SparkFunSuite with
MLlibTestSparkContext
model.transform(densePoints1) // should work
model.transform(sparsePoints1) // should work
-intercept[SparkException] {
+// If the data is local Dataset, it throws AssertionError directly.
+intercept[AssertionError] {
model.transform(densePoints2).collect()
logInfo("Did not throw error when fit, transform were called on vectors
of different lengths")
}
+// If the data is distributed Dataset, it throws SparkException
+// which is the wrapper of AssertionError.
+intercept[SparkException] {
+ model.transform(densePoints2.repartition(2)).collect()
+ logInfo("Did not throw error when fit, transform were called on vectors
of different lengths")
+}
intercept[SparkException] {
vectorIndexer.fit(badPoints)
logInfo("Did not throw error when fitting vectors of different lengths
in same RDD.")
-
To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org
For additional commands, e-mail: commits-h...@spark.apache.org
spark git commit: [SPARK-17704][ML][MLLIB] ChiSqSelector performance improvement.
Repository: spark
Updated Branches:
refs/heads/master a19a1bb59 -> f7082ac12
[SPARK-17704][ML][MLLIB] ChiSqSelector performance improvement.
## What changes were proposed in this pull request?
Several performance improvement for ```ChiSqSelector```:
1, Keep ```selectedFeatures``` ordered ascendent.
```ChiSqSelectorModel.transform``` need ```selectedFeatures``` ordered to make
prediction. We should sort it when training model rather than making
prediction, since users usually train model once and use the model to do
prediction multiple times.
2, When training ```fpr``` type ```ChiSqSelectorModel```, it's not necessary to
sort the ChiSq test result by statistic.
## How was this patch tested?
Existing unit tests.
Author: Yanbo Liang
Closes #15277 from yanboliang/spark-17704.
Project: http://git-wip-us.apache.org/repos/asf/spark/repo
Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/f7082ac1
Tree: http://git-wip-us.apache.org/repos/asf/spark/tree/f7082ac1
Diff: http://git-wip-us.apache.org/repos/asf/spark/diff/f7082ac1
Branch: refs/heads/master
Commit: f7082ac12518ae84d6d1d4b7330a9f12cf95e7c1
Parents: a19a1bb
Author: Yanbo Liang
Authored: Thu Sep 29 04:30:42 2016 -0700
Committer: Yanbo Liang
Committed: Thu Sep 29 04:30:42 2016 -0700
--
.../spark/mllib/feature/ChiSqSelector.scala | 45 +---
project/MimaExcludes.scala | 3 --
2 files changed, 30 insertions(+), 18 deletions(-)
--
http://git-wip-us.apache.org/repos/asf/spark/blob/f7082ac1/mllib/src/main/scala/org/apache/spark/mllib/feature/ChiSqSelector.scala
--
diff --git
a/mllib/src/main/scala/org/apache/spark/mllib/feature/ChiSqSelector.scala
b/mllib/src/main/scala/org/apache/spark/mllib/feature/ChiSqSelector.scala
index 0f7c6e8..706ce78 100644
--- a/mllib/src/main/scala/org/apache/spark/mllib/feature/ChiSqSelector.scala
+++ b/mllib/src/main/scala/org/apache/spark/mllib/feature/ChiSqSelector.scala
@@ -35,12 +35,24 @@ import org.apache.spark.sql.{Row, SparkSession}
/**
* Chi Squared selector model.
*
- * @param selectedFeatures list of indices to select (filter).
+ * @param selectedFeatures list of indices to select (filter). Must be ordered
asc
*/
@Since("1.3.0")
class ChiSqSelectorModel @Since("1.3.0") (
@Since("1.3.0") val selectedFeatures: Array[Int]) extends VectorTransformer
with Saveable {
+ require(isSorted(selectedFeatures), "Array has to be sorted asc")
+
+ protected def isSorted(array: Array[Int]): Boolean = {
+var i = 1
+val len = array.length
+while (i < len) {
+ if (array(i) < array(i-1)) return false
+ i += 1
+}
+true
+ }
+
/**
* Applies transformation on a vector.
*
@@ -57,22 +69,21 @@ class ChiSqSelectorModel @Since("1.3.0") (
* Preserves the order of filtered features the same as their indices are
stored.
* Might be moved to Vector as .slice
* @param features vector
- * @param filterIndices indices of features to filter
+ * @param filterIndices indices of features to filter, must be ordered asc
*/
private def compress(features: Vector, filterIndices: Array[Int]): Vector = {
-val orderedIndices = filterIndices.sorted
features match {
case SparseVector(size, indices, values) =>
-val newSize = orderedIndices.length
+val newSize = filterIndices.length
val newValues = new ArrayBuilder.ofDouble
val newIndices = new ArrayBuilder.ofInt
var i = 0
var j = 0
var indicesIdx = 0
var filterIndicesIdx = 0
-while (i < indices.length && j < orderedIndices.length) {
+while (i < indices.length && j < filterIndices.length) {
indicesIdx = indices(i)
- filterIndicesIdx = orderedIndices(j)
+ filterIndicesIdx = filterIndices(j)
if (indicesIdx == filterIndicesIdx) {
newIndices += j
newValues += values(i)
@@ -90,7 +101,7 @@ class ChiSqSelectorModel @Since("1.3.0") (
Vectors.sparse(newSize, newIndices.result(), newValues.result())
case DenseVector(values) =>
val values = features.toArray
-Vectors.dense(orderedIndices.map(i => values(i)))
+Vectors.dense(filterIndices.map(i => values(i)))
case other =>
throw new UnsupportedOperationException(
s"Only sparse and dense vectors are supported but got
${other.getClass}.")
@@ -220,18 +231,22 @@ class ChiSqSelector @Since("2.1.0") () extends
Serializable {
@Since("1.3.0")
def fit(data: RDD[LabeledPoint]): ChiSqSelectorModel = {
val chiSqTestResult = Statistics.chiSqTest(data)
- .zipWithIndex.sortBy { case (res, _) => -res.statistic }
val features = selectorType match {
- case ChiSqSelector.KBest => chiSqTestResult
spark git commit: [SPARK-14077][ML] Refactor NaiveBayes to support weighted instances
Repository: spark
Updated Branches:
refs/heads/master 74ac1c438 -> 1fad55968
[SPARK-14077][ML] Refactor NaiveBayes to support weighted instances
## What changes were proposed in this pull request?
1,support weighted data
2,use dataset/dataframe instead of rdd
3,make mllib as a wrapper to call ml
## How was this patch tested?
local manual tests in spark-shell
unit tests
Author: Zheng RuiFeng
Closes #12819 from zhengruifeng/weighted_nb.
Project: http://git-wip-us.apache.org/repos/asf/spark/repo
Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/1fad5596
Tree: http://git-wip-us.apache.org/repos/asf/spark/tree/1fad5596
Diff: http://git-wip-us.apache.org/repos/asf/spark/diff/1fad5596
Branch: refs/heads/master
Commit: 1fad5596885aab8b32d2307c0edecbae50d5bd7a
Parents: 74ac1c4
Author: Zheng RuiFeng
Authored: Thu Sep 29 23:55:42 2016 -0700
Committer: Yanbo Liang
Committed: Thu Sep 29 23:55:42 2016 -0700
--
.../spark/ml/classification/NaiveBayes.scala| 154 ++-
.../spark/mllib/classification/NaiveBayes.scala | 99
.../ml/classification/NaiveBayesSuite.scala | 50 +-
3 files changed, 191 insertions(+), 112 deletions(-)
--
http://git-wip-us.apache.org/repos/asf/spark/blob/1fad5596/mllib/src/main/scala/org/apache/spark/ml/classification/NaiveBayes.scala
--
diff --git
a/mllib/src/main/scala/org/apache/spark/ml/classification/NaiveBayes.scala
b/mllib/src/main/scala/org/apache/spark/ml/classification/NaiveBayes.scala
index f939a1c..0d652aa 100644
--- a/mllib/src/main/scala/org/apache/spark/ml/classification/NaiveBayes.scala
+++ b/mllib/src/main/scala/org/apache/spark/ml/classification/NaiveBayes.scala
@@ -19,23 +19,20 @@ package org.apache.spark.ml.classification
import org.apache.hadoop.fs.Path
-import org.apache.spark.SparkException
import org.apache.spark.annotation.Since
import org.apache.spark.ml.PredictorParams
import org.apache.spark.ml.linalg._
import org.apache.spark.ml.param.{DoubleParam, Param, ParamMap,
ParamValidators}
+import org.apache.spark.ml.param.shared.HasWeightCol
import org.apache.spark.ml.util._
-import org.apache.spark.mllib.classification.{NaiveBayes => OldNaiveBayes}
-import org.apache.spark.mllib.classification.{NaiveBayesModel =>
OldNaiveBayesModel}
-import org.apache.spark.mllib.regression.{LabeledPoint => OldLabeledPoint}
-import org.apache.spark.mllib.util.MLUtils
-import org.apache.spark.rdd.RDD
-import org.apache.spark.sql.{Dataset, Row}
+import org.apache.spark.sql.Dataset
+import org.apache.spark.sql.functions.{col, lit}
+import org.apache.spark.sql.types.DoubleType
/**
* Params for Naive Bayes Classifiers.
*/
-private[ml] trait NaiveBayesParams extends PredictorParams {
+private[ml] trait NaiveBayesParams extends PredictorParams with HasWeightCol {
/**
* The smoothing parameter.
@@ -56,7 +53,7 @@ private[ml] trait NaiveBayesParams extends PredictorParams {
*/
final val modelType: Param[String] = new Param[String](this, "modelType",
"The model type " +
"which is a string (case-sensitive). Supported options: multinomial
(default) and bernoulli.",
-ParamValidators.inArray[String](OldNaiveBayes.supportedModelTypes.toArray))
+ParamValidators.inArray[String](NaiveBayes.supportedModelTypes.toArray))
/** @group getParam */
final def getModelType: String = $(modelType)
@@ -64,7 +61,7 @@ private[ml] trait NaiveBayesParams extends PredictorParams {
/**
* Naive Bayes Classifiers.
- * It supports both Multinomial NB
+ * It supports Multinomial NB
*
([[http://nlp.stanford.edu/IR-book/html/htmledition/naive-bayes-text-classification-1.html]])
* which can handle finitely supported discrete data. For example, by
converting documents into
* TF-IDF vectors, it can be used for document classification. By making every
vector a
@@ -78,6 +75,8 @@ class NaiveBayes @Since("1.5.0") (
extends ProbabilisticClassifier[Vector, NaiveBayes, NaiveBayesModel]
with NaiveBayesParams with DefaultParamsWritable {
+ import NaiveBayes.{Bernoulli, Multinomial}
+
@Since("1.5.0")
def this() = this(Identifiable.randomUID("nb"))
@@ -98,7 +97,17 @@ class NaiveBayes @Since("1.5.0") (
*/
@Since("1.5.0")
def setModelType(value: String): this.type = set(modelType, value)
- setDefault(modelType -> OldNaiveBayes.Multinomial)
+ setDefault(modelType -> NaiveBayes.Multinomial)
+
+ /**
+ * Sets the value of param [[weightCol]].
+ * If this is not set or empty, we treat all instance weights as 1.0.
+ * Default is not set, so all instances have weight one.
+ *
+ * @group setParam
+ */
+ @Since("2.1.0")
+ def setWeightCol(value: String): this.type = set(weightCol, value)
override protected def train(dataset: Dataset[_]): NaiveBayesModel = {
val numCl
spark git commit: [SPARK-14077][ML][FOLLOW-UP] Revert change for NB Model's Load to maintain compatibility with the model stored before 2.0
Repository: spark
Updated Branches:
refs/heads/master 1fad55968 -> 8e491af52
[SPARK-14077][ML][FOLLOW-UP] Revert change for NB Model's Load to maintain
compatibility with the model stored before 2.0
## What changes were proposed in this pull request?
Revert change for NB Model's Load to maintain compatibility with the model
stored before 2.0
## How was this patch tested?
local build
Author: Zheng RuiFeng
Closes #15313 from zhengruifeng/revert_save_load.
Project: http://git-wip-us.apache.org/repos/asf/spark/repo
Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/8e491af5
Tree: http://git-wip-us.apache.org/repos/asf/spark/tree/8e491af5
Diff: http://git-wip-us.apache.org/repos/asf/spark/diff/8e491af5
Branch: refs/heads/master
Commit: 8e491af52930886cbe0c54e7d67add3796ddb15f
Parents: 1fad559
Author: Zheng RuiFeng
Authored: Fri Sep 30 08:18:48 2016 -0700
Committer: Yanbo Liang
Committed: Fri Sep 30 08:18:48 2016 -0700
--
.../org/apache/spark/ml/classification/NaiveBayes.scala | 11 +++
1 file changed, 7 insertions(+), 4 deletions(-)
--
http://git-wip-us.apache.org/repos/asf/spark/blob/8e491af5/mllib/src/main/scala/org/apache/spark/ml/classification/NaiveBayes.scala
--
diff --git
a/mllib/src/main/scala/org/apache/spark/ml/classification/NaiveBayes.scala
b/mllib/src/main/scala/org/apache/spark/ml/classification/NaiveBayes.scala
index 0d652aa..6775745 100644
--- a/mllib/src/main/scala/org/apache/spark/ml/classification/NaiveBayes.scala
+++ b/mllib/src/main/scala/org/apache/spark/ml/classification/NaiveBayes.scala
@@ -25,7 +25,8 @@ import org.apache.spark.ml.linalg._
import org.apache.spark.ml.param.{DoubleParam, Param, ParamMap,
ParamValidators}
import org.apache.spark.ml.param.shared.HasWeightCol
import org.apache.spark.ml.util._
-import org.apache.spark.sql.Dataset
+import org.apache.spark.mllib.util.MLUtils
+import org.apache.spark.sql.{Dataset, Row}
import org.apache.spark.sql.functions.{col, lit}
import org.apache.spark.sql.types.DoubleType
@@ -362,9 +363,11 @@ object NaiveBayesModel extends MLReadable[NaiveBayesModel]
{
val metadata = DefaultParamsReader.loadMetadata(path, sc, className)
val dataPath = new Path(path, "data").toString
- val data = sparkSession.read.parquet(dataPath).select("pi",
"theta").head()
- val pi = data.getAs[Vector](0)
- val theta = data.getAs[Matrix](1)
+ val data = sparkSession.read.parquet(dataPath)
+ val vecConverted = MLUtils.convertVectorColumnsToML(data, "pi")
+ val Row(pi: Vector, theta: Matrix) =
MLUtils.convertMatrixColumnsToML(vecConverted, "theta")
+.select("pi", "theta")
+.head()
val model = new NaiveBayesModel(metadata.uid, pi, theta)
DefaultParamsReader.getAndSetParams(model, metadata)
-
To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org
For additional commands, e-mail: commits-h...@spark.apache.org
spark git commit: [SPARK-17744][ML] Parity check between the ml and mllib test suites for NB
Repository: spark
Updated Branches:
refs/heads/master 7d5160883 -> c17f97183
[SPARK-17744][ML] Parity check between the ml and mllib test suites for NB
## What changes were proposed in this pull request?
1,parity check and add missing test suites for ml's NB
2,remove some unused imports
## How was this patch tested?
manual tests in spark-shell
Author: Zheng RuiFeng
Closes #15312 from zhengruifeng/nb_test_parity.
Project: http://git-wip-us.apache.org/repos/asf/spark/repo
Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/c17f9718
Tree: http://git-wip-us.apache.org/repos/asf/spark/tree/c17f9718
Diff: http://git-wip-us.apache.org/repos/asf/spark/diff/c17f9718
Branch: refs/heads/master
Commit: c17f971839816e68f8abe2c8eb4e4db47c57ab67
Parents: 7d51608
Author: Zheng RuiFeng
Authored: Tue Oct 4 06:54:48 2016 -0700
Committer: Yanbo Liang
Committed: Tue Oct 4 06:54:48 2016 -0700
--
.../apache/spark/ml/feature/LabeledPoint.scala | 2 +-
.../spark/ml/feature/QuantileDiscretizer.scala | 2 +-
.../org/apache/spark/ml/python/MLSerDe.scala| 5 --
.../spark/ml/regression/GBTRegressor.scala | 2 +-
.../spark/ml/regression/LinearRegression.scala | 1 -
.../ml/classification/NaiveBayesSuite.scala | 69 +++-
python/pyspark/ml/classification.py | 1 -
7 files changed, 70 insertions(+), 12 deletions(-)
--
http://git-wip-us.apache.org/repos/asf/spark/blob/c17f9718/mllib/src/main/scala/org/apache/spark/ml/feature/LabeledPoint.scala
--
diff --git
a/mllib/src/main/scala/org/apache/spark/ml/feature/LabeledPoint.scala
b/mllib/src/main/scala/org/apache/spark/ml/feature/LabeledPoint.scala
index 6cefa70..7d8e4ad 100644
--- a/mllib/src/main/scala/org/apache/spark/ml/feature/LabeledPoint.scala
+++ b/mllib/src/main/scala/org/apache/spark/ml/feature/LabeledPoint.scala
@@ -25,7 +25,7 @@ import org.apache.spark.ml.linalg.Vector
/**
* :: Experimental ::
*
- * Class that represents the features and labels of a data point.
+ * Class that represents the features and label of a data point.
*
* @param label Label for this data point.
* @param features List of features for this data point.
http://git-wip-us.apache.org/repos/asf/spark/blob/c17f9718/mllib/src/main/scala/org/apache/spark/ml/feature/QuantileDiscretizer.scala
--
diff --git
a/mllib/src/main/scala/org/apache/spark/ml/feature/QuantileDiscretizer.scala
b/mllib/src/main/scala/org/apache/spark/ml/feature/QuantileDiscretizer.scala
index 1e59d71..05e034d 100644
--- a/mllib/src/main/scala/org/apache/spark/ml/feature/QuantileDiscretizer.scala
+++ b/mllib/src/main/scala/org/apache/spark/ml/feature/QuantileDiscretizer.scala
@@ -25,7 +25,7 @@ import org.apache.spark.ml.param._
import org.apache.spark.ml.param.shared.{HasInputCol, HasOutputCol}
import org.apache.spark.ml.util._
import org.apache.spark.sql.Dataset
-import org.apache.spark.sql.types.{DoubleType, StructType}
+import org.apache.spark.sql.types.StructType
/**
* Params for [[QuantileDiscretizer]].
http://git-wip-us.apache.org/repos/asf/spark/blob/c17f9718/mllib/src/main/scala/org/apache/spark/ml/python/MLSerDe.scala
--
diff --git a/mllib/src/main/scala/org/apache/spark/ml/python/MLSerDe.scala
b/mllib/src/main/scala/org/apache/spark/ml/python/MLSerDe.scala
index 4b805e1..da62f85 100644
--- a/mllib/src/main/scala/org/apache/spark/ml/python/MLSerDe.scala
+++ b/mllib/src/main/scala/org/apache/spark/ml/python/MLSerDe.scala
@@ -19,17 +19,12 @@ package org.apache.spark.ml.python
import java.io.OutputStream
import java.nio.{ByteBuffer, ByteOrder}
-import java.util.{ArrayList => JArrayList}
-
-import scala.collection.JavaConverters._
import net.razorvine.pickle._
-import org.apache.spark.api.java.JavaRDD
import org.apache.spark.api.python.SerDeUtil
import org.apache.spark.ml.linalg._
import org.apache.spark.mllib.api.python.SerDeBase
-import org.apache.spark.rdd.RDD
/**
* SerDe utility functions for pyspark.ml.
http://git-wip-us.apache.org/repos/asf/spark/blob/c17f9718/mllib/src/main/scala/org/apache/spark/ml/regression/GBTRegressor.scala
--
diff --git
a/mllib/src/main/scala/org/apache/spark/ml/regression/GBTRegressor.scala
b/mllib/src/main/scala/org/apache/spark/ml/regression/GBTRegressor.scala
index ce35593..bb01f9d 100644
--- a/mllib/src/main/scala/org/apache/spark/ml/regression/GBTRegressor.scala
+++ b/mllib/src/main/scala/org/apache/spark/ml/regression/GBTRegressor.scala
@@ -21,7 +21,7 @@ import com.github.fommil.netlib.BLAS.{getInstance => blas}
import org.json4s.{DefaultFormats, JObject}
import org.json4s.JsonDSL._
-
spark git commit: [MINOR][ML] Avoid 2D array flatten in NB training.
Repository: spark
Updated Branches:
refs/heads/master b678e465a -> 7aeb20be7
[MINOR][ML] Avoid 2D array flatten in NB training.
## What changes were proposed in this pull request?
Avoid 2D array flatten in ```NaiveBayes``` training, since flatten method might
be expensive (It will create another array and copy data there).
## How was this patch tested?
Existing tests.
Author: Yanbo Liang
Closes #15359 from yanboliang/nb-theta.
Project: http://git-wip-us.apache.org/repos/asf/spark/repo
Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/7aeb20be
Tree: http://git-wip-us.apache.org/repos/asf/spark/tree/7aeb20be
Diff: http://git-wip-us.apache.org/repos/asf/spark/diff/7aeb20be
Branch: refs/heads/master
Commit: 7aeb20be7e999523784aca7be1a7c9c99dec125e
Parents: b678e46
Author: Yanbo Liang
Authored: Wed Oct 5 23:03:09 2016 -0700
Committer: Yanbo Liang
Committed: Wed Oct 5 23:03:09 2016 -0700
--
.../org/apache/spark/ml/classification/NaiveBayes.scala | 8
1 file changed, 4 insertions(+), 4 deletions(-)
--
http://git-wip-us.apache.org/repos/asf/spark/blob/7aeb20be/mllib/src/main/scala/org/apache/spark/ml/classification/NaiveBayes.scala
--
diff --git
a/mllib/src/main/scala/org/apache/spark/ml/classification/NaiveBayes.scala
b/mllib/src/main/scala/org/apache/spark/ml/classification/NaiveBayes.scala
index 6775745..e565a6f 100644
--- a/mllib/src/main/scala/org/apache/spark/ml/classification/NaiveBayes.scala
+++ b/mllib/src/main/scala/org/apache/spark/ml/classification/NaiveBayes.scala
@@ -176,8 +176,8 @@ class NaiveBayes @Since("1.5.0") (
val numLabels = aggregated.length
val numDocuments = aggregated.map(_._2._1).sum
-val piArray = Array.fill[Double](numLabels)(0.0)
-val thetaArrays = Array.fill[Double](numLabels, numFeatures)(0.0)
+val piArray = new Array[Double](numLabels)
+val thetaArray = new Array[Double](numLabels * numFeatures)
val lambda = $(smoothing)
val piLogDenom = math.log(numDocuments + numLabels * lambda)
@@ -193,14 +193,14 @@ class NaiveBayes @Since("1.5.0") (
}
var j = 0
while (j < numFeatures) {
-thetaArrays(i)(j) = math.log(sumTermFreqs(j) + lambda) - thetaLogDenom
+thetaArray(i * numFeatures + j) = math.log(sumTermFreqs(j) + lambda) -
thetaLogDenom
j += 1
}
i += 1
}
val pi = Vectors.dense(piArray)
-val theta = new DenseMatrix(numLabels, thetaArrays(0).length,
thetaArrays.flatten, true)
+val theta = new DenseMatrix(numLabels, numFeatures, thetaArray, true)
new NaiveBayesModel(uid, pi, theta)
}
-
To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org
For additional commands, e-mail: commits-h...@spark.apache.org
spark git commit: [SPARK-17792][ML] L-BFGS solver for linear regression does not accept general numeric label column types
Repository: spark
Updated Branches:
refs/heads/master 49d11d499 -> 3713bb199
[SPARK-17792][ML] L-BFGS solver for linear regression does not accept general
numeric label column types
## What changes were proposed in this pull request?
Before, we computed `instances` in LinearRegression in two spots, even though
they did the same thing. One of them did not cast the label column to
`DoubleType`. This patch consolidates the computation and always casts the
label column to `DoubleType`.
## How was this patch tested?
Added a unit test to check all solvers. This test failed before this patch.
Author: sethah
Closes #15364 from sethah/linreg_numeric_type.
Project: http://git-wip-us.apache.org/repos/asf/spark/repo
Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/3713bb19
Tree: http://git-wip-us.apache.org/repos/asf/spark/tree/3713bb19
Diff: http://git-wip-us.apache.org/repos/asf/spark/diff/3713bb19
Branch: refs/heads/master
Commit: 3713bb199142c5e06e2e527c99650f02f41f47b1
Parents: 49d11d4
Author: sethah
Authored: Thu Oct 6 21:10:17 2016 -0700
Committer: Yanbo Liang
Committed: Thu Oct 6 21:10:17 2016 -0700
--
.../spark/ml/regression/LinearRegression.scala | 17 ++---
.../ml/regression/LinearRegressionSuite.scala | 8 +---
2 files changed, 11 insertions(+), 14 deletions(-)
--
http://git-wip-us.apache.org/repos/asf/spark/blob/3713bb19/mllib/src/main/scala/org/apache/spark/ml/regression/LinearRegression.scala
--
diff --git
a/mllib/src/main/scala/org/apache/spark/ml/regression/LinearRegression.scala
b/mllib/src/main/scala/org/apache/spark/ml/regression/LinearRegression.scala
index 536c58f..025ed20 100644
--- a/mllib/src/main/scala/org/apache/spark/ml/regression/LinearRegression.scala
+++ b/mllib/src/main/scala/org/apache/spark/ml/regression/LinearRegression.scala
@@ -188,17 +188,18 @@ class LinearRegression @Since("1.3.0") (@Since("1.3.0")
override val uid: String
val numFeatures =
dataset.select(col($(featuresCol))).first().getAs[Vector](0).size
val w = if (!isDefined(weightCol) || $(weightCol).isEmpty) lit(1.0) else
col($(weightCol))
+val instances: RDD[Instance] = dataset.select(
+ col($(labelCol)).cast(DoubleType), w, col($(featuresCol))).rdd.map {
+ case Row(label: Double, weight: Double, features: Vector) =>
+Instance(label, weight, features)
+}
+
if (($(solver) == "auto" && $(elasticNetParam) == 0.0 &&
numFeatures <= WeightedLeastSquares.MAX_NUM_FEATURES) || $(solver) ==
"normal") {
require($(elasticNetParam) == 0.0, "Only L2 regularization can be used
when normal " +
"solver is used.'")
// For low dimensional data, WeightedLeastSquares is more efficiently
since the
// training algorithm only requires one pass through the data.
(SPARK-10668)
- val instances: RDD[Instance] = dataset.select(
-col($(labelCol)).cast(DoubleType), w, col($(featuresCol))).rdd.map {
- case Row(label: Double, weight: Double, features: Vector) =>
-Instance(label, weight, features)
- }
val optimizer = new WeightedLeastSquares($(fitIntercept), $(regParam),
$(standardization), true)
@@ -221,12 +222,6 @@ class LinearRegression @Since("1.3.0") (@Since("1.3.0")
override val uid: String
return lrModel.setSummary(trainingSummary)
}
-val instances: RDD[Instance] =
- dataset.select(col($(labelCol)), w, col($(featuresCol))).rdd.map {
-case Row(label: Double, weight: Double, features: Vector) =>
- Instance(label, weight, features)
- }
-
val handlePersistence = dataset.rdd.getStorageLevel == StorageLevel.NONE
if (handlePersistence) instances.persist(StorageLevel.MEMORY_AND_DISK)
http://git-wip-us.apache.org/repos/asf/spark/blob/3713bb19/mllib/src/test/scala/org/apache/spark/ml/regression/LinearRegressionSuite.scala
--
diff --git
a/mllib/src/test/scala/org/apache/spark/ml/regression/LinearRegressionSuite.scala
b/mllib/src/test/scala/org/apache/spark/ml/regression/LinearRegressionSuite.scala
index 5ae371b..1c94ec6 100644
---
a/mllib/src/test/scala/org/apache/spark/ml/regression/LinearRegressionSuite.scala
+++
b/mllib/src/test/scala/org/apache/spark/ml/regression/LinearRegressionSuite.scala
@@ -1015,12 +1015,14 @@ class LinearRegressionSuite
}
test("should support all NumericType labels and not support other types") {
-val lr = new LinearRegression().setMaxIter(1)
-MLTestingUtils.checkNumericTypes[LinearRegressionModel, LinearRegression](
- lr, spark, isClassification = false) { (expected, actual) =>
+for (solver <- Seq("auto", "l-bfgs", "normal")) {
+ val lr = new LinearRegression().setMaxIter
spark git commit: [SPARK-17792][ML] L-BFGS solver for linear regression does not accept general numeric label column types
Repository: spark
Updated Branches:
refs/heads/branch-2.0 b1a9c41e8 -> 594a2cf6f
[SPARK-17792][ML] L-BFGS solver for linear regression does not accept general
numeric label column types
## What changes were proposed in this pull request?
Before, we computed `instances` in LinearRegression in two spots, even though
they did the same thing. One of them did not cast the label column to
`DoubleType`. This patch consolidates the computation and always casts the
label column to `DoubleType`.
## How was this patch tested?
Added a unit test to check all solvers. This test failed before this patch.
Author: sethah
Closes #15364 from sethah/linreg_numeric_type.
(cherry picked from commit 3713bb199142c5e06e2e527c99650f02f41f47b1)
Signed-off-by: Yanbo Liang
Project: http://git-wip-us.apache.org/repos/asf/spark/repo
Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/594a2cf6
Tree: http://git-wip-us.apache.org/repos/asf/spark/tree/594a2cf6
Diff: http://git-wip-us.apache.org/repos/asf/spark/diff/594a2cf6
Branch: refs/heads/branch-2.0
Commit: 594a2cf6f7c74c54127b8c3947aadbe0052b404c
Parents: b1a9c41
Author: sethah
Authored: Thu Oct 6 21:10:17 2016 -0700
Committer: Yanbo Liang
Committed: Thu Oct 6 21:14:44 2016 -0700
--
.../spark/ml/regression/LinearRegression.scala | 17 ++---
.../ml/regression/LinearRegressionSuite.scala | 8 +---
2 files changed, 11 insertions(+), 14 deletions(-)
--
http://git-wip-us.apache.org/repos/asf/spark/blob/594a2cf6/mllib/src/main/scala/org/apache/spark/ml/regression/LinearRegression.scala
--
diff --git
a/mllib/src/main/scala/org/apache/spark/ml/regression/LinearRegression.scala
b/mllib/src/main/scala/org/apache/spark/ml/regression/LinearRegression.scala
index f82f2c3..600bbcb 100644
--- a/mllib/src/main/scala/org/apache/spark/ml/regression/LinearRegression.scala
+++ b/mllib/src/main/scala/org/apache/spark/ml/regression/LinearRegression.scala
@@ -163,17 +163,18 @@ class LinearRegression @Since("1.3.0") (@Since("1.3.0")
override val uid: String
val numFeatures =
dataset.select(col($(featuresCol))).first().getAs[Vector](0).size
val w = if (!isDefined(weightCol) || $(weightCol).isEmpty) lit(1.0) else
col($(weightCol))
+val instances: RDD[Instance] = dataset.select(
+ col($(labelCol)).cast(DoubleType), w, col($(featuresCol))).rdd.map {
+ case Row(label: Double, weight: Double, features: Vector) =>
+Instance(label, weight, features)
+}
+
if (($(solver) == "auto" && $(elasticNetParam) == 0.0 &&
numFeatures <= WeightedLeastSquares.MAX_NUM_FEATURES) || $(solver) ==
"normal") {
require($(elasticNetParam) == 0.0, "Only L2 regularization can be used
when normal " +
"solver is used.'")
// For low dimensional data, WeightedLeastSquares is more efficiently
since the
// training algorithm only requires one pass through the data.
(SPARK-10668)
- val instances: RDD[Instance] = dataset.select(
-col($(labelCol)).cast(DoubleType), w, col($(featuresCol))).rdd.map {
- case Row(label: Double, weight: Double, features: Vector) =>
-Instance(label, weight, features)
- }
val optimizer = new WeightedLeastSquares($(fitIntercept), $(regParam),
$(standardization), true)
@@ -196,12 +197,6 @@ class LinearRegression @Since("1.3.0") (@Since("1.3.0")
override val uid: String
return lrModel.setSummary(trainingSummary)
}
-val instances: RDD[Instance] =
- dataset.select(col($(labelCol)), w, col($(featuresCol))).rdd.map {
-case Row(label: Double, weight: Double, features: Vector) =>
- Instance(label, weight, features)
- }
-
val handlePersistence = dataset.rdd.getStorageLevel == StorageLevel.NONE
if (handlePersistence) instances.persist(StorageLevel.MEMORY_AND_DISK)
http://git-wip-us.apache.org/repos/asf/spark/blob/594a2cf6/mllib/src/test/scala/org/apache/spark/ml/regression/LinearRegressionSuite.scala
--
diff --git
a/mllib/src/test/scala/org/apache/spark/ml/regression/LinearRegressionSuite.scala
b/mllib/src/test/scala/org/apache/spark/ml/regression/LinearRegressionSuite.scala
index 265f2f4..df67a3a 100644
---
a/mllib/src/test/scala/org/apache/spark/ml/regression/LinearRegressionSuite.scala
+++
b/mllib/src/test/scala/org/apache/spark/ml/regression/LinearRegressionSuite.scala
@@ -1019,12 +1019,14 @@ class LinearRegressionSuite
}
test("should support all NumericType labels and not support other types") {
-val lr = new LinearRegression().setMaxIter(1)
-MLTestingUtils.checkNumericTypes[LinearRegressionModel, LinearRegression](
- lr, spark, isClassification = false) { (expected, actual) =
spark git commit: [SPARK-15957][ML] RFormula supports forcing to index label
Repository: spark
Updated Branches:
refs/heads/master b515768f2 -> 19401a203
[SPARK-15957][ML] RFormula supports forcing to index label
## What changes were proposed in this pull request?
```RFormula``` will index label only when it is string type currently. If the
label is numeric type and we use ```RFormula``` to present a classification
model, there is no label attributes in label column metadata. The label
attributes are useful when making prediction for classification, so we can
force to index label by ```StringIndexer``` whether it is numeric or string
type for classification. Then SparkR wrappers can extract label attributes from
label column metadata successfully. This feature can help us to fix bug similar
with [SPARK-15153](https://issues.apache.org/jira/browse/SPARK-15153).
For regression, we will still to keep label as numeric type.
In this PR, we add a param ```indexLabel``` to control whether to force to
index label for ```RFormula```.
## How was this patch tested?
Unit tests.
Author: Yanbo Liang
Closes #13675 from yanboliang/spark-15957.
Project: http://git-wip-us.apache.org/repos/asf/spark/repo
Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/19401a20
Tree: http://git-wip-us.apache.org/repos/asf/spark/tree/19401a20
Diff: http://git-wip-us.apache.org/repos/asf/spark/diff/19401a20
Branch: refs/heads/master
Commit: 19401a203b441e3355f0d3fc3fd062b6d5bdee1f
Parents: b515768
Author: Yanbo Liang
Authored: Mon Oct 10 22:50:59 2016 -0700
Committer: Yanbo Liang
Committed: Mon Oct 10 22:50:59 2016 -0700
--
.../org/apache/spark/ml/feature/RFormula.scala | 29 ++--
.../apache/spark/ml/feature/RFormulaSuite.scala | 27 +-
2 files changed, 52 insertions(+), 4 deletions(-)
--
http://git-wip-us.apache.org/repos/asf/spark/blob/19401a20/mllib/src/main/scala/org/apache/spark/ml/feature/RFormula.scala
--
diff --git a/mllib/src/main/scala/org/apache/spark/ml/feature/RFormula.scala
b/mllib/src/main/scala/org/apache/spark/ml/feature/RFormula.scala
index 2ee899b..3898986 100644
--- a/mllib/src/main/scala/org/apache/spark/ml/feature/RFormula.scala
+++ b/mllib/src/main/scala/org/apache/spark/ml/feature/RFormula.scala
@@ -26,7 +26,7 @@ import org.apache.spark.annotation.{Experimental, Since}
import org.apache.spark.ml.{Estimator, Model, Pipeline, PipelineModel,
PipelineStage, Transformer}
import org.apache.spark.ml.attribute.AttributeGroup
import org.apache.spark.ml.linalg.VectorUDT
-import org.apache.spark.ml.param.{Param, ParamMap}
+import org.apache.spark.ml.param.{BooleanParam, Param, ParamMap}
import org.apache.spark.ml.param.shared.{HasFeaturesCol, HasLabelCol}
import org.apache.spark.ml.util._
import org.apache.spark.sql.{DataFrame, Dataset}
@@ -104,6 +104,27 @@ class RFormula @Since("1.5.0") (@Since("1.5.0") override
val uid: String)
@Since("1.5.0")
def setLabelCol(value: String): this.type = set(labelCol, value)
+ /**
+ * Force to index label whether it is numeric or string type.
+ * Usually we index label only when it is string type.
+ * If the formula was used by classification algorithms,
+ * we can force to index label even it is numeric type by setting this param
with true.
+ * Default: false.
+ * @group param
+ */
+ @Since("2.1.0")
+ val forceIndexLabel: BooleanParam = new BooleanParam(this, "forceIndexLabel",
+"Force to index label whether it is numeric or string")
+ setDefault(forceIndexLabel -> false)
+
+ /** @group getParam */
+ @Since("2.1.0")
+ def getForceIndexLabel: Boolean = $(forceIndexLabel)
+
+ /** @group setParam */
+ @Since("2.1.0")
+ def setForceIndexLabel(value: Boolean): this.type = set(forceIndexLabel,
value)
+
/** Whether the formula specifies fitting an intercept. */
private[ml] def hasIntercept: Boolean = {
require(isDefined(formula), "Formula must be defined first.")
@@ -167,8 +188,8 @@ class RFormula @Since("1.5.0") (@Since("1.5.0") override
val uid: String)
encoderStages += new VectorAttributeRewriter($(featuresCol),
prefixesToRewrite.toMap)
encoderStages += new ColumnPruner(tempColumns.toSet)
-if (dataset.schema.fieldNames.contains(resolvedFormula.label) &&
- dataset.schema(resolvedFormula.label).dataType == StringType) {
+if ((dataset.schema.fieldNames.contains(resolvedFormula.label) &&
+ dataset.schema(resolvedFormula.label).dataType == StringType) ||
$(forceIndexLabel)) {
encoderStages += new StringIndexer()
.setInputCol(resolvedFormula.label)
.setOutputCol($(labelCol))
@@ -181,6 +202,8 @@ class RFormula @Since("1.5.0") (@Since("1.5.0") override
val uid: String)
@Since("1.5.0")
// optimistic schema; does not contain any ML attributes
override def transformSchema(schema: StructType):
spark git commit: [SPARK-17745][ML][PYSPARK] update NB python api - add weight col parameter
Repository: spark
Updated Branches:
refs/heads/master 6f20a92ca -> 0d4a69527
[SPARK-17745][ML][PYSPARK] update NB python api - add weight col parameter
## What changes were proposed in this pull request?
update python api for NaiveBayes: add weight col parameter.
## How was this patch tested?
doctests added.
Author: WeichenXu
Closes #15406 from WeichenXu123/nb_python_update.
Project: http://git-wip-us.apache.org/repos/asf/spark/repo
Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/0d4a6952
Tree: http://git-wip-us.apache.org/repos/asf/spark/tree/0d4a6952
Diff: http://git-wip-us.apache.org/repos/asf/spark/diff/0d4a6952
Branch: refs/heads/master
Commit: 0d4a695279c514c76aa0e9288c70ac7aaef91b03
Parents: 6f20a92
Author: WeichenXu
Authored: Wed Oct 12 19:52:57 2016 -0700
Committer: Yanbo Liang
Committed: Wed Oct 12 19:52:57 2016 -0700
--
python/pyspark/ml/classification.py | 26 +-
1 file changed, 13 insertions(+), 13 deletions(-)
--
http://git-wip-us.apache.org/repos/asf/spark/blob/0d4a6952/python/pyspark/ml/classification.py
--
diff --git a/python/pyspark/ml/classification.py
b/python/pyspark/ml/classification.py
index ea60fab..3f763a1 100644
--- a/python/pyspark/ml/classification.py
+++ b/python/pyspark/ml/classification.py
@@ -981,7 +981,7 @@ class GBTClassificationModel(TreeEnsembleModel,
JavaPredictionModel, JavaMLWrita
@inherit_doc
class NaiveBayes(JavaEstimator, HasFeaturesCol, HasLabelCol, HasPredictionCol,
HasProbabilityCol,
- HasRawPredictionCol, HasThresholds, JavaMLWritable,
JavaMLReadable):
+ HasRawPredictionCol, HasThresholds, HasWeightCol,
JavaMLWritable, JavaMLReadable):
"""
Naive Bayes Classifiers.
It supports both Multinomial and Bernoulli NB. `Multinomial NB
@@ -995,23 +995,23 @@ class NaiveBayes(JavaEstimator, HasFeaturesCol,
HasLabelCol, HasPredictionCol, H
>>> from pyspark.sql import Row
>>> from pyspark.ml.linalg import Vectors
>>> df = spark.createDataFrame([
-... Row(label=0.0, features=Vectors.dense([0.0, 0.0])),
-... Row(label=0.0, features=Vectors.dense([0.0, 1.0])),
-... Row(label=1.0, features=Vectors.dense([1.0, 0.0]))])
->>> nb = NaiveBayes(smoothing=1.0, modelType="multinomial")
+... Row(label=0.0, weight=0.1, features=Vectors.dense([0.0, 0.0])),
+... Row(label=0.0, weight=0.5, features=Vectors.dense([0.0, 1.0])),
+... Row(label=1.0, weight=1.0, features=Vectors.dense([1.0, 0.0]))])
+>>> nb = NaiveBayes(smoothing=1.0, modelType="multinomial",
weightCol="weight")
>>> model = nb.fit(df)
>>> model.pi
-DenseVector([-0.51..., -0.91...])
+DenseVector([-0.81..., -0.58...])
>>> model.theta
-DenseMatrix(2, 2, [-1.09..., -0.40..., -0.40..., -1.09...], 1)
+DenseMatrix(2, 2, [-0.91..., -0.51..., -0.40..., -1.09...], 1)
>>> test0 = sc.parallelize([Row(features=Vectors.dense([1.0,
0.0]))]).toDF()
>>> result = model.transform(test0).head()
>>> result.prediction
1.0
>>> result.probability
-DenseVector([0.42..., 0.57...])
+DenseVector([0.32..., 0.67...])
>>> result.rawPrediction
-DenseVector([-1.60..., -1.32...])
+DenseVector([-1.72..., -0.99...])
>>> test1 = sc.parallelize([Row(features=Vectors.sparse(2, [0],
[1.0]))]).toDF()
>>> model.transform(test1).head().prediction
1.0
@@ -1045,11 +1045,11 @@ class NaiveBayes(JavaEstimator, HasFeaturesCol,
HasLabelCol, HasPredictionCol, H
@keyword_only
def __init__(self, featuresCol="features", labelCol="label",
predictionCol="prediction",
probabilityCol="probability",
rawPredictionCol="rawPrediction", smoothing=1.0,
- modelType="multinomial", thresholds=None):
+ modelType="multinomial", thresholds=None, weightCol=None):
"""
__init__(self, featuresCol="features", labelCol="label",
predictionCol="prediction", \
probabilityCol="probability",
rawPredictionCol="rawPrediction", smoothing=1.0, \
- modelType="multinomial", thresholds=None)
+ modelType="multinomial", thresholds=None, weightCol=None)
"""
super(NaiveBayes, self).__init__()
self._java_obj = self._new_java_obj(
@@ -1062,11 +1062,11 @@ class NaiveBayes(JavaEstimator, HasFeaturesCol,
HasLabelCol, HasPredictionCol, H
@since("1.5.0")
def setParams(self, featuresCol="features", labelCol="label",
predictionCol="prediction",
probabilityCol="probability",
rawPredictionCol="rawPrediction", smoothing=1.0,
- modelType="multinomial", thresholds=None):
+ modelType="multinomial", thresholds=None, weightCol=None):
spark git commit: [SPARK-17835][ML][MLLIB] Optimize NaiveBayes mllib wrapper to eliminate extra pass on data
Repository: spark
Updated Branches:
refs/heads/master 0d4a69527 -> 21cb59f1c
[SPARK-17835][ML][MLLIB] Optimize NaiveBayes mllib wrapper to eliminate extra
pass on data
## What changes were proposed in this pull request?
[SPARK-14077](https://issues.apache.org/jira/browse/SPARK-14077) copied the
```NaiveBayes``` implementation from mllib to ml and left mllib as a wrapper.
However, there are some difference between mllib and ml to handle labels:
* mllib allow input labels as {-1, +1}, however, ml assumes the input labels in
range [0, numClasses).
* mllib ```NaiveBayesModel``` expose ```labels``` but ml did not due to the
assumption mention above.
During the copy in
[SPARK-14077](https://issues.apache.org/jira/browse/SPARK-14077), we use
```val labels = data.map(_.label).distinct().collect().sorted```
to get the distinct labels firstly, and then encode the labels for training. It
involves extra Spark job compared with the original implementation. Since
```NaiveBayes``` only do one pass aggregation during training, adding another
one seems less efficient. We can get the labels in a single pass along with
```NaiveBayes``` training and send them to MLlib side.
## How was this patch tested?
Existing tests.
Author: Yanbo Liang
Closes #15402 from yanboliang/spark-17835.
Project: http://git-wip-us.apache.org/repos/asf/spark/repo
Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/21cb59f1
Tree: http://git-wip-us.apache.org/repos/asf/spark/tree/21cb59f1
Diff: http://git-wip-us.apache.org/repos/asf/spark/diff/21cb59f1
Branch: refs/heads/master
Commit: 21cb59f1cd137d96b2596f1abe691b544581cf59
Parents: 0d4a695
Author: Yanbo Liang
Authored: Wed Oct 12 19:56:40 2016 -0700
Committer: Yanbo Liang
Committed: Wed Oct 12 19:56:40 2016 -0700
--
.../spark/ml/classification/NaiveBayes.scala| 46
.../spark/mllib/classification/NaiveBayes.scala | 15 +++
2 files changed, 43 insertions(+), 18 deletions(-)
--
http://git-wip-us.apache.org/repos/asf/spark/blob/21cb59f1/mllib/src/main/scala/org/apache/spark/ml/classification/NaiveBayes.scala
--
diff --git
a/mllib/src/main/scala/org/apache/spark/ml/classification/NaiveBayes.scala
b/mllib/src/main/scala/org/apache/spark/ml/classification/NaiveBayes.scala
index e565a6f..994ed99 100644
--- a/mllib/src/main/scala/org/apache/spark/ml/classification/NaiveBayes.scala
+++ b/mllib/src/main/scala/org/apache/spark/ml/classification/NaiveBayes.scala
@@ -110,16 +110,28 @@ class NaiveBayes @Since("1.5.0") (
@Since("2.1.0")
def setWeightCol(value: String): this.type = set(weightCol, value)
- override protected def train(dataset: Dataset[_]): NaiveBayesModel = {
-val numClasses = getNumClasses(dataset)
+ /**
+ * ml assumes input labels in range [0, numClasses). But this implementation
+ * is also called by mllib NaiveBayes which allows other kinds of input
labels
+ * such as {-1, +1}. Here we use this parameter to switch between different
processing logic.
+ * It should be removed when we remove mllib NaiveBayes.
+ */
+ private[spark] var isML: Boolean = true
-if (isDefined(thresholds)) {
- require($(thresholds).length == numClasses, this.getClass.getSimpleName +
-".train() called with non-matching numClasses and thresholds.length." +
-s" numClasses=$numClasses, but thresholds has length
${$(thresholds).length}")
-}
+ private[spark] def setIsML(isML: Boolean): this.type = {
+this.isML = isML
+this
+ }
-val numFeatures =
dataset.select(col($(featuresCol))).head().getAs[Vector](0).size
+ override protected def train(dataset: Dataset[_]): NaiveBayesModel = {
+if (isML) {
+ val numClasses = getNumClasses(dataset)
+ if (isDefined(thresholds)) {
+require($(thresholds).length == numClasses,
this.getClass.getSimpleName +
+ ".train() called with non-matching numClasses and
thresholds.length." +
+ s" numClasses=$numClasses, but thresholds has length
${$(thresholds).length}")
+ }
+}
val requireNonnegativeValues: Vector => Unit = (v: Vector) => {
val values = v match {
@@ -153,6 +165,7 @@ class NaiveBayes @Since("1.5.0") (
}
}
+val numFeatures =
dataset.select(col($(featuresCol))).head().getAs[Vector](0).size
val w = if (!isDefined(weightCol) || $(weightCol).isEmpty) lit(1.0) else
col($(weightCol))
// Aggregates term frequencies per label.
@@ -176,6 +189,7 @@ class NaiveBayes @Since("1.5.0") (
val numLabels = aggregated.length
val numDocuments = aggregated.map(_._2._1).sum
+val labelArray = new Array[Double](numLabels)
val piArray = new Array[Double](numLabels)
val thetaArray = new Array[Double](numLabels * numFeatures)
@@ -183,6 +197,7 @@ class Nai
spark git commit: [SPARK-15957][FOLLOW-UP][ML][PYSPARK] Add Python API for RFormula forceIndexLabel.
Repository: spark
Updated Branches:
refs/heads/master 9dc0ca060 -> 44cbb61b3
[SPARK-15957][FOLLOW-UP][ML][PYSPARK] Add Python API for RFormula
forceIndexLabel.
## What changes were proposed in this pull request?
Follow-up work of #13675, add Python API for ```RFormula forceIndexLabel```.
## How was this patch tested?
Unit test.
Author: Yanbo Liang
Closes #15430 from yanboliang/spark-15957-python.
Project: http://git-wip-us.apache.org/repos/asf/spark/repo
Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/44cbb61b
Tree: http://git-wip-us.apache.org/repos/asf/spark/tree/44cbb61b
Diff: http://git-wip-us.apache.org/repos/asf/spark/diff/44cbb61b
Branch: refs/heads/master
Commit: 44cbb61b34a98e3e0d8e2543a4eb6e950e0019a5
Parents: 9dc0ca0
Author: Yanbo Liang
Authored: Thu Oct 13 19:44:24 2016 -0700
Committer: Yanbo Liang
Committed: Thu Oct 13 19:44:24 2016 -0700
--
python/pyspark/ml/feature.py | 31 +++
python/pyspark/ml/tests.py | 16
2 files changed, 43 insertions(+), 4 deletions(-)
--
http://git-wip-us.apache.org/repos/asf/spark/blob/44cbb61b/python/pyspark/ml/feature.py
--
diff --git a/python/pyspark/ml/feature.py b/python/pyspark/ml/feature.py
index 64b21ca..a33c3e7 100755
--- a/python/pyspark/ml/feature.py
+++ b/python/pyspark/ml/feature.py
@@ -2494,21 +2494,30 @@ class RFormula(JavaEstimator, HasFeaturesCol,
HasLabelCol, JavaMLReadable, JavaM
formula = Param(Params._dummy(), "formula", "R model formula",
typeConverter=TypeConverters.toString)
+forceIndexLabel = Param(Params._dummy(), "forceIndexLabel",
+"Force to index label whether it is numeric or
string",
+typeConverter=TypeConverters.toBoolean)
+
@keyword_only
-def __init__(self, formula=None, featuresCol="features", labelCol="label"):
+def __init__(self, formula=None, featuresCol="features", labelCol="label",
+ forceIndexLabel=False):
"""
-__init__(self, formula=None, featuresCol="features", labelCol="label")
+__init__(self, formula=None, featuresCol="features", labelCol="label",
\
+ forceIndexLabel=False)
"""
super(RFormula, self).__init__()
self._java_obj =
self._new_java_obj("org.apache.spark.ml.feature.RFormula", self.uid)
+self._setDefault(forceIndexLabel=False)
kwargs = self.__init__._input_kwargs
self.setParams(**kwargs)
@keyword_only
@since("1.5.0")
-def setParams(self, formula=None, featuresCol="features",
labelCol="label"):
+def setParams(self, formula=None, featuresCol="features", labelCol="label",
+ forceIndexLabel=False):
"""
-setParams(self, formula=None, featuresCol="features", labelCol="label")
+setParams(self, formula=None, featuresCol="features",
labelCol="label", \
+ forceIndexLabel=False)
Sets params for RFormula.
"""
kwargs = self.setParams._input_kwargs
@@ -2528,6 +2537,20 @@ class RFormula(JavaEstimator, HasFeaturesCol,
HasLabelCol, JavaMLReadable, JavaM
"""
return self.getOrDefault(self.formula)
+@since("2.1.0")
+def setForceIndexLabel(self, value):
+"""
+Sets the value of :py:attr:`forceIndexLabel`.
+"""
+return self._set(forceIndexLabel=value)
+
+@since("2.1.0")
+def getForceIndexLabel(self):
+"""
+Gets the value of :py:attr:`forceIndexLabel`.
+"""
+return self.getOrDefault(self.forceIndexLabel)
+
def _create_model(self, java_model):
return RFormulaModel(java_model)
http://git-wip-us.apache.org/repos/asf/spark/blob/44cbb61b/python/pyspark/ml/tests.py
--
diff --git a/python/pyspark/ml/tests.py b/python/pyspark/ml/tests.py
index e233549..9d46cc3 100755
--- a/python/pyspark/ml/tests.py
+++ b/python/pyspark/ml/tests.py
@@ -477,6 +477,22 @@ class FeatureTests(SparkSessionTestCase):
feature, expected = r
self.assertEqual(feature, expected)
+def test_rformula_force_index_label(self):
+df = self.spark.createDataFrame([
+(1.0, 1.0, "a"),
+(0.0, 2.0, "b"),
+(1.0, 0.0, "a")], ["y", "x", "s"])
+# Does not index label by default since it's numeric type.
+rf = RFormula(formula="y ~ x + s")
+model = rf.fit(df)
+transformedDF = model.transform(df)
+self.assertEqual(transformedDF.head().label, 1.0)
+# Force to index label.
+rf2 = RFormula(formula="y ~ x + s").setForceIndexLabel(True)
+model2 = rf2.fit(df)
+transformedDF2 = mo
spark git commit: [SPARK-15402][ML][PYSPARK] PySpark ml.evaluation should support save/load
Repository: spark
Updated Branches:
refs/heads/master 2fb12b0a3 -> 1db8feab8
[SPARK-15402][ML][PYSPARK] PySpark ml.evaluation should support save/load
## What changes were proposed in this pull request?
Since ```ml.evaluation``` has supported save/load at Scala side, supporting it
at Python side is very straightforward and easy.
## How was this patch tested?
Add python doctest.
Author: Yanbo Liang
Closes #13194 from yanboliang/spark-15402.
Project: http://git-wip-us.apache.org/repos/asf/spark/repo
Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/1db8feab
Tree: http://git-wip-us.apache.org/repos/asf/spark/tree/1db8feab
Diff: http://git-wip-us.apache.org/repos/asf/spark/diff/1db8feab
Branch: refs/heads/master
Commit: 1db8feab8c564053c05e8bdc1a7f5026fd637d4f
Parents: 2fb12b0
Author: Yanbo Liang
Authored: Fri Oct 14 04:17:03 2016 -0700
Committer: Yanbo Liang
Committed: Fri Oct 14 04:17:03 2016 -0700
--
python/pyspark/ml/evaluation.py | 45
1 file changed, 36 insertions(+), 9 deletions(-)
--
http://git-wip-us.apache.org/repos/asf/spark/blob/1db8feab/python/pyspark/ml/evaluation.py
--
diff --git a/python/pyspark/ml/evaluation.py b/python/pyspark/ml/evaluation.py
index 1fe8772..7aa16fa 100644
--- a/python/pyspark/ml/evaluation.py
+++ b/python/pyspark/ml/evaluation.py
@@ -22,6 +22,7 @@ from pyspark.ml.wrapper import JavaParams
from pyspark.ml.param import Param, Params, TypeConverters
from pyspark.ml.param.shared import HasLabelCol, HasPredictionCol,
HasRawPredictionCol
from pyspark.ml.common import inherit_doc
+from pyspark.ml.util import JavaMLReadable, JavaMLWritable
__all__ = ['Evaluator', 'BinaryClassificationEvaluator', 'RegressionEvaluator',
'MulticlassClassificationEvaluator']
@@ -103,7 +104,8 @@ class JavaEvaluator(JavaParams, Evaluator):
@inherit_doc
-class BinaryClassificationEvaluator(JavaEvaluator, HasLabelCol,
HasRawPredictionCol):
+class BinaryClassificationEvaluator(JavaEvaluator, HasLabelCol,
HasRawPredictionCol,
+JavaMLReadable, JavaMLWritable):
"""
.. note:: Experimental
@@ -121,6 +123,11 @@ class BinaryClassificationEvaluator(JavaEvaluator,
HasLabelCol, HasRawPrediction
0.70...
>>> evaluator.evaluate(dataset, {evaluator.metricName: "areaUnderPR"})
0.83...
+>>> bce_path = temp_path + "/bce"
+>>> evaluator.save(bce_path)
+>>> evaluator2 = BinaryClassificationEvaluator.load(bce_path)
+>>> str(evaluator2.getRawPredictionCol())
+'raw'
.. versionadded:: 1.4.0
"""
@@ -172,7 +179,8 @@ class BinaryClassificationEvaluator(JavaEvaluator,
HasLabelCol, HasRawPrediction
@inherit_doc
-class RegressionEvaluator(JavaEvaluator, HasLabelCol, HasPredictionCol):
+class RegressionEvaluator(JavaEvaluator, HasLabelCol, HasPredictionCol,
+ JavaMLReadable, JavaMLWritable):
"""
.. note:: Experimental
@@ -190,6 +198,11 @@ class RegressionEvaluator(JavaEvaluator, HasLabelCol,
HasPredictionCol):
0.993...
>>> evaluator.evaluate(dataset, {evaluator.metricName: "mae"})
2.649...
+>>> re_path = temp_path + "/re"
+>>> evaluator.save(re_path)
+>>> evaluator2 = RegressionEvaluator.load(re_path)
+>>> str(evaluator2.getPredictionCol())
+'raw'
.. versionadded:: 1.4.0
"""
@@ -244,7 +257,8 @@ class RegressionEvaluator(JavaEvaluator, HasLabelCol,
HasPredictionCol):
@inherit_doc
-class MulticlassClassificationEvaluator(JavaEvaluator, HasLabelCol,
HasPredictionCol):
+class MulticlassClassificationEvaluator(JavaEvaluator, HasLabelCol,
HasPredictionCol,
+JavaMLReadable, JavaMLWritable):
"""
.. note:: Experimental
@@ -260,6 +274,11 @@ class MulticlassClassificationEvaluator(JavaEvaluator,
HasLabelCol, HasPredictio
0.66...
>>> evaluator.evaluate(dataset, {evaluator.metricName: "accuracy"})
0.66...
+>>> mce_path = temp_path + "/mce"
+>>> evaluator.save(mce_path)
+>>> evaluator2 = MulticlassClassificationEvaluator.load(mce_path)
+>>> str(evaluator2.getPredictionCol())
+'prediction'
.. versionadded:: 1.5.0
"""
@@ -311,19 +330,27 @@ class MulticlassClassificationEvaluator(JavaEvaluator,
HasLabelCol, HasPredictio
if __name__ == "__main__":
import doctest
+import tempfile
+import pyspark.ml.evaluation
from pyspark.sql import SparkSession
-globs = globals().copy()
+globs = pyspark.ml.evaluation.__dict__.copy()
# The small batch size here ensures that we see multiple batches,
# even in these small test examples:
spark = SparkSession.builder\
.master("local[2]")\
.appName("ml.evaluation tests")\
.getO
spark git commit: [SPARK-14634][ML] Add BisectingKMeansSummary
Repository: spark
Updated Branches:
refs/heads/master 1db8feab8 -> a1b136d05
[SPARK-14634][ML] Add BisectingKMeansSummary
## What changes were proposed in this pull request?
Add BisectingKMeansSummary
## How was this patch tested?
unit test
Author: Zheng RuiFeng
Closes #12394 from zhengruifeng/biKMSummary.
Project: http://git-wip-us.apache.org/repos/asf/spark/repo
Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/a1b136d0
Tree: http://git-wip-us.apache.org/repos/asf/spark/tree/a1b136d0
Diff: http://git-wip-us.apache.org/repos/asf/spark/diff/a1b136d0
Branch: refs/heads/master
Commit: a1b136d05c6c458ae8211b0844bfc98d7693fa42
Parents: 1db8fea
Author: Zheng RuiFeng
Authored: Fri Oct 14 04:25:14 2016 -0700
Committer: Yanbo Liang
Committed: Fri Oct 14 04:25:14 2016 -0700
--
.../spark/ml/clustering/BisectingKMeans.scala | 74 +++-
.../ml/clustering/BisectingKMeansSuite.scala| 18 -
.../ml/clustering/GaussianMixtureSuite.scala| 2 +-
.../spark/ml/clustering/KMeansSuite.scala | 2 +-
4 files changed, 91 insertions(+), 5 deletions(-)
--
http://git-wip-us.apache.org/repos/asf/spark/blob/a1b136d0/mllib/src/main/scala/org/apache/spark/ml/clustering/BisectingKMeans.scala
--
diff --git
a/mllib/src/main/scala/org/apache/spark/ml/clustering/BisectingKMeans.scala
b/mllib/src/main/scala/org/apache/spark/ml/clustering/BisectingKMeans.scala
index a97bd0f..add8ee2 100644
--- a/mllib/src/main/scala/org/apache/spark/ml/clustering/BisectingKMeans.scala
+++ b/mllib/src/main/scala/org/apache/spark/ml/clustering/BisectingKMeans.scala
@@ -19,6 +19,7 @@ package org.apache.spark.ml.clustering
import org.apache.hadoop.fs.Path
+import org.apache.spark.SparkException
import org.apache.spark.annotation.{Experimental, Since}
import org.apache.spark.ml.{Estimator, Model}
import org.apache.spark.ml.linalg.{Vector, VectorUDT}
@@ -127,6 +128,29 @@ class BisectingKMeansModel private[ml] (
@Since("2.0.0")
override def write: MLWriter = new
BisectingKMeansModel.BisectingKMeansModelWriter(this)
+
+ private var trainingSummary: Option[BisectingKMeansSummary] = None
+
+ private[clustering] def setSummary(summary: BisectingKMeansSummary):
this.type = {
+this.trainingSummary = Some(summary)
+this
+ }
+
+ /**
+ * Return true if there exists summary of model.
+ */
+ @Since("2.1.0")
+ def hasSummary: Boolean = trainingSummary.nonEmpty
+
+ /**
+ * Gets summary of model on training set. An exception is
+ * thrown if `trainingSummary == None`.
+ */
+ @Since("2.1.0")
+ def summary: BisectingKMeansSummary = trainingSummary.getOrElse {
+throw new SparkException(
+ s"No training summary available for the ${this.getClass.getSimpleName}")
+ }
}
object BisectingKMeansModel extends MLReadable[BisectingKMeansModel] {
@@ -228,14 +252,22 @@ class BisectingKMeans @Since("2.0.0") (
case Row(point: Vector) => OldVectors.fromML(point)
}
+val instr = Instrumentation.create(this, rdd)
+instr.logParams(featuresCol, predictionCol, k, maxIter, seed,
minDivisibleClusterSize)
+
val bkm = new MLlibBisectingKMeans()
.setK($(k))
.setMaxIterations($(maxIter))
.setMinDivisibleClusterSize($(minDivisibleClusterSize))
.setSeed($(seed))
val parentModel = bkm.run(rdd)
-val model = new BisectingKMeansModel(uid, parentModel)
-copyValues(model.setParent(this))
+val model = copyValues(new BisectingKMeansModel(uid,
parentModel).setParent(this))
+val summary = new BisectingKMeansSummary(
+ model.transform(dataset), $(predictionCol), $(featuresCol), $(k))
+model.setSummary(summary)
+val m = model.setSummary(summary)
+instr.logSuccess(m)
+m
}
@Since("2.0.0")
@@ -251,3 +283,41 @@ object BisectingKMeans extends
DefaultParamsReadable[BisectingKMeans] {
@Since("2.0.0")
override def load(path: String): BisectingKMeans = super.load(path)
}
+
+
+/**
+ * :: Experimental ::
+ * Summary of BisectingKMeans.
+ *
+ * @param predictions [[DataFrame]] produced by
[[BisectingKMeansModel.transform()]]
+ * @param predictionCol Name for column of predicted clusters in `predictions`
+ * @param featuresCol Name for column of features in `predictions`
+ * @param k Number of clusters
+ */
+@Since("2.1.0")
+@Experimental
+class BisectingKMeansSummary private[clustering] (
+@Since("2.1.0") @transient val predictions: DataFrame,
+@Since("2.1.0") val predictionCol: String,
+@Since("2.1.0") val featuresCol: String,
+@Since("2.1.0") val k: Int) extends Serializable {
+
+ /**
+ * Cluster centers of the transformed data.
+ */
+ @Since("2.1.0")
+ @transient lazy val cluster: DataFrame = predictions.select(predictionCol)
+
+ /**
+ * Size of (number of data points
spark git commit: [SPARK-17986][ML] SQLTransformer should remove temporary tables
Repository: spark
Updated Branches:
refs/heads/master 01b26a064 -> ab3363e9f
[SPARK-17986][ML] SQLTransformer should remove temporary tables
## What changes were proposed in this pull request?
A call to the method `SQLTransformer.transform` previously would create a
temporary table and never delete it. This change adds a call to
`dropTempView()` that deletes this temporary table before returning the result
so that the table will not remain in spark's table catalog. Because `tableName`
is randomized and not exposed, there should be no expected use of this table
outside of the `transform` method.
## How was this patch tested?
A single new assertion was added to the existing test of the
`SQLTransformer.transform` method that all temporary tables are removed.
Without the corresponding code change, this new assertion fails. I am not aware
of any circumstances in which removing this temporary view would be bad for
performance or correctness in other ways, but some expertise here would be
helpful.
Author: Drew Robb
Closes #15526 from drewrobb/SPARK-17986.
Project: http://git-wip-us.apache.org/repos/asf/spark/repo
Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/ab3363e9
Tree: http://git-wip-us.apache.org/repos/asf/spark/tree/ab3363e9
Diff: http://git-wip-us.apache.org/repos/asf/spark/diff/ab3363e9
Branch: refs/heads/master
Commit: ab3363e9f6b1f7fc26682509fe7382c570f91778
Parents: 01b26a0
Author: Drew Robb
Authored: Sat Oct 22 01:59:36 2016 -0700
Committer: Yanbo Liang
Committed: Sat Oct 22 01:59:36 2016 -0700
--
.../main/scala/org/apache/spark/ml/feature/SQLTransformer.scala | 4 +++-
.../scala/org/apache/spark/ml/feature/SQLTransformerSuite.scala | 1 +
2 files changed, 4 insertions(+), 1 deletion(-)
--
http://git-wip-us.apache.org/repos/asf/spark/blob/ab3363e9/mllib/src/main/scala/org/apache/spark/ml/feature/SQLTransformer.scala
--
diff --git
a/mllib/src/main/scala/org/apache/spark/ml/feature/SQLTransformer.scala
b/mllib/src/main/scala/org/apache/spark/ml/feature/SQLTransformer.scala
index 259be26..b25fff9 100644
--- a/mllib/src/main/scala/org/apache/spark/ml/feature/SQLTransformer.scala
+++ b/mllib/src/main/scala/org/apache/spark/ml/feature/SQLTransformer.scala
@@ -67,7 +67,9 @@ class SQLTransformer @Since("1.6.0") (@Since("1.6.0")
override val uid: String)
val tableName = Identifiable.randomUID(uid)
dataset.createOrReplaceTempView(tableName)
val realStatement = $(statement).replace(tableIdentifier, tableName)
-dataset.sparkSession.sql(realStatement)
+val result = dataset.sparkSession.sql(realStatement)
+dataset.sparkSession.catalog.dropTempView(tableName)
+result
}
@Since("1.6.0")
http://git-wip-us.apache.org/repos/asf/spark/blob/ab3363e9/mllib/src/test/scala/org/apache/spark/ml/feature/SQLTransformerSuite.scala
--
diff --git
a/mllib/src/test/scala/org/apache/spark/ml/feature/SQLTransformerSuite.scala
b/mllib/src/test/scala/org/apache/spark/ml/feature/SQLTransformerSuite.scala
index 2346407..753f890 100644
--- a/mllib/src/test/scala/org/apache/spark/ml/feature/SQLTransformerSuite.scala
+++ b/mllib/src/test/scala/org/apache/spark/ml/feature/SQLTransformerSuite.scala
@@ -43,6 +43,7 @@ class SQLTransformerSuite
assert(result.schema.toString == resultSchema.toString)
assert(resultSchema == expected.schema)
assert(result.collect().toSeq == expected.collect().toSeq)
+assert(original.sparkSession.catalog.listTables().count() == 0)
}
test("read/write") {
-
To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org
For additional commands, e-mail: commits-h...@spark.apache.org
spark git commit: [SPARK-17986][ML] SQLTransformer should remove temporary tables
Repository: spark
Updated Branches:
refs/heads/branch-2.0 a0c03c925 -> b959dab32
[SPARK-17986][ML] SQLTransformer should remove temporary tables
## What changes were proposed in this pull request?
A call to the method `SQLTransformer.transform` previously would create a
temporary table and never delete it. This change adds a call to
`dropTempView()` that deletes this temporary table before returning the result
so that the table will not remain in spark's table catalog. Because `tableName`
is randomized and not exposed, there should be no expected use of this table
outside of the `transform` method.
## How was this patch tested?
A single new assertion was added to the existing test of the
`SQLTransformer.transform` method that all temporary tables are removed.
Without the corresponding code change, this new assertion fails. I am not aware
of any circumstances in which removing this temporary view would be bad for
performance or correctness in other ways, but some expertise here would be
helpful.
Author: Drew Robb
Closes #15526 from drewrobb/SPARK-17986.
(cherry picked from commit ab3363e9f6b1f7fc26682509fe7382c570f91778)
Signed-off-by: Yanbo Liang
Project: http://git-wip-us.apache.org/repos/asf/spark/repo
Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/b959dab3
Tree: http://git-wip-us.apache.org/repos/asf/spark/tree/b959dab3
Diff: http://git-wip-us.apache.org/repos/asf/spark/diff/b959dab3
Branch: refs/heads/branch-2.0
Commit: b959dab32a455e0f9a9ea0fd2111e28a5faf796c
Parents: a0c03c9
Author: Drew Robb
Authored: Sat Oct 22 01:59:36 2016 -0700
Committer: Yanbo Liang
Committed: Sat Oct 22 02:00:05 2016 -0700
--
.../main/scala/org/apache/spark/ml/feature/SQLTransformer.scala | 4 +++-
.../scala/org/apache/spark/ml/feature/SQLTransformerSuite.scala | 1 +
2 files changed, 4 insertions(+), 1 deletion(-)
--
http://git-wip-us.apache.org/repos/asf/spark/blob/b959dab3/mllib/src/main/scala/org/apache/spark/ml/feature/SQLTransformer.scala
--
diff --git
a/mllib/src/main/scala/org/apache/spark/ml/feature/SQLTransformer.scala
b/mllib/src/main/scala/org/apache/spark/ml/feature/SQLTransformer.scala
index 259be26..b25fff9 100644
--- a/mllib/src/main/scala/org/apache/spark/ml/feature/SQLTransformer.scala
+++ b/mllib/src/main/scala/org/apache/spark/ml/feature/SQLTransformer.scala
@@ -67,7 +67,9 @@ class SQLTransformer @Since("1.6.0") (@Since("1.6.0")
override val uid: String)
val tableName = Identifiable.randomUID(uid)
dataset.createOrReplaceTempView(tableName)
val realStatement = $(statement).replace(tableIdentifier, tableName)
-dataset.sparkSession.sql(realStatement)
+val result = dataset.sparkSession.sql(realStatement)
+dataset.sparkSession.catalog.dropTempView(tableName)
+result
}
@Since("1.6.0")
http://git-wip-us.apache.org/repos/asf/spark/blob/b959dab3/mllib/src/test/scala/org/apache/spark/ml/feature/SQLTransformerSuite.scala
--
diff --git
a/mllib/src/test/scala/org/apache/spark/ml/feature/SQLTransformerSuite.scala
b/mllib/src/test/scala/org/apache/spark/ml/feature/SQLTransformerSuite.scala
index 1401ea9..9d3c007 100644
--- a/mllib/src/test/scala/org/apache/spark/ml/feature/SQLTransformerSuite.scala
+++ b/mllib/src/test/scala/org/apache/spark/ml/feature/SQLTransformerSuite.scala
@@ -43,6 +43,7 @@ class SQLTransformerSuite
assert(result.schema.toString == resultSchema.toString)
assert(resultSchema == expected.schema)
assert(result.collect().toSeq == expected.collect().toSeq)
+assert(original.sparkSession.catalog.listTables().count() == 0)
}
test("read/write") {
-
To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org
For additional commands, e-mail: commits-h...@spark.apache.org
spark git commit: [SPARK-17748][ML] One pass solver for Weighted Least Squares with ElasticNet
Repository: spark
Updated Branches:
refs/heads/master 483c37c58 -> 78d740a08
[SPARK-17748][ML] One pass solver for Weighted Least Squares with ElasticNet
## What changes were proposed in this pull request?
1. Make a pluggable solver interface for `WeightedLeastSquares`
2. Add a `QuasiNewton` solver to handle elastic net regularization for
`WeightedLeastSquares`
3. Add method `BLAS.dspmv` used by QN solver
4. Add mechanism for WLS to handle singular covariance matrices by falling back
to QN solver when Cholesky fails.
## How was this patch tested?
Unit tests - see below.
## Design choices
**Pluggable Normal Solver**
Before, the `WeightedLeastSquares` package always used the Cholesky
decomposition solver to compute the solution to the normal equations. Now, we
specify the solver as a constructor argument to the `WeightedLeastSquares`. We
introduce a new trait:
scala
private[ml] sealed trait NormalEquationSolver {
def solve(
bBar: Double,
bbBar: Double,
abBar: DenseVector,
aaBar: DenseVector,
aBar: DenseVector): NormalEquationSolution
}
We extend this trait for different variants of normal equation solvers. In the
future, we can easily add others (like QR) using this interface.
**Always train in the standardized space**
The normal solver did not previously standardize the data, but this patch
introduces a change such that we always solve the normal equations in the
standardized space. We convert back to the original space in the same way that
is done for distributed L-BFGS/OWL-QN. We add test cases for zero variance
features/labels.
**Use L-BFGS locally to solve normal equations for singular matrix**
When linear regression with the normal solver is called for a singular matrix,
we initially try to solve with Cholesky. We use the output of `lapack.dppsv` to
determine if the matrix is singular. If it is, we fall back to using L-BFGS
locally to solve the normal equations. We add test cases for this as well.
## Test cases
I found it helpful to enumerate some of the test cases and hopefully it makes
review easier.
**WeightedLeastSquares**
1. Constant columns - Cholesky solver fails with no regularization, Auto solver
falls back to QN, and QN trains successfully.
2. Collinear features - Cholesky solver fails with no regularization, Auto
solver falls back to QN, and QN trains successfully.
3. Label is constant zero - no training is performed regardless of intercept.
Coefficients are zero and intercept is zero.
4. Label is constant - if fitIntercept, then no training is performed and
intercept equals label mean. If not fitIntercept, then we train and return an
answer that matches R's lm package.
5. Test with L1 - go through various combinations of L1/L2, standardization,
fitIntercept and verify that output matches glmnet.
6. Initial intercept - verify that setting the initial intercept to label mean
is correct by training model with strong L1 regularization so that all
coefficients are zero and intercept converges to label mean.
7. Test diagInvAtWA - since we are standardizing features now during training,
we should test that the inverse is computed to match R.
**LinearRegression**
1. For all existing L1 test cases, test the "normal" solver too.
2. Check that using the normal solver now handles singular matrices.
3. Check that using the normal solver with L1 produces an objective history in
the model summary, but does not produce the inverse of AtA.
**BLAS**
1. Test new method `dspmv`.
## Performance Testing
This patch will speed up linear regression with L1/elasticnet penalties when
the feature size is < 4096. I have not conducted performance tests at scale,
only observed by testing locally that there is a speed improvement.
We should decide if this PR needs to be blocked before performance testing is
conducted.
Author: sethah
Closes #15394 from sethah/SPARK-17748.
Project: http://git-wip-us.apache.org/repos/asf/spark/repo
Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/78d740a0
Tree: http://git-wip-us.apache.org/repos/asf/spark/tree/78d740a0
Diff: http://git-wip-us.apache.org/repos/asf/spark/diff/78d740a0
Branch: refs/heads/master
Commit: 78d740a08a04b74b49b5cba4bb6a821631390ab4
Parents: 483c37c
Author: sethah
Authored: Mon Oct 24 23:47:59 2016 -0700
Committer: Yanbo Liang
Committed: Mon Oct 24 23:47:59 2016 -0700
--
.../scala/org/apache/spark/ml/linalg/BLAS.scala | 18 +
.../org/apache/spark/ml/linalg/BLASSuite.scala | 45 ++
.../IterativelyReweightedLeastSquares.scala | 4 +-
.../spark/ml/optim/NormalEquationSolver.scala | 163 +++
.../spark/ml/optim/WeightedLeastSquares.scala | 270 +---
.../GeneralizedLinearRegression.scala | 4 +-
.../spark/ml/regression/LinearRegression.scala | 20 +-
.../mllib/linalg/CholeskyDecomposition.scala| 4 +-
...IterativelyReweightedLeastSquaresSuite.scala |
spark git commit: [SPARK-14634][ML][FOLLOWUP] Delete superfluous line in BisectingKMeans
Repository: spark
Updated Branches:
refs/heads/master 6f31833db -> 38cdd6ccd
[SPARK-14634][ML][FOLLOWUP] Delete superfluous line in BisectingKMeans
## What changes were proposed in this pull request?
As commented by jkbradley in https://github.com/apache/spark/pull/12394,
`model.setSummary(summary)` is superfluous
## How was this patch tested?
existing tests
Author: Zheng RuiFeng
Closes #15619 from zhengruifeng/del_superfluous.
Project: http://git-wip-us.apache.org/repos/asf/spark/repo
Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/38cdd6cc
Tree: http://git-wip-us.apache.org/repos/asf/spark/tree/38cdd6cc
Diff: http://git-wip-us.apache.org/repos/asf/spark/diff/38cdd6cc
Branch: refs/heads/master
Commit: 38cdd6ccdaba7f8da985c4f4efe5bd93a46a2b53
Parents: 6f31833
Author: Zheng RuiFeng
Authored: Tue Oct 25 03:19:50 2016 -0700
Committer: Yanbo Liang
Committed: Tue Oct 25 03:19:50 2016 -0700
--
.../scala/org/apache/spark/ml/clustering/BisectingKMeans.scala | 5 ++---
.../src/main/scala/org/apache/spark/ml/clustering/KMeans.scala | 6 +++---
2 files changed, 5 insertions(+), 6 deletions(-)
--
http://git-wip-us.apache.org/repos/asf/spark/blob/38cdd6cc/mllib/src/main/scala/org/apache/spark/ml/clustering/BisectingKMeans.scala
--
diff --git
a/mllib/src/main/scala/org/apache/spark/ml/clustering/BisectingKMeans.scala
b/mllib/src/main/scala/org/apache/spark/ml/clustering/BisectingKMeans.scala
index add8ee2..ef2d918 100644
--- a/mllib/src/main/scala/org/apache/spark/ml/clustering/BisectingKMeans.scala
+++ b/mllib/src/main/scala/org/apache/spark/ml/clustering/BisectingKMeans.scala
@@ -265,9 +265,8 @@ class BisectingKMeans @Since("2.0.0") (
val summary = new BisectingKMeansSummary(
model.transform(dataset), $(predictionCol), $(featuresCol), $(k))
model.setSummary(summary)
-val m = model.setSummary(summary)
-instr.logSuccess(m)
-m
+instr.logSuccess(model)
+model
}
@Since("2.0.0")
http://git-wip-us.apache.org/repos/asf/spark/blob/38cdd6cc/mllib/src/main/scala/org/apache/spark/ml/clustering/KMeans.scala
--
diff --git a/mllib/src/main/scala/org/apache/spark/ml/clustering/KMeans.scala
b/mllib/src/main/scala/org/apache/spark/ml/clustering/KMeans.scala
index b04e828..0d2405b 100644
--- a/mllib/src/main/scala/org/apache/spark/ml/clustering/KMeans.scala
+++ b/mllib/src/main/scala/org/apache/spark/ml/clustering/KMeans.scala
@@ -324,9 +324,9 @@ class KMeans @Since("1.5.0") (
val model = copyValues(new KMeansModel(uid, parentModel).setParent(this))
val summary = new KMeansSummary(
model.transform(dataset), $(predictionCol), $(featuresCol), $(k))
-val m = model.setSummary(summary)
-instr.logSuccess(m)
-m
+model.setSummary(summary)
+instr.logSuccess(model)
+model
}
@Since("1.5.0")
-
To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org
For additional commands, e-mail: commits-h...@spark.apache.org
spark git commit: [SPARK-17748][FOLLOW-UP][ML] Fix build error for Scala 2.10.
Repository: spark
Updated Branches:
refs/heads/master 38cdd6ccd -> ac8ff920f
[SPARK-17748][FOLLOW-UP][ML] Fix build error for Scala 2.10.
## What changes were proposed in this pull request?
#15394 introduced build error for Scala 2.10, this PR fix it.
## How was this patch tested?
Existing test.
Author: Yanbo Liang
Closes #15625 from yanboliang/spark-17748-scala.
Project: http://git-wip-us.apache.org/repos/asf/spark/repo
Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/ac8ff920
Tree: http://git-wip-us.apache.org/repos/asf/spark/tree/ac8ff920
Diff: http://git-wip-us.apache.org/repos/asf/spark/diff/ac8ff920
Branch: refs/heads/master
Commit: ac8ff920faec6ee06e17212e2b5d2ee117495e87
Parents: 38cdd6c
Author: Yanbo Liang
Authored: Tue Oct 25 10:22:02 2016 -0700
Committer: Yanbo Liang
Committed: Tue Oct 25 10:22:02 2016 -0700
--
.../spark/ml/optim/WeightedLeastSquaresSuite.scala | 13 +++--
1 file changed, 7 insertions(+), 6 deletions(-)
--
http://git-wip-us.apache.org/repos/asf/spark/blob/ac8ff920/mllib/src/test/scala/org/apache/spark/ml/optim/WeightedLeastSquaresSuite.scala
--
diff --git
a/mllib/src/test/scala/org/apache/spark/ml/optim/WeightedLeastSquaresSuite.scala
b/mllib/src/test/scala/org/apache/spark/ml/optim/WeightedLeastSquaresSuite.scala
index 5f638b4..3cdab03 100644
---
a/mllib/src/test/scala/org/apache/spark/ml/optim/WeightedLeastSquaresSuite.scala
+++
b/mllib/src/test/scala/org/apache/spark/ml/optim/WeightedLeastSquaresSuite.scala
@@ -280,7 +280,7 @@ class WeightedLeastSquaresSuite extends SparkFunSuite with
MLlibTestSparkContext
}
// Cholesky also fails when regularization is added but we don't wish to
standardize
-val wls = new WeightedLeastSquares(true, regParam = 0.5, elasticNetParam =
0.0,
+val wls = new WeightedLeastSquares(fitIntercept = true, regParam = 0.5,
elasticNetParam = 0.0,
standardizeFeatures = false, standardizeLabel = false,
solverType = WeightedLeastSquares.Cholesky)
intercept[SingularMatrixException] {
@@ -470,10 +470,11 @@ class WeightedLeastSquaresSuite extends SparkFunSuite
with MLlibTestSparkContext
var idx = 0
for (fitIntercept <- Seq(false, true);
regParam <- Seq(0.1, 0.5, 1.0);
- standardizeFeatures <- Seq(false, true);
+ standardization <- Seq(false, true);
elasticNetParam <- Seq(0.1, 0.5, 1.0)) {
- val wls = new WeightedLeastSquares(fitIntercept, regParam,
elasticNetParam = elasticNetParam,
-standardizeFeatures, standardizeLabel = true, solverType =
WeightedLeastSquares.Auto)
+ val wls = new WeightedLeastSquares(fitIntercept, regParam,
elasticNetParam,
+standardizeFeatures = standardization, standardizeLabel = true,
+solverType = WeightedLeastSquares.Auto)
.fit(instances)
val actual = Vectors.dense(wls.intercept, wls.coefficients(0),
wls.coefficients(1))
assert(actual ~== expected(idx) absTol 1e-4)
@@ -528,10 +529,10 @@ class WeightedLeastSquaresSuite extends SparkFunSuite
with MLlibTestSparkContext
var idx = 0
for (fitIntercept <- Seq(false, true);
regParam <- Seq(0.0, 0.1, 1.0);
- standardizeFeatures <- Seq(false, true)) {
+ standardization <- Seq(false, true)) {
for (solver <- WeightedLeastSquares.supportedSolvers) {
val wls = new WeightedLeastSquares(fitIntercept, regParam,
elasticNetParam = 0.0,
- standardizeFeatures, standardizeLabel = true, solverType = solver)
+ standardizeFeatures = standardization, standardizeLabel = true,
solverType = solver)
.fit(instances)
val actual = Vectors.dense(wls.intercept, wls.coefficients(0),
wls.coefficients(1))
assert(actual ~== expected(idx) absTol 1e-4)
-
To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org
For additional commands, e-mail: commits-h...@spark.apache.org
spark git commit: [SPARK-17748][FOLLOW-UP][ML] Reorg variables of WeightedLeastSquares.
Repository: spark
Updated Branches:
refs/heads/master 4bee95407 -> 312ea3f7f
[SPARK-17748][FOLLOW-UP][ML] Reorg variables of WeightedLeastSquares.
## What changes were proposed in this pull request?
This is follow-up work of #15394.
Reorg some variables of ```WeightedLeastSquares``` and fix one minor issue of
```WeightedLeastSquaresSuite```.
## How was this patch tested?
Existing tests.
Author: Yanbo Liang
Closes #15621 from yanboliang/spark-17748.
Project: http://git-wip-us.apache.org/repos/asf/spark/repo
Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/312ea3f7
Tree: http://git-wip-us.apache.org/repos/asf/spark/tree/312ea3f7
Diff: http://git-wip-us.apache.org/repos/asf/spark/diff/312ea3f7
Branch: refs/heads/master
Commit: 312ea3f7f65532818e11016d6d780ad47485175f
Parents: 4bee954
Author: Yanbo Liang
Authored: Wed Oct 26 09:28:28 2016 -0700
Committer: Yanbo Liang
Committed: Wed Oct 26 09:28:28 2016 -0700
--
.../spark/ml/optim/WeightedLeastSquares.scala | 139 +++
.../ml/optim/WeightedLeastSquaresSuite.scala| 15 +-
2 files changed, 86 insertions(+), 68 deletions(-)
--
http://git-wip-us.apache.org/repos/asf/spark/blob/312ea3f7/mllib/src/main/scala/org/apache/spark/ml/optim/WeightedLeastSquares.scala
--
diff --git
a/mllib/src/main/scala/org/apache/spark/ml/optim/WeightedLeastSquares.scala
b/mllib/src/main/scala/org/apache/spark/ml/optim/WeightedLeastSquares.scala
index 2223f12..90c24e1 100644
--- a/mllib/src/main/scala/org/apache/spark/ml/optim/WeightedLeastSquares.scala
+++ b/mllib/src/main/scala/org/apache/spark/ml/optim/WeightedLeastSquares.scala
@@ -101,23 +101,19 @@ private[ml] class WeightedLeastSquares(
summary.validate()
logInfo(s"Number of instances: ${summary.count}.")
val k = if (fitIntercept) summary.k + 1 else summary.k
+val numFeatures = summary.k
val triK = summary.triK
val wSum = summary.wSum
-val bBar = summary.bBar
-val bbBar = summary.bbBar
-val aBar = summary.aBar
-val aStd = summary.aStd
-val abBar = summary.abBar
-val aaBar = summary.aaBar
-val numFeatures = abBar.size
+
val rawBStd = summary.bStd
+val rawBBar = summary.bBar
// if b is constant (rawBStd is zero), then b cannot be scaled. In this
case
-// setting bStd=abs(bBar) ensures that b is not scaled anymore in l-bfgs
algorithm.
-val bStd = if (rawBStd == 0.0) math.abs(bBar) else rawBStd
+// setting bStd=abs(rawBBar) ensures that b is not scaled anymore in
l-bfgs algorithm.
+val bStd = if (rawBStd == 0.0) math.abs(rawBBar) else rawBStd
if (rawBStd == 0) {
- if (fitIntercept || bBar == 0.0) {
-if (bBar == 0.0) {
+ if (fitIntercept || rawBBar == 0.0) {
+if (rawBBar == 0.0) {
logWarning(s"Mean and standard deviation of the label are zero, so
the coefficients " +
s"and the intercept will all be zero; as a result, training is not
needed.")
} else {
@@ -126,7 +122,7 @@ private[ml] class WeightedLeastSquares(
s"training is not needed.")
}
val coefficients = new DenseVector(Array.ofDim(numFeatures))
-val intercept = bBar
+val intercept = rawBBar
val diagInvAtWA = new DenseVector(Array(0D))
return new WeightedLeastSquaresModel(coefficients, intercept,
diagInvAtWA, Array(0D))
} else {
@@ -137,53 +133,70 @@ private[ml] class WeightedLeastSquares(
}
}
-// scale aBar to standardized space in-place
-val aBarValues = aBar.values
-var j = 0
-while (j < numFeatures) {
- if (aStd(j) == 0.0) {
-aBarValues(j) = 0.0
- } else {
-aBarValues(j) /= aStd(j)
- }
- j += 1
-}
+val bBar = summary.bBar / bStd
+val bbBar = summary.bbBar / (bStd * bStd)
-// scale abBar to standardized space in-place
-val abBarValues = abBar.values
+val aStd = summary.aStd
val aStdValues = aStd.values
-j = 0
-while (j < numFeatures) {
- if (aStdValues(j) == 0.0) {
-abBarValues(j) = 0.0
- } else {
-abBarValues(j) /= (aStdValues(j) * bStd)
+
+val aBar = {
+ val _aBar = summary.aBar
+ val _aBarValues = _aBar.values
+ var i = 0
+ // scale aBar to standardized space in-place
+ while (i < numFeatures) {
+if (aStdValues(i) == 0.0) {
+ _aBarValues(i) = 0.0
+} else {
+ _aBarValues(i) /= aStdValues(i)
+}
+i += 1
}
- j += 1
+ _aBar
}
+val aBarValues = aBar.values
-// scale aaBar to standardized space in-place
-val aaBarValues = aaBar.values
-j = 0
-var p = 0
-while (j < numFeatures) {
- val aStdJ = aStdValues(j)
+val abBar = {
+ val _abBar = sum
spark git commit: [SPARK-18109][ML] Add instrumentation to GMM
Repository: spark
Updated Branches:
refs/heads/master ab5f938bc -> 569788a55
[SPARK-18109][ML] Add instrumentation to GMM
## What changes were proposed in this pull request?
Add instrumentation to GMM
## How was this patch tested?
Test in spark-shell
Author: Zheng RuiFeng
Closes #15636 from zhengruifeng/gmm_instr.
Project: http://git-wip-us.apache.org/repos/asf/spark/repo
Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/569788a5
Tree: http://git-wip-us.apache.org/repos/asf/spark/tree/569788a5
Diff: http://git-wip-us.apache.org/repos/asf/spark/diff/569788a5
Branch: refs/heads/master
Commit: 569788a55e4c6b218fb697e1e54c6138ffe657a6
Parents: ab5f938
Author: Zheng RuiFeng
Authored: Fri Oct 28 00:40:06 2016 -0700
Committer: Yanbo Liang
Committed: Fri Oct 28 00:40:06 2016 -0700
--
.../scala/org/apache/spark/ml/clustering/GaussianMixture.scala | 6 ++
1 file changed, 6 insertions(+)
--
http://git-wip-us.apache.org/repos/asf/spark/blob/569788a5/mllib/src/main/scala/org/apache/spark/ml/clustering/GaussianMixture.scala
--
diff --git
a/mllib/src/main/scala/org/apache/spark/ml/clustering/GaussianMixture.scala
b/mllib/src/main/scala/org/apache/spark/ml/clustering/GaussianMixture.scala
index e3cb92f..8fac63f 100644
--- a/mllib/src/main/scala/org/apache/spark/ml/clustering/GaussianMixture.scala
+++ b/mllib/src/main/scala/org/apache/spark/ml/clustering/GaussianMixture.scala
@@ -323,6 +323,9 @@ class GaussianMixture @Since("2.0.0") (
case Row(point: Vector) => OldVectors.fromML(point)
}
+val instr = Instrumentation.create(this, rdd)
+instr.logParams(featuresCol, predictionCol, probabilityCol, k, maxIter,
seed, tol)
+
val algo = new MLlibGM()
.setK($(k))
.setMaxIterations($(maxIter))
@@ -337,6 +340,9 @@ class GaussianMixture @Since("2.0.0") (
val summary = new GaussianMixtureSummary(model.transform(dataset),
$(predictionCol), $(probabilityCol), $(featuresCol), $(k))
model.setSummary(summary)
+instr.logNumFeatures(model.gaussians.head.mean.size)
+instr.logSuccess(model)
+model
}
@Since("2.0.0")
-
To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org
For additional commands, e-mail: commits-h...@spark.apache.org
spark git commit: [SPARK-18133][EXAMPLES][ML] Python ML Pipeline Example has syntax e…
Repository: spark
Updated Branches:
refs/heads/master 569788a55 -> e9746f87d
[SPARK-18133][EXAMPLES][ML] Python ML Pipeline Example has syntax eâ¦
## What changes were proposed in this pull request?
In Python 3, there is only one integer type (i.e., int), which mostly behaves
like the long type in Python 2. Since Python 3 won't accept "L", so removed "L"
in all examples.
## How was this patch tested?
Unit tests.
â¦rrors]
Author: Jagadeesan
Closes #15660 from jagadeesanas2/SPARK-18133.
Project: http://git-wip-us.apache.org/repos/asf/spark/repo
Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/e9746f87
Tree: http://git-wip-us.apache.org/repos/asf/spark/tree/e9746f87
Diff: http://git-wip-us.apache.org/repos/asf/spark/diff/e9746f87
Branch: refs/heads/master
Commit: e9746f87d0b553b8115948acb79f7e32c23dfd86
Parents: 569788a
Author: Jagadeesan
Authored: Fri Oct 28 02:26:55 2016 -0700
Committer: Yanbo Liang
Committed: Fri Oct 28 02:26:55 2016 -0700
--
examples/src/main/python/ml/cross_validator.py | 8
.../src/main/python/ml/gaussian_mixture_example.py | 2 +-
examples/src/main/python/ml/pipeline_example.py | 16
.../mllib/binary_classification_metrics_example.py | 2 +-
.../python/mllib/multi_class_metrics_example.py | 2 +-
5 files changed, 15 insertions(+), 15 deletions(-)
--
http://git-wip-us.apache.org/repos/asf/spark/blob/e9746f87/examples/src/main/python/ml/cross_validator.py
--
diff --git a/examples/src/main/python/ml/cross_validator.py
b/examples/src/main/python/ml/cross_validator.py
index 907eec6..db70543 100644
--- a/examples/src/main/python/ml/cross_validator.py
+++ b/examples/src/main/python/ml/cross_validator.py
@@ -84,10 +84,10 @@ if __name__ == "__main__":
# Prepare test documents, which are unlabeled.
test = spark.createDataFrame([
-(4L, "spark i j k"),
-(5L, "l m n"),
-(6L, "mapreduce spark"),
-(7L, "apache hadoop")
+(4, "spark i j k"),
+(5, "l m n"),
+(6, "mapreduce spark"),
+(7, "apache hadoop")
], ["id", "text"])
# Make predictions on test documents. cvModel uses the best model found
(lrModel).
http://git-wip-us.apache.org/repos/asf/spark/blob/e9746f87/examples/src/main/python/ml/gaussian_mixture_example.py
--
diff --git a/examples/src/main/python/ml/gaussian_mixture_example.py
b/examples/src/main/python/ml/gaussian_mixture_example.py
index 8ad450b..e4a0d31 100644
--- a/examples/src/main/python/ml/gaussian_mixture_example.py
+++ b/examples/src/main/python/ml/gaussian_mixture_example.py
@@ -38,7 +38,7 @@ if __name__ == "__main__":
# loads data
dataset =
spark.read.format("libsvm").load("data/mllib/sample_kmeans_data.txt")
-gmm = GaussianMixture().setK(2).setSeed(538009335L)
+gmm = GaussianMixture().setK(2).setSeed(538009335)
model = gmm.fit(dataset)
print("Gaussians shown as a DataFrame: ")
http://git-wip-us.apache.org/repos/asf/spark/blob/e9746f87/examples/src/main/python/ml/pipeline_example.py
--
diff --git a/examples/src/main/python/ml/pipeline_example.py
b/examples/src/main/python/ml/pipeline_example.py
index f63e4db..e1fab7c 100644
--- a/examples/src/main/python/ml/pipeline_example.py
+++ b/examples/src/main/python/ml/pipeline_example.py
@@ -35,10 +35,10 @@ if __name__ == "__main__":
# $example on$
# Prepare training documents from a list of (id, text, label) tuples.
training = spark.createDataFrame([
-(0L, "a b c d e spark", 1.0),
-(1L, "b d", 0.0),
-(2L, "spark f g h", 1.0),
-(3L, "hadoop mapreduce", 0.0)
+(0, "a b c d e spark", 1.0),
+(1, "b d", 0.0),
+(2, "spark f g h", 1.0),
+(3, "hadoop mapreduce", 0.0)
], ["id", "text", "label"])
# Configure an ML pipeline, which consists of three stages: tokenizer,
hashingTF, and lr.
@@ -52,10 +52,10 @@ if __name__ == "__main__":
# Prepare test documents, which are unlabeled (id, text) tuples.
test = spark.createDataFrame([
-(4L, "spark i j k"),
-(5L, "l m n"),
-(6L, "spark hadoop spark"),
-(7L, "apache hadoop")
+(4, "spark i j k"),
+(5, "l m n"),
+(6, "spark hadoop spark"),
+(7, "apache hadoop")
], ["id", "text"])
# Make predictions on test documents and print columns of interest.
http://git-wip-us.apache.org/repos/asf/spark/blob/e9746f87/examples/src/main/python/mllib/binary_classification_metrics_example.py
--
diff --git
a/examples/src/main/python/mllib/binary_cla
spark git commit: [SPARK-18177][ML][PYSPARK] Add missing 'subsamplingRate' of pyspark GBTClassifier
Repository: spark Updated Branches: refs/heads/master 0ea5d5b24 -> 9dc9f9a5d [SPARK-18177][ML][PYSPARK] Add missing 'subsamplingRate' of pyspark GBTClassifier ## What changes were proposed in this pull request? Add missing 'subsamplingRate' of pyspark GBTClassifier ## How was this patch tested? existing tests Author: Zheng RuiFeng Closes #15692 from zhengruifeng/gbt_subsamplingRate. Project: http://git-wip-us.apache.org/repos/asf/spark/repo Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/9dc9f9a5 Tree: http://git-wip-us.apache.org/repos/asf/spark/tree/9dc9f9a5 Diff: http://git-wip-us.apache.org/repos/asf/spark/diff/9dc9f9a5 Branch: refs/heads/master Commit: 9dc9f9a5dde37d085808a264cfb9cf4d4f72417d Parents: 0ea5d5b Author: Zheng RuiFeng Authored: Thu Nov 3 07:45:20 2016 -0700 Committer: Yanbo Liang Committed: Thu Nov 3 07:45:20 2016 -0700 -- python/pyspark/ml/classification.py | 10 +- 1 file changed, 5 insertions(+), 5 deletions(-) -- http://git-wip-us.apache.org/repos/asf/spark/blob/9dc9f9a5/python/pyspark/ml/classification.py -- diff --git a/python/pyspark/ml/classification.py b/python/pyspark/ml/classification.py index d9ff356..56c8c62 100644 --- a/python/pyspark/ml/classification.py +++ b/python/pyspark/ml/classification.py @@ -900,19 +900,19 @@ class GBTClassifier(JavaEstimator, HasFeaturesCol, HasLabelCol, HasPredictionCol def __init__(self, featuresCol="features", labelCol="label", predictionCol="prediction", maxDepth=5, maxBins=32, minInstancesPerNode=1, minInfoGain=0.0, maxMemoryInMB=256, cacheNodeIds=False, checkpointInterval=10, lossType="logistic", - maxIter=20, stepSize=0.1, seed=None): + maxIter=20, stepSize=0.1, seed=None, subsamplingRate=1.0): """ __init__(self, featuresCol="features", labelCol="label", predictionCol="prediction", \ maxDepth=5, maxBins=32, minInstancesPerNode=1, minInfoGain=0.0, \ maxMemoryInMB=256, cacheNodeIds=False, checkpointInterval=10, \ - lossType="logistic", maxIter=20, stepSize=0.1, seed=None) + lossType="logistic", maxIter=20, stepSize=0.1, seed=None, subsamplingRate=1.0) """ super(GBTClassifier, self).__init__() self._java_obj = self._new_java_obj( "org.apache.spark.ml.classification.GBTClassifier", self.uid) self._setDefault(maxDepth=5, maxBins=32, minInstancesPerNode=1, minInfoGain=0.0, maxMemoryInMB=256, cacheNodeIds=False, checkpointInterval=10, - lossType="logistic", maxIter=20, stepSize=0.1) + lossType="logistic", maxIter=20, stepSize=0.1, subsamplingRate=1.0) kwargs = self.__init__._input_kwargs self.setParams(**kwargs) @@ -921,12 +921,12 @@ class GBTClassifier(JavaEstimator, HasFeaturesCol, HasLabelCol, HasPredictionCol def setParams(self, featuresCol="features", labelCol="label", predictionCol="prediction", maxDepth=5, maxBins=32, minInstancesPerNode=1, minInfoGain=0.0, maxMemoryInMB=256, cacheNodeIds=False, checkpointInterval=10, - lossType="logistic", maxIter=20, stepSize=0.1, seed=None): + lossType="logistic", maxIter=20, stepSize=0.1, seed=None, subsamplingRate=1.0): """ setParams(self, featuresCol="features", labelCol="label", predictionCol="prediction", \ maxDepth=5, maxBins=32, minInstancesPerNode=1, minInfoGain=0.0, \ maxMemoryInMB=256, cacheNodeIds=False, checkpointInterval=10, \ - lossType="logistic", maxIter=20, stepSize=0.1, seed=None) + lossType="logistic", maxIter=20, stepSize=0.1, seed=None, subsamplingRate=1.0) Sets params for Gradient Boosted Tree Classification. """ kwargs = self.setParams._input_kwargs - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
spark git commit: [SPARK-18177][ML][PYSPARK] Add missing 'subsamplingRate' of pyspark GBTClassifier
Repository: spark Updated Branches: refs/heads/branch-2.1 71104c9c9 -> 99891e56e [SPARK-18177][ML][PYSPARK] Add missing 'subsamplingRate' of pyspark GBTClassifier ## What changes were proposed in this pull request? Add missing 'subsamplingRate' of pyspark GBTClassifier ## How was this patch tested? existing tests Author: Zheng RuiFeng Closes #15692 from zhengruifeng/gbt_subsamplingRate. (cherry picked from commit 9dc9f9a5dde37d085808a264cfb9cf4d4f72417d) Signed-off-by: Yanbo Liang Project: http://git-wip-us.apache.org/repos/asf/spark/repo Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/99891e56 Tree: http://git-wip-us.apache.org/repos/asf/spark/tree/99891e56 Diff: http://git-wip-us.apache.org/repos/asf/spark/diff/99891e56 Branch: refs/heads/branch-2.1 Commit: 99891e56ea286580323fd82e303064d3c0730d85 Parents: 71104c9 Author: Zheng RuiFeng Authored: Thu Nov 3 07:45:20 2016 -0700 Committer: Yanbo Liang Committed: Thu Nov 3 07:45:56 2016 -0700 -- python/pyspark/ml/classification.py | 10 +- 1 file changed, 5 insertions(+), 5 deletions(-) -- http://git-wip-us.apache.org/repos/asf/spark/blob/99891e56/python/pyspark/ml/classification.py -- diff --git a/python/pyspark/ml/classification.py b/python/pyspark/ml/classification.py index d9ff356..56c8c62 100644 --- a/python/pyspark/ml/classification.py +++ b/python/pyspark/ml/classification.py @@ -900,19 +900,19 @@ class GBTClassifier(JavaEstimator, HasFeaturesCol, HasLabelCol, HasPredictionCol def __init__(self, featuresCol="features", labelCol="label", predictionCol="prediction", maxDepth=5, maxBins=32, minInstancesPerNode=1, minInfoGain=0.0, maxMemoryInMB=256, cacheNodeIds=False, checkpointInterval=10, lossType="logistic", - maxIter=20, stepSize=0.1, seed=None): + maxIter=20, stepSize=0.1, seed=None, subsamplingRate=1.0): """ __init__(self, featuresCol="features", labelCol="label", predictionCol="prediction", \ maxDepth=5, maxBins=32, minInstancesPerNode=1, minInfoGain=0.0, \ maxMemoryInMB=256, cacheNodeIds=False, checkpointInterval=10, \ - lossType="logistic", maxIter=20, stepSize=0.1, seed=None) + lossType="logistic", maxIter=20, stepSize=0.1, seed=None, subsamplingRate=1.0) """ super(GBTClassifier, self).__init__() self._java_obj = self._new_java_obj( "org.apache.spark.ml.classification.GBTClassifier", self.uid) self._setDefault(maxDepth=5, maxBins=32, minInstancesPerNode=1, minInfoGain=0.0, maxMemoryInMB=256, cacheNodeIds=False, checkpointInterval=10, - lossType="logistic", maxIter=20, stepSize=0.1) + lossType="logistic", maxIter=20, stepSize=0.1, subsamplingRate=1.0) kwargs = self.__init__._input_kwargs self.setParams(**kwargs) @@ -921,12 +921,12 @@ class GBTClassifier(JavaEstimator, HasFeaturesCol, HasLabelCol, HasPredictionCol def setParams(self, featuresCol="features", labelCol="label", predictionCol="prediction", maxDepth=5, maxBins=32, minInstancesPerNode=1, minInfoGain=0.0, maxMemoryInMB=256, cacheNodeIds=False, checkpointInterval=10, - lossType="logistic", maxIter=20, stepSize=0.1, seed=None): + lossType="logistic", maxIter=20, stepSize=0.1, seed=None, subsamplingRate=1.0): """ setParams(self, featuresCol="features", labelCol="label", predictionCol="prediction", \ maxDepth=5, maxBins=32, minInstancesPerNode=1, minInfoGain=0.0, \ maxMemoryInMB=256, cacheNodeIds=False, checkpointInterval=10, \ - lossType="logistic", maxIter=20, stepSize=0.1, seed=None) + lossType="logistic", maxIter=20, stepSize=0.1, seed=None, subsamplingRate=1.0) Sets params for Gradient Boosted Tree Classification. """ kwargs = self.setParams._input_kwargs - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
spark git commit: [SPARK-18276][ML] ML models should copy the training summary and set parent
Repository: spark
Updated Branches:
refs/heads/master 15d392688 -> 23ce0d1e9
[SPARK-18276][ML] ML models should copy the training summary and set parent
## What changes were proposed in this pull request?
Only some of the models which contain a training summary currently set the
summaries in the copy method. Linear/Logistic regression do, GLR, GMM, KM, and
BKM do not. Additionally, these copy methods did not set the parent pointer of
the copied model. This patch modifies the copy methods of the four models
mentioned above to copy the training summary and set the parent.
## How was this patch tested?
Add unit tests in Linear/Logistic/GeneralizedLinear regression and
GaussianMixture/KMeans/BisectingKMeans to check the parent pointer of the
copied model and check that the copied model has a summary.
Author: sethah
Closes #15773 from sethah/SPARK-18276.
Project: http://git-wip-us.apache.org/repos/asf/spark/repo
Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/23ce0d1e
Tree: http://git-wip-us.apache.org/repos/asf/spark/tree/23ce0d1e
Diff: http://git-wip-us.apache.org/repos/asf/spark/diff/23ce0d1e
Branch: refs/heads/master
Commit: 23ce0d1e91076d90c1a87d698a94d283d08cf899
Parents: 15d3926
Author: sethah
Authored: Sat Nov 5 22:38:07 2016 -0700
Committer: Yanbo Liang
Committed: Sat Nov 5 22:38:07 2016 -0700
--
.../org/apache/spark/ml/clustering/BisectingKMeans.scala | 5 +++--
.../org/apache/spark/ml/clustering/GaussianMixture.scala | 5 +++--
.../scala/org/apache/spark/ml/clustering/KMeans.scala| 5 +++--
.../ml/regression/GeneralizedLinearRegression.scala | 6 --
.../apache/spark/ml/tuning/TrainValidationSplit.scala| 2 +-
.../ml/classification/LogisticRegressionSuite.scala | 11 +++
.../spark/ml/clustering/BisectingKMeansSuite.scala | 10 +-
.../spark/ml/clustering/GaussianMixtureSuite.scala | 10 +-
.../org/apache/spark/ml/clustering/KMeansSuite.scala | 10 +-
.../ml/regression/GeneralizedLinearRegressionSuite.scala | 5 -
.../spark/ml/regression/LinearRegressionSuite.scala | 5 -
.../spark/ml/tuning/TrainValidationSplitSuite.scala | 8 ++--
12 files changed, 62 insertions(+), 20 deletions(-)
--
http://git-wip-us.apache.org/repos/asf/spark/blob/23ce0d1e/mllib/src/main/scala/org/apache/spark/ml/clustering/BisectingKMeans.scala
--
diff --git
a/mllib/src/main/scala/org/apache/spark/ml/clustering/BisectingKMeans.scala
b/mllib/src/main/scala/org/apache/spark/ml/clustering/BisectingKMeans.scala
index 2718dd9..f8a606d 100644
--- a/mllib/src/main/scala/org/apache/spark/ml/clustering/BisectingKMeans.scala
+++ b/mllib/src/main/scala/org/apache/spark/ml/clustering/BisectingKMeans.scala
@@ -94,8 +94,9 @@ class BisectingKMeansModel private[ml] (
@Since("2.0.0")
override def copy(extra: ParamMap): BisectingKMeansModel = {
-val copied = new BisectingKMeansModel(uid, parentModel)
-copyValues(copied, extra)
+val copied = copyValues(new BisectingKMeansModel(uid, parentModel), extra)
+if (trainingSummary.isDefined) copied.setSummary(trainingSummary.get)
+copied.setParent(this.parent)
}
@Since("2.0.0")
http://git-wip-us.apache.org/repos/asf/spark/blob/23ce0d1e/mllib/src/main/scala/org/apache/spark/ml/clustering/GaussianMixture.scala
--
diff --git
a/mllib/src/main/scala/org/apache/spark/ml/clustering/GaussianMixture.scala
b/mllib/src/main/scala/org/apache/spark/ml/clustering/GaussianMixture.scala
index 8fac63f..a0bd66e 100644
--- a/mllib/src/main/scala/org/apache/spark/ml/clustering/GaussianMixture.scala
+++ b/mllib/src/main/scala/org/apache/spark/ml/clustering/GaussianMixture.scala
@@ -89,8 +89,9 @@ class GaussianMixtureModel private[ml] (
@Since("2.0.0")
override def copy(extra: ParamMap): GaussianMixtureModel = {
-val copied = new GaussianMixtureModel(uid, weights, gaussians)
-copyValues(copied, extra).setParent(this.parent)
+val copied = copyValues(new GaussianMixtureModel(uid, weights, gaussians),
extra)
+if (trainingSummary.isDefined) copied.setSummary(trainingSummary.get)
+copied.setParent(this.parent)
}
@Since("2.0.0")
http://git-wip-us.apache.org/repos/asf/spark/blob/23ce0d1e/mllib/src/main/scala/org/apache/spark/ml/clustering/KMeans.scala
--
diff --git a/mllib/src/main/scala/org/apache/spark/ml/clustering/KMeans.scala
b/mllib/src/main/scala/org/apache/spark/ml/clustering/KMeans.scala
index 85bb8c9..a0d481b 100644
--- a/mllib/src/main/scala/org/apache/spark/ml/clustering/KMeans.scala
+++ b/mllib/src/main/scala/org/apache/spark/ml/clustering/KMeans.scala
@@ -108,8 +108,9 @@ cla
spark git commit: [SPARK-18276][ML] ML models should copy the training summary and set parent
Repository: spark
Updated Branches:
refs/heads/branch-2.1 e9f1d4aaa -> c42301f1e
[SPARK-18276][ML] ML models should copy the training summary and set parent
## What changes were proposed in this pull request?
Only some of the models which contain a training summary currently set the
summaries in the copy method. Linear/Logistic regression do, GLR, GMM, KM, and
BKM do not. Additionally, these copy methods did not set the parent pointer of
the copied model. This patch modifies the copy methods of the four models
mentioned above to copy the training summary and set the parent.
## How was this patch tested?
Add unit tests in Linear/Logistic/GeneralizedLinear regression and
GaussianMixture/KMeans/BisectingKMeans to check the parent pointer of the
copied model and check that the copied model has a summary.
Author: sethah
Closes #15773 from sethah/SPARK-18276.
(cherry picked from commit 23ce0d1e91076d90c1a87d698a94d283d08cf899)
Signed-off-by: Yanbo Liang
Project: http://git-wip-us.apache.org/repos/asf/spark/repo
Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/c42301f1
Tree: http://git-wip-us.apache.org/repos/asf/spark/tree/c42301f1
Diff: http://git-wip-us.apache.org/repos/asf/spark/diff/c42301f1
Branch: refs/heads/branch-2.1
Commit: c42301f1eb09565cfaa044b05984ed67879bd946
Parents: e9f1d4a
Author: sethah
Authored: Sat Nov 5 22:38:07 2016 -0700
Committer: Yanbo Liang
Committed: Sat Nov 5 22:38:40 2016 -0700
--
.../org/apache/spark/ml/clustering/BisectingKMeans.scala | 5 +++--
.../org/apache/spark/ml/clustering/GaussianMixture.scala | 5 +++--
.../scala/org/apache/spark/ml/clustering/KMeans.scala| 5 +++--
.../ml/regression/GeneralizedLinearRegression.scala | 6 --
.../apache/spark/ml/tuning/TrainValidationSplit.scala| 2 +-
.../ml/classification/LogisticRegressionSuite.scala | 11 +++
.../spark/ml/clustering/BisectingKMeansSuite.scala | 10 +-
.../spark/ml/clustering/GaussianMixtureSuite.scala | 10 +-
.../org/apache/spark/ml/clustering/KMeansSuite.scala | 10 +-
.../ml/regression/GeneralizedLinearRegressionSuite.scala | 5 -
.../spark/ml/regression/LinearRegressionSuite.scala | 5 -
.../spark/ml/tuning/TrainValidationSplitSuite.scala | 8 ++--
12 files changed, 62 insertions(+), 20 deletions(-)
--
http://git-wip-us.apache.org/repos/asf/spark/blob/c42301f1/mllib/src/main/scala/org/apache/spark/ml/clustering/BisectingKMeans.scala
--
diff --git
a/mllib/src/main/scala/org/apache/spark/ml/clustering/BisectingKMeans.scala
b/mllib/src/main/scala/org/apache/spark/ml/clustering/BisectingKMeans.scala
index 2718dd9..f8a606d 100644
--- a/mllib/src/main/scala/org/apache/spark/ml/clustering/BisectingKMeans.scala
+++ b/mllib/src/main/scala/org/apache/spark/ml/clustering/BisectingKMeans.scala
@@ -94,8 +94,9 @@ class BisectingKMeansModel private[ml] (
@Since("2.0.0")
override def copy(extra: ParamMap): BisectingKMeansModel = {
-val copied = new BisectingKMeansModel(uid, parentModel)
-copyValues(copied, extra)
+val copied = copyValues(new BisectingKMeansModel(uid, parentModel), extra)
+if (trainingSummary.isDefined) copied.setSummary(trainingSummary.get)
+copied.setParent(this.parent)
}
@Since("2.0.0")
http://git-wip-us.apache.org/repos/asf/spark/blob/c42301f1/mllib/src/main/scala/org/apache/spark/ml/clustering/GaussianMixture.scala
--
diff --git
a/mllib/src/main/scala/org/apache/spark/ml/clustering/GaussianMixture.scala
b/mllib/src/main/scala/org/apache/spark/ml/clustering/GaussianMixture.scala
index 8fac63f..a0bd66e 100644
--- a/mllib/src/main/scala/org/apache/spark/ml/clustering/GaussianMixture.scala
+++ b/mllib/src/main/scala/org/apache/spark/ml/clustering/GaussianMixture.scala
@@ -89,8 +89,9 @@ class GaussianMixtureModel private[ml] (
@Since("2.0.0")
override def copy(extra: ParamMap): GaussianMixtureModel = {
-val copied = new GaussianMixtureModel(uid, weights, gaussians)
-copyValues(copied, extra).setParent(this.parent)
+val copied = copyValues(new GaussianMixtureModel(uid, weights, gaussians),
extra)
+if (trainingSummary.isDefined) copied.setSummary(trainingSummary.get)
+copied.setParent(this.parent)
}
@Since("2.0.0")
http://git-wip-us.apache.org/repos/asf/spark/blob/c42301f1/mllib/src/main/scala/org/apache/spark/ml/clustering/KMeans.scala
--
diff --git a/mllib/src/main/scala/org/apache/spark/ml/clustering/KMeans.scala
b/mllib/src/main/scala/org/apache/spark/ml/clustering/KMeans.scala
index 85bb8c9..a0d481b 100644
--- a/mllib/src/main/scala/org/apache/spark/ml/clustering/K
spark git commit: [SPARK-18210][ML] Pipeline.copy does not create an instance with the same UID
Repository: spark
Updated Branches:
refs/heads/branch-2.1 dcbf3fd4b -> d2f2cf68a
[SPARK-18210][ML] Pipeline.copy does not create an instance with the same UID
## What changes were proposed in this pull request?
Motivation:
`org.apache.spark.ml.Pipeline.copy(extra: ParamMap)` does not create an
instance with the same UID. It does not conform to the method specification
from its base class `org.apache.spark.ml.param.Params.copy(extra: ParamMap)`
Solution:
- fix for Pipeline UID
- introduced new tests for `org.apache.spark.ml.Pipeline.copy`
- minor improvements in test for `org.apache.spark.ml.PipelineModel.copy`
## How was this patch tested?
Introduced new unit test: `org.apache.spark.ml.PipelineSuite."Pipeline.copy"`
Improved existing unit test:
`org.apache.spark.ml.PipelineSuite."PipelineModel.copy"`
Author: Wojciech Szymanski
Closes #15759 from wojtek-szymanski/SPARK-18210.
(cherry picked from commit b89d0556dff0520ab35882382242fbfa7d9478eb)
Signed-off-by: Yanbo Liang
Project: http://git-wip-us.apache.org/repos/asf/spark/repo
Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/d2f2cf68
Tree: http://git-wip-us.apache.org/repos/asf/spark/tree/d2f2cf68
Diff: http://git-wip-us.apache.org/repos/asf/spark/diff/d2f2cf68
Branch: refs/heads/branch-2.1
Commit: d2f2cf68a62a3f8beb7cdfef8393acfdcb785975
Parents: dcbf3fd
Author: Wojciech Szymanski
Authored: Sun Nov 6 07:43:13 2016 -0800
Committer: Yanbo Liang
Committed: Sun Nov 6 07:43:36 2016 -0800
--
.../scala/org/apache/spark/ml/Pipeline.scala| 2 +-
.../org/apache/spark/ml/PipelineSuite.scala | 22 ++--
2 files changed, 21 insertions(+), 3 deletions(-)
--
http://git-wip-us.apache.org/repos/asf/spark/blob/d2f2cf68/mllib/src/main/scala/org/apache/spark/ml/Pipeline.scala
--
diff --git a/mllib/src/main/scala/org/apache/spark/ml/Pipeline.scala
b/mllib/src/main/scala/org/apache/spark/ml/Pipeline.scala
index 195a93e..f406f8c 100644
--- a/mllib/src/main/scala/org/apache/spark/ml/Pipeline.scala
+++ b/mllib/src/main/scala/org/apache/spark/ml/Pipeline.scala
@@ -169,7 +169,7 @@ class Pipeline @Since("1.4.0") (
override def copy(extra: ParamMap): Pipeline = {
val map = extractParamMap(extra)
val newStages = map(stages).map(_.copy(extra))
-new Pipeline().setStages(newStages)
+new Pipeline(uid).setStages(newStages)
}
@Since("1.2.0")
http://git-wip-us.apache.org/repos/asf/spark/blob/d2f2cf68/mllib/src/test/scala/org/apache/spark/ml/PipelineSuite.scala
--
diff --git a/mllib/src/test/scala/org/apache/spark/ml/PipelineSuite.scala
b/mllib/src/test/scala/org/apache/spark/ml/PipelineSuite.scala
index 6413ca1..dafc6c2 100644
--- a/mllib/src/test/scala/org/apache/spark/ml/PipelineSuite.scala
+++ b/mllib/src/test/scala/org/apache/spark/ml/PipelineSuite.scala
@@ -101,13 +101,31 @@ class PipelineSuite extends SparkFunSuite with
MLlibTestSparkContext with Defaul
}
}
+ test("Pipeline.copy") {
+val hashingTF = new HashingTF()
+ .setNumFeatures(100)
+val pipeline = new
Pipeline("pipeline").setStages(Array[Transformer](hashingTF))
+val copied = pipeline.copy(ParamMap(hashingTF.numFeatures -> 10))
+
+assert(copied.uid === pipeline.uid,
+ "copy should create an instance with the same UID")
+assert(copied.getStages(0).asInstanceOf[HashingTF].getNumFeatures === 10,
+ "copy should handle extra stage params")
+ }
+
test("PipelineModel.copy") {
val hashingTF = new HashingTF()
.setNumFeatures(100)
-val model = new PipelineModel("pipeline", Array[Transformer](hashingTF))
+val model = new PipelineModel("pipelineModel",
Array[Transformer](hashingTF))
+ .setParent(new Pipeline())
val copied = model.copy(ParamMap(hashingTF.numFeatures -> 10))
-require(copied.stages(0).asInstanceOf[HashingTF].getNumFeatures === 10,
+
+assert(copied.uid === model.uid,
+ "copy should create an instance with the same UID")
+assert(copied.stages(0).asInstanceOf[HashingTF].getNumFeatures === 10,
"copy should handle extra stage params")
+assert(copied.parent === model.parent,
+ "copy should create an instance with the same parent")
}
test("pipeline model constructors") {
-
To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org
For additional commands, e-mail: commits-h...@spark.apache.org
spark git commit: [SPARK-18210][ML] Pipeline.copy does not create an instance with the same UID
Repository: spark
Updated Branches:
refs/heads/master 340f09d10 -> b89d0556d
[SPARK-18210][ML] Pipeline.copy does not create an instance with the same UID
## What changes were proposed in this pull request?
Motivation:
`org.apache.spark.ml.Pipeline.copy(extra: ParamMap)` does not create an
instance with the same UID. It does not conform to the method specification
from its base class `org.apache.spark.ml.param.Params.copy(extra: ParamMap)`
Solution:
- fix for Pipeline UID
- introduced new tests for `org.apache.spark.ml.Pipeline.copy`
- minor improvements in test for `org.apache.spark.ml.PipelineModel.copy`
## How was this patch tested?
Introduced new unit test: `org.apache.spark.ml.PipelineSuite."Pipeline.copy"`
Improved existing unit test:
`org.apache.spark.ml.PipelineSuite."PipelineModel.copy"`
Author: Wojciech Szymanski
Closes #15759 from wojtek-szymanski/SPARK-18210.
Project: http://git-wip-us.apache.org/repos/asf/spark/repo
Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/b89d0556
Tree: http://git-wip-us.apache.org/repos/asf/spark/tree/b89d0556
Diff: http://git-wip-us.apache.org/repos/asf/spark/diff/b89d0556
Branch: refs/heads/master
Commit: b89d0556dff0520ab35882382242fbfa7d9478eb
Parents: 340f09d
Author: Wojciech Szymanski
Authored: Sun Nov 6 07:43:13 2016 -0800
Committer: Yanbo Liang
Committed: Sun Nov 6 07:43:13 2016 -0800
--
.../scala/org/apache/spark/ml/Pipeline.scala| 2 +-
.../org/apache/spark/ml/PipelineSuite.scala | 22 ++--
2 files changed, 21 insertions(+), 3 deletions(-)
--
http://git-wip-us.apache.org/repos/asf/spark/blob/b89d0556/mllib/src/main/scala/org/apache/spark/ml/Pipeline.scala
--
diff --git a/mllib/src/main/scala/org/apache/spark/ml/Pipeline.scala
b/mllib/src/main/scala/org/apache/spark/ml/Pipeline.scala
index 195a93e..f406f8c 100644
--- a/mllib/src/main/scala/org/apache/spark/ml/Pipeline.scala
+++ b/mllib/src/main/scala/org/apache/spark/ml/Pipeline.scala
@@ -169,7 +169,7 @@ class Pipeline @Since("1.4.0") (
override def copy(extra: ParamMap): Pipeline = {
val map = extractParamMap(extra)
val newStages = map(stages).map(_.copy(extra))
-new Pipeline().setStages(newStages)
+new Pipeline(uid).setStages(newStages)
}
@Since("1.2.0")
http://git-wip-us.apache.org/repos/asf/spark/blob/b89d0556/mllib/src/test/scala/org/apache/spark/ml/PipelineSuite.scala
--
diff --git a/mllib/src/test/scala/org/apache/spark/ml/PipelineSuite.scala
b/mllib/src/test/scala/org/apache/spark/ml/PipelineSuite.scala
index 6413ca1..dafc6c2 100644
--- a/mllib/src/test/scala/org/apache/spark/ml/PipelineSuite.scala
+++ b/mllib/src/test/scala/org/apache/spark/ml/PipelineSuite.scala
@@ -101,13 +101,31 @@ class PipelineSuite extends SparkFunSuite with
MLlibTestSparkContext with Defaul
}
}
+ test("Pipeline.copy") {
+val hashingTF = new HashingTF()
+ .setNumFeatures(100)
+val pipeline = new
Pipeline("pipeline").setStages(Array[Transformer](hashingTF))
+val copied = pipeline.copy(ParamMap(hashingTF.numFeatures -> 10))
+
+assert(copied.uid === pipeline.uid,
+ "copy should create an instance with the same UID")
+assert(copied.getStages(0).asInstanceOf[HashingTF].getNumFeatures === 10,
+ "copy should handle extra stage params")
+ }
+
test("PipelineModel.copy") {
val hashingTF = new HashingTF()
.setNumFeatures(100)
-val model = new PipelineModel("pipeline", Array[Transformer](hashingTF))
+val model = new PipelineModel("pipelineModel",
Array[Transformer](hashingTF))
+ .setParent(new Pipeline())
val copied = model.copy(ParamMap(hashingTF.numFeatures -> 10))
-require(copied.stages(0).asInstanceOf[HashingTF].getNumFeatures === 10,
+
+assert(copied.uid === model.uid,
+ "copy should create an instance with the same UID")
+assert(copied.stages(0).asInstanceOf[HashingTF].getNumFeatures === 10,
"copy should handle extra stage params")
+assert(copied.parent === model.parent,
+ "copy should create an instance with the same parent")
}
test("pipeline model constructors") {
-
To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org
For additional commands, e-mail: commits-h...@spark.apache.org
spark git commit: [SPARK-18291][SPARKR][ML] SparkR glm predict should output original label when family = binomial.
Repository: spark
Updated Branches:
refs/heads/master a814eeac6 -> daa975f4b
[SPARK-18291][SPARKR][ML] SparkR glm predict should output original label when
family = binomial.
## What changes were proposed in this pull request?
SparkR ```spark.glm``` predict should output original label when family =
"binomial".
## How was this patch tested?
Add unit test.
You can also run the following code to test:
```R
training <- suppressWarnings(createDataFrame(iris))
training <- training[training$Species %in% c("versicolor", "virginica"), ]
model <- spark.glm(training, Species ~ Sepal_Length + Sepal_Width,family =
binomial(link = "logit"))
showDF(predict(model, training))
```
Before this change:
```
++---++---+--+-+---+
|Sepal_Length|Sepal_Width|Petal_Length|Petal_Width| Species|label|
prediction|
++---++---+--+-+---+
| 7.0|3.2| 4.7|1.4|versicolor| 0.0|
0.8271421517601544|
| 6.4|3.2| 4.5|1.5|versicolor| 0.0|
0.6044595910413112|
| 6.9|3.1| 4.9|1.5|versicolor| 0.0|
0.7916340858281998|
| 5.5|2.3| 4.0|1.3|versicolor|
0.0|0.16080518180591158|
| 6.5|2.8| 4.6|1.5|versicolor| 0.0|
0.6112229217050189|
| 5.7|2.8| 4.5|1.3|versicolor| 0.0|
0.2555087295500885|
| 6.3|3.3| 4.7|1.6|versicolor| 0.0|
0.5681507664364834|
| 4.9|2.4| 3.3|1.0|versicolor|
0.0|0.05990570219972002|
| 6.6|2.9| 4.6|1.3|versicolor| 0.0|
0.6644434078306246|
| 5.2|2.7| 3.9|1.4|versicolor|
0.0|0.11293577405862379|
| 5.0|2.0| 3.5|1.0|versicolor|
0.0|0.06152372321585971|
| 5.9|3.0| 4.2|1.5|versicolor|
0.0|0.35250697207602555|
| 6.0|2.2| 4.0|1.0|versicolor|
0.0|0.32267018290814303|
| 6.1|2.9| 4.7|1.4|versicolor| 0.0|
0.433391153814592|
| 5.6|2.9| 3.6|1.3|versicolor| 0.0|
0.2280744262436993|
| 6.7|3.1| 4.4|1.4|versicolor| 0.0|
0.7219848389339459|
| 5.6|3.0| 4.5|1.5|versicolor|
0.0|0.23527698971404695|
| 5.8|2.7| 4.1|1.0|versicolor| 0.0|
0.285024533520016|
| 6.2|2.2| 4.5|1.5|versicolor| 0.0|
0.4107047877447493|
| 5.6|2.5| 3.9|1.1|versicolor|
0.0|0.20083561961645083|
++---++---+--+-+---+
```
After this change:
```
++---++---+--+-+--+
|Sepal_Length|Sepal_Width|Petal_Length|Petal_Width| Species|label|prediction|
++---++---+--+-+--+
| 7.0|3.2| 4.7|1.4|versicolor| 0.0| virginica|
| 6.4|3.2| 4.5|1.5|versicolor| 0.0| virginica|
| 6.9|3.1| 4.9|1.5|versicolor| 0.0| virginica|
| 5.5|2.3| 4.0|1.3|versicolor| 0.0|versicolor|
| 6.5|2.8| 4.6|1.5|versicolor| 0.0| virginica|
| 5.7|2.8| 4.5|1.3|versicolor| 0.0|versicolor|
| 6.3|3.3| 4.7|1.6|versicolor| 0.0| virginica|
| 4.9|2.4| 3.3|1.0|versicolor| 0.0|versicolor|
| 6.6|2.9| 4.6|1.3|versicolor| 0.0| virginica|
| 5.2|2.7| 3.9|1.4|versicolor| 0.0|versicolor|
| 5.0|2.0| 3.5|1.0|versicolor| 0.0|versicolor|
| 5.9|3.0| 4.2|1.5|versicolor| 0.0|versicolor|
| 6.0|2.2| 4.0|1.0|versicolor| 0.0|versicolor|
| 6.1|2.9| 4.7|1.4|versicolor| 0.0|versicolor|
| 5.6|2.9| 3.6|1.3|versicolor| 0.0|versicolor|
| 6.7|3.1| 4.4|1.4|versicolor| 0.0| virginica|
| 5.6|3.0| 4.5|1.5|versicolor| 0.0|versicolor|
| 5.8|2.7| 4.1|1.0|versicolor| 0.0|versicolor|
| 6.2|2.2| 4.5|1.5|versicolor| 0.0|versicolor|
| 5.6|2.5| 3.9|1.1|versicolor| 0.0|versicolor|
++---++---+--+-+--+
```
Author: Yanbo Liang
Closes #15788 from yanboliang/spark-18291.
Project: http://git-wip-us.apache.org/repos/asf/spark/repo
Commit: http://git-wip-us.apache.org/repos/asf/spark/commi
spark git commit: [SPARK-18291][SPARKR][ML] SparkR glm predict should output original label when family = binomial.
Repository: spark
Updated Branches:
refs/heads/branch-2.1 df40ee2b4 -> 6b332909f
[SPARK-18291][SPARKR][ML] SparkR glm predict should output original label when
family = binomial.
## What changes were proposed in this pull request?
SparkR ```spark.glm``` predict should output original label when family =
"binomial".
## How was this patch tested?
Add unit test.
You can also run the following code to test:
```R
training <- suppressWarnings(createDataFrame(iris))
training <- training[training$Species %in% c("versicolor", "virginica"), ]
model <- spark.glm(training, Species ~ Sepal_Length + Sepal_Width,family =
binomial(link = "logit"))
showDF(predict(model, training))
```
Before this change:
```
++---++---+--+-+---+
|Sepal_Length|Sepal_Width|Petal_Length|Petal_Width| Species|label|
prediction|
++---++---+--+-+---+
| 7.0|3.2| 4.7|1.4|versicolor| 0.0|
0.8271421517601544|
| 6.4|3.2| 4.5|1.5|versicolor| 0.0|
0.6044595910413112|
| 6.9|3.1| 4.9|1.5|versicolor| 0.0|
0.7916340858281998|
| 5.5|2.3| 4.0|1.3|versicolor|
0.0|0.16080518180591158|
| 6.5|2.8| 4.6|1.5|versicolor| 0.0|
0.6112229217050189|
| 5.7|2.8| 4.5|1.3|versicolor| 0.0|
0.2555087295500885|
| 6.3|3.3| 4.7|1.6|versicolor| 0.0|
0.5681507664364834|
| 4.9|2.4| 3.3|1.0|versicolor|
0.0|0.05990570219972002|
| 6.6|2.9| 4.6|1.3|versicolor| 0.0|
0.6644434078306246|
| 5.2|2.7| 3.9|1.4|versicolor|
0.0|0.11293577405862379|
| 5.0|2.0| 3.5|1.0|versicolor|
0.0|0.06152372321585971|
| 5.9|3.0| 4.2|1.5|versicolor|
0.0|0.35250697207602555|
| 6.0|2.2| 4.0|1.0|versicolor|
0.0|0.32267018290814303|
| 6.1|2.9| 4.7|1.4|versicolor| 0.0|
0.433391153814592|
| 5.6|2.9| 3.6|1.3|versicolor| 0.0|
0.2280744262436993|
| 6.7|3.1| 4.4|1.4|versicolor| 0.0|
0.7219848389339459|
| 5.6|3.0| 4.5|1.5|versicolor|
0.0|0.23527698971404695|
| 5.8|2.7| 4.1|1.0|versicolor| 0.0|
0.285024533520016|
| 6.2|2.2| 4.5|1.5|versicolor| 0.0|
0.4107047877447493|
| 5.6|2.5| 3.9|1.1|versicolor|
0.0|0.20083561961645083|
++---++---+--+-+---+
```
After this change:
```
++---++---+--+-+--+
|Sepal_Length|Sepal_Width|Petal_Length|Petal_Width| Species|label|prediction|
++---++---+--+-+--+
| 7.0|3.2| 4.7|1.4|versicolor| 0.0| virginica|
| 6.4|3.2| 4.5|1.5|versicolor| 0.0| virginica|
| 6.9|3.1| 4.9|1.5|versicolor| 0.0| virginica|
| 5.5|2.3| 4.0|1.3|versicolor| 0.0|versicolor|
| 6.5|2.8| 4.6|1.5|versicolor| 0.0| virginica|
| 5.7|2.8| 4.5|1.3|versicolor| 0.0|versicolor|
| 6.3|3.3| 4.7|1.6|versicolor| 0.0| virginica|
| 4.9|2.4| 3.3|1.0|versicolor| 0.0|versicolor|
| 6.6|2.9| 4.6|1.3|versicolor| 0.0| virginica|
| 5.2|2.7| 3.9|1.4|versicolor| 0.0|versicolor|
| 5.0|2.0| 3.5|1.0|versicolor| 0.0|versicolor|
| 5.9|3.0| 4.2|1.5|versicolor| 0.0|versicolor|
| 6.0|2.2| 4.0|1.0|versicolor| 0.0|versicolor|
| 6.1|2.9| 4.7|1.4|versicolor| 0.0|versicolor|
| 5.6|2.9| 3.6|1.3|versicolor| 0.0|versicolor|
| 6.7|3.1| 4.4|1.4|versicolor| 0.0| virginica|
| 5.6|3.0| 4.5|1.5|versicolor| 0.0|versicolor|
| 5.8|2.7| 4.1|1.0|versicolor| 0.0|versicolor|
| 6.2|2.2| 4.5|1.5|versicolor| 0.0|versicolor|
| 5.6|2.5| 3.9|1.1|versicolor| 0.0|versicolor|
++---++---+--+-+--+
```
Author: Yanbo Liang
Closes #15788 from yanboliang/spark-18291.
(cherry picked from commit daa975f4bfa4f904697bf3365a4be9987032e490)
Signed-off-by: Yanbo Liang
Project: http:/
spark git commit: [SPARK-18901][ML] Require in LR LogisticAggregator is redundant
Repository: spark
Updated Branches:
refs/heads/master 776a2c0e9 -> 90264aced
[SPARK-18901][ML] Require in LR LogisticAggregator is redundant
## What changes were proposed in this pull request?
In MultivariateOnlineSummarizer,
`add` and `merge` have check for weights and feature sizes. The checks in LR
are redundant, which are removed from this PR.
## How was this patch tested?
Existing tests.
Author: wm...@hotmail.com
Closes #17478 from wangmiao1981/logit.
Project: http://git-wip-us.apache.org/repos/asf/spark/repo
Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/90264ace
Tree: http://git-wip-us.apache.org/repos/asf/spark/tree/90264ace
Diff: http://git-wip-us.apache.org/repos/asf/spark/diff/90264ace
Branch: refs/heads/master
Commit: 90264aced7cfdf265636517b91e5d1324fe60112
Parents: 776a2c0
Author: wm...@hotmail.com
Authored: Mon Apr 24 23:43:06 2017 +0800
Committer: Yanbo Liang
Committed: Mon Apr 24 23:43:06 2017 +0800
--
.../org/apache/spark/ml/classification/LogisticRegression.scala | 5 -
1 file changed, 5 deletions(-)
--
http://git-wip-us.apache.org/repos/asf/spark/blob/90264ace/mllib/src/main/scala/org/apache/spark/ml/classification/LogisticRegression.scala
--
diff --git
a/mllib/src/main/scala/org/apache/spark/ml/classification/LogisticRegression.scala
b/mllib/src/main/scala/org/apache/spark/ml/classification/LogisticRegression.scala
index bc81546..44b3478 100644
---
a/mllib/src/main/scala/org/apache/spark/ml/classification/LogisticRegression.scala
+++
b/mllib/src/main/scala/org/apache/spark/ml/classification/LogisticRegression.scala
@@ -1571,9 +1571,6 @@ private class LogisticAggregator(
*/
def add(instance: Instance): this.type = {
instance match { case Instance(label, weight, features) =>
- require(numFeatures == features.size, s"Dimensions mismatch when adding
new instance." +
-s" Expecting $numFeatures but got ${features.size}.")
- require(weight >= 0.0, s"instance weight, $weight has to be >= 0.0")
if (weight == 0.0) return this
@@ -1596,8 +1593,6 @@ private class LogisticAggregator(
* @return This LogisticAggregator object.
*/
def merge(other: LogisticAggregator): this.type = {
-require(numFeatures == other.numFeatures, s"Dimensions mismatch when
merging with another " +
- s"LogisticAggregator. Expecting $numFeatures but got
${other.numFeatures}.")
if (other.weightSum != 0.0) {
weightSum += other.weightSum
-
To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org
For additional commands, e-mail: commits-h...@spark.apache.org
spark git commit: [SPARK-18901][ML] Require in LR LogisticAggregator is redundant
Repository: spark
Updated Branches:
refs/heads/branch-2.2 2bef01f64 -> cf16c3250
[SPARK-18901][ML] Require in LR LogisticAggregator is redundant
## What changes were proposed in this pull request?
In MultivariateOnlineSummarizer,
`add` and `merge` have check for weights and feature sizes. The checks in LR
are redundant, which are removed from this PR.
## How was this patch tested?
Existing tests.
Author: wm...@hotmail.com
Closes #17478 from wangmiao1981/logit.
(cherry picked from commit 90264aced7cfdf265636517b91e5d1324fe60112)
Signed-off-by: Yanbo Liang
Project: http://git-wip-us.apache.org/repos/asf/spark/repo
Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/cf16c325
Tree: http://git-wip-us.apache.org/repos/asf/spark/tree/cf16c325
Diff: http://git-wip-us.apache.org/repos/asf/spark/diff/cf16c325
Branch: refs/heads/branch-2.2
Commit: cf16c3250e946c4f89edc999d8764e8fa3dfb056
Parents: 2bef01f
Author: wm...@hotmail.com
Authored: Mon Apr 24 23:43:06 2017 +0800
Committer: Yanbo Liang
Committed: Mon Apr 24 23:43:23 2017 +0800
--
.../org/apache/spark/ml/classification/LogisticRegression.scala | 5 -
1 file changed, 5 deletions(-)
--
http://git-wip-us.apache.org/repos/asf/spark/blob/cf16c325/mllib/src/main/scala/org/apache/spark/ml/classification/LogisticRegression.scala
--
diff --git
a/mllib/src/main/scala/org/apache/spark/ml/classification/LogisticRegression.scala
b/mllib/src/main/scala/org/apache/spark/ml/classification/LogisticRegression.scala
index bc81546..44b3478 100644
---
a/mllib/src/main/scala/org/apache/spark/ml/classification/LogisticRegression.scala
+++
b/mllib/src/main/scala/org/apache/spark/ml/classification/LogisticRegression.scala
@@ -1571,9 +1571,6 @@ private class LogisticAggregator(
*/
def add(instance: Instance): this.type = {
instance match { case Instance(label, weight, features) =>
- require(numFeatures == features.size, s"Dimensions mismatch when adding
new instance." +
-s" Expecting $numFeatures but got ${features.size}.")
- require(weight >= 0.0, s"instance weight, $weight has to be >= 0.0")
if (weight == 0.0) return this
@@ -1596,8 +1593,6 @@ private class LogisticAggregator(
* @return This LogisticAggregator object.
*/
def merge(other: LogisticAggregator): this.type = {
-require(numFeatures == other.numFeatures, s"Dimensions mismatch when
merging with another " +
- s"LogisticAggregator. Expecting $numFeatures but got
${other.numFeatures}.")
if (other.weightSum != 0.0) {
weightSum += other.weightSum
-
To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org
For additional commands, e-mail: commits-h...@spark.apache.org
spark git commit: [SPARK-18901][FOLLOWUP][ML] Require in LR LogisticAggregator is redundant
Repository: spark
Updated Branches:
refs/heads/master 0bc7a9021 -> 387565cf1
[SPARK-18901][FOLLOWUP][ML] Require in LR LogisticAggregator is redundant
## What changes were proposed in this pull request?
This is a follow-up PR of #17478.
## How was this patch tested?
Existing tests
Author: wangmiao1981
Closes #17754 from wangmiao1981/followup.
Project: http://git-wip-us.apache.org/repos/asf/spark/repo
Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/387565cf
Tree: http://git-wip-us.apache.org/repos/asf/spark/tree/387565cf
Diff: http://git-wip-us.apache.org/repos/asf/spark/diff/387565cf
Branch: refs/heads/master
Commit: 387565cf14b490810f9479ff3adbf776e2edecdc
Parents: 0bc7a90
Author: wangmiao1981
Authored: Tue Apr 25 16:30:36 2017 +0800
Committer: Yanbo Liang
Committed: Tue Apr 25 16:30:36 2017 +0800
--
.../scala/org/apache/spark/ml/classification/LinearSVC.scala| 5 ++---
.../scala/org/apache/spark/ml/regression/LinearRegression.scala | 5 -
2 files changed, 2 insertions(+), 8 deletions(-)
--
http://git-wip-us.apache.org/repos/asf/spark/blob/387565cf/mllib/src/main/scala/org/apache/spark/ml/classification/LinearSVC.scala
--
diff --git
a/mllib/src/main/scala/org/apache/spark/ml/classification/LinearSVC.scala
b/mllib/src/main/scala/org/apache/spark/ml/classification/LinearSVC.scala
index f76b14e..7507c75 100644
--- a/mllib/src/main/scala/org/apache/spark/ml/classification/LinearSVC.scala
+++ b/mllib/src/main/scala/org/apache/spark/ml/classification/LinearSVC.scala
@@ -458,9 +458,7 @@ private class LinearSVCAggregator(
*/
def add(instance: Instance): this.type = {
instance match { case Instance(label, weight, features) =>
- require(weight >= 0.0, s"instance weight, $weight has to be >= 0.0")
- require(numFeatures == features.size, s"Dimensions mismatch when adding
new instance." +
-s" Expecting $numFeatures but got ${features.size}.")
+
if (weight == 0.0) return this
val localFeaturesStd = bcFeaturesStd.value
val localCoefficients = coefficientsArray
@@ -512,6 +510,7 @@ private class LinearSVCAggregator(
* @return This LinearSVCAggregator object.
*/
def merge(other: LinearSVCAggregator): this.type = {
+
if (other.weightSum != 0.0) {
weightSum += other.weightSum
lossSum += other.lossSum
http://git-wip-us.apache.org/repos/asf/spark/blob/387565cf/mllib/src/main/scala/org/apache/spark/ml/regression/LinearRegression.scala
--
diff --git
a/mllib/src/main/scala/org/apache/spark/ml/regression/LinearRegression.scala
b/mllib/src/main/scala/org/apache/spark/ml/regression/LinearRegression.scala
index f7e3c8f..eaad549 100644
--- a/mllib/src/main/scala/org/apache/spark/ml/regression/LinearRegression.scala
+++ b/mllib/src/main/scala/org/apache/spark/ml/regression/LinearRegression.scala
@@ -971,9 +971,6 @@ private class LeastSquaresAggregator(
*/
def add(instance: Instance): this.type = {
instance match { case Instance(label, weight, features) =>
- require(dim == features.size, s"Dimensions mismatch when adding new
sample." +
-s" Expecting $dim but got ${features.size}.")
- require(weight >= 0.0, s"instance weight, $weight has to be >= 0.0")
if (weight == 0.0) return this
@@ -1005,8 +1002,6 @@ private class LeastSquaresAggregator(
* @return This LeastSquaresAggregator object.
*/
def merge(other: LeastSquaresAggregator): this.type = {
-require(dim == other.dim, s"Dimensions mismatch when merging with another
" +
- s"LeastSquaresAggregator. Expecting $dim but got ${other.dim}.")
if (other.weightSum != 0) {
totalCnt += other.totalCnt
-
To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org
For additional commands, e-mail: commits-h...@spark.apache.org
spark git commit: [SPARK-18901][FOLLOWUP][ML] Require in LR LogisticAggregator is redundant
Repository: spark
Updated Branches:
refs/heads/branch-2.2 b62ebd91b -> e2591c6d7
[SPARK-18901][FOLLOWUP][ML] Require in LR LogisticAggregator is redundant
## What changes were proposed in this pull request?
This is a follow-up PR of #17478.
## How was this patch tested?
Existing tests
Author: wangmiao1981
Closes #17754 from wangmiao1981/followup.
(cherry picked from commit 387565cf14b490810f9479ff3adbf776e2edecdc)
Signed-off-by: Yanbo Liang
Project: http://git-wip-us.apache.org/repos/asf/spark/repo
Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/e2591c6d
Tree: http://git-wip-us.apache.org/repos/asf/spark/tree/e2591c6d
Diff: http://git-wip-us.apache.org/repos/asf/spark/diff/e2591c6d
Branch: refs/heads/branch-2.2
Commit: e2591c6d74081e9edad2e8982c0125a4f1d21437
Parents: b62ebd9
Author: wangmiao1981
Authored: Tue Apr 25 16:30:36 2017 +0800
Committer: Yanbo Liang
Committed: Tue Apr 25 16:30:53 2017 +0800
--
.../scala/org/apache/spark/ml/classification/LinearSVC.scala| 5 ++---
.../scala/org/apache/spark/ml/regression/LinearRegression.scala | 5 -
2 files changed, 2 insertions(+), 8 deletions(-)
--
http://git-wip-us.apache.org/repos/asf/spark/blob/e2591c6d/mllib/src/main/scala/org/apache/spark/ml/classification/LinearSVC.scala
--
diff --git
a/mllib/src/main/scala/org/apache/spark/ml/classification/LinearSVC.scala
b/mllib/src/main/scala/org/apache/spark/ml/classification/LinearSVC.scala
index f76b14e..7507c75 100644
--- a/mllib/src/main/scala/org/apache/spark/ml/classification/LinearSVC.scala
+++ b/mllib/src/main/scala/org/apache/spark/ml/classification/LinearSVC.scala
@@ -458,9 +458,7 @@ private class LinearSVCAggregator(
*/
def add(instance: Instance): this.type = {
instance match { case Instance(label, weight, features) =>
- require(weight >= 0.0, s"instance weight, $weight has to be >= 0.0")
- require(numFeatures == features.size, s"Dimensions mismatch when adding
new instance." +
-s" Expecting $numFeatures but got ${features.size}.")
+
if (weight == 0.0) return this
val localFeaturesStd = bcFeaturesStd.value
val localCoefficients = coefficientsArray
@@ -512,6 +510,7 @@ private class LinearSVCAggregator(
* @return This LinearSVCAggregator object.
*/
def merge(other: LinearSVCAggregator): this.type = {
+
if (other.weightSum != 0.0) {
weightSum += other.weightSum
lossSum += other.lossSum
http://git-wip-us.apache.org/repos/asf/spark/blob/e2591c6d/mllib/src/main/scala/org/apache/spark/ml/regression/LinearRegression.scala
--
diff --git
a/mllib/src/main/scala/org/apache/spark/ml/regression/LinearRegression.scala
b/mllib/src/main/scala/org/apache/spark/ml/regression/LinearRegression.scala
index f7e3c8f..eaad549 100644
--- a/mllib/src/main/scala/org/apache/spark/ml/regression/LinearRegression.scala
+++ b/mllib/src/main/scala/org/apache/spark/ml/regression/LinearRegression.scala
@@ -971,9 +971,6 @@ private class LeastSquaresAggregator(
*/
def add(instance: Instance): this.type = {
instance match { case Instance(label, weight, features) =>
- require(dim == features.size, s"Dimensions mismatch when adding new
sample." +
-s" Expecting $dim but got ${features.size}.")
- require(weight >= 0.0, s"instance weight, $weight has to be >= 0.0")
if (weight == 0.0) return this
@@ -1005,8 +1002,6 @@ private class LeastSquaresAggregator(
* @return This LeastSquaresAggregator object.
*/
def merge(other: LeastSquaresAggregator): this.type = {
-require(dim == other.dim, s"Dimensions mismatch when merging with another
" +
- s"LeastSquaresAggregator. Expecting $dim but got ${other.dim}.")
if (other.weightSum != 0) {
totalCnt += other.totalCnt
-
To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org
For additional commands, e-mail: commits-h...@spark.apache.org
