GitHub user sueann opened a pull request:
https://github.com/apache/spark/pull/16480
[SPARK-18194][ML] Log instrumentation in OneVsRest, CrossValidator,
TrainValidationSplit
## What changes were proposed in this pull request?
Added instrumentation logging for OneVsRest classifier, CrossValidator,
TrainValidationSplit fit() functions.
## How was this patch tested?
Ran unit tests and checked the log file.
* TrainValidationSplit
```
$ build/sbt "test-only
org.apache.spark.ml.tuning.TrainValidationSplitSuiteâ
$ less mllib/target/unit-tests.log
...
17/01/05 11:21:51.877
pool-1-thread-1-ScalaTest-running-TrainValidationSplitSuite INFO
Instrumentation: TrainValidationSplit-tvs_8e7090288e01-206645650-7: training:
numPartitions=2 storageLevel=StorageLevel(1 replicas)
17/01/05 11:21:51.877
pool-1-thread-1-ScalaTest-running-TrainValidationSplitSuite INFO
Instrumentation: TrainValidationSplit-tvs_8e7090288e01-206645650-7:
{"trainRatio":0.5}
17/01/05 11:21:51.877
pool-1-thread-1-ScalaTest-running-TrainValidationSplitSuite INFO
Instrumentation: TrainValidationSplit-tvs_8e7090288e01-206645650-7:
{"estimator":"LinearRegression","evaluator":"RegressionEvaluator","numModels":4}
17/01/05 11:21:51.877
pool-1-thread-1-ScalaTest-running-TrainValidationSplitSuite INFO
TrainValidationSplit: Train validation split metrics:
WrappedArray(7.79718164555543, 7.734602725775261, 7.79718164555543,
0.11150408019144314)
17/01/05 11:21:51.877
pool-1-thread-1-ScalaTest-running-TrainValidationSplitSuite INFO
TrainValidationSplit: Best set of parameters:
{
linReg_8e7bf3d889f1-maxIter: 10,
linReg_8e7bf3d889f1-regParam: 0.001
}
17/01/05 11:21:51.877
pool-1-thread-1-ScalaTest-running-TrainValidationSplitSuite INFO
TrainValidationSplit: Best train validation split metric: 0.11150408019144314.
...
17/01/05 11:21:52.087
pool-1-thread-1-ScalaTest-running-TrainValidationSplitSuite INFO
Instrumentation: TrainValidationSplit-tvs_8e7090288e01-206645650-7: training
finished
```
* CrossValdator
```
$ build/sbt "test-only org.apache.spark.ml.tuning.CrossValidatorSuite"
$ less mllib/target/unit-tests.log
...
17/01/05 11:27:26.860 pool-1-thread-1-ScalaTest-running-CrossValidatorSuite
INFO Instrumentation: CrossValidator-cv_f60fd2c0f5ce-428174643-13: training:
numPartitions=2 storageLevel=StorageLevel(1 replicas)
17/01/05 11:27:26.860 pool-1-thread-1-ScalaTest-running-CrossValidatorSuite
INFO Instrumentation: CrossValidator-cv_f60fd2c0f5ce-428174643-13:
{"numFolds":3}
17/01/05 11:27:26.860 pool-1-thread-1-ScalaTest-running-CrossValidatorSuite
INFO Instrumentation: CrossValidator-cv_f60fd2c0f5ce-428174643-13:
{"estimator":"LogisticRegression","evaluator":"BinaryClassificationEvaluator","numModels":4}
17/01/05 11:27:26.860 pool-1-thread-1-ScalaTest-running-CrossValidatorSuite
INFO CrossValidator: Average cross-validation metrics: WrappedArray(0.5, 0.5,
0.8333037031406596, 0.8333037031406596)
17/01/05 11:27:26.862 pool-1-thread-1-ScalaTest-running-CrossValidatorSuite
INFO CrossValidator: Best set of parameters:
{
logreg_e763c2efb948-maxIter: 10,
logreg_e763c2efb948-regParam: 0.001
}
17/01/05 11:27:26.863 pool-1-thread-1-ScalaTest-running-CrossValidatorSuite
INFO CrossValidator: Best cross-validation metric: 0.8333037031406596.
...
17/01/05 11:27:27.040 pool-1-thread-1-ScalaTest-running-CrossValidatorSuite
INFO Instrumentation: CrossValidator-cv_f60fd2c0f5ce-428174643-13: training
finished
...
```
* OneVsRest
```
$ build/sbt "test-only org.apache.spark.ml.classification.OneVsRestSuite"
$ less mllib/target/unit-tests.log
...
17/01/05 13:26:36.017 pool-1-thread-1-ScalaTest-running-OneVsRestSuite INFO
Instrumentation: OneVsRest-oneVsRest_551a9c8d41e4-1366242214-12: training:
numPartitions=2 storageLevel=StorageLevel(1 replicas)
17/01/05 13:26:36.018 pool-1-thread-1-ScalaTest-running-OneVsRestSuite INFO
Instrumentation: OneVsRest-oneVsRest_551a9c8d41e4-1366242214-12:
{"labelCol":"indexed","featuresCol":"f","predictionCol":"p"}
17/01/05 13:26:36.018 pool-1-thread-1-ScalaTest-running-OneVsRestSuite INFO
Instrumentation: OneVsRest-oneVsRest_551a9c8d41e4-1366242214-12:
{"numClasses":3}
17/01/05 13:26:36.018 pool-1-thread-1-ScalaTest-running-OneVsRestSuite INFO
Instrumentation: OneVsRest-oneVsRest_551a9c8d41e4-1366242214-12:
{"classifier":"LogisticRegression"}
17/01/05 13:26:36.018 pool-1-thread-1-ScalaTest-running-OneVsRestSuite INFO
Instrumentation: OneVsRest-oneVsRest_551a9c8d41e4-1366242214-12: training
finished
...
```
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/sueann/spark SPARK-18194
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/16480.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #16480
----
commit 84625f2e09b7d35f5487f26c3900c28c5b9d3caa
Author: sueann <sue...@databricks.com>
Date: 2017-01-05T06:39:05Z
[SPARK-18194] log stuff
commit 327c9ff1d809c6bb526916fd20f08e3c5507636e
Author: sueann <sue...@databricks.com>
Date: 2017-01-05T21:36:09Z
[SPARK-18194] instrumentation logging for OneVsRest, CrossValidator,
TrainValidationSplit
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
