Thank, Nick.
This worked for me.
val evaluator = new BinaryClassificationEvaluator().
setLabelCol("label").
setRawPredictionCol("ModelProbability").
setMetricName("areaUnderROC")
val auROC = evaluator.evaluate(testResults)
On M
Typically you pass in the result of a model transform to the evaluator.
So:
val model = estimator.fit(data)
val auc = evaluator.evaluate(model.transform(testData)
Check Scala API docs for some details:
http://spark.apache.org/docs/latest/api/scala/index.html#org.apache.spark.ml.evaluation.BinaryC
Can you please suggest how I can use BinaryClassificationEvaluator? I tried:
scala> import org.apache.spark.ml.evaluation.BinaryClassificationEvaluator
import org.apache.spark.ml.evaluation.BinaryClassificationEvaluator
scala> val evaluator = new BinaryClassificationEvaluator()
evaluator: org.ap
DataFrame.rdd returns an RDD[Row]. You'll need to use map to extract the
doubles from the test score and label DF.
But you may prefer to just use spark.ml evaluators, which work with
DataFrames. Try BinaryClassificationEvaluator.
On Mon, 14 Nov 2016 at 19:30, Bhaarat Sharma wrote:
> I am gettin
I am getting scala.MatchError in the code below. I'm not able to see why
this would be happening. I am using Spark 2.0.1
scala> testResults.columns
res538: Array[String] = Array(TopicVector, subject_id, hadm_id,
isElective, isNewborn, isUrgent, isEmergency, isMale, isFemale,
oasis_score, sapsii_sc
