[ https://issues.apache.org/jira/browse/SPARK-22879?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16301681#comment-16301681 ]
Sean Owen commented on SPARK-22879: ----------------------------------- I'm not quite following here. In both cases the rule is "1 if > threshold". Are you saying that the recomputation of the threshold from t isn't equal to the original one? that's a little different. > LogisticRegression inconsistent prediction when proba == threshold > ------------------------------------------------------------------ > > Key: SPARK-22879 > URL: https://issues.apache.org/jira/browse/SPARK-22879 > Project: Spark > Issue Type: Bug > Components: ML, MLlib > Affects Versions: 1.6.3 > Reporter: Adrien Lavoillotte > Priority: Minor > > I'm using {{org.apache.spark.ml.classification.LogisticRegression}} for > binary classification. > If I predict on a record that yields exactly the probability of the > threshold, then the result of {{transform}} is different depending on whether > the {{rawPredictionCol}} param is empty on the model or not. > If it is empty, as most ML tools I've seen, it correctly predicts 0, the rule > being {{ if (proba > threshold) then 1 else 0 }} (implemented in > {{probability2prediction}}). > If however {{rawPredictionCol}} is set (default), then it avoids > recomputation by calling {{raw2prediction}}, and the rule becomes {{if > (rawPrediction(1) > rawThreshold) 1 else 0}}. The {{rawThreshold = math.log(t > / (1.0 - t))}} is ever-so-slightly below the {{rawPrediction(1)}}, so it > predicts 1. > The use case is that I choose the threshold amongst > {{BinaryClassificationMetrics#thresholds}}, so I get one that corresponds to > the probability for one or more of my test set's records. Re-scoring that > record or one that yields the same probability exhibits this behaviour. > Tested this on Spark 1.6 but the code involved seems to be similar on Spark > 2.2. -- This message was sent by Atlassian JIRA (v6.4.14#64029) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org