[
https://issues.apache.org/jira/browse/SPARK-4547?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Xiangrui Meng updated SPARK-4547:
---------------------------------
Assignee: Sean Owen
> OOM when making bins in BinaryClassificationMetrics
> ---------------------------------------------------
>
> Key: SPARK-4547
> URL: https://issues.apache.org/jira/browse/SPARK-4547
> Project: Spark
> Issue Type: Bug
> Components: MLlib
> Affects Versions: 1.1.0
> Reporter: Sean Owen
> Assignee: Sean Owen
> Priority: Minor
>
> Also following up on
> http://mail-archives.apache.org/mod_mbox/spark-dev/201411.mbox/%3CCAMAsSdK4s4TNkf3_ecLC6yD-pLpys_PpT3WB7Tp6=yoxuxf...@mail.gmail.com%3E
> -- this one I intend to make a PR for a bit later. The conversation was
> basically:
> {quote}
> Recently I was using BinaryClassificationMetrics to build an AUC curve for a
> classifier over a reasonably large number of points (~12M). The scores were
> all probabilities, so tended to be almost entirely unique.
> The computation does some operations by key, and this ran out of memory. It's
> something you can solve with more than the default amount of memory, but in
> this case, it seemed unuseful to create an AUC curve with such fine-grained
> resolution.
> I ended up just binning the scores so there were ~1000 unique values
> and then it was fine.
> {quote}
> and:
> {quote}
> Yes, if there are many distinct values, we need binning to compute the AUC
> curve. Usually, the scores are not evenly distribution, we cannot simply
> truncate the digits. Estimating the quantiles for binning is necessary,
> similar to RangePartitioner:
> https://github.com/apache/spark/blob/master/core/src/main/scala/org/apache/spark/Partitioner.scala#L104
> Limiting the number of bins is definitely useful.
> {quote}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
