[jira] [Commented] (SPARK-21027) Parallel One vs. Rest Classifier
[ https://issues.apache.org/jira/browse/SPARK-21027?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16164120#comment-16164120 ] Apache Spark commented on SPARK-21027: -- User 'WeichenXu123' has created a pull request for this issue: https://github.com/apache/spark/pull/19214 > Parallel One vs. Rest Classifier > > > Key: SPARK-21027 > URL: https://issues.apache.org/jira/browse/SPARK-21027 > Project: Spark > Issue Type: New Feature > Components: PySpark >Affects Versions: 2.2.0, 2.2.1 >Reporter: Ajay Saini >Assignee: Ajay Saini > Fix For: 2.3.0 > > > Currently, the Scala implementation of OneVsRest allows the user to run a > parallel implementation in which each class is evaluated in a different > thread. This implementation allows up to a 2X speedup as determined by > experiments but is not currently not tunable. Furthermore, the python > implementation of OneVsRest does not parallelize at all. It would be useful > to add a parallel, tunable implementation of OneVsRest to the python library > in order to speed up the algorithm. > A ticket for the Scala implementation of this classifier is here: > https://issues.apache.org/jira/browse/SPARK-21028 -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-21027) Parallel One vs. Rest Classifier
[ https://issues.apache.org/jira/browse/SPARK-21027?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16151675#comment-16151675 ] Apache Spark commented on SPARK-21027: -- User 'WeichenXu123' has created a pull request for this issue: https://github.com/apache/spark/pull/19110 > Parallel One vs. Rest Classifier > > > Key: SPARK-21027 > URL: https://issues.apache.org/jira/browse/SPARK-21027 > Project: Spark > Issue Type: New Feature > Components: PySpark >Affects Versions: 2.2.0, 2.2.1 >Reporter: Ajay Saini > > Currently, the Scala implementation of OneVsRest allows the user to run a > parallel implementation in which each class is evaluated in a different > thread. This implementation allows up to a 2X speedup as determined by > experiments but is not currently not tunable. Furthermore, the python > implementation of OneVsRest does not parallelize at all. It would be useful > to add a parallel, tunable implementation of OneVsRest to the python library > in order to speed up the algorithm. > A ticket for the Scala implementation of this classifier is here: > https://issues.apache.org/jira/browse/SPARK-21028 -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-21027) Parallel One vs. Rest Classifier
[
https://issues.apache.org/jira/browse/SPARK-21027?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16047243#comment-16047243
]
Joseph K. Bradley commented on SPARK-21027:
---
Copying from [ML-14450]:
[SPARK-7861] adds a Python wrapper for OneVsRest. Because of possible issues
related to using existing libraries like {{multiprocessing}}, we are not
training multiple models in parallel initially.
This issue is for prototyping, testing, and implementing a way to train
multiple models at once. Speaking with [~joshrosen], a good option might be
the concurrent.futures package:
* Python 3.x:
[https://docs.python.org/3/library/concurrent.futures.html#module-concurrent.futures]
* Python 2.x: [https://pypi.python.org/pypi/futures]
> Parallel One vs. Rest Classifier
>
>
> Key: SPARK-21027
> URL: https://issues.apache.org/jira/browse/SPARK-21027
> Project: Spark
> Issue Type: New Feature
> Components: PySpark
>Affects Versions: 2.2.0, 2.2.1
>Reporter: Ajay Saini
>
> Currently, the Scala implementation of OneVsRest allows the user to run a
> parallel implementation in which each class is evaluated in a different
> thread. This implementation allows up to a 2X speedup as determined by
> experiments but is not currently not tunable. Furthermore, the python
> implementation of OneVsRest does not parallelize at all. It would be useful
> to add a parallel, tunable implementation of OneVsRest to the python library
> in order to speed up the algorithm.
> A ticket for the Scala implementation of this classifier is here:
> https://issues.apache.org/jira/browse/SPARK-21028
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-21027) Parallel One vs. Rest Classifier
[ https://issues.apache.org/jira/browse/SPARK-21027?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16047241#comment-16047241 ] Joseph K. Bradley commented on SPARK-21027: --- Whoops! I realized I'd reported this long ago...and I'd said there might be issues with python multiprocessing. I don't recall what those issues were. Let's investigate. > Parallel One vs. Rest Classifier > > > Key: SPARK-21027 > URL: https://issues.apache.org/jira/browse/SPARK-21027 > Project: Spark > Issue Type: New Feature > Components: PySpark >Affects Versions: 2.2.0, 2.2.1 >Reporter: Ajay Saini > > Currently, the Scala implementation of OneVsRest allows the user to run a > parallel implementation in which each class is evaluated in a different > thread. This implementation allows up to a 2X speedup as determined by > experiments but is not currently not tunable. Furthermore, the python > implementation of OneVsRest does not parallelize at all. It would be useful > to add a parallel, tunable implementation of OneVsRest to the python library > in order to speed up the algorithm. > A ticket for the Scala implementation of this classifier is here: > https://issues.apache.org/jira/browse/SPARK-21028 -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-21027) Parallel One vs. Rest Classifier
[ https://issues.apache.org/jira/browse/SPARK-21027?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16047127#comment-16047127 ] Apache Spark commented on SPARK-21027: -- User 'ajaysaini725' has created a pull request for this issue: https://github.com/apache/spark/pull/18281 > Parallel One vs. Rest Classifier > > > Key: SPARK-21027 > URL: https://issues.apache.org/jira/browse/SPARK-21027 > Project: Spark > Issue Type: New Feature > Components: PySpark >Affects Versions: 2.2.0, 2.2.1 >Reporter: Ajay Saini > > Currently, the Scala implementation of OneVsRest allows the user to run a > parallel implementation in which each class is evaluated in a different > thread. This implementation allows up to a 2X speedup as determined by > experiments but is not currently not tunable. Furthermore, the python > implementation of OneVsRest does not parallelize at all. It would be useful > to add a parallel, tunable implementation of OneVsRest to the python library > in order to speed up the algorithm. > A ticket for the Scala implementation of this classifier is here: > https://issues.apache.org/jira/browse/SPARK-21028 -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-21027) Parallel One vs. Rest Classifier
[ https://issues.apache.org/jira/browse/SPARK-21027?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16045245#comment-16045245 ] Ajay Saini commented on SPARK-21027: Yup! Modified the Scala ticket to indicate the necessity of adding a tunable parallelism parameter. > Parallel One vs. Rest Classifier > > > Key: SPARK-21027 > URL: https://issues.apache.org/jira/browse/SPARK-21027 > Project: Spark > Issue Type: New Feature > Components: PySpark >Affects Versions: 2.2.0, 2.2.1 >Reporter: Ajay Saini > > Currently, the Scala implementation of OneVsRest allows the user to run a > parallel implementation in which each class is evaluated in a different > thread. This implementation allows up to a 2X speedup as determined by > experiments but is not currently not tunable. Furthermore, the python > implementation of OneVsRest does not parallelize at all. It would be useful > to add a parallel, tunable implementation of OneVsRest to the python library > in order to speed up the algorithm. > A ticket for the Scala implementation of this classifier is here: > https://issues.apache.org/jira/browse/SPARK-21028 -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-21027) Parallel One vs. Rest Classifier
[ https://issues.apache.org/jira/browse/SPARK-21027?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16045228#comment-16045228 ] Joseph K. Bradley commented on SPARK-21027: --- One other comment: The Scala implementation can be faster for some tasks but can be slower for others (b/c of conflicting Spark jobs). It'd be nice to permit the user to tune the parallelism, with some reasonable default such as 4 parallel training jobs. > Parallel One vs. Rest Classifier > > > Key: SPARK-21027 > URL: https://issues.apache.org/jira/browse/SPARK-21027 > Project: Spark > Issue Type: New Feature > Components: PySpark >Affects Versions: 2.2.0, 2.2.1 >Reporter: Ajay Saini > > Currently, the Scala implementation of OneVsRest allows the user to run a > parallel implementation in which each class is evaluated in a different > thread. This implementation allows up to a 2X speedup as determined by > experiments but is not currently not tunable. Furthermore, the python > implementation of OneVsRest does not parallelize at all. It would be useful > to add a parallel, tunable implementation of OneVsRest to the python library > in order to speed up the algorithm. > A ticket for the Scala implementation of this classifier is here: > https://issues.apache.org/jira/browse/SPARK-21028 -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-21027) Parallel One vs. Rest Classifier
[ https://issues.apache.org/jira/browse/SPARK-21027?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16044863#comment-16044863 ] Ajay Saini commented on SPARK-21027: Modified the description to include a description of the parallelism and the need for the algorithm. > Parallel One vs. Rest Classifier > > > Key: SPARK-21027 > URL: https://issues.apache.org/jira/browse/SPARK-21027 > Project: Spark > Issue Type: New Feature > Components: PySpark >Affects Versions: 2.2.0, 2.2.1 >Reporter: Ajay Saini > > Currently, the Scala implementation of OneVsRest allows the user to run a > parallel implementation in which each class is evaluated in a different > thread. This implementation allows up to a 2X speedup as determined by > experiments but is not currently not tunable. Furthermore, the python > implementation of OneVsRest does not parallelize at all. It would be useful > to add a parallel, tunable implementation of OneVsRest to the python library > in order to speed up the algorithm. > A ticket for the Scala implementation of this classifier is here: > https://issues.apache.org/jira/browse/SPARK-21028 -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-21027) Parallel One vs. Rest Classifier
[ https://issues.apache.org/jira/browse/SPARK-21027?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16044102#comment-16044102 ] Sean Owen commented on SPARK-21027: --- I don't see a reason for multiple JIRAs. Parallel in what sense? I don't see any detail here. > Parallel One vs. Rest Classifier > > > Key: SPARK-21027 > URL: https://issues.apache.org/jira/browse/SPARK-21027 > Project: Spark > Issue Type: New Feature > Components: PySpark >Affects Versions: 2.2.0, 2.2.1 >Reporter: Ajay Saini > > Adding a class called ParOneVsRest that includes support for a parallelism > parameter in a one vs. rest implementation. A parallel one vs. rest > implementation gives up to a 2X speedup when tested on a dataset with 181024 > points. A ticket for the Scala implementation of this classifier is here: > https://issues.apache.org/jira/browse/SPARK-21028 -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
