Github user MLnick commented on the issue:
https://github.com/apache/spark/pull/18281
Originally in the CV pr Bryan used par collections. Tim pointed out perhaps
futures would be better and to add the ability to use different custom
threadpools. However I think the intention was same as here - i.e. To just
ensure the threadpool is a new dedicated one and not the default in order
not to block multi tenant apps.
I tend to prefer the par collection approach for simplicity and
readability. Also if we can share the param then great - "parallelism"
works I think and is generic enough to be used by multiple components.
Also perhaps this wouldn't work with Scala 2.10? I think that may be ok now
since we intend to drop support in 2.3.x ?
If we agree on approach then the CV PR can be changed back to using par
collections with custom task support.
On Wed, 14 Jun 2017 at 03:36, jkbradley <notificati...@github.com> wrote:
> *@jkbradley* commented on this pull request.
> ------------------------------
>
> In mllib/src/main/scala/org/apache/spark/ml/classification/OneVsRest.scala
> <https://github.com/apache/spark/pull/18281#discussion_r121836808>:
>
> > @@ -325,8 +343,13 @@ final class OneVsRest @Since("1.4.0") (
> multiclassLabeled.persist(StorageLevel.MEMORY_AND_DISK)
> }
>
> + val iters = Range(0, numClasses).par
>
> Discussed offline with @thunterdb <https://github.com/thunterdb> :
> Setting the parallel collection tasksupport value is essentially doing the
> same thing as in @BryanCutler <https://github.com/bryancutler> 's PR:
> ForkJoinTaskSupport is using the ExecutionContext created by ForkJoinPool
> under the hood.
>
> â
> You are receiving this because you were mentioned.
> Reply to this email directly, view it on GitHub
> <https://github.com/apache/spark/pull/18281#discussion_r121836808>, or
mute
> the thread
>
<https://github.com/notifications/unsubscribe-auth/AA_SB35eWKaGmfDIF7gnqVwqrFnwDArMks5sDzkqgaJpZM4N3uuq>
> .
>
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
