[
https://issues.apache.org/jira/browse/SPARK-17904?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15572541#comment-15572541
]
Felix Cheung edited comment on SPARK-17904 at 10/13/16 5:09 PM:
----------------------------------------------------------------
I somewhat disagree, actually. In R, it is very common to use package
management like
https://rstudio.github.io/packrat/
that is not very different from how Python does it.
In fact, Anaconda works with R too:
https://www.continuum.io/blog/developer-blog/anaconda-r-users-sparkr-and-rbokeh
To me, I think we have questions on whether Spark should get into the business
of package management as [~srowen] has pointed out, or not.
Also there are challenges with how Spark does not have access to all
nodes/executors (because it is not a cluster manager) and issues with dynamic
resource allocations and so on as others have pointed out.
was (Author: felixcheung):
I somewhat disagree, actually. In R, it is very common to use package
management like
https://rstudio.github.io/packrat/
that is not very different from how Python does it.
In fact, Anaconda works with R too:
https://www.continuum.io/blog/developer-blog/anaconda-r-users-sparkr-and-rbokeh
> Add a wrapper function to install R packages on each executors.
> ---------------------------------------------------------------
>
> Key: SPARK-17904
> URL: https://issues.apache.org/jira/browse/SPARK-17904
> Project: Spark
> Issue Type: New Feature
> Components: SparkR
> Reporter: Yanbo Liang
>
> SparkR provides {{spark.lappy}} to run local R functions in distributed
> environment, and {{dapply}} to run UDF on SparkDataFrame.
> If users use third-party libraries inside of the function which was passed
> into {{spark.lappy}} or {{dapply}}, they should install required R packages
> on each executor in advance.
> To install dependent R packages on each executors and check it successfully,
> we can run similar code like following:
> (Note: The code is just for example, not the prototype of this proposal. The
> detail implementation should be discussed.)
> {code}
> rdd <- SparkR:::lapplyPartition(SparkR:::parallelize(sc, 1:2, 2L),
> install.packages("Matrix”))
> test <- function(x) { "Matrix" %in% rownames(installed.packages()) }
> rdd <- SparkR:::lapplyPartition(SparkR:::parallelize(sc, 1:2, 2L), test )
> collectRDD(rdd)
> {code}
> It’s cumbersome to run this code snippet each time when you need third-party
> library, since SparkR is an interactive analytics tools, users may call lots
> of libraries during the analytics session. In native R, users can run
> {{install.packages()}} and {{library()}} across the interactive session.
> Should we provide one API to wrapper the work mentioned above, then users can
> install dependent R packages to each executor easily?
> I propose the following API:
> {{spark.installPackages(pkgs, repos)}}
> * pkgs: the name of packages. If repos = NULL, this can be set with a
> local/hdfs path, then SparkR can install packages from local package archives.
> * repos: the base URL(s) of the repositories to use. It can be NULL to
> install from local directories.
> Since SparkR has its own library directories where to install the packages on
> each executor, so I think it will not pollute the native R environment. I'd
> like to know whether it make sense, and feel free to correct me if there is
> misunderstanding.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
