[ https://issues.apache.org/jira/browse/SPARK-14746?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15250229#comment-15250229 ]
Shivaram Venkataraman commented on SPARK-14746: ----------------------------------------------- Yeah I can see that RDD pipe is limited and it would be good to have some better support for calling into R from a Scala / Java environment. However I feel this is introducing a new paradigm in DataFrame / Spark where one can mix and match languages (so far we have built things to be used in Python / R) and I don't know if this is a route we want to go down ([~rxin] ?) . My other reason considering this as lower priority is that this is useful for users who are proficient in more than 1 language which I think is a smaller set of users. > Support transformations in R source code for Dataset/DataFrame > -------------------------------------------------------------- > > Key: SPARK-14746 > URL: https://issues.apache.org/jira/browse/SPARK-14746 > Project: Spark > Issue Type: New Feature > Components: SparkR, SQL > Reporter: Sun Rui > > there actually is a desired scenario mentioned several times in the Spark > mailing list that users are writing Scala/Java Spark applications (not > SparkR) but want to use R functions in some transformations. typically this > can be achieved by calling Pipe() in RDD. However, there are limitations on > pipe(). So we can support applying a R function in source code format to a > Dataset/DataFrame (Thus SparkR is not needed for serializing an R function.) -- This message was sent by Atlassian JIRA (v6.3.4#6332) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org