[jira] [Commented] (SPARK-8506) SparkR does not provide an easy way to depend on Spark Packages when performing init from inside of R
[ https://issues.apache.org/jira/browse/SPARK-8506?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14998000#comment-14998000 ] Girish Reddy commented on SPARK-8506: - Hi [~holdenk] - I am getting an error when specifying multiple packages with a comma separating them. Is there an example showing how multiple packages can be specified in the argument? > SparkR does not provide an easy way to depend on Spark Packages when > performing init from inside of R > - > > Key: SPARK-8506 > URL: https://issues.apache.org/jira/browse/SPARK-8506 > Project: Spark > Issue Type: Bug > Components: SparkR >Affects Versions: 1.4.0 >Reporter: holdenk >Assignee: holdenk >Priority: Minor > Fix For: 1.4.1, 1.5.0 > > > While packages can be specified when using the sparkR or sparkSubmit scripts, > the programming guide tells people to create their spark context using the R > shell + init. The init does have a parameter for jars but no parameter for > packages. Setting the SPARKR_SUBMIT_ARGS overwrites some necessary > information. I think a good solution would just be adding another field to > the init function to allow people to specify packages in the same way as jars. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-8506) SparkR does not provide an easy way to depend on Spark Packages when performing init from inside of R
[ https://issues.apache.org/jira/browse/SPARK-8506?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14595275#comment-14595275 ] Apache Spark commented on SPARK-8506: - User 'holdenk' has created a pull request for this issue: https://github.com/apache/spark/pull/6928 > SparkR does not provide an easy way to depend on Spark Packages when > performing init from inside of R > - > > Key: SPARK-8506 > URL: https://issues.apache.org/jira/browse/SPARK-8506 > Project: Spark > Issue Type: Bug > Components: SparkR >Affects Versions: 1.4.0 >Reporter: holdenk >Priority: Minor > > While packages can be specified when using the sparkR or sparkSubmit scripts, > the programming guide tells people to create their spark context using the R > shell + init. The init does have a parameter for jars but no parameter for > packages. Setting the SPARKR_SUBMIT_ARGS overwrites some necessary > information. I think a good solution would just be adding another field to > the init function to allow people to specify packages in the same way as jars. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-8506) SparkR does not provide an easy way to depend on Spark Packages when performing init from inside of R
[ https://issues.apache.org/jira/browse/SPARK-8506?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14594942#comment-14594942 ] Hao commented on SPARK-8506: This is a good solution. I just ran into the same problem when doing the tutorial. > SparkR does not provide an easy way to depend on Spark Packages when > performing init from inside of R > - > > Key: SPARK-8506 > URL: https://issues.apache.org/jira/browse/SPARK-8506 > Project: Spark > Issue Type: Bug > Components: SparkR >Affects Versions: 1.4.0 >Reporter: holdenk >Priority: Minor > > While packages can be specified when using the sparkR or sparkSubmit scripts, > the programming guide tells people to create their spark context using the R > shell + init. The init does have a parameter for jars but no parameter for > packages. Setting the SPARKR_SUBMIT_ARGS overwrites some necessary > information. I think a good solution would just be adding another field to > the init function to allow people to specify packages in the same way as jars. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-8506) SparkR does not provide an easy way to depend on Spark Packages when performing init from inside of R
[ https://issues.apache.org/jira/browse/SPARK-8506?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14594923#comment-14594923 ] holdenk commented on SPARK-8506: Sounds like a plan :) > SparkR does not provide an easy way to depend on Spark Packages when > performing init from inside of R > - > > Key: SPARK-8506 > URL: https://issues.apache.org/jira/browse/SPARK-8506 > Project: Spark > Issue Type: Bug > Components: SparkR >Affects Versions: 1.4.0 >Reporter: holdenk >Priority: Minor > > While packages can be specified when using the sparkR or sparkSubmit scripts, > the programming guide tells people to create their spark context using the R > shell + init. The init does have a parameter for jars but no parameter for > packages. Setting the SPARKR_SUBMIT_ARGS overwrites some necessary > information. I think a good solution would just be adding another field to > the init function to allow people to specify packages in the same way as jars. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-8506) SparkR does not provide an easy way to depend on Spark Packages when performing init from inside of R
[ https://issues.apache.org/jira/browse/SPARK-8506?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14594870#comment-14594870 ] Shivaram Venkataraman commented on SPARK-8506: -- Yeah sounds good to me. One minor thing is that I'd suggest calling the argument `sparkPackages` and take in a comma separated list (similar to jars) > SparkR does not provide an easy way to depend on Spark Packages when > performing init from inside of R > - > > Key: SPARK-8506 > URL: https://issues.apache.org/jira/browse/SPARK-8506 > Project: Spark > Issue Type: Bug > Components: SparkR >Affects Versions: 1.4.0 >Reporter: holdenk >Priority: Minor > > While packages can be specified when using the sparkR or sparkSubmit scripts, > the programming guide tells people to create their spark context using the R > shell + init. The init does have a parameter for jars but no parameter for > packages. Setting the SPARKR_SUBMIT_ARGS overwrites some necessary > information. I think a good solution would just be adding another field to > the init function to allow people to specify packages in the same way as jars. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-8506) SparkR does not provide an easy way to depend on Spark Packages when performing init from inside of R
[ https://issues.apache.org/jira/browse/SPARK-8506?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14594868#comment-14594868 ] holdenk commented on SPARK-8506: Thats what I thought would be a good solution. I can go ahead and implement if you thinks its a good idea too? > SparkR does not provide an easy way to depend on Spark Packages when > performing init from inside of R > - > > Key: SPARK-8506 > URL: https://issues.apache.org/jira/browse/SPARK-8506 > Project: Spark > Issue Type: Bug > Components: SparkR >Affects Versions: 1.4.0 >Reporter: holdenk >Priority: Minor > > While packages can be specified when using the sparkR or sparkSubmit scripts, > the programming guide tells people to create their spark context using the R > shell + init. The init does have a parameter for jars but no parameter for > packages. Setting the SPARKR_SUBMIT_ARGS overwrites some necessary > information. I think a good solution would just be adding another field to > the init function to allow people to specify packages in the same way as jars. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-8506) SparkR does not provide an easy way to depend on Spark Packages when performing init from inside of R
[ https://issues.apache.org/jira/browse/SPARK-8506?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14594854#comment-14594854 ] Shivaram Venkataraman commented on SPARK-8506: -- Thats a good point - I just ran into this yesterday and created a PR that improves documentation at https://github.com/apache/spark/pull/6916 Do you think adding a new argument to sparkR.init will be good enough ? i.e. something like sparkR.init(master="local", packages="com.databricks.spark-csv_2.10:1.0.3") > SparkR does not provide an easy way to depend on Spark Packages when > performing init from inside of R > - > > Key: SPARK-8506 > URL: https://issues.apache.org/jira/browse/SPARK-8506 > Project: Spark > Issue Type: Bug > Components: SparkR >Affects Versions: 1.4.0 >Reporter: holdenk >Priority: Minor > > While packages can be specified when using the sparkR or sparkSubmit scripts, > the programming guide tells people to create their spark context using the R > shell + init. The init does have a parameter for jars but no parameter for > packages. Setting the SPARKR_SUBMIT_ARGS overwrites some necessary > information. I think a good solution would just be adding another field to > the init function to allow people to specify packages in the same way as jars. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
