[GitHub] [incubator-livy] runzhiwang commented on a change in pull request #196: [LIVY-606][LIVY-152] fix the invalid conf of pyspark.archives and sparkr.package
runzhiwang commented on a change in pull request #196: [LIVY-606][LIVY-152] fix
the invalid conf of pyspark.archives and sparkr.package
URL: https://github.com/apache/incubator-livy/pull/196#discussion_r315023241
##
File path:
server/src/main/scala/org/apache/livy/server/interactive/InteractiveSession.scala
##
@@ -183,7 +183,7 @@ object InteractiveSession extends Logging {
}
def findSparkRArchive(): Option[String] = {
- Option(livyConf.get(RSCConf.Entry.SPARKR_PACKAGE.key())).orElse {
Review comment:
@yiheng I read the SPARKR_PACKAGE from livy-client.conf instead of
livy.conf. The builderProperties has the config of livy-client.conf.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [incubator-livy] runzhiwang commented on a change in pull request #196: [LIVY-606][LIVY-152] fix the invalid conf of pyspark.archives and sparkr.package
runzhiwang commented on a change in pull request #196: [LIVY-606][LIVY-152] fix
the invalid conf of pyspark.archives and sparkr.package
URL: https://github.com/apache/incubator-livy/pull/196#discussion_r314978283
##
File path:
server/src/main/scala/org/apache/livy/server/interactive/InteractiveSession.scala
##
@@ -247,30 +247,26 @@ object InteractiveSession extends Logging {
}
def findPySparkArchives(): Seq[String] = {
- Option(livyConf.get(RSCConf.Entry.PYSPARK_ARCHIVES))
Review comment:
@yiheng I have updated the pr. Please review it again. Thanks.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [incubator-livy] runzhiwang commented on a change in pull request #196: [LIVY-606][LIVY-152] fix the invalid conf of pyspark.archives and sparkr.package
runzhiwang commented on a change in pull request #196: [LIVY-606][LIVY-152] fix the invalid conf of pyspark.archives and sparkr.package URL: https://github.com/apache/incubator-livy/pull/196#discussion_r314978258 ## File path: conf/livy-client.conf.template ## @@ -67,8 +67,6 @@ # livy.rsc.jars = # Location of the SparkR package for running sparkr # livy.rsc.sparkr.package = -# Location of the PySpark package for running pyspark -# livy.rsc.pyspark.archives = Review comment: @jerryshao Sorry, it's wrong to remove this configuration in livy-clent.conf. I have updated the pr. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [incubator-livy] runzhiwang commented on a change in pull request #196: [LIVY-606][LIVY-152] fix the invalid conf of pyspark.archives and sparkr.package
runzhiwang commented on a change in pull request #196: [LIVY-606][LIVY-152] fix the invalid conf of pyspark.archives and sparkr.package URL: https://github.com/apache/incubator-livy/pull/196#discussion_r314564884 ## File path: conf/livy-client.conf.template ## @@ -67,8 +67,6 @@ # livy.rsc.jars = # Location of the SparkR package for running sparkr # livy.rsc.sparkr.package = -# Location of the PySpark package for running pyspark -# livy.rsc.pyspark.archives = Review comment: @jerryshao Because spark will upload the pyspark.zip from local to hdfs, so it is useless to config it in livy-client.conf to avoid uploading . It is also reported in LIVY-152: Livy should use PYSPARK_ARCHIVES_PATH instead of having its own setting. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [incubator-livy] runzhiwang commented on a change in pull request #196: [LIVY-606][LIVY-152] fix the invalid conf of pyspark.archives and sparkr.package
runzhiwang commented on a change in pull request #196: [LIVY-606][LIVY-152] fix the invalid conf of pyspark.archives and sparkr.package URL: https://github.com/apache/incubator-livy/pull/196#discussion_r314564884 ## File path: conf/livy-client.conf.template ## @@ -67,8 +67,6 @@ # livy.rsc.jars = # Location of the SparkR package for running sparkr # livy.rsc.sparkr.package = -# Location of the PySpark package for running pyspark -# livy.rsc.pyspark.archives = Review comment: @jerryshao Because spark will upload the pyspark.zip from local to hdfs, so it is useless to config it in livy-client.conf to avoid uploading . It is also reported in LIVY-152: Livy should use PYSPARK_ARCHIVES_PATH instead of having its own setting. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
