[jira] [Commented] (SPARK-4890) Upgrade Boto to 2.34.0; automatically download Boto from PyPi instead of packaging it
[ https://issues.apache.org/jira/browse/SPARK-4890?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14256581#comment-14256581 ] Apache Spark commented on SPARK-4890: - User 'nchammas' has created a pull request for this issue: https://github.com/apache/spark/pull/3770 Upgrade Boto to 2.34.0; automatically download Boto from PyPi instead of packaging it - Key: SPARK-4890 URL: https://issues.apache.org/jira/browse/SPARK-4890 Project: Spark Issue Type: Improvement Components: EC2 Reporter: Josh Rosen Assignee: Josh Rosen Fix For: 1.3.0 We should upgrade to a newer version of Boto (2.34.0), since this is blocking several features. It looks like newer versions of Boto don't work properly when they're loaded from a zipfile since they try to read a JSON file from a path relative to the Boto library sources. Therefore, I think we should change {{spark-ec2}} to automatically download Boto from PyPi if it's not present in {{SPARK_EC2_DIR/lib}}, similar to what we do in the {{sbt/sbt}} scripts. This shouldn't ben an issue for users since they already need to have an internet connection to launch an EC2 cluster. By performing the downloading in {{spark_ec2.py}} instead of the Bash script, this should also work for Windows users. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-4890) Upgrade Boto to 2.34.0; automatically download Boto from PyPi instead of packaging it
[ https://issues.apache.org/jira/browse/SPARK-4890?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14252339#comment-14252339 ] Apache Spark commented on SPARK-4890: - User 'JoshRosen' has created a pull request for this issue: https://github.com/apache/spark/pull/3737 Upgrade Boto to 2.34.0; automatically download Boto from PyPi instead of packaging it - Key: SPARK-4890 URL: https://issues.apache.org/jira/browse/SPARK-4890 Project: Spark Issue Type: Improvement Components: EC2 Reporter: Josh Rosen Assignee: Josh Rosen We should upgrade to a newer version of Boto (2.34.0), since this is blocking several features. It looks like newer versions of Boto don't work properly when they're loaded from a zipfile since they try to read a JSON file from a path relative to the Boto library sources. Therefore, I think we should change {{spark-ec2}} to automatically download Boto from PyPi if it's not present in {{SPARK_EC2_DIR/lib}}, similar to what we do in the {{sbt/sbt}} scripts. This shouldn't ben an issue for users since they already need to have an internet connection to launch an EC2 cluster. By performing the downloading in {{spark_ec2.py}} instead of the Bash script, this should also work for Windows users. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org