spark git commit: [SPARK-8821] [EC2] Switched to binary mode for file reading
Repository: spark Updated Branches: refs/heads/master 738c10748 - 70beb808e [SPARK-8821] [EC2] Switched to binary mode for file reading Otherwise the script will crash with - Downloading boto... Traceback (most recent call last): File ec2/spark_ec2.py, line 148, in module setup_external_libs(external_libs) File ec2/spark_ec2.py, line 128, in setup_external_libs if hashlib.md5(tar.read()).hexdigest() != lib[md5]: File /usr/lib/python3.4/codecs.py, line 319, in decode (result, consumed) = self._buffer_decode(data, self.errors, final) UnicodeDecodeError: 'utf-8' codec can't decode byte 0x8b in position 1: invalid start byte In case of an utf8 env setting. Author: Simon Hafner hafnersi...@gmail.com Closes #7215 from reactormonk/branch-1.4 and squashes the following commits: e86957a [Simon Hafner] [SPARK-8821] [EC2] Switched to binary mode (cherry picked from commit 83a621a5a8f8a2991c4cfa687279589e5c623d46) Signed-off-by: Shivaram Venkataraman shiva...@cs.berkeley.edu Project: http://git-wip-us.apache.org/repos/asf/spark/repo Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/70beb808 Tree: http://git-wip-us.apache.org/repos/asf/spark/tree/70beb808 Diff: http://git-wip-us.apache.org/repos/asf/spark/diff/70beb808 Branch: refs/heads/master Commit: 70beb808e13f6371968ac87f7cf625ed110375e6 Parents: 738c107 Author: Simon Hafner hafnersi...@gmail.com Authored: Tue Jul 7 09:42:59 2015 -0700 Committer: Shivaram Venkataraman shiva...@cs.berkeley.edu Committed: Tue Jul 7 09:43:16 2015 -0700 -- ec2/spark_ec2.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) -- http://git-wip-us.apache.org/repos/asf/spark/blob/70beb808/ec2/spark_ec2.py -- diff --git a/ec2/spark_ec2.py b/ec2/spark_ec2.py index 18ccbc0..8582d43 100755 --- a/ec2/spark_ec2.py +++ b/ec2/spark_ec2.py @@ -125,7 +125,7 @@ def setup_external_libs(libs): ) with open(tgz_file_path, wb) as tgz_file: tgz_file.write(download_stream.read()) -with open(tgz_file_path) as tar: +with open(tgz_file_path, rb) as tar: if hashlib.md5(tar.read()).hexdigest() != lib[md5]: print(ERROR: Got wrong md5sum for {lib}..format(lib=lib[name]), file=stderr) sys.exit(1) - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
spark git commit: [SPARK-8821] [EC2] Switched to binary mode for file reading
Repository: spark Updated Branches: refs/heads/branch-1.4 bf8b47d17 - 83a621a5a [SPARK-8821] [EC2] Switched to binary mode for file reading Otherwise the script will crash with - Downloading boto... Traceback (most recent call last): File ec2/spark_ec2.py, line 148, in module setup_external_libs(external_libs) File ec2/spark_ec2.py, line 128, in setup_external_libs if hashlib.md5(tar.read()).hexdigest() != lib[md5]: File /usr/lib/python3.4/codecs.py, line 319, in decode (result, consumed) = self._buffer_decode(data, self.errors, final) UnicodeDecodeError: 'utf-8' codec can't decode byte 0x8b in position 1: invalid start byte In case of an utf8 env setting. Author: Simon Hafner hafnersi...@gmail.com Closes #7215 from reactormonk/branch-1.4 and squashes the following commits: e86957a [Simon Hafner] [SPARK-8821] [EC2] Switched to binary mode Project: http://git-wip-us.apache.org/repos/asf/spark/repo Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/83a621a5 Tree: http://git-wip-us.apache.org/repos/asf/spark/tree/83a621a5 Diff: http://git-wip-us.apache.org/repos/asf/spark/diff/83a621a5 Branch: refs/heads/branch-1.4 Commit: 83a621a5a8f8a2991c4cfa687279589e5c623d46 Parents: bf8b47d Author: Simon Hafner hafnersi...@gmail.com Authored: Tue Jul 7 09:42:59 2015 -0700 Committer: Shivaram Venkataraman shiva...@cs.berkeley.edu Committed: Tue Jul 7 09:42:59 2015 -0700 -- ec2/spark_ec2.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) -- http://git-wip-us.apache.org/repos/asf/spark/blob/83a621a5/ec2/spark_ec2.py -- diff --git a/ec2/spark_ec2.py b/ec2/spark_ec2.py index 05fa47f..91f0a24 100755 --- a/ec2/spark_ec2.py +++ b/ec2/spark_ec2.py @@ -127,7 +127,7 @@ def setup_external_libs(libs): ) with open(tgz_file_path, wb) as tgz_file: tgz_file.write(download_stream.read()) -with open(tgz_file_path) as tar: +with open(tgz_file_path, rb) as tar: if hashlib.md5(tar.read()).hexdigest() != lib[md5]: print(ERROR: Got wrong md5sum for {lib}..format(lib=lib[name]), file=stderr) sys.exit(1) - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org