git commit: [SPARK-2652] [PySpark] donot use KyroSerializer as default serializer

Repository: spark Updated Branches: refs/heads/master 0aea22895 - 809c785bc [SPARK-2652] [PySpark] donot use KyroSerializer as default serializer KyroSerializer can not serialize customized class without registered explicitly, use it as default serializer in PySpark will introduce some

git commit: [SPARK-4051] [SQL] [PySpark] Convert Row into dictionary

Repository: spark Updated Branches: refs/heads/master d2987e8f7 - d60a9d440 [SPARK-4051] [SQL] [PySpark] Convert Row into dictionary Added a method to Row to turn row into dict: ``` row = Row(a=1) row.asDict() {'a': 1} ``` Author: Davies Liu dav...@databricks.com Closes #2896 from

git commit: [SPARK-2706][SQL] Enable Spark to support Hive 0.13

Repository: spark Updated Branches: refs/heads/master 0e886610e - 7c89a8f0c [SPARK-2706][SQL] Enable Spark to support Hive 0.13 Given that a lot of users are trying to use hive 0.13 in spark, and the incompatibility between hive-0.12 and hive-0.13 on the API level I want to propose

git commit: [SPARK-4026][Streaming] Write ahead log management

Repository: spark Updated Branches: refs/heads/master 7c89a8f0c - 6a40a7684 [SPARK-4026][Streaming] Write ahead log management As part of the effort to avoid data loss on Spark Streaming driver failure, we want to implement a write ahead log that can write received data to HDFS. This allows

git commit: [SPARK-2713] Executors of same application in same host should only download files jars once

Repository: spark Updated Branches: refs/heads/master 6a40a7684 - 7aacb7bfa [SPARK-2713] Executors of same application in same host should only download files jars once If Spark lunched multiple executors in one host for one application, every executor would download it dependent files and

git commit: [SPARK-4076] Parameter expansion in spark-config is wrong

Repository: spark Updated Branches: refs/heads/master 7aacb7bfa - 30ea2868e [SPARK-4076] Parameter expansion in spark-config is wrong In sbin/spark-config.sh, parameter expansion is used to extract source root as follows. this=${BASH_SOURCE-$0} I think, the parameter expansion should

git commit: [SPARK-4076] Parameter expansion in spark-config is wrong

Repository: spark Updated Branches: refs/heads/branch-1.1 926f8ca53 - 386fc46bc [SPARK-4076] Parameter expansion in spark-config is wrong In sbin/spark-config.sh, parameter expansion is used to extract source root as follows. this=${BASH_SOURCE-$0} I think, the parameter expansion

git commit: [SPARK-4075] [Deploy] Jar url validation is not enough for Jar file

Repository: spark Updated Branches: refs/heads/master 30ea2868e - 098f83c7c [SPARK-4075] [Deploy] Jar url validation is not enough for Jar file In deploy.ClientArguments.isValidJarUrl, the url is checked as follows. def isValidJarUrl(s: String): Boolean = s.matches((.+):(.+)jar) So, it

git commit: [SPARK-4075] [Deploy] Jar url validation is not enough for Jar file

Repository: spark Updated Branches: refs/heads/branch-1.1 386fc46bc - 80dde80a6 [SPARK-4075] [Deploy] Jar url validation is not enough for Jar file In deploy.ClientArguments.isValidJarUrl, the url is checked as follows. def isValidJarUrl(s: String): Boolean = s.matches((.+):(.+)jar) So,

git commit: [SPARK-4013] Do not create multiple actor systems on each executor

Repository: spark Updated Branches: refs/heads/master 098f83c7c - b563987e8 [SPARK-4013] Do not create multiple actor systems on each executor In the existing code, each coarse-grained executor has two concurrently running actor systems. This causes many more error messages to be logged than

git commit: [SPARK-4006] In long running contexts, we encountered the situation of d...

Repository: spark Updated Branches: refs/heads/branch-1.1 80dde80a6 - 59297e951 [SPARK-4006] In long running contexts, we encountered the situation of d... ...ouble registe... ...r without a remove in between. The cause for that is unknown, and assumed a temp network issue. However, since

git commit: [SPARK-4080] Only throw IOException from [write|read][Object|External]

Repository: spark Updated Branches: refs/heads/master 3a906c663 - 6c98c29ae [SPARK-4080] Only throw IOException from [write|read][Object|External] If classes implementing Serializable or Externalizable interfaces throw exceptions other than IOException or ClassNotFoundException from their

git commit: [SPARK-4080] Only throw IOException from [write|read][Object|External]

Repository: spark Updated Branches: refs/heads/branch-1.1 59297e951 - 6c10c2770 [SPARK-4080] Only throw IOException from [write|read][Object|External] If classes implementing Serializable or Externalizable interfaces throw exceptions other than IOException or ClassNotFoundException from their

git commit: [SPARK-4056] Upgrade snappy-java to 1.1.1.5

Repository: spark Updated Branches: refs/heads/branch-1.1 6c10c2770 - b7541ae89 [SPARK-4056] Upgrade snappy-java to 1.1.1.5 This upgrades snappy-java to 1.1.1.5, which improves error messages when attempting to deserialize empty inputs using SnappyInputStream (see

[01/51] [partial] [SQL] Update Hive test harness for Hive 12 and 13

Repository: spark Updated Branches: refs/heads/master 898b22ab1 - 3a845d3c0 http://git-wip-us.apache.org/repos/asf/spark/blob/3a845d3c/sql/hive/src/test/resources/golden/quote2-1-34f3c423b2fb1f0b11457f45a60042b9 -- diff --git

[17/51] [partial] [SQL] Update Hive test harness for Hive 12 and 13

http://git-wip-us.apache.org/repos/asf/spark/blob/3a845d3c/sql/hive/src/test/resources/golden/load_dyn_part3-8-c3b6f0cc860b84851dd60c53b36a6437 -- diff --git

[30/51] [partial] [SQL] Update Hive test harness for Hive 12 and 13

http://git-wip-us.apache.org/repos/asf/spark/blob/3a845d3c/sql/hive/src/test/resources/golden/groupby7_noskew_multi_single_reducer-1-85c4f90b754cd88147d6b74e17d22063 -- diff --git

[12/51] [partial] [SQL] Update Hive test harness for Hive 12 and 13

http://git-wip-us.apache.org/repos/asf/spark/blob/3a845d3c/sql/hive/src/test/resources/golden/load_dyn_part8-2-9e1df41acabef83f62464f52c2396c8a -- diff --git

[18/51] [partial] [SQL] Update Hive test harness for Hive 12 and 13

http://git-wip-us.apache.org/repos/asf/spark/blob/3a845d3c/sql/hive/src/test/resources/golden/load_dyn_part14-2-a4fb8359a2179ec70777aad6366071b7 -- diff --git

[25/51] [partial] [SQL] Update Hive test harness for Hive 12 and 13

http://git-wip-us.apache.org/repos/asf/spark/blob/3a845d3c/sql/hive/src/test/resources/golden/inputddl4-2-7fdf00ff5c22ae284728e0f035396865 -- diff --git

[50/51] [partial] [SQL] Update Hive test harness for Hive 12 and 13

http://git-wip-us.apache.org/repos/asf/spark/blob/3a845d3c/.rat-excludes -- diff --git a/.rat-excludes b/.rat-excludes index b14ad53..ae97456 100644 --- a/.rat-excludes +++ b/.rat-excludes @@ -48,6 +48,7 @@ sbt-launch-lib.bash

[32/51] [partial] [SQL] Update Hive test harness for Hive 12 and 13

http://git-wip-us.apache.org/repos/asf/spark/blob/3a845d3c/sql/hive/src/test/resources/golden/groupby1_limit-0-83c59d378571a6e487aa20217bd87817 -- diff --git

[07/51] [partial] [SQL] Update Hive test harness for Hive 12 and 13

http://git-wip-us.apache.org/repos/asf/spark/blob/3a845d3c/sql/hive/src/test/resources/golden/metadataonly1-2-9eadfd16be30c653625fce7b74048d9d -- diff --git

[34/51] [partial] [SQL] Update Hive test harness for Hive 12 and 13

http://git-wip-us.apache.org/repos/asf/spark/blob/3a845d3c/sql/hive/src/test/resources/golden/dynamic_partition_skip_default-9-3b57aa58995f862f2713624b50db5b65 -- diff --git

[36/51] [partial] [SQL] Update Hive test harness for Hive 12 and 13

http://git-wip-us.apache.org/repos/asf/spark/blob/3a845d3c/sql/hive/src/test/resources/golden/decimal_precision-15-5c49f041326bc5a9e936910094f190ce -- diff --git

[29/51] [partial] [SQL] Update Hive test harness for Hive 12 and 13

http://git-wip-us.apache.org/repos/asf/spark/blob/3a845d3c/sql/hive/src/test/resources/golden/groupby_multi_single_reducer2-6-8566e4c94d34b0fc4a71d34e89529c74 -- diff --git

[23/51] [partial] [SQL] Update Hive test harness for Hive 12 and 13

http://git-wip-us.apache.org/repos/asf/spark/blob/3a845d3c/sql/hive/src/test/resources/golden/join_array-2-16840a0266cad03a1a0b134d105b854f -- diff --git

[22/51] [partial] [SQL] Update Hive test harness for Hive 12 and 13

http://git-wip-us.apache.org/repos/asf/spark/blob/3a845d3c/sql/hive/src/test/resources/golden/join_filters-9-b6372cc006844e8488a3b7836c67daaa -- diff --git

[47/51] [partial] [SQL] Update Hive test harness for Hive 12 and 13

http://git-wip-us.apache.org/repos/asf/spark/blob/3a845d3c/sql/hive/src/test/resources/golden/alter_partition_format_loc-13-fe39b84ddc86b6bf042dc30c1b612321 -- diff --git

[27/51] [partial] [SQL] Update Hive test harness for Hive 12 and 13

http://git-wip-us.apache.org/repos/asf/spark/blob/3a845d3c/sql/hive/src/test/resources/golden/index_creation-20-68c5c98581c683b17ceaf1c0fd192871 -- diff --git

[05/51] [partial] [SQL] Update Hive test harness for Hive 12 and 13

http://git-wip-us.apache.org/repos/asf/spark/blob/3a845d3c/sql/hive/src/test/resources/golden/parallel-9-86a409d8b868dc5f1a3bd1e04c2bc28c -- diff --git

[35/51] [partial] [SQL] Update Hive test harness for Hive 12 and 13

http://git-wip-us.apache.org/repos/asf/spark/blob/3a845d3c/sql/hive/src/test/resources/golden/describe_syntax-23-2b2f2f068fe8e8fcbe10d11506804e49 -- diff --git

[02/51] [partial] [SQL] Update Hive test harness for Hive 12 and 13

http://git-wip-us.apache.org/repos/asf/spark/blob/3a845d3c/sql/hive/src/test/resources/golden/ppr_pushdown-13-2c316c67fd494d878fbbea107d283c3b -- diff --git

[04/51] [partial] [SQL] Update Hive test harness for Hive 12 and 13

http://git-wip-us.apache.org/repos/asf/spark/blob/3a845d3c/sql/hive/src/test/resources/golden/partition_wise_fileformat-1-9fad934993b77eef15a5d10eb203a378 -- diff --git

[03/51] [partial] [SQL] Update Hive test harness for Hive 12 and 13

http://git-wip-us.apache.org/repos/asf/spark/blob/3a845d3c/sql/hive/src/test/resources/golden/partition_wise_fileformat3-5-517aaa22478287fa80eef4a19f2cb9ff -- diff --git

[26/51] [partial] [SQL] Update Hive test harness for Hive 12 and 13

http://git-wip-us.apache.org/repos/asf/spark/blob/3a845d3c/sql/hive/src/test/resources/golden/input32-4-d0e1378a30e21e0198e47d9b668ee1f6 -- diff --git

[19/51] [partial] [SQL] Update Hive test harness for Hive 12 and 13

http://git-wip-us.apache.org/repos/asf/spark/blob/3a845d3c/sql/hive/src/test/resources/golden/load_dyn_part10-2-151ba0c3b8317902f1235ac07d58135e -- diff --git

[46/51] [partial] [SQL] Update Hive test harness for Hive 12 and 13

http://git-wip-us.apache.org/repos/asf/spark/blob/3a845d3c/sql/hive/src/test/resources/golden/annotate_stats_join-19-dea5f8f5c703583c7a3bdcb62cd3d589 -- diff --git

[09/51] [partial] [SQL] Update Hive test harness for Hive 12 and 13

http://git-wip-us.apache.org/repos/asf/spark/blob/3a845d3c/sql/hive/src/test/resources/golden/mapjoin_test_outer-12-80993ab7f757001e1f058bf8609f8420 -- diff --git

[16/51] [partial] [SQL] Update Hive test harness for Hive 12 and 13

http://git-wip-us.apache.org/repos/asf/spark/blob/3a845d3c/sql/hive/src/test/resources/golden/load_dyn_part4-10-297cf42ec45b5aa78b80d9c35a79555a -- diff --git

[37/51] [partial] [SQL] Update Hive test harness for Hive 12 and 13

http://git-wip-us.apache.org/repos/asf/spark/blob/3a845d3c/sql/hive/src/test/resources/golden/database_drop-3-f21dd011aeb417043ed58c03fd5c3bf0 -- diff --git

[11/51] [partial] [SQL] Update Hive test harness for Hive 12 and 13

http://git-wip-us.apache.org/repos/asf/spark/blob/3a845d3c/sql/hive/src/test/resources/golden/load_dyn_part8-9-a7456fceb35f62a282db750384f480db -- diff --git

[42/51] [partial] [SQL] Update Hive test harness for Hive 12 and 13

http://git-wip-us.apache.org/repos/asf/spark/blob/3a845d3c/sql/hive/src/test/resources/golden/auto_sortmerge_join_7-18-cda81d86d127fca0e2fbc2161e91400d -- diff --git

[08/51] [partial] [SQL] Update Hive test harness for Hive 12 and 13

http://git-wip-us.apache.org/repos/asf/spark/blob/3a845d3c/sql/hive/src/test/resources/golden/merge_dynamic_partition-13-16adcdb0e324ad233769e124b5b349da -- diff --git

[45/51] [partial] [SQL] Update Hive test harness for Hive 12 and 13

http://git-wip-us.apache.org/repos/asf/spark/blob/3a845d3c/sql/hive/src/test/resources/golden/auto_join0-0-ce1ef910fff98f174931cc641f7cef3a -- diff --git

[48/51] [partial] [SQL] Update Hive test harness for Hive 12 and 13

http://git-wip-us.apache.org/repos/asf/spark/blob/3a845d3c/sql/hive/src/test/resources/golden/alter3-27-54ad133b447f67c6d1ed7d4c43803a87 -- diff --git

[49/51] [partial] [SQL] Update Hive test harness for Hive 12 and 13

http://git-wip-us.apache.org/repos/asf/spark/blob/3a845d3c/sql/hive/src/test/resources/golden/1Y + 1.0-0-3ad5e3db0d0300312d33231e7c2a6c8d -- diff --git a/sql/hive/src/test/resources/golden/1Y +

[14/51] [partial] [SQL] Update Hive test harness for Hive 12 and 13

http://git-wip-us.apache.org/repos/asf/spark/blob/3a845d3c/sql/hive/src/test/resources/golden/load_dyn_part4-11-1313644c9dad948bfcebd7386c309ab7 -- diff --git

[21/51] [partial] [SQL] Update Hive test harness for Hive 12 and 13

http://git-wip-us.apache.org/repos/asf/spark/blob/3a845d3c/sql/hive/src/test/resources/golden/join_thrift-2-d4bfb7bab262dc2898431680711dec1b -- diff --git

[24/51] [partial] [SQL] Update Hive test harness for Hive 12 and 13

http://git-wip-us.apache.org/repos/asf/spark/blob/3a845d3c/sql/hive/src/test/resources/golden/join1-4-35c7611180562dcd9dab834f41654095 -- diff --git a/sql/hive/src/test/resources/golden/join1-4-35c7611180562dcd9dab834f41654095

[06/51] [partial] [SQL] Update Hive test harness for Hive 12 and 13

http://git-wip-us.apache.org/repos/asf/spark/blob/3a845d3c/sql/hive/src/test/resources/golden/nonreserved_keywords_insert_into1-11-6f59e35684a552a855e4dc3aee667092 -- diff --git

[13/51] [partial] [SQL] Update Hive test harness for Hive 12 and 13

http://git-wip-us.apache.org/repos/asf/spark/blob/3a845d3c/sql/hive/src/test/resources/golden/load_dyn_part4-11-24618a43c4656b72f81683b45708045e -- diff --git

[28/51] [partial] [SQL] Update Hive test harness for Hive 12 and 13

http://git-wip-us.apache.org/repos/asf/spark/blob/3a845d3c/sql/hive/src/test/resources/golden/groupby_sort_4-14-75d59344b6114c0bb20d5eac301c2170 -- diff --git

[31/51] [partial] [SQL] Update Hive test harness for Hive 12 and 13

http://git-wip-us.apache.org/repos/asf/spark/blob/3a845d3c/sql/hive/src/test/resources/golden/groupby6_map-6-adc1ec67836b26b60d8547c4996bfd8f -- diff --git