spark git commit: [SPARK-25746][SQL][FOLLOWUP] do not add unnecessary If expression

2018-10-31 Thread wenchen
Repository: spark Updated Branches: refs/heads/master cc82b9fed -> cd92f25be [SPARK-25746][SQL][FOLLOWUP] do not add unnecessary If expression ## What changes were proposed in this pull request? a followup of https://github.com/apache/spark/pull/22749. When we construct the new serializer

svn commit: r30564 - in /dev/spark/3.0.0-SNAPSHOT-2018_10_31_20_18-cc82b9f-docs: ./ _site/ _site/api/ _site/api/R/ _site/api/java/ _site/api/java/lib/ _site/api/java/org/ _site/api/java/org/apache/ _s

2018-10-31 Thread pwendell
Author: pwendell Date: Thu Nov 1 03:33:19 2018 New Revision: 30564 Log: Apache Spark 3.0.0-SNAPSHOT-2018_10_31_20_18-cc82b9f docs [This commit notification would consist of 1472 parts, which exceeds the limit of 50 ones, so it was shortened to the summary.]

spark git commit: [SPARK-25884][SQL] Add TBLPROPERTIES and COMMENT, and use LOCATION when SHOW CREATE TABLE.

2018-10-31 Thread wenchen
Repository: spark Updated Branches: refs/heads/master c9667aff4 -> cc82b9fed [SPARK-25884][SQL] Add TBLPROPERTIES and COMMENT, and use LOCATION when SHOW CREATE TABLE. ## What changes were proposed in this pull request? When `SHOW CREATE TABLE` for Datasource tables, we are missing

svn commit: r30562 - in /dev/spark/2.4.1-SNAPSHOT-2018_10_31_18_23-3d2fce5-docs: ./ _site/ _site/api/ _site/api/R/ _site/api/java/ _site/api/java/lib/ _site/api/java/org/ _site/api/java/org/apache/ _s

2018-10-31 Thread pwendell
Author: pwendell Date: Thu Nov 1 01:38:17 2018 New Revision: 30562 Log: Apache Spark 2.4.1-SNAPSHOT-2018_10_31_18_23-3d2fce5 docs [This commit notification would consist of 1477 parts, which exceeds the limit of 50 ones, so it was shortened to the summary.]

spark git commit: [SPARK-25672][SQL] schema_of_csv() - schema inference from an example

2018-10-31 Thread gurwls223
Repository: spark Updated Branches: refs/heads/master c5ef477d2 -> c9667aff4 [SPARK-25672][SQL] schema_of_csv() - schema inference from an example ## What changes were proposed in this pull request? In the PR, I propose to add new function - *schema_of_csv()* which infers schema of CSV

svn commit: r30561 - in /dev/spark/3.0.0-SNAPSHOT-2018_10_31_16_28-c5ef477-docs: ./ _site/ _site/api/ _site/api/R/ _site/api/java/ _site/api/java/lib/ _site/api/java/org/ _site/api/java/org/apache/ _s

2018-10-31 Thread pwendell
Author: pwendell Date: Wed Oct 31 23:42:41 2018 New Revision: 30561 Log: Apache Spark 3.0.0-SNAPSHOT-2018_10_31_16_28-c5ef477 docs [This commit notification would consist of 1472 parts, which exceeds the limit of 50 ones, so it was shortened to the summary.]

spark git commit: [INFRA] Close stale PR.

2018-10-31 Thread vanzin
Repository: spark Updated Branches: refs/heads/master 6be3cce75 -> c5ef477d2 [INFRA] Close stale PR. Closes #22860 Project: http://git-wip-us.apache.org/repos/asf/spark/repo Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/c5ef477d Tree:

spark git commit: [SPARK-25899][TESTS] Fix flaky CoarseGrainedSchedulerBackendSuite

2018-10-31 Thread lixiao
Repository: spark Updated Branches: refs/heads/branch-2.4 5cc2987db -> 3d2fce5a3 [SPARK-25899][TESTS] Fix flaky CoarseGrainedSchedulerBackendSuite ## What changes were proposed in this pull request? I saw CoarseGrainedSchedulerBackendSuite failed in my PR and finally reproduced the

spark git commit: [SPARK-25899][TESTS] Fix flaky CoarseGrainedSchedulerBackendSuite

2018-10-31 Thread lixiao
Repository: spark Updated Branches: refs/heads/master bc9f9b4d6 -> 6be3cce75 [SPARK-25899][TESTS] Fix flaky CoarseGrainedSchedulerBackendSuite ## What changes were proposed in this pull request? I saw CoarseGrainedSchedulerBackendSuite failed in my PR and finally reproduced the following

svn commit: r30555 - in /dev/spark/3.0.0-SNAPSHOT-2018_10_31_12_29-bc9f9b4-docs: ./ _site/ _site/api/ _site/api/R/ _site/api/java/ _site/api/java/lib/ _site/api/java/org/ _site/api/java/org/apache/ _s

2018-10-31 Thread pwendell
Author: pwendell Date: Wed Oct 31 19:43:34 2018 New Revision: 30555 Log: Apache Spark 3.0.0-SNAPSHOT-2018_10_31_12_29-bc9f9b4 docs [This commit notification would consist of 1472 parts, which exceeds the limit of 50 ones, so it was shortened to the summary.]

spark git commit: [SPARK-25860][SQL] Replace Literal(null, _) with FalseLiteral whenever possible

2018-10-31 Thread dbtsai
Repository: spark Updated Branches: refs/heads/master 68dde3481 -> bc9f9b4d6 [SPARK-25860][SQL] Replace Literal(null, _) with FalseLiteral whenever possible ## What changes were proposed in this pull request? This PR proposes a new optimization rule that replaces `Literal(null, _)` with

spark git commit: [SPARK-23781][CORE] Merge token renewer functionality into HadoopDelegationTokenManager.

2018-10-31 Thread irashid
Repository: spark Updated Branches: refs/heads/master af3b81607 -> 68dde3481 [SPARK-23781][CORE] Merge token renewer functionality into HadoopDelegationTokenManager. This avoids having two classes to deal with tokens; now the above class is a one-stop shop for dealing with delegation tokens.

spark git commit: [SPARK-25855][CORE] Don't use erasure coding for event logs by default

2018-10-31 Thread vanzin
Repository: spark Updated Branches: refs/heads/master e4cb42ad8 -> af3b81607 [SPARK-25855][CORE] Don't use erasure coding for event logs by default ## What changes were proposed in this pull request? This turns off hdfs erasure coding by default for event logs, regardless of filesystem

spark git commit: [SPARK-25891][PYTHON] Upgrade to Py4J 0.10.8.1

2018-10-31 Thread dongjoon
Repository: spark Updated Branches: refs/heads/master b3af917e7 -> e4cb42ad8 [SPARK-25891][PYTHON] Upgrade to Py4J 0.10.8.1 ## What changes were proposed in this pull request? Py4J 0.10.8.1 is released on October 21st and is the first release of Py4J to support Python 3.7 officially. We had

spark git commit: [SPARK-25893][SQL] Show a directional error message for unsupported Hive Metastore versions

2018-10-31 Thread dongjoon
Repository: spark Updated Branches: refs/heads/master 57eddc718 -> b3af917e7 [SPARK-25893][SQL] Show a directional error message for unsupported Hive Metastore versions ## What changes were proposed in this pull request? When `spark.sql.hive.metastore.version` is misconfigured, we had

svn commit: r30547 - in /dev/spark/3.0.0-SNAPSHOT-2018_10_31_08_15-57eddc7-docs: ./ _site/ _site/api/ _site/api/R/ _site/api/java/ _site/api/java/lib/ _site/api/java/org/ _site/api/java/org/apache/ _s

2018-10-31 Thread pwendell
Author: pwendell Date: Wed Oct 31 15:29:57 2018 New Revision: 30547 Log: Apache Spark 3.0.0-SNAPSHOT-2018_10_31_08_15-57eddc7 docs [This commit notification would consist of 1472 parts, which exceeds the limit of 50 ones, so it was shortened to the summary.]

spark git commit: [SPARK-25886][SQL][MINOR] Improve error message of `FailureSafeParser` and `from_avro` in FAILFAST mode

2018-10-31 Thread gurwls223
Repository: spark Updated Branches: refs/heads/master 3c0e9ce94 -> 57eddc718 [SPARK-25886][SQL][MINOR] Improve error message of `FailureSafeParser` and `from_avro` in FAILFAST mode ## What changes were proposed in this pull request? Currently in `FailureSafeParser` and `from_avro`, the

svn commit: r30546 - in /dev/spark/3.0.0-SNAPSHOT-2018_10_31_04_03-3c0e9ce-docs: ./ _site/ _site/api/ _site/api/R/ _site/api/java/ _site/api/java/lib/ _site/api/java/org/ _site/api/java/org/apache/ _s

2018-10-31 Thread pwendell
Author: pwendell Date: Wed Oct 31 11:18:16 2018 New Revision: 30546 Log: Apache Spark 3.0.0-SNAPSHOT-2018_10_31_04_03-3c0e9ce docs [This commit notification would consist of 1472 parts, which exceeds the limit of 50 ones, so it was shortened to the summary.]

spark git commit: [SPARK-24901][SQL] Merge the codegen of RegularHashMap and fastHashMap to reduce compiler maxCodesize when VectorizedHashMap is false.

2018-10-31 Thread wenchen
Repository: spark Updated Branches: refs/heads/master f8484e49e -> 3c0e9ce94 [SPARK-24901][SQL] Merge the codegen of RegularHashMap and fastHashMap to reduce compiler maxCodesize when VectorizedHashMap is false. ## What changes were proposed in this pull request? Currently, Generate code of

spark git commit: [SPARK-25663][SPARK-25661][SQL][TEST] Refactor BuiltInDataSourceWriteBenchmark, DataSourceWriteBenchmark and AvroWriteBenchmark to use main method

2018-10-31 Thread dongjoon
Repository: spark Updated Branches: refs/heads/master 34c3bc9f1 -> f8484e49e [SPARK-25663][SPARK-25661][SQL][TEST] Refactor BuiltInDataSourceWriteBenchmark, DataSourceWriteBenchmark and AvroWriteBenchmark to use main method ## What changes were proposed in this pull request? Refactor

spark git commit: [SPARK-25618][SQL][TEST] Reduce time taken to execute KafkaContinuousSourceStressForDontFailOnDataLossSuite

2018-10-31 Thread dongjoon
Repository: spark Updated Branches: refs/heads/master 0ad93b093 -> 34c3bc9f1 [SPARK-25618][SQL][TEST] Reduce time taken to execute KafkaContinuousSourceStressForDontFailOnDataLossSuite ## What changes were proposed in this pull request? In this test, i have reduced the test time to 20 secs

svn commit: r30542 - in /dev/spark/3.0.0-SNAPSHOT-2018_10_31_00_02-0ad93b0-docs: ./ _site/ _site/api/ _site/api/R/ _site/api/java/ _site/api/java/lib/ _site/api/java/org/ _site/api/java/org/apache/ _s

2018-10-31 Thread pwendell
Author: pwendell Date: Wed Oct 31 07:17:02 2018 New Revision: 30542 Log: Apache Spark 3.0.0-SNAPSHOT-2018_10_31_00_02-0ad93b0 docs [This commit notification would consist of 1472 parts, which exceeds the limit of 50 ones, so it was shortened to the summary.]

spark git commit: [SPARK-25883][SQL][MINOR] Override method `prettyName` in `from_avro`/`to_avro`

2018-10-31 Thread lixiao
Repository: spark Updated Branches: refs/heads/master 49bea5a7e -> 0ad93b093 [SPARK-25883][SQL][MINOR] Override method `prettyName` in `from_avro`/`to_avro` ## What changes were proposed in this pull request? Previously in from_avro/to_avro, we override the method `simpleString` and `sql`

spark git commit: [SPARK-25833][SQL][DOCS] Update migration guide for Hive view compatibility

2018-10-31 Thread dongjoon
Repository: spark Updated Branches: refs/heads/master 9cf9a83af -> 49bea5a7e [SPARK-25833][SQL][DOCS] Update migration guide for Hive view compatibility ## What changes were proposed in this pull request? Both Spark and Hive support views. However in some cases views created by Hive are not