spark git commit: [SPARK-23914][SQL] Add array_union function

Repository: spark Updated Branches: refs/heads/master 5ad4735bd -> 301bff706 [SPARK-23914][SQL] Add array_union function ## What changes were proposed in this pull request? The PR adds the SQL function `array_union`. The behavior of the function is based on Presto's one. This function retur

svn commit: r28074 - in /dev/spark/2.4.0-SNAPSHOT-2018_07_12_04_01-301bff7-docs: ./ _site/ _site/api/ _site/api/R/ _site/api/java/ _site/api/java/lib/ _site/api/java/org/ _site/api/java/org/apache/ _s

Author: pwendell Date: Thu Jul 12 11:16:20 2018 New Revision: 28074 Log: Apache Spark 2.4.0-SNAPSHOT-2018_07_12_04_01-301bff7 docs [This commit notification would consist of 1467 parts, which exceeds the limit of 50 ones, so it was shortened to the summary.] ---

spark git commit: [SPARK-24691][SQL] Dispatch the type support check in FileFormat implementation

Repository: spark Updated Branches: refs/heads/master 301bff706 -> e6c6f90a5 [SPARK-24691][SQL] Dispatch the type support check in FileFormat implementation ## What changes were proposed in this pull request? With https://github.com/apache/spark/pull/21389, data source schema is validated o

spark git commit: [SPARK-20168][STREAMING KINESIS] Setting the timestamp directly would cause exception on …

Repository: spark Updated Branches: refs/heads/master e6c6f90a5 -> 9fa4a1ed3 [SPARK-20168][STREAMING KINESIS] Setting the timestamp directly would cause exception on … Setting the timestamp directly would cause exception on reading stream, it can be set directly only if the mode is not AT_

svn commit: r28079 - in /dev/spark/2.4.0-SNAPSHOT-2018_07_12_12_01-9fa4a1e-docs: ./ _site/ _site/api/ _site/api/R/ _site/api/java/ _site/api/java/lib/ _site/api/java/org/ _site/api/java/org/apache/ _s

Author: pwendell Date: Thu Jul 12 19:17:14 2018 New Revision: 28079 Log: Apache Spark 2.4.0-SNAPSHOT-2018_07_12_12_01-9fa4a1e docs [This commit notification would consist of 1467 parts, which exceeds the limit of 50 ones, so it was shortened to the summary.] ---

spark git commit: [SPARK-24610] fix reading small files via wholeTextFiles

Repository: spark Updated Branches: refs/heads/master 9fa4a1ed3 -> 1055c94cd [SPARK-24610] fix reading small files via wholeTextFiles ## What changes were proposed in this pull request? The `WholeTextFileInputFormat` determines the `maxSplitSize` for the file/s being read using the `wholeText

[1/2] spark git commit: [SPARK-24768][SQL] Have a built-in AVRO data source implementation

Repository: spark Updated Branches: refs/heads/master 1055c94cd -> 395860a98 http://git-wip-us.apache.org/repos/asf/spark/blob/395860a9/external/avro/src/test/scala/org/apache/spark/sql/avro/AvroSuite.scala -- diff --git a/ext

[2/2] spark git commit: [SPARK-24768][SQL] Have a built-in AVRO data source implementation

[SPARK-24768][SQL] Have a built-in AVRO data source implementation ## What changes were proposed in this pull request? Apache Avro (https://avro.apache.org) is a popular data serialization format. It is widely used in the Spark and Hadoop ecosystem, especially for Kafka-based data pipelines. U

spark git commit: [SPARK-23007][SQL][TEST] Add read schema suite for file-based data sources

Repository: spark Updated Branches: refs/heads/master 395860a98 -> 07704c971 [SPARK-23007][SQL][TEST] Add read schema suite for file-based data sources ## What changes were proposed in this pull request? The reader schema is said to be evolved (or projected) when it changed after the data is

spark git commit: [SPARK-24208][SQL][FOLLOWUP] Move test cases to proper locations

Repository: spark Updated Branches: refs/heads/master 07704c971 -> 11384893b [SPARK-24208][SQL][FOLLOWUP] Move test cases to proper locations ## What changes were proposed in this pull request? The PR is a followup to move the test cases introduced by the original PR in their proper location

svn commit: r28081 - in /dev/spark/2.4.0-SNAPSHOT-2018_07_12_16_01-1138489-docs: ./ _site/ _site/api/ _site/api/R/ _site/api/java/ _site/api/java/lib/ _site/api/java/org/ _site/api/java/org/apache/ _s

Author: pwendell Date: Thu Jul 12 23:15:53 2018 New Revision: 28081 Log: Apache Spark 2.4.0-SNAPSHOT-2018_07_12_16_01-1138489 docs [This commit notification would consist of 1467 parts, which exceeds the limit of 50 ones, so it was shortened to the summary.] ---

spark git commit: [SPARK-24790][SQL] Allow complex aggregate expressions in Pivot

Repository: spark Updated Branches: refs/heads/master 11384893b -> 75725057b [SPARK-24790][SQL] Allow complex aggregate expressions in Pivot ## What changes were proposed in this pull request? Relax the check to allow complex aggregate expressions, like `ceil(sum(col1))` or `sum(col1) + 1`,

spark git commit: [SPARK-24537][R] Add array_remove / array_zip / map_from_arrays / array_distinct

Repository: spark Updated Branches: refs/heads/master 75725057b -> e0f4f206b [SPARK-24537][R] Add array_remove / array_zip / map_from_arrays / array_distinct ## What changes were proposed in this pull request? Add array_remove / array_zip / map_from_arrays / array_distinct functions in SparkR

svn commit: r28083 - in /dev/spark/2.4.0-SNAPSHOT-2018_07_12_20_01-e0f4f20-docs: ./ _site/ _site/api/ _site/api/R/ _site/api/java/ _site/api/java/lib/ _site/api/java/org/ _site/api/java/org/apache/ _s

Author: pwendell Date: Fri Jul 13 03:15:53 2018 New Revision: 28083 Log: Apache Spark 2.4.0-SNAPSHOT-2018_07_12_20_01-e0f4f20 docs [This commit notification would consist of 1467 parts, which exceeds the limit of 50 ones, so it was shortened to the summary.] ---

spark git commit: [SPARK-23486] cache the function name from the external catalog for lookupFunctions

Repository: spark Updated Branches: refs/heads/master e0f4f206b -> 0ce11d0e3 [SPARK-23486] cache the function name from the external catalog for lookupFunctions ## What changes were proposed in this pull request? This PR will cache the function name from external catalog, it is used by look