svn commit: r28565 - in /dev/spark/2.4.0-SNAPSHOT-2018_08_04_20_02-327bb30-docs: ./ _site/ _site/api/ _site/api/R/ _site/api/java/ _site/api/java/lib/ _site/api/java/org/ _site/api/java/org/apache/ _s

2018-08-04 Thread pwendell
Author: pwendell Date: Sun Aug 5 03:15:58 2018 New Revision: 28565 Log: Apache Spark 2.4.0-SNAPSHOT-2018_08_04_20_02-327bb30 docs [This commit notification would consist of 1471 parts, which exceeds the limit of 50 ones, so it was shortened to the summary.]

spark git commit: [SPARK-23911][SQL] Add aggregate function.

2018-08-04 Thread ueshin
Repository: spark Updated Branches: refs/heads/master 5f9633dc9 -> 327bb3007 [SPARK-23911][SQL] Add aggregate function. ## What changes were proposed in this pull request? This pr adds `aggregate` function which applies a binary operator to an initial state and all elements in the array,

svn commit: r28564 - in /dev/spark/2.4.0-SNAPSHOT-2018_08_04_16_02-5f9633d-docs: ./ _site/ _site/api/ _site/api/R/ _site/api/java/ _site/api/java/lib/ _site/api/java/org/ _site/api/java/org/apache/ _s

2018-08-04 Thread pwendell
Author: pwendell Date: Sat Aug 4 23:16:11 2018 New Revision: 28564 Log: Apache Spark 2.4.0-SNAPSHOT-2018_08_04_16_02-5f9633d docs [This commit notification would consist of 1471 parts, which exceeds the limit of 50 ones, so it was shortened to the summary.]

svn commit: r28563 - in /dev/spark/2.3.3-SNAPSHOT-2018_08_04_14_01-136588e-docs: ./ _site/ _site/api/ _site/api/R/ _site/api/java/ _site/api/java/lib/ _site/api/java/org/ _site/api/java/org/apache/ _s

2018-08-04 Thread pwendell
Author: pwendell Date: Sat Aug 4 21:15:17 2018 New Revision: 28563 Log: Apache Spark 2.3.3-SNAPSHOT-2018_08_04_14_01-136588e docs [This commit notification would consist of 1443 parts, which exceeds the limit of 50 ones, so it was shortened to the summary.]

spark git commit: [SPARK-25015][BUILD] Update Hadoop 2.7 to 2.7.7

2018-08-04 Thread srowen
Repository: spark Updated Branches: refs/heads/master b7fdf8eb2 -> 5f9633dc9 [SPARK-25015][BUILD] Update Hadoop 2.7 to 2.7.7 ## What changes were proposed in this pull request? Update Hadoop 2.7 to 2.7.7 to pull in bug and security fixes. ## How was this patch tested? Existing tests.

spark git commit: [SPARK-24987][SS] - Fix Kafka consumer leak when no new offsets for TopicPartition

2018-08-04 Thread koeninger
Repository: spark Updated Branches: refs/heads/branch-2.3 8080c937d -> 14b50d7fe [SPARK-24987][SS] - Fix Kafka consumer leak when no new offsets for TopicPartition ## What changes were proposed in this pull request? This small fix adds a `consumer.release()` call to `KafkaSourceRDD` in the

spark git commit: [SPARK-24987][SS] - Fix Kafka consumer leak when no new offsets for TopicPartition

2018-08-04 Thread koeninger
Repository: spark Updated Branches: refs/heads/master 55e3ae693 -> b7fdf8eb2 [SPARK-24987][SS] - Fix Kafka consumer leak when no new offsets for TopicPartition ## What changes were proposed in this pull request? This small fix adds a `consumer.release()` call to `KafkaSourceRDD` in the case

svn commit: r28562 - in /dev/spark/2.4.0-SNAPSHOT-2018_08_04_12_02-55e3ae6-docs: ./ _site/ _site/api/ _site/api/R/ _site/api/java/ _site/api/java/lib/ _site/api/java/org/ _site/api/java/org/apache/ _s

2018-08-04 Thread pwendell
Author: pwendell Date: Sat Aug 4 19:16:22 2018 New Revision: 28562 Log: Apache Spark 2.4.0-SNAPSHOT-2018_08_04_12_02-55e3ae6 docs [This commit notification would consist of 1471 parts, which exceeds the limit of 50 ones, so it was shortened to the summary.]

spark git commit: [SPARK-25001][BUILD] Fix miscellaneous build warnings

2018-08-04 Thread srowen
Repository: spark Updated Branches: refs/heads/master 70462f291 -> 55e3ae693 [SPARK-25001][BUILD] Fix miscellaneous build warnings ## What changes were proposed in this pull request? There are many warnings in the current build (for instance see

spark git commit: [SPARK-24926][CORE] Ensure numCores is used consistently in all netty configurations

2018-08-04 Thread srowen
Repository: spark Updated Branches: refs/heads/master 684c719cc -> 70462f291 [SPARK-24926][CORE] Ensure numCores is used consistently in all netty configurations ## What changes were proposed in this pull request? Netty could just ignore user-provided configurations. In particular,

svn commit: r28555 - in /dev/spark/2.4.0-SNAPSHOT-2018_08_04_04_02-684c719-docs: ./ _site/ _site/api/ _site/api/R/ _site/api/java/ _site/api/java/lib/ _site/api/java/org/ _site/api/java/org/apache/ _s

2018-08-04 Thread pwendell
Author: pwendell Date: Sat Aug 4 11:20:45 2018 New Revision: 28555 Log: Apache Spark 2.4.0-SNAPSHOT-2018_08_04_04_02-684c719 docs [This commit notification would consist of 1471 parts, which exceeds the limit of 50 ones, so it was shortened to the summary.]

spark git commit: [SPARK-23915][SQL][FOLLOWUP] Add array_except function

2018-08-04 Thread ueshin
Repository: spark Updated Branches: refs/heads/master 0ecc132d6 -> 684c719cc [SPARK-23915][SQL][FOLLOWUP] Add array_except function ## What changes were proposed in this pull request? simplify the codegen: 1. only do real codegen if the type can be specialized by the hash set 2. change the

svn commit: r28554 - in /dev/spark/2.4.0-SNAPSHOT-2018_08_04_00_01-36ea55e-docs: ./ _site/ _site/api/ _site/api/R/ _site/api/java/ _site/api/java/lib/ _site/api/java/org/ _site/api/java/org/apache/ _s

2018-08-04 Thread pwendell
Author: pwendell Date: Sat Aug 4 07:16:08 2018 New Revision: 28554 Log: Apache Spark 2.4.0-SNAPSHOT-2018_08_04_00_01-36ea55e docs [This commit notification would consist of 1471 parts, which exceeds the limit of 50 ones, so it was shortened to the summary.]

spark git commit: [SPARK-23909][SQL] Add filter function.

2018-08-04 Thread ueshin
Repository: spark Updated Branches: refs/heads/master 36ea55e97 -> 0ecc132d6 [SPARK-23909][SQL] Add filter function. ## What changes were proposed in this pull request? This pr adds `filter` function which filters the input array using the given predicate. ```sql > SELECT filter(array(1,

spark git commit: [SPARK-24940][SQL] Coalesce and Repartition Hint for SQL Queries

2018-08-04 Thread lixiao
Repository: spark Updated Branches: refs/heads/master 41c2227a2 -> 36ea55e97 [SPARK-24940][SQL] Coalesce and Repartition Hint for SQL Queries ## What changes were proposed in this pull request? Many Spark SQL users in my company have asked for a way to control the number of output files in

spark git commit: [SPARK-24722][SQL] pivot() with Column type argument

2018-08-04 Thread gurwls223
Repository: spark Updated Branches: refs/heads/master 4c27663cb -> 41c2227a2 [SPARK-24722][SQL] pivot() with Column type argument ## What changes were proposed in this pull request? In the PR, I propose column-based API for the `pivot()` function. It allows using of any column expressions