svn commit: r26496 - in /dev/spark/2.4.0-SNAPSHOT-2018_04_24_20_01-5fea17b-docs: ./ _site/ _site/api/ _site/api/R/ _site/api/java/ _site/api/java/lib/ _site/api/java/org/ _site/api/java/org/apache/ _s

2018-04-24 Thread pwendell
Author: pwendell Date: Wed Apr 25 03:15:17 2018 New Revision: 26496 Log: Apache Spark 2.4.0-SNAPSHOT-2018_04_24_20_01-5fea17b docs [This commit notification would consist of 1460 parts, which exceeds the limit of 50 ones, so it was shortened to the summary.]

spark git commit: [SPARK-23821][SQL] Collection function: flatten

2018-04-24 Thread ueshin
Repository: spark Updated Branches: refs/heads/master d6c26d1c9 -> 5fea17b3b [SPARK-23821][SQL] Collection function: flatten ## What changes were proposed in this pull request? This PR adds a new collection function that transforms an array of arrays into a single array. The PR comprises: -

spark git commit: [SPARK-24038][SS] Refactor continuous writing to its own class

2018-04-24 Thread tdas
Repository: spark Updated Branches: refs/heads/master 7b1e6523a -> d6c26d1c9 [SPARK-24038][SS] Refactor continuous writing to its own class ## What changes were proposed in this pull request? Refactor continuous writing to its own class. See WIP https://github.com/jose-torres/spark/pull/13

svn commit: r26492 - in /dev/spark/2.4.0-SNAPSHOT-2018_04_24_16_01-7b1e652-docs: ./ _site/ _site/api/ _site/api/R/ _site/api/java/ _site/api/java/lib/ _site/api/java/org/ _site/api/java/org/apache/ _s

2018-04-24 Thread pwendell
Author: pwendell Date: Tue Apr 24 23:15:37 2018 New Revision: 26492 Log: Apache Spark 2.4.0-SNAPSHOT-2018_04_24_16_01-7b1e652 docs [This commit notification would consist of 1460 parts, which exceeds the limit of 50 ones, so it was shortened to the summary.]

spark git commit: [SPARK-24056][SS] Make consumer creation lazy in Kafka source for Structured streaming

2018-04-24 Thread tdas
Repository: spark Updated Branches: refs/heads/master 379bffa05 -> 7b1e6523a [SPARK-24056][SS] Make consumer creation lazy in Kafka source for Structured streaming ## What changes were proposed in this pull request? Currently, the driver side of the Kafka source (i.e. KafkaMicroBatchReader)

svn commit: r26486 - in /dev/spark/2.4.0-SNAPSHOT-2018_04_24_12_01-379bffa-docs: ./ _site/ _site/api/ _site/api/R/ _site/api/java/ _site/api/java/lib/ _site/api/java/org/ _site/api/java/org/apache/ _s

2018-04-24 Thread pwendell
Author: pwendell Date: Tue Apr 24 19:15:19 2018 New Revision: 26486 Log: Apache Spark 2.4.0-SNAPSHOT-2018_04_24_12_01-379bffa docs [This commit notification would consist of 1460 parts, which exceeds the limit of 50 ones, so it was shortened to the summary.]

spark git commit: [SPARK-23990][ML] Instruments logging improvements - ML regression package

2018-04-24 Thread jkbradley
Repository: spark Updated Branches: refs/heads/master 83013752e -> 379bffa05 [SPARK-23990][ML] Instruments logging improvements - ML regression package ## What changes were proposed in this pull request? Instruments logging improvements - ML regression package I add an `OptionalInstrument`

spark git commit: [SPARK-23455][ML] Default Params in ML should be saved separately in metadata

2018-04-24 Thread jkbradley
Repository: spark Updated Branches: refs/heads/master ce7ba2e98 -> 83013752e [SPARK-23455][ML] Default Params in ML should be saved separately in metadata ## What changes were proposed in this pull request? We save ML's user-supplied params and default params as one entity in metadata.

spark git commit: [SPARK-23807][BUILD] Add Hadoop 3.1 profile with relevant POM fix ups

2018-04-24 Thread vanzin
Repository: spark Updated Branches: refs/heads/master 2a24c481d -> ce7ba2e98 [SPARK-23807][BUILD] Add Hadoop 3.1 profile with relevant POM fix ups ## What changes were proposed in this pull request? 1. Adds a `hadoop-3.1` profile build depending on the hadoop-3.1 artifacts. 1. In the

spark git commit: [SPARK-23975][ML] Allow Clustering to take Arrays of Double as input features

2018-04-24 Thread jkbradley
Repository: spark Updated Branches: refs/heads/master 55c4ca88a -> 2a24c481d [SPARK-23975][ML] Allow Clustering to take Arrays of Double as input features ## What changes were proposed in this pull request? - Multiple possible input types is added in validateAndTransformSchema() and

spark git commit: [SPARK-22683][CORE] Add a executorAllocationRatio parameter to throttle the parallelism of the dynamic allocation

2018-04-24 Thread tgraves
Repository: spark Updated Branches: refs/heads/master 4926a7c2f -> 55c4ca88a [SPARK-22683][CORE] Add a executorAllocationRatio parameter to throttle the parallelism of the dynamic allocation ## What changes were proposed in this pull request? By default, the dynamic allocation will request

spark git commit: [SPARK-23589][SQL][FOLLOW-UP] Reuse InternalRow in ExternalMapToCatalyst eval

2018-04-24 Thread hvanhovell
Repository: spark Updated Branches: refs/heads/master 87e8a572b -> 4926a7c2f [SPARK-23589][SQL][FOLLOW-UP] Reuse InternalRow in ExternalMapToCatalyst eval ## What changes were proposed in this pull request? This pr is a follow-up of #20980 and fixes code to reuse `InternalRow` for converting

spark git commit: [SPARK-24054][R] Add array_position function / element_at functions

2018-04-24 Thread gurwls223
Repository: spark Updated Branches: refs/heads/master c303b1b67 -> 87e8a572b [SPARK-24054][R] Add array_position function / element_at functions ## What changes were proposed in this pull request? This PR proposes to add array_position and element_at in R side too. array_position: ```r df

spark git commit: [MINOR][DOCS] Fix comments of SQLExecution#withExecutionId

2018-04-24 Thread gurwls223
Repository: spark Updated Branches: refs/heads/branch-2.2 041aec4e1 -> e77d62a72 [MINOR][DOCS] Fix comments of SQLExecution#withExecutionId ## What changes were proposed in this pull request? Fix comment. Change `BroadcastHashJoin.broadcastFuture` to `BroadcastExchangeExec.relationFuture`:

spark git commit: [MINOR][DOCS] Fix comments of SQLExecution#withExecutionId

2018-04-24 Thread gurwls223
Repository: spark Updated Branches: refs/heads/branch-2.3 1c3e8205d -> 096defdd7 [MINOR][DOCS] Fix comments of SQLExecution#withExecutionId ## What changes were proposed in this pull request? Fix comment. Change `BroadcastHashJoin.broadcastFuture` to `BroadcastExchangeExec.relationFuture`:

spark git commit: [MINOR][DOCS] Fix comments of SQLExecution#withExecutionId

2018-04-24 Thread gurwls223
Repository: spark Updated Branches: refs/heads/master 281c1ca0d -> c303b1b67 [MINOR][DOCS] Fix comments of SQLExecution#withExecutionId ## What changes were proposed in this pull request? Fix comment. Change `BroadcastHashJoin.broadcastFuture` to `BroadcastExchangeExec.relationFuture`: