svn commit: r24634 - in /dev/spark/2.3.1-SNAPSHOT-2018_02_01_22_01-2b07452-docs: ./ _site/ _site/api/ _site/api/R/ _site/api/java/ _site/api/java/lib/ _site/api/java/org/ _site/api/java/org/apache/ _s

Author: pwendell Date: Fri Feb 2 06:16:31 2018 New Revision: 24634 Log: Apache Spark 2.3.1-SNAPSHOT-2018_02_01_22_01-2b07452 docs [This commit notification would consist of 1442 parts, which exceeds the limit of 50 ones, so it was shortened to the summary.]

spark git commit: [SPARK-23301][SQL] data source column pruning should work for arbitrary expressions

Repository: spark Updated Branches: refs/heads/branch-2.3 7baae3aef -> 2b07452ca [SPARK-23301][SQL] data source column pruning should work for arbitrary expressions This PR fixes a mistake in the `PushDownOperatorsToDataSource` rule, the column pruning logic is incorrect about `Project`. a

spark git commit: [SPARK-23301][SQL] data source column pruning should work for arbitrary expressions

Repository: spark Updated Branches: refs/heads/master b3a04283f -> 19c7c7ebd [SPARK-23301][SQL] data source column pruning should work for arbitrary expressions ## What changes were proposed in this pull request? This PR fixes a mistake in the `PushDownOperatorsToDataSource` rule, the

spark git commit: [SPARK-23306] Fix the oom caused by contention

Repository: spark Updated Branches: refs/heads/master 969eda4a0 -> b3a04283f [SPARK-23306] Fix the oom caused by contention ## What changes were proposed in this pull request? here is race condition in TaskMemoryManger, which may cause OOM. The memory released may be taken by another task

svn commit: r24631 - in /dev/spark/2.4.0-SNAPSHOT-2018_02_01_20_01-969eda4-docs: ./ _site/ _site/api/ _site/api/R/ _site/api/java/ _site/api/java/lib/ _site/api/java/org/ _site/api/java/org/apache/ _s

Author: pwendell Date: Fri Feb 2 04:15:33 2018 New Revision: 24631 Log: Apache Spark 2.4.0-SNAPSHOT-2018_02_01_20_01-969eda4 docs [This commit notification would consist of 1444 parts, which exceeds the limit of 50 ones, so it was shortened to the summary.]

spark git commit: [SPARK-23020][CORE] Fix another race in the in-process launcher test.

Repository: spark Updated Branches: refs/heads/master 90848d507 -> 969eda4a0 [SPARK-23020][CORE] Fix another race in the in-process launcher test. First the bad news: there's an unfixable race in the launcher code. (By unfixable I mean it would take a lot more effort than this change to fix

spark git commit: [SPARK-23284][SQL] Document the behavior of several ColumnVector's get APIs when accessing null slot

Repository: spark Updated Branches: refs/heads/branch-2.3 ab23785c7 -> 7baae3aef [SPARK-23284][SQL] Document the behavior of several ColumnVector's get APIs when accessing null slot ## What changes were proposed in this pull request? For some ColumnVector get APIs such as getDecimal,

spark git commit: [SPARK-23284][SQL] Document the behavior of several ColumnVector's get APIs when accessing null slot

Repository: spark Updated Branches: refs/heads/master 032c11b83 -> 90848d507 [SPARK-23284][SQL] Document the behavior of several ColumnVector's get APIs when accessing null slot ## What changes were proposed in this pull request? For some ColumnVector get APIs such as getDecimal, getBinary,

svn commit: r24630 - in /dev/spark/2.3.1-SNAPSHOT-2018_02_01_18_01-ab23785-docs: ./ _site/ _site/api/ _site/api/R/ _site/api/java/ _site/api/java/lib/ _site/api/java/org/ _site/api/java/org/apache/ _s

Author: pwendell Date: Fri Feb 2 02:15:06 2018 New Revision: 24630 Log: Apache Spark 2.3.1-SNAPSHOT-2018_02_01_18_01-ab23785 docs [This commit notification would consist of 1442 parts, which exceeds the limit of 50 ones, so it was shortened to the summary.]

svn commit: r24627 - in /dev/spark/2.4.0-SNAPSHOT-2018_02_01_16_01-032c11b-docs: ./ _site/ _site/api/ _site/api/R/ _site/api/java/ _site/api/java/lib/ _site/api/java/org/ _site/api/java/org/apache/ _s

Author: pwendell Date: Fri Feb 2 00:15:49 2018 New Revision: 24627 Log: Apache Spark 2.4.0-SNAPSHOT-2018_02_01_16_01-032c11b docs [This commit notification would consist of 1444 parts, which exceeds the limit of 50 ones, so it was shortened to the summary.]

spark git commit: [SPARK-23296][YARN] Include stacktrace in YARN-app diagnostic

Repository: spark Updated Branches: refs/heads/branch-2.3 07a8f4ddf -> ab23785c7 [SPARK-23296][YARN] Include stacktrace in YARN-app diagnostic ## What changes were proposed in this pull request? Include stacktrace in the diagnostics message upon abnormal unregister from RM ## How was this

spark git commit: [SPARK-23296][YARN] Include stacktrace in YARN-app diagnostic

Repository: spark Updated Branches: refs/heads/master 4bcfdefb9 -> 032c11b83 [SPARK-23296][YARN] Include stacktrace in YARN-app diagnostic ## What changes were proposed in this pull request? Include stacktrace in the diagnostics message upon abnormal unregister from RM ## How was this patch

spark git commit: [INFRA] Close stale PRs.

Repository: spark Updated Branches: refs/heads/master 73da3b696 -> 4bcfdefb9 [INFRA] Close stale PRs. Closes #20334 Closes #20262 Project: http://git-wip-us.apache.org/repos/asf/spark/repo Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/4bcfdefb Tree:

svn commit: r24625 - in /dev/spark/2.3.1-SNAPSHOT-2018_02_01_14_01-07a8f4d-docs: ./ _site/ _site/api/ _site/api/R/ _site/api/java/ _site/api/java/lib/ _site/api/java/org/ _site/api/java/org/apache/ _s

Author: pwendell Date: Thu Feb 1 22:15:59 2018 New Revision: 24625 Log: Apache Spark 2.3.1-SNAPSHOT-2018_02_01_14_01-07a8f4d docs [This commit notification would consist of 1442 parts, which exceeds the limit of 50 ones, so it was shortened to the summary.]

svn commit: r24624 - in /dev/spark/2.4.0-SNAPSHOT-2018_02_01_12_01-73da3b6-docs: ./ _site/ _site/api/ _site/api/R/ _site/api/java/ _site/api/java/lib/ _site/api/java/org/ _site/api/java/org/apache/ _s

Author: pwendell Date: Thu Feb 1 20:15:47 2018 New Revision: 24624 Log: Apache Spark 2.4.0-SNAPSHOT-2018_02_01_12_01-73da3b6 docs [This commit notification would consist of 1444 parts, which exceeds the limit of 50 ones, so it was shortened to the summary.]

spark git commit: [SPARK-23293][SQL] fix data source v2 self join

Repository: spark Updated Branches: refs/heads/branch-2.3 2db7e49db -> 07a8f4ddf [SPARK-23293][SQL] fix data source v2 self join `DataSourceV2Relation` should extend `MultiInstanceRelation`, to take care of self-join. a new test Author: Wenchen Fan Closes #20466

spark git commit: [SPARK-23293][SQL] fix data source v2 self join

Repository: spark Updated Branches: refs/heads/master f051f8340 -> 73da3b696 [SPARK-23293][SQL] fix data source v2 self join ## What changes were proposed in this pull request? `DataSourceV2Relation` should extend `MultiInstanceRelation`, to take care of self-join. ## How was this patch

spark git commit: [SPARK-13983][SQL] Fix HiveThriftServer2 can not get "--hiveconf" and ''--hivevar" variables since 2.0

Repository: spark Updated Branches: refs/heads/branch-2.3 2549beae2 -> 2db7e49db [SPARK-13983][SQL] Fix HiveThriftServer2 can not get "--hiveconf" and ''--hivevar" variables since 2.0 ## What changes were proposed in this pull request? `--hiveconf` and `--hivevar` variables no longer work

spark git commit: [SPARK-13983][SQL] Fix HiveThriftServer2 can not get "--hiveconf" and ''--hivevar" variables since 2.0

Repository: spark Updated Branches: refs/heads/master ec63e2d07 -> f051f8340 [SPARK-13983][SQL] Fix HiveThriftServer2 can not get "--hiveconf" and ''--hivevar" variables since 2.0 ## What changes were proposed in this pull request? `--hiveconf` and `--hivevar` variables no longer work since

svn commit: r24621 - in /dev/spark/2.4.0-SNAPSHOT-2018_02_01_08_01-ec63e2d-docs: ./ _site/ _site/api/ _site/api/R/ _site/api/java/ _site/api/java/lib/ _site/api/java/org/ _site/api/java/org/apache/ _s

Author: pwendell Date: Thu Feb 1 16:15:48 2018 New Revision: 24621 Log: Apache Spark 2.4.0-SNAPSHOT-2018_02_01_08_01-ec63e2d docs [This commit notification would consist of 1444 parts, which exceeds the limit of 50 ones, so it was shortened to the summary.]

svn commit: r24618 - in /dev/spark/2.3.1-SNAPSHOT-2018_02_01_06_01-2549bea-docs: ./ _site/ _site/api/ _site/api/R/ _site/api/java/ _site/api/java/lib/ _site/api/java/org/ _site/api/java/org/apache/ _s

Author: pwendell Date: Thu Feb 1 14:15:13 2018 New Revision: 24618 Log: Apache Spark 2.3.1-SNAPSHOT-2018_02_01_06_01-2549bea docs [This commit notification would consist of 1442 parts, which exceeds the limit of 50 ones, so it was shortened to the summary.]

spark git commit: [SPARK-23289][CORE] OneForOneBlockFetcher.DownloadCallback.onData should write the buffer fully

Repository: spark Updated Branches: refs/heads/branch-2.3 3aa780ef3 -> 2549beae2 [SPARK-23289][CORE] OneForOneBlockFetcher.DownloadCallback.onData should write the buffer fully ## What changes were proposed in this pull request? `channel.write(buf)` may not write the whole buffer since the

spark git commit: [SPARK-23289][CORE] OneForOneBlockFetcher.DownloadCallback.onData should write the buffer fully

Repository: spark Updated Branches: refs/heads/master ffbca8451 -> ec63e2d07 [SPARK-23289][CORE] OneForOneBlockFetcher.DownloadCallback.onData should write the buffer fully ## What changes were proposed in this pull request? `channel.write(buf)` may not write the whole buffer since the

spark git commit: [SPARK-23202][SQL] Add new API in DataSourceWriter: onDataWriterCommit

Repository: spark Updated Branches: refs/heads/master 89e8d556b -> ffbca8451 [SPARK-23202][SQL] Add new API in DataSourceWriter: onDataWriterCommit ## What changes were proposed in this pull request? The current DataSourceWriter API makes it hard to implement `onTaskCommit(taskCommit:

spark git commit: [SPARK-23280][SQL][FOLLOWUP] Enable `MutableColumnarRow.getMap()`.

Repository: spark Updated Branches: refs/heads/master 8bb70b068 -> 89e8d556b [SPARK-23280][SQL][FOLLOWUP] Enable `MutableColumnarRow.getMap()`. ## What changes were proposed in this pull request? This is a followup pr of #20450. We should've enabled `MutableColumnarRow.getMap()` as well. ##

spark git commit: [SPARK-23280][SQL][FOLLOWUP] Enable `MutableColumnarRow.getMap()`.

Repository: spark Updated Branches: refs/heads/branch-2.3 6b6bc9c4e -> 3aa780ef3 [SPARK-23280][SQL][FOLLOWUP] Enable `MutableColumnarRow.getMap()`. ## What changes were proposed in this pull request? This is a followup pr of #20450. We should've enabled `MutableColumnarRow.getMap()` as well.

spark git commit: [SPARK-23280][SQL][FOLLOWUP] Fix Java style check issues.

Repository: spark Updated Branches: refs/heads/branch-2.3 205bce974 -> 6b6bc9c4e [SPARK-23280][SQL][FOLLOWUP] Fix Java style check issues. ## What changes were proposed in this pull request? This is a follow-up of #20450 which broke lint-java checks. This pr fixes the lint-java issues. ```

spark git commit: [SPARK-23280][SQL][FOLLOWUP] Fix Java style check issues.

Repository: spark Updated Branches: refs/heads/master e15da5b14 -> 8bb70b068 [SPARK-23280][SQL][FOLLOWUP] Fix Java style check issues. ## What changes were proposed in this pull request? This is a follow-up of #20450 which broke lint-java checks. This pr fixes the lint-java issues. ```

svn commit: r24617 - in /dev/spark/2.4.0-SNAPSHOT-2018_02_01_04_02-e15da5b-docs: ./ _site/ _site/api/ _site/api/R/ _site/api/java/ _site/api/java/lib/ _site/api/java/org/ _site/api/java/org/apache/ _s

Author: pwendell Date: Thu Feb 1 12:22:59 2018 New Revision: 24617 Log: Apache Spark 2.4.0-SNAPSHOT-2018_02_01_04_02-e15da5b docs [This commit notification would consist of 1443 parts, which exceeds the limit of 50 ones, so it was shortened to the summary.]

svn commit: r24616 - in /dev/spark/2.3.1-SNAPSHOT-2018_02_01_02_01-205bce9-docs: ./ _site/ _site/api/ _site/api/R/ _site/api/java/ _site/api/java/lib/ _site/api/java/org/ _site/api/java/org/apache/ _s

Author: pwendell Date: Thu Feb 1 10:17:58 2018 New Revision: 24616 Log: Apache Spark 2.3.1-SNAPSHOT-2018_02_01_02_01-205bce9 docs [This commit notification would consist of 1442 parts, which exceeds the limit of 50 ones, so it was shortened to the summary.]

spark git commit: [SPARK-23107][ML] ML 2.3 QA: New Scala APIs, docs.

Repository: spark Updated Branches: refs/heads/branch-2.3 871fd48dc -> 205bce974 [SPARK-23107][ML] ML 2.3 QA: New Scala APIs, docs. ## What changes were proposed in this pull request? Audit new APIs and docs in 2.3.0. ## How was this patch tested? No test. Author: Yanbo Liang

spark git commit: [SPARK-23107][ML] ML 2.3 QA: New Scala APIs, docs.

Repository: spark Updated Branches: refs/heads/master 07cee3373 -> e15da5b14 [SPARK-23107][ML] ML 2.3 QA: New Scala APIs, docs. ## What changes were proposed in this pull request? Audit new APIs and docs in 2.3.0. ## How was this patch tested? No test. Author: Yanbo Liang

svn commit: r24615 - in /dev/spark/2.4.0-SNAPSHOT-2018_02_01_00_01-07cee33-docs: ./ _site/ _site/api/ _site/api/R/ _site/api/java/ _site/api/java/lib/ _site/api/java/org/ _site/api/java/org/apache/ _s

Author: pwendell Date: Thu Feb 1 08:17:00 2018 New Revision: 24615 Log: Apache Spark 2.4.0-SNAPSHOT-2018_02_01_00_01-07cee33 docs [This commit notification would consist of 1443 parts, which exceeds the limit of 50 ones, so it was shortened to the summary.]