[2/2] spark git commit: Preparing development version 2.0.1-SNAPSHOT

Preparing development version 2.0.1-SNAPSHOT Project: http://git-wip-us.apache.org/repos/asf/spark/repo Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/b3ebecbb Tree: http://git-wip-us.apache.org/repos/asf/spark/tree/b3ebecbb Diff:

[1/2] spark git commit: Preparing Spark release v2.0.0-rc3

Repository: spark Updated Branches: refs/heads/branch-2.0 f6eda6b30 -> b3ebecbb7 Preparing Spark release v2.0.0-rc3 Project: http://git-wip-us.apache.org/repos/asf/spark/repo Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/48d1fa3e Tree:

[spark] Git Push Summary

Repository: spark Updated Tags: refs/tags/v2.0.0-rc3 [created] 48d1fa3e7 - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org

spark git commit: [SPARK-16503] SparkSession should provide Spark version

Repository: spark Updated Branches: refs/heads/branch-2.0 5244f86cf -> f6eda6b30 [SPARK-16503] SparkSession should provide Spark version ## What changes were proposed in this pull request? This patch enables SparkSession to provide spark version. ## How was this patch tested? Manual test:

[spark] Git Push Summary

Repository: spark Updated Tags: refs/tags/v2.0.0-rc3 [deleted] 47eb9a621 - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org

[2/2] spark git commit: Preparing development version 2.0.1-SNAPSHOT

Preparing development version 2.0.1-SNAPSHOT Project: http://git-wip-us.apache.org/repos/asf/spark/repo Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/5244f86c Tree: http://git-wip-us.apache.org/repos/asf/spark/tree/5244f86c Diff:

[spark] Git Push Summary

Repository: spark Updated Tags: refs/tags/v2.0.0-rc3 [created] 47eb9a621 - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org

[1/2] spark git commit: Preparing Spark release v2.0.0-rc3

Repository: spark Updated Branches: refs/heads/branch-2.0 abb802359 -> 5244f86cf Preparing Spark release v2.0.0-rc3 Project: http://git-wip-us.apache.org/repos/asf/spark/repo Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/47eb9a62 Tree:

spark git commit: [SPARK-16536][SQL][PYSPARK][MINOR] Expose `sql` in PySpark Shell

Repository: spark Updated Branches: refs/heads/master a5f51e216 -> 9c530576a [SPARK-16536][SQL][PYSPARK][MINOR] Expose `sql` in PySpark Shell ## What changes were proposed in this pull request? This PR exposes `sql` in PySpark Shell like Scala/R Shells for consistency. **Background** *

spark git commit: [SPARK-16485][ML][DOC] Fix privacy of GLM members, rename sqlDataTypes for ML, doc fixes

Repository: spark Updated Branches: refs/heads/branch-2.0 550d0e7dc -> abb802359 [SPARK-16485][ML][DOC] Fix privacy of GLM members, rename sqlDataTypes for ML, doc fixes ## What changes were proposed in this pull request? Fixing issues found during 2.0 API checks: *

spark git commit: [SPARK-16482][SQL] Describe Table Command for Tables Requiring Runtime Inferred Schema

Repository: spark Updated Branches: refs/heads/branch-2.0 9e3a59858 -> 550d0e7dc [SPARK-16482][SQL] Describe Table Command for Tables Requiring Runtime Inferred Schema What changes were proposed in this pull request? If we create a table pointing to a parquet/json datasets without

spark git commit: [SPARK-16482][SQL] Describe Table Command for Tables Requiring Runtime Inferred Schema

Repository: spark Updated Branches: refs/heads/master fb2e8eeb0 -> c5ec87982 [SPARK-16482][SQL] Describe Table Command for Tables Requiring Runtime Inferred Schema What changes were proposed in this pull request? If we create a table pointing to a parquet/json datasets without

spark git commit: [SPARKR][DOCS][MINOR] R programming guide to include csv data source example

Repository: spark Updated Branches: refs/heads/master b4baf086c -> fb2e8eeb0 [SPARKR][DOCS][MINOR] R programming guide to include csv data source example ## What changes were proposed in this pull request? Minor documentation update for code example, code style, and missed reference to

spark git commit: [SPARKR][MINOR] R examples and test updates

Repository: spark Updated Branches: refs/heads/master 51a6706b1 -> b4baf086c [SPARKR][MINOR] R examples and test updates ## What changes were proposed in this pull request? Minor example updates ## How was this patch tested? manual shivaram Author: Felix Cheung

spark git commit: [SPARKR][MINOR] R examples and test updates

Repository: spark Updated Branches: refs/heads/branch-2.0 86adc5cfb -> 18255a934 [SPARKR][MINOR] R examples and test updates ## What changes were proposed in this pull request? Minor example updates ## How was this patch tested? manual shivaram Author: Felix Cheung

spark git commit: [SPARK-16114][SQL] updated structured streaming guide

Repository: spark Updated Branches: refs/heads/branch-2.0 7de183d97 -> 86adc5cfb [SPARK-16114][SQL] updated structured streaming guide ## What changes were proposed in this pull request? Updated structured streaming programming guide with new windowed example. ## How was this patch tested?

spark git commit: [SPARK-16114][SQL] updated structured streaming guide

Repository: spark Updated Branches: refs/heads/master 0744d84c9 -> 51a6706b1 [SPARK-16114][SQL] updated structured streaming guide ## What changes were proposed in this pull request? Updated structured streaming programming guide with new windowed example. ## How was this patch tested?

spark git commit: [SPARK-16531][SQL][TEST] Remove timezone setting from DataFrameTimeWindowingSuite

Repository: spark Updated Branches: refs/heads/branch-2.0 2e97f3a08 -> 7de183d97 [SPARK-16531][SQL][TEST] Remove timezone setting from DataFrameTimeWindowingSuite ## What changes were proposed in this pull request? It's unnecessary. `QueryTest` already sets it. Author: Burak Yavuz

spark git commit: [SPARK-16531][SQL][TEST] Remove timezone setting from DataFrameTimeWindowingSuite

Repository: spark Updated Branches: refs/heads/master 01f09b161 -> 0744d84c9 [SPARK-16531][SQL][TEST] Remove timezone setting from DataFrameTimeWindowingSuite ## What changes were proposed in this pull request? It's unnecessary. `QueryTest` already sets it. Author: Burak Yavuz

[1/2] spark git commit: [SPARK-14812][ML][MLLIB][PYTHON] Experimental, DeveloperApi annotation audit for ML

Repository: spark Updated Branches: refs/heads/branch-2.0 90f0e8132 -> 2e97f3a08 http://git-wip-us.apache.org/repos/asf/spark/blob/2e97f3a0/mllib/src/main/scala/org/apache/spark/mllib/clustering/BisectingKMeans.scala -- diff

[1/2] spark git commit: [SPARK-14812][ML][MLLIB][PYTHON] Experimental, DeveloperApi annotation audit for ML

Repository: spark Updated Branches: refs/heads/master d8220c1e5 -> 01f09b161 http://git-wip-us.apache.org/repos/asf/spark/blob/01f09b16/mllib/src/main/scala/org/apache/spark/mllib/clustering/BisectingKMeans.scala -- diff --git

[2/2] spark git commit: [SPARK-14812][ML][MLLIB][PYTHON] Experimental, DeveloperApi annotation audit for ML

[SPARK-14812][ML][MLLIB][PYTHON] Experimental, DeveloperApi annotation audit for ML ## What changes were proposed in this pull request? General decisions to follow, except where noted: * spark.mllib, pyspark.mllib: Remove all Experimental annotations. Leave DeveloperApi annotations alone. *

[2/2] spark git commit: [SPARK-14812][ML][MLLIB][PYTHON] Experimental, DeveloperApi annotation audit for ML

[SPARK-14812][ML][MLLIB][PYTHON] Experimental, DeveloperApi annotation audit for ML ## What changes were proposed in this pull request? General decisions to follow, except where noted: * spark.mllib, pyspark.mllib: Remove all Experimental annotations. Leave DeveloperApi annotations alone. *

spark git commit: [SPARK-16435][YARN][MINOR] Add warning log if initialExecutors is less than minExecutors

Repository: spark Updated Branches: refs/heads/branch-2.0 7d9bd951b -> 90f0e8132 [SPARK-16435][YARN][MINOR] Add warning log if initialExecutors is less than minExecutors ## What changes were proposed in this pull request? Currently if `spark.dynamicAllocation.initialExecutors` is less than

spark git commit: [SPARK-16435][YARN][MINOR] Add warning log if initialExecutors is less than minExecutors

Repository: spark Updated Branches: refs/heads/master f376c3726 -> d8220c1e5 [SPARK-16435][YARN][MINOR] Add warning log if initialExecutors is less than minExecutors ## What changes were proposed in this pull request? Currently if `spark.dynamicAllocation.initialExecutors` is less than

spark git commit: [SPARK-16343][SQL] Improve the PushDownPredicate rule to pushdown predicates correctly in non-deterministic condition.

Repository: spark Updated Branches: refs/heads/master ea06e4ef3 -> f376c3726 [SPARK-16343][SQL] Improve the PushDownPredicate rule to pushdown predicates correctly in non-deterministic condition. ## What changes were proposed in this pull request? Currently our Optimizer may reorder the

spark git commit: [SPARK-16469] enhanced simulate multiply

Repository: spark Updated Branches: refs/heads/branch-2.0 5a71a0501 -> 7d9bd951b [SPARK-16469] enhanced simulate multiply ## What changes were proposed in this pull request? We have a use case of multiplying very big sparse matrices. we have about 1000x1000 distributed block matrices

spark git commit: [SPARK-16469] enhanced simulate multiply

Repository: spark Updated Branches: refs/heads/master 51ade51a9 -> ea06e4ef3 [SPARK-16469] enhanced simulate multiply ## What changes were proposed in this pull request? We have a use case of multiplying very big sparse matrices. we have about 1000x1000 distributed block matrices

spark git commit: [SPARK-16440][MLLIB] Undeleted broadcast variables in Word2Vec causing OoM for long runs

Repository: spark Updated Branches: refs/heads/branch-1.6 fb0933681 -> 4381e2121 [SPARK-16440][MLLIB] Undeleted broadcast variables in Word2Vec causing OoM for long runs ## What changes were proposed in this pull request? Unpersist broadcasted vars in Word2Vec.fit for more timely / reliable

spark git commit: [SPARK-16440][MLLIB] Undeleted broadcast variables in Word2Vec causing OoM for long runs

Repository: spark Updated Branches: refs/heads/master 3d6f679cf -> 51ade51a9 [SPARK-16440][MLLIB] Undeleted broadcast variables in Word2Vec causing OoM for long runs ## What changes were proposed in this pull request? Unpersist broadcasted vars in Word2Vec.fit for more timely / reliable

spark git commit: [SPARK-16440][MLLIB] Undeleted broadcast variables in Word2Vec causing OoM for long runs

Repository: spark Updated Branches: refs/heads/branch-2.0 74ad486dc -> 5a71a0501 [SPARK-16440][MLLIB] Undeleted broadcast variables in Word2Vec causing OoM for long runs ## What changes were proposed in this pull request? Unpersist broadcasted vars in Word2Vec.fit for more timely / reliable

spark git commit: [MINOR][YARN] Fix code error in yarn-cluster unit test

Repository: spark Updated Branches: refs/heads/branch-2.0 a34a54435 -> 74ad486dc [MINOR][YARN] Fix code error in yarn-cluster unit test ## What changes were proposed in this pull request? Fix code error in yarn-cluster unit test. ## How was this patch tested? Use exist tests Author:

spark git commit: [MINOR][YARN] Fix code error in yarn-cluster unit test

Repository: spark Updated Branches: refs/heads/master bf107f1e6 -> 3d6f679cf [MINOR][YARN] Fix code error in yarn-cluster unit test ## What changes were proposed in this pull request? Fix code error in yarn-cluster unit test. ## How was this patch tested? Use exist tests Author: sharkd

spark git commit: [SPARK-16438] Add Asynchronous Actions documentation

Repository: spark Updated Branches: refs/heads/branch-2.0 38787ec93 -> a34a54435 [SPARK-16438] Add Asynchronous Actions documentation ## What changes were proposed in this pull request? Add Asynchronous Actions documentation inside action of programming guide ## How was this patch tested?

spark git commit: [SPARK-16438] Add Asynchronous Actions documentation

Repository: spark Updated Branches: refs/heads/master 83879ebc5 -> bf107f1e6 [SPARK-16438] Add Asynchronous Actions documentation ## What changes were proposed in this pull request? Add Asynchronous Actions documentation inside action of programming guide ## How was this patch tested?

spark git commit: [SPARK-16439] Fix number formatting in SQL UI

Repository: spark Updated Branches: refs/heads/branch-2.0 934e2aa4f -> 38787ec93 [SPARK-16439] Fix number formatting in SQL UI ## What changes were proposed in this pull request? Spark SQL UI display numbers greater than 1000 with u00A0 as grouping separator. Problem exists when server

spark git commit: [SPARK-16439] Fix number formatting in SQL UI

Repository: spark Updated Branches: refs/heads/master f73891e0b -> 83879ebc5 [SPARK-16439] Fix number formatting in SQL UI ## What changes were proposed in this pull request? Spark SQL UI display numbers greater than 1000 with u00A0 as grouping separator. Problem exists when server locale

spark git commit: [MINOR] Fix Java style errors and remove unused imports

Repository: spark Updated Branches: refs/heads/branch-2.0 5301efc17 -> 934e2aa4f [MINOR] Fix Java style errors and remove unused imports Fix Java style errors and remove unused imports, which are randomly found Tested on my local machine. Author: Xin Ren Closes #14161

spark git commit: [MINOR] Fix Java style errors and remove unused imports

Repository: spark Updated Branches: refs/heads/master f156136da -> f73891e0b [MINOR] Fix Java style errors and remove unused imports ## What changes were proposed in this pull request? Fix Java style errors and remove unused imports, which are randomly found ## How was this patch tested?

spark git commit: [SPARK-16375][WEB UI] Fixed misassigned var: numCompletedTasks was assigned to numSkippedTasks

Repository: spark Updated Branches: refs/heads/branch-2.0 4b93a833b -> 5301efc17 [SPARK-16375][WEB UI] Fixed misassigned var: numCompletedTasks was assigned to numSkippedTasks ## What changes were proposed in this pull request? I fixed a misassigned var, numCompletedTasks was assigned to

spark git commit: [SPARK-16375][WEB UI] Fixed misassigned var: numCompletedTasks was assigned to numSkippedTasks

Repository: spark Updated Branches: refs/heads/master c190d89bd -> f156136da [SPARK-16375][WEB UI] Fixed misassigned var: numCompletedTasks was assigned to numSkippedTasks ## What changes were proposed in this pull request? I fixed a misassigned var, numCompletedTasks was assigned to

spark git commit: [SPARK-16375][WEB UI] Fixed misassigned var: numCompletedTasks was assigned to numSkippedTasks

Repository: spark Updated Branches: refs/heads/branch-1.6 980db2bd4 -> fb0933681 [SPARK-16375][WEB UI] Fixed misassigned var: numCompletedTasks was assigned to numSkippedTasks ## What changes were proposed in this pull request? I fixed a misassigned var, numCompletedTasks was assigned to

spark git commit: [SPARK-15889][STREAMING] Follow-up fix to erroneous condition in StreamTest

Repository: spark Updated Branches: refs/heads/master 772c213ec -> c190d89bd [SPARK-15889][STREAMING] Follow-up fix to erroneous condition in StreamTest ## What changes were proposed in this pull request? A second form of AssertQuery now actually invokes the condition; avoids a build

spark git commit: [SPARK-15889][STREAMING] Follow-up fix to erroneous condition in StreamTest

Repository: spark Updated Branches: refs/heads/branch-2.0 5173f847c -> 4b93a833b [SPARK-15889][STREAMING] Follow-up fix to erroneous condition in StreamTest ## What changes were proposed in this pull request? A second form of AssertQuery now actually invokes the condition; avoids a build

spark git commit: [SPARK-16303][DOCS][EXAMPLES] Updated SQL programming guide and examples

Repository: spark Updated Branches: refs/heads/branch-2.0 41df62c59 -> 5173f847c [SPARK-16303][DOCS][EXAMPLES] Updated SQL programming guide and examples - Hard-coded Spark SQL sample snippets were moved into source files under examples sub-project. - Removed the inconsistency between Scala

spark git commit: [SPARK-16303][DOCS][EXAMPLES] Updated SQL programming guide and examples

Repository: spark Updated Branches: refs/heads/master 1c58fa905 -> 772c213ec [SPARK-16303][DOCS][EXAMPLES] Updated SQL programming guide and examples - Hard-coded Spark SQL sample snippets were moved into source files under examples sub-project. - Removed the inconsistency between Scala and

spark git commit: [HOTFIX] Fix build break.

Repository: spark Updated Branches: refs/heads/branch-1.6 7c8a399a2 -> 980db2bd4 [HOTFIX] Fix build break. Project: http://git-wip-us.apache.org/repos/asf/spark/repo Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/980db2bd Tree:

spark git commit: [SPARK-16489][SQL] Guard against variable reuse mistakes in expression code generation

Repository: spark Updated Branches: refs/heads/branch-1.6 d1c992fea -> 7c8a399a2 [SPARK-16489][SQL] Guard against variable reuse mistakes in expression code generation In code generation, it is incorrect for expressions to reuse variable names across different instances of itself. As an

spark git commit: [SPARK-16488] Fix codegen variable namespace collision in pmod and partitionBy

Repository: spark Updated Branches: refs/heads/branch-1.6 9808735e0 -> d1c992fea [SPARK-16488] Fix codegen variable namespace collision in pmod and partitionBy This patch fixes a variable namespace collision bug in pmod and partitionBy Regression test for one possible occurrence. A more

spark git commit: [SPARK-16514][SQL] Fix various regex codegen bugs

Repository: spark Updated Branches: refs/heads/branch-1.6 702178d1f -> 9808735e0 [SPARK-16514][SQL] Fix various regex codegen bugs ## What changes were proposed in this pull request? RegexExtract and RegexReplace currently crash on non-nullable input due use of a hard-coded local variable

spark git commit: [SPARK-16514][SQL] Fix various regex codegen bugs

Repository: spark Updated Branches: refs/heads/branch-2.0 4303d292b -> 41df62c59 [SPARK-16514][SQL] Fix various regex codegen bugs ## What changes were proposed in this pull request? RegexExtract and RegexReplace currently crash on non-nullable input due use of a hard-coded local variable