[31/51] [partial] spark-website git commit: Add Spark 2.1.1 docs

2017-05-02 Thread marmbrus
http://git-wip-us.apache.org/repos/asf/spark-website/blob/d4f0c34a/site/docs/2.1.1/api/R/partitionBy.html -- diff --git a/site/docs/2.1.1/api/R/partitionBy.html b/site/docs/2.1.1/api/R/partitionBy.html new file mode 100644 index

[47/51] [partial] spark-website git commit: Add Spark 2.1.1 docs

2017-05-02 Thread marmbrus
http://git-wip-us.apache.org/repos/asf/spark-website/blob/d4f0c34a/site/docs/2.1.1/api/R/approxCountDistinct.html -- diff --git a/site/docs/2.1.1/api/R/approxCountDistinct.html b/site/docs/2.1.1/api/R/approxCountDistinct.html new

[10/51] [partial] spark-website git commit: Add Spark 2.1.1 docs

2017-05-02 Thread marmbrus
http://git-wip-us.apache.org/repos/asf/spark-website/blob/d4f0c34a/site/docs/2.1.1/api/java/org/apache/spark/AccumulatorParam.IntAccumulatorParam$.html -- diff --git

[37/51] [partial] spark-website git commit: Add Spark 2.1.1 docs

2017-05-02 Thread marmbrus
http://git-wip-us.apache.org/repos/asf/spark-website/blob/d4f0c34a/site/docs/2.1.1/api/R/groupBy.html -- diff --git a/site/docs/2.1.1/api/R/groupBy.html b/site/docs/2.1.1/api/R/groupBy.html new file mode 100644 index

[19/51] [partial] spark-website git commit: Add Spark 2.1.1 docs

2017-05-02 Thread marmbrus
http://git-wip-us.apache.org/repos/asf/spark-website/blob/d4f0c34a/site/docs/2.1.1/api/R/withColumn.html -- diff --git a/site/docs/2.1.1/api/R/withColumn.html b/site/docs/2.1.1/api/R/withColumn.html new file mode 100644 index

[43/51] [partial] spark-website git commit: Add Spark 2.1.1 docs

2017-05-02 Thread marmbrus
http://git-wip-us.apache.org/repos/asf/spark-website/blob/d4f0c34a/site/docs/2.1.1/api/R/covar_pop.html -- diff --git a/site/docs/2.1.1/api/R/covar_pop.html b/site/docs/2.1.1/api/R/covar_pop.html new file mode 100644 index

[15/51] [partial] spark-website git commit: Add Spark 2.1.1 docs

2017-05-02 Thread marmbrus
http://git-wip-us.apache.org/repos/asf/spark-website/blob/d4f0c34a/site/docs/2.1.1/api/java/constant-values.html -- diff --git a/site/docs/2.1.1/api/java/constant-values.html b/site/docs/2.1.1/api/java/constant-values.html new

[14/51] [partial] spark-website git commit: Add Spark 2.1.1 docs

2017-05-02 Thread marmbrus
http://git-wip-us.apache.org/repos/asf/spark-website/blob/d4f0c34a/site/docs/2.1.1/api/java/index-all.html -- diff --git a/site/docs/2.1.1/api/java/index-all.html b/site/docs/2.1.1/api/java/index-all.html new file mode 100644

[13/51] [partial] spark-website git commit: Add Spark 2.1.1 docs

2017-05-02 Thread marmbrus
http://git-wip-us.apache.org/repos/asf/spark-website/blob/d4f0c34a/site/docs/2.1.1/api/java/index.html -- diff --git a/site/docs/2.1.1/api/java/index.html b/site/docs/2.1.1/api/java/index.html new file mode 100644 index

[50/51] [partial] spark-website git commit: Add Spark 2.1.1 docs

2017-05-02 Thread marmbrus
http://git-wip-us.apache.org/repos/asf/spark-website/blob/d4f0c34a/site/docs/2.1.1/README.md -- diff --git a/site/docs/2.1.1/README.md b/site/docs/2.1.1/README.md new file mode 100644 index 000..ffd3b57 --- /dev/null +++

[32/51] [partial] spark-website git commit: Add Spark 2.1.1 docs

2017-05-02 Thread marmbrus
http://git-wip-us.apache.org/repos/asf/spark-website/blob/d4f0c34a/site/docs/2.1.1/api/R/nafunctions.html -- diff --git a/site/docs/2.1.1/api/R/nafunctions.html b/site/docs/2.1.1/api/R/nafunctions.html new file mode 100644 index

[26/51] [partial] spark-website git commit: Add Spark 2.1.1 docs

2017-05-02 Thread marmbrus
http://git-wip-us.apache.org/repos/asf/spark-website/blob/d4f0c34a/site/docs/2.1.1/api/R/show.html -- diff --git a/site/docs/2.1.1/api/R/show.html b/site/docs/2.1.1/api/R/show.html new file mode 100644 index 000..50eb890 ---

[03/51] [partial] spark-website git commit: Add Spark 2.1.1 docs

2017-05-02 Thread marmbrus
http://git-wip-us.apache.org/repos/asf/spark-website/blob/d4f0c34a/site/docs/2.1.1/api/java/org/apache/spark/SparkConf.html -- diff --git a/site/docs/2.1.1/api/java/org/apache/spark/SparkConf.html

[49/51] [partial] spark-website git commit: Add Spark 2.1.1 docs

2017-05-02 Thread marmbrus
http://git-wip-us.apache.org/repos/asf/spark-website/blob/d4f0c34a/site/docs/2.1.1/api/R/00Index.html -- diff --git a/site/docs/2.1.1/api/R/00Index.html b/site/docs/2.1.1/api/R/00Index.html new file mode 100644 index

[42/51] [partial] spark-website git commit: Add Spark 2.1.1 docs

2017-05-02 Thread marmbrus
http://git-wip-us.apache.org/repos/asf/spark-website/blob/d4f0c34a/site/docs/2.1.1/api/R/dapplyCollect.html -- diff --git a/site/docs/2.1.1/api/R/dapplyCollect.html b/site/docs/2.1.1/api/R/dapplyCollect.html new file mode 100644

[46/51] [partial] spark-website git commit: Add Spark 2.1.1 docs

2017-05-02 Thread marmbrus
http://git-wip-us.apache.org/repos/asf/spark-website/blob/d4f0c34a/site/docs/2.1.1/api/R/attach.html -- diff --git a/site/docs/2.1.1/api/R/attach.html b/site/docs/2.1.1/api/R/attach.html new file mode 100644 index

[11/51] [partial] spark-website git commit: Add Spark 2.1.1 docs

2017-05-02 Thread marmbrus
http://git-wip-us.apache.org/repos/asf/spark-website/blob/d4f0c34a/site/docs/2.1.1/api/java/org/apache/spark/Accumulable.html -- diff --git a/site/docs/2.1.1/api/java/org/apache/spark/Accumulable.html

[18/51] [partial] spark-website git commit: Add Spark 2.1.1 docs

2017-05-02 Thread marmbrus
http://git-wip-us.apache.org/repos/asf/spark-website/blob/d4f0c34a/site/docs/2.1.1/api/R/write.orc.html -- diff --git a/site/docs/2.1.1/api/R/write.orc.html b/site/docs/2.1.1/api/R/write.orc.html new file mode 100644 index

[27/51] [partial] spark-website git commit: Add Spark 2.1.1 docs

2017-05-02 Thread marmbrus
http://git-wip-us.apache.org/repos/asf/spark-website/blob/d4f0c34a/site/docs/2.1.1/api/R/select.html -- diff --git a/site/docs/2.1.1/api/R/select.html b/site/docs/2.1.1/api/R/select.html new file mode 100644 index

[05/51] [partial] spark-website git commit: Add Spark 2.1.1 docs

2017-05-02 Thread marmbrus
http://git-wip-us.apache.org/repos/asf/spark-website/blob/d4f0c34a/site/docs/2.1.1/api/java/org/apache/spark/JobExecutionStatus.html -- diff --git a/site/docs/2.1.1/api/java/org/apache/spark/JobExecutionStatus.html

[08/51] [partial] spark-website git commit: Add Spark 2.1.1 docs

2017-05-02 Thread marmbrus
http://git-wip-us.apache.org/repos/asf/spark-website/blob/d4f0c34a/site/docs/2.1.1/api/java/org/apache/spark/ComplexFutureAction.html -- diff --git a/site/docs/2.1.1/api/java/org/apache/spark/ComplexFutureAction.html

[30/51] [partial] spark-website git commit: Add Spark 2.1.1 docs

2017-05-02 Thread marmbrus
http://git-wip-us.apache.org/repos/asf/spark-website/blob/d4f0c34a/site/docs/2.1.1/api/R/randomSplit.html -- diff --git a/site/docs/2.1.1/api/R/randomSplit.html b/site/docs/2.1.1/api/R/randomSplit.html new file mode 100644 index

[3/4] spark-website git commit: Add Spark 2.1.1 release.

2017-05-02 Thread marmbrus
http://git-wip-us.apache.org/repos/asf/spark-website/blob/e4019e64/site/news/spark-1-4-1-released.html -- diff --git a/site/news/spark-1-4-1-released.html b/site/news/spark-1-4-1-released.html index d4327a4..faf7639 100644 ---

[2/4] spark-website git commit: Add Spark 2.1.1 release.

2017-05-02 Thread marmbrus
http://git-wip-us.apache.org/repos/asf/spark-website/blob/e4019e64/site/release-process.html -- diff --git a/site/release-process.html b/site/release-process.html index 4dded93..7782ab0 100644 --- a/site/release-process.html +++

[4/4] spark-website git commit: Add Spark 2.1.1 release.

2017-05-02 Thread marmbrus
Add Spark 2.1.1 release. Project: http://git-wip-us.apache.org/repos/asf/spark-website/repo Commit: http://git-wip-us.apache.org/repos/asf/spark-website/commit/e4019e64 Tree: http://git-wip-us.apache.org/repos/asf/spark-website/tree/e4019e64 Diff:

[1/4] spark-website git commit: Add Spark 2.1.1 release.

2017-05-02 Thread marmbrus
Repository: spark-website Updated Branches: refs/heads/asf-site 09046892b -> e4019e64c http://git-wip-us.apache.org/repos/asf/spark-website/blob/e4019e64/site/sitemap.xml -- diff --git a/site/sitemap.xml b/site/sitemap.xml

[spark] Git Push Summary

2017-05-01 Thread marmbrus
Repository: spark Updated Tags: refs/tags/v2.1.1-rc3 [deleted] 2ed19cff2 - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org

[spark] Git Push Summary

2017-05-01 Thread marmbrus
Repository: spark Updated Tags: refs/tags/v2.1.1-rc2 [deleted] 02b165dcc - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org

[spark] Git Push Summary

2017-05-01 Thread marmbrus
Repository: spark Updated Tags: refs/tags/v2.1.1-rc4 [deleted] 267aca5bd - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org

[spark] Git Push Summary

2017-05-01 Thread marmbrus
Repository: spark Updated Tags: refs/tags/v2.1.1-rc1 [deleted] 30abb95c9 - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org

[spark] Git Push Summary

2017-05-01 Thread marmbrus
Repository: spark Updated Tags: refs/tags/v2.1.1 [created] 267aca5bd - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org

svn commit: r19436 - /dev/spark/spark-2.1.1-rc4/

2017-05-01 Thread marmbrus
Author: marmbrus Date: Tue May 2 01:05:29 2017 New Revision: 19436 Log: Add spark-2.1.1-rc4 Added: dev/spark/spark-2.1.1-rc4/ dev/spark/spark-2.1.1-rc4/SparkR_2.1.1.tar.gz (with props) dev/spark/spark-2.1.1-rc4/SparkR_2.1.1.tar.gz.asc dev/spark/spark-2.1.1-rc4/SparkR_2.1.1

svn commit: r19437 - /dev/spark/spark-2.1.1-rc4/ /release/spark/spark-2.1.1/

2017-05-01 Thread marmbrus
Author: marmbrus Date: Tue May 2 01:06:55 2017 New Revision: 19437 Log: Release Spark 2.1.1 Added: release/spark/spark-2.1.1/ - copied from r19436, dev/spark/spark-2.1.1-rc4/ Removed: dev/spark/spark-2.1.1-rc4

[1/2] spark git commit: [SPARK-18516][SQL] Split state and progress in streaming

2016-11-29 Thread marmbrus
Repository: spark Updated Branches: refs/heads/master 9a02f6821 -> c3d08e2f2 http://git-wip-us.apache.org/repos/asf/spark/blob/c3d08e2f/sql/core/src/main/scala/org/apache/spark/sql/streaming/progress.scala -- diff --git

[1/2] spark git commit: [SPARK-18516][SQL] Split state and progress in streaming

2016-11-29 Thread marmbrus
Repository: spark Updated Branches: refs/heads/branch-2.1 045ae299c -> 28b57c8a1 http://git-wip-us.apache.org/repos/asf/spark/blob/28b57c8a/sql/core/src/main/scala/org/apache/spark/sql/streaming/progress.scala -- diff --git

[2/2] spark git commit: [SPARK-18516][SQL] Split state and progress in streaming

2016-11-29 Thread marmbrus
"3" : 0, "0" : 1 } }, "numRecords" : 3, "inputRowsPerSecond" : 230.76923076923077, "processedRowsPerSecond" : 10.869565217391303 } ] } ``` Additionally, in order to make it possible to correlate progress update

[2/2] spark git commit: [SPARK-18516][SQL] Split state and progress in streaming

2016-11-29 Thread marmbrus
"3" : 0, "0" : 1 } }, "numRecords" : 3, "inputRowsPerSecond" : 230.76923076923077, "processedRowsPerSecond" : 10.869565217391303 } ] } ``` Additionally, in order to make it possible to correlate progress updates across

spark git commit: [SPARK-18498][SQL] Revise HDFSMetadataLog API for better testing

2016-11-29 Thread marmbrus
cks without worrying about batch file name formats. marmbrus zsxwing Existing tests already ensure this API faithfully support core functionality i.e., creation of batch files. Author: Tyson Condie <tcon...@gmail.com> Closes #15924 from tcondie/SPARK-18498. Signed-off-by: Michael Armbr

spark git commit: [SPARK-18498][SQL] Revise HDFSMetadataLog API for better testing

2016-11-29 Thread marmbrus
cks without worrying about batch file name formats. marmbrus zsxwing Existing tests already ensure this API faithfully support core functionality i.e., creation of batch files. Author: Tyson Condie <tcon...@gmail.com> Closes #15924 from tcondie/SPARK-18498. Signed-off-by: Michael Armbr

spark git commit: [SPARK-18461][DOCS][STRUCTUREDSTREAMING] Added more information about monitoring streaming queries

2016-11-16 Thread marmbrus
Repository: spark Updated Branches: refs/heads/master 0048ce7ce -> bb6cdfd9a [SPARK-18461][DOCS][STRUCTUREDSTREAMING] Added more information about monitoring streaming queries ## What changes were proposed in this pull request?

spark git commit: [SPARK-18461][DOCS][STRUCTUREDSTREAMING] Added more information about monitoring streaming queries

2016-11-16 Thread marmbrus
Repository: spark Updated Branches: refs/heads/branch-2.1 b86e962c9 -> 3d4756d56 [SPARK-18461][DOCS][STRUCTUREDSTREAMING] Added more information about monitoring streaming queries ## What changes were proposed in this pull request?

spark git commit: [SPARK-18440][STRUCTURED STREAMING] Pass correct query execution to FileFormatWriter

2016-11-15 Thread marmbrus
Repository: spark Updated Branches: refs/heads/master 5bcb9a7ff -> 1ae4652b7 [SPARK-18440][STRUCTURED STREAMING] Pass correct query execution to FileFormatWriter ## What changes were proposed in this pull request? SPARK-18012 refactored the file write path in FileStreamSink using

spark git commit: [SPARK-18295][SQL] Make to_json function null safe (matching it to from_json)

2016-11-07 Thread marmbrus
Repository: spark Updated Branches: refs/heads/branch-2.1 9873d57f2 -> 4af82d56f [SPARK-18295][SQL] Make to_json function null safe (matching it to from_json) ## What changes were proposed in this pull request? This PR proposes to match up the behaviour of `to_json` to `from_json` function

spark git commit: [SPARK-18295][SQL] Make to_json function null safe (matching it to from_json)

2016-11-07 Thread marmbrus
Repository: spark Updated Branches: refs/heads/master 3a710b94b -> 3eda05703 [SPARK-18295][SQL] Make to_json function null safe (matching it to from_json) ## What changes were proposed in this pull request? This PR proposes to match up the behaviour of `to_json` to `from_json` function for

spark git commit: [SPARK-18212][SS][KAFKA] increase executor poll timeout

2016-11-03 Thread marmbrus
Repository: spark Updated Branches: refs/heads/master 098e4ca9c -> 67659c9af [SPARK-18212][SS][KAFKA] increase executor poll timeout ## What changes were proposed in this pull request? Increase poll timeout to try and address flaky test ## How was this patch tested? Ran existing unit tests

spark git commit: [SPARK-18212][SS][KAFKA] increase executor poll timeout

2016-11-03 Thread marmbrus
Repository: spark Updated Branches: refs/heads/branch-2.1 569f77a11 -> 2daca62cd [SPARK-18212][SS][KAFKA] increase executor poll timeout ## What changes were proposed in this pull request? Increase poll timeout to try and address flaky test ## How was this patch tested? Ran existing unit

spark git commit: [SPARK-17764][SQL] Add `to_json` supporting to convert nested struct column to JSON string

2016-11-01 Thread marmbrus
Repository: spark Updated Branches: refs/heads/master cfac17ee1 -> 01dd00830 [SPARK-17764][SQL] Add `to_json` supporting to convert nested struct column to JSON string ## What changes were proposed in this pull request? This PR proposes to add `to_json` function in contrast with `from_json`

spark git commit: [SPARK-17770][CATALYST] making ObjectType public

2016-10-26 Thread marmbrus
ype. This DataType is used extensively in the JavaBean Encoder, but would also be useful in writing other custom encoders. As mentioned by marmbrus, it is understood that the Expressions API is subject to potential change. ## How was this patch tested? The change only affects the visibil

spark git commit: [SPARK-17900][SQL] Graduate a list of Spark SQL APIs to stable

2016-10-14 Thread marmbrus
Repository: spark Updated Branches: refs/heads/master f00df40cf -> 72adfbf94 [SPARK-17900][SQL] Graduate a list of Spark SQL APIs to stable ## What changes were proposed in this pull request? This patch graduates a list of Spark SQL APIs and mark them stable. The following are marked stable:

spark git commit: [SPARK-11775][PYSPARK][SQL] Allow PySpark to register Java UDF

2016-10-14 Thread marmbrus
Repository: spark Updated Branches: refs/heads/master 5aeb7384c -> f00df40cf [SPARK-11775][PYSPARK][SQL] Allow PySpark to register Java UDF Currently pyspark can only call the builtin java UDF, but can not call custom java UDF. It would be better to allow that. 2 benefits: * Leverage the

spark git commit: [SPARK-16063][SQL] Add storageLevel to Dataset

2016-10-14 Thread marmbrus
Repository: spark Updated Branches: refs/heads/master da9aeb0fd -> 5aeb7384c [SPARK-16063][SQL] Add storageLevel to Dataset [SPARK-11905](https://issues.apache.org/jira/browse/SPARK-11905) added support for `persist`/`cache` for `Dataset`. However, there is no user-facing API to check if a

spark git commit: [SPARK-17368][SQL] Add support for value class serialization and deserialization

2016-10-13 Thread marmbrus
Repository: spark Updated Branches: refs/heads/master adc112429 -> 9dc0ca060 [SPARK-17368][SQL] Add support for value class serialization and deserialization ## What changes were proposed in this pull request? Value classes were unsupported because catalyst data types were obtained through

spark git commit: [SPARK-17830] Annotate spark.sql package with InterfaceStability

2016-10-10 Thread marmbrus
Repository: spark Updated Branches: refs/heads/master 4bafacaa5 -> 689de9200 [SPARK-17830] Annotate spark.sql package with InterfaceStability ## What changes were proposed in this pull request? This patch annotates the InterfaceStability level for top level classes in o.a.spark.sql and

spark git commit: [SPARK-16411][SQL][STREAMING] Add textFile to Structured Streaming.

2016-10-07 Thread marmbrus
Repository: spark Updated Branches: refs/heads/master aa3a6841e -> bb1aaf28e [SPARK-16411][SQL][STREAMING] Add textFile to Structured Streaming. ## What changes were proposed in this pull request? Adds the textFile API which exists in DataFrameReader and serves same purpose. ## How was this

spark git commit: [SPARK-15062][SQL] Backport fix list type infer serializer issue

2016-10-06 Thread marmbrus
Repository: spark Updated Branches: refs/heads/branch-1.6 376545e4d -> d3890deb7 [SPARK-15062][SQL] Backport fix list type infer serializer issue This backports https://github.com/apache/spark/commit/733cbaa3c0ff617a630a9d6937699db37ad2943b to Branch 1.6. It's a pretty simple patch, and

spark git commit: [SPARK-17780][SQL] Report Throwable to user in StreamExecution

2016-10-06 Thread marmbrus
Repository: spark Updated Branches: refs/heads/branch-2.0 225372adf -> a2bf09588 [SPARK-17780][SQL] Report Throwable to user in StreamExecution ## What changes were proposed in this pull request? When using an incompatible source for structured streaming, it may throw NoClassDefFoundError.

spark git commit: [SPARK-17780][SQL] Report Throwable to user in StreamExecution

2016-10-06 Thread marmbrus
Repository: spark Updated Branches: refs/heads/master 79accf45a -> 9a48e60e6 [SPARK-17780][SQL] Report Throwable to user in StreamExecution ## What changes were proposed in this pull request? When using an incompatible source for structured streaming, it may throw NoClassDefFoundError. It's

spark git commit: [SPARK-17153][SQL] Should read partition data when reading new files in filestream without globbing

2016-09-26 Thread marmbrus
Repository: spark Updated Branches: refs/heads/master bde85f8b7 -> 8135e0e5e [SPARK-17153][SQL] Should read partition data when reading new files in filestream without globbing ## What changes were proposed in this pull request? When reading file stream with non-globbing path, the results

spark git commit: [SPARK-16531][SQL][TEST] Remove timezone setting from DataFrameTimeWindowingSuite

2016-07-13 Thread marmbrus
Repository: spark Updated Branches: refs/heads/branch-2.0 2e97f3a08 -> 7de183d97 [SPARK-16531][SQL][TEST] Remove timezone setting from DataFrameTimeWindowingSuite ## What changes were proposed in this pull request? It's unnecessary. `QueryTest` already sets it. Author: Burak Yavuz

spark git commit: [SPARK-16531][SQL][TEST] Remove timezone setting from DataFrameTimeWindowingSuite

2016-07-13 Thread marmbrus
Repository: spark Updated Branches: refs/heads/master 01f09b161 -> 0744d84c9 [SPARK-16531][SQL][TEST] Remove timezone setting from DataFrameTimeWindowingSuite ## What changes were proposed in this pull request? It's unnecessary. `QueryTest` already sets it. Author: Burak Yavuz

spark git commit: [SPARK-16050][TESTS] Remove the flaky test: ConsoleSinkSuite

2016-06-20 Thread marmbrus
Repository: spark Updated Branches: refs/heads/master 905f774b7 -> 5cfabec87 [SPARK-16050][TESTS] Remove the flaky test: ConsoleSinkSuite ## What changes were proposed in this pull request? ConsoleSinkSuite just collects content from stdout and compare them with the expected string.

spark git commit: [SPARK-16050][TESTS] Remove the flaky test: ConsoleSinkSuite

2016-06-20 Thread marmbrus
Repository: spark Updated Branches: refs/heads/branch-2.0 0b0b5fe54 -> 363db9f8b [SPARK-16050][TESTS] Remove the flaky test: ConsoleSinkSuite ## What changes were proposed in this pull request? ConsoleSinkSuite just collects content from stdout and compare them with the expected string.

spark git commit: [SPARK-15915][SQL] Logical plans should use canonicalized plan when override sameResult.

2016-06-14 Thread marmbrus
Repository: spark Updated Branches: refs/heads/branch-2.0 d5e60748b -> 83aa17d44 [SPARK-15915][SQL] Logical plans should use canonicalized plan when override sameResult. ## What changes were proposed in this pull request? `DataFrame` with plan overriding `sameResult` but not using

spark git commit: [SPARK-15915][SQL] Logical plans should use canonicalized plan when override sameResult.

2016-06-14 Thread marmbrus
Repository: spark Updated Branches: refs/heads/master bc02d0112 -> c5b735581 [SPARK-15915][SQL] Logical plans should use canonicalized plan when override sameResult. ## What changes were proposed in this pull request? `DataFrame` with plan overriding `sameResult` but not using canonicalized

spark git commit: [SPARK-15489][SQL] Dataset kryo encoder won't load custom user settings

2016-06-10 Thread marmbrus
Repository: spark Updated Branches: refs/heads/master aec502d91 -> 127a6678d [SPARK-15489][SQL] Dataset kryo encoder won't load custom user settings ## What changes were proposed in this pull request? Serializer instantiation will consider existing SparkConf ## How was this patch tested?

spark git commit: [SPARK-15489][SQL] Dataset kryo encoder won't load custom user settings

2016-06-10 Thread marmbrus
Repository: spark Updated Branches: refs/heads/branch-2.0 bc53422ad -> e6ebb547b [SPARK-15489][SQL] Dataset kryo encoder won't load custom user settings ## What changes were proposed in this pull request? Serializer instantiation will consider existing SparkConf ## How was this patch

spark git commit: [SPARK-6320][SQL] Move planLater method into GenericStrategy.

2016-06-10 Thread marmbrus
Repository: spark Updated Branches: refs/heads/master fb219029d -> 667d4ea7b [SPARK-6320][SQL] Move planLater method into GenericStrategy. ## What changes were proposed in this pull request? This PR moves `QueryPlanner.planLater()` method into `GenericStrategy` for extra strategies to be

spark git commit: [SPARK-6320][SQL] Move planLater method into GenericStrategy.

2016-06-01 Thread marmbrus
Repository: spark Updated Branches: refs/heads/branch-2.0 a780848af -> 71e8aaeaa [SPARK-6320][SQL] Move planLater method into GenericStrategy. ## What changes were proposed in this pull request? This PR is the minimal version of #13147 for `branch-2.0`. ## How was this patch tested? Picked

[1/2] spark git commit: [SPARK-15686][SQL] Move user-facing streaming classes into sql.streaming

2016-06-01 Thread marmbrus
Repository: spark Updated Branches: refs/heads/branch-2.0 9406a3c9a -> a780848af http://git-wip-us.apache.org/repos/asf/spark/blob/a780848a/sql/core/src/main/scala/org/apache/spark/sql/util/ContinuousQueryListener.scala --

[2/2] spark git commit: [SPARK-15686][SQL] Move user-facing streaming classes into sql.streaming

2016-06-01 Thread marmbrus
[SPARK-15686][SQL] Move user-facing streaming classes into sql.streaming ## What changes were proposed in this pull request? This patch moves all user-facing structured streaming classes into sql.streaming. As part of this, I also added some since version annotation to methods and classes that

[1/2] spark git commit: [SPARK-15686][SQL] Move user-facing streaming classes into sql.streaming

2016-06-01 Thread marmbrus
Repository: spark Updated Branches: refs/heads/master d5012c274 -> a71d1364a http://git-wip-us.apache.org/repos/asf/spark/blob/a71d1364/sql/core/src/main/scala/org/apache/spark/sql/util/ContinuousQueryListener.scala -- diff

spark git commit: [SPARK-15517][SQL][STREAMING] Add support for complete output mode in Structure Streaming

2016-05-31 Thread marmbrus
Repository: spark Updated Branches: refs/heads/branch-2.0 8657942ce -> df4f87106 [SPARK-15517][SQL][STREAMING] Add support for complete output mode in Structure Streaming ## What changes were proposed in this pull request? Currently structured streaming only supports append output mode.

spark git commit: [SPARK-15517][SQL][STREAMING] Add support for complete output mode in Structure Streaming

2016-05-31 Thread marmbrus
Repository: spark Updated Branches: refs/heads/master dfe2cbeb4 -> 90b11439b [SPARK-15517][SQL][STREAMING] Add support for complete output mode in Structure Streaming ## What changes were proposed in this pull request? Currently structured streaming only supports append output mode. This PR

spark git commit: [SPARK-15483][SQL] IncrementalExecution should use extra strategies.

2016-05-25 Thread marmbrus
Repository: spark Updated Branches: refs/heads/branch-2.0 409eb28f7 -> 20cc2eb1b [SPARK-15483][SQL] IncrementalExecution should use extra strategies. ## What changes were proposed in this pull request? Extra strategies does not work for streams because `IncrementalExecution` uses modified

spark git commit: [SPARK-15483][SQL] IncrementalExecution should use extra strategies.

2016-05-25 Thread marmbrus
Repository: spark Updated Branches: refs/heads/master 1cb347fbc -> 4b8806741 [SPARK-15483][SQL] IncrementalExecution should use extra strategies. ## What changes were proposed in this pull request? Extra strategies does not work for streams because `IncrementalExecution` uses modified

spark git commit: [MINOR][SQL][DOCS] Add notes of the deterministic assumption on UDF functions

2016-05-23 Thread marmbrus
Repository: spark Updated Branches: refs/heads/branch-2.0 c55a39c97 -> 80bf4ce30 [MINOR][SQL][DOCS] Add notes of the deterministic assumption on UDF functions ## What changes were proposed in this pull request? Spark assumes that UDF functions are deterministic. This PR adds explicit notes

spark git commit: [MINOR][SQL][DOCS] Add notes of the deterministic assumption on UDF functions

2016-05-23 Thread marmbrus
Repository: spark Updated Branches: refs/heads/master 2585d2b32 -> 37c617e4f [MINOR][SQL][DOCS] Add notes of the deterministic assumption on UDF functions ## What changes were proposed in this pull request? Spark assumes that UDF functions are deterministic. This PR adds explicit notes

spark git commit: [SPARK-15471][SQL] ScalaReflection cleanup

2016-05-23 Thread marmbrus
Repository: spark Updated Branches: refs/heads/master 80091b8a6 -> 07c36a2f0 [SPARK-15471][SQL] ScalaReflection cleanup ## What changes were proposed in this pull request? 1. simplify the logic of deserializing option type. 2. simplify the logic of serializing array type, and remove

spark git commit: [SPARK-10216][SQL] Revert "[] Avoid creating empty files during overwrit…

2016-05-20 Thread marmbrus
Michael Armbrust <mich...@databricks.com> Closes #13181 from marmbrus/revert12855. (cherry picked from commit 2ba3ff044900d16d5f6331523526f785864c1e62) Signed-off-by: Michael Armbrust <mich...@databricks.com> Project: http://git-wip-us.apache.org/repos/asf/spark/repo Commit: http://g

spark git commit: [SPARK-10216][SQL] Revert "[] Avoid creating empty files during overwrit…

2016-05-20 Thread marmbrus
Michael Armbrust <mich...@databricks.com> Closes #13181 from marmbrus/revert12855. Project: http://git-wip-us.apache.org/repos/asf/spark/repo Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/2ba3ff04 Tree: http://git-wip-us.apache.org/repos/asf/spark/tree/2ba3ff04 Diff: http://git-wip

spark git commit: [SPARK-15190][SQL] Support using SQLUserDefinedType for case classes

2016-05-20 Thread marmbrus
Repository: spark Updated Branches: refs/heads/branch-2.0 e99b22080 -> 42e63c35a [SPARK-15190][SQL] Support using SQLUserDefinedType for case classes ## What changes were proposed in this pull request? Right now inferring the schema for case classes happens before searching the

spark git commit: [SPARK-15190][SQL] Support using SQLUserDefinedType for case classes

2016-05-20 Thread marmbrus
Repository: spark Updated Branches: refs/heads/master 22947cd02 -> dfa61f7b1 [SPARK-15190][SQL] Support using SQLUserDefinedType for case classes ## What changes were proposed in this pull request? Right now inferring the schema for case classes happens before searching the

spark git commit: [SPARK-15416][SQL] Display a better message for not finding classes removed in Spark 2.0

2016-05-19 Thread marmbrus
Repository: spark Updated Branches: refs/heads/master 664367781 -> 16ba71aba [SPARK-15416][SQL] Display a better message for not finding classes removed in Spark 2.0 ## What changes were proposed in this pull request? If finding `NoClassDefFoundError` or `ClassNotFoundException`, check if

spark git commit: [SPARK-15416][SQL] Display a better message for not finding classes removed in Spark 2.0

2016-05-19 Thread marmbrus
Repository: spark Updated Branches: refs/heads/branch-2.0 e53a8f218 -> 7e25131a9 [SPARK-15416][SQL] Display a better message for not finding classes removed in Spark 2.0 ## What changes were proposed in this pull request? If finding `NoClassDefFoundError` or `ClassNotFoundException`, check

spark git commit: [SPARK-10216][SQL] Avoid creating empty files during overwriting with group by query

2016-05-17 Thread marmbrus
Repository: spark Updated Branches: refs/heads/branch-2.0 adc1c2685 -> af37bdd3a [SPARK-10216][SQL] Avoid creating empty files during overwriting with group by query ## What changes were proposed in this pull request? Currently, `INSERT INTO` with `GROUP BY` query tries to make at least 200

spark git commit: [SPARK-10216][SQL] Avoid creating empty files during overwriting with group by query

2016-05-17 Thread marmbrus
Repository: spark Updated Branches: refs/heads/master 20a89478e -> 8d05a7a98 [SPARK-10216][SQL] Avoid creating empty files during overwriting with group by query ## What changes were proposed in this pull request? Currently, `INSERT INTO` with `GROUP BY` query tries to make at least 200

spark git commit: [SPARK-15077][SQL] Use a fair lock to avoid thread starvation in StreamExecution

2016-05-02 Thread marmbrus
Repository: spark Updated Branches: refs/heads/master 0fd95be3c -> 4e3685ae5 [SPARK-15077][SQL] Use a fair lock to avoid thread starvation in StreamExecution ## What changes were proposed in this pull request? Right now `StreamExecution.awaitBatchLock` uses an unfair lock.

spark git commit: [SPARK-15077][SQL] Use a fair lock to avoid thread starvation in StreamExecution

2016-05-02 Thread marmbrus
Repository: spark Updated Branches: refs/heads/branch-2.0 733cbaa3c -> dcce0aaaf [SPARK-15077][SQL] Use a fair lock to avoid thread starvation in StreamExecution ## What changes were proposed in this pull request? Right now `StreamExecution.awaitBatchLock` uses an unfair lock.

spark git commit: [SPARK-15062][SQL] fix list type infer serializer issue

2016-05-02 Thread marmbrus
Repository: spark Updated Branches: refs/heads/branch-2.0 86167968f -> 733cbaa3c [SPARK-15062][SQL] fix list type infer serializer issue ## What changes were proposed in this pull request? Make serializer correctly inferred if the input type is `List[_]`, since `List[_]` is type of

spark git commit: [SPARK-15062][SQL] fix list type infer serializer issue

2016-05-02 Thread marmbrus
Repository: spark Updated Branches: refs/heads/master 1c19c2769 -> 0fd95be3c [SPARK-15062][SQL] fix list type infer serializer issue ## What changes were proposed in this pull request? Make serializer correctly inferred if the input type is `List[_]`, since `List[_]` is type of `Seq[_]`,

spark git commit: [SPARK-14747][SQL] Add assertStreaming/assertNoneStreaming checks in DataFrameWriter

2016-05-02 Thread marmbrus
Repository: spark Updated Branches: refs/heads/branch-2.0 fbc73f731 -> 65b94f460 [SPARK-14747][SQL] Add assertStreaming/assertNoneStreaming checks in DataFrameWriter ## Problem If an end user happens to write code mixed with continuous-query-oriented methods and

spark git commit: [SPARK-14747][SQL] Add assertStreaming/assertNoneStreaming checks in DataFrameWriter

2016-05-02 Thread marmbrus
Repository: spark Updated Branches: refs/heads/master f362363d1 -> 35d9c8aa6 [SPARK-14747][SQL] Add assertStreaming/assertNoneStreaming checks in DataFrameWriter ## Problem If an end user happens to write code mixed with continuous-query-oriented methods and non-continuous-query-oriented

spark git commit: [SPARK-14830][SQL] Add RemoveRepetitionFromGroupExpressions optimizer.

2016-05-02 Thread marmbrus
Repository: spark Updated Branches: refs/heads/master a35a67a83 -> 6e6320122 [SPARK-14830][SQL] Add RemoveRepetitionFromGroupExpressions optimizer. ## What changes were proposed in this pull request? This PR aims to optimize GroupExpressions by removing repeating expressions.

spark git commit: [SPARK-14830][SQL] Add RemoveRepetitionFromGroupExpressions optimizer.

2016-05-02 Thread marmbrus
Repository: spark Updated Branches: refs/heads/branch-2.0 1c2082b64 -> 972fd22e3 [SPARK-14830][SQL] Add RemoveRepetitionFromGroupExpressions optimizer. ## What changes were proposed in this pull request? This PR aims to optimize GroupExpressions by removing repeating expressions.

spark git commit: [SPARK-14579][SQL] Fix the race condition in StreamExecution.processAllAvailable again

2016-05-02 Thread marmbrus
Repository: spark Updated Branches: refs/heads/branch-2.0 08ae32e61 -> 1c2082b64 [SPARK-14579][SQL] Fix the race condition in StreamExecution.processAllAvailable again ## What changes were proposed in this pull request? #12339 didn't fix the race condition. MemorySinkSuite is still flaky:

spark git commit: [SPARK-14637][SQL] object expressions cleanup

2016-05-02 Thread marmbrus
Repository: spark Updated Branches: refs/heads/branch-2.0 ccb53a20e -> 1145ea01b [SPARK-14637][SQL] object expressions cleanup ## What changes were proposed in this pull request? Simplify and clean up some object expressions: 1. simplify the logic to handle `propagateNull` 2. add

spark git commit: [SPARK-14637][SQL] object expressions cleanup

2016-05-02 Thread marmbrus
Repository: spark Updated Branches: refs/heads/master 214d1be4f -> 0513c3ac9 [SPARK-14637][SQL] object expressions cleanup ## What changes were proposed in this pull request? Simplify and clean up some object expressions: 1. simplify the logic to handle `propagateNull` 2. add

spark git commit: [SPARK-14981][SQL] Throws exception if DESC is specified for sorting columns

2016-04-29 Thread marmbrus
Repository: spark Updated Branches: refs/heads/master 8ebae466a -> a04b1de5f [SPARK-14981][SQL] Throws exception if DESC is specified for sorting columns ## What changes were proposed in this pull request? Currently Spark SQL doesn't support sorting columns in descending order. However, the

spark git commit: [SPARK-14970][SQL] Prevent DataSource from enumerates all files in a directory if there is user specified schema

2016-04-28 Thread marmbrus
Repository: spark Updated Branches: refs/heads/master d5ab42ceb -> 0ee5419b6 [SPARK-14970][SQL] Prevent DataSource from enumerates all files in a directory if there is user specified schema ## What changes were proposed in this pull request? The FileCatalog object gets created even if the

spark git commit: [SPARK-14874][SQL][STREAMING] Remove the obsolete Batch representation

2016-04-27 Thread marmbrus
Repository: spark Updated Branches: refs/heads/master 7dd01d9c0 -> a234cc614 [SPARK-14874][SQL][STREAMING] Remove the obsolete Batch representation ## What changes were proposed in this pull request? The `Batch` class, which had been used to indicate progress in a stream, was abandoned by

<    1   2   3   4   5   6   7   8   9   10   >