[GitHub] spark pull request #21018: [SPARK-23880][SQL] Do not trigger any jobs for ca...

2018-04-22 Thread cloud-fan
Github user cloud-fan commented on a diff in the pull request: https://github.com/apache/spark/pull/21018#discussion_r183269227 --- Diff: sql/core/src/test/scala/org/apache/spark/sql/CachedTableSuite.scala --- @@ -794,4 +794,17 @@ class CachedTableSuite extends QueryTest with SQLT

[GitHub] spark pull request #21018: [SPARK-23880][SQL] Do not trigger any jobs for ca...

2018-04-22 Thread cloud-fan
Github user cloud-fan commented on a diff in the pull request: https://github.com/apache/spark/pull/21018#discussion_r183269323 --- Diff: sql/core/src/test/scala/org/apache/spark/sql/CachedTableSuite.scala --- @@ -794,4 +794,17 @@ class CachedTableSuite extends QueryTest with SQLT

[GitHub] spark issue #21100: [SPARK-24012][SQL] Union of map and other compatible col...

2018-04-22 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21100 **[Test build #89696 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/89696/testReport)** for PR 21100 at commit [`19b5c6a`](https://github.com/apache/spark/commit/19

[GitHub] spark pull request #21100: [SPARK-24012][SQL] Union of map and other compati...

2018-04-22 Thread liutang123
Github user liutang123 commented on a diff in the pull request: https://github.com/apache/spark/pull/21100#discussion_r183269702 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/TypeCoercion.scala --- @@ -171,6 +171,15 @@ object TypeCoercion {

[GitHub] spark pull request #21100: [SPARK-24012][SQL] Union of map and other compati...

2018-04-22 Thread cloud-fan
Github user cloud-fan commented on a diff in the pull request: https://github.com/apache/spark/pull/21100#discussion_r183269995 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/TypeCoercion.scala --- @@ -111,6 +111,18 @@ object TypeCoercion {

[GitHub] spark pull request #21100: [SPARK-24012][SQL] Union of map and other compati...

2018-04-22 Thread cloud-fan
Github user cloud-fan commented on a diff in the pull request: https://github.com/apache/spark/pull/21100#discussion_r183270011 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/TypeCoercion.scala --- @@ -111,6 +111,18 @@ object TypeCoercion {

[GitHub] spark pull request #21100: [SPARK-24012][SQL] Union of map and other compati...

2018-04-22 Thread cloud-fan
Github user cloud-fan commented on a diff in the pull request: https://github.com/apache/spark/pull/21100#discussion_r183270045 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/TypeCoercion.scala --- @@ -111,6 +111,18 @@ object TypeCoercion {

[GitHub] spark pull request #21122: [SPARK-24017] [SQL] Refactor ExternalCatalog to b...

2018-04-22 Thread cloud-fan
Github user cloud-fan commented on a diff in the pull request: https://github.com/apache/spark/pull/21122#discussion_r183270323 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/catalog/ExternalCatalog.scala --- @@ -31,10 +30,16 @@ import org.apache.spark.util.Li

[GitHub] spark pull request #20937: [SPARK-23094][SPARK-23723][SPARK-23724][SQL] Supp...

2018-04-22 Thread cloud-fan
Github user cloud-fan commented on a diff in the pull request: https://github.com/apache/spark/pull/20937#discussion_r183270654 --- Diff: python/pyspark/sql/readwriter.py --- @@ -237,6 +237,9 @@ def json(self, path, schema=None, primitivesAsString=None, prefersDecimal=None,

[GitHub] spark pull request #20937: [SPARK-23094][SPARK-23723][SPARK-23724][SQL] Supp...

2018-04-22 Thread cloud-fan
Github user cloud-fan commented on a diff in the pull request: https://github.com/apache/spark/pull/20937#discussion_r183270773 --- Diff: python/pyspark/sql/readwriter.py --- @@ -773,6 +776,8 @@ def json(self, path, mode=None, compression=None, dateFormat=None, timestampForm

[GitHub] spark pull request #20937: [SPARK-23094][SPARK-23723][SPARK-23724][SQL] Supp...

2018-04-22 Thread cloud-fan
Github user cloud-fan commented on a diff in the pull request: https://github.com/apache/spark/pull/20937#discussion_r183271101 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/json/JSONOptions.scala --- @@ -86,14 +86,41 @@ private[sql] class JSONOptions(

[GitHub] spark pull request #20937: [SPARK-23094][SPARK-23723][SPARK-23724][SQL] Supp...

2018-04-22 Thread cloud-fan
Github user cloud-fan commented on a diff in the pull request: https://github.com/apache/spark/pull/20937#discussion_r183271309 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/json/JacksonParser.scala --- @@ -361,6 +361,12 @@ class JacksonParser( /

[GitHub] spark pull request #20937: [SPARK-23094][SPARK-23723][SPARK-23724][SQL] Supp...

2018-04-22 Thread cloud-fan
Github user cloud-fan commented on a diff in the pull request: https://github.com/apache/spark/pull/20937#discussion_r183271569 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/json/JacksonParser.scala --- @@ -361,6 +361,12 @@ class JacksonParser( /

[GitHub] spark issue #20937: [SPARK-23094][SPARK-23723][SPARK-23724][SQL] Support cus...

2018-04-22 Thread cloud-fan
Github user cloud-fan commented on the issue: https://github.com/apache/spark/pull/20937 LGTM except a few minor comments --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-

[GitHub] spark pull request #21072: [SPARK-23973][SQL] Remove consecutive Sorts

2018-04-22 Thread cloud-fan
Github user cloud-fan commented on a diff in the pull request: https://github.com/apache/spark/pull/21072#discussion_r183272597 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/Optimizer.scala --- @@ -736,12 +736,22 @@ object EliminateSorts extends Rul

[GitHub] spark pull request #21123: [SPARK-24045][SQL]Create base class for file data...

2018-04-22 Thread cloud-fan
Github user cloud-fan commented on a diff in the pull request: https://github.com/apache/spark/pull/21123#discussion_r183274501 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/DataSource.scala --- @@ -89,8 +91,13 @@ case class DataSource(

[GitHub] spark pull request #21123: [SPARK-24045][SQL]Create base class for file data...

2018-04-22 Thread cloud-fan
Github user cloud-fan commented on a diff in the pull request: https://github.com/apache/spark/pull/21123#discussion_r183274588 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/DataSourceStrategy.scala --- @@ -213,6 +215,25 @@ case class DataSourceAnaly

[GitHub] spark pull request #21123: [SPARK-24045][SQL]Create base class for file data...

2018-04-22 Thread cloud-fan
Github user cloud-fan commented on a diff in the pull request: https://github.com/apache/spark/pull/21123#discussion_r183274704 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/FileDataSourceV2.scala --- @@ -0,0 +1,43 @@ +/* + * Licensed to t

[GitHub] spark pull request #21123: [SPARK-24045][SQL]Create base class for file data...

2018-04-22 Thread cloud-fan
Github user cloud-fan commented on a diff in the pull request: https://github.com/apache/spark/pull/21123#discussion_r183274730 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/FileDataSourceV2.scala --- @@ -0,0 +1,43 @@ +/* + * Licensed to t

[GitHub] spark issue #21033: [SPARK-19320][MESOS]allow specifying a hard limit on num...

2018-04-22 Thread felixcheung
Github user felixcheung commented on the issue: https://github.com/apache/spark/pull/21033 ping @yanji84 --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h.

[GitHub] spark issue #20940: [SPARK-23429][CORE] Add executor memory metrics to heart...

2018-04-22 Thread felixcheung
Github user felixcheung commented on the issue: https://github.com/apache/spark/pull/20940 Jenkins, retest this please --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mai

[GitHub] spark issue #20940: [SPARK-23429][CORE] Add executor memory metrics to heart...

2018-04-22 Thread felixcheung
Github user felixcheung commented on the issue: https://github.com/apache/spark/pull/20940 @edwinalu sorry it looks like there are conflicts, we would need to rebase --- - To unsubscribe, e-mail: reviews-unsubscr...

[GitHub] spark issue #20940: [SPARK-23429][CORE] Add executor memory metrics to heart...

2018-04-22 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/20940 **[Test build #89697 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/89697/testReport)** for PR 20940 at commit [`ae8a388`](https://github.com/apache/spark/commit/ae

[GitHub] spark issue #21124: [SPARK-23004][SS] Ensure StateStore.commit is called onl...

2018-04-22 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21124 **[Test build #89694 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/89694/testReport)** for PR 21124 at commit [`304498e`](https://github.com/apache/spark/commit/3

[GitHub] spark issue #21124: [SPARK-23004][SS] Ensure StateStore.commit is called onl...

2018-04-22 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21124 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional comma

[GitHub] spark issue #21124: [SPARK-23004][SS] Ensure StateStore.commit is called onl...

2018-04-22 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21124 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/89694/ Test PASSed. ---

[GitHub] spark issue #21125: [Spark-24024] Fix poisson deviance calculations in GLM t...

2018-04-22 Thread dbtsai
Github user dbtsai commented on the issue: https://github.com/apache/spark/pull/21125 Jenkins, please test this. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: rev

[GitHub] spark issue #20980: [SPARK-23589][SQL] ExternalMapToCatalyst should support ...

2018-04-22 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/20980 **[Test build #89698 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/89698/testReport)** for PR 20980 at commit [`eaef6b3`](https://github.com/apache/spark/commit/ea

[GitHub] spark issue #21125: [Spark-24024] Fix poisson deviance calculations in GLM t...

2018-04-22 Thread dbtsai
Github user dbtsai commented on the issue: https://github.com/apache/spark/pull/21125 ok to test --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.

[GitHub] spark issue #20980: [SPARK-23589][SQL] ExternalMapToCatalyst should support ...

2018-04-22 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/20980 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution/2571/ Tes

[GitHub] spark issue #20980: [SPARK-23589][SQL] ExternalMapToCatalyst should support ...

2018-04-22 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/20980 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional comma

[GitHub] spark issue #21125: [Spark-24024] Fix poisson deviance calculations in GLM t...

2018-04-22 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21125 **[Test build #89699 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/89699/testReport)** for PR 21125 at commit [`3c6a4da`](https://github.com/apache/spark/commit/3c

[GitHub] spark issue #21083: [SPARK-23564][SQL] infer additional filters from constra...

2018-04-22 Thread cloud-fan
Github user cloud-fan commented on the issue: https://github.com/apache/spark/pull/21083 retest this please --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-

[GitHub] spark issue #21100: [SPARK-24012][SQL] Union of map and other compatible col...

2018-04-22 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21100 **[Test build #89700 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/89700/testReport)** for PR 21100 at commit [`0845739`](https://github.com/apache/spark/commit/08

[GitHub] spark issue #21083: [SPARK-23564][SQL] infer additional filters from constra...

2018-04-22 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21083 **[Test build #89701 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/89701/testReport)** for PR 21083 at commit [`787cddf`](https://github.com/apache/spark/commit/78

[GitHub] spark issue #21083: [SPARK-23564][SQL] infer additional filters from constra...

2018-04-22 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21083 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution/2572/ Tes

[GitHub] spark issue #21083: [SPARK-23564][SQL] infer additional filters from constra...

2018-04-22 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21083 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional comma

[GitHub] spark issue #21118: SPARK-23325: Use InternalRow when reading with DataSourc...

2018-04-22 Thread dongjoon-hyun
Github user dongjoon-hyun commented on the issue: https://github.com/apache/spark/pull/21118 @rdblue . Could you fix the remaining `KafkaMicroBatchSourceSuite.scala`, too? ```scala [error] /home/jenkins/workspace/SparkPullRequestBuilder@2/external/kafka-0-10-sql/src/test/

[GitHub] spark pull request #21125: [Spark-24024] Fix poisson deviance calculations i...

2018-04-22 Thread dbtsai
Github user dbtsai commented on a diff in the pull request: https://github.com/apache/spark/pull/21125#discussion_r183283144 --- Diff: mllib/src/test/scala/org/apache/spark/ml/regression/GeneralizedLinearRegressionSuite.scala --- @@ -507,7 +507,8 @@ class GeneralizedLinearRegressi

[GitHub] spark pull request #21125: [Spark-24024] Fix poisson deviance calculations i...

2018-04-22 Thread dbtsai
Github user dbtsai commented on a diff in the pull request: https://github.com/apache/spark/pull/21125#discussion_r183282476 --- Diff: mllib/src/test/scala/org/apache/spark/ml/regression/GeneralizedLinearRegressionSuite.scala --- @@ -495,8 +495,8 @@ class GeneralizedLinearRegressi

[GitHub] spark pull request #21125: [Spark-24024] Fix poisson deviance calculations i...

2018-04-22 Thread dbtsai
Github user dbtsai commented on a diff in the pull request: https://github.com/apache/spark/pull/21125#discussion_r183283116 --- Diff: mllib/src/test/scala/org/apache/spark/ml/regression/GeneralizedLinearRegressionSuite.scala --- @@ -495,8 +495,8 @@ class GeneralizedLinearRegressi

[GitHub] spark pull request #21125: [Spark-24024] Fix poisson deviance calculations i...

2018-04-22 Thread dbtsai
Github user dbtsai commented on a diff in the pull request: https://github.com/apache/spark/pull/21125#discussion_r183282075 --- Diff: mllib/src/main/scala/org/apache/spark/ml/regression/GeneralizedLinearRegression.scala --- @@ -782,8 +782,12 @@ object GeneralizedLinearRegression

[GitHub] spark pull request #21126: [SPARK-24050][SS] Calculate input / processing ra...

2018-04-22 Thread tdas
GitHub user tdas opened a pull request: https://github.com/apache/spark/pull/21126 [SPARK-24050][SS] Calculate input / processing rates correctly for DataSourceV2 streaming sources ## What changes were proposed in this pull request? In some streaming queries, the input and

[GitHub] spark issue #21126: [SPARK-24050][SS] Calculate input / processing rates cor...

2018-04-22 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21126 **[Test build #89702 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/89702/testReport)** for PR 21126 at commit [`d485db8`](https://github.com/apache/spark/commit/d4

[GitHub] spark issue #21126: [SPARK-24050][SS] Calculate input / processing rates cor...

2018-04-22 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21126 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional comma

[GitHub] spark issue #21126: [SPARK-24050][SS] Calculate input / processing rates cor...

2018-04-22 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21126 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution/2573/ Tes

[GitHub] spark pull request #21018: [SPARK-23880][SQL] Do not trigger any jobs for ca...

2018-04-22 Thread maropu
Github user maropu commented on a diff in the pull request: https://github.com/apache/spark/pull/21018#discussion_r183288666 --- Diff: sql/core/src/test/scala/org/apache/spark/sql/CachedTableSuite.scala --- @@ -794,4 +794,17 @@ class CachedTableSuite extends QueryTest with SQLTest

[GitHub] spark pull request #21018: [SPARK-23880][SQL] Do not trigger any jobs for ca...

2018-04-22 Thread maropu
Github user maropu commented on a diff in the pull request: https://github.com/apache/spark/pull/21018#discussion_r183288749 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/columnar/InMemoryRelation.scala --- @@ -55,56 +42,38 @@ object InMemoryRelation { priv

[GitHub] spark issue #21125: [Spark-24024] Fix poisson deviance calculations in GLM t...

2018-04-22 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21125 **[Test build #89699 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/89699/testReport)** for PR 21125 at commit [`3c6a4da`](https://github.com/apache/spark/commit/3

[GitHub] spark issue #21125: [Spark-24024] Fix poisson deviance calculations in GLM t...

2018-04-22 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21125 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional comma

[GitHub] spark issue #21125: [Spark-24024] Fix poisson deviance calculations in GLM t...

2018-04-22 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21125 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/89699/ Test PASSed. ---

<    1   2