[GitHub] spark issue #22192: [SPARK-24918][Core] Executor Plugin API

2018-09-10 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/22192 **[Test build #95862 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/95862/testReport)** for PR 22192 at commit [`447c5e5`](https://github.com/apache/spark/commit/4

[GitHub] spark issue #22192: [SPARK-24918][Core] Executor Plugin API

2018-09-10 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/22192 Merged build finished. Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional comma

[GitHub] spark issue #22376: [SPARK-25021][K8S][BACKPORT] Add spark.executor.pyspark....

2018-09-10 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/22376 **[Test build #95855 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/95855/testReport)** for PR 22376 at commit [`360ef32`](https://github.com/apache/spark/commit/3

[GitHub] spark issue #22375: [WIP][SPARK-25388][Test][SQL] Detect incorrect nullable ...

2018-09-10 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/22375 **[Test build #95861 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/95861/testReport)** for PR 22375 at commit [`51aa9d5`](https://github.com/apache/spark/commit/5

[GitHub] spark issue #22347: [SPARK-25353][SQL] executeTake in SparkPlan is modified ...

2018-09-10 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/22347 **[Test build #95859 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/95859/testReport)** for PR 22347 at commit [`a8f1481`](https://github.com/apache/spark/commit/a

[GitHub] spark issue #22375: [WIP][SPARK-25388][Test][SQL] Detect incorrect nullable ...

2018-09-10 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/22375 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/95861/ Test FAILed. ---

[GitHub] spark issue #22378: [SPARK-25389][SQL] INSERT OVERWRITE DIRECTORY STORED AS ...

2018-09-10 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/22378 **[Test build #95860 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/95860/testReport)** for PR 22378 at commit [`0242576`](https://github.com/apache/spark/commit/0

[GitHub] spark issue #22376: [SPARK-25021][K8S][BACKPORT] Add spark.executor.pyspark....

2018-09-10 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/22376 **[Test build #95856 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/95856/testReport)** for PR 22376 at commit [`4a0cffb`](https://github.com/apache/spark/commit/4

[GitHub] spark issue #22378: [SPARK-25389][SQL] INSERT OVERWRITE DIRECTORY STORED AS ...

2018-09-10 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/22378 Merged build finished. Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional comma

[GitHub] spark issue #22375: [WIP][SPARK-25388][Test][SQL] Detect incorrect nullable ...

2018-09-10 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/22375 Merged build finished. Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional comma

[GitHub] spark issue #22377: [SPARK-24849][SPARK-24911][SQL][FOLLOW-UP] Converting a ...

2018-09-10 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/22377 **[Test build #95858 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/95858/testReport)** for PR 22377 at commit [`97ec077`](https://github.com/apache/spark/commit/9

[GitHub] spark issue #22343: [SPARK-25391][SQL] Make behaviors consistent when conver...

2018-09-10 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/22343 **[Test build #95857 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/95857/testReport)** for PR 22343 at commit [`95673cd`](https://github.com/apache/spark/commit/9

[GitHub] spark issue #22192: [SPARK-24918][Core] Executor Plugin API

2018-09-10 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/22192 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/95862/ Test FAILed. ---

[GitHub] spark issue #22347: [SPARK-25353][SQL] executeTake in SparkPlan is modified ...

2018-09-10 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/22347 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/95859/ Test FAILed. ---

[GitHub] spark issue #22378: [SPARK-25389][SQL] INSERT OVERWRITE DIRECTORY STORED AS ...

2018-09-10 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/22378 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/95860/ Test FAILed. ---

[GitHub] spark issue #22347: [SPARK-25353][SQL] executeTake in SparkPlan is modified ...

2018-09-10 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/22347 Merged build finished. Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional comma

[GitHub] spark issue #22343: [SPARK-25391][SQL] Make behaviors consistent when conver...

2018-09-10 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/22343 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/95857/ Test FAILed. ---

[GitHub] spark issue #22376: [SPARK-25021][K8S][BACKPORT] Add spark.executor.pyspark....

2018-09-10 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/22376 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/95856/ Test FAILed. ---

[GitHub] spark issue #22376: [SPARK-25021][K8S][BACKPORT] Add spark.executor.pyspark....

2018-09-10 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/22376 Merged build finished. Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional comma

[GitHub] spark issue #22376: [SPARK-25021][K8S][BACKPORT] Add spark.executor.pyspark....

2018-09-10 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/22376 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/95855/ Test FAILed. ---

[GitHub] spark issue #22377: [SPARK-24849][SPARK-24911][SQL][FOLLOW-UP] Converting a ...

2018-09-10 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/22377 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/95858/ Test FAILed. ---

[GitHub] spark issue #22343: [SPARK-25391][SQL] Make behaviors consistent when conver...

2018-09-10 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/22343 Merged build finished. Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional comma

[GitHub] spark issue #22377: [SPARK-24849][SPARK-24911][SQL][FOLLOW-UP] Converting a ...

2018-09-10 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/22377 Merged build finished. Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional comma

[GitHub] spark issue #22376: [SPARK-25021][K8S][BACKPORT] Add spark.executor.pyspark....

2018-09-10 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/22376 Merged build finished. Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional comma

[GitHub] spark pull request #22343: [SPARK-25391][SQL] Make behaviors consistent when...

2018-09-10 Thread HyukjinKwon
Github user HyukjinKwon commented on a diff in the pull request: https://github.com/apache/spark/pull/22343#discussion_r216216422 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/parquet/ParquetOptions.scala --- @@ -69,12 +69,25 @@ class ParquetOptions(

[GitHub] spark issue #22377: [SPARK-24849][SPARK-24911][SQL][FOLLOW-UP] Converting a ...

2018-09-10 Thread HyukjinKwon
Github user HyukjinKwon commented on the issue: https://github.com/apache/spark/pull/22377 retest this please --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: review

[GitHub] spark pull request #22370: don't link to deprecated function

2018-09-10 Thread HyukjinKwon
Github user HyukjinKwon commented on a diff in the pull request: https://github.com/apache/spark/pull/22370#discussion_r216217235 --- Diff: R/pkg/R/catalog.R --- @@ -69,7 +69,6 @@ createExternalTable <- function(x, ...) { #' @param ... additional named parameters as options for

[GitHub] spark issue #21968: [SPARK-24999][SQL]Reduce unnecessary 'new' memory operat...

2018-09-10 Thread cloud-fan
Github user cloud-fan commented on the issue: https://github.com/apache/spark/pull/21968 thanks, merging to master! --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail:

[GitHub] spark issue #22377: [SPARK-24849][SPARK-24911][SQL][FOLLOW-UP] Converting a ...

2018-09-10 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/22377 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/2968/

[GitHub] spark issue #22377: [SPARK-24849][SPARK-24911][SQL][FOLLOW-UP] Converting a ...

2018-09-10 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/22377 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional comma

[GitHub] spark issue #22377: [SPARK-24849][SPARK-24911][SQL][FOLLOW-UP] Converting a ...

2018-09-10 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/22377 **[Test build #95863 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/95863/testReport)** for PR 22377 at commit [`97ec077`](https://github.com/apache/spark/commit/97

[GitHub] spark pull request #21968: [SPARK-24999][SQL]Reduce unnecessary 'new' memory...

2018-09-10 Thread asfgit
Github user asfgit closed the pull request at: https://github.com/apache/spark/pull/21968 --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #22343: [SPARK-25391][SQL] Make behaviors consistent when conver...

2018-09-10 Thread cloud-fan
Github user cloud-fan commented on the issue: https://github.com/apache/spark/pull/22343 retest this please --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-

[GitHub] spark pull request #22343: [SPARK-25391][SQL] Make behaviors consistent when...

2018-09-10 Thread cloud-fan
Github user cloud-fan commented on a diff in the pull request: https://github.com/apache/spark/pull/22343#discussion_r216218261 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/parquet/ParquetOptions.scala --- @@ -69,12 +69,25 @@ class ParquetOptions(

[GitHub] spark pull request #22357: [SPARK-25363][SQL] Fix schema pruning in where cl...

2018-09-10 Thread HyukjinKwon
Github user HyukjinKwon commented on a diff in the pull request: https://github.com/apache/spark/pull/22357#discussion_r216218409 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/parquet/ParquetSchemaPruning.scala --- @@ -196,6 +201,9 @@ private[sql] ob

[GitHub] spark issue #22343: [SPARK-25391][SQL] Make behaviors consistent when conver...

2018-09-10 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/22343 **[Test build #95864 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/95864/testReport)** for PR 22343 at commit [`95673cd`](https://github.com/apache/spark/commit/95

[GitHub] spark pull request #22357: [SPARK-25363][SQL] Fix schema pruning in where cl...

2018-09-10 Thread HyukjinKwon
Github user HyukjinKwon commented on a diff in the pull request: https://github.com/apache/spark/pull/22357#discussion_r216218951 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/parquet/ParquetSchemaPruning.scala --- @@ -156,7 +161,7 @@ private[sql] ob

[GitHub] spark issue #22378: [SPARK-25389][SQL] INSERT OVERWRITE DIRECTORY STORED AS ...

2018-09-10 Thread HyukjinKwon
Github user HyukjinKwon commented on the issue: https://github.com/apache/spark/pull/22378 retest this please --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: review

[GitHub] spark pull request #22370: don't link to deprecated function

2018-09-10 Thread MichaelChirico
Github user MichaelChirico commented on a diff in the pull request: https://github.com/apache/spark/pull/22370#discussion_r216220254 --- Diff: R/pkg/R/catalog.R --- @@ -69,7 +69,6 @@ createExternalTable <- function(x, ...) { #' @param ... additional named parameters as options

[GitHub] spark issue #22378: [SPARK-25389][SQL] INSERT OVERWRITE DIRECTORY STORED AS ...

2018-09-10 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/22378 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional comma

[GitHub] spark issue #22378: [SPARK-25389][SQL] INSERT OVERWRITE DIRECTORY STORED AS ...

2018-09-10 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/22378 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/2969/

[GitHub] spark issue #22370: don't link to deprecated function

2018-09-10 Thread MichaelChirico
Github user MichaelChirico commented on the issue: https://github.com/apache/spark/pull/22370 @felixcheung I disagree... what's the point of deprecation if it's going to keep being considered as a co-equal function in the eyes of documentation? If the function is being depreca

[GitHub] spark issue #22378: [SPARK-25389][SQL] INSERT OVERWRITE DIRECTORY STORED AS ...

2018-09-10 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/22378 **[Test build #95865 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/95865/testReport)** for PR 22378 at commit [`0242576`](https://github.com/apache/spark/commit/02

[GitHub] spark pull request #22370: don't link to deprecated function

2018-09-10 Thread MichaelChirico
Github user MichaelChirico commented on a diff in the pull request: https://github.com/apache/spark/pull/22370#discussion_r216222094 --- Diff: R/pkg/R/catalog.R --- @@ -69,7 +69,6 @@ createExternalTable <- function(x, ...) { #' @param ... additional named parameters as options

[GitHub] spark pull request #22370: don't link to deprecated function

2018-09-10 Thread MichaelChirico
Github user MichaelChirico commented on a diff in the pull request: https://github.com/apache/spark/pull/22370#discussion_r216222159 --- Diff: R/pkg/R/catalog.R --- @@ -69,7 +69,6 @@ createExternalTable <- function(x, ...) { #' @param ... additional named parameters as options

[GitHub] spark issue #21596: [SPARK-24601] Bump Jackson version

2018-09-10 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21596 **[Test build #95866 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/95866/testReport)** for PR 21596 at commit [`4f9a9c5`](https://github.com/apache/spark/commit/4f

[GitHub] spark issue #21596: [SPARK-24601] Bump Jackson version

2018-09-10 Thread Fokko
Github user Fokko commented on the issue: https://github.com/apache/spark/pull/21596 @HyukjinKwon I've rebased onto master since the Spark 2.4 branch has been cut. --- - To unsubscribe, e-mail: reviews-unsubscr...@s

[GitHub] spark pull request #22364: [SPARK-25379][SQL] Improve AttributeSet and Colum...

2018-09-10 Thread mgaido91
Github user mgaido91 commented on a diff in the pull request: https://github.com/apache/spark/pull/22364#discussion_r216224645 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/AttributeSet.scala --- @@ -39,10 +41,15 @@ object AttributeSet {

[GitHub] spark issue #22284: [SPARK-25278][SQL] Avoid duplicated Exec nodes when the ...

2018-09-10 Thread mgaido91
Github user mgaido91 commented on the issue: https://github.com/apache/spark/pull/22284 kindly ping @cloud-fan --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: revi

[GitHub] spark pull request #20999: [SPARK-14922][SPARK-17732][SPARK-23866][SQL] Supp...

2018-09-10 Thread mgaido91
Github user mgaido91 commented on a diff in the pull request: https://github.com/apache/spark/pull/20999#discussion_r216227938 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/parser/AstBuilder.scala --- @@ -293,6 +293,28 @@ class AstBuilder(conf: SQLConf) exten

[GitHub] spark pull request #22370: don't link to deprecated function

2018-09-10 Thread HyukjinKwon
Github user HyukjinKwon commented on a diff in the pull request: https://github.com/apache/spark/pull/22370#discussion_r216229836 --- Diff: R/pkg/R/catalog.R --- @@ -69,7 +69,6 @@ createExternalTable <- function(x, ...) { #' @param ... additional named parameters as options for

[GitHub] spark issue #22373: [SPARK-25371][ML] VectorAssembler should not fail with e...

2018-09-10 Thread HyukjinKwon
Github user HyukjinKwon commented on the issue: https://github.com/apache/spark/pull/22373 @mgaido91, BTW are you sure SPARK-21281 introduced that behaviour change? Before: ``` scala> import org.apache.spark.sql.functions.struct import org.apache.spark.sql.fun

[GitHub] spark pull request #22365: [SPARK-25381][SQL] Stratified sampling by Column ...

2018-09-10 Thread HyukjinKwon
Github user HyukjinKwon commented on a diff in the pull request: https://github.com/apache/spark/pull/22365#discussion_r216233066 --- Diff: python/pyspark/sql/dataframe.py --- @@ -880,18 +880,23 @@ def sampleBy(self, col, fractions, seed=None): | 0|5|

[GitHub] spark pull request #22365: [SPARK-25381][SQL] Stratified sampling by Column ...

2018-09-10 Thread HyukjinKwon
Github user HyukjinKwon commented on a diff in the pull request: https://github.com/apache/spark/pull/22365#discussion_r216233575 --- Diff: python/pyspark/sql/dataframe.py --- @@ -880,18 +880,23 @@ def sampleBy(self, col, fractions, seed=None): | 0|5|

[GitHub] spark issue #22316: [SPARK-25048][SQL] Pivoting by multiple columns in Scala...

2018-09-10 Thread HyukjinKwon
Github user HyukjinKwon commented on the issue: https://github.com/apache/spark/pull/22316 Seems fine to me. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews

[GitHub] spark issue #22237: [SPARK-25243][SQL] Use FailureSafeParser in from_json

2018-09-10 Thread HyukjinKwon
Github user HyukjinKwon commented on the issue: https://github.com/apache/spark/pull/22237 Will take a look soon. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: re

[GitHub] spark issue #22237: [SPARK-25243][SQL] Use FailureSafeParser in from_json

2018-09-10 Thread HyukjinKwon
Github user HyukjinKwon commented on the issue: https://github.com/apache/spark/pull/22237 retest this please --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: review

[GitHub] spark issue #22372: [SPARK-25385][BUILD] Upgrade Hadoop 3.1 jackson version ...

2018-09-10 Thread jerryshao
Github user jerryshao commented on the issue: https://github.com/apache/spark/pull/22372 Jackson version below 2.9.5 has CVE issues, I would suggest to upgrade to 2.9.6 as #21596 did. --- - To unsubscribe, e-mail: r

[GitHub] spark issue #22237: [SPARK-25243][SQL] Use FailureSafeParser in from_json

2018-09-10 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/22237 **[Test build #95867 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/95867/testReport)** for PR 22237 at commit [`a433388`](https://github.com/apache/spark/commit/a4

[GitHub] spark issue #22373: [SPARK-25371][ML] VectorAssembler should not fail with e...

2018-09-10 Thread mgaido91
Github user mgaido91 commented on the issue: https://github.com/apache/spark/pull/22373 @HyukjinKwon I am sure, since I tried removing the added check and the UT I added here passed. --- - To unsubscribe, e-mail: re

[GitHub] spark issue #22357: [SPARK-25363][SQL] Fix schema pruning in where clause by...

2018-09-10 Thread viirya
Github user viirya commented on the issue: https://github.com/apache/spark/pull/22357 Thanks @dbtsai and @HyukjinKwon. Your comments are addressed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org F

[GitHub] spark issue #22378: [SPARK-25389][SQL] INSERT OVERWRITE DIRECTORY STORED AS ...

2018-09-10 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/22378 **[Test build #95865 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/95865/testReport)** for PR 22378 at commit [`0242576`](https://github.com/apache/spark/commit/0

[GitHub] spark pull request #22357: [SPARK-25363][SQL] Fix schema pruning in where cl...

2018-09-10 Thread viirya
Github user viirya commented on a diff in the pull request: https://github.com/apache/spark/pull/22357#discussion_r216244620 --- Diff: sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/ParquetSchemaPruningSuite.scala --- @@ -155,6 +161,47 @@ class ParquetS

[GitHub] spark issue #22378: [SPARK-25389][SQL] INSERT OVERWRITE DIRECTORY STORED AS ...

2018-09-10 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/22378 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/95865/ Test FAILed. ---

[GitHub] spark issue #22378: [SPARK-25389][SQL] INSERT OVERWRITE DIRECTORY STORED AS ...

2018-09-10 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/22378 Merged build finished. Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional comma

[GitHub] spark issue #22357: [SPARK-25363][SQL] Fix schema pruning in where clause by...

2018-09-10 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/22357 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/2970/

[GitHub] spark issue #22357: [SPARK-25363][SQL] Fix schema pruning in where clause by...

2018-09-10 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/22357 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional comma

[GitHub] spark issue #22357: [SPARK-25363][SQL] Fix schema pruning in where clause by...

2018-09-10 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/22357 **[Test build #95868 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/95868/testReport)** for PR 22357 at commit [`9168884`](https://github.com/apache/spark/commit/91

[GitHub] spark pull request #22227: [SPARK-25202] [SQL] Implements split with limit s...

2018-09-10 Thread phegstrom
Github user phegstrom commented on a diff in the pull request: https://github.com/apache/spark/pull/7#discussion_r216247365 --- Diff: R/pkg/R/functions.R --- @@ -3404,19 +3404,24 @@ setMethod("collect_set", #' Equivalent to \code{split} SQL function. #' #' @rdname

[GitHub] spark issue #22353: [SPARK-25357][SQL] Abbreviated simpleString in DataSourc...

2018-09-10 Thread LantaoJin
Github user LantaoJin commented on the issue: https://github.com/apache/spark/pull/22353 Thanks @dongjoon-hyun . That would be a problem. Seems setting to 200 or 500 are cause a limited regression on hover text. Hard code to 500 shows: https://user-images.githubusercontent

[GitHub] spark pull request #22379: [SPARK-25393][SQL] Adding new function from_csv()

2018-09-10 Thread MaxGekk
GitHub user MaxGekk opened a pull request: https://github.com/apache/spark/pull/22379 [SPARK-25393][SQL] Adding new function from_csv() ## What changes were proposed in this pull request? The PR adds new function `from_csv()` similar to `from_json()` to parse columns with C

[GitHub] spark issue #22372: [SPARK-25385][BUILD] Upgrade Hadoop 3.1 jackson version ...

2018-09-10 Thread wangyum
Github user wangyum commented on the issue: https://github.com/apache/spark/pull/22372 I did a simple test for 2.9.6. It works well. But that pr for 3.0. It means that a simple test on branch 2.4 will fail: ```scala scala> spark.range(10).write.parquet("/tmp/spark/parquet")

[GitHub] spark issue #21669: [SPARK-23257][K8S] Kerberos Support for Spark on K8S

2018-09-10 Thread rvesse
Github user rvesse commented on the issue: https://github.com/apache/spark/pull/21669 @vanzin I think in the current implementation of this PR the Kerberos login is happening inside the driver pod which is running inside the K8S cluster. The old design from the Spark on K8S fo

[GitHub] spark issue #22379: [SPARK-25393][SQL] Adding new function from_csv()

2018-09-10 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/22379 Can one of the admins verify this patch? --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark issue #22379: [SPARK-25393][SQL] Adding new function from_csv()

2018-09-10 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/22379 Can one of the admins verify this patch? --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark pull request #22357: [SPARK-25363][SQL] Fix schema pruning in where cl...

2018-09-10 Thread viirya
Github user viirya commented on a diff in the pull request: https://github.com/apache/spark/pull/22357#discussion_r216255549 --- Diff: sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/ParquetSchemaPruningSuite.scala --- @@ -155,6 +161,47 @@ class ParquetS

[GitHub] spark pull request #22357: [SPARK-25363][SQL] Fix schema pruning in where cl...

2018-09-10 Thread viirya
Github user viirya commented on a diff in the pull request: https://github.com/apache/spark/pull/22357#discussion_r216256434 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/parquet/ParquetSchemaPruning.scala --- @@ -199,6 +209,15 @@ private[sql] object

[GitHub] spark pull request #22341: [SPARK-24889][Core] Update block info when unpers...

2018-09-10 Thread viirya
Github user viirya commented on a diff in the pull request: https://github.com/apache/spark/pull/22341#discussion_r216257678 --- Diff: core/src/main/scala/org/apache/spark/status/AppStatusListener.scala --- @@ -646,7 +647,47 @@ private[spark] class AppStatusListener( }

[GitHub] spark issue #22353: [SPARK-25357][SQL] Abbreviated simpleString in DataSourc...

2018-09-10 Thread LantaoJin
Github user LantaoJin commented on the issue: https://github.com/apache/spark/pull/22353 The purpose is logging meta info like file input file path to event log. So I revert the changes about simpleString and add back the metadata to SparkPlanInfo interface. This change will log metad

[GitHub] spark issue #22372: [SPARK-25385][BUILD] Upgrade Hadoop 3.1 jackson version ...

2018-09-10 Thread jerryshao
Github user jerryshao commented on the issue: https://github.com/apache/spark/pull/22372 Do we officially support hadoop3 in branch 2.4? If branch 2.4 doesn't target to support Hadoop3 and this fix is only for Hadoop3, then I don't think it is meaningful to have this fix. --- -

[GitHub] spark issue #22372: [SPARK-25385][BUILD] Upgrade Hadoop 3.1 jackson version ...

2018-09-10 Thread jerryshao
Github user jerryshao commented on the issue: https://github.com/apache/spark/pull/22372 Btw, I don't think we can run current Spark with Hadoop 3.1 without any change. --- - To unsubscribe, e-mail: reviews-unsubscr

[GitHub] spark issue #22237: [SPARK-25243][SQL] Use FailureSafeParser in from_json

2018-09-10 Thread MaxGekk
Github user MaxGekk commented on the issue: https://github.com/apache/spark/pull/22237 > Will take a look soon. @HyukjinKwon Thank you. Waiting for you feedback. --- - To unsubscribe, e-mail: reviews-unsubsc

[GitHub] spark issue #21968: [SPARK-24999][SQL]Reduce unnecessary 'new' memory operat...

2018-09-10 Thread heary-cao
Github user heary-cao commented on the issue: https://github.com/apache/spark/pull/21968 @cloud-fan thanks. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews

[GitHub] spark pull request #22372: [SPARK-25385][BUILD] Upgrade Hadoop 3.1 jackson v...

2018-09-10 Thread wangyum
Github user wangyum closed the pull request at: https://github.com/apache/spark/pull/22372 --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #22357: [SPARK-25363][SQL] Fix schema pruning in where clause by...

2018-09-10 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/22357 **[Test build #95868 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/95868/testReport)** for PR 22357 at commit [`9168884`](https://github.com/apache/spark/commit/9

[GitHub] spark issue #22357: [SPARK-25363][SQL] Fix schema pruning in where clause by...

2018-09-10 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/22357 Merged build finished. Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional comma

[GitHub] spark issue #22357: [SPARK-25363][SQL] Fix schema pruning in where clause by...

2018-09-10 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/22357 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/95868/ Test FAILed. ---

[GitHub] spark issue #22373: [SPARK-25371][ML] VectorAssembler should not fail with e...

2018-09-10 Thread mgaido91
Github user mgaido91 commented on the issue: https://github.com/apache/spark/pull/22373 cc @cloud-fan @jerryshao despite a very minor one, this can be considered a regression, so may be considered as a blocker for 2.4/2.3.2? --- --

[GitHub] spark issue #22347: [SPARK-25353][SQL] executeTake in SparkPlan is modified ...

2018-09-10 Thread Dooyoung-Hwang
Github user Dooyoung-Hwang commented on the issue: https://github.com/apache/spark/pull/22347 I tested in my local PC. 3.3 GHz Intel Core i5, and selected 400,000 rows x 25 times. I took a total execution time between decodeUnsafeRows. My tested data is skewed, so gathered rows

[GitHub] spark issue #22348: [SPARK-25354][SQL] Reduce unneeded operation in nextKeyV...

2018-09-10 Thread SongYadong
Github user SongYadong commented on the issue: https://github.com/apache/spark/pull/22348 Could I ask you to review this PR if you have the time? cc @dongjoon-hyun @cloud-fan --- - To unsubscribe, e-mail: reviews-u

[GitHub] spark issue #22377: [SPARK-24849][SPARK-24911][SQL][FOLLOW-UP] Converting a ...

2018-09-10 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/22377 **[Test build #95863 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/95863/testReport)** for PR 22377 at commit [`97ec077`](https://github.com/apache/spark/commit/9

[GitHub] spark issue #22377: [SPARK-24849][SPARK-24911][SQL][FOLLOW-UP] Converting a ...

2018-09-10 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/22377 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional comma

[GitHub] spark issue #22377: [SPARK-24849][SPARK-24911][SQL][FOLLOW-UP] Converting a ...

2018-09-10 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/22377 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/95863/ Test PASSed. ---

[GitHub] spark issue #22343: [SPARK-25391][SQL] Make behaviors consistent when conver...

2018-09-10 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/22343 **[Test build #95864 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/95864/testReport)** for PR 22343 at commit [`95673cd`](https://github.com/apache/spark/commit/9

[GitHub] spark issue #22343: [SPARK-25391][SQL] Make behaviors consistent when conver...

2018-09-10 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/22343 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional comma

[GitHub] spark issue #22343: [SPARK-25391][SQL] Make behaviors consistent when conver...

2018-09-10 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/22343 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/95864/ Test PASSed. ---

[GitHub] spark issue #22357: [SPARK-25363][SQL] Fix schema pruning in where clause by...

2018-09-10 Thread viirya
Github user viirya commented on the issue: https://github.com/apache/spark/pull/22357 retest this please. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h.

[GitHub] spark issue #22377: [SPARK-24849][SPARK-24911][SQL][FOLLOW-UP] Converting a ...

2018-09-10 Thread cloud-fan
Github user cloud-fan commented on the issue: https://github.com/apache/spark/pull/22377 thanks, merging to master/2.4! --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-ma

[GitHub] spark issue #21596: [SPARK-24601] Bump Jackson version

2018-09-10 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21596 **[Test build #95866 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/95866/testReport)** for PR 21596 at commit [`4f9a9c5`](https://github.com/apache/spark/commit/4

[GitHub] spark issue #21596: [SPARK-24601] Bump Jackson version

2018-09-10 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21596 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/95866/ Test FAILed. ---

  1   2   3   4   5   6   7   >