[GitHub] spark pull request #21067: [SPARK-23980][K8S] Resilient Spark driver on Kube...

2018-06-11 Thread mccheah
Github user mccheah commented on a diff in the pull request: https://github.com/apache/spark/pull/21067#discussion_r194558081 --- Diff: resource-managers/kubernetes/core/src/main/scala/org/apache/spark/scheduler/cluster/k8s/KubernetesClusterSchedulerBackend.scala --- @@ -59,16

[GitHub] spark pull request #21452: [MINOR][CORE] Log committer class used by HadoopM...

2018-06-11 Thread asfgit
Github user asfgit closed the pull request at: https://github.com/apache/spark/pull/21452 --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #21532: [SPARK-24524][SQL]Improve aggregateMetrics: reduc...

2018-06-11 Thread gengliangwang
GitHub user gengliangwang opened a pull request: https://github.com/apache/spark/pull/21532 [SPARK-24524][SQL]Improve aggregateMetrics: reduce memory usage and number of loops ## What changes were proposed in this pull request? The function `aggregateMetrics` process

[GitHub] spark issue #21452: [MINOR][CORE] Log committer class used by HadoopMapRedCo...

2018-06-11 Thread srowen
Github user srowen commented on the issue: https://github.com/apache/spark/pull/21452 Merged to master --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail:

[GitHub] spark issue #21496: docs: fix typo

2018-06-11 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21496 **[Test build #4199 has started](https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/4199/testReport)** for PR 21496 at commit

[GitHub] spark issue #21205: [SPARK-24134][Docs]A missing full-stop in doc "Tuning Sp...

2018-06-11 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21205 **[Test build #4198 has started](https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/4198/testReport)** for PR 21205 at commit

[GitHub] spark issue #21517: Testing k9s change - please ignore (13)

2018-06-11 Thread ssuchter
Github user ssuchter commented on the issue: https://github.com/apache/spark/pull/21517 retest this please --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail:

[GitHub] spark issue #21528: [SPARK-24520] Double braces in documentations

2018-06-11 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21528 **[Test build #4197 has started](https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/4197/testReport)** for PR 21528 at commit

[GitHub] spark issue #21527: [SPARK-24519] MapStatus has 2000 hardcoded

2018-06-11 Thread markhamstra
Github user markhamstra commented on the issue: https://github.com/apache/spark/pull/21527 @tgravescs If there is value in making it configurable, that is all fine and good. My argument is against making it configurable just for the sake of making it configurable. If there is more

[GitHub] spark issue #21517: Testing k9s change - please ignore (13)

2018-06-11 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21517 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/20/

[GitHub] spark issue #21517: Testing k9s change - please ignore (13)

2018-06-11 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21517 Merged build finished. Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark issue #21517: Testing k9s change - please ignore (13)

2018-06-11 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21517 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution/3910/

[GitHub] spark issue #21517: Testing k9s change - please ignore (13)

2018-06-11 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21517 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark issue #20636: [SPARK-23415][SQL][TEST] Make behavior of BufferHolderSp...

2018-06-11 Thread kiszk
Github user kiszk commented on the issue: https://github.com/apache/spark/pull/20636 Sure, I will update this --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail:

[GitHub] spark issue #21527: [SPARK-24519] MapStatus has 2000 hardcoded

2018-06-11 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21527 **[Test build #91675 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/91675/testReport)** for PR 21527 at commit

[GitHub] spark issue #21438: [SPARK-24398] [SQL] Improve SQLAppStatusListener.aggrega...

2018-06-11 Thread gatorsmile
Github user gatorsmile commented on the issue: https://github.com/apache/spark/pull/21438 @felixcheung `metricIds` is not needed. We can get rid of it by using `metricTypes ` --- - To unsubscribe, e-mail:

[GitHub] spark issue #21517: Testing k9s change - please ignore (13)

2018-06-11 Thread ssuchter
Github user ssuchter commented on the issue: https://github.com/apache/spark/pull/21517 retest this please --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail:

[GitHub] spark issue #21515: [SPARK-24372][build] Add scripts to help with preparing ...

2018-06-11 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21515 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/19/

[GitHub] spark issue #21515: [SPARK-24372][build] Add scripts to help with preparing ...

2018-06-11 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21515 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark issue #21531: [SPARK-24521][SQL] Fix ineffective test in CachedTableSu...

2018-06-11 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21531 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark issue #21531: [SPARK-24521][SQL] Fix ineffective test in CachedTableSu...

2018-06-11 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21531 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/18/

[GitHub] spark issue #21527: [SPARK-24519] MapStatus has 2000 hardcoded

2018-06-11 Thread tgravescs
Github user tgravescs commented on the issue: https://github.com/apache/spark/pull/21527 yes it is debatable, but with it being configurable it gives you the option to change. the only other way is to change the # of partitions which could be more costly. There have been various

[GitHub] spark issue #21517: Testing k9s change - please ignore (13)

2018-06-11 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21517 Merged build finished. Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark issue #21517: Testing k9s change - please ignore (13)

2018-06-11 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21517 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/17/

[GitHub] spark issue #21508: [SPARK-24488] [SQL] Fix issue when generator is aliased ...

2018-06-11 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21508 **[Test build #91674 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/91674/testReport)** for PR 21508 at commit

[GitHub] spark pull request #21508: [SPARK-24488] [SQL] Fix issue when generator is a...

2018-06-11 Thread bkrieger
Github user bkrieger commented on a diff in the pull request: https://github.com/apache/spark/pull/21508#discussion_r194539275 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/Analyzer.scala --- @@ -1568,11 +1568,13 @@ class Analyzer(

[GitHub] spark pull request #21508: [SPARK-24488] [SQL] Fix issue when generator is a...

2018-06-11 Thread bkrieger
Github user bkrieger commented on a diff in the pull request: https://github.com/apache/spark/pull/21508#discussion_r194539357 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/Analyzer.scala --- @@ -1568,11 +1568,13 @@ class Analyzer(

[GitHub] spark pull request #20708: [SPARK-21209][MLLLIB] Implement Incremental PCA a...

2018-06-11 Thread sandecho
Github user sandecho closed the pull request at: https://github.com/apache/spark/pull/20708 --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #21515: [SPARK-24372][build] Add scripts to help with preparing ...

2018-06-11 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21515 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark issue #21515: [SPARK-24372][build] Add scripts to help with preparing ...

2018-06-11 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21515 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution/3909/

[GitHub] spark issue #21481: [SPARK-24452][SQL][Core] Avoid possible overflow in int ...

2018-06-11 Thread cloud-fan
Github user cloud-fan commented on the issue: https://github.com/apache/spark/pull/21481 @kiszk when you were running findBugs locally, did you find more overflow bugs that are not present in this PR? Let's put all discovered overflow bugs in this PR and have another PR to integrate

[GitHub] spark pull request #21481: [SPARK-24452][SQL][Core] Avoid possible overflow ...

2018-06-11 Thread cloud-fan
Github user cloud-fan commented on a diff in the pull request: https://github.com/apache/spark/pull/21481#discussion_r194537413 --- Diff: sql/catalyst/src/main/java/org/apache/spark/sql/catalyst/expressions/VariableLengthRowBasedKeyValueBatch.java --- @@ -41,7 +41,7 @@

[GitHub] spark pull request #21508: [SPARK-24488] [SQL] Fix issue when generator is a...

2018-06-11 Thread gatorsmile
Github user gatorsmile commented on a diff in the pull request: https://github.com/apache/spark/pull/21508#discussion_r194537508 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/Analyzer.scala --- @@ -1568,11 +1568,13 @@ class Analyzer(

[GitHub] spark pull request #20709: [SPARK-18844][MLLIB] Adding more binary classific...

2018-06-11 Thread sandecho
Github user sandecho closed the pull request at: https://github.com/apache/spark/pull/20709 --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #21481: [SPARK-24452][SQL][Core] Avoid possible overflow ...

2018-06-11 Thread cloud-fan
Github user cloud-fan commented on a diff in the pull request: https://github.com/apache/spark/pull/21481#discussion_r194536447 --- Diff: core/src/main/java/org/apache/spark/unsafe/map/BytesToBytesMap.java --- @@ -703,7 +703,7 @@ public boolean append(Object kbase, long koff, int

[GitHub] spark issue #21531: [SPARK-24521][SQL] Fix ineffective test in CachedTableSu...

2018-06-11 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21531 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution/3908/

[GitHub] spark pull request #21366: [SPARK-24248][K8S] Use level triggering and state...

2018-06-11 Thread mccheah
Github user mccheah commented on a diff in the pull request: https://github.com/apache/spark/pull/21366#discussion_r194536061 --- Diff: resource-managers/kubernetes/core/src/main/scala/org/apache/spark/scheduler/cluster/k8s/ExecutorPodsSnapshotsStoreImpl.scala --- @@ -0,0 +1,95

[GitHub] spark issue #21531: [SPARK-24521][SQL] Fix ineffective test in CachedTableSu...

2018-06-11 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21531 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark pull request #21366: [SPARK-24248][K8S] Use level triggering and state...

2018-06-11 Thread mccheah
Github user mccheah commented on a diff in the pull request: https://github.com/apache/spark/pull/21366#discussion_r194535271 --- Diff: resource-managers/kubernetes/core/src/main/scala/org/apache/spark/scheduler/cluster/k8s/ExecutorPodsSnapshotsStoreImpl.scala --- @@ -0,0 +1,95

[GitHub] spark issue #20636: [SPARK-23415][SQL][TEST] Make behavior of BufferHolderSp...

2018-06-11 Thread cloud-fan
Github user cloud-fan commented on the issue: https://github.com/apache/spark/pull/20636 `BufferHolderSparkSubmitSuite` is really a bad test. What we want to guarantee is, `BufferHolder` is always word-aligned. Is there a better way to test it? ---

[GitHub] spark pull request #21366: [SPARK-24248][K8S] Use level triggering and state...

2018-06-11 Thread skonto
Github user skonto commented on a diff in the pull request: https://github.com/apache/spark/pull/21366#discussion_r194534077 --- Diff: resource-managers/kubernetes/core/src/main/scala/org/apache/spark/scheduler/cluster/k8s/ExecutorPodsSnapshotsStoreImpl.scala --- @@ -0,0 +1,95 @@

[GitHub] spark issue #21515: [SPARK-24372][build] Add scripts to help with preparing ...

2018-06-11 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21515 **[Test build #91673 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/91673/testReport)** for PR 21515 at commit

[GitHub] spark issue #21531: [SPARK-24521][SQL] Fix ineffective test in CachedTableSu...

2018-06-11 Thread gatorsmile
Github user gatorsmile commented on the issue: https://github.com/apache/spark/pull/21531 cc @maryannxue --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail:

[GitHub] spark issue #21517: Testing k9s change - please ignore (13)

2018-06-11 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21517 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution/3907/

[GitHub] spark issue #21517: Testing k9s change - please ignore (13)

2018-06-11 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21517 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark pull request #21531: [SPARK-24521][SQL] Fix ineffective test in Cached...

2018-06-11 Thread icexelloss
Github user icexelloss commented on a diff in the pull request: https://github.com/apache/spark/pull/21531#discussion_r194532610 --- Diff: sql/core/src/test/scala/org/apache/spark/sql/CachedTableSuite.scala --- @@ -83,25 +83,6 @@ class CachedTableSuite extends QueryTest with

[GitHub] spark issue #21513: [SPARK-19826][ML][PYTHON]add spark.ml Python API for PIC

2018-06-11 Thread huaxingao
Github user huaxingao commented on the issue: https://github.com/apache/spark/pull/21513 Thanks a lot for your help! @mengxr @WeichenXu123 --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For

[GitHub] spark issue #21531: [SPARK-24521][SQL] Fix ineffective test in CachedTableSu...

2018-06-11 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21531 **[Test build #91672 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/91672/testReport)** for PR 21531 at commit

[GitHub] spark issue #21517: Testing k9s change - please ignore (13)

2018-06-11 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21517 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark issue #21517: Testing k9s change - please ignore (13)

2018-06-11 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21517 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/16/

[GitHub] spark pull request #21061: [SPARK-23914][SQL] Add array_union function

2018-06-11 Thread ueshin
Github user ueshin commented on a diff in the pull request: https://github.com/apache/spark/pull/21061#discussion_r194527119 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/collectionOperations.scala --- @@ -2189,3 +2189,293 @@ case class

[GitHub] spark pull request #21061: [SPARK-23914][SQL] Add array_union function

2018-06-11 Thread ueshin
Github user ueshin commented on a diff in the pull request: https://github.com/apache/spark/pull/21061#discussion_r194516045 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/collectionOperations.scala --- @@ -2189,3 +2189,293 @@ case class

[GitHub] spark pull request #21061: [SPARK-23914][SQL] Add array_union function

2018-06-11 Thread ueshin
Github user ueshin commented on a diff in the pull request: https://github.com/apache/spark/pull/21061#discussion_r194523480 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/collectionOperations.scala --- @@ -2189,3 +2189,293 @@ case class

[GitHub] spark pull request #21061: [SPARK-23914][SQL] Add array_union function

2018-06-11 Thread ueshin
Github user ueshin commented on a diff in the pull request: https://github.com/apache/spark/pull/21061#discussion_r194520924 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/collectionOperations.scala --- @@ -2189,3 +2189,293 @@ case class

[GitHub] spark pull request #21061: [SPARK-23914][SQL] Add array_union function

2018-06-11 Thread ueshin
Github user ueshin commented on a diff in the pull request: https://github.com/apache/spark/pull/21061#discussion_r194530787 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/collectionOperations.scala --- @@ -2189,3 +2189,293 @@ case class

[GitHub] spark pull request #21061: [SPARK-23914][SQL] Add array_union function

2018-06-11 Thread ueshin
Github user ueshin commented on a diff in the pull request: https://github.com/apache/spark/pull/21061#discussion_r194520589 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/collectionOperations.scala --- @@ -2189,3 +2189,293 @@ case class

[GitHub] spark pull request #21061: [SPARK-23914][SQL] Add array_union function

2018-06-11 Thread ueshin
Github user ueshin commented on a diff in the pull request: https://github.com/apache/spark/pull/21061#discussion_r194520120 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/collectionOperations.scala --- @@ -2189,3 +2189,293 @@ case class

[GitHub] spark pull request #21061: [SPARK-23914][SQL] Add array_union function

2018-06-11 Thread ueshin
Github user ueshin commented on a diff in the pull request: https://github.com/apache/spark/pull/21061#discussion_r194517152 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/collectionOperations.scala --- @@ -2189,3 +2189,293 @@ case class

[GitHub] spark issue #21045: [SPARK-23931][SQL] Adds arrays_zip function to sparksql

2018-06-11 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21045 **[Test build #91671 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/91671/testReport)** for PR 21045 at commit

[GitHub] spark pull request #21531: [SPARK-24521][SQL] Fix ineffective test in Cached...

2018-06-11 Thread icexelloss
GitHub user icexelloss opened a pull request: https://github.com/apache/spark/pull/21531 [SPARK-24521][SQL] Fix ineffective test in CachedTableSuite ## What changes were proposed in this pull request? test("withColumn doesn't invalidate cached dataframe") in

[GitHub] spark pull request #21045: [SPARK-23931][SQL] Adds arrays_zip function to sp...

2018-06-11 Thread DylanGuedes
Github user DylanGuedes commented on a diff in the pull request: https://github.com/apache/spark/pull/21045#discussion_r194530372 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/collectionOperations.scala --- @@ -128,6 +128,170 @@ case class

[GitHub] spark issue #21517: Testing k9s change - please ignore (13)

2018-06-11 Thread ssuchter
Github user ssuchter commented on the issue: https://github.com/apache/spark/pull/21517 comment --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail:

[GitHub] spark issue #21517: Testing k9s change - please ignore (13)

2018-06-11 Thread ssuchter
Github user ssuchter commented on the issue: https://github.com/apache/spark/pull/21517 retest this please --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail:

[GitHub] spark issue #21517: Testing k8s change - please ignore (13)

2018-06-11 Thread ssuchter
Github user ssuchter commented on the issue: https://github.com/apache/spark/pull/21517 retest this please --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail:

[GitHub] spark issue #21517: Testing k8s change - please ignore (13)

2018-06-11 Thread ssuchter
Github user ssuchter commented on the issue: https://github.com/apache/spark/pull/21517 comment --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail:

[GitHub] spark pull request #20636: [SPARK-23415][SQL][TEST] Make behavior of BufferH...

2018-06-11 Thread cloud-fan
Github user cloud-fan commented on a diff in the pull request: https://github.com/apache/spark/pull/20636#discussion_r194526656 --- Diff: sql/catalyst/src/main/java/org/apache/spark/sql/catalyst/expressions/codegen/BufferHolder.java --- @@ -61,6 +61,10 @@ * Grows the

[GitHub] spark issue #21500: Scalable Memory option for HDFSBackedStateStore

2018-06-11 Thread aalobaidi
Github user aalobaidi commented on the issue: https://github.com/apache/spark/pull/21500 @HeartSaVioR 1. As I mentioned before, this option is beneficial for use cases with bigger micro-batches. This way the overhead of loading the state from disk will be spread across

[GitHub] spark issue #21427: [SPARK-24324][PYTHON] Pandas Grouped Map UDF should assi...

2018-06-11 Thread cloud-fan
Github user cloud-fan commented on the issue: https://github.com/apache/spark/pull/21427 Pandas UDF is already in 2 releases(2.3.0 and 2.3.1), we can't just change the behavior. We think the old behavior doesn't make sense and users should change their code, but users may not think

[GitHub] spark issue #21512: [minor][WEB UI] Spark web ui auto refresh every x second...

2018-06-11 Thread tomhanetz
Github user tomhanetz commented on the issue: https://github.com/apache/spark/pull/21512 I don't agree with you. Yes browsers are capable of doing auto refresh through plugins (and actually this is the way I use the spark ui myself), but the it seems like this is the only monitoring

[GitHub] spark pull request #21438: [SPARK-24398] [SQL] Improve SQLAppStatusListener....

2018-06-11 Thread gengliangwang
Github user gengliangwang commented on a diff in the pull request: https://github.com/apache/spark/pull/21438#discussion_r19457 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/ui/SQLAppStatusListener.scala --- @@ -159,7 +159,7 @@ class SQLAppStatusListener(

[GitHub] spark issue #21518: [SPARK-24502][SQL] flaky test: UnsafeRowSerializerSuite

2018-06-11 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21518 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark issue #21518: [SPARK-24502][SQL] flaky test: UnsafeRowSerializerSuite

2018-06-11 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21518 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution/3906/

[GitHub] spark issue #21518: [SPARK-24502][SQL] flaky test: UnsafeRowSerializerSuite

2018-06-11 Thread squito
Github user squito commented on the issue: https://github.com/apache/spark/pull/21518 lgtm assuming tests pass --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail:

[GitHub] spark issue #21518: [SPARK-24502][SQL] flaky test: UnsafeRowSerializerSuite

2018-06-11 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21518 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/15/

[GitHub] spark issue #21518: [SPARK-24502][SQL] flaky test: UnsafeRowSerializerSuite

2018-06-11 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21518 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark issue #21518: [SPARK-24502][SQL] flaky test: UnsafeRowSerializerSuite

2018-06-11 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21518 **[Test build #91670 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/91670/testReport)** for PR 21518 at commit

[GitHub] spark pull request #21513: [SPARK-19826][ML][PYTHON]add spark.ml Python API ...

2018-06-11 Thread asfgit
Github user asfgit closed the pull request at: https://github.com/apache/spark/pull/21513 --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #21513: [SPARK-19826][ML][PYTHON]add spark.ml Python API for PIC

2018-06-11 Thread mengxr
Github user mengxr commented on the issue: https://github.com/apache/spark/pull/21513 LGTM. Merged into master. Thanks! --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands,

[GitHub] spark issue #21517: Testing k8s change - please ignore (13)

2018-06-11 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21517 Kubernetes integration test status success URL: https://amplab.cs.berkeley.edu/jenkins/job/testing-k8s-prb-spark-integration/3759/ ---

[GitHub] spark issue #21512: [minor][WEB UI] Spark web ui auto refresh every x second...

2018-06-11 Thread gengliangwang
Github user gengliangwang commented on the issue: https://github.com/apache/spark/pull/21512 Browsers are able to do the auto refreshing work. I think we should leave it for browsers, so that we can keep the UI code simple. ---

[GitHub] spark pull request #21258: [SPARK-23933][SQL] Add map_from_arrays function

2018-06-11 Thread ueshin
Github user ueshin commented on a diff in the pull request: https://github.com/apache/spark/pull/21258#discussion_r194514576 --- Diff: sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/expressions/ComplexTypeSuite.scala --- @@ -186,6 +186,50 @@ class ComplexTypeSuite

[GitHub] spark pull request #21258: [SPARK-23933][SQL] Add map_from_arrays function

2018-06-11 Thread ueshin
Github user ueshin commented on a diff in the pull request: https://github.com/apache/spark/pull/21258#discussion_r194514611 --- Diff: sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/expressions/ComplexTypeSuite.scala --- @@ -186,6 +186,50 @@ class ComplexTypeSuite

[GitHub] spark issue #21475: [SPARK-24416] Fix configuration specification for killBl...

2018-06-11 Thread squito
Github user squito commented on the issue: https://github.com/apache/spark/pull/21475 lgtm --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail:

[GitHub] spark issue #21517: Testing k8s change - please ignore (13)

2018-06-11 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21517 Merged build finished. Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark issue #21517: Testing k8s change - please ignore (13)

2018-06-11 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21517 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/14/

[GitHub] spark issue #21508: [SPARK-24488] [SQL] Fix issue when generator is aliased ...

2018-06-11 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21508 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark issue #21508: [SPARK-24488] [SQL] Fix issue when generator is aliased ...

2018-06-11 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21508 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/91665/ Test PASSed. ---

[GitHub] spark issue #21527: [SPARK-24519] MapStatus has 2000 hardcoded

2018-06-11 Thread markhamstra
Github user markhamstra commented on the issue: https://github.com/apache/spark/pull/21527 > We should make it configurable. That's a debatable assertion all by itself -- and quite unfortunately, there is no more justification for this claim in the JIRA ticket. Without proof

[GitHub] spark issue #21508: [SPARK-24488] [SQL] Fix issue when generator is aliased ...

2018-06-11 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21508 **[Test build #91665 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/91665/testReport)** for PR 21508 at commit

[GitHub] spark pull request #21045: [SPARK-23931][SQL] Adds zip function to sparksql

2018-06-11 Thread ueshin
Github user ueshin commented on a diff in the pull request: https://github.com/apache/spark/pull/21045#discussion_r194501372 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/collectionOperations.scala --- @@ -128,6 +128,170 @@ case class

[GitHub] spark issue #21517: Testing k8s change - please ignore (13)

2018-06-11 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21517 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark issue #21529: [SPARK-24495][SQL] EnsureRequirement returns wrong plan ...

2018-06-11 Thread cloud-fan
Github user cloud-fan commented on the issue: https://github.com/apache/spark/pull/21529 good catch! thanks! --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail:

[GitHub] spark issue #21517: Testing k8s change - please ignore (13)

2018-06-11 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21517 Kubernetes integration test starting URL: https://amplab.cs.berkeley.edu/jenkins/job/testing-k8s-prb-spark-integration/3759/ ---

[GitHub] spark issue #21517: Testing k8s change - please ignore (13)

2018-06-11 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21517 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution/3905/

[GitHub] spark issue #21527: [SPARK-24519] MapStatus has 2000 hardcoded

2018-06-11 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21527 Merged build finished. Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark issue #21527: [SPARK-24519] MapStatus has 2000 hardcoded

2018-06-11 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21527 **[Test build #91669 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/91669/testReport)** for PR 21527 at commit

[GitHub] spark issue #21527: [SPARK-24519] MapStatus has 2000 hardcoded

2018-06-11 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21527 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/91669/ Test FAILed. ---

[GitHub] spark issue #21527: [SPARK-24519] MapStatus has 2000 hardcoded

2018-06-11 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21527 **[Test build #91669 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/91669/testReport)** for PR 21527 at commit

[GitHub] spark pull request #21529: [SPARK-24495][SQL] EnsureRequirement returns wron...

2018-06-11 Thread cloud-fan
Github user cloud-fan commented on a diff in the pull request: https://github.com/apache/spark/pull/21529#discussion_r194511635 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/exchange/EnsureRequirements.scala --- @@ -227,9 +228,16 @@ case class

[GitHub] spark issue #21529: [SPARK-24495][SQL] EnsureRequirement returns wrong plan ...

2018-06-11 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21529 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

<    1   2   3   4   5   6   >