[GitHub] spark issue #21542: [SPARK-24529][Build][test-maven] Add spotbugs into maven...

2018-06-27 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21542 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark issue #21542: [SPARK-24529][Build][test-maven] Add spotbugs into maven...

2018-06-27 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21542 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/520/

[GitHub] spark issue #21553: [SPARK-24215][PySpark][Follow Up] Implement eager evalua...

2018-06-27 Thread gatorsmile
Github user gatorsmile commented on the issue: https://github.com/apache/spark/pull/21553 LGTM Thanks! Merged to master. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands,

[GitHub] spark issue #21476: [SPARK-24446][yarn] Properly quote library path for YARN...

2018-06-27 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21476 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark issue #21476: [SPARK-24446][yarn] Properly quote library path for YARN...

2018-06-27 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21476 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/92388/ Test PASSed. ---

[GitHub] spark issue #16415: [SPARK-19063][ML]Speedup and optimize the GradientBooste...

2018-06-27 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/16415 Can one of the admins verify this patch? --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark issue #21389: [SPARK-24204][SQL] Verify a schema in Json/Orc/ParquetFi...

2018-06-27 Thread gatorsmile
Github user gatorsmile commented on the issue: https://github.com/apache/spark/pull/21389 retest this please --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail:

[GitHub] spark pull request #21476: [SPARK-24446][yarn] Properly quote library path f...

2018-06-27 Thread asfgit
Github user asfgit closed the pull request at: https://github.com/apache/spark/pull/21476 --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #21511: [SPARK-24491][Kubernetes] Configuration support for requ...

2018-06-27 Thread alexmilowski
Github user alexmilowski commented on the issue: https://github.com/apache/spark/pull/21511 Hello all, I've been thinking about trying to make this more generic given I just ran into a hostPath/volume issue for executors which is similar. I took a look at SPARK-24434 and

[GitHub] spark issue #18544: [SPARK-21318][SQL]Improve exception message thrown by `l...

2018-06-27 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/18544 Can one of the admins verify this patch? --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark issue #21389: [SPARK-24204][SQL] Verify a schema in Json/Orc/ParquetFi...

2018-06-27 Thread HyukjinKwon
Github user HyukjinKwon commented on the issue: https://github.com/apache/spark/pull/21389 retest this please --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail:

[GitHub] spark issue #21389: [SPARK-24204][SQL] Verify a schema in Json/Orc/ParquetFi...

2018-06-27 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21389 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/519/

[GitHub] spark issue #21389: [SPARK-24204][SQL] Verify a schema in Json/Orc/ParquetFi...

2018-06-27 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21389 **[Test build #92390 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/92390/testReport)** for PR 21389 at commit

[GitHub] spark issue #21389: [SPARK-24204][SQL] Verify a schema in Json/Orc/ParquetFi...

2018-06-27 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21389 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark issue #21644: [SPARK-24660][SHS] Show correct error pages when downloa...

2018-06-27 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21644 **[Test build #92384 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/92384/testReport)** for PR 21644 at commit

[GitHub] spark issue #16677: [SPARK-19355][SQL] Use map output statistics to improve ...

2018-06-27 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/16677 **[Test build #92383 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/92383/testReport)** for PR 16677 at commit

[GitHub] spark pull request #18900: [SPARK-21687][SQL] Spark SQL should set createTim...

2018-06-27 Thread asfgit
Github user asfgit closed the pull request at: https://github.com/apache/spark/pull/18900 --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #21636: [SPARK-24533] Typesafe rebranded to lightbend. Changing ...

2018-06-27 Thread tgravescs
Github user tgravescs commented on the issue: https://github.com/apache/spark/pull/21636 +1, lgtm seems downloads.typesafe.com just redirects to downloads.lightbend.com now. --- - To unsubscribe, e-mail:

[GitHub] spark issue #21644: [SPARK-24660][SHS] Show correct error pages when downloa...

2018-06-27 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21644 **[Test build #92392 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/92392/testReport)** for PR 21644 at commit

[GitHub] spark issue #21644: [SPARK-24660][SHS] Show correct error pages when downloa...

2018-06-27 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21644 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark issue #21560: [SPARK-24386][SS] coalesce(1) aggregates in continuous p...

2018-06-27 Thread tdas
Github user tdas commented on the issue: https://github.com/apache/spark/pull/21560 LGTM assuming tests pass. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail:

[GitHub] spark pull request #21635: [SPARK-24594][YARN] Introducing metrics for YARN

2018-06-27 Thread vanzin
Github user vanzin commented on a diff in the pull request: https://github.com/apache/spark/pull/21635#discussion_r198627306 --- Diff: resource-managers/yarn/src/main/scala/org/apache/spark/deploy/yarn/YarnClusterSchedulerSource.scala --- @@ -0,0 +1,49 @@ +/* + * Licensed

[GitHub] spark issue #20451: [SPARK-23146][WIP] Support client mode for Kubernetes cl...

2018-06-27 Thread rayburgemeestre
Github user rayburgemeestre commented on the issue: https://github.com/apache/spark/pull/20451 Yes, absolutely, but I made those changes in a hardcoded way though, just to try it out. So I'm not sure how helpful it is. Near the end of this file:

[GitHub] spark pull request #21635: [SPARK-24594][YARN] Introducing metrics for YARN

2018-06-27 Thread attilapiros
Github user attilapiros commented on a diff in the pull request: https://github.com/apache/spark/pull/21635#discussion_r198629311 --- Diff: resource-managers/yarn/src/main/scala/org/apache/spark/deploy/yarn/YarnClusterSchedulerSource.scala --- @@ -0,0 +1,49 @@ +/* + *

[GitHub] spark pull request #21635: [SPARK-24594][YARN] Introducing metrics for YARN

2018-06-27 Thread attilapiros
Github user attilapiros commented on a diff in the pull request: https://github.com/apache/spark/pull/21635#discussion_r198630307 --- Diff: resource-managers/yarn/src/main/scala/org/apache/spark/deploy/yarn/ApplicationMaster.scala --- @@ -309,6 +312,9 @@ private[spark] class

[GitHub] spark issue #21546: [WIP][SPARK-23030][SQL][PYTHON] Use Arrow stream format ...

2018-06-27 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21546 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/523/

[GitHub] spark issue #21649: [SPARK-23648][R][SQL]Adds more types for hint in SparkR

2018-06-27 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21649 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/92394/ Test PASSed. ---

[GitHub] spark issue #21546: [WIP][SPARK-23030][SQL][PYTHON] Use Arrow stream format ...

2018-06-27 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21546 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark issue #21546: [WIP][SPARK-23030][SQL][PYTHON] Use Arrow stream format ...

2018-06-27 Thread BryanCutler
Github user BryanCutler commented on the issue: https://github.com/apache/spark/pull/21546 _UPDATE_ I changed `toPandas` to write out of order partitions to python as they come in, followed by a list of indices to represent the correct batch order. In python, the batches are

[GitHub] spark issue #21560: [SPARK-24386][SS] coalesce(1) aggregates in continuous p...

2018-06-27 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21560 **[Test build #92387 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/92387/testReport)** for PR 21560 at commit

[GitHub] spark issue #21644: [SPARK-24660][SHS] Show correct error pages when downloa...

2018-06-27 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21644 **[Test build #92386 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/92386/testReport)** for PR 21644 at commit

[GitHub] spark issue #21644: [SPARK-24660][SHS] Show correct error pages when downloa...

2018-06-27 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21644 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark pull request #21440: [SPARK-24307][CORE] Support reading remote cached...

2018-06-27 Thread vanzin
Github user vanzin commented on a diff in the pull request: https://github.com/apache/spark/pull/21440#discussion_r198630017 --- Diff: core/src/main/scala/org/apache/spark/util/io/ChunkedByteBuffer.scala --- @@ -166,6 +170,38 @@ private[spark] class ChunkedByteBuffer(var chunks:

[GitHub] spark pull request #21440: [SPARK-24307][CORE] Support reading remote cached...

2018-06-27 Thread vanzin
Github user vanzin commented on a diff in the pull request: https://github.com/apache/spark/pull/21440#discussion_r198641717 --- Diff: core/src/main/scala/org/apache/spark/util/io/ChunkedByteBufferFileRegion.scala --- @@ -0,0 +1,103 @@ +/* + * Licensed to the Apache

[GitHub] spark issue #21546: [WIP][SPARK-23030][SQL][PYTHON] Use Arrow stream format ...

2018-06-27 Thread BryanCutler
Github user BryanCutler commented on the issue: https://github.com/apache/spark/pull/21546 ### Performance Tests - toPandas Tests run on a 4 node standalone cluster with 32 cores total, 14.04.1-Ubuntu and OpenJDK 8 measured wall clock time to execute `toPandas()` and took

[GitHub] spark issue #21451: [SPARK-24296][CORE][WIP] Replicate large blocks as a str...

2018-06-27 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21451 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark issue #21451: [SPARK-24296][CORE][WIP] Replicate large blocks as a str...

2018-06-27 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21451 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/524/

[GitHub] spark pull request #21440: [SPARK-24307][CORE] Support reading remote cached...

2018-06-27 Thread vanzin
Github user vanzin commented on a diff in the pull request: https://github.com/apache/spark/pull/21440#discussion_r198632416 --- Diff: core/src/main/scala/org/apache/spark/util/io/ChunkedByteBufferFileRegion.scala --- @@ -0,0 +1,103 @@ +/* + * Licensed to the Apache

[GitHub] spark pull request #21440: [SPARK-24307][CORE] Support reading remote cached...

2018-06-27 Thread vanzin
Github user vanzin commented on a diff in the pull request: https://github.com/apache/spark/pull/21440#discussion_r198636259 --- Diff: core/src/main/scala/org/apache/spark/util/io/ChunkedByteBufferFileRegion.scala --- @@ -0,0 +1,103 @@ +/* + * Licensed to the Apache

[GitHub] spark issue #21546: [WIP][SPARK-23030][SQL][PYTHON] Use Arrow stream format ...

2018-06-27 Thread BryanCutler
Github user BryanCutler commented on the issue: https://github.com/apache/spark/pull/21546 ### Performance Tests - createDataFrame Tests run on a 4 node standalone cluster with 32 cores total, 14.04.1-Ubuntu and OpenJDK 8 measured wall clock time to execute

[GitHub] spark pull request #21644: [SPARK-24660][SHS] Show correct error pages when ...

2018-06-27 Thread asfgit
Github user asfgit closed the pull request at: https://github.com/apache/spark/pull/21644 --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #21635: [SPARK-24594][YARN] Introducing metrics for YARN

2018-06-27 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21635 **[Test build #92397 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/92397/testReport)** for PR 21635 at commit

[GitHub] spark issue #21635: [SPARK-24594][YARN] Introducing metrics for YARN

2018-06-27 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21635 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark issue #20944: [SPARK-23831][SQL] Add org.apache.derby to IsolatedClien...

2018-06-27 Thread holdenk
Github user holdenk commented on the issue: https://github.com/apache/spark/pull/20944 So I had this come up while I was testing Spark 2.1.3 RC2 on a machine with an existing YARN cluster with `spark-testing-base`. Haven't had the chance to dig into it fully. ---

[GitHub] spark pull request #21635: [SPARK-24594][YARN] Introducing metrics for YARN

2018-06-27 Thread vanzin
Github user vanzin commented on a diff in the pull request: https://github.com/apache/spark/pull/21635#discussion_r198626433 --- Diff: resource-managers/yarn/src/main/scala/org/apache/spark/deploy/yarn/ApplicationMaster.scala --- @@ -309,6 +312,9 @@ private[spark] class

[GitHub] spark issue #21649: [SPARK-23648][R][SQL]Adds more types for hint in SparkR

2018-06-27 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21649 **[Test build #92394 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/92394/testReport)** for PR 21649 at commit

[GitHub] spark issue #21649: [SPARK-23648][R][SQL]Adds more types for hint in SparkR

2018-06-27 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21649 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark issue #21635: [SPARK-24594][YARN] Introducing metrics for YARN

2018-06-27 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21635 **[Test build #92396 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/92396/testReport)** for PR 21635 at commit

[GitHub] spark pull request #21635: [SPARK-24594][YARN] Introducing metrics for YARN

2018-06-27 Thread attilapiros
Github user attilapiros commented on a diff in the pull request: https://github.com/apache/spark/pull/21635#discussion_r198641094 --- Diff: resource-managers/yarn/src/main/scala/org/apache/spark/deploy/yarn/YarnClusterSchedulerSource.scala --- @@ -0,0 +1,49 @@ +/* + *

[GitHub] spark issue #21644: [SPARK-24660][SHS] Show correct error pages when downloa...

2018-06-27 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21644 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/92386/ Test PASSed. ---

[GitHub] spark issue #21635: [SPARK-24594][YARN] Introducing metrics for YARN

2018-06-27 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21635 **[Test build #92396 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/92396/testReport)** for PR 21635 at commit

[GitHub] spark issue #21635: [SPARK-24594][YARN] Introducing metrics for YARN

2018-06-27 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21635 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/92396/ Test PASSed. ---

[GitHub] spark issue #21635: [SPARK-24594][YARN] Introducing metrics for YARN

2018-06-27 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21635 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark issue #21644: [SPARK-24660][SHS] Show correct error pages when downloa...

2018-06-27 Thread vanzin
Github user vanzin commented on the issue: https://github.com/apache/spark/pull/21644 Merging to master. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail:

[GitHub] spark issue #21636: [SPARK-24533] Typesafe rebranded to lightbend. Changing ...

2018-06-27 Thread vanzin
Github user vanzin commented on the issue: https://github.com/apache/spark/pull/21636 Merging to master. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail:

[GitHub] spark issue #21635: [SPARK-24594][YARN] Introducing metrics for YARN

2018-06-27 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21635 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/92397/ Test PASSed. ---

[GitHub] spark issue #21602: [SPARK-24613][SQL] Cache with UDF could not be matched w...

2018-06-27 Thread gatorsmile
Github user gatorsmile commented on the issue: https://github.com/apache/spark/pull/21602 This is also a regression. Backported to 2.3 branch too. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org

[GitHub] spark pull request #21221: [SPARK-23429][CORE] Add executor memory metrics t...

2018-06-27 Thread squito
Github user squito commented on a diff in the pull request: https://github.com/apache/spark/pull/21221#discussion_r198625661 --- Diff: core/src/test/scala/org/apache/spark/scheduler/EventLoggingListenerSuite.scala --- @@ -251,6 +261,217 @@ class EventLoggingListenerSuite extends

[GitHub] spark pull request #21221: [SPARK-23429][CORE] Add executor memory metrics t...

2018-06-27 Thread squito
Github user squito commented on a diff in the pull request: https://github.com/apache/spark/pull/21221#discussion_r198611581 --- Diff: core/src/main/scala/org/apache/spark/scheduler/EventLoggingListener.scala --- @@ -169,6 +181,28 @@ private[spark] class EventLoggingListener(

[GitHub] spark pull request #21221: [SPARK-23429][CORE] Add executor memory metrics t...

2018-06-27 Thread squito
Github user squito commented on a diff in the pull request: https://github.com/apache/spark/pull/21221#discussion_r198628058 --- Diff: project/MimaExcludes.scala --- @@ -89,7 +89,13 @@ object MimaExcludes {

[GitHub] spark pull request #21221: [SPARK-23429][CORE] Add executor memory metrics t...

2018-06-27 Thread squito
Github user squito commented on a diff in the pull request: https://github.com/apache/spark/pull/21221#discussion_r198618559 --- Diff: core/src/main/scala/org/apache/spark/status/AppStatusListener.scala --- @@ -669,6 +686,29 @@ private[spark] class AppStatusListener(

[GitHub] spark pull request #21221: [SPARK-23429][CORE] Add executor memory metrics t...

2018-06-27 Thread squito
Github user squito commented on a diff in the pull request: https://github.com/apache/spark/pull/21221#discussion_r198624068 --- Diff: core/src/main/scala/org/apache/spark/status/api/v1/api.scala --- @@ -98,14 +101,53 @@ class ExecutorSummary private[spark]( val

[GitHub] spark pull request #21221: [SPARK-23429][CORE] Add executor memory metrics t...

2018-06-27 Thread squito
Github user squito commented on a diff in the pull request: https://github.com/apache/spark/pull/21221#discussion_r198610162 --- Diff: core/src/main/scala/org/apache/spark/metrics/MetricGetter.scala --- @@ -0,0 +1,88 @@ +/* + * Licensed to the Apache Software Foundation

[GitHub] spark pull request #21221: [SPARK-23429][CORE] Add executor memory metrics t...

2018-06-27 Thread squito
Github user squito commented on a diff in the pull request: https://github.com/apache/spark/pull/21221#discussion_r198624819 --- Diff: core/src/main/scala/org/apache/spark/status/api/v1/api.scala --- @@ -98,14 +102,48 @@ class ExecutorSummary private[spark]( val

[GitHub] spark pull request #21221: [SPARK-23429][CORE] Add executor memory metrics t...

2018-06-27 Thread squito
Github user squito commented on a diff in the pull request: https://github.com/apache/spark/pull/21221#discussion_r198613341 --- Diff: core/src/main/scala/org/apache/spark/scheduler/SparkListener.scala --- @@ -264,6 +282,11 @@ private[spark] trait SparkListenerInterface {

[GitHub] spark pull request #21221: [SPARK-23429][CORE] Add executor memory metrics t...

2018-06-27 Thread squito
Github user squito commented on a diff in the pull request: https://github.com/apache/spark/pull/21221#discussion_r198609741 --- Diff: core/src/main/scala/org/apache/spark/Heartbeater.scala --- @@ -0,0 +1,69 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under

[GitHub] spark pull request #21221: [SPARK-23429][CORE] Add executor memory metrics t...

2018-06-27 Thread squito
Github user squito commented on a diff in the pull request: https://github.com/apache/spark/pull/21221#discussion_r198624800 --- Diff: core/src/main/scala/org/apache/spark/status/api/v1/api.scala --- @@ -98,14 +102,48 @@ class ExecutorSummary private[spark]( val

[GitHub] spark pull request #21221: [SPARK-23429][CORE] Add executor memory metrics t...

2018-06-27 Thread squito
Github user squito commented on a diff in the pull request: https://github.com/apache/spark/pull/21221#discussion_r198611872 --- Diff: core/src/main/scala/org/apache/spark/scheduler/PeakExecutorMetrics.scala --- @@ -0,0 +1,127 @@ +/* + * Licensed to the Apache Software

[GitHub] spark pull request #21221: [SPARK-23429][CORE] Add executor memory metrics t...

2018-06-27 Thread squito
Github user squito commented on a diff in the pull request: https://github.com/apache/spark/pull/21221#discussion_r198609800 --- Diff: core/src/main/scala/org/apache/spark/SparkContext.scala --- @@ -1922,6 +1928,12 @@ class SparkContext(config: SparkConf) extends Logging {

[GitHub] spark issue #21546: [WIP][SPARK-23030][SQL][PYTHON] Use Arrow stream format ...

2018-06-27 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21546 **[Test build #92395 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/92395/testReport)** for PR 21546 at commit

[GitHub] spark issue #21560: [SPARK-24386][SS] coalesce(1) aggregates in continuous p...

2018-06-27 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21560 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/92387/ Test PASSed. ---

[GitHub] spark issue #21560: [SPARK-24386][SS] coalesce(1) aggregates in continuous p...

2018-06-27 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21560 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark issue #21635: [SPARK-24594][YARN] Introducing metrics for YARN

2018-06-27 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21635 **[Test build #92397 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/92397/testReport)** for PR 21635 at commit

[GitHub] spark issue #21451: [SPARK-24296][CORE][WIP] Replicate large blocks as a str...

2018-06-27 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21451 **[Test build #92398 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/92398/testReport)** for PR 21451 at commit

[GitHub] spark pull request #21636: [SPARK-24533] Typesafe rebranded to lightbend. Ch...

2018-06-27 Thread asfgit
Github user asfgit closed the pull request at: https://github.com/apache/spark/pull/21636 --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #21075: [SPARK-23988][MESOS] Improve handling of appResource in ...

2018-06-27 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21075 Can one of the admins verify this patch? --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark pull request #21440: [SPARK-24307][CORE] Support reading remote cached...

2018-06-27 Thread squito
Github user squito commented on a diff in the pull request: https://github.com/apache/spark/pull/21440#discussion_r198651582 --- Diff: core/src/main/scala/org/apache/spark/util/io/ChunkedByteBuffer.scala --- @@ -166,6 +170,38 @@ private[spark] class ChunkedByteBuffer(var chunks:

[GitHub] spark issue #16677: [SPARK-19355][SQL] Use map output statistics to improve ...

2018-06-27 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/16677 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark issue #21644: [SPARK-24660][SHS] Show correct error pages when downloa...

2018-06-27 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21644 **[Test build #92380 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/92380/testReport)** for PR 21644 at commit

[GitHub] spark issue #16677: [SPARK-19355][SQL] Use map output statistics to improve ...

2018-06-27 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/16677 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/515/

[GitHub] spark issue #16677: [SPARK-19355][SQL] Use map output statistics to improve ...

2018-06-27 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/16677 **[Test build #92383 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/92383/testReport)** for PR 16677 at commit

[GitHub] spark issue #21495: [SPARK-24418][Build] Upgrade Scala to 2.11.12 and 2.12.6

2018-06-27 Thread jerryshao
Github user jerryshao commented on the issue: https://github.com/apache/spark/pull/21495 > The only change with this PR is that the welcome message will be printed first, and then the Spark URL will be shown latter. It's a minor difference. I think we should create a JIRA to

[GitHub] spark pull request #21651: [SPARK-18258] Sink need access to offset represen...

2018-06-27 Thread ConcurrencyPractitioner
GitHub user ConcurrencyPractitioner opened a pull request: https://github.com/apache/spark/pull/21651 [SPARK-18258] Sink need access to offset representation ## What changes were proposed in this pull request? Currently, sinks only have access to the batchId and the data,

[GitHub] spark issue #21651: [SPARK-18258] Sink need access to offset representation

2018-06-27 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21651 Can one of the admins verify this patch? --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark issue #21651: [SPARK-18258] Sink need access to offset representation

2018-06-27 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21651 Can one of the admins verify this patch? --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark issue #21651: [SPARK-18258] Sink need access to offset representation

2018-06-27 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21651 Can one of the admins verify this patch? --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark issue #21631: [SPARK-24645][SQL] Skip parsing when csvColumnPruning en...

2018-06-27 Thread maropu
Github user maropu commented on the issue: https://github.com/apache/spark/pull/21631 @HyukjinKwon BTW, can you check this? @MaxGekk Probably, I feel you'd be better to file a new jira for the point you're looking into. ---

[GitHub] spark pull request #21221: [SPARK-23429][CORE] Add executor memory metrics t...

2018-06-27 Thread edwinalu
Github user edwinalu commented on a diff in the pull request: https://github.com/apache/spark/pull/21221#discussion_r198682779 --- Diff: core/src/main/scala/org/apache/spark/Heartbeater.scala --- @@ -0,0 +1,69 @@ +/* + * Licensed to the Apache Software Foundation (ASF)

[GitHub] spark pull request #21533: [SPARK-24195][Core] Bug fix for local:/ path in S...

2018-06-27 Thread jerryshao
Github user jerryshao commented on a diff in the pull request: https://github.com/apache/spark/pull/21533#discussion_r198682844 --- Diff: core/src/main/scala/org/apache/spark/SparkContext.scala --- @@ -1519,7 +1519,12 @@ class SparkContext(config: SparkConf) extends Logging {

[GitHub] spark issue #21651: [SPARK-18258] Sink need access to offset representation

2018-06-27 Thread ConcurrencyPractitioner
Github user ConcurrencyPractitioner commented on the issue: https://github.com/apache/spark/pull/21651 cc @koeninger --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands,

[GitHub] spark pull request #21221: [SPARK-23429][CORE] Add executor memory metrics t...

2018-06-27 Thread edwinalu
Github user edwinalu commented on a diff in the pull request: https://github.com/apache/spark/pull/21221#discussion_r198682884 --- Diff: core/src/main/scala/org/apache/spark/scheduler/EventLoggingListener.scala --- @@ -169,6 +181,28 @@ private[spark] class EventLoggingListener(

[GitHub] spark pull request #21221: [SPARK-23429][CORE] Add executor memory metrics t...

2018-06-27 Thread edwinalu
Github user edwinalu commented on a diff in the pull request: https://github.com/apache/spark/pull/21221#discussion_r198682809 --- Diff: core/src/main/scala/org/apache/spark/metrics/MetricGetter.scala --- @@ -0,0 +1,88 @@ +/* + * Licensed to the Apache Software Foundation

[GitHub] spark pull request #21221: [SPARK-23429][CORE] Add executor memory metrics t...

2018-06-27 Thread edwinalu
Github user edwinalu commented on a diff in the pull request: https://github.com/apache/spark/pull/21221#discussion_r198682787 --- Diff: core/src/main/scala/org/apache/spark/SparkContext.scala --- @@ -1922,6 +1928,12 @@ class SparkContext(config: SparkConf) extends Logging {

[GitHub] spark pull request #21221: [SPARK-23429][CORE] Add executor memory metrics t...

2018-06-27 Thread edwinalu
Github user edwinalu commented on a diff in the pull request: https://github.com/apache/spark/pull/21221#discussion_r198682917 --- Diff: core/src/main/scala/org/apache/spark/scheduler/SparkListener.scala --- @@ -264,6 +282,11 @@ private[spark] trait SparkListenerInterface {

[GitHub] spark pull request #21221: [SPARK-23429][CORE] Add executor memory metrics t...

2018-06-27 Thread edwinalu
Github user edwinalu commented on a diff in the pull request: https://github.com/apache/spark/pull/21221#discussion_r198682980 --- Diff: core/src/main/scala/org/apache/spark/status/AppStatusListener.scala --- @@ -669,6 +686,29 @@ private[spark] class AppStatusListener(

[GitHub] spark issue #21651: [SPARK-18258] Sink need access to offset representation

2018-06-27 Thread ConcurrencyPractitioner
Github user ConcurrencyPractitioner commented on the issue: https://github.com/apache/spark/pull/21651 I am uncertain about some of the ways we should transfer the data stored in OffsetSeqs to external storage (e.g. like KafkaSink which I mentioned before). ---

[GitHub] spark issue #21546: [WIP][SPARK-23030][SQL][PYTHON] Use Arrow stream format ...

2018-06-27 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21546 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/92395/ Test PASSed. ---

[GitHub] spark issue #21546: [WIP][SPARK-23030][SQL][PYTHON] Use Arrow stream format ...

2018-06-27 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21546 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark pull request #21221: [SPARK-23429][CORE] Add executor memory metrics t...

2018-06-27 Thread edwinalu
Github user edwinalu commented on a diff in the pull request: https://github.com/apache/spark/pull/21221#discussion_r198683408 --- Diff: core/src/main/scala/org/apache/spark/status/api/v1/api.scala --- @@ -98,14 +102,48 @@ class ExecutorSummary private[spark]( val

[GitHub] spark issue #21546: [WIP][SPARK-23030][SQL][PYTHON] Use Arrow stream format ...

2018-06-27 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21546 **[Test build #92395 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/92395/testReport)** for PR 21546 at commit

<    1   2   3   4   5   >