[GitHub] spark issue #23252: [SPARK-26239] File-based secret key loading for SASL.

2018-12-07 Thread tgravescs
Github user tgravescs commented on the issue: https://github.com/apache/spark/pull/23252 How is this file protected in kubernetes? I actually don't like this idea at least for yarn and other deployments, I see people abusing it (accidentally) and using it in non-secure manner

[GitHub] spark issue #23242: [SPARK-26285][CORE] accumulator metrics sources for Long...

2018-12-07 Thread tgravescs
Github user tgravescs commented on the issue: https://github.com/apache/spark/pull/23242 ok to test --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h

[GitHub] spark issue #23223: [SPARK-26269][YARN]Yarnallocator should have same blackl...

2018-12-06 Thread tgravescs
Github user tgravescs commented on the issue: https://github.com/apache/spark/pull/23223 if you aren't seeing actual issues with this I guess it would be interesting to test it further to see if it does. I can see spark blacklisting when it shouldn't for exit codes like you mention

[GitHub] spark issue #23223: [SPARK-26269][YARN]Yarnallocator should have same blackl...

2018-12-05 Thread tgravescs
Github user tgravescs commented on the issue: https://github.com/apache/spark/pull/23223 the approach here makes sense. Are you seeing actual issues with this blacklisting when it shouldn't? I could see that possible there and if so we should move this to defect and make sure

[GitHub] spark pull request #23223: [SPARK-26269][YARN]Yarnallocator should have same...

2018-12-05 Thread tgravescs
Github user tgravescs commented on a diff in the pull request: https://github.com/apache/spark/pull/23223#discussion_r239110361 --- Diff: resource-managers/yarn/src/main/scala/org/apache/spark/deploy/yarn/YarnAllocator.scala --- @@ -612,11 +612,14 @@ private[yarn] class

[GitHub] spark issue #23223: [SPARK-26269][YARN]Yarnallocator should have same blackl...

2018-12-05 Thread tgravescs
Github user tgravescs commented on the issue: https://github.com/apache/spark/pull/23223 ok to test --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h

[GitHub] spark issue #23166: [SPARK-26201] Fix python broadcast with encryption

2018-11-30 Thread tgravescs
Github user tgravescs commented on the issue: https://github.com/apache/spark/pull/23166 thanks @redsanket @squito, committed master, 2.4, and 2.3.2 --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org

[GitHub] spark pull request #23166: [SPARK-26201] Fix python broadcast with encryptio...

2018-11-30 Thread tgravescs
Github user tgravescs commented on a diff in the pull request: https://github.com/apache/spark/pull/23166#discussion_r237875851 --- Diff: core/src/main/scala/org/apache/spark/api/python/PythonRDD.scala --- @@ -708,16 +709,36 @@ private[spark] class PythonBroadcast(@transient var

[GitHub] spark issue #23160: [SPARK-26196][WebUI] Total tasks title in the stage page...

2018-11-30 Thread tgravescs
Github user tgravescs commented on the issue: https://github.com/apache/spark/pull/23160 reviewing, to clarify @pgandhi999 question, I assume the screenshots above are cut off and the task table is really showing 7 total failed tasks

[GitHub] spark pull request #23166: [SPARK-26201] Fix python broadcast with encryptio...

2018-11-28 Thread tgravescs
Github user tgravescs commented on a diff in the pull request: https://github.com/apache/spark/pull/23166#discussion_r237248890 --- Diff: python/pyspark/broadcast.py --- @@ -134,7 +137,15 @@ def value(self): """ Return the br

[GitHub] spark pull request #23166: [SPARK-26201] Fix python broadcast with encryptio...

2018-11-28 Thread tgravescs
Github user tgravescs commented on a diff in the pull request: https://github.com/apache/spark/pull/23166#discussion_r237212353 --- Diff: python/pyspark/tests/test_broadcast.py --- @@ -67,6 +67,20 @@ def test_broadcast_with_encryption(self): def

[GitHub] spark issue #23166: [SPARK-26201] Fix python broadcast with encryption

2018-11-28 Thread tgravescs
Github user tgravescs commented on the issue: https://github.com/apache/spark/pull/23166 Yeah certainly seems like a good idea. The only question I have is does this cause more memory usage on the driver because it has a reference to that broadcast value or is something else already

[GitHub] spark issue #23166: [SPARK-26201] Fix python broadcast with encryption

2018-11-28 Thread tgravescs
Github user tgravescs commented on the issue: https://github.com/apache/spark/pull/23166 cc @squito --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h

[GitHub] spark issue #23166: [SPARK-26201] Fix python broadcast with encryption

2018-11-28 Thread tgravescs
Github user tgravescs commented on the issue: https://github.com/apache/spark/pull/23166 ok to test --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h

[GitHub] spark issue #21688: [SPARK-21809] : Change Stage Page to use datatables to s...

2018-11-26 Thread tgravescs
Github user tgravescs commented on the issue: https://github.com/apache/spark/pull/21688 +1 , going to merge to master There are a few followup jiras on this. 1) make the timeline visualization better: https://issues.apache.org/jira/browse/SPARK-26130 2) improve

[GitHub] spark issue #21688: [SPARK-21809] : Change Stage Page to use datatables to s...

2018-11-26 Thread tgravescs
Github user tgravescs commented on the issue: https://github.com/apache/spark/pull/21688 Test this please --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail

[GitHub] spark issue #23103: [SPARK-26121] [Structured Streaming] Allow users to defi...

2018-11-22 Thread tgravescs
Github user tgravescs commented on the issue: https://github.com/apache/spark/pull/23103 Lgtm --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h

[GitHub] spark issue #21688: [SPARK-21809] : Change Stage Page to use datatables to s...

2018-11-21 Thread tgravescs
Github user tgravescs commented on the issue: https://github.com/apache/spark/pull/21688 Test this please --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h

[GitHub] spark issue #21688: [SPARK-21809] : Change Stage Page to use datatables to s...

2018-11-21 Thread tgravescs
Github user tgravescs commented on the issue: https://github.com/apache/spark/pull/21688 test this please --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h

[GitHub] spark issue #23103: [SPARK-26121] [Structured Streaming] Allow users to defi...

2018-11-21 Thread tgravescs
Github user tgravescs commented on the issue: https://github.com/apache/spark/pull/23103 ok to test --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h

[GitHub] spark pull request #21688: [SPARK-21809] : Change Stage Page to use datatabl...

2018-11-20 Thread tgravescs
Github user tgravescs commented on a diff in the pull request: https://github.com/apache/spark/pull/21688#discussion_r235164507 --- Diff: core/src/main/scala/org/apache/spark/ui/jobs/StagePage.scala --- @@ -92,6 +92,14 @@ private[ui] class StagePage(parent: StagesTab, store

[GitHub] spark pull request #21688: [SPARK-21809] : Change Stage Page to use datatabl...

2018-11-20 Thread tgravescs
Github user tgravescs commented on a diff in the pull request: https://github.com/apache/spark/pull/21688#discussion_r235157033 --- Diff: core/src/main/scala/org/apache/spark/ui/jobs/StagePage.scala --- @@ -259,7 +278,8 @@ private[ui] class StagePage(parent: StagesTab, store

[GitHub] spark issue #19840: [SPARK-22640][PYSPARK][YARN]switch python exec on execut...

2018-11-20 Thread tgravescs
Github user tgravescs commented on the issue: https://github.com/apache/spark/pull/19840 I didn't read the entire thread here but what you want is this: --archives hdfs:///python36/python36.tgz#python36 --conf spark.pyspark.python=./python36/bin/python3.6 --conf

[GitHub] spark pull request #21688: [SPARK-21809] : Change Stage Page to use datatabl...

2018-11-19 Thread tgravescs
Github user tgravescs commented on a diff in the pull request: https://github.com/apache/spark/pull/21688#discussion_r234765781 --- Diff: core/src/main/scala/org/apache/spark/ui/jobs/StagePage.scala --- @@ -315,187 +241,22 @@ private[ui] class StagePage(parent: StagesTab, store

[GitHub] spark pull request #21688: [SPARK-21809] : Change Stage Page to use datatabl...

2018-11-19 Thread tgravescs
Github user tgravescs commented on a diff in the pull request: https://github.com/apache/spark/pull/21688#discussion_r234757635 --- Diff: core/src/main/scala/org/apache/spark/status/api/v1/StagesResource.scala --- @@ -102,4 +103,120 @@ private[v1] class StagesResource extends

[GitHub] spark issue #23013: [SPARK-25023] More detailed security guidance for K8S

2018-11-19 Thread tgravescs
Github user tgravescs commented on the issue: https://github.com/apache/spark/pull/23013 sorry for my delay as I was on vacation. I would like to clarify if k8s support spark.authenticate and if so how does it do it?Without that, Spark is not secure with k8s unless you

[GitHub] spark pull request #21688: [SPARK-21809] : Change Stage Page to use datatabl...

2018-11-12 Thread tgravescs
Github user tgravescs commented on a diff in the pull request: https://github.com/apache/spark/pull/21688#discussion_r232832492 --- Diff: core/src/main/scala/org/apache/spark/status/api/v1/StagesResource.scala --- @@ -162,26 +165,29 @@ private[v1] class StagesResource extends

[GitHub] spark pull request #21688: [SPARK-21809] : Change Stage Page to use datatabl...

2018-11-12 Thread tgravescs
Github user tgravescs commented on a diff in the pull request: https://github.com/apache/spark/pull/21688#discussion_r232738700 --- Diff: core/src/main/scala/org/apache/spark/status/api/v1/StagesResource.scala --- @@ -102,4 +103,120 @@ private[v1] class StagesResource extends

[GitHub] spark pull request #21688: [SPARK-21809] : Change Stage Page to use datatabl...

2018-11-12 Thread tgravescs
Github user tgravescs commented on a diff in the pull request: https://github.com/apache/spark/pull/21688#discussion_r232736598 --- Diff: core/src/main/scala/org/apache/spark/status/api/v1/StagesResource.scala --- @@ -102,4 +103,120 @@ private[v1] class StagesResource extends

[GitHub] spark pull request #21688: [SPARK-21809] : Change Stage Page to use datatabl...

2018-11-12 Thread tgravescs
Github user tgravescs commented on a diff in the pull request: https://github.com/apache/spark/pull/21688#discussion_r232736621 --- Diff: core/src/main/scala/org/apache/spark/status/api/v1/StagesResource.scala --- @@ -102,4 +103,120 @@ private[v1] class StagesResource extends

[GitHub] spark pull request #21688: [SPARK-21809] : Change Stage Page to use datatabl...

2018-11-12 Thread tgravescs
Github user tgravescs commented on a diff in the pull request: https://github.com/apache/spark/pull/21688#discussion_r232736375 --- Diff: core/src/main/scala/org/apache/spark/status/api/v1/StagesResource.scala --- @@ -102,4 +103,120 @@ private[v1] class StagesResource extends

[GitHub] spark pull request #21688: [SPARK-21809] : Change Stage Page to use datatabl...

2018-11-12 Thread tgravescs
Github user tgravescs commented on a diff in the pull request: https://github.com/apache/spark/pull/21688#discussion_r232735802 --- Diff: core/src/main/scala/org/apache/spark/status/api/v1/StagesResource.scala --- @@ -102,4 +103,120 @@ private[v1] class StagesResource extends

[GitHub] spark pull request #21688: [SPARK-21809] : Change Stage Page to use datatabl...

2018-11-12 Thread tgravescs
Github user tgravescs commented on a diff in the pull request: https://github.com/apache/spark/pull/21688#discussion_r232711679 --- Diff: core/src/main/scala/org/apache/spark/status/api/v1/StagesResource.scala --- @@ -102,4 +103,120 @@ private[v1] class StagesResource extends

[GitHub] spark issue #22144: [SPARK-24935][SQL] : Problem with Executing Hive UDF's f...

2018-11-12 Thread tgravescs
Github user tgravescs commented on the issue: https://github.com/apache/spark/pull/22144 I think we should add the support back. It sounded like some people didn't like this PR for a fix so we would need to investigate something else, @cloud-fan did you have more specifics about

[GitHub] spark issue #23013: [SPARK-25023] More detailed security guidance for K8S

2018-11-12 Thread tgravescs
Github user tgravescs commented on the issue: https://github.com/apache/spark/pull/23013 ok to test --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h

[GitHub] spark issue #22288: [SPARK-22148][SPARK-15815][Scheduler] Acquire new execut...

2018-11-06 Thread tgravescs
Github user tgravescs commented on the issue: https://github.com/apache/spark/pull/22288 merged to master and 2.4 branch, thanks @dhruve --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org

[GitHub] spark issue #22288: [SPARK-22148][SPARK-15815][Scheduler] Acquire new execut...

2018-11-06 Thread tgravescs
Github user tgravescs commented on the issue: https://github.com/apache/spark/pull/22288 +1 --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h

[GitHub] spark pull request #22288: [SPARK-22148][SPARK-15815][Scheduler] Acquire new...

2018-11-02 Thread tgravescs
Github user tgravescs commented on a diff in the pull request: https://github.com/apache/spark/pull/22288#discussion_r230487066 --- Diff: core/src/test/scala/org/apache/spark/scheduler/TaskSchedulerImplSuite.scala --- @@ -503,6 +507,181 @@ class TaskSchedulerImplSuite extends

[GitHub] spark issue #22852: [SPARK-25023] Clarify Spark security documentation

2018-11-02 Thread tgravescs
Github user tgravescs commented on the issue: https://github.com/apache/spark/pull/22852 merged to master and 2.4 --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail

[GitHub] spark issue #21688: [SPARK-21809] : Change Stage Page to use datatables to s...

2018-11-01 Thread tgravescs
Github user tgravescs commented on the issue: https://github.com/apache/spark/pull/21688 test this please --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h

[GitHub] spark issue #22288: [SPARK-22148][SPARK-15815][Scheduler] Acquire new execut...

2018-10-31 Thread tgravescs
Github user tgravescs commented on the issue: https://github.com/apache/spark/pull/22288 @dhruve is this ready to review again? --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark issue #22852: [SPARK-25023] Clarify Spark security documentation

2018-10-31 Thread tgravescs
Github user tgravescs commented on the issue: https://github.com/apache/spark/pull/22852 If no other comments, I'll commit this? I'll leave it open for a bit longer --- - To unsubscribe, e-mail: reviews-unsubscr

[GitHub] spark pull request #21688: [SPARK-21809] : Change Stage Page to use datatabl...

2018-10-30 Thread tgravescs
Github user tgravescs commented on a diff in the pull request: https://github.com/apache/spark/pull/21688#discussion_r229363078 --- Diff: core/src/main/resources/org/apache/spark/ui/static/stagepage.js --- @@ -0,0 +1,959 @@ +/* + * Licensed to the Apache Software

[GitHub] spark issue #22852: [SPARK-25023] Clarify Spark security documentation

2018-10-30 Thread tgravescs
Github user tgravescs commented on the issue: https://github.com/apache/spark/pull/22852 added sections to the resource manager sections. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org

[GitHub] spark issue #22852: [SPARK-25023] Clarify Spark security documentation

2018-10-30 Thread tgravescs
Github user tgravescs commented on the issue: https://github.com/apache/spark/pull/22852 I can add a note here for deployments here and then we can do version specific ones after --- - To unsubscribe, e-mail

[GitHub] spark issue #22852: [SPARK-25023] Clarify Spark security documentation

2018-10-30 Thread tgravescs
Github user tgravescs commented on the issue: https://github.com/apache/spark/pull/22852 Updated to have a section on security in the quickstart and overview, let me know what you think and if wording needs updated. If this ok I can followup with something on the website

[GitHub] spark pull request #21688: [SPARK-21809] : Change Stage Page to use datatabl...

2018-10-29 Thread tgravescs
Github user tgravescs commented on a diff in the pull request: https://github.com/apache/spark/pull/21688#discussion_r229002442 --- Diff: core/src/main/resources/org/apache/spark/ui/static/stagepage.js --- @@ -0,0 +1,944 @@ +/* + * Licensed to the Apache Software

[GitHub] spark pull request #21688: [SPARK-21809] : Change Stage Page to use datatabl...

2018-10-29 Thread tgravescs
Github user tgravescs commented on a diff in the pull request: https://github.com/apache/spark/pull/21688#discussion_r228994601 --- Diff: core/src/main/resources/org/apache/spark/ui/static/stagepage.js --- @@ -0,0 +1,944 @@ +/* + * Licensed to the Apache Software

[GitHub] spark pull request #21688: [SPARK-21809] : Change Stage Page to use datatabl...

2018-10-29 Thread tgravescs
Github user tgravescs commented on a diff in the pull request: https://github.com/apache/spark/pull/21688#discussion_r228982925 --- Diff: core/src/main/resources/org/apache/spark/ui/static/stagepage.js --- @@ -0,0 +1,944 @@ +/* + * Licensed to the Apache Software

[GitHub] spark issue #22852: [SPARK-25023] Clarify Spark security documentation

2018-10-29 Thread tgravescs
Github user tgravescs commented on the issue: https://github.com/apache/spark/pull/22852 I would be fine with adding it more places, including perhaps the overview page: http://spark.apache.org/docs/latest/ and quick start pages. Perhaps we should agree upon the wording here

[GitHub] spark issue #21588: [SPARK-24590][BUILD] Make Jenkins tests passed with hado...

2018-10-29 Thread tgravescs
Github user tgravescs commented on the issue: https://github.com/apache/spark/pull/21588 Can you clarify what you mean by drop builtin metastore support? Are you just saying users must always provide jars to use it or something more

[GitHub] spark pull request #22288: [SPARK-22148][SPARK-15815][Scheduler] Acquire new...

2018-10-26 Thread tgravescs
Github user tgravescs commented on a diff in the pull request: https://github.com/apache/spark/pull/22288#discussion_r228678209 --- Diff: core/src/main/scala/org/apache/spark/scheduler/TaskSchedulerImpl.scala --- @@ -415,9 +420,54 @@ private[spark] class TaskSchedulerImpl

[GitHub] spark issue #22852: [SPARK-25023] Clarify Spark security documentation

2018-10-26 Thread tgravescs
Github user tgravescs commented on the issue: https://github.com/apache/spark/pull/22852 I would rather see someone more familiar with K8s that uses it document it. --- - To unsubscribe, e-mail: reviews-unsubscr

[GitHub] spark issue #22852: [SPARK-25023] Clarify Spark security documentation

2018-10-26 Thread tgravescs
Github user tgravescs commented on the issue: https://github.com/apache/spark/pull/22852 so I had filed a jira to update mesos docs more detail about security things (https://issues.apache.org/jira/browse/SPARK-25024) which I need to follow up on, but I didn't file one for k8s

[GitHub] spark pull request #22288: [SPARK-22148][SPARK-15815][Scheduler] Acquire new...

2018-10-26 Thread tgravescs
Github user tgravescs commented on a diff in the pull request: https://github.com/apache/spark/pull/22288#discussion_r228668756 --- Diff: core/src/main/scala/org/apache/spark/scheduler/TaskSchedulerImpl.scala --- @@ -415,9 +420,54 @@ private[spark] class TaskSchedulerImpl

[GitHub] spark issue #22852: [SPARK-25023] Clarify Spark security documentation

2018-10-26 Thread tgravescs
Github user tgravescs commented on the issue: https://github.com/apache/spark/pull/22852 we can add stronger wording for standalone if you want, I know the text was recently updated (I believe by you) to have the below: > For other resource manag

[GitHub] spark issue #22852: [SPARK-25023] Clarify Spark security documentation

2018-10-26 Thread tgravescs
Github user tgravescs commented on the issue: https://github.com/apache/spark/pull/22852 the intention is not a we told you so, its meant to grab their attention and to get people to think about it because in the end it is their responsibility in my opinion. I'm fine if you

[GitHub] spark pull request #22852: [SPARK-25023] Clarify Spark security documentatio...

2018-10-26 Thread tgravescs
Github user tgravescs commented on a diff in the pull request: https://github.com/apache/spark/pull/22852#discussion_r228544730 --- Diff: docs/security.md --- @@ -6,7 +6,20 @@ title: Security * This will become a table of contents (this text will be scraped). {:toc

[GitHub] spark issue #22852: [SPARK-25023] Clarify Spark security documentation

2018-10-26 Thread tgravescs
Github user tgravescs commented on the issue: https://github.com/apache/spark/pull/22852 @vanzin @srowen --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h

[GitHub] spark pull request #22852: [SPARK-25023] Clarify Spark security documentatio...

2018-10-26 Thread tgravescs
GitHub user tgravescs opened a pull request: https://github.com/apache/spark/pull/22852 [SPARK-25023] Clarify Spark security documentation ## What changes were proposed in this pull request? Clarify documentation about security. ## How was this patch tested

[GitHub] spark issue #22771: [SPARK-25773][Core]Cancel zombie tasks in a result stage...

2018-10-25 Thread tgravescs
Github user tgravescs commented on the issue: https://github.com/apache/spark/pull/22771 no other comments, looks good. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark issue #22144: [SPARK-24935][SQL] : Problem with Executing Hive UDF's f...

2018-10-24 Thread tgravescs
Github user tgravescs commented on the issue: https://github.com/apache/spark/pull/22144 @cloud-fan I agree with you that most issues do not have to block a release but I don't agree with most of your criteria for making that decision. I've already talked about the other

[GitHub] spark issue #22144: [SPARK-24935][SQL] : Problem with Executing Hive UDF's f...

2018-10-24 Thread tgravescs
Github user tgravescs commented on the issue: https://github.com/apache/spark/pull/22144 while I'm ok with not blocking 2.4 for this as well, not for many of the reasons stated though. Note the jira was filed a Major not a blocker. Based on the information we have, the impact

[GitHub] spark pull request #22771: [SPARK-25773][Core]Cancel zombie tasks in a resul...

2018-10-23 Thread tgravescs
Github user tgravescs commented on a diff in the pull request: https://github.com/apache/spark/pull/22771#discussion_r227503789 --- Diff: core/src/main/scala/org/apache/spark/scheduler/DAGScheduler.scala --- @@ -1364,6 +1385,19 @@ private[spark] class DAGScheduler

[GitHub] spark issue #22144: [SPARK-24935][SQL] : Problem with Executing Hive UDF's f...

2018-10-23 Thread tgravescs
Github user tgravescs commented on the issue: https://github.com/apache/spark/pull/22144 The fact that it isn't a new regression shouldn't determine if its a blocker, there are lots of things you don't hit right away. The impact does affect if we decide if this is a blocker

[GitHub] spark issue #22771: [SPARK-25773][Core]Cancel zombie tasks in a result stage...

2018-10-23 Thread tgravescs
Github user tgravescs commented on the issue: https://github.com/apache/spark/pull/22771 @markhamstra thanks for the reference, I was looking for some background on this. I agree those are still issues like mentioned in SPARK-17064 but I don't think that directly impacts this. We

[GitHub] spark issue #22144: [SPARK-24935][SQL] : Problem with Executing Hive UDF's f...

2018-10-23 Thread tgravescs
Github user tgravescs commented on the issue: https://github.com/apache/spark/pull/22144 @cloud-fan I disagree, it seems like you broke functionality that was there and you can't do that in a minor release. 3.0 would be fine to drop I think but we should fix it for 2.4, this PR

[GitHub] spark pull request #21688: [SPARK-21809] : Change Stage Page to use datatabl...

2018-10-22 Thread tgravescs
Github user tgravescs commented on a diff in the pull request: https://github.com/apache/spark/pull/21688#discussion_r227117575 --- Diff: core/src/main/scala/org/apache/spark/status/AppStatusStore.scala --- @@ -547,6 +519,24 @@ private[spark] class AppStatusStore

[GitHub] spark pull request #21688: [SPARK-21809] : Change Stage Page to use datatabl...

2018-10-22 Thread tgravescs
Github user tgravescs commented on a diff in the pull request: https://github.com/apache/spark/pull/21688#discussion_r227116341 --- Diff: core/src/main/resources/org/apache/spark/ui/static/stagepage.js --- @@ -52,22 +52,20 @@ $.extend( $.fn.dataTable.ext.type.order

[GitHub] spark pull request #21688: [SPARK-21809] : Change Stage Page to use datatabl...

2018-10-22 Thread tgravescs
Github user tgravescs commented on a diff in the pull request: https://github.com/apache/spark/pull/21688#discussion_r227116251 --- Diff: core/src/main/resources/org/apache/spark/ui/static/stagepage.js --- @@ -52,22 +52,20 @@ $.extend( $.fn.dataTable.ext.type.order

[GitHub] spark pull request #22504: [SPARK-25118][Submit] Persist Driver Logs in Clie...

2018-10-22 Thread tgravescs
Github user tgravescs commented on a diff in the pull request: https://github.com/apache/spark/pull/22504#discussion_r227036915 --- Diff: docs/configuration.md --- @@ -266,6 +266,41 @@ of the most common options to set are: Only has effect in Spark standalone mode or Mesos

[GitHub] spark pull request #22288: [SPARK-22148][SPARK-15815][Scheduler] Acquire new...

2018-10-22 Thread tgravescs
Github user tgravescs commented on a diff in the pull request: https://github.com/apache/spark/pull/22288#discussion_r226999715 --- Diff: core/src/main/scala/org/apache/spark/scheduler/TaskSchedulerImpl.scala --- @@ -415,9 +420,54 @@ private[spark] class TaskSchedulerImpl

[GitHub] spark pull request #22288: [SPARK-22148][SPARK-15815][Scheduler] Acquire new...

2018-10-22 Thread tgravescs
Github user tgravescs commented on a diff in the pull request: https://github.com/apache/spark/pull/22288#discussion_r227005479 --- Diff: core/src/test/scala/org/apache/spark/scheduler/TaskSchedulerImplSuite.scala --- @@ -503,6 +505,145 @@ class TaskSchedulerImplSuite extends

[GitHub] spark pull request #22288: [SPARK-22148][SPARK-15815][Scheduler] Acquire new...

2018-10-22 Thread tgravescs
Github user tgravescs commented on a diff in the pull request: https://github.com/apache/spark/pull/22288#discussion_r226999389 --- Diff: core/src/main/scala/org/apache/spark/scheduler/TaskSchedulerImpl.scala --- @@ -453,6 +503,22 @@ private[spark] class TaskSchedulerImpl

[GitHub] spark pull request #22288: [SPARK-22148][SPARK-15815][Scheduler] Acquire new...

2018-10-22 Thread tgravescs
Github user tgravescs commented on a diff in the pull request: https://github.com/apache/spark/pull/22288#discussion_r227000658 --- Diff: core/src/main/scala/org/apache/spark/scheduler/TaskSchedulerImpl.scala --- @@ -415,9 +420,54 @@ private[spark] class TaskSchedulerImpl

[GitHub] spark issue #22288: [SPARK-22148][SPARK-15815][Scheduler] Acquire new execut...

2018-10-22 Thread tgravescs
Github user tgravescs commented on the issue: https://github.com/apache/spark/pull/22288 test this please --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h

[GitHub] spark issue #22725: [SPARK-25753][CORE]fix reading small files via BinaryFil...

2018-10-22 Thread tgravescs
Github user tgravescs commented on the issue: https://github.com/apache/spark/pull/22725 merged to master --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h

[GitHub] spark issue #22771: [SPARK-25773][Core]Cancel zombie tasks in a result stage...

2018-10-19 Thread tgravescs
Github user tgravescs commented on the issue: https://github.com/apache/spark/pull/22771 Actually as part of the jiras I mentioned above we were looking at killing other task attempts as soon as one task attempt succeeds rather then waiting for the entire job to finish, thoughts

[GitHub] spark pull request #22771: [SPARK-25773][Core]Cancel zombie tasks in a resul...

2018-10-19 Thread tgravescs
Github user tgravescs commented on a diff in the pull request: https://github.com/apache/spark/pull/22771#discussion_r226651308 --- Diff: core/src/main/scala/org/apache/spark/scheduler/DAGScheduler.scala --- @@ -1364,6 +1385,16 @@ private[spark] class DAGScheduler

[GitHub] spark issue #22771: [SPARK-25773][Core]Cancel zombie tasks in a result stage...

2018-10-19 Thread tgravescs
Github user tgravescs commented on the issue: https://github.com/apache/spark/pull/22771 the change itself seems reasonable we were actually looking at this recently as there are a couple of others bugs around this: https://issues.apache.org/jira/browse/SPARK-25250 and https

[GitHub] spark pull request #22288: [SPARK-22148][SPARK-15815][Scheduler] Acquire new...

2018-10-17 Thread tgravescs
Github user tgravescs commented on a diff in the pull request: https://github.com/apache/spark/pull/22288#discussion_r225998426 --- Diff: core/src/main/scala/org/apache/spark/scheduler/TaskSchedulerImpl.scala --- @@ -415,9 +420,65 @@ private[spark] class TaskSchedulerImpl

[GitHub] spark issue #22725: [SPARK-25753][[CORE][FOLLOW-UP]fix reading small files v...

2018-10-17 Thread tgravescs
Github user tgravescs commented on the issue: https://github.com/apache/spark/pull/22725 +1 Looks good, thanks @10110346 --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e

[GitHub] spark issue #22725: [SPARK-24610][[CORE][FOLLOW-UP]fix reading small files v...

2018-10-16 Thread tgravescs
Github user tgravescs commented on the issue: https://github.com/apache/spark/pull/22725 SPARK-24610 is the original issue, please file a new jira for StreamFileInputFormat --- - To unsubscribe, e-mail: reviews

[GitHub] spark pull request #22288: [SPARK-22148][SPARK-15815][Scheduler] Acquire new...

2018-10-15 Thread tgravescs
Github user tgravescs commented on a diff in the pull request: https://github.com/apache/spark/pull/22288#discussion_r225198423 --- Diff: core/src/test/scala/org/apache/spark/scheduler/TaskSchedulerImplSuite.scala --- @@ -503,6 +505,89 @@ class TaskSchedulerImplSuite extends

[GitHub] spark pull request #22288: [SPARK-22148][SPARK-15815][Scheduler] Acquire new...

2018-10-15 Thread tgravescs
Github user tgravescs commented on a diff in the pull request: https://github.com/apache/spark/pull/22288#discussion_r225193634 --- Diff: core/src/main/scala/org/apache/spark/scheduler/TaskSchedulerImpl.scala --- @@ -415,9 +419,61 @@ private[spark] class TaskSchedulerImpl

[GitHub] spark pull request #22288: [SPARK-22148][SPARK-15815][Scheduler] Acquire new...

2018-10-15 Thread tgravescs
Github user tgravescs commented on a diff in the pull request: https://github.com/apache/spark/pull/22288#discussion_r225185389 --- Diff: core/src/main/scala/org/apache/spark/internal/config/package.scala --- @@ -597,6 +597,16 @@ package object config { .checkValue(v

[GitHub] spark pull request #21688: [SPARK-21809] : Change Stage Page to use datatabl...

2018-10-12 Thread tgravescs
Github user tgravescs commented on a diff in the pull request: https://github.com/apache/spark/pull/21688#discussion_r224864545 --- Diff: core/src/main/scala/org/apache/spark/status/LiveEntity.scala --- @@ -341,7 +341,9 @@ private class LiveExecutorStageSummary

[GitHub] spark pull request #21688: [SPARK-21809] : Change Stage Page to use datatabl...

2018-10-12 Thread tgravescs
Github user tgravescs commented on a diff in the pull request: https://github.com/apache/spark/pull/21688#discussion_r224850397 --- Diff: core/src/main/scala/org/apache/spark/status/LiveEntity.scala --- @@ -341,7 +341,9 @@ private class LiveExecutorStageSummary

[GitHub] spark pull request #21688: [SPARK-21809] : Change Stage Page to use datatabl...

2018-10-10 Thread tgravescs
Github user tgravescs commented on a diff in the pull request: https://github.com/apache/spark/pull/21688#discussion_r224251226 --- Diff: core/src/main/scala/org/apache/spark/ui/jobs/StagePage.scala --- @@ -278,7 +198,7 @@ private[ui] class StagePage(parent: StagesTab, store

[GitHub] spark pull request #21688: [SPARK-21809] : Change Stage Page to use datatabl...

2018-10-10 Thread tgravescs
Github user tgravescs commented on a diff in the pull request: https://github.com/apache/spark/pull/21688#discussion_r224239984 --- Diff: core/src/main/scala/org/apache/spark/status/api/v1/StagesResource.scala --- @@ -102,4 +103,124 @@ private[v1] class StagesResource extends

[GitHub] spark pull request #21688: [SPARK-21809] : Change Stage Page to use datatabl...

2018-10-10 Thread tgravescs
Github user tgravescs commented on a diff in the pull request: https://github.com/apache/spark/pull/21688#discussion_r224238974 --- Diff: core/src/main/scala/org/apache/spark/status/LiveEntity.scala --- @@ -341,7 +341,9 @@ private class LiveExecutorStageSummary

[GitHub] spark pull request #21688: [SPARK-21809] : Change Stage Page to use datatabl...

2018-10-10 Thread tgravescs
Github user tgravescs commented on a diff in the pull request: https://github.com/apache/spark/pull/21688#discussion_r224221297 --- Diff: core/src/main/scala/org/apache/spark/status/AppStatusStore.scala --- @@ -349,7 +349,23 @@ private[spark] class AppStatusStore( def

[GitHub] spark pull request #21688: [SPARK-21809] : Change Stage Page to use datatabl...

2018-10-10 Thread tgravescs
Github user tgravescs commented on a diff in the pull request: https://github.com/apache/spark/pull/21688#discussion_r224217796 --- Diff: core/src/main/resources/org/apache/spark/ui/static/utils.js --- @@ -74,3 +74,99 @@ function getTimeZone() { return new Date().toString

[GitHub] spark pull request #21688: [SPARK-21809] : Change Stage Page to use datatabl...

2018-10-10 Thread tgravescs
Github user tgravescs commented on a diff in the pull request: https://github.com/apache/spark/pull/21688#discussion_r224178567 --- Diff: core/src/main/resources/org/apache/spark/ui/static/stagepage.js --- @@ -0,0 +1,872 @@ +/* + * Licensed to the Apache Software

[GitHub] spark pull request #21688: [SPARK-21809] : Change Stage Page to use datatabl...

2018-10-10 Thread tgravescs
Github user tgravescs commented on a diff in the pull request: https://github.com/apache/spark/pull/21688#discussion_r224176616 --- Diff: core/src/main/resources/org/apache/spark/ui/static/stagespage-template.html --- @@ -0,0 +1,124 @@ + + +

[GitHub] spark pull request #21688: [SPARK-21809] : Change Stage Page to use datatabl...

2018-10-10 Thread tgravescs
Github user tgravescs commented on a diff in the pull request: https://github.com/apache/spark/pull/21688#discussion_r224166110 --- Diff: core/src/main/resources/org/apache/spark/ui/static/stagepage.js --- @@ -0,0 +1,872 @@ +/* + * Licensed to the Apache Software

[GitHub] spark pull request #21688: [SPARK-21809] : Change Stage Page to use datatabl...

2018-10-10 Thread tgravescs
Github user tgravescs commented on a diff in the pull request: https://github.com/apache/spark/pull/21688#discussion_r224164134 --- Diff: core/src/main/resources/org/apache/spark/ui/static/stagepage.js --- @@ -0,0 +1,872 @@ +/* + * Licensed to the Apache Software

[GitHub] spark pull request #21688: [SPARK-21809] : Change Stage Page to use datatabl...

2018-10-10 Thread tgravescs
Github user tgravescs commented on a diff in the pull request: https://github.com/apache/spark/pull/21688#discussion_r224163278 --- Diff: core/src/main/resources/org/apache/spark/ui/static/stagepage.js --- @@ -0,0 +1,872 @@ +/* + * Licensed to the Apache Software

[GitHub] spark pull request #21688: [SPARK-21809] : Change Stage Page to use datatabl...

2018-10-10 Thread tgravescs
Github user tgravescs commented on a diff in the pull request: https://github.com/apache/spark/pull/21688#discussion_r224162695 --- Diff: core/src/main/resources/org/apache/spark/ui/static/stagepage.js --- @@ -0,0 +1,872 @@ +/* + * Licensed to the Apache Software

[GitHub] spark pull request #21688: [SPARK-21809] : Change Stage Page to use datatabl...

2018-10-10 Thread tgravescs
Github user tgravescs commented on a diff in the pull request: https://github.com/apache/spark/pull/21688#discussion_r224162463 --- Diff: core/src/main/resources/org/apache/spark/ui/static/stagepage.js --- @@ -0,0 +1,872 @@ +/* + * Licensed to the Apache Software

  1   2   3   4   5   6   7   8   9   10   >